2024-09-10T18:03:58.927 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-10T18:03:58.944 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T18:03:59.030 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900206 branch: wip-guits-quincy-2024-09-09-1810-quincy description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} email: gabrioux@ibm.com first_in_suite: false job_id: '7900206' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-quincy-2024-09-09-1810-quincy ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) mon_bind_msgr2: false sha1: 5be57395ea235535910512ae73ba8e7ff25fe588 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 5be57395ea235535910512ae73ba8e7ff25fe588 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: quincy-orch-centos-9-teuth sha1: fad5356a94e6044ce1062d748daa5a0f2ed53a7d owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 4625 sha1: 5be57395ea235535910512ae73ba8e7ff25fe588 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: quincy-orch-centos-9-teuth suite_path: /home/teuthworker/src/github.com_adk3798_ceph_fad5356a94e6044ce1062d748daa5a0f2ed53a7d/qa suite_relpath: qa suite_repo: https://github.com/adk3798/ceph.git suite_sha1: fad5356a94e6044ce1062d748daa5a0f2ed53a7d targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHbH2tpVC0ae1misy/DD4ysRcKKOlrsu9nVL9aNxft2FP5gOn287QYxrWEOD2v2qmsKv0OpMoqSeLviwduizJHk= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKL7JwH6XjgEnKVzflClF1zytSvzDWT87kg+knARleyaTPX5D9FZg/AxFftX59+DKFGTVt7c5fGVRy+xBV8EU5I= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install nvmetcli nvme-cli -y - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_17:53:37 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-10T18:03:59.031 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_adk3798_ceph_fad5356a94e6044ce1062d748daa5a0f2ed53a7d/qa; will attempt to use it 2024-09-10T18:03:59.031 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_adk3798_ceph_fad5356a94e6044ce1062d748daa5a0f2ed53a7d/qa/tasks 2024-09-10T18:03:59.032 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-10T18:03:59.033 INFO:teuthology.task.internal:Checking packages... 2024-09-10T18:03:59.054 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5be57395ea235535910512ae73ba8e7ff25fe588' 2024-09-10T18:03:59.054 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T18:03:59.054 INFO:teuthology.packaging:ref: None 2024-09-10T18:03:59.054 INFO:teuthology.packaging:tag: None 2024-09-10T18:03:59.054 INFO:teuthology.packaging:branch: wip-guits-quincy-2024-09-09-1810-quincy 2024-09-10T18:03:59.054 INFO:teuthology.packaging:sha1: 5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:03:59.055 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-quincy-2024-09-09-1810-quincy 2024-09-10T18:03:59.198 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1606.g5be57395 2024-09-10T18:03:59.200 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-10T18:03:59.207 INFO:teuthology.task.internal:no buildpackages task found 2024-09-10T18:03:59.207 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-10T18:03:59.214 INFO:teuthology.task.internal:Saving configuration 2024-09-10T18:03:59.226 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-10T18:03:59.233 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-10T18:03:59.258 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900206', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 17:58:45.973357', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHbH2tpVC0ae1misy/DD4ysRcKKOlrsu9nVL9aNxft2FP5gOn287QYxrWEOD2v2qmsKv0OpMoqSeLviwduizJHk='} 2024-09-10T18:03:59.280 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900206', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 17:58:45.975346', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKL7JwH6XjgEnKVzflClF1zytSvzDWT87kg+knARleyaTPX5D9FZg/AxFftX59+DKFGTVt7c5fGVRy+xBV8EU5I='} 2024-09-10T18:03:59.280 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-10T18:03:59.287 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-10T18:03:59.287 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-10T18:03:59.287 INFO:teuthology.run_tasks:Running task console_log... 2024-09-10T18:03:59.340 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4e6cfe7ac0>, signals=[15]) 2024-09-10T18:03:59.340 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-10T18:03:59.346 INFO:teuthology.task.internal:Opening connections... 2024-09-10T18:03:59.346 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T18:03:59.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:03:59.422 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-09-10T18:03:59.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:03:59.497 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-10T18:03:59.504 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-09-10T18:03:59.523 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-09-10T18:03:59.523 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-09-10T18:03:59.580 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-09-10T18:03:59.580 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-09-10T18:03:59.580 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-09-10T18:03:59.580 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-09-10T18:03:59.580 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-09-10T18:03:59.580 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-09-10T18:03:59.580 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T18:03:59.580 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-09-10T18:03:59.581 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-09-10T18:03:59.581 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T18:03:59.581 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-09-10T18:03:59.581 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T18:03:59.581 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T18:03:59.581 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T18:03:59.582 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-09-10T18:03:59.601 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-09-10T18:03:59.619 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-09-10T18:03:59.620 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-09-10T18:03:59.678 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="CentOS Stream" 2024-09-10T18:03:59.678 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="9" 2024-09-10T18:03:59.678 INFO:teuthology.orchestra.run.smithi204.stdout:ID="centos" 2024-09-10T18:03:59.678 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE="rhel fedora" 2024-09-10T18:03:59.678 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="9" 2024-09-10T18:03:59.678 INFO:teuthology.orchestra.run.smithi204.stdout:PLATFORM_ID="platform:el9" 2024-09-10T18:03:59.679 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T18:03:59.679 INFO:teuthology.orchestra.run.smithi204.stdout:ANSI_COLOR="0;31" 2024-09-10T18:03:59.679 INFO:teuthology.orchestra.run.smithi204.stdout:LOGO="fedora-logo-icon" 2024-09-10T18:03:59.679 INFO:teuthology.orchestra.run.smithi204.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T18:03:59.679 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://centos.org/" 2024-09-10T18:03:59.679 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T18:03:59.679 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T18:03:59.679 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T18:03:59.679 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-09-10T18:03:59.700 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-10T18:03:59.709 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-10T18:03:59.715 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-10T18:03:59.715 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-09-10T18:03:59.718 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-09-10T18:03:59.735 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-10T18:03:59.741 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-10T18:03:59.741 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-09-10T18:03:59.775 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-09-10T18:03:59.866 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-10T18:03:59.904 INFO:teuthology.run_tasks:Running task kernel... 2024-09-10T18:03:59.918 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-10T18:03:59.918 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-10T18:03:59.919 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T18:03:59.919 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T18:03:59.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:03:59.936 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-09-10T18:03:59.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:03:59.937 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-09-10T18:03:59.994 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-503.el9.x86_64 2024-09-10T18:03:59.995 INFO:teuthology.task.kernel:Running kernel on smithi028: 5.14.0-503.el9.x86_64 2024-09-10T18:03:59.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum install -y kernel 2024-09-10T18:03:59.996 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-503.el9.x86_64 2024-09-10T18:03:59.996 INFO:teuthology.task.kernel:Running kernel on smithi204: 5.14.0-503.el9.x86_64 2024-09-10T18:03:59.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum install -y kernel 2024-09-10T18:04:03.545 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.2 MB 00:01 2024-09-10T18:04:04.100 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 3.5 MB/s | 8.2 MB 00:02 2024-09-10T18:04:06.126 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-10T18:04:09.587 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 4.2 MB/s | 20 MB 00:04 2024-09-10T18:04:12.501 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.5 MB 00:02 2024-09-10T18:04:14.032 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-10T18:04:14.319 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 19 kB 00:00 2024-09-10T18:04:15.123 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 33 MB/s | 23 MB 00:00 2024-09-10T18:04:15.930 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 19 kB 00:00 2024-09-10T18:04:17.226 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-10T18:04:20.545 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-10T18:04:22.644 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T18:04:22.644 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T18:04:22.730 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:04:22.736 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:04:22.736 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-09-10T18:04:22.736 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:04:22.736 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:04:22.736 INFO:teuthology.orchestra.run.smithi204.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T18:04:22.736 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-10T18:04:22.736 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T18:04:22.736 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T18:04:22.737 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T18:04:22.737 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:04:22.737 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:04:22.737 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:04:22.737 INFO:teuthology.orchestra.run.smithi204.stdout:Install 4 Packages 2024-09-10T18:04:22.737 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:04:22.737 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 79 M 2024-09-10T18:04:22.738 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 126 M 2024-09-10T18:04:22.738 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:04:22.823 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-10T18:04:23.025 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:01 ago on Tue 10 Sep 2024 06:04:22 PM UTC. 2024-09-10T18:04:23.105 INFO:teuthology.orchestra.run.smithi204.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 199 kB/s | 23 kB 00:00 2024-09-10T18:04:23.506 INFO:teuthology.orchestra.run.smithi204.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-09-10T18:04:23.831 INFO:teuthology.orchestra.run.smithi204.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 41 MB/s | 35 MB 00:00 2024-09-10T18:04:24.106 INFO:teuthology.orchestra.run.smithi204.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 29 MB/s | 29 MB 00:01 2024-09-10T18:04:24.107 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:04:24.107 INFO:teuthology.orchestra.run.smithi204.stdout:Total 58 MB/s | 79 MB 00:01 2024-09-10T18:04:24.789 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:04:24.913 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T18:04:24.913 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T18:04:25.009 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:04:25.014 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:04:25.015 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-09-10T18:04:25.015 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:04:25.015 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:04:25.015 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T18:04:25.015 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-10T18:04:25.015 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T18:04:25.015 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T18:04:25.015 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T18:04:25.015 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:04:25.015 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:04:25.015 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:04:25.016 INFO:teuthology.orchestra.run.smithi028.stdout:Install 4 Packages 2024-09-10T18:04:25.016 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:04:25.016 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 79 M 2024-09-10T18:04:25.016 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 126 M 2024-09-10T18:04:25.016 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:04:25.148 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:04:25.149 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:04:25.383 INFO:teuthology.orchestra.run.smithi028.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 212 kB/s | 23 kB 00:00 2024-09-10T18:04:26.124 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:04:26.125 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:04:27.983 INFO:teuthology.orchestra.run.smithi028.stdout:(2/4): kernel-modules-core-5.14.0-505.el9.x86_6 11 MB/s | 29 MB 00:02 2024-09-10T18:04:28.233 INFO:teuthology.orchestra.run.smithi028.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 5.3 MB/s | 16 MB 00:02 2024-09-10T18:04:28.584 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:04:28.725 INFO:teuthology.orchestra.run.smithi028.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 10 MB/s | 35 MB 00:03 2024-09-10T18:04:28.725 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:04:28.725 INFO:teuthology.orchestra.run.smithi028.stdout:Total 21 MB/s | 79 MB 00:03 2024-09-10T18:04:29.251 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T18:04:29.341 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:04:29.435 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:04:29.830 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:04:29.831 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:04:30.841 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:04:30.842 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:04:32.135 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:04:32.262 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:04:33.416 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:04:34.085 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T18:04:34.167 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:04:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:04:37.086 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:04:37.093 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:04:37.130 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:04:41.975 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:04:42.207 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:04:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:04:47.391 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:05:20.181 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:05:20.220 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:05:20.489 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:05:20.489 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T18:05:20.490 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:05:20.490 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:05:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:05:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:05:21.299 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:05:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T18:05:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T18:05:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T18:05:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T18:05:21.300 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:05:21.300 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:05:21.659 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T18:05:21.743 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T18:05:21.743 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T18:05:21.743 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T18:05:21.743 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T18:05:21.743 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T18:05:21.744 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-10T18:05:21.744 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi204.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T18:05:21.744 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T18:05:21.803 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T18:05:21.803 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T18:05:21.803 DEBUG:teuthology.orchestra.run.smithi204:> sudo rpm -qi grub2-tools 2024-09-10T18:05:21.842 INFO:teuthology.orchestra.run.smithi204.stdout:Name : grub2-tools 2024-09-10T18:05:21.843 INFO:teuthology.orchestra.run.smithi204.stdout:Epoch : 1 2024-09-10T18:05:21.843 INFO:teuthology.orchestra.run.smithi204.stdout:Version : 2.06 2024-09-10T18:05:21.843 INFO:teuthology.orchestra.run.smithi204.stdout:Release : 82.el9 2024-09-10T18:05:21.843 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: x86_64 2024-09-10T18:05:21.843 INFO:teuthology.orchestra.run.smithi204.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T18:05:21.843 INFO:teuthology.orchestra.run.smithi204.stdout:Group : Unspecified 2024-09-10T18:05:21.843 INFO:teuthology.orchestra.run.smithi204.stdout:Size : 8274670 2024-09-10T18:05:21.843 INFO:teuthology.orchestra.run.smithi204.stdout:License : GPLv3+ 2024-09-10T18:05:21.844 INFO:teuthology.orchestra.run.smithi204.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T18:05:21.844 INFO:teuthology.orchestra.run.smithi204.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T18:05:21.844 INFO:teuthology.orchestra.run.smithi204.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T18:05:21.844 INFO:teuthology.orchestra.run.smithi204.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T18:05:21.844 INFO:teuthology.orchestra.run.smithi204.stdout:Packager : builder@centos.org 2024-09-10T18:05:21.844 INFO:teuthology.orchestra.run.smithi204.stdout:Vendor : CentOS 2024-09-10T18:05:21.844 INFO:teuthology.orchestra.run.smithi204.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T18:05:21.844 INFO:teuthology.orchestra.run.smithi204.stdout:Summary : Support tools for GRUB. 2024-09-10T18:05:21.844 INFO:teuthology.orchestra.run.smithi204.stdout:Description : 2024-09-10T18:05:21.844 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:05:21.845 INFO:teuthology.orchestra.run.smithi204.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T18:05:21.845 INFO:teuthology.orchestra.run.smithi204.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T18:05:21.845 INFO:teuthology.orchestra.run.smithi204.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T18:05:21.845 INFO:teuthology.orchestra.run.smithi204.stdout:hardware devices. 2024-09-10T18:05:21.845 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:05:21.845 INFO:teuthology.orchestra.run.smithi204.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T18:05:21.846 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T18:05:21.846 INFO:teuthology.task.kernel:Updating grub on smithi204 to boot 5.14.0-505.el9.x86_64 2024-09-10T18:05:21.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T18:05:22.475 INFO:teuthology.orchestra.run.smithi204.stderr:Generating grub configuration file ... 2024-09-10T18:05:23.743 INFO:teuthology.orchestra.run.smithi204.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T18:05:23.773 INFO:teuthology.orchestra.run.smithi204.stderr:done 2024-09-10T18:05:23.775 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-09-10T18:05:23.791 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.rW94FW9PcJ 2024-09-10T18:05:23.791 DEBUG:teuthology.orchestra.run.smithi204:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rW94FW9PcJ 2024-09-10T18:05:23.858 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.rW94FW9PcJ 2024-09-10T18:05:24.000 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.rW94FW9PcJ is 6KB 2024-09-10T18:05:24.048 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.rW94FW9PcJ 2024-09-10T18:05:24.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T18:05:24.126 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T18:05:24.126 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T18:05:24.126 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T18:05:24.126 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T18:05:24.127 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T18:05:24.299 DEBUG:teuthology.orchestra.run.smithi204:> sudo shutdown -r now 2024-09-10T18:05:26.183 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:05:26.223 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:05:26.577 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:05:26.578 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T18:05:26.578 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:05:26.578 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:05:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:05:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:05:27.461 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:05:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T18:05:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T18:05:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T18:05:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T18:05:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:05:27.461 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:05:27.802 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T18:05:27.889 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T18:05:27.889 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T18:05:27.889 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T18:05:27.889 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T18:05:27.889 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T18:05:27.889 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-10T18:05:27.890 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T18:05:27.890 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T18:05:27.948 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T18:05:27.948 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T18:05:27.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-09-10T18:05:27.989 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-09-10T18:05:27.989 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-09-10T18:05:27.989 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-09-10T18:05:27.989 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 82.el9 2024-09-10T18:05:27.989 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-09-10T18:05:27.990 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T18:05:27.990 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-09-10T18:05:27.990 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8274670 2024-09-10T18:05:27.990 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-09-10T18:05:27.990 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T18:05:27.990 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T18:05:27.990 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T18:05:27.990 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T18:05:27.990 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-09-10T18:05:27.990 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-09-10T18:05:27.990 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T18:05:27.990 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-09-10T18:05:27.990 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-09-10T18:05:27.990 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:05:27.991 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T18:05:27.991 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T18:05:27.991 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T18:05:27.991 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-09-10T18:05:27.991 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:05:27.991 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T18:05:27.992 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T18:05:27.992 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 5.14.0-505.el9.x86_64 2024-09-10T18:05:27.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T18:05:28.626 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-09-10T18:05:29.904 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T18:05:29.923 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-09-10T18:05:29.925 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-09-10T18:05:29.941 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.HgUf2Y676f 2024-09-10T18:05:29.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HgUf2Y676f 2024-09-10T18:05:30.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.HgUf2Y676f 2024-09-10T18:05:30.145 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.HgUf2Y676f is 6KB 2024-09-10T18:05:30.194 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.HgUf2Y676f 2024-09-10T18:05:30.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T18:05:30.279 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T18:05:30.280 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T18:05:30.280 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T18:05:30.280 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T18:05:30.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T18:05:30.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-09-10T18:05:54.317 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-10T18:05:54.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-10T18:05:54.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:06:00.457 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-10T18:06:00.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-10T18:06:00.458 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:06:12.797 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-09-10T18:06:18.941 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-09-10T18:06:21.801 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-10T18:06:21.802 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:06:24.861 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-09-10T18:06:27.945 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-10T18:06:27.945 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:06:36.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-10T18:06:36.869 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:06:46.333 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-09-10T18:06:55.297 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-09-10T18:06:58.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-10T18:06:58.338 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:06:59.252 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-09-10T18:06:59.616 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-10T18:06:59.616 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-10T18:06:59.617 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-09-10T18:06:59.632 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-505.el9.x86_64 2024-09-10T18:06:59.632 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T18:06:59.633 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T18:06:59.633 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T18:07:00.633 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-10T18:07:00.634 DEBUG:teuthology.orchestra.run.smithi028:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T18:07:00.710 INFO:teuthology.orchestra.run.smithi028.stdout:ttyS1 2024-09-10T18:07:00.735 DEBUG:teuthology.parallel:result is None 2024-09-10T18:07:10.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-10T18:07:10.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:07:10.908 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-09-10T18:07:11.436 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-10T18:07:11.436 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-10T18:07:11.436 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-09-10T18:07:11.450 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-505.el9.x86_64 2024-09-10T18:07:11.451 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T18:07:11.451 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T18:07:11.451 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T18:07:12.451 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-10T18:07:12.452 DEBUG:teuthology.orchestra.run.smithi204:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T18:07:12.588 INFO:teuthology.orchestra.run.smithi204.stdout:ttyS1 2024-09-10T18:07:12.616 DEBUG:teuthology.parallel:result is None 2024-09-10T18:07:12.616 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-10T18:07:12.624 INFO:teuthology.task.internal:Creating test directory... 2024-09-10T18:07:12.625 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T18:07:12.628 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T18:07:12.646 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-10T18:07:12.654 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-10T18:07:12.662 INFO:teuthology.task.internal:Creating archive directory... 2024-09-10T18:07:12.662 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T18:07:12.689 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T18:07:12.763 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-10T18:07:12.774 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-10T18:07:12.774 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T18:07:12.785 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T18:07:12.815 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T18:07:12.826 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T18:07:12.827 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T18:07:12.839 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T18:07:12.841 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-10T18:07:12.875 INFO:teuthology.task.internal:Configuring sudo... 2024-09-10T18:07:12.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T18:07:12.879 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T18:07:12.907 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-10T18:07:12.922 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-10T18:07:12.923 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T18:07:12.954 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T18:07:12.969 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T18:07:13.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T18:07:13.122 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T18:07:13.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T18:07:13.259 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T18:07:13.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T18:07:13.330 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T18:07:13.368 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T18:07:13.421 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T18:07:13.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T18:07:13.544 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-10T18:07:13.544 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T18:07:13.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-09-10T18:07:13.613 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-09-10T18:07:13.661 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T18:07:13.700 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T18:07:14.042 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-10T18:07:14.049 INFO:teuthology.task.internal:Starting timer... 2024-09-10T18:07:14.049 INFO:teuthology.run_tasks:Running task pcp... 2024-09-10T18:07:14.121 INFO:teuthology.run_tasks:Running task selinux... 2024-09-10T18:07:14.130 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-10T18:07:14.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-09-10T18:07:14.185 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-09-10T18:07:14.187 DEBUG:teuthology.orchestra.run.smithi204:> sudo service auditd rotate 2024-09-10T18:07:14.246 INFO:teuthology.orchestra.run.smithi204.stdout:Rotating logs: 2024-09-10T18:07:14.248 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-10T18:07:14.248 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-09-10T18:07:14.281 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-09-10T18:07:14.281 DEBUG:teuthology.orchestra.run.smithi204:> /usr/sbin/getenforce 2024-09-10T18:07:14.301 INFO:teuthology.orchestra.run.smithi204.stdout:Permissive 2024-09-10T18:07:14.302 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi204.front.sepia.ceph.com': 'permissive'} 2024-09-10T18:07:14.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T18:07:14.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:07:14.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T18:07:14.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:07:14.377 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-10T18:07:14.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-09-10T18:07:14.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo /usr/sbin/setenforce permissive 2024-09-10T18:07:14.452 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-10T18:07:14.461 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-10T18:07:14.497 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-10T18:07:14.498 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-10T18:10:59.976 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-09-10T18:10:59.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-10T18:10:59.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:11:00.057 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-09-10T18:11:00.141 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-10T18:11:00.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-10T18:11:00.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:11:00.218 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-09-10T18:11:00.305 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-10T18:11:00.306 INFO:teuthology.run_tasks:Running task clock... 2024-09-10T18:11:00.317 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-10T18:11:00.317 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T18:11:00.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T18:11:00.321 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T18:11:00.321 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T18:11:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T18:11:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T18:11:00.375 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T18:11:00.390 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T18:11:00.391 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-09-10T18:11:00.402 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-09-10T18:11:00.416 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: ntpd: command not found 2024-09-10T18:11:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T18:11:00.425 INFO:teuthology.orchestra.run.smithi204.stdout:506 Cannot talk to daemon 2024-09-10T18:11:00.431 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T18:11:00.439 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T18:11:00.453 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T18:11:00.494 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-09-10T18:11:00.498 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T18:11:00.498 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-09-10T18:11:00.498 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:11:00.498 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:11:00.498 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:11:00.498 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:11:00.508 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-09-10T18:11:00.511 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T18:11:00.511 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-09-10T18:11:00.511 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:11:00.511 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:11:00.511 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:11:00.511 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:11:00.512 INFO:teuthology.run_tasks:Running task pexec... 2024-09-10T18:11:00.522 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-10T18:11:00.522 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T18:11:00.523 DEBUG:teuthology.orchestra.run.smithi028:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-10T18:11:00.523 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi204.front.sepia.ceph.com 2024-09-10T18:11:00.523 DEBUG:teuthology.orchestra.run.smithi204:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-10T18:11:00.903 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:11:00.905 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:11:00.915 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:11:00.915 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-09-10T18:11:00.915 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:11:00.916 INFO:teuthology.orchestra.run.smithi204.stdout:Removing: 2024-09-10T18:11:00.916 INFO:teuthology.orchestra.run.smithi204.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-10T18:11:00.916 INFO:teuthology.orchestra.run.smithi204.stdout:Removing unused dependencies: 2024-09-10T18:11:00.916 INFO:teuthology.orchestra.run.smithi204.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-10T18:11:00.916 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:11:00.916 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:11:00.916 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:11:00.916 INFO:teuthology.orchestra.run.smithi204.stdout:Remove 2 Packages 2024-09-10T18:11:00.916 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:11:00.916 INFO:teuthology.orchestra.run.smithi204.stdout:Freed space: 5.5 M 2024-09-10T18:11:00.916 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:11:00.920 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:11:00.920 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:11:00.922 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:11:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-09-10T18:11:00.922 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:11:00.922 INFO:teuthology.orchestra.run.smithi028.stdout:Removing: 2024-09-10T18:11:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-10T18:11:00.923 INFO:teuthology.orchestra.run.smithi028.stdout:Removing unused dependencies: 2024-09-10T18:11:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-10T18:11:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:11:00.923 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:11:00.923 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:11:00.923 INFO:teuthology.orchestra.run.smithi028.stdout:Remove 2 Packages 2024-09-10T18:11:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:11:00.924 INFO:teuthology.orchestra.run.smithi028.stdout:Freed space: 5.5 M 2024-09-10T18:11:00.924 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:11:00.927 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:11:00.927 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:11:00.987 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:11:00.987 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:11:00.991 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:11:00.991 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:11:01.122 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:11:01.133 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:11:01.199 INFO:teuthology.orchestra.run.smithi028.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-10T18:11:01.218 INFO:teuthology.orchestra.run.smithi204.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-10T18:11:01.255 INFO:teuthology.orchestra.run.smithi028.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-10T18:11:01.273 INFO:teuthology.orchestra.run.smithi204.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-10T18:11:01.715 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-10T18:11:01.715 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-10T18:11:01.762 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-10T18:11:01.762 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-10T18:11:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-10T18:11:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:11:01.981 INFO:teuthology.orchestra.run.smithi204.stdout:Removed: 2024-09-10T18:11:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-10T18:11:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:11:01.981 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:11:02.037 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-10T18:11:02.037 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:11:02.037 INFO:teuthology.orchestra.run.smithi028.stdout:Removed: 2024-09-10T18:11:02.037 INFO:teuthology.orchestra.run.smithi028.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-10T18:11:02.037 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:11:02.037 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:11:02.838 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:50 ago on Tue 10 Sep 2024 06:10:12 PM UTC. 2024-09-10T18:11:02.839 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:46 ago on Tue 10 Sep 2024 06:10:16 PM UTC. 2024-09-10T18:11:02.976 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:11:02.977 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:11:02.977 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-09-10T18:11:02.977 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:11:02.977 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:11:02.977 INFO:teuthology.orchestra.run.smithi028.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-10T18:11:02.977 INFO:teuthology.orchestra.run.smithi028.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-10T18:11:02.977 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-10T18:11:02.977 INFO:teuthology.orchestra.run.smithi028.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-10T18:11:02.977 INFO:teuthology.orchestra.run.smithi028.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-10T18:11:02.977 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-10T18:11:02.977 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-10T18:11:02.977 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:11:02.977 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:11:02.978 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:11:02.978 INFO:teuthology.orchestra.run.smithi028.stdout:Install 6 Packages 2024-09-10T18:11:02.978 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:11:02.979 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:11:02.980 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:11:02.980 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-09-10T18:11:02.980 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:11:02.980 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:11:02.980 INFO:teuthology.orchestra.run.smithi204.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-10T18:11:02.980 INFO:teuthology.orchestra.run.smithi204.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-10T18:11:02.980 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-10T18:11:02.980 INFO:teuthology.orchestra.run.smithi204.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-10T18:11:02.981 INFO:teuthology.orchestra.run.smithi204.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-10T18:11:02.981 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-10T18:11:02.981 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-10T18:11:02.981 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:11:02.981 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:11:02.981 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:11:02.981 INFO:teuthology.orchestra.run.smithi204.stdout:Install 6 Packages 2024-09-10T18:11:02.981 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:11:02.986 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 2.0 M 2024-09-10T18:11:02.987 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 9.2 M 2024-09-10T18:11:02.987 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:11:02.989 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 2.0 M 2024-09-10T18:11:02.989 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 9.2 M 2024-09-10T18:11:02.989 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:11:03.116 INFO:teuthology.orchestra.run.smithi028.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 418 kB/s | 44 kB 00:00 2024-09-10T18:11:03.145 INFO:teuthology.orchestra.run.smithi204.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 412 kB/s | 44 kB 00:00 2024-09-10T18:11:03.150 INFO:teuthology.orchestra.run.smithi028.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 757 kB/s | 106 kB 00:00 2024-09-10T18:11:03.187 INFO:teuthology.orchestra.run.smithi204.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 707 kB/s | 106 kB 00:00 2024-09-10T18:11:03.192 INFO:teuthology.orchestra.run.smithi028.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 952 kB/s | 72 kB 00:00 2024-09-10T18:11:03.208 INFO:teuthology.orchestra.run.smithi028.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.4 MB/s | 84 kB 00:00 2024-09-10T18:11:03.221 INFO:teuthology.orchestra.run.smithi204.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 948 kB/s | 72 kB 00:00 2024-09-10T18:11:03.251 INFO:teuthology.orchestra.run.smithi028.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 3.7 MB/s | 906 kB 00:00 2024-09-10T18:11:03.255 INFO:teuthology.orchestra.run.smithi204.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.2 MB/s | 84 kB 00:00 2024-09-10T18:11:03.288 INFO:teuthology.orchestra.run.smithi204.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 3.5 MB/s | 906 kB 00:00 2024-09-10T18:11:03.334 INFO:teuthology.orchestra.run.smithi028.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 5.8 MB/s | 837 kB 00:00 2024-09-10T18:11:03.335 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:11:03.335 INFO:teuthology.orchestra.run.smithi028.stdout:Total 5.8 MB/s | 2.0 MB 00:00 2024-09-10T18:11:03.355 INFO:teuthology.orchestra.run.smithi204.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 6.1 MB/s | 837 kB 00:00 2024-09-10T18:11:03.355 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:11:03.355 INFO:teuthology.orchestra.run.smithi204.stdout:Total 5.5 MB/s | 2.0 MB 00:00 2024-09-10T18:11:03.410 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:11:03.424 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:11:03.424 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:11:03.429 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:11:03.444 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:11:03.444 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:11:03.739 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:11:03.739 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:11:03.741 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:11:03.741 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:11:04.152 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:11:04.185 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:11:04.240 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-10T18:11:04.283 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-10T18:11:04.287 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-10T18:11:04.312 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-10T18:11:04.324 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-10T18:11:04.350 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-10T18:11:04.520 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-10T18:11:04.560 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-10T18:11:04.580 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-10T18:11:04.614 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-10T18:11:05.007 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-10T18:11:05.049 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-10T18:11:05.094 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-10T18:11:05.117 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-10T18:11:06.032 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-10T18:11:06.032 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-10T18:11:06.033 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-10T18:11:06.033 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-10T18:11:06.033 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-10T18:11:06.033 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-10T18:11:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-10T18:11:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-10T18:11:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-10T18:11:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-10T18:11:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-10T18:11:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-10T18:11:06.399 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-10T18:11:06.399 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:11:06.400 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:11:06.400 INFO:teuthology.orchestra.run.smithi204.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-10T18:11:06.400 INFO:teuthology.orchestra.run.smithi204.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-10T18:11:06.400 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-10T18:11:06.400 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:11:06.400 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:11:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-10T18:11:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:11:06.433 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:11:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-10T18:11:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-10T18:11:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-10T18:11:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:11:06.433 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:11:06.703 DEBUG:teuthology.parallel:result is None 2024-09-10T18:11:06.738 DEBUG:teuthology.parallel:result is None 2024-09-10T18:11:06.738 INFO:teuthology.run_tasks:Running task install... 2024-09-10T18:11:06.746 DEBUG:teuthology.task.install:project ceph 2024-09-10T18:11:06.746 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588'}} 2024-09-10T18:11:06.747 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588'} 2024-09-10T18:11:06.747 INFO:teuthology.task.install:Using flavor: default 2024-09-10T18:11:06.752 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-10T18:11:06.752 INFO:teuthology.task.install:extra packages: [] 2024-09-10T18:11:06.753 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588', 'tag': None, 'wait_for_package': False} 2024-09-10T18:11:06.753 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:11:06.755 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588', 'tag': None, 'wait_for_package': False} 2024-09-10T18:11:06.756 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:11:06.910 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/ 2024-09-10T18:11:06.910 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1606.g5be57395 2024-09-10T18:11:06.919 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/ 2024-09-10T18:11:06.919 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1606.g5be57395 2024-09-10T18:11:07.043 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T18:11:07.044 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T18:11:07.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T18:11:07.055 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T18:11:07.055 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-10T18:11:07.055 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T18:11:07.075 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-10T18:11:07.075 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5be57395ea235535910512ae73ba8e7ff25fe588/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T18:11:07.085 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-10T18:11:07.086 DEBUG:teuthology.orchestra.run.smithi204:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5be57395ea235535910512ae73ba8e7ff25fe588/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T18:11:07.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T18:11:07.151 DEBUG:teuthology.orchestra.run.smithi204:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T18:11:07.220 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T18:11:07.226 DEBUG:teuthology.orchestra.run.smithi204:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T18:11:07.287 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-09-10T18:11:07.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-09-10T18:11:07.290 INFO:teuthology.orchestra.run.smithi204.stdout:check_obsoletes = 1 2024-09-10T18:11:07.292 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum clean all 2024-09-10T18:11:07.517 INFO:teuthology.orchestra.run.smithi204.stdout:45 files removed 2024-09-10T18:11:07.537 INFO:teuthology.orchestra.run.smithi028.stdout:45 files removed 2024-09-10T18:11:07.544 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-radosgw 2024-09-10T18:11:07.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw 2024-09-10T18:11:08.095 INFO:teuthology.orchestra.run.smithi204.stdout:ceph packages for x86_64 293 kB/s | 77 kB 00:00 2024-09-10T18:11:08.124 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 297 kB/s | 77 kB 00:00 2024-09-10T18:11:08.378 INFO:teuthology.orchestra.run.smithi204.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-10T18:11:08.391 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 49 kB/s | 11 kB 00:00 2024-09-10T18:11:08.650 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 8.1 kB/s | 1.8 kB 00:00 2024-09-10T18:11:08.671 INFO:teuthology.orchestra.run.smithi204.stdout:ceph source packages 7.0 kB/s | 1.8 kB 00:00 2024-09-10T18:11:10.404 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.2 MB 00:01 2024-09-10T18:11:10.575 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.2 MB 00:01 2024-09-10T18:11:12.651 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-10T18:11:12.714 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 17 MB/s | 20 MB 00:01 2024-09-10T18:11:17.306 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-10T18:11:18.210 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.5 MB 00:01 2024-09-10T18:11:18.966 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 19 kB 00:00 2024-09-10T18:11:19.770 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 33 MB/s | 23 MB 00:00 2024-09-10T18:11:19.844 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 19 kB 00:00 2024-09-10T18:11:20.731 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-10T18:11:25.279 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-09-10T18:11:26.396 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-10T18:11:27.668 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:11:27.670 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:11:27.670 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-10T18:11:27.670 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:11:27.670 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:11:27.670 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw x86_64 2:17.2.7-1606.g5be57395.el9 ceph 14 M 2024-09-10T18:11:27.670 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrading: 2024-09-10T18:11:27.670 INFO:teuthology.orchestra.run.smithi204.stdout: librados2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 3.1 M 2024-09-10T18:11:27.670 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.9 M 2024-09-10T18:11:27.670 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-10T18:11:27.671 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base x86_64 2:17.2.7-1606.g5be57395.el9 ceph 5.0 M 2024-09-10T18:11:27.671 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common x86_64 2:17.2.7-1606.g5be57395.el9 ceph 22 M 2024-09-10T18:11:27.671 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux x86_64 2:17.2.7-1606.g5be57395.el9 ceph 25 k 2024-09-10T18:11:27.671 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T18:11:27.671 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 637 k 2024-09-10T18:11:27.671 INFO:teuthology.orchestra.run.smithi204.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T18:11:27.671 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T18:11:27.671 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T18:11:27.671 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 458 k 2024-09-10T18:11:27.671 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T18:11:27.671 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 4.9 M 2024-09-10T18:11:27.671 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T18:11:27.672 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T18:11:27.672 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 45 k 2024-09-10T18:11:27.672 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common x86_64 2:17.2.7-1606.g5be57395.el9 ceph 120 k 2024-09-10T18:11:27.672 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs x86_64 2:17.2.7-1606.g5be57395.el9 ceph 156 k 2024-09-10T18:11:27.672 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T18:11:27.672 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados x86_64 2:17.2.7-1606.g5be57395.el9 ceph 320 k 2024-09-10T18:11:27.672 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 295 k 2024-09-10T18:11:27.672 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw x86_64 2:17.2.7-1606.g5be57395.el9 ceph 99 k 2024-09-10T18:11:27.672 INFO:teuthology.orchestra.run.smithi204.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T18:11:27.672 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:11:27.672 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:11:27.672 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:11:27.672 INFO:teuthology.orchestra.run.smithi204.stdout:Install 22 Packages 2024-09-10T18:11:27.673 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrade 2 Packages 2024-09-10T18:11:27.673 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:11:27.673 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 58 M 2024-09-10T18:11:27.673 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:11:28.665 INFO:teuthology.orchestra.run.smithi204.stdout:(1/24): ceph-base-17.2.7-1606.g5be57395.el9.x86 5.3 MB/s | 5.0 MB 00:00 2024-09-10T18:11:28.744 INFO:teuthology.orchestra.run.smithi204.stdout:(2/24): ceph-selinux-17.2.7-1606.g5be57395.el9. 319 kB/s | 25 kB 00:00 2024-09-10T18:11:28.881 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:11:28.883 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:11:28.883 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-09-10T18:11:28.884 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:11:28.884 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:11:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:17.2.7-1606.g5be57395.el9 ceph 14 M 2024-09-10T18:11:28.884 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-09-10T18:11:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 3.1 M 2024-09-10T18:11:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.9 M 2024-09-10T18:11:28.884 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-10T18:11:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:17.2.7-1606.g5be57395.el9 ceph 5.0 M 2024-09-10T18:11:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:17.2.7-1606.g5be57395.el9 ceph 22 M 2024-09-10T18:11:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:17.2.7-1606.g5be57395.el9 ceph 25 k 2024-09-10T18:11:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T18:11:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 637 k 2024-09-10T18:11:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T18:11:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T18:11:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T18:11:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 458 k 2024-09-10T18:11:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T18:11:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 4.9 M 2024-09-10T18:11:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T18:11:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T18:11:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 45 k 2024-09-10T18:11:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:17.2.7-1606.g5be57395.el9 ceph 120 k 2024-09-10T18:11:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:17.2.7-1606.g5be57395.el9 ceph 156 k 2024-09-10T18:11:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T18:11:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:17.2.7-1606.g5be57395.el9 ceph 320 k 2024-09-10T18:11:28.886 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 295 k 2024-09-10T18:11:28.886 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:17.2.7-1606.g5be57395.el9 ceph 99 k 2024-09-10T18:11:28.886 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T18:11:28.886 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:11:28.886 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:11:28.886 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:11:28.886 INFO:teuthology.orchestra.run.smithi028.stdout:Install 22 Packages 2024-09-10T18:11:28.886 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 2 Packages 2024-09-10T18:11:28.886 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:11:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 58 M 2024-09-10T18:11:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:11:28.894 INFO:teuthology.orchestra.run.smithi204.stdout:(3/24): libcephfs2-17.2.7-1606.g5be57395.el9.x8 4.1 MB/s | 637 kB 00:00 2024-09-10T18:11:29.045 INFO:teuthology.orchestra.run.smithi204.stdout:(4/24): ceph-radosgw-17.2.7-1606.g5be57395.el9. 11 MB/s | 14 MB 00:01 2024-09-10T18:11:29.070 INFO:teuthology.orchestra.run.smithi204.stdout:(5/24): libradosstriper1-17.2.7-1606.g5be57395. 2.6 MB/s | 458 kB 00:00 2024-09-10T18:11:29.128 INFO:teuthology.orchestra.run.smithi204.stdout:(6/24): python3-ceph-argparse-17.2.7-1606.g5be5 773 kB/s | 45 kB 00:00 2024-09-10T18:11:29.196 INFO:teuthology.orchestra.run.smithi204.stdout:(7/24): python3-ceph-common-17.2.7-1606.g5be573 1.8 MB/s | 120 kB 00:00 2024-09-10T18:11:29.404 INFO:teuthology.orchestra.run.smithi204.stdout:(8/24): ceph-common-17.2.7-1606.g5be57395.el9.x 13 MB/s | 22 MB 00:01 2024-09-10T18:11:29.429 INFO:teuthology.orchestra.run.smithi204.stdout:(9/24): python3-cephfs-17.2.7-1606.g5be57395.el 668 kB/s | 156 kB 00:00 2024-09-10T18:11:29.496 INFO:teuthology.orchestra.run.smithi204.stdout:(10/24): librgw2-17.2.7-1606.g5be57395.el9.x86_ 11 MB/s | 4.9 MB 00:00 2024-09-10T18:11:29.521 INFO:teuthology.orchestra.run.smithi204.stdout:(11/24): python3-rados-17.2.7-1606.g5be57395.el 2.7 MB/s | 320 kB 00:00 2024-09-10T18:11:29.548 INFO:teuthology.orchestra.run.smithi028.stdout:(1/24): ceph-base-17.2.7-1606.g5be57395.el9.x86 10 MB/s | 5.0 MB 00:00 2024-09-10T18:11:29.555 INFO:teuthology.orchestra.run.smithi204.stdout:(12/24): python3-rbd-17.2.7-1606.g5be57395.el9. 2.3 MB/s | 295 kB 00:00 2024-09-10T18:11:29.589 INFO:teuthology.orchestra.run.smithi204.stdout:(13/24): python3-rgw-17.2.7-1606.g5be57395.el9. 1.1 MB/s | 99 kB 00:00 2024-09-10T18:11:29.597 INFO:teuthology.orchestra.run.smithi028.stdout:(2/24): ceph-selinux-17.2.7-1606.g5be57395.el9. 504 kB/s | 25 kB 00:00 2024-09-10T18:11:29.673 INFO:teuthology.orchestra.run.smithi204.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-09-10T18:11:29.681 INFO:teuthology.orchestra.run.smithi028.stdout:(3/24): libcephfs2-17.2.7-1606.g5be57395.el9.x8 7.5 MB/s | 637 kB 00:00 2024-09-10T18:11:29.697 INFO:teuthology.orchestra.run.smithi204.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 912 kB/s | 160 kB 00:00 2024-09-10T18:11:29.748 INFO:teuthology.orchestra.run.smithi204.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 843 kB/s | 42 kB 00:00 2024-09-10T18:11:29.764 INFO:teuthology.orchestra.run.smithi028.stdout:(4/24): libradosstriper1-17.2.7-1606.g5be57395. 5.4 MB/s | 458 kB 00:00 2024-09-10T18:11:29.798 INFO:teuthology.orchestra.run.smithi204.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 6.1 MB/s | 309 kB 00:00 2024-09-10T18:11:29.823 INFO:teuthology.orchestra.run.smithi204.stdout:(18/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-09-10T18:11:29.849 INFO:teuthology.orchestra.run.smithi204.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 967 kB/s | 49 kB 00:00 2024-09-10T18:11:29.883 INFO:teuthology.orchestra.run.smithi204.stdout:(20/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.2 MB/s | 662 kB 00:00 2024-09-10T18:11:29.907 INFO:teuthology.orchestra.run.smithi204.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 804 kB/s | 67 kB 00:00 2024-09-10T18:11:29.950 INFO:teuthology.orchestra.run.smithi204.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-10T18:11:30.131 INFO:teuthology.orchestra.run.smithi028.stdout:(5/24): librgw2-17.2.7-1606.g5be57395.el9.x86_6 13 MB/s | 4.9 MB 00:00 2024-09-10T18:11:30.215 INFO:teuthology.orchestra.run.smithi028.stdout:(6/24): python3-ceph-argparse-17.2.7-1606.g5be5 543 kB/s | 45 kB 00:00 2024-09-10T18:11:30.415 INFO:teuthology.orchestra.run.smithi028.stdout:(7/24): ceph-radosgw-17.2.7-1606.g5be57395.el9. 11 MB/s | 14 MB 00:01 2024-09-10T18:11:30.440 INFO:teuthology.orchestra.run.smithi028.stdout:(8/24): python3-ceph-common-17.2.7-1606.g5be573 534 kB/s | 120 kB 00:00 2024-09-10T18:11:30.474 INFO:teuthology.orchestra.run.smithi028.stdout:(9/24): python3-cephfs-17.2.7-1606.g5be57395.el 2.6 MB/s | 156 kB 00:00 2024-09-10T18:11:30.475 INFO:teuthology.orchestra.run.smithi204.stdout:(23/24): librados2-17.2.7-1606.g5be57395.el9.x8 5.2 MB/s | 3.1 MB 00:00 2024-09-10T18:11:30.499 INFO:teuthology.orchestra.run.smithi028.stdout:(10/24): python3-rados-17.2.7-1606.g5be57395.el 5.4 MB/s | 320 kB 00:00 2024-09-10T18:11:30.525 INFO:teuthology.orchestra.run.smithi204.stdout:(24/24): librbd1-17.2.7-1606.g5be57395.el9.x86_ 4.8 MB/s | 2.9 MB 00:00 2024-09-10T18:11:30.530 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:11:30.530 INFO:teuthology.orchestra.run.smithi204.stdout:Total 20 MB/s | 58 MB 00:02 2024-09-10T18:11:30.541 INFO:teuthology.orchestra.run.smithi028.stdout:(11/24): python3-rbd-17.2.7-1606.g5be57395.el9. 4.3 MB/s | 295 kB 00:00 2024-09-10T18:11:30.566 INFO:teuthology.orchestra.run.smithi028.stdout:(12/24): python3-rgw-17.2.7-1606.g5be57395.el9. 1.5 MB/s | 99 kB 00:00 2024-09-10T18:11:30.593 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:11:30.627 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:11:30.627 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:11:30.808 INFO:teuthology.orchestra.run.smithi028.stdout:(13/24): ceph-common-17.2.7-1606.g5be57395.el9. 13 MB/s | 22 MB 00:01 2024-09-10T18:11:30.833 INFO:teuthology.orchestra.run.smithi028.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 169 kB/s | 45 kB 00:00 2024-09-10T18:11:30.892 INFO:teuthology.orchestra.run.smithi028.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 457 kB/s | 160 kB 00:00 2024-09-10T18:11:30.942 INFO:teuthology.orchestra.run.smithi028.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 845 kB/s | 42 kB 00:00 2024-09-10T18:11:30.993 INFO:teuthology.orchestra.run.smithi028.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 6.1 MB/s | 309 kB 00:00 2024-09-10T18:11:31.018 INFO:teuthology.orchestra.run.smithi028.stdout:(18/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 292 kB 00:00 2024-09-10T18:11:31.043 INFO:teuthology.orchestra.run.smithi028.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-09-10T18:11:31.085 INFO:teuthology.orchestra.run.smithi028.stdout:(20/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.3 MB/s | 662 kB 00:00 2024-09-10T18:11:31.098 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:11:31.098 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:11:31.110 INFO:teuthology.orchestra.run.smithi028.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 731 kB/s | 67 kB 00:00 2024-09-10T18:11:31.152 INFO:teuthology.orchestra.run.smithi028.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-10T18:11:31.319 INFO:teuthology.orchestra.run.smithi028.stdout:(23/24): librbd1-17.2.7-1606.g5be57395.el9.x86_ 14 MB/s | 2.9 MB 00:00 2024-09-10T18:11:31.386 INFO:teuthology.orchestra.run.smithi028.stdout:(24/24): librados2-17.2.7-1606.g5be57395.el9.x8 10 MB/s | 3.1 MB 00:00 2024-09-10T18:11:31.390 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:11:31.391 INFO:teuthology.orchestra.run.smithi028.stdout:Total 23 MB/s | 58 MB 00:02 2024-09-10T18:11:31.454 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:11:31.492 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:11:31.493 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:11:31.989 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:11:31.990 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:11:32.311 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:11:32.644 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-10T18:11:32.761 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T18:11:32.852 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T18:11:32.932 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 3/26 2024-09-10T18:11:32.954 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T18:11:33.026 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T18:11:33.071 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-10T18:11:33.101 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-10T18:11:33.195 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:11:33.550 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-10T18:11:33.575 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-10T18:11:33.582 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 8/26 2024-09-10T18:11:33.628 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 8/26 2024-09-10T18:11:33.646 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T18:11:33.665 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-10T18:11:33.709 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9. 10/26 2024-09-10T18:11:33.739 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T18:11:33.745 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 11/26 2024-09-10T18:11:33.807 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 12/26 2024-09-10T18:11:33.828 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 3/26 2024-09-10T18:11:33.834 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 13/26 2024-09-10T18:11:33.856 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T18:11:33.901 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 13/26 2024-09-10T18:11:33.945 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T18:11:33.950 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-10T18:11:33.993 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-09-10T18:11:33.998 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-10T18:11:34.026 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-10T18:11:34.036 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-10T18:11:34.325 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-10T18:11:34.364 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 18/26 2024-09-10T18:11:34.443 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 18/26 2024-09-10T18:11:34.508 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-10T18:11:34.510 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 19/26 2024-09-10T18:11:34.541 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x8 20/26 2024-09-10T18:11:34.542 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 8/26 2024-09-10T18:11:34.597 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 8/26 2024-09-10T18:11:34.633 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-10T18:11:34.669 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9. 10/26 2024-09-10T18:11:34.705 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 11/26 2024-09-10T18:11:34.769 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 12/26 2024-09-10T18:11:34.794 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 13/26 2024-09-10T18:11:34.869 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 13/26 2024-09-10T18:11:34.910 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-10T18:11:34.945 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-09-10T18:11:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-10T18:11:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-10T18:11:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 18/26 2024-09-10T18:11:35.422 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 18/26 2024-09-10T18:11:35.499 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 19/26 2024-09-10T18:11:35.525 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x8 20/26 2024-09-10T18:11:37.037 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T18:11:37.074 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T18:11:37.604 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T18:11:37.644 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 22/26 2024-09-10T18:11:37.733 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 22/26 2024-09-10T18:11:37.733 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T18:11:37.733 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T18:11:37.734 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:11:37.770 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T18:11:37.982 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T18:11:38.020 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T18:11:38.580 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T18:11:38.622 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 22/26 2024-09-10T18:11:38.736 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 22/26 2024-09-10T18:11:38.736 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T18:11:38.736 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T18:11:38.736 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:11:38.774 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T18:11:47.792 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T18:11:47.792 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /sys 2024-09-10T18:11:47.792 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /proc 2024-09-10T18:11:47.792 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /mnt 2024-09-10T18:11:47.792 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /var/tmp 2024-09-10T18:11:47.793 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /home 2024-09-10T18:11:47.793 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /root 2024-09-10T18:11:47.793 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /tmp 2024-09-10T18:11:47.793 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:11:48.972 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T18:11:48.972 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-09-10T18:11:48.972 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-09-10T18:11:48.972 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-09-10T18:11:48.972 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-09-10T18:11:48.972 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-09-10T18:11:48.972 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-09-10T18:11:48.972 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-09-10T18:11:48.972 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:11:49.103 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 24/26 2024-09-10T18:11:49.181 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 24/26 2024-09-10T18:11:49.182 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:11:49.182 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T18:11:49.182 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T18:11:49.182 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T18:11:49.182 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:11:49.183 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-10T18:11:49.273 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-10T18:11:49.273 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T18:11:50.331 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 24/26 2024-09-10T18:11:50.403 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 24/26 2024-09-10T18:11:50.403 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:11:50.403 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T18:11:50.404 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T18:11:50.404 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T18:11:50.404 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:11:50.405 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-10T18:11:50.492 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-10T18:11:50.492 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T18:11:50.547 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T18:11:50.547 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 1/26 2024-09-10T18:11:50.547 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T18:11:50.548 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 3/26 2024-09-10T18:11:50.548 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T18:11:50.548 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 5/26 2024-09-10T18:11:50.548 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 6/26 2024-09-10T18:11:50.548 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 7/26 2024-09-10T18:11:50.548 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9. 8/26 2024-09-10T18:11:50.548 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x8 9/26 2024-09-10T18:11:50.549 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 10/26 2024-09-10T18:11:50.549 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 11/26 2024-09-10T18:11:50.549 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 12/26 2024-09-10T18:11:50.549 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 13/26 2024-09-10T18:11:50.549 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-10T18:11:50.549 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-10T18:11:50.549 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-10T18:11:50.549 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-10T18:11:50.549 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-10T18:11:50.549 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-09-10T18:11:50.549 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-10T18:11:50.549 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-10T18:11:50.550 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-10T18:11:50.550 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T18:11:50.550 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-10T18:11:50.550 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 25/26 2024-09-10T18:11:51.361 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T18:11:51.361 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:11:51.361 INFO:teuthology.orchestra.run.smithi204.stdout:Upgraded: 2024-09-10T18:11:51.361 INFO:teuthology.orchestra.run.smithi204.stdout: librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:51.361 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:51.361 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:11:51.361 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:51.361 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:51.361 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:51.362 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:51.362 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T18:11:51.362 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:51.362 INFO:teuthology.orchestra.run.smithi204.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T18:11:51.362 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T18:11:51.362 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T18:11:51.362 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:51.362 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T18:11:51.362 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:51.362 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T18:11:51.362 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T18:11:51.362 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:51.362 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:51.362 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:51.363 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T18:11:51.363 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:51.363 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:51.363 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:51.363 INFO:teuthology.orchestra.run.smithi204.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T18:11:51.363 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:11:51.363 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:11:51.720 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-test 2024-09-10T18:11:51.757 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T18:11:51.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 1/26 2024-09-10T18:11:51.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T18:11:51.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 3/26 2024-09-10T18:11:51.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T18:11:51.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 5/26 2024-09-10T18:11:51.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 6/26 2024-09-10T18:11:51.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 7/26 2024-09-10T18:11:51.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9. 8/26 2024-09-10T18:11:51.757 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x8 9/26 2024-09-10T18:11:51.758 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 10/26 2024-09-10T18:11:51.758 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 11/26 2024-09-10T18:11:51.758 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 12/26 2024-09-10T18:11:51.758 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 13/26 2024-09-10T18:11:51.758 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-10T18:11:51.758 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-10T18:11:51.758 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-10T18:11:51.758 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-10T18:11:51.758 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-10T18:11:51.758 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-09-10T18:11:51.758 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-10T18:11:51.758 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-10T18:11:51.758 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-10T18:11:51.758 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T18:11:51.759 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-10T18:11:51.759 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 25/26 2024-09-10T18:11:52.289 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:27 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:11:52.421 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:11:52.422 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:11:52.422 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-10T18:11:52.422 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:11:52.422 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:11:52.422 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test x86_64 2:17.2.7-1606.g5be57395.el9 ceph 48 M 2024-09-10T18:11:52.422 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-10T18:11:52.423 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite x86_64 2:17.2.7-1606.g5be57395.el9 ceph 163 k 2024-09-10T18:11:52.423 INFO:teuthology.orchestra.run.smithi204.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T18:11:52.423 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T18:11:52.423 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:11:52.423 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:11:52.423 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:11:52.423 INFO:teuthology.orchestra.run.smithi204.stdout:Install 4 Packages 2024-09-10T18:11:52.423 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:11:52.424 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 48 M 2024-09-10T18:11:52.424 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 193 M 2024-09-10T18:11:52.424 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:11:52.599 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T18:11:52.599 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:11:52.599 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-09-10T18:11:52.599 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:52.599 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:52.600 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:11:52.600 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:52.600 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:52.600 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:52.600 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:52.600 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T18:11:52.600 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:52.600 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T18:11:52.600 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T18:11:52.600 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T18:11:52.600 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:52.600 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T18:11:52.600 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:52.601 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T18:11:52.601 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T18:11:52.601 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:52.601 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:52.601 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:52.601 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T18:11:52.601 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:52.601 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:52.601 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:11:52.601 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T18:11:52.601 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:11:52.601 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:11:52.679 INFO:teuthology.orchestra.run.smithi204.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.9 MB/s | 304 kB 00:00 2024-09-10T18:11:52.746 INFO:teuthology.orchestra.run.smithi204.stdout:(2/4): libcephsqlite-17.2.7-1606.g5be57395.el9. 738 kB/s | 163 kB 00:00 2024-09-10T18:11:52.771 INFO:teuthology.orchestra.run.smithi204.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 693 kB/s | 64 kB 00:00 2024-09-10T18:11:52.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-test 2024-09-10T18:11:53.557 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:27 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:11:53.699 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:11:53.699 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:11:53.699 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-09-10T18:11:53.700 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:11:53.700 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:11:53.700 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:17.2.7-1606.g5be57395.el9 ceph 48 M 2024-09-10T18:11:53.700 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-10T18:11:53.700 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:17.2.7-1606.g5be57395.el9 ceph 163 k 2024-09-10T18:11:53.700 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T18:11:53.700 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T18:11:53.700 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:11:53.700 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:11:53.700 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:11:53.700 INFO:teuthology.orchestra.run.smithi028.stdout:Install 4 Packages 2024-09-10T18:11:53.700 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:11:53.701 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 48 M 2024-09-10T18:11:53.701 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 193 M 2024-09-10T18:11:53.701 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:11:54.102 INFO:teuthology.orchestra.run.smithi028.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.2 MB/s | 304 kB 00:00 2024-09-10T18:11:54.144 INFO:teuthology.orchestra.run.smithi028.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-10T18:11:54.194 INFO:teuthology.orchestra.run.smithi028.stdout:(3/4): libcephsqlite-17.2.7-1606.g5be57395.el9. 709 kB/s | 163 kB 00:00 2024-09-10T18:11:54.430 INFO:teuthology.orchestra.run.smithi204.stdout:(4/4): ceph-test-17.2.7-1606.g5be57395.el9.x86_ 25 MB/s | 48 MB 00:01 2024-09-10T18:11:54.432 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:11:54.432 INFO:teuthology.orchestra.run.smithi204.stdout:Total 24 MB/s | 48 MB 00:02 2024-09-10T18:11:54.460 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:11:54.479 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:11:54.479 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:11:54.846 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:11:54.847 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:11:55.366 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:11:55.446 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-10T18:11:55.495 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-10T18:11:55.523 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 3/4 2024-09-10T18:11:55.853 INFO:teuthology.orchestra.run.smithi028.stdout:(4/4): ceph-test-17.2.7-1606.g5be57395.el9.x86_ 25 MB/s | 48 MB 00:01 2024-09-10T18:11:55.855 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:11:55.855 INFO:teuthology.orchestra.run.smithi028.stdout:Total 22 MB/s | 48 MB 00:02 2024-09-10T18:11:55.884 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:11:55.905 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:11:55.905 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:11:56.291 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:11:56.291 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:11:56.836 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:11:56.921 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-10T18:11:56.960 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-10T18:11:56.988 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 3/4 2024-09-10T18:11:59.952 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 3/4 2024-09-10T18:12:00.030 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 4/4 2024-09-10T18:12:01.639 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 3/4 2024-09-10T18:12:01.721 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 4/4 2024-09-10T18:12:02.385 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 4/4 2024-09-10T18:12:02.385 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 1/4 2024-09-10T18:12:02.385 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 2/4 2024-09-10T18:12:02.385 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-10T18:12:02.705 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-10T18:12:02.705 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:02.705 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:12:02.705 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:12:02.705 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:12:02.705 INFO:teuthology.orchestra.run.smithi204.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T18:12:02.705 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T18:12:02.705 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:02.705 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:12:02.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph 2024-09-10T18:12:03.554 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:38 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:12:03.696 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:12:03.698 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:03.699 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-10T18:12:03.699 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:03.699 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:12:03.699 INFO:teuthology.orchestra.run.smithi204.stdout: ceph x86_64 2:17.2.7-1606.g5be57395.el9 ceph 6.5 k 2024-09-10T18:12:03.699 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-10T18:12:03.699 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.1 M 2024-09-10T18:12:03.699 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr x86_64 2:17.2.7-1606.g5be57395.el9 ceph 1.4 M 2024-09-10T18:12:03.699 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 240 k 2024-09-10T18:12:03.699 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon x86_64 2:17.2.7-1606.g5be57395.el9 ceph 3.9 M 2024-09-10T18:12:03.699 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 15 M 2024-09-10T18:12:03.699 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T18:12:03.699 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T18:12:03.699 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T18:12:03.700 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T18:12:03.700 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T18:12:03.700 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T18:12:03.700 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T18:12:03.700 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections 2024-09-10T18:12:03.700 INFO:teuthology.orchestra.run.smithi204.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-10T18:12:03.700 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T18:12:03.700 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T18:12:03.700 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T18:12:03.700 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T18:12:03.700 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T18:12:03.700 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T18:12:03.700 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T18:12:03.701 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T18:12:03.701 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T18:12:03.701 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T18:12:03.701 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T18:12:03.701 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T18:12:03.701 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T18:12:03.701 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T18:12:03.701 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T18:12:03.701 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T18:12:03.701 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T18:12:03.701 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T18:12:03.701 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T18:12:03.701 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T18:12:03.701 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:03.702 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:12:03.702 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:03.702 INFO:teuthology.orchestra.run.smithi204.stdout:Install 34 Packages 2024-09-10T18:12:03.702 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:03.702 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 27 M 2024-09-10T18:12:03.702 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 96 M 2024-09-10T18:12:03.702 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:12:03.888 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 4/4 2024-09-10T18:12:03.889 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 1/4 2024-09-10T18:12:03.889 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 2/4 2024-09-10T18:12:03.889 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-10T18:12:04.093 INFO:teuthology.orchestra.run.smithi204.stdout:(1/34): ceph-17.2.7-1606.g5be57395.el9.x86_64.r 50 kB/s | 6.5 kB 00:00 2024-09-10T18:12:04.182 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-10T18:12:04.182 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:04.182 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:12:04.182 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:12:04.182 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:12:04.182 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T18:12:04.182 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T18:12:04.182 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:04.182 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:12:04.277 INFO:teuthology.orchestra.run.smithi204.stdout:(2/34): ceph-mgr-17.2.7-1606.g5be57395.el9.x86_ 4.5 MB/s | 1.4 MB 00:00 2024-09-10T18:12:04.327 INFO:teuthology.orchestra.run.smithi204.stdout:(3/34): ceph-mds-17.2.7-1606.g5be57395.el9.x86_ 5.7 MB/s | 2.1 MB 00:00 2024-09-10T18:12:04.428 INFO:teuthology.orchestra.run.smithi204.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1606.g5be5 2.3 MB/s | 240 kB 00:00 2024-09-10T18:12:04.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph 2024-09-10T18:12:04.486 INFO:teuthology.orchestra.run.smithi204.stdout:(5/34): ceph-mon-17.2.7-1606.g5be57395.el9.x86_ 10 MB/s | 3.9 MB 00:00 2024-09-10T18:12:04.637 INFO:teuthology.orchestra.run.smithi204.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-10T18:12:04.803 INFO:teuthology.orchestra.run.smithi204.stdout:(7/34): ceph-osd-17.2.7-1606.g5be57395.el9.x86_ 29 MB/s | 15 MB 00:00 2024-09-10T18:12:04.820 INFO:teuthology.orchestra.run.smithi204.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 581 kB/s | 106 kB 00:00 2024-09-10T18:12:04.862 INFO:teuthology.orchestra.run.smithi204.stdout:(9/34): python3-cryptography-36.0.1-4.el9.x86_6 3.3 MB/s | 1.2 MB 00:00 2024-09-10T18:12:04.888 INFO:teuthology.orchestra.run.smithi204.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-09-10T18:12:04.913 INFO:teuthology.orchestra.run.smithi204.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-10T18:12:04.938 INFO:teuthology.orchestra.run.smithi204.stdout:(12/34): python3-pycparser-2.20-6.el9.noarch.rp 1.0 MB/s | 135 kB 00:00 2024-09-10T18:12:05.022 INFO:teuthology.orchestra.run.smithi204.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 319 kB/s | 35 kB 00:00 2024-09-10T18:12:05.036 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:39 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:12:05.047 INFO:teuthology.orchestra.run.smithi204.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-10T18:12:05.072 INFO:teuthology.orchestra.run.smithi204.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 576 kB/s | 77 kB 00:00 2024-09-10T18:12:05.115 INFO:teuthology.orchestra.run.smithi204.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 607 kB/s | 51 kB 00:00 2024-09-10T18:12:05.131 INFO:teuthology.orchestra.run.smithi204.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 497 kB/s | 42 kB 00:00 2024-09-10T18:12:05.156 INFO:teuthology.orchestra.run.smithi204.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 519 kB/s | 43 kB 00:00 2024-09-10T18:12:05.184 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:12:05.186 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:05.186 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-09-10T18:12:05.186 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:05.186 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:12:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:17.2.7-1606.g5be57395.el9 ceph 6.5 k 2024-09-10T18:12:05.187 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-10T18:12:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.1 M 2024-09-10T18:12:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:17.2.7-1606.g5be57395.el9 ceph 1.4 M 2024-09-10T18:12:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 240 k 2024-09-10T18:12:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:17.2.7-1606.g5be57395.el9 ceph 3.9 M 2024-09-10T18:12:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 15 M 2024-09-10T18:12:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T18:12:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T18:12:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T18:12:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T18:12:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T18:12:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T18:12:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T18:12:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections 2024-09-10T18:12:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-10T18:12:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T18:12:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T18:12:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T18:12:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T18:12:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T18:12:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T18:12:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T18:12:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T18:12:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T18:12:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T18:12:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T18:12:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T18:12:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T18:12:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T18:12:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T18:12:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T18:12:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T18:12:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T18:12:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T18:12:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T18:12:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:05.189 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:12:05.190 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:05.190 INFO:teuthology.orchestra.run.smithi028.stdout:Install 34 Packages 2024-09-10T18:12:05.190 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:05.190 INFO:teuthology.orchestra.run.smithi204.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 439 kB/s | 11 kB 00:00 2024-09-10T18:12:05.191 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 27 M 2024-09-10T18:12:05.191 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 96 M 2024-09-10T18:12:05.191 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:12:05.215 INFO:teuthology.orchestra.run.smithi204.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-09-10T18:12:05.232 INFO:teuthology.orchestra.run.smithi204.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 442 kB/s | 18 kB 00:00 2024-09-10T18:12:05.249 INFO:teuthology.orchestra.run.smithi204.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 691 kB/s | 23 kB 00:00 2024-09-10T18:12:05.266 INFO:teuthology.orchestra.run.smithi204.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 578 kB/s | 19 kB 00:00 2024-09-10T18:12:05.283 INFO:teuthology.orchestra.run.smithi204.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 589 kB/s | 20 kB 00:00 2024-09-10T18:12:05.300 INFO:teuthology.orchestra.run.smithi204.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-09-10T18:12:05.326 INFO:teuthology.orchestra.run.smithi204.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-10T18:12:05.342 INFO:teuthology.orchestra.run.smithi204.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 998 kB/s | 16 kB 00:00 2024-09-10T18:12:05.367 INFO:teuthology.orchestra.run.smithi204.stdout:(28/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-10T18:12:05.393 INFO:teuthology.orchestra.run.smithi204.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-10T18:12:05.418 INFO:teuthology.orchestra.run.smithi204.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-10T18:12:05.443 INFO:teuthology.orchestra.run.smithi204.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-10T18:12:05.469 INFO:teuthology.orchestra.run.smithi204.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 798 kB/s | 20 kB 00:00 2024-09-10T18:12:05.494 INFO:teuthology.orchestra.run.smithi204.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-10T18:12:05.519 INFO:teuthology.orchestra.run.smithi204.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-10T18:12:05.524 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:12:05.524 INFO:teuthology.orchestra.run.smithi204.stdout:Total 15 MB/s | 27 MB 00:01 2024-09-10T18:12:05.654 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:12:05.691 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:12:05.691 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:12:05.830 INFO:teuthology.orchestra.run.smithi028.stdout:(1/34): ceph-17.2.7-1606.g5be57395.el9.x86_64.r 49 kB/s | 6.5 kB 00:00 2024-09-10T18:12:05.972 INFO:teuthology.orchestra.run.smithi028.stdout:(2/34): ceph-mgr-17.2.7-1606.g5be57395.el9.x86_ 5.1 MB/s | 1.4 MB 00:00 2024-09-10T18:12:06.039 INFO:teuthology.orchestra.run.smithi028.stdout:(3/34): ceph-mds-17.2.7-1606.g5be57395.el9.x86_ 6.1 MB/s | 2.1 MB 00:00 2024-09-10T18:12:06.076 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:12:06.076 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:12:06.123 INFO:teuthology.orchestra.run.smithi028.stdout:(4/34): ceph-mon-17.2.7-1606.g5be57395.el9.x86_ 13 MB/s | 3.9 MB 00:00 2024-09-10T18:12:06.148 INFO:teuthology.orchestra.run.smithi028.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1606.g5be5 2.2 MB/s | 240 kB 00:00 2024-09-10T18:12:06.248 INFO:teuthology.orchestra.run.smithi028.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-09-10T18:12:06.290 INFO:teuthology.orchestra.run.smithi028.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-10T18:12:06.332 INFO:teuthology.orchestra.run.smithi028.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-10T18:12:06.374 INFO:teuthology.orchestra.run.smithi028.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-10T18:12:06.407 INFO:teuthology.orchestra.run.smithi028.stdout:(10/34): python3-cryptography-36.0.1-4.el9.x86_ 4.7 MB/s | 1.2 MB 00:00 2024-09-10T18:12:06.558 INFO:teuthology.orchestra.run.smithi028.stdout:(11/34): ceph-osd-17.2.7-1606.g5be57395.el9.x86 26 MB/s | 15 MB 00:00 2024-09-10T18:12:06.583 INFO:teuthology.orchestra.run.smithi028.stdout:(12/34): python3-urllib3-1.26.5-6.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-09-10T18:12:06.650 INFO:teuthology.orchestra.run.smithi028.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 378 kB/s | 35 kB 00:00 2024-09-10T18:12:06.675 INFO:teuthology.orchestra.run.smithi028.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 644 kB/s | 172 kB 00:00 2024-09-10T18:12:06.700 INFO:teuthology.orchestra.run.smithi028.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 659 kB/s | 77 kB 00:00 2024-09-10T18:12:06.726 INFO:teuthology.orchestra.run.smithi028.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 676 kB/s | 51 kB 00:00 2024-09-10T18:12:06.784 INFO:teuthology.orchestra.run.smithi028.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 383 kB/s | 42 kB 00:00 2024-09-10T18:12:06.835 INFO:teuthology.orchestra.run.smithi028.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 325 kB/s | 43 kB 00:00 2024-09-10T18:12:06.860 INFO:teuthology.orchestra.run.smithi028.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 466 kB/s | 11 kB 00:00 2024-09-10T18:12:06.885 INFO:teuthology.orchestra.run.smithi028.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 706 kB/s | 18 kB 00:00 2024-09-10T18:12:06.911 INFO:teuthology.orchestra.run.smithi028.stdout:(21/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 931 kB/s | 172 kB 00:00 2024-09-10T18:12:06.936 INFO:teuthology.orchestra.run.smithi028.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 459 kB/s | 23 kB 00:00 2024-09-10T18:12:06.961 INFO:teuthology.orchestra.run.smithi028.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-10T18:12:06.986 INFO:teuthology.orchestra.run.smithi028.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-10T18:12:07.012 INFO:teuthology.orchestra.run.smithi028.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-09-10T18:12:07.037 INFO:teuthology.orchestra.run.smithi028.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-10T18:12:07.062 INFO:teuthology.orchestra.run.smithi028.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-10T18:12:07.087 INFO:teuthology.orchestra.run.smithi028.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-10T18:12:07.113 INFO:teuthology.orchestra.run.smithi028.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-10T18:12:07.138 INFO:teuthology.orchestra.run.smithi028.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-10T18:12:07.163 INFO:teuthology.orchestra.run.smithi028.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-10T18:12:07.180 INFO:teuthology.orchestra.run.smithi028.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-10T18:12:07.205 INFO:teuthology.orchestra.run.smithi028.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-10T18:12:07.230 INFO:teuthology.orchestra.run.smithi028.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-10T18:12:07.234 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:12:07.234 INFO:teuthology.orchestra.run.smithi028.stdout:Total 13 MB/s | 27 MB 00:02 2024-09-10T18:12:07.292 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:12:07.347 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-10T18:12:07.366 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:12:07.379 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-10T18:12:07.406 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:12:07.406 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:12:07.412 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-10T18:12:07.505 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-10T18:12:07.583 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-10T18:12:07.626 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-10T18:12:07.661 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-10T18:12:07.691 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-10T18:12:07.726 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-10T18:12:07.759 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-10T18:12:07.800 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-10T18:12:07.806 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:12:07.807 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:12:07.834 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-10T18:12:07.869 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-10T18:12:07.918 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-10T18:12:07.956 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-10T18:12:08.031 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-10T18:12:08.180 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-10T18:12:08.258 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-10T18:12:08.313 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-10T18:12:08.352 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-10T18:12:08.418 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-10T18:12:08.481 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-10T18:12:08.670 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-10T18:12:08.722 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-10T18:12:08.801 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-10T18:12:08.890 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-10T18:12:08.957 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-10T18:12:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:12:09.072 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-10T18:12:09.117 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-10T18:12:09.158 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-10T18:12:09.191 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-10T18:12:09.225 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9. 29/34 2024-09-10T18:12:09.257 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 30/34 2024-09-10T18:12:09.282 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 30/34 2024-09-10T18:12:09.282 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:12:09.282 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T18:12:09.282 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T18:12:09.283 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T18:12:09.283 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:09.288 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-10T18:12:09.363 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-10T18:12:09.411 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-10T18:12:09.445 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-10T18:12:09.475 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-10T18:12:09.509 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-10T18:12:09.541 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-10T18:12:09.573 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-10T18:12:09.616 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-10T18:12:09.650 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-10T18:12:09.700 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-10T18:12:09.745 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-10T18:12:09.824 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-10T18:12:09.976 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-10T18:12:10.059 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-10T18:12:10.118 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-10T18:12:10.155 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-10T18:12:10.213 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-10T18:12:10.277 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-10T18:12:10.484 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-10T18:12:10.543 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-10T18:12:10.624 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-10T18:12:10.650 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 31/34 2024-09-10T18:12:10.678 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 31/34 2024-09-10T18:12:10.678 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:12:10.679 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T18:12:10.679 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T18:12:10.679 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T18:12:10.679 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:10.721 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-10T18:12:10.764 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-10T18:12:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-10T18:12:11.043 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9. 29/34 2024-09-10T18:12:11.067 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 30/34 2024-09-10T18:12:11.068 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 32/34 2024-09-10T18:12:11.092 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 32/34 2024-09-10T18:12:11.092 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:12:11.092 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T18:12:11.093 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T18:12:11.093 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T18:12:11.093 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 30/34 2024-09-10T18:12:11.095 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:12:11.095 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T18:12:11.095 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T18:12:11.095 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T18:12:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:11.301 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 33/34 2024-09-10T18:12:11.325 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 33/34 2024-09-10T18:12:11.326 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:12:11.326 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T18:12:11.326 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T18:12:11.326 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T18:12:11.326 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:11.389 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-2:17.2.7-1606.g5be57395.el9.x86_64 34/34 2024-09-10T18:12:12.485 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 31/34 2024-09-10T18:12:12.513 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 31/34 2024-09-10T18:12:12.513 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:12:12.514 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T18:12:12.514 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T18:12:12.514 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T18:12:12.514 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 32/34 2024-09-10T18:12:12.928 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 32/34 2024-09-10T18:12:12.928 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:12:12.928 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T18:12:12.928 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T18:12:12.928 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T18:12:12.928 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:13.110 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-2:17.2.7-1606.g5be57395.el9.x86_64 34/34 2024-09-10T18:12:13.110 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-2:17.2.7-1606.g5be57395.el9.x86_64 1/34 2024-09-10T18:12:13.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 2/34 2024-09-10T18:12:13.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 3/34 2024-09-10T18:12:13.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 4/34 2024-09-10T18:12:13.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 5/34 2024-09-10T18:12:13.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9. 6/34 2024-09-10T18:12:13.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-10T18:12:13.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-10T18:12:13.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-10T18:12:13.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-10T18:12:13.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-10T18:12:13.112 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-10T18:12:13.112 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-10T18:12:13.112 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-10T18:12:13.112 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-10T18:12:13.112 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-10T18:12:13.112 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-10T18:12:13.112 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-10T18:12:13.112 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-10T18:12:13.112 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-10T18:12:13.112 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-10T18:12:13.112 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-10T18:12:13.112 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-10T18:12:13.112 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-10T18:12:13.112 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-10T18:12:13.113 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-10T18:12:13.113 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-10T18:12:13.113 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-10T18:12:13.113 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-10T18:12:13.113 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-10T18:12:13.113 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-10T18:12:13.113 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-10T18:12:13.113 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-10T18:12:13.144 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 33/34 2024-09-10T18:12:13.170 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 33/34 2024-09-10T18:12:13.170 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:12:13.170 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T18:12:13.170 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T18:12:13.170 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T18:12:13.170 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:13.236 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:17.2.7-1606.g5be57395.el9.x86_64 34/34 2024-09-10T18:12:14.162 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-10T18:12:14.163 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:14.163 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:12:14.163 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:12:14.163 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:12:14.163 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:12:14.163 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:12:14.163 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:12:14.163 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:12:14.163 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T18:12:14.163 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T18:12:14.163 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T18:12:14.163 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T18:12:14.163 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T18:12:14.164 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T18:12:14.164 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T18:12:14.164 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T18:12:14.164 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T18:12:14.164 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T18:12:14.164 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T18:12:14.164 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T18:12:14.164 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T18:12:14.164 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T18:12:14.164 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T18:12:14.164 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T18:12:14.164 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T18:12:14.164 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T18:12:14.164 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T18:12:14.165 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T18:12:14.165 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T18:12:14.165 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T18:12:14.165 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T18:12:14.165 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T18:12:14.165 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T18:12:14.165 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T18:12:14.165 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T18:12:14.165 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T18:12:14.165 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:14.165 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:12:14.575 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-base 2024-09-10T18:12:14.860 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-2:17.2.7-1606.g5be57395.el9.x86_64 34/34 2024-09-10T18:12:14.861 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:17.2.7-1606.g5be57395.el9.x86_64 1/34 2024-09-10T18:12:14.861 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 2/34 2024-09-10T18:12:14.861 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 3/34 2024-09-10T18:12:14.861 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 4/34 2024-09-10T18:12:14.861 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 5/34 2024-09-10T18:12:14.861 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9. 6/34 2024-09-10T18:12:14.861 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-10T18:12:14.861 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-10T18:12:14.861 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-10T18:12:14.861 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-10T18:12:14.861 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-10T18:12:14.862 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-10T18:12:14.862 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-10T18:12:14.862 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-10T18:12:14.862 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-10T18:12:14.862 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-10T18:12:14.862 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-10T18:12:14.862 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-10T18:12:14.862 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-10T18:12:14.862 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-10T18:12:14.862 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-10T18:12:14.862 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-10T18:12:14.862 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-10T18:12:14.862 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-10T18:12:14.863 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-10T18:12:14.863 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-10T18:12:14.863 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-10T18:12:14.863 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-10T18:12:14.863 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-10T18:12:14.863 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-10T18:12:14.863 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-10T18:12:14.863 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-10T18:12:14.863 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-10T18:12:15.145 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:50 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:12:15.237 INFO:teuthology.orchestra.run.smithi204.stdout:Package ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:12:15.279 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:12:15.280 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-10T18:12:15.280 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:12:15.344 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install cephadm 2024-09-10T18:12:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-10T18:12:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:15.747 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:12:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:12:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:12:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:12:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:12:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:12:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:12:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T18:12:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T18:12:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T18:12:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T18:12:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T18:12:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T18:12:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T18:12:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T18:12:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T18:12:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T18:12:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T18:12:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T18:12:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T18:12:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T18:12:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T18:12:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T18:12:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T18:12:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T18:12:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T18:12:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T18:12:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T18:12:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T18:12:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T18:12:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T18:12:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T18:12:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T18:12:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T18:12:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T18:12:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:15.750 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:12:15.909 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:50 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:12:16.040 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:12:16.041 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:16.041 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-10T18:12:16.041 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:16.041 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:12:16.041 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 83 k 2024-09-10T18:12:16.041 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:16.041 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:12:16.041 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:16.041 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-10T18:12:16.041 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:16.041 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 83 k 2024-09-10T18:12:16.041 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 375 k 2024-09-10T18:12:16.041 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:12:16.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-base 2024-09-10T18:12:16.234 INFO:teuthology.orchestra.run.smithi204.stdout:cephadm-17.2.7-1606.g5be57395.el9.noarch.rpm 429 kB/s | 83 kB 00:00 2024-09-10T18:12:16.235 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:12:16.235 INFO:teuthology.orchestra.run.smithi204.stdout:Total 426 kB/s | 83 kB 00:00 2024-09-10T18:12:16.235 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:12:16.240 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:12:16.240 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:12:16.242 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:12:16.243 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:12:16.320 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:12:16.711 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:50 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:12:16.759 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:12:16.803 INFO:teuthology.orchestra.run.smithi028.stdout:Package ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:12:16.847 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:12:16.848 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-09-10T18:12:16.848 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:12:16.863 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:12:16.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install cephadm 2024-09-10T18:12:17.187 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:12:17.437 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:12:17.437 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:17.437 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:12:17.437 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:12:17.437 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:17.437 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:12:17.510 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:51 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:12:17.647 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:12:17.647 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:17.647 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-09-10T18:12:17.647 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:17.647 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:12:17.647 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 83 k 2024-09-10T18:12:17.647 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:17.648 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:12:17.648 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:17.648 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-09-10T18:12:17.648 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:17.648 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 83 k 2024-09-10T18:12:17.648 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 375 k 2024-09-10T18:12:17.649 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:12:17.671 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-immutable-object-cache 2024-09-10T18:12:17.834 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm-17.2.7-1606.g5be57395.el9.noarch.rpm 445 kB/s | 83 kB 00:00 2024-09-10T18:12:17.835 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:12:17.835 INFO:teuthology.orchestra.run.smithi028.stdout:Total 442 kB/s | 83 kB 00:00 2024-09-10T18:12:17.835 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:12:17.841 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:12:17.841 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:12:17.842 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:12:17.843 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:12:17.929 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:12:18.248 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:53 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:12:18.384 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:12:18.385 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:12:18.386 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:18.386 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-09-10T18:12:18.386 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:18.386 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:12:18.386 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1606.g5be57395.el9 ceph 137 k 2024-09-10T18:12:18.386 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:18.386 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:12:18.386 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:18.386 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-10T18:12:18.386 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:18.387 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 137 k 2024-09-10T18:12:18.387 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 379 k 2024-09-10T18:12:18.387 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:12:18.458 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:12:18.591 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-immutable-object-cache-17.2.7-1606.g5be573 675 kB/s | 137 kB 00:00 2024-09-10T18:12:18.591 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:12:18.592 INFO:teuthology.orchestra.run.smithi204.stdout:Total 670 kB/s | 137 kB 00:00 2024-09-10T18:12:18.592 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:12:18.599 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:12:18.599 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:12:18.661 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:12:18.661 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:12:18.817 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:12:18.825 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:12:18.875 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T18:12:18.900 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T18:12:18.900 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:12:18.900 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T18:12:18.900 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:19.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:12:19.047 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:19.047 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:12:19.047 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:12:19.047 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:19.047 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:12:19.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-immutable-object-cache 2024-09-10T18:12:19.831 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T18:12:19.831 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:19.831 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:12:19.831 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:12:19.831 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:19.831 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:12:19.858 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:53 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:12:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:12:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:19.999 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-09-10T18:12:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:12:19.999 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1606.g5be57395.el9 ceph 137 k 2024-09-10T18:12:20.000 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:12:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-09-10T18:12:20.000 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 137 k 2024-09-10T18:12:20.001 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 379 k 2024-09-10T18:12:20.001 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:12:20.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr 2024-09-10T18:12:20.206 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-immutable-object-cache-17.2.7-1606.g5be573 664 kB/s | 137 kB 00:00 2024-09-10T18:12:20.206 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:12:20.207 INFO:teuthology.orchestra.run.smithi028.stdout:Total 660 kB/s | 137 kB 00:00 2024-09-10T18:12:20.207 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:12:20.215 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:12:20.216 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:12:20.280 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:12:20.280 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:12:20.448 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:12:20.524 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T18:12:20.551 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T18:12:20.552 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:12:20.552 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T18:12:20.552 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:20.607 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:55 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:12:20.701 INFO:teuthology.orchestra.run.smithi204.stdout:Package ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:12:20.744 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:12:20.745 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-10T18:12:20.745 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:12:20.808 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-dashboard 2024-09-10T18:12:21.379 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:56 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:12:21.496 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T18:12:21.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:21.496 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:12:21.497 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:12:21.497 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:21.497 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:12:21.514 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:12:21.515 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:21.515 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-10T18:12:21.515 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:21.515 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:12:21.515 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 3.4 M 2024-09-10T18:12:21.515 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-10T18:12:21.516 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 24 k 2024-09-10T18:12:21.516 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 15 k 2024-09-10T18:12:21.516 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-10T18:12:21.516 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T18:12:21.516 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T18:12:21.516 INFO:teuthology.orchestra.run.smithi204.stdout:Installing weak dependencies: 2024-09-10T18:12:21.516 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-10T18:12:21.516 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:21.516 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:12:21.516 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:21.516 INFO:teuthology.orchestra.run.smithi204.stdout:Install 7 Packages 2024-09-10T18:12:21.516 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:21.517 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 3.7 M 2024-09-10T18:12:21.517 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 80 M 2024-09-10T18:12:21.517 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:12:21.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr 2024-09-10T18:12:21.845 INFO:teuthology.orchestra.run.smithi204.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1606.g5be 173 kB/s | 24 kB 00:00 2024-09-10T18:12:21.862 INFO:teuthology.orchestra.run.smithi204.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1606.g5be5 92 kB/s | 15 kB 00:00 2024-09-10T18:12:21.895 INFO:teuthology.orchestra.run.smithi204.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 180 kB/s | 9.0 kB 00:00 2024-09-10T18:12:21.912 INFO:teuthology.orchestra.run.smithi204.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 2.0 MB/s | 31 kB 00:00 2024-09-10T18:12:21.929 INFO:teuthology.orchestra.run.smithi204.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 608 kB/s | 41 kB 00:00 2024-09-10T18:12:21.963 INFO:teuthology.orchestra.run.smithi204.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-10T18:12:22.080 INFO:teuthology.orchestra.run.smithi204.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1606.g5be57395 8.9 MB/s | 3.4 MB 00:00 2024-09-10T18:12:22.084 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:12:22.084 INFO:teuthology.orchestra.run.smithi204.stdout:Total 6.4 MB/s | 3.7 MB 00:00 2024-09-10T18:12:22.084 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:12:22.092 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:12:22.093 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:12:22.177 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:12:22.178 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:12:22.311 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:56 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:12:22.405 INFO:teuthology.orchestra.run.smithi028.stdout:Package ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:12:22.449 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:12:22.450 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-09-10T18:12:22.451 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:12:22.476 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:12:22.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-dashboard 2024-09-10T18:12:22.573 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-10T18:12:22.607 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-10T18:12:22.662 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-10T18:12:22.694 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-10T18:12:22.723 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.n 5/7 2024-09-10T18:12:23.103 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:57 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:12:23.242 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:12:23.242 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:23.242 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-09-10T18:12:23.243 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:23.243 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:12:23.243 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 3.4 M 2024-09-10T18:12:23.243 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-10T18:12:23.243 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 24 k 2024-09-10T18:12:23.243 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 15 k 2024-09-10T18:12:23.243 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-10T18:12:23.243 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T18:12:23.243 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T18:12:23.243 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-09-10T18:12:23.243 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-10T18:12:23.243 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:23.244 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:12:23.244 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:23.244 INFO:teuthology.orchestra.run.smithi028.stdout:Install 7 Packages 2024-09-10T18:12:23.244 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:23.244 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 3.7 M 2024-09-10T18:12:23.244 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 80 M 2024-09-10T18:12:23.245 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:12:23.450 INFO:teuthology.orchestra.run.smithi028.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1606.g5be 168 kB/s | 24 kB 00:00 2024-09-10T18:12:23.475 INFO:teuthology.orchestra.run.smithi028.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1606.g5be5 86 kB/s | 15 kB 00:00 2024-09-10T18:12:23.518 INFO:teuthology.orchestra.run.smithi028.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-10T18:12:23.543 INFO:teuthology.orchestra.run.smithi028.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-10T18:12:23.568 INFO:teuthology.orchestra.run.smithi028.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2024-09-10T18:12:23.593 INFO:teuthology.orchestra.run.smithi028.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-10T18:12:23.702 INFO:teuthology.orchestra.run.smithi028.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1606.g5be57395 8.5 MB/s | 3.4 MB 00:00 2024-09-10T18:12:23.705 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:12:23.706 INFO:teuthology.orchestra.run.smithi028.stdout:Total 7.9 MB/s | 3.7 MB 00:00 2024-09-10T18:12:23.706 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:12:23.714 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:12:23.715 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:12:23.719 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9. 6/7 2024-09-10T18:12:23.760 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 7/7 2024-09-10T18:12:23.802 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:12:23.803 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:12:24.114 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:12:24.178 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-10T18:12:24.213 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-10T18:12:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-10T18:12:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-10T18:12:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.n 5/7 2024-09-10T18:12:24.769 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 7/7 2024-09-10T18:12:24.769 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9. 1/7 2024-09-10T18:12:24.769 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 2/7 2024-09-10T18:12:24.769 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.n 3/7 2024-09-10T18:12:24.770 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-10T18:12:24.770 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-10T18:12:24.770 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-10T18:12:25.044 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-10T18:12:25.044 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:25.045 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:12:25.045 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:12:25.045 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:12:25.045 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:12:25.045 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-10T18:12:25.045 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-10T18:12:25.045 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T18:12:25.045 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T18:12:25.045 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:25.045 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:12:25.251 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-10T18:12:25.384 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9. 6/7 2024-09-10T18:12:25.417 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 7/7 2024-09-10T18:12:25.820 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:00 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:12:25.955 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:12:25.956 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-09-10T18:12:25.956 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-10T18:12:25.956 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-09-10T18:12:25.956 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:12:25.957 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 7.4 M 2024-09-10T18:12:25.957 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-10T18:12:25.957 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T18:12:25.957 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T18:12:25.957 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T18:12:25.957 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T18:12:25.957 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T18:12:25.957 INFO:teuthology.orchestra.run.smithi204.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T18:12:25.957 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T18:12:25.957 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T18:12:25.957 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T18:12:25.957 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T18:12:25.957 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T18:12:25.958 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:12:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-09-10T18:12:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:Install 12 Packages 2024-09-10T18:12:25.958 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 38 M 2024-09-10T18:12:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 212 M 2024-09-10T18:12:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:12:26.230 INFO:teuthology.orchestra.run.smithi204.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-10T18:12:26.305 INFO:teuthology.orchestra.run.smithi204.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.8 MB/s | 801 kB 00:00 2024-09-10T18:12:26.464 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 7/7 2024-09-10T18:12:26.465 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9. 1/7 2024-09-10T18:12:26.465 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 2/7 2024-09-10T18:12:26.465 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.n 3/7 2024-09-10T18:12:26.465 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-10T18:12:26.465 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-10T18:12:26.465 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-10T18:12:26.539 INFO:teuthology.orchestra.run.smithi204.stdout:(3/12): ceph-mgr-diskprediction-local-17.2.7-16 17 MB/s | 7.4 MB 00:00 2024-09-10T18:12:26.564 INFO:teuthology.orchestra.run.smithi204.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 98 kB/s | 33 kB 00:00 2024-09-10T18:12:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-10T18:12:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:26.739 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:12:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:12:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:12:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:12:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-10T18:12:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-10T18:12:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T18:12:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T18:12:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:26.740 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:12:26.756 INFO:teuthology.orchestra.run.smithi204.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 201 kB/s | 38 kB 00:00 2024-09-10T18:12:26.823 INFO:teuthology.orchestra.run.smithi204.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 64 kB/s | 18 kB 00:00 2024-09-10T18:12:26.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-10T18:12:27.290 INFO:teuthology.orchestra.run.smithi204.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 524 kB/s | 245 kB 00:00 2024-09-10T18:12:27.516 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:01 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:12:27.599 INFO:teuthology.orchestra.run.smithi204.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.3 MB/s | 3.0 MB 00:01 2024-09-10T18:12:27.655 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:12:27.656 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-09-10T18:12:27.656 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-09-10T18:12:27.656 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-09-10T18:12:27.656 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:12:27.656 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 7.4 M 2024-09-10T18:12:27.657 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-10T18:12:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T18:12:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T18:12:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T18:12:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T18:12:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T18:12:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T18:12:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T18:12:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T18:12:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T18:12:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T18:12:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T18:12:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:27.657 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:12:27.658 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-09-10T18:12:27.658 INFO:teuthology.orchestra.run.smithi028.stdout:Install 12 Packages 2024-09-10T18:12:27.658 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:27.658 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 38 M 2024-09-10T18:12:27.658 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 212 M 2024-09-10T18:12:27.658 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:12:27.707 INFO:teuthology.orchestra.run.smithi204.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 250 kB 00:00 2024-09-10T18:12:27.924 INFO:teuthology.orchestra.run.smithi204.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.2 MB/s | 4.9 MB 00:01 2024-09-10T18:12:28.266 INFO:teuthology.orchestra.run.smithi204.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.2 MB/s | 5.1 MB 00:00 2024-09-10T18:12:28.361 INFO:teuthology.orchestra.run.smithi028.stdout:(1/12): ceph-mgr-diskprediction-local-17.2.7-16 16 MB/s | 7.4 MB 00:00 2024-09-10T18:12:28.453 INFO:teuthology.orchestra.run.smithi028.stdout:(2/12): libquadmath-11.5.0-2.el9.x86_64.rpm 346 kB/s | 191 kB 00:00 2024-09-10T18:12:28.737 INFO:teuthology.orchestra.run.smithi028.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 958 kB/s | 801 kB 00:00 2024-09-10T18:12:28.979 INFO:teuthology.orchestra.run.smithi028.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 53 kB/s | 33 kB 00:00 2024-09-10T18:12:29.279 INFO:teuthology.orchestra.run.smithi028.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 34 kB/s | 18 kB 00:00 2024-09-10T18:12:29.537 INFO:teuthology.orchestra.run.smithi028.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.8 MB/s | 3.0 MB 00:01 2024-09-10T18:12:29.563 INFO:teuthology.orchestra.run.smithi028.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 67 kB/s | 38 kB 00:00 2024-09-10T18:12:30.296 INFO:teuthology.orchestra.run.smithi028.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 326 kB/s | 245 kB 00:00 2024-09-10T18:12:30.463 INFO:teuthology.orchestra.run.smithi028.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 4.2 MB/s | 4.9 MB 00:01 2024-09-10T18:12:30.680 INFO:teuthology.orchestra.run.smithi028.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.6 MB/s | 5.1 MB 00:01 2024-09-10T18:12:31.147 INFO:teuthology.orchestra.run.smithi028.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 295 kB/s | 250 kB 00:00 2024-09-10T18:12:32.164 INFO:teuthology.orchestra.run.smithi028.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.6 MB/s | 16 MB 00:01 2024-09-10T18:12:32.166 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:12:32.166 INFO:teuthology.orchestra.run.smithi028.stdout:Total 8.5 MB/s | 38 MB 00:04 2024-09-10T18:12:32.537 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:12:32.564 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:12:32.564 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:12:33.087 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:12:33.088 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:12:33.875 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:12:33.935 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-10T18:12:33.983 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-10T18:12:34.083 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-10T18:12:34.122 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-10T18:12:34.314 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-10T18:12:34.337 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-10T18:12:34.416 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-10T18:12:35.101 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-10T18:12:35.229 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-10T18:12:36.555 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T18:12:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-10T18:12:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 12/12 2024-09-10T18:12:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 12/12 2024-09-10T18:12:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 1/12 2024-09-10T18:12:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-10T18:12:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-10T18:12:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-10T18:12:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-10T18:12:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-10T18:12:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-10T18:12:40.168 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-10T18:12:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-10T18:12:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T18:12:40.169 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-10T18:12:40.594 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-10T18:12:40.594 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:40.594 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:12:40.595 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:12:40.595 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T18:12:40.595 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T18:12:40.595 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T18:12:40.595 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T18:12:40.595 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T18:12:40.595 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T18:12:40.595 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T18:12:40.595 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T18:12:40.595 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T18:12:40.595 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T18:12:40.595 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T18:12:40.595 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:40.596 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:12:40.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-rook 2024-09-10T18:12:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:15 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:12:41.736 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:12:41.737 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:41.737 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-09-10T18:12:41.737 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:41.737 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:12:41.737 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 47 k 2024-09-10T18:12:41.737 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-10T18:12:41.737 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T18:12:41.738 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T18:12:41.738 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T18:12:41.738 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T18:12:41.738 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T18:12:41.738 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T18:12:41.738 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T18:12:41.738 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T18:12:41.738 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T18:12:41.738 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T18:12:41.738 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T18:12:41.738 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T18:12:41.738 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:41.738 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:12:41.739 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:41.739 INFO:teuthology.orchestra.run.smithi028.stdout:Install 13 Packages 2024-09-10T18:12:41.739 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:41.739 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 2.2 M 2024-09-10T18:12:41.739 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 26 M 2024-09-10T18:12:41.739 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:12:42.201 INFO:teuthology.orchestra.run.smithi028.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 212 kB/s | 26 kB 00:00 2024-09-10T18:12:42.226 INFO:teuthology.orchestra.run.smithi028.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 131 kB/s | 19 kB 00:00 2024-09-10T18:12:42.251 INFO:teuthology.orchestra.run.smithi028.stdout:(3/13): ceph-mgr-rook-17.2.7-1606.g5be57395.el9 277 kB/s | 47 kB 00:00 2024-09-10T18:12:42.352 INFO:teuthology.orchestra.run.smithi028.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 222 kB 00:00 2024-09-10T18:12:42.402 INFO:teuthology.orchestra.run.smithi028.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 906 kB/s | 159 kB 00:00 2024-09-10T18:12:42.427 INFO:teuthology.orchestra.run.smithi028.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 713 kB/s | 54 kB 00:00 2024-09-10T18:12:42.469 INFO:teuthology.orchestra.run.smithi028.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-09-10T18:12:42.494 INFO:teuthology.orchestra.run.smithi028.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 350 kB/s | 32 kB 00:00 2024-09-10T18:12:42.528 INFO:teuthology.orchestra.run.smithi028.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 141 kB/s | 14 kB 00:00 2024-09-10T18:12:42.553 INFO:teuthology.orchestra.run.smithi028.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2024-09-10T18:12:42.579 INFO:teuthology.orchestra.run.smithi028.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-09-10T18:12:42.604 INFO:teuthology.orchestra.run.smithi028.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.6 MB/s | 220 kB 00:00 2024-09-10T18:12:42.637 INFO:teuthology.orchestra.run.smithi028.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.6 MB/s | 1.0 MB 00:00 2024-09-10T18:12:42.640 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:12:42.640 INFO:teuthology.orchestra.run.smithi028.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-09-10T18:12:42.709 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:12:42.726 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:12:42.727 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:12:42.822 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:12:42.823 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:12:42.957 INFO:teuthology.orchestra.run.smithi204.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 1.1 MB/s | 16 MB 00:15 2024-09-10T18:12:42.958 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:12:42.958 INFO:teuthology.orchestra.run.smithi204.stdout:Total 2.2 MB/s | 38 MB 00:17 2024-09-10T18:12:43.315 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:12:43.341 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:12:43.341 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:12:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:12:43.650 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-10T18:12:43.715 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-10T18:12:43.779 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-10T18:12:43.815 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-10T18:12:43.856 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-10T18:12:43.870 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:12:43.871 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:12:43.962 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-10T18:12:44.177 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-10T18:12:44.233 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-10T18:12:44.637 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:12:44.707 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-10T18:12:44.750 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-10T18:12:44.846 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-10T18:12:44.889 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-10T18:12:44.987 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-10T18:12:45.063 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-10T18:12:45.076 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-10T18:12:45.097 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-10T18:12:45.104 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-10T18:12:45.142 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-10T18:12:45.166 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 13/13 2024-09-10T18:12:45.186 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-10T18:12:45.830 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-10T18:12:45.899 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 13/13 2024-09-10T18:12:45.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 1/13 2024-09-10T18:12:45.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-10T18:12:45.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-10T18:12:45.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-10T18:12:45.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-10T18:12:45.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-10T18:12:45.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-10T18:12:45.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-10T18:12:45.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-10T18:12:45.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-10T18:12:45.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-10T18:12:45.900 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-10T18:12:45.951 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-10T18:12:46.227 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-10T18:12:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:46.228 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:12:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:12:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T18:12:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T18:12:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T18:12:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T18:12:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T18:12:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T18:12:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T18:12:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T18:12:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T18:12:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T18:12:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T18:12:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T18:12:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:46.229 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:12:46.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-cephadm 2024-09-10T18:12:47.040 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:21 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:12:47.180 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:12:47.180 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-09-10T18:12:47.181 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:47.181 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:12:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 124 k 2024-09-10T18:12:47.181 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-10T18:12:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T18:12:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T18:12:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T18:12:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T18:12:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T18:12:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:47.181 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:12:47.181 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:47.181 INFO:teuthology.orchestra.run.smithi028.stdout:Install 6 Packages 2024-09-10T18:12:47.182 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:47.182 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 7.0 M 2024-09-10T18:12:47.182 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 33 M 2024-09-10T18:12:47.182 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:12:47.197 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T18:12:47.480 INFO:teuthology.orchestra.run.smithi028.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-10T18:12:47.505 INFO:teuthology.orchestra.run.smithi028.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1606.g5be57395.e 522 kB/s | 124 kB 00:00 2024-09-10T18:12:47.563 INFO:teuthology.orchestra.run.smithi028.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 993 kB/s | 58 kB 00:00 2024-09-10T18:12:47.588 INFO:teuthology.orchestra.run.smithi028.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-10T18:12:47.614 INFO:teuthology.orchestra.run.smithi028.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.0 MB/s | 548 kB 00:00 2024-09-10T18:12:48.238 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-10T18:12:48.272 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 12/12 2024-09-10T18:12:49.397 INFO:teuthology.orchestra.run.smithi028.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.8 MB/s | 6.0 MB 00:02 2024-09-10T18:12:49.400 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:12:49.400 INFO:teuthology.orchestra.run.smithi028.stdout:Total 3.2 MB/s | 7.0 MB 00:02 2024-09-10T18:12:49.475 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:12:49.487 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:12:49.487 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:12:49.594 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:12:49.594 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:12:49.920 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:12:50.086 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-10T18:12:50.142 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-10T18:12:50.694 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-10T18:12:50.805 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-10T18:12:50.867 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-10T18:12:50.894 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 6/6 2024-09-10T18:12:50.900 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 12/12 2024-09-10T18:12:50.900 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 1/12 2024-09-10T18:12:50.900 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-10T18:12:50.900 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-10T18:12:50.900 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-10T18:12:50.900 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-10T18:12:50.901 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-10T18:12:50.901 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-10T18:12:50.901 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-10T18:12:50.901 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-10T18:12:50.901 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T18:12:50.901 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-10T18:12:51.289 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-10T18:12:51.290 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:51.290 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:12:51.290 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:12:51.290 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T18:12:51.290 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T18:12:51.290 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T18:12:51.290 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T18:12:51.290 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T18:12:51.290 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T18:12:51.290 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T18:12:51.290 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T18:12:51.291 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T18:12:51.291 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T18:12:51.291 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T18:12:51.291 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:51.291 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:12:51.617 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 6/6 2024-09-10T18:12:51.617 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/6 2024-09-10T18:12:51.617 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-10T18:12:51.617 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-10T18:12:51.617 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-10T18:12:51.618 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-10T18:12:51.702 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-rook 2024-09-10T18:12:51.895 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-10T18:12:51.896 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:51.896 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:12:51.896 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:12:51.896 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T18:12:51.896 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T18:12:51.896 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T18:12:51.896 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T18:12:51.896 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T18:12:51.896 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:51.896 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:12:52.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-fuse 2024-09-10T18:12:52.286 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:27 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:12:52.421 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:12:52.423 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:52.423 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-10T18:12:52.423 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:52.423 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:12:52.423 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 47 k 2024-09-10T18:12:52.423 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-10T18:12:52.423 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T18:12:52.423 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T18:12:52.423 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T18:12:52.423 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T18:12:52.423 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T18:12:52.424 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T18:12:52.424 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T18:12:52.424 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T18:12:52.424 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T18:12:52.424 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T18:12:52.424 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T18:12:52.424 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T18:12:52.424 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:52.424 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:12:52.424 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:52.424 INFO:teuthology.orchestra.run.smithi204.stdout:Install 13 Packages 2024-09-10T18:12:52.424 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:52.425 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 2.2 M 2024-09-10T18:12:52.425 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 26 M 2024-09-10T18:12:52.425 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:12:52.745 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:26 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:12:52.885 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:12:52.886 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:52.887 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-09-10T18:12:52.887 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:52.887 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:12:52.887 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 813 k 2024-09-10T18:12:52.887 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-10T18:12:52.887 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T18:12:52.887 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:52.887 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:12:52.888 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:52.888 INFO:teuthology.orchestra.run.smithi028.stdout:Install 2 Packages 2024-09-10T18:12:52.888 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:52.888 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 893 k 2024-09-10T18:12:52.888 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 2.6 M 2024-09-10T18:12:52.888 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:12:52.999 INFO:teuthology.orchestra.run.smithi204.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 165 kB/s | 19 kB 00:00 2024-09-10T18:12:53.024 INFO:teuthology.orchestra.run.smithi204.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 181 kB/s | 26 kB 00:00 2024-09-10T18:12:53.050 INFO:teuthology.orchestra.run.smithi204.stdout:(3/13): ceph-mgr-rook-17.2.7-1606.g5be57395.el9 284 kB/s | 47 kB 00:00 2024-09-10T18:12:53.150 INFO:teuthology.orchestra.run.smithi204.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-10T18:12:53.175 INFO:teuthology.orchestra.run.smithi204.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-09-10T18:12:53.209 INFO:teuthology.orchestra.run.smithi204.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 914 kB/s | 54 kB 00:00 2024-09-10T18:12:53.214 INFO:teuthology.orchestra.run.smithi028.stdout:(1/2): ceph-fuse-17.2.7-1606.g5be57395.el9.x86_ 2.9 MB/s | 813 kB 00:00 2024-09-10T18:12:53.242 INFO:teuthology.orchestra.run.smithi204.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-10T18:12:53.268 INFO:teuthology.orchestra.run.smithi204.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 248 kB/s | 14 kB 00:00 2024-09-10T18:12:53.293 INFO:teuthology.orchestra.run.smithi204.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-09-10T18:12:53.318 INFO:teuthology.orchestra.run.smithi204.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.9 MB/s | 220 kB 00:00 2024-09-10T18:12:53.352 INFO:teuthology.orchestra.run.smithi204.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.3 MB/s | 90 kB 00:00 2024-09-10T18:12:53.377 INFO:teuthology.orchestra.run.smithi204.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 703 kB/s | 59 kB 00:00 2024-09-10T18:12:53.401 INFO:teuthology.orchestra.run.smithi028.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 173 kB/s | 80 kB 00:00 2024-09-10T18:12:53.401 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:12:53.401 INFO:teuthology.orchestra.run.smithi028.stdout:Total 1.7 MB/s | 893 kB 00:00 2024-09-10T18:12:53.411 INFO:teuthology.orchestra.run.smithi204.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-10T18:12:53.412 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:12:53.414 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:12:53.414 INFO:teuthology.orchestra.run.smithi204.stdout:Total 2.2 MB/s | 2.2 MB 00:00 2024-09-10T18:12:53.435 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:12:53.435 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:12:53.477 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:12:53.492 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:12:53.492 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:12:53.522 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:12:53.522 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:12:53.583 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:12:53.585 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:12:53.727 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:12:53.856 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-10T18:12:53.924 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2/2 2024-09-10T18:12:54.089 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:12:54.270 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-10T18:12:54.321 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-10T18:12:54.376 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-10T18:12:54.413 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-10T18:12:54.454 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-10T18:12:54.566 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-10T18:12:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2/2 2024-09-10T18:12:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/2 2024-09-10T18:12:54.675 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-10T18:12:54.731 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-10T18:12:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-10T18:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-10T18:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:12:55.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-volume 2024-09-10T18:12:55.454 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-10T18:12:55.512 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-10T18:12:55.546 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-10T18:12:55.592 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-10T18:12:55.607 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 13/13 2024-09-10T18:12:55.702 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:29 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:12:55.842 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:12:55.843 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:55.843 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-09-10T18:12:55.843 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:55.843 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:12:55.844 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 270 k 2024-09-10T18:12:55.844 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:55.844 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:12:55.844 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:55.844 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-09-10T18:12:55.844 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:55.844 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 270 k 2024-09-10T18:12:55.844 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 1.3 M 2024-09-10T18:12:55.844 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:12:56.069 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-volume-17.2.7-1606.g5be57395.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-10T18:12:56.069 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:12:56.069 INFO:teuthology.orchestra.run.smithi028.stdout:Total 1.2 MB/s | 270 kB 00:00 2024-09-10T18:12:56.070 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:12:56.079 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:12:56.079 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:12:56.127 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:12:56.128 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:12:56.350 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:12:56.367 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 13/13 2024-09-10T18:12:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 1/13 2024-09-10T18:12:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-10T18:12:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-10T18:12:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-10T18:12:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-10T18:12:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-10T18:12:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-10T18:12:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-10T18:12:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-10T18:12:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-10T18:12:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-10T18:12:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-10T18:12:56.421 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:12:56.438 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:12:56.438 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:12:56.438 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-10T18:12:56.438 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:56.690 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-10T18:12:56.690 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:56.690 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:12:56.690 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:12:56.690 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T18:12:56.690 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T18:12:56.690 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T18:12:56.690 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T18:12:56.690 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T18:12:56.690 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T18:12:56.690 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T18:12:56.691 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T18:12:56.691 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T18:12:56.691 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T18:12:56.691 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T18:12:56.691 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T18:12:56.691 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:56.691 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:12:56.943 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-cephadm 2024-09-10T18:12:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:12:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:57.501 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:12:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:12:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:57.501 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:12:57.510 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:32 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:12:57.643 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:12:57.644 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:57.644 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-10T18:12:57.644 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:57.644 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:12:57.644 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 124 k 2024-09-10T18:12:57.644 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-10T18:12:57.645 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T18:12:57.645 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T18:12:57.645 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T18:12:57.645 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T18:12:57.645 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T18:12:57.645 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:57.645 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:12:57.645 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:12:57.645 INFO:teuthology.orchestra.run.smithi204.stdout:Install 6 Packages 2024-09-10T18:12:57.645 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:12:57.646 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 7.0 M 2024-09-10T18:12:57.646 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 33 M 2024-09-10T18:12:57.646 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:12:57.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librados-devel 2024-09-10T18:12:58.384 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:32 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:12:58.464 INFO:teuthology.orchestra.run.smithi204.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-10T18:12:58.517 INFO:teuthology.orchestra.run.smithi204.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-10T18:12:58.524 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:12:58.524 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:58.524 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-09-10T18:12:58.524 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:58.524 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:12:58.525 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:17.2.7-1606.g5be57395.el9 ceph 126 k 2024-09-10T18:12:58.525 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:58.525 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:12:58.525 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:12:58.525 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-09-10T18:12:58.525 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:58.525 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 126 k 2024-09-10T18:12:58.526 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 448 k 2024-09-10T18:12:58.526 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:12:58.539 INFO:teuthology.orchestra.run.smithi204.stdout:(3/6): ceph-mgr-cephadm-17.2.7-1606.g5be57395.e 576 kB/s | 124 kB 00:00 2024-09-10T18:12:58.564 INFO:teuthology.orchestra.run.smithi204.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-10T18:12:58.664 INFO:teuthology.orchestra.run.smithi204.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-09-10T18:12:58.689 INFO:teuthology.orchestra.run.smithi204.stdout:(6/6): python3-typing-extensions-4.12.2-2.el9.n 496 kB/s | 74 kB 00:00 2024-09-10T18:12:58.692 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:12:58.693 INFO:teuthology.orchestra.run.smithi204.stdout:Total 6.7 MB/s | 7.0 MB 00:01 2024-09-10T18:12:58.725 INFO:teuthology.orchestra.run.smithi028.stdout:librados-devel-17.2.7-1606.g5be57395.el9.x86_64 641 kB/s | 126 kB 00:00 2024-09-10T18:12:58.725 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:12:58.725 INFO:teuthology.orchestra.run.smithi028.stdout:Total 637 kB/s | 126 kB 00:00 2024-09-10T18:12:58.725 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:12:58.729 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:12:58.729 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:12:58.763 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:12:58.774 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:12:58.774 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:12:58.797 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:12:58.797 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:12:58.874 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:12:58.876 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:12:58.965 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:12:59.056 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:12:59.178 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:12:59.350 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-10T18:12:59.400 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-10T18:12:59.492 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:12:59.730 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:12:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:59.731 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:12:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:12:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:12:59.731 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:12:59.935 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-10T18:12:59.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install libcephfs2 2024-09-10T18:13:00.028 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-10T18:13:00.091 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-10T18:13:00.121 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 6/6 2024-09-10T18:13:00.587 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:34 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:13:00.681 INFO:teuthology.orchestra.run.smithi028.stdout:Package libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:13:00.726 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:13:00.728 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-09-10T18:13:00.728 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:13:00.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install libcephfs-devel 2024-09-10T18:13:00.902 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 6/6 2024-09-10T18:13:00.902 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/6 2024-09-10T18:13:00.902 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-10T18:13:00.902 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-10T18:13:00.902 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-10T18:13:00.902 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-10T18:13:01.180 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-10T18:13:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:01.181 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:13:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:13:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T18:13:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T18:13:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T18:13:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T18:13:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T18:13:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:01.181 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:13:01.381 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:35 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:13:01.406 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-fuse 2024-09-10T18:13:01.520 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:13:01.521 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:13:01.521 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-09-10T18:13:01.521 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:13:01.521 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:13:01.521 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:17.2.7-1606.g5be57395.el9 ceph 25 k 2024-09-10T18:13:01.522 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:13:01.522 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:13:01.522 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:13:01.522 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-09-10T18:13:01.522 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:13:01.522 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 25 k 2024-09-10T18:13:01.522 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 110 k 2024-09-10T18:13:01.522 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:13:01.667 INFO:teuthology.orchestra.run.smithi028.stdout:libcephfs-devel-17.2.7-1606.g5be57395.el9.x86_6 172 kB/s | 25 kB 00:00 2024-09-10T18:13:01.668 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:13:01.668 INFO:teuthology.orchestra.run.smithi028.stdout:Total 170 kB/s | 25 kB 00:00 2024-09-10T18:13:01.669 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:13:01.672 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:13:01.672 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:13:01.673 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:13:01.674 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:13:01.759 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:13:01.868 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:01.971 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:36 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:13:02.106 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:13:02.106 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:02.106 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-10T18:13:02.106 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:02.106 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:13:02.106 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 813 k 2024-09-10T18:13:02.107 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-10T18:13:02.107 INFO:teuthology.orchestra.run.smithi204.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T18:13:02.107 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:02.107 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:13:02.107 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:02.107 INFO:teuthology.orchestra.run.smithi204.stdout:Install 2 Packages 2024-09-10T18:13:02.107 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:02.107 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 893 k 2024-09-10T18:13:02.108 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 2.6 M 2024-09-10T18:13:02.108 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:13:02.304 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:02.400 INFO:teuthology.orchestra.run.smithi204.stdout:(1/2): ceph-fuse-17.2.7-1606.g5be57395.el9.x86_ 3.0 MB/s | 813 kB 00:00 2024-09-10T18:13:02.533 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:02.533 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:13:02.533 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:13:02.533 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:13:02.534 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:13:02.534 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:13:02.600 INFO:teuthology.orchestra.run.smithi204.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 171 kB/s | 80 kB 00:00 2024-09-10T18:13:02.601 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:13:02.601 INFO:teuthology.orchestra.run.smithi204.stdout:Total 1.8 MB/s | 893 kB 00:00 2024-09-10T18:13:02.613 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:13:02.634 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:13:02.635 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:13:02.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librados2 2024-09-10T18:13:02.715 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:13:02.715 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:13:02.914 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:13:03.047 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-10T18:13:03.117 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2/2 2024-09-10T18:13:03.294 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:37 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:13:03.387 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:13:03.434 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:13:03.435 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-09-10T18:13:03.435 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:13:03.497 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librbd1 2024-09-10T18:13:03.770 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2/2 2024-09-10T18:13:03.770 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/2 2024-09-10T18:13:04.049 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-10T18:13:04.049 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:04.049 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:13:04.049 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-10T18:13:04.049 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:04.049 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:13:04.084 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:38 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:13:04.178 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:13:04.223 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:13:04.224 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-09-10T18:13:04.224 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:13:04.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rados 2024-09-10T18:13:04.307 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-volume 2024-09-10T18:13:04.870 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:38 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:13:04.890 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:39 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:13:04.961 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:13:05.006 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:13:05.007 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-09-10T18:13:05.007 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:13:05.026 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:13:05.027 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:05.027 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-10T18:13:05.027 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:05.028 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:13:05.028 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 270 k 2024-09-10T18:13:05.028 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:05.028 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:13:05.028 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:05.028 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-10T18:13:05.028 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:05.028 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 270 k 2024-09-10T18:13:05.029 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 1.3 M 2024-09-10T18:13:05.029 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:13:05.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rgw 2024-09-10T18:13:05.264 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-volume-17.2.7-1606.g5be57395.el9.noarch.rp 1.1 MB/s | 270 kB 00:00 2024-09-10T18:13:05.264 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:13:05.264 INFO:teuthology.orchestra.run.smithi204.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-10T18:13:05.264 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:13:05.273 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:13:05.273 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:13:05.300 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:13:05.301 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:13:05.516 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:13:05.591 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:13:05.607 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:13:05.607 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:13:05.607 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-10T18:13:05.607 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:05.655 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:39 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:13:05.749 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:13:05.795 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:13:05.796 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-09-10T18:13:05.796 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:13:05.859 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-cephfs 2024-09-10T18:13:06.452 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:40 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:13:06.545 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:13:06.563 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:13:06.563 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:06.563 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:13:06.563 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:13:06.563 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:06.563 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:13:06.590 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:13:06.591 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-09-10T18:13:06.591 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:13:06.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rbd 2024-09-10T18:13:06.829 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados-devel 2024-09-10T18:13:07.245 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:41 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:13:07.337 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:13:07.382 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:13:07.383 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-09-10T18:13:07.383 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:13:07.404 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:42 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:13:07.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-fuse 2024-09-10T18:13:07.544 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:13:07.545 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:07.545 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-09-10T18:13:07.545 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:07.545 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:13:07.545 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel x86_64 2:17.2.7-1606.g5be57395.el9 ceph 126 k 2024-09-10T18:13:07.545 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:07.546 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:13:07.546 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:07.546 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-10T18:13:07.546 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:07.546 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 126 k 2024-09-10T18:13:07.546 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 448 k 2024-09-10T18:13:07.546 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:13:07.744 INFO:teuthology.orchestra.run.smithi204.stdout:librados-devel-17.2.7-1606.g5be57395.el9.x86_64 638 kB/s | 126 kB 00:00 2024-09-10T18:13:07.744 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:13:07.745 INFO:teuthology.orchestra.run.smithi204.stdout:Total 633 kB/s | 126 kB 00:00 2024-09-10T18:13:07.745 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:13:07.750 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:13:07.750 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:13:07.814 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:13:07.814 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:13:07.981 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:13:08.032 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:42 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:13:08.083 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:08.170 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:13:08.171 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:13:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-09-10T18:13:08.171 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:13:08.171 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:13:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 82 k 2024-09-10T18:13:08.172 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:13:08.172 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:13:08.172 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:13:08.172 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-09-10T18:13:08.172 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:13:08.172 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 82 k 2024-09-10T18:13:08.172 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 222 k 2024-09-10T18:13:08.172 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:13:08.362 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-fuse-17.2.7-1606.g5be57395.el9.x86_64.rpm 430 kB/s | 82 kB 00:00 2024-09-10T18:13:08.362 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:13:08.362 INFO:teuthology.orchestra.run.smithi028.stdout:Total 427 kB/s | 82 kB 00:00 2024-09-10T18:13:08.363 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:13:08.370 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:13:08.370 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:13:08.437 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:13:08.438 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:13:08.516 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:08.585 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:13:08.687 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:08.754 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:08.755 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:08.755 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:13:08.755 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:13:08.755 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:08.755 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:13:09.014 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install libcephfs2 2024-09-10T18:13:09.086 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:09.353 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:09.353 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:13:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:13:09.353 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:13:09.353 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:13:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:13:09.591 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:44 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:13:09.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-mirror 2024-09-10T18:13:09.684 INFO:teuthology.orchestra.run.smithi204.stdout:Package libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:13:09.729 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:13:09.730 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-10T18:13:09.730 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:13:09.789 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install libcephfs-devel 2024-09-10T18:13:10.194 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:44 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:13:10.335 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:13:10.336 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:13:10.336 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-09-10T18:13:10.336 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:13:10.336 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:13:10.336 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.9 M 2024-09-10T18:13:10.336 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:13:10.337 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:13:10.337 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:13:10.337 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-09-10T18:13:10.337 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:13:10.337 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 2.9 M 2024-09-10T18:13:10.337 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 12 M 2024-09-10T18:13:10.337 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:13:10.363 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:45 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:13:10.498 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:13:10.499 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-09-10T18:13:10.499 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:10.499 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:13:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel x86_64 2:17.2.7-1606.g5be57395.el9 ceph 25 k 2024-09-10T18:13:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:10.499 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:13:10.499 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:10.499 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-10T18:13:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:10.499 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 25 k 2024-09-10T18:13:10.500 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 110 k 2024-09-10T18:13:10.500 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:13:10.648 INFO:teuthology.orchestra.run.smithi204.stdout:libcephfs-devel-17.2.7-1606.g5be57395.el9.x86_6 168 kB/s | 25 kB 00:00 2024-09-10T18:13:10.649 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:13:10.649 INFO:teuthology.orchestra.run.smithi204.stdout:Total 167 kB/s | 25 kB 00:00 2024-09-10T18:13:10.649 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:13:10.652 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:13:10.652 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:13:10.654 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:13:10.654 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:13:10.656 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-mirror-17.2.7-1606.g5be57395.el9.x86_64.rpm 9.2 MB/s | 2.9 MB 00:00 2024-09-10T18:13:10.657 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:13:10.657 INFO:teuthology.orchestra.run.smithi028.stdout:Total 9.2 MB/s | 2.9 MB 00:00 2024-09-10T18:13:10.657 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:13:10.667 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:13:10.667 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:13:10.742 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:13:10.751 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:13:10.751 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:13:10.838 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:11.195 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:13:11.224 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:11.274 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:11.302 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:11.302 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:13:11.302 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T18:13:11.302 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T18:13:11.302 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T18:13:11.302 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:13:11.479 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:11.479 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:11.479 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:13:11.480 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:13:11.480 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:11.480 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:13:11.680 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados2 2024-09-10T18:13:12.242 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:47 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:13:12.331 INFO:teuthology.orchestra.run.smithi204.stdout:Package librados2-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:13:12.373 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:13:12.374 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-10T18:13:12.375 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:13:12.400 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:12.400 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:13:12.400 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:13:12.400 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:13:12.400 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:13:12.400 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:13:12.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librbd1 2024-09-10T18:13:12.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-nbd 2024-09-10T18:13:13.001 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:47 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:13:13.090 INFO:teuthology.orchestra.run.smithi204.stdout:Package librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:13:13.134 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:13:13.135 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-10T18:13:13.135 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:13:13.195 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rados 2024-09-10T18:13:13.202 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:47 ago on Tue 10 Sep 2024 06:11:26 PM UTC. 2024-09-10T18:13:13.339 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-10T18:13:13.340 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:13:13.340 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-09-10T18:13:13.341 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:13:13.341 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-10T18:13:13.341 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 165 k 2024-09-10T18:13:13.341 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:13:13.341 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-10T18:13:13.341 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-10T18:13:13.341 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-09-10T18:13:13.341 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:13:13.341 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 165 k 2024-09-10T18:13:13.341 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 477 k 2024-09-10T18:13:13.341 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-10T18:13:13.551 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-nbd-17.2.7-1606.g5be57395.el9.x86_64.rpm 788 kB/s | 165 kB 00:00 2024-09-10T18:13:13.552 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:13:13.552 INFO:teuthology.orchestra.run.smithi028.stdout:Total 783 kB/s | 165 kB 00:00 2024-09-10T18:13:13.552 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-10T18:13:13.560 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-10T18:13:13.560 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-10T18:13:13.628 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-10T18:13:13.628 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-10T18:13:13.763 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:48 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:13:13.784 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-10T18:13:13.852 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:13:13.876 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:13.896 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:13:13.897 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-10T18:13:13.897 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:13:13.962 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rgw 2024-09-10T18:13:14.277 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:14.527 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:49 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:13:14.582 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:14.582 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:13:14.582 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-10T18:13:14.582 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:13:14.582 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:13:14.582 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-10T18:13:14.620 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:13:14.662 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:13:14.663 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-10T18:13:14.663 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:13:14.729 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-cephfs 2024-09-10T18:13:14.805 DEBUG:teuthology.parallel:result is None 2024-09-10T18:13:15.300 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:50 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:13:15.391 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:13:15.435 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:13:15.436 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-10T18:13:15.436 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:13:15.498 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rbd 2024-09-10T18:13:16.062 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:51 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:13:16.152 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:13:16.195 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:13:16.196 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-10T18:13:16.196 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:13:16.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-fuse 2024-09-10T18:13:16.833 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:51 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:13:16.967 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:13:16.968 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:16.968 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-09-10T18:13:16.968 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:16.968 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:13:16.968 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 82 k 2024-09-10T18:13:16.968 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:16.968 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:13:16.968 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:16.968 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-10T18:13:16.968 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:16.969 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 82 k 2024-09-10T18:13:16.969 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 222 k 2024-09-10T18:13:16.969 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:13:17.143 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-fuse-17.2.7-1606.g5be57395.el9.x86_64.rpm 470 kB/s | 82 kB 00:00 2024-09-10T18:13:17.143 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:13:17.144 INFO:teuthology.orchestra.run.smithi204.stdout:Total 466 kB/s | 82 kB 00:00 2024-09-10T18:13:17.144 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:13:17.150 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:13:17.150 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:13:17.213 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:13:17.213 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:13:17.355 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:13:17.457 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:17.843 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:18.083 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:18.083 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:18.084 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:13:18.084 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:13:18.084 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:18.084 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:13:18.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-mirror 2024-09-10T18:13:18.908 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:53 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:13:19.051 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:13:19.052 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:19.052 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-09-10T18:13:19.052 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:19.052 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:13:19.052 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.9 M 2024-09-10T18:13:19.052 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:19.053 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:13:19.053 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:19.053 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-10T18:13:19.053 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:19.053 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 2.9 M 2024-09-10T18:13:19.053 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 12 M 2024-09-10T18:13:19.053 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:13:19.364 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-mirror-17.2.7-1606.g5be57395.el9.x86_64.rpm 9.5 MB/s | 2.9 MB 00:00 2024-09-10T18:13:19.364 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:13:19.365 INFO:teuthology.orchestra.run.smithi204.stdout:Total 9.4 MB/s | 2.9 MB 00:00 2024-09-10T18:13:19.365 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:13:19.374 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:13:19.374 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:13:19.456 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:13:19.457 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:13:19.891 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:13:19.948 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:19.974 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:19.974 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:13:19.974 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T18:13:19.974 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T18:13:19.974 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T18:13:19.974 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:21.030 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:21.030 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:21.030 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:13:21.030 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:13:21.030 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:21.030 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:13:21.244 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-nbd 2024-09-10T18:13:21.827 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:56 ago on Tue 10 Sep 2024 06:11:25 PM UTC. 2024-09-10T18:13:21.962 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-10T18:13:21.962 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:21.963 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-09-10T18:13:21.963 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:21.963 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-10T18:13:21.963 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 165 k 2024-09-10T18:13:21.963 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:21.963 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-10T18:13:21.963 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-10T18:13:21.963 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-10T18:13:21.963 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:21.964 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 165 k 2024-09-10T18:13:21.964 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 477 k 2024-09-10T18:13:21.964 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-10T18:13:22.185 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-nbd-17.2.7-1606.g5be57395.el9.x86_64.rpm 750 kB/s | 165 kB 00:00 2024-09-10T18:13:22.185 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:13:22.186 INFO:teuthology.orchestra.run.smithi204.stdout:Total 744 kB/s | 165 kB 00:00 2024-09-10T18:13:22.186 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-10T18:13:22.193 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-10T18:13:22.193 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-10T18:13:22.256 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-10T18:13:22.256 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-10T18:13:22.407 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-10T18:13:22.515 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:22.985 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:23.238 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:13:23.239 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:23.239 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-10T18:13:23.239 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:13:23.239 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:13:23.239 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-10T18:13:23.487 DEBUG:teuthology.parallel:result is None 2024-09-10T18:13:23.488 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:13:23.628 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T18:13:23.654 INFO:teuthology.orchestra.run.smithi028.stdout:17.2.7-1606.g5be57395.el9 2024-09-10T18:13:23.655 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1606.g5be57395.el9 2024-09-10T18:13:23.655 INFO:teuthology.task.install:The correct ceph version 17.2.7-1606.g5be57395 is installed. 2024-09-10T18:13:23.657 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:13:23.795 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T18:13:23.818 INFO:teuthology.orchestra.run.smithi204.stdout:17.2.7-1606.g5be57395.el9 2024-09-10T18:13:23.818 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1606.g5be57395.el9 2024-09-10T18:13:23.818 INFO:teuthology.task.install:The correct ceph version 17.2.7-1606.g5be57395 is installed. 2024-09-10T18:13:23.820 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-10T18:13:23.820 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T18:13:23.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T18:13:23.852 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-10T18:13:23.852 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T18:13:23.885 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-10T18:13:23.885 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T18:13:23.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T18:13:23.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T18:13:23.992 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-10T18:13:23.993 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T18:13:24.018 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T18:13:24.084 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-10T18:13:24.084 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T18:13:24.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T18:13:24.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T18:13:24.184 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-10T18:13:24.184 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T18:13:24.209 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T18:13:24.274 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-10T18:13:24.274 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T18:13:24.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T18:13:24.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T18:13:24.371 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-10T18:13:24.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T18:13:24.396 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T18:13:24.461 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-10T18:13:24.552 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588', 'cephadm_mode': 'root'} 2024-09-10T18:13:24.552 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:13:24.553 INFO:tasks.cephadm:Cluster fsid is 5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:13:24.553 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-10T18:13:24.553 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.28:6789]', 'mon.c': '[v1:172.21.15.28:6790]', 'mon.b': '[v1:172.21.15.204:6789]'} 2024-09-10T18:13:24.554 INFO:tasks.cephadm:First mon is mon.a on smithi028 2024-09-10T18:13:24.554 INFO:tasks.cephadm:First mgr is y 2024-09-10T18:13:24.554 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-10T18:13:24.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo hostname $(hostname -s) 2024-09-10T18:13:24.585 DEBUG:teuthology.orchestra.run.smithi204:> sudo hostname $(hostname -s) 2024-09-10T18:13:24.611 INFO:tasks.cephadm:Downloading cephadm (repo https://git.ceph.com/ceph-ci.git ref 5be57395ea235535910512ae73ba8e7ff25fe588)... 2024-09-10T18:13:24.612 DEBUG:teuthology.orchestra.run.smithi028:> git clone https://git.ceph.com/ceph-ci.git testrepo && cd testrepo && git show 5be57395ea235535910512ae73ba8e7ff25fe588:src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-10T18:13:24.643 INFO:teuthology.orchestra.run.smithi028.stderr:Cloning into 'testrepo'... 2024-09-10T18:14:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:Updating files: 92% (11769/12697) Updating files: 93% (11809/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-09-10T18:14:23.153 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 10 18:14 /home/ubuntu/cephtest/cephadm 2024-09-10T18:14:23.154 DEBUG:teuthology.orchestra.run.smithi204:> git clone https://git.ceph.com/ceph-ci.git testrepo && cd testrepo && git show 5be57395ea235535910512ae73ba8e7ff25fe588:src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-10T18:14:23.174 INFO:teuthology.orchestra.run.smithi204.stderr:Cloning into 'testrepo'... 2024-09-10T18:15:21.121 INFO:teuthology.orchestra.run.smithi204.stderr:Updating files: 97% (12385/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-09-10T18:15:21.154 INFO:teuthology.orchestra.run.smithi204.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 10 18:15 /home/ubuntu/cephtest/cephadm 2024-09-10T18:15:21.155 DEBUG:teuthology.orchestra.run.smithi028:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-10T18:15:21.176 DEBUG:teuthology.orchestra.run.smithi204:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-10T18:15:21.224 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 on all hosts... 2024-09-10T18:15:21.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 pull 2024-09-10T18:15:21.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 pull 2024-09-10T18:15:22.699 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588... 2024-09-10T18:15:24.725 INFO:teuthology.orchestra.run.smithi204.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588... 2024-09-10T18:15:56.516 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:15:56.517 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_version": "ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable)", 2024-09-10T18:15:56.517 INFO:teuthology.orchestra.run.smithi204.stdout: "image_id": "4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9", 2024-09-10T18:15:56.517 INFO:teuthology.orchestra.run.smithi204.stdout: "repo_digests": [ 2024-09-10T18:15:56.517 INFO:teuthology.orchestra.run.smithi204.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16" 2024-09-10T18:15:56.517 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:15:56.517 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:16:11.838 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:16:11.839 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable)", 2024-09-10T18:16:11.839 INFO:teuthology.orchestra.run.smithi028.stdout: "image_id": "4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9", 2024-09-10T18:16:11.839 INFO:teuthology.orchestra.run.smithi028.stdout: "repo_digests": [ 2024-09-10T18:16:11.839 INFO:teuthology.orchestra.run.smithi028.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16" 2024-09-10T18:16:11.839 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:16:11.839 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:16:11.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph 2024-09-10T18:16:11.892 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph 2024-09-10T18:16:11.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /etc/ceph 2024-09-10T18:16:11.960 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /etc/ceph 2024-09-10T18:16:11.994 INFO:tasks.cephadm:Writing seed config... 2024-09-10T18:16:11.996 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-10T18:16:11.996 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-10T18:16:11.996 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-09-10T18:16:11.996 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-09-10T18:16:11.996 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-09-10T18:16:11.996 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-09-10T18:16:11.996 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-10T18:16:11.996 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-10T18:16:11.996 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-10T18:16:11.996 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-10T18:16:11.996 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-10T18:16:11.996 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-10T18:16:11.997 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-10T18:16:11.997 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-10T18:16:11.997 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-10T18:16:11.997 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-10T18:16:11.997 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-10T18:16:11.997 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-10T18:16:11.997 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-09-10T18:16:11.997 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-10T18:16:11.998 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-10T18:16:11.998 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-10T18:16:11.998 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-10T18:16:11.998 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-10T18:16:11.998 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T18:16:11.999 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-10T18:16:12.017 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 5eababb2-6fa0-11ef-bcea-c7b262605968 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-09-10T18:16:12.018 DEBUG:teuthology.orchestra.run.smithi028:mon.a> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mon.a.service 2024-09-10T18:16:12.061 DEBUG:teuthology.orchestra.run.smithi028:mgr.y> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mgr.y.service 2024-09-10T18:16:12.104 INFO:tasks.cephadm:Bootstrapping... 2024-09-10T18:16:12.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 -v bootstrap --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.28:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-10T18:16:12.313 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:16:12.313 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588', '-v', 'bootstrap', '--fsid', '5eababb2-6fa0-11ef-bcea-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.28:6789]', '--skip-admin-label'] 2024-09-10T18:16:12.337 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 5.2.2 2024-09-10T18:16:12.337 INFO:teuthology.orchestra.run.smithi028.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-10T18:16:12.337 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying podman|docker is present... 2024-09-10T18:16:12.359 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 5.2.2 2024-09-10T18:16:12.359 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying lvm2 is present... 2024-09-10T18:16:12.359 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying time synchronization is in place... 2024-09-10T18:16:12.368 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-10T18:16:12.368 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-10T18:16:12.375 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-10T18:16:12.375 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-10T18:16:12.382 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-09-10T18:16:12.388 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-09-10T18:16:12.388 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-09-10T18:16:12.389 INFO:teuthology.orchestra.run.smithi028.stdout:Repeating the final host check... 2024-09-10T18:16:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 5.2.2 2024-09-10T18:16:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-10T18:16:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl is present 2024-09-10T18:16:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:lvcreate is present 2024-09-10T18:16:12.418 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-10T18:16:12.418 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-10T18:16:12.425 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-10T18:16:12.425 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-10T18:16:12.432 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-09-10T18:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-09-10T18:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-09-10T18:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:Host looks OK 2024-09-10T18:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:Cluster fsid: 5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:Acquiring lock 140174713942944 on /run/cephadm/5eababb2-6fa0-11ef-bcea-c7b262605968.lock 2024-09-10T18:16:12.439 INFO:teuthology.orchestra.run.smithi028.stdout:Lock 140174713942944 acquired on /run/cephadm/5eababb2-6fa0-11ef-bcea-c7b262605968.lock 2024-09-10T18:16:12.440 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 6789 ... 2024-09-10T18:16:12.440 INFO:teuthology.orchestra.run.smithi028.stdout:Base mon IP(s) is [172.21.15.28:6789], mon addrv is [v1:172.21.15.28:6789] 2024-09-10T18:16:12.443 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.28 metric 100 2024-09-10T18:16:12.444 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.28 metric 100 2024-09-10T18:16:12.446 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-10T18:16:12.447 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-10T18:16:12.447 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1786sec hoplimit 64 pref medium 2024-09-10T18:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-10T18:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-10T18:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-10T18:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-10T18:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-10T18:16:12.449 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-10T18:16:12.450 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-09-10T18:16:12.450 INFO:teuthology.orchestra.run.smithi028.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-09-10T18:16:12.450 INFO:teuthology.orchestra.run.smithi028.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-10T18:16:12.451 INFO:teuthology.orchestra.run.smithi028.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588... 2024-09-10T18:16:12.879 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 2024-09-10T18:16:12.880 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588... 2024-09-10T18:16:12.880 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Getting image source signatures 2024-09-10T18:16:12.880 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:5df13064c03763eb21a89b455ab82ba4ec435f51a3f340cae76514aaf2996d4c 2024-09-10T18:16:12.880 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-10T18:16:12.880 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying config sha256:4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 2024-09-10T18:16:12.880 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-10T18:16:13.783 INFO:teuthology.orchestra.run.smithi028.stdout:ceph: stdout ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable) 2024-09-10T18:16:13.783 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph version: ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable) 2024-09-10T18:16:13.783 INFO:teuthology.orchestra.run.smithi028.stdout:Extracting ceph user uid/gid from container image... 2024-09-10T18:16:14.756 INFO:teuthology.orchestra.run.smithi028.stdout:stat: stdout 167 167 2024-09-10T18:16:14.756 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial keys... 2024-09-10T18:16:15.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQDvjOBmSGKaAhAAYBQDfBneiguN/hTwrmfKiw== 2024-09-10T18:16:16.541 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQDvjOBmvW4qOBAAOD9wUTI3Qd8nL6fJfr616A== 2024-09-10T18:16:17.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQDwjOBmoTVtNBAAdejKoEmt1k7Dx3yHyzh3PQ== 2024-09-10T18:16:17.432 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial monmap... 2024-09-10T18:16:18.371 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-10T18:16:18.371 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-09-10T18:16:18.371 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:16:18.371 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-10T18:16:18.372 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool for a [v1:172.21.15.28:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-10T18:16:18.372 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = octopus 2024-09-10T18:16:18.372 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: set fsid to 5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:16:18.372 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-10T18:16:18.372 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:16:18.372 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mon... 2024-09-10T18:16:19.267 INFO:teuthology.orchestra.run.smithi028.stdout:create mon.a on 2024-09-10T18:16:19.489 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-10T18:16:19.638 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-10T18:16:19.807 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-5eababb2-6fa0-11ef-bcea-c7b262605968.target → /etc/systemd/system/ceph-5eababb2-6fa0-11ef-bcea-c7b262605968.target. 2024-09-10T18:16:19.807 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-5eababb2-6fa0-11ef-bcea-c7b262605968.target → /etc/systemd/system/ceph-5eababb2-6fa0-11ef-bcea-c7b262605968.target. 2024-09-10T18:16:19.986 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mon.a 2024-09-10T18:16:19.987 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mon.a.service: Unit ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mon.a.service not loaded. 2024-09-10T18:16:20.155 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-5eababb2-6fa0-11ef-bcea-c7b262605968.target.wants/ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mon.a.service → /etc/systemd/system/ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@.service. 2024-09-10T18:16:20.733 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-10T18:16:20.734 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-09-10T18:16:20.741 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-10T18:16:20.742 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-10T18:16:20.742 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-09-10T18:16:20.742 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-09-10T18:16:20.742 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon to start... 2024-09-10T18:16:20.742 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon... 2024-09-10T18:16:21.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:21 smithi028 ceph-mon[28920]: from='client.? v1:172.21.15.28:0/71790561' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-10T18:16:21.762 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout cluster: 2024-09-10T18:16:21.762 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout id: 5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:16:21.762 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-10T18:16:21.762 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-10T18:16:21.762 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout services: 2024-09-10T18:16:21.762 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.551597s) 2024-09-10T18:16:21.762 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-10T18:16:21.762 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-10T18:16:21.762 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-10T18:16:21.762 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout data: 2024-09-10T18:16:21.762 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-10T18:16:21.763 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-10T18:16:21.763 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-10T18:16:21.763 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pgs: 2024-09-10T18:16:21.763 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-10T18:16:21.763 INFO:teuthology.orchestra.run.smithi028.stdout:mon is available 2024-09-10T18:16:21.763 INFO:teuthology.orchestra.run.smithi028.stdout:Assimilating anything we can from ceph.conf... 2024-09-10T18:16:22.701 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-10T18:16:22.701 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-09-10T18:16:22.701 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:16:22.701 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-10T18:16:22.701 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.28:6789] 2024-09-10T18:16:22.701 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-10T18:16:22.701 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-10T18:16:22.702 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-10T18:16:22.702 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-10T18:16:22.702 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-10T18:16:22.702 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-10T18:16:22.702 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-10T18:16:22.702 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-10T18:16:22.702 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-09-10T18:16:22.702 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-10T18:16:22.702 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-10T18:16:22.702 INFO:teuthology.orchestra.run.smithi028.stdout:Generating new minimal ceph.conf... 2024-09-10T18:16:23.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:22 smithi028 ceph-mon[28920]: from='client.? v1:172.21.15.28:0/2081611291' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-10T18:16:23.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:22 smithi028 ceph-mon[28920]: from='client.? v1:172.21.15.28:0/2081611291' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-10T18:16:23.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:23 smithi028 ceph-mon[28920]: from='client.? v1:172.21.15.28:0/2397284133' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:16:23.702 INFO:teuthology.orchestra.run.smithi028.stdout:Restarting the monitor... 2024-09-10T18:16:23.982 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:23 smithi028 systemd[1]: Stopping Ceph mon.a for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:16:24.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:23 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a[28909]: 2024-09-10T18:16:23.980+0000 7fcf83836640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T18:16:24.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:23 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a[28909]: 2024-09-10T18:16:23.980+0000 7fcf83836640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-10T18:16:24.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:24 smithi028 podman[29167]: 2024-09-10 18:16:24.114333206 +0000 UTC m=+0.231098544 container died 80df7d18639dde2a436f7aa784684d0c3b97fb7eabc90d6a0b0e7045df52037b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True) 2024-09-10T18:16:24.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:24 smithi028 podman[29167]: 2024-09-10 18:16:24.387942391 +0000 UTC m=+0.504707730 container cleanup 80df7d18639dde2a436f7aa784684d0c3b97fb7eabc90d6a0b0e7045df52037b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-10T18:16:24.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:24 smithi028 bash[29167]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a 2024-09-10T18:16:24.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:24 smithi028 podman[29179]: 2024-09-10 18:16:24.622313543 +0000 UTC m=+0.503812497 container remove 80df7d18639dde2a436f7aa784684d0c3b97fb7eabc90d6a0b0e7045df52037b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-10T18:16:24.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:24 smithi028 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-10T18:16:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:24 smithi028 systemd[1]: Stopped Ceph mon.a for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:16:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:24 smithi028 systemd[1]: Starting Ceph mon.a for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:16:25.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:24 smithi028 podman[29271]: 2024-09-10 18:16:24.871697318 +0000 UTC m=+0.015991150 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:16:25.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:24 smithi028 podman[29271]: 2024-09-10 18:16:24.973398451 +0000 UTC m=+0.117692319 container create c1cb1eb6243125d19433dfd1a26fe957d501c01e7abf3c4f110ef30df115a11d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True) 2024-09-10T18:16:25.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 podman[29271]: 2024-09-10 18:16:25.217642974 +0000 UTC m=+0.361936801 container init c1cb1eb6243125d19433dfd1a26fe957d501c01e7abf3c4f110ef30df115a11d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-10T18:16:25.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 podman[29271]: 2024-09-10 18:16:25.221053378 +0000 UTC m=+0.365347209 container start c1cb1eb6243125d19433dfd1a26fe957d501c01e7abf3c4f110ef30df115a11d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T18:16:25.314 INFO:teuthology.orchestra.run.smithi028.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-10T18:16:25.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: set uid:gid to 167:167 (ceph:ceph) 2024-09-10T18:16:25.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable), process ceph-mon, pid 2 2024-09-10T18:16:25.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: pidfile_write: ignore empty --pid-file 2024-09-10T18:16:25.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: load: jerasure load: lrc 2024-09-10T18:16:25.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: RocksDB version: 6.15.5 2024-09-10T18:16:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-10T18:16:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Compile date Sep 9 2024 2024-09-10T18:16:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: DB SUMMARY 2024-09-10T18:16:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: DB Session ID: 0WLIND0NVKEUTLPLWSZD 2024-09-10T18:16:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: CURRENT file: CURRENT 2024-09-10T18:16:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: IDENTITY file: IDENTITY 2024-09-10T18:16:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-09-10T18:16:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-10T18:16:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 87772 ; 2024-09-10T18:16:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.error_if_exists: 0 2024-09-10T18:16:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.create_if_missing: 0 2024-09-10T18:16:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.paranoid_checks: 1 2024-09-10T18:16:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-10T18:16:25.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.env: 0x5568b3d1f440 2024-09-10T18:16:25.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.fs: Posix File System 2024-09-10T18:16:25.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.info_log: 0x5568b4cc8e60 2024-09-10T18:16:25.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_file_opening_threads: 16 2024-09-10T18:16:25.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.statistics: (nil) 2024-09-10T18:16:25.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.use_fsync: 0 2024-09-10T18:16:25.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_log_file_size: 0 2024-09-10T18:16:25.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-10T18:16:25.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-10T18:16:25.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.keep_log_file_num: 1000 2024-09-10T18:16:25.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.recycle_log_file_num: 0 2024-09-10T18:16:25.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.allow_fallocate: 1 2024-09-10T18:16:25.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.allow_mmap_reads: 0 2024-09-10T18:16:25.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.allow_mmap_writes: 0 2024-09-10T18:16:25.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.use_direct_reads: 0 2024-09-10T18:16:25.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-10T18:16:25.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.create_missing_column_families: 0 2024-09-10T18:16:25.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.db_log_dir: 2024-09-10T18:16:25.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-10T18:16:25.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-10T18:16:25.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-10T18:16:25.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-10T18:16:25.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-10T18:16:25.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-10T18:16:25.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-10T18:16:25.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.advise_random_on_open: 1 2024-09-10T18:16:25.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.db_write_buffer_size: 0 2024-09-10T18:16:25.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.write_buffer_manager: 0x5568b4d86780 2024-09-10T18:16:25.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-10T18:16:25.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-10T18:16:25.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-10T18:16:25.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-10T18:16:25.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.rate_limiter: (nil) 2024-09-10T18:16:25.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-10T18:16:25.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.wal_recovery_mode: 2 2024-09-10T18:16:25.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.enable_thread_tracking: 0 2024-09-10T18:16:25.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.enable_pipelined_write: 0 2024-09-10T18:16:25.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.unordered_write: 0 2024-09-10T18:16:25.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-10T18:16:25.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-10T18:16:25.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-10T18:16:25.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-10T18:16:25.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.row_cache: None 2024-09-10T18:16:25.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.wal_filter: None 2024-09-10T18:16:25.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-10T18:16:25.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.allow_ingest_behind: 0 2024-09-10T18:16:25.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.preserve_deletes: 0 2024-09-10T18:16:25.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.two_write_queues: 0 2024-09-10T18:16:25.512 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.manual_wal_flush: 0 2024-09-10T18:16:25.512 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.atomic_flush: 0 2024-09-10T18:16:25.512 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-10T18:16:25.512 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-10T18:16:25.512 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-10T18:16:25.512 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.log_readahead_size: 0 2024-09-10T18:16:25.512 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-10T18:16:25.512 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.best_efforts_recovery: 0 2024-09-10T18:16:25.512 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-10T18:16:25.512 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-10T18:16:25.512 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.allow_data_in_errors: 0 2024-09-10T18:16:25.512 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.db_host_id: __hostname__ 2024-09-10T18:16:25.512 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_background_jobs: 2 2024-09-10T18:16:25.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_background_compactions: -1 2024-09-10T18:16:25.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_subcompactions: 1 2024-09-10T18:16:25.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-10T18:16:25.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-10T18:16:25.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-10T18:16:25.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_total_wal_size: 0 2024-09-10T18:16:25.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-10T18:16:25.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-10T18:16:25.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-10T18:16:25.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-10T18:16:25.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_open_files: -1 2024-09-10T18:16:25.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.bytes_per_sync: 0 2024-09-10T18:16:25.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-10T18:16:25.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-10T18:16:25.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compaction_readahead_size: 0 2024-09-10T18:16:25.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_background_flushes: -1 2024-09-10T18:16:25.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Compression algorithms supported: 2024-09-10T18:16:25.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: kZSTD supported: 0 2024-09-10T18:16:25.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: kXpressCompression supported: 0 2024-09-10T18:16:25.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: kBZip2Compression supported: 0 2024-09-10T18:16:25.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-10T18:16:25.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: kLZ4Compression supported: 1 2024-09-10T18:16:25.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: kZlibCompression supported: 1 2024-09-10T18:16:25.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: kLZ4HCCompression supported: 1 2024-09-10T18:16:25.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: kSnappyCompression supported: 1 2024-09-10T18:16:25.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-10T18:16:25.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-09-10T18:16:25.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-10T18:16:25.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-10T18:16:25.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.merge_operator: 2024-09-10T18:16:25.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compaction_filter: None 2024-09-10T18:16:25.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compaction_filter_factory: None 2024-09-10T18:16:25.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.sst_partitioner_factory: None 2024-09-10T18:16:25.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-10T18:16:25.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-10T18:16:25.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5568b4c6fa38) 2024-09-10T18:16:25.515 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-09-10T18:16:25.515 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-10T18:16:25.515 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-10T18:16:25.515 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-09-10T18:16:25.515 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_type: 0 2024-09-10T18:16:25.516 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_index_type: 0 2024-09-10T18:16:25.516 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_shortening: 1 2024-09-10T18:16:25.516 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-10T18:16:25.516 INFO:journalctl@ceph.mon.a.smithi028.stdout: hash_index_allow_collision: 1 2024-09-10T18:16:25.516 INFO:journalctl@ceph.mon.a.smithi028.stdout: checksum: 1 2024-09-10T18:16:25.516 INFO:journalctl@ceph.mon.a.smithi028.stdout: no_block_cache: 0 2024-09-10T18:16:25.516 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache: 0x5568b4ca51f0 2024-09-10T18:16:25.516 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-09-10T18:16:25.516 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_options: 2024-09-10T18:16:25.516 INFO:journalctl@ceph.mon.a.smithi028.stdout: capacity : 536870912 2024-09-10T18:16:25.516 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_shard_bits : 4 2024-09-10T18:16:25.517 INFO:journalctl@ceph.mon.a.smithi028.stdout: strict_capacity_limit : 0 2024-09-10T18:16:25.517 INFO:journalctl@ceph.mon.a.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-09-10T18:16:25.517 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_compressed: (nil) 2024-09-10T18:16:25.517 INFO:journalctl@ceph.mon.a.smithi028.stdout: persistent_cache: (nil) 2024-09-10T18:16:25.517 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size: 4096 2024-09-10T18:16:25.517 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size_deviation: 10 2024-09-10T18:16:25.517 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_restart_interval: 16 2024-09-10T18:16:25.517 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_block_restart_interval: 1 2024-09-10T18:16:25.517 INFO:journalctl@ceph.mon.a.smithi028.stdout: metadata_block_size: 4096 2024-09-10T18:16:25.517 INFO:journalctl@ceph.mon.a.smithi028.stdout: partition_filters: 0 2024-09-10T18:16:25.517 INFO:journalctl@ceph.mon.a.smithi028.stdout: use_delta_encoding: 1 2024-09-10T18:16:25.517 INFO:journalctl@ceph.mon.a.smithi028.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-10T18:16:25.517 INFO:journalctl@ceph.mon.a.smithi028.stdout: whole_key_filtering: 1 2024-09-10T18:16:25.518 INFO:journalctl@ceph.mon.a.smithi028.stdout: verify_compression: 0 2024-09-10T18:16:25.518 INFO:journalctl@ceph.mon.a.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-09-10T18:16:25.518 INFO:journalctl@ceph.mon.a.smithi028.stdout: format_version: 4 2024-09-10T18:16:25.518 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_index_compression: 1 2024-09-10T18:16:25.518 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_align: 0 2024-09-10T18:16:25.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.write_buffer_size: 33554432 2024-09-10T18:16:25.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_write_buffer_number: 2 2024-09-10T18:16:25.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compression: NoCompression 2024-09-10T18:16:25.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.bottommost_compression: Disabled 2024-09-10T18:16:25.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.prefix_extractor: nullptr 2024-09-10T18:16:25.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-10T18:16:25.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.num_levels: 7 2024-09-10T18:16:25.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-10T18:16:25.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-10T18:16:25.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-10T18:16:25.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-10T18:16:25.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-10T18:16:25.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-10T18:16:25.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-10T18:16:25.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-10T18:16:25.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-10T18:16:25.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-10T18:16:25.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-10T18:16:25.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compression_opts.level: 32767 2024-09-10T18:16:25.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compression_opts.strategy: 0 2024-09-10T18:16:25.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-10T18:16:25.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-10T18:16:25.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-10T18:16:25.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compression_opts.enabled: false 2024-09-10T18:16:25.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-10T18:16:25.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-10T18:16:25.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-10T18:16:25.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.target_file_size_base: 67108864 2024-09-10T18:16:25.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-10T18:16:25.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-10T18:16:25.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-10T18:16:25.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-10T18:16:25.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-10T18:16:25.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-10T18:16:25.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-10T18:16:25.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-10T18:16:25.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-10T18:16:25.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-10T18:16:25.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-10T18:16:25.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-10T18:16:25.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-10T18:16:25.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.arena_block_size: 4194304 2024-09-10T18:16:25.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-10T18:16:25.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-10T18:16:25.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-10T18:16:25.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.disable_auto_compactions: 0 2024-09-10T18:16:25.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-10T18:16:25.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-10T18:16:25.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-10T18:16:25.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-10T18:16:25.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-10T18:16:25.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-10T18:16:25.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-10T18:16:25.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-10T18:16:25.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-10T18:16:25.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-10T18:16:25.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-10T18:16:25.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.inplace_update_support: 0 2024-09-10T18:16:25.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-10T18:16:25.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-10T18:16:25.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-10T18:16:25.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-10T18:16:25.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.bloom_locality: 0 2024-09-10T18:16:25.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.max_successive_merges: 0 2024-09-10T18:16:25.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-10T18:16:25.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.paranoid_file_checks: 0 2024-09-10T18:16:25.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.force_consistency_checks: 1 2024-09-10T18:16:25.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.report_bg_io_stats: 0 2024-09-10T18:16:25.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.ttl: 2592000 2024-09-10T18:16:25.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-10T18:16:25.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.enable_blob_files: false 2024-09-10T18:16:25.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.min_blob_size: 0 2024-09-10T18:16:25.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.blob_file_size: 268435456 2024-09-10T18:16:25.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-10T18:16:25.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-10T18:16:25.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-10T18:16:25.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-10T18:16:25.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-09-10T18:16:25.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-09-10T18:16:25.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725992185244896, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-09-10T18:16:25.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-09-10T18:16:25.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-10T18:16:25.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725992185245668, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 83796, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 81907, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 11423, "raw_average_key_size": 48, "raw_value_size": 75621, "raw_average_value_size": 321, "num_data_blocks": 10, "num_entries": 235, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1725992185, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "066e50de-e97b-441d-be34-d9d5e7405cb3", "db_session_id": "0WLIND0NVKEUTLPLWSZD"}} 2024-09-10T18:16:25.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-09-10T18:16:25.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725992185246026, "job": 1, "event": "recovery_finished"} 2024-09-10T18:16:25.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-10T18:16:25.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5568b4cd8700 2024-09-10T18:16:25.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: DB pointer 0x5568b4ca8000 2024-09-10T18:16:25.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-10T18:16:25.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-10T18:16:25.525 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** DB Stats ** 2024-09-10T18:16:25.525 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:16:25.525 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-10T18:16:25.525 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-10T18:16:25.526 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T18:16:25.526 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-10T18:16:25.526 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-10T18:16:25.526 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T18:16:25.526 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-10T18:16:25.526 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-09-10T18:16:25.526 INFO:journalctl@ceph.mon.a.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:16:25.526 INFO:journalctl@ceph.mon.a.smithi028.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:16:25.526 INFO:journalctl@ceph.mon.a.smithi028.stdout: L0 2/0 83.58 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.8 0.00 0.00 1 0.000 0 0 2024-09-10T18:16:25.526 INFO:journalctl@ceph.mon.a.smithi028.stdout: Sum 2/0 83.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.8 0.00 0.00 1 0.000 0 0 2024-09-10T18:16:25.526 INFO:journalctl@ceph.mon.a.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.8 0.00 0.00 1 0.000 0 0 2024-09-10T18:16:25.526 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-10T18:16:25.526 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-09-10T18:16:25.527 INFO:journalctl@ceph.mon.a.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:16:25.527 INFO:journalctl@ceph.mon.a.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:16:25.527 INFO:journalctl@ceph.mon.a.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 161.8 0.00 0.00 1 0.000 0 0 2024-09-10T18:16:25.527 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:16:25.527 INFO:journalctl@ceph.mon.a.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T18:16:25.527 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T18:16:25.527 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T18:16:25.527 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T18:16:25.527 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-10T18:16:25.527 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative compaction: 0.00 GB write, 25.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:16:25.527 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval compaction: 0.00 GB write, 25.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:16:25.527 INFO:journalctl@ceph.mon.a.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T18:16:25.527 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-10T18:16:25.527 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-10T18:16:25.528 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-10T18:16:25.528 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-09-10T18:16:25.528 INFO:journalctl@ceph.mon.a.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:16:25.528 INFO:journalctl@ceph.mon.a.smithi028.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:16:25.528 INFO:journalctl@ceph.mon.a.smithi028.stdout: L0 2/0 83.58 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.8 0.00 0.00 1 0.000 0 0 2024-09-10T18:16:25.528 INFO:journalctl@ceph.mon.a.smithi028.stdout: Sum 2/0 83.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.8 0.00 0.00 1 0.000 0 0 2024-09-10T18:16:25.528 INFO:journalctl@ceph.mon.a.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-10T18:16:25.528 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-10T18:16:25.528 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-09-10T18:16:25.528 INFO:journalctl@ceph.mon.a.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:16:25.528 INFO:journalctl@ceph.mon.a.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:16:25.528 INFO:journalctl@ceph.mon.a.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 161.8 0.00 0.00 1 0.000 0 0 2024-09-10T18:16:25.528 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:16:25.529 INFO:journalctl@ceph.mon.a.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T18:16:25.529 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T18:16:25.529 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T18:16:25.529 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T18:16:25.529 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-10T18:16:25.529 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative compaction: 0.00 GB write, 24.83 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:16:25.529 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:16:25.529 INFO:journalctl@ceph.mon.a.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T18:16:25.529 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-10T18:16:25.529 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-10T18:16:25.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: starting mon.a rank 0 at public addrs v1:172.21.15.28:6789/0 at bind addrs v1:172.21.15.28:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:16:25.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: mon.a@-1(???) e1 preinit fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:16:25.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: mon.a@-1(???).mds e1 new map 2024-09-10T18:16:25.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: mon.a@-1(???).mds e1 print_map 2024-09-10T18:16:25.530 INFO:journalctl@ceph.mon.a.smithi028.stdout: e1 2024-09-10T18:16:25.530 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-10T18:16:25.530 INFO:journalctl@ceph.mon.a.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-10T18:16:25.530 INFO:journalctl@ceph.mon.a.smithi028.stdout: legacy client fscid: -1 2024-09-10T18:16:25.530 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-10T18:16:25.530 INFO:journalctl@ceph.mon.a.smithi028.stdout: No filesystems configured 2024-09-10T18:16:25.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-10T18:16:25.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:16:25.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:16:25.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:16:25.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-10T18:16:25.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-10T18:16:25.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: monmap e1: 1 mons at {a=v1:172.21.15.28:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:16:25.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: fsmap 2024-09-10T18:16:25.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: osdmap e1: 0 total, 0 up, 0 in 2024-09-10T18:16:25.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 ceph-mon[29291]: mgrmap e1: no daemons active 2024-09-10T18:16:25.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 bash[29271]: c1cb1eb6243125d19433dfd1a26fe957d501c01e7abf3c4f110ef30df115a11d 2024-09-10T18:16:25.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:25 smithi028 systemd[1]: Started Ceph mon.a for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:16:26.278 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-10T18:16:26.278 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-10T18:16:26.278 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mgr... 2024-09-10T18:16:26.279 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-10T18:16:26.279 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-10T18:16:26.468 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mgr.y 2024-09-10T18:16:26.469 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mgr.y.service: Unit ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mgr.y.service not loaded. 2024-09-10T18:16:26.621 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-5eababb2-6fa0-11ef-bcea-c7b262605968.target.wants/ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@.service. 2024-09-10T18:16:26.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:26 smithi028 systemd[1]: Starting Ceph mgr.y for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:16:27.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:26 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1073163286' entity='client.admin' 2024-09-10T18:16:27.070 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:26 smithi028 podman[29503]: 2024-09-10 18:16:26.849886251 +0000 UTC m=+0.111798744 container create c0b9aeca110e9d6332e41bb2df929e07e373e3c587a9b0307fa532faa160ac80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T18:16:27.070 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:26 smithi028 podman[29503]: 2024-09-10 18:16:26.75397074 +0000 UTC m=+0.015883245 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:16:27.070 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:27 smithi028 podman[29503]: 2024-09-10 18:16:27.068926042 +0000 UTC m=+0.330838536 container init c0b9aeca110e9d6332e41bb2df929e07e373e3c587a9b0307fa532faa160ac80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-10T18:16:27.176 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-10T18:16:27.177 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-09-10T18:16:27.184 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-10T18:16:27.185 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-10T18:16:27.185 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-09-10T18:16:27.185 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-09-10T18:16:27.194 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-10T18:16:27.194 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-09-10T18:16:27.202 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-10T18:16:27.202 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-10T18:16:27.202 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-09-10T18:16:27.203 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-10T18:16:27.203 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr to start... 2024-09-10T18:16:27.203 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr... 2024-09-10T18:16:27.321 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:27 smithi028 podman[29503]: 2024-09-10 18:16:27.072062376 +0000 UTC m=+0.333974885 container start c0b9aeca110e9d6332e41bb2df929e07e373e3c587a9b0307fa532faa160ac80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-09-10T18:16:27.321 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:27 smithi028 bash[29503]: c0b9aeca110e9d6332e41bb2df929e07e373e3c587a9b0307fa532faa160ac80 2024-09-10T18:16:27.321 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:27 smithi028 systemd[1]: Started Ceph mgr.y for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:16:27.321 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:27 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:27.190+0000 7f0ed2c31ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T18:16:27.322 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:27 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:27.251+0000 7f0ed2c31ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T18:16:27.322 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:27 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:27.318+0000 7f0ed2c31ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T18:16:27.590 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:27 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:27.447+0000 7f0ed2c31ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:16:27.591 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:27 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:27.588+0000 7f0ed2c31ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T18:16:28.254 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:27 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:27.859+0000 7f0ed2c31ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T18:16:28.254 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:27 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:27.921+0000 7f0ed2c31ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T18:16:28.254 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:28 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:28.088+0000 7f0ed2c31ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T18:16:29.254 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:28 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:28.972+0000 7f0ed2c31ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T18:16:29.254 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:29 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:29.039+0000 7f0ed2c31ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T18:16:30.095 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:29 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:29.788+0000 7f0ed2c31ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T18:16:30.095 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:29 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:29.905+0000 7f0ed2c31ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T18:16:30.096 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:30 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:30.030+0000 7f0ed2c31ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:16:30.096 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:30 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:30.093+0000 7f0ed2c31ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T18:16:30.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:30 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1486179416' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-10T18:16:30.364 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:30 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:30.232+0000 7f0ed2c31ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T18:16:30.364 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:30 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:30.302+0000 7f0ed2c31ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T18:16:30.365 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:30 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:30.362+0000 7f0ed2c31ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T18:16:30.637 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:30 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:30.501+0000 7f0ed2c31ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T18:16:30.637 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:30 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:30.567+0000 7f0ed2c31ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T18:16:30.637 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:30 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:30.635+0000 7f0ed2c31ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T18:16:30.943 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:30 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:30.719+0000 7f0ed2c31ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T18:16:30.943 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:30 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:30.779+0000 7f0ed2c31ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T18:16:30.943 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:30 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:30.838+0000 7f0ed2c31ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T18:16:32.004 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:31.624+0000 7f0ed2c31ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T18:16:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:31 smithi028 ceph-mon[29291]: Activating manager daemon y 2024-09-10T18:16:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:31 smithi028 ceph-mon[29291]: mgrmap e2: y(active, starting, since 0.00305509s) 2024-09-10T18:16:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:31 smithi028 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.28:0/4103255368' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T18:16:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:31 smithi028 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.28:0/4103255368' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T18:16:32.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:31 smithi028 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.28:0/4103255368' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T18:16:32.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:31 smithi028 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.28:0/4103255368' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T18:16:32.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:31 smithi028 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.28:0/4103255368' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T18:16:32.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:31 smithi028 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.28:0/4103255368' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T18:16:32.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:31 smithi028 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.28:0/4103255368' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:16:32.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:31 smithi028 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.28:0/4103255368' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T18:16:32.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:31 smithi028 ceph-mon[29291]: Manager daemon y is now available 2024-09-10T18:16:32.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:31 smithi028 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.28:0/4103255368' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:16:32.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:31 smithi028 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.28:0/4103255368' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:16:32.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:31 smithi028 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.28:0/4103255368' entity='mgr.y' 2024-09-10T18:16:32.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:31 smithi028 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.28:0/4103255368' entity='mgr.y' 2024-09-10T18:16:32.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:31 smithi028 ceph-mon[29291]: from='mgr.14100 v1:172.21.15.28:0/4103255368' entity='mgr.y' 2024-09-10T18:16:34.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:33 smithi028 ceph-mon[29291]: mgrmap e3: y(active, since 1.00697s) 2024-09-10T18:16:35.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:34 smithi028 ceph-mon[29291]: mgrmap e4: y(active, since 2s) 2024-09-10T18:16:35.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-10T18:16:35.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:16:35.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "5eababb2-6fa0-11ef-bcea-c7b262605968", 2024-09-10T18:16:35.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-09-10T18:16:35.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-10T18:16:35.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-10T18:16:35.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-10T18:16:35.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:16:35.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-10T18:16:35.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-10T18:16:35.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-09-10T18:16:35.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:16:35.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-10T18:16:35.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-09-10T18:16:35.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:16:35.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-09-10T18:16:35.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-10T18:16:35.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:16:35.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-10T18:16:35.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-10T18:16:35.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:16:35.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-10T18:16:35.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:16:35.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-10T18:16:35.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-10T18:16:35.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-10T18:16:35.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-10T18:16:35.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-10T18:16:35.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-10T18:16:35.549 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:16:35.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-10T18:16:35.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-10T18:16:35.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-10T18:16:35.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-10T18:16:35.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-10T18:16:35.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-10T18:16:35.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-10T18:16:35.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-10T18:16:35.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-10T18:16:35.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:16:35.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-10T18:16:35.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:16:35.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-10T18:16:35.550 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-10T18:16:35.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:16:35.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-10T18:16:35.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-10T18:16:35.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-10T18:16:35.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-10T18:16:35.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-10T18:16:35.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-10T18:16:35.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-09-10T18:16:35.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:16:35.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T18:16:35.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:16:35.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-10T18:16:35.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:16:35.551 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-09-10T18:16:20.680815+0000", 2024-09-10T18:16:35.552 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T18:16:35.552 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:16:35.552 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-10T18:16:35.552 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:16:35.552 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (1/15)... 2024-09-10T18:16:38.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:38 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/414243083' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-10T18:16:38.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-10T18:16:38.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:16:38.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "5eababb2-6fa0-11ef-bcea-c7b262605968", 2024-09-10T18:16:38.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-09-10T18:16:38.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-10T18:16:38.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-10T18:16:38.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-10T18:16:38.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:16:38.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-10T18:16:38.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-10T18:16:38.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-09-10T18:16:38.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:16:38.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-10T18:16:38.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-09-10T18:16:38.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:16:38.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-09-10T18:16:38.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-10T18:16:38.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:16:38.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-10T18:16:38.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-10T18:16:38.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:16:38.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-10T18:16:38.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:16:38.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-10T18:16:38.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-10T18:16:38.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-10T18:16:38.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-10T18:16:38.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-10T18:16:38.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-10T18:16:38.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:16:38.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-10T18:16:38.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-10T18:16:38.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-10T18:16:38.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-10T18:16:38.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-10T18:16:38.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-10T18:16:38.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-10T18:16:38.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-10T18:16:38.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-10T18:16:38.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:16:38.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-10T18:16:38.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:16:38.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-10T18:16:38.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-10T18:16:38.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:16:38.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-10T18:16:38.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-10T18:16:38.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-10T18:16:38.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-10T18:16:38.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-10T18:16:38.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-10T18:16:38.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-09-10T18:16:38.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:16:38.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T18:16:38.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:16:38.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-10T18:16:38.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:16:38.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-09-10T18:16:20.680815+0000", 2024-09-10T18:16:38.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T18:16:38.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:16:38.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-10T18:16:38.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:16:38.731 INFO:teuthology.orchestra.run.smithi028.stdout:mgr is available 2024-09-10T18:16:39.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:39 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1961236098' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-10T18:16:39.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-10T18:16:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-09-10T18:16:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:16:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-10T18:16:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-10T18:16:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-10T18:16:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-10T18:16:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-10T18:16:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-10T18:16:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-10T18:16:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-10T18:16:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-10T18:16:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-09-10T18:16:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-10T18:16:39.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-10T18:16:39.940 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling cephadm module... 2024-09-10T18:16:40.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:40 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2521620821' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-10T18:16:40.884 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:40 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ignoring --setuser ceph since I am not root 2024-09-10T18:16:40.884 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:40 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ignoring --setgroup ceph since I am not root 2024-09-10T18:16:40.884 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:40 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:40.758+0000 7f7ad8411ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T18:16:40.884 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:40 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:40.815+0000 7f7ad8411ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T18:16:40.884 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:40 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:40.881+0000 7f7ad8411ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T18:16:41.153 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:41.010+0000 7f7ad8411ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:16:41.154 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:41.151+0000 7f7ad8411ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T18:16:41.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:41 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2521620821' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-10T18:16:41.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:41 smithi028 ceph-mon[29291]: mgrmap e5: y(active, since 9s) 2024-09-10T18:16:41.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:41.418+0000 7f7ad8411ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T18:16:41.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:41.480+0000 7f7ad8411ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T18:16:41.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:41.649+0000 7f7ad8411ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T18:16:42.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:42 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:42.534+0000 7f7ad8411ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T18:16:42.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:42 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:42.603+0000 7f7ad8411ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T18:16:43.657 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:43 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:43.350+0000 7f7ad8411ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T18:16:43.657 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:43 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:43.468+0000 7f7ad8411ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T18:16:43.657 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:43 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:43.593+0000 7f7ad8411ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:16:43.657 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:43 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:43.654+0000 7f7ad8411ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T18:16:43.916 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:43 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:43.785+0000 7f7ad8411ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T18:16:43.916 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:43 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:43.854+0000 7f7ad8411ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T18:16:43.916 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:43 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:43.914+0000 7f7ad8411ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T18:16:44.190 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:44 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:44.055+0000 7f7ad8411ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T18:16:44.191 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:44 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:44.121+0000 7f7ad8411ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T18:16:44.191 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:44 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:44.188+0000 7f7ad8411ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T18:16:44.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:44 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:44.278+0000 7f7ad8411ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T18:16:44.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:44 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:44.341+0000 7f7ad8411ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T18:16:44.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:44 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:44.400+0000 7f7ad8411ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T18:16:45.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:44 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1822380142' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-10T18:16:45.481 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:45.194+0000 7f7ad8411ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T18:16:45.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:16:45.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-10T18:16:45.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-10T18:16:45.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-10T18:16:45.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-10T18:16:45.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:16:45.485 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-09-10T18:16:45.485 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 5... 2024-09-10T18:16:46.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-mon[29291]: Active manager daemon y restarted 2024-09-10T18:16:46.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-mon[29291]: Activating manager daemon y 2024-09-10T18:16:46.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-mon[29291]: osdmap e2: 0 total, 0 up, 0 in 2024-09-10T18:16:46.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-mon[29291]: mgrmap e6: y(active, starting, since 0.00308847s) 2024-09-10T18:16:46.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:16:46.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T18:16:46.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T18:16:46.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T18:16:46.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T18:16:46.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-mon[29291]: Manager daemon y is now available 2024-09-10T18:16:46.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-mon[29291]: Found migration_current of "None". Setting to last migration. 2024-09-10T18:16:46.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' 2024-09-10T18:16:46.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' 2024-09-10T18:16:46.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:16:46.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:16:46.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:16:46.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:45 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:16:46.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:16:46.784 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-10T18:16:46.784 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-10T18:16:46.784 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:16:46.784 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 5 is available 2024-09-10T18:16:46.784 INFO:teuthology.orchestra.run.smithi028.stdout:Setting orchestrator backend to cephadm... 2024-09-10T18:16:47.341 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:47 smithi028 ceph-mon[29291]: from='client.14126 v1:172.21.15.28:0/2904914393' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-10T18:16:47.341 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:47 smithi028 ceph-mon[29291]: mgrmap e7: y(active, since 1.00508s) 2024-09-10T18:16:47.341 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:47 smithi028 ceph-mon[29291]: from='client.14126 v1:172.21.15.28:0/2904914393' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-10T18:16:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-mon[29291]: mgrmap e8: y(active, since 2s) 2024-09-10T18:16:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-mon[29291]: from='client.14128 v1:172.21.15.28:0/193023500' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:16:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' 2024-09-10T18:16:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:16:48.755 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:48.405+0000 7f7a8a7bf640 -1 log_channel(cephadm) log [ERR] : [10/Sep/2024:18:16:48] ENGINE Error in HTTPServer.serve 2024-09-10T18:16:48.755 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: Traceback (most recent call last): 2024-09-10T18:16:48.755 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:16:48.755 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: self._connections.run(self.expiration_interval) 2024-09-10T18:16:48.755 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:16:48.755 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: self._run(expiration_interval) 2024-09-10T18:16:48.755 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:16:48.755 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:16:48.755 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:16:48.755 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:16:48.755 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:16:48.755 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: s = self.context.wrap_socket( 2024-09-10T18:16:48.756 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:16:48.756 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: return self.sslsocket_class._create( 2024-09-10T18:16:48.756 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:16:48.756 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: self.do_handshake() 2024-09-10T18:16:48.756 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:16:48.756 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: self._sslobj.do_handshake() 2024-09-10T18:16:48.756 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:16:48.756 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:48 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:16:49.013 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-10T18:16:49.013 INFO:teuthology.orchestra.run.smithi028.stdout:Generating ssh key... 2024-09-10T18:16:49.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-mon[29291]: [10/Sep/2024:18:16:48] ENGINE Bus STARTING 2024-09-10T18:16:49.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-mon[29291]: [10/Sep/2024:18:16:48] ENGINE Serving on https://172.21.15.28:7150 2024-09-10T18:16:49.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-mon[29291]: [10/Sep/2024:18:16:48] ENGINE Bus STARTED 2024-09-10T18:16:49.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-mon[29291]: [10/Sep/2024:18:16:48] ENGINE Error in HTTPServer.serve 2024-09-10T18:16:49.677 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-09-10T18:16:49.677 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:16:49.677 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._connections.run(self.expiration_interval) 2024-09-10T18:16:49.677 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:16:49.677 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._run(expiration_interval) 2024-09-10T18:16:49.677 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:16:49.677 INFO:journalctl@ceph.mon.a.smithi028.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:16:49.677 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:16:49.677 INFO:journalctl@ceph.mon.a.smithi028.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:16:49.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:16:49.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: s = self.context.wrap_socket( 2024-09-10T18:16:49.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:16:49.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.sslsocket_class._create( 2024-09-10T18:16:49.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:16:49.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: self.do_handshake() 2024-09-10T18:16:49.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:16:49.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._sslobj.do_handshake() 2024-09-10T18:16:49.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:16:49.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' 2024-09-10T18:16:49.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:16:49.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-mon[29291]: from='client.14130 v1:172.21.15.28:0/930657000' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:16:49.974 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: Generating public/private rsa key pair. 2024-09-10T18:16:49.974 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: Your identification has been saved in /tmp/tmpx53v1wbd/key 2024-09-10T18:16:49.974 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: Your public key has been saved in /tmp/tmpx53v1wbd/key.pub 2024-09-10T18:16:49.974 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: The key fingerprint is: 2024-09-10T18:16:49.975 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: SHA256:Aksv9gESJ/6ZQCLECdh55l0EmlTWlJrUWZeLhYPMZxo ceph-5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:16:49.975 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: The key's randomart image is: 2024-09-10T18:16:49.975 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: +---[RSA 3072]----+ 2024-09-10T18:16:49.975 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: |Oo=.o.+Oo=.... | 2024-09-10T18:16:49.975 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: |o*o=o+. E =.o | 2024-09-10T18:16:49.975 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: | ++*o + = + . | 2024-09-10T18:16:49.975 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: | =.B+ . . . | 2024-09-10T18:16:49.975 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: | B + S | 2024-09-10T18:16:49.975 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: | . o o | 2024-09-10T18:16:49.975 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: | . | 2024-09-10T18:16:49.975 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: | | 2024-09-10T18:16:49.975 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: | | 2024-09-10T18:16:49.975 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:16:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: +----[SHA256]-----+ 2024-09-10T18:16:50.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:50 smithi028 ceph-mon[29291]: from='client.14132 v1:172.21.15.28:0/855747769' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:16:50.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:50 smithi028 ceph-mon[29291]: Generating ssh key... 2024-09-10T18:16:50.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:50 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' 2024-09-10T18:16:50.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:50 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' 2024-09-10T18:16:51.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeGR3NNj2cbsrZ3wIsttcP3l310e9ueLprZAnrGqzvAC0YGxu9DxJEDm/sugkl7CGkop6+1+PEOZ2gFsLhpM+LStcur7LV96AipqjQdbeinBGmzk8zpoYG8kZqGMSxzsZJFC1tDwYVkbht0tYw3D87YkRMSXlUc87IqJeEQFX6EFHVD4iSIMYwhwEWl93qLk47vSaXdgMxT298ihWSaiIF76FnMefUfGCUCGcTwQ4CniTiQtN1p7P74QEKUQPPJX/QhOX2YgZxUdS0XIFPCewxyFU47Wnhv9pJrBB/BzzI2EnAKtrcQO27hKMHN4BneRb3Ng4Z1qUII4kjMQy+CpYIcSNQlq1rYzRTBRczG5e1Z42AG81i//jydE6zuzuV2QHYcUnzUR7l2XZr+0/tHXC4/RTw6qipbNQrcB/t8s7EzuxW6aALQG1v9i+Ikwnbos4IxetCbz+zPlsA0ptL+OzevHeeB4Miu+EtVxp7Nqu6k0/iUFT1Gnw3DH5Z9rlTnas= ceph-5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:16:51.417 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-10T18:16:51.417 INFO:teuthology.orchestra.run.smithi028.stdout:Adding key to root@localhost authorized_keys... 2024-09-10T18:16:51.425 INFO:teuthology.orchestra.run.smithi028.stdout:Adding host smithi028... 2024-09-10T18:16:52.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:51 smithi028 ceph-mon[29291]: from='client.14134 v1:172.21.15.28:0/2336029354' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:16:53.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:52 smithi028 ceph-mon[29291]: from='client.14136 v1:172.21.15.28:0/208563391' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:16:53.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:53 smithi028 ceph-mon[29291]: Deploying cephadm binary to smithi028 2024-09-10T18:16:55.975 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Added host 'smithi028' with addr '172.21.15.28' 2024-09-10T18:16:55.976 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mon service... 2024-09-10T18:16:56.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:56 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' 2024-09-10T18:16:56.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:56 smithi028 ceph-mon[29291]: Added host smithi028 2024-09-10T18:16:56.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:56 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:16:57.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-10T18:16:57.317 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mgr service... 2024-09-10T18:16:57.836 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:57 smithi028 ceph-mon[29291]: from='client.14138 v1:172.21.15.28:0/1067905052' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:16:57.836 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:57 smithi028 ceph-mon[29291]: Saving service mon spec with placement count:5 2024-09-10T18:16:57.836 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:57 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' 2024-09-10T18:16:59.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:58 smithi028 ceph-mon[29291]: from='client.14140 v1:172.21.15.28:0/1870674853' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:16:59.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:58 smithi028 ceph-mon[29291]: Saving service mgr spec with placement count:2 2024-09-10T18:16:59.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:58 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' 2024-09-10T18:16:59.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:16:58 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' 2024-09-10T18:16:59.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-10T18:17:00.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:00 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' 2024-09-10T18:17:00.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:00 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2909058962' entity='client.admin' 2024-09-10T18:17:01.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:01 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' 2024-09-10T18:17:01.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:01 smithi028 ceph-mon[29291]: from='mgr.14116 v1:172.21.15.28:0/2259212339' entity='mgr.y' 2024-09-10T18:17:01.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:01 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1047208304' entity='client.admin' 2024-09-10T18:17:02.181 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling the dashboard module... 2024-09-10T18:17:03.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:02 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/479522861' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-10T18:17:03.189 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:02 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ignoring --setuser ceph since I am not root 2024-09-10T18:17:03.189 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:02 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ignoring --setgroup ceph since I am not root 2024-09-10T18:17:03.189 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:03 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:03.061+0000 7f20f605fec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T18:17:03.189 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:03 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:03.119+0000 7f20f605fec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T18:17:03.189 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:03 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:03.186+0000 7f20f605fec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T18:17:03.462 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:03 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:03.317+0000 7f20f605fec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:17:03.462 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:03 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:03.460+0000 7f20f605fec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T18:17:04.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:03 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/479522861' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-10T18:17:04.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:03 smithi028 ceph-mon[29291]: mgrmap e9: y(active, since 17s) 2024-09-10T18:17:04.004 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:03 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:03.730+0000 7f20f605fec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T18:17:04.004 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:03 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:03.792+0000 7f20f605fec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T18:17:04.004 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:03 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:03.967+0000 7f20f605fec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T18:17:05.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:04 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4091461852' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-10T18:17:05.132 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:04 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:04.869+0000 7f20f605fec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T18:17:05.132 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:04 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:04.938+0000 7f20f605fec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T18:17:05.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:17:05.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-10T18:17:05.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-10T18:17:05.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-10T18:17:05.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-10T18:17:05.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:17:05.419 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-09-10T18:17:05.419 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 9... 2024-09-10T18:17:06.002 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:05 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:05.692+0000 7f20f605fec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T18:17:06.002 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:05 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:05.810+0000 7f20f605fec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T18:17:06.002 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:05 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:05.938+0000 7f20f605fec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:17:06.003 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:06 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:06.000+0000 7f20f605fec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T18:17:06.254 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:06 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:06.131+0000 7f20f605fec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T18:17:06.254 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:06 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:06.201+0000 7f20f605fec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T18:17:06.537 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:06 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:06.261+0000 7f20f605fec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T18:17:06.537 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:06 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:06.401+0000 7f20f605fec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T18:17:06.538 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:06 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:06.467+0000 7f20f605fec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T18:17:06.538 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:06 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:06.535+0000 7f20f605fec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T18:17:07.004 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:06 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:06.619+0000 7f20f605fec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T18:17:07.004 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:06 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:06.679+0000 7f20f605fec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T18:17:07.004 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:06 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:06.738+0000 7f20f605fec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T18:17:07.775 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:07 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:07.518+0000 7f20f605fec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T18:17:07.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:07 smithi028 ceph-mon[29291]: Active manager daemon y restarted 2024-09-10T18:17:07.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:07 smithi028 ceph-mon[29291]: Activating manager daemon y 2024-09-10T18:17:07.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:07 smithi028 ceph-mon[29291]: osdmap e3: 0 total, 0 up, 0 in 2024-09-10T18:17:07.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:07 smithi028 ceph-mon[29291]: mgrmap e10: y(active, starting, since 0.00310944s) 2024-09-10T18:17:07.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:07 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:17:07.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:07 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T18:17:07.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:07 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T18:17:07.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:07 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T18:17:07.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:07 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T18:17:07.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:07 smithi028 ceph-mon[29291]: Manager daemon y is now available 2024-09-10T18:17:07.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:07 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:17:07.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:07 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:17:07.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:07 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:17:09.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:17:09.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-10T18:17:09.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-10T18:17:09.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:17:09.462 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 9 is available 2024-09-10T18:17:09.462 INFO:teuthology.orchestra.run.smithi028.stdout:Generating a dashboard self-signed certificate... 2024-09-10T18:17:09.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:09 smithi028 ceph-mon[29291]: from='client.14152 v1:172.21.15.28:0/3728393971' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-10T18:17:09.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:09 smithi028 ceph-mon[29291]: mgrmap e11: y(active, since 1.00523s) 2024-09-10T18:17:09.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:09 smithi028 ceph-mon[29291]: from='client.14152 v1:172.21.15.28:0/3728393971' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-10T18:17:09.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:09 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:09.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:09 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-mon[29291]: from='client.14160 v1:172.21.15.28:0/48350500' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:17:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-mon[29291]: mgrmap e12: y(active, since 2s) 2024-09-10T18:17:11.520 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-10T18:17:11.520 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial admin user... 2024-09-10T18:17:11.805 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:11.681+0000 7f20a6abc640 -1 log_channel(cephadm) log [ERR] : [10/Sep/2024:18:17:11] ENGINE Error in HTTPServer.serve 2024-09-10T18:17:11.805 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: Traceback (most recent call last): 2024-09-10T18:17:11.806 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:17:11.806 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: self._connections.run(self.expiration_interval) 2024-09-10T18:17:11.806 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:17:11.806 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: self._run(expiration_interval) 2024-09-10T18:17:11.806 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:17:11.806 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:17:11.806 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:17:11.806 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:17:11.806 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:17:11.806 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: s = self.context.wrap_socket( 2024-09-10T18:17:11.806 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:17:11.806 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: return self.sslsocket_class._create( 2024-09-10T18:17:11.806 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:17:11.806 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: self.do_handshake() 2024-09-10T18:17:11.807 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:17:11.807 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: self._sslobj.do_handshake() 2024-09-10T18:17:11.807 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:17:11.807 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:13.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:12 smithi028 ceph-mon[29291]: [10/Sep/2024:18:17:11] ENGINE Bus STARTING 2024-09-10T18:17:13.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:12 smithi028 ceph-mon[29291]: [10/Sep/2024:18:17:11] ENGINE Serving on https://172.21.15.28:7150 2024-09-10T18:17:13.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:12 smithi028 ceph-mon[29291]: [10/Sep/2024:18:17:11] ENGINE Bus STARTED 2024-09-10T18:17:13.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:12 smithi028 ceph-mon[29291]: [10/Sep/2024:18:17:11] ENGINE Error in HTTPServer.serve 2024-09-10T18:17:13.004 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-09-10T18:17:13.004 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:17:13.004 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._connections.run(self.expiration_interval) 2024-09-10T18:17:13.005 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:17:13.005 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._run(expiration_interval) 2024-09-10T18:17:13.005 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:17:13.005 INFO:journalctl@ceph.mon.a.smithi028.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:17:13.005 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:17:13.005 INFO:journalctl@ceph.mon.a.smithi028.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:17:13.005 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:17:13.005 INFO:journalctl@ceph.mon.a.smithi028.stdout: s = self.context.wrap_socket( 2024-09-10T18:17:13.005 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:17:13.005 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.sslsocket_class._create( 2024-09-10T18:17:13.005 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:17:13.005 INFO:journalctl@ceph.mon.a.smithi028.stdout: self.do_handshake() 2024-09-10T18:17:13.005 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:17:13.006 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._sslobj.do_handshake() 2024-09-10T18:17:13.006 INFO:journalctl@ceph.mon.a.smithi028.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:17:13.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:12 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:13.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:12 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:13.090 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$rzvUrnkVGQL2PL7jlqpWzu16eCiysHUYzyHoHwQPak8Ugb3edVl6C", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1725992232, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-10T18:17:13.090 INFO:teuthology.orchestra.run.smithi028.stdout:Fetching dashboard port number... 2024-09-10T18:17:13.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:13 smithi028 ceph-mon[29291]: from='client.14162 v1:172.21.15.28:0/3160994872' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:17:13.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:13 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1596279469' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-10T18:17:14.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 8443 2024-09-10T18:17:14.242 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-10T18:17:14.243 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-09-10T18:17:14.251 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-10T18:17:14.251 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-10T18:17:14.251 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-09-10T18:17:14.251 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-10T18:17:14.253 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph Dashboard is now available at: 2024-09-10T18:17:14.253 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:17:14.253 INFO:teuthology.orchestra.run.smithi028.stdout: URL: https://smithi028.front.sepia.ceph.com:8443/ 2024-09-10T18:17:14.253 INFO:teuthology.orchestra.run.smithi028.stdout: User: admin 2024-09-10T18:17:14.254 INFO:teuthology.orchestra.run.smithi028.stdout: Password: fcyvtmrn4a 2024-09-10T18:17:14.254 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:17:14.254 INFO:teuthology.orchestra.run.smithi028.stdout:Saving cluster configuration to /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config directory 2024-09-10T18:17:14.254 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling autotune for osd_memory_target 2024-09-10T18:17:14.806 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:14 smithi028 ceph-mon[29291]: mgrmap e13: y(active, since 6s) 2024-09-10T18:17:16.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-10T18:17:16.769 INFO:teuthology.orchestra.run.smithi028.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-10T18:17:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:17:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-10T18:17:16.770 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:17:16.770 INFO:teuthology.orchestra.run.smithi028.stdout:Or, if you are only running a single cluster on this host: 2024-09-10T18:17:16.770 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:17:16.770 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-10T18:17:16.770 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:17:16.770 INFO:teuthology.orchestra.run.smithi028.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-10T18:17:16.770 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:17:16.770 INFO:teuthology.orchestra.run.smithi028.stdout: ceph telemetry on 2024-09-10T18:17:16.770 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:17:16.770 INFO:teuthology.orchestra.run.smithi028.stdout:For more information see: 2024-09-10T18:17:16.770 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:17:16.770 INFO:teuthology.orchestra.run.smithi028.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-09-10T18:17:16.771 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:17:16.771 INFO:teuthology.orchestra.run.smithi028.stdout:Bootstrap complete. 2024-09-10T18:17:16.810 INFO:tasks.cephadm:Fetching config... 2024-09-10T18:17:16.810 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T18:17:16.810 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-10T18:17:16.829 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-10T18:17:16.830 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T18:17:16.830 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-10T18:17:16.887 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-10T18:17:16.887 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T18:17:16.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-10T18:17:16.953 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-10T18:17:16.953 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T18:17:16.953 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-10T18:17:17.010 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-10T18:17:17.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeGR3NNj2cbsrZ3wIsttcP3l310e9ueLprZAnrGqzvAC0YGxu9DxJEDm/sugkl7CGkop6+1+PEOZ2gFsLhpM+LStcur7LV96AipqjQdbeinBGmzk8zpoYG8kZqGMSxzsZJFC1tDwYVkbht0tYw3D87YkRMSXlUc87IqJeEQFX6EFHVD4iSIMYwhwEWl93qLk47vSaXdgMxT298ihWSaiIF76FnMefUfGCUCGcTwQ4CniTiQtN1p7P74QEKUQPPJX/QhOX2YgZxUdS0XIFPCewxyFU47Wnhv9pJrBB/BzzI2EnAKtrcQO27hKMHN4BneRb3Ng4Z1qUII4kjMQy+CpYIcSNQlq1rYzRTBRczG5e1Z42AG81i//jydE6zuzuV2QHYcUnzUR7l2XZr+0/tHXC4/RTw6qipbNQrcB/t8s7EzuxW6aALQG1v9i+Ikwnbos4IxetCbz+zPlsA0ptL+OzevHeeB4Miu+EtVxp7Nqu6k0/iUFT1Gnw3DH5Z9rlTnas= ceph-5eababb2-6fa0-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-10T18:17:17.086 INFO:teuthology.orchestra.run.smithi028.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeGR3NNj2cbsrZ3wIsttcP3l310e9ueLprZAnrGqzvAC0YGxu9DxJEDm/sugkl7CGkop6+1+PEOZ2gFsLhpM+LStcur7LV96AipqjQdbeinBGmzk8zpoYG8kZqGMSxzsZJFC1tDwYVkbht0tYw3D87YkRMSXlUc87IqJeEQFX6EFHVD4iSIMYwhwEWl93qLk47vSaXdgMxT298ihWSaiIF76FnMefUfGCUCGcTwQ4CniTiQtN1p7P74QEKUQPPJX/QhOX2YgZxUdS0XIFPCewxyFU47Wnhv9pJrBB/BzzI2EnAKtrcQO27hKMHN4BneRb3Ng4Z1qUII4kjMQy+CpYIcSNQlq1rYzRTBRczG5e1Z42AG81i//jydE6zuzuV2QHYcUnzUR7l2XZr+0/tHXC4/RTw6qipbNQrcB/t8s7EzuxW6aALQG1v9i+Ikwnbos4IxetCbz+zPlsA0ptL+OzevHeeB4Miu+EtVxp7Nqu6k0/iUFT1Gnw3DH5Z9rlTnas= ceph-5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:17:17.097 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeGR3NNj2cbsrZ3wIsttcP3l310e9ueLprZAnrGqzvAC0YGxu9DxJEDm/sugkl7CGkop6+1+PEOZ2gFsLhpM+LStcur7LV96AipqjQdbeinBGmzk8zpoYG8kZqGMSxzsZJFC1tDwYVkbht0tYw3D87YkRMSXlUc87IqJeEQFX6EFHVD4iSIMYwhwEWl93qLk47vSaXdgMxT298ihWSaiIF76FnMefUfGCUCGcTwQ4CniTiQtN1p7P74QEKUQPPJX/QhOX2YgZxUdS0XIFPCewxyFU47Wnhv9pJrBB/BzzI2EnAKtrcQO27hKMHN4BneRb3Ng4Z1qUII4kjMQy+CpYIcSNQlq1rYzRTBRczG5e1Z42AG81i//jydE6zuzuV2QHYcUnzUR7l2XZr+0/tHXC4/RTw6qipbNQrcB/t8s7EzuxW6aALQG1v9i+Ikwnbos4IxetCbz+zPlsA0ptL+OzevHeeB4Miu+EtVxp7Nqu6k0/iUFT1Gnw3DH5Z9rlTnas= ceph-5eababb2-6fa0-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-10T18:17:17.437 INFO:teuthology.orchestra.run.smithi204.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeGR3NNj2cbsrZ3wIsttcP3l310e9ueLprZAnrGqzvAC0YGxu9DxJEDm/sugkl7CGkop6+1+PEOZ2gFsLhpM+LStcur7LV96AipqjQdbeinBGmzk8zpoYG8kZqGMSxzsZJFC1tDwYVkbht0tYw3D87YkRMSXlUc87IqJeEQFX6EFHVD4iSIMYwhwEWl93qLk47vSaXdgMxT298ihWSaiIF76FnMefUfGCUCGcTwQ4CniTiQtN1p7P74QEKUQPPJX/QhOX2YgZxUdS0XIFPCewxyFU47Wnhv9pJrBB/BzzI2EnAKtrcQO27hKMHN4BneRb3Ng4Z1qUII4kjMQy+CpYIcSNQlq1rYzRTBRczG5e1Z42AG81i//jydE6zuzuV2QHYcUnzUR7l2XZr+0/tHXC4/RTw6qipbNQrcB/t8s7EzuxW6aALQG1v9i+Ikwnbos4IxetCbz+zPlsA0ptL+OzevHeeB4Miu+EtVxp7Nqu6k0/iUFT1Gnw3DH5Z9rlTnas= ceph-5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:17:17.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-10T18:17:17.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:17 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1390873364' entity='client.admin' 2024-09-10T18:17:17.707 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:17:19.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:19.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:19.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:17:19.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:19.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:17:19.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:21.676 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-10T18:17:21.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-10T18:17:21.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:21 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2610593833' entity='client.admin' 2024-09-10T18:17:21.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:21 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:17:21.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:21 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:21.973 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:17:24.364 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi204 2024-09-10T18:17:24.364 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-10T18:17:24.364 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.conf 2024-09-10T18:17:24.384 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-10T18:17:24.384 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:17:24.442 INFO:tasks.cephadm:Adding host smithi204 to orchestrator... 2024-09-10T18:17:24.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch host add smithi204 2024-09-10T18:17:24.689 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:17:24.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:24 smithi028 ceph-mon[29291]: from='client.14172 v1:172.21.15.28:0/2702416072' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:17:24.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:24 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:24.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:24 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:17:24.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:24 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:17:24.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:24 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:17:24.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:24 smithi028 ceph-mon[29291]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-10T18:17:24.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:24 smithi028 ceph-mon[29291]: Updating smithi028:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:17:24.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:24 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:24.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:24 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:24.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:24 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:25.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:25 smithi028 ceph-mon[29291]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:17:25.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:25 smithi028 ceph-mon[29291]: Updating smithi028:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:17:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:27 smithi028 ceph-mon[29291]: from='client.14174 v1:172.21.15.28:0/4193831445' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:17:29.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:28 smithi028 ceph-mon[29291]: Deploying cephadm binary to smithi204 2024-09-10T18:17:29.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:28 smithi028 ceph-mon[29291]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:29.625 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi204' with addr '172.21.15.204' 2024-09-10T18:17:30.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch host ls --format=json 2024-09-10T18:17:30.496 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:17:30.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:30 smithi028 ceph-mon[29291]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:30.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:30.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:30 smithi028 ceph-mon[29291]: Added host smithi204 2024-09-10T18:17:30.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:17:30.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:30.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:32.174 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:17:32.174 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.204", "hostname": "smithi204", "labels": [], "status": ""}] 2024-09-10T18:17:32.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:31 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:32.776 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-10T18:17:32.776 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd crush tunables default 2024-09-10T18:17:33.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:32 smithi028 ceph-mon[29291]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:33.024 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:17:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:33 smithi028 ceph-mon[29291]: from='client.14176 v1:172.21.15.28:0/806333114' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:17:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:33 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:33 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:33 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:33 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:33 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:17:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:33 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:17:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:33 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:17:35.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:34 smithi028 ceph-mon[29291]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:35.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:34 smithi028 ceph-mon[29291]: Updating smithi204:/etc/ceph/ceph.conf 2024-09-10T18:17:35.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:34 smithi028 ceph-mon[29291]: Updating smithi204:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:17:35.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:34 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:35.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:34 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:35.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:34 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:35.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:34 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/126667441' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-10T18:17:35.463 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-09-10T18:17:36.049 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:35 smithi028 ceph-mon[29291]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:17:36.049 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:35 smithi028 ceph-mon[29291]: Updating smithi204:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:17:36.049 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:35 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/126667441' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-10T18:17:36.049 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:35 smithi028 ceph-mon[29291]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:17:36.050 INFO:tasks.cephadm:Adding mon.a on smithi028 2024-09-10T18:17:36.050 INFO:tasks.cephadm:Adding mon.c on smithi028 2024-09-10T18:17:36.050 INFO:tasks.cephadm:Adding mon.b on smithi204 2024-09-10T18:17:36.051 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch apply mon '3;smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi204:[v1:172.21.15.204:6789]=b' 2024-09-10T18:17:36.294 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:17:37.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:36 smithi028 ceph-mon[29291]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:38.095 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mon update... 2024-09-10T18:17:38.769 DEBUG:teuthology.orchestra.run.smithi028:mon.c> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mon.c.service 2024-09-10T18:17:38.772 DEBUG:teuthology.orchestra.run.smithi204:mon.b> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mon.b.service 2024-09-10T18:17:38.774 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-10T18:17:38.775 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T18:17:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:38 smithi028 ceph-mon[29291]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:38 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:38 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:17:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:38 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:17:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:38 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:17:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:38 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:38 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:17:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:38 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:17:39.544 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:17:40.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:39 smithi028 ceph-mon[29291]: from='client.14180 v1:172.21.15.204:0/4209441807' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi204:[v1:172.21.15.204:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:17:40.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:39 smithi028 ceph-mon[29291]: Saving service mon spec with placement smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi204:[v1:172.21.15.204:6789]=b;count:3 2024-09-10T18:17:40.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:39 smithi028 ceph-mon[29291]: Deploying daemon mon.b on smithi204 2024-09-10T18:17:41.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:40 smithi028 ceph-mon[29291]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:42.559 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:17:42.560 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":1,"fsid":"5eababb2-6fa0-11ef-bcea-c7b262605968","modified":"2024-09-10T18:16:17.798654Z","created":"2024-09-10T18:16:17.798654Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T18:17:42.560 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 1 2024-09-10T18:17:43.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:42 smithi028 ceph-mon[29291]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:43.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:42 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.204:0/410122418' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:17:43.843 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:43 smithi204 ceph-mon[32357]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-10T18:17:43.843 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:43 smithi204 bash[32338]: edec062d1cefa105b8ec5bf10f41072af5e48fc7c2340ea08909c462279cea02 2024-09-10T18:17:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:43 smithi204 systemd[1]: Started Ceph mon.b for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:17:44.338 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-10T18:17:44.339 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T18:17:44.591 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:17:49.067 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:17:49.067 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":2,"fsid":"5eababb2-6fa0-11ef-bcea-c7b262605968","modified":"2024-09-10T18:17:43.816650Z","created":"2024-09-10T18:16:17.798654Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-10T18:17:49.067 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 2 2024-09-10T18:17:49.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:49.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:17:49.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:49.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: mon.a calling monitor election 2024-09-10T18:17:49.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: mon.b calling monitor election 2024-09-10T18:17:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-10T18:17:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: monmap e2: 2 mons at {a=v1:172.21.15.28:6789/0,b=v1:172.21.15.204:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:17:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: fsmap 2024-09-10T18:17:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:17:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: mgrmap e13: y(active, since 41s) 2024-09-10T18:17:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: overall HEALTH_OK 2024-09-10T18:17:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:49.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:49.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:49.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:17:49.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:17:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:17:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:49.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: mon.a calling monitor election 2024-09-10T18:17:49.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:49.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:49.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:49.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: mon.b calling monitor election 2024-09-10T18:17:49.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:49.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:49.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:49.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:49.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-10T18:17:49.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: monmap e2: 2 mons at {a=v1:172.21.15.28:6789/0,b=v1:172.21.15.204:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:17:49.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: fsmap 2024-09-10T18:17:49.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:17:49.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: mgrmap e13: y(active, since 41s) 2024-09-10T18:17:49.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: overall HEALTH_OK 2024-09-10T18:17:49.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:49.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:49.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:49.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:17:49.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:17:50.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:49 smithi028 ceph-mon[29291]: Deploying daemon mon.c on smithi028 2024-09-10T18:17:50.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:49 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.204:0/4278963414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:17:50.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:49 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:49 smithi204 ceph-mon[32357]: Deploying daemon mon.c on smithi028 2024-09-10T18:17:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:49 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.204:0/4278963414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:17:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:49 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:50.654 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-10T18:17:50.655 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T18:17:50.903 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:17:51.014 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:50 smithi204 ceph-mon[32357]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:51.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:50 smithi028 ceph-mon[29291]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:51.255 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:50 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:50.816+0000 7f20da5f1640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-10T18:17:52.001 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:51 smithi028 podman[39201]: 2024-09-10 18:17:51.858565522 +0000 UTC m=+0.016608577 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T18:17:52.001 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:51 smithi028 podman[39201]: 2024-09-10 18:17:51.99922625 +0000 UTC m=+0.157269348 container create 9f8c661432c8db15ed754ed487235fa154e42bdbece867bc97c207bc9cd9625b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-10T18:17:52.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 podman[39201]: 2024-09-10 18:17:52.26023171 +0000 UTC m=+0.418274758 container init 9f8c661432c8db15ed754ed487235fa154e42bdbece867bc97c207bc9cd9625b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-10T18:17:52.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 podman[39201]: 2024-09-10 18:17:52.26462601 +0000 UTC m=+0.422669055 container start 9f8c661432c8db15ed754ed487235fa154e42bdbece867bc97c207bc9cd9625b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-10T18:17:52.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: set uid:gid to 167:167 (ceph:ceph) 2024-09-10T18:17:52.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable), process ceph-mon, pid 2 2024-09-10T18:17:52.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: pidfile_write: ignore empty --pid-file 2024-09-10T18:17:52.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: load: jerasure load: lrc 2024-09-10T18:17:52.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: RocksDB version: 6.15.5 2024-09-10T18:17:52.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-10T18:17:52.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Compile date Sep 9 2024 2024-09-10T18:17:52.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: DB SUMMARY 2024-09-10T18:17:52.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: DB Session ID: 6RMU5UCWCT9HSGIIUO74 2024-09-10T18:17:52.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: CURRENT file: CURRENT 2024-09-10T18:17:52.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: IDENTITY file: IDENTITY 2024-09-10T18:17:52.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-10T18:17:52.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-10T18:17:52.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 566 ; 2024-09-10T18:17:52.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.error_if_exists: 0 2024-09-10T18:17:52.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.create_if_missing: 0 2024-09-10T18:17:52.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.paranoid_checks: 1 2024-09-10T18:17:52.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-10T18:17:52.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.env: 0x56292119f440 2024-09-10T18:17:52.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.fs: Posix File System 2024-09-10T18:17:52.759 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.info_log: 0x562921a4ce80 2024-09-10T18:17:52.759 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_file_opening_threads: 16 2024-09-10T18:17:52.759 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.statistics: (nil) 2024-09-10T18:17:52.759 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.use_fsync: 0 2024-09-10T18:17:52.759 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_log_file_size: 0 2024-09-10T18:17:52.759 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-10T18:17:52.760 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-10T18:17:52.760 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.keep_log_file_num: 1000 2024-09-10T18:17:52.760 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.recycle_log_file_num: 0 2024-09-10T18:17:52.760 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.allow_fallocate: 1 2024-09-10T18:17:52.760 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.allow_mmap_reads: 0 2024-09-10T18:17:52.760 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.allow_mmap_writes: 0 2024-09-10T18:17:52.761 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.use_direct_reads: 0 2024-09-10T18:17:52.761 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-10T18:17:52.761 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.create_missing_column_families: 0 2024-09-10T18:17:52.761 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.db_log_dir: 2024-09-10T18:17:52.761 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-09-10T18:17:52.761 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-10T18:17:52.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-10T18:17:52.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-10T18:17:52.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-10T18:17:52.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-10T18:17:52.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-10T18:17:52.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.advise_random_on_open: 1 2024-09-10T18:17:52.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.db_write_buffer_size: 0 2024-09-10T18:17:52.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.write_buffer_manager: 0x562921b0a840 2024-09-10T18:17:52.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-10T18:17:52.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-10T18:17:52.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-10T18:17:52.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-10T18:17:52.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.rate_limiter: (nil) 2024-09-10T18:17:52.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-10T18:17:52.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.wal_recovery_mode: 2 2024-09-10T18:17:52.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.enable_thread_tracking: 0 2024-09-10T18:17:52.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.enable_pipelined_write: 0 2024-09-10T18:17:52.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.unordered_write: 0 2024-09-10T18:17:52.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-10T18:17:52.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-10T18:17:52.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-10T18:17:52.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-10T18:17:52.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.row_cache: None 2024-09-10T18:17:52.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.wal_filter: None 2024-09-10T18:17:52.765 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-10T18:17:52.765 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.allow_ingest_behind: 0 2024-09-10T18:17:52.765 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.preserve_deletes: 0 2024-09-10T18:17:52.765 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.two_write_queues: 0 2024-09-10T18:17:52.765 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.manual_wal_flush: 0 2024-09-10T18:17:52.765 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.atomic_flush: 0 2024-09-10T18:17:52.765 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-10T18:17:52.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-10T18:17:52.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-10T18:17:52.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.log_readahead_size: 0 2024-09-10T18:17:52.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-10T18:17:52.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.best_efforts_recovery: 0 2024-09-10T18:17:52.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-10T18:17:52.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-10T18:17:52.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.allow_data_in_errors: 0 2024-09-10T18:17:52.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.db_host_id: __hostname__ 2024-09-10T18:17:52.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_background_jobs: 2 2024-09-10T18:17:52.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_background_compactions: -1 2024-09-10T18:17:52.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_subcompactions: 1 2024-09-10T18:17:52.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-10T18:17:52.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-10T18:17:52.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-10T18:17:52.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_total_wal_size: 0 2024-09-10T18:17:52.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-10T18:17:52.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-10T18:17:52.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-10T18:17:52.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-10T18:17:52.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_open_files: -1 2024-09-10T18:17:52.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.bytes_per_sync: 0 2024-09-10T18:17:52.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-10T18:17:52.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-10T18:17:52.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compaction_readahead_size: 0 2024-09-10T18:17:52.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_background_flushes: -1 2024-09-10T18:17:52.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Compression algorithms supported: 2024-09-10T18:17:52.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: kZSTD supported: 0 2024-09-10T18:17:52.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: kXpressCompression supported: 0 2024-09-10T18:17:52.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: kBZip2Compression supported: 0 2024-09-10T18:17:52.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-10T18:17:52.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: kLZ4Compression supported: 1 2024-09-10T18:17:52.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: kZlibCompression supported: 1 2024-09-10T18:17:52.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: kLZ4HCCompression supported: 1 2024-09-10T18:17:52.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: kSnappyCompression supported: 1 2024-09-10T18:17:52.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-10T18:17:52.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-09-10T18:17:52.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-10T18:17:52.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-10T18:17:52.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.merge_operator: 2024-09-10T18:17:52.769 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compaction_filter: None 2024-09-10T18:17:52.769 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compaction_filter_factory: None 2024-09-10T18:17:52.769 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.sst_partitioner_factory: None 2024-09-10T18:17:52.769 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-10T18:17:52.769 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-10T18:17:52.769 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5629219f3a38) 2024-09-10T18:17:52.769 INFO:journalctl@ceph.mon.c.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-09-10T18:17:52.769 INFO:journalctl@ceph.mon.c.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-10T18:17:52.769 INFO:journalctl@ceph.mon.c.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-10T18:17:52.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-09-10T18:17:52.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: index_type: 0 2024-09-10T18:17:52.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: data_block_index_type: 0 2024-09-10T18:17:52.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: index_shortening: 1 2024-09-10T18:17:52.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-10T18:17:52.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: hash_index_allow_collision: 1 2024-09-10T18:17:52.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: checksum: 1 2024-09-10T18:17:52.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: no_block_cache: 0 2024-09-10T18:17:52.770 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_cache: 0x562921a291f0 2024-09-10T18:17:52.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-09-10T18:17:52.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_cache_options: 2024-09-10T18:17:52.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: capacity : 536870912 2024-09-10T18:17:52.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: num_shard_bits : 4 2024-09-10T18:17:52.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: strict_capacity_limit : 0 2024-09-10T18:17:52.771 INFO:journalctl@ceph.mon.c.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-09-10T18:17:52.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_cache_compressed: (nil) 2024-09-10T18:17:52.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: persistent_cache: (nil) 2024-09-10T18:17:52.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_size: 4096 2024-09-10T18:17:52.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_size_deviation: 10 2024-09-10T18:17:52.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_restart_interval: 16 2024-09-10T18:17:52.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: index_block_restart_interval: 1 2024-09-10T18:17:52.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: metadata_block_size: 4096 2024-09-10T18:17:52.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: partition_filters: 0 2024-09-10T18:17:52.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: use_delta_encoding: 1 2024-09-10T18:17:52.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-10T18:17:52.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: whole_key_filtering: 1 2024-09-10T18:17:52.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: verify_compression: 0 2024-09-10T18:17:52.772 INFO:journalctl@ceph.mon.c.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-09-10T18:17:52.773 INFO:journalctl@ceph.mon.c.smithi028.stdout: format_version: 4 2024-09-10T18:17:52.773 INFO:journalctl@ceph.mon.c.smithi028.stdout: enable_index_compression: 1 2024-09-10T18:17:52.773 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_align: 0 2024-09-10T18:17:52.773 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.write_buffer_size: 33554432 2024-09-10T18:17:52.773 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_write_buffer_number: 2 2024-09-10T18:17:52.773 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compression: NoCompression 2024-09-10T18:17:52.773 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.bottommost_compression: Disabled 2024-09-10T18:17:52.773 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.prefix_extractor: nullptr 2024-09-10T18:17:52.773 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-10T18:17:52.773 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.num_levels: 7 2024-09-10T18:17:52.773 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-10T18:17:52.773 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-10T18:17:52.774 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-10T18:17:52.774 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-10T18:17:52.774 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-10T18:17:52.774 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-10T18:17:52.774 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-10T18:17:52.774 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-10T18:17:52.774 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-10T18:17:52.774 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-10T18:17:52.774 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-10T18:17:52.774 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compression_opts.level: 32767 2024-09-10T18:17:52.774 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compression_opts.strategy: 0 2024-09-10T18:17:52.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-10T18:17:52.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-10T18:17:52.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-10T18:17:52.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compression_opts.enabled: false 2024-09-10T18:17:52.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-10T18:17:52.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-10T18:17:52.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-10T18:17:52.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.target_file_size_base: 67108864 2024-09-10T18:17:52.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-10T18:17:52.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-10T18:17:52.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-10T18:17:52.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-10T18:17:52.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-10T18:17:52.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-10T18:17:52.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-10T18:17:52.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-10T18:17:52.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-10T18:17:52.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-10T18:17:52.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-10T18:17:52.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-10T18:17:52.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-10T18:17:52.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.arena_block_size: 4194304 2024-09-10T18:17:52.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-10T18:17:52.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-10T18:17:52.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-10T18:17:52.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.disable_auto_compactions: 0 2024-09-10T18:17:52.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-10T18:17:52.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-10T18:17:52.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-10T18:17:52.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-10T18:17:52.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-10T18:17:52.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-10T18:17:52.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-10T18:17:52.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-10T18:17:52.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-10T18:17:52.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-10T18:17:52.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-10T18:17:52.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.inplace_update_support: 0 2024-09-10T18:17:52.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-10T18:17:52.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-10T18:17:52.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-10T18:17:52.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-10T18:17:52.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.bloom_locality: 0 2024-09-10T18:17:52.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.max_successive_merges: 0 2024-09-10T18:17:52.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-10T18:17:52.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.paranoid_file_checks: 0 2024-09-10T18:17:52.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.force_consistency_checks: 1 2024-09-10T18:17:52.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.report_bg_io_stats: 0 2024-09-10T18:17:52.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.ttl: 2592000 2024-09-10T18:17:52.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-10T18:17:52.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.enable_blob_files: false 2024-09-10T18:17:52.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.min_blob_size: 0 2024-09-10T18:17:52.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.blob_file_size: 268435456 2024-09-10T18:17:52.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-10T18:17:52.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-10T18:17:52.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-10T18:17:52.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-10T18:17:52.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-10T18:17:52.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-09-10T18:17:52.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725992272289114, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-10T18:17:52.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-09-10T18:17:52.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-10T18:17:52.780 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725992272289433, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1626, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1725992272, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "39b5828e-4b68-46a4-8644-c6787dadafbc", "db_session_id": "6RMU5UCWCT9HSGIIUO74"}} 2024-09-10T18:17:52.780 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-09-10T18:17:52.780 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725992272289769, "job": 1, "event": "recovery_finished"} 2024-09-10T18:17:52.780 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-10T18:17:52.780 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x562921a5c700 2024-09-10T18:17:52.780 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: DB pointer 0x562921a2c000 2024-09-10T18:17:52.780 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-10T18:17:52.780 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-10T18:17:52.780 INFO:journalctl@ceph.mon.c.smithi028.stdout: ** DB Stats ** 2024-09-10T18:17:52.780 INFO:journalctl@ceph.mon.c.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:17:52.780 INFO:journalctl@ceph.mon.c.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-10T18:17:52.780 INFO:journalctl@ceph.mon.c.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-10T18:17:52.781 INFO:journalctl@ceph.mon.c.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T18:17:52.781 INFO:journalctl@ceph.mon.c.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-10T18:17:52.781 INFO:journalctl@ceph.mon.c.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-10T18:17:52.781 INFO:journalctl@ceph.mon.c.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T18:17:52.781 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-09-10T18:17:52.781 INFO:journalctl@ceph.mon.c.smithi028.stdout: ** Compaction Stats [default] ** 2024-09-10T18:17:52.781 INFO:journalctl@ceph.mon.c.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:17:52.781 INFO:journalctl@ceph.mon.c.smithi028.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:17:52.781 INFO:journalctl@ceph.mon.c.smithi028.stdout: L0 1/0 1.59 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-09-10T18:17:52.781 INFO:journalctl@ceph.mon.c.smithi028.stdout: Sum 1/0 1.59 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-09-10T18:17:52.782 INFO:journalctl@ceph.mon.c.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-09-10T18:17:52.782 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-09-10T18:17:52.782 INFO:journalctl@ceph.mon.c.smithi028.stdout: ** Compaction Stats [default] ** 2024-09-10T18:17:52.782 INFO:journalctl@ceph.mon.c.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:17:52.782 INFO:journalctl@ceph.mon.c.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:17:52.782 INFO:journalctl@ceph.mon.c.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-09-10T18:17:52.782 INFO:journalctl@ceph.mon.c.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:17:52.782 INFO:journalctl@ceph.mon.c.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T18:17:52.782 INFO:journalctl@ceph.mon.c.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T18:17:52.783 INFO:journalctl@ceph.mon.c.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T18:17:52.783 INFO:journalctl@ceph.mon.c.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T18:17:52.783 INFO:journalctl@ceph.mon.c.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-10T18:17:52.783 INFO:journalctl@ceph.mon.c.smithi028.stdout: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:17:52.783 INFO:journalctl@ceph.mon.c.smithi028.stdout: Interval compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:17:52.783 INFO:journalctl@ceph.mon.c.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T18:17:52.783 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-09-10T18:17:52.783 INFO:journalctl@ceph.mon.c.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-10T18:17:52.783 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-09-10T18:17:52.783 INFO:journalctl@ceph.mon.c.smithi028.stdout: ** Compaction Stats [default] ** 2024-09-10T18:17:52.783 INFO:journalctl@ceph.mon.c.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:17:52.783 INFO:journalctl@ceph.mon.c.smithi028.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:17:52.783 INFO:journalctl@ceph.mon.c.smithi028.stdout: L0 1/0 1.59 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-09-10T18:17:52.783 INFO:journalctl@ceph.mon.c.smithi028.stdout: Sum 1/0 1.59 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-09-10T18:17:52.783 INFO:journalctl@ceph.mon.c.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-10T18:17:52.784 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-09-10T18:17:52.784 INFO:journalctl@ceph.mon.c.smithi028.stdout: ** Compaction Stats [default] ** 2024-09-10T18:17:52.784 INFO:journalctl@ceph.mon.c.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:17:52.784 INFO:journalctl@ceph.mon.c.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:17:52.784 INFO:journalctl@ceph.mon.c.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-09-10T18:17:52.784 INFO:journalctl@ceph.mon.c.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:17:52.784 INFO:journalctl@ceph.mon.c.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T18:17:52.784 INFO:journalctl@ceph.mon.c.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T18:17:52.784 INFO:journalctl@ceph.mon.c.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T18:17:52.784 INFO:journalctl@ceph.mon.c.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T18:17:52.784 INFO:journalctl@ceph.mon.c.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-10T18:17:52.784 INFO:journalctl@ceph.mon.c.smithi028.stdout: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:17:52.785 INFO:journalctl@ceph.mon.c.smithi028.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:17:52.785 INFO:journalctl@ceph.mon.c.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T18:17:52.785 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-09-10T18:17:52.785 INFO:journalctl@ceph.mon.c.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-10T18:17:52.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-10T18:17:52.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: using public_addrv v1:172.21.15.28:6790/0 2024-09-10T18:17:52.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: starting mon.c rank -1 at public addrs v1:172.21.15.28:6790/0 at bind addrs v1:172.21.15.28:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:17:52.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.c@-1(???) e0 preinit fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:17:52.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.c@-1(synchronizing).mds e1 new map 2024-09-10T18:17:52.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-10T18:17:52.785 INFO:journalctl@ceph.mon.c.smithi028.stdout: e1 2024-09-10T18:17:52.785 INFO:journalctl@ceph.mon.c.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-10T18:17:52.785 INFO:journalctl@ceph.mon.c.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-10T18:17:52.786 INFO:journalctl@ceph.mon.c.smithi028.stdout: legacy client fscid: -1 2024-09-10T18:17:52.786 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-09-10T18:17:52.786 INFO:journalctl@ceph.mon.c.smithi028.stdout: No filesystems configured 2024-09-10T18:17:52.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-10T18:17:52.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-10T18:17:52.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-10T18:17:52.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-10T18:17:52.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-10T18:17:52.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-10T18:17:52.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-10T18:17:52.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:17:52.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:17:52.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:17:52.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:52.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:52.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:52.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:17:52.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:17:52.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:17:52.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:52.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:17:52.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:17:52.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='client.14180 v1:172.21.15.204:0/4209441807' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi204:[v1:172.21.15.204:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:17:52.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: Saving service mon spec with placement smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi204:[v1:172.21.15.204:6789]=b;count:3 2024-09-10T18:17:52.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: Deploying daemon mon.b on smithi204 2024-09-10T18:17:52.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:52.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:52.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.204:0/410122418' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:17:52.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:52.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:17:52.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:52.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.a calling monitor election 2024-09-10T18:17:52.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:52.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:52.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:52.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.b calling monitor election 2024-09-10T18:17:52.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:52.789 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:52.789 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:52.789 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:52.789 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-10T18:17:52.789 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: monmap e2: 2 mons at {a=v1:172.21.15.28:6789/0,b=v1:172.21.15.204:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:17:52.789 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: fsmap 2024-09-10T18:17:52.789 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:17:52.789 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mgrmap e13: y(active, since 41s) 2024-09-10T18:17:52.789 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: overall HEALTH_OK 2024-09-10T18:17:52.789 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:52.789 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:52.789 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:52.789 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:17:52.790 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:17:52.790 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: Deploying daemon mon.c on smithi028 2024-09-10T18:17:52.790 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.204:0/4278963414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:17:52.790 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:52.790 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:52.790 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 ceph-mon[39220]: mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-09-10T18:17:52.790 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 bash[39201]: 9f8c661432c8db15ed754ed487235fa154e42bdbece867bc97c207bc9cd9625b 2024-09-10T18:17:52.790 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:52 smithi028 systemd[1]: Started Ceph mon.c for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:17:57.570 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-10T18:17:57.570 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":3,"fsid":"5eababb2-6fa0-11ef-bcea-c7b262605968","modified":"2024-09-10T18:17:52.308273Z","created":"2024-09-10T18:16:17.798654Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6790","nonce":0}]},"addr":"172.21.15.28:6790/0","public_addr":"172.21.15.28:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-10T18:17:57.570 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 3 2024-09-10T18:17:57.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:17:57.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: mon.a calling monitor election 2024-09-10T18:17:57.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: mon.b calling monitor election 2024-09-10T18:17:57.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:57.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: mon.c calling monitor election 2024-09-10T18:17:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-10T18:17:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: monmap e3: 3 mons at {a=v1:172.21.15.28:6789/0,b=v1:172.21.15.204:6789/0,c=v1:172.21.15.28:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:17:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: fsmap 2024-09-10T18:17:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:17:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: mgrmap e13: y(active, since 49s) 2024-09-10T18:17:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: overall HEALTH_OK 2024-09-10T18:17:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:57.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:57.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:57.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:57.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:17:57.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: mon.a calling monitor election 2024-09-10T18:17:57.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: mon.b calling monitor election 2024-09-10T18:17:57.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:57.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:57.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: mon.c calling monitor election 2024-09-10T18:17:57.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:57.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-10T18:17:57.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: monmap e3: 3 mons at {a=v1:172.21.15.28:6789/0,b=v1:172.21.15.204:6789/0,c=v1:172.21.15.28:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:17:57.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: fsmap 2024-09-10T18:17:57.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:17:57.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: mgrmap e13: y(active, since 49s) 2024-09-10T18:17:57.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: overall HEALTH_OK 2024-09-10T18:17:57.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:57.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:57.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:57.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:57 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:17:57.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: mon.a calling monitor election 2024-09-10T18:17:57.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: mon.b calling monitor election 2024-09-10T18:17:57.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:17:57.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:57.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: mon.c calling monitor election 2024-09-10T18:17:57.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:57.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:57.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-10T18:17:57.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: monmap e3: 3 mons at {a=v1:172.21.15.28:6789/0,b=v1:172.21.15.204:6789/0,c=v1:172.21.15.28:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:17:57.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: fsmap 2024-09-10T18:17:57.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:17:57.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: mgrmap e13: y(active, since 49s) 2024-09-10T18:17:57.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: overall HEALTH_OK 2024-09-10T18:17:57.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:57.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:57.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:57.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:57 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:58.142 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-10T18:17:58.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph config generate-minimal-conf 2024-09-10T18:17:58.420 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:17:58.421 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:58 smithi204 ceph-mon[32357]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:58.421 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:58.421 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:58 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.204:0/996185129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:17:58.421 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:17:58.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:17:58.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:58 smithi028 ceph-mon[39220]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:58.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:58.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:58 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.204:0/996185129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:17:58.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:58.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:17:58.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:58 smithi028 ceph-mon[29291]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:17:58.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:58.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:58 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.204:0/996185129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:17:58.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:17:59.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:17:59.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:17:59.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[29291]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-10T18:17:59.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[29291]: Updating smithi204:/etc/ceph/ceph.conf 2024-09-10T18:17:59.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:17:59.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:17:59.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:17:59.533 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.533 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:17:59.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:17:59.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[39220]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-10T18:17:59.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[39220]: Updating smithi204:/etc/ceph/ceph.conf 2024-09-10T18:17:59.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.535 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:17:59.535 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:17:59.535 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:17:59.535 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:17:59 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:17:59.307+0000 7f20da5f1640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-10T18:17:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:59 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:59 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:59 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:59 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:59 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:17:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:59 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:17:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:59 smithi204 ceph-mon[32357]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-10T18:17:59.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:59 smithi204 ceph-mon[32357]: Updating smithi204:/etc/ceph/ceph.conf 2024-09-10T18:17:59.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:59 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:59 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:59 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:59 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:59 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:17:59.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:59 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:17:59.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:59 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:17:59.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:17:59 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:00.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:00 smithi028 ceph-mon[29291]: Updating smithi204:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:18:00.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:00 smithi028 ceph-mon[29291]: Updating smithi028:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:18:00.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:00 smithi028 ceph-mon[29291]: Reconfiguring mon.a (unknown last config time)... 2024-09-10T18:18:00.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:00 smithi028 ceph-mon[29291]: Reconfiguring daemon mon.a on smithi028 2024-09-10T18:18:00.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:00 smithi028 ceph-mon[29291]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:00.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:00 smithi028 ceph-mon[39220]: Updating smithi204:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:18:00.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:00 smithi028 ceph-mon[39220]: Updating smithi028:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:18:00.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:00 smithi028 ceph-mon[39220]: Reconfiguring mon.a (unknown last config time)... 2024-09-10T18:18:00.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:00 smithi028 ceph-mon[39220]: Reconfiguring daemon mon.a on smithi028 2024-09-10T18:18:00.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:00 smithi028 ceph-mon[39220]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:00 smithi204 ceph-mon[32357]: Updating smithi204:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:18:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:00 smithi204 ceph-mon[32357]: Updating smithi028:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:18:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:00 smithi204 ceph-mon[32357]: Reconfiguring mon.a (unknown last config time)... 2024-09-10T18:18:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:00 smithi204 ceph-mon[32357]: Reconfiguring daemon mon.a on smithi028 2024-09-10T18:18:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:00 smithi204 ceph-mon[32357]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:00.908 INFO:teuthology.orchestra.run.smithi028.stdout:# minimal ceph.conf for 5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:18:00.908 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-09-10T18:18:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:18:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: mon_host = 172.21.15.28:6789/0 172.21.15.204:6789/0 v1:172.21.15.28:6790/0 2024-09-10T18:18:01.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:01 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3329891193' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:01.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:01 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:01.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:01 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:01.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:01 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:18:01.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:01 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:18:01.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:01 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:01.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:01 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3329891193' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:01.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:01 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:01.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:01 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:01.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:01 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:18:01.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:01 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:18:01.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:01 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:01.803 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-10T18:18:01.803 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T18:18:01.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.conf 2024-09-10T18:18:01.831 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T18:18:01.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:18:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:01 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3329891193' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:01 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:01 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:01 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:18:01.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:01 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:18:01.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:01 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:01.900 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-10T18:18:01.900 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.conf 2024-09-10T18:18:01.929 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-10T18:18:01.929 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:18:01.997 INFO:tasks.cephadm:Adding mgr.y on smithi028 2024-09-10T18:18:01.997 INFO:tasks.cephadm:Adding mgr.x on smithi204 2024-09-10T18:18:01.997 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch apply mgr '2;smithi028=y;smithi204=x' 2024-09-10T18:18:02.273 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:18:02.645 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:02 smithi204 ceph-mon[32357]: Reconfiguring mon.c (monmap changed)... 2024-09-10T18:18:02.646 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:02 smithi204 ceph-mon[32357]: Reconfiguring daemon mon.c on smithi028 2024-09-10T18:18:02.646 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:02 smithi204 ceph-mon[32357]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:02.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:02 smithi028 ceph-mon[39220]: Reconfiguring mon.c (monmap changed)... 2024-09-10T18:18:02.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:02 smithi028 ceph-mon[39220]: Reconfiguring daemon mon.c on smithi028 2024-09-10T18:18:02.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:02 smithi028 ceph-mon[39220]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:02.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:02 smithi028 ceph-mon[29291]: Reconfiguring mon.c (monmap changed)... 2024-09-10T18:18:02.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:02 smithi028 ceph-mon[29291]: Reconfiguring daemon mon.c on smithi028 2024-09-10T18:18:02.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:02 smithi028 ceph-mon[29291]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:04 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:04 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:04 smithi204 ceph-mon[32357]: Reconfiguring mon.b (monmap changed)... 2024-09-10T18:18:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:04 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:18:04.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:04 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:18:04.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:04 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:04.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:04 smithi204 ceph-mon[32357]: Reconfiguring daemon mon.b on smithi204 2024-09-10T18:18:04.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:04 smithi204 ceph-mon[32357]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:04.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:04 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:04.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:04 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:04.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:04 smithi028 ceph-mon[29291]: Reconfiguring mon.b (monmap changed)... 2024-09-10T18:18:04.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:04 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:18:04.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:04 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:18:04.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:04 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:04.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:04 smithi028 ceph-mon[29291]: Reconfiguring daemon mon.b on smithi204 2024-09-10T18:18:04.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:04 smithi028 ceph-mon[29291]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:04.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:04 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:04.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:04 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:04.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:04 smithi028 ceph-mon[39220]: Reconfiguring mon.b (monmap changed)... 2024-09-10T18:18:04.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:04 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:18:04.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:04 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:18:04.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:04 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:04.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:04 smithi028 ceph-mon[39220]: Reconfiguring daemon mon.b on smithi204 2024-09-10T18:18:04.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:04 smithi028 ceph-mon[39220]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:04.655 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mgr update... 2024-09-10T18:18:05.519 DEBUG:teuthology.orchestra.run.smithi204:mgr.x> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mgr.x.service 2024-09-10T18:18:05.521 INFO:tasks.cephadm:Deploying OSDs... 2024-09-10T18:18:05.522 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T18:18:05.522 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T18:18:05.542 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T18:18:05.542 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-09-10T18:18:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T18:18:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:18:05.603 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 954 Links: 1 2024-09-10T18:18:05.603 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:18:05.603 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T18:18:05.603 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-10 18:17:51.680241222 +0000 2024-09-10T18:18:05.603 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-10 18:11:05.209288317 +0000 2024-09-10T18:18:05.603 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-10 18:11:05.209288317 +0000 2024-09-10T18:18:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-10 18:11:05.209288317 +0000 2024-09-10T18:18:05.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T18:18:05.673 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-10T18:18:05.674 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-10T18:18:05.674 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000130717 s, 3.9 MB/s 2024-09-10T18:18:05.675 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T18:18:05.735 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-09-10T18:18:05.776 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:05 smithi204 ceph-mon[32357]: from='client.24106 v1:172.21.15.204:0/2992021804' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi028=y;smithi204=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:18:05.777 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:05 smithi204 ceph-mon[32357]: Saving service mgr spec with placement smithi028=y;smithi204=x;count:2 2024-09-10T18:18:05.777 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:05 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:05.777 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:05 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:05.777 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:05 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:05.777 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:05 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:05.777 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:05 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:05.778 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:05 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:05.778 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:05 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:05.778 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:05 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:18:05.778 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:05 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-10T18:18:05.778 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:05 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:18:05.778 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:05 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:05.779 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:05 smithi204 ceph-mon[32357]: Deploying daemon mgr.x on smithi204 2024-09-10T18:18:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T18:18:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:18:05.797 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 939 Links: 1 2024-09-10T18:18:05.797 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:18:05.797 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T18:18:05.797 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-10 18:17:51.680241222 +0000 2024-09-10T18:18:05.797 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-10 18:11:05.207288375 +0000 2024-09-10T18:18:05.797 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-10 18:11:05.207288375 +0000 2024-09-10T18:18:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-10 18:11:05.207288375 +0000 2024-09-10T18:18:05.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T18:18:05.863 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-10T18:18:05.863 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-10T18:18:05.864 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.00012019 s, 4.3 MB/s 2024-09-10T18:18:05.865 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T18:18:05.925 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-09-10T18:18:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T18:18:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:18:05.982 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 948 Links: 1 2024-09-10T18:18:05.982 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:18:05.982 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T18:18:05.982 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-10 18:17:51.681241193 +0000 2024-09-10T18:18:05.982 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-10 18:11:05.208288346 +0000 2024-09-10T18:18:05.983 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-10 18:11:05.208288346 +0000 2024-09-10T18:18:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-10 18:11:05.208288346 +0000 2024-09-10T18:18:05.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T18:18:06.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[29291]: from='client.24106 v1:172.21.15.204:0/2992021804' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi028=y;smithi204=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:18:06.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[29291]: Saving service mgr spec with placement smithi028=y;smithi204=x;count:2 2024-09-10T18:18:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:06.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:06.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:18:06.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-10T18:18:06.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:18:06.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:06.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[29291]: Deploying daemon mgr.x on smithi204 2024-09-10T18:18:06.007 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[39220]: from='client.24106 v1:172.21.15.204:0/2992021804' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi028=y;smithi204=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:18:06.007 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[39220]: Saving service mgr spec with placement smithi028=y;smithi204=x;count:2 2024-09-10T18:18:06.007 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:06.008 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:06.008 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:06.008 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:06.008 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:06.008 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:06.008 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:06.008 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:18:06.009 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-10T18:18:06.009 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:18:06.009 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:06.009 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:05 smithi028 ceph-mon[39220]: Deploying daemon mgr.x on smithi204 2024-09-10T18:18:06.035 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-10T18:18:06.036 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-10T18:18:06.036 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000121438 s, 4.2 MB/s 2024-09-10T18:18:06.037 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T18:18:06.096 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-09-10T18:18:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T18:18:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:18:06.157 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 955 Links: 1 2024-09-10T18:18:06.157 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:18:06.157 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T18:18:06.157 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-10 18:17:51.681241193 +0000 2024-09-10T18:18:06.157 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-10 18:11:05.209288317 +0000 2024-09-10T18:18:06.157 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-10 18:11:05.209288317 +0000 2024-09-10T18:18:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-10 18:11:05.209288317 +0000 2024-09-10T18:18:06.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T18:18:06.222 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-10T18:18:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-10T18:18:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000127085 s, 4.0 MB/s 2024-09-10T18:18:06.224 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T18:18:06.282 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-10T18:18:06.282 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T18:18:06.299 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T18:18:06.299 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-09-10T18:18:06.355 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T18:18:06.355 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:18:06.355 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 967 Links: 1 2024-09-10T18:18:06.356 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:18:06.356 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T18:18:06.356 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-10 18:17:42.913083675 +0000 2024-09-10T18:18:06.356 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-10 18:11:05.114477008 +0000 2024-09-10T18:18:06.356 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-10 18:11:05.114477008 +0000 2024-09-10T18:18:06.356 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-10 18:11:05.114477008 +0000 2024-09-10T18:18:06.356 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T18:18:06.419 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-10T18:18:06.420 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-10T18:18:06.420 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000190119 s, 2.7 MB/s 2024-09-10T18:18:06.421 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T18:18:06.477 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-09-10T18:18:06.533 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T18:18:06.533 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:18:06.533 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 983 Links: 1 2024-09-10T18:18:06.533 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:18:06.533 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T18:18:06.533 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-10 18:17:42.914083661 +0000 2024-09-10T18:18:06.533 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-10 18:11:05.116476981 +0000 2024-09-10T18:18:06.533 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-10 18:11:05.116476981 +0000 2024-09-10T18:18:06.533 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-10 18:11:05.116476981 +0000 2024-09-10T18:18:06.534 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T18:18:06.712 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:06 smithi204 ceph-mon[32357]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:06.732 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-10T18:18:06.732 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-10T18:18:06.732 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000204986 s, 2.5 MB/s 2024-09-10T18:18:06.733 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T18:18:06.749 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-09-10T18:18:06.809 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T18:18:06.809 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:18:06.809 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 974 Links: 1 2024-09-10T18:18:06.810 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:18:06.810 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T18:18:06.810 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-10 18:18:06.681749168 +0000 2024-09-10T18:18:06.810 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-10 18:11:05.115476995 +0000 2024-09-10T18:18:06.810 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-10 18:11:05.115476995 +0000 2024-09-10T18:18:06.810 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-10 18:11:05.115476995 +0000 2024-09-10T18:18:06.810 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T18:18:06.914 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-10T18:18:06.914 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-10T18:18:06.914 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000235826 s, 2.2 MB/s 2024-09-10T18:18:06.915 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T18:18:06.931 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-09-10T18:18:06.988 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T18:18:06.988 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:18:06.988 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 975 Links: 1 2024-09-10T18:18:06.988 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:18:06.988 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T18:18:06.988 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-10 18:18:06.869746522 +0000 2024-09-10T18:18:06.988 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-10 18:11:05.115476995 +0000 2024-09-10T18:18:06.988 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-10 18:11:05.115476995 +0000 2024-09-10T18:18:06.988 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-10 18:11:05.115476995 +0000 2024-09-10T18:18:06.989 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T18:18:07.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:06 smithi028 ceph-mon[29291]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:07.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:06 smithi028 ceph-mon[39220]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:07.051 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:06 smithi204 systemd[1]: Starting Ceph mgr.x for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:18:07.055 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-10T18:18:07.055 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-10T18:18:07.055 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.00016926 s, 3.0 MB/s 2024-09-10T18:18:07.056 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T18:18:07.112 INFO:tasks.cephadm:Deploying osd.0 on smithi028 with /dev/vg_nvme/lv_4... 2024-09-10T18:18:07.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-10T18:18:07.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:07 smithi204 podman[34141]: 2024-09-10 18:18:07.040761977 +0000 UTC m=+0.016022095 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T18:18:07.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:07 smithi204 podman[34141]: 2024-09-10 18:18:07.174796562 +0000 UTC m=+0.150056664 container create 0d6adb1156510a78cb8d96dbf8d82449cc570459dd5b977915f9f8206c345904 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T18:18:07.373 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:18:07.761 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:07 smithi204 podman[34141]: 2024-09-10 18:18:07.385474639 +0000 UTC m=+0.360734742 container init 0d6adb1156510a78cb8d96dbf8d82449cc570459dd5b977915f9f8206c345904 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-10T18:18:07.761 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:07 smithi204 podman[34141]: 2024-09-10 18:18:07.38895466 +0000 UTC m=+0.364214763 container start 0d6adb1156510a78cb8d96dbf8d82449cc570459dd5b977915f9f8206c345904 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-10T18:18:07.761 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:07 smithi204 bash[34141]: 0d6adb1156510a78cb8d96dbf8d82449cc570459dd5b977915f9f8206c345904 2024-09-10T18:18:07.761 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:07 smithi204 systemd[1]: Started Ceph mgr.x for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:18:07.761 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:07.501+0000 7fdfec1b9ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T18:18:07.761 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:07.560+0000 7fdfec1b9ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T18:18:07.762 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:07.627+0000 7fdfec1b9ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T18:18:07.762 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:07.758+0000 7fdfec1b9ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:18:08.104 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:07.911+0000 7fdfec1b9ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T18:18:08.465 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:08.188+0000 7fdfec1b9ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T18:18:08.466 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:08.251+0000 7fdfec1b9ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T18:18:08.466 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:08.420+0000 7fdfec1b9ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T18:18:08.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:08 smithi028 ceph-mon[29291]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:08.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:08 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:08.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:08 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:08.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:08 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:08.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:08 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:08.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:08 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:08.795 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:08 smithi028 ceph-mon[39220]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:08.795 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:08 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:08.795 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:08 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:08.795 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:08 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:08.795 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:08 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:08.796 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:08 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:08.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:08 smithi204 ceph-mon[32357]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:08.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:08 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:08.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:08 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:08.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:08 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:08.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:08 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:08.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:08 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-10T18:18:09.268 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-10T18:18:09.279 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-09-10T18:18:09.279 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-09-10T18:18:09.279 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00857684 s, 1.2 GB/s 2024-09-10T18:18:09.279 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-09-10T18:18:09.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:09.282+0000 7fdfec1b9ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T18:18:09.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:09.347+0000 7fdfec1b9ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T18:18:09.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:09.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:09.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:09.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:09.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:09.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[29291]: Reconfiguring mgr.y (unknown last config time)... 2024-09-10T18:18:09.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:18:09.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:18:09.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:09.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[29291]: Reconfiguring daemon mgr.y on smithi028 2024-09-10T18:18:09.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:09.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:09.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:09.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:09.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:09.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[39220]: Reconfiguring mgr.y (unknown last config time)... 2024-09-10T18:18:09.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:18:09.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:18:09.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:09.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:09 smithi028 ceph-mon[39220]: Reconfiguring daemon mgr.y on smithi028 2024-09-10T18:18:09.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_4 2024-09-10T18:18:10.062 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:09 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:10.063 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:09 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:10.063 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:09 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:10.063 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:09 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:10.063 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:09 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:10.063 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:09 smithi204 ceph-mon[32357]: Reconfiguring mgr.y (unknown last config time)... 2024-09-10T18:18:10.064 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:09 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:18:10.064 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:09 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:18:10.064 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:09 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:10.064 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:09 smithi204 ceph-mon[32357]: Reconfiguring daemon mgr.y on smithi028 2024-09-10T18:18:10.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:10 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:10.060+0000 7fdfec1b9ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T18:18:10.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:10 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:10.172+0000 7fdfec1b9ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T18:18:10.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:10 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:10.292+0000 7fdfec1b9ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:18:10.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:10 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:10.352+0000 7fdfec1b9ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T18:18:10.605 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:10 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:10.477+0000 7fdfec1b9ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T18:18:10.605 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:10 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:10.544+0000 7fdfec1b9ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T18:18:10.605 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:10 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:10.602+0000 7fdfec1b9ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T18:18:10.812 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:10 smithi028 ceph-mon[39220]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:10.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:10 smithi028 ceph-mon[29291]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:10.868 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:10 smithi204 ceph-mon[32357]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:10.868 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:10 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:10.737+0000 7fdfec1b9ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T18:18:10.869 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:10 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:10.800+0000 7fdfec1b9ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T18:18:10.869 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:10 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:10.864+0000 7fdfec1b9ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T18:18:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:18:11.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:10 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:10.946+0000 7fdfec1b9ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T18:18:11.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:11 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:11.004+0000 7fdfec1b9ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T18:18:11.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:11 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:11.061+0000 7fdfec1b9ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T18:18:12.039 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:12.039 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:12.039 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:12.039 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:12.039 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:12.039 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:12.039 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[39220]: Standby manager daemon x started 2024-09-10T18:18:12.039 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[39220]: from='mgr.? v1:172.21.15.204:0/88519371' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T18:18:12.039 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[39220]: from='mgr.? v1:172.21.15.204:0/88519371' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T18:18:12.039 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[39220]: from='mgr.? v1:172.21.15.204:0/88519371' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T18:18:12.039 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[39220]: from='mgr.? v1:172.21.15.204:0/88519371' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T18:18:12.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:12.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:12.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:12.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:12.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:12.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:12.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[29291]: Standby manager daemon x started 2024-09-10T18:18:12.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[29291]: from='mgr.? v1:172.21.15.204:0/88519371' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T18:18:12.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[29291]: from='mgr.? v1:172.21.15.204:0/88519371' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T18:18:12.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[29291]: from='mgr.? v1:172.21.15.204:0/88519371' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T18:18:12.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:11 smithi028 ceph-mon[29291]: from='mgr.? v1:172.21.15.204:0/88519371' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T18:18:12.104 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:18:11 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:18:11.802+0000 7fdfec1b9ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T18:18:12.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:11 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:12.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:11 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:12.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:11 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:12.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:11 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:12.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:11 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:12.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:11 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:12.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:11 smithi204 ceph-mon[32357]: Standby manager daemon x started 2024-09-10T18:18:12.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:11 smithi204 ceph-mon[32357]: from='mgr.? v1:172.21.15.204:0/88519371' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T18:18:12.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:11 smithi204 ceph-mon[32357]: from='mgr.? v1:172.21.15.204:0/88519371' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T18:18:12.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:11 smithi204 ceph-mon[32357]: from='mgr.? v1:172.21.15.204:0/88519371' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T18:18:12.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:11 smithi204 ceph-mon[32357]: from='mgr.? v1:172.21.15.204:0/88519371' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T18:18:13.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:12 smithi028 ceph-mon[29291]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:13.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:12 smithi028 ceph-mon[29291]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-10T18:18:13.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:12 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T18:18:13.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:12 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:13.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:12 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:18:13.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:12 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:18:13.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:12 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:13.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:12 smithi028 ceph-mon[39220]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:13.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:12 smithi028 ceph-mon[39220]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-10T18:18:13.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:12 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T18:18:13.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:12 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:13.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:12 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:18:13.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:12 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:18:13.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:12 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:12 smithi204 ceph-mon[32357]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:12 smithi204 ceph-mon[32357]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-10T18:18:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:12 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T18:18:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:12 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:12 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:18:13.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:12 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:18:13.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:12 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:13 smithi028 ceph-mon[29291]: from='client.34115 v1:172.21.15.28:0/2007385672' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:18:14.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:13 smithi028 ceph-mon[39220]: from='client.34115 v1:172.21.15.28:0/2007385672' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:18:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:13 smithi204 ceph-mon[32357]: from='client.34115 v1:172.21.15.28:0/2007385672' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:18:15.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:14 smithi028 ceph-mon[29291]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:15.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:14 smithi028 ceph-mon[39220]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:14 smithi204 ceph-mon[32357]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:15 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1832851505' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1c834d9-252a-4077-b25e-169181d70251"}]: dispatch 2024-09-10T18:18:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:15 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1832851505' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1c834d9-252a-4077-b25e-169181d70251"}]': finished 2024-09-10T18:18:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:15 smithi028 ceph-mon[39220]: osdmap e5: 1 total, 0 up, 1 in 2024-09-10T18:18:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:15 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:15 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4139810362' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:18:16.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:15 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1832851505' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1c834d9-252a-4077-b25e-169181d70251"}]: dispatch 2024-09-10T18:18:16.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:15 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1832851505' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1c834d9-252a-4077-b25e-169181d70251"}]': finished 2024-09-10T18:18:16.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:15 smithi028 ceph-mon[29291]: osdmap e5: 1 total, 0 up, 1 in 2024-09-10T18:18:16.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:15 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:16.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:15 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4139810362' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:18:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:15 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1832851505' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1c834d9-252a-4077-b25e-169181d70251"}]: dispatch 2024-09-10T18:18:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:15 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1832851505' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1c834d9-252a-4077-b25e-169181d70251"}]': finished 2024-09-10T18:18:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:15 smithi204 ceph-mon[32357]: osdmap e5: 1 total, 0 up, 1 in 2024-09-10T18:18:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:15 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:15 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4139810362' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:18:17.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:16 smithi028 ceph-mon[29291]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:17.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:16 smithi028 ceph-mon[39220]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:16 smithi204 ceph-mon[32357]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:19.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:18 smithi028 ceph-mon[29291]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:19.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:18 smithi028 ceph-mon[39220]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:18 smithi204 ceph-mon[32357]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:21.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:20 smithi028 ceph-mon[29291]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:21.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:20 smithi028 ceph-mon[39220]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:21.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:20 smithi204 ceph-mon[32357]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:22.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:21 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-10T18:18:22.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:21 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:22.168 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:21 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-10T18:18:22.169 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:21 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:21 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-10T18:18:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:21 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:23.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:22 smithi028 ceph-mon[29291]: Deploying daemon osd.0 on smithi028 2024-09-10T18:18:23.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:22 smithi028 ceph-mon[29291]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:23.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:22 smithi028 ceph-mon[39220]: Deploying daemon osd.0 on smithi028 2024-09-10T18:18:23.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:22 smithi028 ceph-mon[39220]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:22 smithi204 ceph-mon[32357]: Deploying daemon osd.0 on smithi028 2024-09-10T18:18:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:22 smithi204 ceph-mon[32357]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:24.988 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:24 smithi028 ceph-mon[29291]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:24.989 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:24 smithi028 ceph-mon[39220]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:24 smithi204 ceph-mon[32357]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:26.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:26 smithi028 ceph-mon[29291]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:26.294 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:26 smithi028 ceph-mon[39220]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:26 smithi204 ceph-mon[32357]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:27.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:27.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:27.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:28 smithi204 ceph-mon[32357]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:28.734 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:28 smithi028 ceph-mon[29291]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:28.734 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:28 smithi028 ceph-mon[39220]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:29.914 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 0 on host 'smithi028' 2024-09-10T18:18:30.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:30.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:30.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:30.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:30.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:30.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[39220]: from='osd.0 v1:172.21.15.28:6801/3188372351' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T18:18:30.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[39220]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T18:18:30.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[39220]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:30.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:30.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:30.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:30.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:30.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:30.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:30.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:30.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:30.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[29291]: from='osd.0 v1:172.21.15.28:6801/3188372351' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T18:18:30.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[29291]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T18:18:30.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[29291]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:30.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:30.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:30.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:30.566 DEBUG:teuthology.orchestra.run.smithi028:osd.0> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.0.service 2024-09-10T18:18:30.568 INFO:tasks.cephadm:Deploying osd.1 on smithi028 with /dev/vg_nvme/lv_3... 2024-09-10T18:18:30.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-10T18:18:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:30.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:30.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:30.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:30.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:30 smithi204 ceph-mon[32357]: from='osd.0 v1:172.21.15.28:6801/3188372351' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T18:18:30.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:30 smithi204 ceph-mon[32357]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T18:18:30.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:30 smithi204 ceph-mon[32357]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:30.606 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:30.606 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:30.606 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:31.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:31 smithi028 ceph-mon[29291]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-10T18:18:31.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:31 smithi028 ceph-mon[29291]: osdmap e6: 1 total, 0 up, 1 in 2024-09-10T18:18:31.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:31 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:31.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:31 smithi028 ceph-mon[29291]: from='osd.0 v1:172.21.15.28:6801/3188372351' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:18:31.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:31 smithi028 ceph-mon[29291]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:18:31.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:31 smithi028 ceph-mon[39220]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-10T18:18:31.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:31 smithi028 ceph-mon[39220]: osdmap e6: 1 total, 0 up, 1 in 2024-09-10T18:18:31.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:31 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:31.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:31 smithi028 ceph-mon[39220]: from='osd.0 v1:172.21.15.28:6801/3188372351' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:18:31.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:31 smithi028 ceph-mon[39220]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:18:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:31 smithi204 ceph-mon[32357]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-10T18:18:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:31 smithi204 ceph-mon[32357]: osdmap e6: 1 total, 0 up, 1 in 2024-09-10T18:18:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:31 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:31 smithi204 ceph-mon[32357]: from='osd.0 v1:172.21.15.28:6801/3188372351' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:18:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:31 smithi204 ceph-mon[32357]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:18:31.355 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:18:31.504 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:18:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-0[43198]: 2024-09-10T18:18:31.136+0000 7fa7bacca640 -1 osd.0 0 waiting for initial osdmap 2024-09-10T18:18:31.504 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:18:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-0[43198]: 2024-09-10T18:18:31.138+0000 7fa7b5300640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T18:18:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:32 smithi028 ceph-mon[29291]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-10T18:18:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:32 smithi028 ceph-mon[29291]: osdmap e7: 1 total, 0 up, 1 in 2024-09-10T18:18:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:32 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:32 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:32 smithi028 ceph-mon[29291]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:32.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:32 smithi028 ceph-mon[39220]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-10T18:18:32.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:32 smithi028 ceph-mon[39220]: osdmap e7: 1 total, 0 up, 1 in 2024-09-10T18:18:32.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:32 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:32.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:32 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:32.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:32 smithi028 ceph-mon[39220]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:32 smithi204 ceph-mon[32357]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-10T18:18:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:32 smithi204 ceph-mon[32357]: osdmap e7: 1 total, 0 up, 1 in 2024-09-10T18:18:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:32 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:32 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:32 smithi204 ceph-mon[32357]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:18:33.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:33 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:18:33.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:33 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:18:33.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:33 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:33.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:33 smithi028 ceph-mon[29291]: osd.0 v1:172.21.15.28:6801/3188372351 boot 2024-09-10T18:18:33.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:33 smithi028 ceph-mon[29291]: osdmap e8: 1 total, 1 up, 1 in 2024-09-10T18:18:33.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:33 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:33.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:33 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:18:33.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:33 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:18:33.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:33 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:33.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:33 smithi028 ceph-mon[39220]: osd.0 v1:172.21.15.28:6801/3188372351 boot 2024-09-10T18:18:33.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:33 smithi028 ceph-mon[39220]: osdmap e8: 1 total, 1 up, 1 in 2024-09-10T18:18:33.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:33 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:33 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:18:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:33 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:18:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:33 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:33 smithi204 ceph-mon[32357]: osd.0 v1:172.21.15.28:6801/3188372351 boot 2024-09-10T18:18:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:33 smithi204 ceph-mon[32357]: osdmap e8: 1 total, 1 up, 1 in 2024-09-10T18:18:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:33 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:18:34.229 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-10T18:18:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:34 smithi028 ceph-mon[29291]: osdmap e9: 1 total, 1 up, 1 in 2024-09-10T18:18:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:34 smithi028 ceph-mon[29291]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:34 smithi028 ceph-mon[39220]: osdmap e9: 1 total, 1 up, 1 in 2024-09-10T18:18:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:34 smithi028 ceph-mon[39220]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-10T18:18:34.320 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-09-10T18:18:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-09-10T18:18:34.320 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00892545 s, 1.2 GB/s 2024-09-10T18:18:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-09-10T18:18:34.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:34 smithi204 ceph-mon[32357]: osdmap e9: 1 total, 1 up, 1 in 2024-09-10T18:18:34.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:34 smithi204 ceph-mon[32357]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:35.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_3 2024-09-10T18:18:35.405 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:18:35.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[29291]: Detected new or changed devices on smithi028 2024-09-10T18:18:35.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:35.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:35.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:18:35.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[29291]: Adjusting osd_memory_target on smithi028 to 16068M 2024-09-10T18:18:35.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:35.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:35.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:35.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:35.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[39220]: Detected new or changed devices on smithi028 2024-09-10T18:18:35.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:35.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:35.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:18:35.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[39220]: Adjusting osd_memory_target on smithi028 to 16068M 2024-09-10T18:18:35.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:35.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:35.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:35.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:35.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:35 smithi204 ceph-mon[32357]: Detected new or changed devices on smithi028 2024-09-10T18:18:35.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:35.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:35.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:18:35.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:35 smithi204 ceph-mon[32357]: Adjusting osd_memory_target on smithi028 to 16068M 2024-09-10T18:18:35.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:35.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:35.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:35.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:36.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:36 smithi028 ceph-mon[29291]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:36.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:36 smithi028 ceph-mon[39220]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:36.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:36 smithi204 ceph-mon[32357]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:37.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:37 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:18:37.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:37 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:18:37.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:37 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:37.889 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:37 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:18:37.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:37 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:18:37.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:37 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:37.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:37 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:18:37.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:37 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:18:37.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:37 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:38.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:38 smithi028 ceph-mon[39220]: from='client.14241 v1:172.21.15.28:0/3773371684' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:18:38.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:38 smithi028 ceph-mon[39220]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:38 smithi028 ceph-mon[29291]: from='client.14241 v1:172.21.15.28:0/3773371684' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:18:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:38 smithi028 ceph-mon[29291]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:38.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:38 smithi204 ceph-mon[32357]: from='client.14241 v1:172.21.15.28:0/3773371684' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:18:38.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:38 smithi204 ceph-mon[32357]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:40.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:40 smithi204 ceph-mon[32357]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:40.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:40 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2466178680' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81f1855d-b946-4ca3-b246-38977b9e20fd"}]: dispatch 2024-09-10T18:18:40.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:40 smithi204 ceph-mon[32357]: from='client.34148 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81f1855d-b946-4ca3-b246-38977b9e20fd"}]: dispatch 2024-09-10T18:18:40.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:40 smithi204 ceph-mon[32357]: from='client.34148 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81f1855d-b946-4ca3-b246-38977b9e20fd"}]': finished 2024-09-10T18:18:40.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:40 smithi204 ceph-mon[32357]: osdmap e10: 2 total, 1 up, 2 in 2024-09-10T18:18:40.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:40 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:40.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:40 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/357104567' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:18:41.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:40 smithi028 ceph-mon[39220]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:41.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:40 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2466178680' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81f1855d-b946-4ca3-b246-38977b9e20fd"}]: dispatch 2024-09-10T18:18:41.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:40 smithi028 ceph-mon[39220]: from='client.34148 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81f1855d-b946-4ca3-b246-38977b9e20fd"}]: dispatch 2024-09-10T18:18:41.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:40 smithi028 ceph-mon[39220]: from='client.34148 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81f1855d-b946-4ca3-b246-38977b9e20fd"}]': finished 2024-09-10T18:18:41.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:40 smithi028 ceph-mon[39220]: osdmap e10: 2 total, 1 up, 2 in 2024-09-10T18:18:41.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:40 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:41.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:40 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/357104567' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:18:41.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:40 smithi028 ceph-mon[29291]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:41.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:40 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2466178680' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81f1855d-b946-4ca3-b246-38977b9e20fd"}]: dispatch 2024-09-10T18:18:41.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:40 smithi028 ceph-mon[29291]: from='client.34148 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81f1855d-b946-4ca3-b246-38977b9e20fd"}]: dispatch 2024-09-10T18:18:41.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:40 smithi028 ceph-mon[29291]: from='client.34148 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81f1855d-b946-4ca3-b246-38977b9e20fd"}]': finished 2024-09-10T18:18:41.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:40 smithi028 ceph-mon[29291]: osdmap e10: 2 total, 1 up, 2 in 2024-09-10T18:18:41.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:40 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:41.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:40 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/357104567' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:18:42.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:42 smithi204 ceph-mon[32357]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:42.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:42 smithi028 ceph-mon[29291]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:42.927 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:42 smithi028 ceph-mon[39220]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:44.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:44 smithi028 ceph-mon[29291]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:44.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:44 smithi028 ceph-mon[39220]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:44.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:44 smithi204 ceph-mon[32357]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:46 smithi028 ceph-mon[29291]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:46 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-10T18:18:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:46 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:46 smithi028 ceph-mon[29291]: Deploying daemon osd.1 on smithi028 2024-09-10T18:18:46.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:46 smithi028 ceph-mon[39220]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:46 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-10T18:18:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:46 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:46 smithi028 ceph-mon[39220]: Deploying daemon osd.1 on smithi028 2024-09-10T18:18:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:46 smithi204 ceph-mon[32357]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:46 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-10T18:18:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:46 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:46 smithi204 ceph-mon[32357]: Deploying daemon osd.1 on smithi028 2024-09-10T18:18:48.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:48 smithi028 ceph-mon[29291]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:48.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:48 smithi028 ceph-mon[39220]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:49.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:48 smithi204 ceph-mon[32357]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:51.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:50 smithi028 ceph-mon[29291]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:51.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:50 smithi028 ceph-mon[39220]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:51.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:50 smithi204 ceph-mon[32357]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:52.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:51 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:52.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:51 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:52.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:51 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:52.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:51 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:52.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:51 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:52.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:51 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:52.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:51 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:52.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:51 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:52.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:51 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:52.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:52 smithi028 ceph-mon[29291]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:52.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:52 smithi028 ceph-mon[39220]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:52 smithi204 ceph-mon[32357]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:54.470 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 1 on host 'smithi028' 2024-09-10T18:18:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:54.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:54.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:54.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[39220]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:54.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[39220]: from='osd.1 v1:172.21.15.28:6805/1369912678' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T18:18:54.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[39220]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T18:18:54.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:54.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:54.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:54.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:54.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:54.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[29291]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:54.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[29291]: from='osd.1 v1:172.21.15.28:6805/1369912678' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T18:18:54.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[29291]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T18:18:54.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:54.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:54.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:54 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:54 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:54 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:54 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:54 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:54 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:54 smithi204 ceph-mon[32357]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:54.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:54 smithi204 ceph-mon[32357]: from='osd.1 v1:172.21.15.28:6805/1369912678' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T18:18:54.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:54 smithi204 ceph-mon[32357]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T18:18:54.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:54 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:18:54.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:54 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:54.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:54 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:55.113 DEBUG:teuthology.orchestra.run.smithi028:osd.1> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.1.service 2024-09-10T18:18:55.116 INFO:tasks.cephadm:Deploying osd.2 on smithi028 with /dev/vg_nvme/lv_2... 2024-09-10T18:18:55.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-10T18:18:55.602 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:55 smithi028 ceph-mon[29291]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T18:18:55.602 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:55 smithi028 ceph-mon[29291]: osdmap e11: 2 total, 1 up, 2 in 2024-09-10T18:18:55.602 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:55 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:55.602 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:55 smithi028 ceph-mon[29291]: from='osd.1 v1:172.21.15.28:6805/1369912678' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:18:55.602 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:55 smithi028 ceph-mon[29291]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:18:55.602 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:55 smithi028 ceph-mon[39220]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T18:18:55.603 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:55 smithi028 ceph-mon[39220]: osdmap e11: 2 total, 1 up, 2 in 2024-09-10T18:18:55.603 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:55 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:55.603 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:55 smithi028 ceph-mon[39220]: from='osd.1 v1:172.21.15.28:6805/1369912678' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:18:55.603 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:55 smithi028 ceph-mon[39220]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:18:55.603 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:18:55 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-1[49149]: 2024-09-10T18:18:55.521+0000 7fcc53af9640 -1 osd.1 0 waiting for initial osdmap 2024-09-10T18:18:55.603 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:18:55 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-1[49149]: 2024-09-10T18:18:55.525+0000 7fcc4e930640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T18:18:55.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:55 smithi204 ceph-mon[32357]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T18:18:55.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:55 smithi204 ceph-mon[32357]: osdmap e11: 2 total, 1 up, 2 in 2024-09-10T18:18:55.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:55 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:55.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:55 smithi204 ceph-mon[32357]: from='osd.1 v1:172.21.15.28:6805/1369912678' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:18:55.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:55 smithi204 ceph-mon[32357]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:18:55.933 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:18:56.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:56 smithi028 ceph-mon[29291]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-10T18:18:56.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:56 smithi028 ceph-mon[29291]: osdmap e12: 2 total, 1 up, 2 in 2024-09-10T18:18:56.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:56 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:56.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:56 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:56.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:56 smithi028 ceph-mon[29291]: pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:56.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:56 smithi028 ceph-mon[39220]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-10T18:18:56.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:56 smithi028 ceph-mon[39220]: osdmap e12: 2 total, 1 up, 2 in 2024-09-10T18:18:56.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:56 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:56.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:56 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:56.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:56 smithi028 ceph-mon[39220]: pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:56 smithi204 ceph-mon[32357]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-10T18:18:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:56 smithi204 ceph-mon[32357]: osdmap e12: 2 total, 1 up, 2 in 2024-09-10T18:18:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:56 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:56 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:56 smithi204 ceph-mon[32357]: pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:18:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:57 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:18:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:57 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:18:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:57 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:57 smithi028 ceph-mon[39220]: osd.1 v1:172.21.15.28:6805/1369912678 boot 2024-09-10T18:18:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:57 smithi028 ceph-mon[39220]: osdmap e13: 2 total, 2 up, 2 in 2024-09-10T18:18:57.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:57 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:57.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:57 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:18:57.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:57 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:18:57.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:57 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:57.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:57 smithi028 ceph-mon[29291]: osd.1 v1:172.21.15.28:6805/1369912678 boot 2024-09-10T18:18:57.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:57 smithi028 ceph-mon[29291]: osdmap e13: 2 total, 2 up, 2 in 2024-09-10T18:18:57.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:57 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:57 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:18:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:57 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:18:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:57 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:57 smithi204 ceph-mon[32357]: osd.1 v1:172.21.15.28:6805/1369912678 boot 2024-09-10T18:18:57.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:57 smithi204 ceph-mon[32357]: osdmap e13: 2 total, 2 up, 2 in 2024-09-10T18:18:57.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:57 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:18:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:58 smithi204 ceph-mon[32357]: osdmap e14: 2 total, 2 up, 2 in 2024-09-10T18:18:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:58 smithi204 ceph-mon[32357]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:18:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:18:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:18:58.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:58.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:58.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:58.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:58.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[29291]: osdmap e14: 2 total, 2 up, 2 in 2024-09-10T18:18:58.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[29291]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:18:58.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:58.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:58.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:18:58.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:18:58.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:58.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:58.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:58.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:58.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[39220]: osdmap e14: 2 total, 2 up, 2 in 2024-09-10T18:18:58.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[39220]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:18:58.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:58.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:58.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:18:58.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:18:58.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:58.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:18:58.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:18:58.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:18:59.438 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-10T18:18:59.516 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-10T18:18:59.526 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-09-10T18:18:59.526 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-09-10T18:18:59.526 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00837809 s, 1.3 GB/s 2024-09-10T18:18:59.527 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-09-10T18:18:59.839 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:59 smithi204 ceph-mon[32357]: Detected new or changed devices on smithi028 2024-09-10T18:18:59.839 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:18:59 smithi204 ceph-mon[32357]: Adjusting osd_memory_target on smithi028 to 8034M 2024-09-10T18:18:59.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:59 smithi028 ceph-mon[29291]: Detected new or changed devices on smithi028 2024-09-10T18:18:59.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:18:59 smithi028 ceph-mon[29291]: Adjusting osd_memory_target on smithi028 to 8034M 2024-09-10T18:18:59.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:59 smithi028 ceph-mon[39220]: Detected new or changed devices on smithi028 2024-09-10T18:18:59.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:18:59 smithi028 ceph-mon[39220]: Adjusting osd_memory_target on smithi028 to 8034M 2024-09-10T18:19:00.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_2 2024-09-10T18:19:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:19:00.847 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:00 smithi028 ceph-mon[29291]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:00.847 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:00 smithi028 ceph-mon[39220]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:00 smithi204 ceph-mon[32357]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:02.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:02 smithi028 ceph-mon[39220]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:02.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:02 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:19:02.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:02 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:19:02.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:02 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:02.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:02 smithi028 ceph-mon[29291]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:02.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:02 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:19:02.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:02 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:19:02.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:02 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:02 smithi204 ceph-mon[32357]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:02 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:19:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:02 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:19:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:02 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:03.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:03 smithi028 ceph-mon[29291]: from='client.14268 v1:172.21.15.28:0/1636739973' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:19:03.814 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:03 smithi028 ceph-mon[39220]: from='client.14268 v1:172.21.15.28:0/1636739973' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:19:04.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:03 smithi204 ceph-mon[32357]: from='client.14268 v1:172.21.15.28:0/1636739973' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:19:05.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:04 smithi028 ceph-mon[29291]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:05.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:04 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4048186059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ad55d2a-36fa-4756-bf22-58d4b049486e"}]: dispatch 2024-09-10T18:19:05.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:04 smithi028 ceph-mon[29291]: from='client.34175 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ad55d2a-36fa-4756-bf22-58d4b049486e"}]: dispatch 2024-09-10T18:19:05.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:04 smithi028 ceph-mon[29291]: from='client.34175 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ad55d2a-36fa-4756-bf22-58d4b049486e"}]': finished 2024-09-10T18:19:05.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:04 smithi028 ceph-mon[29291]: osdmap e15: 3 total, 2 up, 3 in 2024-09-10T18:19:05.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:04 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:05.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:04 smithi028 ceph-mon[39220]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:05.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:04 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4048186059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ad55d2a-36fa-4756-bf22-58d4b049486e"}]: dispatch 2024-09-10T18:19:05.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:04 smithi028 ceph-mon[39220]: from='client.34175 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ad55d2a-36fa-4756-bf22-58d4b049486e"}]: dispatch 2024-09-10T18:19:05.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:04 smithi028 ceph-mon[39220]: from='client.34175 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ad55d2a-36fa-4756-bf22-58d4b049486e"}]': finished 2024-09-10T18:19:05.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:04 smithi028 ceph-mon[39220]: osdmap e15: 3 total, 2 up, 3 in 2024-09-10T18:19:05.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:04 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:05.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:04 smithi204 ceph-mon[32357]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:05.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:04 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4048186059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ad55d2a-36fa-4756-bf22-58d4b049486e"}]: dispatch 2024-09-10T18:19:05.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:04 smithi204 ceph-mon[32357]: from='client.34175 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ad55d2a-36fa-4756-bf22-58d4b049486e"}]: dispatch 2024-09-10T18:19:05.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:04 smithi204 ceph-mon[32357]: from='client.34175 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ad55d2a-36fa-4756-bf22-58d4b049486e"}]': finished 2024-09-10T18:19:05.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:04 smithi204 ceph-mon[32357]: osdmap e15: 3 total, 2 up, 3 in 2024-09-10T18:19:05.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:04 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:06.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:05 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1223476361' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:19:06.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:05 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1223476361' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:19:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:05 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1223476361' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:19:07.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:06 smithi028 ceph-mon[29291]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:07.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:06 smithi028 ceph-mon[39220]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:07.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:06 smithi204 ceph-mon[32357]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:08.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:08 smithi028 ceph-mon[29291]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:08.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:08 smithi028 ceph-mon[39220]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:09.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:08 smithi204 ceph-mon[32357]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:11.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:10 smithi028 ceph-mon[29291]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:11.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:10 smithi028 ceph-mon[39220]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:11.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:10 smithi204 ceph-mon[32357]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:11.988 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:11 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-10T18:19:11.988 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:11 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:11.988 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:11 smithi028 ceph-mon[29291]: Deploying daemon osd.2 on smithi028 2024-09-10T18:19:11.988 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:11 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-10T18:19:11.988 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:11 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:11.988 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:11 smithi028 ceph-mon[39220]: Deploying daemon osd.2 on smithi028 2024-09-10T18:19:12.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:11 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-10T18:19:12.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:11 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:12.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:11 smithi204 ceph-mon[32357]: Deploying daemon osd.2 on smithi028 2024-09-10T18:19:12.851 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:12 smithi028 ceph-mon[29291]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:12.851 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:12 smithi028 ceph-mon[39220]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:13.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:12 smithi204 ceph-mon[32357]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:14.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:14 smithi028 ceph-mon[29291]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:14.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:14 smithi028 ceph-mon[39220]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:15.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:14 smithi204 ceph-mon[32357]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:16.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:16 smithi028 ceph-mon[39220]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:16.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:16 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:19:16.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:16 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:16.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:16 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:16.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:16 smithi028 ceph-mon[29291]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:16.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:16 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:19:16.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:16 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:16.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:16 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:17.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:16 smithi204 ceph-mon[32357]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:17.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:16 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:19:17.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:16 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:17.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:16 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:18.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:18 smithi028 ceph-mon[29291]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:18.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:18 smithi028 ceph-mon[39220]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:19.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:18 smithi204 ceph-mon[32357]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:19.649 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 2 on host 'smithi028' 2024-09-10T18:19:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[29291]: from='osd.2 v1:172.21.15.28:6809/1243406003' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-10T18:19:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[29291]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-10T18:19:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:19:20.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:20.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:19:20.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:20.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:20.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:20.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:20.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[39220]: from='osd.2 v1:172.21.15.28:6809/1243406003' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-10T18:19:20.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[39220]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-10T18:19:20.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:20.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:19:20.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:20.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:19:20.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:20.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:19 smithi204 ceph-mon[32357]: from='osd.2 v1:172.21.15.28:6809/1243406003' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-10T18:19:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:19 smithi204 ceph-mon[32357]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-10T18:19:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:19:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:20.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:19:20.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:20.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:20.316 DEBUG:teuthology.orchestra.run.smithi028:osd.2> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.2.service 2024-09-10T18:19:20.318 INFO:tasks.cephadm:Deploying osd.3 on smithi028 with /dev/vg_nvme/lv_1... 2024-09-10T18:19:20.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-10T18:19:20.986 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:19:21.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:20 smithi028 ceph-mon[29291]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:21.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:20 smithi028 ceph-mon[29291]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-10T18:19:21.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:20 smithi028 ceph-mon[29291]: osdmap e16: 3 total, 2 up, 3 in 2024-09-10T18:19:21.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:20 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:21.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:20 smithi028 ceph-mon[29291]: from='osd.2 v1:172.21.15.28:6809/1243406003' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:19:21.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:20 smithi028 ceph-mon[29291]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:19:21.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:20 smithi028 ceph-mon[39220]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:21.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:20 smithi028 ceph-mon[39220]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-10T18:19:21.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:20 smithi028 ceph-mon[39220]: osdmap e16: 3 total, 2 up, 3 in 2024-09-10T18:19:21.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:20 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:21.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:20 smithi028 ceph-mon[39220]: from='osd.2 v1:172.21.15.28:6809/1243406003' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:19:21.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:20 smithi028 ceph-mon[39220]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:19:21.005 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:19:20 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-2[54713]: 2024-09-10T18:19:20.790+0000 7fea7d842640 -1 osd.2 0 waiting for initial osdmap 2024-09-10T18:19:21.005 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:19:20 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-2[54713]: 2024-09-10T18:19:20.793+0000 7fea7a67d640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T18:19:21.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:20 smithi204 ceph-mon[32357]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:21.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:20 smithi204 ceph-mon[32357]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-10T18:19:21.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:20 smithi204 ceph-mon[32357]: osdmap e16: 3 total, 2 up, 3 in 2024-09-10T18:19:21.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:20 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:21.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:20 smithi204 ceph-mon[32357]: from='osd.2 v1:172.21.15.28:6809/1243406003' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:19:21.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:20 smithi204 ceph-mon[32357]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:19:22.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:21 smithi204 ceph-mon[32357]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-10T18:19:22.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:21 smithi204 ceph-mon[32357]: osdmap e17: 3 total, 2 up, 3 in 2024-09-10T18:19:22.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:21 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:22.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:21 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:22.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:21 smithi028 ceph-mon[39220]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-10T18:19:22.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:21 smithi028 ceph-mon[39220]: osdmap e17: 3 total, 2 up, 3 in 2024-09-10T18:19:22.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:21 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:22.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:21 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:22.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:21 smithi028 ceph-mon[29291]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-10T18:19:22.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:21 smithi028 ceph-mon[29291]: osdmap e17: 3 total, 2 up, 3 in 2024-09-10T18:19:22.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:21 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:22.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:21 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:23.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:22 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:19:23.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:22 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:19:23.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:22 smithi028 ceph-mon[39220]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:23.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:22 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:23.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:22 smithi028 ceph-mon[39220]: osd.2 v1:172.21.15.28:6809/1243406003 boot 2024-09-10T18:19:23.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:22 smithi028 ceph-mon[39220]: osdmap e18: 3 total, 3 up, 3 in 2024-09-10T18:19:23.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:22 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:23.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:22 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:19:23.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:22 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:19:23.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:22 smithi028 ceph-mon[29291]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:23.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:22 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:23.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:22 smithi028 ceph-mon[29291]: osd.2 v1:172.21.15.28:6809/1243406003 boot 2024-09-10T18:19:23.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:22 smithi028 ceph-mon[29291]: osdmap e18: 3 total, 3 up, 3 in 2024-09-10T18:19:23.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:22 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:23.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:22 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:19:23.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:22 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:19:23.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:22 smithi204 ceph-mon[32357]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:19:23.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:22 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:23.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:22 smithi204 ceph-mon[32357]: osd.2 v1:172.21.15.28:6809/1243406003 boot 2024-09-10T18:19:23.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:22 smithi204 ceph-mon[32357]: osdmap e18: 3 total, 3 up, 3 in 2024-09-10T18:19:23.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:22 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:19:23.902 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-10T18:19:23.999 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-10T18:19:24.009 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-09-10T18:19:24.009 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-09-10T18:19:24.010 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0084798 s, 1.2 GB/s 2024-09-10T18:19:24.010 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-09-10T18:19:24.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:24 smithi028 ceph-mon[39220]: osdmap e19: 3 total, 3 up, 3 in 2024-09-10T18:19:24.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:24 smithi028 ceph-mon[29291]: osdmap e19: 3 total, 3 up, 3 in 2024-09-10T18:19:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:24 smithi204 ceph-mon[32357]: osdmap e19: 3 total, 3 up, 3 in 2024-09-10T18:19:24.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_1 2024-09-10T18:19:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:19:25.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[39220]: pgmap v77: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:25.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-10T18:19:25.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-10T18:19:25.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[39220]: osdmap e20: 3 total, 3 up, 3 in 2024-09-10T18:19:25.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:19:25.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[39220]: Detected new or changed devices on smithi028 2024-09-10T18:19:25.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:25.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:25.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:25.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:25.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:25.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[39220]: Adjusting osd_memory_target on smithi028 to 5356M 2024-09-10T18:19:25.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:25.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:25.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:19:25.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:25.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[29291]: pgmap v77: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:25.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-10T18:19:25.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-10T18:19:25.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[29291]: osdmap e20: 3 total, 3 up, 3 in 2024-09-10T18:19:25.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:19:25.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[29291]: Detected new or changed devices on smithi028 2024-09-10T18:19:25.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:25.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:25.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:25.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:25.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:25.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[29291]: Adjusting osd_memory_target on smithi028 to 5356M 2024-09-10T18:19:25.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:25.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:25.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:19:25.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:25 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:25 smithi204 ceph-mon[32357]: pgmap v77: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:25 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-10T18:19:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:25 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-10T18:19:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:25 smithi204 ceph-mon[32357]: osdmap e20: 3 total, 3 up, 3 in 2024-09-10T18:19:25.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:25 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:19:25.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:25 smithi204 ceph-mon[32357]: Detected new or changed devices on smithi028 2024-09-10T18:19:25.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:25 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:25.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:25 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:25.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:25 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:25.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:25 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:25.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:25 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:25.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:25 smithi204 ceph-mon[32357]: Adjusting osd_memory_target on smithi028 to 5356M 2024-09-10T18:19:25.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:25 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:25.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:25 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:25.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:25 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:19:25.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:25 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:26.004 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:19:25 smithi028 sudo[58955]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T18:19:26.004 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:19:25 smithi028 sudo[58955]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:19:26.004 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:19:25 smithi028 sudo[58955]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:19:26.004 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:19:25 smithi028 sudo[58955]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:19:26.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:26 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:19:26.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:26 smithi028 ceph-mon[29291]: osdmap e21: 3 total, 3 up, 3 in 2024-09-10T18:19:26.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:26 smithi028 ceph-mon[29291]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:26.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:26 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:19:26.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:26 smithi028 ceph-mon[39220]: osdmap e21: 3 total, 3 up, 3 in 2024-09-10T18:19:26.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:26 smithi028 ceph-mon[39220]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:26.332 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:19:26 smithi028 sudo[58962]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T18:19:26.332 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:19:26 smithi028 sudo[58962]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:19:26.332 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:19:26 smithi028 sudo[58962]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:19:26.332 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:19:26 smithi028 sudo[58962]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:19:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:26 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:19:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:26 smithi204 ceph-mon[32357]: osdmap e21: 3 total, 3 up, 3 in 2024-09-10T18:19:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:26 smithi204 ceph-mon[32357]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:26.879 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:19:26 smithi028 sudo[58991]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T18:19:26.879 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:19:26 smithi028 sudo[58991]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:19:26.879 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:19:26 smithi028 sudo[58991]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:19:26.879 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:19:26 smithi028 sudo[58991]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:19:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:27 smithi028 ceph-mon[29291]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-10T18:19:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:27 smithi028 ceph-mon[29291]: osdmap e22: 3 total, 3 up, 3 in 2024-09-10T18:19:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:19:27.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:19:27.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:27.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:27.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:27 smithi028 ceph-mon[39220]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-10T18:19:27.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:27 smithi028 ceph-mon[39220]: osdmap e22: 3 total, 3 up, 3 in 2024-09-10T18:19:27.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:19:27.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:19:27.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:27.506 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:19:27 smithi028 sudo[59082]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T18:19:27.506 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:19:27 smithi028 sudo[59082]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:19:27.506 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:19:27 smithi028 sudo[59082]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:19:27.506 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:19:27 smithi028 sudo[59082]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:19:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:27 smithi204 ceph-mon[32357]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-10T18:19:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:27 smithi204 ceph-mon[32357]: osdmap e22: 3 total, 3 up, 3 in 2024-09-10T18:19:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:19:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:19:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:27.848 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:19:27 smithi028 sudo[59141]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T18:19:27.848 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:19:27 smithi028 sudo[59141]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:19:27.849 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:19:27 smithi028 sudo[59141]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:19:27.849 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:19:27 smithi028 sudo[59141]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:19:28.254 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:19:28 smithi028 sudo[59161]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T18:19:28.254 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:19:28 smithi028 sudo[59161]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:19:28.254 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:19:28 smithi028 sudo[59161]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:19:28.254 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:19:28 smithi028 sudo[59161]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:19:28.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:28 smithi028 sudo[59168]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T18:19:28.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:28 smithi028 sudo[59168]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:19:28.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:28 smithi028 sudo[59168]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:19:28.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:28 smithi028 sudo[59168]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:19:29.218 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:28 smithi204 ceph-mon[32357]: from='client.14301 v1:172.21.15.28:0/3621618828' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:19:29.218 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:28 smithi204 ceph-mon[32357]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:29.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:28 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:19:29.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:28 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:19:29.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:28 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:19:29.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:28 smithi204 ceph-mon[32357]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:19:29.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:29 smithi204 sudo[34848]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T18:19:29.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:29 smithi204 sudo[34848]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:19:29.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:29 smithi204 sudo[34848]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:19:29.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:29 smithi204 sudo[34848]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:19:29.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:28 smithi028 ceph-mon[39220]: from='client.14301 v1:172.21.15.28:0/3621618828' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:19:29.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:28 smithi028 ceph-mon[39220]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:29.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:28 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:19:29.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:28 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:19:29.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:28 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:19:29.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:28 smithi028 ceph-mon[39220]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:19:29.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:28 smithi028 sudo[59199]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T18:19:29.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:28 smithi028 sudo[59199]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:19:29.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:28 smithi028 sudo[59199]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:19:29.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:28 smithi028 sudo[59199]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:19:29.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:28 smithi028 ceph-mon[29291]: from='client.14301 v1:172.21.15.28:0/3621618828' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:19:29.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:28 smithi028 ceph-mon[29291]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:29.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:28 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:19:29.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:28 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:19:29.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:28 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:19:29.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:28 smithi028 ceph-mon[29291]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:19:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:29 smithi204 sudo[34851]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T18:19:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:29 smithi204 sudo[34851]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:19:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:29 smithi204 sudo[34851]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:19:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:29 smithi204 sudo[34851]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:19:30.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:29 smithi028 sudo[59275]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T18:19:30.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:29 smithi028 sudo[59275]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:19:30.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:29 smithi028 sudo[59275]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:19:30.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:29 smithi028 sudo[59275]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:19:30.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[29291]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:19:30.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:19:30.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:19:30.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[29291]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:19:30.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:19:30.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3664478297' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "242729a5-a5e8-44d5-97c8-14d14e970bff"}]: dispatch 2024-09-10T18:19:30.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[29291]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:19:30.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3664478297' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "242729a5-a5e8-44d5-97c8-14d14e970bff"}]': finished 2024-09-10T18:19:30.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[29291]: osdmap e23: 4 total, 3 up, 4 in 2024-09-10T18:19:30.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:30.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:30.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:19:30.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:19:30.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:19:30.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 sudo[59279]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T18:19:30.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 sudo[59279]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:19:30.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 sudo[59279]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:19:30.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 sudo[59279]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:19:30.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[39220]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:19:30.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:19:30.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:19:30.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[39220]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:19:30.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:19:30.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3664478297' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "242729a5-a5e8-44d5-97c8-14d14e970bff"}]: dispatch 2024-09-10T18:19:30.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[39220]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:19:30.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3664478297' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "242729a5-a5e8-44d5-97c8-14d14e970bff"}]': finished 2024-09-10T18:19:30.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[39220]: osdmap e23: 4 total, 3 up, 4 in 2024-09-10T18:19:30.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:30.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:30.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:19:30.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:19:30.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:19:30.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:30 smithi204 ceph-mon[32357]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:19:30.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:19:30.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:19:30.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:30 smithi204 ceph-mon[32357]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:19:30.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:19:30.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:30 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3664478297' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "242729a5-a5e8-44d5-97c8-14d14e970bff"}]: dispatch 2024-09-10T18:19:30.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:30 smithi204 ceph-mon[32357]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:19:30.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:30 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3664478297' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "242729a5-a5e8-44d5-97c8-14d14e970bff"}]': finished 2024-09-10T18:19:30.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:30 smithi204 ceph-mon[32357]: osdmap e23: 4 total, 3 up, 4 in 2024-09-10T18:19:30.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:30.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:30.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:19:30.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:19:30.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:19:31.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:31 smithi028 ceph-mon[29291]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:19:31.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:31 smithi028 ceph-mon[29291]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:31.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:31 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3772665486' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:19:31.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:31 smithi028 ceph-mon[29291]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:19:31.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:31 smithi028 ceph-mon[39220]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:19:31.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:31 smithi028 ceph-mon[39220]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:31.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:31 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3772665486' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:19:31.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:31 smithi028 ceph-mon[39220]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:19:31.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:31 smithi204 ceph-mon[32357]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:19:31.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:31 smithi204 ceph-mon[32357]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:31.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:31 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3772665486' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:19:31.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:31 smithi204 ceph-mon[32357]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:19:32.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:32 smithi028 ceph-mon[29291]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:32.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:32 smithi028 ceph-mon[39220]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:32.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:32 smithi204 ceph-mon[32357]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:34.849 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:34 smithi028 ceph-mon[29291]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-10T18:19:34.849 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:34 smithi028 ceph-mon[39220]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-10T18:19:35.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:34 smithi204 ceph-mon[32357]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-10T18:19:36.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:36 smithi028 ceph-mon[29291]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-10T18:19:36.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:36 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-10T18:19:36.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:36 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:36.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:36 smithi028 ceph-mon[29291]: Deploying daemon osd.3 on smithi028 2024-09-10T18:19:36.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:36 smithi028 ceph-mon[39220]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-10T18:19:36.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:36 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-10T18:19:36.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:36 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:36.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:36 smithi028 ceph-mon[39220]: Deploying daemon osd.3 on smithi028 2024-09-10T18:19:37.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:36 smithi204 ceph-mon[32357]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-10T18:19:37.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:36 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-10T18:19:37.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:36 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:37.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:36 smithi204 ceph-mon[32357]: Deploying daemon osd.3 on smithi028 2024-09-10T18:19:38.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:38 smithi028 ceph-mon[29291]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-10T18:19:38.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:38 smithi028 ceph-mon[39220]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-10T18:19:39.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:38 smithi204 ceph-mon[32357]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-10T18:19:40.745 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:40 smithi028 ceph-mon[29291]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-10T18:19:40.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:40 smithi028 ceph-mon[39220]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-10T18:19:41.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:40 smithi204 ceph-mon[32357]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-10T18:19:41.834 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:41 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:19:41.834 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:41 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:41.834 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:41 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:41.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:41 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:19:41.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:41 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:41.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:41 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:41 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:19:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:41 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:41 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:43.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:42 smithi028 ceph-mon[29291]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-10T18:19:43.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:42 smithi028 ceph-mon[39220]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-10T18:19:43.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:42 smithi204 ceph-mon[32357]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-10T18:19:43.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:43 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:19:43.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:43 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:19:43.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:43 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:19:43.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:43 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:19:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:43 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:19:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:43 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:19:44.491 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 3 on host 'smithi028' 2024-09-10T18:19:44.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[39220]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-10T18:19:44.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[39220]: osdmap e24: 4 total, 3 up, 4 in 2024-09-10T18:19:44.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:44.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:44.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:44.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:44.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:19:44.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:44.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[39220]: from='osd.3 v1:172.21.15.28:6813/67703283' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T18:19:44.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[39220]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T18:19:44.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:19:44.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:44.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:44.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[29291]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-10T18:19:44.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[29291]: osdmap e24: 4 total, 3 up, 4 in 2024-09-10T18:19:44.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:44.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:44.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:44.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:44.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:19:44.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:44.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[29291]: from='osd.3 v1:172.21.15.28:6813/67703283' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T18:19:44.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[29291]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T18:19:44.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:19:44.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:44.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:44 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:45.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:44 smithi204 ceph-mon[32357]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-10T18:19:45.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:44 smithi204 ceph-mon[32357]: osdmap e24: 4 total, 3 up, 4 in 2024-09-10T18:19:45.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:44 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:45.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:44 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:45.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:44 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:45.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:44 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:45.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:44 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:19:45.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:44 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:45.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:44 smithi204 ceph-mon[32357]: from='osd.3 v1:172.21.15.28:6813/67703283' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T18:19:45.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:44 smithi204 ceph-mon[32357]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T18:19:45.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:44 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:19:45.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:44 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:45.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:44 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:45.321 DEBUG:teuthology.orchestra.run.smithi028:osd.3> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.3.service 2024-09-10T18:19:45.323 INFO:tasks.cephadm:Deploying osd.4 on smithi204 with /dev/vg_nvme/lv_4... 2024-09-10T18:19:45.323 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-10T18:19:45.568 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:19:46.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:45 smithi028 ceph-mon[39220]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-10T18:19:46.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:45 smithi028 ceph-mon[39220]: osdmap e25: 4 total, 3 up, 4 in 2024-09-10T18:19:46.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:45 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:46.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:45 smithi028 ceph-mon[39220]: from='osd.3 v1:172.21.15.28:6813/67703283' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:19:46.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:45 smithi028 ceph-mon[39220]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:19:46.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:45 smithi028 ceph-mon[29291]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-10T18:19:46.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:45 smithi028 ceph-mon[29291]: osdmap e25: 4 total, 3 up, 4 in 2024-09-10T18:19:46.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:45 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:46.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:45 smithi028 ceph-mon[29291]: from='osd.3 v1:172.21.15.28:6813/67703283' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:19:46.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:45 smithi028 ceph-mon[29291]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:19:46.005 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 10 18:19:45 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-3[60920]: 2024-09-10T18:19:45.720+0000 7f93b0b83640 -1 osd.3 0 waiting for initial osdmap 2024-09-10T18:19:46.006 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 10 18:19:45 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-3[60920]: 2024-09-10T18:19:45.724+0000 7f93ad1bd640 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T18:19:46.063 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:45 smithi204 ceph-mon[32357]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-10T18:19:46.063 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:45 smithi204 ceph-mon[32357]: osdmap e25: 4 total, 3 up, 4 in 2024-09-10T18:19:46.063 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:45 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:46.063 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:45 smithi204 ceph-mon[32357]: from='osd.3 v1:172.21.15.28:6813/67703283' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:19:46.063 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:45 smithi204 ceph-mon[32357]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-10T18:19:46.948 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:46 smithi204 ceph-mon[32357]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:46.949 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:46 smithi204 ceph-mon[32357]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-10T18:19:46.949 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:46 smithi204 ceph-mon[32357]: osdmap e26: 4 total, 3 up, 4 in 2024-09-10T18:19:46.949 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:46 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:46.949 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:46 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:47.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:46 smithi028 ceph-mon[29291]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:47.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:46 smithi028 ceph-mon[29291]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-10T18:19:47.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:46 smithi028 ceph-mon[29291]: osdmap e26: 4 total, 3 up, 4 in 2024-09-10T18:19:47.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:46 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:47.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:46 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:47.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:46 smithi028 ceph-mon[39220]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:47.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:46 smithi028 ceph-mon[39220]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-10T18:19:47.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:46 smithi028 ceph-mon[39220]: osdmap e26: 4 total, 3 up, 4 in 2024-09-10T18:19:47.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:46 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:47.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:46 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:47.307 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-10T18:19:47.393 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-10T18:19:47.406 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-09-10T18:19:47.406 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-09-10T18:19:47.406 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114714 s, 914 MB/s 2024-09-10T18:19:47.406 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-09-10T18:19:48.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:47 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:19:48.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:47 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:19:48.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:47 smithi028 ceph-mon[39220]: osd.3 v1:172.21.15.28:6813/67703283 boot 2024-09-10T18:19:48.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:47 smithi028 ceph-mon[39220]: osdmap e27: 4 total, 4 up, 4 in 2024-09-10T18:19:48.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:47 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:48.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:47 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:19:48.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:47 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:19:48.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:47 smithi028 ceph-mon[29291]: osd.3 v1:172.21.15.28:6813/67703283 boot 2024-09-10T18:19:48.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:47 smithi028 ceph-mon[29291]: osdmap e27: 4 total, 4 up, 4 in 2024-09-10T18:19:48.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:47 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:48.015 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:47 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:19:48.015 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:47 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:19:48.015 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:47 smithi204 ceph-mon[32357]: osd.3 v1:172.21.15.28:6813/67703283 boot 2024-09-10T18:19:48.015 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:47 smithi204 ceph-mon[32357]: osdmap e27: 4 total, 4 up, 4 in 2024-09-10T18:19:48.015 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:47 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:19:48.038 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_4 2024-09-10T18:19:48.285 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:19:49.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[39220]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:49.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[39220]: osdmap e28: 4 total, 4 up, 4 in 2024-09-10T18:19:49.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:49.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:49.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:49.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:49.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:49.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:49.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:49.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:49.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:19:49.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:49.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[29291]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:49.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[29291]: osdmap e28: 4 total, 4 up, 4 in 2024-09-10T18:19:49.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:49.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:49.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:49.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:49.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:49.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:49.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:49.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:49.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:19:49.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:49.046 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:48 smithi204 ceph-mon[32357]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:19:49.046 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:48 smithi204 ceph-mon[32357]: osdmap e28: 4 total, 4 up, 4 in 2024-09-10T18:19:49.046 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:49.046 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:49.046 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:49.046 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:49.046 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:49.046 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:19:49.046 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:49.047 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:49.047 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:19:49.047 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:19:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:49 smithi204 ceph-mon[32357]: Detected new or changed devices on smithi028 2024-09-10T18:19:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:49 smithi204 ceph-mon[32357]: Adjusting osd_memory_target on smithi028 to 4017M 2024-09-10T18:19:50.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:49 smithi028 ceph-mon[39220]: Detected new or changed devices on smithi028 2024-09-10T18:19:50.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:49 smithi028 ceph-mon[39220]: Adjusting osd_memory_target on smithi028 to 4017M 2024-09-10T18:19:50.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:49 smithi028 ceph-mon[29291]: Detected new or changed devices on smithi028 2024-09-10T18:19:50.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:49 smithi028 ceph-mon[29291]: Adjusting osd_memory_target on smithi028 to 4017M 2024-09-10T18:19:50.989 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:50 smithi204 ceph-mon[32357]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:19:50.990 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:50 smithi204 ceph-mon[32357]: from='client.24226 v1:172.21.15.204:0/103109644' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:19:50.990 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:50 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:19:50.990 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:50 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:19:50.990 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:50 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:51.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:50 smithi028 ceph-mon[29291]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:19:51.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:50 smithi028 ceph-mon[29291]: from='client.24226 v1:172.21.15.204:0/103109644' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:19:51.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:50 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:19:51.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:50 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:19:51.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:50 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:51.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:50 smithi028 ceph-mon[39220]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:19:51.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:50 smithi028 ceph-mon[39220]: from='client.24226 v1:172.21.15.204:0/103109644' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:19:51.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:50 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:19:51.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:50 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:19:51.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:50 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:53.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:52 smithi028 ceph-mon[39220]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:19:53.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:52 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.204:0/1149978229' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e38b8ef-e6ed-493f-8eca-755354754787"}]: dispatch 2024-09-10T18:19:53.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:52 smithi028 ceph-mon[39220]: from='client.24232 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e38b8ef-e6ed-493f-8eca-755354754787"}]: dispatch 2024-09-10T18:19:53.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:52 smithi028 ceph-mon[39220]: from='client.24232 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e38b8ef-e6ed-493f-8eca-755354754787"}]': finished 2024-09-10T18:19:53.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:52 smithi028 ceph-mon[39220]: osdmap e29: 5 total, 4 up, 5 in 2024-09-10T18:19:53.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:19:53.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:52 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.204:0/2401844840' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:19:53.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:52 smithi028 ceph-mon[29291]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:19:53.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:52 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.204:0/1149978229' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e38b8ef-e6ed-493f-8eca-755354754787"}]: dispatch 2024-09-10T18:19:53.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:52 smithi028 ceph-mon[29291]: from='client.24232 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e38b8ef-e6ed-493f-8eca-755354754787"}]: dispatch 2024-09-10T18:19:53.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:52 smithi028 ceph-mon[29291]: from='client.24232 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e38b8ef-e6ed-493f-8eca-755354754787"}]': finished 2024-09-10T18:19:53.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:52 smithi028 ceph-mon[29291]: osdmap e29: 5 total, 4 up, 5 in 2024-09-10T18:19:53.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:52 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:19:53.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:52 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.204:0/2401844840' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:19:53.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:52 smithi204 ceph-mon[32357]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:19:53.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:52 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.204:0/1149978229' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e38b8ef-e6ed-493f-8eca-755354754787"}]: dispatch 2024-09-10T18:19:53.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:52 smithi204 ceph-mon[32357]: from='client.24232 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e38b8ef-e6ed-493f-8eca-755354754787"}]: dispatch 2024-09-10T18:19:53.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:52 smithi204 ceph-mon[32357]: from='client.24232 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e38b8ef-e6ed-493f-8eca-755354754787"}]': finished 2024-09-10T18:19:53.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:52 smithi204 ceph-mon[32357]: osdmap e29: 5 total, 4 up, 5 in 2024-09-10T18:19:53.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:52 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:19:53.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:52 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.204:0/2401844840' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:19:55.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:54 smithi028 ceph-mon[29291]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:19:55.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:54 smithi028 ceph-mon[39220]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:19:55.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:54 smithi204 ceph-mon[32357]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:19:57.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:56 smithi028 ceph-mon[29291]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:19:57.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:56 smithi028 ceph-mon[39220]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:19:57.013 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:56 smithi204 ceph-mon[32357]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:19:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:58 smithi204 ceph-mon[32357]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:19:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-10T18:19:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:59.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:58 smithi028 ceph-mon[29291]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:19:59.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-10T18:19:59.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:59.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:58 smithi028 ceph-mon[39220]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:19:59.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-10T18:19:59.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:19:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:19:59 smithi204 ceph-mon[32357]: Deploying daemon osd.4 on smithi204 2024-09-10T18:20:00.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:19:59 smithi028 ceph-mon[29291]: Deploying daemon osd.4 on smithi204 2024-09-10T18:20:00.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:19:59 smithi028 ceph-mon[39220]: Deploying daemon osd.4 on smithi204 2024-09-10T18:20:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:00 smithi204 ceph-mon[32357]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:00 smithi204 ceph-mon[32357]: overall HEALTH_OK 2024-09-10T18:20:01.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:00 smithi028 ceph-mon[29291]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:01.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:00 smithi028 ceph-mon[29291]: overall HEALTH_OK 2024-09-10T18:20:01.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:00 smithi028 ceph-mon[39220]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:01.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:00 smithi028 ceph-mon[39220]: overall HEALTH_OK 2024-09-10T18:20:03.036 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:02 smithi204 ceph-mon[32357]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:03.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:02 smithi028 ceph-mon[29291]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:03.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:02 smithi028 ceph-mon[39220]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:04.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:03 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:04.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:03 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:04.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:03 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:04.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:03 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:04.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:03 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:04.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:03 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:03 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:03 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:03 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:05.011 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:04 smithi204 ceph-mon[32357]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:05.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:04 smithi028 ceph-mon[29291]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:05.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:04 smithi028 ceph-mon[39220]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:05 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:20:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:05 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:20:06.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:05 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:06.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:05 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:05 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:20:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:05 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:20:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:05 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:05 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:06.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:05 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:20:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:05 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:20:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:05 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:05 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:06.473 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 4 on host 'smithi204' 2024-09-10T18:20:07.048 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:06 smithi204 ceph-mon[32357]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:07.048 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:06 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:07.048 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:06 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:07.048 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:06 smithi204 ceph-mon[32357]: osdmap e30: 5 total, 4 up, 5 in 2024-09-10T18:20:07.048 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:06 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:20:07.048 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:06 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:07.049 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:06 smithi204 ceph-mon[32357]: from='osd.4 v1:172.21.15.204:6800/1619669076' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T18:20:07.049 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:06 smithi204 ceph-mon[32357]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T18:20:07.049 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:06 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:07.049 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:06 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:07.049 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:06 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:07.071 DEBUG:teuthology.orchestra.run.smithi204:osd.4> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.4.service 2024-09-10T18:20:07.074 INFO:tasks.cephadm:Deploying osd.5 on smithi204 with /dev/vg_nvme/lv_3... 2024-09-10T18:20:07.074 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-10T18:20:07.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[29291]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:07.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:07.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[29291]: osdmap e30: 5 total, 4 up, 5 in 2024-09-10T18:20:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:20:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[29291]: from='osd.4 v1:172.21.15.204:6800/1619669076' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T18:20:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[29291]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T18:20:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:07.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:07.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:07.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[39220]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:07.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:07.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:07.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[39220]: osdmap e30: 5 total, 4 up, 5 in 2024-09-10T18:20:07.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:20:07.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:07.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[39220]: from='osd.4 v1:172.21.15.204:6800/1619669076' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T18:20:07.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[39220]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T18:20:07.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:07.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:07.258 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:06 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:07.832 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:20:07.936 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:07 smithi204 ceph-mon[32357]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-10T18:20:07.937 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:07 smithi204 ceph-mon[32357]: osdmap e31: 5 total, 4 up, 5 in 2024-09-10T18:20:07.937 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:07 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:20:07.937 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:07 smithi204 ceph-mon[32357]: from='osd.4 v1:172.21.15.204:6800/1619669076' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:07.937 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:07 smithi204 ceph-mon[32357]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:07.937 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:20:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:20:07.783+0000 7f9c66c0e640 -1 osd.4 0 waiting for initial osdmap 2024-09-10T18:20:07.937 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:20:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:20:07.788+0000 7f9c5fa41640 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T18:20:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:07 smithi028 ceph-mon[39220]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-10T18:20:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:07 smithi028 ceph-mon[39220]: osdmap e31: 5 total, 4 up, 5 in 2024-09-10T18:20:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:07 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:20:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:07 smithi028 ceph-mon[39220]: from='osd.4 v1:172.21.15.204:6800/1619669076' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:07 smithi028 ceph-mon[39220]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:08.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:07 smithi028 ceph-mon[29291]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-10T18:20:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:07 smithi028 ceph-mon[29291]: osdmap e31: 5 total, 4 up, 5 in 2024-09-10T18:20:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:07 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:20:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:07 smithi028 ceph-mon[29291]: from='osd.4 v1:172.21.15.204:6800/1619669076' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:07 smithi028 ceph-mon[29291]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:08.841 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:08 smithi204 ceph-mon[32357]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:08.842 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:08 smithi204 ceph-mon[32357]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-10T18:20:08.842 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:08 smithi204 ceph-mon[32357]: osdmap e32: 5 total, 4 up, 5 in 2024-09-10T18:20:08.842 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:08 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:20:08.842 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:08 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:20:09.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:08 smithi028 ceph-mon[29291]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:09.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:08 smithi028 ceph-mon[29291]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-10T18:20:09.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:08 smithi028 ceph-mon[29291]: osdmap e32: 5 total, 4 up, 5 in 2024-09-10T18:20:09.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:08 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:20:09.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:08 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:20:09.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:08 smithi028 ceph-mon[39220]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:08 smithi028 ceph-mon[39220]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-10T18:20:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:08 smithi028 ceph-mon[39220]: osdmap e32: 5 total, 4 up, 5 in 2024-09-10T18:20:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:08 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:20:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:08 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:20:09.948 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:09 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:20:09.949 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:09 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:20:09.949 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:09 smithi204 ceph-mon[32357]: osd.4 v1:172.21.15.204:6800/1619669076 boot 2024-09-10T18:20:09.949 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:09 smithi204 ceph-mon[32357]: osdmap e33: 5 total, 5 up, 5 in 2024-09-10T18:20:09.949 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:09 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:20:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:09 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:20:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:09 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:20:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:09 smithi028 ceph-mon[29291]: osd.4 v1:172.21.15.204:6800/1619669076 boot 2024-09-10T18:20:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:09 smithi028 ceph-mon[29291]: osdmap e33: 5 total, 5 up, 5 in 2024-09-10T18:20:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:09 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:20:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:09 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:20:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:09 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:20:10.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:09 smithi028 ceph-mon[39220]: osd.4 v1:172.21.15.204:6800/1619669076 boot 2024-09-10T18:20:10.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:09 smithi028 ceph-mon[39220]: osdmap e33: 5 total, 5 up, 5 in 2024-09-10T18:20:10.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:09 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:20:10.332 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-10T18:20:10.422 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-10T18:20:10.437 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-09-10T18:20:10.437 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-09-10T18:20:10.437 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126446 s, 829 MB/s 2024-09-10T18:20:10.437 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-09-10T18:20:11.062 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:10 smithi204 ceph-mon[32357]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:11.062 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:10 smithi204 ceph-mon[32357]: osdmap e34: 5 total, 5 up, 5 in 2024-09-10T18:20:11.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_3 2024-09-10T18:20:11.155 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:10 smithi028 ceph-mon[39220]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:11.156 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:10 smithi028 ceph-mon[39220]: osdmap e34: 5 total, 5 up, 5 in 2024-09-10T18:20:11.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:10 smithi028 ceph-mon[29291]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:20:11.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:10 smithi028 ceph-mon[29291]: osdmap e34: 5 total, 5 up, 5 in 2024-09-10T18:20:11.538 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:20:12.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:11 smithi204 ceph-mon[32357]: osdmap e35: 5 total, 5 up, 5 in 2024-09-10T18:20:12.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:11 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:12.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:11 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:12.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:11 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:12.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:11 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:12.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:11 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:12.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:11 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:12.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:11 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:12.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:11 smithi028 ceph-mon[29291]: osdmap e35: 5 total, 5 up, 5 in 2024-09-10T18:20:12.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:11 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:11 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:11 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:11 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:11 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:11 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:11 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:12.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:11 smithi028 ceph-mon[39220]: osdmap e35: 5 total, 5 up, 5 in 2024-09-10T18:20:12.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:11 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:12.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:11 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:12.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:11 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:12.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:11 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:12.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:11 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:12.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:11 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:12.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:11 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:12.923 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:12 smithi204 ceph-mon[32357]: Detected new or changed devices on smithi204 2024-09-10T18:20:12.923 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:12 smithi204 ceph-mon[32357]: Adjusting osd_memory_target on smithi204 to 17088M 2024-09-10T18:20:12.923 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:12 smithi204 ceph-mon[32357]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:13.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:12 smithi028 ceph-mon[29291]: Detected new or changed devices on smithi204 2024-09-10T18:20:13.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:12 smithi028 ceph-mon[29291]: Adjusting osd_memory_target on smithi204 to 17088M 2024-09-10T18:20:13.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:12 smithi028 ceph-mon[29291]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:13.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:12 smithi028 ceph-mon[39220]: Detected new or changed devices on smithi204 2024-09-10T18:20:13.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:12 smithi028 ceph-mon[39220]: Adjusting osd_memory_target on smithi204 to 17088M 2024-09-10T18:20:13.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:12 smithi028 ceph-mon[39220]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:14.030 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:13 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:20:14.030 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:13 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:20:14.030 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:13 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:14.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:13 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:20:14.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:13 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:20:14.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:13 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:13 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:20:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:13 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:20:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:13 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:14.887 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:14 smithi204 ceph-mon[32357]: from='client.24253 v1:172.21.15.204:0/2622319188' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:20:14.887 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:14 smithi204 ceph-mon[32357]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-09-10T18:20:15.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:14 smithi028 ceph-mon[29291]: from='client.24253 v1:172.21.15.204:0/2622319188' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:20:15.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:14 smithi028 ceph-mon[29291]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-09-10T18:20:15.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:14 smithi028 ceph-mon[39220]: from='client.24253 v1:172.21.15.204:0/2622319188' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:20:15.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:14 smithi028 ceph-mon[39220]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-09-10T18:20:16.067 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:15 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.204:0/530605509' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a92e0441-66c9-498e-a0ac-6465acdbb1f5"}]: dispatch 2024-09-10T18:20:16.067 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:15 smithi204 ceph-mon[32357]: from='client.24259 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a92e0441-66c9-498e-a0ac-6465acdbb1f5"}]: dispatch 2024-09-10T18:20:16.067 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:15 smithi204 ceph-mon[32357]: from='client.24259 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a92e0441-66c9-498e-a0ac-6465acdbb1f5"}]': finished 2024-09-10T18:20:16.067 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:15 smithi204 ceph-mon[32357]: osdmap e36: 6 total, 5 up, 6 in 2024-09-10T18:20:16.067 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:15 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:20:16.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:15 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.204:0/530605509' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a92e0441-66c9-498e-a0ac-6465acdbb1f5"}]: dispatch 2024-09-10T18:20:16.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:15 smithi028 ceph-mon[29291]: from='client.24259 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a92e0441-66c9-498e-a0ac-6465acdbb1f5"}]: dispatch 2024-09-10T18:20:16.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:15 smithi028 ceph-mon[29291]: from='client.24259 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a92e0441-66c9-498e-a0ac-6465acdbb1f5"}]': finished 2024-09-10T18:20:16.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:15 smithi028 ceph-mon[29291]: osdmap e36: 6 total, 5 up, 6 in 2024-09-10T18:20:16.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:15 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:20:16.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:15 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.204:0/530605509' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a92e0441-66c9-498e-a0ac-6465acdbb1f5"}]: dispatch 2024-09-10T18:20:16.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:15 smithi028 ceph-mon[39220]: from='client.24259 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a92e0441-66c9-498e-a0ac-6465acdbb1f5"}]: dispatch 2024-09-10T18:20:16.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:15 smithi028 ceph-mon[39220]: from='client.24259 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a92e0441-66c9-498e-a0ac-6465acdbb1f5"}]': finished 2024-09-10T18:20:16.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:15 smithi028 ceph-mon[39220]: osdmap e36: 6 total, 5 up, 6 in 2024-09-10T18:20:16.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:15 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:20:17.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:16 smithi204 ceph-mon[32357]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T18:20:17.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:16 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.204:0/3719983246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:20:17.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:16 smithi028 ceph-mon[29291]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T18:20:17.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:16 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.204:0/3719983246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:20:17.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:16 smithi028 ceph-mon[39220]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T18:20:17.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:16 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.204:0/3719983246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:20:18.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:18 smithi204 ceph-mon[32357]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-10T18:20:19.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:18 smithi028 ceph-mon[39220]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-10T18:20:19.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:18 smithi028 ceph-mon[29291]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-10T18:20:20.925 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:20 smithi204 ceph-mon[32357]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-10T18:20:21.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:20 smithi028 ceph-mon[29291]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-10T18:20:21.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:20 smithi028 ceph-mon[39220]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-10T18:20:22.073 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:21 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-10T18:20:22.073 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:21 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:22.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:21 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-10T18:20:22.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:21 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:22.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:21 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-10T18:20:22.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:21 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:23.101 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:22 smithi204 ceph-mon[32357]: Deploying daemon osd.5 on smithi204 2024-09-10T18:20:23.102 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:22 smithi204 ceph-mon[32357]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T18:20:23.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:22 smithi028 ceph-mon[29291]: Deploying daemon osd.5 on smithi204 2024-09-10T18:20:23.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:22 smithi028 ceph-mon[29291]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T18:20:23.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:22 smithi028 ceph-mon[39220]: Deploying daemon osd.5 on smithi204 2024-09-10T18:20:23.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:22 smithi028 ceph-mon[39220]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T18:20:25.100 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:24 smithi204 ceph-mon[32357]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:25.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:24 smithi028 ceph-mon[29291]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:25.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:24 smithi028 ceph-mon[39220]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:26 smithi204 ceph-mon[32357]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:26.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:26 smithi028 ceph-mon[29291]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:26.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:26 smithi028 ceph-mon[39220]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:27.531 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:27.531 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:27.531 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:27.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:27.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:27.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:27.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:27.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:27.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:28 smithi204 ceph-mon[32357]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:28.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:28 smithi028 ceph-mon[29291]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:28.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:28 smithi028 ceph-mon[39220]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:29.595 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 5 on host 'smithi204' 2024-09-10T18:20:29.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:29 smithi204 ceph-mon[32357]: from='osd.5 v1:172.21.15.204:6804/2717802880' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T18:20:29.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:29 smithi204 ceph-mon[32357]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T18:20:29.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:29 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:29.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:29 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:30.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:29 smithi028 ceph-mon[29291]: from='osd.5 v1:172.21.15.204:6804/2717802880' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T18:20:30.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:29 smithi028 ceph-mon[29291]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T18:20:30.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:29 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:30.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:29 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:30.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:29 smithi028 ceph-mon[39220]: from='osd.5 v1:172.21.15.204:6804/2717802880' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T18:20:30.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:29 smithi028 ceph-mon[39220]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T18:20:30.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:29 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:30.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:29 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:30.221 DEBUG:teuthology.orchestra.run.smithi204:osd.5> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.5.service 2024-09-10T18:20:30.223 INFO:tasks.cephadm:Deploying osd.6 on smithi204 with /dev/vg_nvme/lv_2... 2024-09-10T18:20:30.223 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-10T18:20:30.521 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:20:30.965 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:30 smithi204 ceph-mon[32357]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:30.965 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:30 smithi204 ceph-mon[32357]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-10T18:20:30.966 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:30 smithi204 ceph-mon[32357]: osdmap e37: 6 total, 5 up, 6 in 2024-09-10T18:20:30.966 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:20:30.966 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:30 smithi204 ceph-mon[32357]: from='osd.5 v1:172.21.15.204:6804/2717802880' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:30.966 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:30 smithi204 ceph-mon[32357]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:30.966 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:30.966 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:30 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:30.966 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:20:30 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:20:30.644+0000 7fb200b0a640 -1 osd.5 0 waiting for initial osdmap 2024-09-10T18:20:30.966 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:20:30 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:20:30.649+0000 7fb1fa93f640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T18:20:31.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:30 smithi028 ceph-mon[29291]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:31.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:30 smithi028 ceph-mon[29291]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-10T18:20:31.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:30 smithi028 ceph-mon[29291]: osdmap e37: 6 total, 5 up, 6 in 2024-09-10T18:20:31.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:20:31.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:30 smithi028 ceph-mon[29291]: from='osd.5 v1:172.21.15.204:6804/2717802880' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:31.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:30 smithi028 ceph-mon[29291]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:31.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:31.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:30 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:31.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:30 smithi028 ceph-mon[39220]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:31.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:30 smithi028 ceph-mon[39220]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-10T18:20:31.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:30 smithi028 ceph-mon[39220]: osdmap e37: 6 total, 5 up, 6 in 2024-09-10T18:20:31.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:20:31.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:30 smithi028 ceph-mon[39220]: from='osd.5 v1:172.21.15.204:6804/2717802880' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:31.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:30 smithi028 ceph-mon[39220]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:31.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:31.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:30 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:31.837 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:31 smithi204 ceph-mon[32357]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-10T18:20:31.837 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:31 smithi204 ceph-mon[32357]: osdmap e38: 6 total, 5 up, 6 in 2024-09-10T18:20:31.837 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:31 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:20:31.837 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:31 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:20:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:31 smithi028 ceph-mon[29291]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-10T18:20:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:31 smithi028 ceph-mon[29291]: osdmap e38: 6 total, 5 up, 6 in 2024-09-10T18:20:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:31 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:20:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:31 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:20:32.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:31 smithi028 ceph-mon[39220]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-10T18:20:32.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:31 smithi028 ceph-mon[39220]: osdmap e38: 6 total, 5 up, 6 in 2024-09-10T18:20:32.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:31 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:20:32.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:31 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:20:32.202 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-10T18:20:32.302 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-10T18:20:32.316 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-09-10T18:20:32.316 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-09-10T18:20:32.316 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123413 s, 850 MB/s 2024-09-10T18:20:32.316 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-09-10T18:20:33.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:32 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:20:33.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:32 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:20:33.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:32 smithi028 ceph-mon[29291]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:33.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:32 smithi028 ceph-mon[29291]: osd.5 v1:172.21.15.204:6804/2717802880 boot 2024-09-10T18:20:33.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:32 smithi028 ceph-mon[29291]: osdmap e39: 6 total, 6 up, 6 in 2024-09-10T18:20:33.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:32 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:20:33.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:32 smithi028 ceph-mon[29291]: osdmap e40: 6 total, 6 up, 6 in 2024-09-10T18:20:33.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:32 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:20:33.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:32 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:20:33.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:32 smithi028 ceph-mon[39220]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:33.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:32 smithi028 ceph-mon[39220]: osd.5 v1:172.21.15.204:6804/2717802880 boot 2024-09-10T18:20:33.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:32 smithi028 ceph-mon[39220]: osdmap e39: 6 total, 6 up, 6 in 2024-09-10T18:20:33.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:32 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:20:33.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:32 smithi028 ceph-mon[39220]: osdmap e40: 6 total, 6 up, 6 in 2024-09-10T18:20:33.011 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:32 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:20:33.012 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:32 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:20:33.012 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:32 smithi204 ceph-mon[32357]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:20:33.012 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:32 smithi204 ceph-mon[32357]: osd.5 v1:172.21.15.204:6804/2717802880 boot 2024-09-10T18:20:33.012 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:32 smithi204 ceph-mon[32357]: osdmap e39: 6 total, 6 up, 6 in 2024-09-10T18:20:33.012 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:32 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:20:33.012 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:32 smithi204 ceph-mon[32357]: osdmap e40: 6 total, 6 up, 6 in 2024-09-10T18:20:33.295 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_2 2024-09-10T18:20:34.152 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:20:34.808 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:34 smithi204 ceph-mon[32357]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:34.809 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:34 smithi204 ceph-mon[32357]: osdmap e41: 6 total, 6 up, 6 in 2024-09-10T18:20:34.809 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:34 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:20:34.809 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:34 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:20:35.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:34 smithi028 ceph-mon[29291]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:35.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:34 smithi028 ceph-mon[29291]: osdmap e41: 6 total, 6 up, 6 in 2024-09-10T18:20:35.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:34 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:20:35.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:34 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:20:35.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:34 smithi028 ceph-mon[39220]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:35.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:34 smithi028 ceph-mon[39220]: osdmap e41: 6 total, 6 up, 6 in 2024-09-10T18:20:35.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:34 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:20:35.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:34 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:20:35.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:35 smithi204 ceph-mon[32357]: osdmap e42: 6 total, 6 up, 6 in 2024-09-10T18:20:35.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:35.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:35.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:35.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:35.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:35.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:35.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:35.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:35.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:35.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:35.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:35.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:35 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:36.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[29291]: osdmap e42: 6 total, 6 up, 6 in 2024-09-10T18:20:36.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:36.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:36.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:36.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:36.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:36.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:36.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[39220]: osdmap e42: 6 total, 6 up, 6 in 2024-09-10T18:20:36.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:36.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:36.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:35 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:37.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:36 smithi028 ceph-mon[29291]: Detected new or changed devices on smithi204 2024-09-10T18:20:37.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:36 smithi028 ceph-mon[29291]: Adjusting osd_memory_target on smithi204 to 8544M 2024-09-10T18:20:37.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:36 smithi028 ceph-mon[29291]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:37.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:36 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:20:37.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:36 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:20:37.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:36 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:37.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:36 smithi028 ceph-mon[39220]: Detected new or changed devices on smithi204 2024-09-10T18:20:37.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:36 smithi028 ceph-mon[39220]: Adjusting osd_memory_target on smithi204 to 8544M 2024-09-10T18:20:37.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:36 smithi028 ceph-mon[39220]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:37.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:36 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:20:37.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:36 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:20:37.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:36 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:37.044 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:36 smithi204 ceph-mon[32357]: Detected new or changed devices on smithi204 2024-09-10T18:20:37.045 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:36 smithi204 ceph-mon[32357]: Adjusting osd_memory_target on smithi204 to 8544M 2024-09-10T18:20:37.045 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:36 smithi204 ceph-mon[32357]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:37.045 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:36 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:20:37.045 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:36 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:20:37.045 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:36 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:37.922 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:37 smithi204 ceph-mon[32357]: from='client.24280 v1:172.21.15.204:0/1644735513' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:20:38.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:37 smithi028 ceph-mon[29291]: from='client.24280 v1:172.21.15.204:0/1644735513' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:20:38.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:37 smithi028 ceph-mon[39220]: from='client.24280 v1:172.21.15.204:0/1644735513' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:20:38.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:38 smithi204 ceph-mon[32357]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:39.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:38 smithi028 ceph-mon[29291]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:39.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:38 smithi028 ceph-mon[39220]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:39.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:39 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.204:0/4016346782' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12b02875-6653-45ed-8d2b-b4d33bd88525"}]: dispatch 2024-09-10T18:20:39.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:39 smithi204 ceph-mon[32357]: from='client.24286 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12b02875-6653-45ed-8d2b-b4d33bd88525"}]: dispatch 2024-09-10T18:20:39.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:39 smithi204 ceph-mon[32357]: from='client.24286 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12b02875-6653-45ed-8d2b-b4d33bd88525"}]': finished 2024-09-10T18:20:39.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:39 smithi204 ceph-mon[32357]: osdmap e43: 7 total, 6 up, 7 in 2024-09-10T18:20:39.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:39 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:20:39.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:39 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.204:0/3636433164' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:20:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:39 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.204:0/4016346782' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12b02875-6653-45ed-8d2b-b4d33bd88525"}]: dispatch 2024-09-10T18:20:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:39 smithi028 ceph-mon[29291]: from='client.24286 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12b02875-6653-45ed-8d2b-b4d33bd88525"}]: dispatch 2024-09-10T18:20:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:39 smithi028 ceph-mon[29291]: from='client.24286 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12b02875-6653-45ed-8d2b-b4d33bd88525"}]': finished 2024-09-10T18:20:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:39 smithi028 ceph-mon[29291]: osdmap e43: 7 total, 6 up, 7 in 2024-09-10T18:20:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:39 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:20:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:39 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.204:0/3636433164' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:20:40.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:39 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.204:0/4016346782' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12b02875-6653-45ed-8d2b-b4d33bd88525"}]: dispatch 2024-09-10T18:20:40.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:39 smithi028 ceph-mon[39220]: from='client.24286 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12b02875-6653-45ed-8d2b-b4d33bd88525"}]: dispatch 2024-09-10T18:20:40.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:39 smithi028 ceph-mon[39220]: from='client.24286 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12b02875-6653-45ed-8d2b-b4d33bd88525"}]': finished 2024-09-10T18:20:40.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:39 smithi028 ceph-mon[39220]: osdmap e43: 7 total, 6 up, 7 in 2024-09-10T18:20:40.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:39 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:20:40.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:39 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.204:0/3636433164' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:20:41.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:40 smithi028 ceph-mon[29291]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:40 smithi028 ceph-mon[39220]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:40 smithi204 ceph-mon[32357]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:43.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:42 smithi028 ceph-mon[39220]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:43.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:42 smithi028 ceph-mon[29291]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:43.270 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:42 smithi204 ceph-mon[32357]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:45.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:44 smithi204 ceph-mon[32357]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:45.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:44 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-10T18:20:45.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:44 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:45.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:44 smithi028 ceph-mon[39220]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:45.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:44 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-10T18:20:45.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:44 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:44 smithi028 ceph-mon[29291]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:44 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-10T18:20:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:44 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:46.164 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:45 smithi204 ceph-mon[32357]: Deploying daemon osd.6 on smithi204 2024-09-10T18:20:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:45 smithi028 ceph-mon[29291]: Deploying daemon osd.6 on smithi204 2024-09-10T18:20:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:45 smithi028 ceph-mon[39220]: Deploying daemon osd.6 on smithi204 2024-09-10T18:20:47.164 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:46 smithi204 ceph-mon[32357]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:47.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:46 smithi028 ceph-mon[39220]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:47.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:46 smithi028 ceph-mon[29291]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:49.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:48 smithi204 ceph-mon[32357]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:49.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:48 smithi028 ceph-mon[29291]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:49.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:48 smithi028 ceph-mon[39220]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:50.135 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:49 smithi204 ceph-mon[32357]: 1.0 deep-scrub starts 2024-09-10T18:20:50.135 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:49 smithi204 ceph-mon[32357]: 1.0 deep-scrub ok 2024-09-10T18:20:50.135 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:49 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:50.136 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:49 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:50.136 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:49 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:49 smithi028 ceph-mon[39220]: 1.0 deep-scrub starts 2024-09-10T18:20:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:49 smithi028 ceph-mon[39220]: 1.0 deep-scrub ok 2024-09-10T18:20:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:49 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:49 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:49 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:50.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:49 smithi028 ceph-mon[29291]: 1.0 deep-scrub starts 2024-09-10T18:20:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:49 smithi028 ceph-mon[29291]: 1.0 deep-scrub ok 2024-09-10T18:20:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:49 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:49 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:49 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:51.042 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:50 smithi204 ceph-mon[32357]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:51.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:50 smithi028 ceph-mon[29291]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:51.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:50 smithi028 ceph-mon[39220]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:53.039 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 6 on host 'smithi204' 2024-09-10T18:20:53.040 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:52 smithi204 ceph-mon[32357]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:53.040 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:52 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:53.040 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:52 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:53.040 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:52 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:53.040 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:52 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:53.040 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:52 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:53.040 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:52 smithi204 ceph-mon[32357]: from='osd.6 v1:172.21.15.204:6808/3377682607' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:20:53.041 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:52 smithi204 ceph-mon[32357]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:20:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:52 smithi028 ceph-mon[29291]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:52 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:52 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:52 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:52 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:52 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:52 smithi028 ceph-mon[29291]: from='osd.6 v1:172.21.15.204:6808/3377682607' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:20:53.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:52 smithi028 ceph-mon[29291]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:20:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:52 smithi028 ceph-mon[39220]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:52 smithi028 ceph-mon[39220]: from='osd.6 v1:172.21.15.204:6808/3377682607' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:20:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:52 smithi028 ceph-mon[39220]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:20:53.713 DEBUG:teuthology.orchestra.run.smithi204:osd.6> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.6.service 2024-09-10T18:20:53.716 INFO:tasks.cephadm:Deploying osd.7 on smithi204 with /dev/vg_nvme/lv_1... 2024-09-10T18:20:53.716 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-10T18:20:53.963 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:53 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:53.963 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:53 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:53.963 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:53 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:53.963 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:53 smithi204 ceph-mon[32357]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-10T18:20:53.963 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:53 smithi204 ceph-mon[32357]: osdmap e44: 7 total, 6 up, 7 in 2024-09-10T18:20:53.963 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:53 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:20:53.963 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:53 smithi204 ceph-mon[32357]: from='osd.6 v1:172.21.15.204:6808/3377682607' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:53.963 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:53 smithi204 ceph-mon[32357]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:53 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:53 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:53 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:53 smithi028 ceph-mon[29291]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-10T18:20:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:53 smithi028 ceph-mon[29291]: osdmap e44: 7 total, 6 up, 7 in 2024-09-10T18:20:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:53 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:20:54.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:53 smithi028 ceph-mon[29291]: from='osd.6 v1:172.21.15.204:6808/3377682607' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:54.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:53 smithi028 ceph-mon[29291]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:53 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:20:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:53 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:53 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:54.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:53 smithi028 ceph-mon[39220]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-10T18:20:54.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:53 smithi028 ceph-mon[39220]: osdmap e44: 7 total, 6 up, 7 in 2024-09-10T18:20:54.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:53 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:20:54.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:53 smithi028 ceph-mon[39220]: from='osd.6 v1:172.21.15.204:6808/3377682607' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:54.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:53 smithi028 ceph-mon[39220]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:20:54.349 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:20:54.738 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:20:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[48595]: 2024-09-10T18:20:54.450+0000 7f2a77783640 -1 osd.6 0 waiting for initial osdmap 2024-09-10T18:20:54.739 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:20:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[48595]: 2024-09-10T18:20:54.456+0000 7f2a735bc640 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T18:20:54.999 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:54 smithi204 ceph-mon[32357]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:54.999 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:54 smithi204 ceph-mon[32357]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-10T18:20:54.999 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:54 smithi204 ceph-mon[32357]: osdmap e45: 7 total, 6 up, 7 in 2024-09-10T18:20:54.999 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:54 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:20:54.999 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:54 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:20:55.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:54 smithi028 ceph-mon[29291]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:55.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:54 smithi028 ceph-mon[29291]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-10T18:20:55.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:54 smithi028 ceph-mon[29291]: osdmap e45: 7 total, 6 up, 7 in 2024-09-10T18:20:55.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:54 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:20:55.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:54 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:20:55.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:54 smithi028 ceph-mon[39220]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:20:55.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:54 smithi028 ceph-mon[39220]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-10T18:20:55.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:54 smithi028 ceph-mon[39220]: osdmap e45: 7 total, 6 up, 7 in 2024-09-10T18:20:55.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:54 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:20:55.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:54 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:20:56.471 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-10T18:20:56.556 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-10T18:20:56.573 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-09-10T18:20:56.573 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-09-10T18:20:56.573 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0146736 s, 715 MB/s 2024-09-10T18:20:56.573 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-09-10T18:20:56.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:56 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:20:56.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:56 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:20:56.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:56 smithi028 ceph-mon[29291]: osd.6 v1:172.21.15.204:6808/3377682607 boot 2024-09-10T18:20:56.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:56 smithi028 ceph-mon[29291]: osdmap e46: 7 total, 7 up, 7 in 2024-09-10T18:20:56.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:56 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:20:56.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:56 smithi028 ceph-mon[29291]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:20:56.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:56 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:20:56.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:56 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:20:56.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:56 smithi028 ceph-mon[39220]: osd.6 v1:172.21.15.204:6808/3377682607 boot 2024-09-10T18:20:56.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:56 smithi028 ceph-mon[39220]: osdmap e46: 7 total, 7 up, 7 in 2024-09-10T18:20:56.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:56 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:20:56.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:56 smithi028 ceph-mon[39220]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:20:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:56 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:20:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:56 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:20:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:56 smithi204 ceph-mon[32357]: osd.6 v1:172.21.15.204:6808/3377682607 boot 2024-09-10T18:20:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:56 smithi204 ceph-mon[32357]: osdmap e46: 7 total, 7 up, 7 in 2024-09-10T18:20:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:56 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:20:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:56 smithi204 ceph-mon[32357]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:20:57.430 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_1 2024-09-10T18:20:57.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:57 smithi028 ceph-mon[29291]: osdmap e47: 7 total, 7 up, 7 in 2024-09-10T18:20:57.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:57 smithi028 ceph-mon[39220]: osdmap e47: 7 total, 7 up, 7 in 2024-09-10T18:20:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:57 smithi204 ceph-mon[32357]: osdmap e47: 7 total, 7 up, 7 in 2024-09-10T18:20:58.189 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:20:58.571 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:58 smithi204 ceph-mon[32357]: osdmap e48: 7 total, 7 up, 7 in 2024-09-10T18:20:58.571 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:58 smithi204 ceph-mon[32357]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:20:58.571 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:58.571 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:58.571 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:58.571 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:58.571 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:58.571 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:58.571 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:58.571 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:58.571 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:58 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:58.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[29291]: osdmap e48: 7 total, 7 up, 7 in 2024-09-10T18:20:58.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[29291]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:20:58.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:58.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:58.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:58.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:58.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:58.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:58.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:58.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:58.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:58.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[39220]: osdmap e48: 7 total, 7 up, 7 in 2024-09-10T18:20:58.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[39220]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:20:58.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:58.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:58.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:58.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:58.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:20:58.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:58.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:20:58.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:20:58.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:58 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:20:59.657 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:59 smithi204 ceph-mon[32357]: Detected new or changed devices on smithi204 2024-09-10T18:20:59.657 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:59 smithi204 ceph-mon[32357]: Adjusting osd_memory_target on smithi204 to 5696M 2024-09-10T18:20:59.657 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:59 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:20:59.657 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:59 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:20:59.658 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:20:59 smithi204 ceph-mon[32357]: osdmap e49: 7 total, 7 up, 7 in 2024-09-10T18:20:59.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:59 smithi028 ceph-mon[29291]: Detected new or changed devices on smithi204 2024-09-10T18:20:59.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:59 smithi028 ceph-mon[29291]: Adjusting osd_memory_target on smithi204 to 5696M 2024-09-10T18:20:59.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:59 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:20:59.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:59 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:20:59.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:20:59 smithi028 ceph-mon[29291]: osdmap e49: 7 total, 7 up, 7 in 2024-09-10T18:20:59.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:59 smithi028 ceph-mon[39220]: Detected new or changed devices on smithi204 2024-09-10T18:20:59.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:59 smithi028 ceph-mon[39220]: Adjusting osd_memory_target on smithi204 to 5696M 2024-09-10T18:20:59.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:59 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:20:59.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:59 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:20:59.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:20:59 smithi028 ceph-mon[39220]: osdmap e49: 7 total, 7 up, 7 in 2024-09-10T18:21:00.573 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:00 smithi204 ceph-mon[32357]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:00.573 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:00 smithi204 ceph-mon[32357]: from='client.24307 v1:172.21.15.204:0/3638253495' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:00.573 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:00 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:21:00.573 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:00 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:21:00.573 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:00 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:00.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:00 smithi028 ceph-mon[29291]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:00.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:00 smithi028 ceph-mon[29291]: from='client.24307 v1:172.21.15.204:0/3638253495' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:00.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:00 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:21:00.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:00 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:21:00.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:00 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:00.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:00 smithi028 ceph-mon[39220]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:00.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:00 smithi028 ceph-mon[39220]: from='client.24307 v1:172.21.15.204:0/3638253495' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:00.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:00 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:21:00.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:00 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:21:00.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:00 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:03.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:02 smithi028 ceph-mon[29291]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T18:21:03.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:02 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.204:0/2299983537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c4f7aa9-ff95-4192-a146-ca576eee907f"}]: dispatch 2024-09-10T18:21:03.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:02 smithi028 ceph-mon[29291]: from='client.24313 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c4f7aa9-ff95-4192-a146-ca576eee907f"}]: dispatch 2024-09-10T18:21:03.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:02 smithi028 ceph-mon[29291]: from='client.24313 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c4f7aa9-ff95-4192-a146-ca576eee907f"}]': finished 2024-09-10T18:21:03.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:02 smithi028 ceph-mon[29291]: osdmap e50: 8 total, 7 up, 8 in 2024-09-10T18:21:03.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:02 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:03.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:02 smithi028 ceph-mon[39220]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T18:21:03.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:02 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.204:0/2299983537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c4f7aa9-ff95-4192-a146-ca576eee907f"}]: dispatch 2024-09-10T18:21:03.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:02 smithi028 ceph-mon[39220]: from='client.24313 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c4f7aa9-ff95-4192-a146-ca576eee907f"}]: dispatch 2024-09-10T18:21:03.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:02 smithi028 ceph-mon[39220]: from='client.24313 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c4f7aa9-ff95-4192-a146-ca576eee907f"}]': finished 2024-09-10T18:21:03.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:02 smithi028 ceph-mon[39220]: osdmap e50: 8 total, 7 up, 8 in 2024-09-10T18:21:03.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:02 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:02 smithi204 ceph-mon[32357]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T18:21:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:02 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.204:0/2299983537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c4f7aa9-ff95-4192-a146-ca576eee907f"}]: dispatch 2024-09-10T18:21:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:02 smithi204 ceph-mon[32357]: from='client.24313 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c4f7aa9-ff95-4192-a146-ca576eee907f"}]: dispatch 2024-09-10T18:21:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:02 smithi204 ceph-mon[32357]: from='client.24313 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c4f7aa9-ff95-4192-a146-ca576eee907f"}]': finished 2024-09-10T18:21:03.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:02 smithi204 ceph-mon[32357]: osdmap e50: 8 total, 7 up, 8 in 2024-09-10T18:21:03.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:02 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:04.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:03 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.204:0/3818307659' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:21:04.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:03 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.204:0/3818307659' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:21:04.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:03 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.204:0/3818307659' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:21:04.895 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:04 smithi204 ceph-mon[32357]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-09-10T18:21:05.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:04 smithi028 ceph-mon[29291]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-09-10T18:21:05.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:04 smithi028 ceph-mon[39220]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-09-10T18:21:05.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:05 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:21:05.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:05 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:21:05.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:05 smithi204 ceph-mon[32357]: osdmap e51: 8 total, 7 up, 8 in 2024-09-10T18:21:05.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:05 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:06.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:05 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:21:06.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:05 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:21:06.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:05 smithi028 ceph-mon[29291]: osdmap e51: 8 total, 7 up, 8 in 2024-09-10T18:21:06.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:05 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:06.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:05 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:21:06.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:05 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:21:06.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:05 smithi028 ceph-mon[39220]: osdmap e51: 8 total, 7 up, 8 in 2024-09-10T18:21:06.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:05 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:06.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:06 smithi204 ceph-mon[32357]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-09-10T18:21:07.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:06 smithi028 ceph-mon[29291]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-09-10T18:21:07.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:06 smithi028 ceph-mon[39220]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-09-10T18:21:08.793 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:08 smithi204 ceph-mon[32357]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T18:21:08.794 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:08 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-10T18:21:08.794 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:08 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:09.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:08 smithi028 ceph-mon[29291]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T18:21:09.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:08 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-10T18:21:09.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:08 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:09.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:08 smithi028 ceph-mon[39220]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T18:21:09.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:08 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-10T18:21:09.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:08 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:09.697 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:09 smithi204 ceph-mon[32357]: Deploying daemon osd.7 on smithi204 2024-09-10T18:21:10.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:09 smithi028 ceph-mon[29291]: Deploying daemon osd.7 on smithi204 2024-09-10T18:21:10.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:09 smithi028 ceph-mon[39220]: Deploying daemon osd.7 on smithi204 2024-09-10T18:21:10.848 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:10 smithi204 ceph-mon[32357]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:11.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:10 smithi028 ceph-mon[29291]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:11.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:10 smithi028 ceph-mon[39220]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:13.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:12 smithi028 ceph-mon[29291]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:13.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:12 smithi028 ceph-mon[39220]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:13.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:12 smithi204 ceph-mon[32357]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:14.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:13 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:21:14.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:13 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:14.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:13 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:14.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:13 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:21:14.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:13 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:14.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:13 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:14.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:13 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:21:14.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:13 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:14.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:13 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:15.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:14 smithi204 ceph-mon[32357]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:15.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:14 smithi028 ceph-mon[29291]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:15.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:14 smithi028 ceph-mon[39220]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:16.093 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 7 on host 'smithi204' 2024-09-10T18:21:16.670 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:16 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:16.671 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:16 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:16.671 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:16 smithi204 ceph-mon[32357]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:16.671 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:16 smithi204 ceph-mon[32357]: from='osd.7 v1:172.21.15.204:6812/3511791879' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T18:21:16.671 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:16 smithi204 ceph-mon[32357]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T18:21:16.671 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:16 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:16.671 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:16 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:16.698 DEBUG:teuthology.orchestra.run.smithi204:osd.7> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.7.service 2024-09-10T18:21:16.700 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-10T18:21:16.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd stat -f json 2024-09-10T18:21:16.967 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:21:17.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:16 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:17.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:16 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:17.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:16 smithi028 ceph-mon[39220]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:17.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:16 smithi028 ceph-mon[39220]: from='osd.7 v1:172.21.15.204:6812/3511791879' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T18:21:17.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:16 smithi028 ceph-mon[39220]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T18:21:17.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:16 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:17.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:16 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:17.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:16 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:17.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:16 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:17.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:16 smithi028 ceph-mon[29291]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:17.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:16 smithi028 ceph-mon[29291]: from='osd.7 v1:172.21.15.204:6812/3511791879' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T18:21:17.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:16 smithi028 ceph-mon[29291]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T18:21:17.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:16 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:17.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:16 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:17.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:17 smithi028 ceph-mon[29291]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-10T18:21:17.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:17 smithi028 ceph-mon[29291]: osdmap e52: 8 total, 7 up, 8 in 2024-09-10T18:21:17.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:17 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:17.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:17 smithi028 ceph-mon[29291]: from='osd.7 v1:172.21.15.204:6812/3511791879' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:21:17.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:17 smithi028 ceph-mon[29291]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:21:17.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:17 smithi028 ceph-mon[39220]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-10T18:21:17.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:17 smithi028 ceph-mon[39220]: osdmap e52: 8 total, 7 up, 8 in 2024-09-10T18:21:17.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:17 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:17.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:17 smithi028 ceph-mon[39220]: from='osd.7 v1:172.21.15.204:6812/3511791879' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:21:17.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:17 smithi028 ceph-mon[39220]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:21:17.810 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:17 smithi204 ceph-mon[32357]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-10T18:21:17.810 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:17 smithi204 ceph-mon[32357]: osdmap e52: 8 total, 7 up, 8 in 2024-09-10T18:21:17.810 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:17 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:17.810 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:17 smithi204 ceph-mon[32357]: from='osd.7 v1:172.21.15.204:6812/3511791879' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:21:17.810 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:17 smithi204 ceph-mon[32357]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:21:17.811 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:21:17 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:21:17.556+0000 7f484a6b9640 -1 osd.7 0 waiting for initial osdmap 2024-09-10T18:21:17.811 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:21:17 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:21:17.562+0000 7f48434ec640 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T18:21:18.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:18 smithi028 ceph-mon[29291]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-10T18:21:18.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:18 smithi028 ceph-mon[29291]: osdmap e53: 8 total, 7 up, 8 in 2024-09-10T18:21:18.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:18 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:18.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:18 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:18.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:18 smithi028 ceph-mon[29291]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:18.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:18 smithi028 ceph-mon[39220]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-10T18:21:18.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:18 smithi028 ceph-mon[39220]: osdmap e53: 8 total, 7 up, 8 in 2024-09-10T18:21:18.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:18 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:18.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:18 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:18.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:18 smithi028 ceph-mon[39220]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:18.764 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:21:18.822 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:18 smithi204 ceph-mon[32357]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-10T18:21:18.822 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:18 smithi204 ceph-mon[32357]: osdmap e53: 8 total, 7 up, 8 in 2024-09-10T18:21:18.822 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:18 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:18.822 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:18 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:18.822 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:18 smithi204 ceph-mon[32357]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:21:19.465 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1725992478,"num_in_osds":8,"osd_in_since":1725992462,"num_remapped_pgs":1} 2024-09-10T18:21:19.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T18:21:19.744 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:21:19.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:21:19.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:21:19.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: osd.7 v1:172.21.15.204:6812/3511791879 boot 2024-09-10T18:21:19.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: osdmap e54: 8 total, 8 up, 8 in 2024-09-10T18:21:19.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:19.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3373334010' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T18:21:19.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:19.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:19.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:21:19.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:21:19.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:21:19.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:21:19.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:19.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:19.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:21:19.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:19.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:21:19.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:19.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:21:19.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:21:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:21:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: osd.7 v1:172.21.15.204:6812/3511791879 boot 2024-09-10T18:21:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: osdmap e54: 8 total, 8 up, 8 in 2024-09-10T18:21:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3373334010' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T18:21:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:21:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:21:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:21:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:21:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:19.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:19.748 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:21:19.748 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:19.748 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:21:19.748 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:19.748 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:21:19.748 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:19 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:19.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:21:19.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:21:19.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: osd.7 v1:172.21.15.204:6812/3511791879 boot 2024-09-10T18:21:19.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: osdmap e54: 8 total, 8 up, 8 in 2024-09-10T18:21:19.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:21:19.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3373334010' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T18:21:19.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:19.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:19.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:21:19.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:21:19.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:21:19.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:21:19.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:19.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:19.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:21:19.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:19.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:21:19.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:19.856 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:21:19.857 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:19 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:20.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:20 smithi204 ceph-mon[32357]: Detected new or changed devices on smithi204 2024-09-10T18:21:20.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:20 smithi204 ceph-mon[32357]: Adjusting osd_memory_target on smithi204 to 4272M 2024-09-10T18:21:20.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:20 smithi204 ceph-mon[32357]: osdmap e55: 8 total, 8 up, 8 in 2024-09-10T18:21:20.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:20 smithi204 ceph-mon[32357]: pgmap v171: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:21:20.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:20 smithi028 ceph-mon[29291]: Detected new or changed devices on smithi204 2024-09-10T18:21:20.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:20 smithi028 ceph-mon[29291]: Adjusting osd_memory_target on smithi204 to 4272M 2024-09-10T18:21:20.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:20 smithi028 ceph-mon[29291]: osdmap e55: 8 total, 8 up, 8 in 2024-09-10T18:21:20.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:20 smithi028 ceph-mon[29291]: pgmap v171: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:21:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:20 smithi028 ceph-mon[39220]: Detected new or changed devices on smithi204 2024-09-10T18:21:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:20 smithi028 ceph-mon[39220]: Adjusting osd_memory_target on smithi204 to 4272M 2024-09-10T18:21:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:20 smithi028 ceph-mon[39220]: osdmap e55: 8 total, 8 up, 8 in 2024-09-10T18:21:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:20 smithi028 ceph-mon[39220]: pgmap v171: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:21:21.523 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:21:21.523 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":56,"fsid":"5eababb2-6fa0-11ef-bcea-c7b262605968","created":"2024-09-10T18:16:20.680373+0000","modified":"2024-09-10T18:21:20.564756+0000","last_up_change":"2024-09-10T18:21:18.554378+0000","last_in_change":"2024-09-10T18:21:02.114621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T18:19:24.013536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f1c834d9-252a-4077-b25e-169181d70251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6801","nonce":3188372351}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6802","nonce":3188372351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6804","nonce":3188372351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6803","nonce":3188372351}]},"public_addr":"172.21.15.28:6801/3188372351","cluster_addr":"172.21.15.28:6802/3188372351","heartbeat_back_addr":"172.21.15.28:6804/3188372351","heartbeat_front_addr":"172.21.15.28:6803/3188372351","state":["exists","up"]},{"osd":1,"uuid":"81f1855d-b946-4ca3-b246-38977b9e20fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6805","nonce":1369912678}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6806","nonce":1369912678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6808","nonce":1369912678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6807","nonce":1369912678}]},"public_addr":"172.21.15.28:6805/1369912678","cluster_addr":"172.21.15.28:6806/1369912678","heartbeat_back_addr":"172.21.15.28:6808/1369912678","heartbeat_front_addr":"172.21.15.28:6807/1369912678","state":["exists","up"]},{"osd":2,"uuid":"2ad55d2a-36fa-4756-bf22-58d4b049486e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6809","nonce":1243406003}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6810","nonce":1243406003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6812","nonce":1243406003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6811","nonce":1243406003}]},"public_addr":"172.21.15.28:6809/1243406003","cluster_addr":"172.21.15.28:6810/1243406003","heartbeat_back_addr":"172.21.15.28:6812/1243406003","heartbeat_front_addr":"172.21.15.28:6811/1243406003","state":["exists","up"]},{"osd":3,"uuid":"242729a5-a5e8-44d5-97c8-14d14e970bff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6813","nonce":67703283}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6814","nonce":67703283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6816","nonce":67703283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6815","nonce":67703283}]},"public_addr":"172.21.15.28:6813/67703283","cluster_addr":"172.21.15.28:6814/67703283","heartbeat_back_addr":"172.21.15.28:6816/67703283","heartbeat_front_addr":"172.21.15.28:6815/67703283","state":["exists","up"]},{"osd":4,"uuid":"6e38b8ef-e6ed-493f-8eca-755354754787","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6800","nonce":1619669076}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6801","nonce":1619669076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6803","nonce":1619669076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6802","nonce":1619669076}]},"public_addr":"172.21.15.204:6800/1619669076","cluster_addr":"172.21.15.204:6801/1619669076","heartbeat_back_addr":"172.21.15.204:6803/1619669076","heartbeat_front_addr":"172.21.15.204:6802/1619669076","state":["exists","up"]},{"osd":5,"uuid":"a92e0441-66c9-498e-a0ac-6465acdbb1f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6804","nonce":2717802880}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6805","nonce":2717802880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6807","nonce":2717802880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6806","nonce":2717802880}]},"public_addr":"172.21.15.204:6804/2717802880","cluster_addr":"172.21.15.204:6805/2717802880","heartbeat_back_addr":"172.21.15.204:6807/2717802880","heartbeat_front_addr":"172.21.15.204:6806/2717802880","state":["exists","up"]},{"osd":6,"uuid":"12b02875-6653-45ed-8d2b-b4d33bd88525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6808","nonce":3377682607}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6809","nonce":3377682607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6811","nonce":3377682607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6810","nonce":3377682607}]},"public_addr":"172.21.15.204:6808/3377682607","cluster_addr":"172.21.15.204:6809/3377682607","heartbeat_back_addr":"172.21.15.204:6811/3377682607","heartbeat_front_addr":"172.21.15.204:6810/3377682607","state":["exists","up"]},{"osd":7,"uuid":"0c4f7aa9-ff95-4192-a146-ca576eee907f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6812","nonce":3511791879}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6813","nonce":3511791879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6815","nonce":3511791879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6814","nonce":3511791879}]},"public_addr":"172.21.15.204:6812/3511791879","cluster_addr":"172.21.15.204:6813/3511791879","heartbeat_back_addr":"172.21.15.204:6815/3511791879","heartbeat_front_addr":"172.21.15.204:6814/3511791879","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:20:58.436749+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:20:04.829886+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:20:33.992966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:21:04.315918+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:20:07.196588+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:20:30.586420+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:20:53.767012+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:21:17.055966+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/2564333506":"2024-09-11T18:17:07.521165+0000","172.21.15.28:0/2874160765":"2024-09-11T18:17:07.521165+0000","172.21.15.28:6800/3310949126":"2024-09-11T18:17:07.521165+0000","172.21.15.28:0/1746797014":"2024-09-11T18:16:45.197204+0000","172.21.15.28:0/421194352":"2024-09-11T18:17:07.521165+0000","172.21.15.28:0/1029045895":"2024-09-11T18:16:45.197204+0000","172.21.15.28:6800/3760775432":"2024-09-11T18:16:45.197204+0000","172.21.15.28:0/3866220552":"2024-09-11T18:16:45.197204+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T18:21:21.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:21 smithi028 ceph-mon[29291]: osdmap e56: 8 total, 8 up, 8 in 2024-09-10T18:21:21.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:21 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2055450931' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:21:21.818 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:21 smithi028 ceph-mon[39220]: osdmap e56: 8 total, 8 up, 8 in 2024-09-10T18:21:21.818 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:21 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2055450931' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:21:21.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:21 smithi204 ceph-mon[32357]: osdmap e56: 8 total, 8 up, 8 in 2024-09-10T18:21:21.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:21 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2055450931' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:21:22.159 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T18:19:24.013536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-10T18:21:22.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-10T18:21:22.417 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:21:22.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:22 smithi204 ceph-mon[32357]: pgmap v173: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:21:22.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:22 smithi028 ceph-mon[29291]: pgmap v173: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:21:22.863 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:22 smithi028 ceph-mon[39220]: pgmap v173: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:21:24.252 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-10T18:21:24.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:24 smithi028 ceph-mon[29291]: pgmap v174: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:21:24.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:24 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3569802924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-10T18:21:24.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:24 smithi028 ceph-mon[39220]: pgmap v174: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:21:24.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:24 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3569802924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-10T18:21:24.968 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi028 2024-09-10T18:21:24.968 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi028=foo.a' 2024-09-10T18:21:24.998 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:24 smithi204 ceph-mon[32357]: pgmap v174: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:21:24.998 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:24 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3569802924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-10T18:21:25.225 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:21:26.546 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:26 smithi204 ceph-mon[32357]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-10T18:21:26.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:26 smithi028 ceph-mon[29291]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-10T18:21:26.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:26 smithi028 ceph-mon[39220]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-10T18:21:26.950 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled rgw.foo.a update... 2024-09-10T18:21:27.572 DEBUG:teuthology.orchestra.run.smithi028:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@rgw.foo.a.service 2024-09-10T18:21:27.574 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi204 2024-09-10T18:21:27.574 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-09-10T18:21:27.855 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:21:28.209 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:27 smithi204 ceph-mon[32357]: from='client.24352 v1:172.21.15.204:0/3788298928' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi028=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:28.209 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:27 smithi204 ceph-mon[32357]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-09-10T18:21:28.209 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:28.209 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:21:28.209 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:28.210 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:21:28.210 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:28.210 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-10T18:21:28.210 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-10T18:21:28.210 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:28.210 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:27 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[29291]: from='client.24352 v1:172.21.15.204:0/3788298928' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi028=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[29291]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-09-10T18:21:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:21:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:21:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-10T18:21:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-10T18:21:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:28.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[39220]: from='client.24352 v1:172.21.15.204:0/3788298928' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi028=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:28.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[39220]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-09-10T18:21:28.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:28.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:21:28.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:28.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:21:28.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:28.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-10T18:21:28.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-10T18:21:28.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:28.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:27 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:29.081 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:28 smithi028 ceph-mon[29291]: Deploying daemon rgw.foo.a on smithi028 2024-09-10T18:21:29.082 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:28 smithi028 ceph-mon[29291]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T18:21:29.082 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:28 smithi028 ceph-mon[39220]: Deploying daemon rgw.foo.a on smithi028 2024-09-10T18:21:29.082 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:28 smithi028 ceph-mon[39220]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T18:21:29.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:28 smithi204 ceph-mon[32357]: Deploying daemon rgw.foo.a on smithi028 2024-09-10T18:21:29.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:28 smithi204 ceph-mon[32357]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T18:21:29.487 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Sep 10 18:21:29 smithi028 systemd[1]: Starting Ceph rgw.foo.a for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:21:29.754 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Sep 10 18:21:29 smithi028 podman[65703]: 2024-09-10 18:21:29.38832607 +0000 UTC m=+0.016113046 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T18:21:29.755 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Sep 10 18:21:29 smithi028 podman[65703]: 2024-09-10 18:21:29.490019303 +0000 UTC m=+0.117806325 container create cb342bdaacb4a7cda1a920996d21595af0d5ace824fa182730b397aa2c89c5a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-rgw-foo-a, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-10T18:21:29.755 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Sep 10 18:21:29 smithi028 podman[65703]: 2024-09-10 18:21:29.717878683 +0000 UTC m=+0.345665841 container init cb342bdaacb4a7cda1a920996d21595af0d5ace824fa182730b397aa2c89c5a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-rgw-foo-a, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T18:21:29.755 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Sep 10 18:21:29 smithi028 podman[65703]: 2024-09-10 18:21:29.721071654 +0000 UTC m=+0.348858619 container start cb342bdaacb4a7cda1a920996d21595af0d5ace824fa182730b397aa2c89c5a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-rgw-foo-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux ) 2024-09-10T18:21:30.254 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Sep 10 18:21:29 smithi028 bash[65703]: cb342bdaacb4a7cda1a920996d21595af0d5ace824fa182730b397aa2c89c5a2 2024-09-10T18:21:30.254 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Sep 10 18:21:29 smithi028 systemd[1]: Started Ceph rgw.foo.a for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:21:30.519 INFO:teuthology.orchestra.run.smithi204.stderr:pool 'datapool' created 2024-09-10T18:21:30.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:30 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.204:0/1787703232' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T18:21:30.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:30 smithi028 ceph-mon[29291]: from='client.24358 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T18:21:30.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:30 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.204:0/1787703232' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T18:21:30.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:30 smithi028 ceph-mon[39220]: from='client.24358 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T18:21:30.828 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:30 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.204:0/1787703232' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T18:21:30.828 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:30 smithi204 ceph-mon[32357]: from='client.24358 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T18:21:31.146 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- rbd pool init datapool 2024-09-10T18:21:31.390 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:21:31.799 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:31 smithi204 ceph-mon[32357]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T18:21:31.799 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:31 smithi204 ceph-mon[32357]: from='client.24358 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-10T18:21:31.799 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:31 smithi204 ceph-mon[32357]: osdmap e57: 8 total, 8 up, 8 in 2024-09-10T18:21:31.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:31 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:31.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:31 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/601695668' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T18:21:31.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:31 smithi204 ceph-mon[32357]: from='client.34334 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T18:21:31.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:31 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:31.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:31 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:31.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:31 smithi204 ceph-mon[32357]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-09-10T18:21:31.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:31 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:31.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:31 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:31.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:31 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:21:31.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:31 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:21:31.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:31 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:21:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[29291]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T18:21:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[29291]: from='client.24358 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-10T18:21:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[29291]: osdmap e57: 8 total, 8 up, 8 in 2024-09-10T18:21:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/601695668' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T18:21:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[29291]: from='client.34334 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T18:21:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[29291]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-09-10T18:21:32.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:32.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:32.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:21:32.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:21:32.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:21:32.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[39220]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T18:21:32.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[39220]: from='client.24358 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-10T18:21:32.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[39220]: osdmap e57: 8 total, 8 up, 8 in 2024-09-10T18:21:32.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:32.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/601695668' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T18:21:32.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[39220]: from='client.34334 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T18:21:32.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:32.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:32.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[39220]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-09-10T18:21:32.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:32.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:32.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:21:32.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:21:32.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:31 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:21:32.730 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:32 smithi204 ceph-mon[32357]: from='client.34334 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-10T18:21:32.730 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:32 smithi204 ceph-mon[32357]: osdmap e58: 8 total, 8 up, 8 in 2024-09-10T18:21:32.730 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:32 smithi204 ceph-mon[32357]: pgmap v180: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T18:21:32.730 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:32 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:21:32.730 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:32 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:21:32.730 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:32 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:32.730 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:32 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:32.730 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:32 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:32.731 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:32 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:21:33.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[29291]: from='client.34334 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-10T18:21:33.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[29291]: osdmap e58: 8 total, 8 up, 8 in 2024-09-10T18:21:33.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[29291]: pgmap v180: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T18:21:33.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:21:33.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:21:33.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:33.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:33.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:33.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:21:33.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[39220]: from='client.34334 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-10T18:21:33.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[39220]: osdmap e58: 8 total, 8 up, 8 in 2024-09-10T18:21:33.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[39220]: pgmap v180: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T18:21:33.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:21:33.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:21:33.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:33.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:33.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:33.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:32 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:21:33.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:33 smithi204 ceph-mon[32357]: Checking dashboard <-> RGW credentials 2024-09-10T18:21:33.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:33 smithi204 ceph-mon[32357]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:21:33.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:33 smithi204 ceph-mon[32357]: osdmap e59: 8 total, 8 up, 8 in 2024-09-10T18:21:33.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:33 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:33.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:33 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/601695668' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:21:33.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:33 smithi204 ceph-mon[32357]: from='client.34334 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:21:33.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:33 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3347113795' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:21:33.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:33 smithi204 ceph-mon[32357]: from='client.34358 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:21:33.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:33 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.204:0/2873224768' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T18:21:33.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:33 smithi204 ceph-mon[32357]: from='client.24397 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T18:21:34.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[39220]: Checking dashboard <-> RGW credentials 2024-09-10T18:21:34.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[39220]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:21:34.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[39220]: osdmap e59: 8 total, 8 up, 8 in 2024-09-10T18:21:34.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:34.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/601695668' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:21:34.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[39220]: from='client.34334 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:21:34.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3347113795' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:21:34.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[39220]: from='client.34358 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:21:34.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.204:0/2873224768' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T18:21:34.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[39220]: from='client.24397 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T18:21:34.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[29291]: Checking dashboard <-> RGW credentials 2024-09-10T18:21:34.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[29291]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:21:34.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[29291]: osdmap e59: 8 total, 8 up, 8 in 2024-09-10T18:21:34.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:34.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/601695668' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:21:34.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[29291]: from='client.34334 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:21:34.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3347113795' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:21:34.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[29291]: from='client.34358 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:21:34.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.204:0/2873224768' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T18:21:34.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:33 smithi028 ceph-mon[29291]: from='client.24397 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T18:21:34.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:34 smithi204 ceph-mon[32357]: pgmap v182: 68 pgs: 13 creating+peering, 32 unknown, 23 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 170 B/s wr, 0 op/s 2024-09-10T18:21:34.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:34 smithi204 ceph-mon[32357]: from='client.34334 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T18:21:34.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:34 smithi204 ceph-mon[32357]: from='client.34358 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T18:21:34.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:34 smithi204 ceph-mon[32357]: from='client.24397 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-10T18:21:34.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:34 smithi204 ceph-mon[32357]: osdmap e60: 8 total, 8 up, 8 in 2024-09-10T18:21:35.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:34 smithi028 ceph-mon[29291]: pgmap v182: 68 pgs: 13 creating+peering, 32 unknown, 23 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 170 B/s wr, 0 op/s 2024-09-10T18:21:35.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:34 smithi028 ceph-mon[29291]: from='client.34334 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T18:21:35.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:34 smithi028 ceph-mon[29291]: from='client.34358 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T18:21:35.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:34 smithi028 ceph-mon[29291]: from='client.24397 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-10T18:21:35.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:34 smithi028 ceph-mon[29291]: osdmap e60: 8 total, 8 up, 8 in 2024-09-10T18:21:35.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:34 smithi028 ceph-mon[39220]: pgmap v182: 68 pgs: 13 creating+peering, 32 unknown, 23 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 170 B/s wr, 0 op/s 2024-09-10T18:21:35.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:34 smithi028 ceph-mon[39220]: from='client.34334 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T18:21:35.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:34 smithi028 ceph-mon[39220]: from='client.34358 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T18:21:35.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:34 smithi028 ceph-mon[39220]: from='client.24397 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-10T18:21:35.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:34 smithi028 ceph-mon[39220]: osdmap e60: 8 total, 8 up, 8 in 2024-09-10T18:21:35.984 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:35 smithi204 ceph-mon[32357]: osdmap e61: 8 total, 8 up, 8 in 2024-09-10T18:21:35.984 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:35 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/601695668' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:21:35.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:35 smithi204 ceph-mon[32357]: from='client.34334 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:21:35.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:35 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3347113795' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:21:35.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:35 smithi204 ceph-mon[32357]: from='client.34358 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:21:36.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:35 smithi028 ceph-mon[39220]: osdmap e61: 8 total, 8 up, 8 in 2024-09-10T18:21:36.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:35 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/601695668' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:21:36.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:35 smithi028 ceph-mon[39220]: from='client.34334 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:21:36.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:35 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3347113795' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:21:36.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:35 smithi028 ceph-mon[39220]: from='client.34358 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:21:36.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:35 smithi028 ceph-mon[29291]: osdmap e61: 8 total, 8 up, 8 in 2024-09-10T18:21:36.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:35 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/601695668' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:21:36.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:35 smithi028 ceph-mon[29291]: from='client.34334 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:21:36.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:35 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3347113795' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:21:36.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:35 smithi028 ceph-mon[29291]: from='client.34358 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:21:36.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.204 --placement '1;smithi204=iscsi.a' 2024-09-10T18:21:36.491 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:21:36.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:36 smithi204 ceph-mon[32357]: pgmap v185: 100 pgs: 20 creating+peering, 32 unknown, 48 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1006 B/s wr, 5 op/s 2024-09-10T18:21:36.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:36 smithi204 ceph-mon[32357]: from='client.34334 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T18:21:36.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:36 smithi204 ceph-mon[32357]: from='client.34358 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T18:21:36.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:36 smithi204 ceph-mon[32357]: osdmap e62: 8 total, 8 up, 8 in 2024-09-10T18:21:37.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:36 smithi028 ceph-mon[29291]: pgmap v185: 100 pgs: 20 creating+peering, 32 unknown, 48 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1006 B/s wr, 5 op/s 2024-09-10T18:21:37.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:36 smithi028 ceph-mon[29291]: from='client.34334 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T18:21:37.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:36 smithi028 ceph-mon[29291]: from='client.34358 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T18:21:37.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:36 smithi028 ceph-mon[29291]: osdmap e62: 8 total, 8 up, 8 in 2024-09-10T18:21:37.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:36 smithi028 ceph-mon[39220]: pgmap v185: 100 pgs: 20 creating+peering, 32 unknown, 48 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1006 B/s wr, 5 op/s 2024-09-10T18:21:37.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:36 smithi028 ceph-mon[39220]: from='client.34334 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T18:21:37.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:36 smithi028 ceph-mon[39220]: from='client.34358 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T18:21:37.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:36 smithi028 ceph-mon[39220]: osdmap e62: 8 total, 8 up, 8 in 2024-09-10T18:21:37.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:37 smithi204 ceph-mon[32357]: osdmap e63: 8 total, 8 up, 8 in 2024-09-10T18:21:37.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:37 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3100642218' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:21:37.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:37 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1487830206' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:21:37.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:37 smithi204 ceph-mon[32357]: from='client.34376 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:21:37.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:37 smithi204 ceph-mon[32357]: from='client.24406 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:21:38.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:37 smithi028 ceph-mon[39220]: osdmap e63: 8 total, 8 up, 8 in 2024-09-10T18:21:38.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:37 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3100642218' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:21:38.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:37 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1487830206' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:21:38.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:37 smithi028 ceph-mon[39220]: from='client.34376 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:21:38.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:37 smithi028 ceph-mon[39220]: from='client.24406 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:21:38.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:37 smithi028 ceph-mon[29291]: osdmap e63: 8 total, 8 up, 8 in 2024-09-10T18:21:38.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:37 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3100642218' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:21:38.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:37 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1487830206' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:21:38.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:37 smithi028 ceph-mon[29291]: from='client.34376 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:21:38.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:37 smithi028 ceph-mon[29291]: from='client.24406 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:21:38.242 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled iscsi.datapool update... 2024-09-10T18:21:38.833 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:38 smithi204 ceph-mon[32357]: pgmap v188: 132 pgs: 27 creating+peering, 41 unknown, 64 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-10T18:21:38.833 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:38 smithi204 ceph-mon[32357]: from='client.34376 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T18:21:38.833 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:38 smithi204 ceph-mon[32357]: from='client.24406 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T18:21:38.833 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:38 smithi204 ceph-mon[32357]: osdmap e64: 8 total, 8 up, 8 in 2024-09-10T18:21:38.833 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:38 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3100642218' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:21:38.833 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:38 smithi204 ceph-mon[32357]: from='client.34376 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:21:38.834 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:38 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1487830206' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:21:38.834 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:38 smithi204 ceph-mon[32357]: from='client.24406 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:21:38.834 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:38 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:38.860 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-10T18:21:38.860 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T18:21:38.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-10T18:21:38.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[29291]: pgmap v188: 132 pgs: 27 creating+peering, 41 unknown, 64 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-10T18:21:38.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[29291]: from='client.34376 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T18:21:38.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[29291]: from='client.24406 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T18:21:38.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[29291]: osdmap e64: 8 total, 8 up, 8 in 2024-09-10T18:21:38.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3100642218' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:21:38.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[29291]: from='client.34376 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:21:38.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1487830206' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:21:38.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[29291]: from='client.24406 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:21:38.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:38.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[39220]: pgmap v188: 132 pgs: 27 creating+peering, 41 unknown, 64 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-10T18:21:38.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[39220]: from='client.34376 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T18:21:38.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[39220]: from='client.24406 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T18:21:38.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[39220]: osdmap e64: 8 total, 8 up, 8 in 2024-09-10T18:21:38.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3100642218' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:21:38.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[39220]: from='client.34376 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:21:38.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1487830206' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:21:38.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[39220]: from='client.24406 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:21:38.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:38 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:38.893 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-10T18:21:38.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-10T18:21:38.922 DEBUG:teuthology.orchestra.run.smithi204:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@iscsi.iscsi.a.service 2024-09-10T18:21:38.964 INFO:tasks.cephadm:Adding prometheus.a on smithi204 2024-09-10T18:21:38.965 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch apply prometheus '1;smithi204=a' 2024-09-10T18:21:39.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:39 smithi204 ceph-mon[32357]: from='client.24415 v1:172.21.15.204:0/3727103084' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.204", "placement": "1;smithi204=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:39.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:39 smithi204 ceph-mon[32357]: Saving service iscsi.datapool spec with placement smithi204=iscsi.a;count:1 2024-09-10T18:21:39.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:39 smithi204 ceph-mon[32357]: from='client.34376 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T18:21:39.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:39 smithi204 ceph-mon[32357]: from='client.24406 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T18:21:39.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:39 smithi204 ceph-mon[32357]: osdmap e65: 8 total, 8 up, 8 in 2024-09-10T18:21:39.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:39 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:39.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:39 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:39.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:39 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:39.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:39 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-10T18:21:39.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:39 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-10T18:21:39.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:39 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:39.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:39 smithi204 ceph-mon[32357]: Deploying daemon iscsi.iscsi.a on smithi204 2024-09-10T18:21:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[29291]: from='client.24415 v1:172.21.15.204:0/3727103084' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.204", "placement": "1;smithi204=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[29291]: Saving service iscsi.datapool spec with placement smithi204=iscsi.a;count:1 2024-09-10T18:21:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[29291]: from='client.34376 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T18:21:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[29291]: from='client.24406 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T18:21:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[29291]: osdmap e65: 8 total, 8 up, 8 in 2024-09-10T18:21:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:40.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:40.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-10T18:21:40.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-10T18:21:40.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:40.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[29291]: Deploying daemon iscsi.iscsi.a on smithi204 2024-09-10T18:21:40.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[39220]: from='client.24415 v1:172.21.15.204:0/3727103084' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.204", "placement": "1;smithi204=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:40.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[39220]: Saving service iscsi.datapool spec with placement smithi204=iscsi.a;count:1 2024-09-10T18:21:40.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[39220]: from='client.34376 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T18:21:40.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[39220]: from='client.24406 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T18:21:40.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[39220]: osdmap e65: 8 total, 8 up, 8 in 2024-09-10T18:21:40.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:40.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:40.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:40.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-10T18:21:40.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-10T18:21:40.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:40.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:39 smithi028 ceph-mon[39220]: Deploying daemon iscsi.iscsi.a on smithi204 2024-09-10T18:21:40.302 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:21:41.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:40 smithi028 ceph-mon[29291]: pgmap v191: 132 pgs: 4 creating+activating, 14 creating+peering, 5 unknown, 109 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-09-10T18:21:41.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:40 smithi028 ceph-mon[29291]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:21:41.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:40 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:21:41.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:40 smithi028 ceph-mon[29291]: osdmap e66: 8 total, 8 up, 8 in 2024-09-10T18:21:41.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:40 smithi028 ceph-mon[39220]: pgmap v191: 132 pgs: 4 creating+activating, 14 creating+peering, 5 unknown, 109 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-09-10T18:21:41.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:40 smithi028 ceph-mon[39220]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:21:41.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:40 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:21:41.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:40 smithi028 ceph-mon[39220]: osdmap e66: 8 total, 8 up, 8 in 2024-09-10T18:21:41.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:40 smithi204 ceph-mon[32357]: pgmap v191: 132 pgs: 4 creating+activating, 14 creating+peering, 5 unknown, 109 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-09-10T18:21:41.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:40 smithi204 ceph-mon[32357]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:21:41.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:40 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:21:41.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:40 smithi204 ceph-mon[32357]: osdmap e66: 8 total, 8 up, 8 in 2024-09-10T18:21:42.111 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled prometheus update... 2024-09-10T18:21:42.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:42 smithi204 ceph-mon[32357]: pgmap v193: 132 pgs: 4 creating+activating, 7 creating+peering, 121 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 3.2 KiB/s wr, 85 op/s 2024-09-10T18:21:42.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:42 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:43.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:42 smithi028 ceph-mon[29291]: pgmap v193: 132 pgs: 4 creating+activating, 7 creating+peering, 121 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 3.2 KiB/s wr, 85 op/s 2024-09-10T18:21:43.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:42 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:43.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:42 smithi028 ceph-mon[39220]: pgmap v193: 132 pgs: 4 creating+activating, 7 creating+peering, 121 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 3.2 KiB/s wr, 85 op/s 2024-09-10T18:21:43.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:42 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:43.007 DEBUG:teuthology.orchestra.run.smithi204:prometheus.a> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@prometheus.a.service 2024-09-10T18:21:43.009 INFO:tasks.cephadm:Adding node-exporter.a on smithi028 2024-09-10T18:21:43.010 INFO:tasks.cephadm:Adding node-exporter.b on smithi204 2024-09-10T18:21:43.010 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch apply node-exporter '2;smithi028=a;smithi204=b' 2024-09-10T18:21:43.714 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:21:43.805 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:43 smithi204 ceph-mon[32357]: from='client.24436 v1:172.21.15.204:0/978176816' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:43.806 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:43 smithi204 ceph-mon[32357]: Saving service prometheus spec with placement smithi204=a;count:1 2024-09-10T18:21:43.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:43 smithi028 ceph-mon[29291]: from='client.24436 v1:172.21.15.204:0/978176816' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:43.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:43 smithi028 ceph-mon[29291]: Saving service prometheus spec with placement smithi204=a;count:1 2024-09-10T18:21:43.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:43 smithi028 ceph-mon[39220]: from='client.24436 v1:172.21.15.204:0/978176816' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:43.906 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:43 smithi028 ceph-mon[39220]: Saving service prometheus spec with placement smithi204=a;count:1 2024-09-10T18:21:44.718 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:44 smithi204 ceph-mon[32357]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 5.8 KiB/s wr, 186 op/s 2024-09-10T18:21:44.718 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:44 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:44.719 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:44 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:44.719 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:44 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:44.719 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:44 smithi204 ceph-mon[32357]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-10T18:21:44.719 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:44 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:44.719 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:44 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:21:44.719 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:44 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.204:0/1066353315' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-10T18:21:45.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:44 smithi028 ceph-mon[29291]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 5.8 KiB/s wr, 186 op/s 2024-09-10T18:21:45.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:44 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:45.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:44 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:45.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:44 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:45.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:44 smithi028 ceph-mon[29291]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-10T18:21:45.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:44 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:45.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:44 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:21:45.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:44 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.204:0/1066353315' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-10T18:21:45.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:44 smithi028 ceph-mon[39220]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 5.8 KiB/s wr, 186 op/s 2024-09-10T18:21:45.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:44 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:45.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:44 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:45.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:44 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:45.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:44 smithi028 ceph-mon[39220]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-10T18:21:45.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:44 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:45.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:44 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:21:45.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:44 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.204:0/1066353315' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-10T18:21:45.392 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled node-exporter update... 2024-09-10T18:21:46.101 DEBUG:teuthology.orchestra.run.smithi028:node-exporter.a> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@node-exporter.a.service 2024-09-10T18:21:46.103 DEBUG:teuthology.orchestra.run.smithi204:node-exporter.b> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@node-exporter.b.service 2024-09-10T18:21:46.106 INFO:tasks.cephadm:Adding alertmanager.a on smithi028 2024-09-10T18:21:46.106 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch apply alertmanager '1;smithi028=a' 2024-09-10T18:21:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:46 smithi204 ceph-mon[32357]: from='client.24472 v1:172.21.15.204:0/1877851178' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi028=a;smithi204=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:46 smithi204 ceph-mon[32357]: Saving service node-exporter spec with placement smithi028=a;smithi204=b;count:2 2024-09-10T18:21:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:46 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:46 smithi204 ceph-mon[32357]: osdmap e67: 8 total, 8 up, 8 in 2024-09-10T18:21:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:46 smithi204 ceph-mon[32357]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 5.2 KiB/s wr, 162 op/s 2024-09-10T18:21:46.622 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:21:46.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:46 smithi028 ceph-mon[39220]: from='client.24472 v1:172.21.15.204:0/1877851178' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi028=a;smithi204=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:46.771 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:46 smithi028 ceph-mon[39220]: Saving service node-exporter spec with placement smithi028=a;smithi204=b;count:2 2024-09-10T18:21:46.771 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:46 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:46.771 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:46 smithi028 ceph-mon[39220]: osdmap e67: 8 total, 8 up, 8 in 2024-09-10T18:21:46.771 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:46 smithi028 ceph-mon[39220]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 5.2 KiB/s wr, 162 op/s 2024-09-10T18:21:46.772 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:46 smithi028 ceph-mon[29291]: from='client.24472 v1:172.21.15.204:0/1877851178' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi028=a;smithi204=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:46.772 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:46 smithi028 ceph-mon[29291]: Saving service node-exporter spec with placement smithi028=a;smithi204=b;count:2 2024-09-10T18:21:46.772 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:46 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:46.772 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:46 smithi028 ceph-mon[29291]: osdmap e67: 8 total, 8 up, 8 in 2024-09-10T18:21:46.772 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:46 smithi028 ceph-mon[29291]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 5.2 KiB/s wr, 162 op/s 2024-09-10T18:21:47.820 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:47 smithi204 ceph-mon[32357]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-10T18:21:48.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:47 smithi028 ceph-mon[39220]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-10T18:21:48.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:47 smithi028 ceph-mon[29291]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-10T18:21:48.303 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled alertmanager update... 2024-09-10T18:21:48.769 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:48 smithi204 ceph-mon[32357]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 57 KiB/s rd, 4.2 KiB/s wr, 140 op/s 2024-09-10T18:21:48.769 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:48.769 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:48 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:49.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:48 smithi028 ceph-mon[39220]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 57 KiB/s rd, 4.2 KiB/s wr, 140 op/s 2024-09-10T18:21:49.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:48 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:49.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:48 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:49.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:48 smithi028 ceph-mon[29291]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 57 KiB/s rd, 4.2 KiB/s wr, 140 op/s 2024-09-10T18:21:49.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:49.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:48 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:49.068 DEBUG:teuthology.orchestra.run.smithi028:alertmanager.a> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@alertmanager.a.service 2024-09-10T18:21:49.071 INFO:tasks.cephadm:Adding grafana.a on smithi204 2024-09-10T18:21:49.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph orch apply grafana '1;smithi204=a' 2024-09-10T18:21:49.423 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:21:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:49 smithi204 ceph-mon[32357]: from='client.24478 v1:172.21.15.204:0/3934914647' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi028=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:49 smithi204 ceph-mon[32357]: Saving service alertmanager spec with placement smithi028=a;count:1 2024-09-10T18:21:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:49 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:49 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:49 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:49 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:21:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:49 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:50.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:49 smithi028 ceph-mon[29291]: from='client.24478 v1:172.21.15.204:0/3934914647' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi028=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:50.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:49 smithi028 ceph-mon[29291]: Saving service alertmanager spec with placement smithi028=a;count:1 2024-09-10T18:21:50.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:49 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:50.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:49 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:50.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:49 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:50.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:49 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:21:50.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:49 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:50.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:49 smithi028 ceph-mon[39220]: from='client.24478 v1:172.21.15.204:0/3934914647' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi028=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:50.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:49 smithi028 ceph-mon[39220]: Saving service alertmanager spec with placement smithi028=a;count:1 2024-09-10T18:21:50.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:49 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:50.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:49 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:50.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:49 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:21:50.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:49 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:21:50.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:49 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:50.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:50 smithi204 ceph-mon[32357]: Deploying daemon prometheus.a on smithi204 2024-09-10T18:21:50.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:50 smithi204 ceph-mon[32357]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 33 KiB/s rd, 2.5 KiB/s wr, 79 op/s 2024-09-10T18:21:50.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:50 smithi204 ceph-mon[32357]: osdmap e68: 8 total, 8 up, 8 in 2024-09-10T18:21:51.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:50 smithi028 ceph-mon[29291]: Deploying daemon prometheus.a on smithi204 2024-09-10T18:21:51.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:50 smithi028 ceph-mon[29291]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 33 KiB/s rd, 2.5 KiB/s wr, 79 op/s 2024-09-10T18:21:51.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:50 smithi028 ceph-mon[29291]: osdmap e68: 8 total, 8 up, 8 in 2024-09-10T18:21:51.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:50 smithi028 ceph-mon[39220]: Deploying daemon prometheus.a on smithi204 2024-09-10T18:21:51.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:50 smithi028 ceph-mon[39220]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 33 KiB/s rd, 2.5 KiB/s wr, 79 op/s 2024-09-10T18:21:51.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:50 smithi028 ceph-mon[39220]: osdmap e68: 8 total, 8 up, 8 in 2024-09-10T18:21:51.183 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled grafana update... 2024-09-10T18:21:51.846 DEBUG:teuthology.orchestra.run.smithi204:grafana.a> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@grafana.a.service 2024-09-10T18:21:51.848 INFO:tasks.cephadm:Setting up client nodes... 2024-09-10T18:21:51.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-10T18:21:52.114 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:21:52.465 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:52 smithi204 ceph-mon[32357]: from='client.24484 v1:172.21.15.204:0/655819331' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:52.465 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:52 smithi204 ceph-mon[32357]: Saving service grafana spec with placement smithi204=a;count:1 2024-09-10T18:21:52.465 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:52 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:52.465 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:52 smithi204 ceph-mon[32357]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-09-10T18:21:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:52 smithi028 ceph-mon[29291]: from='client.24484 v1:172.21.15.204:0/655819331' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:52 smithi028 ceph-mon[29291]: Saving service grafana spec with placement smithi204=a;count:1 2024-09-10T18:21:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:52 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:52 smithi028 ceph-mon[29291]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-09-10T18:21:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:52 smithi028 ceph-mon[39220]: from='client.24484 v1:172.21.15.204:0/655819331' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi204=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:21:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:52 smithi028 ceph-mon[39220]: Saving service grafana spec with placement smithi204=a;count:1 2024-09-10T18:21:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:52 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:52 smithi028 ceph-mon[39220]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-09-10T18:21:53.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:53 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:21:53.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:53 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:21:53.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:53 smithi028 ceph-mon[29291]: osdmap e69: 8 total, 8 up, 8 in 2024-09-10T18:21:53.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:53 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:53.476 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:53 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:21:53.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:53 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:21:53.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:53 smithi028 ceph-mon[39220]: osdmap e69: 8 total, 8 up, 8 in 2024-09-10T18:21:53.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:53 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:53 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:21:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:53 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:21:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:53 smithi204 ceph-mon[32357]: osdmap e69: 8 total, 8 up, 8 in 2024-09-10T18:21:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:53 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:21:54.082 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:21:53 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:21:54.170 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-10T18:21:54.171 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBCjuBm1crgCRAAraqXdISoutQSGsVYXatKrg== 2024-09-10T18:21:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:54 smithi204 ceph-mon[32357]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:21:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:54 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:21:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:54 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1719636069' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:21:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:54 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1719636069' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T18:21:54.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:54 smithi028 ceph-mon[39220]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:21:54.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:54 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:21:54.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:54 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1719636069' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:21:54.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:54 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1719636069' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T18:21:54.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:54 smithi028 ceph-mon[29291]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:21:54.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:54 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:21:54.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:54 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1719636069' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:21:54.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:54 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1719636069' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T18:21:54.795 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-10T18:21:54.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-10T18:21:54.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-10T18:21:54.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-10T18:21:55.173 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:21:55.489 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:55 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:21:55.489 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:55 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:21:55.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:55 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:21:55.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:55 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:21:55.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:55 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:21:55.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:55 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:21:56.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:56 smithi028 ceph-mon[29291]: osdmap e70: 8 total, 8 up, 8 in 2024-09-10T18:21:56.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:56 smithi028 ceph-mon[29291]: osdmap e71: 8 total, 8 up, 8 in 2024-09-10T18:21:56.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:56 smithi028 ceph-mon[29291]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1008 B/s rd, 0 op/s 2024-09-10T18:21:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:56 smithi028 ceph-mon[39220]: osdmap e70: 8 total, 8 up, 8 in 2024-09-10T18:21:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:56 smithi028 ceph-mon[39220]: osdmap e71: 8 total, 8 up, 8 in 2024-09-10T18:21:56.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:56 smithi028 ceph-mon[39220]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1008 B/s rd, 0 op/s 2024-09-10T18:21:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:56 smithi204 ceph-mon[32357]: osdmap e70: 8 total, 8 up, 8 in 2024-09-10T18:21:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:56 smithi204 ceph-mon[32357]: osdmap e71: 8 total, 8 up, 8 in 2024-09-10T18:21:56.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:56 smithi204 ceph-mon[32357]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1008 B/s rd, 0 op/s 2024-09-10T18:21:58.814 INFO:teuthology.orchestra.run.smithi204.stdout:[client.1] 2024-09-10T18:21:58.814 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBGjuBmwgYpMBAAq8YeBYQ9fjrrJKJFEWvqFA== 2024-09-10T18:21:58.814 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:58 smithi204 ceph-mon[32357]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:21:59.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:58 smithi028 ceph-mon[29291]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:21:59.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:58 smithi028 ceph-mon[39220]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:21:59.845 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:59 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.204:0/3437863352' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:21:59.845 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:59 smithi204 ceph-mon[32357]: from='client.24496 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:21:59.845 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:21:59 smithi204 ceph-mon[32357]: from='client.24496 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T18:22:00.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:59 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.204:0/3437863352' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:22:00.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:59 smithi028 ceph-mon[29291]: from='client.24496 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:22:00.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:21:59 smithi028 ceph-mon[29291]: from='client.24496 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T18:22:00.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:59 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.204:0/3437863352' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:22:00.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:59 smithi028 ceph-mon[39220]: from='client.24496 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:22:00.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:21:59 smithi028 ceph-mon[39220]: from='client.24496 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T18:22:00.156 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-10T18:22:00.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-10T18:22:00.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-10T18:22:00.198 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-10T18:22:00.198 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-10T18:22:00.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph mgr dump --format=json 2024-09-10T18:22:00.463 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:00.833 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:00 smithi204 ceph-mon[32357]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:22:00.833 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:00 smithi204 ceph-mon[32357]: osdmap e72: 8 total, 8 up, 8 in 2024-09-10T18:22:00.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:00 smithi028 ceph-mon[29291]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:22:00.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:00 smithi028 ceph-mon[29291]: osdmap e72: 8 total, 8 up, 8 in 2024-09-10T18:22:00.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:00 smithi028 ceph-mon[39220]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:22:00.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:00 smithi028 ceph-mon[39220]: osdmap e72: 8 total, 8 up, 8 in 2024-09-10T18:22:02.473 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:22:02.560 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 systemd[1]: Starting Ceph prometheus.a for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:22:02.817 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 podman[61034]: 2024-09-10 18:22:02.462751891 +0000 UTC m=+0.016835974 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-10T18:22:02.817 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 podman[61034]: 2024-09-10 18:22:02.573860784 +0000 UTC m=+0.127944857 container create 0bff8348e47dc6f65d05a5343cd38c7fe487e2f07d57961011ae3bfb1e15fe7b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T18:22:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-mon[32357]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 800 B/s rd, 0 op/s 2024-09-10T18:22:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/273877279' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-10T18:22:03.105 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 podman[61034]: 2024-09-10 18:22:02.843326404 +0000 UTC m=+0.397410477 container init 0bff8348e47dc6f65d05a5343cd38c7fe487e2f07d57961011ae3bfb1e15fe7b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T18:22:03.105 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 podman[61034]: 2024-09-10 18:22:02.846276806 +0000 UTC m=+0.400360878 container start 0bff8348e47dc6f65d05a5343cd38c7fe487e2f07d57961011ae3bfb1e15fe7b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T18:22:03.105 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.870Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-10T18:22:03.105 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.870Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-10T18:22:03.105 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.870Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi204 (none))" 2024-09-10T18:22:03.105 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.870Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-10T18:22:03.105 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.870Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-10T18:22:03.105 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.871Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-10T18:22:03.105 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.872Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-10T18:22:03.105 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.873Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-10T18:22:03.105 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.873Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-10T18:22:03.105 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.875Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-10T18:22:03.106 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.875Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.3µs 2024-09-10T18:22:03.106 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.875Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-10T18:22:03.106 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.875Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-10T18:22:03.106 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.875Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=26.192µs wal_replay_duration=363.38µs wbl_replay_duration=123ns total_replay_duration=418.364µs 2024-09-10T18:22:03.106 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.876Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-10T18:22:03.106 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.876Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-10T18:22:03.106 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.876Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-10T18:22:03.106 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.892Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.832185ms db_storage=724ns remote_storage=1.036µs web_handler=386ns query_engine=530ns scrape=404.336µs scrape_sd=16.685µs notify=725ns notify_sd=1.478µs rules=15.20269ms tracing=7.244µs 2024-09-10T18:22:03.106 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.892Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-10T18:22:03.106 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:22:02.892Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-10T18:22:03.106 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 bash[61034]: 0bff8348e47dc6f65d05a5343cd38c7fe487e2f07d57961011ae3bfb1e15fe7b 2024-09-10T18:22:03.107 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:22:02 smithi204 systemd[1]: Started Ceph prometheus.a for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:22:03.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:02 smithi028 ceph-mon[29291]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 800 B/s rd, 0 op/s 2024-09-10T18:22:03.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:02 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/273877279' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-10T18:22:03.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:02 smithi028 ceph-mon[39220]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 800 B/s rd, 0 op/s 2024-09-10T18:22:03.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:02 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/273877279' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-10T18:22:03.222 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":16,"flags":0,"active_gid":14148,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6800","nonce":1077377902}]},"active_addr":"172.21.15.28:6800/1077377902","active_change":"2024-09-10T18:17:07.521269+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24112,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.28:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":717120167}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":2620179974}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":1759116360}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3428757300}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":1745574083}]}]} 2024-09-10T18:22:03.226 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-10T18:22:03.226 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-10T18:22:03.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T18:22:03.486 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:04.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:03 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:22:04.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:03 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:22:04.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:03 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:22:04.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:03 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-10T18:22:04.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:03 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:22:04.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:03 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:22:04.105 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ignoring --setuser ceph since I am not root 2024-09-10T18:22:04.105 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ignoring --setgroup ceph since I am not root 2024-09-10T18:22:04.106 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:22:03 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:22:04.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:03 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:22:04.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:03 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:22:04.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:03 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:22:04.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:03 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-10T18:22:04.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:03 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:22:04.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:03 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:22:04.203 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:03 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:22:04.203 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:03 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:22:04.203 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:03 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' 2024-09-10T18:22:04.203 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:03 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-10T18:22:04.203 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:03 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:22:04.203 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:03 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:22:04.203 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:04 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ignoring --setuser ceph since I am not root 2024-09-10T18:22:04.203 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:04 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ignoring --setgroup ceph since I am not root 2024-09-10T18:22:04.204 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:04 smithi028 ceph-mgr[29522]: -- 172.21.15.28:0/17370652 <== mon.2 v1:172.21.15.28:6790/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (unknown 974846825 0 0) 0x55c83f594820 con 0x55c83f5a0c00 2024-09-10T18:22:04.204 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:04 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:04.140+0000 7f2e27866ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T18:22:04.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:04 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:04.199+0000 7f2e27866ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T18:22:04.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:04 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:04.290+0000 7f2e27866ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T18:22:04.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:04 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:04.420+0000 7f2e27866ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:22:04.527 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:04.141+0000 7f0fcbf54ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T18:22:04.527 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:04.197+0000 7f0fcbf54ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T18:22:04.527 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:04.264+0000 7f0fcbf54ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T18:22:04.528 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:04.388+0000 7f0fcbf54ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:22:04.791 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:04.525+0000 7f0fcbf54ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T18:22:04.846 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:04 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:04.562+0000 7f2e27866ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T18:22:04.846 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:04 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:04.843+0000 7f2e27866ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T18:22:05.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:04 smithi204 ceph-mon[32357]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-10T18:22:05.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:04 smithi204 ceph-mon[32357]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-10T18:22:05.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:04 smithi204 ceph-mon[32357]: osdmap e73: 8 total, 8 up, 8 in 2024-09-10T18:22:05.104 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:04.789+0000 7f0fcbf54ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T18:22:05.104 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:04.849+0000 7f0fcbf54ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T18:22:05.105 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:05 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:05.014+0000 7f0fcbf54ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T18:22:05.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:04 smithi028 ceph-mon[29291]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-10T18:22:05.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:04 smithi028 ceph-mon[29291]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-10T18:22:05.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:04 smithi028 ceph-mon[29291]: osdmap e73: 8 total, 8 up, 8 in 2024-09-10T18:22:05.214 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:04 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:04.905+0000 7f2e27866ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T18:22:05.214 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:05 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:05.084+0000 7f2e27866ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T18:22:05.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:04 smithi028 ceph-mon[39220]: from='mgr.14148 v1:172.21.15.28:0/3124638065' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-10T18:22:05.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:04 smithi028 ceph-mon[39220]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-10T18:22:05.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:04 smithi028 ceph-mon[39220]: osdmap e73: 8 total, 8 up, 8 in 2024-09-10T18:22:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:22:05.467 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":73,"fsid":"5eababb2-6fa0-11ef-bcea-c7b262605968","created":"2024-09-10T18:16:20.680373+0000","modified":"2024-09-10T18:22:03.994302+0000","last_up_change":"2024-09-10T18:21:18.554378+0000","last_in_change":"2024-09-10T18:21:02.114621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T18:19:24.013536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-10T18:21:30.065209+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":62,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-10T18:21:30.506441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-10T18:21:32.008130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-10T18:21:34.060571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-10T18:21:36.127330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"f1c834d9-252a-4077-b25e-169181d70251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6801","nonce":3188372351}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6802","nonce":3188372351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6804","nonce":3188372351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6803","nonce":3188372351}]},"public_addr":"172.21.15.28:6801/3188372351","cluster_addr":"172.21.15.28:6802/3188372351","heartbeat_back_addr":"172.21.15.28:6804/3188372351","heartbeat_front_addr":"172.21.15.28:6803/3188372351","state":["exists","up"]},{"osd":1,"uuid":"81f1855d-b946-4ca3-b246-38977b9e20fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6805","nonce":1369912678}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6806","nonce":1369912678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6808","nonce":1369912678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6807","nonce":1369912678}]},"public_addr":"172.21.15.28:6805/1369912678","cluster_addr":"172.21.15.28:6806/1369912678","heartbeat_back_addr":"172.21.15.28:6808/1369912678","heartbeat_front_addr":"172.21.15.28:6807/1369912678","state":["exists","up"]},{"osd":2,"uuid":"2ad55d2a-36fa-4756-bf22-58d4b049486e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6809","nonce":1243406003}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6810","nonce":1243406003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6812","nonce":1243406003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6811","nonce":1243406003}]},"public_addr":"172.21.15.28:6809/1243406003","cluster_addr":"172.21.15.28:6810/1243406003","heartbeat_back_addr":"172.21.15.28:6812/1243406003","heartbeat_front_addr":"172.21.15.28:6811/1243406003","state":["exists","up"]},{"osd":3,"uuid":"242729a5-a5e8-44d5-97c8-14d14e970bff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6813","nonce":67703283}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6814","nonce":67703283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6816","nonce":67703283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6815","nonce":67703283}]},"public_addr":"172.21.15.28:6813/67703283","cluster_addr":"172.21.15.28:6814/67703283","heartbeat_back_addr":"172.21.15.28:6816/67703283","heartbeat_front_addr":"172.21.15.28:6815/67703283","state":["exists","up"]},{"osd":4,"uuid":"6e38b8ef-e6ed-493f-8eca-755354754787","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6800","nonce":1619669076}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6801","nonce":1619669076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6803","nonce":1619669076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6802","nonce":1619669076}]},"public_addr":"172.21.15.204:6800/1619669076","cluster_addr":"172.21.15.204:6801/1619669076","heartbeat_back_addr":"172.21.15.204:6803/1619669076","heartbeat_front_addr":"172.21.15.204:6802/1619669076","state":["exists","up"]},{"osd":5,"uuid":"a92e0441-66c9-498e-a0ac-6465acdbb1f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6804","nonce":2717802880}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6805","nonce":2717802880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6807","nonce":2717802880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6806","nonce":2717802880}]},"public_addr":"172.21.15.204:6804/2717802880","cluster_addr":"172.21.15.204:6805/2717802880","heartbeat_back_addr":"172.21.15.204:6807/2717802880","heartbeat_front_addr":"172.21.15.204:6806/2717802880","state":["exists","up"]},{"osd":6,"uuid":"12b02875-6653-45ed-8d2b-b4d33bd88525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6808","nonce":3377682607}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6809","nonce":3377682607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6811","nonce":3377682607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6810","nonce":3377682607}]},"public_addr":"172.21.15.204:6808/3377682607","cluster_addr":"172.21.15.204:6809/3377682607","heartbeat_back_addr":"172.21.15.204:6811/3377682607","heartbeat_front_addr":"172.21.15.204:6810/3377682607","state":["exists","up"]},{"osd":7,"uuid":"0c4f7aa9-ff95-4192-a146-ca576eee907f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6812","nonce":3511791879}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6813","nonce":3511791879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6815","nonce":3511791879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6814","nonce":3511791879}]},"public_addr":"172.21.15.204:6812/3511791879","cluster_addr":"172.21.15.204:6813/3511791879","heartbeat_back_addr":"172.21.15.204:6815/3511791879","heartbeat_front_addr":"172.21.15.204:6814/3511791879","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:22:03.348971+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:21:31.930690+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:21:54.771209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:21:04.315918+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:21:30.569793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:21:51.675189+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:20:53.767012+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:21:17.055966+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/2564333506":"2024-09-11T18:17:07.521165+0000","172.21.15.28:0/2874160765":"2024-09-11T18:17:07.521165+0000","172.21.15.28:6800/3310949126":"2024-09-11T18:17:07.521165+0000","172.21.15.28:0/1746797014":"2024-09-11T18:16:45.197204+0000","172.21.15.28:0/421194352":"2024-09-11T18:17:07.521165+0000","172.21.15.28:0/1029045895":"2024-09-11T18:16:45.197204+0000","172.21.15.28:6800/3760775432":"2024-09-11T18:16:45.197204+0000","172.21.15.28:0/3866220552":"2024-09-11T18:16:45.197204+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T18:22:06.058 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:06 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/893084938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:22:06.058 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:06 smithi028 ceph-mon[29291]: osdmap e74: 8 total, 8 up, 8 in 2024-09-10T18:22:06.059 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:05 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:05.987+0000 7f2e27866ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T18:22:06.059 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:06 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/893084938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:22:06.059 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:06 smithi028 ceph-mon[39220]: osdmap e74: 8 total, 8 up, 8 in 2024-09-10T18:22:06.142 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-10T18:22:06.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T18:22:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:06 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/893084938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:22:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:06 smithi204 ceph-mon[32357]: osdmap e74: 8 total, 8 up, 8 in 2024-09-10T18:22:06.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:05 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:05.878+0000 7f0fcbf54ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T18:22:06.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:05 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:05.943+0000 7f0fcbf54ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T18:22:06.400 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:06.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:06 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:06.056+0000 7f2e27866ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T18:22:06.953 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:06 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:06.660+0000 7f0fcbf54ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T18:22:06.954 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:06 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:06.771+0000 7f0fcbf54ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T18:22:06.954 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:06 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:06.892+0000 7f0fcbf54ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:22:06.954 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:06 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:06.951+0000 7f0fcbf54ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T18:22:07.068 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:06 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:06.823+0000 7f2e27866ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T18:22:07.068 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:06 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:06.939+0000 7f2e27866ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T18:22:07.068 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:07 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:07.066+0000 7f2e27866ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:22:07.205 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:07.078+0000 7f0fcbf54ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T18:22:07.251 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:07.145+0000 7f0fcbf54ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T18:22:07.251 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:07.203+0000 7f0fcbf54ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T18:22:07.332 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:07 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:07.128+0000 7f2e27866ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T18:22:07.332 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:07 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:07.259+0000 7f2e27866ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T18:22:07.332 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:07 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:07.330+0000 7f2e27866ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T18:22:07.472 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:07.338+0000 7f0fcbf54ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T18:22:07.472 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:07.402+0000 7f0fcbf54ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T18:22:07.599 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:07 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:07.390+0000 7f2e27866ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T18:22:07.599 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:07 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:07.531+0000 7f2e27866ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T18:22:07.854 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:07.467+0000 7f0fcbf54ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T18:22:07.854 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:07.548+0000 7f0fcbf54ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T18:22:07.854 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:07.605+0000 7f0fcbf54ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T18:22:07.854 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:07.666+0000 7f0fcbf54ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T18:22:07.877 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:07 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:07.597+0000 7f2e27866ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T18:22:07.877 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:07 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:07.670+0000 7f2e27866ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T18:22:07.877 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:07 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:07.755+0000 7f2e27866ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T18:22:07.877 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:07 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:07.815+0000 7f2e27866ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T18:22:08.167 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:07 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:07.875+0000 7f2e27866ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T18:22:08.353 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:22:08.353 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":74,"fsid":"5eababb2-6fa0-11ef-bcea-c7b262605968","created":"2024-09-10T18:16:20.680373+0000","modified":"2024-09-10T18:22:05.515932+0000","last_up_change":"2024-09-10T18:21:18.554378+0000","last_in_change":"2024-09-10T18:21:02.114621+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T18:19:24.013536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-10T18:21:30.065209+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":62,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-10T18:21:30.506441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-10T18:21:32.008130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-10T18:21:34.060571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-10T18:21:36.127330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"f1c834d9-252a-4077-b25e-169181d70251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6801","nonce":3188372351}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6802","nonce":3188372351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6804","nonce":3188372351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6803","nonce":3188372351}]},"public_addr":"172.21.15.28:6801/3188372351","cluster_addr":"172.21.15.28:6802/3188372351","heartbeat_back_addr":"172.21.15.28:6804/3188372351","heartbeat_front_addr":"172.21.15.28:6803/3188372351","state":["exists","up"]},{"osd":1,"uuid":"81f1855d-b946-4ca3-b246-38977b9e20fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6805","nonce":1369912678}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6806","nonce":1369912678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6808","nonce":1369912678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6807","nonce":1369912678}]},"public_addr":"172.21.15.28:6805/1369912678","cluster_addr":"172.21.15.28:6806/1369912678","heartbeat_back_addr":"172.21.15.28:6808/1369912678","heartbeat_front_addr":"172.21.15.28:6807/1369912678","state":["exists","up"]},{"osd":2,"uuid":"2ad55d2a-36fa-4756-bf22-58d4b049486e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6809","nonce":1243406003}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6810","nonce":1243406003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6812","nonce":1243406003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6811","nonce":1243406003}]},"public_addr":"172.21.15.28:6809/1243406003","cluster_addr":"172.21.15.28:6810/1243406003","heartbeat_back_addr":"172.21.15.28:6812/1243406003","heartbeat_front_addr":"172.21.15.28:6811/1243406003","state":["exists","up"]},{"osd":3,"uuid":"242729a5-a5e8-44d5-97c8-14d14e970bff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6813","nonce":67703283}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6814","nonce":67703283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6816","nonce":67703283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6815","nonce":67703283}]},"public_addr":"172.21.15.28:6813/67703283","cluster_addr":"172.21.15.28:6814/67703283","heartbeat_back_addr":"172.21.15.28:6816/67703283","heartbeat_front_addr":"172.21.15.28:6815/67703283","state":["exists","up"]},{"osd":4,"uuid":"6e38b8ef-e6ed-493f-8eca-755354754787","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6800","nonce":1619669076}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6801","nonce":1619669076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6803","nonce":1619669076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6802","nonce":1619669076}]},"public_addr":"172.21.15.204:6800/1619669076","cluster_addr":"172.21.15.204:6801/1619669076","heartbeat_back_addr":"172.21.15.204:6803/1619669076","heartbeat_front_addr":"172.21.15.204:6802/1619669076","state":["exists","up"]},{"osd":5,"uuid":"a92e0441-66c9-498e-a0ac-6465acdbb1f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6804","nonce":2717802880}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6805","nonce":2717802880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6807","nonce":2717802880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6806","nonce":2717802880}]},"public_addr":"172.21.15.204:6804/2717802880","cluster_addr":"172.21.15.204:6805/2717802880","heartbeat_back_addr":"172.21.15.204:6807/2717802880","heartbeat_front_addr":"172.21.15.204:6806/2717802880","state":["exists","up"]},{"osd":6,"uuid":"12b02875-6653-45ed-8d2b-b4d33bd88525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6808","nonce":3377682607}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6809","nonce":3377682607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6811","nonce":3377682607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6810","nonce":3377682607}]},"public_addr":"172.21.15.204:6808/3377682607","cluster_addr":"172.21.15.204:6809/3377682607","heartbeat_back_addr":"172.21.15.204:6811/3377682607","heartbeat_front_addr":"172.21.15.204:6810/3377682607","state":["exists","up"]},{"osd":7,"uuid":"0c4f7aa9-ff95-4192-a146-ca576eee907f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6812","nonce":3511791879}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6813","nonce":3511791879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6815","nonce":3511791879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6814","nonce":3511791879}]},"public_addr":"172.21.15.204:6812/3511791879","cluster_addr":"172.21.15.204:6813/3511791879","heartbeat_back_addr":"172.21.15.204:6815/3511791879","heartbeat_front_addr":"172.21.15.204:6814/3511791879","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:22:03.348971+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:21:31.930690+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:21:54.771209+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:21:04.315918+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:21:30.569793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:21:51.675189+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:20:53.767012+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:21:17.055966+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/2564333506":"2024-09-11T18:17:07.521165+0000","172.21.15.28:0/2874160765":"2024-09-11T18:17:07.521165+0000","172.21.15.28:6800/3310949126":"2024-09-11T18:17:07.521165+0000","172.21.15.28:0/1746797014":"2024-09-11T18:16:45.197204+0000","172.21.15.28:0/421194352":"2024-09-11T18:17:07.521165+0000","172.21.15.28:0/1029045895":"2024-09-11T18:16:45.197204+0000","172.21.15.28:6800/3760775432":"2024-09-11T18:16:45.197204+0000","172.21.15.28:0/3866220552":"2024-09-11T18:16:45.197204+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T18:22:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:08 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/713413397' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:22:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:08 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/713413397' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:22:08.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:08 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/713413397' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:22:08.854 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:08.420+0000 7f0fcbf54ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T18:22:08.854 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: [10/Sep/2024:18:22:08] ENGINE Bus STARTING 2024-09-10T18:22:08.854 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: CherryPy Checker: 2024-09-10T18:22:08.854 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: The Application mounted at '' has an empty config. 2024-09-10T18:22:08.855 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: 2024-09-10T18:22:08.855 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: [10/Sep/2024:18:22:08] ENGINE Serving on http://:::9283 2024-09-10T18:22:08.855 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: [10/Sep/2024:18:22:08] ENGINE Bus STARTED 2024-09-10T18:22:08.876 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:08 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:08.676+0000 7f2e27866ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T18:22:08.876 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:08 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:22:08] ENGINE Bus STARTING 2024-09-10T18:22:08.876 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:08 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: CherryPy Checker: 2024-09-10T18:22:08.876 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:08 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: The Application mounted at '' has an empty config. 2024-09-10T18:22:08.876 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:08 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:09.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-10T18:22:09.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-10T18:22:09.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-10T18:22:09.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-10T18:22:09.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-10T18:22:09.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-10T18:22:09.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-10T18:22:09.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-10T18:22:09.161 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:08 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:22:08] ENGINE Serving on http://:::9283 2024-09-10T18:22:09.162 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:08 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:22:08] ENGINE Bus STARTED 2024-09-10T18:22:09.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: Standby manager daemon x restarted 2024-09-10T18:22:09.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: Standby manager daemon x started 2024-09-10T18:22:09.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.? v1:172.21.15.204:0/2016791548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T18:22:09.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.? v1:172.21.15.204:0/2016791548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T18:22:09.646 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:09.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: Standby manager daemon x restarted 2024-09-10T18:22:09.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: Standby manager daemon x started 2024-09-10T18:22:09.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.? v1:172.21.15.204:0/2016791548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T18:22:09.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.? v1:172.21.15.204:0/2016791548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T18:22:09.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.? v1:172.21.15.204:0/2016791548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T18:22:09.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.? v1:172.21.15.204:0/2016791548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T18:22:09.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: Active manager daemon y restarted 2024-09-10T18:22:09.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: Activating manager daemon y 2024-09-10T18:22:09.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: osdmap e75: 8 total, 8 up, 8 in 2024-09-10T18:22:09.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: mgrmap e18: y(active, starting, since 0.0066482s), standbys: x 2024-09-10T18:22:09.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:22:09.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:22:09.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:22:09.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T18:22:09.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T18:22:09.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:22:09.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:22:09.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:22:09.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:22:09.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:22:09.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:22:09.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:22:09.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:22:09.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T18:22:09.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T18:22:09.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T18:22:09.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: Manager daemon y is now available 2024-09-10T18:22:09.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:22:09.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:09.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:22:09.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:22:09.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:22:09.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:22:09.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:22:09.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.? v1:172.21.15.204:0/2016791548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T18:22:09.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.? v1:172.21.15.204:0/2016791548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T18:22:09.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: Active manager daemon y restarted 2024-09-10T18:22:09.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: Activating manager daemon y 2024-09-10T18:22:09.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: osdmap e75: 8 total, 8 up, 8 in 2024-09-10T18:22:09.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: mgrmap e18: y(active, starting, since 0.0066482s), standbys: x 2024-09-10T18:22:09.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:22:09.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:22:09.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:22:09.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T18:22:09.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T18:22:09.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:22:09.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:22:09.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:22:09.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:22:09.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:22:09.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:22:09.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:22:09.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:22:09.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T18:22:09.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T18:22:09.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T18:22:09.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: Manager daemon y is now available 2024-09-10T18:22:09.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:22:09.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:09.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:22:09.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:22:09.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:22:09.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:22:09.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:22:09.799 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: Standby manager daemon x restarted 2024-09-10T18:22:09.799 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: Standby manager daemon x started 2024-09-10T18:22:09.799 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.? v1:172.21.15.204:0/2016791548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T18:22:09.799 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.? v1:172.21.15.204:0/2016791548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T18:22:09.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.? v1:172.21.15.204:0/2016791548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T18:22:09.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.? v1:172.21.15.204:0/2016791548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T18:22:09.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: Active manager daemon y restarted 2024-09-10T18:22:09.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: Activating manager daemon y 2024-09-10T18:22:09.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: osdmap e75: 8 total, 8 up, 8 in 2024-09-10T18:22:09.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: mgrmap e18: y(active, starting, since 0.0066482s), standbys: x 2024-09-10T18:22:09.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:22:09.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:22:09.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:22:09.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T18:22:09.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T18:22:09.800 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:22:09.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:22:09.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:22:09.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:22:09.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:22:09.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:22:09.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:22:09.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:22:09.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T18:22:09.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T18:22:09.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T18:22:09.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: Manager daemon y is now available 2024-09-10T18:22:09.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:22:09.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:09.801 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:22:09.802 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:22:09.802 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:22:09.802 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:22:09.802 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:22:09.965 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:09.969 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:10.796 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:10 smithi204 ceph-mon[32357]: mgrmap e19: y(active, since 1.01496s), standbys: x 2024-09-10T18:22:10.796 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:10 smithi204 ceph-mon[32357]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:22:10.796 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:10 smithi204 ceph-mon[32357]: osdmap e76: 8 total, 8 up, 8 in 2024-09-10T18:22:10.990 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:10 smithi028 ceph-mon[29291]: mgrmap e19: y(active, since 1.01496s), standbys: x 2024-09-10T18:22:10.990 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:10 smithi028 ceph-mon[29291]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:22:10.990 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:10 smithi028 ceph-mon[29291]: osdmap e76: 8 total, 8 up, 8 in 2024-09-10T18:22:10.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:10 smithi028 ceph-mon[39220]: mgrmap e19: y(active, since 1.01496s), standbys: x 2024-09-10T18:22:10.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:10 smithi028 ceph-mon[39220]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:22:10.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:10 smithi028 ceph-mon[39220]: osdmap e76: 8 total, 8 up, 8 in 2024-09-10T18:22:11.695 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:11.465+0000 7f2dc65e0640 -1 log_channel(cephadm) log [ERR] : [10/Sep/2024:18:22:11] ENGINE Error in HTTPServer.serve 2024-09-10T18:22:11.696 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: Traceback (most recent call last): 2024-09-10T18:22:11.696 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:22:11.696 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: self._connections.run(self.expiration_interval) 2024-09-10T18:22:11.696 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:22:11.696 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: self._run(expiration_interval) 2024-09-10T18:22:11.696 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:22:11.696 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:22:11.696 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:22:11.696 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:22:11.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:22:11.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: s = self.context.wrap_socket( 2024-09-10T18:22:11.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:22:11.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: return self.sslsocket_class._create( 2024-09-10T18:22:11.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:22:11.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: self.do_handshake() 2024-09-10T18:22:11.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:22:11.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: self._sslobj.do_handshake() 2024-09-10T18:22:11.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:22:11.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:22:11.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:11 smithi204 ceph-mon[32357]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:22:11.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:11 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:11.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:11 smithi204 ceph-mon[32357]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-10T18:22:11.962 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-mon[39220]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:22:11.962 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:11.962 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-mon[39220]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-10T18:22:11.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-mon[29291]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:22:11.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:11.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:11 smithi028 ceph-mon[29291]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-10T18:22:12.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:22:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:22:12.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:22:12.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:22:12.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: [10/Sep/2024:18:22:11] ENGINE Bus STARTING 2024-09-10T18:22:12.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: [10/Sep/2024:18:22:11] ENGINE Serving on https://172.21.15.28:7150 2024-09-10T18:22:12.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: [10/Sep/2024:18:22:11] ENGINE Bus STARTED 2024-09-10T18:22:12.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: [10/Sep/2024:18:22:11] ENGINE Error in HTTPServer.serve 2024-09-10T18:22:12.936 INFO:journalctl@ceph.mon.c.smithi028.stdout: Traceback (most recent call last): 2024-09-10T18:22:12.936 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:22:12.936 INFO:journalctl@ceph.mon.c.smithi028.stdout: self._connections.run(self.expiration_interval) 2024-09-10T18:22:12.936 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:22:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout: self._run(expiration_interval) 2024-09-10T18:22:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:22:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:22:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:22:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:22:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:22:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout: s = self.context.wrap_socket( 2024-09-10T18:22:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:22:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout: return self.sslsocket_class._create( 2024-09-10T18:22:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:22:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout: self.do_handshake() 2024-09-10T18:22:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:22:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout: self._sslobj.do_handshake() 2024-09-10T18:22:12.937 INFO:journalctl@ceph.mon.c.smithi028.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:22:12.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: osdmap e77: 8 total, 8 up, 8 in 2024-09-10T18:22:12.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:12.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:12.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:12.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:12.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:12.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:12.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:12.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:12.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:12.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:12.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:12.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:12.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:12.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:22:12.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:22:12.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: [10/Sep/2024:18:22:11] ENGINE Bus STARTING 2024-09-10T18:22:12.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: [10/Sep/2024:18:22:11] ENGINE Serving on https://172.21.15.28:7150 2024-09-10T18:22:12.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: [10/Sep/2024:18:22:11] ENGINE Bus STARTED 2024-09-10T18:22:12.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: [10/Sep/2024:18:22:11] ENGINE Error in HTTPServer.serve 2024-09-10T18:22:12.939 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-09-10T18:22:12.939 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:22:12.939 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._connections.run(self.expiration_interval) 2024-09-10T18:22:12.939 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:22:12.940 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._run(expiration_interval) 2024-09-10T18:22:12.940 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:22:12.940 INFO:journalctl@ceph.mon.a.smithi028.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:22:12.940 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:22:12.940 INFO:journalctl@ceph.mon.a.smithi028.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:22:12.940 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:22:12.940 INFO:journalctl@ceph.mon.a.smithi028.stdout: s = self.context.wrap_socket( 2024-09-10T18:22:12.940 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:22:12.940 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.sslsocket_class._create( 2024-09-10T18:22:12.940 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:22:12.940 INFO:journalctl@ceph.mon.a.smithi028.stdout: self.do_handshake() 2024-09-10T18:22:12.940 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:22:12.940 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._sslobj.do_handshake() 2024-09-10T18:22:12.940 INFO:journalctl@ceph.mon.a.smithi028.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:22:12.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: osdmap e77: 8 total, 8 up, 8 in 2024-09-10T18:22:12.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:12.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:12.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:12.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:12.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:12.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:12.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:12.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:12.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:12.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:12.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:12.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:12.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:12 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:13.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:22:13.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:22:13.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: [10/Sep/2024:18:22:11] ENGINE Bus STARTING 2024-09-10T18:22:13.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: [10/Sep/2024:18:22:11] ENGINE Serving on https://172.21.15.28:7150 2024-09-10T18:22:13.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: [10/Sep/2024:18:22:11] ENGINE Bus STARTED 2024-09-10T18:22:13.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: [10/Sep/2024:18:22:11] ENGINE Error in HTTPServer.serve 2024-09-10T18:22:13.105 INFO:journalctl@ceph.mon.b.smithi204.stdout: Traceback (most recent call last): 2024-09-10T18:22:13.105 INFO:journalctl@ceph.mon.b.smithi204.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:22:13.105 INFO:journalctl@ceph.mon.b.smithi204.stdout: self._connections.run(self.expiration_interval) 2024-09-10T18:22:13.105 INFO:journalctl@ceph.mon.b.smithi204.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:22:13.105 INFO:journalctl@ceph.mon.b.smithi204.stdout: self._run(expiration_interval) 2024-09-10T18:22:13.105 INFO:journalctl@ceph.mon.b.smithi204.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:22:13.105 INFO:journalctl@ceph.mon.b.smithi204.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:22:13.106 INFO:journalctl@ceph.mon.b.smithi204.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:22:13.106 INFO:journalctl@ceph.mon.b.smithi204.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:22:13.106 INFO:journalctl@ceph.mon.b.smithi204.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:22:13.106 INFO:journalctl@ceph.mon.b.smithi204.stdout: s = self.context.wrap_socket( 2024-09-10T18:22:13.106 INFO:journalctl@ceph.mon.b.smithi204.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:22:13.106 INFO:journalctl@ceph.mon.b.smithi204.stdout: return self.sslsocket_class._create( 2024-09-10T18:22:13.106 INFO:journalctl@ceph.mon.b.smithi204.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:22:13.106 INFO:journalctl@ceph.mon.b.smithi204.stdout: self.do_handshake() 2024-09-10T18:22:13.106 INFO:journalctl@ceph.mon.b.smithi204.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:22:13.106 INFO:journalctl@ceph.mon.b.smithi204.stdout: self._sslobj.do_handshake() 2024-09-10T18:22:13.106 INFO:journalctl@ceph.mon.b.smithi204.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:22:13.106 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: osdmap e77: 8 total, 8 up, 8 in 2024-09-10T18:22:13.106 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:13.107 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:13.107 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:13.107 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:13.107 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:13.107 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:13.107 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:13.107 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:13.107 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:13.107 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:13.107 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:13.107 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:13.107 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:12 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:13.696 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:13.698 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:13.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:13 smithi028 ceph-mon[29291]: Adjusting osd_memory_target on smithi204 to 3760M 2024-09-10T18:22:13.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:13 smithi028 ceph-mon[29291]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:22:13.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:13 smithi028 ceph-mon[29291]: mgrmap e21: y(active, since 4s), standbys: x 2024-09-10T18:22:13.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:13 smithi028 ceph-mon[39220]: Adjusting osd_memory_target on smithi204 to 3760M 2024-09-10T18:22:13.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:13 smithi028 ceph-mon[39220]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:22:13.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:13 smithi028 ceph-mon[39220]: mgrmap e21: y(active, since 4s), standbys: x 2024-09-10T18:22:14.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:13 smithi204 ceph-mon[32357]: Adjusting osd_memory_target on smithi204 to 3760M 2024-09-10T18:22:14.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:13 smithi204 ceph-mon[32357]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:22:14.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:13 smithi204 ceph-mon[32357]: mgrmap e21: y(active, since 4s), standbys: x 2024-09-10T18:22:14.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:22:13 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:22:14.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:14 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:22:14.923 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:14 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:22:15.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:14 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:22:15.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:15 smithi028 ceph-mon[39220]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:22:15.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:15 smithi028 ceph-mon[39220]: osdmap e78: 8 total, 8 up, 8 in 2024-09-10T18:22:15.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:15 smithi028 ceph-mon[29291]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:22:15.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:15 smithi028 ceph-mon[29291]: osdmap e78: 8 total, 8 up, 8 in 2024-09-10T18:22:16.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:15 smithi204 ceph-mon[32357]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:22:16.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:15 smithi204 ceph-mon[32357]: osdmap e78: 8 total, 8 up, 8 in 2024-09-10T18:22:17.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:17 smithi028 ceph-mon[29291]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-10T18:22:17.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:17 smithi028 ceph-mon[29291]: 1.0 deep-scrub starts 2024-09-10T18:22:17.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:17 smithi028 ceph-mon[29291]: 1.0 deep-scrub ok 2024-09-10T18:22:17.994 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:17 smithi028 ceph-mon[39220]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-10T18:22:17.994 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:17 smithi028 ceph-mon[39220]: 1.0 deep-scrub starts 2024-09-10T18:22:17.994 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:17 smithi028 ceph-mon[39220]: 1.0 deep-scrub ok 2024-09-10T18:22:18.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:17 smithi204 ceph-mon[32357]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-10T18:22:18.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:17 smithi204 ceph-mon[32357]: 1.0 deep-scrub starts 2024-09-10T18:22:18.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:17 smithi204 ceph-mon[32357]: 1.0 deep-scrub ok 2024-09-10T18:22:19.471 INFO:teuthology.orchestra.run.smithi028.stdout:115964117025 2024-09-10T18:22:19.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-10T18:22:19.990 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:19 smithi028 ceph-mon[29291]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-10T18:22:19.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:19 smithi028 ceph-mon[39220]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-10T18:22:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:19 smithi204 ceph-mon[32357]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-10T18:22:20.642 INFO:teuthology.orchestra.run.smithi028.stdout:167503724567 2024-09-10T18:22:20.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-10T18:22:20.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:20 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:22:20.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:20 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:22:20.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:20 smithi028 ceph-mon[29291]: osdmap e79: 8 total, 8 up, 8 in 2024-09-10T18:22:20.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:20 smithi028 ceph-mon[29291]: osdmap e80: 8 total, 8 up, 8 in 2024-09-10T18:22:20.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:20 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:22:20.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:20 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:22:20.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:20 smithi028 ceph-mon[39220]: osdmap e79: 8 total, 8 up, 8 in 2024-09-10T18:22:20.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:20 smithi028 ceph-mon[39220]: osdmap e80: 8 total, 8 up, 8 in 2024-09-10T18:22:21.008 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:20 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:22:21.009 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:20 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:22:21.009 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:20 smithi204 ceph-mon[32357]: osdmap e79: 8 total, 8 up, 8 in 2024-09-10T18:22:21.009 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:20 smithi204 ceph-mon[32357]: osdmap e80: 8 total, 8 up, 8 in 2024-09-10T18:22:22.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:21 smithi028 ceph-mon[29291]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-10T18:22:22.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:21 smithi028 ceph-mon[39220]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-10T18:22:22.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:21 smithi204 ceph-mon[32357]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-10T18:22:22.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:22:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:22:23.134 INFO:teuthology.orchestra.run.smithi028.stdout:197568495634 2024-09-10T18:22:23.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-10T18:22:24.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:23 smithi028 ceph-mon[29291]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 713 B/s rd, 0 op/s 2024-09-10T18:22:24.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:23 smithi028 ceph-mon[39220]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 713 B/s rd, 0 op/s 2024-09-10T18:22:24.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:23 smithi204 ceph-mon[32357]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 713 B/s rd, 0 op/s 2024-09-10T18:22:24.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:22:23 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:22:24.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:22:24.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:24 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:22:24.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:22:24.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:24 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:22:25.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:22:25.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:24 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:22:25.990 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:25 smithi028 ceph-mon[29291]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:22:25.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:25 smithi028 ceph-mon[39220]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:22:26.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:25 smithi204 ceph-mon[32357]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:22:26.424 INFO:teuthology.orchestra.run.smithi028.stdout:77309411367 2024-09-10T18:22:26.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-10T18:22:27.322 INFO:teuthology.orchestra.run.smithi028.stdout:141733920797 2024-09-10T18:22:27.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-10T18:22:27.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:27 smithi028 ceph-mon[29291]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:22:27.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:27 smithi028 ceph-mon[39220]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:22:27.940 INFO:teuthology.orchestra.run.smithi028.stdout:231928233997 2024-09-10T18:22:27.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-10T18:22:28.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:27 smithi204 ceph-mon[32357]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:22:28.259 INFO:teuthology.orchestra.run.smithi028.stdout:55834574892 2024-09-10T18:22:28.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-10T18:22:28.765 INFO:teuthology.orchestra.run.smithi028.stdout:34359738416 2024-09-10T18:22:28.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-10T18:22:28.855 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:28.855 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:29.159 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:30.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:29 smithi028 ceph-mon[29291]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-09-10T18:22:30.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:29 smithi028 ceph-mon[39220]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-09-10T18:22:30.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:29 smithi204 ceph-mon[32357]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-09-10T18:22:31.719 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:31 smithi028 ceph-mon[29291]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-09-10T18:22:31.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:31 smithi028 ceph-mon[39220]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-09-10T18:22:31.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:31 smithi204 ceph-mon[32357]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-09-10T18:22:32.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:22:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:22:34.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:33 smithi028 ceph-mon[29291]: 3.d scrub starts 2024-09-10T18:22:34.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:33 smithi028 ceph-mon[29291]: 3.d scrub ok 2024-09-10T18:22:34.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:33 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:34.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:33 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:34.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:33 smithi028 ceph-mon[29291]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:22:34.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:33 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:22:34.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:33 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:22:34.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:33 smithi028 ceph-mon[39220]: 3.d scrub starts 2024-09-10T18:22:34.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:33 smithi028 ceph-mon[39220]: 3.d scrub ok 2024-09-10T18:22:34.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:33 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:34.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:33 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:34.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:33 smithi028 ceph-mon[39220]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:22:34.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:33 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:22:34.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:33 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:22:34.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:22:33 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:22:34.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:33 smithi204 ceph-mon[32357]: 3.d scrub starts 2024-09-10T18:22:34.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:33 smithi204 ceph-mon[32357]: 3.d scrub ok 2024-09-10T18:22:34.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:33 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:34.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:33 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:34.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:33 smithi204 ceph-mon[32357]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:22:34.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:33 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:22:34.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:33 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:22:34.715 INFO:teuthology.orchestra.run.smithi028.stdout:141733920798 2024-09-10T18:22:34.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:34 smithi028 ceph-mon[39220]: 4.14 scrub starts 2024-09-10T18:22:34.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:34 smithi028 ceph-mon[39220]: 4.14 scrub ok 2024-09-10T18:22:34.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:34 smithi028 ceph-mon[39220]: osdmap e81: 8 total, 8 up, 8 in 2024-09-10T18:22:34.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:34 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:22:34.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:34 smithi028 ceph-mon[29291]: 4.14 scrub starts 2024-09-10T18:22:34.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:34 smithi028 ceph-mon[29291]: 4.14 scrub ok 2024-09-10T18:22:34.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:34 smithi028 ceph-mon[29291]: osdmap e81: 8 total, 8 up, 8 in 2024-09-10T18:22:34.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:34 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:22:35.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:34 smithi204 ceph-mon[32357]: 4.14 scrub starts 2024-09-10T18:22:35.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:34 smithi204 ceph-mon[32357]: 4.14 scrub ok 2024-09-10T18:22:35.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:34 smithi204 ceph-mon[32357]: osdmap e81: 8 total, 8 up, 8 in 2024-09-10T18:22:35.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:34 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:22:35.908 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:35 smithi028 ceph-mon[29291]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T18:22:35.908 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:35 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3914203279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T18:22:35.908 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:35 smithi028 ceph-mon[29291]: osdmap e82: 8 total, 8 up, 8 in 2024-09-10T18:22:35.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:35 smithi028 ceph-mon[39220]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T18:22:35.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:35 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3914203279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T18:22:35.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:35 smithi028 ceph-mon[39220]: osdmap e82: 8 total, 8 up, 8 in 2024-09-10T18:22:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:35 smithi204 ceph-mon[32357]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T18:22:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:35 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3914203279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T18:22:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:35 smithi204 ceph-mon[32357]: osdmap e82: 8 total, 8 up, 8 in 2024-09-10T18:22:36.293 INFO:teuthology.orchestra.run.smithi028.stdout:115964117028 2024-09-10T18:22:36.586 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920797 got 141733920798 for osd.4 2024-09-10T18:22:36.587 DEBUG:teuthology.parallel:result is None 2024-09-10T18:22:37.086 INFO:teuthology.orchestra.run.smithi028.stdout:167503724571 2024-09-10T18:22:37.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:37.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:37.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: Adjusting osd_memory_target on smithi028 to 3761M 2024-09-10T18:22:37.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:37.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:22:37.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:22:37.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-10T18:22:37.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: Updating smithi204:/etc/ceph/ceph.conf 2024-09-10T18:22:37.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/443123089' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T18:22:37.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:37.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:37.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:37.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:37.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: Adjusting osd_memory_target on smithi028 to 3761M 2024-09-10T18:22:37.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:37.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:22:37.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:22:37.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-10T18:22:37.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: Updating smithi204:/etc/ceph/ceph.conf 2024-09-10T18:22:37.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/443123089' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T18:22:37.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:37.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:36 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:22:37.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: Adjusting osd_memory_target on smithi028 to 3761M 2024-09-10T18:22:37.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:37.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:22:37.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:22:37.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-10T18:22:37.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: Updating smithi204:/etc/ceph/ceph.conf 2024-09-10T18:22:37.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/443123089' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T18:22:37.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:37.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:36 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:38.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[29291]: 5.12 scrub starts 2024-09-10T18:22:38.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[29291]: 5.12 scrub ok 2024-09-10T18:22:38.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[29291]: Updating smithi028:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:22:38.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[29291]: Updating smithi204:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:22:38.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[29291]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:22:38.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[29291]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:22:38.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[29291]: Updating smithi204:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:22:38.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[29291]: Updating smithi028:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:22:38.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[29291]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:22:38.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:38.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:38.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:38.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[29291]: Deploying daemon alertmanager.a on smithi028 2024-09-10T18:22:38.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1009653420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T18:22:38.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[39220]: 5.12 scrub starts 2024-09-10T18:22:38.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[39220]: 5.12 scrub ok 2024-09-10T18:22:38.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[39220]: Updating smithi028:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:22:38.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[39220]: Updating smithi204:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:22:38.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[39220]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:22:38.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[39220]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:22:38.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[39220]: Updating smithi204:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:22:38.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[39220]: Updating smithi028:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:22:38.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[39220]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:22:38.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:38.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:38.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:38.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[39220]: Deploying daemon alertmanager.a on smithi028 2024-09-10T18:22:38.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:37 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1009653420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T18:22:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:37 smithi204 ceph-mon[32357]: 5.12 scrub starts 2024-09-10T18:22:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:37 smithi204 ceph-mon[32357]: 5.12 scrub ok 2024-09-10T18:22:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:37 smithi204 ceph-mon[32357]: Updating smithi028:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:22:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:37 smithi204 ceph-mon[32357]: Updating smithi204:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:22:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:37 smithi204 ceph-mon[32357]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:22:38.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:37 smithi204 ceph-mon[32357]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:22:38.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:37 smithi204 ceph-mon[32357]: Updating smithi204:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:22:38.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:37 smithi204 ceph-mon[32357]: Updating smithi028:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:22:38.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:37 smithi204 ceph-mon[32357]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:22:38.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:37 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:38.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:37 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:38.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:37 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:38.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:37 smithi204 ceph-mon[32357]: Deploying daemon alertmanager.a on smithi028 2024-09-10T18:22:38.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:37 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1009653420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T18:22:38.370 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117025 got 115964117028 for osd.3 2024-09-10T18:22:38.370 DEBUG:teuthology.parallel:result is None 2024-09-10T18:22:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:38 smithi028 ceph-mon[29291]: 4.1 scrub starts 2024-09-10T18:22:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:38 smithi028 ceph-mon[29291]: 4.1 scrub ok 2024-09-10T18:22:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:38 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:22:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:38 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:22:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:38 smithi028 ceph-mon[29291]: osdmap e83: 8 total, 8 up, 8 in 2024-09-10T18:22:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:38 smithi028 ceph-mon[29291]: 3.12 scrub starts 2024-09-10T18:22:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:38 smithi028 ceph-mon[29291]: 3.12 scrub ok 2024-09-10T18:22:39.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:38 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:22:39.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:38 smithi028 ceph-mon[39220]: 4.1 scrub starts 2024-09-10T18:22:39.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:38 smithi028 ceph-mon[39220]: 4.1 scrub ok 2024-09-10T18:22:39.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:38 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:22:39.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:38 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:22:39.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:38 smithi028 ceph-mon[39220]: osdmap e83: 8 total, 8 up, 8 in 2024-09-10T18:22:39.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:38 smithi028 ceph-mon[39220]: 3.12 scrub starts 2024-09-10T18:22:39.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:38 smithi028 ceph-mon[39220]: 3.12 scrub ok 2024-09-10T18:22:39.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:38 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:22:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:38 smithi204 ceph-mon[32357]: 4.1 scrub starts 2024-09-10T18:22:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:38 smithi204 ceph-mon[32357]: 4.1 scrub ok 2024-09-10T18:22:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:38 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:22:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:38 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:22:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:38 smithi204 ceph-mon[32357]: osdmap e83: 8 total, 8 up, 8 in 2024-09-10T18:22:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:38 smithi204 ceph-mon[32357]: 3.12 scrub starts 2024-09-10T18:22:39.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:38 smithi204 ceph-mon[32357]: 3.12 scrub ok 2024-09-10T18:22:39.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:38 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:22:39.891 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724567 got 167503724571 for osd.5 2024-09-10T18:22:39.891 DEBUG:teuthology.parallel:result is None 2024-09-10T18:22:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:39 smithi028 ceph-mon[39220]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:22:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:39 smithi028 ceph-mon[39220]: 5.2 scrub starts 2024-09-10T18:22:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:39 smithi028 ceph-mon[39220]: 5.2 scrub ok 2024-09-10T18:22:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:39 smithi028 ceph-mon[39220]: 5.1d scrub starts 2024-09-10T18:22:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:39 smithi028 ceph-mon[39220]: 5.1d scrub ok 2024-09-10T18:22:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:39 smithi028 ceph-mon[29291]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:22:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:39 smithi028 ceph-mon[29291]: 5.2 scrub starts 2024-09-10T18:22:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:39 smithi028 ceph-mon[29291]: 5.2 scrub ok 2024-09-10T18:22:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:39 smithi028 ceph-mon[29291]: 5.1d scrub starts 2024-09-10T18:22:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:39 smithi028 ceph-mon[29291]: 5.1d scrub ok 2024-09-10T18:22:40.314 INFO:teuthology.orchestra.run.smithi028.stdout:77309411369 2024-09-10T18:22:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:39 smithi204 ceph-mon[32357]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:22:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:39 smithi204 ceph-mon[32357]: 5.2 scrub starts 2024-09-10T18:22:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:39 smithi204 ceph-mon[32357]: 5.2 scrub ok 2024-09-10T18:22:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:39 smithi204 ceph-mon[32357]: 5.1d scrub starts 2024-09-10T18:22:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:39 smithi204 ceph-mon[32357]: 5.1d scrub ok 2024-09-10T18:22:40.451 INFO:teuthology.orchestra.run.smithi028.stdout:55834574894 2024-09-10T18:22:40.476 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:40.477 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:41.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:40 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1611359052' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T18:22:41.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:40 smithi028 ceph-mon[29291]: 4.f deep-scrub starts 2024-09-10T18:22:41.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:40 smithi028 ceph-mon[29291]: 4.f deep-scrub ok 2024-09-10T18:22:41.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:40 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2169070166' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T18:22:41.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:40 smithi028 ceph-mon[29291]: osdmap e84: 8 total, 8 up, 8 in 2024-09-10T18:22:41.096 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:40 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1611359052' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T18:22:41.096 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:40 smithi028 ceph-mon[39220]: 4.f deep-scrub starts 2024-09-10T18:22:41.096 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:40 smithi028 ceph-mon[39220]: 4.f deep-scrub ok 2024-09-10T18:22:41.096 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:40 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2169070166' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T18:22:41.096 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:40 smithi028 ceph-mon[39220]: osdmap e84: 8 total, 8 up, 8 in 2024-09-10T18:22:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:40 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1611359052' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T18:22:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:40 smithi204 ceph-mon[32357]: 4.f deep-scrub starts 2024-09-10T18:22:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:40 smithi204 ceph-mon[32357]: 4.f deep-scrub ok 2024-09-10T18:22:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:40 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2169070166' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T18:22:41.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:40 smithi204 ceph-mon[32357]: osdmap e84: 8 total, 8 up, 8 in 2024-09-10T18:22:42.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:41 smithi028 ceph-mon[29291]: 5.8 scrub starts 2024-09-10T18:22:42.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:41 smithi028 ceph-mon[29291]: 5.8 scrub ok 2024-09-10T18:22:42.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:41 smithi028 ceph-mon[29291]: 3.16 deep-scrub starts 2024-09-10T18:22:42.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:41 smithi028 ceph-mon[29291]: 3.16 deep-scrub ok 2024-09-10T18:22:42.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:41 smithi028 ceph-mon[29291]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T18:22:42.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:41 smithi028 ceph-mon[29291]: 6.0 scrub starts 2024-09-10T18:22:42.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:41 smithi028 ceph-mon[29291]: 6.0 scrub ok 2024-09-10T18:22:42.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:41 smithi028 ceph-mon[39220]: 5.8 scrub starts 2024-09-10T18:22:42.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:41 smithi028 ceph-mon[39220]: 5.8 scrub ok 2024-09-10T18:22:42.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:41 smithi028 ceph-mon[39220]: 3.16 deep-scrub starts 2024-09-10T18:22:42.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:41 smithi028 ceph-mon[39220]: 3.16 deep-scrub ok 2024-09-10T18:22:42.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:41 smithi028 ceph-mon[39220]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T18:22:42.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:41 smithi028 ceph-mon[39220]: 6.0 scrub starts 2024-09-10T18:22:42.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:41 smithi028 ceph-mon[39220]: 6.0 scrub ok 2024-09-10T18:22:42.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:22:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:22:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:41 smithi204 ceph-mon[32357]: 5.8 scrub starts 2024-09-10T18:22:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:41 smithi204 ceph-mon[32357]: 5.8 scrub ok 2024-09-10T18:22:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:41 smithi204 ceph-mon[32357]: 3.16 deep-scrub starts 2024-09-10T18:22:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:41 smithi204 ceph-mon[32357]: 3.16 deep-scrub ok 2024-09-10T18:22:42.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:41 smithi204 ceph-mon[32357]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T18:22:42.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:41 smithi204 ceph-mon[32357]: 6.0 scrub starts 2024-09-10T18:22:42.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:41 smithi204 ceph-mon[32357]: 6.0 scrub ok 2024-09-10T18:22:42.378 INFO:teuthology.orchestra.run.smithi028.stdout:231928234001 2024-09-10T18:22:43.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:42 smithi028 ceph-mon[39220]: 2.0 scrub starts 2024-09-10T18:22:43.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:42 smithi028 ceph-mon[39220]: 2.0 scrub ok 2024-09-10T18:22:43.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:42 smithi028 ceph-mon[39220]: 5.a scrub starts 2024-09-10T18:22:43.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:42 smithi028 ceph-mon[39220]: 5.a scrub ok 2024-09-10T18:22:43.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:42 smithi028 ceph-mon[39220]: 4.12 scrub starts 2024-09-10T18:22:43.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:42 smithi028 ceph-mon[39220]: 4.12 scrub ok 2024-09-10T18:22:43.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:42 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2034607298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T18:22:43.242 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:42 smithi028 ceph-mon[29291]: 2.0 scrub starts 2024-09-10T18:22:43.242 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:42 smithi028 ceph-mon[29291]: 2.0 scrub ok 2024-09-10T18:22:43.242 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:42 smithi028 ceph-mon[29291]: 5.a scrub starts 2024-09-10T18:22:43.242 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:42 smithi028 ceph-mon[29291]: 5.a scrub ok 2024-09-10T18:22:43.242 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:42 smithi028 ceph-mon[29291]: 4.12 scrub starts 2024-09-10T18:22:43.242 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:42 smithi028 ceph-mon[29291]: 4.12 scrub ok 2024-09-10T18:22:43.242 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:42 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2034607298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T18:22:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:42 smithi204 ceph-mon[32357]: 2.0 scrub starts 2024-09-10T18:22:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:42 smithi204 ceph-mon[32357]: 2.0 scrub ok 2024-09-10T18:22:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:42 smithi204 ceph-mon[32357]: 5.a scrub starts 2024-09-10T18:22:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:42 smithi204 ceph-mon[32357]: 5.a scrub ok 2024-09-10T18:22:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:42 smithi204 ceph-mon[32357]: 4.12 scrub starts 2024-09-10T18:22:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:42 smithi204 ceph-mon[32357]: 4.12 scrub ok 2024-09-10T18:22:43.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:42 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2034607298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T18:22:43.479 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574894 for osd.1 2024-09-10T18:22:43.479 DEBUG:teuthology.parallel:result is None 2024-09-10T18:22:44.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:43 smithi028 ceph-mon[29291]: 5.b scrub starts 2024-09-10T18:22:44.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:43 smithi028 ceph-mon[29291]: 5.b scrub ok 2024-09-10T18:22:44.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:43 smithi028 ceph-mon[29291]: 3.3 scrub starts 2024-09-10T18:22:44.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:43 smithi028 ceph-mon[29291]: 3.3 scrub ok 2024-09-10T18:22:44.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:43 smithi028 ceph-mon[29291]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 713 B/s rd, 0 op/s 2024-09-10T18:22:44.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:43 smithi028 ceph-mon[29291]: 5.5 scrub starts 2024-09-10T18:22:44.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:43 smithi028 ceph-mon[29291]: 5.5 scrub ok 2024-09-10T18:22:44.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:43 smithi028 ceph-mon[39220]: 5.b scrub starts 2024-09-10T18:22:44.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:43 smithi028 ceph-mon[39220]: 5.b scrub ok 2024-09-10T18:22:44.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:43 smithi028 ceph-mon[39220]: 3.3 scrub starts 2024-09-10T18:22:44.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:43 smithi028 ceph-mon[39220]: 3.3 scrub ok 2024-09-10T18:22:44.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:43 smithi028 ceph-mon[39220]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 713 B/s rd, 0 op/s 2024-09-10T18:22:44.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:43 smithi028 ceph-mon[39220]: 5.5 scrub starts 2024-09-10T18:22:44.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:43 smithi028 ceph-mon[39220]: 5.5 scrub ok 2024-09-10T18:22:44.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:22:43 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:22:44.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:43 smithi204 ceph-mon[32357]: 5.b scrub starts 2024-09-10T18:22:44.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:43 smithi204 ceph-mon[32357]: 5.b scrub ok 2024-09-10T18:22:44.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:43 smithi204 ceph-mon[32357]: 3.3 scrub starts 2024-09-10T18:22:44.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:43 smithi204 ceph-mon[32357]: 3.3 scrub ok 2024-09-10T18:22:44.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:43 smithi204 ceph-mon[32357]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 713 B/s rd, 0 op/s 2024-09-10T18:22:44.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:43 smithi204 ceph-mon[32357]: 5.5 scrub starts 2024-09-10T18:22:44.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:43 smithi204 ceph-mon[32357]: 5.5 scrub ok 2024-09-10T18:22:45.240 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:44 smithi028 ceph-mon[39220]: 3.15 scrub starts 2024-09-10T18:22:45.240 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:44 smithi028 ceph-mon[39220]: 3.15 scrub ok 2024-09-10T18:22:45.240 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:44 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:22:45.240 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:44 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:22:45.240 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:44 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:22:45.240 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:44 smithi028 ceph-mon[39220]: osdmap e85: 8 total, 8 up, 8 in 2024-09-10T18:22:45.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:44 smithi028 ceph-mon[29291]: 3.15 scrub starts 2024-09-10T18:22:45.241 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:44 smithi028 ceph-mon[29291]: 3.15 scrub ok 2024-09-10T18:22:45.241 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:44 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:22:45.241 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:44 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:22:45.241 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:44 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:22:45.241 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:44 smithi028 ceph-mon[29291]: osdmap e85: 8 total, 8 up, 8 in 2024-09-10T18:22:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:45 smithi204 ceph-mon[32357]: 3.15 scrub starts 2024-09-10T18:22:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:45 smithi204 ceph-mon[32357]: 3.15 scrub ok 2024-09-10T18:22:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:45 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:22:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:45 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:22:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:45 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:22:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:45 smithi204 ceph-mon[32357]: osdmap e85: 8 total, 8 up, 8 in 2024-09-10T18:22:45.741 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411369 for osd.2 2024-09-10T18:22:45.742 DEBUG:teuthology.parallel:result is None 2024-09-10T18:22:45.867 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233997 got 231928234001 for osd.7 2024-09-10T18:22:45.868 DEBUG:teuthology.parallel:result is None 2024-09-10T18:22:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:45 smithi028 ceph-mon[29291]: 6.1d scrub starts 2024-09-10T18:22:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:45 smithi028 ceph-mon[29291]: 6.1d scrub ok 2024-09-10T18:22:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:45 smithi028 ceph-mon[29291]: 4.1b deep-scrub starts 2024-09-10T18:22:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:45 smithi028 ceph-mon[29291]: 4.1b deep-scrub ok 2024-09-10T18:22:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:45 smithi028 ceph-mon[29291]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 761 B/s rd, 0 op/s 2024-09-10T18:22:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:45 smithi028 ceph-mon[29291]: osdmap e86: 8 total, 8 up, 8 in 2024-09-10T18:22:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:45 smithi028 ceph-mon[39220]: 6.1d scrub starts 2024-09-10T18:22:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:45 smithi028 ceph-mon[39220]: 6.1d scrub ok 2024-09-10T18:22:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:45 smithi028 ceph-mon[39220]: 4.1b deep-scrub starts 2024-09-10T18:22:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:45 smithi028 ceph-mon[39220]: 4.1b deep-scrub ok 2024-09-10T18:22:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:45 smithi028 ceph-mon[39220]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 761 B/s rd, 0 op/s 2024-09-10T18:22:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:45 smithi028 ceph-mon[39220]: osdmap e86: 8 total, 8 up, 8 in 2024-09-10T18:22:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:46 smithi204 ceph-mon[32357]: 6.1d scrub starts 2024-09-10T18:22:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:46 smithi204 ceph-mon[32357]: 6.1d scrub ok 2024-09-10T18:22:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:46 smithi204 ceph-mon[32357]: 4.1b deep-scrub starts 2024-09-10T18:22:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:46 smithi204 ceph-mon[32357]: 4.1b deep-scrub ok 2024-09-10T18:22:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:46 smithi204 ceph-mon[32357]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 761 B/s rd, 0 op/s 2024-09-10T18:22:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:46 smithi204 ceph-mon[32357]: osdmap e86: 8 total, 8 up, 8 in 2024-09-10T18:22:47.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:47 smithi028 ceph-mon[29291]: 3.2 scrub starts 2024-09-10T18:22:47.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:47 smithi028 ceph-mon[29291]: 3.2 scrub ok 2024-09-10T18:22:47.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:47 smithi028 ceph-mon[29291]: 6.18 scrub starts 2024-09-10T18:22:47.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:47 smithi028 ceph-mon[29291]: 6.18 scrub ok 2024-09-10T18:22:47.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:47 smithi028 ceph-mon[39220]: 3.2 scrub starts 2024-09-10T18:22:47.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:47 smithi028 ceph-mon[39220]: 3.2 scrub ok 2024-09-10T18:22:47.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:47 smithi028 ceph-mon[39220]: 6.18 scrub starts 2024-09-10T18:22:47.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:47 smithi028 ceph-mon[39220]: 6.18 scrub ok 2024-09-10T18:22:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:47 smithi204 ceph-mon[32357]: 3.2 scrub starts 2024-09-10T18:22:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:47 smithi204 ceph-mon[32357]: 3.2 scrub ok 2024-09-10T18:22:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:47 smithi204 ceph-mon[32357]: 6.18 scrub starts 2024-09-10T18:22:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:47 smithi204 ceph-mon[32357]: 6.18 scrub ok 2024-09-10T18:22:48.008 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:48 smithi028 ceph-mon[39220]: 3.19 scrub starts 2024-09-10T18:22:48.008 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:48 smithi028 ceph-mon[39220]: 3.19 scrub ok 2024-09-10T18:22:48.008 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:48 smithi028 ceph-mon[39220]: 2.1 scrub starts 2024-09-10T18:22:48.008 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:48 smithi028 ceph-mon[39220]: 2.1 scrub ok 2024-09-10T18:22:48.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:48 smithi028 ceph-mon[29291]: 3.19 scrub starts 2024-09-10T18:22:48.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:48 smithi028 ceph-mon[29291]: 3.19 scrub ok 2024-09-10T18:22:48.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:48 smithi028 ceph-mon[29291]: 2.1 scrub starts 2024-09-10T18:22:48.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:48 smithi028 ceph-mon[29291]: 2.1 scrub ok 2024-09-10T18:22:48.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:48 smithi028 ceph-mon[29291]: 6.13 scrub starts 2024-09-10T18:22:48.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:48 smithi028 ceph-mon[29291]: 6.13 scrub ok 2024-09-10T18:22:48.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:48 smithi028 ceph-mon[29291]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 829 B/s rd, 0 op/s 2024-09-10T18:22:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:48 smithi028 ceph-mon[39220]: 6.13 scrub starts 2024-09-10T18:22:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:48 smithi028 ceph-mon[39220]: 6.13 scrub ok 2024-09-10T18:22:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:48 smithi028 ceph-mon[39220]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 829 B/s rd, 0 op/s 2024-09-10T18:22:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:48 smithi204 ceph-mon[32357]: 3.19 scrub starts 2024-09-10T18:22:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:48 smithi204 ceph-mon[32357]: 3.19 scrub ok 2024-09-10T18:22:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:48 smithi204 ceph-mon[32357]: 2.1 scrub starts 2024-09-10T18:22:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:48 smithi204 ceph-mon[32357]: 2.1 scrub ok 2024-09-10T18:22:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:48 smithi204 ceph-mon[32357]: 6.13 scrub starts 2024-09-10T18:22:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:48 smithi204 ceph-mon[32357]: 6.13 scrub ok 2024-09-10T18:22:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:48 smithi204 ceph-mon[32357]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 829 B/s rd, 0 op/s 2024-09-10T18:22:49.039 INFO:teuthology.orchestra.run.smithi028.stdout:34359738421 2024-09-10T18:22:49.086 INFO:teuthology.orchestra.run.smithi028.stdout:197568495640 2024-09-10T18:22:49.087 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:49 smithi028 ceph-mon[39220]: 5.1 scrub starts 2024-09-10T18:22:49.087 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:49 smithi028 ceph-mon[39220]: 5.1 scrub ok 2024-09-10T18:22:49.087 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:49 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1753649254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T18:22:49.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:49 smithi028 ceph-mon[29291]: 5.1 scrub starts 2024-09-10T18:22:49.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:49 smithi028 ceph-mon[29291]: 5.1 scrub ok 2024-09-10T18:22:49.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:49 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1753649254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T18:22:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:49 smithi204 ceph-mon[32357]: 5.1 scrub starts 2024-09-10T18:22:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:49 smithi204 ceph-mon[32357]: 5.1 scrub ok 2024-09-10T18:22:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:49 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1753649254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T18:22:49.907 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495634 got 197568495640 for osd.6 2024-09-10T18:22:49.907 DEBUG:teuthology.parallel:result is None 2024-09-10T18:22:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[39220]: 5.14 scrub starts 2024-09-10T18:22:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[39220]: 5.14 scrub ok 2024-09-10T18:22:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[39220]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:22:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[39220]: 5.1f scrub starts 2024-09-10T18:22:50.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[39220]: 5.1f scrub ok 2024-09-10T18:22:50.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3289271913' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T18:22:50.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[39220]: 4.1e deep-scrub starts 2024-09-10T18:22:50.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[39220]: 4.1e deep-scrub ok 2024-09-10T18:22:50.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[39220]: 5.11 scrub starts 2024-09-10T18:22:50.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[39220]: 5.11 scrub ok 2024-09-10T18:22:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[29291]: 5.14 scrub starts 2024-09-10T18:22:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[29291]: 5.14 scrub ok 2024-09-10T18:22:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[29291]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:22:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[29291]: 5.1f scrub starts 2024-09-10T18:22:50.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[29291]: 5.1f scrub ok 2024-09-10T18:22:50.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3289271913' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T18:22:50.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[29291]: 4.1e deep-scrub starts 2024-09-10T18:22:50.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[29291]: 4.1e deep-scrub ok 2024-09-10T18:22:50.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[29291]: 5.11 scrub starts 2024-09-10T18:22:50.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:50 smithi028 ceph-mon[29291]: 5.11 scrub ok 2024-09-10T18:22:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:50 smithi204 ceph-mon[32357]: 5.14 scrub starts 2024-09-10T18:22:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:50 smithi204 ceph-mon[32357]: 5.14 scrub ok 2024-09-10T18:22:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:50 smithi204 ceph-mon[32357]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:22:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:50 smithi204 ceph-mon[32357]: 5.1f scrub starts 2024-09-10T18:22:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:50 smithi204 ceph-mon[32357]: 5.1f scrub ok 2024-09-10T18:22:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:50 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3289271913' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T18:22:50.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:50 smithi204 ceph-mon[32357]: 4.1e deep-scrub starts 2024-09-10T18:22:50.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:50 smithi204 ceph-mon[32357]: 4.1e deep-scrub ok 2024-09-10T18:22:50.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:50 smithi204 ceph-mon[32357]: 5.11 scrub starts 2024-09-10T18:22:50.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:50 smithi204 ceph-mon[32357]: 5.11 scrub ok 2024-09-10T18:22:50.446 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738416 got 34359738421 for osd.0 2024-09-10T18:22:50.446 DEBUG:teuthology.parallel:result is None 2024-09-10T18:22:50.446 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-10T18:22:50.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:22:51.258 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:51.272 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:51 smithi204 ceph-mon[32357]: 5.17 scrub starts 2024-09-10T18:22:51.272 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:51 smithi204 ceph-mon[32357]: 5.17 scrub ok 2024-09-10T18:22:51.373 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:51 smithi028 ceph-mon[39220]: 5.17 scrub starts 2024-09-10T18:22:51.373 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:51 smithi028 ceph-mon[39220]: 5.17 scrub ok 2024-09-10T18:22:51.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:51 smithi028 ceph-mon[29291]: 5.17 scrub starts 2024-09-10T18:22:51.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:51 smithi028 ceph-mon[29291]: 5.17 scrub ok 2024-09-10T18:22:52.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:22:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:22:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:22:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:52 smithi204 ceph-mon[32357]: 6.c scrub starts 2024-09-10T18:22:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:52 smithi204 ceph-mon[32357]: 6.c scrub ok 2024-09-10T18:22:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:52 smithi204 ceph-mon[32357]: 5.1c scrub starts 2024-09-10T18:22:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:52 smithi204 ceph-mon[32357]: 5.1c scrub ok 2024-09-10T18:22:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:52 smithi204 ceph-mon[32357]: 5.15 scrub starts 2024-09-10T18:22:52.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:52 smithi204 ceph-mon[32357]: 5.15 scrub ok 2024-09-10T18:22:52.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:52 smithi204 ceph-mon[32357]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:22:52.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:52 smithi204 ceph-mon[32357]: 4.6 scrub starts 2024-09-10T18:22:52.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:52 smithi204 ceph-mon[32357]: 4.6 scrub ok 2024-09-10T18:22:52.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[39220]: 6.c scrub starts 2024-09-10T18:22:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[39220]: 6.c scrub ok 2024-09-10T18:22:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[39220]: 5.1c scrub starts 2024-09-10T18:22:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[39220]: 5.1c scrub ok 2024-09-10T18:22:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[39220]: 5.15 scrub starts 2024-09-10T18:22:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[39220]: 5.15 scrub ok 2024-09-10T18:22:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[39220]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:22:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[39220]: 4.6 scrub starts 2024-09-10T18:22:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[39220]: 4.6 scrub ok 2024-09-10T18:22:52.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[29291]: 6.c scrub starts 2024-09-10T18:22:52.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[29291]: 6.c scrub ok 2024-09-10T18:22:52.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[29291]: 5.1c scrub starts 2024-09-10T18:22:52.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[29291]: 5.1c scrub ok 2024-09-10T18:22:52.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[29291]: 5.15 scrub starts 2024-09-10T18:22:52.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[29291]: 5.15 scrub ok 2024-09-10T18:22:52.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[29291]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:22:52.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[29291]: 4.6 scrub starts 2024-09-10T18:22:52.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:52 smithi028 ceph-mon[29291]: 4.6 scrub ok 2024-09-10T18:22:53.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:53 smithi028 ceph-mon[29291]: 5.d scrub starts 2024-09-10T18:22:53.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:53 smithi028 ceph-mon[29291]: 5.d scrub ok 2024-09-10T18:22:53.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:53 smithi028 ceph-mon[29291]: 4.8 scrub starts 2024-09-10T18:22:53.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:53 smithi028 ceph-mon[29291]: 4.8 scrub ok 2024-09-10T18:22:53.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:53 smithi028 ceph-mon[29291]: 6.10 scrub starts 2024-09-10T18:22:53.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:53 smithi028 ceph-mon[29291]: 6.10 scrub ok 2024-09-10T18:22:53.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:53 smithi028 ceph-mon[39220]: 5.d scrub starts 2024-09-10T18:22:53.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:53 smithi028 ceph-mon[39220]: 5.d scrub ok 2024-09-10T18:22:53.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:53 smithi028 ceph-mon[39220]: 4.8 scrub starts 2024-09-10T18:22:53.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:53 smithi028 ceph-mon[39220]: 4.8 scrub ok 2024-09-10T18:22:53.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:53 smithi028 ceph-mon[39220]: 6.10 scrub starts 2024-09-10T18:22:53.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:53 smithi028 ceph-mon[39220]: 6.10 scrub ok 2024-09-10T18:22:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:53 smithi204 ceph-mon[32357]: 5.d scrub starts 2024-09-10T18:22:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:53 smithi204 ceph-mon[32357]: 5.d scrub ok 2024-09-10T18:22:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:53 smithi204 ceph-mon[32357]: 4.8 scrub starts 2024-09-10T18:22:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:53 smithi204 ceph-mon[32357]: 4.8 scrub ok 2024-09-10T18:22:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:53 smithi204 ceph-mon[32357]: 6.10 scrub starts 2024-09-10T18:22:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:53 smithi204 ceph-mon[32357]: 6.10 scrub ok 2024-09-10T18:22:53.558 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:22:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:22:54.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-mon[39220]: 3.0 scrub starts 2024-09-10T18:22:54.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-mon[39220]: 3.0 scrub ok 2024-09-10T18:22:54.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-mon[39220]: 6.f scrub starts 2024-09-10T18:22:54.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-mon[39220]: 6.f scrub ok 2024-09-10T18:22:54.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-mon[39220]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:22:54.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-mon[39220]: 5.1b scrub starts 2024-09-10T18:22:54.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-mon[39220]: 5.1b scrub ok 2024-09-10T18:22:54.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:22:54.177 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-mon[29291]: 3.0 scrub starts 2024-09-10T18:22:54.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-mon[29291]: 3.0 scrub ok 2024-09-10T18:22:54.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-mon[29291]: 6.f scrub starts 2024-09-10T18:22:54.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-mon[29291]: 6.f scrub ok 2024-09-10T18:22:54.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-mon[29291]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:22:54.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-mon[29291]: 5.1b scrub starts 2024-09-10T18:22:54.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-mon[29291]: 5.1b scrub ok 2024-09-10T18:22:54.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:22:54.356 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:22:53 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:22:54.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:54 smithi204 ceph-mon[32357]: 3.0 scrub starts 2024-09-10T18:22:54.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:54 smithi204 ceph-mon[32357]: 3.0 scrub ok 2024-09-10T18:22:54.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:54 smithi204 ceph-mon[32357]: 6.f scrub starts 2024-09-10T18:22:54.357 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:54 smithi204 ceph-mon[32357]: 6.f scrub ok 2024-09-10T18:22:54.357 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:54 smithi204 ceph-mon[32357]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:22:54.357 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:54 smithi204 ceph-mon[32357]: 5.1b scrub starts 2024-09-10T18:22:54.357 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:54 smithi204 ceph-mon[32357]: 5.1b scrub ok 2024-09-10T18:22:54.357 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:54 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:22:54.438 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:22:54] ENGINE Bus STOPPING 2024-09-10T18:22:54.732 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:22:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T18:22:54.733 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:22:54] ENGINE Bus STOPPED 2024-09-10T18:22:54.733 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:22:54] ENGINE Bus STARTING 2024-09-10T18:22:54.733 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:22:54] ENGINE Serving on http://:::9283 2024-09-10T18:22:54.733 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:22:54 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:22:54] ENGINE Bus STARTED 2024-09-10T18:22:54.758 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-09-10T18:22:52.700398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":924,"num_read_kb":781,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":564,"ondisk_log_size":564,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2387608,"kb_used_data":8768,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747573608,"statfs":{"total":767960285184,"available":765515374592,"internally_reserved":0,"allocated":8978432,"data_stored":4748729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.713092"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198233+0000","last_change":"2024-09-10T18:21:31.526401+0000","last_active":"2024-09-10T18:22:44.198233+0000","last_peered":"2024-09-10T18:22:44.198233+0000","last_clean":"2024-09-10T18:22:44.198233+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:22:44.198233+0000","last_undegraded":"2024-09-10T18:22:44.198233+0000","last_fullsized":"2024-09-10T18:22:44.198233+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:58.540140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.990927+0000","last_change":"2024-09-10T18:21:33.614228+0000","last_active":"2024-09-10T18:22:43.990927+0000","last_peered":"2024-09-10T18:22:43.990927+0000","last_clean":"2024-09-10T18:22:43.990927+0000","last_became_active":"2024-09-10T18:21:33.613935+0000","last_became_peered":"2024-09-10T18:21:33.613935+0000","last_unstale":"2024-09-10T18:22:43.990927+0000","last_undegraded":"2024-09-10T18:22:43.990927+0000","last_fullsized":"2024-09-10T18:22:43.990927+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:00.978849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.107309+0000","last_change":"2024-09-10T18:21:35.619936+0000","last_active":"2024-09-10T18:22:44.107309+0000","last_peered":"2024-09-10T18:22:44.107309+0000","last_clean":"2024-09-10T18:22:44.107309+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:22:44.107309+0000","last_undegraded":"2024-09-10T18:22:44.107309+0000","last_fullsized":"2024-09-10T18:22:44.107309+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:52.780616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991044+0000","last_change":"2024-09-10T18:21:37.642760+0000","last_active":"2024-09-10T18:22:43.991044+0000","last_peered":"2024-09-10T18:22:43.991044+0000","last_clean":"2024-09-10T18:22:43.991044+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:22:43.991044+0000","last_undegraded":"2024-09-10T18:22:43.991044+0000","last_fullsized":"2024-09-10T18:22:43.991044+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989014+0000","last_change":"2024-09-10T18:21:38.565483+0000","last_active":"2024-09-10T18:22:43.989014+0000","last_peered":"2024-09-10T18:22:43.989014+0000","last_clean":"2024-09-10T18:22:43.989014+0000","last_became_active":"2024-09-10T18:21:38.565118+0000","last_became_peered":"2024-09-10T18:21:38.565118+0000","last_unstale":"2024-09-10T18:22:43.989014+0000","last_undegraded":"2024-09-10T18:22:43.989014+0000","last_fullsized":"2024-09-10T18:22:43.989014+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:55.214070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989692+0000","last_change":"2024-09-10T18:21:31.528038+0000","last_active":"2024-09-10T18:22:43.989692+0000","last_peered":"2024-09-10T18:22:43.989692+0000","last_clean":"2024-09-10T18:22:43.989692+0000","last_became_active":"2024-09-10T18:21:31.527582+0000","last_became_peered":"2024-09-10T18:21:31.527582+0000","last_unstale":"2024-09-10T18:22:43.989692+0000","last_undegraded":"2024-09-10T18:22:43.989692+0000","last_fullsized":"2024-09-10T18:22:43.989692+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:51.649992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.988961+0000","last_change":"2024-09-10T18:21:33.613829+0000","last_active":"2024-09-10T18:22:43.988961+0000","last_peered":"2024-09-10T18:22:43.988961+0000","last_clean":"2024-09-10T18:22:43.988961+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:22:43.988961+0000","last_undegraded":"2024-09-10T18:22:43.988961+0000","last_fullsized":"2024-09-10T18:22:43.988961+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:55.849000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.990886+0000","last_change":"2024-09-10T18:21:35.620924+0000","last_active":"2024-09-10T18:22:43.990886+0000","last_peered":"2024-09-10T18:22:43.990886+0000","last_clean":"2024-09-10T18:22:43.990886+0000","last_became_active":"2024-09-10T18:21:35.620585+0000","last_became_peered":"2024-09-10T18:21:35.620585+0000","last_unstale":"2024-09-10T18:22:43.990886+0000","last_undegraded":"2024-09-10T18:22:43.990886+0000","last_fullsized":"2024-09-10T18:22:43.990886+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110623+0000","last_change":"2024-09-10T18:21:31.524898+0000","last_active":"2024-09-10T18:22:44.110623+0000","last_peered":"2024-09-10T18:22:44.110623+0000","last_clean":"2024-09-10T18:22:44.110623+0000","last_became_active":"2024-09-10T18:21:31.524620+0000","last_became_peered":"2024-09-10T18:21:31.524620+0000","last_unstale":"2024-09-10T18:22:44.110623+0000","last_undegraded":"2024-09-10T18:22:44.110623+0000","last_fullsized":"2024-09-10T18:22:44.110623+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:58.405475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991268+0000","last_change":"2024-09-10T18:21:33.610754+0000","last_active":"2024-09-10T18:22:43.991268+0000","last_peered":"2024-09-10T18:22:43.991268+0000","last_clean":"2024-09-10T18:22:43.991268+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:22:43.991268+0000","last_undegraded":"2024-09-10T18:22:43.991268+0000","last_fullsized":"2024-09-10T18:22:43.991268+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:51.091191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110130+0000","last_change":"2024-09-10T18:21:35.624087+0000","last_active":"2024-09-10T18:22:44.110130+0000","last_peered":"2024-09-10T18:22:44.110130+0000","last_clean":"2024-09-10T18:22:44.110130+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:22:44.110130+0000","last_undegraded":"2024-09-10T18:22:44.110130+0000","last_fullsized":"2024-09-10T18:22:44.110130+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.513248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.328351+0000","last_change":"2024-09-10T18:22:46.328322+0000","last_active":"2024-09-10T18:22:46.328351+0000","last_peered":"2024-09-10T18:22:46.328351+0000","last_clean":"2024-09-10T18:22:46.328351+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:22:46.328351+0000","last_undegraded":"2024-09-10T18:22:46.328351+0000","last_fullsized":"2024-09-10T18:22:46.328351+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:46.328262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:46.328262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:50.476851+0000","scrub_duration":0.0024608049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110305+0000","last_change":"2024-09-10T18:21:31.528616+0000","last_active":"2024-09-10T18:22:44.110305+0000","last_peered":"2024-09-10T18:22:44.110305+0000","last_clean":"2024-09-10T18:22:44.110305+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:22:44.110305+0000","last_undegraded":"2024-09-10T18:22:44.110305+0000","last_fullsized":"2024-09-10T18:22:44.110305+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.563751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"85'7","reported_seq":56,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.365355+0000","last_change":"2024-09-10T18:22:44.365273+0000","last_active":"2024-09-10T18:22:44.365355+0000","last_peered":"2024-09-10T18:22:44.365355+0000","last_clean":"2024-09-10T18:22:44.365355+0000","last_became_active":"2024-09-10T18:21:33.615429+0000","last_became_peered":"2024-09-10T18:21:33.615429+0000","last_unstale":"2024-09-10T18:22:44.365355+0000","last_undegraded":"2024-09-10T18:22:44.365355+0000","last_fullsized":"2024-09-10T18:22:44.365355+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'7","last_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:22:44.365196+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:58.822595+0000","scrub_duration":0.0033059909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198357+0000","last_change":"2024-09-10T18:21:35.624352+0000","last_active":"2024-09-10T18:22:44.198357+0000","last_peered":"2024-09-10T18:22:44.198357+0000","last_clean":"2024-09-10T18:22:44.198357+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:22:44.198357+0000","last_undegraded":"2024-09-10T18:22:44.198357+0000","last_fullsized":"2024-09-10T18:22:44.198357+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:56.347578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.109933+0000","last_change":"2024-09-10T18:21:37.639437+0000","last_active":"2024-09-10T18:22:44.109933+0000","last_peered":"2024-09-10T18:22:44.109933+0000","last_clean":"2024-09-10T18:22:44.109933+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:22:44.109933+0000","last_undegraded":"2024-09-10T18:22:44.109933+0000","last_fullsized":"2024-09-10T18:22:44.109933+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:52.337945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991686+0000","last_change":"2024-09-10T18:21:38.563964+0000","last_active":"2024-09-10T18:22:43.991686+0000","last_peered":"2024-09-10T18:22:43.991686+0000","last_clean":"2024-09-10T18:22:43.991686+0000","last_became_active":"2024-09-10T18:21:38.563570+0000","last_became_peered":"2024-09-10T18:21:38.563570+0000","last_unstale":"2024-09-10T18:22:43.991686+0000","last_undegraded":"2024-09-10T18:22:43.991686+0000","last_fullsized":"2024-09-10T18:22:43.991686+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:47.491396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198587+0000","last_change":"2024-09-10T18:21:31.529843+0000","last_active":"2024-09-10T18:22:44.198587+0000","last_peered":"2024-09-10T18:22:44.198587+0000","last_clean":"2024-09-10T18:22:44.198587+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:22:44.198587+0000","last_undegraded":"2024-09-10T18:22:44.198587+0000","last_fullsized":"2024-09-10T18:22:44.198587+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:58.167486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198662+0000","last_change":"2024-09-10T18:21:33.617848+0000","last_active":"2024-09-10T18:22:44.198662+0000","last_peered":"2024-09-10T18:22:44.198662+0000","last_clean":"2024-09-10T18:22:44.198662+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:22:44.198662+0000","last_undegraded":"2024-09-10T18:22:44.198662+0000","last_fullsized":"2024-09-10T18:22:44.198662+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:58.180270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.107878+0000","last_change":"2024-09-10T18:22:39.007672+0000","last_active":"2024-09-10T18:22:44.107878+0000","last_peered":"2024-09-10T18:22:44.107878+0000","last_clean":"2024-09-10T18:22:44.107878+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:22:44.107878+0000","last_undegraded":"2024-09-10T18:22:44.107878+0000","last_fullsized":"2024-09-10T18:22:44.107878+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:39.007614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:39.007614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:59.162407+0000","scrub_duration":0.0021069719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989795+0000","last_change":"2024-09-10T18:21:38.565017+0000","last_active":"2024-09-10T18:22:43.989795+0000","last_peered":"2024-09-10T18:22:43.989795+0000","last_clean":"2024-09-10T18:22:43.989795+0000","last_became_active":"2024-09-10T18:21:38.564681+0000","last_became_peered":"2024-09-10T18:21:38.564681+0000","last_unstale":"2024-09-10T18:22:43.989795+0000","last_undegraded":"2024-09-10T18:22:43.989795+0000","last_fullsized":"2024-09-10T18:22:43.989795+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.656974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991392+0000","last_change":"2024-09-10T18:21:31.528392+0000","last_active":"2024-09-10T18:22:43.991392+0000","last_peered":"2024-09-10T18:22:43.991392+0000","last_clean":"2024-09-10T18:22:43.991392+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:22:43.991392+0000","last_undegraded":"2024-09-10T18:22:43.991392+0000","last_fullsized":"2024-09-10T18:22:43.991392+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":58,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989576+0000","last_change":"2024-09-10T18:21:33.617612+0000","last_active":"2024-09-10T18:22:43.989576+0000","last_peered":"2024-09-10T18:22:43.989576+0000","last_clean":"2024-09-10T18:22:43.989576+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:22:43.989576+0000","last_undegraded":"2024-09-10T18:22:43.989576+0000","last_fullsized":"2024-09-10T18:22:43.989576+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.240219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991282+0000","last_change":"2024-09-10T18:21:35.624988+0000","last_active":"2024-09-10T18:22:43.991282+0000","last_peered":"2024-09-10T18:22:43.991282+0000","last_clean":"2024-09-10T18:22:43.991282+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:22:43.991282+0000","last_undegraded":"2024-09-10T18:22:43.991282+0000","last_fullsized":"2024-09-10T18:22:43.991282+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198650+0000","last_change":"2024-09-10T18:21:37.646675+0000","last_active":"2024-09-10T18:22:44.198650+0000","last_peered":"2024-09-10T18:22:44.198650+0000","last_clean":"2024-09-10T18:22:44.198650+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:22:44.198650+0000","last_undegraded":"2024-09-10T18:22:44.198650+0000","last_fullsized":"2024-09-10T18:22:44.198650+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:48.412991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.141701+0000","last_change":"2024-09-10T18:22:46.141545+0000","last_active":"2024-09-10T18:22:46.141701+0000","last_peered":"2024-09-10T18:22:46.141701+0000","last_clean":"2024-09-10T18:22:46.141701+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:22:46.141701+0000","last_undegraded":"2024-09-10T18:22:46.141701+0000","last_fullsized":"2024-09-10T18:22:46.141701+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:46.141408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:46.141408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:12.563610+0000","scrub_duration":0.0022977269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"86'15","reported_seq":64,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:49.298538+0000","last_change":"2024-09-10T18:22:49.298511+0000","last_active":"2024-09-10T18:22:49.298538+0000","last_peered":"2024-09-10T18:22:49.298538+0000","last_clean":"2024-09-10T18:22:49.298538+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:22:49.298538+0000","last_undegraded":"2024-09-10T18:22:49.298538+0000","last_fullsized":"2024-09-10T18:22:49.298538+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:22:49.298455+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:10.207495+0000","scrub_duration":0.0038705900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":68,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:48.795265+0000","last_change":"2024-09-10T18:22:48.795234+0000","last_active":"2024-09-10T18:22:48.795265+0000","last_peered":"2024-09-10T18:22:48.795265+0000","last_clean":"2024-09-10T18:22:48.795265+0000","last_became_active":"2024-09-10T18:21:35.624964+0000","last_became_peered":"2024-09-10T18:21:35.624964+0000","last_unstale":"2024-09-10T18:22:48.795265+0000","last_undegraded":"2024-09-10T18:22:48.795265+0000","last_fullsized":"2024-09-10T18:22:48.795265+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:22:48.795158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:48.795158+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:11.053077+0000","scrub_duration":0.0017552570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"83'22","reported_seq":75,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.108054+0000","last_change":"2024-09-10T18:22:40.007784+0000","last_active":"2024-09-10T18:22:44.108054+0000","last_peered":"2024-09-10T18:22:44.108054+0000","last_clean":"2024-09-10T18:22:44.108054+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:22:44.108054+0000","last_undegraded":"2024-09-10T18:22:44.108054+0000","last_fullsized":"2024-09-10T18:22:44.108054+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:22:40.007685+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:46.637635+0000","scrub_duration":0.0077173049999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.988922+0000","last_change":"2024-09-10T18:21:31.533221+0000","last_active":"2024-09-10T18:22:43.988922+0000","last_peered":"2024-09-10T18:22:43.988922+0000","last_clean":"2024-09-10T18:22:43.988922+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:22:43.988922+0000","last_undegraded":"2024-09-10T18:22:43.988922+0000","last_fullsized":"2024-09-10T18:22:43.988922+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:48.727349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":58,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.766721+0000","last_change":"2024-09-10T18:21:35.624518+0000","last_active":"2024-09-10T18:22:44.766721+0000","last_peered":"2024-09-10T18:22:44.766721+0000","last_clean":"2024-09-10T18:22:44.766721+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:22:44.766721+0000","last_undegraded":"2024-09-10T18:22:44.766721+0000","last_fullsized":"2024-09-10T18:22:44.766721+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.011869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110294+0000","last_change":"2024-09-10T18:21:37.644088+0000","last_active":"2024-09-10T18:22:44.110294+0000","last_peered":"2024-09-10T18:22:44.110294+0000","last_clean":"2024-09-10T18:22:44.110294+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:22:44.110294+0000","last_undegraded":"2024-09-10T18:22:44.110294+0000","last_fullsized":"2024-09-10T18:22:44.110294+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:55.585853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":67,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.988640+0000","last_change":"2024-09-10T18:21:34.047843+0000","last_active":"2024-09-10T18:22:43.988640+0000","last_peered":"2024-09-10T18:22:43.988640+0000","last_clean":"2024-09-10T18:22:43.988640+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:22:43.988640+0000","last_undegraded":"2024-09-10T18:22:43.988640+0000","last_fullsized":"2024-09-10T18:22:43.988640+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:55.071791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989941+0000","last_change":"2024-09-10T18:21:31.536352+0000","last_active":"2024-09-10T18:22:43.989941+0000","last_peered":"2024-09-10T18:22:43.989941+0000","last_clean":"2024-09-10T18:22:43.989941+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:22:43.989941+0000","last_undegraded":"2024-09-10T18:22:43.989941+0000","last_fullsized":"2024-09-10T18:22:43.989941+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:48.615680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:47.386424+0000","last_change":"2024-09-10T18:22:47.386371+0000","last_active":"2024-09-10T18:22:47.386424+0000","last_peered":"2024-09-10T18:22:47.386424+0000","last_clean":"2024-09-10T18:22:47.386424+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:22:47.386424+0000","last_undegraded":"2024-09-10T18:22:47.386424+0000","last_fullsized":"2024-09-10T18:22:47.386424+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:47.386303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:47.386303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:01.335164+0000","scrub_duration":0.001725484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991180+0000","last_change":"2024-09-10T18:21:37.642918+0000","last_active":"2024-09-10T18:22:43.991180+0000","last_peered":"2024-09-10T18:22:43.991180+0000","last_clean":"2024-09-10T18:22:43.991180+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:22:43.991180+0000","last_undegraded":"2024-09-10T18:22:43.991180+0000","last_fullsized":"2024-09-10T18:22:43.991180+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991114+0000","last_change":"2024-09-10T18:22:37.322150+0000","last_active":"2024-09-10T18:22:43.991114+0000","last_peered":"2024-09-10T18:22:43.991114+0000","last_clean":"2024-09-10T18:22:43.991114+0000","last_became_active":"2024-09-10T18:21:33.607574+0000","last_became_peered":"2024-09-10T18:21:33.607574+0000","last_unstale":"2024-09-10T18:22:43.991114+0000","last_undegraded":"2024-09-10T18:22:43.991114+0000","last_fullsized":"2024-09-10T18:22:43.991114+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-10T18:22:37.322094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:37.322094+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:41.781070+0000","scrub_duration":0.001716817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198915+0000","last_change":"2024-09-10T18:21:31.531508+0000","last_active":"2024-09-10T18:22:44.198915+0000","last_peered":"2024-09-10T18:22:44.198915+0000","last_clean":"2024-09-10T18:22:44.198915+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:22:44.198915+0000","last_undegraded":"2024-09-10T18:22:44.198915+0000","last_fullsized":"2024-09-10T18:22:44.198915+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.249073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":56,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.766485+0000","last_change":"2024-09-10T18:21:35.623304+0000","last_active":"2024-09-10T18:22:44.766485+0000","last_peered":"2024-09-10T18:22:44.766485+0000","last_clean":"2024-09-10T18:22:44.766485+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:22:44.766485+0000","last_undegraded":"2024-09-10T18:22:44.766485+0000","last_fullsized":"2024-09-10T18:22:44.766485+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:51.799895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198454+0000","last_change":"2024-09-10T18:21:38.566158+0000","last_active":"2024-09-10T18:22:44.198454+0000","last_peered":"2024-09-10T18:22:44.198454+0000","last_clean":"2024-09-10T18:22:44.198454+0000","last_became_active":"2024-09-10T18:21:38.565165+0000","last_became_peered":"2024-09-10T18:21:38.565165+0000","last_unstale":"2024-09-10T18:22:44.198454+0000","last_undegraded":"2024-09-10T18:22:44.198454+0000","last_fullsized":"2024-09-10T18:22:44.198454+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.107686+0000","last_change":"2024-09-10T18:21:33.614039+0000","last_active":"2024-09-10T18:22:44.107686+0000","last_peered":"2024-09-10T18:22:44.107686+0000","last_clean":"2024-09-10T18:22:44.107686+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:22:44.107686+0000","last_undegraded":"2024-09-10T18:22:44.107686+0000","last_fullsized":"2024-09-10T18:22:44.107686+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.533971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110662+0000","last_change":"2024-09-10T18:21:31.534170+0000","last_active":"2024-09-10T18:22:44.110662+0000","last_peered":"2024-09-10T18:22:44.110662+0000","last_clean":"2024-09-10T18:22:44.110662+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:22:44.110662+0000","last_undegraded":"2024-09-10T18:22:44.110662+0000","last_fullsized":"2024-09-10T18:22:44.110662+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:54.289170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":55,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.766432+0000","last_change":"2024-09-10T18:21:35.618920+0000","last_active":"2024-09-10T18:22:44.766432+0000","last_peered":"2024-09-10T18:22:44.766432+0000","last_clean":"2024-09-10T18:22:44.766432+0000","last_became_active":"2024-09-10T18:21:35.617821+0000","last_became_peered":"2024-09-10T18:21:35.617821+0000","last_unstale":"2024-09-10T18:22:44.766432+0000","last_undegraded":"2024-09-10T18:22:44.766432+0000","last_fullsized":"2024-09-10T18:22:44.766432+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:59.357441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.197873+0000","last_change":"2024-09-10T18:22:41.399132+0000","last_active":"2024-09-10T18:22:44.197873+0000","last_peered":"2024-09-10T18:22:44.197873+0000","last_clean":"2024-09-10T18:22:44.197873+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:22:44.197873+0000","last_undegraded":"2024-09-10T18:22:44.197873+0000","last_fullsized":"2024-09-10T18:22:44.197873+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:41.399078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:41.399078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:48.673347+0000","scrub_duration":0.001245323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198369+0000","last_change":"2024-09-10T18:21:34.046468+0000","last_active":"2024-09-10T18:22:44.198369+0000","last_peered":"2024-09-10T18:22:44.198369+0000","last_clean":"2024-09-10T18:22:44.198369+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:22:44.198369+0000","last_undegraded":"2024-09-10T18:22:44.198369+0000","last_fullsized":"2024-09-10T18:22:44.198369+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:47.310301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.108359+0000","last_change":"2024-09-10T18:21:31.530156+0000","last_active":"2024-09-10T18:22:44.108359+0000","last_peered":"2024-09-10T18:22:44.108359+0000","last_clean":"2024-09-10T18:22:44.108359+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:22:44.108359+0000","last_undegraded":"2024-09-10T18:22:44.108359+0000","last_fullsized":"2024-09-10T18:22:44.108359+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:59.013351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.172564+0000","last_change":"2024-09-10T18:22:38.853925+0000","last_active":"2024-09-10T18:22:44.172564+0000","last_peered":"2024-09-10T18:22:44.172564+0000","last_clean":"2024-09-10T18:22:44.172564+0000","last_became_active":"2024-09-10T18:21:35.617132+0000","last_became_peered":"2024-09-10T18:21:35.617132+0000","last_unstale":"2024-09-10T18:22:44.172564+0000","last_undegraded":"2024-09-10T18:22:44.172564+0000","last_fullsized":"2024-09-10T18:22:44.172564+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:38.853839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:38.853839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:39.087399+0000","scrub_duration":0.0031500500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.107278+0000","last_change":"2024-09-10T18:21:38.564042+0000","last_active":"2024-09-10T18:22:44.107278+0000","last_peered":"2024-09-10T18:22:44.107278+0000","last_clean":"2024-09-10T18:22:44.107278+0000","last_became_active":"2024-09-10T18:21:38.563629+0000","last_became_peered":"2024-09-10T18:21:38.563629+0000","last_unstale":"2024-09-10T18:22:44.107278+0000","last_undegraded":"2024-09-10T18:22:44.107278+0000","last_fullsized":"2024-09-10T18:22:44.107278+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:04.135213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":96,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.766446+0000","last_change":"2024-09-10T18:21:33.618915+0000","last_active":"2024-09-10T18:22:44.766446+0000","last_peered":"2024-09-10T18:22:44.766446+0000","last_clean":"2024-09-10T18:22:44.766446+0000","last_became_active":"2024-09-10T18:21:33.618616+0000","last_became_peered":"2024-09-10T18:21:33.618616+0000","last_unstale":"2024-09-10T18:22:44.766446+0000","last_undegraded":"2024-09-10T18:22:44.766446+0000","last_fullsized":"2024-09-10T18:22:44.766446+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.559464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991579+0000","last_change":"2024-09-10T18:22:42.335215+0000","last_active":"2024-09-10T18:22:43.991579+0000","last_peered":"2024-09-10T18:22:43.991579+0000","last_clean":"2024-09-10T18:22:43.991579+0000","last_became_active":"2024-09-10T18:21:31.526161+0000","last_became_peered":"2024-09-10T18:21:31.526161+0000","last_unstale":"2024-09-10T18:22:43.991579+0000","last_undegraded":"2024-09-10T18:22:43.991579+0000","last_fullsized":"2024-09-10T18:22:43.991579+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:42.335152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:42.335152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:02.796505+0000","scrub_duration":0.0013614720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"62'2","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110347+0000","last_change":"2024-09-10T18:21:35.613488+0000","last_active":"2024-09-10T18:22:44.110347+0000","last_peered":"2024-09-10T18:22:44.110347+0000","last_clean":"2024-09-10T18:22:44.110347+0000","last_became_active":"2024-09-10T18:21:31.528970+0000","last_became_peered":"2024-09-10T18:21:31.528970+0000","last_unstale":"2024-09-10T18:22:44.110347+0000","last_undegraded":"2024-09-10T18:22:44.110347+0000","last_fullsized":"2024-09-10T18:22:44.110347+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:51.822022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198385+0000","last_change":"2024-09-10T18:22:43.309177+0000","last_active":"2024-09-10T18:22:44.198385+0000","last_peered":"2024-09-10T18:22:44.198385+0000","last_clean":"2024-09-10T18:22:44.198385+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:22:44.198385+0000","last_undegraded":"2024-09-10T18:22:44.198385+0000","last_fullsized":"2024-09-10T18:22:44.198385+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:43.309115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:43.309115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:08.721305+0000","scrub_duration":0.0015388929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989504+0000","last_change":"2024-09-10T18:21:37.641789+0000","last_active":"2024-09-10T18:22:43.989504+0000","last_peered":"2024-09-10T18:22:43.989504+0000","last_clean":"2024-09-10T18:22:43.989504+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:22:43.989504+0000","last_undegraded":"2024-09-10T18:22:43.989504+0000","last_fullsized":"2024-09-10T18:22:43.989504+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:47.975239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.108136+0000","last_change":"2024-09-10T18:21:33.618163+0000","last_active":"2024-09-10T18:22:44.108136+0000","last_peered":"2024-09-10T18:22:44.108136+0000","last_clean":"2024-09-10T18:22:44.108136+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:22:44.108136+0000","last_undegraded":"2024-09-10T18:22:44.108136+0000","last_fullsized":"2024-09-10T18:22:44.108136+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:50.603962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.107532+0000","last_change":"2024-09-10T18:21:31.532902+0000","last_active":"2024-09-10T18:22:44.107532+0000","last_peered":"2024-09-10T18:22:44.107532+0000","last_clean":"2024-09-10T18:22:44.107532+0000","last_became_active":"2024-09-10T18:21:31.532624+0000","last_became_peered":"2024-09-10T18:21:31.532624+0000","last_unstale":"2024-09-10T18:22:44.107532+0000","last_undegraded":"2024-09-10T18:22:44.107532+0000","last_fullsized":"2024-09-10T18:22:44.107532+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:47.086852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":52,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.337307+0000","last_change":"2024-09-10T18:22:46.337257+0000","last_active":"2024-09-10T18:22:46.337307+0000","last_peered":"2024-09-10T18:22:46.337307+0000","last_clean":"2024-09-10T18:22:46.337307+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:22:46.337307+0000","last_undegraded":"2024-09-10T18:22:46.337307+0000","last_fullsized":"2024-09-10T18:22:46.337307+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:22:46.337186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:46.337186+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:11.152611+0000","scrub_duration":0.0015240410000000001,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198864+0000","last_change":"2024-09-10T18:21:35.621582+0000","last_active":"2024-09-10T18:22:44.198864+0000","last_peered":"2024-09-10T18:22:44.198864+0000","last_clean":"2024-09-10T18:22:44.198864+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:22:44.198864+0000","last_undegraded":"2024-09-10T18:22:44.198864+0000","last_fullsized":"2024-09-10T18:22:44.198864+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.088889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198757+0000","last_change":"2024-09-10T18:21:38.565470+0000","last_active":"2024-09-10T18:22:44.198757+0000","last_peered":"2024-09-10T18:22:44.198757+0000","last_clean":"2024-09-10T18:22:44.198757+0000","last_became_active":"2024-09-10T18:21:38.565183+0000","last_became_peered":"2024-09-10T18:21:38.565183+0000","last_unstale":"2024-09-10T18:22:44.198757+0000","last_undegraded":"2024-09-10T18:22:44.198757+0000","last_fullsized":"2024-09-10T18:22:44.198757+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:04.594058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":52,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198758+0000","last_change":"2024-09-10T18:21:33.613784+0000","last_active":"2024-09-10T18:22:44.198758+0000","last_peered":"2024-09-10T18:22:44.198758+0000","last_clean":"2024-09-10T18:22:44.198758+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:22:44.198758+0000","last_undegraded":"2024-09-10T18:22:44.198758+0000","last_fullsized":"2024-09-10T18:22:44.198758+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198058+0000","last_change":"2024-09-10T18:21:31.534516+0000","last_active":"2024-09-10T18:22:44.198058+0000","last_peered":"2024-09-10T18:22:44.198058+0000","last_clean":"2024-09-10T18:22:44.198058+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:22:44.198058+0000","last_undegraded":"2024-09-10T18:22:44.198058+0000","last_fullsized":"2024-09-10T18:22:44.198058+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":125,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:47.911832+0000","last_change":"2024-09-10T18:22:41.256321+0000","last_active":"2024-09-10T18:22:47.911832+0000","last_peered":"2024-09-10T18:22:47.911832+0000","last_clean":"2024-09-10T18:22:47.911832+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:22:47.911832+0000","last_undegraded":"2024-09-10T18:22:47.911832+0000","last_fullsized":"2024-09-10T18:22:47.911832+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:22:41.256255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:41.256255+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:01.661114+0000","scrub_duration":0.0015771470000000001,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":66,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110722+0000","last_change":"2024-09-10T18:21:35.619757+0000","last_active":"2024-09-10T18:22:44.110722+0000","last_peered":"2024-09-10T18:22:44.110722+0000","last_clean":"2024-09-10T18:22:44.110722+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:22:44.110722+0000","last_undegraded":"2024-09-10T18:22:44.110722+0000","last_fullsized":"2024-09-10T18:22:44.110722+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:52.027113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.107999+0000","last_change":"2024-09-10T18:21:37.639920+0000","last_active":"2024-09-10T18:22:44.107999+0000","last_peered":"2024-09-10T18:22:44.107999+0000","last_clean":"2024-09-10T18:22:44.107999+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:22:44.107999+0000","last_undegraded":"2024-09-10T18:22:44.107999+0000","last_fullsized":"2024-09-10T18:22:44.107999+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:04.764182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":56,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.173038+0000","last_change":"2024-09-10T18:21:34.046018+0000","last_active":"2024-09-10T18:22:44.173038+0000","last_peered":"2024-09-10T18:22:44.173038+0000","last_clean":"2024-09-10T18:22:44.173038+0000","last_became_active":"2024-09-10T18:21:34.045549+0000","last_became_peered":"2024-09-10T18:21:34.045549+0000","last_unstale":"2024-09-10T18:22:44.173038+0000","last_undegraded":"2024-09-10T18:22:44.173038+0000","last_fullsized":"2024-09-10T18:22:44.173038+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:52.816356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:45.383886+0000","last_change":"2024-09-10T18:22:45.383833+0000","last_active":"2024-09-10T18:22:45.383886+0000","last_peered":"2024-09-10T18:22:45.383886+0000","last_clean":"2024-09-10T18:22:45.383886+0000","last_became_active":"2024-09-10T18:21:31.529816+0000","last_became_peered":"2024-09-10T18:21:31.529816+0000","last_unstale":"2024-09-10T18:22:45.383886+0000","last_undegraded":"2024-09-10T18:22:45.383886+0000","last_fullsized":"2024-09-10T18:22:45.383886+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:45.383768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:45.383768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:06.766629+0000","scrub_duration":0.001512844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":80,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198758+0000","last_change":"2024-09-10T18:22:17.024622+0000","last_active":"2024-09-10T18:22:44.198758+0000","last_peered":"2024-09-10T18:22:44.198758+0000","last_clean":"2024-09-10T18:22:44.198758+0000","last_became_active":"2024-09-10T18:21:20.570983+0000","last_became_peered":"2024-09-10T18:21:20.570983+0000","last_unstale":"2024-09-10T18:22:44.198758+0000","last_undegraded":"2024-09-10T18:22:44.198758+0000","last_fullsized":"2024-09-10T18:22:44.198758+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:22:17.024553+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:21.328129+0000","scrub_duration":0.003148245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198776+0000","last_change":"2024-09-10T18:21:35.621370+0000","last_active":"2024-09-10T18:22:44.198776+0000","last_peered":"2024-09-10T18:22:44.198776+0000","last_clean":"2024-09-10T18:22:44.198776+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:22:44.198776+0000","last_undegraded":"2024-09-10T18:22:44.198776+0000","last_fullsized":"2024-09-10T18:22:44.198776+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:56.537273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110658+0000","last_change":"2024-09-10T18:21:37.638814+0000","last_active":"2024-09-10T18:22:44.110658+0000","last_peered":"2024-09-10T18:22:44.110658+0000","last_clean":"2024-09-10T18:22:44.110658+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:22:44.110658+0000","last_undegraded":"2024-09-10T18:22:44.110658+0000","last_fullsized":"2024-09-10T18:22:44.110658+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:06.014250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991781+0000","last_change":"2024-09-10T18:21:33.614556+0000","last_active":"2024-09-10T18:22:43.991781+0000","last_peered":"2024-09-10T18:22:43.991781+0000","last_clean":"2024-09-10T18:22:43.991781+0000","last_became_active":"2024-09-10T18:21:33.614090+0000","last_became_peered":"2024-09-10T18:21:33.614090+0000","last_unstale":"2024-09-10T18:22:43.991781+0000","last_undegraded":"2024-09-10T18:22:43.991781+0000","last_fullsized":"2024-09-10T18:22:43.991781+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:57.016893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991498+0000","last_change":"2024-09-10T18:21:31.529705+0000","last_active":"2024-09-10T18:22:43.991498+0000","last_peered":"2024-09-10T18:22:43.991498+0000","last_clean":"2024-09-10T18:22:43.991498+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:22:43.991498+0000","last_undegraded":"2024-09-10T18:22:43.991498+0000","last_fullsized":"2024-09-10T18:22:43.991498+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:56.165261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110581+0000","last_change":"2024-09-10T18:21:35.620645+0000","last_active":"2024-09-10T18:22:44.110581+0000","last_peered":"2024-09-10T18:22:44.110581+0000","last_clean":"2024-09-10T18:22:44.110581+0000","last_became_active":"2024-09-10T18:21:35.620331+0000","last_became_peered":"2024-09-10T18:21:35.620331+0000","last_unstale":"2024-09-10T18:22:44.110581+0000","last_undegraded":"2024-09-10T18:22:44.110581+0000","last_fullsized":"2024-09-10T18:22:44.110581+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:55.576218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989251+0000","last_change":"2024-09-10T18:21:38.564048+0000","last_active":"2024-09-10T18:22:43.989251+0000","last_peered":"2024-09-10T18:22:43.989251+0000","last_clean":"2024-09-10T18:22:43.989251+0000","last_became_active":"2024-09-10T18:21:38.563450+0000","last_became_peered":"2024-09-10T18:21:38.563450+0000","last_unstale":"2024-09-10T18:22:43.989251+0000","last_undegraded":"2024-09-10T18:22:43.989251+0000","last_fullsized":"2024-09-10T18:22:43.989251+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:46.597152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991713+0000","last_change":"2024-09-10T18:21:33.611478+0000","last_active":"2024-09-10T18:22:43.991713+0000","last_peered":"2024-09-10T18:22:43.991713+0000","last_clean":"2024-09-10T18:22:43.991713+0000","last_became_active":"2024-09-10T18:21:33.611203+0000","last_became_peered":"2024-09-10T18:21:33.611203+0000","last_unstale":"2024-09-10T18:22:43.991713+0000","last_undegraded":"2024-09-10T18:22:43.991713+0000","last_fullsized":"2024-09-10T18:22:43.991713+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:58.663030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"58'1","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.172795+0000","last_change":"2024-09-10T18:21:31.528999+0000","last_active":"2024-09-10T18:22:44.172795+0000","last_peered":"2024-09-10T18:22:44.172795+0000","last_clean":"2024-09-10T18:22:44.172795+0000","last_became_active":"2024-09-10T18:21:31.528748+0000","last_became_peered":"2024-09-10T18:21:31.528748+0000","last_unstale":"2024-09-10T18:22:44.172795+0000","last_undegraded":"2024-09-10T18:22:44.172795+0000","last_fullsized":"2024-09-10T18:22:44.172795+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:00.198506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.107651+0000","last_change":"2024-09-10T18:21:35.625340+0000","last_active":"2024-09-10T18:22:44.107651+0000","last_peered":"2024-09-10T18:22:44.107651+0000","last_clean":"2024-09-10T18:22:44.107651+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:22:44.107651+0000","last_undegraded":"2024-09-10T18:22:44.107651+0000","last_fullsized":"2024-09-10T18:22:44.107651+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:51.126655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198641+0000","last_change":"2024-09-10T18:21:37.642943+0000","last_active":"2024-09-10T18:22:44.198641+0000","last_peered":"2024-09-10T18:22:44.198641+0000","last_clean":"2024-09-10T18:22:44.198641+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:22:44.198641+0000","last_undegraded":"2024-09-10T18:22:44.198641+0000","last_fullsized":"2024-09-10T18:22:44.198641+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:45.701597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991335+0000","last_change":"2024-09-10T18:21:33.614985+0000","last_active":"2024-09-10T18:22:43.991335+0000","last_peered":"2024-09-10T18:22:43.991335+0000","last_clean":"2024-09-10T18:22:43.991335+0000","last_became_active":"2024-09-10T18:21:33.614689+0000","last_became_peered":"2024-09-10T18:21:33.614689+0000","last_unstale":"2024-09-10T18:22:43.991335+0000","last_undegraded":"2024-09-10T18:22:43.991335+0000","last_fullsized":"2024-09-10T18:22:43.991335+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:57.940565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989276+0000","last_change":"2024-09-10T18:21:31.535458+0000","last_active":"2024-09-10T18:22:43.989276+0000","last_peered":"2024-09-10T18:22:43.989276+0000","last_clean":"2024-09-10T18:22:43.989276+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:22:43.989276+0000","last_undegraded":"2024-09-10T18:22:43.989276+0000","last_fullsized":"2024-09-10T18:22:43.989276+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:55.260904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":55,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.766433+0000","last_change":"2024-09-10T18:21:35.625424+0000","last_active":"2024-09-10T18:22:44.766433+0000","last_peered":"2024-09-10T18:22:44.766433+0000","last_clean":"2024-09-10T18:22:44.766433+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:22:44.766433+0000","last_undegraded":"2024-09-10T18:22:44.766433+0000","last_fullsized":"2024-09-10T18:22:44.766433+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991650+0000","last_change":"2024-09-10T18:21:37.642233+0000","last_active":"2024-09-10T18:22:43.991650+0000","last_peered":"2024-09-10T18:22:43.991650+0000","last_clean":"2024-09-10T18:22:43.991650+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:22:43.991650+0000","last_undegraded":"2024-09-10T18:22:43.991650+0000","last_fullsized":"2024-09-10T18:22:43.991650+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198288+0000","last_change":"2024-09-10T18:21:33.614532+0000","last_active":"2024-09-10T18:22:44.198288+0000","last_peered":"2024-09-10T18:22:44.198288+0000","last_clean":"2024-09-10T18:22:44.198288+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:22:44.198288+0000","last_undegraded":"2024-09-10T18:22:44.198288+0000","last_fullsized":"2024-09-10T18:22:44.198288+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:57.959502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"58'1","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110399+0000","last_change":"2024-09-10T18:21:31.534677+0000","last_active":"2024-09-10T18:22:44.110399+0000","last_peered":"2024-09-10T18:22:44.110399+0000","last_clean":"2024-09-10T18:22:44.110399+0000","last_became_active":"2024-09-10T18:21:31.534441+0000","last_became_peered":"2024-09-10T18:21:31.534441+0000","last_unstale":"2024-09-10T18:22:44.110399+0000","last_undegraded":"2024-09-10T18:22:44.110399+0000","last_fullsized":"2024-09-10T18:22:44.110399+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.618552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198361+0000","last_change":"2024-09-10T18:22:41.279395+0000","last_active":"2024-09-10T18:22:44.198361+0000","last_peered":"2024-09-10T18:22:44.198361+0000","last_clean":"2024-09-10T18:22:44.198361+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:22:44.198361+0000","last_undegraded":"2024-09-10T18:22:44.198361+0000","last_fullsized":"2024-09-10T18:22:44.198361+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:41.279342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:41.279342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:48.532838+0000","scrub_duration":0.0017787790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198014+0000","last_change":"2024-09-10T18:21:37.641253+0000","last_active":"2024-09-10T18:22:44.198014+0000","last_peered":"2024-09-10T18:22:44.198014+0000","last_clean":"2024-09-10T18:22:44.198014+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:22:44.198014+0000","last_undegraded":"2024-09-10T18:22:44.198014+0000","last_fullsized":"2024-09-10T18:22:44.198014+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.172596+0000","last_change":"2024-09-10T18:21:34.046749+0000","last_active":"2024-09-10T18:22:44.172596+0000","last_peered":"2024-09-10T18:22:44.172596+0000","last_clean":"2024-09-10T18:22:44.172596+0000","last_became_active":"2024-09-10T18:21:34.046453+0000","last_became_peered":"2024-09-10T18:21:34.046453+0000","last_unstale":"2024-09-10T18:22:44.172596+0000","last_undegraded":"2024-09-10T18:22:44.172596+0000","last_fullsized":"2024-09-10T18:22:44.172596+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.046130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.199059+0000","last_change":"2024-09-10T18:22:32.137670+0000","last_active":"2024-09-10T18:22:44.199059+0000","last_peered":"2024-09-10T18:22:44.199059+0000","last_clean":"2024-09-10T18:22:44.199059+0000","last_became_active":"2024-09-10T18:21:31.531951+0000","last_became_peered":"2024-09-10T18:21:31.531951+0000","last_unstale":"2024-09-10T18:22:44.199059+0000","last_undegraded":"2024-09-10T18:22:44.199059+0000","last_fullsized":"2024-09-10T18:22:44.199059+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:32.137600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:32.137600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:41.301280+0000","scrub_duration":0.00151761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198130+0000","last_change":"2024-09-10T18:22:42.312458+0000","last_active":"2024-09-10T18:22:44.198130+0000","last_peered":"2024-09-10T18:22:44.198130+0000","last_clean":"2024-09-10T18:22:44.198130+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:22:44.198130+0000","last_undegraded":"2024-09-10T18:22:44.198130+0000","last_fullsized":"2024-09-10T18:22:44.198130+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:42.312403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:42.312403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:42.394403+0000","scrub_duration":0.0014316489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198593+0000","last_change":"2024-09-10T18:21:37.642296+0000","last_active":"2024-09-10T18:22:44.198593+0000","last_peered":"2024-09-10T18:22:44.198593+0000","last_clean":"2024-09-10T18:22:44.198593+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:22:44.198593+0000","last_undegraded":"2024-09-10T18:22:44.198593+0000","last_fullsized":"2024-09-10T18:22:44.198593+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":57,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.990999+0000","last_change":"2024-09-10T18:21:33.615440+0000","last_active":"2024-09-10T18:22:43.990999+0000","last_peered":"2024-09-10T18:22:43.990999+0000","last_clean":"2024-09-10T18:22:43.990999+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:22:43.990999+0000","last_undegraded":"2024-09-10T18:22:43.990999+0000","last_fullsized":"2024-09-10T18:22:43.990999+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.199220+0000","last_change":"2024-09-10T18:21:31.531998+0000","last_active":"2024-09-10T18:22:44.199220+0000","last_peered":"2024-09-10T18:22:44.199220+0000","last_clean":"2024-09-10T18:22:44.199220+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:22:44.199220+0000","last_undegraded":"2024-09-10T18:22:44.199220+0000","last_fullsized":"2024-09-10T18:22:44.199220+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:57.379141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.197789+0000","last_change":"2024-09-10T18:22:40.275165+0000","last_active":"2024-09-10T18:22:44.197789+0000","last_peered":"2024-09-10T18:22:44.197789+0000","last_clean":"2024-09-10T18:22:44.197789+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:22:44.197789+0000","last_undegraded":"2024-09-10T18:22:44.197789+0000","last_fullsized":"2024-09-10T18:22:44.197789+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:40.275069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:40.275069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:54.251771+0000","scrub_duration":0.001522121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989104+0000","last_change":"2024-09-10T18:21:37.644889+0000","last_active":"2024-09-10T18:22:43.989104+0000","last_peered":"2024-09-10T18:22:43.989104+0000","last_clean":"2024-09-10T18:22:43.989104+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:22:43.989104+0000","last_undegraded":"2024-09-10T18:22:43.989104+0000","last_fullsized":"2024-09-10T18:22:43.989104+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:59.247137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.109888+0000","last_change":"2024-09-10T18:21:34.047130+0000","last_active":"2024-09-10T18:22:44.109888+0000","last_peered":"2024-09-10T18:22:44.109888+0000","last_clean":"2024-09-10T18:22:44.109888+0000","last_became_active":"2024-09-10T18:21:34.046741+0000","last_became_peered":"2024-09-10T18:21:34.046741+0000","last_unstale":"2024-09-10T18:22:44.109888+0000","last_undegraded":"2024-09-10T18:22:44.109888+0000","last_fullsized":"2024-09-10T18:22:44.109888+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.199062+0000","last_change":"2024-09-10T18:21:31.530630+0000","last_active":"2024-09-10T18:22:44.199062+0000","last_peered":"2024-09-10T18:22:44.199062+0000","last_clean":"2024-09-10T18:22:44.199062+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:22:44.199062+0000","last_undegraded":"2024-09-10T18:22:44.199062+0000","last_fullsized":"2024-09-10T18:22:44.199062+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":55,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.766795+0000","last_change":"2024-09-10T18:21:35.627197+0000","last_active":"2024-09-10T18:22:44.766795+0000","last_peered":"2024-09-10T18:22:44.766795+0000","last_clean":"2024-09-10T18:22:44.766795+0000","last_became_active":"2024-09-10T18:21:35.626744+0000","last_became_peered":"2024-09-10T18:21:35.626744+0000","last_unstale":"2024-09-10T18:22:44.766795+0000","last_undegraded":"2024-09-10T18:22:44.766795+0000","last_fullsized":"2024-09-10T18:22:44.766795+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:58.371060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.109996+0000","last_change":"2024-09-10T18:21:38.563824+0000","last_active":"2024-09-10T18:22:44.109996+0000","last_peered":"2024-09-10T18:22:44.109996+0000","last_clean":"2024-09-10T18:22:44.109996+0000","last_became_active":"2024-09-10T18:21:38.563550+0000","last_became_peered":"2024-09-10T18:21:38.563550+0000","last_unstale":"2024-09-10T18:22:44.109996+0000","last_undegraded":"2024-09-10T18:22:44.109996+0000","last_fullsized":"2024-09-10T18:22:44.109996+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:05.155686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.172510+0000","last_change":"2024-09-10T18:21:31.527287+0000","last_active":"2024-09-10T18:22:44.172510+0000","last_peered":"2024-09-10T18:22:44.172510+0000","last_clean":"2024-09-10T18:22:44.172510+0000","last_became_active":"2024-09-10T18:21:31.526818+0000","last_became_peered":"2024-09-10T18:21:31.526818+0000","last_unstale":"2024-09-10T18:22:44.172510+0000","last_undegraded":"2024-09-10T18:22:44.172510+0000","last_fullsized":"2024-09-10T18:22:44.172510+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:53.818450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989883+0000","last_change":"2024-09-10T18:21:34.046070+0000","last_active":"2024-09-10T18:22:43.989883+0000","last_peered":"2024-09-10T18:22:43.989883+0000","last_clean":"2024-09-10T18:22:43.989883+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:22:43.989883+0000","last_undegraded":"2024-09-10T18:22:43.989883+0000","last_fullsized":"2024-09-10T18:22:43.989883+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:54.221783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110546+0000","last_change":"2024-09-10T18:21:35.622254+0000","last_active":"2024-09-10T18:22:44.110546+0000","last_peered":"2024-09-10T18:22:44.110546+0000","last_clean":"2024-09-10T18:22:44.110546+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:22:44.110546+0000","last_undegraded":"2024-09-10T18:22:44.110546+0000","last_fullsized":"2024-09-10T18:22:44.110546+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:54.549955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198867+0000","last_change":"2024-09-10T18:21:38.565959+0000","last_active":"2024-09-10T18:22:44.198867+0000","last_peered":"2024-09-10T18:22:44.198867+0000","last_clean":"2024-09-10T18:22:44.198867+0000","last_became_active":"2024-09-10T18:21:38.565675+0000","last_became_peered":"2024-09-10T18:21:38.565675+0000","last_unstale":"2024-09-10T18:22:44.198867+0000","last_undegraded":"2024-09-10T18:22:44.198867+0000","last_fullsized":"2024-09-10T18:22:44.198867+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198837+0000","last_change":"2024-09-10T18:21:34.047798+0000","last_active":"2024-09-10T18:22:44.198837+0000","last_peered":"2024-09-10T18:22:44.198837+0000","last_clean":"2024-09-10T18:22:44.198837+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:22:44.198837+0000","last_undegraded":"2024-09-10T18:22:44.198837+0000","last_fullsized":"2024-09-10T18:22:44.198837+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:47.847418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198648+0000","last_change":"2024-09-10T18:21:31.532155+0000","last_active":"2024-09-10T18:22:44.198648+0000","last_peered":"2024-09-10T18:22:44.198648+0000","last_clean":"2024-09-10T18:22:44.198648+0000","last_became_active":"2024-09-10T18:21:31.531895+0000","last_became_peered":"2024-09-10T18:21:31.531895+0000","last_unstale":"2024-09-10T18:22:44.198648+0000","last_undegraded":"2024-09-10T18:22:44.198648+0000","last_fullsized":"2024-09-10T18:22:44.198648+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:49.301934+0000","last_change":"2024-09-10T18:22:49.301887+0000","last_active":"2024-09-10T18:22:49.301934+0000","last_peered":"2024-09-10T18:22:49.301934+0000","last_clean":"2024-09-10T18:22:49.301934+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:22:49.301934+0000","last_undegraded":"2024-09-10T18:22:49.301934+0000","last_fullsized":"2024-09-10T18:22:49.301934+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:49.301823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:49.301823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:06.288915+0000","scrub_duration":0.001243433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198276+0000","last_change":"2024-09-10T18:21:37.637599+0000","last_active":"2024-09-10T18:22:44.198276+0000","last_peered":"2024-09-10T18:22:44.198276+0000","last_clean":"2024-09-10T18:22:44.198276+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:22:44.198276+0000","last_undegraded":"2024-09-10T18:22:44.198276+0000","last_fullsized":"2024-09-10T18:22:44.198276+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:02.280321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110579+0000","last_change":"2024-09-10T18:21:34.047045+0000","last_active":"2024-09-10T18:22:44.110579+0000","last_peered":"2024-09-10T18:22:44.110579+0000","last_clean":"2024-09-10T18:22:44.110579+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:22:44.110579+0000","last_undegraded":"2024-09-10T18:22:44.110579+0000","last_fullsized":"2024-09-10T18:22:44.110579+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.769358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198144+0000","last_change":"2024-09-10T18:22:38.445504+0000","last_active":"2024-09-10T18:22:44.198144+0000","last_peered":"2024-09-10T18:22:44.198144+0000","last_clean":"2024-09-10T18:22:44.198144+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:22:44.198144+0000","last_undegraded":"2024-09-10T18:22:44.198144+0000","last_fullsized":"2024-09-10T18:22:44.198144+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:38.445448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:38.445448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:05.044685+0000","scrub_duration":0.0018359400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":66,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:48.317787+0000","last_change":"2024-09-10T18:22:48.317756+0000","last_active":"2024-09-10T18:22:48.317787+0000","last_peered":"2024-09-10T18:22:48.317787+0000","last_clean":"2024-09-10T18:22:48.317787+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:22:48.317787+0000","last_undegraded":"2024-09-10T18:22:48.317787+0000","last_fullsized":"2024-09-10T18:22:48.317787+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:22:48.317687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:48.317687+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:01.400638+0000","scrub_duration":0.0022162499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991842+0000","last_change":"2024-09-10T18:21:37.640015+0000","last_active":"2024-09-10T18:22:43.991842+0000","last_peered":"2024-09-10T18:22:43.991842+0000","last_clean":"2024-09-10T18:22:43.991842+0000","last_became_active":"2024-09-10T18:21:37.639706+0000","last_became_peered":"2024-09-10T18:21:37.639706+0000","last_unstale":"2024-09-10T18:22:43.991842+0000","last_undegraded":"2024-09-10T18:22:43.991842+0000","last_fullsized":"2024-09-10T18:22:43.991842+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989634+0000","last_change":"2024-09-10T18:22:33.342047+0000","last_active":"2024-09-10T18:22:43.989634+0000","last_peered":"2024-09-10T18:22:43.989634+0000","last_clean":"2024-09-10T18:22:43.989634+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:22:43.989634+0000","last_undegraded":"2024-09-10T18:22:43.989634+0000","last_fullsized":"2024-09-10T18:22:43.989634+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-10T18:22:33.341980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:33.341980+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:47.079316+0000","scrub_duration":0.0021839350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198817+0000","last_change":"2024-09-10T18:21:31.532402+0000","last_active":"2024-09-10T18:22:44.198817+0000","last_peered":"2024-09-10T18:22:44.198817+0000","last_clean":"2024-09-10T18:22:44.198817+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:22:44.198817+0000","last_undegraded":"2024-09-10T18:22:44.198817+0000","last_fullsized":"2024-09-10T18:22:44.198817+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":55,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.766736+0000","last_change":"2024-09-10T18:21:35.618966+0000","last_active":"2024-09-10T18:22:44.766736+0000","last_peered":"2024-09-10T18:22:44.766736+0000","last_clean":"2024-09-10T18:22:44.766736+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:22:44.766736+0000","last_undegraded":"2024-09-10T18:22:44.766736+0000","last_fullsized":"2024-09-10T18:22:44.766736+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198472+0000","last_change":"2024-09-10T18:21:37.644078+0000","last_active":"2024-09-10T18:22:44.198472+0000","last_peered":"2024-09-10T18:22:44.198472+0000","last_clean":"2024-09-10T18:22:44.198472+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:22:44.198472+0000","last_undegraded":"2024-09-10T18:22:44.198472+0000","last_fullsized":"2024-09-10T18:22:44.198472+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991187+0000","last_change":"2024-09-10T18:21:33.611605+0000","last_active":"2024-09-10T18:22:43.991187+0000","last_peered":"2024-09-10T18:22:43.991187+0000","last_clean":"2024-09-10T18:22:43.991187+0000","last_became_active":"2024-09-10T18:21:33.611286+0000","last_became_peered":"2024-09-10T18:21:33.611286+0000","last_unstale":"2024-09-10T18:22:43.991187+0000","last_undegraded":"2024-09-10T18:22:43.991187+0000","last_fullsized":"2024-09-10T18:22:43.991187+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:53.952740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991512+0000","last_change":"2024-09-10T18:21:31.530449+0000","last_active":"2024-09-10T18:22:43.991512+0000","last_peered":"2024-09-10T18:22:43.991512+0000","last_clean":"2024-09-10T18:22:43.991512+0000","last_became_active":"2024-09-10T18:21:31.530158+0000","last_became_peered":"2024-09-10T18:21:31.530158+0000","last_unstale":"2024-09-10T18:22:43.991512+0000","last_undegraded":"2024-09-10T18:22:43.991512+0000","last_fullsized":"2024-09-10T18:22:43.991512+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.107405+0000","last_change":"2024-09-10T18:22:36.049767+0000","last_active":"2024-09-10T18:22:44.107405+0000","last_peered":"2024-09-10T18:22:44.107405+0000","last_clean":"2024-09-10T18:22:44.107405+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:22:44.107405+0000","last_undegraded":"2024-09-10T18:22:44.107405+0000","last_fullsized":"2024-09-10T18:22:44.107405+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:36.049705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:36.049705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:39.206398+0000","scrub_duration":0.0030879660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.988587+0000","last_change":"2024-09-10T18:21:37.642856+0000","last_active":"2024-09-10T18:22:43.988587+0000","last_peered":"2024-09-10T18:22:43.988587+0000","last_clean":"2024-09-10T18:22:43.988587+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:22:43.988587+0000","last_undegraded":"2024-09-10T18:22:43.988587+0000","last_fullsized":"2024-09-10T18:22:43.988587+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:04.616953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.107805+0000","last_change":"2024-09-10T18:22:42.015349+0000","last_active":"2024-09-10T18:22:44.107805+0000","last_peered":"2024-09-10T18:22:44.107805+0000","last_clean":"2024-09-10T18:22:44.107805+0000","last_became_active":"2024-09-10T18:21:33.612401+0000","last_became_peered":"2024-09-10T18:21:33.612401+0000","last_unstale":"2024-09-10T18:22:44.107805+0000","last_undegraded":"2024-09-10T18:22:44.107805+0000","last_fullsized":"2024-09-10T18:22:44.107805+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:22:42.015292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:42.015292+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:07.052542+0000","scrub_duration":0.001986785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198978+0000","last_change":"2024-09-10T18:22:43.250096+0000","last_active":"2024-09-10T18:22:44.198978+0000","last_peered":"2024-09-10T18:22:44.198978+0000","last_clean":"2024-09-10T18:22:44.198978+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:22:44.198978+0000","last_undegraded":"2024-09-10T18:22:44.198978+0000","last_fullsized":"2024-09-10T18:22:44.198978+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:43.250030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:43.250030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:12.038178+0000","scrub_duration":0.001581381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989164+0000","last_change":"2024-09-10T18:21:35.623087+0000","last_active":"2024-09-10T18:22:43.989164+0000","last_peered":"2024-09-10T18:22:43.989164+0000","last_clean":"2024-09-10T18:22:43.989164+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:22:43.989164+0000","last_undegraded":"2024-09-10T18:22:43.989164+0000","last_fullsized":"2024-09-10T18:22:43.989164+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:03.160666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.197662+0000","last_change":"2024-09-10T18:21:37.641754+0000","last_active":"2024-09-10T18:22:44.197662+0000","last_peered":"2024-09-10T18:22:44.197662+0000","last_clean":"2024-09-10T18:22:44.197662+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:22:44.197662+0000","last_undegraded":"2024-09-10T18:22:44.197662+0000","last_fullsized":"2024-09-10T18:22:44.197662+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":54,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.988726+0000","last_change":"2024-09-10T18:21:34.046343+0000","last_active":"2024-09-10T18:22:43.988726+0000","last_peered":"2024-09-10T18:22:43.988726+0000","last_clean":"2024-09-10T18:22:43.988726+0000","last_became_active":"2024-09-10T18:21:34.046024+0000","last_became_peered":"2024-09-10T18:21:34.046024+0000","last_unstale":"2024-09-10T18:22:43.988726+0000","last_undegraded":"2024-09-10T18:22:43.988726+0000","last_fullsized":"2024-09-10T18:22:43.988726+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:50.603367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110567+0000","last_change":"2024-09-10T18:22:40.686003+0000","last_active":"2024-09-10T18:22:44.110567+0000","last_peered":"2024-09-10T18:22:44.110567+0000","last_clean":"2024-09-10T18:22:44.110567+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:22:44.110567+0000","last_undegraded":"2024-09-10T18:22:44.110567+0000","last_fullsized":"2024-09-10T18:22:44.110567+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:22:40.685879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:01.722544+0000","scrub_duration":0.001767794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198179+0000","last_change":"2024-09-10T18:21:35.626825+0000","last_active":"2024-09-10T18:22:44.198179+0000","last_peered":"2024-09-10T18:22:44.198179+0000","last_clean":"2024-09-10T18:22:44.198179+0000","last_became_active":"2024-09-10T18:21:35.626537+0000","last_became_peered":"2024-09-10T18:21:35.626537+0000","last_unstale":"2024-09-10T18:22:44.198179+0000","last_undegraded":"2024-09-10T18:22:44.198179+0000","last_fullsized":"2024-09-10T18:22:44.198179+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:56.248415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.343231+0000","last_change":"2024-09-10T18:22:46.343197+0000","last_active":"2024-09-10T18:22:46.343231+0000","last_peered":"2024-09-10T18:22:46.343231+0000","last_clean":"2024-09-10T18:22:46.343231+0000","last_became_active":"2024-09-10T18:21:38.564314+0000","last_became_peered":"2024-09-10T18:21:38.564314+0000","last_unstale":"2024-09-10T18:22:46.343231+0000","last_undegraded":"2024-09-10T18:22:46.343231+0000","last_fullsized":"2024-09-10T18:22:46.343231+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:46.343137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:46.343137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:10.438171+0000","scrub_duration":0.0016021189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.988577+0000","last_change":"2024-09-10T18:21:33.618322+0000","last_active":"2024-09-10T18:22:43.988577+0000","last_peered":"2024-09-10T18:22:43.988577+0000","last_clean":"2024-09-10T18:22:43.988577+0000","last_became_active":"2024-09-10T18:21:33.617670+0000","last_became_peered":"2024-09-10T18:21:33.617670+0000","last_unstale":"2024-09-10T18:22:43.988577+0000","last_undegraded":"2024-09-10T18:22:43.988577+0000","last_fullsized":"2024-09-10T18:22:43.988577+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:58.493392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198306+0000","last_change":"2024-09-10T18:21:31.530146+0000","last_active":"2024-09-10T18:22:44.198306+0000","last_peered":"2024-09-10T18:22:44.198306+0000","last_clean":"2024-09-10T18:22:44.198306+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:22:44.198306+0000","last_undegraded":"2024-09-10T18:22:44.198306+0000","last_fullsized":"2024-09-10T18:22:44.198306+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:52.583420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.172552+0000","last_change":"2024-09-10T18:21:35.620681+0000","last_active":"2024-09-10T18:22:44.172552+0000","last_peered":"2024-09-10T18:22:44.172552+0000","last_clean":"2024-09-10T18:22:44.172552+0000","last_became_active":"2024-09-10T18:21:35.620419+0000","last_became_peered":"2024-09-10T18:21:35.620419+0000","last_unstale":"2024-09-10T18:22:44.172552+0000","last_undegraded":"2024-09-10T18:22:44.172552+0000","last_fullsized":"2024-09-10T18:22:44.172552+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198343+0000","last_change":"2024-09-10T18:21:37.643084+0000","last_active":"2024-09-10T18:22:44.198343+0000","last_peered":"2024-09-10T18:22:44.198343+0000","last_clean":"2024-09-10T18:22:44.198343+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:22:44.198343+0000","last_undegraded":"2024-09-10T18:22:44.198343+0000","last_fullsized":"2024-09-10T18:22:44.198343+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:05.246215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.108529+0000","last_change":"2024-09-10T18:22:44.108462+0000","last_active":"2024-09-10T18:22:44.108529+0000","last_peered":"2024-09-10T18:22:44.108529+0000","last_clean":"2024-09-10T18:22:44.108529+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:22:44.108529+0000","last_undegraded":"2024-09-10T18:22:44.108529+0000","last_fullsized":"2024-09-10T18:22:44.108529+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:44.108373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:44.108373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:51.902386+0000","scrub_duration":0.0024985559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.988765+0000","last_change":"2024-09-10T18:21:31.535545+0000","last_active":"2024-09-10T18:22:43.988765+0000","last_peered":"2024-09-10T18:22:43.988765+0000","last_clean":"2024-09-10T18:22:43.988765+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:22:43.988765+0000","last_undegraded":"2024-09-10T18:22:43.988765+0000","last_fullsized":"2024-09-10T18:22:43.988765+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:56.890432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.173162+0000","last_change":"2024-09-10T18:21:33.613827+0000","last_active":"2024-09-10T18:22:44.173162+0000","last_peered":"2024-09-10T18:22:44.173162+0000","last_clean":"2024-09-10T18:22:44.173162+0000","last_became_active":"2024-09-10T18:21:33.613487+0000","last_became_peered":"2024-09-10T18:21:33.613487+0000","last_unstale":"2024-09-10T18:22:44.173162+0000","last_undegraded":"2024-09-10T18:22:44.173162+0000","last_fullsized":"2024-09-10T18:22:44.173162+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:50.340408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198608+0000","last_change":"2024-09-10T18:21:35.620819+0000","last_active":"2024-09-10T18:22:44.198608+0000","last_peered":"2024-09-10T18:22:44.198608+0000","last_clean":"2024-09-10T18:22:44.198608+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:22:44.198608+0000","last_undegraded":"2024-09-10T18:22:44.198608+0000","last_fullsized":"2024-09-10T18:22:44.198608+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":405,"ondisk_log_size":405,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":66,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234003,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298812,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446340,"statfs":{"total":95995035648,"available":95689052160,"internally_reserved":0,"allocated":1495040,"data_stored":962545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.46300000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.44},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.58699999999999997}]},{"osd":2,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.49399999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52700000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.40799999999999997}]},{"osd":5,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.51200000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.61199999999999999}]}]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298804,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446348,"statfs":{"total":95995035648,"available":95689060352,"internally_reserved":0,"allocated":1478656,"data_stored":962717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.747}]},{"osd":1,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.53500000000000003}]},{"osd":2,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.58899999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.59999999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.48299999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.625}]},{"osd":7,"last update":"Tue Sep 10 18:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.70299999999999996}]}]},{"osd":5,"up_from":39,"seq":167503724574,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298220,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446932,"statfs":{"total":95995035648,"available":95689658368,"internally_reserved":0,"allocated":884736,"data_stored":372010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.36399999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.42499999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.65400000000000003}]},{"osd":1,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.56499999999999995,"15min":0.56100000000000005},"min":{"1min":0.42399999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.377,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.53700000000000003}]},{"osd":2,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55100000000000005,"15min":0.55200000000000005},"min":{"1min":0.38800000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.38,"5min":0.255,"15min":0.255},"max":{"1min":0.75600000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.54900000000000004}]},{"osd":3,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.54800000000000004,"15min":0.54300000000000004},"min":{"1min":0.434,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.52900000000000003,"15min":0.53600000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.57699999999999996}]},{"osd":4,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.36499999999999999,"15min":0.35899999999999999},"min":{"1min":0.29499999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.38400000000000001},"min":{"1min":0.28599999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.442}]},{"osd":6,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.624}]},{"osd":7,"last update":"Tue Sep 10 18:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.60299999999999998}]}]},{"osd":4,"up_from":33,"seq":141733920802,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298236,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446916,"statfs":{"total":95995035648,"available":95689641984,"internally_reserved":0,"allocated":901120,"data_stored":371988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56599999999999995,"15min":0.57499999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.72699999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.746,"5min":0.61799999999999999,"15min":0.59699999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.8170000000000002,"5min":4.8170000000000002,"15min":4.8170000000000002},"last":0.377}]},{"osd":1,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.61499999999999999,"15min":0.59699999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":4.7729999999999997,"5min":4.7729999999999997,"15min":4.7729999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.626}]},{"osd":2,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.60699999999999998,"15min":0.57899999999999996},"min":{"1min":0.42799999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.8040000000000003,"5min":4.8040000000000003,"15min":4.8040000000000003},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.57599999999999996,"15min":0.58599999999999997},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71699999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.45800000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.66600000000000004,"5min":0.751,"15min":0.751},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59099999999999997,"15min":0.59599999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.747,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.434}]},{"osd":5,"last update":"Tue Sep 10 18:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.39000000000000001,"15min":0.38300000000000001},"min":{"1min":0.32800000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42499999999999999,"15min":0.42399999999999999},"min":{"1min":0.28299999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60099999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.54600000000000004}]},{"osd":6,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.56499999999999995}]},{"osd":7,"last update":"Tue Sep 10 18:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.58299999999999996}]}]},{"osd":3,"up_from":27,"seq":115964117032,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298244,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446908,"statfs":{"total":95995035648,"available":95689633792,"internally_reserved":0,"allocated":913408,"data_stored":372832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:21:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.377,"15min":0.371},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.39400000000000002,"15min":0.39100000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.57999999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.28399999999999997}]},{"osd":1,"last update":"Tue Sep 10 18:21:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.432,"15min":0.433},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.40200000000000002,"15min":0.39600000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.32100000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:21:52 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.441,"15min":0.437},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.39700000000000002,"15min":0.39200000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.30099999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:22:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.55600000000000005,"15min":0.54000000000000004},"min":{"1min":0.378,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.56499999999999995,"15min":0.55500000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.44900000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:22:34 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":0.73899999999999999,"15min":0.63400000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":14.885,"5min":14.885,"15min":14.885},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.61899999999999999,"15min":0.60699999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.499}]},{"osd":6,"last update":"Tue Sep 10 18:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.47399999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":0.44},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.52200000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298208,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446944,"statfs":{"total":95995035648,"available":95689670656,"internally_reserved":0,"allocated":876544,"data_stored":371240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.36199999999999999,"15min":0.33400000000000002},"min":{"1min":0.22,"5min":0.219,"15min":0.219},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.371,"15min":0.34899999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.41499999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.35599999999999998,"15min":0.33800000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.63100000000000001,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.39100000000000001,"15min":0.35499999999999998},"min":{"1min":0.28199999999999997,"5min":0.22,"15min":0.22},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.40100000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46400000000000002,"15min":0.46300000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.436,"15min":0.433},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.53800000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.56799999999999995,"15min":0.56499999999999995},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.76300000000000001,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.76000000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.61499999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.61499999999999999,"15min":0.59399999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":2.3279999999999998,"5min":2.3279999999999998,"15min":2.3279999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.63,"15min":0.61599999999999999},"min":{"1min":0.44500000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.47499999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.503},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.66700000000000004}]},{"osd":7,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.57399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298272,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446880,"statfs":{"total":95995035648,"available":95689605120,"internally_reserved":0,"allocated":937984,"data_stored":373168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.434},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39000000000000001,"15min":0.373},"min":{"1min":0.27700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.32400000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.379},"min":{"1min":0.25700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.38200000000000001,"15min":0.36399999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.64800000000000002,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.39100000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:21:51 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.46200000000000002,"15min":0.46800000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68200000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.41499999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.85899999999999999,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.94799999999999995,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.66400000000000003}]},{"osd":5,"last update":"Tue Sep 10 18:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.55100000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.54600000000000004}]},{"osd":6,"last update":"Tue Sep 10 18:21:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.628},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.56699999999999995}]},{"osd":7,"last update":"Tue Sep 10 18:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.60099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298812,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446340,"statfs":{"total":95995035648,"available":95689052160,"internally_reserved":0,"allocated":1490944,"data_stored":962229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36199999999999999,"15min":0.34999999999999998},"min":{"1min":0.23999999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.57999999999999996,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.372,"15min":0.35199999999999998},"min":{"1min":0.25600000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.58599999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.439}]},{"osd":2,"last update":"Tue Sep 10 18:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40999999999999998,"15min":0.40699999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53500000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.40600000000000003,"15min":0.40200000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.47899999999999998,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.376}]},{"osd":3,"last update":"Tue Sep 10 18:21:51 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.44600000000000001,"15min":0.44900000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.41999999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:22:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.624,"15min":0.63100000000000001},"min":{"1min":0.433,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.83299999999999996,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.65000000000000002,"15min":0.66100000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81899999999999995,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.66700000000000004}]},{"osd":5,"last update":"Tue Sep 10 18:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.59399999999999997,"15min":0.60299999999999998},"min":{"1min":0.437,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.72099999999999997,"5min":0.879,"15min":0.879},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.625,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.68400000000000005}]},{"osd":6,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.73499999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.65000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:22:54.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:22:55.026 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:55.283 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[39220]: 3.14 scrub starts 2024-09-10T18:22:55.283 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[39220]: 3.14 scrub ok 2024-09-10T18:22:55.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[39220]: 5.6 deep-scrub starts 2024-09-10T18:22:55.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[39220]: 5.6 deep-scrub ok 2024-09-10T18:22:55.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[39220]: from='client.14721 v1:172.21.15.28:0/193279802' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:22:55.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:22:55.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T18:22:55.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[29291]: 3.14 scrub starts 2024-09-10T18:22:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[29291]: 3.14 scrub ok 2024-09-10T18:22:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[29291]: 5.6 deep-scrub starts 2024-09-10T18:22:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[29291]: 5.6 deep-scrub ok 2024-09-10T18:22:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[29291]: from='client.14721 v1:172.21.15.28:0/193279802' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:22:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:22:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T18:22:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:55 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:55 smithi204 ceph-mon[32357]: 3.14 scrub starts 2024-09-10T18:22:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:55 smithi204 ceph-mon[32357]: 3.14 scrub ok 2024-09-10T18:22:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:55 smithi204 ceph-mon[32357]: 5.6 deep-scrub starts 2024-09-10T18:22:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:55 smithi204 ceph-mon[32357]: 5.6 deep-scrub ok 2024-09-10T18:22:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:55 smithi204 ceph-mon[32357]: from='client.14721 v1:172.21.15.28:0/193279802' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:22:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:55 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:22:55.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:55 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:55 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:55 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:55 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:55 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:55 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:55.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:55 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T18:22:55.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:55 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:56.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[29291]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T18:22:56.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[29291]: Deploying daemon grafana.a on smithi204 2024-09-10T18:22:56.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[29291]: 4.9 scrub starts 2024-09-10T18:22:56.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[29291]: 4.9 scrub ok 2024-09-10T18:22:56.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[29291]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:22:56.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[29291]: 4.a deep-scrub starts 2024-09-10T18:22:56.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[29291]: 4.a deep-scrub ok 2024-09-10T18:22:56.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[29291]: 3.1 scrub starts 2024-09-10T18:22:56.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[29291]: 3.1 scrub ok 2024-09-10T18:22:56.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[39220]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T18:22:56.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[39220]: Deploying daemon grafana.a on smithi204 2024-09-10T18:22:56.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[39220]: 4.9 scrub starts 2024-09-10T18:22:56.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[39220]: 4.9 scrub ok 2024-09-10T18:22:56.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[39220]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:22:56.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[39220]: 4.a deep-scrub starts 2024-09-10T18:22:56.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[39220]: 4.a deep-scrub ok 2024-09-10T18:22:56.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[39220]: 3.1 scrub starts 2024-09-10T18:22:56.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:56 smithi028 ceph-mon[39220]: 3.1 scrub ok 2024-09-10T18:22:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:56 smithi204 ceph-mon[32357]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T18:22:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:56 smithi204 ceph-mon[32357]: Deploying daemon grafana.a on smithi204 2024-09-10T18:22:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:56 smithi204 ceph-mon[32357]: 4.9 scrub starts 2024-09-10T18:22:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:56 smithi204 ceph-mon[32357]: 4.9 scrub ok 2024-09-10T18:22:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:56 smithi204 ceph-mon[32357]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:22:56.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:56 smithi204 ceph-mon[32357]: 4.a deep-scrub starts 2024-09-10T18:22:56.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:56 smithi204 ceph-mon[32357]: 4.a deep-scrub ok 2024-09-10T18:22:56.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:56 smithi204 ceph-mon[32357]: 3.1 scrub starts 2024-09-10T18:22:56.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:56 smithi204 ceph-mon[32357]: 3.1 scrub ok 2024-09-10T18:22:56.845 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:22:56.847 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:22:57.170 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:57 smithi028 ceph-mon[39220]: 6.6 scrub starts 2024-09-10T18:22:57.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:57 smithi028 ceph-mon[39220]: 6.6 scrub ok 2024-09-10T18:22:57.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:57 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:22:57.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:57 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:22:57.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:57 smithi028 ceph-mon[39220]: 3.1c scrub starts 2024-09-10T18:22:57.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:57 smithi028 ceph-mon[39220]: 3.1c scrub ok 2024-09-10T18:22:57.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:57 smithi028 ceph-mon[29291]: 6.6 scrub starts 2024-09-10T18:22:57.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:57 smithi028 ceph-mon[29291]: 6.6 scrub ok 2024-09-10T18:22:57.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:57 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:22:57.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:57 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:22:57.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:57 smithi028 ceph-mon[29291]: 3.1c scrub starts 2024-09-10T18:22:57.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:57 smithi028 ceph-mon[29291]: 3.1c scrub ok 2024-09-10T18:22:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:57 smithi204 ceph-mon[32357]: 6.6 scrub starts 2024-09-10T18:22:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:57 smithi204 ceph-mon[32357]: 6.6 scrub ok 2024-09-10T18:22:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:57 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:22:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:57 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:22:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:57 smithi204 ceph-mon[32357]: 3.1c scrub starts 2024-09-10T18:22:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:57 smithi204 ceph-mon[32357]: 3.1c scrub ok 2024-09-10T18:22:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-09-10T18:22:56.701151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":929,"num_read_kb":786,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":564,"ondisk_log_size":564,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2387584,"kb_used_data":8768,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747573632,"statfs":{"total":767960285184,"available":765515399168,"internally_reserved":0,"allocated":8978432,"data_stored":4748729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.172613"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198233+0000","last_change":"2024-09-10T18:21:31.526401+0000","last_active":"2024-09-10T18:22:44.198233+0000","last_peered":"2024-09-10T18:22:44.198233+0000","last_clean":"2024-09-10T18:22:44.198233+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:22:44.198233+0000","last_undegraded":"2024-09-10T18:22:44.198233+0000","last_fullsized":"2024-09-10T18:22:44.198233+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:58.540140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.990927+0000","last_change":"2024-09-10T18:21:33.614228+0000","last_active":"2024-09-10T18:22:43.990927+0000","last_peered":"2024-09-10T18:22:43.990927+0000","last_clean":"2024-09-10T18:22:43.990927+0000","last_became_active":"2024-09-10T18:21:33.613935+0000","last_became_peered":"2024-09-10T18:21:33.613935+0000","last_unstale":"2024-09-10T18:22:43.990927+0000","last_undegraded":"2024-09-10T18:22:43.990927+0000","last_fullsized":"2024-09-10T18:22:43.990927+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:00.978849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.107309+0000","last_change":"2024-09-10T18:21:35.619936+0000","last_active":"2024-09-10T18:22:44.107309+0000","last_peered":"2024-09-10T18:22:44.107309+0000","last_clean":"2024-09-10T18:22:44.107309+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:22:44.107309+0000","last_undegraded":"2024-09-10T18:22:44.107309+0000","last_fullsized":"2024-09-10T18:22:44.107309+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:52.780616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991044+0000","last_change":"2024-09-10T18:21:37.642760+0000","last_active":"2024-09-10T18:22:43.991044+0000","last_peered":"2024-09-10T18:22:43.991044+0000","last_clean":"2024-09-10T18:22:43.991044+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:22:43.991044+0000","last_undegraded":"2024-09-10T18:22:43.991044+0000","last_fullsized":"2024-09-10T18:22:43.991044+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989014+0000","last_change":"2024-09-10T18:21:38.565483+0000","last_active":"2024-09-10T18:22:43.989014+0000","last_peered":"2024-09-10T18:22:43.989014+0000","last_clean":"2024-09-10T18:22:43.989014+0000","last_became_active":"2024-09-10T18:21:38.565118+0000","last_became_peered":"2024-09-10T18:21:38.565118+0000","last_unstale":"2024-09-10T18:22:43.989014+0000","last_undegraded":"2024-09-10T18:22:43.989014+0000","last_fullsized":"2024-09-10T18:22:43.989014+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:55.214070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989692+0000","last_change":"2024-09-10T18:21:31.528038+0000","last_active":"2024-09-10T18:22:43.989692+0000","last_peered":"2024-09-10T18:22:43.989692+0000","last_clean":"2024-09-10T18:22:43.989692+0000","last_became_active":"2024-09-10T18:21:31.527582+0000","last_became_peered":"2024-09-10T18:21:31.527582+0000","last_unstale":"2024-09-10T18:22:43.989692+0000","last_undegraded":"2024-09-10T18:22:43.989692+0000","last_fullsized":"2024-09-10T18:22:43.989692+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:51.649992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.988961+0000","last_change":"2024-09-10T18:21:33.613829+0000","last_active":"2024-09-10T18:22:43.988961+0000","last_peered":"2024-09-10T18:22:43.988961+0000","last_clean":"2024-09-10T18:22:43.988961+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:22:43.988961+0000","last_undegraded":"2024-09-10T18:22:43.988961+0000","last_fullsized":"2024-09-10T18:22:43.988961+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:55.849000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.990886+0000","last_change":"2024-09-10T18:21:35.620924+0000","last_active":"2024-09-10T18:22:43.990886+0000","last_peered":"2024-09-10T18:22:43.990886+0000","last_clean":"2024-09-10T18:22:43.990886+0000","last_became_active":"2024-09-10T18:21:35.620585+0000","last_became_peered":"2024-09-10T18:21:35.620585+0000","last_unstale":"2024-09-10T18:22:43.990886+0000","last_undegraded":"2024-09-10T18:22:43.990886+0000","last_fullsized":"2024-09-10T18:22:43.990886+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110623+0000","last_change":"2024-09-10T18:21:31.524898+0000","last_active":"2024-09-10T18:22:44.110623+0000","last_peered":"2024-09-10T18:22:44.110623+0000","last_clean":"2024-09-10T18:22:44.110623+0000","last_became_active":"2024-09-10T18:21:31.524620+0000","last_became_peered":"2024-09-10T18:21:31.524620+0000","last_unstale":"2024-09-10T18:22:44.110623+0000","last_undegraded":"2024-09-10T18:22:44.110623+0000","last_fullsized":"2024-09-10T18:22:44.110623+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:58.405475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991268+0000","last_change":"2024-09-10T18:21:33.610754+0000","last_active":"2024-09-10T18:22:43.991268+0000","last_peered":"2024-09-10T18:22:43.991268+0000","last_clean":"2024-09-10T18:22:43.991268+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:22:43.991268+0000","last_undegraded":"2024-09-10T18:22:43.991268+0000","last_fullsized":"2024-09-10T18:22:43.991268+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:51.091191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:53.577957+0000","last_change":"2024-09-10T18:22:53.577905+0000","last_active":"2024-09-10T18:22:53.577957+0000","last_peered":"2024-09-10T18:22:53.577957+0000","last_clean":"2024-09-10T18:22:53.577957+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:22:53.577957+0000","last_undegraded":"2024-09-10T18:22:53.577957+0000","last_fullsized":"2024-09-10T18:22:53.577957+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:53.577826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:53.577826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:19.539271+0000","scrub_duration":0.001440577,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.328351+0000","last_change":"2024-09-10T18:22:46.328322+0000","last_active":"2024-09-10T18:22:46.328351+0000","last_peered":"2024-09-10T18:22:46.328351+0000","last_clean":"2024-09-10T18:22:46.328351+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:22:46.328351+0000","last_undegraded":"2024-09-10T18:22:46.328351+0000","last_fullsized":"2024-09-10T18:22:46.328351+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:46.328262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:46.328262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:50.476851+0000","scrub_duration":0.0024608049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110305+0000","last_change":"2024-09-10T18:21:31.528616+0000","last_active":"2024-09-10T18:22:44.110305+0000","last_peered":"2024-09-10T18:22:44.110305+0000","last_clean":"2024-09-10T18:22:44.110305+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:22:44.110305+0000","last_undegraded":"2024-09-10T18:22:44.110305+0000","last_fullsized":"2024-09-10T18:22:44.110305+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.563751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"85'7","reported_seq":56,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.365355+0000","last_change":"2024-09-10T18:22:44.365273+0000","last_active":"2024-09-10T18:22:44.365355+0000","last_peered":"2024-09-10T18:22:44.365355+0000","last_clean":"2024-09-10T18:22:44.365355+0000","last_became_active":"2024-09-10T18:21:33.615429+0000","last_became_peered":"2024-09-10T18:21:33.615429+0000","last_unstale":"2024-09-10T18:22:44.365355+0000","last_undegraded":"2024-09-10T18:22:44.365355+0000","last_fullsized":"2024-09-10T18:22:44.365355+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'7","last_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:22:44.365196+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:58.822595+0000","scrub_duration":0.0033059909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198357+0000","last_change":"2024-09-10T18:21:35.624352+0000","last_active":"2024-09-10T18:22:44.198357+0000","last_peered":"2024-09-10T18:22:44.198357+0000","last_clean":"2024-09-10T18:22:44.198357+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:22:44.198357+0000","last_undegraded":"2024-09-10T18:22:44.198357+0000","last_fullsized":"2024-09-10T18:22:44.198357+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:56.347578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.109933+0000","last_change":"2024-09-10T18:21:37.639437+0000","last_active":"2024-09-10T18:22:44.109933+0000","last_peered":"2024-09-10T18:22:44.109933+0000","last_clean":"2024-09-10T18:22:44.109933+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:22:44.109933+0000","last_undegraded":"2024-09-10T18:22:44.109933+0000","last_fullsized":"2024-09-10T18:22:44.109933+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:52.337945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991686+0000","last_change":"2024-09-10T18:21:38.563964+0000","last_active":"2024-09-10T18:22:43.991686+0000","last_peered":"2024-09-10T18:22:43.991686+0000","last_clean":"2024-09-10T18:22:43.991686+0000","last_became_active":"2024-09-10T18:21:38.563570+0000","last_became_peered":"2024-09-10T18:21:38.563570+0000","last_unstale":"2024-09-10T18:22:43.991686+0000","last_undegraded":"2024-09-10T18:22:43.991686+0000","last_fullsized":"2024-09-10T18:22:43.991686+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:47.491396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198587+0000","last_change":"2024-09-10T18:21:31.529843+0000","last_active":"2024-09-10T18:22:44.198587+0000","last_peered":"2024-09-10T18:22:44.198587+0000","last_clean":"2024-09-10T18:22:44.198587+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:22:44.198587+0000","last_undegraded":"2024-09-10T18:22:44.198587+0000","last_fullsized":"2024-09-10T18:22:44.198587+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:58.167486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198662+0000","last_change":"2024-09-10T18:21:33.617848+0000","last_active":"2024-09-10T18:22:44.198662+0000","last_peered":"2024-09-10T18:22:44.198662+0000","last_clean":"2024-09-10T18:22:44.198662+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:22:44.198662+0000","last_undegraded":"2024-09-10T18:22:44.198662+0000","last_fullsized":"2024-09-10T18:22:44.198662+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:58.180270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.107878+0000","last_change":"2024-09-10T18:22:39.007672+0000","last_active":"2024-09-10T18:22:44.107878+0000","last_peered":"2024-09-10T18:22:44.107878+0000","last_clean":"2024-09-10T18:22:44.107878+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:22:44.107878+0000","last_undegraded":"2024-09-10T18:22:44.107878+0000","last_fullsized":"2024-09-10T18:22:44.107878+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:39.007614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:39.007614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:59.162407+0000","scrub_duration":0.0021069719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989795+0000","last_change":"2024-09-10T18:21:38.565017+0000","last_active":"2024-09-10T18:22:43.989795+0000","last_peered":"2024-09-10T18:22:43.989795+0000","last_clean":"2024-09-10T18:22:43.989795+0000","last_became_active":"2024-09-10T18:21:38.564681+0000","last_became_peered":"2024-09-10T18:21:38.564681+0000","last_unstale":"2024-09-10T18:22:43.989795+0000","last_undegraded":"2024-09-10T18:22:43.989795+0000","last_fullsized":"2024-09-10T18:22:43.989795+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.656974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991392+0000","last_change":"2024-09-10T18:21:31.528392+0000","last_active":"2024-09-10T18:22:43.991392+0000","last_peered":"2024-09-10T18:22:43.991392+0000","last_clean":"2024-09-10T18:22:43.991392+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:22:43.991392+0000","last_undegraded":"2024-09-10T18:22:43.991392+0000","last_fullsized":"2024-09-10T18:22:43.991392+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":58,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989576+0000","last_change":"2024-09-10T18:21:33.617612+0000","last_active":"2024-09-10T18:22:43.989576+0000","last_peered":"2024-09-10T18:22:43.989576+0000","last_clean":"2024-09-10T18:22:43.989576+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:22:43.989576+0000","last_undegraded":"2024-09-10T18:22:43.989576+0000","last_fullsized":"2024-09-10T18:22:43.989576+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.240219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:50.436394+0000","last_change":"2024-09-10T18:22:50.436357+0000","last_active":"2024-09-10T18:22:50.436394+0000","last_peered":"2024-09-10T18:22:50.436394+0000","last_clean":"2024-09-10T18:22:50.436394+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:22:50.436394+0000","last_undegraded":"2024-09-10T18:22:50.436394+0000","last_fullsized":"2024-09-10T18:22:50.436394+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:50.436294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:50.436294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:07.783088+0000","scrub_duration":0.001715187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198650+0000","last_change":"2024-09-10T18:21:37.646675+0000","last_active":"2024-09-10T18:22:44.198650+0000","last_peered":"2024-09-10T18:22:44.198650+0000","last_clean":"2024-09-10T18:22:44.198650+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:22:44.198650+0000","last_undegraded":"2024-09-10T18:22:44.198650+0000","last_fullsized":"2024-09-10T18:22:44.198650+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:48.412991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.141701+0000","last_change":"2024-09-10T18:22:46.141545+0000","last_active":"2024-09-10T18:22:46.141701+0000","last_peered":"2024-09-10T18:22:46.141701+0000","last_clean":"2024-09-10T18:22:46.141701+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:22:46.141701+0000","last_undegraded":"2024-09-10T18:22:46.141701+0000","last_fullsized":"2024-09-10T18:22:46.141701+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:46.141408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:46.141408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:12.563610+0000","scrub_duration":0.0022977269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"86'15","reported_seq":64,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:49.298538+0000","last_change":"2024-09-10T18:22:49.298511+0000","last_active":"2024-09-10T18:22:49.298538+0000","last_peered":"2024-09-10T18:22:49.298538+0000","last_clean":"2024-09-10T18:22:49.298538+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:22:49.298538+0000","last_undegraded":"2024-09-10T18:22:49.298538+0000","last_fullsized":"2024-09-10T18:22:49.298538+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:22:49.298455+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:10.207495+0000","scrub_duration":0.0038705900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":69,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:48.795282+0000","last_change":"2024-09-10T18:22:48.795234+0000","last_active":"2024-09-10T18:22:48.795282+0000","last_peered":"2024-09-10T18:22:48.795282+0000","last_clean":"2024-09-10T18:22:48.795282+0000","last_became_active":"2024-09-10T18:21:35.624964+0000","last_became_peered":"2024-09-10T18:21:35.624964+0000","last_unstale":"2024-09-10T18:22:48.795282+0000","last_undegraded":"2024-09-10T18:22:48.795282+0000","last_fullsized":"2024-09-10T18:22:48.795282+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:22:48.795158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:48.795158+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:11.053077+0000","scrub_duration":0.0017552570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"83'22","reported_seq":75,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.108054+0000","last_change":"2024-09-10T18:22:40.007784+0000","last_active":"2024-09-10T18:22:44.108054+0000","last_peered":"2024-09-10T18:22:44.108054+0000","last_clean":"2024-09-10T18:22:44.108054+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:22:44.108054+0000","last_undegraded":"2024-09-10T18:22:44.108054+0000","last_fullsized":"2024-09-10T18:22:44.108054+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:22:40.007685+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:46.637635+0000","scrub_duration":0.0077173049999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.988922+0000","last_change":"2024-09-10T18:21:31.533221+0000","last_active":"2024-09-10T18:22:43.988922+0000","last_peered":"2024-09-10T18:22:43.988922+0000","last_clean":"2024-09-10T18:22:43.988922+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:22:43.988922+0000","last_undegraded":"2024-09-10T18:22:43.988922+0000","last_fullsized":"2024-09-10T18:22:43.988922+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:48.727349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":59,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.142536+0000","last_change":"2024-09-10T18:21:35.624518+0000","last_active":"2024-09-10T18:22:46.142536+0000","last_peered":"2024-09-10T18:22:46.142536+0000","last_clean":"2024-09-10T18:22:46.142536+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:22:46.142536+0000","last_undegraded":"2024-09-10T18:22:46.142536+0000","last_fullsized":"2024-09-10T18:22:46.142536+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110294+0000","last_change":"2024-09-10T18:21:37.644088+0000","last_active":"2024-09-10T18:22:44.110294+0000","last_peered":"2024-09-10T18:22:44.110294+0000","last_clean":"2024-09-10T18:22:44.110294+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:22:44.110294+0000","last_undegraded":"2024-09-10T18:22:44.110294+0000","last_fullsized":"2024-09-10T18:22:44.110294+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:55.585853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":67,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.988640+0000","last_change":"2024-09-10T18:21:34.047843+0000","last_active":"2024-09-10T18:22:43.988640+0000","last_peered":"2024-09-10T18:22:43.988640+0000","last_clean":"2024-09-10T18:22:43.988640+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:22:43.988640+0000","last_undegraded":"2024-09-10T18:22:43.988640+0000","last_fullsized":"2024-09-10T18:22:43.988640+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:55.071791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989941+0000","last_change":"2024-09-10T18:21:31.536352+0000","last_active":"2024-09-10T18:22:43.989941+0000","last_peered":"2024-09-10T18:22:43.989941+0000","last_clean":"2024-09-10T18:22:43.989941+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:22:43.989941+0000","last_undegraded":"2024-09-10T18:22:43.989941+0000","last_fullsized":"2024-09-10T18:22:43.989941+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:48.615680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:47.386424+0000","last_change":"2024-09-10T18:22:47.386371+0000","last_active":"2024-09-10T18:22:47.386424+0000","last_peered":"2024-09-10T18:22:47.386424+0000","last_clean":"2024-09-10T18:22:47.386424+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:22:47.386424+0000","last_undegraded":"2024-09-10T18:22:47.386424+0000","last_fullsized":"2024-09-10T18:22:47.386424+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:47.386303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:47.386303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:01.335164+0000","scrub_duration":0.001725484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991180+0000","last_change":"2024-09-10T18:21:37.642918+0000","last_active":"2024-09-10T18:22:43.991180+0000","last_peered":"2024-09-10T18:22:43.991180+0000","last_clean":"2024-09-10T18:22:43.991180+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:22:43.991180+0000","last_undegraded":"2024-09-10T18:22:43.991180+0000","last_fullsized":"2024-09-10T18:22:43.991180+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991114+0000","last_change":"2024-09-10T18:22:37.322150+0000","last_active":"2024-09-10T18:22:43.991114+0000","last_peered":"2024-09-10T18:22:43.991114+0000","last_clean":"2024-09-10T18:22:43.991114+0000","last_became_active":"2024-09-10T18:21:33.607574+0000","last_became_peered":"2024-09-10T18:21:33.607574+0000","last_unstale":"2024-09-10T18:22:43.991114+0000","last_undegraded":"2024-09-10T18:22:43.991114+0000","last_fullsized":"2024-09-10T18:22:43.991114+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-10T18:22:37.322094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:37.322094+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:41.781070+0000","scrub_duration":0.001716817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198915+0000","last_change":"2024-09-10T18:21:31.531508+0000","last_active":"2024-09-10T18:22:44.198915+0000","last_peered":"2024-09-10T18:22:44.198915+0000","last_clean":"2024-09-10T18:22:44.198915+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:22:44.198915+0000","last_undegraded":"2024-09-10T18:22:44.198915+0000","last_fullsized":"2024-09-10T18:22:44.198915+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.249073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":57,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.142894+0000","last_change":"2024-09-10T18:21:35.623304+0000","last_active":"2024-09-10T18:22:46.142894+0000","last_peered":"2024-09-10T18:22:46.142894+0000","last_clean":"2024-09-10T18:22:46.142894+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:22:46.142894+0000","last_undegraded":"2024-09-10T18:22:46.142894+0000","last_fullsized":"2024-09-10T18:22:46.142894+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:51.799895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198454+0000","last_change":"2024-09-10T18:21:38.566158+0000","last_active":"2024-09-10T18:22:44.198454+0000","last_peered":"2024-09-10T18:22:44.198454+0000","last_clean":"2024-09-10T18:22:44.198454+0000","last_became_active":"2024-09-10T18:21:38.565165+0000","last_became_peered":"2024-09-10T18:21:38.565165+0000","last_unstale":"2024-09-10T18:22:44.198454+0000","last_undegraded":"2024-09-10T18:22:44.198454+0000","last_fullsized":"2024-09-10T18:22:44.198454+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.107686+0000","last_change":"2024-09-10T18:21:33.614039+0000","last_active":"2024-09-10T18:22:44.107686+0000","last_peered":"2024-09-10T18:22:44.107686+0000","last_clean":"2024-09-10T18:22:44.107686+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:22:44.107686+0000","last_undegraded":"2024-09-10T18:22:44.107686+0000","last_fullsized":"2024-09-10T18:22:44.107686+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.533971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110662+0000","last_change":"2024-09-10T18:21:31.534170+0000","last_active":"2024-09-10T18:22:44.110662+0000","last_peered":"2024-09-10T18:22:44.110662+0000","last_clean":"2024-09-10T18:22:44.110662+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:22:44.110662+0000","last_undegraded":"2024-09-10T18:22:44.110662+0000","last_fullsized":"2024-09-10T18:22:44.110662+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:54.289170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":56,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.327188+0000","last_change":"2024-09-10T18:21:35.618920+0000","last_active":"2024-09-10T18:22:46.327188+0000","last_peered":"2024-09-10T18:22:46.327188+0000","last_clean":"2024-09-10T18:22:46.327188+0000","last_became_active":"2024-09-10T18:21:35.617821+0000","last_became_peered":"2024-09-10T18:21:35.617821+0000","last_unstale":"2024-09-10T18:22:46.327188+0000","last_undegraded":"2024-09-10T18:22:46.327188+0000","last_fullsized":"2024-09-10T18:22:46.327188+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:59.357441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.197873+0000","last_change":"2024-09-10T18:22:41.399132+0000","last_active":"2024-09-10T18:22:44.197873+0000","last_peered":"2024-09-10T18:22:44.197873+0000","last_clean":"2024-09-10T18:22:44.197873+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:22:44.197873+0000","last_undegraded":"2024-09-10T18:22:44.197873+0000","last_fullsized":"2024-09-10T18:22:44.197873+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:41.399078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:41.399078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:48.673347+0000","scrub_duration":0.001245323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198369+0000","last_change":"2024-09-10T18:21:34.046468+0000","last_active":"2024-09-10T18:22:44.198369+0000","last_peered":"2024-09-10T18:22:44.198369+0000","last_clean":"2024-09-10T18:22:44.198369+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:22:44.198369+0000","last_undegraded":"2024-09-10T18:22:44.198369+0000","last_fullsized":"2024-09-10T18:22:44.198369+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:47.310301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.108359+0000","last_change":"2024-09-10T18:21:31.530156+0000","last_active":"2024-09-10T18:22:44.108359+0000","last_peered":"2024-09-10T18:22:44.108359+0000","last_clean":"2024-09-10T18:22:44.108359+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:22:44.108359+0000","last_undegraded":"2024-09-10T18:22:44.108359+0000","last_fullsized":"2024-09-10T18:22:44.108359+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:59.013351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.172564+0000","last_change":"2024-09-10T18:22:38.853925+0000","last_active":"2024-09-10T18:22:44.172564+0000","last_peered":"2024-09-10T18:22:44.172564+0000","last_clean":"2024-09-10T18:22:44.172564+0000","last_became_active":"2024-09-10T18:21:35.617132+0000","last_became_peered":"2024-09-10T18:21:35.617132+0000","last_unstale":"2024-09-10T18:22:44.172564+0000","last_undegraded":"2024-09-10T18:22:44.172564+0000","last_fullsized":"2024-09-10T18:22:44.172564+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:38.853839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:38.853839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:39.087399+0000","scrub_duration":0.0031500500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.107278+0000","last_change":"2024-09-10T18:21:38.564042+0000","last_active":"2024-09-10T18:22:44.107278+0000","last_peered":"2024-09-10T18:22:44.107278+0000","last_clean":"2024-09-10T18:22:44.107278+0000","last_became_active":"2024-09-10T18:21:38.563629+0000","last_became_peered":"2024-09-10T18:21:38.563629+0000","last_unstale":"2024-09-10T18:22:44.107278+0000","last_undegraded":"2024-09-10T18:22:44.107278+0000","last_fullsized":"2024-09-10T18:22:44.107278+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:04.135213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":97,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.140250+0000","last_change":"2024-09-10T18:21:33.618915+0000","last_active":"2024-09-10T18:22:46.140250+0000","last_peered":"2024-09-10T18:22:46.140250+0000","last_clean":"2024-09-10T18:22:46.140250+0000","last_became_active":"2024-09-10T18:21:33.618616+0000","last_became_peered":"2024-09-10T18:21:33.618616+0000","last_unstale":"2024-09-10T18:22:46.140250+0000","last_undegraded":"2024-09-10T18:22:46.140250+0000","last_fullsized":"2024-09-10T18:22:46.140250+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991579+0000","last_change":"2024-09-10T18:22:42.335215+0000","last_active":"2024-09-10T18:22:43.991579+0000","last_peered":"2024-09-10T18:22:43.991579+0000","last_clean":"2024-09-10T18:22:43.991579+0000","last_became_active":"2024-09-10T18:21:31.526161+0000","last_became_peered":"2024-09-10T18:21:31.526161+0000","last_unstale":"2024-09-10T18:22:43.991579+0000","last_undegraded":"2024-09-10T18:22:43.991579+0000","last_fullsized":"2024-09-10T18:22:43.991579+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:42.335152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:42.335152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:02.796505+0000","scrub_duration":0.0013614720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"62'2","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110347+0000","last_change":"2024-09-10T18:21:35.613488+0000","last_active":"2024-09-10T18:22:44.110347+0000","last_peered":"2024-09-10T18:22:44.110347+0000","last_clean":"2024-09-10T18:22:44.110347+0000","last_became_active":"2024-09-10T18:21:31.528970+0000","last_became_peered":"2024-09-10T18:21:31.528970+0000","last_unstale":"2024-09-10T18:22:44.110347+0000","last_undegraded":"2024-09-10T18:22:44.110347+0000","last_fullsized":"2024-09-10T18:22:44.110347+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:51.822022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198385+0000","last_change":"2024-09-10T18:22:43.309177+0000","last_active":"2024-09-10T18:22:44.198385+0000","last_peered":"2024-09-10T18:22:44.198385+0000","last_clean":"2024-09-10T18:22:44.198385+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:22:44.198385+0000","last_undegraded":"2024-09-10T18:22:44.198385+0000","last_fullsized":"2024-09-10T18:22:44.198385+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:43.309115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:43.309115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:08.721305+0000","scrub_duration":0.0015388929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989504+0000","last_change":"2024-09-10T18:21:37.641789+0000","last_active":"2024-09-10T18:22:43.989504+0000","last_peered":"2024-09-10T18:22:43.989504+0000","last_clean":"2024-09-10T18:22:43.989504+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:22:43.989504+0000","last_undegraded":"2024-09-10T18:22:43.989504+0000","last_fullsized":"2024-09-10T18:22:43.989504+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:47.975239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.108136+0000","last_change":"2024-09-10T18:21:33.618163+0000","last_active":"2024-09-10T18:22:44.108136+0000","last_peered":"2024-09-10T18:22:44.108136+0000","last_clean":"2024-09-10T18:22:44.108136+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:22:44.108136+0000","last_undegraded":"2024-09-10T18:22:44.108136+0000","last_fullsized":"2024-09-10T18:22:44.108136+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:50.603962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:52.160864+0000","last_change":"2024-09-10T18:22:52.160807+0000","last_active":"2024-09-10T18:22:52.160864+0000","last_peered":"2024-09-10T18:22:52.160864+0000","last_clean":"2024-09-10T18:22:52.160864+0000","last_became_active":"2024-09-10T18:21:31.532624+0000","last_became_peered":"2024-09-10T18:21:31.532624+0000","last_unstale":"2024-09-10T18:22:52.160864+0000","last_undegraded":"2024-09-10T18:22:52.160864+0000","last_fullsized":"2024-09-10T18:22:52.160864+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:52.160732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:52.160732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:11.270210+0000","scrub_duration":0.0015264289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":52,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.337307+0000","last_change":"2024-09-10T18:22:46.337257+0000","last_active":"2024-09-10T18:22:46.337307+0000","last_peered":"2024-09-10T18:22:46.337307+0000","last_clean":"2024-09-10T18:22:46.337307+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:22:46.337307+0000","last_undegraded":"2024-09-10T18:22:46.337307+0000","last_fullsized":"2024-09-10T18:22:46.337307+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:22:46.337186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:46.337186+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:11.152611+0000","scrub_duration":0.0015240410000000001,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:53.488519+0000","last_change":"2024-09-10T18:22:53.488490+0000","last_active":"2024-09-10T18:22:53.488519+0000","last_peered":"2024-09-10T18:22:53.488519+0000","last_clean":"2024-09-10T18:22:53.488519+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:22:53.488519+0000","last_undegraded":"2024-09-10T18:22:53.488519+0000","last_fullsized":"2024-09-10T18:22:53.488519+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:22:53.488434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:15.445322+0000","scrub_duration":0.001619882,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198757+0000","last_change":"2024-09-10T18:21:38.565470+0000","last_active":"2024-09-10T18:22:44.198757+0000","last_peered":"2024-09-10T18:22:44.198757+0000","last_clean":"2024-09-10T18:22:44.198757+0000","last_became_active":"2024-09-10T18:21:38.565183+0000","last_became_peered":"2024-09-10T18:21:38.565183+0000","last_unstale":"2024-09-10T18:22:44.198757+0000","last_undegraded":"2024-09-10T18:22:44.198757+0000","last_fullsized":"2024-09-10T18:22:44.198757+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:04.594058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":62,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:51.280304+0000","last_change":"2024-09-10T18:22:51.280276+0000","last_active":"2024-09-10T18:22:51.280304+0000","last_peered":"2024-09-10T18:22:51.280304+0000","last_clean":"2024-09-10T18:22:51.280304+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:22:51.280304+0000","last_undegraded":"2024-09-10T18:22:51.280304+0000","last_fullsized":"2024-09-10T18:22:51.280304+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:22:51.280219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:51.280219+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:13.028302+0000","scrub_duration":0.0021524019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198058+0000","last_change":"2024-09-10T18:21:31.534516+0000","last_active":"2024-09-10T18:22:44.198058+0000","last_peered":"2024-09-10T18:22:44.198058+0000","last_clean":"2024-09-10T18:22:44.198058+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:22:44.198058+0000","last_undegraded":"2024-09-10T18:22:44.198058+0000","last_fullsized":"2024-09-10T18:22:44.198058+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":130,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:52.919656+0000","last_change":"2024-09-10T18:22:41.256321+0000","last_active":"2024-09-10T18:22:52.919656+0000","last_peered":"2024-09-10T18:22:52.919656+0000","last_clean":"2024-09-10T18:22:52.919656+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:22:52.919656+0000","last_undegraded":"2024-09-10T18:22:52.919656+0000","last_fullsized":"2024-09-10T18:22:52.919656+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:22:41.256255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:41.256255+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:01.661114+0000","scrub_duration":0.0015771470000000001,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110722+0000","last_change":"2024-09-10T18:21:35.619757+0000","last_active":"2024-09-10T18:22:44.110722+0000","last_peered":"2024-09-10T18:22:44.110722+0000","last_clean":"2024-09-10T18:22:44.110722+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:22:44.110722+0000","last_undegraded":"2024-09-10T18:22:44.110722+0000","last_fullsized":"2024-09-10T18:22:44.110722+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:52.027113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.107999+0000","last_change":"2024-09-10T18:21:37.639920+0000","last_active":"2024-09-10T18:22:44.107999+0000","last_peered":"2024-09-10T18:22:44.107999+0000","last_clean":"2024-09-10T18:22:44.107999+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:22:44.107999+0000","last_undegraded":"2024-09-10T18:22:44.107999+0000","last_fullsized":"2024-09-10T18:22:44.107999+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:04.764182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":56,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.173038+0000","last_change":"2024-09-10T18:21:34.046018+0000","last_active":"2024-09-10T18:22:44.173038+0000","last_peered":"2024-09-10T18:22:44.173038+0000","last_clean":"2024-09-10T18:22:44.173038+0000","last_became_active":"2024-09-10T18:21:34.045549+0000","last_became_peered":"2024-09-10T18:21:34.045549+0000","last_unstale":"2024-09-10T18:22:44.173038+0000","last_undegraded":"2024-09-10T18:22:44.173038+0000","last_fullsized":"2024-09-10T18:22:44.173038+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:52.816356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:45.383886+0000","last_change":"2024-09-10T18:22:45.383833+0000","last_active":"2024-09-10T18:22:45.383886+0000","last_peered":"2024-09-10T18:22:45.383886+0000","last_clean":"2024-09-10T18:22:45.383886+0000","last_became_active":"2024-09-10T18:21:31.529816+0000","last_became_peered":"2024-09-10T18:21:31.529816+0000","last_unstale":"2024-09-10T18:22:45.383886+0000","last_undegraded":"2024-09-10T18:22:45.383886+0000","last_fullsized":"2024-09-10T18:22:45.383886+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:45.383768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:45.383768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:06.766629+0000","scrub_duration":0.001512844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":80,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198758+0000","last_change":"2024-09-10T18:22:17.024622+0000","last_active":"2024-09-10T18:22:44.198758+0000","last_peered":"2024-09-10T18:22:44.198758+0000","last_clean":"2024-09-10T18:22:44.198758+0000","last_became_active":"2024-09-10T18:21:20.570983+0000","last_became_peered":"2024-09-10T18:21:20.570983+0000","last_unstale":"2024-09-10T18:22:44.198758+0000","last_undegraded":"2024-09-10T18:22:44.198758+0000","last_fullsized":"2024-09-10T18:22:44.198758+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:22:17.024553+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:21.328129+0000","scrub_duration":0.003148245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198776+0000","last_change":"2024-09-10T18:21:35.621370+0000","last_active":"2024-09-10T18:22:44.198776+0000","last_peered":"2024-09-10T18:22:44.198776+0000","last_clean":"2024-09-10T18:22:44.198776+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:22:44.198776+0000","last_undegraded":"2024-09-10T18:22:44.198776+0000","last_fullsized":"2024-09-10T18:22:44.198776+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:56.537273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110658+0000","last_change":"2024-09-10T18:21:37.638814+0000","last_active":"2024-09-10T18:22:44.110658+0000","last_peered":"2024-09-10T18:22:44.110658+0000","last_clean":"2024-09-10T18:22:44.110658+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:22:44.110658+0000","last_undegraded":"2024-09-10T18:22:44.110658+0000","last_fullsized":"2024-09-10T18:22:44.110658+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:06.014250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991781+0000","last_change":"2024-09-10T18:21:33.614556+0000","last_active":"2024-09-10T18:22:43.991781+0000","last_peered":"2024-09-10T18:22:43.991781+0000","last_clean":"2024-09-10T18:22:43.991781+0000","last_became_active":"2024-09-10T18:21:33.614090+0000","last_became_peered":"2024-09-10T18:21:33.614090+0000","last_unstale":"2024-09-10T18:22:43.991781+0000","last_undegraded":"2024-09-10T18:22:43.991781+0000","last_fullsized":"2024-09-10T18:22:43.991781+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:57.016893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991498+0000","last_change":"2024-09-10T18:21:31.529705+0000","last_active":"2024-09-10T18:22:43.991498+0000","last_peered":"2024-09-10T18:22:43.991498+0000","last_clean":"2024-09-10T18:22:43.991498+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:22:43.991498+0000","last_undegraded":"2024-09-10T18:22:43.991498+0000","last_fullsized":"2024-09-10T18:22:43.991498+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:56.165261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110581+0000","last_change":"2024-09-10T18:21:35.620645+0000","last_active":"2024-09-10T18:22:44.110581+0000","last_peered":"2024-09-10T18:22:44.110581+0000","last_clean":"2024-09-10T18:22:44.110581+0000","last_became_active":"2024-09-10T18:21:35.620331+0000","last_became_peered":"2024-09-10T18:21:35.620331+0000","last_unstale":"2024-09-10T18:22:44.110581+0000","last_undegraded":"2024-09-10T18:22:44.110581+0000","last_fullsized":"2024-09-10T18:22:44.110581+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:55.576218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:50.298425+0000","last_change":"2024-09-10T18:22:50.298390+0000","last_active":"2024-09-10T18:22:50.298425+0000","last_peered":"2024-09-10T18:22:50.298425+0000","last_clean":"2024-09-10T18:22:50.298425+0000","last_became_active":"2024-09-10T18:21:38.563450+0000","last_became_peered":"2024-09-10T18:21:38.563450+0000","last_unstale":"2024-09-10T18:22:50.298425+0000","last_undegraded":"2024-09-10T18:22:50.298425+0000","last_fullsized":"2024-09-10T18:22:50.298425+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:50.298320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:50.298320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:19.836783+0000","scrub_duration":0.0021610420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991713+0000","last_change":"2024-09-10T18:21:33.611478+0000","last_active":"2024-09-10T18:22:43.991713+0000","last_peered":"2024-09-10T18:22:43.991713+0000","last_clean":"2024-09-10T18:22:43.991713+0000","last_became_active":"2024-09-10T18:21:33.611203+0000","last_became_peered":"2024-09-10T18:21:33.611203+0000","last_unstale":"2024-09-10T18:22:43.991713+0000","last_undegraded":"2024-09-10T18:22:43.991713+0000","last_fullsized":"2024-09-10T18:22:43.991713+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:58.663030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"58'1","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.172795+0000","last_change":"2024-09-10T18:21:31.528999+0000","last_active":"2024-09-10T18:22:44.172795+0000","last_peered":"2024-09-10T18:22:44.172795+0000","last_clean":"2024-09-10T18:22:44.172795+0000","last_became_active":"2024-09-10T18:21:31.528748+0000","last_became_peered":"2024-09-10T18:21:31.528748+0000","last_unstale":"2024-09-10T18:22:44.172795+0000","last_undegraded":"2024-09-10T18:22:44.172795+0000","last_fullsized":"2024-09-10T18:22:44.172795+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:00.198506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.107651+0000","last_change":"2024-09-10T18:21:35.625340+0000","last_active":"2024-09-10T18:22:44.107651+0000","last_peered":"2024-09-10T18:22:44.107651+0000","last_clean":"2024-09-10T18:22:44.107651+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:22:44.107651+0000","last_undegraded":"2024-09-10T18:22:44.107651+0000","last_fullsized":"2024-09-10T18:22:44.107651+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:51.126655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:52.486839+0000","last_change":"2024-09-10T18:22:52.486807+0000","last_active":"2024-09-10T18:22:52.486839+0000","last_peered":"2024-09-10T18:22:52.486839+0000","last_clean":"2024-09-10T18:22:52.486839+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:22:52.486839+0000","last_undegraded":"2024-09-10T18:22:52.486839+0000","last_fullsized":"2024-09-10T18:22:52.486839+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:52.486737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:52.486737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:59.120450+0000","scrub_duration":0.0015615,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991335+0000","last_change":"2024-09-10T18:21:33.614985+0000","last_active":"2024-09-10T18:22:43.991335+0000","last_peered":"2024-09-10T18:22:43.991335+0000","last_clean":"2024-09-10T18:22:43.991335+0000","last_became_active":"2024-09-10T18:21:33.614689+0000","last_became_peered":"2024-09-10T18:21:33.614689+0000","last_unstale":"2024-09-10T18:22:43.991335+0000","last_undegraded":"2024-09-10T18:22:43.991335+0000","last_fullsized":"2024-09-10T18:22:43.991335+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:57.940565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989276+0000","last_change":"2024-09-10T18:21:31.535458+0000","last_active":"2024-09-10T18:22:43.989276+0000","last_peered":"2024-09-10T18:22:43.989276+0000","last_clean":"2024-09-10T18:22:43.989276+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:22:43.989276+0000","last_undegraded":"2024-09-10T18:22:43.989276+0000","last_fullsized":"2024-09-10T18:22:43.989276+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:55.260904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":66,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:51.492728+0000","last_change":"2024-09-10T18:22:51.492683+0000","last_active":"2024-09-10T18:22:51.492728+0000","last_peered":"2024-09-10T18:22:51.492728+0000","last_clean":"2024-09-10T18:22:51.492728+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:22:51.492728+0000","last_undegraded":"2024-09-10T18:22:51.492728+0000","last_fullsized":"2024-09-10T18:22:51.492728+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:22:51.492580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:51.492580+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:00.192242+0000","scrub_duration":0.002035987,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991650+0000","last_change":"2024-09-10T18:21:37.642233+0000","last_active":"2024-09-10T18:22:43.991650+0000","last_peered":"2024-09-10T18:22:43.991650+0000","last_clean":"2024-09-10T18:22:43.991650+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:22:43.991650+0000","last_undegraded":"2024-09-10T18:22:43.991650+0000","last_fullsized":"2024-09-10T18:22:43.991650+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198288+0000","last_change":"2024-09-10T18:21:33.614532+0000","last_active":"2024-09-10T18:22:44.198288+0000","last_peered":"2024-09-10T18:22:44.198288+0000","last_clean":"2024-09-10T18:22:44.198288+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:22:44.198288+0000","last_undegraded":"2024-09-10T18:22:44.198288+0000","last_fullsized":"2024-09-10T18:22:44.198288+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:57.959502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"58'1","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110399+0000","last_change":"2024-09-10T18:21:31.534677+0000","last_active":"2024-09-10T18:22:44.110399+0000","last_peered":"2024-09-10T18:22:44.110399+0000","last_clean":"2024-09-10T18:22:44.110399+0000","last_became_active":"2024-09-10T18:21:31.534441+0000","last_became_peered":"2024-09-10T18:21:31.534441+0000","last_unstale":"2024-09-10T18:22:44.110399+0000","last_undegraded":"2024-09-10T18:22:44.110399+0000","last_fullsized":"2024-09-10T18:22:44.110399+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.618552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198361+0000","last_change":"2024-09-10T18:22:41.279395+0000","last_active":"2024-09-10T18:22:44.198361+0000","last_peered":"2024-09-10T18:22:44.198361+0000","last_clean":"2024-09-10T18:22:44.198361+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:22:44.198361+0000","last_undegraded":"2024-09-10T18:22:44.198361+0000","last_fullsized":"2024-09-10T18:22:44.198361+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:41.279342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:41.279342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:48.532838+0000","scrub_duration":0.0017787790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198014+0000","last_change":"2024-09-10T18:21:37.641253+0000","last_active":"2024-09-10T18:22:44.198014+0000","last_peered":"2024-09-10T18:22:44.198014+0000","last_clean":"2024-09-10T18:22:44.198014+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:22:44.198014+0000","last_undegraded":"2024-09-10T18:22:44.198014+0000","last_fullsized":"2024-09-10T18:22:44.198014+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.172596+0000","last_change":"2024-09-10T18:21:34.046749+0000","last_active":"2024-09-10T18:22:44.172596+0000","last_peered":"2024-09-10T18:22:44.172596+0000","last_clean":"2024-09-10T18:22:44.172596+0000","last_became_active":"2024-09-10T18:21:34.046453+0000","last_became_peered":"2024-09-10T18:21:34.046453+0000","last_unstale":"2024-09-10T18:22:44.172596+0000","last_undegraded":"2024-09-10T18:22:44.172596+0000","last_fullsized":"2024-09-10T18:22:44.172596+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.046130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.199059+0000","last_change":"2024-09-10T18:22:32.137670+0000","last_active":"2024-09-10T18:22:44.199059+0000","last_peered":"2024-09-10T18:22:44.199059+0000","last_clean":"2024-09-10T18:22:44.199059+0000","last_became_active":"2024-09-10T18:21:31.531951+0000","last_became_peered":"2024-09-10T18:21:31.531951+0000","last_unstale":"2024-09-10T18:22:44.199059+0000","last_undegraded":"2024-09-10T18:22:44.199059+0000","last_fullsized":"2024-09-10T18:22:44.199059+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:32.137600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:32.137600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:41.301280+0000","scrub_duration":0.00151761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198130+0000","last_change":"2024-09-10T18:22:42.312458+0000","last_active":"2024-09-10T18:22:44.198130+0000","last_peered":"2024-09-10T18:22:44.198130+0000","last_clean":"2024-09-10T18:22:44.198130+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:22:44.198130+0000","last_undegraded":"2024-09-10T18:22:44.198130+0000","last_fullsized":"2024-09-10T18:22:44.198130+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:42.312403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:42.312403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:42.394403+0000","scrub_duration":0.0014316489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198593+0000","last_change":"2024-09-10T18:21:37.642296+0000","last_active":"2024-09-10T18:22:44.198593+0000","last_peered":"2024-09-10T18:22:44.198593+0000","last_clean":"2024-09-10T18:22:44.198593+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:22:44.198593+0000","last_undegraded":"2024-09-10T18:22:44.198593+0000","last_fullsized":"2024-09-10T18:22:44.198593+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":67,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:54.449673+0000","last_change":"2024-09-10T18:22:54.449637+0000","last_active":"2024-09-10T18:22:54.449673+0000","last_peered":"2024-09-10T18:22:54.449673+0000","last_clean":"2024-09-10T18:22:54.449673+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:22:54.449673+0000","last_undegraded":"2024-09-10T18:22:54.449673+0000","last_fullsized":"2024-09-10T18:22:54.449673+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:22:54.449573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:54.449573+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:07.055885+0000","scrub_duration":0.0022198460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.199220+0000","last_change":"2024-09-10T18:21:31.531998+0000","last_active":"2024-09-10T18:22:44.199220+0000","last_peered":"2024-09-10T18:22:44.199220+0000","last_clean":"2024-09-10T18:22:44.199220+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:22:44.199220+0000","last_undegraded":"2024-09-10T18:22:44.199220+0000","last_fullsized":"2024-09-10T18:22:44.199220+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:57.379141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.197789+0000","last_change":"2024-09-10T18:22:40.275165+0000","last_active":"2024-09-10T18:22:44.197789+0000","last_peered":"2024-09-10T18:22:44.197789+0000","last_clean":"2024-09-10T18:22:44.197789+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:22:44.197789+0000","last_undegraded":"2024-09-10T18:22:44.197789+0000","last_fullsized":"2024-09-10T18:22:44.197789+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:40.275069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:40.275069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:54.251771+0000","scrub_duration":0.001522121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989104+0000","last_change":"2024-09-10T18:21:37.644889+0000","last_active":"2024-09-10T18:22:43.989104+0000","last_peered":"2024-09-10T18:22:43.989104+0000","last_clean":"2024-09-10T18:22:43.989104+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:22:43.989104+0000","last_undegraded":"2024-09-10T18:22:43.989104+0000","last_fullsized":"2024-09-10T18:22:43.989104+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:59.247137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":69,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:51.599694+0000","last_change":"2024-09-10T18:22:51.599652+0000","last_active":"2024-09-10T18:22:51.599694+0000","last_peered":"2024-09-10T18:22:51.599694+0000","last_clean":"2024-09-10T18:22:51.599694+0000","last_became_active":"2024-09-10T18:21:34.046741+0000","last_became_peered":"2024-09-10T18:21:34.046741+0000","last_unstale":"2024-09-10T18:22:51.599694+0000","last_undegraded":"2024-09-10T18:22:51.599694+0000","last_fullsized":"2024-09-10T18:22:51.599694+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-10T18:22:51.599559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:51.599559+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:02.843581+0000","scrub_duration":0.0019152609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.199062+0000","last_change":"2024-09-10T18:21:31.530630+0000","last_active":"2024-09-10T18:22:44.199062+0000","last_peered":"2024-09-10T18:22:44.199062+0000","last_clean":"2024-09-10T18:22:44.199062+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:22:44.199062+0000","last_undegraded":"2024-09-10T18:22:44.199062+0000","last_fullsized":"2024-09-10T18:22:44.199062+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":56,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.176047+0000","last_change":"2024-09-10T18:21:35.627197+0000","last_active":"2024-09-10T18:22:46.176047+0000","last_peered":"2024-09-10T18:22:46.176047+0000","last_clean":"2024-09-10T18:22:46.176047+0000","last_became_active":"2024-09-10T18:21:35.626744+0000","last_became_peered":"2024-09-10T18:21:35.626744+0000","last_unstale":"2024-09-10T18:22:46.176047+0000","last_undegraded":"2024-09-10T18:22:46.176047+0000","last_fullsized":"2024-09-10T18:22:46.176047+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:58.371060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.109996+0000","last_change":"2024-09-10T18:21:38.563824+0000","last_active":"2024-09-10T18:22:44.109996+0000","last_peered":"2024-09-10T18:22:44.109996+0000","last_clean":"2024-09-10T18:22:44.109996+0000","last_became_active":"2024-09-10T18:21:38.563550+0000","last_became_peered":"2024-09-10T18:21:38.563550+0000","last_unstale":"2024-09-10T18:22:44.109996+0000","last_undegraded":"2024-09-10T18:22:44.109996+0000","last_fullsized":"2024-09-10T18:22:44.109996+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:05.155686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.172510+0000","last_change":"2024-09-10T18:21:31.527287+0000","last_active":"2024-09-10T18:22:44.172510+0000","last_peered":"2024-09-10T18:22:44.172510+0000","last_clean":"2024-09-10T18:22:44.172510+0000","last_became_active":"2024-09-10T18:21:31.526818+0000","last_became_peered":"2024-09-10T18:21:31.526818+0000","last_unstale":"2024-09-10T18:22:44.172510+0000","last_undegraded":"2024-09-10T18:22:44.172510+0000","last_fullsized":"2024-09-10T18:22:44.172510+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:53.818450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989883+0000","last_change":"2024-09-10T18:21:34.046070+0000","last_active":"2024-09-10T18:22:43.989883+0000","last_peered":"2024-09-10T18:22:43.989883+0000","last_clean":"2024-09-10T18:22:43.989883+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:22:43.989883+0000","last_undegraded":"2024-09-10T18:22:43.989883+0000","last_fullsized":"2024-09-10T18:22:43.989883+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:54.221783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110546+0000","last_change":"2024-09-10T18:21:35.622254+0000","last_active":"2024-09-10T18:22:44.110546+0000","last_peered":"2024-09-10T18:22:44.110546+0000","last_clean":"2024-09-10T18:22:44.110546+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:22:44.110546+0000","last_undegraded":"2024-09-10T18:22:44.110546+0000","last_fullsized":"2024-09-10T18:22:44.110546+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:54.549955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198867+0000","last_change":"2024-09-10T18:21:38.565959+0000","last_active":"2024-09-10T18:22:44.198867+0000","last_peered":"2024-09-10T18:22:44.198867+0000","last_clean":"2024-09-10T18:22:44.198867+0000","last_became_active":"2024-09-10T18:21:38.565675+0000","last_became_peered":"2024-09-10T18:21:38.565675+0000","last_unstale":"2024-09-10T18:22:44.198867+0000","last_undegraded":"2024-09-10T18:22:44.198867+0000","last_fullsized":"2024-09-10T18:22:44.198867+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198837+0000","last_change":"2024-09-10T18:21:34.047798+0000","last_active":"2024-09-10T18:22:44.198837+0000","last_peered":"2024-09-10T18:22:44.198837+0000","last_clean":"2024-09-10T18:22:44.198837+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:22:44.198837+0000","last_undegraded":"2024-09-10T18:22:44.198837+0000","last_fullsized":"2024-09-10T18:22:44.198837+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:47.847418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198648+0000","last_change":"2024-09-10T18:21:31.532155+0000","last_active":"2024-09-10T18:22:44.198648+0000","last_peered":"2024-09-10T18:22:44.198648+0000","last_clean":"2024-09-10T18:22:44.198648+0000","last_became_active":"2024-09-10T18:21:31.531895+0000","last_became_peered":"2024-09-10T18:21:31.531895+0000","last_unstale":"2024-09-10T18:22:44.198648+0000","last_undegraded":"2024-09-10T18:22:44.198648+0000","last_fullsized":"2024-09-10T18:22:44.198648+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:49.301934+0000","last_change":"2024-09-10T18:22:49.301887+0000","last_active":"2024-09-10T18:22:49.301934+0000","last_peered":"2024-09-10T18:22:49.301934+0000","last_clean":"2024-09-10T18:22:49.301934+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:22:49.301934+0000","last_undegraded":"2024-09-10T18:22:49.301934+0000","last_fullsized":"2024-09-10T18:22:49.301934+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:49.301823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:49.301823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:06.288915+0000","scrub_duration":0.001243433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198276+0000","last_change":"2024-09-10T18:21:37.637599+0000","last_active":"2024-09-10T18:22:44.198276+0000","last_peered":"2024-09-10T18:22:44.198276+0000","last_clean":"2024-09-10T18:22:44.198276+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:22:44.198276+0000","last_undegraded":"2024-09-10T18:22:44.198276+0000","last_fullsized":"2024-09-10T18:22:44.198276+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:02.280321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110579+0000","last_change":"2024-09-10T18:21:34.047045+0000","last_active":"2024-09-10T18:22:44.110579+0000","last_peered":"2024-09-10T18:22:44.110579+0000","last_clean":"2024-09-10T18:22:44.110579+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:22:44.110579+0000","last_undegraded":"2024-09-10T18:22:44.110579+0000","last_fullsized":"2024-09-10T18:22:44.110579+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:49.769358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198144+0000","last_change":"2024-09-10T18:22:38.445504+0000","last_active":"2024-09-10T18:22:44.198144+0000","last_peered":"2024-09-10T18:22:44.198144+0000","last_clean":"2024-09-10T18:22:44.198144+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:22:44.198144+0000","last_undegraded":"2024-09-10T18:22:44.198144+0000","last_fullsized":"2024-09-10T18:22:44.198144+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:38.445448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:38.445448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:05.044685+0000","scrub_duration":0.0018359400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":67,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:48.317806+0000","last_change":"2024-09-10T18:22:48.317756+0000","last_active":"2024-09-10T18:22:48.317806+0000","last_peered":"2024-09-10T18:22:48.317806+0000","last_clean":"2024-09-10T18:22:48.317806+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:22:48.317806+0000","last_undegraded":"2024-09-10T18:22:48.317806+0000","last_fullsized":"2024-09-10T18:22:48.317806+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:22:48.317687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:48.317687+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:01.400638+0000","scrub_duration":0.0022162499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991842+0000","last_change":"2024-09-10T18:21:37.640015+0000","last_active":"2024-09-10T18:22:43.991842+0000","last_peered":"2024-09-10T18:22:43.991842+0000","last_clean":"2024-09-10T18:22:43.991842+0000","last_became_active":"2024-09-10T18:21:37.639706+0000","last_became_peered":"2024-09-10T18:21:37.639706+0000","last_unstale":"2024-09-10T18:22:43.991842+0000","last_undegraded":"2024-09-10T18:22:43.991842+0000","last_fullsized":"2024-09-10T18:22:43.991842+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989634+0000","last_change":"2024-09-10T18:22:33.342047+0000","last_active":"2024-09-10T18:22:43.989634+0000","last_peered":"2024-09-10T18:22:43.989634+0000","last_clean":"2024-09-10T18:22:43.989634+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:22:43.989634+0000","last_undegraded":"2024-09-10T18:22:43.989634+0000","last_fullsized":"2024-09-10T18:22:43.989634+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-10T18:22:33.341980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:33.341980+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:47.079316+0000","scrub_duration":0.0021839350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198817+0000","last_change":"2024-09-10T18:21:31.532402+0000","last_active":"2024-09-10T18:22:44.198817+0000","last_peered":"2024-09-10T18:22:44.198817+0000","last_clean":"2024-09-10T18:22:44.198817+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:22:44.198817+0000","last_undegraded":"2024-09-10T18:22:44.198817+0000","last_fullsized":"2024-09-10T18:22:44.198817+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":66,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:50.618322+0000","last_change":"2024-09-10T18:22:50.618285+0000","last_active":"2024-09-10T18:22:50.618322+0000","last_peered":"2024-09-10T18:22:50.618322+0000","last_clean":"2024-09-10T18:22:50.618322+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:22:50.618322+0000","last_undegraded":"2024-09-10T18:22:50.618322+0000","last_fullsized":"2024-09-10T18:22:50.618322+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:22:50.618215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:50.618215+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:08.321441+0000","scrub_duration":0.0017651660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198472+0000","last_change":"2024-09-10T18:21:37.644078+0000","last_active":"2024-09-10T18:22:44.198472+0000","last_peered":"2024-09-10T18:22:44.198472+0000","last_clean":"2024-09-10T18:22:44.198472+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:22:44.198472+0000","last_undegraded":"2024-09-10T18:22:44.198472+0000","last_fullsized":"2024-09-10T18:22:44.198472+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.991187+0000","last_change":"2024-09-10T18:21:33.611605+0000","last_active":"2024-09-10T18:22:43.991187+0000","last_peered":"2024-09-10T18:22:43.991187+0000","last_clean":"2024-09-10T18:22:43.991187+0000","last_became_active":"2024-09-10T18:21:33.611286+0000","last_became_peered":"2024-09-10T18:21:33.611286+0000","last_unstale":"2024-09-10T18:22:43.991187+0000","last_undegraded":"2024-09-10T18:22:43.991187+0000","last_fullsized":"2024-09-10T18:22:43.991187+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:53.952740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:53.433111+0000","last_change":"2024-09-10T18:22:53.433078+0000","last_active":"2024-09-10T18:22:53.433111+0000","last_peered":"2024-09-10T18:22:53.433111+0000","last_clean":"2024-09-10T18:22:53.433111+0000","last_became_active":"2024-09-10T18:21:31.530158+0000","last_became_peered":"2024-09-10T18:21:31.530158+0000","last_unstale":"2024-09-10T18:22:53.433111+0000","last_undegraded":"2024-09-10T18:22:53.433111+0000","last_fullsized":"2024-09-10T18:22:53.433111+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:53.433012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:53.433012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:19.414323+0000","scrub_duration":0.0013676739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.107405+0000","last_change":"2024-09-10T18:22:36.049767+0000","last_active":"2024-09-10T18:22:44.107405+0000","last_peered":"2024-09-10T18:22:44.107405+0000","last_clean":"2024-09-10T18:22:44.107405+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:22:44.107405+0000","last_undegraded":"2024-09-10T18:22:44.107405+0000","last_fullsized":"2024-09-10T18:22:44.107405+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:36.049705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:36.049705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:39.206398+0000","scrub_duration":0.0030879660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.988587+0000","last_change":"2024-09-10T18:21:37.642856+0000","last_active":"2024-09-10T18:22:43.988587+0000","last_peered":"2024-09-10T18:22:43.988587+0000","last_clean":"2024-09-10T18:22:43.988587+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:22:43.988587+0000","last_undegraded":"2024-09-10T18:22:43.988587+0000","last_fullsized":"2024-09-10T18:22:43.988587+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:04.616953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.107805+0000","last_change":"2024-09-10T18:22:42.015349+0000","last_active":"2024-09-10T18:22:44.107805+0000","last_peered":"2024-09-10T18:22:44.107805+0000","last_clean":"2024-09-10T18:22:44.107805+0000","last_became_active":"2024-09-10T18:21:33.612401+0000","last_became_peered":"2024-09-10T18:21:33.612401+0000","last_unstale":"2024-09-10T18:22:44.107805+0000","last_undegraded":"2024-09-10T18:22:44.107805+0000","last_fullsized":"2024-09-10T18:22:44.107805+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:22:42.015292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:42.015292+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:07.052542+0000","scrub_duration":0.001986785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198978+0000","last_change":"2024-09-10T18:22:43.250096+0000","last_active":"2024-09-10T18:22:44.198978+0000","last_peered":"2024-09-10T18:22:44.198978+0000","last_clean":"2024-09-10T18:22:44.198978+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:22:44.198978+0000","last_undegraded":"2024-09-10T18:22:44.198978+0000","last_fullsized":"2024-09-10T18:22:44.198978+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:43.250030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:43.250030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:12.038178+0000","scrub_duration":0.001581381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.989164+0000","last_change":"2024-09-10T18:21:35.623087+0000","last_active":"2024-09-10T18:22:43.989164+0000","last_peered":"2024-09-10T18:22:43.989164+0000","last_clean":"2024-09-10T18:22:43.989164+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:22:43.989164+0000","last_undegraded":"2024-09-10T18:22:43.989164+0000","last_fullsized":"2024-09-10T18:22:43.989164+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:03.160666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:52.262661+0000","last_change":"2024-09-10T18:22:52.262629+0000","last_active":"2024-09-10T18:22:52.262661+0000","last_peered":"2024-09-10T18:22:52.262661+0000","last_clean":"2024-09-10T18:22:52.262661+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:22:52.262661+0000","last_undegraded":"2024-09-10T18:22:52.262661+0000","last_fullsized":"2024-09-10T18:22:52.262661+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:52.262563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:52.262563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:20.159979+0000","scrub_duration":0.001452381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":54,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.988726+0000","last_change":"2024-09-10T18:21:34.046343+0000","last_active":"2024-09-10T18:22:43.988726+0000","last_peered":"2024-09-10T18:22:43.988726+0000","last_clean":"2024-09-10T18:22:43.988726+0000","last_became_active":"2024-09-10T18:21:34.046024+0000","last_became_peered":"2024-09-10T18:21:34.046024+0000","last_unstale":"2024-09-10T18:22:43.988726+0000","last_undegraded":"2024-09-10T18:22:43.988726+0000","last_fullsized":"2024-09-10T18:22:43.988726+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:50.603367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.110567+0000","last_change":"2024-09-10T18:22:40.686003+0000","last_active":"2024-09-10T18:22:44.110567+0000","last_peered":"2024-09-10T18:22:44.110567+0000","last_clean":"2024-09-10T18:22:44.110567+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:22:44.110567+0000","last_undegraded":"2024-09-10T18:22:44.110567+0000","last_fullsized":"2024-09-10T18:22:44.110567+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:22:40.685879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:01.722544+0000","scrub_duration":0.001767794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198179+0000","last_change":"2024-09-10T18:21:35.626825+0000","last_active":"2024-09-10T18:22:44.198179+0000","last_peered":"2024-09-10T18:22:44.198179+0000","last_clean":"2024-09-10T18:22:44.198179+0000","last_became_active":"2024-09-10T18:21:35.626537+0000","last_became_peered":"2024-09-10T18:21:35.626537+0000","last_unstale":"2024-09-10T18:22:44.198179+0000","last_undegraded":"2024-09-10T18:22:44.198179+0000","last_fullsized":"2024-09-10T18:22:44.198179+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:56.248415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.343231+0000","last_change":"2024-09-10T18:22:46.343197+0000","last_active":"2024-09-10T18:22:46.343231+0000","last_peered":"2024-09-10T18:22:46.343231+0000","last_clean":"2024-09-10T18:22:46.343231+0000","last_became_active":"2024-09-10T18:21:38.564314+0000","last_became_peered":"2024-09-10T18:21:38.564314+0000","last_unstale":"2024-09-10T18:22:46.343231+0000","last_undegraded":"2024-09-10T18:22:46.343231+0000","last_fullsized":"2024-09-10T18:22:46.343231+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:46.343137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:46.343137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:10.438171+0000","scrub_duration":0.0016021189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.988577+0000","last_change":"2024-09-10T18:21:33.618322+0000","last_active":"2024-09-10T18:22:43.988577+0000","last_peered":"2024-09-10T18:22:43.988577+0000","last_clean":"2024-09-10T18:22:43.988577+0000","last_became_active":"2024-09-10T18:21:33.617670+0000","last_became_peered":"2024-09-10T18:21:33.617670+0000","last_unstale":"2024-09-10T18:22:43.988577+0000","last_undegraded":"2024-09-10T18:22:43.988577+0000","last_fullsized":"2024-09-10T18:22:43.988577+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:58.493392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198306+0000","last_change":"2024-09-10T18:21:31.530146+0000","last_active":"2024-09-10T18:22:44.198306+0000","last_peered":"2024-09-10T18:22:44.198306+0000","last_clean":"2024-09-10T18:22:44.198306+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:22:44.198306+0000","last_undegraded":"2024-09-10T18:22:44.198306+0000","last_fullsized":"2024-09-10T18:22:44.198306+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:52.583420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:49.757559+0000","last_change":"2024-09-10T18:22:49.757505+0000","last_active":"2024-09-10T18:22:49.757559+0000","last_peered":"2024-09-10T18:22:49.757559+0000","last_clean":"2024-09-10T18:22:49.757559+0000","last_became_active":"2024-09-10T18:21:35.620419+0000","last_became_peered":"2024-09-10T18:21:35.620419+0000","last_unstale":"2024-09-10T18:22:49.757559+0000","last_undegraded":"2024-09-10T18:22:49.757559+0000","last_fullsized":"2024-09-10T18:22:49.757559+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:49.757435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:49.757435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:00.814048+0000","scrub_duration":0.0014410040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198343+0000","last_change":"2024-09-10T18:21:37.643084+0000","last_active":"2024-09-10T18:22:44.198343+0000","last_peered":"2024-09-10T18:22:44.198343+0000","last_clean":"2024-09-10T18:22:44.198343+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:22:44.198343+0000","last_undegraded":"2024-09-10T18:22:44.198343+0000","last_fullsized":"2024-09-10T18:22:44.198343+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:05.246215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.108529+0000","last_change":"2024-09-10T18:22:44.108462+0000","last_active":"2024-09-10T18:22:44.108529+0000","last_peered":"2024-09-10T18:22:44.108529+0000","last_clean":"2024-09-10T18:22:44.108529+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:22:44.108529+0000","last_undegraded":"2024-09-10T18:22:44.108529+0000","last_fullsized":"2024-09-10T18:22:44.108529+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:44.108373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:44.108373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:51.902386+0000","scrub_duration":0.0024985559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:43.988765+0000","last_change":"2024-09-10T18:21:31.535545+0000","last_active":"2024-09-10T18:22:43.988765+0000","last_peered":"2024-09-10T18:22:43.988765+0000","last_clean":"2024-09-10T18:22:43.988765+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:22:43.988765+0000","last_undegraded":"2024-09-10T18:22:43.988765+0000","last_fullsized":"2024-09-10T18:22:43.988765+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:56.890432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.173162+0000","last_change":"2024-09-10T18:21:33.613827+0000","last_active":"2024-09-10T18:22:44.173162+0000","last_peered":"2024-09-10T18:22:44.173162+0000","last_clean":"2024-09-10T18:22:44.173162+0000","last_became_active":"2024-09-10T18:21:33.613487+0000","last_became_peered":"2024-09-10T18:21:33.613487+0000","last_unstale":"2024-09-10T18:22:44.173162+0000","last_undegraded":"2024-09-10T18:22:44.173162+0000","last_fullsized":"2024-09-10T18:22:44.173162+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:50.340408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-10T18:22:44.198608+0000","last_change":"2024-09-10T18:21:35.620819+0000","last_active":"2024-09-10T18:22:44.198608+0000","last_peered":"2024-09-10T18:22:44.198608+0000","last_clean":"2024-09-10T18:22:44.198608+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:22:44.198608+0000","last_undegraded":"2024-09-10T18:22:44.198608+0000","last_fullsized":"2024-09-10T18:22:44.198608+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":405,"ondisk_log_size":405,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298812,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446340,"statfs":{"total":95995035648,"available":95689052160,"internally_reserved":0,"allocated":1495040,"data_stored":962545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.434},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.48299999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.503}]},{"osd":2,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":2.9569999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":2.9100000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":2.9289999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.40999999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.39500000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.41899999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.46800000000000003}]}]},{"osd":6,"up_from":46,"seq":197568495642,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298796,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446356,"statfs":{"total":95995035648,"available":95689068544,"internally_reserved":0,"allocated":1478656,"data_stored":962717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.47199999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.49399999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.44400000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.45300000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.42999999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.35599999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.38200000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724575,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298216,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446936,"statfs":{"total":95995035648,"available":95689662464,"internally_reserved":0,"allocated":884736,"data_stored":372010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.36399999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.42499999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.63}]},{"osd":1,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.56499999999999995,"15min":0.56100000000000005},"min":{"1min":0.42399999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.377,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.55100000000000005}]},{"osd":2,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55100000000000005,"15min":0.55200000000000005},"min":{"1min":0.38800000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.38,"5min":0.255,"15min":0.255},"max":{"1min":0.75600000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.51500000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.54800000000000004,"15min":0.54300000000000004},"min":{"1min":0.434,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.52900000000000003,"15min":0.53600000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.42799999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.36499999999999999,"15min":0.35899999999999999},"min":{"1min":0.29499999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.38400000000000001},"min":{"1min":0.28599999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.46200000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.503},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.34100000000000003}]},{"osd":7,"last update":"Tue Sep 10 18:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.372},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.378}]}]},{"osd":4,"up_from":33,"seq":141733920803,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":901120,"data_stored":371988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56599999999999995,"15min":0.57499999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.72699999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.61799999999999999,"15min":0.59699999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.8170000000000002,"5min":4.8170000000000002,"15min":4.8170000000000002},"last":0.45900000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.61499999999999999,"15min":0.59699999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":4.7729999999999997,"5min":4.7729999999999997,"15min":4.7729999999999997},"last":0.375},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.504}]},{"osd":2,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.60699999999999998,"15min":0.57899999999999996},"min":{"1min":0.42799999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.8040000000000003,"5min":4.8040000000000003,"15min":4.8040000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.57599999999999996,"15min":0.58599999999999997},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71699999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.34999999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.66600000000000004,"5min":0.751,"15min":0.751},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59099999999999997,"15min":0.59599999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.747,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.47099999999999997}]},{"osd":5,"last update":"Tue Sep 10 18:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.39000000000000001,"15min":0.38300000000000001},"min":{"1min":0.32800000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42499999999999999,"15min":0.42399999999999999},"min":{"1min":0.28299999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60099999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.308}]},{"osd":6,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.36299999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.40699999999999997}]}]},{"osd":3,"up_from":27,"seq":115964117033,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298244,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446908,"statfs":{"total":95995035648,"available":95689633792,"internally_reserved":0,"allocated":913408,"data_stored":372832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.39800000000000002,"15min":0.378},"min":{"1min":0.23000000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.40999999999999998,"15min":0.39600000000000002},"min":{"1min":0.255,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.46200000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.437,"15min":0.435},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.41599999999999998,"15min":0.40100000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.442}]},{"osd":2,"last update":"Tue Sep 10 18:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.45100000000000001,"15min":0.44},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.46999999999999997,"15min":0.41599999999999998},"min":{"1min":0.27700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":7.2750000000000004,"5min":7.2750000000000004,"15min":7.2750000000000004},"last":0.43099999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:22:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.55600000000000005,"15min":0.54000000000000004},"min":{"1min":0.378,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.56499999999999995,"15min":0.55500000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.502}]},{"osd":5,"last update":"Tue Sep 10 18:22:34 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":0.73899999999999999,"15min":0.63400000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":14.885,"5min":14.885,"15min":14.885},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.61899999999999999,"15min":0.60699999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.53400000000000003}]},{"osd":6,"last update":"Tue Sep 10 18:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.48299999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.55700000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298208,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446944,"statfs":{"total":95995035648,"available":95689670656,"internally_reserved":0,"allocated":876544,"data_stored":371240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.36199999999999999,"15min":0.33400000000000002},"min":{"1min":0.22,"5min":0.219,"15min":0.219},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.371,"15min":0.34899999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.224}]},{"osd":1,"last update":"Tue Sep 10 18:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.35599999999999998,"15min":0.33800000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.63100000000000001,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.39100000000000001,"15min":0.35499999999999998},"min":{"1min":0.28199999999999997,"5min":0.22,"15min":0.22},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.29499999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46400000000000002,"15min":0.46300000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.313},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.436,"15min":0.433},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.25}]},{"osd":4,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.56799999999999995,"15min":0.56499999999999995},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.76300000000000001,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.76000000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.61699999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.61499999999999999,"15min":0.59399999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":2.3279999999999998,"5min":2.3279999999999998,"15min":2.3279999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.63,"15min":0.61599999999999999},"min":{"1min":0.44500000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.63900000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.56200000000000006}]},{"osd":7,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.627}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298268,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446884,"statfs":{"total":95995035648,"available":95689609216,"internally_reserved":0,"allocated":937984,"data_stored":373168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39000000000000001,"15min":0.373},"min":{"1min":0.27700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.47799999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.379},"min":{"1min":0.25700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.38200000000000001,"15min":0.36399999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.64800000000000002,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.41499999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.26800000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.59399999999999997,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.45100000000000001,"15min":0.46400000000000002},"min":{"1min":0.30499999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.63300000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.45500000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.85899999999999999,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.94799999999999995,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.59799999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.55100000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.64300000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:21:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.61299999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.626}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298808,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446344,"statfs":{"total":95995035648,"available":95689056256,"internally_reserved":0,"allocated":1490944,"data_stored":962229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:22:01 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36199999999999999,"15min":0.34999999999999998},"min":{"1min":0.23999999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.57999999999999996,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.378},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.372,"15min":0.35199999999999998},"min":{"1min":0.25600000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.58599999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.28899999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40999999999999998,"15min":0.40699999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53500000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.40600000000000003,"15min":0.40200000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.47899999999999998,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.53400000000000003}]},{"osd":3,"last update":"Tue Sep 10 18:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.44400000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.55500000000000005,"5min":0.629,"15min":0.629},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.434,"15min":0.44500000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.54100000000000004,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.50800000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:22:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.624,"15min":0.63100000000000001},"min":{"1min":0.433,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.83299999999999996,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.65000000000000002,"15min":0.66100000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81899999999999995,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.52200000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.59399999999999997,"15min":0.60299999999999998},"min":{"1min":0.437,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.72099999999999997,"5min":0.879,"15min":0.879},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.625,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.42999999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.498}]},{"osd":7,"last update":"Tue Sep 10 18:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.48099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:22:57.530 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-10T18:22:57.530 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-10T18:22:57.530 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-10T18:22:57.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph health --format=json 2024-09-10T18:22:57.795 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:22:58.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:58 smithi028 ceph-mon[39220]: 3.11 deep-scrub starts 2024-09-10T18:22:58.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:58 smithi028 ceph-mon[39220]: 3.11 deep-scrub ok 2024-09-10T18:22:58.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:58 smithi028 ceph-mon[39220]: 3.7 scrub starts 2024-09-10T18:22:58.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:58 smithi028 ceph-mon[39220]: 3.7 scrub ok 2024-09-10T18:22:58.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:58 smithi028 ceph-mon[39220]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-09-10T18:22:58.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:58 smithi028 ceph-mon[39220]: from='client.34619 v1:172.21.15.28:0/343266528' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:22:58.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:58 smithi028 ceph-mon[39220]: osdmap e87: 8 total, 8 up, 8 in 2024-09-10T18:22:58.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:58 smithi028 ceph-mon[29291]: 3.11 deep-scrub starts 2024-09-10T18:22:58.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:58 smithi028 ceph-mon[29291]: 3.11 deep-scrub ok 2024-09-10T18:22:58.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:58 smithi028 ceph-mon[29291]: 3.7 scrub starts 2024-09-10T18:22:58.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:58 smithi028 ceph-mon[29291]: 3.7 scrub ok 2024-09-10T18:22:58.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:58 smithi028 ceph-mon[29291]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-09-10T18:22:58.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:58 smithi028 ceph-mon[29291]: from='client.34619 v1:172.21.15.28:0/343266528' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:22:58.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:58 smithi028 ceph-mon[29291]: osdmap e87: 8 total, 8 up, 8 in 2024-09-10T18:22:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:58 smithi204 ceph-mon[32357]: 3.11 deep-scrub starts 2024-09-10T18:22:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:58 smithi204 ceph-mon[32357]: 3.11 deep-scrub ok 2024-09-10T18:22:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:58 smithi204 ceph-mon[32357]: 3.7 scrub starts 2024-09-10T18:22:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:58 smithi204 ceph-mon[32357]: 3.7 scrub ok 2024-09-10T18:22:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:58 smithi204 ceph-mon[32357]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-09-10T18:22:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:58 smithi204 ceph-mon[32357]: from='client.34619 v1:172.21.15.28:0/343266528' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:22:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:58 smithi204 ceph-mon[32357]: osdmap e87: 8 total, 8 up, 8 in 2024-09-10T18:22:59.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:59 smithi028 ceph-mon[29291]: 4.2 scrub starts 2024-09-10T18:22:59.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:59 smithi028 ceph-mon[29291]: 4.2 scrub ok 2024-09-10T18:22:59.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:59 smithi028 ceph-mon[29291]: 6.1a scrub starts 2024-09-10T18:22:59.099 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:59 smithi028 ceph-mon[29291]: 6.1a scrub ok 2024-09-10T18:22:59.099 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:22:59 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:59.099 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:59 smithi028 ceph-mon[39220]: 4.2 scrub starts 2024-09-10T18:22:59.099 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:59 smithi028 ceph-mon[39220]: 4.2 scrub ok 2024-09-10T18:22:59.099 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:59 smithi028 ceph-mon[39220]: 6.1a scrub starts 2024-09-10T18:22:59.099 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:59 smithi028 ceph-mon[39220]: 6.1a scrub ok 2024-09-10T18:22:59.099 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:22:59 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:59 smithi204 ceph-mon[32357]: 4.2 scrub starts 2024-09-10T18:22:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:59 smithi204 ceph-mon[32357]: 4.2 scrub ok 2024-09-10T18:22:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:59 smithi204 ceph-mon[32357]: 6.1a scrub starts 2024-09-10T18:22:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:59 smithi204 ceph-mon[32357]: 6.1a scrub ok 2024-09-10T18:22:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:22:59 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:22:59.483 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:22:59.483 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-10T18:23:00.130 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-10T18:23:00.130 INFO:tasks.cephadm:Setup complete, yielding 2024-09-10T18:23:00.130 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-10T18:23:00.142 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-10T18:23:00.142 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-guits-quincy-2024-09-09-1810-quincy'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'quincy-orch-centos-9-teuth', 'sha1': 'fad5356a94e6044ce1062d748daa5a0f2ed53a7d'}} 2024-09-10T18:23:00.142 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-10T18:23:00.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-10T18:23:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:00 smithi204 ceph-mon[32357]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:23:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:00 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/984384650' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-10T18:23:00.401 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:00.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:00 smithi028 ceph-mon[29291]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:23:00.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:00 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/984384650' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-10T18:23:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:00 smithi028 ceph-mon[39220]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:23:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:00 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/984384650' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-10T18:23:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:01 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:23:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:01 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:23:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:01 smithi204 ceph-mon[32357]: osdmap e88: 8 total, 8 up, 8 in 2024-09-10T18:23:01.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:01 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:23:01.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:01 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:23:01.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:01 smithi028 ceph-mon[29291]: osdmap e88: 8 total, 8 up, 8 in 2024-09-10T18:23:01.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:01 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:23:01.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:01 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:23:01.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:01 smithi028 ceph-mon[39220]: osdmap e88: 8 total, 8 up, 8 in 2024-09-10T18:23:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-mon[29291]: 3.13 scrub starts 2024-09-10T18:23:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-mon[29291]: 3.13 scrub ok 2024-09-10T18:23:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-mon[29291]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:23:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-mon[29291]: 4.1f scrub starts 2024-09-10T18:23:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-mon[29291]: 4.1f scrub ok 2024-09-10T18:23:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-mon[29291]: 6.8 scrub starts 2024-09-10T18:23:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-mon[29291]: 6.8 scrub ok 2024-09-10T18:23:02.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-mon[39220]: 3.13 scrub starts 2024-09-10T18:23:02.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-mon[39220]: 3.13 scrub ok 2024-09-10T18:23:02.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-mon[39220]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:23:02.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-mon[39220]: 4.1f scrub starts 2024-09-10T18:23:02.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-mon[39220]: 4.1f scrub ok 2024-09-10T18:23:02.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-mon[39220]: 6.8 scrub starts 2024-09-10T18:23:02.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-mon[39220]: 6.8 scrub ok 2024-09-10T18:23:02.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:23:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:23:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:23:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:02 smithi204 ceph-mon[32357]: 3.13 scrub starts 2024-09-10T18:23:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:02 smithi204 ceph-mon[32357]: 3.13 scrub ok 2024-09-10T18:23:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:02 smithi204 ceph-mon[32357]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:23:02.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:02 smithi204 ceph-mon[32357]: 4.1f scrub starts 2024-09-10T18:23:02.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:02 smithi204 ceph-mon[32357]: 4.1f scrub ok 2024-09-10T18:23:02.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:02 smithi204 ceph-mon[32357]: 6.8 scrub starts 2024-09-10T18:23:02.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:02 smithi204 ceph-mon[32357]: 6.8 scrub ok 2024-09-10T18:23:02.561 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:02] ENGINE Bus STOPPING 2024-09-10T18:23:02.951 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:02] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T18:23:02.951 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:02] ENGINE Bus STOPPED 2024-09-10T18:23:02.952 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:02] ENGINE Bus STARTING 2024-09-10T18:23:02.952 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:02] ENGINE Serving on http://:::9283 2024-09-10T18:23:02.952 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:02 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:02] ENGINE Bus STARTED 2024-09-10T18:23:02.977 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-10T18:23:02.978 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-10T18:23:02.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:23:03.234 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:03.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:03 smithi028 ceph-mon[39220]: 3.8 scrub starts 2024-09-10T18:23:03.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:03 smithi028 ceph-mon[39220]: 3.8 scrub ok 2024-09-10T18:23:03.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:03 smithi028 ceph-mon[39220]: 6.3 deep-scrub starts 2024-09-10T18:23:03.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:03 smithi028 ceph-mon[39220]: 6.3 deep-scrub ok 2024-09-10T18:23:03.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:03 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2613315252' entity='client.admin' 2024-09-10T18:23:03.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:03 smithi028 ceph-mon[29291]: 3.8 scrub starts 2024-09-10T18:23:03.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:03 smithi028 ceph-mon[29291]: 3.8 scrub ok 2024-09-10T18:23:03.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:03 smithi028 ceph-mon[29291]: 6.3 deep-scrub starts 2024-09-10T18:23:03.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:03 smithi028 ceph-mon[29291]: 6.3 deep-scrub ok 2024-09-10T18:23:03.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:03 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2613315252' entity='client.admin' 2024-09-10T18:23:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:03 smithi204 ceph-mon[32357]: 3.8 scrub starts 2024-09-10T18:23:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:03 smithi204 ceph-mon[32357]: 3.8 scrub ok 2024-09-10T18:23:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:03 smithi204 ceph-mon[32357]: 6.3 deep-scrub starts 2024-09-10T18:23:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:03 smithi204 ceph-mon[32357]: 6.3 deep-scrub ok 2024-09-10T18:23:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:03 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2613315252' entity='client.admin' 2024-09-10T18:23:04.169 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:23:03 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:23:04.169 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:04 smithi204 ceph-mon[32357]: 4.1d deep-scrub starts 2024-09-10T18:23:04.169 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:04 smithi204 ceph-mon[32357]: 4.1d deep-scrub ok 2024-09-10T18:23:04.169 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:04 smithi204 ceph-mon[32357]: 4.1c scrub starts 2024-09-10T18:23:04.169 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:04 smithi204 ceph-mon[32357]: 4.1c scrub ok 2024-09-10T18:23:04.169 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:04 smithi204 ceph-mon[32357]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:23:04.170 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:04 smithi204 ceph-mon[32357]: 4.4 scrub starts 2024-09-10T18:23:04.170 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:04 smithi204 ceph-mon[32357]: 4.4 scrub ok 2024-09-10T18:23:04.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:04 smithi028 ceph-mon[29291]: 4.1d deep-scrub starts 2024-09-10T18:23:04.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:04 smithi028 ceph-mon[29291]: 4.1d deep-scrub ok 2024-09-10T18:23:04.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:04 smithi028 ceph-mon[29291]: 4.1c scrub starts 2024-09-10T18:23:04.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:04 smithi028 ceph-mon[29291]: 4.1c scrub ok 2024-09-10T18:23:04.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:04 smithi028 ceph-mon[29291]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:23:04.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:04 smithi028 ceph-mon[29291]: 4.4 scrub starts 2024-09-10T18:23:04.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:04 smithi028 ceph-mon[29291]: 4.4 scrub ok 2024-09-10T18:23:04.295 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:04 smithi028 ceph-mon[39220]: 4.1d deep-scrub starts 2024-09-10T18:23:04.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:04 smithi028 ceph-mon[39220]: 4.1d deep-scrub ok 2024-09-10T18:23:04.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:04 smithi028 ceph-mon[39220]: 4.1c scrub starts 2024-09-10T18:23:04.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:04 smithi028 ceph-mon[39220]: 4.1c scrub ok 2024-09-10T18:23:04.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:04 smithi028 ceph-mon[39220]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:23:04.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:04 smithi028 ceph-mon[39220]: 4.4 scrub starts 2024-09-10T18:23:04.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:04 smithi028 ceph-mon[39220]: 4.4 scrub ok 2024-09-10T18:23:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:23:04.981 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:23:05.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:05 smithi028 ceph-mon[29291]: 6.17 scrub starts 2024-09-10T18:23:05.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:05 smithi028 ceph-mon[29291]: 6.17 scrub ok 2024-09-10T18:23:05.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:05 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:05.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:05 smithi028 ceph-mon[29291]: 3.c scrub starts 2024-09-10T18:23:05.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:05 smithi028 ceph-mon[29291]: 3.c scrub ok 2024-09-10T18:23:05.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:05 smithi028 ceph-mon[39220]: 6.17 scrub starts 2024-09-10T18:23:05.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:05 smithi028 ceph-mon[39220]: 6.17 scrub ok 2024-09-10T18:23:05.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:05 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:05.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:05 smithi028 ceph-mon[39220]: 3.c scrub starts 2024-09-10T18:23:05.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:05 smithi028 ceph-mon[39220]: 3.c scrub ok 2024-09-10T18:23:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:05 smithi204 ceph-mon[32357]: 6.17 scrub starts 2024-09-10T18:23:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:05 smithi204 ceph-mon[32357]: 6.17 scrub ok 2024-09-10T18:23:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:05 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:05 smithi204 ceph-mon[32357]: 3.c scrub starts 2024-09-10T18:23:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:05 smithi204 ceph-mon[32357]: 3.c scrub ok 2024-09-10T18:23:05.560 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-09-10T18:23:04.702605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":934,"num_read_kb":791,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":573,"ondisk_log_size":573,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2386956,"kb_used_data":6832,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747574260,"statfs":{"total":767960285184,"available":765516042240,"internally_reserved":0,"allocated":6995968,"data_stored":4164069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001453"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143504+0000","last_change":"2024-09-10T18:21:31.526401+0000","last_active":"2024-09-10T18:22:57.143504+0000","last_peered":"2024-09-10T18:22:57.143504+0000","last_clean":"2024-09-10T18:22:57.143504+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:22:57.143504+0000","last_undegraded":"2024-09-10T18:22:57.143504+0000","last_fullsized":"2024-09-10T18:22:57.143504+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029127+0000","last_change":"2024-09-10T18:21:33.614228+0000","last_active":"2024-09-10T18:22:57.029127+0000","last_peered":"2024-09-10T18:22:57.029127+0000","last_clean":"2024-09-10T18:22:57.029127+0000","last_became_active":"2024-09-10T18:21:33.613935+0000","last_became_peered":"2024-09-10T18:21:33.613935+0000","last_unstale":"2024-09-10T18:22:57.029127+0000","last_undegraded":"2024-09-10T18:22:57.029127+0000","last_fullsized":"2024-09-10T18:22:57.029127+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:00.978849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.140654+0000","last_change":"2024-09-10T18:21:35.619936+0000","last_active":"2024-09-10T18:22:46.140654+0000","last_peered":"2024-09-10T18:22:46.140654+0000","last_clean":"2024-09-10T18:22:46.140654+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:22:46.140654+0000","last_undegraded":"2024-09-10T18:22:46.140654+0000","last_fullsized":"2024-09-10T18:22:46.140654+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.532528+0000","last_change":"2024-09-10T18:22:57.532472+0000","last_active":"2024-09-10T18:22:57.532528+0000","last_peered":"2024-09-10T18:22:57.532528+0000","last_clean":"2024-09-10T18:22:57.532528+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:22:57.532528+0000","last_undegraded":"2024-09-10T18:22:57.532528+0000","last_fullsized":"2024-09-10T18:22:57.532528+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:57.532385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:57.532385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:25.022831+0000","scrub_duration":0.0015832260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.142012+0000","last_change":"2024-09-10T18:21:38.565483+0000","last_active":"2024-09-10T18:22:46.142012+0000","last_peered":"2024-09-10T18:22:46.142012+0000","last_clean":"2024-09-10T18:22:46.142012+0000","last_became_active":"2024-09-10T18:21:38.565118+0000","last_became_peered":"2024-09-10T18:21:38.565118+0000","last_unstale":"2024-09-10T18:22:46.142012+0000","last_undegraded":"2024-09-10T18:22:46.142012+0000","last_fullsized":"2024-09-10T18:22:46.142012+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.142856+0000","last_change":"2024-09-10T18:21:31.528038+0000","last_active":"2024-09-10T18:22:46.142856+0000","last_peered":"2024-09-10T18:22:46.142856+0000","last_clean":"2024-09-10T18:22:46.142856+0000","last_became_active":"2024-09-10T18:21:31.527582+0000","last_became_peered":"2024-09-10T18:21:31.527582+0000","last_unstale":"2024-09-10T18:22:46.142856+0000","last_undegraded":"2024-09-10T18:22:46.142856+0000","last_fullsized":"2024-09-10T18:22:46.142856+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":61,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.141730+0000","last_change":"2024-09-10T18:21:33.613829+0000","last_active":"2024-09-10T18:22:46.141730+0000","last_peered":"2024-09-10T18:22:46.141730+0000","last_clean":"2024-09-10T18:22:46.141730+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:22:46.141730+0000","last_undegraded":"2024-09-10T18:22:46.141730+0000","last_fullsized":"2024-09-10T18:22:46.141730+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":44,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-10T18:23:04.633739+0000","last_change":"2024-09-10T18:23:04.633711+0000","last_active":"2024-09-10T18:23:04.633739+0000","last_peered":"2024-09-10T18:23:04.633739+0000","last_clean":"2024-09-10T18:23:04.633739+0000","last_became_active":"2024-09-10T18:21:35.620585+0000","last_became_peered":"2024-09-10T18:21:35.620585+0000","last_unstale":"2024-09-10T18:23:04.633739+0000","last_undegraded":"2024-09-10T18:23:04.633739+0000","last_fullsized":"2024-09-10T18:23:04.633739+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:23:04.633647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:09.146682+0000","scrub_duration":0.0012299979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099491+0000","last_change":"2024-09-10T18:21:31.524898+0000","last_active":"2024-09-10T18:22:57.099491+0000","last_peered":"2024-09-10T18:22:57.099491+0000","last_clean":"2024-09-10T18:22:57.099491+0000","last_became_active":"2024-09-10T18:21:31.524620+0000","last_became_peered":"2024-09-10T18:21:31.524620+0000","last_unstale":"2024-09-10T18:22:57.099491+0000","last_undegraded":"2024-09-10T18:22:57.099491+0000","last_fullsized":"2024-09-10T18:22:57.099491+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029552+0000","last_change":"2024-09-10T18:21:33.610754+0000","last_active":"2024-09-10T18:22:57.029552+0000","last_peered":"2024-09-10T18:22:57.029552+0000","last_clean":"2024-09-10T18:22:57.029552+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:22:57.029552+0000","last_undegraded":"2024-09-10T18:22:57.029552+0000","last_fullsized":"2024-09-10T18:22:57.029552+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099176+0000","last_change":"2024-09-10T18:22:53.577905+0000","last_active":"2024-09-10T18:22:57.099176+0000","last_peered":"2024-09-10T18:22:57.099176+0000","last_clean":"2024-09-10T18:22:57.099176+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:22:57.099176+0000","last_undegraded":"2024-09-10T18:22:57.099176+0000","last_fullsized":"2024-09-10T18:22:57.099176+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:53.577826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:53.577826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:19.539271+0000","scrub_duration":0.001440577,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143080+0000","last_change":"2024-09-10T18:22:46.328322+0000","last_active":"2024-09-10T18:22:57.143080+0000","last_peered":"2024-09-10T18:22:57.143080+0000","last_clean":"2024-09-10T18:22:57.143080+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:22:57.143080+0000","last_undegraded":"2024-09-10T18:22:57.143080+0000","last_fullsized":"2024-09-10T18:22:57.143080+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:46.328262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:46.328262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:50.476851+0000","scrub_duration":0.0024608049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099496+0000","last_change":"2024-09-10T18:22:56.588562+0000","last_active":"2024-09-10T18:22:57.099496+0000","last_peered":"2024-09-10T18:22:57.099496+0000","last_clean":"2024-09-10T18:22:57.099496+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:22:57.099496+0000","last_undegraded":"2024-09-10T18:22:57.099496+0000","last_fullsized":"2024-09-10T18:22:57.099496+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:56.588510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:56.588510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:07.012424+0000","scrub_duration":0.0011336320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"85'7","reported_seq":58,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029493+0000","last_change":"2024-09-10T18:22:44.365273+0000","last_active":"2024-09-10T18:22:57.029493+0000","last_peered":"2024-09-10T18:22:57.029493+0000","last_clean":"2024-09-10T18:22:57.029493+0000","last_became_active":"2024-09-10T18:21:33.615429+0000","last_became_peered":"2024-09-10T18:21:33.615429+0000","last_unstale":"2024-09-10T18:22:57.029493+0000","last_undegraded":"2024-09-10T18:22:57.029493+0000","last_fullsized":"2024-09-10T18:22:57.029493+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'7","last_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:22:44.365196+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:58.822595+0000","scrub_duration":0.0033059909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.176275+0000","last_change":"2024-09-10T18:21:35.624352+0000","last_active":"2024-09-10T18:22:46.176275+0000","last_peered":"2024-09-10T18:22:46.176275+0000","last_clean":"2024-09-10T18:22:46.176275+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:22:46.176275+0000","last_undegraded":"2024-09-10T18:22:46.176275+0000","last_fullsized":"2024-09-10T18:22:46.176275+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099034+0000","last_change":"2024-09-10T18:21:37.639437+0000","last_active":"2024-09-10T18:22:57.099034+0000","last_peered":"2024-09-10T18:22:57.099034+0000","last_clean":"2024-09-10T18:22:57.099034+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:22:57.099034+0000","last_undegraded":"2024-09-10T18:22:57.099034+0000","last_fullsized":"2024-09-10T18:22:57.099034+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029965+0000","last_change":"2024-09-10T18:21:38.563964+0000","last_active":"2024-09-10T18:22:57.029965+0000","last_peered":"2024-09-10T18:22:57.029965+0000","last_clean":"2024-09-10T18:22:57.029965+0000","last_became_active":"2024-09-10T18:21:38.563570+0000","last_became_peered":"2024-09-10T18:21:38.563570+0000","last_unstale":"2024-09-10T18:22:57.029965+0000","last_undegraded":"2024-09-10T18:22:57.029965+0000","last_fullsized":"2024-09-10T18:22:57.029965+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143518+0000","last_change":"2024-09-10T18:21:31.529843+0000","last_active":"2024-09-10T18:22:57.143518+0000","last_peered":"2024-09-10T18:22:57.143518+0000","last_clean":"2024-09-10T18:22:57.143518+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:22:57.143518+0000","last_undegraded":"2024-09-10T18:22:57.143518+0000","last_fullsized":"2024-09-10T18:22:57.143518+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":60,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:45.529962+0000","last_change":"2024-09-10T18:21:33.617848+0000","last_active":"2024-09-10T18:22:45.529962+0000","last_peered":"2024-09-10T18:22:45.529962+0000","last_clean":"2024-09-10T18:22:45.529962+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:22:45.529962+0000","last_undegraded":"2024-09-10T18:22:45.529962+0000","last_fullsized":"2024-09-10T18:22:45.529962+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:58.180270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.141643+0000","last_change":"2024-09-10T18:22:39.007672+0000","last_active":"2024-09-10T18:22:46.141643+0000","last_peered":"2024-09-10T18:22:46.141643+0000","last_clean":"2024-09-10T18:22:46.141643+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:22:46.141643+0000","last_undegraded":"2024-09-10T18:22:46.141643+0000","last_fullsized":"2024-09-10T18:22:46.141643+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:39.007614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:39.007614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:59.162407+0000","scrub_duration":0.0021069719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.142523+0000","last_change":"2024-09-10T18:21:38.565017+0000","last_active":"2024-09-10T18:22:46.142523+0000","last_peered":"2024-09-10T18:22:46.142523+0000","last_clean":"2024-09-10T18:22:46.142523+0000","last_became_active":"2024-09-10T18:21:38.564681+0000","last_became_peered":"2024-09-10T18:21:38.564681+0000","last_unstale":"2024-09-10T18:22:46.142523+0000","last_undegraded":"2024-09-10T18:22:46.142523+0000","last_fullsized":"2024-09-10T18:22:46.142523+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029667+0000","last_change":"2024-09-10T18:21:31.528392+0000","last_active":"2024-09-10T18:22:57.029667+0000","last_peered":"2024-09-10T18:22:57.029667+0000","last_clean":"2024-09-10T18:22:57.029667+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:22:57.029667+0000","last_undegraded":"2024-09-10T18:22:57.029667+0000","last_fullsized":"2024-09-10T18:22:57.029667+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":59,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.142192+0000","last_change":"2024-09-10T18:21:33.617612+0000","last_active":"2024-09-10T18:22:46.142192+0000","last_peered":"2024-09-10T18:22:46.142192+0000","last_clean":"2024-09-10T18:22:46.142192+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:22:46.142192+0000","last_undegraded":"2024-09-10T18:22:46.142192+0000","last_fullsized":"2024-09-10T18:22:46.142192+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029475+0000","last_change":"2024-09-10T18:22:50.436357+0000","last_active":"2024-09-10T18:22:57.029475+0000","last_peered":"2024-09-10T18:22:57.029475+0000","last_clean":"2024-09-10T18:22:57.029475+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:22:57.029475+0000","last_undegraded":"2024-09-10T18:22:57.029475+0000","last_fullsized":"2024-09-10T18:22:57.029475+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:50.436294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:50.436294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:07.783088+0000","scrub_duration":0.001715187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.176411+0000","last_change":"2024-09-10T18:21:37.646675+0000","last_active":"2024-09-10T18:22:46.176411+0000","last_peered":"2024-09-10T18:22:46.176411+0000","last_clean":"2024-09-10T18:22:46.176411+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:22:46.176411+0000","last_undegraded":"2024-09-10T18:22:46.176411+0000","last_fullsized":"2024-09-10T18:22:46.176411+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.141728+0000","last_change":"2024-09-10T18:22:46.141545+0000","last_active":"2024-09-10T18:22:46.141728+0000","last_peered":"2024-09-10T18:22:46.141728+0000","last_clean":"2024-09-10T18:22:46.141728+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:22:46.141728+0000","last_undegraded":"2024-09-10T18:22:46.141728+0000","last_fullsized":"2024-09-10T18:22:46.141728+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:46.141408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:46.141408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:12.563610+0000","scrub_duration":0.0022977269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"86'15","reported_seq":66,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143429+0000","last_change":"2024-09-10T18:22:49.298511+0000","last_active":"2024-09-10T18:22:57.143429+0000","last_peered":"2024-09-10T18:22:57.143429+0000","last_clean":"2024-09-10T18:22:57.143429+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:22:57.143429+0000","last_undegraded":"2024-09-10T18:22:57.143429+0000","last_fullsized":"2024-09-10T18:22:57.143429+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:22:49.298455+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:10.207495+0000","scrub_duration":0.0038705900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:59.769656+0000","last_change":"2024-09-10T18:22:48.795234+0000","last_active":"2024-09-10T18:22:59.769656+0000","last_peered":"2024-09-10T18:22:59.769656+0000","last_clean":"2024-09-10T18:22:59.769656+0000","last_became_active":"2024-09-10T18:21:35.624964+0000","last_became_peered":"2024-09-10T18:21:35.624964+0000","last_unstale":"2024-09-10T18:22:59.769656+0000","last_undegraded":"2024-09-10T18:22:59.769656+0000","last_fullsized":"2024-09-10T18:22:59.769656+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:22:48.795158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:48.795158+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:11.053077+0000","scrub_duration":0.0017552570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"83'22","reported_seq":76,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.142002+0000","last_change":"2024-09-10T18:22:40.007784+0000","last_active":"2024-09-10T18:22:46.142002+0000","last_peered":"2024-09-10T18:22:46.142002+0000","last_clean":"2024-09-10T18:22:46.142002+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:22:46.142002+0000","last_undegraded":"2024-09-10T18:22:46.142002+0000","last_fullsized":"2024-09-10T18:22:46.142002+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:22:40.007685+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:46.637635+0000","scrub_duration":0.0077173049999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.143450+0000","last_change":"2024-09-10T18:21:31.533221+0000","last_active":"2024-09-10T18:22:46.143450+0000","last_peered":"2024-09-10T18:22:46.143450+0000","last_clean":"2024-09-10T18:22:46.143450+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:22:46.143450+0000","last_undegraded":"2024-09-10T18:22:46.143450+0000","last_fullsized":"2024-09-10T18:22:46.143450+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:59.769676+0000","last_change":"2024-09-10T18:21:35.624518+0000","last_active":"2024-09-10T18:22:59.769676+0000","last_peered":"2024-09-10T18:22:59.769676+0000","last_clean":"2024-09-10T18:22:59.769676+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:22:59.769676+0000","last_undegraded":"2024-09-10T18:22:59.769676+0000","last_fullsized":"2024-09-10T18:22:59.769676+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099151+0000","last_change":"2024-09-10T18:21:37.644088+0000","last_active":"2024-09-10T18:22:57.099151+0000","last_peered":"2024-09-10T18:22:57.099151+0000","last_clean":"2024-09-10T18:22:57.099151+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:22:57.099151+0000","last_undegraded":"2024-09-10T18:22:57.099151+0000","last_fullsized":"2024-09-10T18:22:57.099151+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":68,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.143028+0000","last_change":"2024-09-10T18:21:34.047843+0000","last_active":"2024-09-10T18:22:46.143028+0000","last_peered":"2024-09-10T18:22:46.143028+0000","last_clean":"2024-09-10T18:22:46.143028+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:22:46.143028+0000","last_undegraded":"2024-09-10T18:22:46.143028+0000","last_fullsized":"2024-09-10T18:22:46.143028+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":48,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:56.357009+0000","last_change":"2024-09-10T18:22:56.356952+0000","last_active":"2024-09-10T18:22:56.357009+0000","last_peered":"2024-09-10T18:22:56.357009+0000","last_clean":"2024-09-10T18:22:56.357009+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:22:56.357009+0000","last_undegraded":"2024-09-10T18:22:56.357009+0000","last_fullsized":"2024-09-10T18:22:56.357009+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:56.356885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:56.356885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:15.154888+0000","scrub_duration":0.0016221829999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029215+0000","last_change":"2024-09-10T18:22:47.386371+0000","last_active":"2024-09-10T18:22:57.029215+0000","last_peered":"2024-09-10T18:22:57.029215+0000","last_clean":"2024-09-10T18:22:57.029215+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:22:57.029215+0000","last_undegraded":"2024-09-10T18:22:57.029215+0000","last_fullsized":"2024-09-10T18:22:57.029215+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:47.386303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:47.386303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:01.335164+0000","scrub_duration":0.001725484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029479+0000","last_change":"2024-09-10T18:21:37.642918+0000","last_active":"2024-09-10T18:22:57.029479+0000","last_peered":"2024-09-10T18:22:57.029479+0000","last_clean":"2024-09-10T18:22:57.029479+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:22:57.029479+0000","last_undegraded":"2024-09-10T18:22:57.029479+0000","last_fullsized":"2024-09-10T18:22:57.029479+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029339+0000","last_change":"2024-09-10T18:22:37.322150+0000","last_active":"2024-09-10T18:22:57.029339+0000","last_peered":"2024-09-10T18:22:57.029339+0000","last_clean":"2024-09-10T18:22:57.029339+0000","last_became_active":"2024-09-10T18:21:33.607574+0000","last_became_peered":"2024-09-10T18:21:33.607574+0000","last_unstale":"2024-09-10T18:22:57.029339+0000","last_undegraded":"2024-09-10T18:22:57.029339+0000","last_fullsized":"2024-09-10T18:22:57.029339+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-10T18:22:37.322094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:37.322094+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:41.781070+0000","scrub_duration":0.001716817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143607+0000","last_change":"2024-09-10T18:21:31.531508+0000","last_active":"2024-09-10T18:22:57.143607+0000","last_peered":"2024-09-10T18:22:57.143607+0000","last_clean":"2024-09-10T18:22:57.143607+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:22:57.143607+0000","last_undegraded":"2024-09-10T18:22:57.143607+0000","last_fullsized":"2024-09-10T18:22:57.143607+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":59,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:54.768311+0000","last_change":"2024-09-10T18:21:35.623304+0000","last_active":"2024-09-10T18:22:54.768311+0000","last_peered":"2024-09-10T18:22:54.768311+0000","last_clean":"2024-09-10T18:22:54.768311+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:22:54.768311+0000","last_undegraded":"2024-09-10T18:22:54.768311+0000","last_fullsized":"2024-09-10T18:22:54.768311+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.176058+0000","last_change":"2024-09-10T18:21:38.566158+0000","last_active":"2024-09-10T18:22:46.176058+0000","last_peered":"2024-09-10T18:22:46.176058+0000","last_clean":"2024-09-10T18:22:46.176058+0000","last_became_active":"2024-09-10T18:21:38.565165+0000","last_became_peered":"2024-09-10T18:21:38.565165+0000","last_unstale":"2024-09-10T18:22:46.176058+0000","last_undegraded":"2024-09-10T18:22:46.176058+0000","last_fullsized":"2024-09-10T18:22:46.176058+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.098139+0000","last_change":"2024-09-10T18:22:57.098114+0000","last_active":"2024-09-10T18:22:57.098139+0000","last_peered":"2024-09-10T18:22:57.098139+0000","last_clean":"2024-09-10T18:22:57.098139+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:22:57.098139+0000","last_undegraded":"2024-09-10T18:22:57.098139+0000","last_fullsized":"2024-09-10T18:22:57.098139+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-10T18:22:57.098059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:57.098059+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:07.299393+0000","scrub_duration":0.0025842040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099580+0000","last_change":"2024-09-10T18:21:31.534170+0000","last_active":"2024-09-10T18:22:57.099580+0000","last_peered":"2024-09-10T18:22:57.099580+0000","last_clean":"2024-09-10T18:22:57.099580+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:22:57.099580+0000","last_undegraded":"2024-09-10T18:22:57.099580+0000","last_fullsized":"2024-09-10T18:22:57.099580+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:59.769529+0000","last_change":"2024-09-10T18:21:35.618920+0000","last_active":"2024-09-10T18:22:59.769529+0000","last_peered":"2024-09-10T18:22:59.769529+0000","last_clean":"2024-09-10T18:22:59.769529+0000","last_became_active":"2024-09-10T18:21:35.617821+0000","last_became_peered":"2024-09-10T18:21:35.617821+0000","last_unstale":"2024-09-10T18:22:59.769529+0000","last_undegraded":"2024-09-10T18:22:59.769529+0000","last_fullsized":"2024-09-10T18:22:59.769529+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143182+0000","last_change":"2024-09-10T18:22:41.399132+0000","last_active":"2024-09-10T18:22:57.143182+0000","last_peered":"2024-09-10T18:22:57.143182+0000","last_clean":"2024-09-10T18:22:57.143182+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:22:57.143182+0000","last_undegraded":"2024-09-10T18:22:57.143182+0000","last_fullsized":"2024-09-10T18:22:57.143182+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:41.399078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:41.399078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:48.673347+0000","scrub_duration":0.001245323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143300+0000","last_change":"2024-09-10T18:21:34.046468+0000","last_active":"2024-09-10T18:22:57.143300+0000","last_peered":"2024-09-10T18:22:57.143300+0000","last_clean":"2024-09-10T18:22:57.143300+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:22:57.143300+0000","last_undegraded":"2024-09-10T18:22:57.143300+0000","last_fullsized":"2024-09-10T18:22:57.143300+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.141646+0000","last_change":"2024-09-10T18:21:31.530156+0000","last_active":"2024-09-10T18:22:46.141646+0000","last_peered":"2024-09-10T18:22:46.141646+0000","last_clean":"2024-09-10T18:22:46.141646+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:22:46.141646+0000","last_undegraded":"2024-09-10T18:22:46.141646+0000","last_fullsized":"2024-09-10T18:22:46.141646+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:59.013351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143474+0000","last_change":"2024-09-10T18:22:38.853925+0000","last_active":"2024-09-10T18:22:57.143474+0000","last_peered":"2024-09-10T18:22:57.143474+0000","last_clean":"2024-09-10T18:22:57.143474+0000","last_became_active":"2024-09-10T18:21:35.617132+0000","last_became_peered":"2024-09-10T18:21:35.617132+0000","last_unstale":"2024-09-10T18:22:57.143474+0000","last_undegraded":"2024-09-10T18:22:57.143474+0000","last_fullsized":"2024-09-10T18:22:57.143474+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:38.853839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:38.853839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:39.087399+0000","scrub_duration":0.0031500500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.140596+0000","last_change":"2024-09-10T18:21:38.564042+0000","last_active":"2024-09-10T18:22:46.140596+0000","last_peered":"2024-09-10T18:22:46.140596+0000","last_clean":"2024-09-10T18:22:46.140596+0000","last_became_active":"2024-09-10T18:21:38.563629+0000","last_became_peered":"2024-09-10T18:21:38.563629+0000","last_unstale":"2024-09-10T18:22:46.140596+0000","last_undegraded":"2024-09-10T18:22:46.140596+0000","last_fullsized":"2024-09-10T18:22:46.140596+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:04.135213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":99,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:54.768341+0000","last_change":"2024-09-10T18:21:33.618915+0000","last_active":"2024-09-10T18:22:54.768341+0000","last_peered":"2024-09-10T18:22:54.768341+0000","last_clean":"2024-09-10T18:22:54.768341+0000","last_became_active":"2024-09-10T18:21:33.618616+0000","last_became_peered":"2024-09-10T18:21:33.618616+0000","last_unstale":"2024-09-10T18:22:54.768341+0000","last_undegraded":"2024-09-10T18:22:54.768341+0000","last_fullsized":"2024-09-10T18:22:54.768341+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029848+0000","last_change":"2024-09-10T18:22:42.335215+0000","last_active":"2024-09-10T18:22:57.029848+0000","last_peered":"2024-09-10T18:22:57.029848+0000","last_clean":"2024-09-10T18:22:57.029848+0000","last_became_active":"2024-09-10T18:21:31.526161+0000","last_became_peered":"2024-09-10T18:21:31.526161+0000","last_unstale":"2024-09-10T18:22:57.029848+0000","last_undegraded":"2024-09-10T18:22:57.029848+0000","last_fullsized":"2024-09-10T18:22:57.029848+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:42.335152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:42.335152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:02.796505+0000","scrub_duration":0.0013614720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"62'2","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099444+0000","last_change":"2024-09-10T18:21:35.613488+0000","last_active":"2024-09-10T18:22:57.099444+0000","last_peered":"2024-09-10T18:22:57.099444+0000","last_clean":"2024-09-10T18:22:57.099444+0000","last_became_active":"2024-09-10T18:21:31.528970+0000","last_became_peered":"2024-09-10T18:21:31.528970+0000","last_unstale":"2024-09-10T18:22:57.099444+0000","last_undegraded":"2024-09-10T18:22:57.099444+0000","last_fullsized":"2024-09-10T18:22:57.099444+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143376+0000","last_change":"2024-09-10T18:22:43.309177+0000","last_active":"2024-09-10T18:22:57.143376+0000","last_peered":"2024-09-10T18:22:57.143376+0000","last_clean":"2024-09-10T18:22:57.143376+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:22:57.143376+0000","last_undegraded":"2024-09-10T18:22:57.143376+0000","last_fullsized":"2024-09-10T18:22:57.143376+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:43.309115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:43.309115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:08.721305+0000","scrub_duration":0.0015388929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:55.388648+0000","last_change":"2024-09-10T18:22:55.388615+0000","last_active":"2024-09-10T18:22:55.388648+0000","last_peered":"2024-09-10T18:22:55.388648+0000","last_clean":"2024-09-10T18:22:55.388648+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:22:55.388648+0000","last_undegraded":"2024-09-10T18:22:55.388648+0000","last_fullsized":"2024-09-10T18:22:55.388648+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:55.388564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:55.388564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:13.516819+0000","scrub_duration":0.001337956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":61,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.142045+0000","last_change":"2024-09-10T18:21:33.618163+0000","last_active":"2024-09-10T18:22:46.142045+0000","last_peered":"2024-09-10T18:22:46.142045+0000","last_clean":"2024-09-10T18:22:46.142045+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:22:46.142045+0000","last_undegraded":"2024-09-10T18:22:46.142045+0000","last_fullsized":"2024-09-10T18:22:46.142045+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:52.160895+0000","last_change":"2024-09-10T18:22:52.160807+0000","last_active":"2024-09-10T18:22:52.160895+0000","last_peered":"2024-09-10T18:22:52.160895+0000","last_clean":"2024-09-10T18:22:52.160895+0000","last_became_active":"2024-09-10T18:21:31.532624+0000","last_became_peered":"2024-09-10T18:21:31.532624+0000","last_unstale":"2024-09-10T18:22:52.160895+0000","last_undegraded":"2024-09-10T18:22:52.160895+0000","last_fullsized":"2024-09-10T18:22:52.160895+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:52.160732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:52.160732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:11.270210+0000","scrub_duration":0.0015264289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":53,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.337329+0000","last_change":"2024-09-10T18:22:46.337257+0000","last_active":"2024-09-10T18:22:46.337329+0000","last_peered":"2024-09-10T18:22:46.337329+0000","last_clean":"2024-09-10T18:22:46.337329+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:22:46.337329+0000","last_undegraded":"2024-09-10T18:22:46.337329+0000","last_fullsized":"2024-09-10T18:22:46.337329+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:22:46.337186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:46.337186+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:11.152611+0000","scrub_duration":0.0015240410000000001,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:53.488530+0000","last_change":"2024-09-10T18:22:53.488490+0000","last_active":"2024-09-10T18:22:53.488530+0000","last_peered":"2024-09-10T18:22:53.488530+0000","last_clean":"2024-09-10T18:22:53.488530+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:22:53.488530+0000","last_undegraded":"2024-09-10T18:22:53.488530+0000","last_fullsized":"2024-09-10T18:22:53.488530+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:22:53.488434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:15.445322+0000","scrub_duration":0.001619882,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.176468+0000","last_change":"2024-09-10T18:21:38.565470+0000","last_active":"2024-09-10T18:22:46.176468+0000","last_peered":"2024-09-10T18:22:46.176468+0000","last_clean":"2024-09-10T18:22:46.176468+0000","last_became_active":"2024-09-10T18:21:38.565183+0000","last_became_peered":"2024-09-10T18:21:38.565183+0000","last_unstale":"2024-09-10T18:22:46.176468+0000","last_undegraded":"2024-09-10T18:22:46.176468+0000","last_fullsized":"2024-09-10T18:22:46.176468+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:04.594058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143506+0000","last_change":"2024-09-10T18:22:51.280276+0000","last_active":"2024-09-10T18:22:57.143506+0000","last_peered":"2024-09-10T18:22:57.143506+0000","last_clean":"2024-09-10T18:22:57.143506+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:22:57.143506+0000","last_undegraded":"2024-09-10T18:22:57.143506+0000","last_fullsized":"2024-09-10T18:22:57.143506+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:22:51.280219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:51.280219+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:13.028302+0000","scrub_duration":0.0021524019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143200+0000","last_change":"2024-09-10T18:22:55.325500+0000","last_active":"2024-09-10T18:22:57.143200+0000","last_peered":"2024-09-10T18:22:57.143200+0000","last_clean":"2024-09-10T18:22:57.143200+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:22:57.143200+0000","last_undegraded":"2024-09-10T18:22:57.143200+0000","last_fullsized":"2024-09-10T18:22:57.143200+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:55.325446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:55.325446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:08.340184+0000","scrub_duration":0.0014498149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":136,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.925246+0000","last_change":"2024-09-10T18:22:41.256321+0000","last_active":"2024-09-10T18:22:57.925246+0000","last_peered":"2024-09-10T18:22:57.925246+0000","last_clean":"2024-09-10T18:22:57.925246+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:22:57.925246+0000","last_undegraded":"2024-09-10T18:22:57.925246+0000","last_fullsized":"2024-09-10T18:22:57.925246+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:22:41.256255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:41.256255+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:01.661114+0000","scrub_duration":0.0015771470000000001,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":81,"num_read_kb":76,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099771+0000","last_change":"2024-09-10T18:21:35.619757+0000","last_active":"2024-09-10T18:22:57.099771+0000","last_peered":"2024-09-10T18:22:57.099771+0000","last_clean":"2024-09-10T18:22:57.099771+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:22:57.099771+0000","last_undegraded":"2024-09-10T18:22:57.099771+0000","last_fullsized":"2024-09-10T18:22:57.099771+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.141097+0000","last_change":"2024-09-10T18:21:37.639920+0000","last_active":"2024-09-10T18:22:46.141097+0000","last_peered":"2024-09-10T18:22:46.141097+0000","last_clean":"2024-09-10T18:22:46.141097+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:22:46.141097+0000","last_undegraded":"2024-09-10T18:22:46.141097+0000","last_fullsized":"2024-09-10T18:22:46.141097+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:04.764182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":58,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.144019+0000","last_change":"2024-09-10T18:21:34.046018+0000","last_active":"2024-09-10T18:22:57.144019+0000","last_peered":"2024-09-10T18:22:57.144019+0000","last_clean":"2024-09-10T18:22:57.144019+0000","last_became_active":"2024-09-10T18:21:34.045549+0000","last_became_peered":"2024-09-10T18:21:34.045549+0000","last_unstale":"2024-09-10T18:22:57.144019+0000","last_undegraded":"2024-09-10T18:22:57.144019+0000","last_fullsized":"2024-09-10T18:22:57.144019+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":48,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.142358+0000","last_change":"2024-09-10T18:22:45.383833+0000","last_active":"2024-09-10T18:22:46.142358+0000","last_peered":"2024-09-10T18:22:46.142358+0000","last_clean":"2024-09-10T18:22:46.142358+0000","last_became_active":"2024-09-10T18:21:31.529816+0000","last_became_peered":"2024-09-10T18:21:31.529816+0000","last_unstale":"2024-09-10T18:22:46.142358+0000","last_undegraded":"2024-09-10T18:22:46.142358+0000","last_fullsized":"2024-09-10T18:22:46.142358+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:45.383768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:45.383768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:06.766629+0000","scrub_duration":0.001512844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":81,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.176319+0000","last_change":"2024-09-10T18:22:17.024622+0000","last_active":"2024-09-10T18:22:46.176319+0000","last_peered":"2024-09-10T18:22:46.176319+0000","last_clean":"2024-09-10T18:22:46.176319+0000","last_became_active":"2024-09-10T18:21:20.570983+0000","last_became_peered":"2024-09-10T18:21:20.570983+0000","last_unstale":"2024-09-10T18:22:46.176319+0000","last_undegraded":"2024-09-10T18:22:46.176319+0000","last_fullsized":"2024-09-10T18:22:46.176319+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:22:17.024553+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:21.328129+0000","scrub_duration":0.003148245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.176521+0000","last_change":"2024-09-10T18:21:35.621370+0000","last_active":"2024-09-10T18:22:46.176521+0000","last_peered":"2024-09-10T18:22:46.176521+0000","last_clean":"2024-09-10T18:22:46.176521+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:22:46.176521+0000","last_undegraded":"2024-09-10T18:22:46.176521+0000","last_fullsized":"2024-09-10T18:22:46.176521+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099717+0000","last_change":"2024-09-10T18:21:37.638814+0000","last_active":"2024-09-10T18:22:57.099717+0000","last_peered":"2024-09-10T18:22:57.099717+0000","last_clean":"2024-09-10T18:22:57.099717+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:22:57.099717+0000","last_undegraded":"2024-09-10T18:22:57.099717+0000","last_fullsized":"2024-09-10T18:22:57.099717+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:06.014250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.030046+0000","last_change":"2024-09-10T18:21:33.614556+0000","last_active":"2024-09-10T18:22:57.030046+0000","last_peered":"2024-09-10T18:22:57.030046+0000","last_clean":"2024-09-10T18:22:57.030046+0000","last_became_active":"2024-09-10T18:21:33.614090+0000","last_became_peered":"2024-09-10T18:21:33.614090+0000","last_unstale":"2024-09-10T18:22:57.030046+0000","last_undegraded":"2024-09-10T18:22:57.030046+0000","last_fullsized":"2024-09-10T18:22:57.030046+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029508+0000","last_change":"2024-09-10T18:21:31.529705+0000","last_active":"2024-09-10T18:22:57.029508+0000","last_peered":"2024-09-10T18:22:57.029508+0000","last_clean":"2024-09-10T18:22:57.029508+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:22:57.029508+0000","last_undegraded":"2024-09-10T18:22:57.029508+0000","last_fullsized":"2024-09-10T18:22:57.029508+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099644+0000","last_change":"2024-09-10T18:21:35.620645+0000","last_active":"2024-09-10T18:22:57.099644+0000","last_peered":"2024-09-10T18:22:57.099644+0000","last_clean":"2024-09-10T18:22:57.099644+0000","last_became_active":"2024-09-10T18:21:35.620331+0000","last_became_peered":"2024-09-10T18:21:35.620331+0000","last_unstale":"2024-09-10T18:22:57.099644+0000","last_undegraded":"2024-09-10T18:22:57.099644+0000","last_fullsized":"2024-09-10T18:22:57.099644+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:50.298439+0000","last_change":"2024-09-10T18:22:50.298390+0000","last_active":"2024-09-10T18:22:50.298439+0000","last_peered":"2024-09-10T18:22:50.298439+0000","last_clean":"2024-09-10T18:22:50.298439+0000","last_became_active":"2024-09-10T18:21:38.563450+0000","last_became_peered":"2024-09-10T18:21:38.563450+0000","last_unstale":"2024-09-10T18:22:50.298439+0000","last_undegraded":"2024-09-10T18:22:50.298439+0000","last_fullsized":"2024-09-10T18:22:50.298439+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:50.298320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:50.298320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:19.836783+0000","scrub_duration":0.0021610420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029981+0000","last_change":"2024-09-10T18:21:33.611478+0000","last_active":"2024-09-10T18:22:57.029981+0000","last_peered":"2024-09-10T18:22:57.029981+0000","last_clean":"2024-09-10T18:22:57.029981+0000","last_became_active":"2024-09-10T18:21:33.611203+0000","last_became_peered":"2024-09-10T18:21:33.611203+0000","last_unstale":"2024-09-10T18:22:57.029981+0000","last_undegraded":"2024-09-10T18:22:57.029981+0000","last_fullsized":"2024-09-10T18:22:57.029981+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"58'1","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143989+0000","last_change":"2024-09-10T18:21:31.528999+0000","last_active":"2024-09-10T18:22:57.143989+0000","last_peered":"2024-09-10T18:22:57.143989+0000","last_clean":"2024-09-10T18:22:57.143989+0000","last_became_active":"2024-09-10T18:21:31.528748+0000","last_became_peered":"2024-09-10T18:21:31.528748+0000","last_unstale":"2024-09-10T18:22:57.143989+0000","last_undegraded":"2024-09-10T18:22:57.143989+0000","last_fullsized":"2024-09-10T18:22:57.143989+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.141212+0000","last_change":"2024-09-10T18:21:35.625340+0000","last_active":"2024-09-10T18:22:46.141212+0000","last_peered":"2024-09-10T18:22:46.141212+0000","last_clean":"2024-09-10T18:22:46.141212+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:22:46.141212+0000","last_undegraded":"2024-09-10T18:22:46.141212+0000","last_fullsized":"2024-09-10T18:22:46.141212+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:52.486850+0000","last_change":"2024-09-10T18:22:52.486807+0000","last_active":"2024-09-10T18:22:52.486850+0000","last_peered":"2024-09-10T18:22:52.486850+0000","last_clean":"2024-09-10T18:22:52.486850+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:22:52.486850+0000","last_undegraded":"2024-09-10T18:22:52.486850+0000","last_fullsized":"2024-09-10T18:22:52.486850+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:52.486737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:52.486737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:59.120450+0000","scrub_duration":0.0015615,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029682+0000","last_change":"2024-09-10T18:21:33.614985+0000","last_active":"2024-09-10T18:22:57.029682+0000","last_peered":"2024-09-10T18:22:57.029682+0000","last_clean":"2024-09-10T18:22:57.029682+0000","last_became_active":"2024-09-10T18:21:33.614689+0000","last_became_peered":"2024-09-10T18:21:33.614689+0000","last_unstale":"2024-09-10T18:22:57.029682+0000","last_undegraded":"2024-09-10T18:22:57.029682+0000","last_fullsized":"2024-09-10T18:22:57.029682+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.142594+0000","last_change":"2024-09-10T18:21:31.535458+0000","last_active":"2024-09-10T18:22:46.142594+0000","last_peered":"2024-09-10T18:22:46.142594+0000","last_clean":"2024-09-10T18:22:46.142594+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:22:46.142594+0000","last_undegraded":"2024-09-10T18:22:46.142594+0000","last_fullsized":"2024-09-10T18:22:46.142594+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":68,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:54.768322+0000","last_change":"2024-09-10T18:22:51.492683+0000","last_active":"2024-09-10T18:22:54.768322+0000","last_peered":"2024-09-10T18:22:54.768322+0000","last_clean":"2024-09-10T18:22:54.768322+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:22:54.768322+0000","last_undegraded":"2024-09-10T18:22:54.768322+0000","last_fullsized":"2024-09-10T18:22:54.768322+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:22:51.492580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:51.492580+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:00.192242+0000","scrub_duration":0.002035987,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029768+0000","last_change":"2024-09-10T18:21:37.642233+0000","last_active":"2024-09-10T18:22:57.029768+0000","last_peered":"2024-09-10T18:22:57.029768+0000","last_clean":"2024-09-10T18:22:57.029768+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:22:57.029768+0000","last_undegraded":"2024-09-10T18:22:57.029768+0000","last_fullsized":"2024-09-10T18:22:57.029768+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143202+0000","last_change":"2024-09-10T18:21:33.614532+0000","last_active":"2024-09-10T18:22:57.143202+0000","last_peered":"2024-09-10T18:22:57.143202+0000","last_clean":"2024-09-10T18:22:57.143202+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:22:57.143202+0000","last_undegraded":"2024-09-10T18:22:57.143202+0000","last_fullsized":"2024-09-10T18:22:57.143202+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"58'1","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-10T18:23:04.541328+0000","last_change":"2024-09-10T18:23:04.541297+0000","last_active":"2024-09-10T18:23:04.541328+0000","last_peered":"2024-09-10T18:23:04.541328+0000","last_clean":"2024-09-10T18:23:04.541328+0000","last_became_active":"2024-09-10T18:21:31.534441+0000","last_became_peered":"2024-09-10T18:21:31.534441+0000","last_unstale":"2024-09-10T18:23:04.541328+0000","last_undegraded":"2024-09-10T18:23:04.541328+0000","last_fullsized":"2024-09-10T18:23:04.541328+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-10T18:23:04.541238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:23:04.541238+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:18.491111+0000","scrub_duration":0.0017464519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:45.529609+0000","last_change":"2024-09-10T18:22:41.279395+0000","last_active":"2024-09-10T18:22:45.529609+0000","last_peered":"2024-09-10T18:22:45.529609+0000","last_clean":"2024-09-10T18:22:45.529609+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:22:45.529609+0000","last_undegraded":"2024-09-10T18:22:45.529609+0000","last_fullsized":"2024-09-10T18:22:45.529609+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:41.279342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:41.279342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:48.532838+0000","scrub_duration":0.0017787790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143228+0000","last_change":"2024-09-10T18:21:37.641253+0000","last_active":"2024-09-10T18:22:57.143228+0000","last_peered":"2024-09-10T18:22:57.143228+0000","last_clean":"2024-09-10T18:22:57.143228+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:22:57.143228+0000","last_undegraded":"2024-09-10T18:22:57.143228+0000","last_fullsized":"2024-09-10T18:22:57.143228+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"86'28","reported_seq":86,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143556+0000","last_change":"2024-09-10T18:22:54.863374+0000","last_active":"2024-09-10T18:22:57.143556+0000","last_peered":"2024-09-10T18:22:57.143556+0000","last_clean":"2024-09-10T18:22:57.143556+0000","last_became_active":"2024-09-10T18:21:34.046453+0000","last_became_peered":"2024-09-10T18:21:34.046453+0000","last_unstale":"2024-09-10T18:22:57.143556+0000","last_undegraded":"2024-09-10T18:22:57.143556+0000","last_fullsized":"2024-09-10T18:22:57.143556+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'28","last_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:22:54.863323+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:20.254372+0000","scrub_duration":0.0053712059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":47,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.176695+0000","last_change":"2024-09-10T18:22:32.137670+0000","last_active":"2024-09-10T18:22:46.176695+0000","last_peered":"2024-09-10T18:22:46.176695+0000","last_clean":"2024-09-10T18:22:46.176695+0000","last_became_active":"2024-09-10T18:21:31.531951+0000","last_became_peered":"2024-09-10T18:21:31.531951+0000","last_unstale":"2024-09-10T18:22:46.176695+0000","last_undegraded":"2024-09-10T18:22:46.176695+0000","last_fullsized":"2024-09-10T18:22:46.176695+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:32.137600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:32.137600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:41.301280+0000","scrub_duration":0.00151761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:45.529595+0000","last_change":"2024-09-10T18:22:42.312458+0000","last_active":"2024-09-10T18:22:45.529595+0000","last_peered":"2024-09-10T18:22:45.529595+0000","last_clean":"2024-09-10T18:22:45.529595+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:22:45.529595+0000","last_undegraded":"2024-09-10T18:22:45.529595+0000","last_fullsized":"2024-09-10T18:22:45.529595+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:42.312403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:42.312403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:42.394403+0000","scrub_duration":0.0014316489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.176216+0000","last_change":"2024-09-10T18:21:37.642296+0000","last_active":"2024-09-10T18:22:46.176216+0000","last_peered":"2024-09-10T18:22:46.176216+0000","last_clean":"2024-09-10T18:22:46.176216+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:22:46.176216+0000","last_undegraded":"2024-09-10T18:22:46.176216+0000","last_fullsized":"2024-09-10T18:22:46.176216+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":69,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029573+0000","last_change":"2024-09-10T18:22:54.449637+0000","last_active":"2024-09-10T18:22:57.029573+0000","last_peered":"2024-09-10T18:22:57.029573+0000","last_clean":"2024-09-10T18:22:57.029573+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:22:57.029573+0000","last_undegraded":"2024-09-10T18:22:57.029573+0000","last_fullsized":"2024-09-10T18:22:57.029573+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:22:54.449573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:54.449573+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:07.055885+0000","scrub_duration":0.0022198460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.176823+0000","last_change":"2024-09-10T18:21:31.531998+0000","last_active":"2024-09-10T18:22:46.176823+0000","last_peered":"2024-09-10T18:22:46.176823+0000","last_clean":"2024-09-10T18:22:46.176823+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:22:46.176823+0000","last_undegraded":"2024-09-10T18:22:46.176823+0000","last_fullsized":"2024-09-10T18:22:46.176823+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:57.379141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:45.529422+0000","last_change":"2024-09-10T18:22:40.275165+0000","last_active":"2024-09-10T18:22:45.529422+0000","last_peered":"2024-09-10T18:22:45.529422+0000","last_clean":"2024-09-10T18:22:45.529422+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:22:45.529422+0000","last_undegraded":"2024-09-10T18:22:45.529422+0000","last_fullsized":"2024-09-10T18:22:45.529422+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:40.275069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:40.275069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:54.251771+0000","scrub_duration":0.001522121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.142371+0000","last_change":"2024-09-10T18:21:37.644889+0000","last_active":"2024-09-10T18:22:46.142371+0000","last_peered":"2024-09-10T18:22:46.142371+0000","last_clean":"2024-09-10T18:22:46.142371+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:22:46.142371+0000","last_undegraded":"2024-09-10T18:22:46.142371+0000","last_fullsized":"2024-09-10T18:22:46.142371+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:59.247137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":71,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.098980+0000","last_change":"2024-09-10T18:22:51.599652+0000","last_active":"2024-09-10T18:22:57.098980+0000","last_peered":"2024-09-10T18:22:57.098980+0000","last_clean":"2024-09-10T18:22:57.098980+0000","last_became_active":"2024-09-10T18:21:34.046741+0000","last_became_peered":"2024-09-10T18:21:34.046741+0000","last_unstale":"2024-09-10T18:22:57.098980+0000","last_undegraded":"2024-09-10T18:22:57.098980+0000","last_fullsized":"2024-09-10T18:22:57.098980+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-10T18:22:51.599559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:51.599559+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:02.843581+0000","scrub_duration":0.0019152609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":54,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.176816+0000","last_change":"2024-09-10T18:21:31.530630+0000","last_active":"2024-09-10T18:22:46.176816+0000","last_peered":"2024-09-10T18:22:46.176816+0000","last_clean":"2024-09-10T18:22:46.176816+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:22:46.176816+0000","last_undegraded":"2024-09-10T18:22:46.176816+0000","last_fullsized":"2024-09-10T18:22:46.176816+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":58,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:54.768408+0000","last_change":"2024-09-10T18:21:35.627197+0000","last_active":"2024-09-10T18:22:54.768408+0000","last_peered":"2024-09-10T18:22:54.768408+0000","last_clean":"2024-09-10T18:22:54.768408+0000","last_became_active":"2024-09-10T18:21:35.626744+0000","last_became_peered":"2024-09-10T18:21:35.626744+0000","last_unstale":"2024-09-10T18:22:54.768408+0000","last_undegraded":"2024-09-10T18:22:54.768408+0000","last_fullsized":"2024-09-10T18:22:54.768408+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:58.371060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099184+0000","last_change":"2024-09-10T18:21:38.563824+0000","last_active":"2024-09-10T18:22:57.099184+0000","last_peered":"2024-09-10T18:22:57.099184+0000","last_clean":"2024-09-10T18:22:57.099184+0000","last_became_active":"2024-09-10T18:21:38.563550+0000","last_became_peered":"2024-09-10T18:21:38.563550+0000","last_unstale":"2024-09-10T18:22:57.099184+0000","last_undegraded":"2024-09-10T18:22:57.099184+0000","last_fullsized":"2024-09-10T18:22:57.099184+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:05.155686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143848+0000","last_change":"2024-09-10T18:21:31.527287+0000","last_active":"2024-09-10T18:22:57.143848+0000","last_peered":"2024-09-10T18:22:57.143848+0000","last_clean":"2024-09-10T18:22:57.143848+0000","last_became_active":"2024-09-10T18:21:31.526818+0000","last_became_peered":"2024-09-10T18:21:31.526818+0000","last_unstale":"2024-09-10T18:22:57.143848+0000","last_undegraded":"2024-09-10T18:22:57.143848+0000","last_fullsized":"2024-09-10T18:22:57.143848+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.142787+0000","last_change":"2024-09-10T18:21:34.046070+0000","last_active":"2024-09-10T18:22:46.142787+0000","last_peered":"2024-09-10T18:22:46.142787+0000","last_clean":"2024-09-10T18:22:46.142787+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:22:46.142787+0000","last_undegraded":"2024-09-10T18:22:46.142787+0000","last_fullsized":"2024-09-10T18:22:46.142787+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099420+0000","last_change":"2024-09-10T18:21:35.622254+0000","last_active":"2024-09-10T18:22:57.099420+0000","last_peered":"2024-09-10T18:22:57.099420+0000","last_clean":"2024-09-10T18:22:57.099420+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:22:57.099420+0000","last_undegraded":"2024-09-10T18:22:57.099420+0000","last_fullsized":"2024-09-10T18:22:57.099420+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.176441+0000","last_change":"2024-09-10T18:21:38.565959+0000","last_active":"2024-09-10T18:22:46.176441+0000","last_peered":"2024-09-10T18:22:46.176441+0000","last_clean":"2024-09-10T18:22:46.176441+0000","last_became_active":"2024-09-10T18:21:38.565675+0000","last_became_peered":"2024-09-10T18:21:38.565675+0000","last_unstale":"2024-09-10T18:22:46.176441+0000","last_undegraded":"2024-09-10T18:22:46.176441+0000","last_fullsized":"2024-09-10T18:22:46.176441+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143615+0000","last_change":"2024-09-10T18:21:34.047798+0000","last_active":"2024-09-10T18:22:57.143615+0000","last_peered":"2024-09-10T18:22:57.143615+0000","last_clean":"2024-09-10T18:22:57.143615+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:22:57.143615+0000","last_undegraded":"2024-09-10T18:22:57.143615+0000","last_fullsized":"2024-09-10T18:22:57.143615+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":47,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:56.100131+0000","last_change":"2024-09-10T18:22:56.100073+0000","last_active":"2024-09-10T18:22:56.100131+0000","last_peered":"2024-09-10T18:22:56.100131+0000","last_clean":"2024-09-10T18:22:56.100131+0000","last_became_active":"2024-09-10T18:21:31.531895+0000","last_became_peered":"2024-09-10T18:21:31.531895+0000","last_unstale":"2024-09-10T18:22:56.100131+0000","last_undegraded":"2024-09-10T18:22:56.100131+0000","last_fullsized":"2024-09-10T18:22:56.100131+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:22:56.100012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:13.102252+0000","scrub_duration":0.001414212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":44,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:49.301955+0000","last_change":"2024-09-10T18:22:49.301887+0000","last_active":"2024-09-10T18:22:49.301955+0000","last_peered":"2024-09-10T18:22:49.301955+0000","last_clean":"2024-09-10T18:22:49.301955+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:22:49.301955+0000","last_undegraded":"2024-09-10T18:22:49.301955+0000","last_fullsized":"2024-09-10T18:22:49.301955+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:49.301823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:49.301823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:06.288915+0000","scrub_duration":0.001243433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:45.529679+0000","last_change":"2024-09-10T18:21:37.637599+0000","last_active":"2024-09-10T18:22:45.529679+0000","last_peered":"2024-09-10T18:22:45.529679+0000","last_clean":"2024-09-10T18:22:45.529679+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:22:45.529679+0000","last_undegraded":"2024-09-10T18:22:45.529679+0000","last_fullsized":"2024-09-10T18:22:45.529679+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:02.280321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099447+0000","last_change":"2024-09-10T18:21:34.047045+0000","last_active":"2024-09-10T18:22:57.099447+0000","last_peered":"2024-09-10T18:22:57.099447+0000","last_clean":"2024-09-10T18:22:57.099447+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:22:57.099447+0000","last_undegraded":"2024-09-10T18:22:57.099447+0000","last_fullsized":"2024-09-10T18:22:57.099447+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143142+0000","last_change":"2024-09-10T18:22:38.445504+0000","last_active":"2024-09-10T18:22:57.143142+0000","last_peered":"2024-09-10T18:22:57.143142+0000","last_clean":"2024-09-10T18:22:57.143142+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:22:57.143142+0000","last_undegraded":"2024-09-10T18:22:57.143142+0000","last_fullsized":"2024-09-10T18:22:57.143142+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:38.445448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:38.445448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:05.044685+0000","scrub_duration":0.0018359400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":69,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:54.768605+0000","last_change":"2024-09-10T18:22:48.317756+0000","last_active":"2024-09-10T18:22:54.768605+0000","last_peered":"2024-09-10T18:22:54.768605+0000","last_clean":"2024-09-10T18:22:54.768605+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:22:54.768605+0000","last_undegraded":"2024-09-10T18:22:54.768605+0000","last_fullsized":"2024-09-10T18:22:54.768605+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:22:48.317687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:48.317687+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:01.400638+0000","scrub_duration":0.0022162499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-10T18:23:03.608800+0000","last_change":"2024-09-10T18:23:03.608764+0000","last_active":"2024-09-10T18:23:03.608800+0000","last_peered":"2024-09-10T18:23:03.608800+0000","last_clean":"2024-09-10T18:23:03.608800+0000","last_became_active":"2024-09-10T18:21:37.639706+0000","last_became_peered":"2024-09-10T18:21:37.639706+0000","last_unstale":"2024-09-10T18:23:03.608800+0000","last_undegraded":"2024-09-10T18:23:03.608800+0000","last_fullsized":"2024-09-10T18:23:03.608800+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:03.608698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:23:03.608698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:24.670346+0000","scrub_duration":0.0015379020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":61,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.143299+0000","last_change":"2024-09-10T18:22:33.342047+0000","last_active":"2024-09-10T18:22:46.143299+0000","last_peered":"2024-09-10T18:22:46.143299+0000","last_clean":"2024-09-10T18:22:46.143299+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:22:46.143299+0000","last_undegraded":"2024-09-10T18:22:46.143299+0000","last_fullsized":"2024-09-10T18:22:46.143299+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-10T18:22:33.341980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:33.341980+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:47.079316+0000","scrub_duration":0.0021839350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.176369+0000","last_change":"2024-09-10T18:21:31.532402+0000","last_active":"2024-09-10T18:22:46.176369+0000","last_peered":"2024-09-10T18:22:46.176369+0000","last_clean":"2024-09-10T18:22:46.176369+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:22:46.176369+0000","last_undegraded":"2024-09-10T18:22:46.176369+0000","last_fullsized":"2024-09-10T18:22:46.176369+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:59.769979+0000","last_change":"2024-09-10T18:22:50.618285+0000","last_active":"2024-09-10T18:22:59.769979+0000","last_peered":"2024-09-10T18:22:59.769979+0000","last_clean":"2024-09-10T18:22:59.769979+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:22:59.769979+0000","last_undegraded":"2024-09-10T18:22:59.769979+0000","last_fullsized":"2024-09-10T18:22:59.769979+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:22:50.618215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:50.618215+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:08.321441+0000","scrub_duration":0.0017651660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143200+0000","last_change":"2024-09-10T18:21:37.644078+0000","last_active":"2024-09-10T18:22:57.143200+0000","last_peered":"2024-09-10T18:22:57.143200+0000","last_clean":"2024-09-10T18:22:57.143200+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:22:57.143200+0000","last_undegraded":"2024-09-10T18:22:57.143200+0000","last_fullsized":"2024-09-10T18:22:57.143200+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029576+0000","last_change":"2024-09-10T18:21:33.611605+0000","last_active":"2024-09-10T18:22:57.029576+0000","last_peered":"2024-09-10T18:22:57.029576+0000","last_clean":"2024-09-10T18:22:57.029576+0000","last_became_active":"2024-09-10T18:21:33.611286+0000","last_became_peered":"2024-09-10T18:21:33.611286+0000","last_unstale":"2024-09-10T18:22:57.029576+0000","last_undegraded":"2024-09-10T18:22:57.029576+0000","last_fullsized":"2024-09-10T18:22:57.029576+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029856+0000","last_change":"2024-09-10T18:22:53.433078+0000","last_active":"2024-09-10T18:22:57.029856+0000","last_peered":"2024-09-10T18:22:57.029856+0000","last_clean":"2024-09-10T18:22:57.029856+0000","last_became_active":"2024-09-10T18:21:31.530158+0000","last_became_peered":"2024-09-10T18:21:31.530158+0000","last_unstale":"2024-09-10T18:22:57.029856+0000","last_undegraded":"2024-09-10T18:22:57.029856+0000","last_fullsized":"2024-09-10T18:22:57.029856+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:53.433012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:53.433012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:19.414323+0000","scrub_duration":0.0013676739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.141341+0000","last_change":"2024-09-10T18:22:36.049767+0000","last_active":"2024-09-10T18:22:46.141341+0000","last_peered":"2024-09-10T18:22:46.141341+0000","last_clean":"2024-09-10T18:22:46.141341+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:22:46.141341+0000","last_undegraded":"2024-09-10T18:22:46.141341+0000","last_fullsized":"2024-09-10T18:22:46.141341+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:36.049705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:36.049705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:39.206398+0000","scrub_duration":0.0030879660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.141980+0000","last_change":"2024-09-10T18:21:37.642856+0000","last_active":"2024-09-10T18:22:46.141980+0000","last_peered":"2024-09-10T18:22:46.141980+0000","last_clean":"2024-09-10T18:22:46.141980+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:22:46.141980+0000","last_undegraded":"2024-09-10T18:22:46.141980+0000","last_fullsized":"2024-09-10T18:22:46.141980+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:04.616953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":60,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.140242+0000","last_change":"2024-09-10T18:22:42.015349+0000","last_active":"2024-09-10T18:22:46.140242+0000","last_peered":"2024-09-10T18:22:46.140242+0000","last_clean":"2024-09-10T18:22:46.140242+0000","last_became_active":"2024-09-10T18:21:33.612401+0000","last_became_peered":"2024-09-10T18:21:33.612401+0000","last_unstale":"2024-09-10T18:22:46.140242+0000","last_undegraded":"2024-09-10T18:22:46.140242+0000","last_fullsized":"2024-09-10T18:22:46.140242+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:22:42.015292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:42.015292+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:07.052542+0000","scrub_duration":0.001986785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":47,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.176613+0000","last_change":"2024-09-10T18:22:43.250096+0000","last_active":"2024-09-10T18:22:46.176613+0000","last_peered":"2024-09-10T18:22:46.176613+0000","last_clean":"2024-09-10T18:22:46.176613+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:22:46.176613+0000","last_undegraded":"2024-09-10T18:22:46.176613+0000","last_fullsized":"2024-09-10T18:22:46.176613+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:43.250030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:43.250030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:12.038178+0000","scrub_duration":0.001581381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.142393+0000","last_change":"2024-09-10T18:21:35.623087+0000","last_active":"2024-09-10T18:22:46.142393+0000","last_peered":"2024-09-10T18:22:46.142393+0000","last_clean":"2024-09-10T18:22:46.142393+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:22:46.142393+0000","last_undegraded":"2024-09-10T18:22:46.142393+0000","last_fullsized":"2024-09-10T18:22:46.142393+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:03.160666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.142674+0000","last_change":"2024-09-10T18:22:52.262629+0000","last_active":"2024-09-10T18:22:57.142674+0000","last_peered":"2024-09-10T18:22:57.142674+0000","last_clean":"2024-09-10T18:22:57.142674+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:22:57.142674+0000","last_undegraded":"2024-09-10T18:22:57.142674+0000","last_fullsized":"2024-09-10T18:22:57.142674+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:52.262563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:52.262563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:20.159979+0000","scrub_duration":0.001452381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":55,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.142190+0000","last_change":"2024-09-10T18:21:34.046343+0000","last_active":"2024-09-10T18:22:46.142190+0000","last_peered":"2024-09-10T18:22:46.142190+0000","last_clean":"2024-09-10T18:22:46.142190+0000","last_became_active":"2024-09-10T18:21:34.046024+0000","last_became_peered":"2024-09-10T18:21:34.046024+0000","last_unstale":"2024-09-10T18:22:46.142190+0000","last_undegraded":"2024-09-10T18:22:46.142190+0000","last_fullsized":"2024-09-10T18:22:46.142190+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099492+0000","last_change":"2024-09-10T18:22:40.686003+0000","last_active":"2024-09-10T18:22:57.099492+0000","last_peered":"2024-09-10T18:22:57.099492+0000","last_clean":"2024-09-10T18:22:57.099492+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:22:57.099492+0000","last_undegraded":"2024-09-10T18:22:57.099492+0000","last_fullsized":"2024-09-10T18:22:57.099492+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:22:40.685879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:01.722544+0000","scrub_duration":0.001767794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.175777+0000","last_change":"2024-09-10T18:21:35.626825+0000","last_active":"2024-09-10T18:22:46.175777+0000","last_peered":"2024-09-10T18:22:46.175777+0000","last_clean":"2024-09-10T18:22:46.175777+0000","last_became_active":"2024-09-10T18:21:35.626537+0000","last_became_peered":"2024-09-10T18:21:35.626537+0000","last_unstale":"2024-09-10T18:22:46.175777+0000","last_undegraded":"2024-09-10T18:22:46.175777+0000","last_fullsized":"2024-09-10T18:22:46.175777+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.343243+0000","last_change":"2024-09-10T18:22:46.343197+0000","last_active":"2024-09-10T18:22:46.343243+0000","last_peered":"2024-09-10T18:22:46.343243+0000","last_clean":"2024-09-10T18:22:46.343243+0000","last_became_active":"2024-09-10T18:21:38.564314+0000","last_became_peered":"2024-09-10T18:21:38.564314+0000","last_unstale":"2024-09-10T18:22:46.343243+0000","last_undegraded":"2024-09-10T18:22:46.343243+0000","last_fullsized":"2024-09-10T18:22:46.343243+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:46.343137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:46.343137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:10.438171+0000","scrub_duration":0.0016021189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.142191+0000","last_change":"2024-09-10T18:21:33.618322+0000","last_active":"2024-09-10T18:22:46.142191+0000","last_peered":"2024-09-10T18:22:46.142191+0000","last_clean":"2024-09-10T18:22:46.142191+0000","last_became_active":"2024-09-10T18:21:33.617670+0000","last_became_peered":"2024-09-10T18:21:33.617670+0000","last_unstale":"2024-09-10T18:22:46.142191+0000","last_undegraded":"2024-09-10T18:22:46.142191+0000","last_fullsized":"2024-09-10T18:22:46.142191+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:22:58.493392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143575+0000","last_change":"2024-09-10T18:21:31.530146+0000","last_active":"2024-09-10T18:22:57.143575+0000","last_peered":"2024-09-10T18:22:57.143575+0000","last_clean":"2024-09-10T18:22:57.143575+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:22:57.143575+0000","last_undegraded":"2024-09-10T18:22:57.143575+0000","last_fullsized":"2024-09-10T18:22:57.143575+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.144303+0000","last_change":"2024-09-10T18:22:49.757505+0000","last_active":"2024-09-10T18:22:57.144303+0000","last_peered":"2024-09-10T18:22:57.144303+0000","last_clean":"2024-09-10T18:22:57.144303+0000","last_became_active":"2024-09-10T18:21:35.620419+0000","last_became_peered":"2024-09-10T18:21:35.620419+0000","last_unstale":"2024-09-10T18:22:57.144303+0000","last_undegraded":"2024-09-10T18:22:57.144303+0000","last_fullsized":"2024-09-10T18:22:57.144303+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:49.757435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:49.757435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:00.814048+0000","scrub_duration":0.0014410040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.176069+0000","last_change":"2024-09-10T18:21:37.643084+0000","last_active":"2024-09-10T18:22:46.176069+0000","last_peered":"2024-09-10T18:22:46.176069+0000","last_clean":"2024-09-10T18:22:46.176069+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:22:46.176069+0000","last_undegraded":"2024-09-10T18:22:46.176069+0000","last_fullsized":"2024-09-10T18:22:46.176069+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:05.246215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.141364+0000","last_change":"2024-09-10T18:22:44.108462+0000","last_active":"2024-09-10T18:22:46.141364+0000","last_peered":"2024-09-10T18:22:46.141364+0000","last_clean":"2024-09-10T18:22:46.141364+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:22:46.141364+0000","last_undegraded":"2024-09-10T18:22:46.141364+0000","last_fullsized":"2024-09-10T18:22:46.141364+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:44.108373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:44.108373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:51.902386+0000","scrub_duration":0.0024985559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-10T18:22:46.143303+0000","last_change":"2024-09-10T18:21:31.535545+0000","last_active":"2024-09-10T18:22:46.143303+0000","last_peered":"2024-09-10T18:22:46.143303+0000","last_clean":"2024-09-10T18:22:46.143303+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:22:46.143303+0000","last_undegraded":"2024-09-10T18:22:46.143303+0000","last_fullsized":"2024-09-10T18:22:46.143303+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":65,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-10T18:23:00.864553+0000","last_change":"2024-09-10T18:23:00.864502+0000","last_active":"2024-09-10T18:23:00.864553+0000","last_peered":"2024-09-10T18:23:00.864553+0000","last_clean":"2024-09-10T18:23:00.864553+0000","last_became_active":"2024-09-10T18:21:33.613487+0000","last_became_peered":"2024-09-10T18:21:33.613487+0000","last_unstale":"2024-09-10T18:23:00.864553+0000","last_undegraded":"2024-09-10T18:23:00.864553+0000","last_fullsized":"2024-09-10T18:23:00.864553+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:23:00.864408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:23:00.864408+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:02.969515+0000","scrub_duration":0.0025444080000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143363+0000","last_change":"2024-09-10T18:21:35.620819+0000","last_active":"2024-09-10T18:22:57.143363+0000","last_peered":"2024-09-10T18:22:57.143363+0000","last_clean":"2024-09-10T18:22:57.143363+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:22:57.143363+0000","last_undegraded":"2024-09-10T18:22:57.143363+0000","last_fullsized":"2024-09-10T18:22:57.143363+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":414,"ondisk_log_size":414,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":81,"num_read_kb":76,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234005,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298828,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446324,"statfs":{"total":95995035648,"available":95689035776,"internally_reserved":0,"allocated":1261568,"data_stored":895509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.26900000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.36599999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.48299999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.505},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.434}]},{"osd":4,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.44400000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.55700000000000005}]},{"osd":6,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.53800000000000003}]}]},{"osd":6,"up_from":46,"seq":197568495644,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298584,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446568,"statfs":{"total":95995035648,"available":95689285632,"internally_reserved":0,"allocated":1216512,"data_stored":883588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55100000000000005,"15min":0.56000000000000005},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71299999999999997,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53900000000000003,"15min":0.54000000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.747,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.61299999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.55400000000000005,"15min":0.56000000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.65400000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54200000000000004,"15min":0.54600000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74099999999999999,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.54500000000000004}]},{"osd":2,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.71199999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54900000000000004,"15min":0.55200000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73999999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.64600000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.68100000000000005,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.625},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.70899999999999996,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.501}]},{"osd":4,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.42299999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.438,"15min":0.434},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.57699999999999996}]},{"osd":5,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.443,"15min":0.435},"min":{"1min":0.317,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.44600000000000001,"15min":0.439},"min":{"1min":0.315,"5min":0.308,"15min":0.308},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.45000000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.56599999999999995}]}]},{"osd":5,"up_from":39,"seq":167503724577,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297960,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447192,"statfs":{"total":95995035648,"available":95689924608,"internally_reserved":0,"allocated":622592,"data_stored":292881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.36399999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.42499999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.65600000000000003}]},{"osd":1,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.56499999999999995,"15min":0.56100000000000005},"min":{"1min":0.42399999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.377,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.69899999999999995}]},{"osd":2,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55100000000000005,"15min":0.55200000000000005},"min":{"1min":0.38800000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.38,"5min":0.255,"15min":0.255},"max":{"1min":0.75600000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.51100000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.54800000000000004,"15min":0.54300000000000004},"min":{"1min":0.434,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.52900000000000003,"15min":0.53600000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.67400000000000004}]},{"osd":4,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.36499999999999999,"15min":0.35899999999999999},"min":{"1min":0.29499999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.38400000000000001},"min":{"1min":0.28599999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.36199999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:22:58 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.42399999999999999,"15min":0.40999999999999998},"min":{"1min":0.36599999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.44800000000000001,"15min":0.43099999999999999},"min":{"1min":0.34100000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.38}]},{"osd":7,"last update":"Tue Sep 10 18:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.41199999999999998}]}]},{"osd":4,"up_from":33,"seq":141733920805,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297976,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447176,"statfs":{"total":95995035648,"available":95689908224,"internally_reserved":0,"allocated":638976,"data_stored":292859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56599999999999995,"15min":0.57499999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.72699999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.746,"5min":0.61799999999999999,"15min":0.59699999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.8170000000000002,"5min":4.8170000000000002,"15min":4.8170000000000002},"last":0.60899999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.61499999999999999,"15min":0.59699999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":4.7729999999999997,"5min":4.7729999999999997,"15min":4.7729999999999997},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.64100000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.60699999999999998,"15min":0.57899999999999996},"min":{"1min":0.42799999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.8040000000000003,"5min":4.8040000000000003,"15min":4.8040000000000003},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.57599999999999996,"15min":0.58599999999999997},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71699999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.57599999999999996}]},{"osd":3,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.66600000000000004,"5min":0.751,"15min":0.751},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59099999999999997,"15min":0.59599999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.747,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.65800000000000003}]},{"osd":5,"last update":"Tue Sep 10 18:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.39000000000000001,"15min":0.38300000000000001},"min":{"1min":0.32800000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42499999999999999,"15min":0.42399999999999999},"min":{"1min":0.28299999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60099999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.45500000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:22:59 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40400000000000003},"min":{"1min":0.29299999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.61599999999999999,"5min":0.629,"15min":0.629},"last":0.496},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.46999999999999997,"15min":0.46600000000000003},"min":{"1min":0.36299999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64100000000000001,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.47599999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.53200000000000003}]}]},{"osd":3,"up_from":27,"seq":115964117034,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298260,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446892,"statfs":{"total":95995035648,"available":95689617408,"internally_reserved":0,"allocated":679936,"data_stored":305796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.39800000000000002,"15min":0.378},"min":{"1min":0.23000000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.40999999999999998,"15min":0.39600000000000002},"min":{"1min":0.255,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.38300000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.437,"15min":0.435},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.41599999999999998,"15min":0.40100000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.34000000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.45100000000000001,"15min":0.44},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.46999999999999997,"15min":0.41599999999999998},"min":{"1min":0.27700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":7.2750000000000004,"5min":7.2750000000000004,"15min":7.2750000000000004},"last":0.33000000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:22:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.55600000000000005,"15min":0.54000000000000004},"min":{"1min":0.378,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.56499999999999995,"15min":0.55500000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.40899999999999997}]},{"osd":5,"last update":"Tue Sep 10 18:22:34 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":0.73899999999999999,"15min":0.63400000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":14.885,"5min":14.885,"15min":14.885},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.61899999999999999,"15min":0.60699999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.45700000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.46600000000000003}]},{"osd":7,"last update":"Tue Sep 10 18:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.47699999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":643072,"data_stored":304204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.36199999999999999,"15min":0.33400000000000002},"min":{"1min":0.22,"5min":0.219,"15min":0.219},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.371,"15min":0.34899999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.34899999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.35599999999999998,"15min":0.33800000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.63100000000000001,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.39100000000000001,"15min":0.35499999999999998},"min":{"1min":0.28199999999999997,"5min":0.22,"15min":0.22},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.31}]},{"osd":3,"last update":"Tue Sep 10 18:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.48399999999999999,"15min":0.46999999999999997},"min":{"1min":0.313,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.41699999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.56799999999999995,"15min":0.56499999999999995},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.76300000000000001,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.76000000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.51000000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.61499999999999999,"15min":0.59399999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":2.3279999999999998,"5min":2.3279999999999998,"15min":2.3279999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.63,"15min":0.61599999999999999},"min":{"1min":0.44500000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.48599999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:21:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.54700000000000004}]},{"osd":7,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.56399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298284,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446868,"statfs":{"total":95995035648,"available":95689592832,"internally_reserved":0,"allocated":704512,"data_stored":306132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39000000000000001,"15min":0.373},"min":{"1min":0.27700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.41199999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.379},"min":{"1min":0.25700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.38200000000000001,"15min":0.36399999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.64800000000000002,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38400000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.26800000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.59399999999999997,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.45100000000000001,"15min":0.46400000000000002},"min":{"1min":0.30499999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.63300000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.45200000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.85899999999999999,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.94799999999999995,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.51000000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.55100000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.496}]},{"osd":6,"last update":"Tue Sep 10 18:21:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.53400000000000003}]},{"osd":7,"last update":"Tue Sep 10 18:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.56299999999999994}]}]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298840,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446312,"statfs":{"total":95995035648,"available":95689023488,"internally_reserved":0,"allocated":1228800,"data_stored":883100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:23:02 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.35999999999999999,"15min":0.34899999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.45000000000000001,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.252},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.377,"15min":0.35399999999999998},"min":{"1min":0.25,"5min":0.249,"15min":0.249},"max":{"1min":0.49199999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.33200000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40999999999999998,"15min":0.40699999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53500000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.40600000000000003,"15min":0.40200000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.47899999999999998,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.38800000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.44400000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.55500000000000005,"5min":0.629,"15min":0.629},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.434,"15min":0.44500000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.54100000000000004,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.30399999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:22:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.624,"15min":0.63100000000000001},"min":{"1min":0.433,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.83299999999999996,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.65000000000000002,"15min":0.66100000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81899999999999995,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.52200000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.59399999999999997,"15min":0.60299999999999998},"min":{"1min":0.437,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.72099999999999997,"5min":0.879,"15min":0.879},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.625,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.496}]},{"osd":6,"last update":"Tue Sep 10 18:23:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60899999999999999,"15min":0.61299999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.61899999999999999,"15min":0.624},"min":{"1min":0.46000000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.76800000000000002,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.56000000000000005}]},{"osd":7,"last update":"Tue Sep 10 18:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.53400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:23:05.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:23:05.824 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:06 smithi028 ceph-mon[29291]: 6.1f scrub starts 2024-09-10T18:23:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:06 smithi028 ceph-mon[29291]: 6.1f scrub ok 2024-09-10T18:23:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:06 smithi028 ceph-mon[29291]: 5.18 deep-scrub starts 2024-09-10T18:23:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:06 smithi028 ceph-mon[29291]: 5.18 deep-scrub ok 2024-09-10T18:23:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:06 smithi028 ceph-mon[29291]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:23:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:06 smithi028 ceph-mon[29291]: 4.5 scrub starts 2024-09-10T18:23:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:06 smithi028 ceph-mon[29291]: 4.5 scrub ok 2024-09-10T18:23:06.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:06 smithi028 ceph-mon[29291]: from='client.14745 v1:172.21.15.28:0/834877356' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:23:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:06 smithi028 ceph-mon[39220]: 6.1f scrub starts 2024-09-10T18:23:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:06 smithi028 ceph-mon[39220]: 6.1f scrub ok 2024-09-10T18:23:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:06 smithi028 ceph-mon[39220]: 5.18 deep-scrub starts 2024-09-10T18:23:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:06 smithi028 ceph-mon[39220]: 5.18 deep-scrub ok 2024-09-10T18:23:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:06 smithi028 ceph-mon[39220]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:23:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:06 smithi028 ceph-mon[39220]: 4.5 scrub starts 2024-09-10T18:23:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:06 smithi028 ceph-mon[39220]: 4.5 scrub ok 2024-09-10T18:23:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:06 smithi028 ceph-mon[39220]: from='client.14745 v1:172.21.15.28:0/834877356' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:23:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:06 smithi204 ceph-mon[32357]: 6.1f scrub starts 2024-09-10T18:23:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:06 smithi204 ceph-mon[32357]: 6.1f scrub ok 2024-09-10T18:23:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:06 smithi204 ceph-mon[32357]: 5.18 deep-scrub starts 2024-09-10T18:23:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:06 smithi204 ceph-mon[32357]: 5.18 deep-scrub ok 2024-09-10T18:23:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:06 smithi204 ceph-mon[32357]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:23:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:06 smithi204 ceph-mon[32357]: 4.5 scrub starts 2024-09-10T18:23:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:06 smithi204 ceph-mon[32357]: 4.5 scrub ok 2024-09-10T18:23:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:06 smithi204 ceph-mon[32357]: from='client.14745 v1:172.21.15.28:0/834877356' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:23:07.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-mon[32357]: 4.11 scrub starts 2024-09-10T18:23:07.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-mon[32357]: 4.11 scrub ok 2024-09-10T18:23:07.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:07 smithi028 ceph-mon[29291]: 4.11 scrub starts 2024-09-10T18:23:07.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:07 smithi028 ceph-mon[29291]: 4.11 scrub ok 2024-09-10T18:23:07.295 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:07 smithi028 ceph-mon[39220]: 4.11 scrub starts 2024-09-10T18:23:07.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:07 smithi028 ceph-mon[39220]: 4.11 scrub ok 2024-09-10T18:23:07.490 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 systemd[1]: Started Ceph grafana.a for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:23:07.690 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:23:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:23:07.743 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=settings t=2024-09-10T18:23:07.489408645Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-10T18:23:07.743 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=settings t=2024-09-10T18:23:07.489613149Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-10T18:23:07.743 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=settings t=2024-09-10T18:23:07.489677166Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-10T18:23:07.743 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=settings t=2024-09-10T18:23:07.489721414Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-10T18:23:07.743 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=settings t=2024-09-10T18:23:07.489758749Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-10T18:23:07.743 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=settings t=2024-09-10T18:23:07.489772146Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-10T18:23:07.743 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=settings t=2024-09-10T18:23:07.489782912Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-10T18:23:07.743 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=settings t=2024-09-10T18:23:07.489794095Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-10T18:23:07.743 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=settings t=2024-09-10T18:23:07.489805096Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-10T18:23:07.743 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=settings t=2024-09-10T18:23:07.489815685Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-10T18:23:07.744 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=settings t=2024-09-10T18:23:07.489826097Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-10T18:23:07.744 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=settings t=2024-09-10T18:23:07.489836291Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-10T18:23:07.744 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=settings t=2024-09-10T18:23:07.489846856Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-10T18:23:07.744 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=settings t=2024-09-10T18:23:07.489857471Z level=info msg="App mode production" 2024-09-10T18:23:07.744 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=sqlstore t=2024-09-10T18:23:07.489930093Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-10T18:23:07.744 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=sqlstore t=2024-09-10T18:23:07.48995564Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-10T18:23:07.744 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.490898232Z level=info msg="Starting DB migrations" 2024-09-10T18:23:07.744 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.491450474Z level=info msg="Executing migration" id="create migration_log table" 2024-09-10T18:23:07.744 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.491975739Z level=info msg="Executing migration" id="create user table" 2024-09-10T18:23:07.745 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.49241628Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-10T18:23:07.745 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.492850363Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-10T18:23:07.745 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.493295807Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-10T18:23:07.746 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.493831639Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-10T18:23:07.746 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.494242597Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-10T18:23:07.746 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.494926191Z level=info msg="Executing migration" id="create user table v2" 2024-09-10T18:23:07.746 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.495328131Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-10T18:23:07.746 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.49580116Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-10T18:23:07.746 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.496244624Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-10T18:23:07.746 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.496685486Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-10T18:23:07.746 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.497060476Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-10T18:23:07.746 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.497935138Z level=info msg="Executing migration" id="Update user table charset" 2024-09-10T18:23:07.746 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.498941665Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-10T18:23:07.746 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.499619464Z level=info msg="Executing migration" id="Add missing user data" 2024-09-10T18:23:07.746 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.499946173Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-10T18:23:07.746 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.50037616Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-10T18:23:07.747 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.500823841Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-10T18:23:07.747 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.501266887Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-10T18:23:07.747 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.502380163Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-10T18:23:07.747 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.503242456Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-10T18:23:07.747 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.503684803Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-10T18:23:07.747 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.506339377Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-10T18:23:07.747 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.509063922Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-10T18:23:07.747 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.511708475Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-10T18:23:07.747 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.514291691Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-10T18:23:07.747 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.517016096Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-10T18:23:07.747 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.519661691Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-10T18:23:07.747 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.522409396Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-10T18:23:07.747 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.525110844Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-10T18:23:07.747 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.52784058Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-10T18:23:07.748 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.530654058Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-10T18:23:07.748 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.533418296Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-10T18:23:07.748 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.53624533Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-10T18:23:07.748 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.538906636Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-10T18:23:07.748 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.541523438Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-10T18:23:07.748 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.544248869Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-10T18:23:07.748 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.546783412Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-10T18:23:07.748 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.547248919Z level=info msg="Executing migration" id="create star table" 2024-09-10T18:23:07.748 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.550054976Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-10T18:23:07.748 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.550590921Z level=info msg="Executing migration" id="create org table v1" 2024-09-10T18:23:07.748 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.552698919Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-10T18:23:07.748 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.555176406Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-10T18:23:07.748 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.557833633Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-10T18:23:07.748 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.560524849Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-10T18:23:07.749 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.563210676Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-10T18:23:07.749 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.563684675Z level=info msg="Executing migration" id="Update org table charset" 2024-09-10T18:23:07.749 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.566413874Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-10T18:23:07.749 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.569043618Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-10T18:23:07.749 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.571499757Z level=info msg="Executing migration" id="create dashboard table" 2024-09-10T18:23:07.749 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.571967003Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-10T18:23:07.749 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.574666947Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-10T18:23:07.749 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.577229952Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-10T18:23:07.749 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.579948245Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-10T18:23:07.749 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.580399049Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-10T18:23:07.749 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.583010785Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-10T18:23:07.749 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.584012799Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-10T18:23:07.749 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.584451074Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-10T18:23:07.750 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.585279884Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-10T18:23:07.750 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.585751837Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-10T18:23:07.750 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.586654321Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-10T18:23:07.750 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.587330307Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-10T18:23:07.750 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.588321003Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-10T18:23:07.750 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.589006239Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-10T18:23:07.750 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.589843703Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-10T18:23:07.750 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.59077099Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-10T18:23:07.750 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.591374075Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-10T18:23:07.750 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.59185509Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-10T18:23:07.750 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.592317404Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-10T18:23:07.750 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.592726734Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-10T18:23:07.750 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.593059135Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-10T18:23:07.751 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.593533907Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-10T18:23:07.751 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.594381606Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-10T18:23:07.751 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.594916026Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-10T18:23:07.751 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.595798566Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-10T18:23:07.751 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.596674329Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-10T18:23:07.751 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.59739452Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-10T18:23:07.751 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.597847339Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-10T18:23:07.751 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.598616557Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-10T18:23:07.751 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.59945531Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-10T18:23:07.751 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.600328474Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-10T18:23:07.751 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.600775329Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-10T18:23:07.751 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.601716308Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-10T18:23:07.751 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.602695174Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-10T18:23:07.751 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.603361577Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-10T18:23:07.752 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.60384292Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-10T18:23:07.752 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.604677978Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-10T18:23:07.752 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.605542859Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-10T18:23:07.752 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.606402442Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-10T18:23:07.752 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.606867603Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-10T18:23:07.752 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.607723704Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-10T18:23:07.752 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.610387877Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-10T18:23:07.752 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.613218357Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-10T18:23:07.752 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.6139122Z level=info msg="Executing migration" id="create data_source table" 2024-09-10T18:23:07.752 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.616671874Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-10T18:23:07.752 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.619471769Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-10T18:23:07.752 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.62226903Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-10T18:23:07.752 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.622692285Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-10T18:23:07.752 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.625520832Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-10T18:23:07.752 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.628301822Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-10T18:23:07.753 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.630862077Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-10T18:23:07.753 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.631334676Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-10T18:23:07.753 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.634189685Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-10T18:23:07.753 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.636809418Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-10T18:23:07.753 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.639678501Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-10T18:23:07.753 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.640286229Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-10T18:23:07.753 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.643186791Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-10T18:23:07.753 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.645675202Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-10T18:23:07.753 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.648438119Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-10T18:23:07.753 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.648909775Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-10T18:23:07.753 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.651425271Z level=info msg="Executing migration" id="Add uid column" 2024-09-10T18:23:07.753 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.65415197Z level=info msg="Executing migration" id="Update uid value" 2024-09-10T18:23:07.753 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.65674842Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-10T18:23:07.754 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.65729066Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-10T18:23:07.754 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.659848533Z level=info msg="Executing migration" id="create api_key table" 2024-09-10T18:23:07.754 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.662637316Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-10T18:23:07.754 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.665313078Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-10T18:23:07.754 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.665959441Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-10T18:23:07.754 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.668559122Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-10T18:23:07.754 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.671152313Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-10T18:23:07.754 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.673668031Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-10T18:23:07.754 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.676398996Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-10T18:23:07.754 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.678965607Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-10T18:23:07.754 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.681613635Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-10T18:23:07.754 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.684057113Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-10T18:23:07.754 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.68675853Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-10T18:23:07.754 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.687470714Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-10T18:23:07.755 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.688306677Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-10T18:23:07.755 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.68891414Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-10T18:23:07.755 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.689217763Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-10T18:23:07.755 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.689984767Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-10T18:23:07.755 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.690571586Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-10T18:23:07.755 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.69086439Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-10T18:23:07.755 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.691424254Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-10T18:23:07.755 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.691990672Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-10T18:23:07.755 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.692413793Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-10T18:23:07.756 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.692780859Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-10T18:23:07.756 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.693218382Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-10T18:23:07.756 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.693673178Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-10T18:23:07.756 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.694452299Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-10T18:23:07.756 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.695274168Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-10T18:23:07.756 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.696048656Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-10T18:23:07.756 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.696359724Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-10T18:23:07.756 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.696960195Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-10T18:23:07.756 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.697529706Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-10T18:23:07.756 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.697835732Z level=info msg="Executing migration" id="create quota table v1" 2024-09-10T18:23:07.757 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.698266029Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-10T18:23:07.757 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.698720458Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-10T18:23:07.757 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.699034656Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-10T18:23:07.757 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.699435819Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-10T18:23:07.757 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.699884223Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-10T18:23:07.757 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.700480668Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-10T18:23:07.757 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.700746787Z level=info msg="Executing migration" id="create session table" 2024-09-10T18:23:07.757 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.701201667Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-10T18:23:07.757 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.701489724Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-10T18:23:07.757 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.70175645Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-10T18:23:07.757 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.702145297Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-10T18:23:07.757 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.702550924Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-10T18:23:07.757 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.702805024Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-10T18:23:07.757 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.703103927Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-10T18:23:07.758 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.703398088Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-10T18:23:07.758 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.703668915Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-10T18:23:07.758 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.704087259Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-10T18:23:07.758 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.704368698Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-10T18:23:07.758 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.704933869Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-10T18:23:07.758 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.705241555Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-10T18:23:07.758 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.705819804Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-10T18:23:07.758 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.706433403Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-10T18:23:07.758 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.706705466Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-10T18:23:07.758 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.707131989Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-10T18:23:07.758 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.707584822Z level=info msg="Executing migration" id="create alert table v1" 2024-09-10T18:23:07.758 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.708422976Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-10T18:23:07.758 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.70927953Z level=info msg="Executing migration" id="add index alert state" 2024-09-10T18:23:07.758 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.710080219Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-10T18:23:07.758 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.710515877Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-10T18:23:07.759 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.710897334Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-10T18:23:07.759 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.711333974Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-10T18:23:07.759 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.711705498Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-10T18:23:07.759 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.713101965Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-10T18:23:07.759 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.713492064Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-10T18:23:07.759 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.713915959Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-10T18:23:07.759 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.714233138Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-10T18:23:07.759 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.714569723Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-10T18:23:07.759 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.71493434Z level=info msg="Executing migration" id="Add column is_default" 2024-09-10T18:23:07.759 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.71555301Z level=info msg="Executing migration" id="Add column frequency" 2024-09-10T18:23:07.759 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.716230809Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-10T18:23:07.759 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.716861454Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-10T18:23:07.759 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.717537337Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-10T18:23:07.760 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.717967537Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-10T18:23:07.760 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.718256094Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-10T18:23:07.760 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.718520698Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-10T18:23:07.760 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.718900228Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-10T18:23:07.760 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.719350412Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-10T18:23:07.760 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.719713325Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-10T18:23:07.760 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.720130234Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-10T18:23:07.760 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.720578625Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-10T18:23:07.760 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.721459205Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-10T18:23:07.760 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.722289095Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-10T18:23:07.760 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.72316576Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-10T18:23:07.760 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.724003436Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-10T18:23:07.760 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.724883195Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-10T18:23:07.760 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.725729315Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-10T18:23:07.761 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.726658355Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-10T18:23:07.761 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.727406782Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-10T18:23:07.761 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.728481071Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-10T18:23:07.761 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.729129316Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-10T18:23:07.761 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.730002459Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-10T18:23:07.761 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.731045459Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-10T18:23:07.761 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.733783252Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-10T18:23:07.761 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.736249682Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-10T18:23:07.761 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.738969423Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-10T18:23:07.761 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.74154055Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-10T18:23:07.761 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.744320637Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-10T18:23:07.993 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.746983226Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-10T18:23:07.994 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.749631458Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-10T18:23:07.994 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.752404139Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-10T18:23:07.994 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.755028506Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-10T18:23:07.994 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.757665077Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-10T18:23:07.994 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.76041174Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-10T18:23:07.994 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.763096885Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-10T18:23:07.994 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.76602735Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-10T18:23:07.994 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.768720305Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-10T18:23:07.994 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.771446297Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-10T18:23:07.994 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.774155324Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-10T18:23:07.994 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.777068728Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-10T18:23:07.994 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.779773473Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-10T18:23:07.995 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.782394847Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-10T18:23:07.995 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.785110777Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-10T18:23:07.995 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.78762404Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-10T18:23:07.995 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.790073573Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-10T18:23:07.995 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.7927974Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-10T18:23:07.996 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.7954223Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-10T18:23:07.996 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.798056904Z level=info msg="Executing migration" id="Move region to single row" 2024-09-10T18:23:07.996 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.800775185Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-10T18:23:07.996 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.803301956Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-10T18:23:07.996 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.805869179Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-10T18:23:07.996 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.808713054Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-10T18:23:07.996 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.811263726Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-10T18:23:07.997 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.814083784Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-10T18:23:07.997 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.816684544Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-10T18:23:07.997 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.819355282Z level=info msg="Executing migration" id="create test_data table" 2024-09-10T18:23:07.997 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.821853138Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-10T18:23:07.997 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.822297016Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-10T18:23:07.997 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.82505272Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-10T18:23:07.997 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.825780338Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-10T18:23:07.997 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.82662424Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-10T18:23:07.997 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.827352536Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-10T18:23:07.997 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.828244052Z level=info msg="Executing migration" id="create team table" 2024-09-10T18:23:07.997 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.828985473Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-10T18:23:07.997 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.829917134Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-10T18:23:07.997 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.830593932Z level=info msg="Executing migration" id="create team member table" 2024-09-10T18:23:07.997 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.831577362Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-10T18:23:07.998 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.832196831Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-10T18:23:07.998 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.833002189Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-10T18:23:07.998 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.833997817Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-10T18:23:07.998 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.834916109Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-10T18:23:07.998 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.835662769Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-10T18:23:07.998 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.836495644Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-10T18:23:07.998 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.837297253Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-10T18:23:07.998 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.838097939Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-10T18:23:07.998 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.841526265Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-10T18:23:07.998 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.8421703Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-10T18:23:07.998 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.842991771Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-10T18:23:07.998 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.843417674Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-10T18:23:07.998 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.84385476Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-10T18:23:07.998 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.844302838Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-10T18:23:07.998 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.844672939Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-10T18:23:07.999 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.8449588Z level=info msg="Executing migration" id="create tag table" 2024-09-10T18:23:07.999 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.845343374Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-10T18:23:07.999 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.845894687Z level=info msg="Executing migration" id="create login attempt table" 2024-09-10T18:23:07.999 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.846313885Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-10T18:23:07.999 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.846746881Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-10T18:23:07.999 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.847234134Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-10T18:23:07.999 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.849227581Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-10T18:23:07.999 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.849626739Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-10T18:23:07.999 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.850070903Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-10T18:23:07.999 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.850400814Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-10T18:23:07.999 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.850754204Z level=info msg="Executing migration" id="create user auth table" 2024-09-10T18:23:07.999 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.851161332Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-10T18:23:07.999 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.851589188Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-10T18:23:07.999 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.851868079Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-10T18:23:08.000 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.8526283Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-10T18:23:08.000 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.853409605Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-10T18:23:08.000 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.854222653Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-10T18:23:08.000 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.854979266Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-10T18:23:08.000 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.855413237Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-10T18:23:08.000 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.856170451Z level=info msg="Executing migration" id="create server_lock table" 2024-09-10T18:23:08.000 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.856574566Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-10T18:23:08.000 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.857473439Z level=info msg="Executing migration" id="create user auth token table" 2024-09-10T18:23:08.000 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.858320394Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-10T18:23:08.000 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.859143691Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-10T18:23:08.000 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.859612766Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-10T18:23:08.000 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.860620842Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-10T18:23:08.000 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.861488837Z level=info msg="Executing migration" id="create cache_data table" 2024-09-10T18:23:08.000 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.862362566Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-10T18:23:08.001 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.862818959Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-10T18:23:08.001 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.86379171Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-10T18:23:08.001 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.864475329Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-10T18:23:08.001 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.865341917Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-10T18:23:08.001 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.865891675Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-10T18:23:08.001 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.866341517Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-10T18:23:08.001 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.8690511Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-10T18:23:08.001 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.871866495Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-10T18:23:08.001 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.874428499Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-10T18:23:08.001 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.874855573Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-10T18:23:08.001 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.877819457Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-10T18:23:08.001 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.880417756Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-10T18:23:08.001 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.88318128Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-10T18:23:08.001 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.883990775Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-10T18:23:08.001 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.886748833Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-10T18:23:08.002 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.889272909Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-10T18:23:08.002 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.892032283Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-10T18:23:08.002 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.892468551Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-10T18:23:08.002 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.895194711Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-10T18:23:08.002 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.895517181Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-10T18:23:08.002 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.897993146Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-10T18:23:08.002 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.898541069Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-10T18:23:08.002 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.900643617Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-10T18:23:08.002 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.901108272Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-10T18:23:08.002 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.901889881Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-10T18:23:08.002 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.904684752Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-10T18:23:08.002 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.907269803Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-10T18:23:08.002 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.910611757Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-10T18:23:08.002 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.913941413Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-10T18:23:08.003 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.916673254Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-10T18:23:08.003 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.919331175Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-10T18:23:08.003 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.921956309Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-10T18:23:08.003 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.92240949Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-10T18:23:08.003 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.924993572Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-10T18:23:08.003 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.927486976Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-10T18:23:08.003 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.930150639Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-10T18:23:08.003 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.930514997Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-10T18:23:08.003 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.933086874Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-10T18:23:08.003 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.93576093Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-10T18:23:08.003 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.938408184Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-10T18:23:08.003 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.938859037Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-10T18:23:08.003 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.939659275Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-10T18:23:08.003 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.940499865Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-10T18:23:08.004 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.941423202Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-10T18:23:08.004 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.941882014Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-10T18:23:08.004 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.942716918Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-10T18:23:08.004 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.943546278Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-10T18:23:08.004 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.944509817Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-10T18:23:08.004 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.945097518Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-10T18:23:08.004 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.946056887Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-10T18:23:08.005 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.946697667Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-10T18:23:08.005 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.947528374Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-10T18:23:08.005 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.948634833Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-10T18:23:08.005 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.949480796Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-10T18:23:08.005 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.950351826Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-10T18:23:08.005 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.951215246Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-10T18:23:08.005 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.952163526Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-10T18:23:08.005 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.952480773Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-10T18:23:08.005 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.953320707Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-10T18:23:08.005 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.954204248Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-10T18:23:08.005 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.955043037Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-10T18:23:08.005 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.955917318Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-10T18:23:08.005 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.95681923Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-10T18:23:08.005 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.957789896Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-10T18:23:08.006 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.958614911Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-10T18:23:08.006 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.959099674Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-10T18:23:08.006 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.959986779Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-10T18:23:08.006 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.960806409Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-10T18:23:08.006 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.961793288Z level=info msg="Executing migration" id="create alert_image table" 2024-09-10T18:23:08.006 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.962219389Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-10T18:23:08.006 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.963168933Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-10T18:23:08.006 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.963963227Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-10T18:23:08.006 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.964921484Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-10T18:23:08.006 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.965493932Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-10T18:23:08.006 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.96556934Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-10T18:23:08.006 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.966233867Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-10T18:23:08.006 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.966619254Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-10T18:23:08.006 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.967108439Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-10T18:23:08.007 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.967225496Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-10T18:23:08.007 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.967330414Z level=info msg="alerts found to migrate" alerts=0 2024-09-10T18:23:08.007 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.970016057Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-10T18:23:08.007 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.970627198Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-10T18:23:08.007 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.973561097Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-10T18:23:08.007 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.976022241Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-10T18:23:08.007 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.978726989Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-10T18:23:08.007 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.979222383Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-10T18:23:08.007 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.981960076Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-10T18:23:08.007 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.984642419Z level=info msg="Executing migration" id="create data_keys table" 2024-09-10T18:23:08.007 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.987455185Z level=info msg="Executing migration" id="create secrets table" 2024-09-10T18:23:08.007 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.988054832Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-10T18:23:08.007 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.99230874Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-10T18:23:08.008 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.995089621Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-10T18:23:08.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[29291]: 3.1e scrub starts 2024-09-10T18:23:08.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[29291]: 3.1e scrub ok 2024-09-10T18:23:08.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[29291]: 6.14 scrub starts 2024-09-10T18:23:08.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[29291]: 6.14 scrub ok 2024-09-10T18:23:08.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[29291]: 3.1a scrub starts 2024-09-10T18:23:08.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[29291]: 3.1a scrub ok 2024-09-10T18:23:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[29291]: 4.15 scrub starts 2024-09-10T18:23:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[29291]: 4.15 scrub ok 2024-09-10T18:23:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[29291]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T18:23:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[29291]: 5.1e scrub starts 2024-09-10T18:23:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[29291]: 5.1e scrub ok 2024-09-10T18:23:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[29291]: 4.7 scrub starts 2024-09-10T18:23:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[29291]: 4.7 scrub ok 2024-09-10T18:23:08.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[39220]: 3.1e scrub starts 2024-09-10T18:23:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[39220]: 3.1e scrub ok 2024-09-10T18:23:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[39220]: 6.14 scrub starts 2024-09-10T18:23:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[39220]: 6.14 scrub ok 2024-09-10T18:23:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[39220]: 3.1a scrub starts 2024-09-10T18:23:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[39220]: 3.1a scrub ok 2024-09-10T18:23:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[39220]: 4.15 scrub starts 2024-09-10T18:23:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[39220]: 4.15 scrub ok 2024-09-10T18:23:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[39220]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T18:23:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[39220]: 5.1e scrub starts 2024-09-10T18:23:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[39220]: 5.1e scrub ok 2024-09-10T18:23:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:08.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:08.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[39220]: 4.7 scrub starts 2024-09-10T18:23:08.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:08 smithi028 ceph-mon[39220]: 4.7 scrub ok 2024-09-10T18:23:08.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-mon[32357]: 3.1e scrub starts 2024-09-10T18:23:08.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-mon[32357]: 3.1e scrub ok 2024-09-10T18:23:08.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-mon[32357]: 6.14 scrub starts 2024-09-10T18:23:08.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-mon[32357]: 6.14 scrub ok 2024-09-10T18:23:08.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-mon[32357]: 3.1a scrub starts 2024-09-10T18:23:08.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-mon[32357]: 3.1a scrub ok 2024-09-10T18:23:08.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-mon[32357]: 4.15 scrub starts 2024-09-10T18:23:08.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-mon[32357]: 4.15 scrub ok 2024-09-10T18:23:08.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-mon[32357]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T18:23:08.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-mon[32357]: 5.1e scrub starts 2024-09-10T18:23:08.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-mon[32357]: 5.1e scrub ok 2024-09-10T18:23:08.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:08.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:08.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:08.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:08.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-mon[32357]: 4.7 scrub starts 2024-09-10T18:23:08.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-mon[32357]: 4.7 scrub ok 2024-09-10T18:23:08.268 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:07.997745339Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-10T18:23:08.269 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.005241221Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-10T18:23:08.269 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.009638974Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-10T18:23:08.269 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.012151447Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-10T18:23:08.269 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.014923772Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-10T18:23:08.269 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.015398317Z level=info msg="Executing migration" id="create permission table" 2024-09-10T18:23:08.269 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.018130811Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-10T18:23:08.269 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.020636606Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-10T18:23:08.269 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.02333876Z level=info msg="Executing migration" id="create role table" 2024-09-10T18:23:08.269 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.023822845Z level=info msg="Executing migration" id="add column display_name" 2024-09-10T18:23:08.269 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.026624742Z level=info msg="Executing migration" id="add column group_name" 2024-09-10T18:23:08.269 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.029202247Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-10T18:23:08.269 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.031941716Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-10T18:23:08.269 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.032543238Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-10T18:23:08.270 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.035020352Z level=info msg="Executing migration" id="create team role table" 2024-09-10T18:23:08.270 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.03777911Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-10T18:23:08.270 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.040247098Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-10T18:23:08.270 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.040765955Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-10T18:23:08.270 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.043462266Z level=info msg="Executing migration" id="create user role table" 2024-09-10T18:23:08.270 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.045929878Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-10T18:23:08.270 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.048511632Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-10T18:23:08.270 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.049006464Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-10T18:23:08.270 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.049971637Z level=info msg="Executing migration" id="create builtin role table" 2024-09-10T18:23:08.270 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.050617745Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-10T18:23:08.270 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.051560016Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-10T18:23:08.270 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.052147482Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-10T18:23:08.270 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.053156984Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-10T18:23:08.270 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.053976232Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-10T18:23:08.270 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.054950305Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-10T18:23:08.271 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.055463742Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-10T18:23:08.271 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.056258814Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-10T18:23:08.271 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.05709483Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-10T18:23:08.271 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.057569997Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-10T18:23:08.271 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.058573221Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-10T18:23:08.271 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.059002845Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-10T18:23:08.271 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.059487221Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-10T18:23:08.271 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.059781444Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-10T18:23:08.271 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.060240681Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-10T18:23:08.271 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.060701969Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-10T18:23:08.271 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.061108575Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-10T18:23:08.271 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.061443252Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-10T18:23:08.271 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.061800841Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-10T18:23:08.271 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.062235519Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-10T18:23:08.272 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.062710425Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-10T18:23:08.272 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.0636951Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-10T18:23:08.272 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.064038292Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-10T18:23:08.272 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.064486517Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-10T18:23:08.272 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.064952268Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-10T18:23:08.272 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.065402582Z level=info msg="Executing migration" id="add correlation config column" 2024-09-10T18:23:08.272 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.066421589Z level=info msg="Executing migration" id="create entity_events table" 2024-09-10T18:23:08.272 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.066886623Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-10T18:23:08.272 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.067321774Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-10T18:23:08.272 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.067426717Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-10T18:23:08.272 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.067727302Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-10T18:23:08.272 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.067828669Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-10T18:23:08.272 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.069753791Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-10T18:23:08.272 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.072453699Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-10T18:23:08.273 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.075194143Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-10T18:23:08.273 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.077736866Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-10T18:23:08.273 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.080551192Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-10T18:23:08.273 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.083096349Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-10T18:23:08.273 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.085837214Z level=info msg="Executing migration" id="Drop public config table" 2024-09-10T18:23:08.273 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.08866619Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-10T18:23:08.273 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.091157317Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-10T18:23:08.273 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.093935959Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-10T18:23:08.273 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.096706642Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-10T18:23:08.273 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.099297921Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-10T18:23:08.273 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.102446659Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-10T18:23:08.273 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.105156613Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-10T18:23:08.273 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.107939818Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-10T18:23:08.273 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.110627003Z level=info msg="Executing migration" id="add share column" 2024-09-10T18:23:08.274 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.113409896Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-10T18:23:08.274 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.116093618Z level=info msg="Executing migration" id="create file table" 2024-09-10T18:23:08.274 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.118928588Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-10T18:23:08.274 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.12152652Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-10T18:23:08.274 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.124059831Z level=info msg="Executing migration" id="create file_meta table" 2024-09-10T18:23:08.274 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.124740112Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-10T18:23:08.274 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.127362104Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-10T18:23:08.274 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.129852728Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-10T18:23:08.274 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.132488737Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-10T18:23:08.274 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.134976692Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-10T18:23:08.274 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.137694739Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-10T18:23:08.274 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.14024716Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-10T18:23:08.274 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.142905935Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-10T18:23:08.274 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.145631145Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-10T18:23:08.275 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.148123899Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-10T18:23:08.275 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.150805297Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-10T18:23:08.275 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.15349685Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-10T18:23:08.275 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.156143235Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-10T18:23:08.275 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.158820026Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-10T18:23:08.275 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.161388875Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-10T18:23:08.275 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.171156498Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-10T18:23:08.275 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.17197409Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-10T18:23:08.275 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.172926711Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-10T18:23:08.275 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.176432641Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-10T18:23:08.275 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.177236217Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-10T18:23:08.275 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.178038209Z level=info msg="Executing migration" id="create folder table" 2024-09-10T18:23:08.275 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.178856243Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-10T18:23:08.275 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.179670571Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-10T18:23:08.276 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.180462492Z level=info msg="Executing migration" id="Update folder title length" 2024-09-10T18:23:08.276 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.181398373Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-10T18:23:08.276 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=migrator t=2024-09-10T18:23:08.182098711Z level=info msg="migrations completed" performed=485 skipped=0 duration=690.668459ms 2024-09-10T18:23:08.276 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=sqlstore t=2024-09-10T18:23:08.182497197Z level=info msg="Created default organization" 2024-09-10T18:23:08.276 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=plugin.loader t=2024-09-10T18:23:08.237443068Z level=info msg="Plugin registered" pluginID=input 2024-09-10T18:23:08.276 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=plugin.loader t=2024-09-10T18:23:08.265515303Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-10T18:23:08.276 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=plugin.loader t=2024-09-10T18:23:08.265548722Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-10T18:23:08.276 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=secrets t=2024-09-10T18:23:08.265592629Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-10T18:23:08.276 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=query_data t=2024-09-10T18:23:08.267586506Z level=info msg="Query Service initialization" 2024-09-10T18:23:08.361 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-09-10T18:23:06.703099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":939,"num_read_kb":796,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":578,"ondisk_log_size":578,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385956,"kb_used_data":6720,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575260,"statfs":{"total":767960285184,"available":765517066240,"internally_reserved":0,"allocated":6881280,"data_stored":4115697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.676539"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143504+0000","last_change":"2024-09-10T18:21:31.526401+0000","last_active":"2024-09-10T18:22:57.143504+0000","last_peered":"2024-09-10T18:22:57.143504+0000","last_clean":"2024-09-10T18:22:57.143504+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:22:57.143504+0000","last_undegraded":"2024-09-10T18:22:57.143504+0000","last_fullsized":"2024-09-10T18:22:57.143504+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029127+0000","last_change":"2024-09-10T18:21:33.614228+0000","last_active":"2024-09-10T18:22:57.029127+0000","last_peered":"2024-09-10T18:22:57.029127+0000","last_clean":"2024-09-10T18:22:57.029127+0000","last_became_active":"2024-09-10T18:21:33.613935+0000","last_became_peered":"2024-09-10T18:21:33.613935+0000","last_unstale":"2024-09-10T18:22:57.029127+0000","last_undegraded":"2024-09-10T18:22:57.029127+0000","last_fullsized":"2024-09-10T18:22:57.029127+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:00.978849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.096923+0000","last_change":"2024-09-10T18:21:35.619936+0000","last_active":"2024-09-10T18:22:57.096923+0000","last_peered":"2024-09-10T18:22:57.096923+0000","last_clean":"2024-09-10T18:22:57.096923+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:22:57.096923+0000","last_undegraded":"2024-09-10T18:22:57.096923+0000","last_fullsized":"2024-09-10T18:22:57.096923+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.532528+0000","last_change":"2024-09-10T18:22:57.532472+0000","last_active":"2024-09-10T18:22:57.532528+0000","last_peered":"2024-09-10T18:22:57.532528+0000","last_clean":"2024-09-10T18:22:57.532528+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:22:57.532528+0000","last_undegraded":"2024-09-10T18:22:57.532528+0000","last_fullsized":"2024-09-10T18:22:57.532528+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:57.532385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:57.532385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:25.022831+0000","scrub_duration":0.0015832260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.027788+0000","last_change":"2024-09-10T18:21:38.565483+0000","last_active":"2024-09-10T18:22:57.027788+0000","last_peered":"2024-09-10T18:22:57.027788+0000","last_clean":"2024-09-10T18:22:57.027788+0000","last_became_active":"2024-09-10T18:21:38.565118+0000","last_became_peered":"2024-09-10T18:21:38.565118+0000","last_unstale":"2024-09-10T18:22:57.027788+0000","last_undegraded":"2024-09-10T18:22:57.027788+0000","last_fullsized":"2024-09-10T18:22:57.027788+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.028541+0000","last_change":"2024-09-10T18:21:31.528038+0000","last_active":"2024-09-10T18:22:57.028541+0000","last_peered":"2024-09-10T18:22:57.028541+0000","last_clean":"2024-09-10T18:22:57.028541+0000","last_became_active":"2024-09-10T18:21:31.527582+0000","last_became_peered":"2024-09-10T18:21:31.527582+0000","last_unstale":"2024-09-10T18:22:57.028541+0000","last_undegraded":"2024-09-10T18:22:57.028541+0000","last_fullsized":"2024-09-10T18:22:57.028541+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":62,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.028016+0000","last_change":"2024-09-10T18:21:33.613829+0000","last_active":"2024-09-10T18:22:57.028016+0000","last_peered":"2024-09-10T18:22:57.028016+0000","last_clean":"2024-09-10T18:22:57.028016+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:22:57.028016+0000","last_undegraded":"2024-09-10T18:22:57.028016+0000","last_fullsized":"2024-09-10T18:22:57.028016+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":44,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-10T18:23:04.633739+0000","last_change":"2024-09-10T18:23:04.633711+0000","last_active":"2024-09-10T18:23:04.633739+0000","last_peered":"2024-09-10T18:23:04.633739+0000","last_clean":"2024-09-10T18:23:04.633739+0000","last_became_active":"2024-09-10T18:21:35.620585+0000","last_became_peered":"2024-09-10T18:21:35.620585+0000","last_unstale":"2024-09-10T18:23:04.633739+0000","last_undegraded":"2024-09-10T18:23:04.633739+0000","last_fullsized":"2024-09-10T18:23:04.633739+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:23:04.633647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:09.146682+0000","scrub_duration":0.0012299979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099491+0000","last_change":"2024-09-10T18:21:31.524898+0000","last_active":"2024-09-10T18:22:57.099491+0000","last_peered":"2024-09-10T18:22:57.099491+0000","last_clean":"2024-09-10T18:22:57.099491+0000","last_became_active":"2024-09-10T18:21:31.524620+0000","last_became_peered":"2024-09-10T18:21:31.524620+0000","last_unstale":"2024-09-10T18:22:57.099491+0000","last_undegraded":"2024-09-10T18:22:57.099491+0000","last_fullsized":"2024-09-10T18:22:57.099491+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029552+0000","last_change":"2024-09-10T18:21:33.610754+0000","last_active":"2024-09-10T18:22:57.029552+0000","last_peered":"2024-09-10T18:22:57.029552+0000","last_clean":"2024-09-10T18:22:57.029552+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:22:57.029552+0000","last_undegraded":"2024-09-10T18:22:57.029552+0000","last_fullsized":"2024-09-10T18:22:57.029552+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099176+0000","last_change":"2024-09-10T18:22:53.577905+0000","last_active":"2024-09-10T18:22:57.099176+0000","last_peered":"2024-09-10T18:22:57.099176+0000","last_clean":"2024-09-10T18:22:57.099176+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:22:57.099176+0000","last_undegraded":"2024-09-10T18:22:57.099176+0000","last_fullsized":"2024-09-10T18:22:57.099176+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:53.577826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:53.577826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:19.539271+0000","scrub_duration":0.001440577,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143080+0000","last_change":"2024-09-10T18:22:46.328322+0000","last_active":"2024-09-10T18:22:57.143080+0000","last_peered":"2024-09-10T18:22:57.143080+0000","last_clean":"2024-09-10T18:22:57.143080+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:22:57.143080+0000","last_undegraded":"2024-09-10T18:22:57.143080+0000","last_fullsized":"2024-09-10T18:22:57.143080+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:46.328262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:46.328262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:50.476851+0000","scrub_duration":0.0024608049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099496+0000","last_change":"2024-09-10T18:22:56.588562+0000","last_active":"2024-09-10T18:22:57.099496+0000","last_peered":"2024-09-10T18:22:57.099496+0000","last_clean":"2024-09-10T18:22:57.099496+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:22:57.099496+0000","last_undegraded":"2024-09-10T18:22:57.099496+0000","last_fullsized":"2024-09-10T18:22:57.099496+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:56.588510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:56.588510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:07.012424+0000","scrub_duration":0.0011336320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"85'7","reported_seq":58,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029493+0000","last_change":"2024-09-10T18:22:44.365273+0000","last_active":"2024-09-10T18:22:57.029493+0000","last_peered":"2024-09-10T18:22:57.029493+0000","last_clean":"2024-09-10T18:22:57.029493+0000","last_became_active":"2024-09-10T18:21:33.615429+0000","last_became_peered":"2024-09-10T18:21:33.615429+0000","last_unstale":"2024-09-10T18:22:57.029493+0000","last_undegraded":"2024-09-10T18:22:57.029493+0000","last_fullsized":"2024-09-10T18:22:57.029493+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'7","last_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:22:44.365196+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:58.822595+0000","scrub_duration":0.0033059909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.144395+0000","last_change":"2024-09-10T18:21:35.624352+0000","last_active":"2024-09-10T18:22:57.144395+0000","last_peered":"2024-09-10T18:22:57.144395+0000","last_clean":"2024-09-10T18:22:57.144395+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:22:57.144395+0000","last_undegraded":"2024-09-10T18:22:57.144395+0000","last_fullsized":"2024-09-10T18:22:57.144395+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099034+0000","last_change":"2024-09-10T18:21:37.639437+0000","last_active":"2024-09-10T18:22:57.099034+0000","last_peered":"2024-09-10T18:22:57.099034+0000","last_clean":"2024-09-10T18:22:57.099034+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:22:57.099034+0000","last_undegraded":"2024-09-10T18:22:57.099034+0000","last_fullsized":"2024-09-10T18:22:57.099034+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029965+0000","last_change":"2024-09-10T18:21:38.563964+0000","last_active":"2024-09-10T18:22:57.029965+0000","last_peered":"2024-09-10T18:22:57.029965+0000","last_clean":"2024-09-10T18:22:57.029965+0000","last_became_active":"2024-09-10T18:21:38.563570+0000","last_became_peered":"2024-09-10T18:21:38.563570+0000","last_unstale":"2024-09-10T18:22:57.029965+0000","last_undegraded":"2024-09-10T18:22:57.029965+0000","last_fullsized":"2024-09-10T18:22:57.029965+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143518+0000","last_change":"2024-09-10T18:21:31.529843+0000","last_active":"2024-09-10T18:22:57.143518+0000","last_peered":"2024-09-10T18:22:57.143518+0000","last_clean":"2024-09-10T18:22:57.143518+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:22:57.143518+0000","last_undegraded":"2024-09-10T18:22:57.143518+0000","last_fullsized":"2024-09-10T18:22:57.143518+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":71,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-10T18:23:02.499455+0000","last_change":"2024-09-10T18:23:02.499426+0000","last_active":"2024-09-10T18:23:02.499455+0000","last_peered":"2024-09-10T18:23:02.499455+0000","last_clean":"2024-09-10T18:23:02.499455+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:23:02.499455+0000","last_undegraded":"2024-09-10T18:23:02.499455+0000","last_fullsized":"2024-09-10T18:23:02.499455+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-10T18:23:02.499365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:23:02.499365+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:06.760263+0000","scrub_duration":0.001894258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.097532+0000","last_change":"2024-09-10T18:22:39.007672+0000","last_active":"2024-09-10T18:22:57.097532+0000","last_peered":"2024-09-10T18:22:57.097532+0000","last_clean":"2024-09-10T18:22:57.097532+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:22:57.097532+0000","last_undegraded":"2024-09-10T18:22:57.097532+0000","last_fullsized":"2024-09-10T18:22:57.097532+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:39.007614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:39.007614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:59.162407+0000","scrub_duration":0.0021069719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-10T18:23:04.413254+0000","last_change":"2024-09-10T18:23:04.413228+0000","last_active":"2024-09-10T18:23:04.413254+0000","last_peered":"2024-09-10T18:23:04.413254+0000","last_clean":"2024-09-10T18:23:04.413254+0000","last_became_active":"2024-09-10T18:21:38.564681+0000","last_became_peered":"2024-09-10T18:21:38.564681+0000","last_unstale":"2024-09-10T18:23:04.413254+0000","last_undegraded":"2024-09-10T18:23:04.413254+0000","last_fullsized":"2024-09-10T18:23:04.413254+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:04.413172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:23:04.413172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:17.019734+0000","scrub_duration":0.001699489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029667+0000","last_change":"2024-09-10T18:21:31.528392+0000","last_active":"2024-09-10T18:22:57.029667+0000","last_peered":"2024-09-10T18:22:57.029667+0000","last_clean":"2024-09-10T18:22:57.029667+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:22:57.029667+0000","last_undegraded":"2024-09-10T18:22:57.029667+0000","last_fullsized":"2024-09-10T18:22:57.029667+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"88'17","reported_seq":75,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-10T18:23:02.323823+0000","last_change":"2024-09-10T18:23:02.323792+0000","last_active":"2024-09-10T18:23:02.323823+0000","last_peered":"2024-09-10T18:23:02.323823+0000","last_clean":"2024-09-10T18:23:02.323823+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:23:02.323823+0000","last_undegraded":"2024-09-10T18:23:02.323823+0000","last_fullsized":"2024-09-10T18:23:02.323823+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:23:02.323736+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:09.338427+0000","scrub_duration":0.0041732410000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029475+0000","last_change":"2024-09-10T18:22:50.436357+0000","last_active":"2024-09-10T18:22:57.029475+0000","last_peered":"2024-09-10T18:22:57.029475+0000","last_clean":"2024-09-10T18:22:57.029475+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:22:57.029475+0000","last_undegraded":"2024-09-10T18:22:57.029475+0000","last_fullsized":"2024-09-10T18:22:57.029475+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:50.436294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:50.436294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:07.783088+0000","scrub_duration":0.001715187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.144212+0000","last_change":"2024-09-10T18:21:37.646675+0000","last_active":"2024-09-10T18:22:57.144212+0000","last_peered":"2024-09-10T18:22:57.144212+0000","last_clean":"2024-09-10T18:22:57.144212+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:22:57.144212+0000","last_undegraded":"2024-09-10T18:22:57.144212+0000","last_fullsized":"2024-09-10T18:22:57.144212+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.097056+0000","last_change":"2024-09-10T18:22:46.141545+0000","last_active":"2024-09-10T18:22:57.097056+0000","last_peered":"2024-09-10T18:22:57.097056+0000","last_clean":"2024-09-10T18:22:57.097056+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:22:57.097056+0000","last_undegraded":"2024-09-10T18:22:57.097056+0000","last_fullsized":"2024-09-10T18:22:57.097056+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:46.141408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:46.141408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:12.563610+0000","scrub_duration":0.0022977269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"86'15","reported_seq":66,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143429+0000","last_change":"2024-09-10T18:22:49.298511+0000","last_active":"2024-09-10T18:22:57.143429+0000","last_peered":"2024-09-10T18:22:57.143429+0000","last_clean":"2024-09-10T18:22:57.143429+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:22:57.143429+0000","last_undegraded":"2024-09-10T18:22:57.143429+0000","last_fullsized":"2024-09-10T18:22:57.143429+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:22:49.298455+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:10.207495+0000","scrub_duration":0.0038705900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:59.769656+0000","last_change":"2024-09-10T18:22:48.795234+0000","last_active":"2024-09-10T18:22:59.769656+0000","last_peered":"2024-09-10T18:22:59.769656+0000","last_clean":"2024-09-10T18:22:59.769656+0000","last_became_active":"2024-09-10T18:21:35.624964+0000","last_became_peered":"2024-09-10T18:21:35.624964+0000","last_unstale":"2024-09-10T18:22:59.769656+0000","last_undegraded":"2024-09-10T18:22:59.769656+0000","last_fullsized":"2024-09-10T18:22:59.769656+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:22:48.795158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:48.795158+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:11.053077+0000","scrub_duration":0.0017552570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"83'22","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.097817+0000","last_change":"2024-09-10T18:22:40.007784+0000","last_active":"2024-09-10T18:22:57.097817+0000","last_peered":"2024-09-10T18:22:57.097817+0000","last_clean":"2024-09-10T18:22:57.097817+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:22:57.097817+0000","last_undegraded":"2024-09-10T18:22:57.097817+0000","last_fullsized":"2024-09-10T18:22:57.097817+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:22:40.007685+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:46.637635+0000","scrub_duration":0.0077173049999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-10T18:23:01.321273+0000","last_change":"2024-09-10T18:23:01.321217+0000","last_active":"2024-09-10T18:23:01.321273+0000","last_peered":"2024-09-10T18:23:01.321273+0000","last_clean":"2024-09-10T18:23:01.321273+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:23:01.321273+0000","last_undegraded":"2024-09-10T18:23:01.321273+0000","last_fullsized":"2024-09-10T18:23:01.321273+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:01.321141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:23:01.321141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:06.428741+0000","scrub_duration":0.0015244830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:59.769676+0000","last_change":"2024-09-10T18:21:35.624518+0000","last_active":"2024-09-10T18:22:59.769676+0000","last_peered":"2024-09-10T18:22:59.769676+0000","last_clean":"2024-09-10T18:22:59.769676+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:22:59.769676+0000","last_undegraded":"2024-09-10T18:22:59.769676+0000","last_fullsized":"2024-09-10T18:22:59.769676+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099151+0000","last_change":"2024-09-10T18:21:37.644088+0000","last_active":"2024-09-10T18:22:57.099151+0000","last_peered":"2024-09-10T18:22:57.099151+0000","last_clean":"2024-09-10T18:22:57.099151+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:22:57.099151+0000","last_undegraded":"2024-09-10T18:22:57.099151+0000","last_fullsized":"2024-09-10T18:22:57.099151+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":69,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.027733+0000","last_change":"2024-09-10T18:21:34.047843+0000","last_active":"2024-09-10T18:22:57.027733+0000","last_peered":"2024-09-10T18:22:57.027733+0000","last_clean":"2024-09-10T18:22:57.027733+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:22:57.027733+0000","last_undegraded":"2024-09-10T18:22:57.027733+0000","last_fullsized":"2024-09-10T18:22:57.027733+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.028657+0000","last_change":"2024-09-10T18:22:56.356952+0000","last_active":"2024-09-10T18:22:57.028657+0000","last_peered":"2024-09-10T18:22:57.028657+0000","last_clean":"2024-09-10T18:22:57.028657+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:22:57.028657+0000","last_undegraded":"2024-09-10T18:22:57.028657+0000","last_fullsized":"2024-09-10T18:22:57.028657+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:56.356885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:56.356885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:15.154888+0000","scrub_duration":0.0016221829999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029215+0000","last_change":"2024-09-10T18:22:47.386371+0000","last_active":"2024-09-10T18:22:57.029215+0000","last_peered":"2024-09-10T18:22:57.029215+0000","last_clean":"2024-09-10T18:22:57.029215+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:22:57.029215+0000","last_undegraded":"2024-09-10T18:22:57.029215+0000","last_fullsized":"2024-09-10T18:22:57.029215+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:47.386303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:47.386303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:01.335164+0000","scrub_duration":0.001725484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029479+0000","last_change":"2024-09-10T18:21:37.642918+0000","last_active":"2024-09-10T18:22:57.029479+0000","last_peered":"2024-09-10T18:22:57.029479+0000","last_clean":"2024-09-10T18:22:57.029479+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:22:57.029479+0000","last_undegraded":"2024-09-10T18:22:57.029479+0000","last_fullsized":"2024-09-10T18:22:57.029479+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029339+0000","last_change":"2024-09-10T18:22:37.322150+0000","last_active":"2024-09-10T18:22:57.029339+0000","last_peered":"2024-09-10T18:22:57.029339+0000","last_clean":"2024-09-10T18:22:57.029339+0000","last_became_active":"2024-09-10T18:21:33.607574+0000","last_became_peered":"2024-09-10T18:21:33.607574+0000","last_unstale":"2024-09-10T18:22:57.029339+0000","last_undegraded":"2024-09-10T18:22:57.029339+0000","last_fullsized":"2024-09-10T18:22:57.029339+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-10T18:22:37.322094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:37.322094+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:41.781070+0000","scrub_duration":0.001716817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143607+0000","last_change":"2024-09-10T18:21:31.531508+0000","last_active":"2024-09-10T18:22:57.143607+0000","last_peered":"2024-09-10T18:22:57.143607+0000","last_clean":"2024-09-10T18:22:57.143607+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:22:57.143607+0000","last_undegraded":"2024-09-10T18:22:57.143607+0000","last_fullsized":"2024-09-10T18:22:57.143607+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:59.769544+0000","last_change":"2024-09-10T18:21:35.623304+0000","last_active":"2024-09-10T18:22:59.769544+0000","last_peered":"2024-09-10T18:22:59.769544+0000","last_clean":"2024-09-10T18:22:59.769544+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:22:59.769544+0000","last_undegraded":"2024-09-10T18:22:59.769544+0000","last_fullsized":"2024-09-10T18:22:59.769544+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-10T18:23:02.011695+0000","last_change":"2024-09-10T18:23:02.011664+0000","last_active":"2024-09-10T18:23:02.011695+0000","last_peered":"2024-09-10T18:23:02.011695+0000","last_clean":"2024-09-10T18:23:02.011695+0000","last_became_active":"2024-09-10T18:21:38.565165+0000","last_became_peered":"2024-09-10T18:21:38.565165+0000","last_unstale":"2024-09-10T18:23:02.011695+0000","last_undegraded":"2024-09-10T18:23:02.011695+0000","last_fullsized":"2024-09-10T18:23:02.011695+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:02.011610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:02.011610+0000","last_clean_scrub_stamp":"2024-09-10T18:23:02.011610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:15.053175+0000","scrub_duration":0.0017054329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.098157+0000","last_change":"2024-09-10T18:22:57.098114+0000","last_active":"2024-09-10T18:22:57.098157+0000","last_peered":"2024-09-10T18:22:57.098157+0000","last_clean":"2024-09-10T18:22:57.098157+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:22:57.098157+0000","last_undegraded":"2024-09-10T18:22:57.098157+0000","last_fullsized":"2024-09-10T18:22:57.098157+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-10T18:22:57.098059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:57.098059+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:07.299393+0000","scrub_duration":0.0025842040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099580+0000","last_change":"2024-09-10T18:21:31.534170+0000","last_active":"2024-09-10T18:22:57.099580+0000","last_peered":"2024-09-10T18:22:57.099580+0000","last_clean":"2024-09-10T18:22:57.099580+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:22:57.099580+0000","last_undegraded":"2024-09-10T18:22:57.099580+0000","last_fullsized":"2024-09-10T18:22:57.099580+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:59.769529+0000","last_change":"2024-09-10T18:21:35.618920+0000","last_active":"2024-09-10T18:22:59.769529+0000","last_peered":"2024-09-10T18:22:59.769529+0000","last_clean":"2024-09-10T18:22:59.769529+0000","last_became_active":"2024-09-10T18:21:35.617821+0000","last_became_peered":"2024-09-10T18:21:35.617821+0000","last_unstale":"2024-09-10T18:22:59.769529+0000","last_undegraded":"2024-09-10T18:22:59.769529+0000","last_fullsized":"2024-09-10T18:22:59.769529+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143182+0000","last_change":"2024-09-10T18:22:41.399132+0000","last_active":"2024-09-10T18:22:57.143182+0000","last_peered":"2024-09-10T18:22:57.143182+0000","last_clean":"2024-09-10T18:22:57.143182+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:22:57.143182+0000","last_undegraded":"2024-09-10T18:22:57.143182+0000","last_fullsized":"2024-09-10T18:22:57.143182+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:41.399078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:41.399078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:48.673347+0000","scrub_duration":0.001245323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143300+0000","last_change":"2024-09-10T18:21:34.046468+0000","last_active":"2024-09-10T18:22:57.143300+0000","last_peered":"2024-09-10T18:22:57.143300+0000","last_clean":"2024-09-10T18:22:57.143300+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:22:57.143300+0000","last_undegraded":"2024-09-10T18:22:57.143300+0000","last_fullsized":"2024-09-10T18:22:57.143300+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.097449+0000","last_change":"2024-09-10T18:21:31.530156+0000","last_active":"2024-09-10T18:22:57.097449+0000","last_peered":"2024-09-10T18:22:57.097449+0000","last_clean":"2024-09-10T18:22:57.097449+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:22:57.097449+0000","last_undegraded":"2024-09-10T18:22:57.097449+0000","last_fullsized":"2024-09-10T18:22:57.097449+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143474+0000","last_change":"2024-09-10T18:22:38.853925+0000","last_active":"2024-09-10T18:22:57.143474+0000","last_peered":"2024-09-10T18:22:57.143474+0000","last_clean":"2024-09-10T18:22:57.143474+0000","last_became_active":"2024-09-10T18:21:35.617132+0000","last_became_peered":"2024-09-10T18:21:35.617132+0000","last_unstale":"2024-09-10T18:22:57.143474+0000","last_undegraded":"2024-09-10T18:22:57.143474+0000","last_fullsized":"2024-09-10T18:22:57.143474+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:38.853839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:38.853839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:39.087399+0000","scrub_duration":0.0031500500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.096873+0000","last_change":"2024-09-10T18:21:38.564042+0000","last_active":"2024-09-10T18:22:57.096873+0000","last_peered":"2024-09-10T18:22:57.096873+0000","last_clean":"2024-09-10T18:22:57.096873+0000","last_became_active":"2024-09-10T18:21:38.563629+0000","last_became_peered":"2024-09-10T18:21:38.563629+0000","last_unstale":"2024-09-10T18:22:57.096873+0000","last_undegraded":"2024-09-10T18:22:57.096873+0000","last_fullsized":"2024-09-10T18:22:57.096873+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:04.135213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":111,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-10T18:23:03.940479+0000","last_change":"2024-09-10T18:23:03.940452+0000","last_active":"2024-09-10T18:23:03.940479+0000","last_peered":"2024-09-10T18:23:03.940479+0000","last_clean":"2024-09-10T18:23:03.940479+0000","last_became_active":"2024-09-10T18:21:33.618616+0000","last_became_peered":"2024-09-10T18:21:33.618616+0000","last_unstale":"2024-09-10T18:23:03.940479+0000","last_undegraded":"2024-09-10T18:23:03.940479+0000","last_fullsized":"2024-09-10T18:23:03.940479+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-09-10T18:23:03.940389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:23:03.940389+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:16.735312+0000","scrub_duration":0.0020414500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029848+0000","last_change":"2024-09-10T18:22:42.335215+0000","last_active":"2024-09-10T18:22:57.029848+0000","last_peered":"2024-09-10T18:22:57.029848+0000","last_clean":"2024-09-10T18:22:57.029848+0000","last_became_active":"2024-09-10T18:21:31.526161+0000","last_became_peered":"2024-09-10T18:21:31.526161+0000","last_unstale":"2024-09-10T18:22:57.029848+0000","last_undegraded":"2024-09-10T18:22:57.029848+0000","last_fullsized":"2024-09-10T18:22:57.029848+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:42.335152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:42.335152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:02.796505+0000","scrub_duration":0.0013614720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"62'2","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099444+0000","last_change":"2024-09-10T18:21:35.613488+0000","last_active":"2024-09-10T18:22:57.099444+0000","last_peered":"2024-09-10T18:22:57.099444+0000","last_clean":"2024-09-10T18:22:57.099444+0000","last_became_active":"2024-09-10T18:21:31.528970+0000","last_became_peered":"2024-09-10T18:21:31.528970+0000","last_unstale":"2024-09-10T18:22:57.099444+0000","last_undegraded":"2024-09-10T18:22:57.099444+0000","last_fullsized":"2024-09-10T18:22:57.099444+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143376+0000","last_change":"2024-09-10T18:22:43.309177+0000","last_active":"2024-09-10T18:22:57.143376+0000","last_peered":"2024-09-10T18:22:57.143376+0000","last_clean":"2024-09-10T18:22:57.143376+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:22:57.143376+0000","last_undegraded":"2024-09-10T18:22:57.143376+0000","last_fullsized":"2024-09-10T18:22:57.143376+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:43.309115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:43.309115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:08.721305+0000","scrub_duration":0.0015388929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.028163+0000","last_change":"2024-09-10T18:22:55.388615+0000","last_active":"2024-09-10T18:22:57.028163+0000","last_peered":"2024-09-10T18:22:57.028163+0000","last_clean":"2024-09-10T18:22:57.028163+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:22:57.028163+0000","last_undegraded":"2024-09-10T18:22:57.028163+0000","last_fullsized":"2024-09-10T18:22:57.028163+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:55.388564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:55.388564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:13.516819+0000","scrub_duration":0.001337956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":62,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.097806+0000","last_change":"2024-09-10T18:21:33.618163+0000","last_active":"2024-09-10T18:22:57.097806+0000","last_peered":"2024-09-10T18:22:57.097806+0000","last_clean":"2024-09-10T18:22:57.097806+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:22:57.097806+0000","last_undegraded":"2024-09-10T18:22:57.097806+0000","last_fullsized":"2024-09-10T18:22:57.097806+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":47,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.097160+0000","last_change":"2024-09-10T18:22:52.160807+0000","last_active":"2024-09-10T18:22:57.097160+0000","last_peered":"2024-09-10T18:22:57.097160+0000","last_clean":"2024-09-10T18:22:57.097160+0000","last_became_active":"2024-09-10T18:21:31.532624+0000","last_became_peered":"2024-09-10T18:21:31.532624+0000","last_unstale":"2024-09-10T18:22:57.097160+0000","last_undegraded":"2024-09-10T18:22:57.097160+0000","last_fullsized":"2024-09-10T18:22:57.097160+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:52.160732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:52.160732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:11.270210+0000","scrub_duration":0.0015264289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":54,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.142830+0000","last_change":"2024-09-10T18:22:46.337257+0000","last_active":"2024-09-10T18:22:57.142830+0000","last_peered":"2024-09-10T18:22:57.142830+0000","last_clean":"2024-09-10T18:22:57.142830+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:22:57.142830+0000","last_undegraded":"2024-09-10T18:22:57.142830+0000","last_fullsized":"2024-09-10T18:22:57.142830+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:22:46.337186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:46.337186+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:11.152611+0000","scrub_duration":0.0015240410000000001,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143821+0000","last_change":"2024-09-10T18:22:53.488490+0000","last_active":"2024-09-10T18:22:57.143821+0000","last_peered":"2024-09-10T18:22:57.143821+0000","last_clean":"2024-09-10T18:22:57.143821+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:22:57.143821+0000","last_undegraded":"2024-09-10T18:22:57.143821+0000","last_fullsized":"2024-09-10T18:22:57.143821+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:22:53.488434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:15.445322+0000","scrub_duration":0.001619882,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.144533+0000","last_change":"2024-09-10T18:21:38.565470+0000","last_active":"2024-09-10T18:22:57.144533+0000","last_peered":"2024-09-10T18:22:57.144533+0000","last_clean":"2024-09-10T18:22:57.144533+0000","last_became_active":"2024-09-10T18:21:38.565183+0000","last_became_peered":"2024-09-10T18:21:38.565183+0000","last_unstale":"2024-09-10T18:22:57.144533+0000","last_undegraded":"2024-09-10T18:22:57.144533+0000","last_fullsized":"2024-09-10T18:22:57.144533+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:04.594058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143506+0000","last_change":"2024-09-10T18:22:51.280276+0000","last_active":"2024-09-10T18:22:57.143506+0000","last_peered":"2024-09-10T18:22:57.143506+0000","last_clean":"2024-09-10T18:22:57.143506+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:22:57.143506+0000","last_undegraded":"2024-09-10T18:22:57.143506+0000","last_fullsized":"2024-09-10T18:22:57.143506+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:22:51.280219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:51.280219+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:13.028302+0000","scrub_duration":0.0021524019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143200+0000","last_change":"2024-09-10T18:22:55.325500+0000","last_active":"2024-09-10T18:22:57.143200+0000","last_peered":"2024-09-10T18:22:57.143200+0000","last_clean":"2024-09-10T18:22:57.143200+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:22:57.143200+0000","last_undegraded":"2024-09-10T18:22:57.143200+0000","last_fullsized":"2024-09-10T18:22:57.143200+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:55.325446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:55.325446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:08.340184+0000","scrub_duration":0.0014498149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":142,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-10T18:23:02.930747+0000","last_change":"2024-09-10T18:22:41.256321+0000","last_active":"2024-09-10T18:23:02.930747+0000","last_peered":"2024-09-10T18:23:02.930747+0000","last_clean":"2024-09-10T18:23:02.930747+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:23:02.930747+0000","last_undegraded":"2024-09-10T18:23:02.930747+0000","last_fullsized":"2024-09-10T18:23:02.930747+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:22:41.256255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:41.256255+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:01.661114+0000","scrub_duration":0.0015771470000000001,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":86,"num_read_kb":81,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099771+0000","last_change":"2024-09-10T18:21:35.619757+0000","last_active":"2024-09-10T18:22:57.099771+0000","last_peered":"2024-09-10T18:22:57.099771+0000","last_clean":"2024-09-10T18:22:57.099771+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:22:57.099771+0000","last_undegraded":"2024-09-10T18:22:57.099771+0000","last_fullsized":"2024-09-10T18:22:57.099771+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.097107+0000","last_change":"2024-09-10T18:21:37.639920+0000","last_active":"2024-09-10T18:22:57.097107+0000","last_peered":"2024-09-10T18:22:57.097107+0000","last_clean":"2024-09-10T18:22:57.097107+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:22:57.097107+0000","last_undegraded":"2024-09-10T18:22:57.097107+0000","last_fullsized":"2024-09-10T18:22:57.097107+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:04.764182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":58,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.144019+0000","last_change":"2024-09-10T18:21:34.046018+0000","last_active":"2024-09-10T18:22:57.144019+0000","last_peered":"2024-09-10T18:22:57.144019+0000","last_clean":"2024-09-10T18:22:57.144019+0000","last_became_active":"2024-09-10T18:21:34.045549+0000","last_became_peered":"2024-09-10T18:21:34.045549+0000","last_unstale":"2024-09-10T18:22:57.144019+0000","last_undegraded":"2024-09-10T18:22:57.144019+0000","last_fullsized":"2024-09-10T18:22:57.144019+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.028200+0000","last_change":"2024-09-10T18:22:45.383833+0000","last_active":"2024-09-10T18:22:57.028200+0000","last_peered":"2024-09-10T18:22:57.028200+0000","last_clean":"2024-09-10T18:22:57.028200+0000","last_became_active":"2024-09-10T18:21:31.529816+0000","last_became_peered":"2024-09-10T18:21:31.529816+0000","last_unstale":"2024-09-10T18:22:57.028200+0000","last_undegraded":"2024-09-10T18:22:57.028200+0000","last_fullsized":"2024-09-10T18:22:57.028200+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:45.383768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:45.383768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:06.766629+0000","scrub_duration":0.001512844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143579+0000","last_change":"2024-09-10T18:22:17.024622+0000","last_active":"2024-09-10T18:22:57.143579+0000","last_peered":"2024-09-10T18:22:57.143579+0000","last_clean":"2024-09-10T18:22:57.143579+0000","last_became_active":"2024-09-10T18:21:20.570983+0000","last_became_peered":"2024-09-10T18:21:20.570983+0000","last_unstale":"2024-09-10T18:22:57.143579+0000","last_undegraded":"2024-09-10T18:22:57.143579+0000","last_fullsized":"2024-09-10T18:22:57.143579+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:22:17.024553+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:21.328129+0000","scrub_duration":0.003148245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.144355+0000","last_change":"2024-09-10T18:21:35.621370+0000","last_active":"2024-09-10T18:22:57.144355+0000","last_peered":"2024-09-10T18:22:57.144355+0000","last_clean":"2024-09-10T18:22:57.144355+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:22:57.144355+0000","last_undegraded":"2024-09-10T18:22:57.144355+0000","last_fullsized":"2024-09-10T18:22:57.144355+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099717+0000","last_change":"2024-09-10T18:21:37.638814+0000","last_active":"2024-09-10T18:22:57.099717+0000","last_peered":"2024-09-10T18:22:57.099717+0000","last_clean":"2024-09-10T18:22:57.099717+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:22:57.099717+0000","last_undegraded":"2024-09-10T18:22:57.099717+0000","last_fullsized":"2024-09-10T18:22:57.099717+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:06.014250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.030046+0000","last_change":"2024-09-10T18:21:33.614556+0000","last_active":"2024-09-10T18:22:57.030046+0000","last_peered":"2024-09-10T18:22:57.030046+0000","last_clean":"2024-09-10T18:22:57.030046+0000","last_became_active":"2024-09-10T18:21:33.614090+0000","last_became_peered":"2024-09-10T18:21:33.614090+0000","last_unstale":"2024-09-10T18:22:57.030046+0000","last_undegraded":"2024-09-10T18:22:57.030046+0000","last_fullsized":"2024-09-10T18:22:57.030046+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029508+0000","last_change":"2024-09-10T18:21:31.529705+0000","last_active":"2024-09-10T18:22:57.029508+0000","last_peered":"2024-09-10T18:22:57.029508+0000","last_clean":"2024-09-10T18:22:57.029508+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:22:57.029508+0000","last_undegraded":"2024-09-10T18:22:57.029508+0000","last_fullsized":"2024-09-10T18:22:57.029508+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099644+0000","last_change":"2024-09-10T18:21:35.620645+0000","last_active":"2024-09-10T18:22:57.099644+0000","last_peered":"2024-09-10T18:22:57.099644+0000","last_clean":"2024-09-10T18:22:57.099644+0000","last_became_active":"2024-09-10T18:21:35.620331+0000","last_became_peered":"2024-09-10T18:21:35.620331+0000","last_unstale":"2024-09-10T18:22:57.099644+0000","last_undegraded":"2024-09-10T18:22:57.099644+0000","last_fullsized":"2024-09-10T18:22:57.099644+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.028016+0000","last_change":"2024-09-10T18:22:50.298390+0000","last_active":"2024-09-10T18:22:57.028016+0000","last_peered":"2024-09-10T18:22:57.028016+0000","last_clean":"2024-09-10T18:22:57.028016+0000","last_became_active":"2024-09-10T18:21:38.563450+0000","last_became_peered":"2024-09-10T18:21:38.563450+0000","last_unstale":"2024-09-10T18:22:57.028016+0000","last_undegraded":"2024-09-10T18:22:57.028016+0000","last_fullsized":"2024-09-10T18:22:57.028016+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:50.298320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:50.298320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:19.836783+0000","scrub_duration":0.0021610420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029981+0000","last_change":"2024-09-10T18:21:33.611478+0000","last_active":"2024-09-10T18:22:57.029981+0000","last_peered":"2024-09-10T18:22:57.029981+0000","last_clean":"2024-09-10T18:22:57.029981+0000","last_became_active":"2024-09-10T18:21:33.611203+0000","last_became_peered":"2024-09-10T18:21:33.611203+0000","last_unstale":"2024-09-10T18:22:57.029981+0000","last_undegraded":"2024-09-10T18:22:57.029981+0000","last_fullsized":"2024-09-10T18:22:57.029981+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"58'1","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143989+0000","last_change":"2024-09-10T18:21:31.528999+0000","last_active":"2024-09-10T18:22:57.143989+0000","last_peered":"2024-09-10T18:22:57.143989+0000","last_clean":"2024-09-10T18:22:57.143989+0000","last_became_active":"2024-09-10T18:21:31.528748+0000","last_became_peered":"2024-09-10T18:21:31.528748+0000","last_unstale":"2024-09-10T18:22:57.143989+0000","last_undegraded":"2024-09-10T18:22:57.143989+0000","last_fullsized":"2024-09-10T18:22:57.143989+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.097346+0000","last_change":"2024-09-10T18:21:35.625340+0000","last_active":"2024-09-10T18:22:57.097346+0000","last_peered":"2024-09-10T18:22:57.097346+0000","last_clean":"2024-09-10T18:22:57.097346+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:22:57.097346+0000","last_undegraded":"2024-09-10T18:22:57.097346+0000","last_fullsized":"2024-09-10T18:22:57.097346+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143895+0000","last_change":"2024-09-10T18:22:52.486807+0000","last_active":"2024-09-10T18:22:57.143895+0000","last_peered":"2024-09-10T18:22:57.143895+0000","last_clean":"2024-09-10T18:22:57.143895+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:22:57.143895+0000","last_undegraded":"2024-09-10T18:22:57.143895+0000","last_fullsized":"2024-09-10T18:22:57.143895+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:52.486737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:52.486737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:59.120450+0000","scrub_duration":0.0015615,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"65'10","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029682+0000","last_change":"2024-09-10T18:21:33.614985+0000","last_active":"2024-09-10T18:22:57.029682+0000","last_peered":"2024-09-10T18:22:57.029682+0000","last_clean":"2024-09-10T18:22:57.029682+0000","last_became_active":"2024-09-10T18:21:33.614689+0000","last_became_peered":"2024-09-10T18:21:33.614689+0000","last_unstale":"2024-09-10T18:22:57.029682+0000","last_undegraded":"2024-09-10T18:22:57.029682+0000","last_fullsized":"2024-09-10T18:22:57.029682+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.028222+0000","last_change":"2024-09-10T18:21:31.535458+0000","last_active":"2024-09-10T18:22:57.028222+0000","last_peered":"2024-09-10T18:22:57.028222+0000","last_clean":"2024-09-10T18:22:57.028222+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:22:57.028222+0000","last_undegraded":"2024-09-10T18:22:57.028222+0000","last_fullsized":"2024-09-10T18:22:57.028222+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:59.769511+0000","last_change":"2024-09-10T18:22:51.492683+0000","last_active":"2024-09-10T18:22:59.769511+0000","last_peered":"2024-09-10T18:22:59.769511+0000","last_clean":"2024-09-10T18:22:59.769511+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:22:59.769511+0000","last_undegraded":"2024-09-10T18:22:59.769511+0000","last_fullsized":"2024-09-10T18:22:59.769511+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:22:51.492580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:51.492580+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:00.192242+0000","scrub_duration":0.002035987,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029768+0000","last_change":"2024-09-10T18:21:37.642233+0000","last_active":"2024-09-10T18:22:57.029768+0000","last_peered":"2024-09-10T18:22:57.029768+0000","last_clean":"2024-09-10T18:22:57.029768+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:22:57.029768+0000","last_undegraded":"2024-09-10T18:22:57.029768+0000","last_fullsized":"2024-09-10T18:22:57.029768+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143202+0000","last_change":"2024-09-10T18:21:33.614532+0000","last_active":"2024-09-10T18:22:57.143202+0000","last_peered":"2024-09-10T18:22:57.143202+0000","last_clean":"2024-09-10T18:22:57.143202+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:22:57.143202+0000","last_undegraded":"2024-09-10T18:22:57.143202+0000","last_fullsized":"2024-09-10T18:22:57.143202+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"58'1","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-10T18:23:04.541328+0000","last_change":"2024-09-10T18:23:04.541297+0000","last_active":"2024-09-10T18:23:04.541328+0000","last_peered":"2024-09-10T18:23:04.541328+0000","last_clean":"2024-09-10T18:23:04.541328+0000","last_became_active":"2024-09-10T18:21:31.534441+0000","last_became_peered":"2024-09-10T18:21:31.534441+0000","last_unstale":"2024-09-10T18:23:04.541328+0000","last_undegraded":"2024-09-10T18:23:04.541328+0000","last_fullsized":"2024-09-10T18:23:04.541328+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-10T18:23:04.541238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:23:04.541238+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:18.491111+0000","scrub_duration":0.0017464519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.142778+0000","last_change":"2024-09-10T18:22:41.279395+0000","last_active":"2024-09-10T18:22:57.142778+0000","last_peered":"2024-09-10T18:22:57.142778+0000","last_clean":"2024-09-10T18:22:57.142778+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:22:57.142778+0000","last_undegraded":"2024-09-10T18:22:57.142778+0000","last_fullsized":"2024-09-10T18:22:57.142778+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:41.279342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:41.279342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:48.532838+0000","scrub_duration":0.0017787790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143228+0000","last_change":"2024-09-10T18:21:37.641253+0000","last_active":"2024-09-10T18:22:57.143228+0000","last_peered":"2024-09-10T18:22:57.143228+0000","last_clean":"2024-09-10T18:22:57.143228+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:22:57.143228+0000","last_undegraded":"2024-09-10T18:22:57.143228+0000","last_fullsized":"2024-09-10T18:22:57.143228+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"86'28","reported_seq":86,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143556+0000","last_change":"2024-09-10T18:22:54.863374+0000","last_active":"2024-09-10T18:22:57.143556+0000","last_peered":"2024-09-10T18:22:57.143556+0000","last_clean":"2024-09-10T18:22:57.143556+0000","last_became_active":"2024-09-10T18:21:34.046453+0000","last_became_peered":"2024-09-10T18:21:34.046453+0000","last_unstale":"2024-09-10T18:22:57.143556+0000","last_undegraded":"2024-09-10T18:22:57.143556+0000","last_fullsized":"2024-09-10T18:22:57.143556+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'28","last_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:22:54.863323+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:20.254372+0000","scrub_duration":0.0053712059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.144699+0000","last_change":"2024-09-10T18:22:32.137670+0000","last_active":"2024-09-10T18:22:57.144699+0000","last_peered":"2024-09-10T18:22:57.144699+0000","last_clean":"2024-09-10T18:22:57.144699+0000","last_became_active":"2024-09-10T18:21:31.531951+0000","last_became_peered":"2024-09-10T18:21:31.531951+0000","last_unstale":"2024-09-10T18:22:57.144699+0000","last_undegraded":"2024-09-10T18:22:57.144699+0000","last_fullsized":"2024-09-10T18:22:57.144699+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:32.137600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:32.137600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:41.301280+0000","scrub_duration":0.00151761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.142758+0000","last_change":"2024-09-10T18:22:42.312458+0000","last_active":"2024-09-10T18:22:57.142758+0000","last_peered":"2024-09-10T18:22:57.142758+0000","last_clean":"2024-09-10T18:22:57.142758+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:22:57.142758+0000","last_undegraded":"2024-09-10T18:22:57.142758+0000","last_fullsized":"2024-09-10T18:22:57.142758+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:42.312403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:42.312403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:42.394403+0000","scrub_duration":0.0014316489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-10T18:23:01.047165+0000","last_change":"2024-09-10T18:23:01.047127+0000","last_active":"2024-09-10T18:23:01.047165+0000","last_peered":"2024-09-10T18:23:01.047165+0000","last_clean":"2024-09-10T18:23:01.047165+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:23:01.047165+0000","last_undegraded":"2024-09-10T18:23:01.047165+0000","last_fullsized":"2024-09-10T18:23:01.047165+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:01.047039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:23:01.047039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:06.384141+0000","scrub_duration":0.0015193260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":69,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029573+0000","last_change":"2024-09-10T18:22:54.449637+0000","last_active":"2024-09-10T18:22:57.029573+0000","last_peered":"2024-09-10T18:22:57.029573+0000","last_clean":"2024-09-10T18:22:57.029573+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:22:57.029573+0000","last_undegraded":"2024-09-10T18:22:57.029573+0000","last_fullsized":"2024-09-10T18:22:57.029573+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:22:54.449573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:54.449573+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:07.055885+0000","scrub_duration":0.0022198460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.144776+0000","last_change":"2024-09-10T18:21:31.531998+0000","last_active":"2024-09-10T18:22:57.144776+0000","last_peered":"2024-09-10T18:22:57.144776+0000","last_clean":"2024-09-10T18:22:57.144776+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:22:57.144776+0000","last_undegraded":"2024-09-10T18:22:57.144776+0000","last_fullsized":"2024-09-10T18:22:57.144776+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.142674+0000","last_change":"2024-09-10T18:22:40.275165+0000","last_active":"2024-09-10T18:22:57.142674+0000","last_peered":"2024-09-10T18:22:57.142674+0000","last_clean":"2024-09-10T18:22:57.142674+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:22:57.142674+0000","last_undegraded":"2024-09-10T18:22:57.142674+0000","last_fullsized":"2024-09-10T18:22:57.142674+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:40.275069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:40.275069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:54.251771+0000","scrub_duration":0.001522121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.028052+0000","last_change":"2024-09-10T18:21:37.644889+0000","last_active":"2024-09-10T18:22:57.028052+0000","last_peered":"2024-09-10T18:22:57.028052+0000","last_clean":"2024-09-10T18:22:57.028052+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:22:57.028052+0000","last_undegraded":"2024-09-10T18:22:57.028052+0000","last_fullsized":"2024-09-10T18:22:57.028052+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":71,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.098980+0000","last_change":"2024-09-10T18:22:51.599652+0000","last_active":"2024-09-10T18:22:57.098980+0000","last_peered":"2024-09-10T18:22:57.098980+0000","last_clean":"2024-09-10T18:22:57.098980+0000","last_became_active":"2024-09-10T18:21:34.046741+0000","last_became_peered":"2024-09-10T18:21:34.046741+0000","last_unstale":"2024-09-10T18:22:57.098980+0000","last_undegraded":"2024-09-10T18:22:57.098980+0000","last_fullsized":"2024-09-10T18:22:57.098980+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-10T18:22:51.599559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:51.599559+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:02.843581+0000","scrub_duration":0.0019152609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.144854+0000","last_change":"2024-09-10T18:21:31.530630+0000","last_active":"2024-09-10T18:22:57.144854+0000","last_peered":"2024-09-10T18:22:57.144854+0000","last_clean":"2024-09-10T18:22:57.144854+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:22:57.144854+0000","last_undegraded":"2024-09-10T18:22:57.144854+0000","last_fullsized":"2024-09-10T18:22:57.144854+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:59.769689+0000","last_change":"2024-09-10T18:21:35.627197+0000","last_active":"2024-09-10T18:22:59.769689+0000","last_peered":"2024-09-10T18:22:59.769689+0000","last_clean":"2024-09-10T18:22:59.769689+0000","last_became_active":"2024-09-10T18:21:35.626744+0000","last_became_peered":"2024-09-10T18:21:35.626744+0000","last_unstale":"2024-09-10T18:22:59.769689+0000","last_undegraded":"2024-09-10T18:22:59.769689+0000","last_fullsized":"2024-09-10T18:22:59.769689+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099184+0000","last_change":"2024-09-10T18:21:38.563824+0000","last_active":"2024-09-10T18:22:57.099184+0000","last_peered":"2024-09-10T18:22:57.099184+0000","last_clean":"2024-09-10T18:22:57.099184+0000","last_became_active":"2024-09-10T18:21:38.563550+0000","last_became_peered":"2024-09-10T18:21:38.563550+0000","last_unstale":"2024-09-10T18:22:57.099184+0000","last_undegraded":"2024-09-10T18:22:57.099184+0000","last_fullsized":"2024-09-10T18:22:57.099184+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:05.155686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143848+0000","last_change":"2024-09-10T18:21:31.527287+0000","last_active":"2024-09-10T18:22:57.143848+0000","last_peered":"2024-09-10T18:22:57.143848+0000","last_clean":"2024-09-10T18:22:57.143848+0000","last_became_active":"2024-09-10T18:21:31.526818+0000","last_became_peered":"2024-09-10T18:21:31.526818+0000","last_unstale":"2024-09-10T18:22:57.143848+0000","last_undegraded":"2024-09-10T18:22:57.143848+0000","last_fullsized":"2024-09-10T18:22:57.143848+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":46,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.028616+0000","last_change":"2024-09-10T18:21:34.046070+0000","last_active":"2024-09-10T18:22:57.028616+0000","last_peered":"2024-09-10T18:22:57.028616+0000","last_clean":"2024-09-10T18:22:57.028616+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:22:57.028616+0000","last_undegraded":"2024-09-10T18:22:57.028616+0000","last_fullsized":"2024-09-10T18:22:57.028616+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099420+0000","last_change":"2024-09-10T18:21:35.622254+0000","last_active":"2024-09-10T18:22:57.099420+0000","last_peered":"2024-09-10T18:22:57.099420+0000","last_clean":"2024-09-10T18:22:57.099420+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:22:57.099420+0000","last_undegraded":"2024-09-10T18:22:57.099420+0000","last_fullsized":"2024-09-10T18:22:57.099420+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.144509+0000","last_change":"2024-09-10T18:21:38.565959+0000","last_active":"2024-09-10T18:22:57.144509+0000","last_peered":"2024-09-10T18:22:57.144509+0000","last_clean":"2024-09-10T18:22:57.144509+0000","last_became_active":"2024-09-10T18:21:38.565675+0000","last_became_peered":"2024-09-10T18:21:38.565675+0000","last_unstale":"2024-09-10T18:22:57.144509+0000","last_undegraded":"2024-09-10T18:22:57.144509+0000","last_fullsized":"2024-09-10T18:22:57.144509+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143615+0000","last_change":"2024-09-10T18:21:34.047798+0000","last_active":"2024-09-10T18:22:57.143615+0000","last_peered":"2024-09-10T18:22:57.143615+0000","last_clean":"2024-09-10T18:22:57.143615+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:22:57.143615+0000","last_undegraded":"2024-09-10T18:22:57.143615+0000","last_fullsized":"2024-09-10T18:22:57.143615+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.144327+0000","last_change":"2024-09-10T18:22:56.100073+0000","last_active":"2024-09-10T18:22:57.144327+0000","last_peered":"2024-09-10T18:22:57.144327+0000","last_clean":"2024-09-10T18:22:57.144327+0000","last_became_active":"2024-09-10T18:21:31.531895+0000","last_became_peered":"2024-09-10T18:21:31.531895+0000","last_unstale":"2024-09-10T18:22:57.144327+0000","last_undegraded":"2024-09-10T18:22:57.144327+0000","last_fullsized":"2024-09-10T18:22:57.144327+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:22:56.100012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:13.102252+0000","scrub_duration":0.001414212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.028567+0000","last_change":"2024-09-10T18:22:49.301887+0000","last_active":"2024-09-10T18:22:57.028567+0000","last_peered":"2024-09-10T18:22:57.028567+0000","last_clean":"2024-09-10T18:22:57.028567+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:22:57.028567+0000","last_undegraded":"2024-09-10T18:22:57.028567+0000","last_fullsized":"2024-09-10T18:22:57.028567+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:49.301823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:49.301823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:06.288915+0000","scrub_duration":0.001243433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.142759+0000","last_change":"2024-09-10T18:21:37.637599+0000","last_active":"2024-09-10T18:22:57.142759+0000","last_peered":"2024-09-10T18:22:57.142759+0000","last_clean":"2024-09-10T18:22:57.142759+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:22:57.142759+0000","last_undegraded":"2024-09-10T18:22:57.142759+0000","last_fullsized":"2024-09-10T18:22:57.142759+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:02.280321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099447+0000","last_change":"2024-09-10T18:21:34.047045+0000","last_active":"2024-09-10T18:22:57.099447+0000","last_peered":"2024-09-10T18:22:57.099447+0000","last_clean":"2024-09-10T18:22:57.099447+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:22:57.099447+0000","last_undegraded":"2024-09-10T18:22:57.099447+0000","last_fullsized":"2024-09-10T18:22:57.099447+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143142+0000","last_change":"2024-09-10T18:22:38.445504+0000","last_active":"2024-09-10T18:22:57.143142+0000","last_peered":"2024-09-10T18:22:57.143142+0000","last_clean":"2024-09-10T18:22:57.143142+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:22:57.143142+0000","last_undegraded":"2024-09-10T18:22:57.143142+0000","last_fullsized":"2024-09-10T18:22:57.143142+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:38.445448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:38.445448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:05.044685+0000","scrub_duration":0.0018359400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":71,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:59.769728+0000","last_change":"2024-09-10T18:22:48.317756+0000","last_active":"2024-09-10T18:22:59.769728+0000","last_peered":"2024-09-10T18:22:59.769728+0000","last_clean":"2024-09-10T18:22:59.769728+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:22:59.769728+0000","last_undegraded":"2024-09-10T18:22:59.769728+0000","last_fullsized":"2024-09-10T18:22:59.769728+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:22:48.317687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:48.317687+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:01.400638+0000","scrub_duration":0.0022162499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-10T18:23:03.608800+0000","last_change":"2024-09-10T18:23:03.608764+0000","last_active":"2024-09-10T18:23:03.608800+0000","last_peered":"2024-09-10T18:23:03.608800+0000","last_clean":"2024-09-10T18:23:03.608800+0000","last_became_active":"2024-09-10T18:21:37.639706+0000","last_became_peered":"2024-09-10T18:21:37.639706+0000","last_unstale":"2024-09-10T18:23:03.608800+0000","last_undegraded":"2024-09-10T18:23:03.608800+0000","last_fullsized":"2024-09-10T18:23:03.608800+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:03.608698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:23:03.608698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:24.670346+0000","scrub_duration":0.0015379020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":62,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.028403+0000","last_change":"2024-09-10T18:22:33.342047+0000","last_active":"2024-09-10T18:22:57.028403+0000","last_peered":"2024-09-10T18:22:57.028403+0000","last_clean":"2024-09-10T18:22:57.028403+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:22:57.028403+0000","last_undegraded":"2024-09-10T18:22:57.028403+0000","last_fullsized":"2024-09-10T18:22:57.028403+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-10T18:22:33.341980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:33.341980+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:47.079316+0000","scrub_duration":0.0021839350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":48,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-10T18:23:00.076263+0000","last_change":"2024-09-10T18:23:00.076202+0000","last_active":"2024-09-10T18:23:00.076263+0000","last_peered":"2024-09-10T18:23:00.076263+0000","last_clean":"2024-09-10T18:23:00.076263+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:23:00.076263+0000","last_undegraded":"2024-09-10T18:23:00.076263+0000","last_fullsized":"2024-09-10T18:23:00.076263+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:00.076117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:23:00.076117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:25.553562+0000","scrub_duration":0.002299382,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:59.769979+0000","last_change":"2024-09-10T18:22:50.618285+0000","last_active":"2024-09-10T18:22:59.769979+0000","last_peered":"2024-09-10T18:22:59.769979+0000","last_clean":"2024-09-10T18:22:59.769979+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:22:59.769979+0000","last_undegraded":"2024-09-10T18:22:59.769979+0000","last_fullsized":"2024-09-10T18:22:59.769979+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:22:50.618215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:50.618215+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:08.321441+0000","scrub_duration":0.0017651660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143200+0000","last_change":"2024-09-10T18:21:37.644078+0000","last_active":"2024-09-10T18:22:57.143200+0000","last_peered":"2024-09-10T18:22:57.143200+0000","last_clean":"2024-09-10T18:22:57.143200+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:22:57.143200+0000","last_undegraded":"2024-09-10T18:22:57.143200+0000","last_fullsized":"2024-09-10T18:22:57.143200+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029576+0000","last_change":"2024-09-10T18:21:33.611605+0000","last_active":"2024-09-10T18:22:57.029576+0000","last_peered":"2024-09-10T18:22:57.029576+0000","last_clean":"2024-09-10T18:22:57.029576+0000","last_became_active":"2024-09-10T18:21:33.611286+0000","last_became_peered":"2024-09-10T18:21:33.611286+0000","last_unstale":"2024-09-10T18:22:57.029576+0000","last_undegraded":"2024-09-10T18:22:57.029576+0000","last_fullsized":"2024-09-10T18:22:57.029576+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.029856+0000","last_change":"2024-09-10T18:22:53.433078+0000","last_active":"2024-09-10T18:22:57.029856+0000","last_peered":"2024-09-10T18:22:57.029856+0000","last_clean":"2024-09-10T18:22:57.029856+0000","last_became_active":"2024-09-10T18:21:31.530158+0000","last_became_peered":"2024-09-10T18:21:31.530158+0000","last_unstale":"2024-09-10T18:22:57.029856+0000","last_undegraded":"2024-09-10T18:22:57.029856+0000","last_fullsized":"2024-09-10T18:22:57.029856+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:53.433012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:53.433012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:19.414323+0000","scrub_duration":0.0013676739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.097082+0000","last_change":"2024-09-10T18:22:36.049767+0000","last_active":"2024-09-10T18:22:57.097082+0000","last_peered":"2024-09-10T18:22:57.097082+0000","last_clean":"2024-09-10T18:22:57.097082+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:22:57.097082+0000","last_undegraded":"2024-09-10T18:22:57.097082+0000","last_fullsized":"2024-09-10T18:22:57.097082+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:36.049705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:36.049705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:39.206398+0000","scrub_duration":0.0030879660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.027687+0000","last_change":"2024-09-10T18:21:37.642856+0000","last_active":"2024-09-10T18:22:57.027687+0000","last_peered":"2024-09-10T18:22:57.027687+0000","last_clean":"2024-09-10T18:22:57.027687+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:22:57.027687+0000","last_undegraded":"2024-09-10T18:22:57.027687+0000","last_fullsized":"2024-09-10T18:22:57.027687+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:04.616953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.097314+0000","last_change":"2024-09-10T18:22:42.015349+0000","last_active":"2024-09-10T18:22:57.097314+0000","last_peered":"2024-09-10T18:22:57.097314+0000","last_clean":"2024-09-10T18:22:57.097314+0000","last_became_active":"2024-09-10T18:21:33.612401+0000","last_became_peered":"2024-09-10T18:21:33.612401+0000","last_unstale":"2024-09-10T18:22:57.097314+0000","last_undegraded":"2024-09-10T18:22:57.097314+0000","last_fullsized":"2024-09-10T18:22:57.097314+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:22:42.015292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:42.015292+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:07.052542+0000","scrub_duration":0.001986785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.144634+0000","last_change":"2024-09-10T18:22:43.250096+0000","last_active":"2024-09-10T18:22:57.144634+0000","last_peered":"2024-09-10T18:22:57.144634+0000","last_clean":"2024-09-10T18:22:57.144634+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:22:57.144634+0000","last_undegraded":"2024-09-10T18:22:57.144634+0000","last_fullsized":"2024-09-10T18:22:57.144634+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:43.250030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:43.250030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:12.038178+0000","scrub_duration":0.001581381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.028070+0000","last_change":"2024-09-10T18:21:35.623087+0000","last_active":"2024-09-10T18:22:57.028070+0000","last_peered":"2024-09-10T18:22:57.028070+0000","last_clean":"2024-09-10T18:22:57.028070+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:22:57.028070+0000","last_undegraded":"2024-09-10T18:22:57.028070+0000","last_fullsized":"2024-09-10T18:22:57.028070+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:03.160666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.142674+0000","last_change":"2024-09-10T18:22:52.262629+0000","last_active":"2024-09-10T18:22:57.142674+0000","last_peered":"2024-09-10T18:22:57.142674+0000","last_clean":"2024-09-10T18:22:57.142674+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:22:57.142674+0000","last_undegraded":"2024-09-10T18:22:57.142674+0000","last_fullsized":"2024-09-10T18:22:57.142674+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:52.262563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:52.262563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:20.159979+0000","scrub_duration":0.001452381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.027824+0000","last_change":"2024-09-10T18:21:34.046343+0000","last_active":"2024-09-10T18:22:57.027824+0000","last_peered":"2024-09-10T18:22:57.027824+0000","last_clean":"2024-09-10T18:22:57.027824+0000","last_became_active":"2024-09-10T18:21:34.046024+0000","last_became_peered":"2024-09-10T18:21:34.046024+0000","last_unstale":"2024-09-10T18:22:57.027824+0000","last_undegraded":"2024-09-10T18:22:57.027824+0000","last_fullsized":"2024-09-10T18:22:57.027824+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.099492+0000","last_change":"2024-09-10T18:22:40.686003+0000","last_active":"2024-09-10T18:22:57.099492+0000","last_peered":"2024-09-10T18:22:57.099492+0000","last_clean":"2024-09-10T18:22:57.099492+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:22:57.099492+0000","last_undegraded":"2024-09-10T18:22:57.099492+0000","last_fullsized":"2024-09-10T18:22:57.099492+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:22:40.685879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:01.722544+0000","scrub_duration":0.001767794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143311+0000","last_change":"2024-09-10T18:21:35.626825+0000","last_active":"2024-09-10T18:22:57.143311+0000","last_peered":"2024-09-10T18:22:57.143311+0000","last_clean":"2024-09-10T18:22:57.143311+0000","last_became_active":"2024-09-10T18:21:35.626537+0000","last_became_peered":"2024-09-10T18:21:35.626537+0000","last_unstale":"2024-09-10T18:22:57.143311+0000","last_undegraded":"2024-09-10T18:22:57.143311+0000","last_fullsized":"2024-09-10T18:22:57.143311+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.028166+0000","last_change":"2024-09-10T18:22:46.343197+0000","last_active":"2024-09-10T18:22:57.028166+0000","last_peered":"2024-09-10T18:22:57.028166+0000","last_clean":"2024-09-10T18:22:57.028166+0000","last_became_active":"2024-09-10T18:21:38.564314+0000","last_became_peered":"2024-09-10T18:21:38.564314+0000","last_unstale":"2024-09-10T18:22:57.028166+0000","last_undegraded":"2024-09-10T18:22:57.028166+0000","last_fullsized":"2024-09-10T18:22:57.028166+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:46.343137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:46.343137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:10.438171+0000","scrub_duration":0.0016021189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.027675+0000","last_change":"2024-09-10T18:21:33.618322+0000","last_active":"2024-09-10T18:22:57.027675+0000","last_peered":"2024-09-10T18:22:57.027675+0000","last_clean":"2024-09-10T18:22:57.027675+0000","last_became_active":"2024-09-10T18:21:33.617670+0000","last_became_peered":"2024-09-10T18:21:33.617670+0000","last_unstale":"2024-09-10T18:22:57.027675+0000","last_undegraded":"2024-09-10T18:22:57.027675+0000","last_fullsized":"2024-09-10T18:22:57.027675+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:21:32.587099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143575+0000","last_change":"2024-09-10T18:21:31.530146+0000","last_active":"2024-09-10T18:22:57.143575+0000","last_peered":"2024-09-10T18:22:57.143575+0000","last_clean":"2024-09-10T18:22:57.143575+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:22:57.143575+0000","last_undegraded":"2024-09-10T18:22:57.143575+0000","last_fullsized":"2024-09-10T18:22:57.143575+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.144303+0000","last_change":"2024-09-10T18:22:49.757505+0000","last_active":"2024-09-10T18:22:57.144303+0000","last_peered":"2024-09-10T18:22:57.144303+0000","last_clean":"2024-09-10T18:22:57.144303+0000","last_became_active":"2024-09-10T18:21:35.620419+0000","last_became_peered":"2024-09-10T18:21:35.620419+0000","last_unstale":"2024-09-10T18:22:57.144303+0000","last_undegraded":"2024-09-10T18:22:57.144303+0000","last_fullsized":"2024-09-10T18:22:57.144303+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:49.757435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:49.757435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:00.814048+0000","scrub_duration":0.0014410040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.144396+0000","last_change":"2024-09-10T18:21:37.643084+0000","last_active":"2024-09-10T18:22:57.144396+0000","last_peered":"2024-09-10T18:22:57.144396+0000","last_clean":"2024-09-10T18:22:57.144396+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:22:57.144396+0000","last_undegraded":"2024-09-10T18:22:57.144396+0000","last_fullsized":"2024-09-10T18:22:57.144396+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:05.246215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.097204+0000","last_change":"2024-09-10T18:22:44.108462+0000","last_active":"2024-09-10T18:22:57.097204+0000","last_peered":"2024-09-10T18:22:57.097204+0000","last_clean":"2024-09-10T18:22:57.097204+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:22:57.097204+0000","last_undegraded":"2024-09-10T18:22:57.097204+0000","last_fullsized":"2024-09-10T18:22:57.097204+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:44.108373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:44.108373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:23:51.902386+0000","scrub_duration":0.0024985559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.027978+0000","last_change":"2024-09-10T18:21:31.535545+0000","last_active":"2024-09-10T18:22:57.027978+0000","last_peered":"2024-09-10T18:22:57.027978+0000","last_clean":"2024-09-10T18:22:57.027978+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:22:57.027978+0000","last_undegraded":"2024-09-10T18:22:57.027978+0000","last_fullsized":"2024-09-10T18:22:57.027978+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:21:30.506537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":65,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-10T18:23:00.864553+0000","last_change":"2024-09-10T18:23:00.864502+0000","last_active":"2024-09-10T18:23:00.864553+0000","last_peered":"2024-09-10T18:23:00.864553+0000","last_clean":"2024-09-10T18:23:00.864553+0000","last_became_active":"2024-09-10T18:21:33.613487+0000","last_became_peered":"2024-09-10T18:21:33.613487+0000","last_unstale":"2024-09-10T18:23:00.864553+0000","last_undegraded":"2024-09-10T18:23:00.864553+0000","last_fullsized":"2024-09-10T18:23:00.864553+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:23:00.864408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:23:00.864408+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:24:02.969515+0000","scrub_duration":0.0025444080000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-10T18:22:57.143363+0000","last_change":"2024-09-10T18:21:35.620819+0000","last_active":"2024-09-10T18:22:57.143363+0000","last_peered":"2024-09-10T18:22:57.143363+0000","last_clean":"2024-09-10T18:22:57.143363+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:22:57.143363+0000","last_undegraded":"2024-09-10T18:22:57.143363+0000","last_fullsized":"2024-09-10T18:22:57.143363+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":419,"ondisk_log_size":419,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":81,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234006,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298556,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446596,"statfs":{"total":95995035648,"available":95689314304,"internally_reserved":0,"allocated":1232896,"data_stored":883416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.46200000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.56499999999999995}]},{"osd":2,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.48299999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.502},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.61499999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.439}]},{"osd":5,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.59199999999999997}]},{"osd":6,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.53100000000000003}]}]},{"osd":6,"up_from":46,"seq":197568495644,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298584,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446568,"statfs":{"total":95995035648,"available":95689285632,"internally_reserved":0,"allocated":1216512,"data_stored":883588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55100000000000005,"15min":0.56000000000000005},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71299999999999997,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53900000000000003,"15min":0.54000000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.747,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.61299999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.55400000000000005,"15min":0.56000000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.65400000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54200000000000004,"15min":0.54600000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74099999999999999,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.54500000000000004}]},{"osd":2,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.71199999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54900000000000004,"15min":0.55200000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73999999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.64600000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.68100000000000005,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.625},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.70899999999999996,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.501}]},{"osd":4,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.42299999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.438,"15min":0.434},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.57699999999999996}]},{"osd":5,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.443,"15min":0.435},"min":{"1min":0.317,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.44600000000000001,"15min":0.439},"min":{"1min":0.315,"5min":0.308,"15min":0.308},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.45000000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.56599999999999995}]}]},{"osd":5,"up_from":39,"seq":167503724577,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297960,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447192,"statfs":{"total":95995035648,"available":95689924608,"internally_reserved":0,"allocated":622592,"data_stored":292881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.36399999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.42499999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.65600000000000003}]},{"osd":1,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.56499999999999995,"15min":0.56100000000000005},"min":{"1min":0.42399999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.377,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.69899999999999995}]},{"osd":2,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55100000000000005,"15min":0.55200000000000005},"min":{"1min":0.38800000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.38,"5min":0.255,"15min":0.255},"max":{"1min":0.75600000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.51100000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.54800000000000004,"15min":0.54300000000000004},"min":{"1min":0.434,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.52900000000000003,"15min":0.53600000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.67400000000000004}]},{"osd":4,"last update":"Tue Sep 10 18:22:36 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.36499999999999999,"15min":0.35899999999999999},"min":{"1min":0.29499999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.38400000000000001},"min":{"1min":0.28599999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.36199999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:22:58 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.42399999999999999,"15min":0.40999999999999998},"min":{"1min":0.36599999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.44800000000000001,"15min":0.43099999999999999},"min":{"1min":0.34100000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.38}]},{"osd":7,"last update":"Tue Sep 10 18:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.41199999999999998}]}]},{"osd":4,"up_from":33,"seq":141733920805,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297976,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447176,"statfs":{"total":95995035648,"available":95689908224,"internally_reserved":0,"allocated":638976,"data_stored":292859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56599999999999995,"15min":0.57499999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.72699999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.746,"5min":0.61799999999999999,"15min":0.59699999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.8170000000000002,"5min":4.8170000000000002,"15min":4.8170000000000002},"last":0.60899999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.61499999999999999,"15min":0.59699999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":4.7729999999999997,"5min":4.7729999999999997,"15min":4.7729999999999997},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.64100000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.60699999999999998,"15min":0.57899999999999996},"min":{"1min":0.42799999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.8040000000000003,"5min":4.8040000000000003,"15min":4.8040000000000003},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.57599999999999996,"15min":0.58599999999999997},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71699999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.57599999999999996}]},{"osd":3,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.66600000000000004,"5min":0.751,"15min":0.751},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59099999999999997,"15min":0.59599999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.747,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.65800000000000003}]},{"osd":5,"last update":"Tue Sep 10 18:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.39000000000000001,"15min":0.38300000000000001},"min":{"1min":0.32800000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42499999999999999,"15min":0.42399999999999999},"min":{"1min":0.28299999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60099999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.45500000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:22:59 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40400000000000003},"min":{"1min":0.29299999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.61599999999999999,"5min":0.629,"15min":0.629},"last":0.496},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.46999999999999997,"15min":0.46600000000000003},"min":{"1min":0.36299999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64100000000000001,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.47599999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.53200000000000003}]}]},{"osd":3,"up_from":27,"seq":115964117035,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298032,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447120,"statfs":{"total":95995035648,"available":95689850880,"internally_reserved":0,"allocated":651264,"data_stored":293703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.39800000000000002,"15min":0.378},"min":{"1min":0.23000000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.40999999999999998,"15min":0.39600000000000002},"min":{"1min":0.255,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.222}]},{"osd":1,"last update":"Tue Sep 10 18:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.437,"15min":0.435},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.41599999999999998,"15min":0.40100000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.23799999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.45100000000000001,"15min":0.44},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.46999999999999997,"15min":0.41599999999999998},"min":{"1min":0.27700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":7.2750000000000004,"5min":7.2750000000000004,"15min":7.2750000000000004},"last":0.30399999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:22:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.55600000000000005,"15min":0.54000000000000004},"min":{"1min":0.378,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.56499999999999995,"15min":0.55500000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.44700000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:22:34 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":0.73899999999999999,"15min":0.63400000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":14.885,"5min":14.885,"15min":14.885},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.61899999999999999,"15min":0.60699999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.52500000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.63500000000000001,"15min":0.629},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.64400000000000002,"15min":0.64600000000000002},"min":{"1min":0.41299999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.499}]},{"osd":7,"last update":"Tue Sep 10 18:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.57099999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297952,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447200,"statfs":{"total":95995035648,"available":95689932800,"internally_reserved":0,"allocated":614400,"data_stored":292111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.36199999999999999,"15min":0.33400000000000002},"min":{"1min":0.22,"5min":0.219,"15min":0.219},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.371,"15min":0.34899999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.29899999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.35599999999999998,"15min":0.33800000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.63100000000000001,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.39100000000000001,"15min":0.35499999999999998},"min":{"1min":0.28199999999999997,"5min":0.22,"15min":0.22},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.245}]},{"osd":3,"last update":"Tue Sep 10 18:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.48399999999999999,"15min":0.46999999999999997},"min":{"1min":0.313,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.371},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.35799999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.56799999999999995,"15min":0.56499999999999995},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.76300000000000001,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.316},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.76000000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.42999999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.61499999999999999,"15min":0.59399999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":2.3279999999999998,"5min":2.3279999999999998,"15min":2.3279999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.63,"15min":0.61599999999999999},"min":{"1min":0.44500000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.40100000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:23:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.88300000000000001,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.625,"15min":0.628},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.39000000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.41399999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298056,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447096,"statfs":{"total":95995035648,"available":95689826304,"internally_reserved":0,"allocated":675840,"data_stored":294039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39000000000000001,"15min":0.373},"min":{"1min":0.27700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.47899999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.379},"min":{"1min":0.25700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.38200000000000001,"15min":0.36399999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.64800000000000002,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.46100000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.26800000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.59399999999999997,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.45100000000000001,"15min":0.46400000000000002},"min":{"1min":0.30499999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.63300000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.56000000000000005}]},{"osd":4,"last update":"Tue Sep 10 18:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.85899999999999999,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.94799999999999995,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69399999999999995}]},{"osd":5,"last update":"Tue Sep 10 18:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.55100000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.76000000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:23:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.60899999999999999,"15min":0.61099999999999999},"min":{"1min":0.435,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78700000000000003,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84199999999999997,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.84199999999999997}]},{"osd":7,"last update":"Tue Sep 10 18:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.77300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298840,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446312,"statfs":{"total":95995035648,"available":95689023488,"internally_reserved":0,"allocated":1228800,"data_stored":883100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:23:02 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.35999999999999999,"15min":0.34899999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.45000000000000001,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.252},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.377,"15min":0.35399999999999998},"min":{"1min":0.25,"5min":0.249,"15min":0.249},"max":{"1min":0.49199999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.33200000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40999999999999998,"15min":0.40699999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53500000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.40600000000000003,"15min":0.40200000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.47899999999999998,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.38800000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.44400000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.55500000000000005,"5min":0.629,"15min":0.629},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.434,"15min":0.44500000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.54100000000000004,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.30399999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:22:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.624,"15min":0.63100000000000001},"min":{"1min":0.433,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.83299999999999996,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.65000000000000002,"15min":0.66100000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81899999999999995,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.52200000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.59399999999999997,"15min":0.60299999999999998},"min":{"1min":0.437,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.72099999999999997,"5min":0.879,"15min":0.879},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.625,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.496}]},{"osd":6,"last update":"Tue Sep 10 18:23:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60899999999999999,"15min":0.61299999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.61899999999999999,"15min":0.624},"min":{"1min":0.46000000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.76800000000000002,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.56000000000000005}]},{"osd":7,"last update":"Tue Sep 10 18:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.53400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:23:08.367 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-10T18:23:08.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd dump 2024-09-10T18:23:08.604 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=live.push_http t=2024-09-10T18:23:08.2702361Z level=info msg="Live Push Gateway initialization" 2024-09-10T18:23:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:08.919 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:08 smithi028 systemd[1]: Starting Ceph node-exporter.a for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:23:09.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[39220]: 3.f scrub starts 2024-09-10T18:23:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[39220]: 3.f scrub ok 2024-09-10T18:23:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[39220]: 5.0 deep-scrub starts 2024-09-10T18:23:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[39220]: 5.0 deep-scrub ok 2024-09-10T18:23:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[39220]: Deploying daemon node-exporter.a on smithi028 2024-09-10T18:23:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[39220]: 6.2 scrub starts 2024-09-10T18:23:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[39220]: 6.2 scrub ok 2024-09-10T18:23:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[39220]: from='client.24673 v1:172.21.15.28:0/606041390' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:23:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-10T18:23:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-10T18:23:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T18:23:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-10T18:23:09.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-10T18:23:09.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T18:23:09.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:09.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:23:09.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[29291]: 3.f scrub starts 2024-09-10T18:23:09.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[29291]: 3.f scrub ok 2024-09-10T18:23:09.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[29291]: 5.0 deep-scrub starts 2024-09-10T18:23:09.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[29291]: 5.0 deep-scrub ok 2024-09-10T18:23:09.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[29291]: Deploying daemon node-exporter.a on smithi028 2024-09-10T18:23:09.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[29291]: 6.2 scrub starts 2024-09-10T18:23:09.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[29291]: 6.2 scrub ok 2024-09-10T18:23:09.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[29291]: from='client.24673 v1:172.21.15.28:0/606041390' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:23:09.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-10T18:23:09.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-10T18:23:09.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T18:23:09.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-10T18:23:09.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-10T18:23:09.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T18:23:09.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:09.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:23:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-mon[32357]: 3.f scrub starts 2024-09-10T18:23:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-mon[32357]: 3.f scrub ok 2024-09-10T18:23:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-mon[32357]: 5.0 deep-scrub starts 2024-09-10T18:23:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-mon[32357]: 5.0 deep-scrub ok 2024-09-10T18:23:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-mon[32357]: Deploying daemon node-exporter.a on smithi028 2024-09-10T18:23:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-mon[32357]: 6.2 scrub starts 2024-09-10T18:23:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-mon[32357]: 6.2 scrub ok 2024-09-10T18:23:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-mon[32357]: from='client.24673 v1:172.21.15.28:0/606041390' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:23:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-10T18:23:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-10T18:23:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T18:23:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-10T18:23:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-10T18:23:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T18:23:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:09.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:23:09.356 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=infra.usagestats.collector t=2024-09-10T18:23:09.049200734Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-10T18:23:09.356 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=provisioning.datasources t=2024-09-10T18:23:09.049583893Z level=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-09-10T18:23:09.356 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=provisioning.datasources t=2024-09-10T18:23:09.083013691Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-10T18:23:09.356 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=provisioning.datasources t=2024-09-10T18:23:09.115390462Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-10T18:23:09.356 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=provisioning.alerting t=2024-09-10T18:23:09.137101531Z level=info msg="starting to provision alerting" 2024-09-10T18:23:09.356 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=provisioning.alerting t=2024-09-10T18:23:09.137122083Z level=info msg="finished to provision alerting" 2024-09-10T18:23:09.356 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=grafanaStorageLogger t=2024-09-10T18:23:09.137259429Z level=info msg="storage starting" 2024-09-10T18:23:09.356 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=ngalert.state.manager t=2024-09-10T18:23:09.13749471Z level=info msg="Warming state cache for startup" 2024-09-10T18:23:09.356 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=http.server t=2024-09-10T18:23:09.140389219Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-10T18:23:09.356 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=ngalert.state.manager t=2024-09-10T18:23:09.147592477Z level=info msg="State cache has been initialized" states=0 duration=10.097059ms 2024-09-10T18:23:09.357 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=ngalert.multiorg.alertmanager t=2024-09-10T18:23:09.147638509Z level=info msg="starting MultiOrg Alertmanager" 2024-09-10T18:23:09.357 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=ticker t=2024-09-10T18:23:09.147671248Z level=info msg=starting first_tick=2024-09-10T18:23:10Z 2024-09-10T18:23:09.357 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=sqlstore.transactions t=2024-09-10T18:23:09.147679189Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-10T18:23:09.357 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=sqlstore.transactions t=2024-09-10T18:23:09.181542906Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-10T18:23:09.357 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:23:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=sqlstore.transactions t=2024-09-10T18:23:09.225050355Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-10T18:23:10.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:10 smithi028 ceph-mon[39220]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:23:10.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:10 smithi028 ceph-mon[39220]: 3.10 deep-scrub starts 2024-09-10T18:23:10.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:10 smithi028 ceph-mon[39220]: 3.10 deep-scrub ok 2024-09-10T18:23:10.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:10 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-10T18:23:10.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:10 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-09-10T18:23:10.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:10 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-10T18:23:10.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:10 smithi028 ceph-mon[39220]: osdmap e89: 8 total, 8 up, 8 in 2024-09-10T18:23:10.195 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:10 smithi028 bash[73332]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-10T18:23:10.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:10 smithi028 ceph-mon[29291]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:23:10.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:10 smithi028 ceph-mon[29291]: 3.10 deep-scrub starts 2024-09-10T18:23:10.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:10 smithi028 ceph-mon[29291]: 3.10 deep-scrub ok 2024-09-10T18:23:10.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:10 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-10T18:23:10.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:10 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-09-10T18:23:10.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:10 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-10T18:23:10.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:10 smithi028 ceph-mon[29291]: osdmap e89: 8 total, 8 up, 8 in 2024-09-10T18:23:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:10 smithi204 ceph-mon[32357]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:23:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:10 smithi204 ceph-mon[32357]: 3.10 deep-scrub starts 2024-09-10T18:23:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:10 smithi204 ceph-mon[32357]: 3.10 deep-scrub ok 2024-09-10T18:23:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:10 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-10T18:23:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:10 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-09-10T18:23:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:10 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-10T18:23:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:10 smithi204 ceph-mon[32357]: osdmap e89: 8 total, 8 up, 8 in 2024-09-10T18:23:10.504 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:10 smithi028 bash[73332]: Getting image source signatures 2024-09-10T18:23:10.504 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:10 smithi028 bash[73332]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-10T18:23:10.504 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:10 smithi028 bash[73332]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-10T18:23:10.504 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:10 smithi028 bash[73332]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-10T18:23:10.568 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 90 2024-09-10T18:23:10.568 INFO:teuthology.orchestra.run.smithi028.stdout:fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 2024-09-10T18:23:10.568 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-09-10T18:16:20.680373+0000 2024-09-10T18:23:10.568 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-09-10T18:23:10.059417+0000 2024-09-10T18:23:10.568 INFO:teuthology.orchestra.run.smithi028.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-10T18:23:10.568 INFO:teuthology.orchestra.run.smithi028.stdout:crush_version 18 2024-09-10T18:23:10.568 INFO:teuthology.orchestra.run.smithi028.stdout:full_ratio 0.95 2024-09-10T18:23:10.568 INFO:teuthology.orchestra.run.smithi028.stdout:backfillfull_ratio 0.9 2024-09-10T18:23:10.568 INFO:teuthology.orchestra.run.smithi028.stdout:nearfull_ratio 0.85 2024-09-10T18:23:10.568 INFO:teuthology.orchestra.run.smithi028.stdout:require_min_compat_client luminous 2024-09-10T18:23:10.568 INFO:teuthology.orchestra.run.smithi028.stdout:min_compat_client luminous 2024-09-10T18:23:10.569 INFO:teuthology.orchestra.run.smithi028.stdout:require_osd_release quincy 2024-09-10T18:23:10.569 INFO:teuthology.orchestra.run.smithi028.stdout:stretch_mode_enabled false 2024-09-10T18:23:10.569 INFO:teuthology.orchestra.run.smithi028.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2024-09-10T18:23:10.569 INFO:teuthology.orchestra.run.smithi028.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 62 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd 2024-09-10T18:23:10.569 INFO:teuthology.orchestra.run.smithi028.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 59 flags hashpspool stripe_width 0 application rgw 2024-09-10T18:23:10.569 INFO:teuthology.orchestra.run.smithi028.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 61 flags hashpspool stripe_width 0 application rgw 2024-09-10T18:23:10.569 INFO:teuthology.orchestra.run.smithi028.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 63 flags hashpspool stripe_width 0 application rgw 2024-09-10T18:23:10.569 INFO:teuthology.orchestra.run.smithi028.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 65 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw 2024-09-10T18:23:10.569 INFO:teuthology.orchestra.run.smithi028.stdout:max_osd 8 2024-09-10T18:23:10.569 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 up in weight 1 up_from 8 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6801/3188372351 v1:172.21.15.28:6802/3188372351 exists,up f1c834d9-252a-4077-b25e-169181d70251 2024-09-10T18:23:10.569 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 up in weight 1 up_from 13 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6805/1369912678 v1:172.21.15.28:6806/1369912678 exists,up 81f1855d-b946-4ca3-b246-38977b9e20fd 2024-09-10T18:23:10.569 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 up in weight 1 up_from 18 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6809/1243406003 v1:172.21.15.28:6810/1243406003 exists,up 2ad55d2a-36fa-4756-bf22-58d4b049486e 2024-09-10T18:23:10.569 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 up in weight 1 up_from 27 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6813/67703283 v1:172.21.15.28:6814/67703283 exists,up 242729a5-a5e8-44d5-97c8-14d14e970bff 2024-09-10T18:23:10.570 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 up in weight 1 up_from 33 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.204:6800/1619669076 v1:172.21.15.204:6801/1619669076 exists,up 6e38b8ef-e6ed-493f-8eca-755354754787 2024-09-10T18:23:10.570 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 up in weight 1 up_from 39 up_thru 89 down_at 0 last_clean_interval [0,0) v1:172.21.15.204:6804/2717802880 v1:172.21.15.204:6805/2717802880 exists,up a92e0441-66c9-498e-a0ac-6465acdbb1f5 2024-09-10T18:23:10.570 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 up in weight 1 up_from 46 up_thru 89 down_at 0 last_clean_interval [0,0) v1:172.21.15.204:6808/3377682607 v1:172.21.15.204:6809/3377682607 exists,up 12b02875-6653-45ed-8d2b-b4d33bd88525 2024-09-10T18:23:10.570 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 up in weight 1 up_from 54 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.204:6812/3511791879 v1:172.21.15.204:6813/3511791879 exists,up 0c4f7aa9-ff95-4192-a146-ca576eee907f 2024-09-10T18:23:10.570 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.4 [1,5] 2024-09-10T18:23:10.570 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.a [1,2] 2024-09-10T18:23:10.570 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.1f [1,2] 2024-09-10T18:23:10.570 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1745574083 expires 2024-09-11T18:22:08.678985+0000 2024-09-10T18:23:10.570 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/3866220552 expires 2024-09-11T18:16:45.197204+0000 2024-09-10T18:23:10.570 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/3760775432 expires 2024-09-11T18:16:45.197204+0000 2024-09-10T18:23:10.570 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1746797014 expires 2024-09-11T18:16:45.197204+0000 2024-09-10T18:23:10.570 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1029045895 expires 2024-09-11T18:16:45.197204+0000 2024-09-10T18:23:10.570 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/3310949126 expires 2024-09-11T18:17:07.521165+0000 2024-09-10T18:23:10.571 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2874160765 expires 2024-09-11T18:17:07.521165+0000 2024-09-10T18:23:10.571 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2620179974 expires 2024-09-11T18:22:08.678985+0000 2024-09-10T18:23:10.571 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/421194352 expires 2024-09-11T18:17:07.521165+0000 2024-09-10T18:23:10.571 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/1077377902 expires 2024-09-11T18:22:08.678985+0000 2024-09-10T18:23:10.571 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/3428757300 expires 2024-09-11T18:22:08.678985+0000 2024-09-10T18:23:10.571 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1759116360 expires 2024-09-11T18:22:08.678985+0000 2024-09-10T18:23:10.571 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2564333506 expires 2024-09-11T18:17:07.521165+0000 2024-09-10T18:23:10.571 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/717120167 expires 2024-09-11T18:22:08.678985+0000 2024-09-10T18:23:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:11 smithi204 ceph-mon[32357]: 6.15 scrub starts 2024-09-10T18:23:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:11 smithi204 ceph-mon[32357]: 6.15 scrub ok 2024-09-10T18:23:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:11 smithi204 ceph-mon[32357]: 6.e scrub starts 2024-09-10T18:23:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:11 smithi204 ceph-mon[32357]: 6.e scrub ok 2024-09-10T18:23:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:11 smithi204 ceph-mon[32357]: 3.a scrub starts 2024-09-10T18:23:11.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:11 smithi204 ceph-mon[32357]: 3.a scrub ok 2024-09-10T18:23:11.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:11 smithi204 ceph-mon[32357]: osdmap e90: 8 total, 8 up, 8 in 2024-09-10T18:23:11.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:11 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2803590550' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-10T18:23:11.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:11 smithi204 ceph-mon[32357]: 2.2 scrub starts 2024-09-10T18:23:11.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:11 smithi204 ceph-mon[32357]: 2.2 scrub ok 2024-09-10T18:23:11.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:11 smithi204 ceph-mon[32357]: pgmap v49: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:23:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[29291]: 6.15 scrub starts 2024-09-10T18:23:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[29291]: 6.15 scrub ok 2024-09-10T18:23:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[29291]: 6.e scrub starts 2024-09-10T18:23:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[29291]: 6.e scrub ok 2024-09-10T18:23:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[29291]: 3.a scrub starts 2024-09-10T18:23:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[29291]: 3.a scrub ok 2024-09-10T18:23:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[29291]: osdmap e90: 8 total, 8 up, 8 in 2024-09-10T18:23:11.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2803590550' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-10T18:23:11.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[29291]: 2.2 scrub starts 2024-09-10T18:23:11.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[29291]: 2.2 scrub ok 2024-09-10T18:23:11.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[29291]: pgmap v49: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:23:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[39220]: 6.15 scrub starts 2024-09-10T18:23:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[39220]: 6.15 scrub ok 2024-09-10T18:23:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[39220]: 6.e scrub starts 2024-09-10T18:23:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[39220]: 6.e scrub ok 2024-09-10T18:23:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[39220]: 3.a scrub starts 2024-09-10T18:23:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[39220]: 3.a scrub ok 2024-09-10T18:23:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[39220]: osdmap e90: 8 total, 8 up, 8 in 2024-09-10T18:23:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2803590550' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-10T18:23:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[39220]: 2.2 scrub starts 2024-09-10T18:23:11.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[39220]: 2.2 scrub ok 2024-09-10T18:23:11.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:11 smithi028 ceph-mon[39220]: pgmap v49: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:23:12.268 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6801/3188372351 v1:172.21.15.28:6802/3188372351 exists,up f1c834d9-252a-4077-b25e-169181d70251', 'osd.1 up in weight 1 up_from 13 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6805/1369912678 v1:172.21.15.28:6806/1369912678 exists,up 81f1855d-b946-4ca3-b246-38977b9e20fd', 'osd.2 up in weight 1 up_from 18 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6809/1243406003 v1:172.21.15.28:6810/1243406003 exists,up 2ad55d2a-36fa-4756-bf22-58d4b049486e', 'osd.3 up in weight 1 up_from 27 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6813/67703283 v1:172.21.15.28:6814/67703283 exists,up 242729a5-a5e8-44d5-97c8-14d14e970bff', 'osd.4 up in weight 1 up_from 33 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.204:6800/1619669076 v1:172.21.15.204:6801/1619669076 exists,up 6e38b8ef-e6ed-493f-8eca-755354754787', 'osd.5 up in weight 1 up_from 39 up_thru 89 down_at 0 last_clean_interval [0,0) v1:172.21.15.204:6804/2717802880 v1:172.21.15.204:6805/2717802880 exists,up a92e0441-66c9-498e-a0ac-6465acdbb1f5', 'osd.6 up in weight 1 up_from 46 up_thru 89 down_at 0 last_clean_interval [0,0) v1:172.21.15.204:6808/3377682607 v1:172.21.15.204:6809/3377682607 exists,up 12b02875-6653-45ed-8d2b-b4d33bd88525', 'osd.7 up in weight 1 up_from 54 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.204:6812/3511791879 v1:172.21.15.204:6813/3511791879 exists,up 0c4f7aa9-ff95-4192-a146-ca576eee907f'] 2024-09-10T18:23:12.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon mon.a config show 2024-09-10T18:23:12.343 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 bash[73332]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-10T18:23:12.343 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-mon[29291]: 6.1e scrub starts 2024-09-10T18:23:12.343 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-mon[29291]: 6.1e scrub ok 2024-09-10T18:23:12.343 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-mon[29291]: 5.7 scrub starts 2024-09-10T18:23:12.343 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-mon[29291]: 5.7 scrub ok 2024-09-10T18:23:12.343 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-mon[39220]: 6.1e scrub starts 2024-09-10T18:23:12.344 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-mon[39220]: 6.1e scrub ok 2024-09-10T18:23:12.344 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-mon[39220]: 5.7 scrub starts 2024-09-10T18:23:12.344 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-mon[39220]: 5.7 scrub ok 2024-09-10T18:23:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:12 smithi204 ceph-mon[32357]: 6.1e scrub starts 2024-09-10T18:23:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:12 smithi204 ceph-mon[32357]: 6.1e scrub ok 2024-09-10T18:23:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:12 smithi204 ceph-mon[32357]: 5.7 scrub starts 2024-09-10T18:23:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:12 smithi204 ceph-mon[32357]: 5.7 scrub ok 2024-09-10T18:23:12.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:23:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:23:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:23:12.669 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 bash[73332]: Writing manifest to image destination 2024-09-10T18:23:12.949 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 podman[73332]: 2024-09-10 18:23:12.61439757 +0000 UTC m=+2.674106261 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-10T18:23:12.949 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 podman[73332]: 2024-09-10 18:23:12.695728172 +0000 UTC m=+2.755436864 container create 5922a8d5ce88c530d86d87987c0ef8d2851e1bdf9a5a9b8914607aead6e26939 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T18:23:12.949 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 podman[73332]: 2024-09-10 18:23:12.948214999 +0000 UTC m=+3.007923700 container init 5922a8d5ce88c530d86d87987c0ef8d2851e1bdf9a5a9b8914607aead6e26939 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T18:23:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:13.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:13 smithi028 ceph-mon[29291]: 4.17 scrub starts 2024-09-10T18:23:13.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:13 smithi028 ceph-mon[29291]: 4.17 scrub ok 2024-09-10T18:23:13.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:13 smithi028 ceph-mon[29291]: 5.e scrub starts 2024-09-10T18:23:13.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:13 smithi028 ceph-mon[29291]: 5.e scrub ok 2024-09-10T18:23:13.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:13 smithi028 ceph-mon[29291]: 6.16 scrub starts 2024-09-10T18:23:13.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:13 smithi028 ceph-mon[29291]: 6.16 scrub ok 2024-09-10T18:23:13.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:13 smithi028 ceph-mon[29291]: pgmap v50: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:23:13.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:13 smithi028 ceph-mon[39220]: 4.17 scrub starts 2024-09-10T18:23:13.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:13 smithi028 ceph-mon[39220]: 4.17 scrub ok 2024-09-10T18:23:13.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:13 smithi028 ceph-mon[39220]: 5.e scrub starts 2024-09-10T18:23:13.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:13 smithi028 ceph-mon[39220]: 5.e scrub ok 2024-09-10T18:23:13.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:13 smithi028 ceph-mon[39220]: 6.16 scrub starts 2024-09-10T18:23:13.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:13 smithi028 ceph-mon[39220]: 6.16 scrub ok 2024-09-10T18:23:13.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:13 smithi028 ceph-mon[39220]: pgmap v50: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:23:13.210 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 podman[73332]: 2024-09-10 18:23:12.951351311 +0000 UTC m=+3.011060003 container start 5922a8d5ce88c530d86d87987c0ef8d2851e1bdf9a5a9b8914607aead6e26939 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T18:23:13.211 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.955Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-10T18:23:13.211 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.955Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-10T18:23:13.211 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.955Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-10T18:23:13.211 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.955Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-10T18:23:13.211 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-10T18:23:13.211 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-10T18:23:13.211 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-10T18:23:13.211 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=arp 2024-09-10T18:23:13.211 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-10T18:23:13.211 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-10T18:23:13.211 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-10T18:23:13.211 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-10T18:23:13.211 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-10T18:23:13.212 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-10T18:23:13.212 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-10T18:23:13.212 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-10T18:23:13.212 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=edac 2024-09-10T18:23:13.212 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-10T18:23:13.212 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-10T18:23:13.212 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-10T18:23:13.212 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-10T18:23:13.212 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-10T18:23:13.212 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-10T18:23:13.212 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-10T18:23:13.212 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-10T18:23:13.212 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-10T18:23:13.212 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-10T18:23:13.212 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-10T18:23:13.213 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-10T18:23:13.213 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-10T18:23:13.213 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-10T18:23:13.213 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-10T18:23:13.213 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-10T18:23:13.213 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=os 2024-09-10T18:23:13.213 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-10T18:23:13.213 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-10T18:23:13.213 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-10T18:23:13.213 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-10T18:23:13.213 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-10T18:23:13.213 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-10T18:23:13.213 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-10T18:23:13.213 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=stat 2024-09-10T18:23:13.214 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-10T18:23:13.214 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-10T18:23:13.214 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-10T18:23:13.214 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=time 2024-09-10T18:23:13.214 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-10T18:23:13.214 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=uname 2024-09-10T18:23:13.214 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-10T18:23:13.214 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-10T18:23:13.214 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.956Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-10T18:23:13.214 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.957Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-10T18:23:13.214 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:12 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a[73526]: ts=2024-09-10T18:23:12.957Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-10T18:23:13.214 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:13 smithi028 bash[73332]: 5922a8d5ce88c530d86d87987c0ef8d2851e1bdf9a5a9b8914607aead6e26939 2024-09-10T18:23:13.214 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:23:13 smithi028 systemd[1]: Started Ceph node-exporter.a for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:23:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:13 smithi204 ceph-mon[32357]: 4.17 scrub starts 2024-09-10T18:23:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:13 smithi204 ceph-mon[32357]: 4.17 scrub ok 2024-09-10T18:23:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:13 smithi204 ceph-mon[32357]: 5.e scrub starts 2024-09-10T18:23:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:13 smithi204 ceph-mon[32357]: 5.e scrub ok 2024-09-10T18:23:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:13 smithi204 ceph-mon[32357]: 6.16 scrub starts 2024-09-10T18:23:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:13 smithi204 ceph-mon[32357]: 6.16 scrub ok 2024-09-10T18:23:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:13 smithi204 ceph-mon[32357]: pgmap v50: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:23:14.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:23:13 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:23:14.101 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:13 smithi204 bash[63159]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-10T18:23:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[29291]: 4.0 scrub starts 2024-09-10T18:23:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[29291]: 4.0 scrub ok 2024-09-10T18:23:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[29291]: 6.1c scrub starts 2024-09-10T18:23:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[29291]: 6.1c scrub ok 2024-09-10T18:23:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[29291]: Deploying daemon node-exporter.b on smithi204 2024-09-10T18:23:14.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[29291]: 6.9 scrub starts 2024-09-10T18:23:14.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[29291]: 6.9 scrub ok 2024-09-10T18:23:14.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:14.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[39220]: 4.0 scrub starts 2024-09-10T18:23:14.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[39220]: 4.0 scrub ok 2024-09-10T18:23:14.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[39220]: 6.1c scrub starts 2024-09-10T18:23:14.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[39220]: 6.1c scrub ok 2024-09-10T18:23:14.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:14.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:14.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:14.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[39220]: Deploying daemon node-exporter.b on smithi204 2024-09-10T18:23:14.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[39220]: 6.9 scrub starts 2024-09-10T18:23:14.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[39220]: 6.9 scrub ok 2024-09-10T18:23:14.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:14 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:14 smithi204 ceph-mon[32357]: 4.0 scrub starts 2024-09-10T18:23:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:14 smithi204 ceph-mon[32357]: 4.0 scrub ok 2024-09-10T18:23:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:14 smithi204 ceph-mon[32357]: 6.1c scrub starts 2024-09-10T18:23:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:14 smithi204 ceph-mon[32357]: 6.1c scrub ok 2024-09-10T18:23:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:14 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:14 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:14 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:14 smithi204 ceph-mon[32357]: Deploying daemon node-exporter.b on smithi204 2024-09-10T18:23:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:14 smithi204 ceph-mon[32357]: 6.9 scrub starts 2024-09-10T18:23:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:14 smithi204 ceph-mon[32357]: 6.9 scrub ok 2024-09-10T18:23:14.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:14 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:14.355 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:14 smithi204 bash[63159]: Getting image source signatures 2024-09-10T18:23:14.355 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:14 smithi204 bash[63159]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-10T18:23:14.355 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:14 smithi204 bash[63159]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-10T18:23:14.355 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:14 smithi204 bash[63159]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-10T18:23:14.946 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:23:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mon.a", 2024-09-10T18:23:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster": "ceph", 2024-09-10T18:23:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-10T18:23:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "admin_socket_mode": "", 2024-09-10T18:23:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-10T18:23:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_client_required": "cephx, none", 2024-09-10T18:23:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_cluster_required": "cephx", 2024-09-10T18:23:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_debug": "true", 2024-09-10T18:23:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-10T18:23:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-10T18:23:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_service_required": "cephx", 2024-09-10T18:23:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-10T18:23:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_supported": "", 2024-09-10T18:23:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio": "true", 2024-09-10T18:23:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-10T18:23:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio_poll_ms": "250", 2024-09-10T18:23:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio_reap_max": "16", 2024-09-10T18:23:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_async_discard": "false", 2024-09-10T18:23:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_block_size": "4096", 2024-09-10T18:23:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_aio": "false", 2024-09-10T18:23:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-10T18:23:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-10T18:23:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_inflight_ios": "false", 2024-09-10T18:23:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_enable_discard": "false", 2024-09-10T18:23:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_flock_retry": "3", 2024-09-10T18:23:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-10T18:23:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_inject_crash": "0", 2024-09-10T18:23:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-10T18:23:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_ioring": "false", 2024-09-10T18:23:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_ioring_hipri": "false", 2024-09-10T18:23:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-10T18:23:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-10T18:23:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-10T18:23:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-10T18:23:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_type": "", 2024-09-10T18:23:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_alloc_size": "1048576", 2024-09-10T18:23:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_allocator": "hybrid", 2024-09-10T18:23:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_buffered_io": "true", 2024-09-10T18:23:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_check_for_zeros": "false", 2024-09-10T18:23:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-10T18:23:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-10T18:23:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_compact_log_sync": "false", 2024-09-10T18:23:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-10T18:23:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-10T18:23:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-10T18:23:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-10T18:23:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_max_log_runway": "4194304", 2024-09-10T18:23:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_max_prefetch": "1048576", 2024-09-10T18:23:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_min_flush_size": "524288", 2024-09-10T18:23:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_min_log_runway": "1048576", 2024-09-10T18:23:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_replay_recovery": "false", 2024-09-10T18:23:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-10T18:23:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-10T18:23:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_sync_write": "false", 2024-09-10T18:23:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-10T18:23:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-10T18:23:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-10T18:23:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_allocation_from_file": "true", 2024-09-10T18:23:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_allocator": "hybrid", 2024-09-10T18:23:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_async_db_compaction": "true", 2024-09-10T18:23:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-10T18:23:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-10T18:23:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-10T18:23:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-10T18:23:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-10T18:23:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-10T18:23:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-10T18:23:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_create": "true", 2024-09-10T18:23:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_db_create": "false", 2024-09-10T18:23:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_db_path": "", 2024-09-10T18:23:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_db_size": "0", 2024-09-10T18:23:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_path": "", 2024-09-10T18:23:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_preallocate_file": "false", 2024-09-10T18:23:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_size": "107374182400", 2024-09-10T18:23:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_wal_create": "false", 2024-09-10T18:23:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_wal_path": "", 2024-09-10T18:23:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_wal_size": "100663296", 2024-09-10T18:23:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs": "true", 2024-09-10T18:23:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-10T18:23:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-10T18:23:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-10T18:23:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-10T18:23:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-10T18:23:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-10T18:23:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-10T18:23:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-10T18:23:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_autotune": "true", 2024-09-10T18:23:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-10T18:23:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-10T18:23:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-10T18:23:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-10T18:23:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_size": "0", 2024-09-10T18:23:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-10T18:23:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-10T18:23:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-10T18:23:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-10T18:23:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_type": "2q", 2024-09-10T18:23:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-10T18:23:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_clone_cow": "true", 2024-09-10T18:23:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-10T18:23:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-10T18:23:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-10T18:23:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-10T18:23:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-10T18:23:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-10T18:23:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-10T18:23:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_mode": "none", 2024-09-10T18:23:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-10T18:23:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_csum_type": "crc32c", 2024-09-10T18:23:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-10T18:23:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_freelist": "false", 2024-09-10T18:23:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-10T18:23:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-10T18:23:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_inject_bug21040": "false", 2024-09-10T18:23:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-10T18:23:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-10T18:23:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-10T18:23:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-10T18:23:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-10T18:23:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-10T18:23:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-10T18:23:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_prefill": "0.000000", 2024-09-10T18:23:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-10T18:23:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-10T18:23:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-10T18:23:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_small_allocations": "0", 2024-09-10T18:23:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-10T18:23:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_default_buffered_read": "true", 2024-09-10T18:23:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_default_buffered_write": "false", 2024-09-10T18:23:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-10T18:23:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-10T18:23:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-10T18:23:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-10T18:23:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-10T18:23:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-10T18:23:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-10T18:23:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-10T18:23:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fail_eio": "false", 2024-09-10T18:23:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-10T18:23:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-10T18:23:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-10T18:23:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-10T18:23:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-10T18:23:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-10T18:23:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mount": "false", 2024-09-10T18:23:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-10T18:23:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_umount": "false", 2024-09-10T18:23:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-10T18:23:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-10T18:23:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-10T18:23:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-10T18:23:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-10T18:23:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-10T18:23:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-10T18:23:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-10T18:23:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_ignore_data_csum": "false", 2024-09-10T18:23:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-10T18:23:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-10T18:23:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-10T18:23:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-10T18:23:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_log_op_age": "5.000000", 2024-09-10T18:23:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_alloc_size": "0", 2024-09-10T18:23:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_blob_size": "0", 2024-09-10T18:23:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-10T18:23:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-10T18:23:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-10T18:23:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_deferred_txc": "32", 2024-09-10T18:23:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_min_alloc_size": "0", 2024-09-10T18:23:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-10T18:23:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-10T18:23:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_nid_prealloc": "1024", 2024-09-10T18:23:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-10T18:23:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-10T18:23:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-10T18:23:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-10T18:23:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_retry_disk_reads": "3", 2024-09-10T18:23:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_cf": "true", 2024-09-10T18:23:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-09-10T18:23:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-09-10T18:23:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-10T18:23:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-10T18:23:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-10T18:23:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-10T18:23:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_mem": "512", 2024-09-10T18:23:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-10T18:23:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-10T18:23:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-10T18:23:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-10T18:23:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-10T18:23:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-10T18:23:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-10T18:23:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_tracing": "false", 2024-09-10T18:23:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-10T18:23:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-10T18:23:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-10T18:23:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-10T18:23:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-10T18:23:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-10T18:23:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-10T18:23:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-10T18:23:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-10T18:23:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_zero_block_detection": "false", 2024-09-10T18:23:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-10T18:23:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-10T18:23:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-10T18:23:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-10T18:23:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-10T18:23:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-10T18:23:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-10T18:23:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-10T18:23:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-10T18:23:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-10T18:23:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-10T18:23:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-10T18:23:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-10T18:23:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_cluster_require_signatures": "false", 2024-09-10T18:23:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_cluster_require_version": "2", 2024-09-10T18:23:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_require_signatures": "false", 2024-09-10T18:23:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_require_version": "2", 2024-09-10T18:23:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_service_require_signatures": "false", 2024-09-10T18:23:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_service_require_version": "2", 2024-09-10T18:23:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_sign_messages": "true", 2024-09-10T18:23:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "chdir": "", 2024-09-10T18:23:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "client_acl_type": "", 2024-09-10T18:23:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "client_asio_thread_count": "2", 2024-09-10T18:23:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "client_cache_mid": "0.750000", 2024-09-10T18:23:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "client_cache_size": "16384", 2024-09-10T18:23:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "client_caps_release_delay": "5", 2024-09-10T18:23:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "client_check_pool_perm": "true", 2024-09-10T18:23:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-10T18:23:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "client_debug_force_sync_read": "false", 2024-09-10T18:23:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "client_debug_getattr_caps": "false", 2024-09-10T18:23:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "client_debug_inject_features": "", 2024-09-10T18:23:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "client_debug_inject_tick_delay": "0", 2024-09-10T18:23:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-10T18:23:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "client_die_on_failed_remount": "false", 2024-09-10T18:23:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "client_dirsize_rbytes": "true", 2024-09-10T18:23:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "client_force_lazyio": "false", 2024-09-10T18:23:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "client_fs": "", 2024-09-10T18:23:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-10T18:23:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "client_inject_release_failure": "false", 2024-09-10T18:23:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "client_max_inline_size": "4096", 2024-09-10T18:23:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-10T18:23:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mds_namespace": "", 2024-09-10T18:23:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "client_metadata": "", 2024-09-10T18:23:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mount_gid": "-1", 2024-09-10T18:23:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mount_timeout": "300", 2024-09-10T18:23:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mount_uid": "-1", 2024-09-10T18:23:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mountpoint": "/", 2024-09-10T18:23:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "client_notify_timeout": "10", 2024-09-10T18:23:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc": "true", 2024-09-10T18:23:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_max_dirty": "104857600", 2024-09-10T18:23:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-10T18:23:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_max_objects": "1000", 2024-09-10T18:23:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_size": "209715200", 2024-09-10T18:23:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_target_dirty": "8388608", 2024-09-10T18:23:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_permissions": "true", 2024-09-10T18:23:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_quota": "true", 2024-09-10T18:23:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_quota_df": "true", 2024-09-10T18:23:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_readahead_max_bytes": "0", 2024-09-10T18:23:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_readahead_max_periods": "4", 2024-09-10T18:23:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_readahead_min": "131072", 2024-09-10T18:23:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_reconnect_stale": "false", 2024-09-10T18:23:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_shutdown_timeout": "30", 2024-09-10T18:23:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_snapdir": ".snap", 2024-09-10T18:23:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_tick_interval": "1", 2024-09-10T18:23:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_trace": "", 2024-09-10T18:23:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_try_dentry_invalidate": "false", 2024-09-10T18:23:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "client_use_faked_inos": "false", 2024-09-10T18:23:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "client_use_random_mds": "false", 2024-09-10T18:23:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_graylog": "false", 2024-09-10T18:23:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-10T18:23:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_graylog_port": "12201", 2024-09-10T18:23:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_monitors": "default=true", 2024-09-10T18:23:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_syslog": "false", 2024-09-10T18:23:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-10T18:23:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_syslog_level": "info", 2024-09-10T18:23:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "-", 2024-09-10T18:23:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_network": "", 2024-09-10T18:23:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_network_interface": "", 2024-09-10T18:23:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zlib_isal": "false", 2024-09-10T18:23:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zlib_level": "5", 2024-09-10T18:23:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zlib_winsize": "-15", 2024-09-10T18:23:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zstd_level": "1", 2024-09-10T18:23:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16", 2024-09-10T18:23:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-10T18:23:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_alien_op_num_threads": "6", 2024-09-10T18:23:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-10T18:23:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_osd_obc_lru_size": "10", 2024-09-10T18:23:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-10T18:23:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "", 2024-09-10T18:23:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location_hook": "", 2024-09-10T18:23:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location_hook_timeout": "10", 2024-09-10T18:23:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "daemonize": "false", 2024-09-10T18:23:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_alienstore": "0/5", 2024-09-10T18:23:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_allow_any_pool_priority": "false", 2024-09-10T18:23:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_asok": "1/5", 2024-09-10T18:23:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_asok_assert_abort": "false", 2024-09-10T18:23:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_asserts_on_shutdown": "true", 2024-09-10T18:23:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_auth": "1/5", 2024-09-10T18:23:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_bdev": "1/3", 2024-09-10T18:23:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_bluefs": "1/5", 2024-09-10T18:23:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_bluestore": "1/5", 2024-09-10T18:23:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_buffer": "0/1", 2024-09-10T18:23:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_ceph_exporter": "1/5", 2024-09-10T18:23:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_cephfs_mirror": "0/5", 2024-09-10T18:23:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_cephsqlite": "0/5", 2024-09-10T18:23:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_civetweb": "1/10", 2024-09-10T18:23:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_client": "0/5", 2024-09-10T18:23:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_compressor": "1/5", 2024-09-10T18:23:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_context": "0/1", 2024-09-10T18:23:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_crush": "1/1", 2024-09-10T18:23:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_crypto": "1/5", 2024-09-10T18:23:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_deliberately_leak_memory": "false", 2024-09-10T18:23:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_disable_randomized_ping": "false", 2024-09-10T18:23:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_dpdk": "1/5", 2024-09-10T18:23:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_eventtrace": "1/5", 2024-09-10T18:23:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_filer": "0/1", 2024-09-10T18:23:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_filestore": "1/3", 2024-09-10T18:23:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_finisher": "1/1", 2024-09-10T18:23:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_fuse": "1/5", 2024-09-10T18:23:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_heartbeat_testing_span": "0", 2024-09-10T18:23:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_heartbeatmap": "1/5", 2024-09-10T18:23:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-10T18:23:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_javaclient": "1/5", 2024-09-10T18:23:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_journal": "1/3", 2024-09-10T18:23:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_journaler": "0/5", 2024-09-10T18:23:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_kstore": "1/5", 2024-09-10T18:23:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_leveldb": "4/5", 2024-09-10T18:23:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_lockdep": "0/1", 2024-09-10T18:23:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mclock": "1/5", 2024-09-10T18:23:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds": "1/5", 2024-09-10T18:23:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_balancer": "1/5", 2024-09-10T18:23:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_locker": "1/5", 2024-09-10T18:23:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_log": "1/5", 2024-09-10T18:23:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_log_expire": "1/5", 2024-09-10T18:23:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_migrator": "1/5", 2024-09-10T18:23:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_memdb": "4/5", 2024-09-10T18:23:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mgr": "2/5", 2024-09-10T18:23:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mgrc": "1/5", 2024-09-10T18:23:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mon": "20/20", 2024-09-10T18:23:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_monc": "0/10", 2024-09-10T18:23:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_ms": "1/1", 2024-09-10T18:23:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_none": "0/5", 2024-09-10T18:23:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_objclass": "0/5", 2024-09-10T18:23:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_objectcacher": "0/5", 2024-09-10T18:23:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_objecter": "0/1", 2024-09-10T18:23:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_optracker": "0/5", 2024-09-10T18:23:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_osd": "1/5", 2024-09-10T18:23:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_paxos": "20/20", 2024-09-10T18:23:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_perfcounter": "1/5", 2024-09-10T18:23:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_prioritycache": "1/5", 2024-09-10T18:23:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rados": "0/5", 2024-09-10T18:23:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd": "0/5", 2024-09-10T18:23:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd_mirror": "0/5", 2024-09-10T18:23:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd_pwl": "0/5", 2024-09-10T18:23:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd_replay": "0/5", 2024-09-10T18:23:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_refs": "0/0", 2024-09-10T18:23:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_reserver": "1/1", 2024-09-10T18:23:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw": "1/5", 2024-09-10T18:23:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_datacache": "1/5", 2024-09-10T18:23:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_sync": "1/5", 2024-09-10T18:23:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rocksdb": "4/5", 2024-09-10T18:23:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore": "0/5", 2024-09-10T18:23:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_cache": "0/5", 2024-09-10T18:23:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_cleaner": "0/5", 2024-09-10T18:23:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_device": "0/5", 2024-09-10T18:23:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_journal": "0/5", 2024-09-10T18:23:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_lba": "0/5", 2024-09-10T18:23:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_odata": "0/5", 2024-09-10T18:23:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_omap": "0/5", 2024-09-10T18:23:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_onode": "0/5", 2024-09-10T18:23:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_tm": "0/5", 2024-09-10T18:23:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_striper": "0/1", 2024-09-10T18:23:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_test": "0/5", 2024-09-10T18:23:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_throttle": "1/1", 2024-09-10T18:23:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_timer": "0/1", 2024-09-10T18:23:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_tp": "0/5", 2024-09-10T18:23:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-10T18:23:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "device_failure_prediction_mode": "none", 2024-09-10T18:23:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-10T18:23:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-10T18:23:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_graylog": "false", 2024-09-10T18:23:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_journald": "false", 2024-09-10T18:23:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_stderr": "true", 2024-09-10T18:23:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_syslog": "false", 2024-09-10T18:23:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event_tracing": "false", 2024-09-10T18:23:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_addr": "0.0.0.0", 2024-09-10T18:23:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_http_port": "9926", 2024-09-10T18:23:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_prio_limit": "5", 2024-09-10T18:23:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-10T18:23:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_sort_metrics": "true", 2024-09-10T18:23:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_stats_period": "5", 2024-09-10T18:23:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "fake_statfs_for_testing": "0", 2024-09-10T18:23:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "fatal_signal_handlers": "true", 2024-09-10T18:23:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "filer_max_purge_ops": "10", 2024-09-10T18:23:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "filer_max_truncate_ops": "128", 2024-09-10T18:23:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_apply_finisher_threads": "1", 2024-09-10T18:23:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_blackhole": "false", 2024-09-10T18:23:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_btrfs_clone_range": "true", 2024-09-10T18:23:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_btrfs_snap": "true", 2024-09-10T18:23:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_caller_concurrency": "10", 2024-09-10T18:23:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_collect_device_partition_information": "true", 2024-09-10T18:23:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_commit_timeout": "600.000000", 2024-09-10T18:23:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_inject_read_err": "false", 2024-09-10T18:23:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_omap_check": "false", 2024-09-10T18:23:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-10T18:23:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_verify_split": "false", 2024-09-10T18:23:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_dump_file": "", 2024-09-10T18:23:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-10T18:23:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-10T18:23:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fadvise": "true", 2024-09-10T18:23:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fail_eio": "true", 2024-09-10T18:23:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fd_cache_shards": "16", 2024-09-10T18:23:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fd_cache_size": "128", 2024-09-10T18:23:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fiemap": "false", 2024-09-10T18:23:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fiemap_threshold": "4096", 2024-09-10T18:23:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-10T18:23:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-10T18:23:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_inject_stall": "0", 2024-09-10T18:23:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_journal_parallel": "false", 2024-09-10T18:23:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_journal_trailing": "false", 2024-09-10T18:23:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_journal_writeahead": "false", 2024-09-10T18:23:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_kill_at": "0", 2024-09-10T18:23:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-10T18:23:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-10T18:23:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-10T18:23:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-10T18:23:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-10T18:23:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs": "0", 2024-09-10T18:23:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-10T18:23:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-10T18:23:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-10T18:23:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-10T18:23:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size": "0", 2024-09-10T18:23:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-10T18:23:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-10T18:23:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-10T18:23:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_merge_threshold": "-10", 2024-09-10T18:23:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-10T18:23:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_odsync_write": "false", 2024-09-10T18:23:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_omap_backend": "rocksdb", 2024-09-10T18:23:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_omap_backend_path": "", 2024-09-10T18:23:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-10T18:23:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-10T18:23:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-10T18:23:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_op_thread_timeout": "60", 2024-09-10T18:23:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_op_threads": "2", 2024-09-10T18:23:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_punch_hole": "false", 2024-09-10T18:23:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-10T18:23:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-10T18:23:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-10T18:23:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-10T18:23:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-10T18:23:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-10T18:23:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-10T18:23:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-10T18:23:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-10T18:23:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_ops": "50", 2024-09-10T18:23:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-10T18:23:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_seek_data_hole": "false", 2024-09-10T18:23:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_sloppy_crc": "false", 2024-09-10T18:23:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-10T18:23:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_splice": "false", 2024-09-10T18:23:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_split_multiple": "2", 2024-09-10T18:23:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_split_rand_factor": "20", 2024-09-10T18:23:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_update_to": "1000", 2024-09-10T18:23:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-10T18:23:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-10T18:23:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-10T18:23:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-10T18:23:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-10T18:23:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-10T18:23:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_enable": "true", 2024-09-10T18:23:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-10T18:23:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-10T18:23:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-10T18:23:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-10T18:23:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-10T18:23:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-10T18:23:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_xfs_extsize": "false", 2024-09-10T18:23:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_zfs_snap": "false", 2024-09-10T18:23:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "fio_dir": "/tmp/fio", 2024-09-10T18:23:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "5eababb2-6fa0-11ef-bcea-c7b262605968", 2024-09-10T18:23:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_allow_other": "true", 2024-09-10T18:23:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_atomic_o_trunc": "true", 2024-09-10T18:23:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_big_writes": "true", 2024-09-10T18:23:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_debug": "false", 2024-09-10T18:23:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_default_permissions": "false", 2024-09-10T18:23:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_disable_pagecache": "false", 2024-09-10T18:23:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_max_write": "0", 2024-09-10T18:23:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_multithreaded": "true", 2024-09-10T18:23:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_require_active_mds": "true", 2024-09-10T18:23:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_set_user_groups": "true", 2024-09-10T18:23:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_splice_move": "true", 2024-09-10T18:23:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_splice_read": "true", 2024-09-10T18:23:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_splice_write": "true", 2024-09-10T18:23:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-10T18:23:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_use_invalidate_cb": "true", 2024-09-10T18:23:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-10T18:23:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "gss_target_name": "ceph", 2024-09-10T18:23:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_file": "", 2024-09-10T18:23:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_inject_failure": "0", 2024-09-10T18:23:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_interval": "5", 2024-09-10T18:23:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "host": "smithi028", 2024-09-10T18:23:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-10T18:23:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-10T18:23:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-10T18:23:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-10T18:23:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-10T18:23:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-10T18:23:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-10T18:23:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-10T18:23:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-10T18:23:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-10T18:23:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-10T18:23:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-10T18:23:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-10T18:23:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "inject_early_sigterm": "false", 2024-09-10T18:23:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "jaeger_tracing_enable": "false", 2024-09-10T18:23:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_aio": "true", 2024-09-10T18:23:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_align_min_size": "65536", 2024-09-10T18:23:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_block_align": "true", 2024-09-10T18:23:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_block_size": "4096", 2024-09-10T18:23:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_dio": "true", 2024-09-10T18:23:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_discard": "false", 2024-09-10T18:23:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_force_aio": "false", 2024-09-10T18:23:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_ignore_corruption": "false", 2024-09-10T18:23:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_max_write_bytes": "10485760", 2024-09-10T18:23:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_max_write_entries": "100", 2024-09-10T18:23:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_replay_from": "0", 2024-09-10T18:23:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-10T18:23:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-10T18:23:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-10T18:23:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-10T18:23:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_write_header_frequency": "0", 2024-09-10T18:23:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_zero_on_create": "false", 2024-09-10T18:23:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "journaler_prefetch_periods": "10", 2024-09-10T18:23:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "journaler_prezero_periods": "5", 2024-09-10T18:23:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "journaler_write_head_interval": "15", 2024-09-10T18:23:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "", 2024-09-10T18:23:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "keyfile": "", 2024-09-10T18:23:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-10T18:23:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_backend": "rocksdb", 2024-09-10T18:23:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_default_stripe_size": "65536", 2024-09-10T18:23:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_fsck_on_mount": "false", 2024-09-10T18:23:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-10T18:23:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_max_bytes": "67108864", 2024-09-10T18:23:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_max_ops": "512", 2024-09-10T18:23:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_nid_prealloc": "1024", 2024-09-10T18:23:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_onode_map_size": "1024", 2024-09-10T18:23:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-10T18:23:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_sync_submit_transaction": "false", 2024-09-10T18:23:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_sync_transaction": "false", 2024-09-10T18:23:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_block_size": "65536", 2024-09-10T18:23:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_bloom_size": "0", 2024-09-10T18:23:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_cache_size": "536870912", 2024-09-10T18:23:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_compact_on_mount": "false", 2024-09-10T18:23:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_compression": "false", 2024-09-10T18:23:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_log": "", 2024-09-10T18:23:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_log_to_ceph_log": "true", 2024-09-10T18:23:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_max_open_files": "0", 2024-09-10T18:23:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_paranoid": "false", 2024-09-10T18:23:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_write_buffer_size": "33554432", 2024-09-10T18:23:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "librados_thread_count": "2", 2024-09-10T18:23:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "lockdep": "false", 2024-09-10T18:23:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "lockdep_force_backtrace": "false", 2024-09-10T18:23:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "log_coarse_timestamps": "true", 2024-09-10T18:23:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-10T18:23:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "log_flush_on_exit": "false", 2024-09-10T18:23:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "log_graylog_host": "127.0.0.1", 2024-09-10T18:23:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "log_graylog_port": "12201", 2024-09-10T18:23:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "log_max_new": "1000", 2024-09-10T18:23:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "log_max_recent": "10000", 2024-09-10T18:23:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "log_stderr_prefix": "", 2024-09-10T18:23:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "log_stop_at_utilization": "0.970000", 2024-09-10T18:23:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": "true", 2024-09-10T18:23:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_graylog": "false", 2024-09-10T18:23:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_journald": "false", 2024-09-10T18:23:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_stderr": "false", 2024-09-10T18:23:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_syslog": "false", 2024-09-10T18:23:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "max_rotating_auth_attempts": "10", 2024-09-10T18:23:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-10T18:23:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_action_on_write_error": "1", 2024-09-10T18:23:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_alternate_name_max": "8192", 2024-09-10T18:23:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_asio_thread_count": "2", 2024-09-10T18:23:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_export_pin": "true", 2024-09-10T18:23:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_dirs": "true", 2024-09-10T18:23:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-10T18:23:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_interval": "5", 2024-09-10T18:23:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-10T18:23:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-10T18:23:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_interval": "10", 2024-09-10T18:23:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_max": "-1", 2024-09-10T18:23:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_max_until": "-1", 2024-09-10T18:23:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_merge_size": "50", 2024-09-10T18:23:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_midchunk": "0.300000", 2024-09-10T18:23:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-10T18:23:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_min_start": "0.200000", 2024-09-10T18:23:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_minchunk": "0.001000", 2024-09-10T18:23:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_mode": "0", 2024-09-10T18:23:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_need_max": "1.200000", 2024-09-10T18:23:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_need_min": "0.800000", 2024-09-10T18:23:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-10T18:23:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-10T18:23:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_bits": "3", 2024-09-10T18:23:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-10T18:23:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_size": "10000", 2024-09-10T18:23:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-10T18:23:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_target_decay": "10.000000", 2024-09-10T18:23:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-10T18:23:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_beacon_grace": "15.000000", 2024-09-10T18:23:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_beacon_interval": "4.000000", 2024-09-10T18:23:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-10T18:23:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-10T18:23:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_mid": "0.700000", 2024-09-10T18:23:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_release_free_interval": "10", 2024-09-10T18:23:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_reservation": "0.050000", 2024-09-10T18:23:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-10T18:23:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_trim_interval": "1", 2024-09-10T18:23:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_trim_threshold": "262144", 2024-09-10T18:23:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-10T18:23:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-10T18:23:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-10T18:23:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_client_prealloc_inos": "1000", 2024-09-10T18:23:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-10T18:23:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_connect_bootstrapping": "false", 2024-09-10T18:23:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_damage_table_max_entries": "10000", 2024-09-10T18:23:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-10T18:23:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_auth_pins": "false", 2024-09-10T18:23:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_frag": "false", 2024-09-10T18:23:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_scatterstat": "false", 2024-09-10T18:23:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_subtrees": "false", 2024-09-10T18:23:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_decay_halflife": "5.000000", 2024-09-10T18:23:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_default_dir_hash": "2", 2024-09-10T18:23:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_defer_session_stale": "true", 2024-09-10T18:23:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_deny_all_reconnect": "false", 2024-09-10T18:23:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_keys_per_op": "16384", 2024-09-10T18:23:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_max_commit_size": "10", 2024-09-10T18:23:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_max_entries": "0", 2024-09-10T18:23:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-10T18:23:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-10T18:23:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_on_map": "false", 2024-09-10T18:23:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-10T18:23:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-10T18:23:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_early_reply": "true", 2024-09-10T18:23:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_enable_op_tracker": "true", 2024-09-10T18:23:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_enforce_unique_name": "true", 2024-09-10T18:23:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-10T18:23:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-10T18:23:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_random": "true", 2024-09-10T18:23:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-10T18:23:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-10T18:23:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-10T18:23:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-10T18:23:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-10T18:23:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-10T18:23:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_health_summarize_threshold": "10", 2024-09-10T18:23:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-10T18:23:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-10T18:23:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_health_dummy": "false", 2024-09-10T18:23:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-10T18:23:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_migrator_session_race": "false", 2024-09-10T18:23:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-10T18:23:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-10T18:23:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-10T18:23:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_join_fs": "", 2024-09-10T18:23:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_journal_format": "1", 2024-09-10T18:23:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_create_at": "0", 2024-09-10T18:23:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_export_at": "0", 2024-09-10T18:23:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_import_at": "0", 2024-09-10T18:23:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_journal_at": "0", 2024-09-10T18:23:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_journal_expire_at": "0", 2024-09-10T18:23:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_journal_replay_at": "0", 2024-09-10T18:23:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_link_at": "0", 2024-09-10T18:23:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_mdstable_at": "0", 2024-09-10T18:23:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_openc_at": "0", 2024-09-10T18:23:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_rename_at": "0", 2024-09-10T18:23:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-09-10T18:23:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_events_per_segment": "1024", 2024-09-10T18:23:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_max_events": "-1", 2024-09-10T18:23:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_max_segments": "128", 2024-09-10T18:23:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_pause": "false", 2024-09-10T18:23:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_segment_size": "0", 2024-09-10T18:23:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-10T18:23:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_warn_factor": "2.000000", 2024-09-10T18:23:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_caps_per_client": "1048576", 2024-09-10T18:23:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_completed_flushes": "100000", 2024-09-10T18:23:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_completed_requests": "100000", 2024-09-10T18:23:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_export_size": "20971520", 2024-09-10T18:23:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_file_recover": "32", 2024-09-10T18:23:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_purge_files": "64", 2024-09-10T18:23:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_purge_ops": "8192", 2024-09-10T18:23:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-10T18:23:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-10T18:23:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_snaps_per_dir": "100", 2024-09-10T18:23:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_xattr_pairs_size": "65536", 2024-09-10T18:23:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_metrics_update_interval": "2", 2024-09-10T18:23:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_min_caps_per_client": "100", 2024-09-10T18:23:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_min_caps_working_set": "10000", 2024-09-10T18:23:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-10T18:23:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_numa_node": "-1", 2024-09-10T18:23:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-10T18:23:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_complaint_time": "30.000000", 2024-09-10T18:23:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_duration": "600", 2024-09-10T18:23:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_size": "20", 2024-09-10T18:23:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_slow_op_size": "20", 2024-09-10T18:23:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-10T18:23:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_log_threshold": "5", 2024-09-10T18:23:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_ping_grace": "15", 2024-09-10T18:23:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_ping_interval": "5", 2024-09-10T18:23:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-10T18:23:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-10T18:23:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_max_caps": "30000", 2024-09-10T18:23:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-10T18:23:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-10T18:23:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-10T18:23:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_warning_threshold": "262144", 2024-09-10T18:23:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-10T18:23:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_replay_interval": "1.000000", 2024-09-10T18:23:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-10T18:23:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-10T18:23:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_root_ino_gid": "0", 2024-09-10T18:23:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_root_ino_uid": "0", 2024-09-10T18:23:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-10T18:23:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-10T18:23:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-10T18:23:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-10T18:23:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-10T18:23:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-10T18:23:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-10T18:23:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-10T18:23:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-10T18:23:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-10T18:23:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_shutdown_check": "0", 2024-09-10T18:23:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_skip_ino": "0", 2024-09-10T18:23:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-10T18:23:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_snap_max_uid": "4294967294", 2024-09-10T18:23:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_snap_min_uid": "0", 2024-09-10T18:23:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_snap_rstat": "false", 2024-09-10T18:23:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_standby_replay_damaged": "false", 2024-09-10T18:23:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_symlink_recovery": "true", 2024-09-10T18:23:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-10T18:23:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_thrash_exports": "0", 2024-09-10T18:23:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_thrash_fragments": "0", 2024-09-10T18:23:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_tick_interval": "5.000000", 2024-09-10T18:23:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_valgrind_exit": "false", 2024-09-10T18:23:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_verify_backtrace": "1", 2024-09-10T18:23:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_verify_scatter": "false", 2024-09-10T18:23:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-10T18:23:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_wipe_sessions": "false", 2024-09-10T18:23:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool_debug": "false", 2024-09-10T18:23:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-10T18:23:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_device_bytes": "1073741824", 2024-09-10T18:23:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_page_set": "false", 2024-09-10T18:23:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_page_size": "65536", 2024-09-10T18:23:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_client_bytes": "134217728", 2024-09-10T18:23:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_client_messages": "512", 2024-09-10T18:23:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-10T18:23:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-10T18:23:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-10T18:23:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-10T18:23:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_disabled_modules": "diskprediction_local", 2024-09-10T18:23:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-10T18:23:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_max_pg_num_change": "128", 2024-09-10T18:23:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mds_bytes": "134217728", 2024-09-10T18:23:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mds_messages": "128", 2024-09-10T18:23:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-10T18:23:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mon_bytes": "134217728", 2024-09-10T18:23:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mon_messages": "128", 2024-09-10T18:23:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_osd_bytes": "536870912", 2024-09-10T18:23:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_osd_messages": "8192", 2024-09-10T18:23:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_pool": "true", 2024-09-10T18:23:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-10T18:23:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_standby_modules": "true", 2024-09-10T18:23:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_stats_period": "5", 2024-09-10T18:23:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_stats_threshold": "5", 2024-09-10T18:23:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_tick_period": "2", 2024-09-10T18:23:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-10T18:23:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-10T18:23:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_allow_pool_delete": "true", 2024-09-10T18:23:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_allow_pool_size_one": "false", 2024-09-10T18:23:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_auth_validate_all_caps": "true", 2024-09-10T18:23:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-10T18:23:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-10T18:23:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_bytes": "104857600", 2024-09-10T18:23:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_directed_command_retry": "2", 2024-09-10T18:23:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-10T18:23:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-10T18:23:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-10T18:23:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-10T18:23:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_parallel": "3", 2024-09-10T18:23:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_log_interval": "1.000000", 2024-09-10T18:23:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-10T18:23:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_ping_interval": "10.000000", 2024-09-10T18:23:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-10T18:23:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-10T18:23:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-10T18:23:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-10T18:23:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_level": "debug", 2024-09-10T18:23:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_file": "false", 2024-09-10T18:23:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-10T18:23:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-10T18:23:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-10T18:23:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_journald": "true", 2024-09-10T18:23:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-10T18:23:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-10T18:23:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-10T18:23:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_compact_on_bootstrap": "false", 2024-09-10T18:23:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_compact_on_start": "false", 2024-09-10T18:23:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_compact_on_trim": "true", 2024-09-10T18:23:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-10T18:23:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-10T18:23:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-10T18:23:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cpu_threads": "4", 2024-09-10T18:23:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_crush_min_required_version": "hammer", 2024-09-10T18:23:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_daemon_bytes": "419430400", 2024-09-10T18:23:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-10T18:23:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data_avail_crit": "5", 2024-09-10T18:23:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data_avail_warn": "5", 2024-09-10T18:23:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data_size_warn": "16106127360", 2024-09-10T18:23:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-10T18:23:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-10T18:23:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_dump_json": "false", 2024-09-10T18:23:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-10T18:23:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_dump_transactions": "false", 2024-09-10T18:23:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_extra_checks": "false", 2024-09-10T18:23:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-10T18:23:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-10T18:23:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_require_pacific": "false", 2024-09-10T18:23:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_require_quincy": "false", 2024-09-10T18:23:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-10T18:23:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-10T18:23:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-10T18:23:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_down_mkfs_grace": "60", 2024-09-10T18:23:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_election_default_strategy": "1", 2024-09-10T18:23:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_election_timeout": "5.000000", 2024-09-10T18:23:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-10T18:23:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_elector_ping_divisor": "2", 2024-09-10T18:23:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-10T18:23:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_enable_op_tracker": "true", 2024-09-10T18:23:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_fake_pool_delete": "false", 2024-09-10T18:23:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_force_quorum_join": "false", 2024-09-10T18:23:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_globalid_prealloc": "10000", 2024-09-10T18:23:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_detail_to_clog": "true", 2024-09-10T18:23:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_log_update_period": "5", 2024-09-10T18:23:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_max_detail": "50", 2024-09-10T18:23:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_to_clog": "true", 2024-09-10T18:23:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_to_clog_interval": "600", 2024-09-10T18:23:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-10T18:23:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_host": "172.21.15.28:6789/0", 2024-09-10T18:23:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_host_override": "", 2024-09-10T18:23:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_initial_members": "", 2024-09-10T18:23:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-10T18:23:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-10T18:23:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-10T18:23:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-10T18:23:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-10T18:23:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_lease": "5.000000", 2024-09-10T18:23:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-10T18:23:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-10T18:23:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_log_full_interval": "50", 2024-09-10T18:23:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_log_max": "10000", 2024-09-10T18:23:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_log_max_summary": "50", 2024-09-10T18:23:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-10T18:23:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_log_epochs": "500", 2024-09-10T18:23:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-10T18:23:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-10T18:23:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_osd": "10000", 2024-09-10T18:23:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_pg_per_osd": "10000", 2024-09-10T18:23:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_pool_pg_num": "65536", 2024-09-10T18:23:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-10T18:23:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-10T18:23:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mds_force_trim_to": "0", 2024-09-10T18:23:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mds_skip_sanity": "false", 2024-09-10T18:23:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_memory_autotune": "true", 2024-09-10T18:23:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_memory_target": "2147483648", 2024-09-10T18:23:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_beacon_grace": "30", 2024-09-10T18:23:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-10T18:23:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_digest_period": "5", 2024-09-10T18:23:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_inactive_grace": "60", 2024-09-10T18:23:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-10T18:23:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-10T18:23:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_min_osdmap_epochs": "50", 2024-09-10T18:23:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_complaint_time": "30", 2024-09-10T18:23:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_duration": "600", 2024-09-10T18:23:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_size": "20", 2024-09-10T18:23:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_slow_op_size": "20", 2024-09-10T18:23:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-10T18:23:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_log_threshold": "5", 2024-09-10T18:23:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-10T18:23:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-10T18:23:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-10T18:23:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_auto_mark_in": "false", 2024-09-10T18:23:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-10T18:23:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-10T18:23:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-10T18:23:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_cache_size": "500", 2024-09-10T18:23:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-10T18:23:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-10T18:23:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-10T18:23:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_down_out_interval": "600", 2024-09-10T18:23:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-10T18:23:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-10T18:23:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_force_trim_to": "0", 2024-09-10T18:23:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-10T18:23:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-10T18:23:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-10T18:23:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-10T18:23:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-10T18:23:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-10T18:23:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-10T18:23:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-10T18:23:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_min_down_reporters": "2", 2024-09-10T18:23:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-10T18:23:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-10T18:23:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-10T18:23:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-10T18:23:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-10T18:23:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-10T18:23:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_report_timeout": "900", 2024-09-10T18:23:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-10T18:23:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-10T18:23:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-10T18:23:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-10T18:23:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-10T18:23:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-10T18:23:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-10T18:23:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-10T18:23:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-10T18:23:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_stuck_threshold": "60", 2024-09-10T18:23:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-10T18:23:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-10T18:23:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-10T18:23:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-10T18:23:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-10T18:23:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-10T18:23:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_probe_timeout": "2.000000", 2024-09-10T18:23:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_max_change": "0.050000", 2024-09-10T18:23:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_max_osds": "4", 2024-09-10T18:23:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-10T18:23:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-10T18:23:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-10T18:23:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-10T18:23:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-10T18:23:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_interval": "86400", 2024-09-10T18:23:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_max_keys": "100", 2024-09-10T18:23:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_timeout": "300", 2024-09-10T18:23:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_session_timeout": "300", 2024-09-10T18:23:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_smart_report_timeout": "5", 2024-09-10T18:23:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stat_smooth_intervals": "6", 2024-09-10T18:23:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-10T18:23:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_pool_min_size": "2", 2024-09-10T18:23:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_pool_size": "4", 2024-09-10T18:23:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-10T18:23:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-10T18:23:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_debug": "false", 2024-09-10T18:23:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-10T18:23:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-10T18:23:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_provider_kill_at": "0", 2024-09-10T18:23:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_requester_kill_at": "0", 2024-09-10T18:23:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_timeout": "60.000000", 2024-09-10T18:23:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_target_pg_per_osd": "100", 2024-09-10T18:23:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_tick_interval": "5", 2024-09-10T18:23:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_timecheck_interval": "300.000000", 2024-09-10T18:23:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-10T18:23:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_older_version_delay": "604800", 2024-09-10T18:23:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-10T18:23:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-10T18:23:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-10T18:23:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-10T18:23:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-10T18:23:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-10T18:23:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-10T18:23:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_misplaced": "false", 2024-09-10T18:23:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-10T18:23:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_older_version": "true", 2024-09-10T18:23:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-10T18:23:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-10T18:23:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-10T18:23:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-10T18:23:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-10T18:23:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-10T18:23:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-10T18:23:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-10T18:23:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-10T18:23:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": "", 2024-09-10T18:23:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_op_threads": "3", 2024-09-10T18:23:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-10T18:23:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_cm": "false", 2024-09-10T18:23:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_device_name": "", 2024-09-10T18:23:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_dscp": "96", 2024-09-10T18:23:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-10T18:23:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-10T18:23:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_local_gid": "", 2024-09-10T18:23:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-10T18:23:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_port_num": "1", 2024-09-10T18:23:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-10T18:23:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-10T18:23:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-10T18:23:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-10T18:23:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_sl": "3", 2024-09-10T18:23:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_support_srq": "true", 2024-09-10T18:23:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_type": "ib", 2024-09-10T18:23:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_reap_threshold": "5", 2024-09-10T18:23:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_before_connect": "false", 2024-09-10T18:23:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_ipv4": "true", 2024-09-10T18:23:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_ipv6": "false", 2024-09-10T18:23:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_msgr1": "true", 2024-09-10T18:23:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_msgr2": "false", 2024-09-10T18:23:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_port_max": "7568", 2024-09-10T18:23:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_port_min": "6800", 2024-09-10T18:23:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-10T18:23:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_retry_count": "3", 2024-09-10T18:23:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_retry_delay": "5", 2024-09-10T18:23:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_client": "false", 2024-09-10T18:23:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_mds": "false", 2024-09-10T18:23:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_mgr": "false", 2024-09-10T18:23:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_mon": "false", 2024-09-10T18:23:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_osd": "false", 2024-09-10T18:23:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_client_mode": "crc secure", 2024-09-10T18:23:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_cluster_mode": "crc secure", 2024-09-10T18:23:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_cluster_type": "", 2024-09-10T18:23:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_compress_secure": "false", 2024-09-10T18:23:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_connection_idle_timeout": "900", 2024-09-10T18:23:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_connection_ready_timeout": "10", 2024-09-10T18:23:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_crc_data": "true", 2024-09-10T18:23:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_crc_header": "true", 2024-09-10T18:23:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_bad_msg": "false", 2024-09-10T18:23:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_bug": "true", 2024-09-10T18:23:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_old_message": "true", 2024-09-10T18:23:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_skipped_message": "false", 2024-09-10T18:23:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-10T18:23:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-10T18:23:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_coremask": "0xF", 2024-09-10T18:23:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-10T18:23:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-10T18:23:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_enable_tso": "true", 2024-09-10T18:23:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-10T18:23:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-10T18:23:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_hugepages": "", 2024-09-10T18:23:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-10T18:23:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-10T18:23:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_lro": "true", 2024-09-10T18:23:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_memory_channel": "4", 2024-09-10T18:23:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-10T18:23:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_pmd": "", 2024-09-10T18:23:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_port_id": "0", 2024-09-10T18:23:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-10T18:23:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-10T18:23:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dump_on_send": "false", 2024-09-10T18:23:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_initial_backoff": "0.200000", 2024-09-10T18:23:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_delay_max": "1.000000", 2024-09-10T18:23:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-10T18:23:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_delay_type": "", 2024-09-10T18:23:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-10T18:23:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_socket_failures": "0", 2024-09-10T18:23:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_learn_addr_from_peer": "true", 2024-09-10T18:23:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_max_accept_failures": "4", 2024-09-10T18:23:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_max_backoff": "15.000000", 2024-09-10T18:23:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_mon_client_mode": "secure crc", 2024-09-10T18:23:15.027 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-10T18:23:15.027 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_mon_service_mode": "secure crc", 2024-09-10T18:23:15.027 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_osd_compress_min_size": "1024", 2024-09-10T18:23:15.027 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_osd_compress_mode": "none", 2024-09-10T18:23:15.027 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_osd_compression_algorithm": "snappy zlib zstd lz4", 2024-09-10T18:23:15.027 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-10T18:23:15.027 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_pq_min_cost": "65536", 2024-09-10T18:23:15.027 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_public_type": "", 2024-09-10T18:23:15.027 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_service_mode": "crc secure", 2024-09-10T18:23:15.027 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_listen_backlog": "512", 2024-09-10T18:23:15.027 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_nodelay": "true", 2024-09-10T18:23:15.027 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-10T18:23:15.028 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_rcvbuf": "0", 2024-09-10T18:23:15.028 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_type": "async+posix", 2024-09-10T18:23:15.028 INFO:teuthology.orchestra.run.smithi028.stdout: "no_config_file": "false", 2024-09-10T18:23:15.028 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_completion_locks_per_session": "32", 2024-09-10T18:23:15.028 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-10T18:23:15.028 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-10T18:23:15.028 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_inflight_ops": "1024", 2024-09-10T18:23:15.028 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-10T18:23:15.028 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-10T18:23:15.028 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_tick_interval": "5.000000", 2024-09-10T18:23:15.028 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_timeout": "10.000000", 2024-09-10T18:23:15.028 INFO:teuthology.orchestra.run.smithi028.stdout: "objectstore_blackhole": "false", 2024-09-10T18:23:15.028 INFO:teuthology.orchestra.run.smithi028.stdout: "openssl_engine_opts": "", 2024-09-10T18:23:15.029 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_delay_time": "5.000000", 2024-09-10T18:23:15.029 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_hist_halflife": "1000", 2024-09-10T18:23:15.029 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_max_low_ops": "2", 2024-09-10T18:23:15.029 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_max_ops": "4", 2024-09-10T18:23:15.029 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-10T18:23:15.029 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-10T18:23:15.029 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_slop": "0.020000", 2024-09-10T18:23:15.029 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-10T18:23:15.029 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-10T18:23:15.029 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_asio_thread_count": "2", 2024-09-10T18:23:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_async_recovery_min_cost": "100", 2024-09-10T18:23:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-10T18:23:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-10T18:23:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backfill_scan_max": "512", 2024-09-10T18:23:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backfill_scan_min": "64", 2024-09-10T18:23:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backoff_on_degraded": "false", 2024-09-10T18:23:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backoff_on_peering": "false", 2024-09-10T18:23:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backoff_on_unfound": "true", 2024-09-10T18:23:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_beacon_report_interval": "300", 2024-09-10T18:23:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_duration": "30", 2024-09-10T18:23:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-10T18:23:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_max_block_size": "67108864", 2024-09-10T18:23:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-10T18:23:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_blkin_trace_all": "false", 2024-09-10T18:23:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-10T18:23:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-10T18:23:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_check_for_log_corruption": "false", 2024-09-10T18:23:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-10T18:23:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-10T18:23:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-10T18:23:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-10T18:23:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_update_on_start": "true", 2024-09-10T18:23:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_message_cap": "256", 2024-09-10T18:23:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_message_size_cap": "524288000", 2024-09-10T18:23:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_op_priority": "63", 2024-09-10T18:23:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_watch_timeout": "30", 2024-09-10T18:23:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_command_max_records": "256", 2024-09-10T18:23:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_compact_on_start": "false", 2024-09-10T18:23:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-10T18:23:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-10T18:23:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-10T18:23:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_update_on_start": "true", 2024-09-10T18:23:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_update_weight_set": "true", 2024-09-10T18:23:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-10T18:23:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-10T18:23:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-10T18:23:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-10T18:23:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-10T18:23:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_feed_pullee": "-1", 2024-09-10T18:23:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-10T18:23:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-10T18:23:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-10T18:23:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_misdirected_ops": "false", 2024-09-10T18:23:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_no_acting_change": "false", 2024-09-10T18:23:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_no_purge_strays": "false", 2024-09-10T18:23:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_op_order": "false", 2024-09-10T18:23:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-10T18:23:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-10T18:23:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-10T18:23:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-10T18:23:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_shutdown": "false", 2024-09-10T18:23:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-10T18:23:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-10T18:23:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-10T18:23:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-10T18:23:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_snaps": "false", 2024-09-10T18:23:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-10T18:23:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-10T18:23:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_keys": "1024", 2024-09-10T18:23:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-10T18:23:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-10T18:23:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-10T18:23:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_stride": "524288", 2024-09-10T18:23:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-10T18:23:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-10T18:23:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_default_notify_timeout": "30", 2024-09-10T18:23:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep": "0.000000", 2024-09-10T18:23:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-10T18:23:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-10T18:23:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-10T18:23:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_discard_disconnected_ops": "true", 2024-09-10T18:23:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_enable_op_tracker": "true", 2024-09-10T18:23:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-10T18:23:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-10T18:23:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-10T18:23:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_info": "true", 2024-09-10T18:23:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_shutdown": "true", 2024-09-10T18:23:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-10T18:23:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-10T18:23:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-10T18:23:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-10T18:23:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-10T18:23:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_function_tracing": "false", 2024-09-10T18:23:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_grace": "20", 2024-09-10T18:23:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_interval": "6", 2024-09-10T18:23:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-10T18:23:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_min_peers": "10", 2024-09-10T18:23:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_min_size": "2000", 2024-09-10T18:23:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_stale": "600", 2024-09-10T18:23:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-10T18:23:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_hit_set_max_size": "100000", 2024-09-10T18:23:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_hit_set_min_size": "1000", 2024-09-10T18:23:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-10T18:23:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-10T18:23:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-10T18:23:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-10T18:23:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-10T18:23:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-10T18:23:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_journal_size": "5120", 2024-09-10T18:23:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_kill_backfill_at": "0", 2024-09-10T18:23:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-10T18:23:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_cache_size": "50", 2024-09-10T18:23:15.038 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_dedup": "true", 2024-09-10T18:23:15.038 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_message_max": "40", 2024-09-10T18:23:15.038 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-10T18:23:15.038 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_share_max_epochs": "40", 2024-09-10T18:23:15.038 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_attr_name_len": "100", 2024-09-10T18:23:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_attr_size": "0", 2024-09-10T18:23:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_backfills": "1", 2024-09-10T18:23:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_markdown_count": "5", 2024-09-10T18:23:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_markdown_period": "600", 2024-09-10T18:23:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_object_name_len": "2048", 2024-09-10T18:23:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_object_namespace_len": "256", 2024-09-10T18:23:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_object_size": "134217728", 2024-09-10T18:23:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-10T18:23:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-10T18:23:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pg_blocked_by": "16", 2024-09-10T18:23:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pg_log_entries": "10000", 2024-09-10T18:23:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-10T18:23:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pgls": "1024", 2024-09-10T18:23:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_push_cost": "8388608", 2024-09-10T18:23:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_push_objects": "10", 2024-09-10T18:23:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_scrubs": "3", 2024-09-10T18:23:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-10T18:23:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_trimming_pgs": "2", 2024-09-10T18:23:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_write_op_reply_len": "64", 2024-09-10T18:23:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_write_size": "90", 2024-09-10T18:23:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-10T18:23:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-10T18:23:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-10T18:23:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-10T18:23:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-10T18:23:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-10T18:23:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-10T18:23:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-10T18:23:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_profile": "balanced", 2024-09-10T18:23:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-10T18:23:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-10T18:23:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-10T18:23:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-10T18:23:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-10T18:23:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-10T18:23:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-10T18:23:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-10T18:23:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-10T18:23:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-10T18:23:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-10T18:23:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_base": "805306368", 2024-09-10T18:23:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_cache_min": "134217728", 2024-09-10T18:23:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-10T18:23:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-10T18:23:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_target": "4294967296", 2024-09-10T18:23:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_target_autotune": "false", 2024-09-10T18:23:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-10T18:23:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_min_pg_log_entries": "250", 2024-09-10T18:23:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_min_recovery_priority": "0", 2024-09-10T18:23:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-10T18:23:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-10T18:23:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_report_interval": "5", 2024-09-10T18:23:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-10T18:23:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-10T18:23:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_num_cache_shards": "32", 2024-09-10T18:23:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_num_op_tracker_shard": "32", 2024-09-10T18:23:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_numa_auto_affinity": "true", 2024-09-10T18:23:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_numa_node": "-1", 2024-09-10T18:23:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_numa_prefer_iface": "true", 2024-09-10T18:23:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-10T18:23:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objecter_finishers": "1", 2024-09-10T18:23:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objectstore": "bluestore", 2024-09-10T18:23:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objectstore_fuse": "false", 2024-09-10T18:23:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objectstore_tracing": "false", 2024-09-10T18:23:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_complaint_time": "30.000000", 2024-09-10T18:23:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_duration": "600", 2024-09-10T18:23:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_size": "20", 2024-09-10T18:23:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_slow_op_size": "20", 2024-09-10T18:23:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-10T18:23:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_log_threshold": "5", 2024-09-10T18:23:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_shards": "0", 2024-09-10T18:23:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_shards_hdd": "5", 2024-09-10T18:23:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_shards_ssd": "8", 2024-09-10T18:23:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-10T18:23:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-10T18:23:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-10T18:23:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-10T18:23:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_pq_min_cost": "65536", 2024-09-10T18:23:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-10T18:23:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_queue_cut_off": "high", 2024-09-10T18:23:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-10T18:23:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_thread_timeout": "15", 2024-09-10T18:23:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_open_classes_on_start": "true", 2024-09-10T18:23:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_os_flags": "0", 2024-09-10T18:23:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_peering_op_priority": "255", 2024-09-10T18:23:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_delete_cost": "1048576", 2024-09-10T18:23:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_delete_priority": "5", 2024-09-10T18:23:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-10T18:23:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-10T18:23:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-10T18:23:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_log_trim_max": "10000", 2024-09-10T18:23:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_log_trim_min": "100", 2024-09-10T18:23:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-10T18:23:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-10T18:23:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_stat_report_interval_max": "500", 2024-09-10T18:23:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-10T18:23:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-10T18:23:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-10T18:23:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-10T18:23:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-10T18:23:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-10T18:23:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-10T18:23:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-10T18:23:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-10T18:23:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-10T18:23:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-10T18:23:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-10T18:23:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-10T18:23:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-10T18:23:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flags": "0", 2024-09-10T18:23:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-10T18:23:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_min_size": "0", 2024-09-10T18:23:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-10T18:23:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_pg_num": "32", 2024-09-10T18:23:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_pgp_num": "0", 2024-09-10T18:23:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-10T18:23:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_size": "3", 2024-09-10T18:23:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_type": "replicated", 2024-09-10T18:23:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-10T18:23:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-10T18:23:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_push_per_object_cost": "1000", 2024-09-10T18:23:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-10T18:23:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recover_clone_overlap": "true", 2024-09-10T18:23:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-10T18:23:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_cost": "20971520", 2024-09-10T18:23:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-10T18:23:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_active": "0", 2024-09-10T18:23:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-10T18:23:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-10T18:23:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-10T18:23:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-10T18:23:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_single_start": "1", 2024-09-10T18:23:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_op_priority": "3", 2024-09-10T18:23:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-10T18:23:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_priority": "5", 2024-09-10T18:23:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-10T18:23:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep": "0.000000", 2024-09-10T18:23:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-10T18:23:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-10T18:23:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-10T18:23:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_repair_during_recovery": "false", 2024-09-10T18:23:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_requested_scrub_priority": "120", 2024-09-10T18:23:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-10T18:23:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-10T18:23:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_auto_repair": "false", 2024-09-10T18:23:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-10T18:23:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-10T18:23:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_begin_hour": "0", 2024-09-10T18:23:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_begin_week_day": "0", 2024-09-10T18:23:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_chunk_max": "25", 2024-09-10T18:23:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_chunk_min": "5", 2024-09-10T18:23:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_cost": "52428800", 2024-09-10T18:23:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_during_recovery": "false", 2024-09-10T18:23:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_end_hour": "0", 2024-09-10T18:23:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_end_week_day": "0", 2024-09-10T18:23:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_event_cost": "4096", 2024-09-10T18:23:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-10T18:23:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-10T18:23:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_invalid_stats": "true", 2024-09-10T18:23:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-10T18:23:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-10T18:23:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_max_preemptions": "5", 2024-09-10T18:23:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-10T18:23:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_priority": "5", 2024-09-10T18:23:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_sleep": "0.000000", 2024-09-10T18:23:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-10T18:23:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_skip_data_digest": "false", 2024-09-10T18:23:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_smart_report_timeout": "5", 2024-09-10T18:23:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_cost": "1048576", 2024-09-10T18:23:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_priority": "5", 2024-09-10T18:23:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-10T18:23:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-10T18:23:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-10T18:23:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-10T18:23:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-10T18:23:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_target_transaction_size": "30", 2024-09-10T18:23:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-10T18:23:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-10T18:23:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-10T18:23:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-10T18:23:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-10T18:23:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-10T18:23:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-10T18:23:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-10T18:23:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-10T18:23:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-10T18:23:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tracing": "false", 2024-09-10T18:23:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_use_stale_snap": "false", 2024-09-10T18:23:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-10T18:23:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "osdc_blkin_trace_all": "false", 2024-09-10T18:23:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_kill_at": "0", 2024-09-10T18:23:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_max_join_drift": "10", 2024-09-10T18:23:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_min": "500", 2024-09-10T18:23:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_min_wait": "0.050000", 2024-09-10T18:23:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_propose_interval": "1.000000", 2024-09-10T18:23:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_service_trim_max": "500", 2024-09-10T18:23:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-10T18:23:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_service_trim_min": "10", 2024-09-10T18:23:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_stash_full_interval": "25", 2024-09-10T18:23:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_trim_max": "500", 2024-09-10T18:23:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_trim_min": "250", 2024-09-10T18:23:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "perf": "true", 2024-09-10T18:23:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "pid_file": "", 2024-09-10T18:23:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-10T18:23:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-10T18:23:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "-", 2024-09-10T18:23:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrv": "", 2024-09-10T18:23:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "public_bind_addr": "-", 2024-09-10T18:23:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "public_network": "", 2024-09-10T18:23:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "public_network_interface": "", 2024-09-10T18:23:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "qat_compressor_enabled": "false", 2024-09-10T18:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rados_mon_op_timeout": "0", 2024-09-10T18:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rados_osd_op_timeout": "0", 2024-09-10T18:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rados_tracing": "false", 2024-09-10T18:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_atime_update_interval": "60", 2024-09-10T18:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-10T18:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_balance_parent_reads": "false", 2024-09-10T18:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_balance_snap_reads": "false", 2024-09-10T18:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_blkin_trace_all": "false", 2024-09-10T18:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-10T18:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-10T18:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache": "true", 2024-09-10T18:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-10T18:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-10T18:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-10T18:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-10T18:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_policy": "writearound", 2024-09-10T18:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_size": "33554432", 2024-09-10T18:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-10T18:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-10T18:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_clone_copy_on_read": "false", 2024-09-10T18:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_compression_hint": "none", 2024-09-10T18:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_concurrent_management_ops": "10", 2024-09-10T18:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-10T18:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_clone_format": "auto", 2024-09-10T18:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_data_pool": "", 2024-09-10T18:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_features": "61", 2024-09-10T18:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_format": "2", 2024-09-10T18:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_map_options": "", 2024-09-10T18:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_order": "22", 2024-09-10T18:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_pool": "rbd", 2024-09-10T18:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-10T18:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_stripe_count": "0", 2024-09-10T18:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_stripe_unit": "0", 2024-09-10T18:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-10T18:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-10T18:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-10T18:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_enable_alloc_hint": "true", 2024-09-10T18:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-10T18:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_io_scheduler": "simple", 2024-09-10T18:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-10T18:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-10T18:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-10T18:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-10T18:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-10T18:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-10T18:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-10T18:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-10T18:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-10T18:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_order": "24", 2024-09-10T18:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_pool": "", 2024-09-10T18:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_splay_width": "4", 2024-09-10T18:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_localize_parent_reads": "false", 2024-09-10T18:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_localize_snap_reads": "false", 2024-09-10T18:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-10T18:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-10T18:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-10T18:23:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-10T18:23:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-10T18:23:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-10T18:23:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-10T18:23:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-10T18:23:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-10T18:23:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-10T18:23:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-10T18:23:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-10T18:23:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-10T18:23:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-10T18:23:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-10T18:23:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-10T18:23:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-10T18:23:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-10T18:23:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-10T18:23:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-10T18:23:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-10T18:23:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-10T18:23:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-10T18:23:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-10T18:23:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-10T18:23:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-10T18:23:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-10T18:23:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-10T18:23:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-10T18:23:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-10T18:23:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-10T18:23:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mtime_update_interval": "60", 2024-09-10T18:23:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_non_blocking_aio": "true", 2024-09-10T18:23:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_op_thread_timeout": "60", 2024-09-10T18:23:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_op_threads": "1", 2024-09-10T18:23:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_parent_cache_enabled": "false", 2024-09-10T18:23:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-10T18:23:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-10T18:23:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-10T18:23:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_plugins": "", 2024-09-10T18:23:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_bps_burst": "0", 2024-09-10T18:23:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-10T18:23:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_bps_limit": "0", 2024-09-10T18:23:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_exclude_ops": "0", 2024-09-10T18:23:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_iops_burst": "0", 2024-09-10T18:23:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-10T18:23:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_iops_limit": "0", 2024-09-10T18:23:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-10T18:23:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-10T18:23:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-10T18:23:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-10T18:23:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-10T18:23:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-10T18:23:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-10T18:23:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-10T18:23:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-10T18:23:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-10T18:23:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-10T18:23:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-10T18:23:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-10T18:23:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-10T18:23:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_read_from_replica_policy": "default", 2024-09-10T18:23:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-10T18:23:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-10T18:23:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-10T18:23:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-10T18:23:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_skip_partial_discard": "true", 2024-09-10T18:23:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-10T18:23:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_tracing": "false", 2024-09-10T18:23:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_validate_names": "true", 2024-09-10T18:23:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_validate_pool": "true", 2024-09-10T18:23:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "restapi_base_url": "", 2024-09-10T18:23:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "restapi_log_level": "", 2024-09-10T18:23:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_acl_grants_max_num": "100", 2024-09-10T18:23:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_admin_entry": "admin", 2024-09-10T18:23:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_backend_store": "rados", 2024-09-10T18:23:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_barbican_url": "", 2024-09-10T18:23:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_beast_enable_async": "true", 2024-09-10T18:23:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-10T18:23:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-10T18:23:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-10T18:23:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-10T18:23:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-10T18:23:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cache_enabled": "true", 2024-09-10T18:23:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cache_expiry_interval": "900", 2024-09-10T18:23:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cache_lru_size": "10000", 2024-09-10T18:23:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_content_length_compat": "false", 2024-09-10T18:23:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_copy_obj_progress": "true", 2024-09-10T18:23:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-10T18:23:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cors_rules_max_num": "100", 2024-09-10T18:23:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cross_domain_policy": "", 2024-09-10T18:23:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-10T18:23:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_addr": "", 2024-09-10T18:23:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-10T18:23:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-10T18:23:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-10T18:23:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-10T18:23:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_password": "", 2024-09-10T18:23:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-10T18:23:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_username": "", 2024-09-10T18:23:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_require_ssl": "true", 2024-09-10T18:23:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-10T18:23:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-10T18:23:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-10T18:23:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-10T18:23:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-10T18:23:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-10T18:23:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-10T18:23:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-10T18:23:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-10T18:23:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-10T18:23:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-10T18:23:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-10T18:23:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-10T18:23:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-10T18:23:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-10T18:23:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_addr": "", 2024-09-10T18:23:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_auth": "token", 2024-09-10T18:23:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_namespace": "", 2024-09-10T18:23:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_prefix": "", 2024-09-10T18:23:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-10T18:23:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-10T18:23:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-10T18:23:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-10T18:23:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_token_file": "", 2024-09-10T18:23:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-10T18:23:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_buffersize": "524288", 2024-09-10T18:23:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-10T18:23:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_low_speed_time": "300", 2024-09-10T18:23:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-10T18:23:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-10T18:23:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-10T18:23:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-10T18:23:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-10T18:23:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-10T18:23:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-10T18:23:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-10T18:23:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-10T18:23:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-10T18:23:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_changes_size": "1000", 2024-09-10T18:23:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_num_shards": "128", 2024-09-10T18:23:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-10T18:23:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_window": "30", 2024-09-10T18:23:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_notify_interval_msec": "200", 2024-09-10T18:23:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-10T18:23:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-10T18:23:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-10T18:23:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-10T18:23:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-10T18:23:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-10T18:23:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-10T18:23:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-10T18:23:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-10T18:23:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-10T18:23:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-10T18:23:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-10T18:23:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-10T18:23:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-10T18:23:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-10T18:23:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-10T18:23:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-10T18:23:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-10T18:23:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-10T18:23:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-10T18:23:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-10T18:23:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-10T18:23:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dns_name": "", 2024-09-10T18:23:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dns_s3website_name": "", 2024-09-10T18:23:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dynamic_resharding": "true", 2024-09-10T18:23:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-10T18:23:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_gc_threads": "true", 2024-09-10T18:23:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_lc_threads": "true", 2024-09-10T18:23:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_ops_log": "false", 2024-09-10T18:23:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_quota_threads": "true", 2024-09-10T18:23:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_static_website": "false", 2024-09-10T18:23:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_usage_log": "false", 2024-09-10T18:23:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enforce_swift_acls": "true", 2024-09-10T18:23:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_exit_timeout_secs": "120", 2024-09-10T18:23:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_expose_bucket": "false", 2024-09-10T18:23:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_extended_http_attrs": "", 2024-09-10T18:23:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-10T18:23:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_frontends": "beast port=7480", 2024-09-10T18:23:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-10T18:23:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_deferred": "50", 2024-09-10T18:23:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-10T18:23:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_objs": "32", 2024-09-10T18:23:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-10T18:23:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-10T18:23:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-10T18:23:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-10T18:23:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_processor_period": "3600", 2024-09-10T18:23:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-10T18:23:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-10T18:23:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-10T18:23:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-10T18:23:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_init_timeout": "300", 2024-09-10T18:23:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-10T18:23:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_kafka_connection_idle": "300", 2024-09-10T18:23:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-10T18:23:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-10T18:23:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-10T18:23:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_domain": "", 2024-09-10T18:23:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_password": "", 2024-09-10T18:23:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_password_path": "", 2024-09-10T18:23:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_project": "", 2024-09-10T18:23:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_tenant": "", 2024-09-10T18:23:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_token": "", 2024-09-10T18:23:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_token_path": "", 2024-09-10T18:23:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_user": "", 2024-09-10T18:23:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_api_version": "2", 2024-09-10T18:23:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_domain": "", 2024-09-10T18:23:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_password": "", 2024-09-10T18:23:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_project": "", 2024-09-10T18:23:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-10T18:23:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_user": "", 2024-09-10T18:23:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-10T18:23:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-10T18:23:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-10T18:23:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-10T18:23:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-10T18:23:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_url": "", 2024-09-10T18:23:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-10T18:23:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_debug_interval": "-1", 2024-09-10T18:23:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_lock_max_time": "90", 2024-09-10T18:23:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_objs": "32", 2024-09-10T18:23:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_rules": "1000", 2024-09-10T18:23:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_worker": "3", 2024-09-10T18:23:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-10T18:23:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_thread_delay": "0", 2024-09-10T18:23:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-10T18:23:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_dnattr": "uid", 2024-09-10T18:23:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-10T18:23:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_searchfilter": "", 2024-09-10T18:23:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-10T18:23:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_uri": "", 2024-09-10T18:23:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-10T18:23:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-10T18:23:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-10T18:23:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_http_headers": "", 2024-09-10T18:23:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-10T18:23:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-10T18:23:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_object_name_utc": "false", 2024-09-10T18:23:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-09-10T18:23:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_attr_name_len": "0", 2024-09-10T18:23:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_attr_size": "0", 2024-09-10T18:23:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-10T18:23:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_chunk_size": "4194304", 2024-09-10T18:23:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-10T18:23:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-10T18:23:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_listing_results": "1000", 2024-09-10T18:23:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_notify_retries": "10", 2024-09-10T18:23:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-10T18:23:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_put_param_size": "1048576", 2024-09-10T18:23:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_put_size": "5368709120", 2024-09-10T18:23:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_slo_entries": "1000", 2024-09-10T18:23:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_md_log_max_shards": "64", 2024-09-10T18:23:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-10T18:23:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-10T18:23:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_mp_lock_max_time": "600", 2024-09-10T18:23:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-10T18:23:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-10T18:23:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-10T18:23:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-10T18:23:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-10T18:23:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-10T18:23:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-10T18:23:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_max_gc": "300", 2024-09-10T18:23:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-10T18:23:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-10T18:23:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-10T18:23:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-10T18:23:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-10T18:23:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-10T18:23:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-10T18:23:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_num_async_rados_threads": "32", 2024-09-10T18:23:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_num_control_oids": "8", 2024-09-10T18:23:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_numa_node": "-1", 2024-09-10T18:23:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-10T18:23:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-10T18:23:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_objexp_chunk_size": "100", 2024-09-10T18:23:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_objexp_gc_interval": "600", 2024-09-10T18:23:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-10T18:23:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-10T18:23:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-10T18:23:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_op_thread_timeout": "600", 2024-09-10T18:23:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_op_tracing": "false", 2024-09-10T18:23:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_opa_token": "", 2024-09-10T18:23:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_opa_url": "", 2024-09-10T18:23:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_opa_verify_ssl": "true", 2024-09-10T18:23:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-10T18:23:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_file_path": "", 2024-09-10T18:23:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_rados": "true", 2024-09-10T18:23:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_socket_path": "", 2024-09-10T18:23:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-10T18:23:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-10T18:23:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_push_interval": "2.000000", 2024-09-10T18:23:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-10T18:23:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-10T18:23:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_print_continue": "true", 2024-09-10T18:23:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-10T18:23:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-10T18:23:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-10T18:23:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-10T18:23:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-10T18:23:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rados_tracing": "false", 2024-09-10T18:23:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_realm": "", 2024-09-10T18:23:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_realm_id": "", 2024-09-10T18:23:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-10T18:23:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_region": "", 2024-09-10T18:23:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-10T18:23:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-10T18:23:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-10T18:23:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-10T18:23:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_request_uri": "", 2024-09-10T18:23:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_batch_size": "64", 2024-09-10T18:23:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-10T18:23:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_max_aio": "128", 2024-09-10T18:23:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_num_logs": "16", 2024-09-10T18:23:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_thread_interval": "600", 2024-09-10T18:23:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_resolve_cname": "false", 2024-09-10T18:23:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-10T18:23:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_run_sync_thread": "true", 2024-09-10T18:23:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-10T18:23:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-10T18:23:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-10T18:23:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-10T18:23:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-10T18:23:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-10T18:23:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-10T18:23:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-10T18:23:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-10T18:23:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_scheduler_type": "throttler", 2024-09-10T18:23:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_script_uri": "", 2024-09-10T18:23:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_service_provider_name": "", 2024-09-10T18:23:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-10T18:23:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_client_id": "", 2024-09-10T18:23:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_client_secret": "", 2024-09-10T18:23:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_entry": "sts", 2024-09-10T18:23:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_key": "sts", 2024-09-10T18:23:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-10T18:23:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_min_session_duration": "900", 2024-09-10T18:23:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_token_introspection_url": "", 2024-09-10T18:23:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_account_in_url": "false", 2024-09-10T18:23:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_auth_entry": "auth", 2024-09-10T18:23:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_auth_url": "", 2024-09-10T18:23:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_custom_header": "", 2024-09-10T18:23:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-10T18:23:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_need_stats": "true", 2024-09-10T18:23:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_tenant_name": "", 2024-09-10T18:23:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_token_expiration": "86400", 2024-09-10T18:23:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_url": "", 2024-09-10T18:23:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_url_prefix": "swift", 2024-09-10T18:23:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-10T18:23:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-10T18:23:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_lease_period": "120", 2024-09-10T18:23:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-10T18:23:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-10T18:23:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-10T18:23:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-10T18:23:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-10T18:23:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-10T18:23:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-10T18:23:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-10T18:23:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-10T18:23:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_thread_pool_size": "512", 2024-09-10T18:23:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_comment": "", 2024-09-10T18:23:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_createby": "", 2024-09-10T18:23:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_encoding": "", 2024-09-10T18:23:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_flag": "false", 2024-09-10T18:23:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_origin": "", 2024-09-10T18:23:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-10T18:23:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_tracker": "", 2024-09-10T18:23:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_trust_forwarded_https": "false", 2024-09-10T18:23:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-10T18:23:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-10T18:23:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_max_shards": "32", 2024-09-10T18:23:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_max_user_shards": "1", 2024-09-10T18:23:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_use_opa_authz": "false", 2024-09-10T18:23:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-10T18:23:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-10T18:23:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_max_buckets": "1000", 2024-09-10T18:23:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-10T18:23:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-10T18:23:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-10T18:23:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-10T18:23:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_unique_email": "true", 2024-09-10T18:23:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_verify_ssl": "true", 2024-09-10T18:23:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-10T18:23:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zone": "", 2024-09-10T18:23:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zone_id": "", 2024-09-10T18:23:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-10T18:23:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zonegroup": "", 2024-09-10T18:23:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zonegroup_id": "", 2024-09-10T18:23:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-10T18:23:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_block_size": "4096", 2024-09-10T18:23:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-10T18:23:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-10T18:23:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-10T18:23:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-10T18:23:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-10T18:23:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_size": "536870912", 2024-09-10T18:23:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-10T18:23:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-10T18:23:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-10T18:23:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-10T18:23:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-10T18:23:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-10T18:23:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-10T18:23:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-10T18:23:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_index_type": "binary_search", 2024-09-10T18:23:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-10T18:23:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-10T18:23:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_partition_filters": "false", 2024-09-10T18:23:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_perf": "false", 2024-09-10T18:23:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-10T18:23:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-10T18:23:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-10T18:23:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "run_dir": "/var/run/ceph", 2024-09-10T18:23:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_block_create": "true", 2024-09-10T18:23:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_cache_lru_size": "67108864", 2024-09-10T18:23:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_default_max_object_size": "16777216", 2024-09-10T18:23:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-10T18:23:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_device_size": "107374182400", 2024-09-10T18:23:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_init_rewrite_segments_num_per_device": "6", 2024-09-10T18:23:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_batch_capacity": "16", 2024-09-10T18:23:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-10T18:23:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-10T18:23:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-10T18:23:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_segment_size": "67108864", 2024-09-10T18:23:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "setgroup": "ceph", 2024-09-10T18:23:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "setuser": "ceph", 2024-09-10T18:23:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "setuser_match_path": "", 2024-09-10T18:23:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-10T18:23:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "thp": "false", 2024-09-10T18:23:15.093 INFO:teuthology.orchestra.run.smithi028.stdout: "threadpool_default_timeout": "60", 2024-09-10T18:23:15.093 INFO:teuthology.orchestra.run.smithi028.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-10T18:23:15.093 INFO:teuthology.orchestra.run.smithi028.stdout: "throttler_perf_counter": "true" 2024-09-10T18:23:15.093 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:23:15.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[29291]: 6.1b scrub starts 2024-09-10T18:23:15.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[29291]: 6.1b scrub ok 2024-09-10T18:23:15.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:23:15.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:23:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[29291]: 4.1a scrub starts 2024-09-10T18:23:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[29291]: 4.1a scrub ok 2024-09-10T18:23:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[29291]: osdmap e91: 8 total, 8 up, 8 in 2024-09-10T18:23:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[29291]: 4.3 scrub starts 2024-09-10T18:23:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[29291]: 4.3 scrub ok 2024-09-10T18:23:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[29291]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 96 B/s, 3 objects/s recovering 2024-09-10T18:23:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[29291]: 5.10 scrub starts 2024-09-10T18:23:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[29291]: 5.10 scrub ok 2024-09-10T18:23:15.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[39220]: 6.1b scrub starts 2024-09-10T18:23:15.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[39220]: 6.1b scrub ok 2024-09-10T18:23:15.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:23:15.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:23:15.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[39220]: 4.1a scrub starts 2024-09-10T18:23:15.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[39220]: 4.1a scrub ok 2024-09-10T18:23:15.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[39220]: osdmap e91: 8 total, 8 up, 8 in 2024-09-10T18:23:15.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[39220]: 4.3 scrub starts 2024-09-10T18:23:15.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[39220]: 4.3 scrub ok 2024-09-10T18:23:15.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[39220]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 96 B/s, 3 objects/s recovering 2024-09-10T18:23:15.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[39220]: 5.10 scrub starts 2024-09-10T18:23:15.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:15 smithi028 ceph-mon[39220]: 5.10 scrub ok 2024-09-10T18:23:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:15 smithi204 ceph-mon[32357]: 6.1b scrub starts 2024-09-10T18:23:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:15 smithi204 ceph-mon[32357]: 6.1b scrub ok 2024-09-10T18:23:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:15 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:23:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:15 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:23:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:15 smithi204 ceph-mon[32357]: 4.1a scrub starts 2024-09-10T18:23:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:15 smithi204 ceph-mon[32357]: 4.1a scrub ok 2024-09-10T18:23:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:15 smithi204 ceph-mon[32357]: osdmap e91: 8 total, 8 up, 8 in 2024-09-10T18:23:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:15 smithi204 ceph-mon[32357]: 4.3 scrub starts 2024-09-10T18:23:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:15 smithi204 ceph-mon[32357]: 4.3 scrub ok 2024-09-10T18:23:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:15 smithi204 ceph-mon[32357]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 96 B/s, 3 objects/s recovering 2024-09-10T18:23:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:15 smithi204 ceph-mon[32357]: 5.10 scrub starts 2024-09-10T18:23:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:15 smithi204 ceph-mon[32357]: 5.10 scrub ok 2024-09-10T18:23:15.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-10T18:23:16.104 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:15 smithi204 bash[63159]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-10T18:23:16.104 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 bash[63159]: Writing manifest to image destination 2024-09-10T18:23:16.156 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:16.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:16 smithi028 ceph-mon[29291]: 4.13 scrub starts 2024-09-10T18:23:16.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:16 smithi028 ceph-mon[29291]: 4.13 scrub ok 2024-09-10T18:23:16.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:16 smithi028 ceph-mon[39220]: 4.13 scrub starts 2024-09-10T18:23:16.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:16 smithi028 ceph-mon[39220]: 4.13 scrub ok 2024-09-10T18:23:16.563 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-mon[32357]: 4.13 scrub starts 2024-09-10T18:23:16.563 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-mon[32357]: 4.13 scrub ok 2024-09-10T18:23:16.564 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 podman[63159]: 2024-09-10 18:23:16.29445861 +0000 UTC m=+2.402406969 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-10T18:23:16.564 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 podman[63159]: 2024-09-10 18:23:16.359652868 +0000 UTC m=+2.467601274 container create 4c2732cbae6f7c897b6115f82ff2d8b57c44c9239f3f32c8ec4ad9b511c9721d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T18:23:16.854 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 podman[63159]: 2024-09-10 18:23:16.562146171 +0000 UTC m=+2.670094532 container init 4c2732cbae6f7c897b6115f82ff2d8b57c44c9239f3f32c8ec4ad9b511c9721d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T18:23:16.855 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 podman[63159]: 2024-09-10 18:23:16.565207756 +0000 UTC m=+2.673156124 container start 4c2732cbae6f7c897b6115f82ff2d8b57c44c9239f3f32c8ec4ad9b511c9721d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T18:23:16.855 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.569Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-10T18:23:16.855 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.569Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-10T18:23:16.855 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.569Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-10T18:23:16.855 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.569Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-10T18:23:16.855 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.569Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-10T18:23:16.855 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-10T18:23:16.855 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-10T18:23:16.855 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=arp 2024-09-10T18:23:16.855 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-10T18:23:16.855 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-10T18:23:16.855 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-10T18:23:16.856 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-10T18:23:16.856 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-10T18:23:16.856 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-10T18:23:16.856 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-10T18:23:16.856 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-10T18:23:16.856 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=edac 2024-09-10T18:23:16.857 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-10T18:23:16.857 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-10T18:23:16.857 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-10T18:23:16.857 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-10T18:23:16.857 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-10T18:23:16.857 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-10T18:23:16.857 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-10T18:23:16.857 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-10T18:23:16.857 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-10T18:23:16.857 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-10T18:23:16.857 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-10T18:23:16.857 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-10T18:23:16.857 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-10T18:23:16.858 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-10T18:23:16.858 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-10T18:23:16.858 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-10T18:23:16.858 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=os 2024-09-10T18:23:16.858 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.570Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-10T18:23:16.858 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-10T18:23:16.858 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-10T18:23:16.858 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-10T18:23:16.858 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-10T18:23:16.858 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-10T18:23:16.858 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-10T18:23:16.858 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=node_exporter.go:117 level=info collector=stat 2024-09-10T18:23:16.858 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-10T18:23:16.858 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-10T18:23:16.858 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-10T18:23:16.859 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=node_exporter.go:117 level=info collector=time 2024-09-10T18:23:16.859 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-10T18:23:16.859 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=node_exporter.go:117 level=info collector=uname 2024-09-10T18:23:16.859 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-10T18:23:16.859 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-10T18:23:16.859 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-10T18:23:16.859 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-10T18:23:16.859 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b[63233]: ts=2024-09-10T18:23:16.571Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-10T18:23:16.859 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 bash[63159]: 4c2732cbae6f7c897b6115f82ff2d8b57c44c9239f3f32c8ec4ad9b511c9721d 2024-09-10T18:23:16.860 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:23:16 smithi204 systemd[1]: Started Ceph node-exporter.b for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:23:17.629 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a: {} 2024-09-10T18:23:17.629 INFO:teuthology.orchestra.run.smithi028.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T18:23:17.636 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b: {} 2024-09-10T18:23:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T18:23:17.644 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c: {} 2024-09-10T18:23:17.645 INFO:teuthology.orchestra.run.smithi028.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T18:23:18.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[29291]: 3.9 scrub starts 2024-09-10T18:23:18.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[29291]: 3.9 scrub ok 2024-09-10T18:23:18.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[29291]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 75 B/s, 3 objects/s recovering 2024-09-10T18:23:18.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:18.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:18.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:18.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:18.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:18.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:23:18.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3192942027' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:23:18.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[39220]: 3.9 scrub starts 2024-09-10T18:23:18.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[39220]: 3.9 scrub ok 2024-09-10T18:23:18.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[39220]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 75 B/s, 3 objects/s recovering 2024-09-10T18:23:18.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:18.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:18.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:18.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:18.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:18.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:23:18.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:17 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3192942027' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:23:18.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:17 smithi204 ceph-mon[32357]: 3.9 scrub starts 2024-09-10T18:23:18.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:17 smithi204 ceph-mon[32357]: 3.9 scrub ok 2024-09-10T18:23:18.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:17 smithi204 ceph-mon[32357]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 75 B/s, 3 objects/s recovering 2024-09-10T18:23:18.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:17 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:18.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:17 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:18.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:17 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:18.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:17 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:18.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:17 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:18.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:17 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:23:18.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:17 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3192942027' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:23:18.512 INFO:teuthology.run_tasks:Running task full_sequential... 2024-09-10T18:23:18.524 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-10T18:23:18.526 INFO:tasks.radosbench:Beginning radosbench... 2024-09-10T18:23:18.526 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-10T18:23:18.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-10T18:23:18.527 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-10T18:23:18.527 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T18:23:18.527 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:23:18.527 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-10T18:23:18.527 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-10T18:23:18.527 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-10T18:23:18.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-10T18:23:18.528 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-10T18:23:18.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:18.529 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-10T18:23:18.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:23:18.529 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-10T18:23:18.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T18:23:18.530 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-10T18:23:18.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:23:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:18.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:18.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:18.663 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:18.685 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:18.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:18.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:18.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:18.790 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:23:18.810 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:18.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:18.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:18.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:18 smithi204 ceph-mon[32357]: 5.1a scrub starts 2024-09-10T18:23:18.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:18 smithi204 ceph-mon[32357]: 5.1a scrub ok 2024-09-10T18:23:18.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:18 smithi204 ceph-mon[32357]: 3.b scrub starts 2024-09-10T18:23:18.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:18 smithi204 ceph-mon[32357]: 3.b scrub ok 2024-09-10T18:23:18.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:18 smithi204 ceph-mon[32357]: 6.19 deep-scrub starts 2024-09-10T18:23:18.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:18 smithi204 ceph-mon[32357]: 6.19 deep-scrub ok 2024-09-10T18:23:18.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:18 smithi028 ceph-mon[29291]: 5.1a scrub starts 2024-09-10T18:23:18.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:18 smithi028 ceph-mon[29291]: 5.1a scrub ok 2024-09-10T18:23:18.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:18 smithi028 ceph-mon[29291]: 3.b scrub starts 2024-09-10T18:23:18.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:18 smithi028 ceph-mon[29291]: 3.b scrub ok 2024-09-10T18:23:18.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:18 smithi028 ceph-mon[29291]: 6.19 deep-scrub starts 2024-09-10T18:23:18.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:18 smithi028 ceph-mon[29291]: 6.19 deep-scrub ok 2024-09-10T18:23:18.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:18 smithi028 ceph-mon[39220]: 5.1a scrub starts 2024-09-10T18:23:18.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:18 smithi028 ceph-mon[39220]: 5.1a scrub ok 2024-09-10T18:23:18.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:18 smithi028 ceph-mon[39220]: 3.b scrub starts 2024-09-10T18:23:18.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:18 smithi028 ceph-mon[39220]: 3.b scrub ok 2024-09-10T18:23:18.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:18 smithi028 ceph-mon[39220]: 6.19 deep-scrub starts 2024-09-10T18:23:18.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:18 smithi028 ceph-mon[39220]: 6.19 deep-scrub ok 2024-09-10T18:23:18.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:18.935 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:18.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:18.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:19.037 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:23:19.059 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:19.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:19.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:23:19.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:19.194 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:19.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:19.296 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:19.317 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:19.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:19.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:19.419 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:19.440 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:19.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:19.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:19.542 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:23:19.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:19.565 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:19.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:23:19.666 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:23:19.688 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:19.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:19.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:23:19.790 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:19.812 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:19.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:19.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:19.914 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:19.936 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:19.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:19.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:19 smithi028 ceph-mon[29291]: 4.e scrub starts 2024-09-10T18:23:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:19 smithi028 ceph-mon[29291]: 4.e scrub ok 2024-09-10T18:23:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:19 smithi028 ceph-mon[29291]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.4 KiB/s rd, 473 B/s wr, 9 op/s; 66 B/s, 2 objects/s recovering 2024-09-10T18:23:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:19 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:19 smithi028 ceph-mon[29291]: 3.5 scrub starts 2024-09-10T18:23:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:19 smithi028 ceph-mon[29291]: 3.5 scrub ok 2024-09-10T18:23:20.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:19 smithi028 ceph-mon[39220]: 4.e scrub starts 2024-09-10T18:23:20.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:19 smithi028 ceph-mon[39220]: 4.e scrub ok 2024-09-10T18:23:20.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:19 smithi028 ceph-mon[39220]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.4 KiB/s rd, 473 B/s wr, 9 op/s; 66 B/s, 2 objects/s recovering 2024-09-10T18:23:20.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:19 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:20.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:19 smithi028 ceph-mon[39220]: 3.5 scrub starts 2024-09-10T18:23:20.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:19 smithi028 ceph-mon[39220]: 3.5 scrub ok 2024-09-10T18:23:20.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:20.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:20.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:20.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:19 smithi204 ceph-mon[32357]: 4.e scrub starts 2024-09-10T18:23:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:19 smithi204 ceph-mon[32357]: 4.e scrub ok 2024-09-10T18:23:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:19 smithi204 ceph-mon[32357]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.4 KiB/s rd, 473 B/s wr, 9 op/s; 66 B/s, 2 objects/s recovering 2024-09-10T18:23:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:19 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:19 smithi204 ceph-mon[32357]: 3.5 scrub starts 2024-09-10T18:23:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:19 smithi204 ceph-mon[32357]: 3.5 scrub ok 2024-09-10T18:23:20.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:20.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:20.186 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:20.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:20.287 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:20.308 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:20.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:20.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:20.410 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:20.431 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:20.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:20.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:20.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:20.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:20.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:20.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:20.682 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:20.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:20.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:20.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:20.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:20.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:20.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:20.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:20.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:20.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:20.945 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:20 smithi204 ceph-mon[32357]: 5.4 scrub starts 2024-09-10T18:23:20.945 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:20 smithi204 ceph-mon[32357]: 5.4 scrub ok 2024-09-10T18:23:20.945 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:20 smithi204 ceph-mon[32357]: 5.16 scrub starts 2024-09-10T18:23:20.945 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:20 smithi204 ceph-mon[32357]: 5.16 scrub ok 2024-09-10T18:23:21.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:21.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:21.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:21.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:20 smithi028 ceph-mon[29291]: 5.4 scrub starts 2024-09-10T18:23:21.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:20 smithi028 ceph-mon[29291]: 5.4 scrub ok 2024-09-10T18:23:21.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:20 smithi028 ceph-mon[29291]: 5.16 scrub starts 2024-09-10T18:23:21.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:20 smithi028 ceph-mon[29291]: 5.16 scrub ok 2024-09-10T18:23:21.079 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:20 smithi028 ceph-mon[39220]: 5.4 scrub starts 2024-09-10T18:23:21.079 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:20 smithi028 ceph-mon[39220]: 5.4 scrub ok 2024-09-10T18:23:21.079 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:20 smithi028 ceph-mon[39220]: 5.16 scrub starts 2024-09-10T18:23:21.080 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:20 smithi028 ceph-mon[39220]: 5.16 scrub ok 2024-09-10T18:23:21.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:21.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:21.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:21.213 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:23:21.213 INFO:teuthology.orchestra.run.smithi204.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-10T18:23:21.213 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:23:21.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:21.304 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:21.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:21.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:21.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:21.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:21.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:21.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:21.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:21.554 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:21.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:21.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:21.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:21.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:21.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:21.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:21.779 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:21.801 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:21.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:21.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:21.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:21.926 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:21.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:21.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:22.019 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:21 smithi204 ceph-mon[32357]: 4.19 scrub starts 2024-09-10T18:23:22.019 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:21 smithi204 ceph-mon[32357]: 4.19 scrub ok 2024-09-10T18:23:22.019 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:21 smithi204 ceph-mon[32357]: 4.16 scrub starts 2024-09-10T18:23:22.019 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:21 smithi204 ceph-mon[32357]: 4.16 scrub ok 2024-09-10T18:23:22.019 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:21 smithi204 ceph-mon[32357]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.3 KiB/s rd, 409 B/s wr, 8 op/s; 57 B/s, 2 objects/s recovering 2024-09-10T18:23:22.028 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:22.050 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:22.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:22.151 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:21 smithi028 ceph-mon[29291]: 4.19 scrub starts 2024-09-10T18:23:22.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:21 smithi028 ceph-mon[29291]: 4.19 scrub ok 2024-09-10T18:23:22.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:21 smithi028 ceph-mon[29291]: 4.16 scrub starts 2024-09-10T18:23:22.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:21 smithi028 ceph-mon[29291]: 4.16 scrub ok 2024-09-10T18:23:22.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:21 smithi028 ceph-mon[29291]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.3 KiB/s rd, 409 B/s wr, 8 op/s; 57 B/s, 2 objects/s recovering 2024-09-10T18:23:22.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:21 smithi028 ceph-mon[39220]: 4.19 scrub starts 2024-09-10T18:23:22.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:21 smithi028 ceph-mon[39220]: 4.19 scrub ok 2024-09-10T18:23:22.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:21 smithi028 ceph-mon[39220]: 4.16 scrub starts 2024-09-10T18:23:22.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:21 smithi028 ceph-mon[39220]: 4.16 scrub ok 2024-09-10T18:23:22.153 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:21 smithi028 ceph-mon[39220]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.3 KiB/s rd, 409 B/s wr, 8 op/s; 57 B/s, 2 objects/s recovering 2024-09-10T18:23:22.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:22.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:22.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:22.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:22.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:22.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:22.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:23:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:23:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:23:22.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:22.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:22.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:22.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:22.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:22.553 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:22.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:22.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:22.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:22.676 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:22.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:22.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:22.779 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:22.799 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:22.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:22.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:22.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:22.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:22.924 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:22.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:23.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:23:23.047 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:23.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:23.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:23:23.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:23:23.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:23:23.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[29291]: osdmap e92: 8 total, 8 up, 8 in 2024-09-10T18:23:23.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[29291]: 3.e scrub starts 2024-09-10T18:23:23.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[29291]: 3.e scrub ok 2024-09-10T18:23:23.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[29291]: 5.f scrub starts 2024-09-10T18:23:23.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[29291]: 5.f scrub ok 2024-09-10T18:23:23.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:23.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:23.092 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:23:23.092 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:23:23.092 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[39220]: osdmap e92: 8 total, 8 up, 8 in 2024-09-10T18:23:23.092 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[39220]: 3.e scrub starts 2024-09-10T18:23:23.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[39220]: 3.e scrub ok 2024-09-10T18:23:23.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[39220]: 5.f scrub starts 2024-09-10T18:23:23.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[39220]: 5.f scrub ok 2024-09-10T18:23:23.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:23.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:22 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:23.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:22 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:23:23.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:22 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:23:23.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:22 smithi204 ceph-mon[32357]: osdmap e92: 8 total, 8 up, 8 in 2024-09-10T18:23:23.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:22 smithi204 ceph-mon[32357]: 3.e scrub starts 2024-09-10T18:23:23.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:22 smithi204 ceph-mon[32357]: 3.e scrub ok 2024-09-10T18:23:23.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:22 smithi204 ceph-mon[32357]: 5.f scrub starts 2024-09-10T18:23:23.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:22 smithi204 ceph-mon[32357]: 5.f scrub ok 2024-09-10T18:23:23.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:22 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:23.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:22 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:23.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:23.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:23.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:23.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:23:23.293 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:23.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:23.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:23:23.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:23.418 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:23.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:23.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:23.520 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:23.542 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:23.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:23.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:23.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:23.666 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:23.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:23.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:23.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:23.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:23.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:23.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:23.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:23.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:24.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:24.041 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:24.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:24.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:24.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:23 smithi204 ceph-mon[32357]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 476 B/s wr, 9 op/s; 67 B/s, 2 objects/s recovering 2024-09-10T18:23:24.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:23 smithi204 ceph-mon[32357]: 6.d scrub starts 2024-09-10T18:23:24.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:23 smithi204 ceph-mon[32357]: 6.d scrub ok 2024-09-10T18:23:24.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:23 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:23:24.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:23:23 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:23:24.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:24.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:23 smithi028 ceph-mon[29291]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 476 B/s wr, 9 op/s; 67 B/s, 2 objects/s recovering 2024-09-10T18:23:24.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:23 smithi028 ceph-mon[29291]: 6.d scrub starts 2024-09-10T18:23:24.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:23 smithi028 ceph-mon[29291]: 6.d scrub ok 2024-09-10T18:23:24.154 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:23 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:23:24.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:23 smithi028 ceph-mon[39220]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 476 B/s wr, 9 op/s; 67 B/s, 2 objects/s recovering 2024-09-10T18:23:24.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:23 smithi028 ceph-mon[39220]: 6.d scrub starts 2024-09-10T18:23:24.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:23 smithi028 ceph-mon[39220]: 6.d scrub ok 2024-09-10T18:23:24.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:23 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:23:24.165 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:24.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:24.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:23:24.290 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:24.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:24.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:23:24.392 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:24.413 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:24.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:24.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:24.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:24.539 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:24.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:24.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:24.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:24.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:24.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:24.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:24.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:24.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:24.914 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:24.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:24.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:24.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:24 smithi028 ceph-mon[29291]: 4.c deep-scrub starts 2024-09-10T18:23:24.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:24 smithi028 ceph-mon[29291]: 4.c deep-scrub ok 2024-09-10T18:23:24.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:24 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:24.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:24 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:24.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:24 smithi028 ceph-mon[29291]: 5.9 scrub starts 2024-09-10T18:23:24.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:24 smithi028 ceph-mon[29291]: 5.9 scrub ok 2024-09-10T18:23:24.987 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:24 smithi028 ceph-mon[39220]: 4.c deep-scrub starts 2024-09-10T18:23:24.987 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:24 smithi028 ceph-mon[39220]: 4.c deep-scrub ok 2024-09-10T18:23:24.987 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:24 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:24.987 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:24 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:24.987 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:24 smithi028 ceph-mon[39220]: 5.9 scrub starts 2024-09-10T18:23:24.987 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:24 smithi028 ceph-mon[39220]: 5.9 scrub ok 2024-09-10T18:23:25.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:25.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:25.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:25.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:24 smithi204 ceph-mon[32357]: 4.c deep-scrub starts 2024-09-10T18:23:25.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:24 smithi204 ceph-mon[32357]: 4.c deep-scrub ok 2024-09-10T18:23:25.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:24 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:25.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:24 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:25.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:24 smithi204 ceph-mon[32357]: 5.9 scrub starts 2024-09-10T18:23:25.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:24 smithi204 ceph-mon[32357]: 5.9 scrub ok 2024-09-10T18:23:25.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:25.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:25.165 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:25.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:25.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:25.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:25.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:25.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:25.395 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:23:25.396 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:23:25.412 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:23:25.413 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:23:25.416 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:25.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:25.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:25.428 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:23:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:23:25.463 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:23:25.463 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:23:25.479 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:23:25.479 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:23:25.494 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:23:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:23:25.510 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:23:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:23:25.519 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:25.526 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:23:25.526 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:23:25.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:25.544 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:25.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:25.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:25.668 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:25.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:25.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:25.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:25.787 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_0' created 2024-09-10T18:23:25.797 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:25.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:25.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:25.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:25.902 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:23:25.902 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:23:25.902 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:25 smithi028 ceph-mon[39220]: 3.18 scrub starts 2024-09-10T18:23:25.902 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:25 smithi028 ceph-mon[39220]: 3.18 scrub ok 2024-09-10T18:23:25.902 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:25 smithi028 ceph-mon[39220]: 3.6 scrub starts 2024-09-10T18:23:25.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:25 smithi028 ceph-mon[39220]: 3.6 scrub ok 2024-09-10T18:23:25.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:25 smithi028 ceph-mon[39220]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:23:25.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:25 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1426110545' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-10T18:23:25.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:25 smithi028 ceph-mon[39220]: from='client.34667 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-10T18:23:25.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:25 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1698365425' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:23:25.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:25 smithi028 ceph-mon[29291]: 3.18 scrub starts 2024-09-10T18:23:25.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:25 smithi028 ceph-mon[29291]: 3.18 scrub ok 2024-09-10T18:23:25.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:25 smithi028 ceph-mon[29291]: 3.6 scrub starts 2024-09-10T18:23:25.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:25 smithi028 ceph-mon[29291]: 3.6 scrub ok 2024-09-10T18:23:25.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:25 smithi028 ceph-mon[29291]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:23:25.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:25 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1426110545' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-10T18:23:25.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:25 smithi028 ceph-mon[29291]: from='client.34667 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-10T18:23:25.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:25 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1698365425' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:23:25.923 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:25.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:26.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:26.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:26.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:26.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:26.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:25 smithi204 ceph-mon[32357]: 3.18 scrub starts 2024-09-10T18:23:26.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:25 smithi204 ceph-mon[32357]: 3.18 scrub ok 2024-09-10T18:23:26.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:25 smithi204 ceph-mon[32357]: 3.6 scrub starts 2024-09-10T18:23:26.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:25 smithi204 ceph-mon[32357]: 3.6 scrub ok 2024-09-10T18:23:26.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:25 smithi204 ceph-mon[32357]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:23:26.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:25 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1426110545' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-10T18:23:26.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:25 smithi204 ceph-mon[32357]: from='client.34667 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-10T18:23:26.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:25 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1698365425' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:23:26.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:26.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:26.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:26.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:26.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:26.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:26.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:26.398 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:26.424 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:26.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:26.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:26.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:23:26.551 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:26.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:26.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:23:26.653 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:26.677 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:26.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:26.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:26.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:26.810 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:26.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:26.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:26.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:26.935 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:26.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:26.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:27.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:27.044 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T18:23:27.045 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:23:27.045 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T18:23:27.045 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-10T18:23:27.045 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.6 2024-09-10T18:23:27.057 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[39220]: 5.c scrub starts 2024-09-10T18:23:27.058 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[39220]: 5.c scrub ok 2024-09-10T18:23:27.058 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[39220]: 4.d scrub starts 2024-09-10T18:23:27.058 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[39220]: 4.d scrub ok 2024-09-10T18:23:27.058 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[39220]: 3.17 scrub starts 2024-09-10T18:23:27.058 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[39220]: 3.17 scrub ok 2024-09-10T18:23:27.058 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[39220]: from='client.34667 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-10T18:23:27.058 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[39220]: osdmap e93: 8 total, 8 up, 8 in 2024-09-10T18:23:27.058 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[39220]: 6.5 scrub starts 2024-09-10T18:23:27.058 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[39220]: 6.5 scrub ok 2024-09-10T18:23:27.058 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[39220]: 3.1d scrub starts 2024-09-10T18:23:27.058 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[39220]: 3.1d scrub ok 2024-09-10T18:23:27.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[29291]: 5.c scrub starts 2024-09-10T18:23:27.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[29291]: 5.c scrub ok 2024-09-10T18:23:27.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[29291]: 4.d scrub starts 2024-09-10T18:23:27.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[29291]: 4.d scrub ok 2024-09-10T18:23:27.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[29291]: 3.17 scrub starts 2024-09-10T18:23:27.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[29291]: 3.17 scrub ok 2024-09-10T18:23:27.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[29291]: from='client.34667 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-10T18:23:27.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[29291]: osdmap e93: 8 total, 8 up, 8 in 2024-09-10T18:23:27.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[29291]: 6.5 scrub starts 2024-09-10T18:23:27.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[29291]: 6.5 scrub ok 2024-09-10T18:23:27.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[29291]: 3.1d scrub starts 2024-09-10T18:23:27.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:26 smithi028 ceph-mon[29291]: 3.1d scrub ok 2024-09-10T18:23:27.061 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:27.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:27.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:26 smithi204 ceph-mon[32357]: 5.c scrub starts 2024-09-10T18:23:27.085 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:26 smithi204 ceph-mon[32357]: 5.c scrub ok 2024-09-10T18:23:27.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:26 smithi204 ceph-mon[32357]: 4.d scrub starts 2024-09-10T18:23:27.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:26 smithi204 ceph-mon[32357]: 4.d scrub ok 2024-09-10T18:23:27.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:26 smithi204 ceph-mon[32357]: 3.17 scrub starts 2024-09-10T18:23:27.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:26 smithi204 ceph-mon[32357]: 3.17 scrub ok 2024-09-10T18:23:27.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:26 smithi204 ceph-mon[32357]: from='client.34667 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-10T18:23:27.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:26 smithi204 ceph-mon[32357]: osdmap e93: 8 total, 8 up, 8 in 2024-09-10T18:23:27.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:26 smithi204 ceph-mon[32357]: 6.5 scrub starts 2024-09-10T18:23:27.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:26 smithi204 ceph-mon[32357]: 6.5 scrub ok 2024-09-10T18:23:27.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:26 smithi204 ceph-mon[32357]: 3.1d scrub starts 2024-09-10T18:23:27.086 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:26 smithi204 ceph-mon[32357]: 3.1d scrub ok 2024-09-10T18:23:27.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:27.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:27.187 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:27.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:27.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:27.310 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:27.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:27.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:27.354 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:23:27 smithi204 systemd[1]: Stopping Ceph osd.6 for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:23:27.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:27.434 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:27.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:27.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:27.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:27.559 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:27.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:27.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:27.661 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:27.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:27.683 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:27.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:27.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-10T18:23:27.785 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:27.788 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:23:27 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[48595]: 2024-09-10T18:23:27.417+0000 7f2a787a7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T18:23:27.788 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:23:27 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[48595]: 2024-09-10T18:23:27.417+0000 7f2a787a7640 -1 osd.6 94 *** Got signal Terminated *** 2024-09-10T18:23:27.788 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:23:27 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[48595]: 2024-09-10T18:23:27.417+0000 7f2a787a7640 -1 osd.6 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T18:23:27.795 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:23:27.812 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:27.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:27.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:27.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:27.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:27.938 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:27.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:28.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[29291]: 4.10 scrub starts 2024-09-10T18:23:28.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[29291]: 4.10 scrub ok 2024-09-10T18:23:28.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[29291]: 4.18 scrub starts 2024-09-10T18:23:28.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[29291]: 4.18 scrub ok 2024-09-10T18:23:28.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[29291]: pgmap v60: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:23:28.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[29291]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:23:28.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[29291]: osdmap e94: 8 total, 8 up, 8 in 2024-09-10T18:23:28.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:23:28.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:23:28.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3307389410' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:23:28.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[29291]: from='client.34679 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:23:28.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[29291]: osd.6 marked itself down and dead 2024-09-10T18:23:28.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[39220]: 4.10 scrub starts 2024-09-10T18:23:28.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[39220]: 4.10 scrub ok 2024-09-10T18:23:28.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[39220]: 4.18 scrub starts 2024-09-10T18:23:28.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[39220]: 4.18 scrub ok 2024-09-10T18:23:28.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[39220]: pgmap v60: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:23:28.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[39220]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:23:28.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[39220]: osdmap e94: 8 total, 8 up, 8 in 2024-09-10T18:23:28.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:23:28.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:23:28.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3307389410' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:23:28.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[39220]: from='client.34679 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:23:28.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:27 smithi028 ceph-mon[39220]: osd.6 marked itself down and dead 2024-09-10T18:23:28.039 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:28.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:28.062 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:28.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:28.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:27 smithi204 ceph-mon[32357]: 4.10 scrub starts 2024-09-10T18:23:28.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:27 smithi204 ceph-mon[32357]: 4.10 scrub ok 2024-09-10T18:23:28.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:27 smithi204 ceph-mon[32357]: 4.18 scrub starts 2024-09-10T18:23:28.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:27 smithi204 ceph-mon[32357]: 4.18 scrub ok 2024-09-10T18:23:28.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:27 smithi204 ceph-mon[32357]: pgmap v60: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:23:28.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:27 smithi204 ceph-mon[32357]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:23:28.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:27 smithi204 ceph-mon[32357]: osdmap e94: 8 total, 8 up, 8 in 2024-09-10T18:23:28.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:27 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:23:28.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:27 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:23:28.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:27 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3307389410' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:23:28.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:27 smithi204 ceph-mon[32357]: from='client.34679 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:23:28.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:27 smithi204 ceph-mon[32357]: osd.6 marked itself down and dead 2024-09-10T18:23:28.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:28.186 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:28.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:28.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:28.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:28.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:28.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:28.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:28.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:28.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-10T18:23:28.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:23:28.435 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:28.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:28.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:28.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:28.560 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:28.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:28.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:28.604 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:23:28 smithi204 podman[64606]: 2024-09-10 18:23:28.331146751 +0000 UTC m=+1.074449607 container died 7367c9aeed8f25d5e2911a518412c6024b8e89a8b0a5fcbafcd95e62a38cadbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-10T18:23:28.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:28.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:28.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:28.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:28.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:28.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:28.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:28.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:28.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:28.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:29.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:28 smithi204 ceph-mon[32357]: 4.b scrub starts 2024-09-10T18:23:29.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:28 smithi204 ceph-mon[32357]: 4.b scrub ok 2024-09-10T18:23:29.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:28 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:23:29.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:28 smithi204 ceph-mon[32357]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-10T18:23:29.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:28 smithi204 ceph-mon[32357]: from='client.34679 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:23:29.004 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:28 smithi204 ceph-mon[32357]: osdmap e95: 8 total, 7 up, 8 in 2024-09-10T18:23:29.004 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:23:28 smithi204 podman[64606]: 2024-09-10 18:23:28.701061981 +0000 UTC m=+1.444364837 container cleanup 7367c9aeed8f25d5e2911a518412c6024b8e89a8b0a5fcbafcd95e62a38cadbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-10T18:23:29.004 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:23:28 smithi204 bash[64606]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6 2024-09-10T18:23:29.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:29.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:29.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:29.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:29.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:28 smithi028 ceph-mon[39220]: 4.b scrub starts 2024-09-10T18:23:29.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:28 smithi028 ceph-mon[39220]: 4.b scrub ok 2024-09-10T18:23:29.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:28 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:23:29.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:28 smithi028 ceph-mon[39220]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-10T18:23:29.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:28 smithi028 ceph-mon[39220]: from='client.34679 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:23:29.099 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:28 smithi028 ceph-mon[39220]: osdmap e95: 8 total, 7 up, 8 in 2024-09-10T18:23:29.099 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:28 smithi028 ceph-mon[29291]: 4.b scrub starts 2024-09-10T18:23:29.099 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:28 smithi028 ceph-mon[29291]: 4.b scrub ok 2024-09-10T18:23:29.099 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:28 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:23:29.099 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:28 smithi028 ceph-mon[29291]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-10T18:23:29.099 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:28 smithi028 ceph-mon[29291]: from='client.34679 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:23:29.099 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:28 smithi028 ceph-mon[29291]: osdmap e95: 8 total, 7 up, 8 in 2024-09-10T18:23:29.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:29.189 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:29.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:29.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:29.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:29.314 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:29.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:29.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:29.328 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:23:29 smithi204 podman[64691]: 2024-09-10 18:23:29.002744212 +0000 UTC m=+0.671510036 container remove 7367c9aeed8f25d5e2911a518412c6024b8e89a8b0a5fcbafcd95e62a38cadbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T18:23:29.415 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:29.437 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:29.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:29.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:29.561 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:29.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:29.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:29.604 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:23:29 smithi204 podman[64829]: 2024-09-10 18:23:29.229504234 +0000 UTC m=+0.015506735 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T18:23:29.604 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:23:29 smithi204 podman[64829]: 2024-09-10 18:23:29.337049606 +0000 UTC m=+0.123052146 container create 1ec7180d937b011cc2be9000bc1a9ac93a0bffc8c09f2ee98d59477145ebbba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-10T18:23:29.643 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:29.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:29.677 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:29.690 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:29.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:29.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:29.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:29.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:29.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:29.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:29.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:29.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:29.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:30.041 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:30.062 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:30.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:30.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:30.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:29 smithi028 ceph-mon[29291]: pgmap v63: 148 pgs: 8 stale+active+clean, 16 unknown, 124 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:23:30.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:29 smithi028 ceph-mon[29291]: osdmap e96: 8 total, 7 up, 8 in 2024-09-10T18:23:30.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:29 smithi028 ceph-mon[39220]: pgmap v63: 148 pgs: 8 stale+active+clean, 16 unknown, 124 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:23:30.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:29 smithi028 ceph-mon[39220]: osdmap e96: 8 total, 7 up, 8 in 2024-09-10T18:23:30.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:29 smithi204 ceph-mon[32357]: pgmap v63: 148 pgs: 8 stale+active+clean, 16 unknown, 124 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:23:30.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:29 smithi204 ceph-mon[32357]: osdmap e96: 8 total, 7 up, 8 in 2024-09-10T18:23:30.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:23:29 smithi204 podman[64829]: 2024-09-10 18:23:29.606239004 +0000 UTC m=+0.392241488 container init 1ec7180d937b011cc2be9000bc1a9ac93a0bffc8c09f2ee98d59477145ebbba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-10T18:23:30.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:23:29 smithi204 podman[64829]: 2024-09-10 18:23:29.609679614 +0000 UTC m=+0.395682110 container start 1ec7180d937b011cc2be9000bc1a9ac93a0bffc8c09f2ee98d59477145ebbba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-10T18:23:30.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:23:29 smithi204 podman[64829]: 2024-09-10 18:23:29.68961189 +0000 UTC m=+0.475614376 container attach 1ec7180d937b011cc2be9000bc1a9ac93a0bffc8c09f2ee98d59477145ebbba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-10T18:23:30.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:23:29 smithi204 podman[64829]: 2024-09-10 18:23:29.783581703 +0000 UTC m=+0.569584198 container died 1ec7180d937b011cc2be9000bc1a9ac93a0bffc8c09f2ee98d59477145ebbba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-deactivate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-09-10T18:23:30.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:30.186 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:30.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:30.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:30.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:30.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:30.309 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:30.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:30.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:30.434 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:30.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:30.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:30.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:30.562 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:30.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:30.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:30.664 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:30.686 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:30.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:30.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:30.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:30.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:30.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:30.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:23:30.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:30.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:31.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:31.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:31.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:31.163 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:31.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:31.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:31.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:31.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:31.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:31.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:31.435 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:31.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:31.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:31.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:31.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:31.561 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:31.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:31.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:31.686 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:31.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:31.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:31.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:31 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:31.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:31 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:31.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:31 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:23:31.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:31 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:23:31.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:31 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:31.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:31 smithi028 ceph-mon[29291]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-10T18:23:31.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:31 smithi028 ceph-mon[29291]: Reconfiguring daemon alertmanager.a on smithi028 2024-09-10T18:23:31.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:31 smithi028 ceph-mon[29291]: pgmap v65: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 73/588 objects degraded (12.415%) 2024-09-10T18:23:31.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:31 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:31.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:31 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:31.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:31 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:23:31.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:31 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:23:31.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:31 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:31.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:31 smithi028 ceph-mon[39220]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-10T18:23:31.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:31 smithi028 ceph-mon[39220]: Reconfiguring daemon alertmanager.a on smithi028 2024-09-10T18:23:31.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:31 smithi028 ceph-mon[39220]: pgmap v65: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 73/588 objects degraded (12.415%) 2024-09-10T18:23:31.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:31.814 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:31.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:31.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:31.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:31 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:31.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:31 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:31.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:31 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:23:31.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:31 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:23:31.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:31 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:31.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:31 smithi204 ceph-mon[32357]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-10T18:23:31.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:31 smithi204 ceph-mon[32357]: Reconfiguring daemon alertmanager.a on smithi028 2024-09-10T18:23:31.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:31 smithi204 ceph-mon[32357]: pgmap v65: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 73/588 objects degraded (12.415%) 2024-09-10T18:23:31.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:31.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:31.939 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:31.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:32.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:32.063 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:32.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:32.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:32.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:32.189 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:32.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:32.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:32.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:32.314 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:32.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:32.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:23:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:23:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:23:32.414 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:32.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:32.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:32.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:32.543 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:32.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:32.568 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:32.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:32.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:32.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:32.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:32 smithi028 ceph-mon[29291]: Health check failed: Degraded data redundancy: 73/588 objects degraded (12.415%), 19 pgs degraded (PG_DEGRADED) 2024-09-10T18:23:32.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:32 smithi028 ceph-mon[39220]: Health check failed: Degraded data redundancy: 73/588 objects degraded (12.415%), 19 pgs degraded (PG_DEGRADED) 2024-09-10T18:23:32.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:32.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:32.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:32.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:32.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:32 smithi204 ceph-mon[32357]: Health check failed: Degraded data redundancy: 73/588 objects degraded (12.415%), 19 pgs degraded (PG_DEGRADED) 2024-09-10T18:23:32.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:32.943 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:32.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:32.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:33.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:33.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:33.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:33.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:33.194 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:33.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:33.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:33.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:33.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:33.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:33.421 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:33.447 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:33.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:33.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:33.549 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:33.573 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:33.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:33.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:33.650 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:33 smithi028 ceph-mon[39220]: pgmap v66: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 73/588 objects degraded (12.415%) 2024-09-10T18:23:33.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:33 smithi028 ceph-mon[29291]: pgmap v66: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 73/588 objects degraded (12.415%) 2024-09-10T18:23:33.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:33.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:33.711 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:33.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:33.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:33.839 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:33.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:33.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:33.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:33 smithi204 ceph-mon[32357]: pgmap v66: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 73/588 objects degraded (12.415%) 2024-09-10T18:23:33.903 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:23:33.903 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:23:33.918 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:23:33.918 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:23:33.934 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:23:33.934 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:23:33.940 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:33.948 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:23:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:23:33.964 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:23:33.964 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:23:33.966 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:33.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:33.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:33.980 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:23:33.981 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:23:33.981 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-10T18:23:33.997 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:23:33.997 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:23:34.067 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:34.093 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:34.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:34.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:34.195 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:34.219 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:34.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:34.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:34.321 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:34.347 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:34.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:34.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:34.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:23:33 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:23:34.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:34.474 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:34.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:34.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:34.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:34.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:34.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:34.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:34.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:34.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:34.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:34.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:34.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:34 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1860853163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:23:34.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:34 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:34.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:34 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1860853163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:23:34.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:34 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:34.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:34.852 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:34.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:34.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:34.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:34 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1860853163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:23:34.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:34 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:34.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:34.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:34.978 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:34.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:34.990 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.6.service' 2024-09-10T18:23:35.078 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:35.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:35.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:35.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:35.111 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T18:23:35.112 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-10T18:23:35.112 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.6 2024-09-10T18:23:35.112 INFO:tasks.thrashosds.thrasher:remote for osd 6 is ubuntu@smithi204.front.sepia.ceph.com 2024-09-10T18:23:35.112 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -n osd.6 --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op list-pgs 2024-09-10T18:23:35.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:35.226 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:35.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:35.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:35.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:35.344 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/osd.6/config 2024-09-10T18:23:35.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:35.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:35.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:35.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:35.474 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:35.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:35.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:35.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:35.599 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:35.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:35.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:35.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:35 smithi028 ceph-mon[39220]: pgmap v67: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 258 B/s rd, 0 op/s; 73/588 objects degraded (12.415%) 2024-09-10T18:23:35.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:35 smithi028 ceph-mon[29291]: pgmap v67: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 258 B/s rd, 0 op/s; 73/588 objects degraded (12.415%) 2024-09-10T18:23:35.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:35.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:35.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:35.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:35.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:35.848 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:35.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:35.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:35.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:35 smithi204 ceph-mon[32357]: pgmap v67: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 258 B/s rd, 0 op/s; 73/588 objects degraded (12.415%) 2024-09-10T18:23:35.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:35.973 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:35.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:35.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:36.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:36.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:36.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:36.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:36.117 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T18:23:36.117 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T18:23:36.200 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:36.227 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:36.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:36.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:36.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:36.355 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:36.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:36.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:36.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:36.480 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:36.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:36.582 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:36.609 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:36.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:36.652 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:36.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:36.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:36.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:36.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:36.858 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:36.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:36.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:36.960 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:36.986 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:36.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:36.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:37.088 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:37.113 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:37.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:37.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:37.215 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:37.241 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:37.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:37.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:37.343 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:37.368 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:37.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:37.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:37.470 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:37.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:37.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:37.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:37.623 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:37.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:37.725 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:37.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:37.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:37.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:37.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:37.879 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:37.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:37.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:37.981 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:38.008 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:38.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:38.013 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:37 smithi204 ceph-mon[32357]: pgmap v68: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 229 B/s rd, 0 op/s; 73/588 objects degraded (12.415%) 2024-09-10T18:23:38.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:38.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:23:38.134 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:38.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:38.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:38.157 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:37 smithi028 ceph-mon[29291]: pgmap v68: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 229 B/s rd, 0 op/s; 73/588 objects degraded (12.415%) 2024-09-10T18:23:38.157 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:37 smithi028 ceph-mon[39220]: pgmap v68: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 229 B/s rd, 0 op/s; 73/588 objects degraded (12.415%) 2024-09-10T18:23:38.236 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:38.262 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:38.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:38.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:38.386 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:38.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:38.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:38.488 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:38.514 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:38.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:38.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:38.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:38.644 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:38.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:38.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:38.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:38.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:38.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:38.871 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:38.893 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:38.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:38.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:38.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:39.017 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:39.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:39.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:39.119 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:39.141 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:39.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:39.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:39.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:39.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:39.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:39.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:39.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:39.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:39.411 INFO:teuthology.orchestra.run.smithi204.stdout:7.b 2024-09-10T18:23:39.412 INFO:teuthology.orchestra.run.smithi204.stdout:6.5 2024-09-10T18:23:39.412 INFO:teuthology.orchestra.run.smithi204.stdout:6.3 2024-09-10T18:23:39.412 INFO:teuthology.orchestra.run.smithi204.stdout:6.1f 2024-09-10T18:23:39.412 INFO:teuthology.orchestra.run.smithi204.stdout:6.c 2024-09-10T18:23:39.412 INFO:teuthology.orchestra.run.smithi204.stdout:6.1b 2024-09-10T18:23:39.412 INFO:teuthology.orchestra.run.smithi204.stdout:6.1 2024-09-10T18:23:39.412 INFO:teuthology.orchestra.run.smithi204.stdout:6.15 2024-09-10T18:23:39.412 INFO:teuthology.orchestra.run.smithi204.stdout:5.9 2024-09-10T18:23:39.412 INFO:teuthology.orchestra.run.smithi204.stdout:5.3 2024-09-10T18:23:39.413 INFO:teuthology.orchestra.run.smithi204.stdout:3.a 2024-09-10T18:23:39.413 INFO:teuthology.orchestra.run.smithi204.stdout:3.14 2024-09-10T18:23:39.413 INFO:teuthology.orchestra.run.smithi204.stdout:3.3 2024-09-10T18:23:39.413 INFO:teuthology.orchestra.run.smithi204.stdout:3.1e 2024-09-10T18:23:39.413 INFO:teuthology.orchestra.run.smithi204.stdout:3.1d 2024-09-10T18:23:39.413 INFO:teuthology.orchestra.run.smithi204.stdout:6.13 2024-09-10T18:23:39.413 INFO:teuthology.orchestra.run.smithi204.stdout:5.1f 2024-09-10T18:23:39.413 INFO:teuthology.orchestra.run.smithi204.stdout:3.2 2024-09-10T18:23:39.413 INFO:teuthology.orchestra.run.smithi204.stdout:6.1e 2024-09-10T18:23:39.413 INFO:teuthology.orchestra.run.smithi204.stdout:5.f 2024-09-10T18:23:39.413 INFO:teuthology.orchestra.run.smithi204.stdout:4.8 2024-09-10T18:23:39.413 INFO:teuthology.orchestra.run.smithi204.stdout:3.10 2024-09-10T18:23:39.413 INFO:teuthology.orchestra.run.smithi204.stdout:7.1 2024-09-10T18:23:39.413 INFO:teuthology.orchestra.run.smithi204.stdout:4.1f 2024-09-10T18:23:39.414 INFO:teuthology.orchestra.run.smithi204.stdout:3.d 2024-09-10T18:23:39.414 INFO:teuthology.orchestra.run.smithi204.stdout:7.d 2024-09-10T18:23:39.414 INFO:teuthology.orchestra.run.smithi204.stdout:3.11 2024-09-10T18:23:39.414 INFO:teuthology.orchestra.run.smithi204.stdout:3.0 2024-09-10T18:23:39.414 INFO:teuthology.orchestra.run.smithi204.stdout:5.2 2024-09-10T18:23:39.414 INFO:teuthology.orchestra.run.smithi204.stdout:5.10 2024-09-10T18:23:39.414 INFO:teuthology.orchestra.run.smithi204.stdout:1.0 2024-09-10T18:23:39.414 INFO:teuthology.orchestra.run.smithi204.stdout:4.12 2024-09-10T18:23:39.414 INFO:teuthology.orchestra.run.smithi204.stdout:7.7 2024-09-10T18:23:39.414 INFO:teuthology.orchestra.run.smithi204.stdout:5.11 2024-09-10T18:23:39.414 INFO:teuthology.orchestra.run.smithi204.stdout:4.13 2024-09-10T18:23:39.414 INFO:teuthology.orchestra.run.smithi204.stdout:7.c 2024-09-10T18:23:39.414 INFO:teuthology.orchestra.run.smithi204.stdout:4.a 2024-09-10T18:23:39.414 INFO:teuthology.orchestra.run.smithi204.stdout:4.10 2024-09-10T18:23:39.414 INFO:teuthology.orchestra.run.smithi204.stdout:4.18 2024-09-10T18:23:39.415 INFO:teuthology.orchestra.run.smithi204.stdout:3.c 2024-09-10T18:23:39.415 INFO:teuthology.orchestra.run.smithi204.stdout:4.c 2024-09-10T18:23:39.415 INFO:teuthology.orchestra.run.smithi204.stdout:4.1 2024-09-10T18:23:39.415 INFO:teuthology.orchestra.run.smithi204.stdout:4.11 2024-09-10T18:23:39.415 INFO:teuthology.orchestra.run.smithi204.stdout:4.5 2024-09-10T18:23:39.415 INFO:teuthology.orchestra.run.smithi204.stdout:6.a 2024-09-10T18:23:39.415 INFO:teuthology.orchestra.run.smithi204.stdout:2.2 2024-09-10T18:23:39.415 INFO:teuthology.orchestra.run.smithi204.stdout:4.e 2024-09-10T18:23:39.415 INFO:teuthology.orchestra.run.smithi204.stdout:5.18 2024-09-10T18:23:39.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:39.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:39.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:39.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:39.741 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:39.767 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:39.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:39.770 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:39.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:39.771 INFO:teuthology.orchestra.run.smithi028.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-10T18:23:39.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:39.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:39.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:39.996 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:39 smithi028 ceph-mon[29291]: pgmap v69: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s; 73/588 objects degraded (12.415%) 2024-09-10T18:23:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:39 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:23:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:39 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1276443400' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:23:40.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:39 smithi028 ceph-mon[39220]: pgmap v69: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s; 73/588 objects degraded (12.415%) 2024-09-10T18:23:40.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:39 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:40.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:23:40.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:39 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1276443400' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:23:40.018 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:40.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:40.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:40.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:39 smithi204 ceph-mon[32357]: pgmap v69: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s; 73/588 objects degraded (12.415%) 2024-09-10T18:23:40.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:39 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:40.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:23:40.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:39 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1276443400' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:23:40.119 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:40.143 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:40.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:40.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:40.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:40.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:40.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:40.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:40.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:40.494 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:40.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:40.516 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:40.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:40.549 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -n osd.6 --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op export --pgid 4.1 --file /var/log/ceph/exp.4.1.6 2024-09-10T18:23:40.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:40.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:40.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:40.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:40.766 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:40.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:40.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:40.774 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:40 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1276443400' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:23:40.775 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:40 smithi204 ceph-mon[32357]: osdmap e97: 8 total, 7 up, 8 in 2024-09-10T18:23:40.775 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:40 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/513040664' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:23:40.775 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:40 smithi204 ceph-mon[32357]: from='client.34697 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:23:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:23:40.782 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/osd.6/config 2024-09-10T18:23:40.867 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:40.889 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:40.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:40.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:40.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:41.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:41.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:41.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:40 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1276443400' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:23:41.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:40 smithi028 ceph-mon[29291]: osdmap e97: 8 total, 7 up, 8 in 2024-09-10T18:23:41.086 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:40 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/513040664' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:23:41.086 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:40 smithi028 ceph-mon[29291]: from='client.34697 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:23:41.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:40 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1276443400' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:23:41.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:40 smithi028 ceph-mon[39220]: osdmap e97: 8 total, 7 up, 8 in 2024-09-10T18:23:41.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:40 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/513040664' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:23:41.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:40 smithi028 ceph-mon[39220]: from='client.34697 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:23:41.087 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:41 smithi028 podman[77439]: 2024-09-10 18:23:41.084310567 +0000 UTC m=+0.944119677 volume remove d4ad530f3d597c6fceb86136eb2d3031489b2c6a029a9c34193dde9cb5b90102 2024-09-10T18:23:41.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:41.137 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:41.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:41.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:41.239 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:41.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:41.262 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:41.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:41.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:41.384 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:41.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:41.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:41.485 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:41.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:41.508 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:41.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:41.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:41.633 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:41.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:41.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:41.698 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:23:41.698 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-10T18:23:41.698 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-10T18:23:41.698 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-10T18:23:41.698 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:23:41.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:41.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:41.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:41.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:41.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:41.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:41.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:41.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:41.992 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:41 smithi028 ceph-mon[39220]: pgmap v71: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 73/588 objects degraded (12.415%) 2024-09-10T18:23:41.992 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:41 smithi028 ceph-mon[39220]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:23:41.992 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:41 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:23:41.992 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:41 smithi028 ceph-mon[39220]: from='client.34697 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:23:41.992 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:41 smithi028 ceph-mon[39220]: osdmap e98: 8 total, 7 up, 8 in 2024-09-10T18:23:41.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:41 smithi028 ceph-mon[29291]: pgmap v71: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 73/588 objects degraded (12.415%) 2024-09-10T18:23:41.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:41 smithi028 ceph-mon[29291]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:23:41.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:41 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:23:41.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:41 smithi028 ceph-mon[29291]: from='client.34697 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:23:41.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:41 smithi028 ceph-mon[29291]: osdmap e98: 8 total, 7 up, 8 in 2024-09-10T18:23:42.008 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:42.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:42.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:42.051 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:41 smithi204 ceph-mon[32357]: pgmap v71: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 73/588 objects degraded (12.415%) 2024-09-10T18:23:42.051 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:41 smithi204 ceph-mon[32357]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:23:42.051 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:41 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:23:42.051 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:41 smithi204 ceph-mon[32357]: from='client.34697 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:23:42.051 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:41 smithi204 ceph-mon[32357]: osdmap e98: 8 total, 7 up, 8 in 2024-09-10T18:23:42.110 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:42.132 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:42.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:42.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:42.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:42.254 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:42 smithi028 podman[77491]: 2024-09-10 18:23:42.117141154 +0000 UTC m=+1.548120853 container died 19ddaf34619851dd86b89d1e37b3d9b5aa25d3957c9e6c7132967fd9fa5450cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=pensive_pike, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-10T18:23:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:42.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:42.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:42.344 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:23:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:23:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:23:42.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:42.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:42.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:42.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:42.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:42.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:42.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:42.609 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:42.632 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:42.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:42.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:42.654 INFO:teuthology.orchestra.run.smithi204.stderr:Exporting 4.1 info 4.1( v 65'14 (0'0,65'14] local-lis/les=59/60 n=7 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) 2024-09-10T18:23:42.655 INFO:teuthology.orchestra.run.smithi204.stderr:Read #4:80542eaa:::obj_delete_at_hint.0000000068:head# 2024-09-10T18:23:42.655 INFO:teuthology.orchestra.run.smithi204.stderr:Read #4:80b6fe23:reshard::reshard.0000000011:head# 2024-09-10T18:23:42.655 INFO:teuthology.orchestra.run.smithi204.stderr:Read #4:80d23011:::obj_delete_at_hint.0000000085:head# 2024-09-10T18:23:42.656 INFO:teuthology.orchestra.run.smithi204.stderr:Read #4:8411e274:::obj_delete_at_hint.0000000073:head# 2024-09-10T18:23:42.656 INFO:teuthology.orchestra.run.smithi204.stderr:Read #4:84d19476:::obj_delete_at_hint.0000000038:head# 2024-09-10T18:23:42.656 INFO:teuthology.orchestra.run.smithi204.stderr:Read #4:856b9b26:reshard::reshard.0000000002:head# 2024-09-10T18:23:42.656 INFO:teuthology.orchestra.run.smithi204.stderr:Read #4:86cc8a85:::obj_delete_at_hint.0000000118:head# 2024-09-10T18:23:42.656 INFO:teuthology.orchestra.run.smithi204.stderr:Export successful 2024-09-10T18:23:42.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:42.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:42.858 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:42.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:42.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:42.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:42.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:43.004 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:43.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:43.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:43.106 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:43.127 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:43.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:43.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:43.229 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:43.251 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:43.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:43.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:43.374 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:43.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:43.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:43.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:43.498 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:43.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:43.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:43.600 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:43.622 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:43.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:43.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:43.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:43.747 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:43.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:43.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:43.790 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -n osd.6 --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --force --op remove --pgid 4.1 2024-09-10T18:23:43.801 DEBUG:teuthology.orchestra.run.smithi028:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-10T18:23:43.803 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-10T18:23:43.803 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-10T18:23:43.849 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:43.870 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:43.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:43.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:43.877 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:hints = 1 2024-09-10T18:23:43.877 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-10T18:23:43.877 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object prefix: benchmark_data_smithi028_77856 2024-09-10T18:23:43.877 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:23:43.877 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 0 0 0 0 0 0 - 0 2024-09-10T18:23:43.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:44.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:43 smithi028 ceph-mon[29291]: pgmap v73: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 73/588 objects degraded (12.415%) 2024-09-10T18:23:44.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:43 smithi028 ceph-mon[39220]: pgmap v73: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 73/588 objects degraded (12.415%) 2024-09-10T18:23:44.004 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:43 smithi028 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-10T18:23:44.004 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:43 smithi028 systemd[1]: Stopped Ceph alertmanager.a for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:23:44.004 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:43 smithi028 systemd[1]: Starting Ceph alertmanager.a for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:23:44.006 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/osd.6/config 2024-09-10T18:23:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:44.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:44.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:43 smithi204 ceph-mon[32357]: pgmap v73: 148 pgs: 29 active+undersized, 19 active+undersized+degraded, 100 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 73/588 objects degraded (12.415%) 2024-09-10T18:23:44.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:23:43 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:23:44.116 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:44.140 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:44.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:44.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:44.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:44.290 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:44.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:44.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:44.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:44.434 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:44.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:44.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:44.577 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:44.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:44.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:44.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:44.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-10T18:23:44.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:44.731 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:44.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:44.789 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:23:44.789 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:23:44.790 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:23:44.790 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:23:44.790 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:23:44.790 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:23:44.790 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:23:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:23:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:23:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:23:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:23:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:23:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:23:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-10T18:23:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:23:44.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:44.873 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:44.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:44.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:44.877 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1 16 2679 2663 166.427 166.438 0.010943 0.00598705 2024-09-10T18:23:44.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:44 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:44.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:44 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2718422303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:23:44.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:44 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:44.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:44 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2718422303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:23:44.975 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:45.001 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:45.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:45.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:45.102 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:45.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:44 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:45.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:44 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2718422303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:23:45.126 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:45.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:45.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:45.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:23:45.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:45.293 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:45.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:45.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:45.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:45.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:45.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:45.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:45.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:45.564 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:45.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:45.607 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T18:23:45.608 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T18:23:45.665 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:45.684 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:45.689 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:45.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:45.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:45.791 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:45.817 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:45.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:45.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:45.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:45 smithi028 ceph-mon[29291]: pgmap v74: 148 pgs: 25 active+undersized, 23 active+undersized+degraded, 100 active+clean; 105 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 13 MiB/s wr, 210 op/s; 586/5610 objects degraded (10.446%) 2024-09-10T18:23:45.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:45 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 586/5610 objects degraded (10.446%), 23 pgs degraded (PG_DEGRADED) 2024-09-10T18:23:45.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:45 smithi028 ceph-mon[39220]: pgmap v74: 148 pgs: 25 active+undersized, 23 active+undersized+degraded, 100 active+clean; 105 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 13 MiB/s wr, 210 op/s; 586/5610 objects degraded (10.446%) 2024-09-10T18:23:45.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:45 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 586/5610 objects degraded (10.446%), 23 pgs degraded (PG_DEGRADED) 2024-09-10T18:23:45.878 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2 16 5161 5145 160.758 155.125 0.00932605 0.00620618 2024-09-10T18:23:45.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:45.940 INFO:teuthology.orchestra.run.smithi204.stdout: marking collection for removal 2024-09-10T18:23:45.940 INFO:teuthology.orchestra.run.smithi204.stdout:setting '_remove' omap key 2024-09-10T18:23:45.941 INFO:teuthology.orchestra.run.smithi204.stdout:finish_remove_pgs 4.1_head removing 4.1 2024-09-10T18:23:45.941 INFO:teuthology.orchestra.run.smithi204.stdout:Remove successful 2024-09-10T18:23:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:45.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:45.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:46.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:46.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:46.103 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:46.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:46.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:45 smithi204 ceph-mon[32357]: pgmap v74: 148 pgs: 25 active+undersized, 23 active+undersized+degraded, 100 active+clean; 105 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 13 MiB/s wr, 210 op/s; 586/5610 objects degraded (10.446%) 2024-09-10T18:23:46.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:45 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 586/5610 objects degraded (10.446%), 23 pgs degraded (PG_DEGRADED) 2024-09-10T18:23:46.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:46.244 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:46.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:46.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:46.335 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:46 smithi028 podman[78259]: 2024-09-10 18:23:46.201956023 +0000 UTC m=+0.476221813 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-10T18:23:46.347 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:46.373 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:46.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:46.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:46.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:46.503 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:46.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:46.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:46.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:46.647 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:46.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:46.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:46.740 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:46 smithi028 podman[78259]: 2024-09-10 18:23:46.738574388 +0000 UTC m=+1.012840153 volume create 8e8b5fa0e1bf9b688f601f74fbca3acb39777d64fe48f98b032eb068148e50a1 2024-09-10T18:23:46.749 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:46.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:46.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:46.878 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3 16 8175 8159 169.958 188.375 0.00364075 0.00587346 2024-09-10T18:23:46.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:46.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:46.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:46.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:46.995 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:46 smithi028 podman[78259]: 2024-09-10 18:23:46.855358159 +0000 UTC m=+1.129623928 container create dc1a8a4268b4437f96588c0d28faa4fa187b90695b65b5909b59da02a7ac333f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T18:23:47.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:47.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:47.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:47.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -n osd.6 --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.1.6 2024-09-10T18:23:47.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:47.210 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:47.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:47.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:47.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:47.351 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:47.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:47.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:47.374 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/osd.6/config 2024-09-10T18:23:47.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:47.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:47.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:47.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:47.599 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:47.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:23:47.624 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:47.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:47.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:47.651 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:47 smithi028 podman[78259]: 2024-09-10 18:23:47.42983795 +0000 UTC m=+1.704103732 container init dc1a8a4268b4437f96588c0d28faa4fa187b90695b65b5909b59da02a7ac333f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T18:23:47.651 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:47 smithi028 podman[78259]: 2024-09-10 18:23:47.435953797 +0000 UTC m=+1.710219572 container start dc1a8a4268b4437f96588c0d28faa4fa187b90695b65b5909b59da02a7ac333f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T18:23:47.651 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:47 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a[78449]: ts=2024-09-10T18:23:47.465Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-10T18:23:47.651 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:47 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a[78449]: ts=2024-09-10T18:23:47.465Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-10T18:23:47.651 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:47 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a[78449]: ts=2024-09-10T18:23:47.466Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.28 port=9094 2024-09-10T18:23:47.651 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:47 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a[78449]: ts=2024-09-10T18:23:47.468Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-10T18:23:47.651 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:47 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a[78449]: ts=2024-09-10T18:23:47.517Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-10T18:23:47.651 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:47 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a[78449]: ts=2024-09-10T18:23:47.518Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-10T18:23:47.651 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:47 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a[78449]: ts=2024-09-10T18:23:47.522Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-10T18:23:47.651 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:47 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a[78449]: ts=2024-09-10T18:23:47.522Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-10T18:23:47.726 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:47.750 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:47.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:47.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:47.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:47.878 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4 16 10973 10957 171.183 174.875 0.00806511 0.00583363 2024-09-10T18:23:47.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:47.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:47.906 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:47 smithi028 ceph-mon[39220]: pgmap v75: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 142 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 283 op/s; 717/7371 objects degraded (9.727%) 2024-09-10T18:23:47.906 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:47 smithi028 ceph-mon[29291]: pgmap v75: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 142 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 283 op/s; 717/7371 objects degraded (9.727%) 2024-09-10T18:23:47.906 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:47 smithi028 bash[78259]: dc1a8a4268b4437f96588c0d28faa4fa187b90695b65b5909b59da02a7ac333f 2024-09-10T18:23:47.906 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:47 smithi028 systemd[1]: Started Ceph alertmanager.a for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:23:47.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:48.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:48.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:48.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:47 smithi204 ceph-mon[32357]: pgmap v75: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 142 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 283 op/s; 717/7371 objects degraded (9.727%) 2024-09-10T18:23:48.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:48.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:48.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:48.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:48.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:48.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:48.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:48.482 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:48.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:48.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:48.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:48.613 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:48.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:48.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:48.715 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:48.741 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:48.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:48.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:48.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:48.870 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:48.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:48.878 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5 16 13540 13524 169.031 160.438 0.0101521 0.00590826 2024-09-10T18:23:48.972 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:48.998 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:48.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:48.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:49.100 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:49.108 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:48 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:23:49.108 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:48 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:23:49.108 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:48 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:49.108 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:48 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:49.108 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:48 smithi204 ceph-mon[32357]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-10T18:23:49.109 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:48 smithi204 ceph-mon[32357]: Reconfiguring daemon prometheus.a on smithi204 2024-09-10T18:23:49.109 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:48 smithi204 ceph-mon[32357]: osdmap e99: 8 total, 7 up, 8 in 2024-09-10T18:23:49.126 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:49.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:49.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:49.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:49.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:48 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:23:49.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:48 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:23:49.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:48 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:49.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:48 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:49.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:48 smithi028 ceph-mon[39220]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-10T18:23:49.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:48 smithi028 ceph-mon[39220]: Reconfiguring daemon prometheus.a on smithi204 2024-09-10T18:23:49.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:48 smithi028 ceph-mon[39220]: osdmap e99: 8 total, 7 up, 8 in 2024-09-10T18:23:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:48 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:23:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:48 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:23:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:48 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:48 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:48 smithi028 ceph-mon[29291]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-10T18:23:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:48 smithi028 ceph-mon[29291]: Reconfiguring daemon prometheus.a on smithi204 2024-09-10T18:23:49.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:48 smithi028 ceph-mon[29291]: osdmap e99: 8 total, 7 up, 8 in 2024-09-10T18:23:49.266 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:49.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:49.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:49.368 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:49.396 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:49.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:49.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:49.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:49.541 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:49.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:49.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:49.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:49.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:49.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:49.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:49.713 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:49 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a[78449]: ts=2024-09-10T18:23:49.469Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001028248s 2024-09-10T18:23:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:49.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:49.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:49.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:49.878 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 6 16 16159 16143 168.135 163.688 0.00640761 0.00594015 2024-09-10T18:23:49.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:49.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:49.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:49.979 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:49 smithi028 ceph-mon[39220]: pgmap v76: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 142 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 572 B/s rd, 16 MiB/s wr, 253 op/s; 717/7371 objects degraded (9.727%) 2024-09-10T18:23:49.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:49 smithi028 ceph-mon[29291]: pgmap v76: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 142 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 572 B/s rd, 16 MiB/s wr, 253 op/s; 717/7371 objects degraded (9.727%) 2024-09-10T18:23:50.062 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:50.087 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:50.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:50.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:50.159 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:49 smithi204 ceph-mon[32357]: pgmap v76: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 142 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 572 B/s rd, 16 MiB/s wr, 253 op/s; 717/7371 objects degraded (9.727%) 2024-09-10T18:23:50.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:50.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:50.312 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:23:50.313 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:23:50.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:50.374 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:50.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:50.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:50.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:50.477 INFO:teuthology.orchestra.run.smithi204.stdout:get_pg_num_history pg_num_history pg_num_history(e94 pg_nums {1={20=1},2={57=3},3={57=32},4={59=32},5={61=32},6={63=32},7={93=16}} deleted_pools ) 2024-09-10T18:23:50.478 INFO:teuthology.orchestra.run.smithi204.stdout:Importing pgid 4.1 2024-09-10T18:23:50.478 INFO:teuthology.orchestra.run.smithi204.stdout:Write #4:80542eaa:::obj_delete_at_hint.0000000068:head# 2024-09-10T18:23:50.478 INFO:teuthology.orchestra.run.smithi204.stdout:snapset 0=[]:{} 2024-09-10T18:23:50.478 INFO:teuthology.orchestra.run.smithi204.stdout:Write #4:80b6fe23:reshard::reshard.0000000011:head# 2024-09-10T18:23:50.478 INFO:teuthology.orchestra.run.smithi204.stdout:snapset 0=[]:{} 2024-09-10T18:23:50.478 INFO:teuthology.orchestra.run.smithi204.stdout:Write #4:80d23011:::obj_delete_at_hint.0000000085:head# 2024-09-10T18:23:50.478 INFO:teuthology.orchestra.run.smithi204.stdout:snapset 0=[]:{} 2024-09-10T18:23:50.478 INFO:teuthology.orchestra.run.smithi204.stdout:Write #4:8411e274:::obj_delete_at_hint.0000000073:head# 2024-09-10T18:23:50.478 INFO:teuthology.orchestra.run.smithi204.stdout:snapset 0=[]:{} 2024-09-10T18:23:50.478 INFO:teuthology.orchestra.run.smithi204.stdout:Write #4:84d19476:::obj_delete_at_hint.0000000038:head# 2024-09-10T18:23:50.478 INFO:teuthology.orchestra.run.smithi204.stdout:snapset 0=[]:{} 2024-09-10T18:23:50.479 INFO:teuthology.orchestra.run.smithi204.stdout:Write #4:856b9b26:reshard::reshard.0000000002:head# 2024-09-10T18:23:50.479 INFO:teuthology.orchestra.run.smithi204.stdout:snapset 0=[]:{} 2024-09-10T18:23:50.479 INFO:teuthology.orchestra.run.smithi204.stdout:Write #4:86cc8a85:::obj_delete_at_hint.0000000118:head# 2024-09-10T18:23:50.479 INFO:teuthology.orchestra.run.smithi204.stdout:snapset 0=[]:{} 2024-09-10T18:23:50.479 INFO:teuthology.orchestra.run.smithi204.stdout:write_pg epoch 82 info 4.1( v 65'14 (0'0,65'14] local-lis/les=59/60 n=7 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) 2024-09-10T18:23:50.480 INFO:teuthology.orchestra.run.smithi204.stdout:Import successful 2024-09-10T18:23:50.503 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:50.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:50.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:50.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:50.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:50.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:50.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:50.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:50.773 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:50.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:50.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:50.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:50.878 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 7 16 18425 18409 164.346 141.625 0.00716672 0.00607744 2024-09-10T18:23:50.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:50.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:50.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:51.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:50 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 717/7371 objects degraded (9.727%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:23:51.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:50 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 717/7371 objects degraded (9.727%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:23:51.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:51.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:51.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:51.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:51.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:50 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 717/7371 objects degraded (9.727%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:23:51.163 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:51.189 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:51.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:51.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:51.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T18:23:51.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:51.339 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:51.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:51.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:51.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:51.470 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:51.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:51.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:51.572 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:51.598 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:51.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:51.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:51.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:51.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:51.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:51.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:51.878 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 8 16 20823 20807 162.535 149.875 0.00602341 0.00614643 2024-09-10T18:23:51.883 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:51.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:51.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:51.900 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:23:51.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:52.026 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:52.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:52.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:52.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:51 smithi204 ceph-mon[32357]: pgmap v78: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 989 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 99 MiB/s wr, 1.59k op/s; 4920/48015 objects degraded (10.247%) 2024-09-10T18:23:52.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:51 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1703949501' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:23:52.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:52.169 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:52.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:52.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:52.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:51 smithi028 ceph-mon[29291]: pgmap v78: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 989 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 99 MiB/s wr, 1.59k op/s; 4920/48015 objects degraded (10.247%) 2024-09-10T18:23:52.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:51 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1703949501' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:23:52.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:51 smithi028 ceph-mon[39220]: pgmap v78: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 989 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 99 MiB/s wr, 1.59k op/s; 4920/48015 objects degraded (10.247%) 2024-09-10T18:23:52.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:51 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1703949501' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:23:52.271 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:52.297 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:52.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:52.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:52.423 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:52.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:52.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:52.525 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:52.551 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:52.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:52.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:52.560 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:23:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:23:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:23:52.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:52.693 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:52.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:52.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:52.794 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:52.821 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:52.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:52.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:52.854 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:52 smithi204 systemd[1]: Stopping Ceph prometheus.a for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:23:52.879 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 9 16 23217 23201 161.097 149.625 0.00731936 0.00620128 2024-09-10T18:23:52.923 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:52.949 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:52.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:52.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:53.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:52 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:23:53.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:52 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1703949501' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:23:53.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:52 smithi028 ceph-mon[29291]: osdmap e100: 8 total, 7 up, 8 in 2024-09-10T18:23:53.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:52 smithi028 ceph-mon[29291]: 5.19 scrub starts 2024-09-10T18:23:53.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:52 smithi028 ceph-mon[29291]: 5.19 scrub ok 2024-09-10T18:23:53.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:52 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:23:53.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:52 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1703949501' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:23:53.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:52 smithi028 ceph-mon[39220]: osdmap e100: 8 total, 7 up, 8 in 2024-09-10T18:23:53.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:52 smithi028 ceph-mon[39220]: 5.19 scrub starts 2024-09-10T18:23:53.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:52 smithi028 ceph-mon[39220]: 5.19 scrub ok 2024-09-10T18:23:53.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:53.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:53.090 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:53.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:53.145 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:52 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:23:53.145 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:52 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1703949501' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:23:53.145 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:52 smithi204 ceph-mon[32357]: osdmap e100: 8 total, 7 up, 8 in 2024-09-10T18:23:53.145 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:52 smithi204 ceph-mon[32357]: 5.19 scrub starts 2024-09-10T18:23:53.145 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:52 smithi204 ceph-mon[32357]: 5.19 scrub ok 2024-09-10T18:23:53.146 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/exp.4.1.6 2024-09-10T18:23:53.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:53.249 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:53.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:53.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:53.288 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:23:53.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:53.381 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:53.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:53.483 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:53.512 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:53.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:53.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:53.614 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:53.641 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:53.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:53.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:53.742 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:53.768 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:53.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:53.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:53.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:53.879 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 10 16 25403 25387 158.647 136.625 0.00632219 0.00629591 2024-09-10T18:23:53.911 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:53.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:53.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:54.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:53 smithi028 ceph-mon[39220]: 5.b scrub starts 2024-09-10T18:23:54.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:53 smithi028 ceph-mon[39220]: 5.b scrub ok 2024-09-10T18:23:54.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:53 smithi028 ceph-mon[39220]: pgmap v80: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 989 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 110 MiB/s wr, 1.77k op/s; 4920/48015 objects degraded (10.247%) 2024-09-10T18:23:54.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:53 smithi028 ceph-mon[39220]: 6.12 scrub starts 2024-09-10T18:23:54.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:53 smithi028 ceph-mon[39220]: 6.12 scrub ok 2024-09-10T18:23:54.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:53 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:54.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:53 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:23:54.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:53 smithi028 ceph-mon[29291]: 5.b scrub starts 2024-09-10T18:23:54.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:53 smithi028 ceph-mon[29291]: 5.b scrub ok 2024-09-10T18:23:54.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:53 smithi028 ceph-mon[29291]: pgmap v80: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 989 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 110 MiB/s wr, 1.77k op/s; 4920/48015 objects degraded (10.247%) 2024-09-10T18:23:54.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:53 smithi028 ceph-mon[29291]: 6.12 scrub starts 2024-09-10T18:23:54.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:53 smithi028 ceph-mon[29291]: 6.12 scrub ok 2024-09-10T18:23:54.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:53 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:54.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:53 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:23:54.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:54.013 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:53 smithi204 ceph-mon[32357]: 5.b scrub starts 2024-09-10T18:23:54.013 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:53 smithi204 ceph-mon[32357]: 5.b scrub ok 2024-09-10T18:23:54.013 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:53 smithi204 ceph-mon[32357]: pgmap v80: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 989 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 110 MiB/s wr, 1.77k op/s; 4920/48015 objects degraded (10.247%) 2024-09-10T18:23:54.013 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:53 smithi204 ceph-mon[32357]: 6.12 scrub starts 2024-09-10T18:23:54.014 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:53 smithi204 ceph-mon[32357]: 6.12 scrub ok 2024-09-10T18:23:54.014 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:53 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:54.014 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:53 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:23:54.014 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:23:53 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:23:54.014 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:23:54.010Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-10T18:23:54.014 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:23:54.010Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-10T18:23:54.014 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:23:54.010Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-10T18:23:54.014 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:23:54.010Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-10T18:23:54.014 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:23:54.010Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-10T18:23:54.015 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:23:54.010Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-10T18:23:54.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:54.062 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:54.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:54.118 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:23:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:23:54.159 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:23:54.160 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:23:54.163 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:54.188 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:54.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:54.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:54.197 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:23:54.197 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:23:54.237 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:23:54.237 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:23:54.268 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:23:54.268 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:23:54.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:54.299 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:23:54.300 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:23:54.301 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-10T18:23:54.313 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:23:54.010Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-10T18:23:54.313 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:23:54.011Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-10T18:23:54.314 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:23:54.011Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-10T18:23:54.314 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:23:54.011Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-10T18:23:54.314 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:23:54.010Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-10T18:23:54.314 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[61048]: ts=2024-09-10T18:23:54.012Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-10T18:23:54.314 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 podman[68417]: 2024-09-10 18:23:54.021138249 +0000 UTC m=+0.639533957 container died 0bff8348e47dc6f65d05a5343cd38c7fe487e2f07d57961011ae3bfb1e15fe7b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T18:23:54.329 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:54.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:54.332 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:23:54.332 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:23:54.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:54.471 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:54.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:54.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:54.604 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 podman[68417]: 2024-09-10 18:23:54.419017907 +0000 UTC m=+1.037413614 container cleanup 0bff8348e47dc6f65d05a5343cd38c7fe487e2f07d57961011ae3bfb1e15fe7b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T18:23:54.604 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 bash[68417]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a 2024-09-10T18:23:54.610 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:54.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:54.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:54.739 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:54.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:54.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:54.764 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:23:54.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:54.880 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 11 16 27282 27266 154.885 117.438 0.0108264 0.00644964 2024-09-10T18:23:54.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:54.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:54.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:55.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:55.032 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:55.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:55.082 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-mon[32357]: 5.a scrub starts 2024-09-10T18:23:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-mon[32357]: 5.a scrub ok 2024-09-10T18:23:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-mon[32357]: 3.1b scrub starts 2024-09-10T18:23:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-mon[32357]: 3.1b scrub ok 2024-09-10T18:23:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/624003239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:23:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-mon[32357]: 6.7 scrub starts 2024-09-10T18:23:55.083 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:54 smithi204 ceph-mon[32357]: 6.7 scrub ok 2024-09-10T18:23:55.083 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 podman[68489]: 2024-09-10 18:23:54.712568888 +0000 UTC m=+0.690233078 container remove 0bff8348e47dc6f65d05a5343cd38c7fe487e2f07d57961011ae3bfb1e15fe7b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T18:23:55.083 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-10T18:23:55.083 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 systemd[1]: Stopped Ceph prometheus.a for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:23:55.083 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@prometheus.a.service: Consumed 1.186s CPU time. 2024-09-10T18:23:55.084 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:54 smithi204 systemd[1]: Starting Ceph prometheus.a for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:23:55.133 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:55.161 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:55.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:55.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:55.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:54 smithi028 ceph-mon[39220]: 5.a scrub starts 2024-09-10T18:23:55.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:54 smithi028 ceph-mon[39220]: 5.a scrub ok 2024-09-10T18:23:55.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:54 smithi028 ceph-mon[39220]: 3.1b scrub starts 2024-09-10T18:23:55.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:54 smithi028 ceph-mon[39220]: 3.1b scrub ok 2024-09-10T18:23:55.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:54 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:55.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:54 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/624003239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:23:55.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:54 smithi028 ceph-mon[39220]: 6.7 scrub starts 2024-09-10T18:23:55.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:54 smithi028 ceph-mon[39220]: 6.7 scrub ok 2024-09-10T18:23:55.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:54 smithi028 ceph-mon[29291]: 5.a scrub starts 2024-09-10T18:23:55.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:54 smithi028 ceph-mon[29291]: 5.a scrub ok 2024-09-10T18:23:55.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:54 smithi028 ceph-mon[29291]: 3.1b scrub starts 2024-09-10T18:23:55.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:54 smithi028 ceph-mon[29291]: 3.1b scrub ok 2024-09-10T18:23:55.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:54 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:23:55.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:54 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/624003239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:23:55.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:54 smithi028 ceph-mon[29291]: 6.7 scrub starts 2024-09-10T18:23:55.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:54 smithi028 ceph-mon[29291]: 6.7 scrub ok 2024-09-10T18:23:55.263 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:55.291 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:55.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:55.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:55.354 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 podman[68595]: 2024-09-10 18:23:54.985787735 +0000 UTC m=+0.020396503 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-10T18:23:55.354 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 podman[68595]: 2024-09-10 18:23:55.147095901 +0000 UTC m=+0.181704645 container create 18fcf77c26a7c6edeb2ec241aeac651efd7c8e965b327fdabe9ca92764b541bd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T18:23:55.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:55.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:55.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:55.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:55.546 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:55.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:55.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:55.648 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:55.672 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:55.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:55.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:55.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:55] ENGINE Bus STOPPING 2024-09-10T18:23:55.774 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:55.795 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:55.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:55.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:55.855 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 podman[68595]: 2024-09-10 18:23:55.383417716 +0000 UTC m=+0.418026461 container init 18fcf77c26a7c6edeb2ec241aeac651efd7c8e965b327fdabe9ca92764b541bd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T18:23:55.855 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 podman[68595]: 2024-09-10 18:23:55.387461075 +0000 UTC m=+0.422069825 container start 18fcf77c26a7c6edeb2ec241aeac651efd7c8e965b327fdabe9ca92764b541bd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T18:23:55.855 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.415Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-10T18:23:55.855 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.416Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-10T18:23:55.855 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.416Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi204 (none))" 2024-09-10T18:23:55.855 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.416Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-10T18:23:55.855 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.416Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-10T18:23:55.855 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.417Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-10T18:23:55.855 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.417Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-10T18:23:55.856 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.419Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-10T18:23:55.856 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.419Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-10T18:23:55.856 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.421Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-10T18:23:55.856 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.421Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.477µs 2024-09-10T18:23:55.856 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.421Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-10T18:23:55.856 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.421Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-10T18:23:55.856 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.421Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-10T18:23:55.856 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.421Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.2µs wal_replay_duration=402.647µs wbl_replay_duration=88ns total_replay_duration=438.693µs 2024-09-10T18:23:55.856 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.422Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-10T18:23:55.856 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.422Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-10T18:23:55.856 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.422Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-10T18:23:55.856 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.436Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.486193ms db_storage=765ns remote_storage=1.248µs web_handler=385ns query_engine=524ns scrape=391.646µs scrape_sd=23.725µs notify=24.438µs notify_sd=6.363µs rules=13.815431ms tracing=6.15µs 2024-09-10T18:23:55.857 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.436Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-10T18:23:55.857 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:23:55.436Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-10T18:23:55.857 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 bash[68595]: 18fcf77c26a7c6edeb2ec241aeac651efd7c8e965b327fdabe9ca92764b541bd 2024-09-10T18:23:55.857 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:23:55 smithi204 systemd[1]: Started Ceph prometheus.a for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:23:55.880 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 12 16 28670 28654 149.206 86.75 0.00777034 0.00669551 2024-09-10T18:23:55.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:55.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:55.944 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:55.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:56.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:56.083 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:56.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:56.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:56.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:56.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:56.254 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T18:23:56.254 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:55] ENGINE Bus STOPPED 2024-09-10T18:23:56.254 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:55] ENGINE Bus STARTING 2024-09-10T18:23:56.254 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:55] ENGINE Serving on http://:::9283 2024-09-10T18:23:56.254 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:55] ENGINE Bus STARTED 2024-09-10T18:23:56.254 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:55] ENGINE Bus STOPPING 2024-09-10T18:23:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[39220]: pgmap v81: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 1.6 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 193 MiB/s wr, 3.08k op/s; 8381/81444 objects degraded (10.291%) 2024-09-10T18:23:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[39220]: 3.1f scrub starts 2024-09-10T18:23:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[39220]: 3.1f scrub ok 2024-09-10T18:23:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T18:23:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T18:23:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T18:23:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T18:23:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 8381/81444 objects degraded (10.291%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:23:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T18:23:56.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T18:23:56.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:56.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:23:56.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[29291]: pgmap v81: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 1.6 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 193 MiB/s wr, 3.08k op/s; 8381/81444 objects degraded (10.291%) 2024-09-10T18:23:56.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[29291]: 3.1f scrub starts 2024-09-10T18:23:56.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[29291]: 3.1f scrub ok 2024-09-10T18:23:56.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:56.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:56.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T18:23:56.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T18:23:56.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:56.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T18:23:56.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T18:23:56.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:56.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 8381/81444 objects degraded (10.291%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:23:56.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T18:23:56.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T18:23:56.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:56.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:55 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:23:56.270 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-mon[32357]: pgmap v81: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 1.6 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 193 MiB/s wr, 3.08k op/s; 8381/81444 objects degraded (10.291%) 2024-09-10T18:23:56.270 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-mon[32357]: 3.1f scrub starts 2024-09-10T18:23:56.270 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-mon[32357]: 3.1f scrub ok 2024-09-10T18:23:56.270 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:56.271 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:56.271 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T18:23:56.271 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T18:23:56.271 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:56.271 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T18:23:56.271 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T18:23:56.271 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:56.271 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 8381/81444 objects degraded (10.291%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:23:56.271 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T18:23:56.271 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T18:23:56.271 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:23:56.271 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:55 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:23:56.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:56.354 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:56.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:56.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:56.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:56.487 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:56.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:56.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:56.588 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:56.610 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:56.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:56.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:56.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:56.733 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:56.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:56.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:56.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T18:23:56.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:56] ENGINE Bus STOPPED 2024-09-10T18:23:56.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:56] ENGINE Bus STARTING 2024-09-10T18:23:56.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:56] ENGINE Serving on http://:::9283 2024-09-10T18:23:56.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:56] ENGINE Bus STARTED 2024-09-10T18:23:56.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:56] ENGINE Bus STOPPING 2024-09-10T18:23:56.838 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:56.865 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:56.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:56.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:56.880 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 13 16 30059 30043 144.407 86.8125 0.00398321 0.00691879 2024-09-10T18:23:56.966 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:56.991 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:56.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:57.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:57.133 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:57.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:57.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:57.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:57.254 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T18:23:57.255 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:56] ENGINE Bus STOPPED 2024-09-10T18:23:57.255 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:56] ENGINE Bus STARTING 2024-09-10T18:23:57.255 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:57 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:57] ENGINE Serving on http://:::9283 2024-09-10T18:23:57.255 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:23:57 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:23:57] ENGINE Bus STARTED 2024-09-10T18:23:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[39220]: 6.b scrub starts 2024-09-10T18:23:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[39220]: 6.b scrub ok 2024-09-10T18:23:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[39220]: 5.8 scrub starts 2024-09-10T18:23:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[39220]: 5.8 scrub ok 2024-09-10T18:23:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[39220]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T18:23:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[39220]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T18:23:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[39220]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T18:23:57.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[39220]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T18:23:57.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[39220]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T18:23:57.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[39220]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T18:23:57.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[29291]: 6.b scrub starts 2024-09-10T18:23:57.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[29291]: 6.b scrub ok 2024-09-10T18:23:57.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[29291]: 5.8 scrub starts 2024-09-10T18:23:57.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[29291]: 5.8 scrub ok 2024-09-10T18:23:57.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[29291]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T18:23:57.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[29291]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T18:23:57.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[29291]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T18:23:57.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[29291]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T18:23:57.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[29291]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T18:23:57.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:56 smithi028 ceph-mon[29291]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T18:23:57.275 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:57.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:57.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:56 smithi204 ceph-mon[32357]: 6.b scrub starts 2024-09-10T18:23:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:56 smithi204 ceph-mon[32357]: 6.b scrub ok 2024-09-10T18:23:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:56 smithi204 ceph-mon[32357]: 5.8 scrub starts 2024-09-10T18:23:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:56 smithi204 ceph-mon[32357]: 5.8 scrub ok 2024-09-10T18:23:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:56 smithi204 ceph-mon[32357]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T18:23:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:56 smithi204 ceph-mon[32357]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T18:23:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:56 smithi204 ceph-mon[32357]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T18:23:57.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:56 smithi204 ceph-mon[32357]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi204.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T18:23:57.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:56 smithi204 ceph-mon[32357]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T18:23:57.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:56 smithi204 ceph-mon[32357]: from='mon.? v1:172.21.15.28:0/3901794868' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi204.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T18:23:57.377 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:57.401 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:57.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:57.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:57.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:57.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:57.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:57.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:57.669 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:57.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:57.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:57.754 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:23:57 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a[78449]: ts=2024-09-10T18:23:57.472Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00447464s 2024-09-10T18:23:57.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:57.811 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:57.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:57.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:57.880 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 14 16 31768 31752 141.721 106.812 0.00149869 0.00703106 2024-09-10T18:23:57.913 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:57.934 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:57.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:57.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:58.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:58.072 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:58.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:58.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:58.173 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:58.181 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-09-10T18:23:58.181 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:23:58.181 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-10T18:23:58.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-10T18:23:58.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:58.198 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:58.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:58.218 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:57 smithi028 ceph-mon[39220]: pgmap v82: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 1.6 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 193 MiB/s wr, 3.08k op/s; 8381/81444 objects degraded (10.291%) 2024-09-10T18:23:58.218 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:57 smithi028 ceph-mon[39220]: 6.0 scrub starts 2024-09-10T18:23:58.218 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:57 smithi028 ceph-mon[39220]: 6.0 scrub ok 2024-09-10T18:23:58.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:57 smithi028 ceph-mon[29291]: pgmap v82: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 1.6 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 193 MiB/s wr, 3.08k op/s; 8381/81444 objects degraded (10.291%) 2024-09-10T18:23:58.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:57 smithi028 ceph-mon[29291]: 6.0 scrub starts 2024-09-10T18:23:58.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:57 smithi028 ceph-mon[29291]: 6.0 scrub ok 2024-09-10T18:23:58.251 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:57 smithi204 ceph-mon[32357]: pgmap v82: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 1.6 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 193 MiB/s wr, 3.08k op/s; 8381/81444 objects degraded (10.291%) 2024-09-10T18:23:58.251 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:57 smithi204 ceph-mon[32357]: 6.0 scrub starts 2024-09-10T18:23:58.251 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:57 smithi204 ceph-mon[32357]: 6.0 scrub ok 2024-09-10T18:23:58.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:58.350 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:58.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:58.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:58.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:58.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:23:58.600 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:23:58.626 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:58.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:58.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:23:58.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:23:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:58.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:58.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:23:58.866 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:58.880 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 15 16 33417 33401 139.143 103.062 0.00758034 0.00718053 2024-09-10T18:23:58.891 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:58.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:58.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:58.992 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:23:59.022 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:59.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:59.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:23:59.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:23:59.160 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:59.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:59.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:23:59.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:58 smithi028 ceph-mon[29291]: 5.13 scrub starts 2024-09-10T18:23:59.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:58 smithi028 ceph-mon[29291]: 5.13 scrub ok 2024-09-10T18:23:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:58 smithi028 ceph-mon[39220]: 5.13 scrub starts 2024-09-10T18:23:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:58 smithi028 ceph-mon[39220]: 5.13 scrub ok 2024-09-10T18:23:59.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:59.286 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:59.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:59.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:58 smithi204 ceph-mon[32357]: 5.13 scrub starts 2024-09-10T18:23:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:58 smithi204 ceph-mon[32357]: 5.13 scrub ok 2024-09-10T18:23:59.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:23:59.414 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:23:59.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:59.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:23:59.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:59.556 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:59.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:59.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:59.703 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:59.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:59.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:59.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:23:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:59.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:59.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:23:59.880 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 16 16 34948 34932 136.427 95.6875 0.00413387 0.00732338 2024-09-10T18:23:59.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:23:59.970 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:23:59.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:23:59.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:00.072 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:00.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:00.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:00.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:00.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:00.244 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:00.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:00.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:00.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:23:59 smithi028 ceph-mon[39220]: pgmap v83: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 1.6 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 155 MiB/s wr, 2.49k op/s; 8381/81444 objects degraded (10.291%) 2024-09-10T18:24:00.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:23:59 smithi028 ceph-mon[29291]: pgmap v83: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 1.6 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 155 MiB/s wr, 2.49k op/s; 8381/81444 objects degraded (10.291%) 2024-09-10T18:24:00.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:23:59 smithi204 ceph-mon[32357]: pgmap v83: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 1.6 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 155 MiB/s wr, 2.49k op/s; 8381/81444 objects degraded (10.291%) 2024-09-10T18:24:00.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:00.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:00.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:00.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:00.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:00.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:00.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:00.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:00.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:00.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:00.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:00.808 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:00.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:00.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:00.881 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 17 16 36526 36510 134.203 98.625 0.00540486 0.00744539 2024-09-10T18:24:00.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:00.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:00.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:00.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:01.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:01.085 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:01.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:01.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:01.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:01.226 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:01.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:01.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[29291]: 3.4 scrub starts 2024-09-10T18:24:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[29291]: 3.4 scrub ok 2024-09-10T18:24:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:24:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:24:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[29291]: 4.1b scrub starts 2024-09-10T18:24:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[29291]: 4.1b scrub ok 2024-09-10T18:24:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[29291]: osdmap e101: 8 total, 7 up, 8 in 2024-09-10T18:24:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[29291]: 6.11 scrub starts 2024-09-10T18:24:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[29291]: 6.11 scrub ok 2024-09-10T18:24:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[29291]: osdmap e102: 8 total, 7 up, 8 in 2024-09-10T18:24:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[39220]: 3.4 scrub starts 2024-09-10T18:24:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[39220]: 3.4 scrub ok 2024-09-10T18:24:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:24:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:24:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[39220]: 4.1b scrub starts 2024-09-10T18:24:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[39220]: 4.1b scrub ok 2024-09-10T18:24:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[39220]: osdmap e101: 8 total, 7 up, 8 in 2024-09-10T18:24:01.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[39220]: 6.11 scrub starts 2024-09-10T18:24:01.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[39220]: 6.11 scrub ok 2024-09-10T18:24:01.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:00 smithi028 ceph-mon[39220]: osdmap e102: 8 total, 7 up, 8 in 2024-09-10T18:24:01.306 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:00 smithi204 ceph-mon[32357]: 3.4 scrub starts 2024-09-10T18:24:01.306 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:00 smithi204 ceph-mon[32357]: 3.4 scrub ok 2024-09-10T18:24:01.306 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:00 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:24:01.306 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:00 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:24:01.306 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:00 smithi204 ceph-mon[32357]: 4.1b scrub starts 2024-09-10T18:24:01.306 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:00 smithi204 ceph-mon[32357]: 4.1b scrub ok 2024-09-10T18:24:01.307 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:00 smithi204 ceph-mon[32357]: osdmap e101: 8 total, 7 up, 8 in 2024-09-10T18:24:01.307 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:00 smithi204 ceph-mon[32357]: 6.11 scrub starts 2024-09-10T18:24:01.307 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:00 smithi204 ceph-mon[32357]: 6.11 scrub ok 2024-09-10T18:24:01.307 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:00 smithi204 ceph-mon[32357]: osdmap e102: 8 total, 7 up, 8 in 2024-09-10T18:24:01.327 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:01.356 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:01.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:01.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:01.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:01.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:01.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:01.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:01.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:01.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:01.743 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:01.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:24:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:24:01] "GET /metrics HTTP/1.1" 200 223216 "" "Prometheus/2.43.0" 2024-09-10T18:24:01.763 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:01.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:01.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:01.865 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:01.880 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 18 16 37925 37909 131.605 87.4375 0.00797332 0.00759194 2024-09-10T18:24:01.890 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:01.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:01.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:01.992 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:02.016 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:02.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:02.118 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:02.144 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:02.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:02.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:02.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:01 smithi028 ceph-mon[29291]: 6.4 scrub starts 2024-09-10T18:24:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:01 smithi028 ceph-mon[29291]: 6.4 scrub ok 2024-09-10T18:24:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:01 smithi028 ceph-mon[29291]: pgmap v86: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 2.1 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.36k op/s; 10774/104742 objects degraded (10.286%) 2024-09-10T18:24:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:01 smithi028 ceph-mon[29291]: 6.18 scrub starts 2024-09-10T18:24:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:01 smithi028 ceph-mon[29291]: 6.18 scrub ok 2024-09-10T18:24:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:01 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 10774/104742 objects degraded (10.286%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:01 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:01 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:02.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:01 smithi028 ceph-mon[39220]: 6.4 scrub starts 2024-09-10T18:24:02.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:01 smithi028 ceph-mon[39220]: 6.4 scrub ok 2024-09-10T18:24:02.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:01 smithi028 ceph-mon[39220]: pgmap v86: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 2.1 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.36k op/s; 10774/104742 objects degraded (10.286%) 2024-09-10T18:24:02.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:01 smithi028 ceph-mon[39220]: 6.18 scrub starts 2024-09-10T18:24:02.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:01 smithi028 ceph-mon[39220]: 6.18 scrub ok 2024-09-10T18:24:02.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:01 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 10774/104742 objects degraded (10.286%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:02.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:01 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:02.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:01 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:02.283 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:02.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:02.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:02.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:24:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:24:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:24:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:01 smithi204 ceph-mon[32357]: 6.4 scrub starts 2024-09-10T18:24:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:01 smithi204 ceph-mon[32357]: 6.4 scrub ok 2024-09-10T18:24:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:01 smithi204 ceph-mon[32357]: pgmap v86: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 2.1 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.36k op/s; 10774/104742 objects degraded (10.286%) 2024-09-10T18:24:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:01 smithi204 ceph-mon[32357]: 6.18 scrub starts 2024-09-10T18:24:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:01 smithi204 ceph-mon[32357]: 6.18 scrub ok 2024-09-10T18:24:02.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:01 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 10774/104742 objects degraded (10.286%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:02.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:01 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:02.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:01 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:02.385 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:02.411 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:02.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:02.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:02.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:02.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:02.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:02.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:02.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:02.714 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T18:24:02.714 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T18:24:02.803 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:02.827 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:02.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:02.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:02.882 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 19 16 39659 39643 130.37 108.375 0.00759285 0.00766514 2024-09-10T18:24:02.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:02.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:02.971 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:02.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:03.072 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:03.095 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:03.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:03.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:03.197 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:03.221 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:03.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:03.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:24:03.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:02 smithi028 ceph-mon[39220]: 6.f scrub starts 2024-09-10T18:24:03.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:02 smithi028 ceph-mon[39220]: 6.f scrub ok 2024-09-10T18:24:03.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:02 smithi028 ceph-mon[39220]: 3.16 scrub starts 2024-09-10T18:24:03.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:02 smithi028 ceph-mon[39220]: 3.16 scrub ok 2024-09-10T18:24:03.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:02 smithi028 ceph-mon[29291]: 6.f scrub starts 2024-09-10T18:24:03.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:02 smithi028 ceph-mon[29291]: 6.f scrub ok 2024-09-10T18:24:03.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:02 smithi028 ceph-mon[29291]: 3.16 scrub starts 2024-09-10T18:24:03.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:02 smithi028 ceph-mon[29291]: 3.16 scrub ok 2024-09-10T18:24:03.323 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:03.348 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:03.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:03.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:02 smithi204 ceph-mon[32357]: 6.f scrub starts 2024-09-10T18:24:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:02 smithi204 ceph-mon[32357]: 6.f scrub ok 2024-09-10T18:24:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:02 smithi204 ceph-mon[32357]: 3.16 scrub starts 2024-09-10T18:24:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:02 smithi204 ceph-mon[32357]: 3.16 scrub ok 2024-09-10T18:24:03.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:03.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:03.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:24:03.597 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:03.622 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:03.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:03.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:03.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:03.762 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:03.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:03.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:03.864 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:03.882 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:24:03.881876+0000 min lat: 0.00103822 max lat: 0.253055 avg lat: 0.00773807 2024-09-10T18:24:03.882 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:24:03.883 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 20 16 41335 41319 129.089 104.75 0.00353951 0.00773807 2024-09-10T18:24:03.887 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:03.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:03.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:03.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:04.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:04.040 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:24:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:24:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:41461 7.d 7:b97c8a4d:::benchmark_data_smithi028_77856_object41460:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-10T18:24:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:24:03.988813+0000", 2024-09-10T18:24:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.002947889, 2024-09-10T18:24:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029926269999999999, 2024-09-10T18:24:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:24:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:24:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:24:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:24:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:24:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 41461 2024-09-10T18:24:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:24:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:24:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:24:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:03.988813+0000", 2024-09-10T18:24:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:24:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:24:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:24:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:03.988813+0000", 2024-09-10T18:24:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:24:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:24:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:24:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:03.988811+0000", 2024-09-10T18:24:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-10T18:24:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:24:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:24:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:03.988848+0000", 2024-09-10T18:24:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7267000000000001e-05 2024-09-10T18:24:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:24:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:24:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:03.988861+0000", 2024-09-10T18:24:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2986999999999999e-05 2024-09-10T18:24:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:24:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:24:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:03.988874+0000", 2024-09-10T18:24:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.275e-05 2024-09-10T18:24:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:24:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:24:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:03.988927+0000", 2024-09-10T18:24:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3567999999999999e-05 2024-09-10T18:24:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:24:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:24:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:03.989065+0000", 2024-09-10T18:24:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013798099999999999 2024-09-10T18:24:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:24:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-10T18:24:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:03.989224+0000", 2024-09-10T18:24:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015903299999999999 2024-09-10T18:24:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:24:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:24:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:24:04.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:24:04.045 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:24:04.045 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 1 2024-09-10T18:24:04.045 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:24:04.134 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:04.159 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:04.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:03 smithi028 ceph-mon[39220]: 5.12 scrub starts 2024-09-10T18:24:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:03 smithi028 ceph-mon[39220]: 5.12 scrub ok 2024-09-10T18:24:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:03 smithi028 ceph-mon[39220]: 4.14 scrub starts 2024-09-10T18:24:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:03 smithi028 ceph-mon[39220]: 4.14 scrub ok 2024-09-10T18:24:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:03 smithi028 ceph-mon[39220]: pgmap v87: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 2.1 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 970 op/s; 10774/104742 objects degraded (10.286%) 2024-09-10T18:24:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:03 smithi028 ceph-mon[39220]: 5.1 scrub starts 2024-09-10T18:24:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:03 smithi028 ceph-mon[39220]: 5.1 scrub ok 2024-09-10T18:24:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:03 smithi028 ceph-mon[29291]: 5.12 scrub starts 2024-09-10T18:24:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:03 smithi028 ceph-mon[29291]: 5.12 scrub ok 2024-09-10T18:24:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:03 smithi028 ceph-mon[29291]: 4.14 scrub starts 2024-09-10T18:24:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:03 smithi028 ceph-mon[29291]: 4.14 scrub ok 2024-09-10T18:24:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:03 smithi028 ceph-mon[29291]: pgmap v87: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 2.1 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 970 op/s; 10774/104742 objects degraded (10.286%) 2024-09-10T18:24:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:03 smithi028 ceph-mon[29291]: 5.1 scrub starts 2024-09-10T18:24:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:03 smithi028 ceph-mon[29291]: 5.1 scrub ok 2024-09-10T18:24:04.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:04.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:04.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:04.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:24:03 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:24:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:03 smithi204 ceph-mon[32357]: 5.12 scrub starts 2024-09-10T18:24:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:03 smithi204 ceph-mon[32357]: 5.12 scrub ok 2024-09-10T18:24:04.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:03 smithi204 ceph-mon[32357]: 4.14 scrub starts 2024-09-10T18:24:04.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:03 smithi204 ceph-mon[32357]: 4.14 scrub ok 2024-09-10T18:24:04.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:03 smithi204 ceph-mon[32357]: pgmap v87: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 2.1 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 970 op/s; 10774/104742 objects degraded (10.286%) 2024-09-10T18:24:04.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:03 smithi204 ceph-mon[32357]: 5.1 scrub starts 2024-09-10T18:24:04.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:03 smithi204 ceph-mon[32357]: 5.1 scrub ok 2024-09-10T18:24:04.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:04.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:04.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:04.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:04.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:04.577 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:04.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:04.679 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:04.702 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:04.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:04.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:04.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:24:04.804 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:04.827 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:04.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:04.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:04.884 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 21 16 42791 42775 127.269 91 0.0178911 0.00785156 2024-09-10T18:24:04.928 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:04.953 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:04.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:04.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:05.055 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:05.079 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:05.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:05.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:05.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:04 smithi204 ceph-mon[32357]: 4.f scrub starts 2024-09-10T18:24:05.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:04 smithi204 ceph-mon[32357]: 4.f scrub ok 2024-09-10T18:24:05.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:04 smithi204 ceph-mon[32357]: 2.0 scrub starts 2024-09-10T18:24:05.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:04 smithi204 ceph-mon[32357]: 2.0 scrub ok 2024-09-10T18:24:05.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:04 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:05.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:05.228 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:05.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:05.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:05.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:04 smithi028 ceph-mon[39220]: 4.f scrub starts 2024-09-10T18:24:05.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:04 smithi028 ceph-mon[39220]: 4.f scrub ok 2024-09-10T18:24:05.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:04 smithi028 ceph-mon[39220]: 2.0 scrub starts 2024-09-10T18:24:05.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:04 smithi028 ceph-mon[39220]: 2.0 scrub ok 2024-09-10T18:24:05.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:04 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:05.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:04 smithi028 ceph-mon[29291]: 4.f scrub starts 2024-09-10T18:24:05.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:04 smithi028 ceph-mon[29291]: 4.f scrub ok 2024-09-10T18:24:05.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:04 smithi028 ceph-mon[29291]: 2.0 scrub starts 2024-09-10T18:24:05.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:04 smithi028 ceph-mon[29291]: 2.0 scrub ok 2024-09-10T18:24:05.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:04 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:05.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:05.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-10T18:24:05.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:05.387 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:05.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:05.488 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:05.511 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:05.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:05.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:05.613 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:05.637 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:05.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:05.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:05.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:05.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:05.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:05.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:05.885 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 22 16 44142 44126 125.312 84.4375 0.0115025 0.00797325 2024-09-10T18:24:05.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:05.970 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:05.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:06.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:05 smithi028 ceph-mon[29291]: 5.14 scrub starts 2024-09-10T18:24:06.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:05 smithi028 ceph-mon[29291]: 5.14 scrub ok 2024-09-10T18:24:06.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:05 smithi028 ceph-mon[29291]: 5.d scrub starts 2024-09-10T18:24:06.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:05 smithi028 ceph-mon[29291]: 5.d scrub ok 2024-09-10T18:24:06.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:05 smithi028 ceph-mon[29291]: pgmap v88: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 2.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 12681/120723 objects degraded (10.504%) 2024-09-10T18:24:06.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:05 smithi028 ceph-mon[29291]: 3.12 scrub starts 2024-09-10T18:24:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:05 smithi028 ceph-mon[29291]: 3.12 scrub ok 2024-09-10T18:24:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:05 smithi028 ceph-mon[29291]: osdmap e103: 8 total, 7 up, 8 in 2024-09-10T18:24:06.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:05 smithi028 ceph-mon[39220]: 5.14 scrub starts 2024-09-10T18:24:06.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:05 smithi028 ceph-mon[39220]: 5.14 scrub ok 2024-09-10T18:24:06.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:05 smithi028 ceph-mon[39220]: 5.d scrub starts 2024-09-10T18:24:06.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:05 smithi028 ceph-mon[39220]: 5.d scrub ok 2024-09-10T18:24:06.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:05 smithi028 ceph-mon[39220]: pgmap v88: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 2.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 12681/120723 objects degraded (10.504%) 2024-09-10T18:24:06.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:05 smithi028 ceph-mon[39220]: 3.12 scrub starts 2024-09-10T18:24:06.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:05 smithi028 ceph-mon[39220]: 3.12 scrub ok 2024-09-10T18:24:06.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:05 smithi028 ceph-mon[39220]: osdmap e103: 8 total, 7 up, 8 in 2024-09-10T18:24:06.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:06.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:06.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:06.259 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:06.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:06.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:05 smithi204 ceph-mon[32357]: 5.14 scrub starts 2024-09-10T18:24:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:05 smithi204 ceph-mon[32357]: 5.14 scrub ok 2024-09-10T18:24:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:05 smithi204 ceph-mon[32357]: 5.d scrub starts 2024-09-10T18:24:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:05 smithi204 ceph-mon[32357]: 5.d scrub ok 2024-09-10T18:24:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:05 smithi204 ceph-mon[32357]: pgmap v88: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 2.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 12681/120723 objects degraded (10.504%) 2024-09-10T18:24:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:05 smithi204 ceph-mon[32357]: 3.12 scrub starts 2024-09-10T18:24:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:05 smithi204 ceph-mon[32357]: 3.12 scrub ok 2024-09-10T18:24:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:05 smithi204 ceph-mon[32357]: osdmap e103: 8 total, 7 up, 8 in 2024-09-10T18:24:06.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:06.386 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:06.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:06.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:06.488 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:06.511 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:06.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:06.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:06.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:06.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:06.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:06.764 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:06.788 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:06.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:06.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:06.886 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 23 16 45800 45784 124.369 103.625 0.00358542 0.00803553 2024-09-10T18:24:06.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:06.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:06.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:06.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:07.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:07.076 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:07.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:07.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:07.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:07.228 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:07.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:07.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:07.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:07 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 13185/128298 objects degraded (10.277%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:07.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:07 smithi028 ceph-mon[29291]: osdmap e104: 8 total, 7 up, 8 in 2024-09-10T18:24:07.312 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:07 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 13185/128298 objects degraded (10.277%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:07.312 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:07 smithi028 ceph-mon[39220]: osdmap e104: 8 total, 7 up, 8 in 2024-09-10T18:24:07.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:07 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 13185/128298 objects degraded (10.277%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:07 smithi204 ceph-mon[32357]: osdmap e104: 8 total, 7 up, 8 in 2024-09-10T18:24:07.370 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:07.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:07.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:07.472 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:07.495 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:07.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:07.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:07.597 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:07.622 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:07.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:07.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:07.724 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:07.749 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:07.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:07.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:07.850 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:07.877 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:07.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:07.886 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 24 16 47394 47378 123.336 99.625 0.00962378 0.00810234 2024-09-10T18:24:07.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:08.002 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:08.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:08.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:08.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:08.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:08.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:08 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:24:08.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:08 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:24:08.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:08 smithi028 ceph-mon[29291]: pgmap v90: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 2.6 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 755 B/s rd, 72 MiB/s wr, 1.16k op/s; 13185/128298 objects degraded (10.277%) 2024-09-10T18:24:08.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:08 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:24:08.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:08 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:24:08.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:08 smithi028 ceph-mon[39220]: pgmap v90: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 2.6 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 755 B/s rd, 72 MiB/s wr, 1.16k op/s; 13185/128298 objects degraded (10.277%) 2024-09-10T18:24:08.258 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:08.281 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:08.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:08.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:08 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:24:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:08 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:24:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:08 smithi204 ceph-mon[32357]: pgmap v90: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 2.6 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 755 B/s rd, 72 MiB/s wr, 1.16k op/s; 13185/128298 objects degraded (10.277%) 2024-09-10T18:24:08.385 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:08.408 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:08.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:08.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:08.463 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:24:08.510 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:08.534 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:08.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:08.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:08.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:08.660 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:08.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:08.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:08.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:08.805 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:08.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:08.840 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:24:08.886 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 25 16 48843 48827 122.025 90.5625 0.00952827 0.00819029 2024-09-10T18:24:08.907 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:08.951 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:08.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:09.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:09.053 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:24:09.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:09.101 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:09.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:09.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:09.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:09.246 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:09.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:09 smithi028 ceph-mon[39220]: 4.9 scrub starts 2024-09-10T18:24:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:09 smithi028 ceph-mon[39220]: 4.9 scrub ok 2024-09-10T18:24:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:09 smithi028 ceph-mon[39220]: 6.1d scrub starts 2024-09-10T18:24:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:09 smithi028 ceph-mon[39220]: 6.1d scrub ok 2024-09-10T18:24:09.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:09 smithi028 ceph-mon[39220]: 5.1e scrub starts 2024-09-10T18:24:09.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:09 smithi028 ceph-mon[39220]: 5.1e scrub ok 2024-09-10T18:24:09.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:09 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/237374684' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:24:09.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:24:09.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:09 smithi028 ceph-mon[29291]: 4.9 scrub starts 2024-09-10T18:24:09.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:09 smithi028 ceph-mon[29291]: 4.9 scrub ok 2024-09-10T18:24:09.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:09 smithi028 ceph-mon[29291]: 6.1d scrub starts 2024-09-10T18:24:09.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:09 smithi028 ceph-mon[29291]: 6.1d scrub ok 2024-09-10T18:24:09.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:09 smithi028 ceph-mon[29291]: 5.1e scrub starts 2024-09-10T18:24:09.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:09 smithi028 ceph-mon[29291]: 5.1e scrub ok 2024-09-10T18:24:09.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:09 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/237374684' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:24:09.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:24:09.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:09 smithi204 ceph-mon[32357]: 4.9 scrub starts 2024-09-10T18:24:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:09 smithi204 ceph-mon[32357]: 4.9 scrub ok 2024-09-10T18:24:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:09 smithi204 ceph-mon[32357]: 6.1d scrub starts 2024-09-10T18:24:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:09 smithi204 ceph-mon[32357]: 6.1d scrub ok 2024-09-10T18:24:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:09 smithi204 ceph-mon[32357]: 5.1e scrub starts 2024-09-10T18:24:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:09 smithi204 ceph-mon[32357]: 5.1e scrub ok 2024-09-10T18:24:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:09 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/237374684' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:24:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:24:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:09.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:09.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:09.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-10T18:24:09.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:09.526 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:09.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:09.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:09.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:09.685 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:09.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:09.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:09.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:09.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:09.886 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 26 16 50332 50316 120.909 93.0625 0.0128944 0.00826598 2024-09-10T18:24:09.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:09.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:09.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:10.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:10.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:10.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:10.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:10.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:10 smithi028 ceph-mon[29291]: pgmap v92: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 2.6 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 981 op/s; 13185/128298 objects degraded (10.277%) 2024-09-10T18:24:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:10 smithi028 ceph-mon[29291]: 6.8 scrub starts 2024-09-10T18:24:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:10 smithi028 ceph-mon[29291]: 6.8 scrub ok 2024-09-10T18:24:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:10 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:24:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:10 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/237374684' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:24:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:10 smithi028 ceph-mon[29291]: osdmap e105: 8 total, 7 up, 8 in 2024-09-10T18:24:10.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:10 smithi028 ceph-mon[39220]: pgmap v92: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 2.6 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 981 op/s; 13185/128298 objects degraded (10.277%) 2024-09-10T18:24:10.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:10 smithi028 ceph-mon[39220]: 6.8 scrub starts 2024-09-10T18:24:10.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:10 smithi028 ceph-mon[39220]: 6.8 scrub ok 2024-09-10T18:24:10.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:10 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:24:10.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:10 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/237374684' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:24:10.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:10 smithi028 ceph-mon[39220]: osdmap e105: 8 total, 7 up, 8 in 2024-09-10T18:24:10.283 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:10.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:10.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:10 smithi204 ceph-mon[32357]: pgmap v92: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 2.6 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 981 op/s; 13185/128298 objects degraded (10.277%) 2024-09-10T18:24:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:10 smithi204 ceph-mon[32357]: 6.8 scrub starts 2024-09-10T18:24:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:10 smithi204 ceph-mon[32357]: 6.8 scrub ok 2024-09-10T18:24:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:10 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:24:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:10 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/237374684' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:24:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:10 smithi204 ceph-mon[32357]: osdmap e105: 8 total, 7 up, 8 in 2024-09-10T18:24:10.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:10.428 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:10.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:10.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:10.554 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:10.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:10.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:10.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:10.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:10.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:10.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:10.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:10.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:10.886 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 27 16 51888 51872 120.033 97.25 0.00507123 0.00832505 2024-09-10T18:24:10.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:10.994 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:10.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:10.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:11.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:11.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:11.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:11.247 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:11.271 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:11.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:11.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:11.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:11.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:11.423 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:11.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:11.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:24:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:24:11] "GET /metrics HTTP/1.1" 200 223493 "" "Prometheus/2.43.0" 2024-09-10T18:24:11.524 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:11.549 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:11.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:11.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:11.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:11.692 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:11.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:11.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:11.794 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:11.818 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:11.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:11.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:11.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:11 smithi204 ceph-mon[32357]: 5.1d scrub starts 2024-09-10T18:24:11.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:11 smithi204 ceph-mon[32357]: 5.1d scrub ok 2024-09-10T18:24:11.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:11 smithi204 ceph-mon[32357]: osdmap e106: 8 total, 7 up, 8 in 2024-09-10T18:24:11.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:11 smithi204 ceph-mon[32357]: 3.1c scrub starts 2024-09-10T18:24:11.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:11 smithi204 ceph-mon[32357]: 3.1c scrub ok 2024-09-10T18:24:11.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:11 smithi204 ceph-mon[32357]: 5.1c scrub starts 2024-09-10T18:24:11.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:11 smithi204 ceph-mon[32357]: 5.1c scrub ok 2024-09-10T18:24:11.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:11 smithi204 ceph-mon[32357]: pgmap v95: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 990 B/s rd, 91 MiB/s wr, 1.46k op/s; 15546/150861 objects degraded (10.305%) 2024-09-10T18:24:11.887 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 28 16 53431 53415 119.19 96.4375 0.00354868 0.00838456 2024-09-10T18:24:11.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:11.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:11 smithi028 ceph-mon[39220]: 5.1d scrub starts 2024-09-10T18:24:11.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:11 smithi028 ceph-mon[39220]: 5.1d scrub ok 2024-09-10T18:24:11.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:11 smithi028 ceph-mon[39220]: osdmap e106: 8 total, 7 up, 8 in 2024-09-10T18:24:11.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:11 smithi028 ceph-mon[39220]: 3.1c scrub starts 2024-09-10T18:24:11.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:11 smithi028 ceph-mon[39220]: 3.1c scrub ok 2024-09-10T18:24:11.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:11 smithi028 ceph-mon[39220]: 5.1c scrub starts 2024-09-10T18:24:11.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:11 smithi028 ceph-mon[39220]: 5.1c scrub ok 2024-09-10T18:24:11.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:11 smithi028 ceph-mon[39220]: pgmap v95: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 990 B/s rd, 91 MiB/s wr, 1.46k op/s; 15546/150861 objects degraded (10.305%) 2024-09-10T18:24:11.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:11 smithi028 ceph-mon[29291]: 5.1d scrub starts 2024-09-10T18:24:11.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:11 smithi028 ceph-mon[29291]: 5.1d scrub ok 2024-09-10T18:24:11.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:11 smithi028 ceph-mon[29291]: osdmap e106: 8 total, 7 up, 8 in 2024-09-10T18:24:11.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:11 smithi028 ceph-mon[29291]: 3.1c scrub starts 2024-09-10T18:24:11.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:11 smithi028 ceph-mon[29291]: 3.1c scrub ok 2024-09-10T18:24:11.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:11 smithi028 ceph-mon[29291]: 5.1c scrub starts 2024-09-10T18:24:11.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:11 smithi028 ceph-mon[29291]: 5.1c scrub ok 2024-09-10T18:24:11.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:11 smithi028 ceph-mon[29291]: pgmap v95: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 990 B/s rd, 91 MiB/s wr, 1.46k op/s; 15546/150861 objects degraded (10.305%) 2024-09-10T18:24:11.956 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:11.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:12.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:12.083 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:12.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:12.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:12.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:12.210 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:12.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:12.312 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:12.334 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:12.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:12.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:12.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:12.461 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:12.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:12.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:12.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:12.570 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:24:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:24:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:24:12.587 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:12.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:12.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:12.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:12.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:12.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:12.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:12.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:12 smithi204 ceph-mon[32357]: 4.2 scrub starts 2024-09-10T18:24:12.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:12 smithi204 ceph-mon[32357]: 4.2 scrub ok 2024-09-10T18:24:12.876 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:12.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:12.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:12.887 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 29 16 54765 54749 117.955 83.375 0.00867372 0.00847204 2024-09-10T18:24:12.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:24:12.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:12 smithi028 ceph-mon[29291]: 4.2 scrub starts 2024-09-10T18:24:12.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:12 smithi028 ceph-mon[29291]: 4.2 scrub ok 2024-09-10T18:24:12.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:12 smithi028 ceph-mon[39220]: 4.2 scrub starts 2024-09-10T18:24:12.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:12 smithi028 ceph-mon[39220]: 4.2 scrub ok 2024-09-10T18:24:12.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:13.003 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:13.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:13.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:13.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:13.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:13.154 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:13.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:13.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:13.310 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:13.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:13.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:13.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:13.461 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:13.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:13.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:13.587 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:13.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:13.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:13.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:13.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:13.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:13.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:13 smithi028 ceph-mon[39220]: pgmap v96: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 78 MiB/s wr, 1.25k op/s; 15546/150861 objects degraded (10.305%) 2024-09-10T18:24:13.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:13 smithi028 ceph-mon[39220]: 3.15 deep-scrub starts 2024-09-10T18:24:13.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:13 smithi028 ceph-mon[39220]: 3.15 deep-scrub ok 2024-09-10T18:24:13.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:13 smithi028 ceph-mon[29291]: pgmap v96: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 78 MiB/s wr, 1.25k op/s; 15546/150861 objects degraded (10.305%) 2024-09-10T18:24:13.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:13 smithi028 ceph-mon[29291]: 3.15 deep-scrub starts 2024-09-10T18:24:13.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:13 smithi028 ceph-mon[29291]: 3.15 deep-scrub ok 2024-09-10T18:24:13.844 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:13.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:13 smithi204 ceph-mon[32357]: pgmap v96: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 78 MiB/s wr, 1.25k op/s; 15546/150861 objects degraded (10.305%) 2024-09-10T18:24:13.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:13 smithi204 ceph-mon[32357]: 3.15 deep-scrub starts 2024-09-10T18:24:13.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:13 smithi204 ceph-mon[32357]: 3.15 deep-scrub ok 2024-09-10T18:24:13.868 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:13.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:13.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:13.888 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 30 16 56113 56097 116.828 84.25 0.0150287 0.00855385 2024-09-10T18:24:13.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:14.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:14.016 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:14.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:14.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:14.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:14.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:14.283 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:14.305 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:14.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:14.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:14.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:24:13 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:24:14.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:14.449 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:14.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:14.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:14.551 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:14.574 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:14.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:14.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:14.616 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:24:14.616 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:24:14.652 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:24:14.652 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:24:14.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:14.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:14 smithi028 ceph-mon[29291]: 3.19 scrub starts 2024-09-10T18:24:14.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:14 smithi028 ceph-mon[29291]: 3.19 scrub ok 2024-09-10T18:24:14.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:14 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:14.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:14 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1541241088' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:24:14.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:14 smithi028 ceph-mon[39220]: 3.19 scrub starts 2024-09-10T18:24:14.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:14 smithi028 ceph-mon[39220]: 3.19 scrub ok 2024-09-10T18:24:14.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:14 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:14.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:14 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1541241088' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:24:14.692 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:24:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:24:14.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:14.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:14.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:14.725 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:24:14.725 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:24:14.755 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:24:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:24:14.775 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:24:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:24:14.798 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:24:14.798 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:24:14.798 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-10T18:24:14.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:14.829 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:24:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:24:14.835 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:24:14.835 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:24:14.849 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:14.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:14.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:14.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:14 smithi204 ceph-mon[32357]: 3.19 scrub starts 2024-09-10T18:24:14.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:14 smithi204 ceph-mon[32357]: 3.19 scrub ok 2024-09-10T18:24:14.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:14 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:14.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:14 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1541241088' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:24:14.888 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 31 16 57347 57331 115.547 77.125 0.01274 0.00864821 2024-09-10T18:24:14.951 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:14.975 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:14.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:14.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:15.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:15.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:15.114 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:15.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:15.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:15.307 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:15.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:15.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:15.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:15.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:15.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:15.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:15.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:15.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:15.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:15.740 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:15.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:15.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:15 smithi028 ceph-mon[29291]: pgmap v97: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 85 MiB/s wr, 1.37k op/s; 16865/160038 objects degraded (10.538%) 2024-09-10T18:24:15.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:15 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 15546/150861 objects degraded (10.305%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:15.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:15 smithi028 ceph-mon[29291]: osdmap e107: 8 total, 7 up, 8 in 2024-09-10T18:24:15.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:15 smithi028 ceph-mon[39220]: pgmap v97: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 85 MiB/s wr, 1.37k op/s; 16865/160038 objects degraded (10.538%) 2024-09-10T18:24:15.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:15 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 15546/150861 objects degraded (10.305%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:15.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:15 smithi028 ceph-mon[39220]: osdmap e107: 8 total, 7 up, 8 in 2024-09-10T18:24:15.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:15.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-10T18:24:15.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:15 smithi204 ceph-mon[32357]: pgmap v97: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 85 MiB/s wr, 1.37k op/s; 16865/160038 objects degraded (10.538%) 2024-09-10T18:24:15.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:15 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 15546/150861 objects degraded (10.305%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:15.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:15 smithi204 ceph-mon[32357]: osdmap e107: 8 total, 7 up, 8 in 2024-09-10T18:24:15.868 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:15.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:15.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:15.888 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 32 16 58815 58799 114.803 91.75 0.00403645 0.00870557 2024-09-10T18:24:15.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:16.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:16.020 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:16.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:16.121 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:16.144 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:16.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:16.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:16.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:16.270 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:16.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:16.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:16.372 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:16.397 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:16.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:16.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:16.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:16.541 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:16.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:16.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:16.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:16.683 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:16.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:16.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:16.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:16 smithi028 ceph-mon[29291]: 5.15 scrub starts 2024-09-10T18:24:16.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:16 smithi028 ceph-mon[29291]: 5.15 scrub ok 2024-09-10T18:24:16.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:16 smithi028 ceph-mon[39220]: 5.15 scrub starts 2024-09-10T18:24:16.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:16 smithi028 ceph-mon[39220]: 5.15 scrub ok 2024-09-10T18:24:16.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:16.835 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:16.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:16.889 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 33 16 60185 60169 113.918 85.625 0.00946126 0.00877323 2024-09-10T18:24:16.936 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:16.962 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:16.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:16.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:17.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:17.088 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:17.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:17.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:17.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:16 smithi204 ceph-mon[32357]: 5.15 scrub starts 2024-09-10T18:24:17.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:16 smithi204 ceph-mon[32357]: 5.15 scrub ok 2024-09-10T18:24:17.190 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:17.214 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:17.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:17.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:17.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:17.355 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:17.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:17.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:17.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:17.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:17.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:17.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:17.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:17.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:17.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:17.728 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T18:24:17.728 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T18:24:17.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:17.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:17.773 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:17.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:17.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:17.889 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 34 16 61649 61633 113.258 91.5 0.0106402 0.00882421 2024-09-10T18:24:17.910 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:17.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:17.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:18.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:17 smithi028 ceph-mon[29291]: pgmap v99: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 119 MiB/s wr, 1.90k op/s; 17833/172161 objects degraded (10.358%) 2024-09-10T18:24:18.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:17 smithi028 ceph-mon[39220]: pgmap v99: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 119 MiB/s wr, 1.90k op/s; 17833/172161 objects degraded (10.358%) 2024-09-10T18:24:18.013 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:18.034 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:18.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:18.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:18.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:17 smithi204 ceph-mon[32357]: pgmap v99: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 119 MiB/s wr, 1.90k op/s; 17833/172161 objects degraded (10.358%) 2024-09-10T18:24:18.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:18.177 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:18.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:18.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:18.279 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:18.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:18.305 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:18.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:18.406 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:18.431 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:18.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:18.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:18.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:18.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:18.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:18.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:18.723 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:18.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:18.824 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:18.848 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:18.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:18.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:18.889 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 35 16 62759 62743 112.005 69.375 0.00780225 0.00892267 2024-09-10T18:24:18.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:18.974 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:18.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:18.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:19.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:18 smithi028 ceph-mon[29291]: 3.1 deep-scrub starts 2024-09-10T18:24:19.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:18 smithi028 ceph-mon[29291]: 3.1 deep-scrub ok 2024-09-10T18:24:19.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:18 smithi028 ceph-mon[39220]: 3.1 deep-scrub starts 2024-09-10T18:24:19.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:18 smithi028 ceph-mon[39220]: 3.1 deep-scrub ok 2024-09-10T18:24:19.076 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:19.101 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:19.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:19.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:19.103 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:18 smithi204 ceph-mon[32357]: 3.1 deep-scrub starts 2024-09-10T18:24:19.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:18 smithi204 ceph-mon[32357]: 3.1 deep-scrub ok 2024-09-10T18:24:19.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:19.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:19.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:19.348 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:19.371 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:19.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:19.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:19.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:19.496 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:19.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:19.597 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:19.620 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:19.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:19.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:19.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:19.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:24:19.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:19.764 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:19.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:19.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:19.891 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 36 16 64033 64017 111.099 79.625 0.00799849 0.00899493 2024-09-10T18:24:19.913 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:19.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:19.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:19 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:24:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:19 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:24:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:19 smithi028 ceph-mon[29291]: 4.1c scrub starts 2024-09-10T18:24:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:19 smithi028 ceph-mon[29291]: 4.1c scrub ok 2024-09-10T18:24:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:19 smithi028 ceph-mon[29291]: osdmap e108: 8 total, 7 up, 8 in 2024-09-10T18:24:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:19 smithi028 ceph-mon[29291]: pgmap v101: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 887 op/s; 17833/172161 objects degraded (10.358%) 2024-09-10T18:24:20.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:19 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:24:20.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:19 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:24:20.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:19 smithi028 ceph-mon[39220]: 4.1c scrub starts 2024-09-10T18:24:20.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:19 smithi028 ceph-mon[39220]: 4.1c scrub ok 2024-09-10T18:24:20.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:19 smithi028 ceph-mon[39220]: osdmap e108: 8 total, 7 up, 8 in 2024-09-10T18:24:20.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:19 smithi028 ceph-mon[39220]: pgmap v101: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 887 op/s; 17833/172161 objects degraded (10.358%) 2024-09-10T18:24:20.015 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:20.037 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:20.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:20.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:19 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:24:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:19 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:24:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:19 smithi204 ceph-mon[32357]: 4.1c scrub starts 2024-09-10T18:24:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:19 smithi204 ceph-mon[32357]: 4.1c scrub ok 2024-09-10T18:24:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:19 smithi204 ceph-mon[32357]: osdmap e108: 8 total, 7 up, 8 in 2024-09-10T18:24:20.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:19 smithi204 ceph-mon[32357]: pgmap v101: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 887 op/s; 17833/172161 objects degraded (10.358%) 2024-09-10T18:24:20.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:20.163 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:20.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:20.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:20.265 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:20.290 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:20.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:20.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:20.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:20.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:20.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:20.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:20.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:20.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:20.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:20.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:20.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:20.719 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:20.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:20.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:20.821 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:20.845 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:20.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:20.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:20.891 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 37 16 65366 65350 110.348 83.3125 0.0112101 0.00905677 2024-09-10T18:24:20.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:20.970 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:20.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:21.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:20 smithi028 ceph-mon[29291]: 2.1 scrub starts 2024-09-10T18:24:21.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:20 smithi028 ceph-mon[29291]: 2.1 scrub ok 2024-09-10T18:24:21.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:20 smithi028 ceph-mon[29291]: 3.1a scrub starts 2024-09-10T18:24:21.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:20 smithi028 ceph-mon[29291]: 3.1a scrub ok 2024-09-10T18:24:21.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:20 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 17833/172161 objects degraded (10.358%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:21.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:20 smithi028 ceph-mon[29291]: osdmap e109: 8 total, 7 up, 8 in 2024-09-10T18:24:21.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:20 smithi028 ceph-mon[39220]: 2.1 scrub starts 2024-09-10T18:24:21.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:20 smithi028 ceph-mon[39220]: 2.1 scrub ok 2024-09-10T18:24:21.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:20 smithi028 ceph-mon[39220]: 3.1a scrub starts 2024-09-10T18:24:21.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:20 smithi028 ceph-mon[39220]: 3.1a scrub ok 2024-09-10T18:24:21.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:20 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 17833/172161 objects degraded (10.358%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:21.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:20 smithi028 ceph-mon[39220]: osdmap e109: 8 total, 7 up, 8 in 2024-09-10T18:24:21.005 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:20 smithi204 ceph-mon[32357]: 2.1 scrub starts 2024-09-10T18:24:21.005 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:20 smithi204 ceph-mon[32357]: 2.1 scrub ok 2024-09-10T18:24:21.005 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:20 smithi204 ceph-mon[32357]: 3.1a scrub starts 2024-09-10T18:24:21.005 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:20 smithi204 ceph-mon[32357]: 3.1a scrub ok 2024-09-10T18:24:21.005 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:20 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 17833/172161 objects degraded (10.358%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:21.005 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:20 smithi204 ceph-mon[32357]: osdmap e109: 8 total, 7 up, 8 in 2024-09-10T18:24:21.072 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:21.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:21.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:21.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:21.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:21.223 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:21.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:21.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:21.325 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:21.349 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:21.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:21.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:21.451 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:21.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:21.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:21.576 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:24:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:24:21] "GET /metrics HTTP/1.1" 200 223493 "" "Prometheus/2.43.0" 2024-09-10T18:24:21.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:21.634 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:21.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:21.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:21.680 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:24:21.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:21.786 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:21.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:21.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:21.888 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:21.891 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 38 16 66631 66615 109.524 79.0625 0.00991505 0.00912556 2024-09-10T18:24:21.914 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:21.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:21.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:21.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:21 smithi028 ceph-mon[39220]: 4.7 scrub starts 2024-09-10T18:24:21.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:21 smithi028 ceph-mon[39220]: 4.7 scrub ok 2024-09-10T18:24:21.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:21 smithi028 ceph-mon[39220]: pgmap v103: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 111 MiB/s wr, 1.78k op/s; 19873/192051 objects degraded (10.348%) 2024-09-10T18:24:21.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:21 smithi028 ceph-mon[29291]: 4.7 scrub starts 2024-09-10T18:24:21.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:21 smithi028 ceph-mon[29291]: 4.7 scrub ok 2024-09-10T18:24:21.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:21 smithi028 ceph-mon[29291]: pgmap v103: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 111 MiB/s wr, 1.78k op/s; 19873/192051 objects degraded (10.348%) 2024-09-10T18:24:22.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:22.041 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:22.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:22.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:22.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:21 smithi204 ceph-mon[32357]: 4.7 scrub starts 2024-09-10T18:24:22.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:21 smithi204 ceph-mon[32357]: 4.7 scrub ok 2024-09-10T18:24:22.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:21 smithi204 ceph-mon[32357]: pgmap v103: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 111 MiB/s wr, 1.78k op/s; 19873/192051 objects degraded (10.348%) 2024-09-10T18:24:22.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:22.169 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:22.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:22.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:22.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:22.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:22.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:22.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:22.457 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:22.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:22.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:22.559 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:22.581 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:22.582 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:24:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:24:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:24:22.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:22.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:22.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:22.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:22.733 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:22.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:22.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:22 smithi028 ceph-mon[39220]: 5.6 scrub starts 2024-09-10T18:24:22.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:22 smithi028 ceph-mon[39220]: 5.6 scrub ok 2024-09-10T18:24:22.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:22 smithi028 ceph-mon[39220]: 4.1a scrub starts 2024-09-10T18:24:22.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:22 smithi028 ceph-mon[39220]: 4.1a scrub ok 2024-09-10T18:24:22.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:22 smithi028 ceph-mon[39220]: 5.5 scrub starts 2024-09-10T18:24:22.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:22 smithi028 ceph-mon[39220]: 5.5 scrub ok 2024-09-10T18:24:22.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:22 smithi028 ceph-mon[29291]: 5.6 scrub starts 2024-09-10T18:24:22.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:22 smithi028 ceph-mon[29291]: 5.6 scrub ok 2024-09-10T18:24:22.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:22 smithi028 ceph-mon[29291]: 4.1a scrub starts 2024-09-10T18:24:22.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:22 smithi028 ceph-mon[29291]: 4.1a scrub ok 2024-09-10T18:24:22.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:22 smithi028 ceph-mon[29291]: 5.5 scrub starts 2024-09-10T18:24:22.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:22 smithi028 ceph-mon[29291]: 5.5 scrub ok 2024-09-10T18:24:22.835 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:22.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:22 smithi204 ceph-mon[32357]: 5.6 scrub starts 2024-09-10T18:24:22.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:22 smithi204 ceph-mon[32357]: 5.6 scrub ok 2024-09-10T18:24:22.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:22 smithi204 ceph-mon[32357]: 4.1a scrub starts 2024-09-10T18:24:22.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:22 smithi204 ceph-mon[32357]: 4.1a scrub ok 2024-09-10T18:24:22.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:22 smithi204 ceph-mon[32357]: 5.5 scrub starts 2024-09-10T18:24:22.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:22 smithi204 ceph-mon[32357]: 5.5 scrub ok 2024-09-10T18:24:22.859 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:22.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:22.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:22.892 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 39 16 67841 67825 108.655 75.625 0.0157881 0.00919783 2024-09-10T18:24:22.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:23.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:23.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:23.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:23.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:23.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:23.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:23.291 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:23.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:23.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:23.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:23.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:23.418 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:23.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:23.519 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:23.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:23.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:23.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:23.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:23.687 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:23.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:23.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:23.741 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:24:23.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:23.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:23.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:23.891 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:24:23.890711+0000 min lat: 0.00103822 max lat: 0.253055 avg lat: 0.0092728 2024-09-10T18:24:23.891 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:24:23.892 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 40 16 69022 69006 107.784 73.8125 0.0190485 0.0092728 2024-09-10T18:24:23.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:23.978 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:23.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:23.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:24.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:23 smithi028 ceph-mon[39220]: 5.17 scrub starts 2024-09-10T18:24:24.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:23 smithi028 ceph-mon[39220]: 5.17 scrub ok 2024-09-10T18:24:24.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:23 smithi028 ceph-mon[39220]: pgmap v104: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 714 B/s rd, 58 MiB/s wr, 925 op/s; 19873/192051 objects degraded (10.348%) 2024-09-10T18:24:24.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:23 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3759820141' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:24:24.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:23 smithi028 ceph-mon[39220]: 4.1e scrub starts 2024-09-10T18:24:24.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:23 smithi028 ceph-mon[39220]: 4.1e scrub ok 2024-09-10T18:24:24.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:23 smithi028 ceph-mon[39220]: 5.1b scrub starts 2024-09-10T18:24:24.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:23 smithi028 ceph-mon[39220]: 5.1b scrub ok 2024-09-10T18:24:24.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:23 smithi028 ceph-mon[29291]: 5.17 scrub starts 2024-09-10T18:24:24.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:23 smithi028 ceph-mon[29291]: 5.17 scrub ok 2024-09-10T18:24:24.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:23 smithi028 ceph-mon[29291]: pgmap v104: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 714 B/s rd, 58 MiB/s wr, 925 op/s; 19873/192051 objects degraded (10.348%) 2024-09-10T18:24:24.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:23 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3759820141' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:24:24.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:23 smithi028 ceph-mon[29291]: 4.1e scrub starts 2024-09-10T18:24:24.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:23 smithi028 ceph-mon[29291]: 4.1e scrub ok 2024-09-10T18:24:24.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:23 smithi028 ceph-mon[29291]: 5.1b scrub starts 2024-09-10T18:24:24.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:23 smithi028 ceph-mon[29291]: 5.1b scrub ok 2024-09-10T18:24:24.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:24.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:24:23 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:24:24.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:23 smithi204 ceph-mon[32357]: 5.17 scrub starts 2024-09-10T18:24:24.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:23 smithi204 ceph-mon[32357]: 5.17 scrub ok 2024-09-10T18:24:24.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:23 smithi204 ceph-mon[32357]: pgmap v104: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 714 B/s rd, 58 MiB/s wr, 925 op/s; 19873/192051 objects degraded (10.348%) 2024-09-10T18:24:24.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:23 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3759820141' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:24:24.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:23 smithi204 ceph-mon[32357]: 4.1e scrub starts 2024-09-10T18:24:24.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:23 smithi204 ceph-mon[32357]: 4.1e scrub ok 2024-09-10T18:24:24.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:23 smithi204 ceph-mon[32357]: 5.1b scrub starts 2024-09-10T18:24:24.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:23 smithi204 ceph-mon[32357]: 5.1b scrub ok 2024-09-10T18:24:24.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:24.106 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:24.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:24.207 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:24.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:24.232 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:24.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:24.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:24.376 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:24.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:24.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:24.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:24.522 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:24.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:24.624 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:24.649 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:24.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:24.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:24.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:24.782 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:24.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:24.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:24.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:24.891 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 41 16 70267 70251 107.053 77.8125 0.0192349 0.00933615 2024-09-10T18:24:24.925 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:24.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:24.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:25.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:24 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:24:25.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:24 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3759820141' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:24:25.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:24 smithi028 ceph-mon[39220]: osdmap e110: 8 total, 7 up, 8 in 2024-09-10T18:24:25.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:24:25.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:24 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:25.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:24 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:24:25.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:24 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:24:25.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:24 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:24:25.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:24 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3759820141' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:24:25.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:24 smithi028 ceph-mon[29291]: osdmap e110: 8 total, 7 up, 8 in 2024-09-10T18:24:25.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:24:25.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:24 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:25.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:24 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:24:25.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:24 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:24:25.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:25.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:25.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:24 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:24:25.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:24 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3759820141' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:24:25.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:24 smithi204 ceph-mon[32357]: osdmap e110: 8 total, 7 up, 8 in 2024-09-10T18:24:25.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:24:25.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:24 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:25.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:24 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:24:25.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:24 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:24:25.170 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:25.192 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:25.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:25.294 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:25.318 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:25.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:25.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:25.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:25.458 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:25.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:25.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:25.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:25.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:25.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:25.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:25.743 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:25.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:25.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:25.845 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:25.871 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:25.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:25.891 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 42 16 71522 71506 106.372 78.4375 0.0243637 0.00939675 2024-09-10T18:24:25.973 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:25.997 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:25.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:25.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:26.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:25 smithi028 ceph-mon[29291]: pgmap v106: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 21074/200352 objects degraded (10.518%) 2024-09-10T18:24:26.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:25 smithi028 ceph-mon[29291]: osdmap e111: 8 total, 7 up, 8 in 2024-09-10T18:24:26.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:25 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 19873/192051 objects degraded (10.348%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:26.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:25 smithi028 ceph-mon[39220]: pgmap v106: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 21074/200352 objects degraded (10.518%) 2024-09-10T18:24:26.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:25 smithi028 ceph-mon[39220]: osdmap e111: 8 total, 7 up, 8 in 2024-09-10T18:24:26.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:25 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 19873/192051 objects degraded (10.348%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:26.099 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:26.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:25 smithi204 ceph-mon[32357]: pgmap v106: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 21074/200352 objects degraded (10.518%) 2024-09-10T18:24:26.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:25 smithi204 ceph-mon[32357]: osdmap e111: 8 total, 7 up, 8 in 2024-09-10T18:24:26.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:25 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 19873/192051 objects degraded (10.348%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:26.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:26.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:26.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:26.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:26.276 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:26.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:26.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:26.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:26.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:26.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:26.518 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:26.542 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:26.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:26.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:26.599 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:24:26.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:26.696 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:26.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:26.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:26.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:26.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:26.837 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:26.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:26.892 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 43 16 72760 72744 105.698 77.375 0.0125816 0.00945619 2024-09-10T18:24:26.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:26.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:26.977 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:26.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:27.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:27.118 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:27.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:27.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:27.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:27.265 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:27.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:27.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:27.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:27.392 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:27.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:27.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:27.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:27.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:27.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:27.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:27.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:27.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-10T18:24:27.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:27.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:27.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:27.779 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:27.802 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:27.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:27.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:27.892 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 44 16 73959 73943 104.998 74.9375 0.0103455 0.00951846 2024-09-10T18:24:27.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:27.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:27 smithi028 ceph-mon[39220]: pgmap v108: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 829 B/s rd, 63 MiB/s wr, 1.01k op/s; 21820/210855 objects degraded (10.348%) 2024-09-10T18:24:27.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:27 smithi028 ceph-mon[29291]: pgmap v108: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 829 B/s rd, 63 MiB/s wr, 1.01k op/s; 21820/210855 objects degraded (10.348%) 2024-09-10T18:24:27.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:27.958 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:27.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:28.059 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:28.082 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:28.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:28.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:28.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:27 smithi204 ceph-mon[32357]: pgmap v108: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 829 B/s rd, 63 MiB/s wr, 1.01k op/s; 21820/210855 objects degraded (10.348%) 2024-09-10T18:24:28.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:28.223 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:28.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:28.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:28.326 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:28.352 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:28.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:28.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:28.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:28.510 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:28.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:28.611 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:28.635 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:28.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:28.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:28.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:24:28.737 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:28.759 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:28.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:28.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:28.861 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:28.886 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:28.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:28.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:28.892 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 45 16 75110 75094 104.264 71.9375 0.0207601 0.00958615 2024-09-10T18:24:28.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:28 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:24:28.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:28 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:24:28.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:28 smithi028 ceph-mon[29291]: osdmap e112: 8 total, 7 up, 8 in 2024-09-10T18:24:28.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:28 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:24:28.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:28 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:24:28.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:28 smithi028 ceph-mon[39220]: osdmap e112: 8 total, 7 up, 8 in 2024-09-10T18:24:28.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:29.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:29.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:29.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:28 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:24:29.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:28 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:24:29.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:28 smithi204 ceph-mon[32357]: osdmap e112: 8 total, 7 up, 8 in 2024-09-10T18:24:29.133 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:29.157 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:29.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:29.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:29.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:29.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:29.400 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:29.425 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:29.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:29.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:29.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:29.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:29.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:29.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:29.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:29.722 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:29.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:29.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:29.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:29.867 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:29.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:29.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:29.892 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 46 16 76305 76289 103.621 74.6875 0.00943654 0.00964571 2024-09-10T18:24:29.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:30.048 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:24:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:30.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:30.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:30.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:29 smithi204 ceph-mon[32357]: pgmap v110: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 851 B/s rd, 65 MiB/s wr, 1.04k op/s; 21820/210855 objects degraded (10.348%) 2024-09-10T18:24:30.150 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:30.176 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:30.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:30.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:30.207 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:29 smithi028 ceph-mon[29291]: pgmap v110: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 851 B/s rd, 65 MiB/s wr, 1.04k op/s; 21820/210855 objects degraded (10.348%) 2024-09-10T18:24:30.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:29 smithi028 ceph-mon[39220]: pgmap v110: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 851 B/s rd, 65 MiB/s wr, 1.04k op/s; 21820/210855 objects degraded (10.348%) 2024-09-10T18:24:30.278 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:30.302 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:30.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:30.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:30.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:30.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:30.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:30.551 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:30.577 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:30.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:30.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:30.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:30.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:30.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:30.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:30.872 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:30.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:30.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:30.892 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 47 16 77415 77399 102.892 69.375 0.0410828 0.00970928 2024-09-10T18:24:30.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:30.998 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:30.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:30.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:31.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:30 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 21820/210855 objects degraded (10.348%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:31.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:30 smithi028 ceph-mon[39220]: osdmap e113: 8 total, 7 up, 8 in 2024-09-10T18:24:31.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:30 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 21820/210855 objects degraded (10.348%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:31.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:30 smithi028 ceph-mon[29291]: osdmap e113: 8 total, 7 up, 8 in 2024-09-10T18:24:31.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:31.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:30 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 21820/210855 objects degraded (10.348%), 24 pgs degraded (PG_DEGRADED) 2024-09-10T18:24:31.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:30 smithi204 ceph-mon[32357]: osdmap e113: 8 total, 7 up, 8 in 2024-09-10T18:24:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:31.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:31.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:31.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:31.269 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:31.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:31.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:31.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:31.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:31.395 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:31.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:31.418 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:24:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:24:31] "GET /metrics HTTP/1.1" 200 223568 "" "Prometheus/2.43.0" 2024-09-10T18:24:31.496 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:31.520 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:31.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:31.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:31.622 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:31.646 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:31.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:31.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:31.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:31.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:31.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:31.892 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 48 16 78531 78515 102.202 69.75 0.0181655 0.00977947 2024-09-10T18:24:31.897 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:31.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:31.921 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:31.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:32.004 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:24:32.004 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:24:32.022 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:32.046 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:32.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:32.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:32.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:31 smithi204 ceph-mon[32357]: pgmap v112: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 851 B/s rd, 99 MiB/s wr, 1.58k op/s; 23692/228909 objects degraded (10.350%) 2024-09-10T18:24:32.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:32.185 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:32.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:32.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:31 smithi028 ceph-mon[39220]: pgmap v112: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 851 B/s rd, 99 MiB/s wr, 1.58k op/s; 23692/228909 objects degraded (10.350%) 2024-09-10T18:24:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:31 smithi028 ceph-mon[29291]: pgmap v112: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 851 B/s rd, 99 MiB/s wr, 1.58k op/s; 23692/228909 objects degraded (10.350%) 2024-09-10T18:24:32.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:32.313 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:32.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:32.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:32.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:32.457 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:32.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:32.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:32.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:32.597 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:32.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:32.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:32.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:24:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:24:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:24:32.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:32.737 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:32.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:32.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:32.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-10T18:24:32.839 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:32.862 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:32.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:32.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:32.893 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 49 16 79706 79690 101.613 73.4375 0.00905765 0.00983562 2024-09-10T18:24:32.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:32.987 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:32.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:32.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:33.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:33.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:33.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:33.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:33.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:33.289 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:33.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:33.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:33.406 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:24:33.496 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:33.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:33.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:33.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:33.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:33.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:33.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:33.796 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:33.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:33.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:33.893 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 50 16 80649 80633 100.76 58.9375 0.0202704 0.00991872 2024-09-10T18:24:33.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:33.945 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:33.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:33.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:34.047 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:34.056 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:33 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:24:34.056 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:33 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:24:34.056 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:33 smithi028 ceph-mon[29291]: pgmap v113: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.19k op/s; 23692/228909 objects degraded (10.350%) 2024-09-10T18:24:34.056 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:33 smithi028 ceph-mon[29291]: osdmap e114: 8 total, 7 up, 8 in 2024-09-10T18:24:34.056 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:33 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:34.056 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:33 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:34.056 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:33 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3318737106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:24:34.057 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:33 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:24:34.057 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:33 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:24:34.057 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:33 smithi028 ceph-mon[39220]: pgmap v113: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.19k op/s; 23692/228909 objects degraded (10.350%) 2024-09-10T18:24:34.057 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:33 smithi028 ceph-mon[39220]: osdmap e114: 8 total, 7 up, 8 in 2024-09-10T18:24:34.057 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:33 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:34.057 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:33 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:34.057 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:33 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3318737106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:24:34.069 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:34.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:34.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:34.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:24:33 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:24:34.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:33 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:24:34.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:33 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:24:34.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:33 smithi204 ceph-mon[32357]: pgmap v113: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.19k op/s; 23692/228909 objects degraded (10.350%) 2024-09-10T18:24:34.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:33 smithi204 ceph-mon[32357]: osdmap e114: 8 total, 7 up, 8 in 2024-09-10T18:24:34.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:33 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:34.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:33 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:34.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:33 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3318737106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:24:34.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:34.196 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:34.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:34.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:34.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:34.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:34.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:34.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:34.478 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:34.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:34.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:34.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:34.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:34.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:34.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:34.750 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:34.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:34.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:34.853 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:34.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:34.880 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:34.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:34.893 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 51 16 81535 81519 99.8702 55.375 0.0105728 0.0100076 2024-09-10T18:24:34.963 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:24:34.964 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:24:34.964 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:24:34.964 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:24:34.964 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:24:34.964 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:24:34.964 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:24:34.964 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:24:34.964 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:24:34.964 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:24:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:24:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:24:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:24:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-10T18:24:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:24:34.981 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:35.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:35.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:35.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:35.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:34 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:35.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:35.151 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:35.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:35.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:35.244 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:34 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:35.245 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:34 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:35.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:35.308 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:35.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:35.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:35.410 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:35.432 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:35.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:35.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:35.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:35.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:35.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:35.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:35.783 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:35.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:35.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:35.893 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 52 16 82456 82440 99.0566 57.5625 0.0360402 0.0100856 2024-09-10T18:24:35.913 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:35.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:36.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:35 smithi028 ceph-mon[39220]: pgmap v115: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 24684/235920 objects degraded (10.463%) 2024-09-10T18:24:36.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:35 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:24:36.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:35 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:24:36.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:35 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:36.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:35 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 24684/235920 objects degraded (10.463%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:24:36.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:35 smithi028 ceph-mon[39220]: osdmap e115: 8 total, 7 up, 8 in 2024-09-10T18:24:36.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:35 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:24:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:35 smithi028 ceph-mon[29291]: pgmap v115: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 24684/235920 objects degraded (10.463%) 2024-09-10T18:24:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:35 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:24:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:35 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:24:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:35 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:35 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 24684/235920 objects degraded (10.463%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:24:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:35 smithi028 ceph-mon[29291]: osdmap e115: 8 total, 7 up, 8 in 2024-09-10T18:24:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:35 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:24:36.015 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:36.042 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:36.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:36.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:35 smithi204 ceph-mon[32357]: pgmap v115: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 24684/235920 objects degraded (10.463%) 2024-09-10T18:24:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:35 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:24:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:35 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:24:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:35 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:35 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 24684/235920 objects degraded (10.463%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:24:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:35 smithi204 ceph-mon[32357]: osdmap e115: 8 total, 7 up, 8 in 2024-09-10T18:24:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:35 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:24:36.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:36.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:36.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:36.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:36.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:36.314 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:36.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:36.414 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:36.440 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:36.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:36.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:36.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:36.570 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:36.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:36.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:36.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:36.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:36.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:36.812 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T18:24:36.812 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T18:24:36.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:36.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:36.857 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:36.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:36.893 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 53 16 83536 83520 98.4612 67.5 0.0101217 0.0101511 2024-09-10T18:24:36.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:37.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:37.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:37.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:37.163 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:37.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:37.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:37.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:37.308 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:37.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:37.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:37.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:37.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:37.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:37.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:37.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:37.581 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:37.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:37.682 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:37.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:37.708 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:37.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:37.809 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:37.838 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:37.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:37.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:37.893 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 54 16 84684 84668 97.9666 71.75 0.010712 0.0102016 2024-09-10T18:24:37.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:37.993 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:37.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:37.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:38.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:37 smithi028 ceph-mon[39220]: pgmap v117: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 827 B/s rd, 53 MiB/s wr, 852 op/s; 25310/244707 objects degraded (10.343%) 2024-09-10T18:24:38.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:37 smithi028 ceph-mon[29291]: pgmap v117: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 827 B/s rd, 53 MiB/s wr, 852 op/s; 25310/244707 objects degraded (10.343%) 2024-09-10T18:24:38.005 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:24:38.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:38.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:37 smithi204 ceph-mon[32357]: pgmap v117: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 827 B/s rd, 53 MiB/s wr, 852 op/s; 25310/244707 objects degraded (10.343%) 2024-09-10T18:24:38.141 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:38.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:38.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:38.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:38.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:38.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:38.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:38.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:38.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:38.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:38.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:38.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:38.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:38.675 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:38.698 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:38.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:38.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:38.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:38.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:24:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:38.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:38.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:38.893 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 55 16 85695 85679 97.3342 63.1875 0.0105758 0.010268 2024-09-10T18:24:38.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:38.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:38 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:24:38.971 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:38 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:24:39.024 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:39.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:39.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:39.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:38 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:24:39.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:39.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:39.180 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:39.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:39.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:39.310 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:39.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:39.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:39.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:39.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:39.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:39.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:39.564 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:39.590 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:39.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:39.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:39.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:39.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:39.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:39.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:39.853 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:24:39.859 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:39.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:39.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:39.893 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 56 16 86749 86733 96.7725 65.875 0.00368277 0.0103283 2024-09-10T18:24:39.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:39.985 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:39 smithi028 ceph-mon[39220]: pgmap v118: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 658 op/s; 25310/244707 objects degraded (10.343%) 2024-09-10T18:24:39.985 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:39 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3328004157' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:24:39.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:39 smithi028 ceph-mon[29291]: pgmap v118: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 658 op/s; 25310/244707 objects degraded (10.343%) 2024-09-10T18:24:39.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:39 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3328004157' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:24:39.988 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:39.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:39.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:40.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-10T18:24:40.089 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:40.115 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:40.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:40.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:40.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:40.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:40.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:39 smithi204 ceph-mon[32357]: pgmap v118: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 658 op/s; 25310/244707 objects degraded (10.343%) 2024-09-10T18:24:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:39 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3328004157' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:24:40.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:40.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:40.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:40.523 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:40.547 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:40.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:40.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:40.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:40.682 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:40.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:40.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:40.822 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:40.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:40.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 57 16 87800 87784 96.2271 65.6875 0.00261654 0.0103854 2024-09-10T18:24:40.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:40.963 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:40.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:40.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:41.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:41.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:41.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:41.207 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:41.234 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:41.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:41.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:40 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:24:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:40 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3328004157' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:24:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:40 smithi028 ceph-mon[39220]: osdmap e116: 8 total, 7 up, 8 in 2024-09-10T18:24:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:40 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 25310/244707 objects degraded (10.343%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:24:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:40 smithi028 ceph-mon[39220]: osdmap e117: 8 total, 7 up, 8 in 2024-09-10T18:24:41.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:40 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:41.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:40 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:41.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:40 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:24:41.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:40 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3328004157' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:24:41.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:40 smithi028 ceph-mon[29291]: osdmap e116: 8 total, 7 up, 8 in 2024-09-10T18:24:41.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:40 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 25310/244707 objects degraded (10.343%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:24:41.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:40 smithi028 ceph-mon[29291]: osdmap e117: 8 total, 7 up, 8 in 2024-09-10T18:24:41.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:40 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:41.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:40 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:41.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:40 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:24:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:40 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3328004157' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:24:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:40 smithi204 ceph-mon[32357]: osdmap e116: 8 total, 7 up, 8 in 2024-09-10T18:24:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:40 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 25310/244707 objects degraded (10.343%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:24:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:40 smithi204 ceph-mon[32357]: osdmap e117: 8 total, 7 up, 8 in 2024-09-10T18:24:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:40 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:41.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:40 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:41.371 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:41.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:41.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:41.473 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:41.497 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:41.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:41.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:41.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:41.636 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:41.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:41.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:41.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:41.753 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:24:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:24:41] "GET /metrics HTTP/1.1" 200 223614 "" "Prometheus/2.43.0" 2024-09-10T18:24:41.760 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:41.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:41.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:41.893 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 58 16 88845 88829 95.6941 65.3125 0.0150801 0.0104447 2024-09-10T18:24:41.903 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:41.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:41.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:42.005 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:42.029 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:42.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:42.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:42.120 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:41 smithi204 ceph-mon[32357]: pgmap v121: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.36k op/s; 26969/260349 objects degraded (10.359%) 2024-09-10T18:24:42.122 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:41 smithi028 ceph-mon[29291]: pgmap v121: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.36k op/s; 26969/260349 objects degraded (10.359%) 2024-09-10T18:24:42.122 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:41 smithi028 ceph-mon[39220]: pgmap v121: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.36k op/s; 26969/260349 objects degraded (10.359%) 2024-09-10T18:24:42.131 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:42.155 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:42.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:42.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:42.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:42.299 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:42.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:42.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:42.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:42.444 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:42.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:42.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:42.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:24:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:24:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:24:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:42.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:42.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:42.715 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:42.738 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:42.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:42.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:42.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:42.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:42.893 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 59 16 89956 89940 95.2491 69.4375 0.013203 0.0104928 2024-09-10T18:24:42.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:43.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:43.046 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:43.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:24:43.110 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:24:43.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:43.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:43.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:43.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:43.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:43.354 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:43.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:43.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:43.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:43.483 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:43.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:43.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:43.585 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:43.608 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:43.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:43.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:43.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:43.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:43.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:43.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:43.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:24:43.893152+0000 min lat: 0.00103822 max lat: 0.253055 avg lat: 0.0105559 2024-09-10T18:24:43.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:24:43.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 60 16 90937 90921 94.6835 61.3125 0.014401 0.0105559 2024-09-10T18:24:43.908 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:43.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:44.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:44.059 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:44.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:44.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:44.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:44.185 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:44.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:44.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:44.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:43 smithi028 ceph-mon[29291]: pgmap v122: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 712 B/s rd, 45 MiB/s wr, 726 op/s; 26969/260349 objects degraded (10.359%) 2024-09-10T18:24:44.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:43 smithi028 ceph-mon[39220]: pgmap v122: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 712 B/s rd, 45 MiB/s wr, 726 op/s; 26969/260349 objects degraded (10.359%) 2024-09-10T18:24:44.258 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:24:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:24:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:91316 7.2 e117/93 7:428f8843:::benchmark_data_smithi028_77856_object91315:head v 117'5659, mlcod=117'5657)", 2024-09-10T18:24:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:24:44.248269+0000", 2024-09-10T18:24:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0053159490000000004, 2024-09-10T18:24:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0053552030000000002, 2024-09-10T18:24:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:24:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:24:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:24:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:24:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:44.248269+0000", 2024-09-10T18:24:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:24:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:24:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:24:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:44.248269+0000", 2024-09-10T18:24:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:24:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:24:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:24:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:44.248267+0000", 2024-09-10T18:24:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:24:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:24:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:24:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:44.248293+0000", 2024-09-10T18:24:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6440000000000001e-05 2024-09-10T18:24:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:24:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:24:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:44.248308+0000", 2024-09-10T18:24:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4474999999999999e-05 2024-09-10T18:24:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:24:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:24:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:44.248314+0000", 2024-09-10T18:24:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1940000000000003e-06 2024-09-10T18:24:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:24:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:24:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:44.248378+0000", 2024-09-10T18:24:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4339999999999997e-05 2024-09-10T18:24:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:24:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:24:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:24:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:44.248399+0000", 2024-09-10T18:24:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0356999999999998e-05 2024-09-10T18:24:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:24:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:24:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:24:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:24:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:24:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 1 2024-09-10T18:24:44.262 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:24:44.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:44.324 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:44.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:44.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:43 smithi204 ceph-mon[32357]: pgmap v122: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 712 B/s rd, 45 MiB/s wr, 726 op/s; 26969/260349 objects degraded (10.359%) 2024-09-10T18:24:44.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:24:43 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:24:44.426 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:44.450 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:44.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:44.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:44.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:44.591 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:44.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:44.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:44.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:44.740 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:44.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:44.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:44.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:44.882 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:44.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:44.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:44.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 61 15 92032 92017 94.2542 68.5 0.0285348 0.0106041 2024-09-10T18:24:44.984 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:45.006 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:45.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:45.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:45.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:24:45.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:45.163 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:45.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:45.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:45.206 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:44 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:45.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:44 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:45.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:45.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:45.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:45.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:44 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:45.412 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:45.436 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:45.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:45.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:45.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:45.581 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:45.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:45.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:45.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:45.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:45.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:45.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:45.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 62 16 93126 93110 93.8358 68.3125 0.00700142 0.010652 2024-09-10T18:24:45.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:46.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-10T18:24:46.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:46.026 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:46.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:46.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:46.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:45 smithi028 ceph-mon[39220]: pgmap v123: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 938 op/s; 27937/267213 objects degraded (10.455%) 2024-09-10T18:24:46.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:45 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 27937/267213 objects degraded (10.455%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:24:46.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:45 smithi028 ceph-mon[29291]: pgmap v123: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 938 op/s; 27937/267213 objects degraded (10.455%) 2024-09-10T18:24:46.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:45 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 27937/267213 objects degraded (10.455%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:24:46.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:46.205 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:46.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:46.306 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:46.330 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:46.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:46.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:45 smithi204 ceph-mon[32357]: pgmap v123: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 938 op/s; 27937/267213 objects degraded (10.455%) 2024-09-10T18:24:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:45 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 27937/267213 objects degraded (10.455%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:24:46.433 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:46.453 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:46.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:46.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:46.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:46.597 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:46.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:46.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:46.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:46.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:46.773 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:46.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:46.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:46.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 63 16 94076 94060 93.2888 59.375 0.0106474 0.0107144 2024-09-10T18:24:46.917 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:46.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:46.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:47.022 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:47.046 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:47.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:47.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:47.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:47.172 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:47.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:47.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:47.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:47.319 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:47.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:47.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:47.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:47.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:47.476 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:47.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:47.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:47.629 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:47.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:47.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:47.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:47.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:47.804 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:47.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:47.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 64 16 95118 95102 92.8487 65.125 0.015344 0.0107647 2024-09-10T18:24:47.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:47.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:47.952 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:47.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:48.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:48.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:48.097 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:48.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:48.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:47 smithi028 ceph-mon[39220]: pgmap v124: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 28627/276234 objects degraded (10.363%) 2024-09-10T18:24:48.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:47 smithi028 ceph-mon[29291]: pgmap v124: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 28627/276234 objects degraded (10.363%) 2024-09-10T18:24:48.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:48.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:48.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:47 smithi204 ceph-mon[32357]: pgmap v124: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 28627/276234 objects degraded (10.363%) 2024-09-10T18:24:48.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:48.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:48.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:48.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:24:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:24:48.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:48.619 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:48.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:48.720 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:48.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:48.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:48.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:48.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:48.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 65 16 96120 96104 92.3837 62.625 0.0151644 0.0108192 2024-09-10T18:24:48.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:48.905 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:48.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:49.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:49.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:49.053 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:49.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:49.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:49.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:49.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:49.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:49.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:49.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:49.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:49.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:49.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:49.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:49.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:49.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:49.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:49.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:49.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:49.811 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:49.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:49.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 66 15 97190 97175 91.9981 66.9375 0.0269271 0.0108648 2024-09-10T18:24:49.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:49.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:49.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:49.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:50.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:49 smithi028 ceph-mon[39220]: pgmap v125: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 574 B/s rd, 37 MiB/s wr, 594 op/s; 28627/276234 objects degraded (10.363%) 2024-09-10T18:24:50.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:49 smithi028 ceph-mon[29291]: pgmap v125: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 574 B/s rd, 37 MiB/s wr, 594 op/s; 28627/276234 objects degraded (10.363%) 2024-09-10T18:24:50.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:50.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:50.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:50.203 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:50.228 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:50.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:50.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:50.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:49 smithi204 ceph-mon[32357]: pgmap v125: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 574 B/s rd, 37 MiB/s wr, 594 op/s; 28627/276234 objects degraded (10.363%) 2024-09-10T18:24:50.368 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:50.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:50.470 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:50.495 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:50.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:50.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:50.597 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:50.620 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:50.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:50.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:50.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:50.744 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:50.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:50.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:50.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:50.857 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:24:50.858 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:24:50.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 67 16 98260 98244 91.6222 66.8125 0.0124321 0.0109091 2024-09-10T18:24:50.903 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:24:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:24:50.904 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:50.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:50.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:50.933 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:24:50.933 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:24:50.981 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:24:50.982 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:24:51.006 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:51.013 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:24:51.014 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:24:51.029 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:51.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:51.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:51.042 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:24:51.042 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:24:51.044 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-10T18:24:51.083 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:24:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:24:51.130 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:51.155 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:51.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:51.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:51.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:50 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 28627/276234 objects degraded (10.363%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:24:51.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:50 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1686331518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:24:51.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:50 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 28627/276234 objects degraded (10.363%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:24:51.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:50 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1686331518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:24:51.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:50 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 28627/276234 objects degraded (10.363%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:24:51.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:50 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1686331518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:24:51.257 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:51.281 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:51.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:51.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:51.383 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:51.409 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:51.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:51.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:51.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:24:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:24:51] "GET /metrics HTTP/1.1" 200 223614 "" "Prometheus/2.43.0" 2024-09-10T18:24:51.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:51.559 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:51.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:51.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:51.661 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:51.687 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:51.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:51.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:51.789 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:51.814 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:51.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:51.894 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 68 16 99348 99332 91.2748 68 0.0127399 0.0109514 2024-09-10T18:24:51.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:51.955 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:51.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:52.057 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:52.085 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:52.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:52.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:52.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:51 smithi204 ceph-mon[32357]: pgmap v126: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1004 B/s rd, 64 MiB/s wr, 1.03k op/s; 30248/291723 objects degraded (10.369%) 2024-09-10T18:24:52.187 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:52.209 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:52.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:52.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:51 smithi028 ceph-mon[29291]: pgmap v126: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1004 B/s rd, 64 MiB/s wr, 1.03k op/s; 30248/291723 objects degraded (10.369%) 2024-09-10T18:24:52.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:51 smithi028 ceph-mon[39220]: pgmap v126: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1004 B/s rd, 64 MiB/s wr, 1.03k op/s; 30248/291723 objects degraded (10.369%) 2024-09-10T18:24:52.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:52.346 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:52.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:52.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:52.448 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:52.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:52.470 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:52.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:52.571 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:52.594 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:52.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:52.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:52.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:24:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:24:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:24:52.696 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:52.721 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:52.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:52.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:52.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:52.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:52.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:52.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 69 15 100274 100259 90.7917 57.9375 0.0103341 0.0110093 2024-09-10T18:24:52.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:53.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:53.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:53.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:53.160 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:53.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:53.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:53.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:53.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:53.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:53.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:53.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:53.436 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:53.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:53.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:53.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:53.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:53.562 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:53.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:53.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:53.705 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:53.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:53.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:53.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:53.846 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:53.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:53.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 70 16 101312 101296 90.4205 64.8125 0.0139229 0.0110543 2024-09-10T18:24:53.907 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T18:24:53.908 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T18:24:53.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:53.990 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:53.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:53.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:54.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:54.115 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:54.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:54.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:54.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:53 smithi028 ceph-mon[29291]: pgmap v127: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 54 MiB/s wr, 871 op/s; 30248/291723 objects degraded (10.369%) 2024-09-10T18:24:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:53 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:24:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:53 smithi028 ceph-mon[39220]: pgmap v127: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 54 MiB/s wr, 871 op/s; 30248/291723 objects degraded (10.369%) 2024-09-10T18:24:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:53 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:24:54.255 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:54.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:54.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:54.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-10T18:24:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:53 smithi204 ceph-mon[32357]: pgmap v127: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 54 MiB/s wr, 871 op/s; 30248/291723 objects degraded (10.369%) 2024-09-10T18:24:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:53 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:24:54.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:24:53 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:24:54.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:54.410 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:54.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:54.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:54.512 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:54.535 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:54.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:54.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:54.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:54.694 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:24:54.694 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:24:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:54.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:54.796 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:54.820 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:54.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:54.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:54.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 71 16 102386 102370 90.0924 67.125 0.010521 0.0110947 2024-09-10T18:24:54.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:54.969 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:54.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:54.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:55.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:54 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:55.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:54 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:55.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:55.096 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:55.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:55.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:55.197 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:55.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:55.220 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:55.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:55.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:54 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:24:55.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:55.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:55.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:55.502 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:55.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:55.604 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:55.628 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:55.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:55.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:55.756 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:55.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:55.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:55.858 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:55.882 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:55.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:55.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:55.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 72 16 103486 103470 89.7959 68.75 0.00698668 0.0111314 2024-09-10T18:24:55.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:24:55.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:56.041 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:56.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:56.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-10T18:24:56.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:56.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:55 smithi028 ceph-mon[29291]: pgmap v128: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 1.00k op/s; 30585/296376 objects degraded (10.320%) 2024-09-10T18:24:56.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:55 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 30585/296376 objects degraded (10.320%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:24:56.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:55 smithi028 ceph-mon[39220]: pgmap v128: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 1.00k op/s; 30585/296376 objects degraded (10.320%) 2024-09-10T18:24:56.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:55 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 30585/296376 objects degraded (10.320%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:24:56.212 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:56.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:56.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:56.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:55 smithi204 ceph-mon[32357]: pgmap v128: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 1.00k op/s; 30585/296376 objects degraded (10.320%) 2024-09-10T18:24:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:55 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 30585/296376 objects degraded (10.320%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:24:56.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:56.392 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:56.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:24:56.493 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:56.515 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:56.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:56.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:56.617 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:56.644 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:56.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:56.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:56.746 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:56.768 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:56.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:56.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:56.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 73 16 104355 104339 89.3098 54.3125 0.0447649 0.0111922 2024-09-10T18:24:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:24:56.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:56.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:56.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:57.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:57.072 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:57.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:57.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:57.174 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:57.202 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:57.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:57.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:57.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:56 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1615040907' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:24:57.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:56 smithi028 ceph-mon[39220]: from='client.24799 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:24:57.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:56 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1615040907' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:24:57.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:56 smithi028 ceph-mon[29291]: from='client.24799 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:24:57.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:57.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:57.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:57.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:56 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1615040907' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:24:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:56 smithi204 ceph-mon[32357]: from='client.24799 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:24:57.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:57.483 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:57.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:57.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:57.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:57.622 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:57.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:57.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:57.712 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:24:57.724 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:57.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:57.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:57.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:57.849 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:57.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:57.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:57.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:57.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 74 16 105444 105428 89.0227 68.0625 0.0053802 0.0112276 2024-09-10T18:24:57.975 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:57.996 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:57.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:57.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:58.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:58.143 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:58.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:58.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:58.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:58.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[29291]: pgmap v129: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.11k op/s; 31890/307323 objects degraded (10.377%) 2024-09-10T18:24:58.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:24:58.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[29291]: from='client.24799 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:24:58.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[29291]: osdmap e118: 8 total, 7 up, 8 in 2024-09-10T18:24:58.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[29291]: 3.8 scrub starts 2024-09-10T18:24:58.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[29291]: 3.8 scrub ok 2024-09-10T18:24:58.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:58.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:58.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:24:58.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:24:58.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:58.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[39220]: pgmap v129: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.11k op/s; 31890/307323 objects degraded (10.377%) 2024-09-10T18:24:58.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:24:58.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[39220]: from='client.24799 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:24:58.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[39220]: osdmap e118: 8 total, 7 up, 8 in 2024-09-10T18:24:58.258 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[39220]: 3.8 scrub starts 2024-09-10T18:24:58.258 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[39220]: 3.8 scrub ok 2024-09-10T18:24:58.258 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:58.258 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:58.258 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:24:58.258 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:24:58.258 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:57 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:58.282 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:58.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:58.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:57 smithi204 ceph-mon[32357]: pgmap v129: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.11k op/s; 31890/307323 objects degraded (10.377%) 2024-09-10T18:24:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:57 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:24:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:57 smithi204 ceph-mon[32357]: from='client.24799 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:24:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:57 smithi204 ceph-mon[32357]: osdmap e118: 8 total, 7 up, 8 in 2024-09-10T18:24:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:57 smithi204 ceph-mon[32357]: 3.8 scrub starts 2024-09-10T18:24:58.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:57 smithi204 ceph-mon[32357]: 3.8 scrub ok 2024-09-10T18:24:58.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:57 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:58.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:57 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:58.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:57 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:24:58.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:57 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:24:58.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:57 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:24:58.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:24:58.423 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:58.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:58.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:24:58.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:58.564 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:58.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:58.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:58.666 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:24:58.691 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:58.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:58.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:24:58.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:58.830 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:58.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:58.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 75 16 106425 106409 88.6532 61.3125 0.00366825 0.0112747 2024-09-10T18:24:58.933 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:24:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:58.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:58.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:24:59.057 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:24:59.082 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:59.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:59.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:24:59.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:59.223 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:59.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:59.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:58 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:24:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:58 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:24:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:58 smithi028 ceph-mon[39220]: 6.17 scrub starts 2024-09-10T18:24:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:58 smithi028 ceph-mon[39220]: 6.17 scrub ok 2024-09-10T18:24:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:58 smithi028 ceph-mon[39220]: 4.6 scrub starts 2024-09-10T18:24:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:58 smithi028 ceph-mon[39220]: osdmap e119: 8 total, 7 up, 8 in 2024-09-10T18:24:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:58 smithi028 ceph-mon[39220]: 4.6 scrub ok 2024-09-10T18:24:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:58 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:24:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:58 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:24:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:58 smithi028 ceph-mon[29291]: 6.17 scrub starts 2024-09-10T18:24:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:58 smithi028 ceph-mon[29291]: 6.17 scrub ok 2024-09-10T18:24:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:58 smithi028 ceph-mon[29291]: 4.6 scrub starts 2024-09-10T18:24:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:58 smithi028 ceph-mon[29291]: osdmap e119: 8 total, 7 up, 8 in 2024-09-10T18:24:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:58 smithi028 ceph-mon[29291]: 4.6 scrub ok 2024-09-10T18:24:59.325 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:59.349 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:59.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:59.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:58 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:24:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:58 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:24:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:58 smithi204 ceph-mon[32357]: 6.17 scrub starts 2024-09-10T18:24:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:58 smithi204 ceph-mon[32357]: 6.17 scrub ok 2024-09-10T18:24:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:58 smithi204 ceph-mon[32357]: 4.6 scrub starts 2024-09-10T18:24:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:58 smithi204 ceph-mon[32357]: osdmap e119: 8 total, 7 up, 8 in 2024-09-10T18:24:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:58 smithi204 ceph-mon[32357]: 4.6 scrub ok 2024-09-10T18:24:59.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:24:59.489 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:59.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:59.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:24:59.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:59.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:59.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:24:59.737 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:24:59.760 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:24:59.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:59.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:24:59.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:24:59.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 76 15 107459 107444 88.3378 64.6875 0.0095263 0.0113147 2024-09-10T18:24:59.905 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:24:59.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:59.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:00.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:59 smithi028 ceph-mon[39220]: 5.c scrub starts 2024-09-10T18:25:00.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:59 smithi028 ceph-mon[39220]: 5.c scrub ok 2024-09-10T18:25:00.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:59 smithi028 ceph-mon[39220]: pgmap v132: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 650 op/s; 31890/307323 objects degraded (10.377%) 2024-09-10T18:25:00.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:59 smithi028 ceph-mon[39220]: 6.1c scrub starts 2024-09-10T18:25:00.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:59 smithi028 ceph-mon[39220]: 6.1c scrub ok 2024-09-10T18:25:00.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:59 smithi028 ceph-mon[39220]: 6.10 scrub starts 2024-09-10T18:25:00.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:24:59 smithi028 ceph-mon[39220]: 6.10 scrub ok 2024-09-10T18:25:00.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:59 smithi028 ceph-mon[29291]: 5.c scrub starts 2024-09-10T18:25:00.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:59 smithi028 ceph-mon[29291]: 5.c scrub ok 2024-09-10T18:25:00.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:59 smithi028 ceph-mon[29291]: pgmap v132: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 650 op/s; 31890/307323 objects degraded (10.377%) 2024-09-10T18:25:00.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:59 smithi028 ceph-mon[29291]: 6.1c scrub starts 2024-09-10T18:25:00.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:59 smithi028 ceph-mon[29291]: 6.1c scrub ok 2024-09-10T18:25:00.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:59 smithi028 ceph-mon[29291]: 6.10 scrub starts 2024-09-10T18:25:00.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:24:59 smithi028 ceph-mon[29291]: 6.10 scrub ok 2024-09-10T18:25:00.007 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:00.030 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:00.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:00.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:00.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:00.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:00.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:00.274 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:00.297 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:00.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:59 smithi204 ceph-mon[32357]: 5.c scrub starts 2024-09-10T18:25:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:59 smithi204 ceph-mon[32357]: 5.c scrub ok 2024-09-10T18:25:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:59 smithi204 ceph-mon[32357]: pgmap v132: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 650 op/s; 31890/307323 objects degraded (10.377%) 2024-09-10T18:25:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:59 smithi204 ceph-mon[32357]: 6.1c scrub starts 2024-09-10T18:25:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:59 smithi204 ceph-mon[32357]: 6.1c scrub ok 2024-09-10T18:25:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:59 smithi204 ceph-mon[32357]: 6.10 scrub starts 2024-09-10T18:25:00.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:24:59 smithi204 ceph-mon[32357]: 6.10 scrub ok 2024-09-10T18:25:00.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:00.440 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:00.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:00.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:00.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:00.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:00.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:00.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:00.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:00.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:00.849 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:00.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:00.895 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 77 16 108551 108535 88.0761 68.1875 0.0188654 0.0113488 2024-09-10T18:25:00.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:00.974 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:00.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:00.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:00.989 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:25:01.076 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:01.100 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:01.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:01.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:01.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:01.240 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:01.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:01.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:01.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:00 smithi028 ceph-mon[39220]: 5.1a scrub starts 2024-09-10T18:25:01.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:00 smithi028 ceph-mon[39220]: 5.1a scrub ok 2024-09-10T18:25:01.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:00 smithi028 ceph-mon[39220]: 4.1d scrub starts 2024-09-10T18:25:01.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:00 smithi028 ceph-mon[39220]: 4.1d scrub ok 2024-09-10T18:25:01.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:00 smithi028 ceph-mon[39220]: 4.4 scrub starts 2024-09-10T18:25:01.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:00 smithi028 ceph-mon[39220]: 4.4 scrub ok 2024-09-10T18:25:01.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:00 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 31890/307323 objects degraded (10.377%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:00 smithi028 ceph-mon[29291]: 5.1a scrub starts 2024-09-10T18:25:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:00 smithi028 ceph-mon[29291]: 5.1a scrub ok 2024-09-10T18:25:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:00 smithi028 ceph-mon[29291]: 4.1d scrub starts 2024-09-10T18:25:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:00 smithi028 ceph-mon[29291]: 4.1d scrub ok 2024-09-10T18:25:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:00 smithi028 ceph-mon[29291]: 4.4 scrub starts 2024-09-10T18:25:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:00 smithi028 ceph-mon[29291]: 4.4 scrub ok 2024-09-10T18:25:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:00 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 31890/307323 objects degraded (10.377%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:01.343 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:00 smithi204 ceph-mon[32357]: 5.1a scrub starts 2024-09-10T18:25:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:00 smithi204 ceph-mon[32357]: 5.1a scrub ok 2024-09-10T18:25:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:00 smithi204 ceph-mon[32357]: 4.1d scrub starts 2024-09-10T18:25:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:00 smithi204 ceph-mon[32357]: 4.1d scrub ok 2024-09-10T18:25:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:00 smithi204 ceph-mon[32357]: 4.4 scrub starts 2024-09-10T18:25:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:00 smithi204 ceph-mon[32357]: 4.4 scrub ok 2024-09-10T18:25:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:00 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 31890/307323 objects degraded (10.377%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:01.366 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:01.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:01.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:01.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:01.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:01.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:01.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:01.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:01.645 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:01.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:01.746 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:01.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:25:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:25:01] "GET /metrics HTTP/1.1" 200 223614 "" "Prometheus/2.43.0" 2024-09-10T18:25:01.771 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:01.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:01.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:01.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:01.896 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 78 16 109643 109627 87.8219 68.25 0.0108346 0.0113819 2024-09-10T18:25:01.917 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:01.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:01.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:02.018 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:02.041 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:02.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:02.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:02.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:02.182 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:02.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:02.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:02.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:01 smithi028 ceph-mon[39220]: 5.19 scrub starts 2024-09-10T18:25:02.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:01 smithi028 ceph-mon[39220]: 5.19 scrub ok 2024-09-10T18:25:02.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:01 smithi028 ceph-mon[39220]: pgmap v133: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s; 33461/322614 objects degraded (10.372%) 2024-09-10T18:25:02.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:01 smithi028 ceph-mon[39220]: 6.d scrub starts 2024-09-10T18:25:02.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:01 smithi028 ceph-mon[39220]: 6.d scrub ok 2024-09-10T18:25:02.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:01 smithi028 ceph-mon[29291]: 5.19 scrub starts 2024-09-10T18:25:02.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:01 smithi028 ceph-mon[29291]: 5.19 scrub ok 2024-09-10T18:25:02.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:01 smithi028 ceph-mon[29291]: pgmap v133: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s; 33461/322614 objects degraded (10.372%) 2024-09-10T18:25:02.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:01 smithi028 ceph-mon[29291]: 6.d scrub starts 2024-09-10T18:25:02.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:01 smithi028 ceph-mon[29291]: 6.d scrub ok 2024-09-10T18:25:02.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:02.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:02.311 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:02.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:02.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:25:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:25:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:25:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:01 smithi204 ceph-mon[32357]: 5.19 scrub starts 2024-09-10T18:25:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:01 smithi204 ceph-mon[32357]: 5.19 scrub ok 2024-09-10T18:25:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:01 smithi204 ceph-mon[32357]: pgmap v133: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s; 33461/322614 objects degraded (10.372%) 2024-09-10T18:25:02.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:01 smithi204 ceph-mon[32357]: 6.d scrub starts 2024-09-10T18:25:02.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:01 smithi204 ceph-mon[32357]: 6.d scrub ok 2024-09-10T18:25:02.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:02.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:02.454 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:02.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:02.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:02.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:25:02.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:02.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:02.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:02.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:02.737 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:02.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:02.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:02.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:02.896 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 79 16 110688 110672 87.5369 65.3125 0.00852311 0.0114186 2024-09-10T18:25:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:02.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:02.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:02.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:03.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:03.024 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:03.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:02 smithi204 ceph-mon[32357]: 3.13 scrub starts 2024-09-10T18:25:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:02 smithi204 ceph-mon[32357]: 3.13 scrub ok 2024-09-10T18:25:03.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:03.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:03.163 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:03.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:03.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:02 smithi028 ceph-mon[39220]: 3.13 scrub starts 2024-09-10T18:25:03.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:02 smithi028 ceph-mon[39220]: 3.13 scrub ok 2024-09-10T18:25:03.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:02 smithi028 ceph-mon[29291]: 3.13 scrub starts 2024-09-10T18:25:03.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:02 smithi028 ceph-mon[29291]: 3.13 scrub ok 2024-09-10T18:25:03.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:03.303 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:03.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:03.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:03.404 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:03.428 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:03.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:03.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:03.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:03.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:03.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:03.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:03.709 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:03.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:03.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:03.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:03.835 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:03.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:03.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:03.896 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:25:03.894975+0000 min lat: 0.00103822 max lat: 0.253055 avg lat: 0.0114646 2024-09-10T18:25:03.896 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:25:03.896 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 80 16 111641 111625 87.1872 59.5625 0.0148579 0.0114646 2024-09-10T18:25:03.936 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:03.962 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:03.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:03.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:04.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:04.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:04.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:04.208 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:04.232 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:04.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:04.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:03 smithi028 ceph-mon[39220]: 6.14 scrub starts 2024-09-10T18:25:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:03 smithi028 ceph-mon[39220]: 6.14 scrub ok 2024-09-10T18:25:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:03 smithi028 ceph-mon[39220]: pgmap v134: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s; 33461/322614 objects degraded (10.372%) 2024-09-10T18:25:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:03 smithi028 ceph-mon[39220]: 3.f scrub starts 2024-09-10T18:25:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:03 smithi028 ceph-mon[39220]: 3.f scrub ok 2024-09-10T18:25:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:03 smithi028 ceph-mon[39220]: 6.6 scrub starts 2024-09-10T18:25:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:03 smithi028 ceph-mon[39220]: 6.6 scrub ok 2024-09-10T18:25:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:03 smithi028 ceph-mon[29291]: 6.14 scrub starts 2024-09-10T18:25:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:03 smithi028 ceph-mon[29291]: 6.14 scrub ok 2024-09-10T18:25:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:03 smithi028 ceph-mon[29291]: pgmap v134: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s; 33461/322614 objects degraded (10.372%) 2024-09-10T18:25:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:03 smithi028 ceph-mon[29291]: 3.f scrub starts 2024-09-10T18:25:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:03 smithi028 ceph-mon[29291]: 3.f scrub ok 2024-09-10T18:25:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:03 smithi028 ceph-mon[29291]: 6.6 scrub starts 2024-09-10T18:25:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:03 smithi028 ceph-mon[29291]: 6.6 scrub ok 2024-09-10T18:25:04.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:03 smithi204 ceph-mon[32357]: 6.14 scrub starts 2024-09-10T18:25:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:03 smithi204 ceph-mon[32357]: 6.14 scrub ok 2024-09-10T18:25:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:03 smithi204 ceph-mon[32357]: pgmap v134: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s; 33461/322614 objects degraded (10.372%) 2024-09-10T18:25:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:03 smithi204 ceph-mon[32357]: 3.f scrub starts 2024-09-10T18:25:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:03 smithi204 ceph-mon[32357]: 3.f scrub ok 2024-09-10T18:25:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:03 smithi204 ceph-mon[32357]: 6.6 scrub starts 2024-09-10T18:25:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:03 smithi204 ceph-mon[32357]: 6.6 scrub ok 2024-09-10T18:25:04.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:25:03 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:25:04.375 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:04.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:04.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:04.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:04.538 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:04.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:04.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:04.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:04.683 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:04.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:04.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:04.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:04.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:04.836 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:04.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:04.898 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 81 16 112615 112599 86.8599 60.875 0.0213092 0.0115079 2024-09-10T18:25:04.937 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:04.962 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:04.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:04.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:05.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:05.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:05.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:05.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[39220]: 7.e scrub starts 2024-09-10T18:25:05.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[39220]: 7.5 scrub starts 2024-09-10T18:25:05.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[39220]: 6.1a scrub starts 2024-09-10T18:25:05.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[39220]: 6.1a scrub ok 2024-09-10T18:25:05.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:05.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[39220]: 3.7 scrub starts 2024-09-10T18:25:05.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[39220]: 3.7 scrub ok 2024-09-10T18:25:05.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[39220]: 6.16 scrub starts 2024-09-10T18:25:05.146 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[39220]: 6.16 scrub ok 2024-09-10T18:25:05.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[29291]: 7.e scrub starts 2024-09-10T18:25:05.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[29291]: 7.5 scrub starts 2024-09-10T18:25:05.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[29291]: 6.1a scrub starts 2024-09-10T18:25:05.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[29291]: 6.1a scrub ok 2024-09-10T18:25:05.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:05.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[29291]: 3.7 scrub starts 2024-09-10T18:25:05.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[29291]: 3.7 scrub ok 2024-09-10T18:25:05.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[29291]: 6.16 scrub starts 2024-09-10T18:25:05.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:04 smithi028 ceph-mon[29291]: 6.16 scrub ok 2024-09-10T18:25:05.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:05.226 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:25:05.227 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:05.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:05.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:05.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:04 smithi204 ceph-mon[32357]: 7.e scrub starts 2024-09-10T18:25:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:04 smithi204 ceph-mon[32357]: 7.5 scrub starts 2024-09-10T18:25:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:04 smithi204 ceph-mon[32357]: 6.1a scrub starts 2024-09-10T18:25:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:04 smithi204 ceph-mon[32357]: 6.1a scrub ok 2024-09-10T18:25:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:04 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:04 smithi204 ceph-mon[32357]: 3.7 scrub starts 2024-09-10T18:25:05.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:04 smithi204 ceph-mon[32357]: 3.7 scrub ok 2024-09-10T18:25:05.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:04 smithi204 ceph-mon[32357]: 6.16 scrub starts 2024-09-10T18:25:05.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:04 smithi204 ceph-mon[32357]: 6.16 scrub ok 2024-09-10T18:25:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:05.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:05.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:05.482 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:05.506 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:05.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:05.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:05.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:05.651 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:05.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:05.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:05.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:05.799 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:05.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:05.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:05.898 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 82 16 113486 113470 86.4646 54.4375 0.0251256 0.0115605 2024-09-10T18:25:05.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:05.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:05.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:05.987 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:25:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:25:06.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[39220]: 7.8 scrub starts 2024-09-10T18:25:06.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[39220]: 3.e scrub starts 2024-09-10T18:25:06.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[39220]: 3.e scrub ok 2024-09-10T18:25:06.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[39220]: pgmap v135: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 758 op/s; 33785/325509 objects degraded (10.379%) 2024-09-10T18:25:06.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[39220]: 6.e scrub starts 2024-09-10T18:25:06.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[39220]: 4.3 scrub starts 2024-09-10T18:25:06.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[39220]: 4.3 scrub ok 2024-09-10T18:25:06.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 33785/325509 objects degraded (10.379%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:06.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4072911912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:25:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[29291]: 7.8 scrub starts 2024-09-10T18:25:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[29291]: 3.e scrub starts 2024-09-10T18:25:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[29291]: 3.e scrub ok 2024-09-10T18:25:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[29291]: pgmap v135: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 758 op/s; 33785/325509 objects degraded (10.379%) 2024-09-10T18:25:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[29291]: 6.e scrub starts 2024-09-10T18:25:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[29291]: 4.3 scrub starts 2024-09-10T18:25:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[29291]: 4.3 scrub ok 2024-09-10T18:25:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 33785/325509 objects degraded (10.379%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:05 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4072911912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:25:06.010 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:25:06.010 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:25:06.057 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:25:06.058 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:25:06.059 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:06.082 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:06.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:06.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:06.095 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:25:06.095 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:25:06.130 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:25:06.131 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:25:06.164 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:25:06.164 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:25:06.172 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-10T18:25:06.184 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:06.207 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:06.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:06.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:06.216 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:25:06.216 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:25:06.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:05 smithi204 ceph-mon[32357]: 7.8 scrub starts 2024-09-10T18:25:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:05 smithi204 ceph-mon[32357]: 3.e scrub starts 2024-09-10T18:25:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:05 smithi204 ceph-mon[32357]: 3.e scrub ok 2024-09-10T18:25:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:05 smithi204 ceph-mon[32357]: pgmap v135: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 758 op/s; 33785/325509 objects degraded (10.379%) 2024-09-10T18:25:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:05 smithi204 ceph-mon[32357]: 6.e scrub starts 2024-09-10T18:25:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:05 smithi204 ceph-mon[32357]: 4.3 scrub starts 2024-09-10T18:25:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:05 smithi204 ceph-mon[32357]: 4.3 scrub ok 2024-09-10T18:25:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:05 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 33785/325509 objects degraded (10.379%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:05 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4072911912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:25:06.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:06.357 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:06.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:06.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:06.486 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:06.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:06.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:06.587 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:06.611 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:06.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:06.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:06.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:06.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:06.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:06.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:06.898 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 83 16 114264 114248 86.0087 48.625 0.0363366 0.0116221 2024-09-10T18:25:06.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:06.901 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:06.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:07.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:07.042 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:07.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:07.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:07.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-10T18:25:07.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:07.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:07.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:07.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:07.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:07.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:07.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:07.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:07.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:07.519 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:07.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:07.620 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:07.643 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:07.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:07.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:07.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:07.767 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:07.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:07.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:07.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:07.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:07.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:07.898 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 84 16 115062 115046 85.5785 49.875 0.00715458 0.0116786 2024-09-10T18:25:07.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:08.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:08.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:08.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:08.161 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:08.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:08.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:07 smithi028 ceph-mon[39220]: 7.5 scrub ok 2024-09-10T18:25:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:07 smithi028 ceph-mon[39220]: pgmap v136: 148 pgs: 3 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 35073/338148 objects degraded (10.372%) 2024-09-10T18:25:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:07 smithi028 ceph-mon[39220]: 4.d scrub starts 2024-09-10T18:25:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:07 smithi028 ceph-mon[39220]: 4.d scrub ok 2024-09-10T18:25:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:07 smithi028 ceph-mon[39220]: 7.8 scrub ok 2024-09-10T18:25:08.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:07 smithi028 ceph-mon[29291]: 7.5 scrub ok 2024-09-10T18:25:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:07 smithi028 ceph-mon[29291]: pgmap v136: 148 pgs: 3 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 35073/338148 objects degraded (10.372%) 2024-09-10T18:25:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:07 smithi028 ceph-mon[29291]: 4.d scrub starts 2024-09-10T18:25:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:07 smithi028 ceph-mon[29291]: 4.d scrub ok 2024-09-10T18:25:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:07 smithi028 ceph-mon[29291]: 7.8 scrub ok 2024-09-10T18:25:08.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:08.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:08.301 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:08.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:07 smithi204 ceph-mon[32357]: 7.5 scrub ok 2024-09-10T18:25:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:07 smithi204 ceph-mon[32357]: pgmap v136: 148 pgs: 3 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 35073/338148 objects degraded (10.372%) 2024-09-10T18:25:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:07 smithi204 ceph-mon[32357]: 4.d scrub starts 2024-09-10T18:25:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:07 smithi204 ceph-mon[32357]: 4.d scrub ok 2024-09-10T18:25:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:07 smithi204 ceph-mon[32357]: 7.8 scrub ok 2024-09-10T18:25:08.402 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:08.426 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:08.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:08.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:08.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:08.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:08.585 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:08.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:08.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:08.697 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T18:25:08.697 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T18:25:08.727 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:08.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:08.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:08.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:08.851 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:08.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:08.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:08.898 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 85 16 116032 116016 85.285 60.625 0.0104299 0.0117202 2024-09-10T18:25:08.941 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:25:08.942 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-10T18:25:08.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-10T18:25:08.942 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:25:08.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.942 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:28260 7.8 7:1fde44db:::benchmark_data_smithi028_77856_object28259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:25:08.942 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:55.399724+0000", 2024-09-10T18:25:08.942 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.536422837999993, 2024-09-10T18:25:08.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.12490164400000001, 2024-09-10T18:25:08.942 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:08.942 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:08.942 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:08.942 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:08.943 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:08.943 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 28260 2024-09-10T18:25:08.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.943 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:08.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:08.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.399724+0000", 2024-09-10T18:25:08.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:08.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.399724+0000", 2024-09-10T18:25:08.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:08.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.399722+0000", 2024-09-10T18:25:08.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:25:08.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:08.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.399747+0000", 2024-09-10T18:25:08.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5327999999999999e-05 2024-09-10T18:25:08.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:08.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.399758+0000", 2024-09-10T18:25:08.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0573e-05 2024-09-10T18:25:08.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:08.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.399764+0000", 2024-09-10T18:25:08.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0610000000000001e-06 2024-09-10T18:25:08.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:08.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.399815+0000", 2024-09-10T18:25:08.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0937999999999999e-05 2024-09-10T18:25:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.399949+0000", 2024-09-10T18:25:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013424699999999999 2024-09-10T18:25:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,7", 2024-09-10T18:25:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.400158+0000", 2024-09-10T18:25:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020943000000000001 2024-09-10T18:25:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.524202+0000", 2024-09-10T18:25:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.12404319699999999 2024-09-10T18:25:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.524235+0000", 2024-09-10T18:25:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2965e-05 2024-09-10T18:25:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.524304+0000", 2024-09-10T18:25:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8989e-05 2024-09-10T18:25:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.524537+0000", 2024-09-10T18:25:08.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023318199999999999 2024-09-10T18:25:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.524625+0000", 2024-09-10T18:25:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8740999999999996e-05 2024-09-10T18:25:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:28265 7.8 7:108777f1:::benchmark_data_smithi028_77856_object28264:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:25:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:55.400012+0000", 2024-09-10T18:25:08.948 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.536134215000004, 2024-09-10T18:25:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.12631735499999999, 2024-09-10T18:25:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 28265 2024-09-10T18:25:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:08.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.400012+0000", 2024-09-10T18:25:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.400012+0000", 2024-09-10T18:25:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.400011+0000", 2024-09-10T18:25:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:25:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.400036+0000", 2024-09-10T18:25:08.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4712e-05 2024-09-10T18:25:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.400045+0000", 2024-09-10T18:25:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8720000000000008e-06 2024-09-10T18:25:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.400050+0000", 2024-09-10T18:25:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6699999999999999e-06 2024-09-10T18:25:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.402395+0000", 2024-09-10T18:25:08.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023442910000000001 2024-09-10T18:25:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.402508+0000", 2024-09-10T18:25:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011372 2024-09-10T18:25:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,7", 2024-09-10T18:25:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.402656+0000", 2024-09-10T18:25:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014803599999999999 2024-09-10T18:25:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.524262+0000", 2024-09-10T18:25:08.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.12160552199999999 2024-09-10T18:25:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.524289+0000", 2024-09-10T18:25:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6835e-05 2024-09-10T18:25:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.526007+0000", 2024-09-10T18:25:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017181570000000001 2024-09-10T18:25:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.526211+0000", 2024-09-10T18:25:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020422100000000001 2024-09-10T18:25:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.526330+0000", 2024-09-10T18:25:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000118658 2024-09-10T18:25:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.14808.0:28260 7.8 e100/93 ondisk, result = 0)", 2024-09-10T18:25:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:55.401554+0000", 2024-09-10T18:25:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.534592334999999, 2024-09-10T18:25:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.12265061400000001, 2024-09-10T18:25:08.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:25:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.401554+0000", 2024-09-10T18:25:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.401554+0000", 2024-09-10T18:25:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:08.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.401553+0000", 2024-09-10T18:25:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:25:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.401559+0000", 2024-09-10T18:25:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7919999999999998e-06 2024-09-10T18:25:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.401568+0000", 2024-09-10T18:25:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7370000000000001e-06 2024-09-10T18:25:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:08.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.401578+0000", 2024-09-10T18:25:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5170000000000005e-06 2024-09-10T18:25:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.524187+0000", 2024-09-10T18:25:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.122609024 2024-09-10T18:25:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.524195+0000", 2024-09-10T18:25:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9240000000000007e-06 2024-09-10T18:25:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:08.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.524205+0000", 2024-09-10T18:25:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9890000000000004e-06 2024-09-10T18:25:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.14808.0:28260 7.8 e100/93 ondisk, result = 0)", 2024-09-10T18:25:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:55.401732+0000", 2024-09-10T18:25:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.534414537999993, 2024-09-10T18:25:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.122503742, 2024-09-10T18:25:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:25:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:08.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.401732+0000", 2024-09-10T18:25:08.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:08.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.401732+0000", 2024-09-10T18:25:08.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:08.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.401731+0000", 2024-09-10T18:25:08.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:25:08.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:08.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.401736+0000", 2024-09-10T18:25:08.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8749999999999999e-06 2024-09-10T18:25:08.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:08.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.401748+0000", 2024-09-10T18:25:08.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.172e-05 2024-09-10T18:25:08.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:08.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.401752+0000", 2024-09-10T18:25:08.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5959999999999997e-06 2024-09-10T18:25:08.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:08.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.524224+0000", 2024-09-10T18:25:08.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.122471787 2024-09-10T18:25:08.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:08.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.524230+0000", 2024-09-10T18:25:08.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9370000000000001e-06 2024-09-10T18:25:08.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:08.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.524236+0000", 2024-09-10T18:25:08.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9200000000000001e-06 2024-09-10T18:25:08.961 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.961 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:08.961 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.962 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.14808.0:28265 7.8 e100/93 ondisk, result = 0)", 2024-09-10T18:25:08.962 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:55.404318+0000", 2024-09-10T18:25:08.962 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.531829025999997, 2024-09-10T18:25:08.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.11994569099999999, 2024-09-10T18:25:08.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:08.962 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:25:08.962 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:08.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:08.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.404318+0000", 2024-09-10T18:25:08.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:08.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.404318+0000", 2024-09-10T18:25:08.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:08.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.404316+0000", 2024-09-10T18:25:08.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:25:08.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:08.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.404322+0000", 2024-09-10T18:25:08.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0519999999999998e-06 2024-09-10T18:25:08.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:08.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.522990+0000", 2024-09-10T18:25:08.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.11866784900000001 2024-09-10T18:25:08.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:08.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.523002+0000", 2024-09-10T18:25:08.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2077e-05 2024-09-10T18:25:08.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:08.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.524252+0000", 2024-09-10T18:25:08.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001249607 2024-09-10T18:25:08.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:08.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.524257+0000", 2024-09-10T18:25:08.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4999999999999999e-06 2024-09-10T18:25:08.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:08.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.524263+0000", 2024-09-10T18:25:08.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9989999999999997e-06 2024-09-10T18:25:08.965 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.965 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:08.965 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.966 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:28630 7.e 7:73051138:::benchmark_data_smithi028_77856_object28629:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:25:08.966 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:55.725848+0000", 2024-09-10T18:25:08.966 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.210298539999997, 2024-09-10T18:25:08.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13805567499999999, 2024-09-10T18:25:08.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:08.966 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:08.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:08.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:08.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:08.966 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 28630 2024-09-10T18:25:08.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.966 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:08.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:08.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.725848+0000", 2024-09-10T18:25:08.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:08.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.725848+0000", 2024-09-10T18:25:08.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:08.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.725846+0000", 2024-09-10T18:25:08.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:25:08.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:08.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.725881+0000", 2024-09-10T18:25:08.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4428000000000003e-05 2024-09-10T18:25:08.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:08.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.725891+0000", 2024-09-10T18:25:08.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0454e-05 2024-09-10T18:25:08.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:08.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.725897+0000", 2024-09-10T18:25:08.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3669999999999996e-06 2024-09-10T18:25:08.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:08.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:08.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726070+0000", 2024-09-10T18:25:08.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017355699999999999 2024-09-10T18:25:08.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:08.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726200+0000", 2024-09-10T18:25:08.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129784 2024-09-10T18:25:08.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:25:08.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726386+0000", 2024-09-10T18:25:08.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018569100000000001 2024-09-10T18:25:08.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:08.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.727974+0000", 2024-09-10T18:25:08.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001588173 2024-09-10T18:25:08.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:08.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.727989+0000", 2024-09-10T18:25:08.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5153999999999999e-05 2024-09-10T18:25:08.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:08.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.863819+0000", 2024-09-10T18:25:08.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.135830595 2024-09-10T18:25:08.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:08.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.863864+0000", 2024-09-10T18:25:08.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4137000000000001e-05 2024-09-10T18:25:08.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:08.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.863904+0000", 2024-09-10T18:25:08.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0163999999999997e-05 2024-09-10T18:25:08.973 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.973 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:08.973 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.973 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29819 7.e 7:7d2523b5:::benchmark_data_smithi028_77856_object29818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:25:08.973 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.539630+0000", 2024-09-10T18:25:08.974 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.396517071999995, 2024-09-10T18:25:08.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25094886500000002, 2024-09-10T18:25:08.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:08.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:08.974 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:08.974 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:08.974 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:08.974 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29819 2024-09-10T18:25:08.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.974 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:08.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:08.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.539630+0000", 2024-09-10T18:25:08.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:08.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.539630+0000", 2024-09-10T18:25:08.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:08.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.539627+0000", 2024-09-10T18:25:08.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:25:08.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:08.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.539654+0000", 2024-09-10T18:25:08.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.614e-05 2024-09-10T18:25:08.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:08.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.539665+0000", 2024-09-10T18:25:08.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1272e-05 2024-09-10T18:25:08.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:08.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.539670+0000", 2024-09-10T18:25:08.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5670000000000002e-06 2024-09-10T18:25:08.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:08.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.539791+0000", 2024-09-10T18:25:08.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012051 2024-09-10T18:25:08.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:08.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.539913+0000", 2024-09-10T18:25:08.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012243599999999999 2024-09-10T18:25:08.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:25:08.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.540029+0000", 2024-09-10T18:25:08.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000115837 2024-09-10T18:25:08.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:08.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.542070+0000", 2024-09-10T18:25:08.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020411959999999999 2024-09-10T18:25:08.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:08.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.542345+0000", 2024-09-10T18:25:08.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027425000000000003 2024-09-10T18:25:08.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:08.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790496+0000", 2024-09-10T18:25:08.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.248150961 2024-09-10T18:25:08.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:08.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790543+0000", 2024-09-10T18:25:08.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7553999999999999e-05 2024-09-10T18:25:08.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:08.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790578+0000", 2024-09-10T18:25:08.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5324000000000003e-05 2024-09-10T18:25:08.980 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.980 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:08.980 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.980 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29835 7.8 7:10b6d885:::benchmark_data_smithi028_77856_object29834:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:25:08.980 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.545862+0000", 2024-09-10T18:25:08.980 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.390284956000002, 2024-09-10T18:25:08.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.243747676, 2024-09-10T18:25:08.980 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:08.980 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:08.981 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:08.981 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:08.981 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:08.981 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29835 2024-09-10T18:25:08.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.981 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:08.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:08.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545862+0000", 2024-09-10T18:25:08.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:08.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545862+0000", 2024-09-10T18:25:08.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:08.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545860+0000", 2024-09-10T18:25:08.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:25:08.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:08.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545975+0000", 2024-09-10T18:25:08.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000115219 2024-09-10T18:25:08.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:08.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545986+0000", 2024-09-10T18:25:08.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1394e-05 2024-09-10T18:25:08.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:08.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545993+0000", 2024-09-10T18:25:08.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0430000000000004e-06 2024-09-10T18:25:08.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:08.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546086+0000", 2024-09-10T18:25:08.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3540000000000002e-05 2024-09-10T18:25:08.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:08.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546185+0000", 2024-09-10T18:25:08.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9204999999999996e-05 2024-09-10T18:25:08.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,7", 2024-09-10T18:25:08.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546298+0000", 2024-09-10T18:25:08.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000112613 2024-09-10T18:25:08.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:08.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547603+0000", 2024-09-10T18:25:08.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013053979999999999 2024-09-10T18:25:08.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789116+0000", 2024-09-10T18:25:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24151288300000001 2024-09-10T18:25:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789526+0000", 2024-09-10T18:25:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00040943499999999999 2024-09-10T18:25:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789569+0000", 2024-09-10T18:25:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3273999999999999e-05 2024-09-10T18:25:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:08.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789609+0000", 2024-09-10T18:25:08.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0463999999999997e-05 2024-09-10T18:25:08.986 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.986 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:08.986 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.986 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29836 7.e 7:7efd3370:::benchmark_data_smithi028_77856_object29835:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:25:08.986 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.546009+0000", 2024-09-10T18:25:08.986 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.390137847000005, 2024-09-10T18:25:08.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24472428600000001, 2024-09-10T18:25:08.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:08.986 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:08.986 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:08.986 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:08.987 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:08.987 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29836 2024-09-10T18:25:08.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.987 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:08.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:08.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546009+0000", 2024-09-10T18:25:08.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:08.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546009+0000", 2024-09-10T18:25:08.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:08.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546008+0000", 2024-09-10T18:25:08.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:25:08.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:08.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546030+0000", 2024-09-10T18:25:08.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2206000000000001e-05 2024-09-10T18:25:08.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:08.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546038+0000", 2024-09-10T18:25:08.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7430000000000002e-06 2024-09-10T18:25:08.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546046+0000", 2024-09-10T18:25:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.405e-06 2024-09-10T18:25:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546430+0000", 2024-09-10T18:25:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00038401400000000001 2024-09-10T18:25:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546549+0000", 2024-09-10T18:25:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000118958 2024-09-10T18:25:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:25:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546680+0000", 2024-09-10T18:25:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013058199999999999 2024-09-10T18:25:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548016+0000", 2024-09-10T18:25:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013362160000000001 2024-09-10T18:25:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790618+0000", 2024-09-10T18:25:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24260189800000001 2024-09-10T18:25:08.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:08.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790633+0000", 2024-09-10T18:25:08.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5290000000000001e-05 2024-09-10T18:25:08.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:08.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790672+0000", 2024-09-10T18:25:08.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8587999999999997e-05 2024-09-10T18:25:08.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:08.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790733+0000", 2024-09-10T18:25:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1480000000000001e-05 2024-09-10T18:25:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:29831 7.a e100/93 7:5aa0b980:::benchmark_data_smithi028_77856_object29830:head v 100'1871, mlcod=100'1869)", 2024-09-10T18:25:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.546864+0000", 2024-09-10T18:25:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.389282168999998, 2024-09-10T18:25:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.242727832, 2024-09-10T18:25:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546864+0000", 2024-09-10T18:25:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546864+0000", 2024-09-10T18:25:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546863+0000", 2024-09-10T18:25:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:25:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:08.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546893+0000", 2024-09-10T18:25:08.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0133000000000001e-05 2024-09-10T18:25:08.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:08.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546902+0000", 2024-09-10T18:25:08.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0380000000000005e-06 2024-09-10T18:25:08.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:08.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546907+0000", 2024-09-10T18:25:08.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0200000000000002e-06 2024-09-10T18:25:08.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:08.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546986+0000", 2024-09-10T18:25:08.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8712000000000001e-05 2024-09-10T18:25:08.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:08.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547004+0000", 2024-09-10T18:25:08.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8357000000000001e-05 2024-09-10T18:25:08.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:08.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789563+0000", 2024-09-10T18:25:08.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24255862 2024-09-10T18:25:08.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:08.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789592+0000", 2024-09-10T18:25:08.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9212000000000001e-05 2024-09-10T18:25:08.996 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.996 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:08.996 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:08.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.996 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:29834 7.5 e100/93 7:a74714fe:::benchmark_data_smithi028_77856_object29833:head v 100'1873, mlcod=100'1871)", 2024-09-10T18:25:08.996 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.547005+0000", 2024-09-10T18:25:08.996 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.389141903999999, 2024-09-10T18:25:08.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24378228900000001, 2024-09-10T18:25:08.996 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:08.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:08.997 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:08.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:08.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547005+0000", 2024-09-10T18:25:08.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:08.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547005+0000", 2024-09-10T18:25:08.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:08.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:08.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547004+0000", 2024-09-10T18:25:08.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:25:08.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:08.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547045+0000", 2024-09-10T18:25:08.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1198000000000001e-05 2024-09-10T18:25:08.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:08.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547054+0000", 2024-09-10T18:25:08.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8040000000000007e-06 2024-09-10T18:25:08.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:08.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:08.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:08.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547058+0000", 2024-09-10T18:25:08.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.707e-06 2024-09-10T18:25:08.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:09.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547151+0000", 2024-09-10T18:25:09.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2991999999999993e-05 2024-09-10T18:25:09.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:09.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547180+0000", 2024-09-10T18:25:09.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8696000000000001e-05 2024-09-10T18:25:09.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:09.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790683+0000", 2024-09-10T18:25:09.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24350297100000001 2024-09-10T18:25:09.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:09.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790787+0000", 2024-09-10T18:25:09.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010414 2024-09-10T18:25:09.001 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.001 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:09.001 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.001 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.14808.0:29835 7.8 e100/93 ondisk, result = 0)", 2024-09-10T18:25:09.001 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.548280+0000", 2024-09-10T18:25:09.002 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.387866955000007, 2024-09-10T18:25:09.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24084339199999999, 2024-09-10T18:25:09.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:09.002 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:25:09.002 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:09.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:09.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548280+0000", 2024-09-10T18:25:09.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:09.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548280+0000", 2024-09-10T18:25:09.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:09.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548279+0000", 2024-09-10T18:25:09.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:25:09.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:09.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548284+0000", 2024-09-10T18:25:09.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5119999999999999e-06 2024-09-10T18:25:09.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:09.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548293+0000", 2024-09-10T18:25:09.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2609999999999999e-06 2024-09-10T18:25:09.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:09.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548298+0000", 2024-09-10T18:25:09.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.707e-06 2024-09-10T18:25:09.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:09.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789037+0000", 2024-09-10T18:25:09.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.240738535 2024-09-10T18:25:09.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:09.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789096+0000", 2024-09-10T18:25:09.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9354999999999997e-05 2024-09-10T18:25:09.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:09.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789123+0000", 2024-09-10T18:25:09.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7143000000000001e-05 2024-09-10T18:25:09.005 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.005 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:09.005 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.005 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29837 7.8 7:106d3607:::benchmark_data_smithi028_77856_object29836:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:25:09.006 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.548334+0000", 2024-09-10T18:25:09.006 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.387812244000003, 2024-09-10T18:25:09.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24611195, 2024-09-10T18:25:09.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:09.006 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:09.006 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:09.006 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:09.006 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:09.006 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29837 2024-09-10T18:25:09.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.006 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:09.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:09.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548334+0000", 2024-09-10T18:25:09.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:09.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548334+0000", 2024-09-10T18:25:09.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:09.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548332+0000", 2024-09-10T18:25:09.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:25:09.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:09.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548359+0000", 2024-09-10T18:25:09.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6567000000000002e-05 2024-09-10T18:25:09.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:09.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548401+0000", 2024-09-10T18:25:09.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1902e-05 2024-09-10T18:25:09.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:09.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548407+0000", 2024-09-10T18:25:09.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4500000000000001e-06 2024-09-10T18:25:09.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:09.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789639+0000", 2024-09-10T18:25:09.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24123214800000001 2024-09-10T18:25:09.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:09.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789814+0000", 2024-09-10T18:25:09.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017425399999999999 2024-09-10T18:25:09.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,7", 2024-09-10T18:25:09.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789956+0000", 2024-09-10T18:25:09.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014267300000000001 2024-09-10T18:25:09.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:09.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.794173+0000", 2024-09-10T18:25:09.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042164480000000002 2024-09-10T18:25:09.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:09.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.794180+0000", 2024-09-10T18:25:09.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9619999999999996e-06 2024-09-10T18:25:09.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:09.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.794367+0000", 2024-09-10T18:25:09.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000186947 2024-09-10T18:25:09.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:09.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.794413+0000", 2024-09-10T18:25:09.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6644000000000003e-05 2024-09-10T18:25:09.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:09.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.794446+0000", 2024-09-10T18:25:09.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3011e-05 2024-09-10T18:25:09.011 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.011 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:09.011 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.012 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.14808.0:29835 7.8 e100/93 ondisk, result = 0)", 2024-09-10T18:25:09.012 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.548378+0000", 2024-09-10T18:25:09.012 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.387768711000007, 2024-09-10T18:25:09.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.241239025, 2024-09-10T18:25:09.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:09.012 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:25:09.012 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:09.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:09.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548378+0000", 2024-09-10T18:25:09.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:09.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548378+0000", 2024-09-10T18:25:09.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:09.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548377+0000", 2024-09-10T18:25:09.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:25:09.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:09.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548382+0000", 2024-09-10T18:25:09.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7269999999999998e-06 2024-09-10T18:25:09.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:09.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548416+0000", 2024-09-10T18:25:09.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3991e-05 2024-09-10T18:25:09.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:09.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548419+0000", 2024-09-10T18:25:09.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.298e-06 2024-09-10T18:25:09.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:09.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789514+0000", 2024-09-10T18:25:09.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.241094586 2024-09-10T18:25:09.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:09.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789520+0000", 2024-09-10T18:25:09.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6680000000000004e-06 2024-09-10T18:25:09.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:09.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789617+0000", 2024-09-10T18:25:09.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6757999999999998e-05 2024-09-10T18:25:09.015 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.015 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:09.015 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.015 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29838 7.e 7:7d62ddc4:::benchmark_data_smithi028_77856_object29837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:25:09.016 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.548432+0000", 2024-09-10T18:25:09.016 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.387714537999997, 2024-09-10T18:25:09.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25066205800000002, 2024-09-10T18:25:09.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:09.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:09.016 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:09.016 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:09.016 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:09.016 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29838 2024-09-10T18:25:09.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.016 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:09.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:09.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548432+0000", 2024-09-10T18:25:09.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:09.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548432+0000", 2024-09-10T18:25:09.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:09.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548431+0000", 2024-09-10T18:25:09.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:25:09.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:09.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548453+0000", 2024-09-10T18:25:09.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1889e-05 2024-09-10T18:25:09.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:09.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548462+0000", 2024-09-10T18:25:09.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3370000000000006e-06 2024-09-10T18:25:09.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:09.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548466+0000", 2024-09-10T18:25:09.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7809999999999999e-06 2024-09-10T18:25:09.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:09.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548524+0000", 2024-09-10T18:25:09.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7541999999999999e-05 2024-09-10T18:25:09.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:09.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548653+0000", 2024-09-10T18:25:09.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129154 2024-09-10T18:25:09.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:25:09.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789234+0000", 2024-09-10T18:25:09.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.240580764 2024-09-10T18:25:09.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:09.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790805+0000", 2024-09-10T18:25:09.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001571486 2024-09-10T18:25:09.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:09.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790966+0000", 2024-09-10T18:25:09.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016126199999999999 2024-09-10T18:25:09.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:09.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.799018+0000", 2024-09-10T18:25:09.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080513049999999999 2024-09-10T18:25:09.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:09.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.799062+0000", 2024-09-10T18:25:09.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4209000000000003e-05 2024-09-10T18:25:09.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:09.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.799094+0000", 2024-09-10T18:25:09.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2149e-05 2024-09-10T18:25:09.021 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.021 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:09.021 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.022 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.14808.0:29819 7.e e100/93 ondisk, result = 0)", 2024-09-10T18:25:09.022 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.549107+0000", 2024-09-10T18:25:09.022 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.387040076000005, 2024-09-10T18:25:09.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.241481533, 2024-09-10T18:25:09.022 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:09.022 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:25:09.022 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:09.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:09.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549107+0000", 2024-09-10T18:25:09.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:09.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549107+0000", 2024-09-10T18:25:09.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:09.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549105+0000", 2024-09-10T18:25:09.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:25:09.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:09.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549111+0000", 2024-09-10T18:25:09.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7409999999999998e-06 2024-09-10T18:25:09.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:09.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789574+0000", 2024-09-10T18:25:09.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.240462388 2024-09-10T18:25:09.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:09.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789580+0000", 2024-09-10T18:25:09.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4570000000000002e-06 2024-09-10T18:25:09.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:09.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790483+0000", 2024-09-10T18:25:09.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00090302299999999998 2024-09-10T18:25:09.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:09.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790490+0000", 2024-09-10T18:25:09.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.674e-06 2024-09-10T18:25:09.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:09.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790588+0000", 2024-09-10T18:25:09.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8424000000000005e-05 2024-09-10T18:25:09.025 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.026 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:09.026 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.026 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:29844 7.5 e100/93 7:a491b2d4:::benchmark_data_smithi028_77856_object29843:head v 100'1874, mlcod=100'1871)", 2024-09-10T18:25:09.026 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.550888+0000", 2024-09-10T18:25:09.026 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.385258563999997, 2024-09-10T18:25:09.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.241289533, 2024-09-10T18:25:09.027 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:09.027 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:09.027 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:09.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:09.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.550888+0000", 2024-09-10T18:25:09.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:09.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.550888+0000", 2024-09-10T18:25:09.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:09.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.550886+0000", 2024-09-10T18:25:09.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:25:09.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:09.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.550927+0000", 2024-09-10T18:25:09.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1322999999999997e-05 2024-09-10T18:25:09.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:09.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.788973+0000", 2024-09-10T18:25:09.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23804528899999999 2024-09-10T18:25:09.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:09.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789001+0000", 2024-09-10T18:25:09.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8816e-05 2024-09-10T18:25:09.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:09.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789552+0000", 2024-09-10T18:25:09.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00055091099999999996 2024-09-10T18:25:09.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:09.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789574+0000", 2024-09-10T18:25:09.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1883e-05 2024-09-10T18:25:09.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:09.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.792111+0000", 2024-09-10T18:25:09.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025364960000000001 2024-09-10T18:25:09.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:09.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.792178+0000", 2024-09-10T18:25:09.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6939000000000004e-05 2024-09-10T18:25:09.031 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.031 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:09.031 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.031 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:38175 7.e 7:75d822d5:::benchmark_data_smithi028_77856_object38174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-10T18:25:09.031 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:24:02.009669+0000", 2024-09-10T18:25:09.031 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 66.926477992000002, 2024-09-10T18:25:09.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.087222025999999994, 2024-09-10T18:25:09.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:09.032 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:09.032 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:09.032 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:09.032 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:09.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 38175 2024-09-10T18:25:09.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.032 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:09.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:09.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:02.009669+0000", 2024-09-10T18:25:09.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:09.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:02.009669+0000", 2024-09-10T18:25:09.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:09.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:02.009666+0000", 2024-09-10T18:25:09.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:25:09.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:09.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:02.009691+0000", 2024-09-10T18:25:09.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4375999999999998e-05 2024-09-10T18:25:09.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:09.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:02.009703+0000", 2024-09-10T18:25:09.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1795e-05 2024-09-10T18:25:09.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:09.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:02.009708+0000", 2024-09-10T18:25:09.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8289999999999996e-06 2024-09-10T18:25:09.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:09.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:02.009758+0000", 2024-09-10T18:25:09.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0115999999999999e-05 2024-09-10T18:25:09.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:09.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:02.009886+0000", 2024-09-10T18:25:09.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000127408 2024-09-10T18:25:09.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:25:09.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:02.010009+0000", 2024-09-10T18:25:09.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012354400000000001 2024-09-10T18:25:09.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:09.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:02.011864+0000", 2024-09-10T18:25:09.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018547360000000001 2024-09-10T18:25:09.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:09.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:02.012134+0000", 2024-09-10T18:25:09.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026960100000000001 2024-09-10T18:25:09.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:09.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:02.096789+0000", 2024-09-10T18:25:09.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.084655307999999999 2024-09-10T18:25:09.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:09.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:02.096847+0000", 2024-09-10T18:25:09.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8439000000000001e-05 2024-09-10T18:25:09.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:09.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:24:02.096891+0000", 2024-09-10T18:25:09.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3174999999999999e-05 2024-09-10T18:25:09.037 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.037 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:09.037 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.038 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:115043 7.e 7:77710db5:::benchmark_data_smithi028_77856_object115042:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-10T18:25:09.038 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:07.833264+0000", 2024-09-10T18:25:09.038 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.102882527, 2024-09-10T18:25:09.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.088276322000000004, 2024-09-10T18:25:09.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:09.038 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:09.038 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:09.038 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:09.038 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:09.038 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 115043 2024-09-10T18:25:09.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.038 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:09.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.039 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:09.039 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:07.833264+0000", 2024-09-10T18:25:09.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.039 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:09.039 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:07.833264+0000", 2024-09-10T18:25:09.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.039 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:09.039 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:07.833261+0000", 2024-09-10T18:25:09.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-10T18:25:09.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.040 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:09.040 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:07.833290+0000", 2024-09-10T18:25:09.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8398000000000002e-05 2024-09-10T18:25:09.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.040 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:09.040 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:07.833305+0000", 2024-09-10T18:25:09.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5668999999999999e-05 2024-09-10T18:25:09.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.040 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:09.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:07.833313+0000", 2024-09-10T18:25:09.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5290000000000003e-06 2024-09-10T18:25:09.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:09.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:07.833897+0000", 2024-09-10T18:25:09.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000583842 2024-09-10T18:25:09.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:09.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:07.834023+0000", 2024-09-10T18:25:09.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012648100000000001 2024-09-10T18:25:09.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:25:09.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:07.834195+0000", 2024-09-10T18:25:09.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017148599999999999 2024-09-10T18:25:09.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:09.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:07.836913+0000", 2024-09-10T18:25:09.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002718622 2024-09-10T18:25:09.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:09.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:07.837061+0000", 2024-09-10T18:25:09.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014757400000000001 2024-09-10T18:25:09.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:09.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:07.920354+0000", 2024-09-10T18:25:09.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.083293052000000006 2024-09-10T18:25:09.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:09.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:07.920878+0000", 2024-09-10T18:25:09.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00052420400000000001 2024-09-10T18:25:09.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:09.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:07.921540+0000", 2024-09-10T18:25:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066235399999999998 2024-09-10T18:25:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:115274 7.8 7:19cbcfc6:::benchmark_data_smithi028_77856_object115273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T18:25:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:08.110629+0000", 2024-09-10T18:25:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.825517736, 2024-09-10T18:25:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.088974991000000003, 2024-09-10T18:25:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:09.044 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:09.045 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:09.045 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 115274 2024-09-10T18:25:09.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.045 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:09.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:09.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:08.110629+0000", 2024-09-10T18:25:09.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:09.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:08.110629+0000", 2024-09-10T18:25:09.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:09.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:09.046 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:08.110628+0000", 2024-09-10T18:25:09.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:25:09.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:09.046 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:08.110651+0000", 2024-09-10T18:25:09.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3363999999999999e-05 2024-09-10T18:25:09.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:09.046 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:08.110660+0000", 2024-09-10T18:25:09.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6440000000000006e-06 2024-09-10T18:25:09.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:08.110665+0000", 2024-09-10T18:25:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8300000000000003e-06 2024-09-10T18:25:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:08.114981+0000", 2024-09-10T18:25:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0043157040000000001 2024-09-10T18:25:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:08.115086+0000", 2024-09-10T18:25:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010539699999999999 2024-09-10T18:25:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,7", 2024-09-10T18:25:09.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:08.115215+0000", 2024-09-10T18:25:09.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012894499999999999 2024-09-10T18:25:09.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:09.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:08.122860+0000", 2024-09-10T18:25:09.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076445949999999997 2024-09-10T18:25:09.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:09.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:08.122892+0000", 2024-09-10T18:25:09.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2506999999999999e-05 2024-09-10T18:25:09.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.049 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:09.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:08.199510+0000", 2024-09-10T18:25:09.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.076618048999999994 2024-09-10T18:25:09.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.049 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:09.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:08.199562+0000", 2024-09-10T18:25:09.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1848000000000002e-05 2024-09-10T18:25:09.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:09.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:09.049 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:09.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:08.199604+0000", 2024-09-10T18:25:09.050 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1927e-05 2024-09-10T18:25:09.050 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.050 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:09.050 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.050 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:09.050 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:09.050 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:25:09.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:09.051 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:09.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:09.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:09.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:09.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:09.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:09.242 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: 5.b scrub starts 2024-09-10T18:25:09.242 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: 5.b scrub ok 2024-09-10T18:25:09.242 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:25:09.242 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:25:09.242 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: 5.4 scrub starts 2024-09-10T18:25:09.242 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: 5.4 scrub ok 2024-09-10T18:25:09.242 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: osdmap e120: 8 total, 7 up, 8 in 2024-09-10T18:25:09.242 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: 7.e scrub ok 2024-09-10T18:25:09.242 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: 4.17 scrub starts 2024-09-10T18:25:09.242 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: 4.17 scrub ok 2024-09-10T18:25:09.243 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:25:09.243 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: 5.b scrub starts 2024-09-10T18:25:09.243 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: 5.b scrub ok 2024-09-10T18:25:09.243 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:25:09.243 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:25:09.243 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: 5.4 scrub starts 2024-09-10T18:25:09.243 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: 5.4 scrub ok 2024-09-10T18:25:09.243 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: osdmap e120: 8 total, 7 up, 8 in 2024-09-10T18:25:09.243 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: 7.e scrub ok 2024-09-10T18:25:09.243 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: 4.17 scrub starts 2024-09-10T18:25:09.243 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: 4.17 scrub ok 2024-09-10T18:25:09.244 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:25:09.296 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:09.319 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:09.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:09.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:08 smithi204 ceph-mon[32357]: 5.b scrub starts 2024-09-10T18:25:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:08 smithi204 ceph-mon[32357]: 5.b scrub ok 2024-09-10T18:25:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:08 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:25:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:08 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:25:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:08 smithi204 ceph-mon[32357]: 5.4 scrub starts 2024-09-10T18:25:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:08 smithi204 ceph-mon[32357]: 5.4 scrub ok 2024-09-10T18:25:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:08 smithi204 ceph-mon[32357]: osdmap e120: 8 total, 7 up, 8 in 2024-09-10T18:25:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:08 smithi204 ceph-mon[32357]: 7.e scrub ok 2024-09-10T18:25:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:08 smithi204 ceph-mon[32357]: 4.17 scrub starts 2024-09-10T18:25:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:08 smithi204 ceph-mon[32357]: 4.17 scrub ok 2024-09-10T18:25:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:08 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:25:09.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:09.465 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:09.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:09.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:09.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:09.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:09.623 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:09.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:09.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:09.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:09.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:09.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:09.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:09.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 86 16 117055 117039 85.0363 63.9375 0.0160601 0.0117552 2024-09-10T18:25:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:09.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:09.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:10.018 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:10.041 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:10.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:10.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:10.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-10T18:25:10.142 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:10.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:10.167 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:10.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: 5.8 scrub starts 2024-09-10T18:25:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: 5.8 scrub ok 2024-09-10T18:25:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:25:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:25:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: pgmap v138: 148 pgs: 3 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.03k op/s; 35073/338148 objects degraded (10.372%) 2024-09-10T18:25:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: 6.2 scrub starts 2024-09-10T18:25:10.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: 6.2 scrub ok 2024-09-10T18:25:10.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[39220]: osdmap e121: 8 total, 7 up, 8 in 2024-09-10T18:25:10.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: 5.8 scrub starts 2024-09-10T18:25:10.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: 5.8 scrub ok 2024-09-10T18:25:10.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:25:10.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:25:10.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: pgmap v138: 148 pgs: 3 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.03k op/s; 35073/338148 objects degraded (10.372%) 2024-09-10T18:25:10.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: 6.2 scrub starts 2024-09-10T18:25:10.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: 6.2 scrub ok 2024-09-10T18:25:10.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:09 smithi028 ceph-mon[29291]: osdmap e121: 8 total, 7 up, 8 in 2024-09-10T18:25:10.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:10.322 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:10.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:10.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:09 smithi204 ceph-mon[32357]: 5.8 scrub starts 2024-09-10T18:25:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:09 smithi204 ceph-mon[32357]: 5.8 scrub ok 2024-09-10T18:25:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:09 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:25:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:09 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:25:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:09 smithi204 ceph-mon[32357]: pgmap v138: 148 pgs: 3 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.03k op/s; 35073/338148 objects degraded (10.372%) 2024-09-10T18:25:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:09 smithi204 ceph-mon[32357]: 6.2 scrub starts 2024-09-10T18:25:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:09 smithi204 ceph-mon[32357]: 6.2 scrub ok 2024-09-10T18:25:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:09 smithi204 ceph-mon[32357]: osdmap e121: 8 total, 7 up, 8 in 2024-09-10T18:25:10.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:10.447 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:10.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:10.548 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:10.572 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:10.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:10.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:10.606 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:25:10.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:10.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:25:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:10.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:10.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:10.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:10.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:10.876 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:10.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:10.899 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 87 16 118041 118025 84.7671 61.625 0.0117489 0.011792 2024-09-10T18:25:10.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:11.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:11.038 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:11.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:11.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:11.163 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:11.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:11.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:11.265 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:11.289 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:11.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:11.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:11.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:11 smithi028 ceph-mon[39220]: 6.4 scrub starts 2024-09-10T18:25:11.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:11 smithi028 ceph-mon[39220]: 6.4 scrub ok 2024-09-10T18:25:11.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:11 smithi028 ceph-mon[39220]: 5.e scrub starts 2024-09-10T18:25:11.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:11 smithi028 ceph-mon[39220]: 5.e scrub ok 2024-09-10T18:25:11.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:11 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 35073/338148 objects degraded (10.372%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:11.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:11 smithi028 ceph-mon[39220]: osdmap e122: 8 total, 7 up, 8 in 2024-09-10T18:25:11.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:11 smithi028 ceph-mon[29291]: 6.4 scrub starts 2024-09-10T18:25:11.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:11 smithi028 ceph-mon[29291]: 6.4 scrub ok 2024-09-10T18:25:11.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:11 smithi028 ceph-mon[29291]: 5.e scrub starts 2024-09-10T18:25:11.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:11 smithi028 ceph-mon[29291]: 5.e scrub ok 2024-09-10T18:25:11.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:11 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 35073/338148 objects degraded (10.372%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:11.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:11 smithi028 ceph-mon[29291]: osdmap e122: 8 total, 7 up, 8 in 2024-09-10T18:25:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:11 smithi204 ceph-mon[32357]: 6.4 scrub starts 2024-09-10T18:25:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:11 smithi204 ceph-mon[32357]: 6.4 scrub ok 2024-09-10T18:25:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:11 smithi204 ceph-mon[32357]: 5.e scrub starts 2024-09-10T18:25:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:11 smithi204 ceph-mon[32357]: 5.e scrub ok 2024-09-10T18:25:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:11 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 35073/338148 objects degraded (10.372%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:11 smithi204 ceph-mon[32357]: osdmap e122: 8 total, 7 up, 8 in 2024-09-10T18:25:11.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:11.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:11.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:11.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:11.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:11.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:11.669 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:25:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:25:11] "GET /metrics HTTP/1.1" 200 223653 "" "Prometheus/2.43.0" 2024-09-10T18:25:11.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:11.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:11.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:11.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:11.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:11.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:11.899 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 88 16 119047 119031 84.5183 62.875 0.008843 0.0118265 2024-09-10T18:25:11.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:12.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:12.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:12.106 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:12.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:12.130 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:12.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:12.155 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:25:12.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:12.255 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:12.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:12.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:12.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[29291]: 6.1d scrub starts 2024-09-10T18:25:12.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[29291]: 6.1d scrub ok 2024-09-10T18:25:12.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[29291]: 6.f scrub starts 2024-09-10T18:25:12.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[29291]: 6.f scrub ok 2024-09-10T18:25:12.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[29291]: pgmap v141: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 90 MiB/s wr, 1.44k op/s; 36367/351360 objects degraded (10.350%) 2024-09-10T18:25:12.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[29291]: 6.12 scrub starts 2024-09-10T18:25:12.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[29291]: 6.12 scrub ok 2024-09-10T18:25:12.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[29291]: 5.0 scrub starts 2024-09-10T18:25:12.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[29291]: 5.0 scrub ok 2024-09-10T18:25:12.268 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[39220]: 6.1d scrub starts 2024-09-10T18:25:12.268 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[39220]: 6.1d scrub ok 2024-09-10T18:25:12.268 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[39220]: 6.f scrub starts 2024-09-10T18:25:12.268 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[39220]: 6.f scrub ok 2024-09-10T18:25:12.268 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[39220]: pgmap v141: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 90 MiB/s wr, 1.44k op/s; 36367/351360 objects degraded (10.350%) 2024-09-10T18:25:12.269 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[39220]: 6.12 scrub starts 2024-09-10T18:25:12.269 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[39220]: 6.12 scrub ok 2024-09-10T18:25:12.269 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[39220]: 5.0 scrub starts 2024-09-10T18:25:12.269 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:12 smithi028 ceph-mon[39220]: 5.0 scrub ok 2024-09-10T18:25:12.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:25:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:25:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:25:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:12 smithi204 ceph-mon[32357]: 6.1d scrub starts 2024-09-10T18:25:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:12 smithi204 ceph-mon[32357]: 6.1d scrub ok 2024-09-10T18:25:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:12 smithi204 ceph-mon[32357]: 6.f scrub starts 2024-09-10T18:25:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:12 smithi204 ceph-mon[32357]: 6.f scrub ok 2024-09-10T18:25:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:12 smithi204 ceph-mon[32357]: pgmap v141: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 90 MiB/s wr, 1.44k op/s; 36367/351360 objects degraded (10.350%) 2024-09-10T18:25:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:12 smithi204 ceph-mon[32357]: 6.12 scrub starts 2024-09-10T18:25:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:12 smithi204 ceph-mon[32357]: 6.12 scrub ok 2024-09-10T18:25:12.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:12 smithi204 ceph-mon[32357]: 5.0 scrub starts 2024-09-10T18:25:12.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:12 smithi204 ceph-mon[32357]: 5.0 scrub ok 2024-09-10T18:25:12.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:12.402 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:12.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:12.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:12.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:12.526 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:12.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:12.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:12.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:12.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:12.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:12.794 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:12.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:12.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:12.896 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:12.899 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 89 16 120070 120054 84.287 63.9375 0.0176496 0.0118599 2024-09-10T18:25:12.920 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:12.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:12.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:13.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:13.032 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:25:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:13.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:13.173 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:13.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:13 smithi028 ceph-mon[39220]: 4.0 scrub starts 2024-09-10T18:25:13.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:13 smithi028 ceph-mon[39220]: 4.0 scrub ok 2024-09-10T18:25:13.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:13 smithi028 ceph-mon[39220]: 4.15 scrub starts 2024-09-10T18:25:13.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:13 smithi028 ceph-mon[39220]: 4.15 scrub ok 2024-09-10T18:25:13.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:13 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/504448486' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:25:13.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:13 smithi028 ceph-mon[39220]: from='client.24817 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:25:13.180 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:13 smithi028 ceph-mon[29291]: 4.0 scrub starts 2024-09-10T18:25:13.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:13 smithi028 ceph-mon[29291]: 4.0 scrub ok 2024-09-10T18:25:13.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:13 smithi028 ceph-mon[29291]: 4.15 scrub starts 2024-09-10T18:25:13.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:13 smithi028 ceph-mon[29291]: 4.15 scrub ok 2024-09-10T18:25:13.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:13 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/504448486' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:25:13.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:13 smithi028 ceph-mon[29291]: from='client.24817 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:25:13.197 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:13.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:13.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:13.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:13.340 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:13.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:13.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:13 smithi204 ceph-mon[32357]: 4.0 scrub starts 2024-09-10T18:25:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:13 smithi204 ceph-mon[32357]: 4.0 scrub ok 2024-09-10T18:25:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:13 smithi204 ceph-mon[32357]: 4.15 scrub starts 2024-09-10T18:25:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:13 smithi204 ceph-mon[32357]: 4.15 scrub ok 2024-09-10T18:25:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:13 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/504448486' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:25:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:13 smithi204 ceph-mon[32357]: from='client.24817 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:25:13.442 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:13.467 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:13.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:13.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:13.569 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:13.591 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:13.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:13.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:13.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:13.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:13.738 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:13.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:13.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:13.880 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:13.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:13.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:13.899 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total time run: 90.0076 2024-09-10T18:25:13.899 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total writes made: 121097 2024-09-10T18:25:13.899 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Write size: 65536 2024-09-10T18:25:13.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object size: 65536 2024-09-10T18:25:13.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Bandwidth (MB/sec): 84.0881 2024-09-10T18:25:13.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Bandwidth: 30.6494 2024-09-10T18:25:13.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max bandwidth (MB/sec): 188.375 2024-09-10T18:25:13.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min bandwidth (MB/sec): 48.625 2024-09-10T18:25:13.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average IOPS: 1345 2024-09-10T18:25:13.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev IOPS: 490.39 2024-09-10T18:25:13.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max IOPS: 3014 2024-09-10T18:25:13.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min IOPS: 778 2024-09-10T18:25:13.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average Latency(s): 0.0118882 2024-09-10T18:25:13.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Latency(s): 0.00934618 2024-09-10T18:25:13.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max latency(s): 0.253055 2024-09-10T18:25:13.900 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min latency(s): 0.00103822 2024-09-10T18:25:13.902 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Cleaning up (deleting benchmark objects) 2024-09-10T18:25:13.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:14.008 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:14.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:14.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:14.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:14.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:14.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:14.259 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:14.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:14.282 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:14.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:14.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:25:14 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:25:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:14 smithi204 ceph-mon[32357]: 5.d scrub starts 2024-09-10T18:25:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:14 smithi204 ceph-mon[32357]: 5.d scrub ok 2024-09-10T18:25:14.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:14 smithi204 ceph-mon[32357]: pgmap v142: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 46 MiB/s wr, 734 op/s; 36367/351360 objects degraded (10.350%) 2024-09-10T18:25:14.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:14 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:25:14.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:14 smithi204 ceph-mon[32357]: from='client.24817 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:25:14.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:14 smithi204 ceph-mon[32357]: osdmap e123: 8 total, 7 up, 8 in 2024-09-10T18:25:14.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:14 smithi204 ceph-mon[32357]: 4.19 deep-scrub starts 2024-09-10T18:25:14.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:14 smithi204 ceph-mon[32357]: 4.19 deep-scrub ok 2024-09-10T18:25:14.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:14 smithi204 ceph-mon[32357]: 3.5 scrub starts 2024-09-10T18:25:14.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:14 smithi204 ceph-mon[32357]: 3.5 scrub ok 2024-09-10T18:25:14.383 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:14.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:14.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:14.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:14.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[29291]: 5.d scrub starts 2024-09-10T18:25:14.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[29291]: 5.d scrub ok 2024-09-10T18:25:14.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[29291]: pgmap v142: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 46 MiB/s wr, 734 op/s; 36367/351360 objects degraded (10.350%) 2024-09-10T18:25:14.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:25:14.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[29291]: from='client.24817 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:25:14.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[29291]: osdmap e123: 8 total, 7 up, 8 in 2024-09-10T18:25:14.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[29291]: 4.19 deep-scrub starts 2024-09-10T18:25:14.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[29291]: 4.19 deep-scrub ok 2024-09-10T18:25:14.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[29291]: 3.5 scrub starts 2024-09-10T18:25:14.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[29291]: 3.5 scrub ok 2024-09-10T18:25:14.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[39220]: 5.d scrub starts 2024-09-10T18:25:14.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[39220]: 5.d scrub ok 2024-09-10T18:25:14.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[39220]: pgmap v142: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 46 MiB/s wr, 734 op/s; 36367/351360 objects degraded (10.350%) 2024-09-10T18:25:14.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:25:14.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[39220]: from='client.24817 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:25:14.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[39220]: osdmap e123: 8 total, 7 up, 8 in 2024-09-10T18:25:14.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[39220]: 4.19 deep-scrub starts 2024-09-10T18:25:14.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[39220]: 4.19 deep-scrub ok 2024-09-10T18:25:14.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[39220]: 3.5 scrub starts 2024-09-10T18:25:14.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:14 smithi028 ceph-mon[39220]: 3.5 scrub ok 2024-09-10T18:25:14.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:14.568 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:14.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:14.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:14.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:14.705 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:14.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:14.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:14.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:14.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:14.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:14.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:14.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:14.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:15.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:15.125 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:15.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:15.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:15.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:15.245 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:15 smithi028 ceph-mon[39220]: 2.0 scrub starts 2024-09-10T18:25:15.245 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:15 smithi028 ceph-mon[39220]: 2.0 scrub ok 2024-09-10T18:25:15.245 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:15 smithi028 ceph-mon[39220]: 3.9 scrub starts 2024-09-10T18:25:15.245 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:15 smithi028 ceph-mon[39220]: 3.9 scrub ok 2024-09-10T18:25:15.245 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:15 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:15.245 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:15 smithi028 ceph-mon[39220]: 7.a deep-scrub starts 2024-09-10T18:25:15.245 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:15 smithi028 ceph-mon[39220]: 6.9 scrub starts 2024-09-10T18:25:15.245 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:15 smithi028 ceph-mon[39220]: 6.9 scrub ok 2024-09-10T18:25:15.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:15 smithi028 ceph-mon[29291]: 2.0 scrub starts 2024-09-10T18:25:15.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:15 smithi028 ceph-mon[29291]: 2.0 scrub ok 2024-09-10T18:25:15.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:15 smithi028 ceph-mon[29291]: 3.9 scrub starts 2024-09-10T18:25:15.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:15 smithi028 ceph-mon[29291]: 3.9 scrub ok 2024-09-10T18:25:15.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:15 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:15.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:15 smithi028 ceph-mon[29291]: 7.a deep-scrub starts 2024-09-10T18:25:15.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:15 smithi028 ceph-mon[29291]: 6.9 scrub starts 2024-09-10T18:25:15.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:15 smithi028 ceph-mon[29291]: 6.9 scrub ok 2024-09-10T18:25:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:25:15.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:15.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:15 smithi204 ceph-mon[32357]: 2.0 scrub starts 2024-09-10T18:25:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:15 smithi204 ceph-mon[32357]: 2.0 scrub ok 2024-09-10T18:25:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:15 smithi204 ceph-mon[32357]: 3.9 scrub starts 2024-09-10T18:25:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:15 smithi204 ceph-mon[32357]: 3.9 scrub ok 2024-09-10T18:25:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:15 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:15 smithi204 ceph-mon[32357]: 7.a deep-scrub starts 2024-09-10T18:25:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:15 smithi204 ceph-mon[32357]: 6.9 scrub starts 2024-09-10T18:25:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:15 smithi204 ceph-mon[32357]: 6.9 scrub ok 2024-09-10T18:25:15.383 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:15.407 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:15.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:15.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:15.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:15.550 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:15.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:15.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:15.651 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:15.675 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:15.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:15.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:15.777 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:15.802 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:15.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:15.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:15.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:15.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:15.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:16.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:16.104 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:16.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:16.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:16.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:16.230 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:16.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:16.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:16.244 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:16 smithi028 ceph-mon[39220]: pgmap v144: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 54 MiB/s wr, 888 op/s; 36608/353364 objects degraded (10.360%) 2024-09-10T18:25:16.244 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:16 smithi028 ceph-mon[39220]: 3.b scrub starts 2024-09-10T18:25:16.244 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:16 smithi028 ceph-mon[39220]: 3.b scrub ok 2024-09-10T18:25:16.244 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:16 smithi028 ceph-mon[39220]: 7.4 scrub starts 2024-09-10T18:25:16.244 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:16 smithi028 ceph-mon[39220]: 7.0 scrub starts 2024-09-10T18:25:16.244 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:16 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 36608/353364 objects degraded (10.360%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:16.245 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:16 smithi028 ceph-mon[29291]: pgmap v144: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 54 MiB/s wr, 888 op/s; 36608/353364 objects degraded (10.360%) 2024-09-10T18:25:16.245 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:16 smithi028 ceph-mon[29291]: 3.b scrub starts 2024-09-10T18:25:16.245 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:16 smithi028 ceph-mon[29291]: 3.b scrub ok 2024-09-10T18:25:16.245 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:16 smithi028 ceph-mon[29291]: 7.4 scrub starts 2024-09-10T18:25:16.245 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:16 smithi028 ceph-mon[29291]: 7.0 scrub starts 2024-09-10T18:25:16.245 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:16 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 36608/353364 objects degraded (10.360%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:16.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:16 smithi204 ceph-mon[32357]: pgmap v144: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 7.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 54 MiB/s wr, 888 op/s; 36608/353364 objects degraded (10.360%) 2024-09-10T18:25:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:16 smithi204 ceph-mon[32357]: 3.b scrub starts 2024-09-10T18:25:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:16 smithi204 ceph-mon[32357]: 3.b scrub ok 2024-09-10T18:25:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:16 smithi204 ceph-mon[32357]: 7.4 scrub starts 2024-09-10T18:25:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:16 smithi204 ceph-mon[32357]: 7.0 scrub starts 2024-09-10T18:25:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:16 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 36608/353364 objects degraded (10.360%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:16.366 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:16.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:16.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:16.468 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:16.493 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:16.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:16.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:16.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:25:16.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:16.643 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:16.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:16.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:16.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:16.835 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:16.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:16.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:16.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:16.989 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:16.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:17.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:17.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:17.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:17.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:17.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:17 smithi028 ceph-mon[29291]: 7.2 deep-scrub starts 2024-09-10T18:25:17.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:17 smithi028 ceph-mon[29291]: 3.17 deep-scrub starts 2024-09-10T18:25:17.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:17 smithi028 ceph-mon[39220]: 7.2 deep-scrub starts 2024-09-10T18:25:17.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:17 smithi028 ceph-mon[39220]: 3.17 deep-scrub starts 2024-09-10T18:25:17.237 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:17.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:17.262 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:17.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:17 smithi204 ceph-mon[32357]: 7.2 deep-scrub starts 2024-09-10T18:25:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:17 smithi204 ceph-mon[32357]: 3.17 deep-scrub starts 2024-09-10T18:25:17.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:17.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:17.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:17.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:17.566 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:17.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:17.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:17.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:17.710 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:17.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:17.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:17.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:17.834 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:17.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:17.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:17.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:17.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:17.978 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:17.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:18.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:18.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:18 smithi028 ceph-mon[29291]: pgmap v145: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 69 MiB/s wr, 1.20k op/s; 37503/361578 objects degraded (10.372%) 2024-09-10T18:25:18.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:18 smithi028 ceph-mon[29291]: 4.1b scrub starts 2024-09-10T18:25:18.104 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:18 smithi028 ceph-mon[39220]: pgmap v145: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 69 MiB/s wr, 1.20k op/s; 37503/361578 objects degraded (10.372%) 2024-09-10T18:25:18.104 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:18 smithi028 ceph-mon[39220]: 4.1b scrub starts 2024-09-10T18:25:18.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:18.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:18.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:18.197 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:25:18.198 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:25:18.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:18.240 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:25:18.241 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:25:18.246 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:18.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:18.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:18.286 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:25:18.286 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:25:18.328 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:25:18.328 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:25:18.347 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:18 smithi204 ceph-mon[32357]: pgmap v145: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 69 MiB/s wr, 1.20k op/s; 37503/361578 objects degraded (10.372%) 2024-09-10T18:25:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:18 smithi204 ceph-mon[32357]: 4.1b scrub starts 2024-09-10T18:25:18.360 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:25:18.361 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:25:18.369 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:18.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:18.391 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:25:18.391 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:25:18.391 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:25:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:25:18.393 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-10T18:25:18.431 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:25:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:25:18.471 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:18.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:18.494 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:18.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:18.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:18.617 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:18.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:18.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:18.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:18.755 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:18.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:18.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:18.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:18.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:18.880 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:18.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:18.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:19.028 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:19.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:19.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:19.131 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:19.153 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:19.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:19.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:19.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:19.293 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:19.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:19.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:19.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:19 smithi204 ceph-mon[32357]: 5.7 scrub starts 2024-09-10T18:25:19.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:19 smithi204 ceph-mon[32357]: 7.0 scrub ok 2024-09-10T18:25:19.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:19 smithi204 ceph-mon[32357]: 3.18 deep-scrub starts 2024-09-10T18:25:19.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:19 smithi204 ceph-mon[32357]: 3.18 deep-scrub ok 2024-09-10T18:25:19.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:19 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/69346240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:25:19.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:19 smithi204 ceph-mon[32357]: 7.f deep-scrub starts 2024-09-10T18:25:19.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:19 smithi204 ceph-mon[32357]: 7.4 scrub ok 2024-09-10T18:25:19.357 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:19 smithi204 ceph-mon[32357]: 6.19 scrub starts 2024-09-10T18:25:19.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:19 smithi028 ceph-mon[39220]: 5.7 scrub starts 2024-09-10T18:25:19.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:19 smithi028 ceph-mon[39220]: 7.0 scrub ok 2024-09-10T18:25:19.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:19 smithi028 ceph-mon[39220]: 3.18 deep-scrub starts 2024-09-10T18:25:19.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:19 smithi028 ceph-mon[39220]: 3.18 deep-scrub ok 2024-09-10T18:25:19.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:19 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/69346240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:25:19.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:19 smithi028 ceph-mon[39220]: 7.f deep-scrub starts 2024-09-10T18:25:19.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:19 smithi028 ceph-mon[39220]: 7.4 scrub ok 2024-09-10T18:25:19.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:19 smithi028 ceph-mon[39220]: 6.19 scrub starts 2024-09-10T18:25:19.365 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:19 smithi028 ceph-mon[29291]: 5.7 scrub starts 2024-09-10T18:25:19.365 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:19 smithi028 ceph-mon[29291]: 7.0 scrub ok 2024-09-10T18:25:19.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:19 smithi028 ceph-mon[29291]: 3.18 deep-scrub starts 2024-09-10T18:25:19.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:19 smithi028 ceph-mon[29291]: 3.18 deep-scrub ok 2024-09-10T18:25:19.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:19 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/69346240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:25:19.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:19 smithi028 ceph-mon[29291]: 7.f deep-scrub starts 2024-09-10T18:25:19.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:19 smithi028 ceph-mon[29291]: 7.4 scrub ok 2024-09-10T18:25:19.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:19 smithi028 ceph-mon[29291]: 6.19 scrub starts 2024-09-10T18:25:19.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:19.432 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:19.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:19.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:19.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:19.579 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:19.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:19.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:19.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:19.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:19.704 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:19.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:19.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:19.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:19.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:19.944 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:19.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:19.969 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:19.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:20.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:20.106 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:20.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:20.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:20.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:20.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:20.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:20.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:20 smithi204 ceph-mon[32357]: 4.2 deep-scrub starts 2024-09-10T18:25:20.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:20 smithi204 ceph-mon[32357]: 4.2 deep-scrub ok 2024-09-10T18:25:20.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:20 smithi204 ceph-mon[32357]: pgmap v146: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 750 B/s rd, 32 MiB/s wr, 604 op/s; 37503/361578 objects degraded (10.372%) 2024-09-10T18:25:20.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:20.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:20.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:20.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:20.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:20.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:20 smithi028 ceph-mon[39220]: 4.2 deep-scrub starts 2024-09-10T18:25:20.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:20 smithi028 ceph-mon[39220]: 4.2 deep-scrub ok 2024-09-10T18:25:20.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:20 smithi028 ceph-mon[39220]: pgmap v146: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 750 B/s rd, 32 MiB/s wr, 604 op/s; 37503/361578 objects degraded (10.372%) 2024-09-10T18:25:20.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:20 smithi028 ceph-mon[29291]: 4.2 deep-scrub starts 2024-09-10T18:25:20.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:20 smithi028 ceph-mon[29291]: 4.2 deep-scrub ok 2024-09-10T18:25:20.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:20 smithi028 ceph-mon[29291]: pgmap v146: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 750 B/s rd, 32 MiB/s wr, 604 op/s; 37503/361578 objects degraded (10.372%) 2024-09-10T18:25:20.523 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:20.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:20.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:20.625 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:20.647 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:20.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:20.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:20.749 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:20.772 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:20.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:20.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:20.909 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:20.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:20.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:21.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-10T18:25:21.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:21.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:21.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:21.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:21.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:21.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:21.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:21 smithi028 ceph-mon[39220]: 4.9 deep-scrub starts 2024-09-10T18:25:21.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:21 smithi028 ceph-mon[39220]: 7.6 scrub starts 2024-09-10T18:25:21.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:21 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 37503/361578 objects degraded (10.372%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:21.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:21 smithi028 ceph-mon[29291]: 4.9 deep-scrub starts 2024-09-10T18:25:21.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:21 smithi028 ceph-mon[29291]: 7.6 scrub starts 2024-09-10T18:25:21.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:21 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 37503/361578 objects degraded (10.372%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:21.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:21.338 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:21.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:21.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:21.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:21 smithi204 ceph-mon[32357]: 4.9 deep-scrub starts 2024-09-10T18:25:21.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:21 smithi204 ceph-mon[32357]: 7.6 scrub starts 2024-09-10T18:25:21.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:21 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 37503/361578 objects degraded (10.372%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:21.441 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:21.465 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:21.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:21.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:21.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:21.590 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:21.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:21.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:21.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:21.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:21.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:21.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:25:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:25:21] "GET /metrics HTTP/1.1" 200 223653 "" "Prometheus/2.43.0" 2024-09-10T18:25:21.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:21.851 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:21.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:21.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:21.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:25:21.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:21.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:22.022 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T18:25:22.022 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T18:25:22.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:22.145 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:22.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:22.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:22.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:22 smithi028 ceph-mon[29291]: pgmap v147: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 7.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 867 op/s; 36378/350391 objects degraded (10.382%) 2024-09-10T18:25:22.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:22 smithi028 ceph-mon[39220]: pgmap v147: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 7.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 867 op/s; 36378/350391 objects degraded (10.382%) 2024-09-10T18:25:22.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:22.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:22.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:22 smithi204 ceph-mon[32357]: pgmap v147: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 7.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 867 op/s; 36378/350391 objects degraded (10.382%) 2024-09-10T18:25:22.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:25:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:25:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:25:22.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:22.420 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:22.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:22.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:22.522 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:22.545 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:22.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:22.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:22.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:22.684 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:22.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:22.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:22.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-10T18:25:22.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:22.810 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:22.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:22.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:22.912 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:22.933 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:22.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:22.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:23.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:23.057 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:23.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:23.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:23.159 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:23.180 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:23.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:23.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:23.282 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:23.303 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:23.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:23.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:23 smithi204 ceph-mon[32357]: 5.16 deep-scrub starts 2024-09-10T18:25:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:23 smithi204 ceph-mon[32357]: 3.15 scrub starts 2024-09-10T18:25:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:23 smithi204 ceph-mon[32357]: 4.16 scrub starts 2024-09-10T18:25:23.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:23 smithi028 ceph-mon[29291]: 5.16 deep-scrub starts 2024-09-10T18:25:23.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:23 smithi028 ceph-mon[29291]: 3.15 scrub starts 2024-09-10T18:25:23.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:23 smithi028 ceph-mon[29291]: 4.16 scrub starts 2024-09-10T18:25:23.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:23 smithi028 ceph-mon[39220]: 5.16 deep-scrub starts 2024-09-10T18:25:23.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:23 smithi028 ceph-mon[39220]: 3.15 scrub starts 2024-09-10T18:25:23.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:23 smithi028 ceph-mon[39220]: 4.16 scrub starts 2024-09-10T18:25:23.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:23.443 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:23.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:23.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:23.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:23.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:23.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:23.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:23.690 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:23.711 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:23.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:23.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:23.728 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:25:23.733 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:25:23.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:23.876 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:23.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:23.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:23.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:24.001 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:24.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:24.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:24.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:25:24.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:24.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:24.147 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:24.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:24.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:24 smithi028 ceph-mon[29291]: pgmap v148: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 7.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 867 op/s; 36378/350391 objects degraded (10.382%) 2024-09-10T18:25:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:24 smithi028 ceph-mon[29291]: 7.9 scrub starts 2024-09-10T18:25:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:24 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:25:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:25:24.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:24 smithi028 ceph-mon[39220]: pgmap v148: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 7.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 867 op/s; 36378/350391 objects degraded (10.382%) 2024-09-10T18:25:24.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:24 smithi028 ceph-mon[39220]: 7.9 scrub starts 2024-09-10T18:25:24.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:24 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:25:24.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:25:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:24.306 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:24.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:24.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:25:24 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:25:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:24 smithi204 ceph-mon[32357]: pgmap v148: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 7.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 867 op/s; 36378/350391 objects degraded (10.382%) 2024-09-10T18:25:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:24 smithi204 ceph-mon[32357]: 7.9 scrub starts 2024-09-10T18:25:24.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:24 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:25:24.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:25:24.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:24.449 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:24.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:24.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:24.589 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:24.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:24.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:24.713 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:24.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:24.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:24.815 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:24.838 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:24.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:24.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:24.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:24.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:24.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:24.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:25.087 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:25.110 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:25.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:25.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:25.212 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:25.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:25.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:25.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:25.335 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:25 smithi204 ceph-mon[32357]: 5.12 deep-scrub starts 2024-09-10T18:25:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:25 smithi204 ceph-mon[32357]: 6.7 scrub starts 2024-09-10T18:25:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:25 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:25 smithi204 ceph-mon[32357]: 6.e scrub starts 2024-09-10T18:25:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:25 smithi204 ceph-mon[32357]: 6.e scrub ok 2024-09-10T18:25:25.358 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:25.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:25.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:25.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:25.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:25.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:25 smithi028 ceph-mon[29291]: 5.12 deep-scrub starts 2024-09-10T18:25:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:25 smithi028 ceph-mon[29291]: 6.7 scrub starts 2024-09-10T18:25:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:25 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:25 smithi028 ceph-mon[29291]: 6.e scrub starts 2024-09-10T18:25:25.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:25 smithi028 ceph-mon[29291]: 6.e scrub ok 2024-09-10T18:25:25.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:25 smithi028 ceph-mon[39220]: 5.12 deep-scrub starts 2024-09-10T18:25:25.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:25 smithi028 ceph-mon[39220]: 6.7 scrub starts 2024-09-10T18:25:25.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:25 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:25.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:25 smithi028 ceph-mon[39220]: 6.e scrub starts 2024-09-10T18:25:25.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:25 smithi028 ceph-mon[39220]: 6.e scrub ok 2024-09-10T18:25:25.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:25.642 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:25.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:25.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:25.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:25.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:25.789 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:25.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:25.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:25.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:25.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:25.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:26.037 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:26.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:26.060 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:26.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:26.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:26.183 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:26.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:26.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:26 smithi028 ceph-mon[39220]: pgmap v149: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 22 MiB/s wr, 790 op/s; 36196/348582 objects degraded (10.384%) 2024-09-10T18:25:26.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:26 smithi028 ceph-mon[39220]: 7.9 scrub ok 2024-09-10T18:25:26.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:26 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 36196/348582 objects degraded (10.384%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:26.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:26 smithi028 ceph-mon[29291]: pgmap v149: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 22 MiB/s wr, 790 op/s; 36196/348582 objects degraded (10.384%) 2024-09-10T18:25:26.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:26 smithi028 ceph-mon[29291]: 7.9 scrub ok 2024-09-10T18:25:26.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:26 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 36196/348582 objects degraded (10.384%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:26.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:26.321 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:26.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:26 smithi204 ceph-mon[32357]: pgmap v149: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 7.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 22 MiB/s wr, 790 op/s; 36196/348582 objects degraded (10.384%) 2024-09-10T18:25:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:26 smithi204 ceph-mon[32357]: 7.9 scrub ok 2024-09-10T18:25:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:26 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 36196/348582 objects degraded (10.384%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:26.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:26.463 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:26.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:26.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:26.606 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:26.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:26.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:26.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:26.748 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:26.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:26.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:26.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:26.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:26.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:26.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:27.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:27.047 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:27.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:27.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:27.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:27.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:27.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:27.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:27 smithi028 ceph-mon[29291]: 5.a scrub starts 2024-09-10T18:25:27.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:27 smithi028 ceph-mon[29291]: 3.16 scrub starts 2024-09-10T18:25:27.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:27 smithi028 ceph-mon[29291]: 3.16 scrub ok 2024-09-10T18:25:27.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:27 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:25:27.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:27 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:25:27.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:27 smithi028 ceph-mon[29291]: 6.8 scrub starts 2024-09-10T18:25:27.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:27 smithi028 ceph-mon[39220]: 5.a scrub starts 2024-09-10T18:25:27.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:27 smithi028 ceph-mon[39220]: 3.16 scrub starts 2024-09-10T18:25:27.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:27 smithi028 ceph-mon[39220]: 3.16 scrub ok 2024-09-10T18:25:27.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:27 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:25:27.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:27 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:25:27.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:27 smithi028 ceph-mon[39220]: 6.8 scrub starts 2024-09-10T18:25:27.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:27.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:27.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:27.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:27.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:27.499 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:27.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:27.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:27.601 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:27 smithi204 ceph-mon[32357]: 5.a scrub starts 2024-09-10T18:25:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:27 smithi204 ceph-mon[32357]: 3.16 scrub starts 2024-09-10T18:25:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:27 smithi204 ceph-mon[32357]: 3.16 scrub ok 2024-09-10T18:25:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:27 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:25:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:27 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:25:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:27 smithi204 ceph-mon[32357]: 6.8 scrub starts 2024-09-10T18:25:27.622 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:27.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:27.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:27.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:27.744 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-10T18:25:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:27.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:27.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:27.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:27.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:27.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:28.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:28.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:28.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:28.159 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:28.181 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:28.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:28.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:28.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:28.324 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:28.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:28.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:28.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:28.478 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:28 smithi028 ceph-mon[39220]: 4.f scrub starts 2024-09-10T18:25:28.478 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:28 smithi028 ceph-mon[39220]: pgmap v150: 148 pgs: 1 active+clean+scrubbing, 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 96 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 938 B/s rd, 18 MiB/s wr, 900 op/s; 35434/341241 objects degraded (10.384%) 2024-09-10T18:25:28.478 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:28 smithi028 ceph-mon[39220]: 4.1b scrub starts 2024-09-10T18:25:28.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:28 smithi028 ceph-mon[39220]: osdmap e124: 8 total, 7 up, 8 in 2024-09-10T18:25:28.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:28 smithi028 ceph-mon[29291]: 4.f scrub starts 2024-09-10T18:25:28.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:28 smithi028 ceph-mon[29291]: pgmap v150: 148 pgs: 1 active+clean+scrubbing, 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 96 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 938 B/s rd, 18 MiB/s wr, 900 op/s; 35434/341241 objects degraded (10.384%) 2024-09-10T18:25:28.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:28 smithi028 ceph-mon[29291]: 4.1b scrub starts 2024-09-10T18:25:28.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:28 smithi028 ceph-mon[29291]: osdmap e124: 8 total, 7 up, 8 in 2024-09-10T18:25:28.570 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:28.593 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:28.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:28.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:28 smithi204 ceph-mon[32357]: 4.f scrub starts 2024-09-10T18:25:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:28 smithi204 ceph-mon[32357]: pgmap v150: 148 pgs: 1 active+clean+scrubbing, 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 96 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 938 B/s rd, 18 MiB/s wr, 900 op/s; 35434/341241 objects degraded (10.384%) 2024-09-10T18:25:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:28 smithi204 ceph-mon[32357]: 4.1b scrub starts 2024-09-10T18:25:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:28 smithi204 ceph-mon[32357]: osdmap e124: 8 total, 7 up, 8 in 2024-09-10T18:25:28.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:28.718 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:28.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:28.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:28.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:28.857 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:28.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:28.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:28.959 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:28.982 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:28.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:28.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:29.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:29.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:29.141 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:29.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:29.241 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:29.264 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:29.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:29.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:29.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:29.403 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:29.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:29.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:29.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:29 smithi028 ceph-mon[39220]: 3.6 scrub starts 2024-09-10T18:25:29.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:29 smithi028 ceph-mon[39220]: 3.6 scrub ok 2024-09-10T18:25:29.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:29 smithi028 ceph-mon[29291]: 3.6 scrub starts 2024-09-10T18:25:29.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:29 smithi028 ceph-mon[29291]: 3.6 scrub ok 2024-09-10T18:25:29.505 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:29.526 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:29.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:29.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:29 smithi204 ceph-mon[32357]: 3.6 scrub starts 2024-09-10T18:25:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:29 smithi204 ceph-mon[32357]: 3.6 scrub ok 2024-09-10T18:25:29.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:29.665 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:29.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:29.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:29.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:29.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:29.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:29.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:29.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:29.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:30.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:30.076 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:30.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:30.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:30.135 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:25:30.178 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:30.201 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:30.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:30.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:30.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:30 smithi028 ceph-mon[39220]: 5.a scrub starts 2024-09-10T18:25:30.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:30 smithi028 ceph-mon[39220]: 5.7 scrub starts 2024-09-10T18:25:30.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:30 smithi028 ceph-mon[39220]: 5.7 scrub ok 2024-09-10T18:25:30.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:30 smithi028 ceph-mon[39220]: pgmap v152: 148 pgs: 1 active+clean+scrubbing, 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 96 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 678 op/s; 35434/341241 objects degraded (10.384%) 2024-09-10T18:25:30.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:30 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2004521685' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:25:30.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:30 smithi028 ceph-mon[39220]: from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:25:30.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:30 smithi028 ceph-mon[29291]: 5.a scrub starts 2024-09-10T18:25:30.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:30 smithi028 ceph-mon[29291]: 5.7 scrub starts 2024-09-10T18:25:30.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:30 smithi028 ceph-mon[29291]: 5.7 scrub ok 2024-09-10T18:25:30.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:30 smithi028 ceph-mon[29291]: pgmap v152: 148 pgs: 1 active+clean+scrubbing, 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 96 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 678 op/s; 35434/341241 objects degraded (10.384%) 2024-09-10T18:25:30.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:30 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2004521685' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:25:30.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:30 smithi028 ceph-mon[29291]: from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:25:30.303 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:30.327 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:30.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:30.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:30.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:30.441 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T18:25:30.441 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2024-09-10T18:25:30.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-10T18:25:30.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:30.473 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:30.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:30.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:30 smithi204 ceph-mon[32357]: 5.a scrub starts 2024-09-10T18:25:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:30 smithi204 ceph-mon[32357]: 5.7 scrub starts 2024-09-10T18:25:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:30 smithi204 ceph-mon[32357]: 5.7 scrub ok 2024-09-10T18:25:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:30 smithi204 ceph-mon[32357]: pgmap v152: 148 pgs: 1 active+clean+scrubbing, 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 96 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 678 op/s; 35434/341241 objects degraded (10.384%) 2024-09-10T18:25:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:30 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2004521685' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:25:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:30 smithi204 ceph-mon[32357]: from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:25:30.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:30.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:30.727 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:30.753 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:30.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:30.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:30.877 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:30.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:30.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:25:30.979 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:31.002 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:31.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:31.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:31.104 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:31.127 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:31.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:31.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:31.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:31.237 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:25:31.237 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:25:31.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:31.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:31.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:31 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:25:31.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:31 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:25:31.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:31 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:25:31.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:31 smithi028 ceph-mon[29291]: from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:25:31.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:31 smithi028 ceph-mon[29291]: osdmap e125: 8 total, 7 up, 8 in 2024-09-10T18:25:31.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:31 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 35434/341241 objects degraded (10.384%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:31.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:31 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:25:31.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:31 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:25:31.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:31 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:25:31.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:31 smithi028 ceph-mon[39220]: from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:25:31.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:31 smithi028 ceph-mon[39220]: osdmap e125: 8 total, 7 up, 8 in 2024-09-10T18:25:31.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:31 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 35434/341241 objects degraded (10.384%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:31.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:31.414 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:31.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:31.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:31.516 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:31.539 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:31.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:31.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:31.599 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:25:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:25:31] "GET /metrics HTTP/1.1" 200 223729 "" "Prometheus/2.43.0" 2024-09-10T18:25:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:31 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:25:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:31 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:25:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:31 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:25:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:31 smithi204 ceph-mon[32357]: from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:25:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:31 smithi204 ceph-mon[32357]: osdmap e125: 8 total, 7 up, 8 in 2024-09-10T18:25:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:31 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 35434/341241 objects degraded (10.384%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:31.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:31.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:31.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:31.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:31.828 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:31.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:31.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:31.964 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:31.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:31.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:32.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:32.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:32.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:32.214 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:32.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:32.238 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:32.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:32.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:32.374 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:32.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:32.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:32.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:32.498 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:32.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:32.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:32 smithi028 ceph-mon[39220]: pgmap v154: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 749 op/s; 34521/332412 objects degraded (10.385%) 2024-09-10T18:25:32.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:32 smithi028 ceph-mon[29291]: pgmap v154: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 749 op/s; 34521/332412 objects degraded (10.385%) 2024-09-10T18:25:32.599 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:32 smithi204 ceph-mon[32357]: pgmap v154: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 749 op/s; 34521/332412 objects degraded (10.385%) 2024-09-10T18:25:32.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:25:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:25:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:25:32.622 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:32.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:32.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:32.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:32.757 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:32.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:32.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:32.859 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:32.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:32.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:32.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:33.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:33.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:33.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:33.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:33.163 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:33.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:33.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:33.264 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:33.286 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:33.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:33.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:33.409 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:33.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:33.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:33.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:33.535 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:33.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:33.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:33.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:33.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:33.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:33.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:33.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:33.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:33.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:33.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:33.933 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:33.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:34.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:34.057 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:34.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:34.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:34.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-10T18:25:34.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:34.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:34.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:34.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:34.346 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:34.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:34 smithi204 ceph-mon[32357]: pgmap v155: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 673 op/s; 34521/332412 objects degraded (10.385%) 2024-09-10T18:25:34.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:25:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:25:34.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:34 smithi028 ceph-mon[29291]: pgmap v155: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 673 op/s; 34521/332412 objects degraded (10.385%) 2024-09-10T18:25:34.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:34 smithi028 ceph-mon[39220]: pgmap v155: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 673 op/s; 34521/332412 objects degraded (10.385%) 2024-09-10T18:25:34.447 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:34.470 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:34.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:34.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:34.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:34.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:34.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:34.714 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:34.737 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:34.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:34.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:34.756 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:25:34.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:34.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:34.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:34.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:34.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:35.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:35.038 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:35.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:35.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:35.162 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:35.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:35.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:35.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:35.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:35.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:35.320 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:35 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:35.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:35 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:35.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:35.424 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:35.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:35.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:35.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:35.548 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:35.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:35.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:35.550 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:25:35.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:35 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:35.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:35.674 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:35.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:35.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:35.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:35.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:35.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:35.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:35.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:35.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:35.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:35.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:36.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:36.078 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:36.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:36.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:36.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:36.213 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:36.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:36.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:36.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:25:36.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:36.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:36.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:36.455 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:36.468 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:36 smithi028 ceph-mon[39220]: pgmap v156: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 439 op/s; 34340/330714 objects degraded (10.384%) 2024-09-10T18:25:36.468 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:36 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 34340/330714 objects degraded (10.384%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:36.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:36 smithi028 ceph-mon[29291]: pgmap v156: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 439 op/s; 34340/330714 objects degraded (10.384%) 2024-09-10T18:25:36.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:36 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 34340/330714 objects degraded (10.384%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:36.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:36.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:36.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:36.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:36.602 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:36.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:36.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:36.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:36 smithi204 ceph-mon[32357]: pgmap v156: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 439 op/s; 34340/330714 objects degraded (10.384%) 2024-09-10T18:25:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:36 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 34340/330714 objects degraded (10.384%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:36.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:36.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:36.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:36.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:36.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:36.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:36.993 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:37.015 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:37.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:37.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:37.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:37.175 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:37.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:37.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:37.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:37.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:37.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:37.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:37.461 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:37.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:37.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:37.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:37 smithi028 ceph-mon[29291]: pgmap v157: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 618 op/s; 33591/323298 objects degraded (10.390%) 2024-09-10T18:25:37.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:37 smithi028 ceph-mon[39220]: pgmap v157: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 618 op/s; 33591/323298 objects degraded (10.390%) 2024-09-10T18:25:37.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:37.585 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:37.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:37.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:37 smithi204 ceph-mon[32357]: pgmap v157: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 618 op/s; 33591/323298 objects degraded (10.390%) 2024-09-10T18:25:37.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:37.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:37.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:37.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:37.853 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:37.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:37.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:37.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:37.989 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:37.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:37.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:38.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:38.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:38.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:38.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:38.258 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:38.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:38.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:38.359 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:38.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:38 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:25:38.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:38 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:25:38.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:38 smithi028 ceph-mon[39220]: osdmap e126: 8 total, 7 up, 8 in 2024-09-10T18:25:38.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:38 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:25:38.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:38 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:25:38.370 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:38 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:25:38.370 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:38 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:25:38.370 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:38 smithi028 ceph-mon[29291]: osdmap e126: 8 total, 7 up, 8 in 2024-09-10T18:25:38.370 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:38 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:25:38.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:38 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:25:38.381 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:38.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:38.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:38.483 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:38.505 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:38.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:38.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:38 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:25:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:38 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:25:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:38 smithi204 ceph-mon[32357]: osdmap e126: 8 total, 7 up, 8 in 2024-09-10T18:25:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:38 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:25:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:38 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:25:38.607 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:38.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:38.629 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:38.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:38.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:38.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:38.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:38.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:38.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:38.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:39.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:39.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:39.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:39.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:39.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:39.205 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:39.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:39.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:39.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:39 smithi028 ceph-mon[39220]: osdmap e127: 8 total, 7 up, 8 in 2024-09-10T18:25:39.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:39 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:25:39.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:25:39.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:39 smithi028 ceph-mon[39220]: pgmap v160: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 380 op/s; 33591/323298 objects degraded (10.390%) 2024-09-10T18:25:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:39 smithi028 ceph-mon[29291]: osdmap e127: 8 total, 7 up, 8 in 2024-09-10T18:25:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:39 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:25:39.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:25:39.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:39 smithi028 ceph-mon[29291]: pgmap v160: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 380 op/s; 33591/323298 objects degraded (10.390%) 2024-09-10T18:25:39.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:39.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:39.336 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:39.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:39.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:39.460 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:39.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:39.561 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:39.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:39.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:39.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:39 smithi204 ceph-mon[32357]: osdmap e127: 8 total, 7 up, 8 in 2024-09-10T18:25:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:39 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:25:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:25:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:39 smithi204 ceph-mon[32357]: pgmap v160: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 380 op/s; 33591/323298 objects degraded (10.390%) 2024-09-10T18:25:39.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:39.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:39.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:39.837 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:39.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:39.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:39.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:39.962 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:40.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:40.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:40.102 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:40.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:40.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:40.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:40.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:40.262 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:40.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:40.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:40.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:40.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:40.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:40.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:40.524 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:40.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:40.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:40.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:40.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:40.675 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:40.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:40.776 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:40.798 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:40.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:40.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:40.900 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:40.922 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:40.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:41.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:40 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 33591/323298 objects degraded (10.390%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:41.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:40 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 33591/323298 objects degraded (10.390%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:41.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:41.047 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:25:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:25:41.069 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:41.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:41.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:41.089 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:25:41.089 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:25:41.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:40 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 33591/323298 objects degraded (10.390%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:41.135 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:25:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:25:41.171 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:41.172 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:25:41.172 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:25:41.194 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:41.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:41.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:41.206 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:25:41.206 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:25:41.240 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:25:41.240 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:25:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-10T18:25:41.276 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:25:41.276 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:25:41.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:41.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:41.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:41.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:41.448 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:41.471 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:41.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:41.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:41.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:41.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:41.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:41.657 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:25:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:25:41] "GET /metrics HTTP/1.1" 200 223776 "" "Prometheus/2.43.0" 2024-09-10T18:25:41.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:41.751 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:41.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:41.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:41.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:41.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:41.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:41.983 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:41 smithi028 ceph-mon[29291]: pgmap v161: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 727 op/s; 32720/314952 objects degraded (10.389%) 2024-09-10T18:25:41.983 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:41 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1239707100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:25:41.983 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:41 smithi028 ceph-mon[39220]: pgmap v161: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 727 op/s; 32720/314952 objects degraded (10.389%) 2024-09-10T18:25:41.983 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:41 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1239707100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:25:41.992 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:42.014 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:42.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:42.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:41 smithi204 ceph-mon[32357]: pgmap v161: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 727 op/s; 32720/314952 objects degraded (10.389%) 2024-09-10T18:25:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:41 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1239707100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:25:42.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:42.135 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:25:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-10T18:25:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-10T18:25:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:25:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:28609 7.a 7:5057c4e8:::benchmark_data_smithi028_77856_object28608:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:25:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:55.715797+0000", 2024-09-10T18:25:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 106.413775511, 2024-09-10T18:25:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14500564599999999, 2024-09-10T18:25:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 28609 2024-09-10T18:25:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.715797+0000", 2024-09-10T18:25:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.715797+0000", 2024-09-10T18:25:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.715796+0000", 2024-09-10T18:25:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:25:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.715819+0000", 2024-09-10T18:25:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3201999999999999e-05 2024-09-10T18:25:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.715827+0000", 2024-09-10T18:25:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6049999999999998e-06 2024-09-10T18:25:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.715832+0000", 2024-09-10T18:25:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.994e-06 2024-09-10T18:25:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.717225+0000", 2024-09-10T18:25:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013928300000000001 2024-09-10T18:25:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.717382+0000", 2024-09-10T18:25:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015649199999999999 2024-09-10T18:25:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:25:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.717535+0000", 2024-09-10T18:25:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000153392 2024-09-10T18:25:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.718954+0000", 2024-09-10T18:25:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001418402 2024-09-10T18:25:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.721010+0000", 2024-09-10T18:25:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002056048 2024-09-10T18:25:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726296+0000", 2024-09-10T18:25:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00528642 2024-09-10T18:25:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726328+0000", 2024-09-10T18:25:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2327999999999999e-05 2024-09-10T18:25:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.860803+0000", 2024-09-10T18:25:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13447430499999999 2024-09-10T18:25:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29831 7.a 7:5aa0b980:::benchmark_data_smithi028_77856_object29830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:25:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.545597+0000", 2024-09-10T18:25:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 105.583975575, 2024-09-10T18:25:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24445233, 2024-09-10T18:25:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29831 2024-09-10T18:25:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545597+0000", 2024-09-10T18:25:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545597+0000", 2024-09-10T18:25:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545595+0000", 2024-09-10T18:25:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:25:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545618+0000", 2024-09-10T18:25:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2614000000000001e-05 2024-09-10T18:25:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545633+0000", 2024-09-10T18:25:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4814000000000001e-05 2024-09-10T18:25:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545639+0000", 2024-09-10T18:25:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9560000000000002e-06 2024-09-10T18:25:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545683+0000", 2024-09-10T18:25:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3893000000000001e-05 2024-09-10T18:25:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545798+0000", 2024-09-10T18:25:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000115179 2024-09-10T18:25:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:25:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545952+0000", 2024-09-10T18:25:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015383399999999999 2024-09-10T18:25:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548658+0000", 2024-09-10T18:25:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027061089999999999 2024-09-10T18:25:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548665+0000", 2024-09-10T18:25:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9929999999999998e-06 2024-09-10T18:25:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789955+0000", 2024-09-10T18:25:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.241290227 2024-09-10T18:25:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790008+0000", 2024-09-10T18:25:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3362999999999997e-05 2024-09-10T18:25:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790049+0000", 2024-09-10T18:25:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1037e-05 2024-09-10T18:25:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29839 7.a 7:508240a2:::benchmark_data_smithi028_77856_object29838:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:25:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.549223+0000", 2024-09-10T18:25:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 105.580349437, 2024-09-10T18:25:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.244164943, 2024-09-10T18:25:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29839 2024-09-10T18:25:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549223+0000", 2024-09-10T18:25:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549223+0000", 2024-09-10T18:25:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549221+0000", 2024-09-10T18:25:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:25:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549254+0000", 2024-09-10T18:25:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2920999999999999e-05 2024-09-10T18:25:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549268+0000", 2024-09-10T18:25:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4418e-05 2024-09-10T18:25:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549273+0000", 2024-09-10T18:25:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.22e-06 2024-09-10T18:25:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549316+0000", 2024-09-10T18:25:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2519999999999999e-05 2024-09-10T18:25:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549937+0000", 2024-09-10T18:25:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00062122000000000002 2024-09-10T18:25:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:25:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.550171+0000", 2024-09-10T18:25:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023401499999999999 2024-09-10T18:25:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.551601+0000", 2024-09-10T18:25:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0014293909999999999 2024-09-10T18:25:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.551669+0000", 2024-09-10T18:25:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8378000000000002e-05 2024-09-10T18:25:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793314+0000", 2024-09-10T18:25:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24164548 2024-09-10T18:25:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793355+0000", 2024-09-10T18:25:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0368999999999999e-05 2024-09-10T18:25:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793388+0000", 2024-09-10T18:25:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3241999999999998e-05 2024-09-10T18:25:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29852 7.2 7:40708469:::benchmark_data_smithi028_77856_object29851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:25:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.555115+0000", 2024-09-10T18:25:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 105.574457349, 2024-09-10T18:25:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.238455156, 2024-09-10T18:25:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29852 2024-09-10T18:25:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555115+0000", 2024-09-10T18:25:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555115+0000", 2024-09-10T18:25:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555113+0000", 2024-09-10T18:25:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:25:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555141+0000", 2024-09-10T18:25:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8665000000000001e-05 2024-09-10T18:25:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555157+0000", 2024-09-10T18:25:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5668999999999999e-05 2024-09-10T18:25:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555163+0000", 2024-09-10T18:25:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5490000000000003e-06 2024-09-10T18:25:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555251+0000", 2024-09-10T18:25:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7559999999999995e-05 2024-09-10T18:25:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555369+0000", 2024-09-10T18:25:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011843 2024-09-10T18:25:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:25:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555505+0000", 2024-09-10T18:25:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000135115 2024-09-10T18:25:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.557478+0000", 2024-09-10T18:25:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019733139999999999 2024-09-10T18:25:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.557928+0000", 2024-09-10T18:25:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00044986800000000002 2024-09-10T18:25:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793508+0000", 2024-09-10T18:25:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23558014999999999 2024-09-10T18:25:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793542+0000", 2024-09-10T18:25:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4235e-05 2024-09-10T18:25:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793570+0000", 2024-09-10T18:25:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8206999999999999e-05 2024-09-10T18:25:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29853 7.3 7:c42b9ada:::benchmark_data_smithi028_77856_object29852:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:25:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.555209+0000", 2024-09-10T18:25:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 105.574363791, 2024-09-10T18:25:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23772209899999999, 2024-09-10T18:25:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29853 2024-09-10T18:25:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555209+0000", 2024-09-10T18:25:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555209+0000", 2024-09-10T18:25:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555208+0000", 2024-09-10T18:25:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:25:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:42.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555230+0000", 2024-09-10T18:25:42.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2101999999999999e-05 2024-09-10T18:25:42.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555240+0000", 2024-09-10T18:25:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8439999999999993e-06 2024-09-10T18:25:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555245+0000", 2024-09-10T18:25:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4190000000000001e-06 2024-09-10T18:25:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555294+0000", 2024-09-10T18:25:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8812e-05 2024-09-10T18:25:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555386+0000", 2024-09-10T18:25:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2119000000000004e-05 2024-09-10T18:25:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,4", 2024-09-10T18:25:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555520+0000", 2024-09-10T18:25:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013431399999999999 2024-09-10T18:25:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.557268+0000", 2024-09-10T18:25:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017474999999999999 2024-09-10T18:25:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.557731+0000", 2024-09-10T18:25:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00046338800000000002 2024-09-10T18:25:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.792820+0000", 2024-09-10T18:25:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23508900299999999 2024-09-10T18:25:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.792864+0000", 2024-09-10T18:25:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3778e-05 2024-09-10T18:25:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.792931+0000", 2024-09-10T18:25:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6889000000000003e-05 2024-09-10T18:25:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29854 7.a 7:5609dd1f:::benchmark_data_smithi028_77856_object29853:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:25:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.555316+0000", 2024-09-10T18:25:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 105.574256633, 2024-09-10T18:25:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.238163033, 2024-09-10T18:25:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29854 2024-09-10T18:25:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555316+0000", 2024-09-10T18:25:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555316+0000", 2024-09-10T18:25:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555314+0000", 2024-09-10T18:25:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:25:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555341+0000", 2024-09-10T18:25:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6951e-05 2024-09-10T18:25:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555353+0000", 2024-09-10T18:25:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1398e-05 2024-09-10T18:25:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555360+0000", 2024-09-10T18:25:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2010000000000002e-06 2024-09-10T18:25:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555397+0000", 2024-09-10T18:25:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7163000000000003e-05 2024-09-10T18:25:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555506+0000", 2024-09-10T18:25:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109188 2024-09-10T18:25:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:25:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555633+0000", 2024-09-10T18:25:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000126391 2024-09-10T18:25:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.557918+0000", 2024-09-10T18:25:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00228583 2024-09-10T18:25:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.557935+0000", 2024-09-10T18:25:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6439e-05 2024-09-10T18:25:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793418+0000", 2024-09-10T18:25:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23548359199999999 2024-09-10T18:25:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793451+0000", 2024-09-10T18:25:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2656e-05 2024-09-10T18:25:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793479+0000", 2024-09-10T18:25:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7838e-05 2024-09-10T18:25:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29858 7.a 7:56ca4896:::benchmark_data_smithi028_77856_object29857:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:25:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.559039+0000", 2024-09-10T18:25:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 105.570534005, 2024-09-10T18:25:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23462707999999999, 2024-09-10T18:25:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29858 2024-09-10T18:25:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559039+0000", 2024-09-10T18:25:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559039+0000", 2024-09-10T18:25:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559036+0000", 2024-09-10T18:25:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-10T18:25:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559060+0000", 2024-09-10T18:25:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3879999999999998e-05 2024-09-10T18:25:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559073+0000", 2024-09-10T18:25:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3339000000000001e-05 2024-09-10T18:25:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559079+0000", 2024-09-10T18:25:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.088e-06 2024-09-10T18:25:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559130+0000", 2024-09-10T18:25:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0688e-05 2024-09-10T18:25:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559243+0000", 2024-09-10T18:25:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011288199999999999 2024-09-10T18:25:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:25:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559381+0000", 2024-09-10T18:25:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013798799999999999 2024-09-10T18:25:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.561064+0000", 2024-09-10T18:25:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001683323 2024-09-10T18:25:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.561094+0000", 2024-09-10T18:25:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0114e-05 2024-09-10T18:25:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793598+0000", 2024-09-10T18:25:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.232503821 2024-09-10T18:25:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793638+0000", 2024-09-10T18:25:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0005000000000001e-05 2024-09-10T18:25:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793666+0000", 2024-09-10T18:25:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7633000000000001e-05 2024-09-10T18:25:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29859 7.a 7:5b51003a:::benchmark_data_smithi028_77856_object29858:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:25:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.559489+0000", 2024-09-10T18:25:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 105.570083135, 2024-09-10T18:25:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23708596400000001, 2024-09-10T18:25:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29859 2024-09-10T18:25:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559489+0000", 2024-09-10T18:25:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559489+0000", 2024-09-10T18:25:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559488+0000", 2024-09-10T18:25:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:25:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559514+0000", 2024-09-10T18:25:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5272000000000001e-05 2024-09-10T18:25:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559526+0000", 2024-09-10T18:25:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.237e-05 2024-09-10T18:25:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559532+0000", 2024-09-10T18:25:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7790000000000001e-06 2024-09-10T18:25:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.560346+0000", 2024-09-10T18:25:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081443200000000002 2024-09-10T18:25:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.560426+0000", 2024-09-10T18:25:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9412000000000005e-05 2024-09-10T18:25:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:25:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.560503+0000", 2024-09-10T18:25:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7825999999999997e-05 2024-09-10T18:25:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.561194+0000", 2024-09-10T18:25:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00069056 2024-09-10T18:25:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.561443+0000", 2024-09-10T18:25:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024952600000000002 2024-09-10T18:25:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.796494+0000", 2024-09-10T18:25:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23505084200000001 2024-09-10T18:25:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.796537+0000", 2024-09-10T18:25:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3089e-05 2024-09-10T18:25:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.796575+0000", 2024-09-10T18:25:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7976999999999999e-05 2024-09-10T18:25:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:121357 7.2 7:454496b5:::benchmark_data_smithi028_77856_object257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:25:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:14.123608+0000", 2024-09-10T18:25:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.005964842000001, 2024-09-10T18:25:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15093262199999999, 2024-09-10T18:25:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 121357 2024-09-10T18:25:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.123608+0000", 2024-09-10T18:25:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.123608+0000", 2024-09-10T18:25:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.123606+0000", 2024-09-10T18:25:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:25:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.123613+0000", 2024-09-10T18:25:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2259999999999999e-06 2024-09-10T18:25:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.123624+0000", 2024-09-10T18:25:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1232e-05 2024-09-10T18:25:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.123629+0000", 2024-09-10T18:25:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4550000000000003e-06 2024-09-10T18:25:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.164453+0000", 2024-09-10T18:25:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.040823267000000003 2024-09-10T18:25:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.164554+0000", 2024-09-10T18:25:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010128000000000001 2024-09-10T18:25:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:25:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.164688+0000", 2024-09-10T18:25:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000133664 2024-09-10T18:25:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.174257+0000", 2024-09-10T18:25:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095690600000000008 2024-09-10T18:25:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.184293+0000", 2024-09-10T18:25:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010035983 2024-09-10T18:25:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.274414+0000", 2024-09-10T18:25:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.090121323000000003 2024-09-10T18:25:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.274501+0000", 2024-09-10T18:25:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7332999999999999e-05 2024-09-10T18:25:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.274540+0000", 2024-09-10T18:25:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8940999999999999e-05 2024-09-10T18:25:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:121682 7.a 7:5cc2bbec:::benchmark_data_smithi028_77856_object582:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:25:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:14.512524+0000", 2024-09-10T18:25:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.617048165, 2024-09-10T18:25:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16768606599999999, 2024-09-10T18:25:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 121682 2024-09-10T18:25:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512524+0000", 2024-09-10T18:25:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512524+0000", 2024-09-10T18:25:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512524+0000", 2024-09-10T18:25:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:25:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512527+0000", 2024-09-10T18:25:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.467e-06 2024-09-10T18:25:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512533+0000", 2024-09-10T18:25:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5679999999999999e-06 2024-09-10T18:25:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512538+0000", 2024-09-10T18:25:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7119999999999999e-06 2024-09-10T18:25:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521276+0000", 2024-09-10T18:25:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087382520000000002 2024-09-10T18:25:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521386+0000", 2024-09-10T18:25:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110188 2024-09-10T18:25:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:25:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521505+0000", 2024-09-10T18:25:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000119021 2024-09-10T18:25:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.530749+0000", 2024-09-10T18:25:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.009244169 2024-09-10T18:25:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.530781+0000", 2024-09-10T18:25:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2119999999999997e-05 2024-09-10T18:25:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680127+0000", 2024-09-10T18:25:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14934560899999999 2024-09-10T18:25:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680182+0000", 2024-09-10T18:25:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5398000000000001e-05 2024-09-10T18:25:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680210+0000", 2024-09-10T18:25:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8191000000000002e-05 2024-09-10T18:25:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:121685 7.a 7:556e904c:::benchmark_data_smithi028_77856_object585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:25:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:14.516970+0000", 2024-09-10T18:25:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.612602434999999, 2024-09-10T18:25:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16365090800000001, 2024-09-10T18:25:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 121685 2024-09-10T18:25:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516970+0000", 2024-09-10T18:25:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516970+0000", 2024-09-10T18:25:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516968+0000", 2024-09-10T18:25:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:25:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516977+0000", 2024-09-10T18:25:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1600000000000004e-06 2024-09-10T18:25:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516989+0000", 2024-09-10T18:25:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2094e-05 2024-09-10T18:25:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516995+0000", 2024-09-10T18:25:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1199999999999999e-06 2024-09-10T18:25:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.524202+0000", 2024-09-10T18:25:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0072076129999999999 2024-09-10T18:25:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.524301+0000", 2024-09-10T18:25:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8858999999999997e-05 2024-09-10T18:25:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:25:42.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.524413+0000", 2024-09-10T18:25:42.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011134899999999999 2024-09-10T18:25:42.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.530754+0000", 2024-09-10T18:25:42.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006341218 2024-09-10T18:25:42.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.533574+0000", 2024-09-10T18:25:42.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002820473 2024-09-10T18:25:42.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680527+0000", 2024-09-10T18:25:42.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.146952895 2024-09-10T18:25:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680579+0000", 2024-09-10T18:25:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1777999999999998e-05 2024-09-10T18:25:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680621+0000", 2024-09-10T18:25:42.211 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:42.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1953999999999998e-05 2024-09-10T18:25:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:121686 7.2 7:471809bb:::benchmark_data_smithi028_77856_object586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:25:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:14.518358+0000", 2024-09-10T18:25:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.611214814, 2024-09-10T18:25:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16205149299999999, 2024-09-10T18:25:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 121686 2024-09-10T18:25:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518358+0000", 2024-09-10T18:25:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518358+0000", 2024-09-10T18:25:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518356+0000", 2024-09-10T18:25:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:25:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518362+0000", 2024-09-10T18:25:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8889999999999999e-06 2024-09-10T18:25:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518373+0000", 2024-09-10T18:25:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0375e-05 2024-09-10T18:25:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518380+0000", 2024-09-10T18:25:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.092e-06 2024-09-10T18:25:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.526805+0000", 2024-09-10T18:25:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0084254840000000004 2024-09-10T18:25:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.526929+0000", 2024-09-10T18:25:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012368700000000001 2024-09-10T18:25:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:25:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.527045+0000", 2024-09-10T18:25:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116277 2024-09-10T18:25:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.533582+0000", 2024-09-10T18:25:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0065365359999999999 2024-09-10T18:25:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.542247+0000", 2024-09-10T18:25:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0086657750000000006 2024-09-10T18:25:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680327+0000", 2024-09-10T18:25:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13807940899999999 2024-09-10T18:25:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680372+0000", 2024-09-10T18:25:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5173000000000003e-05 2024-09-10T18:25:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680409+0000", 2024-09-10T18:25:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7194000000000003e-05 2024-09-10T18:25:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:121693 7.2 7:4f645b06:::benchmark_data_smithi028_77856_object593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:25:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:14.521071+0000", 2024-09-10T18:25:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.608501839999999, 2024-09-10T18:25:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.159677503, 2024-09-10T18:25:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 121693 2024-09-10T18:25:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521071+0000", 2024-09-10T18:25:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521071+0000", 2024-09-10T18:25:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521070+0000", 2024-09-10T18:25:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:25:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521075+0000", 2024-09-10T18:25:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2220000000000001e-06 2024-09-10T18:25:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521083+0000", 2024-09-10T18:25:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6370000000000005e-06 2024-09-10T18:25:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521088+0000", 2024-09-10T18:25:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8930000000000004e-06 2024-09-10T18:25:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.530803+0000", 2024-09-10T18:25:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0097147029999999999 2024-09-10T18:25:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.530925+0000", 2024-09-10T18:25:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122376 2024-09-10T18:25:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:25:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.531026+0000", 2024-09-10T18:25:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010061 2024-09-10T18:25:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.534573+0000", 2024-09-10T18:25:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035470860000000001 2024-09-10T18:25:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.542273+0000", 2024-09-10T18:25:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076997999999999997 2024-09-10T18:25:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680669+0000", 2024-09-10T18:25:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13839658699999999 2024-09-10T18:25:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680712+0000", 2024-09-10T18:25:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2979999999999998e-05 2024-09-10T18:25:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680748+0000", 2024-09-10T18:25:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5775999999999998e-05 2024-09-10T18:25:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:121705 7.a 7:5e5a4369:::benchmark_data_smithi028_77856_object605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:25:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:14.535020+0000", 2024-09-10T18:25:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.594552461999999, 2024-09-10T18:25:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14584269799999999, 2024-09-10T18:25:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 121705 2024-09-10T18:25:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.535020+0000", 2024-09-10T18:25:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.535020+0000", 2024-09-10T18:25:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.535018+0000", 2024-09-10T18:25:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:25:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.535025+0000", 2024-09-10T18:25:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6390000000000003e-06 2024-09-10T18:25:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.535035+0000", 2024-09-10T18:25:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9080000000000007e-06 2024-09-10T18:25:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.535040+0000", 2024-09-10T18:25:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4129999999999997e-06 2024-09-10T18:25:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.542296+0000", 2024-09-10T18:25:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0072563039999999999 2024-09-10T18:25:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.542419+0000", 2024-09-10T18:25:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012223099999999999 2024-09-10T18:25:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:25:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.542539+0000", 2024-09-10T18:25:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012066300000000001 2024-09-10T18:25:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680220+0000", 2024-09-10T18:25:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.137680412 2024-09-10T18:25:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680285+0000", 2024-09-10T18:25:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5134000000000004e-05 2024-09-10T18:25:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680790+0000", 2024-09-10T18:25:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00050506200000000002 2024-09-10T18:25:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680830+0000", 2024-09-10T18:25:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9814000000000002e-05 2024-09-10T18:25:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680863+0000", 2024-09-10T18:25:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2977e-05 2024-09-10T18:25:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:123139 7.2 7:41b385ad:::benchmark_data_smithi028_77856_object2039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:25:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:16.257335+0000", 2024-09-10T18:25:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 25.872237201000001, 2024-09-10T18:25:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14337841500000001, 2024-09-10T18:25:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 123139 2024-09-10T18:25:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:16.257335+0000", 2024-09-10T18:25:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:16.257335+0000", 2024-09-10T18:25:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:16.257333+0000", 2024-09-10T18:25:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:25:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:16.257340+0000", 2024-09-10T18:25:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.109e-06 2024-09-10T18:25:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:16.257351+0000", 2024-09-10T18:25:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0954999999999999e-05 2024-09-10T18:25:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:16.257356+0000", 2024-09-10T18:25:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1989999999999997e-06 2024-09-10T18:25:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:16.298402+0000", 2024-09-10T18:25:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.041045524 2024-09-10T18:25:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:16.298535+0000", 2024-09-10T18:25:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013261499999999999 2024-09-10T18:25:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:25:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:16.298656+0000", 2024-09-10T18:25:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012154699999999999 2024-09-10T18:25:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:16.307416+0000", 2024-09-10T18:25:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087603149999999994 2024-09-10T18:25:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:16.319022+0000", 2024-09-10T18:25:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011605189 2024-09-10T18:25:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:16.400628+0000", 2024-09-10T18:25:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.081606149000000003 2024-09-10T18:25:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:16.400676+0000", 2024-09-10T18:25:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8099000000000001e-05 2024-09-10T18:25:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:16.400714+0000", 2024-09-10T18:25:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.786e-05 2024-09-10T18:25:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:124429 7.2 7:4838b573:::benchmark_data_smithi028_77856_object3329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:25:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:18.054003+0000", 2024-09-10T18:25:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 24.075569595000001, 2024-09-10T18:25:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.166994589, 2024-09-10T18:25:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 124429 2024-09-10T18:25:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054003+0000", 2024-09-10T18:25:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054003+0000", 2024-09-10T18:25:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054002+0000", 2024-09-10T18:25:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:25:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054005+0000", 2024-09-10T18:25:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4559999999999999e-06 2024-09-10T18:25:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054009+0000", 2024-09-10T18:25:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3669999999999999e-06 2024-09-10T18:25:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054013+0000", 2024-09-10T18:25:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.343e-06 2024-09-10T18:25:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.126700+0000", 2024-09-10T18:25:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.072687286000000004 2024-09-10T18:25:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.126814+0000", 2024-09-10T18:25:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114442 2024-09-10T18:25:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:25:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.126963+0000", 2024-09-10T18:25:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014888499999999999 2024-09-10T18:25:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.140882+0000", 2024-09-10T18:25:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013919124 2024-09-10T18:25:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.141158+0000", 2024-09-10T18:25:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000276042 2024-09-10T18:25:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.220924+0000", 2024-09-10T18:25:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.079766112 2024-09-10T18:25:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.220962+0000", 2024-09-10T18:25:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7707999999999999e-05 2024-09-10T18:25:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.220998+0000", 2024-09-10T18:25:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5349999999999999e-05 2024-09-10T18:25:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:124435 7.2 7:409dca55:::benchmark_data_smithi028_77856_object3335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:25:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:18.066838+0000", 2024-09-10T18:25:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 24.062734751000001, 2024-09-10T18:25:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.154273204, 2024-09-10T18:25:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 124435 2024-09-10T18:25:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.066838+0000", 2024-09-10T18:25:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.066838+0000", 2024-09-10T18:25:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.066835+0000", 2024-09-10T18:25:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:25:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.066844+0000", 2024-09-10T18:25:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5019999999999992e-06 2024-09-10T18:25:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.066856+0000", 2024-09-10T18:25:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1878e-05 2024-09-10T18:25:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.066863+0000", 2024-09-10T18:25:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.525e-06 2024-09-10T18:25:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.130593+0000", 2024-09-10T18:25:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.063729443999999996 2024-09-10T18:25:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.130674+0000", 2024-09-10T18:25:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0990999999999998e-05 2024-09-10T18:25:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:25:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.130810+0000", 2024-09-10T18:25:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013623800000000001 2024-09-10T18:25:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.140887+0000", 2024-09-10T18:25:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010077579 2024-09-10T18:25:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.141623+0000", 2024-09-10T18:25:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00073552000000000003 2024-09-10T18:25:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.221053+0000", 2024-09-10T18:25:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.079429581999999999 2024-09-10T18:25:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.221083+0000", 2024-09-10T18:25:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0586999999999997e-05 2024-09-10T18:25:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.221111+0000", 2024-09-10T18:25:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7912999999999999e-05 2024-09-10T18:25:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:124437 7.a 7:5b85f91c:::benchmark_data_smithi028_77856_object3337:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:25:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:18.074202+0000", 2024-09-10T18:25:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 24.05537073, 2024-09-10T18:25:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14705718700000001, 2024-09-10T18:25:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 124437 2024-09-10T18:25:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.074202+0000", 2024-09-10T18:25:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.074202+0000", 2024-09-10T18:25:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.074200+0000", 2024-09-10T18:25:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:25:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.074207+0000", 2024-09-10T18:25:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2030000000000003e-06 2024-09-10T18:25:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.074219+0000", 2024-09-10T18:25:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.219e-05 2024-09-10T18:25:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.074225+0000", 2024-09-10T18:25:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1709999999999999e-06 2024-09-10T18:25:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.133264+0000", 2024-09-10T18:25:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.059038395 2024-09-10T18:25:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.133390+0000", 2024-09-10T18:25:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012608899999999999 2024-09-10T18:25:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:25:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.133512+0000", 2024-09-10T18:25:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121685 2024-09-10T18:25:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.140822+0000", 2024-09-10T18:25:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073108180000000002 2024-09-10T18:25:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.140892+0000", 2024-09-10T18:25:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9956999999999994e-05 2024-09-10T18:25:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.221197+0000", 2024-09-10T18:25:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.080304679000000004 2024-09-10T18:25:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.221229+0000", 2024-09-10T18:25:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2218999999999998e-05 2024-09-10T18:25:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.221259+0000", 2024-09-10T18:25:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9794999999999998e-05 2024-09-10T18:25:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:138134 7.2 7:43f8b39b:::benchmark_data_smithi028_77856_object17034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:25:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:41.224966+0000", 2024-09-10T18:25:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.90460680100000002, 2024-09-10T18:25:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.152545974, 2024-09-10T18:25:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 138134 2024-09-10T18:25:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.224966+0000", 2024-09-10T18:25:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.224966+0000", 2024-09-10T18:25:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.224964+0000", 2024-09-10T18:25:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:25:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.224971+0000", 2024-09-10T18:25:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9739999999999996e-06 2024-09-10T18:25:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.224980+0000", 2024-09-10T18:25:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0939999999999997e-06 2024-09-10T18:25:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.224985+0000", 2024-09-10T18:25:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.57e-06 2024-09-10T18:25:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.335800+0000", 2024-09-10T18:25:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.11081452 2024-09-10T18:25:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.335922+0000", 2024-09-10T18:25:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012243200000000001 2024-09-10T18:25:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:25:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.336058+0000", 2024-09-10T18:25:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013559100000000001 2024-09-10T18:25:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.350044+0000", 2024-09-10T18:25:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013986343 2024-09-10T18:25:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.353752+0000", 2024-09-10T18:25:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037080559999999999 2024-09-10T18:25:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.375979+0000", 2024-09-10T18:25:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022226844999999999 2024-09-10T18:25:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.377471+0000", 2024-09-10T18:25:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001491491 2024-09-10T18:25:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.377512+0000", 2024-09-10T18:25:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0948999999999997e-05 2024-09-10T18:25:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:138382 7.a 7:5f41d747:::benchmark_data_smithi028_77856_object17282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:25:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:41.628063+0000", 2024-09-10T18:25:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.50150914700000004, 2024-09-10T18:25:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14914930700000001, 2024-09-10T18:25:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:25:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:25:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:25:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:25:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:25:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 138382 2024-09-10T18:25:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:25:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:25:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.628063+0000", 2024-09-10T18:25:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:25:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.628063+0000", 2024-09-10T18:25:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:25:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:25:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.628062+0000", 2024-09-10T18:25:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:25:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:25:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.628069+0000", 2024-09-10T18:25:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1099999999999997e-06 2024-09-10T18:25:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:25:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.628078+0000", 2024-09-10T18:25:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3850000000000008e-06 2024-09-10T18:25:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:25:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.628083+0000", 2024-09-10T18:25:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2349999999999999e-06 2024-09-10T18:25:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:25:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.681773+0000", 2024-09-10T18:25:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.053689751000000001 2024-09-10T18:25:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:25:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.681889+0000", 2024-09-10T18:25:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000115821 2024-09-10T18:25:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:25:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.682003+0000", 2024-09-10T18:25:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114307 2024-09-10T18:25:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.684489+0000", 2024-09-10T18:25:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024851919999999998 2024-09-10T18:25:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:25:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.696630+0000", 2024-09-10T18:25:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012141216 2024-09-10T18:25:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:25:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.777125+0000", 2024-09-10T18:25:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.080495400999999994 2024-09-10T18:25:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:25:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.777173+0000", 2024-09-10T18:25:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7981e-05 2024-09-10T18:25:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:42.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:25:42.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:41.777213+0000", 2024-09-10T18:25:42.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9591000000000001e-05 2024-09-10T18:25:42.279 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.279 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.279 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.279 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:42.279 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:42.279 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:25:42.313 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:42.336 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:42.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:42.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:25:42.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:42.494 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:42.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:42.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:42.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:25:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:25:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:25:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:42.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:42.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:42.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:42.786 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:42.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:42.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:42.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:42.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:42.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:43.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:43.054 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:43.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:43.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:43.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:43.178 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:43.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:43.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:43.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:43.321 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:43.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:43.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:43.423 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:43.444 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:43.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:43.446 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T18:25:43.446 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T18:25:43.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:43.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:43.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:43.695 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:43.718 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:43.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:43.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:43.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:43.841 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:43.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:43.943 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:43.964 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:43.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:43.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:44.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:43 smithi028 ceph-mon[39220]: pgmap v162: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 656 op/s; 32720/314952 objects degraded (10.389%) 2024-09-10T18:25:44.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:43 smithi028 ceph-mon[29291]: pgmap v162: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 656 op/s; 32720/314952 objects degraded (10.389%) 2024-09-10T18:25:44.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:44.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:44.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:44.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:44.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:44.239 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:44.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:44.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:44.341 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:43 smithi204 ceph-mon[32357]: pgmap v162: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 656 op/s; 32720/314952 objects degraded (10.389%) 2024-09-10T18:25:44.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:25:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:25:44.365 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:44.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:44.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:44.466 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:44.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:44.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:44.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:44.590 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:44.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:44.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:44.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:44.616 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-10T18:25:44.714 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:44.737 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:44.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:44.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:44.839 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:44.863 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:44.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:44.942 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:25:44.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:45.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:45.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:45.065 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:44 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:45.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:45.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:45.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:45.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:44 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:45.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:44 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:45.289 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:45.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:45.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:45.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:45.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:45.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:45.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:25:45.532 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:45.557 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:45.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:45.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:45.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:45.764 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:45.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:45.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:45.866 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:45.888 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:45.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:45.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:45.950 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:25:45.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:46.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:45 smithi028 ceph-mon[29291]: pgmap v163: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 417 op/s; 32543/313296 objects degraded (10.387%) 2024-09-10T18:25:46.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:45 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 32543/313296 objects degraded (10.387%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:46.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:45 smithi028 ceph-mon[39220]: pgmap v163: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 417 op/s; 32543/313296 objects degraded (10.387%) 2024-09-10T18:25:46.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:45 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 32543/313296 objects degraded (10.387%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:46.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:46.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:46.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:46.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:46.191 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:45 smithi204 ceph-mon[32357]: pgmap v163: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 417 op/s; 32543/313296 objects degraded (10.387%) 2024-09-10T18:25:46.191 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:45 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 32543/313296 objects degraded (10.387%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:46.203 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:46.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:46.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:46.304 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:46.330 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:46.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:46.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:46.432 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:46.453 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:46.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:46.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:46.554 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-09-10T18:25:46.554 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:25:46.555 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-10T18:25:46.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T18:25:46.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:46.578 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:46.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:46.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:46.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:46.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:46.745 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:25:46.745 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:25:46.745 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-10T18:25:46.745 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:25:46.822 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:46.845 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:46.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:46.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:46.893 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:25:46.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:46.968 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:46.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:47.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:47.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:47.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:47.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:47.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:46 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/672860335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:25:47.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:46 smithi204 ceph-mon[32357]: from='client.34811 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:25:47.194 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:47.220 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:47.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:47.251 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:46 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/672860335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:25:47.251 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:46 smithi028 ceph-mon[39220]: from='client.34811 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:25:47.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:46 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/672860335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:25:47.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:46 smithi028 ceph-mon[29291]: from='client.34811 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:25:47.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:47.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:47.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:47.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:47.489 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:47.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:47.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:47.542 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-10T18:25:47.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:47.633 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:47.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:47.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:47.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:47.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:47.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:47.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:47.912 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:25:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:47.937 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:47.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:48.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:48.080 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:48.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:48.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:48.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:47 smithi204 ceph-mon[32357]: pgmap v164: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 589 op/s; 31812/306258 objects degraded (10.387%) 2024-09-10T18:25:48.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:47 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:25:48.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:47 smithi204 ceph-mon[32357]: from='client.34811 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:25:48.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:47 smithi204 ceph-mon[32357]: osdmap e128: 8 total, 7 up, 8 in 2024-09-10T18:25:48.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:48.215 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:48.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:48.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:47 smithi028 ceph-mon[39220]: pgmap v164: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 589 op/s; 31812/306258 objects degraded (10.387%) 2024-09-10T18:25:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:47 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:25:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:47 smithi028 ceph-mon[39220]: from='client.34811 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:25:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:47 smithi028 ceph-mon[39220]: osdmap e128: 8 total, 7 up, 8 in 2024-09-10T18:25:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:47 smithi028 ceph-mon[29291]: pgmap v164: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 589 op/s; 31812/306258 objects degraded (10.387%) 2024-09-10T18:25:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:47 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:25:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:47 smithi028 ceph-mon[29291]: from='client.34811 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:25:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:47 smithi028 ceph-mon[29291]: osdmap e128: 8 total, 7 up, 8 in 2024-09-10T18:25:48.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:48.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:48.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:48.484 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:48.510 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:48.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:48.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:48.612 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:48.635 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:48.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:48.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:48.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:48.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:48.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:48.837 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:25:48.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:48.891 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:25:48.891 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:25:48.933 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:48.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:48.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:48.942 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:25:48.942 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:25:48.976 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:25:48.976 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:25:49.018 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:25:49.018 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:25:49.035 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:49.050 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:25:49.050 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:25:49.079 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:49.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:49.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:49.084 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:25:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:25:49.085 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-10T18:25:49.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:48 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1821330916' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:25:49.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:48 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1821330916' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:25:49.117 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:48 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1821330916' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:25:49.117 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:25:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:25:49.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:49.217 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:49.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:49.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:49.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:49.358 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:49.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:49.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:49.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:49.485 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:49.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:49.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:49.587 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:49.626 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:49.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:49.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:49.772 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:49.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:49.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:49.775 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:25:49.775 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:25:49.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:49.917 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:49.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:49.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:50.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:50.044 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:50.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:50.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:50.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:49 smithi204 ceph-mon[32357]: pgmap v166: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 568 op/s; 31812/306258 objects degraded (10.387%) 2024-09-10T18:25:50.147 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:50.169 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:50.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:50.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:50.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:49 smithi028 ceph-mon[29291]: pgmap v166: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 568 op/s; 31812/306258 objects degraded (10.387%) 2024-09-10T18:25:50.218 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:49 smithi028 ceph-mon[39220]: pgmap v166: 148 pgs: 3 active+clean+scrubbing+deep, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 568 op/s; 31812/306258 objects degraded (10.387%) 2024-09-10T18:25:50.258 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T18:25:50.259 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T18:25:50.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:50.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:50.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:50.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:50.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:25:50.396 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:50.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:50.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:50.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:50.561 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:50.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:50.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:50.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:50.702 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:50.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:50.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:50.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-10T18:25:50.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:50.847 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:50.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:50.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:50.949 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:50.976 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:50.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:51.012 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:25:51.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:50 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 31812/306258 objects degraded (10.387%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:51.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:50 smithi028 ceph-mon[29291]: osdmap e129: 8 total, 7 up, 8 in 2024-09-10T18:25:51.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:50 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 31812/306258 objects degraded (10.387%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:51.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:50 smithi028 ceph-mon[39220]: osdmap e129: 8 total, 7 up, 8 in 2024-09-10T18:25:51.077 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:51.101 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:51.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:51.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:50 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 31812/306258 objects degraded (10.387%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:51.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:50 smithi204 ceph-mon[32357]: osdmap e129: 8 total, 7 up, 8 in 2024-09-10T18:25:51.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:51.216 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:25:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:51.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:51.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:51.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:51.375 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:51.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:51.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:51.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:51.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:51.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:51.614 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:51.638 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:51.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:51.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:51.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:51.748 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:25:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:25:51] "GET /metrics HTTP/1.1" 200 223776 "" "Prometheus/2.43.0" 2024-09-10T18:25:51.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:51.778 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:51.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:51.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:25:51.911 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:25:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 129, 2024-09-10T18:25:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "5eababb2-6fa0-11ef-bcea-c7b262605968", 2024-09-10T18:25:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-10T18:16:20.680373+0000", 2024-09-10T18:25:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-10T18:25:50.596243+0000", 2024-09-10T18:25:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-09-10T18:23:27.783937+0000", 2024-09-10T18:25:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-09-10T18:21:02.114621+0000", 2024-09-10T18:25:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T18:25:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5806080, 2024-09-10T18:25:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-09-10T18:25:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "nodeep-scrub", 2024-09-10T18:25:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "noscrub", 2024-09-10T18:25:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-09-10T18:25:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-09-10T18:25:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-09-10T18:25:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-09-10T18:25:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:25:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 18, 2024-09-10T18:25:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T18:25:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T18:25:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T18:25:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-09-10T18:25:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 7, 2024-09-10T18:25:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-09-10T18:25:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-09-10T18:25:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-09-10T18:25:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "quincy", 2024-09-10T18:25:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-09-10T18:25:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-09-10T18:25:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-09-10T18:25:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:19:24.013536+0000", 2024-09-10T18:25:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:25:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:25:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:25:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:25:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:25:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:25:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:25:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:25:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:25:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:25:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:25:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:25:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 1, 2024-09-10T18:25:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 1, 2024-09-10T18:25:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 1, 2024-09-10T18:25:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 1, 2024-09-10T18:25:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 1, 2024-09-10T18:25:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:25:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:25:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:25:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:25:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:25:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:25:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:25:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "22", 2024-09-10T18:25:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:25:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:25:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:25:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:25:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:25:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:25:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:25:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:25:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:25:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:25:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:25:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:25:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:25:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:25:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:25:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:25:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:25:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:25:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:25:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:25:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:25:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:25:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:25:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:25:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:25:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:25:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:25:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:25:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:25:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:25:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:25:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:25:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:25:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:25:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:25:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:25:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:25:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-10T18:25:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-09-10T18:25:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-09-10T18:25:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:25:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-09-10T18:25:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-09-10T18:25:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-09-10T18:25:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:30.065209+0000", 2024-09-10T18:25:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-09-10T18:25:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T18:25:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:25:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:25:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:25:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:25:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:25:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:25:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:25:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:25:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:25:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:25:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-09-10T18:25:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-09-10T18:25:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-09-10T18:25:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-09-10T18:25:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-09-10T18:25:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:25:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:25:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:25:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:25:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:25:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:25:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:25:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "62", 2024-09-10T18:25:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:25:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:25:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:25:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:25:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:25:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 3, 2024-09-10T18:25:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 62, 2024-09-10T18:25:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:25:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:25:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:25:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:25:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:25:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:25:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:25:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:25:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:25:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:25:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:25:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:25:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:25:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:25:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:25:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:25:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:25:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:25:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:25:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:25:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:25:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:25:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:25:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:25:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:25:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:25:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:25:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:25:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:25:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:25:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:25:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:25:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-09-10T18:25:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-09-10T18:25:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-09-10T18:25:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:30.506441+0000", 2024-09-10T18:25:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:25:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:25:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:25:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:25:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:25:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:25:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:25:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:25:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:25:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:25:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:25:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:25:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:25:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:25:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:25:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:25:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:25:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:25:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:25:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:25:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:25:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:25:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:25:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:25:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "59", 2024-09-10T18:25:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:25:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:25:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:25:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:25:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:25:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:25:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:25:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:25:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:25:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:25:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:25:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:25:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:25:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:25:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:25:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:25:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:25:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:25:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:25:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:25:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:25:51.928 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:25:51.928 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:25:51.928 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:25:51.928 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:25:51.928 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:25:51.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:25:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:25:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:25:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:25:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:25:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:25:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:25:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:25:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:25:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:25:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:25:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:25:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:25:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:25:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-09-10T18:25:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-09-10T18:25:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:32.008130+0000", 2024-09-10T18:25:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:25:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:25:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:25:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:25:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:25:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:25:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:25:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:25:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:25:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:25:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:25:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:25:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:25:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:25:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:25:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:25:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:25:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:25:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:25:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:25:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:25:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:25:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:25:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:25:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "61", 2024-09-10T18:25:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:25:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:25:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:25:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:25:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:25:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:25:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:25:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:25:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:25:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:25:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:25:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:25:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:25:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:25:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:25:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:25:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:25:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:25:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:25:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:25:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:25:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:25:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:25:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:25:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:25:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:25:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:25:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:25:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:25:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:25:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:25:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:25:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:25:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:25:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:25:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:25:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:25:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:25:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:25:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:25:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-09-10T18:25:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-09-10T18:25:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:34.060571+0000", 2024-09-10T18:25:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:25:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:25:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:25:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:25:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:25:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:25:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:25:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:25:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:25:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:25:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:25:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:25:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:25:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:25:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:25:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:25:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:25:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:25:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:25:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:25:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:25:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:25:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:25:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:25:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "63", 2024-09-10T18:25:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:25:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:25:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:25:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:25:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:25:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:25:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:25:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:25:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:25:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:25:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:25:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:25:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:25:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:25:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:25:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:25:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:25:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:25:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:25:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:25:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:25:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:25:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:25:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:25:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:25:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:25:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:25:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:25:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:25:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:25:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:25:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:25:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:25:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:25:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:25:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:25:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:25:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:25:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:25:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:25:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-09-10T18:25:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-09-10T18:25:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:36.127330+0000", 2024-09-10T18:25:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:25:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:25:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:25:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:25:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:25:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:25:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:25:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:25:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:25:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:25:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:25:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:25:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:25:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:25:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:25:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:25:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:25:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:25:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:25:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:25:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:25:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:25:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:25:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:25:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "65", 2024-09-10T18:25:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:25:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:25:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:25:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:25:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:25:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:25:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:25:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:25:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:25:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:25:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:25:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:25:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:25:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:25:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:25:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:25:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:25:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:25:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:25:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:25:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:25:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:25:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:25:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:25:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:25:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:25:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:25:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:25:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:25:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:25:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:25:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:25:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:25:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:25:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:25:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:25:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:25:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-10T18:25:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-09-10T18:25:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:25:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:25:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 7, 2024-09-10T18:25:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "unique_pool_0", 2024-09-10T18:25:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:23:25.513798+0000", 2024-09-10T18:25:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:25:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:25:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:25:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:25:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:25:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:25:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:25:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:25:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:25:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:25:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:25:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:25:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 16, 2024-09-10T18:25:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 16, 2024-09-10T18:25:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 16, 2024-09-10T18:25:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 16, 2024-09-10T18:25:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 16, 2024-09-10T18:25:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:25:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:25:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:25:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:25:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:25:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:25:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:25:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "97", 2024-09-10T18:25:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:25:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:25:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:25:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:25:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:25:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:25:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:25:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:25:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:25:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:25:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:25:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:25:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:25:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:25:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:25:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:25:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:25:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:25:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:25:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:25:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:25:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:25:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:25:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:25:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:25:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:25:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:25:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:25:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:25:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:25:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:25:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:25:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:25:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:25:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:25:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:25:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:25:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:25:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:25:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-09-10T18:25:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:25:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:25:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-10T18:25:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "f1c834d9-252a-4077-b25e-169181d70251", 2024-09-10T18:25:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:25:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:25:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:25:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:25:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:25:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:25:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-09-10T18:25:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 95, 2024-09-10T18:25:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:25:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:25:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:25:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-09-10T18:25:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:25:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:25:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-09-10T18:25:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:25:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:25:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-09-10T18:25:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:25:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:25:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-09-10T18:25:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:25:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/3188372351", 2024-09-10T18:25:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/3188372351", 2024-09-10T18:25:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/3188372351", 2024-09-10T18:25:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/3188372351", 2024-09-10T18:25:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:25:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:25:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:25:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-10T18:25:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "81f1855d-b946-4ca3-b246-38977b9e20fd", 2024-09-10T18:25:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:25:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:25:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:25:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:25:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:25:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:25:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-10T18:25:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 95, 2024-09-10T18:25:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:25:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:25:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:25:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-09-10T18:25:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:25:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:25:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-09-10T18:25:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:25:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:25:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-09-10T18:25:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:25:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:25:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-09-10T18:25:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:25:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1369912678", 2024-09-10T18:25:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1369912678", 2024-09-10T18:25:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1369912678", 2024-09-10T18:25:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1369912678", 2024-09-10T18:25:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:25:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:25:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:25:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-10T18:25:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "2ad55d2a-36fa-4756-bf22-58d4b049486e", 2024-09-10T18:25:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:25:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:25:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:25:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:25:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:25:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:25:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-09-10T18:25:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 95, 2024-09-10T18:25:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:25:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:25:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:25:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-09-10T18:25:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:25:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:25:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-09-10T18:25:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:25:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:25:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-09-10T18:25:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:25:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:25:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-09-10T18:25:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:25:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/1243406003", 2024-09-10T18:25:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/1243406003", 2024-09-10T18:25:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/1243406003", 2024-09-10T18:25:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1243406003", 2024-09-10T18:25:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:25:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:25:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:25:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:51.965 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:51.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-10T18:25:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "242729a5-a5e8-44d5-97c8-14d14e970bff", 2024-09-10T18:25:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:25:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:25:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:25:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:25:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:25:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:25:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-09-10T18:25:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 95, 2024-09-10T18:25:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:25:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:25:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:25:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-09-10T18:25:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:25:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:25:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-09-10T18:25:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:25:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:25:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-09-10T18:25:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:25:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:25:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-09-10T18:25:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:25:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/67703283", 2024-09-10T18:25:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/67703283", 2024-09-10T18:25:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/67703283", 2024-09-10T18:25:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/67703283", 2024-09-10T18:25:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:25:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:25:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:25:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-10T18:25:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "6e38b8ef-e6ed-493f-8eca-755354754787", 2024-09-10T18:25:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:25:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:25:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:25:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:25:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:25:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:25:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 33, 2024-09-10T18:25:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 95, 2024-09-10T18:25:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:25:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:25:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:25:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6800", 2024-09-10T18:25:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:25:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:25:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6801", 2024-09-10T18:25:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:25:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:25:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6803", 2024-09-10T18:25:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:25:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:25:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6802", 2024-09-10T18:25:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:25:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6800/1619669076", 2024-09-10T18:25:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6801/1619669076", 2024-09-10T18:25:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6803/1619669076", 2024-09-10T18:25:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6802/1619669076", 2024-09-10T18:25:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:25:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:25:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:25:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-10T18:25:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "a92e0441-66c9-498e-a0ac-6465acdbb1f5", 2024-09-10T18:25:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:25:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:25:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:25:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:25:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:25:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:25:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 39, 2024-09-10T18:25:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 95, 2024-09-10T18:25:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:25:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:25:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:25:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6804", 2024-09-10T18:25:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:25:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:25:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6805", 2024-09-10T18:25:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:25:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:25:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6807", 2024-09-10T18:25:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:25:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:25:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6806", 2024-09-10T18:25:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:25:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6804/2717802880", 2024-09-10T18:25:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6805/2717802880", 2024-09-10T18:25:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6807/2717802880", 2024-09-10T18:25:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6806/2717802880", 2024-09-10T18:25:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:25:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:25:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:25:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-09-10T18:25:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "12b02875-6653-45ed-8d2b-b4d33bd88525", 2024-09-10T18:25:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 0, 2024-09-10T18:25:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:25:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:25:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:25:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:25:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:25:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 46, 2024-09-10T18:25:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 93, 2024-09-10T18:25:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 95, 2024-09-10T18:25:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:25:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:25:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6808", 2024-09-10T18:25:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3377682607 2024-09-10T18:25:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:25:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6809", 2024-09-10T18:25:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3377682607 2024-09-10T18:25:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:25:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6811", 2024-09-10T18:25:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3377682607 2024-09-10T18:25:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:25:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6810", 2024-09-10T18:25:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3377682607 2024-09-10T18:25:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6808/3377682607", 2024-09-10T18:25:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6809/3377682607", 2024-09-10T18:25:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6811/3377682607", 2024-09-10T18:25:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6810/3377682607", 2024-09-10T18:25:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:25:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: "exists" 2024-09-10T18:25:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-09-10T18:25:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "0c4f7aa9-ff95-4192-a146-ca576eee907f", 2024-09-10T18:25:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:25:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:25:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:25:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:25:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:25:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:25:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 54, 2024-09-10T18:25:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 95, 2024-09-10T18:25:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:25:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:25:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:25:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6812", 2024-09-10T18:25:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:25:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:25:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6813", 2024-09-10T18:25:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:25:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:25:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6815", 2024-09-10T18:25:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:25:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:25:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:25:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:25:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6814", 2024-09-10T18:25:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:25:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6812/3511791879", 2024-09-10T18:25:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6813/3511791879", 2024-09-10T18:25:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6815/3511791879", 2024-09-10T18:25:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6814/3511791879", 2024-09-10T18:25:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:25:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:25:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:25:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:25:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-09-10T18:25:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-10T18:25:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:25:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:25:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:25:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:25:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:25:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:25:37.226997+0000", 2024-09-10T18:25:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:25:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-10T18:25:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:25:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:25:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:25:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:25:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:25:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:24:57.789498+0000", 2024-09-10T18:25:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:25:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-10T18:25:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:25:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:25:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:25:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:25:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:25:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:25:29.443786+0000", 2024-09-10T18:25:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:25:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-10T18:25:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:25:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:25:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:25:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:25:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:25:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:25:36.450533+0000", 2024-09-10T18:25:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:25:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-10T18:25:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:25:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:25:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:25:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:25:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:25:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:25:07.585348+0000", 2024-09-10T18:25:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:25:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-10T18:25:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:25:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:25:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:25:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:25:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:25:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:25:26.538402+0000", 2024-09-10T18:25:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:25:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-09-10T18:25:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:25:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:25:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:25:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:25:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:25:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:23:21.545352+0000", 2024-09-10T18:25:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 94 2024-09-10T18:25:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-09-10T18:25:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:25:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:25:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:25:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:25:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:25:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:25:08.449074+0000", 2024-09-10T18:25:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:25:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:25:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [], 2024-09-10T18:25:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-09-10T18:25:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.4", 2024-09-10T18:25:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:25:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:25:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-09-10T18:25:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.a", 2024-09-10T18:25:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:25:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:25:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-09-10T18:25:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1f", 2024-09-10T18:25:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:25:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:25:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:25:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-09-10T18:25:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:25:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:25:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-09-10T18:25:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-09-10T18:25:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-09-10T18:25:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1745574083": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:25:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3866220552": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:25:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3760775432": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:25:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1746797014": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:25:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1029045895": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:25:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3310949126": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:25:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2874160765": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:25:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2620179974": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:25:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/421194352": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:25:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/1077377902": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:25:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3428757300": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:25:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1759116360": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:25:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2564333506": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:25:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/717120167": "2024-09-11T18:22:08.678985+0000" 2024-09-10T18:25:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-09-10T18:25:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-09-10T18:25:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-09-10T18:25:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-09-10T18:25:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-09-10T18:25:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-09-10T18:25:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-09-10T18:25:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-09-10T18:25:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:25:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-09-10T18:25:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-09-10T18:25:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-09-10T18:25:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-09-10T18:25:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-09-10T18:25:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-09-10T18:25:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-09-10T18:25:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-09-10T18:25:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-09-10T18:25:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-09-10T18:25:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-09-10T18:25:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:25:51.998 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:25:52.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:51 smithi028 ceph-mon[29291]: pgmap v168: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 901 op/s; 30490/293319 objects degraded (10.395%) 2024-09-10T18:25:52.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:51 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/682847125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:25:52.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:51 smithi028 ceph-mon[39220]: pgmap v168: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 901 op/s; 30490/293319 objects degraded (10.395%) 2024-09-10T18:25:52.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:51 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/682847125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:25:52.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:52.103 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:52.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:52.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:52.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:51 smithi204 ceph-mon[32357]: pgmap v168: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 901 op/s; 30490/293319 objects degraded (10.395%) 2024-09-10T18:25:52.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:51 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/682847125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:25:52.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:52.240 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:52.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:52.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:52.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:25:52.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:52.378 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:25:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:25:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:25:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:52.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:52.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:52.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:52.534 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:52.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:52.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:52.639 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:52.665 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:52.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:52.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:52.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:52.774 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:25:52.774 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:25:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:52.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:52.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:52.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:52.959 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:52.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:52.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:53.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:53.106 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:53.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:53.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:53.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:53.258 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:53.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:53.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:53.360 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:53.383 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:53.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:53.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:53.485 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:53.509 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:53.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:53.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:53.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-10T18:25:53.611 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:53.645 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:53.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:53.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:53.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:53.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:53.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:53.889 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:25:53.896 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:53.924 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:53.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:53.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:54.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:54.051 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:54.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:54.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:54.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:54.162 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:53 smithi204 ceph-mon[32357]: pgmap v169: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 832 op/s; 30490/293319 objects degraded (10.395%) 2024-09-10T18:25:54.162 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:53 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:25:54.162 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:25:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:25:54.192 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:54.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:54.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:54.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:53 smithi028 ceph-mon[29291]: pgmap v169: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 832 op/s; 30490/293319 objects degraded (10.395%) 2024-09-10T18:25:54.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:53 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:25:54.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:53 smithi028 ceph-mon[39220]: pgmap v169: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 832 op/s; 30490/293319 objects degraded (10.395%) 2024-09-10T18:25:54.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:53 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:25:54.295 INFO:tasks.thrashosds.thrasher:j is {'epoch': 129, 'fsid': '5eababb2-6fa0-11ef-bcea-c7b262605968', 'created': '2024-09-10T18:16:20.680373+0000', 'modified': '2024-09-10T18:25:50.596243+0000', 'last_up_change': '2024-09-10T18:23:27.783937+0000', 'last_in_change': '2024-09-10T18:21:02.114621+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T18:19:24.013536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-10T18:21:30.065209+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 62, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T18:21:30.506441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T18:21:32.008130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T18:21:34.060571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T18:21:36.127330+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-10T18:23:25.513798+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'f1c834d9-252a-4077-b25e-169181d70251', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 3188372351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 3188372351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 3188372351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 3188372351}]}, 'public_addr': '172.21.15.28:6801/3188372351', 'cluster_addr': '172.21.15.28:6802/3188372351', 'heartbeat_back_addr': '172.21.15.28:6804/3188372351', 'heartbeat_front_addr': '172.21.15.28:6803/3188372351', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '81f1855d-b946-4ca3-b246-38977b9e20fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1369912678}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1369912678}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1369912678}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1369912678}]}, 'public_addr': '172.21.15.28:6805/1369912678', 'cluster_addr': '172.21.15.28:6806/1369912678', 'heartbeat_back_addr': '172.21.15.28:6808/1369912678', 'heartbeat_front_addr': '172.21.15.28:6807/1369912678', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2ad55d2a-36fa-4756-bf22-58d4b049486e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 1243406003}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 1243406003}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 1243406003}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1243406003}]}, 'public_addr': '172.21.15.28:6809/1243406003', 'cluster_addr': '172.21.15.28:6810/1243406003', 'heartbeat_back_addr': '172.21.15.28:6812/1243406003', 'heartbeat_front_addr': '172.21.15.28:6811/1243406003', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '242729a5-a5e8-44d5-97c8-14d14e970bff', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 67703283}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 67703283}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 67703283}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 67703283}]}, 'public_addr': '172.21.15.28:6813/67703283', 'cluster_addr': '172.21.15.28:6814/67703283', 'heartbeat_back_addr': '172.21.15.28:6816/67703283', 'heartbeat_front_addr': '172.21.15.28:6815/67703283', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6e38b8ef-e6ed-493f-8eca-755354754787', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6800', 'nonce': 1619669076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 1619669076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 1619669076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6802', 'nonce': 1619669076}]}, 'public_addr': '172.21.15.204:6800/1619669076', 'cluster_addr': '172.21.15.204:6801/1619669076', 'heartbeat_back_addr': '172.21.15.204:6803/1619669076', 'heartbeat_front_addr': '172.21.15.204:6802/1619669076', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a92e0441-66c9-498e-a0ac-6465acdbb1f5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6804', 'nonce': 2717802880}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 2717802880}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 2717802880}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6806', 'nonce': 2717802880}]}, 'public_addr': '172.21.15.204:6804/2717802880', 'cluster_addr': '172.21.15.204:6805/2717802880', 'heartbeat_back_addr': '172.21.15.204:6807/2717802880', 'heartbeat_front_addr': '172.21.15.204:6806/2717802880', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '12b02875-6653-45ed-8d2b-b4d33bd88525', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 93, 'down_at': 95, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6808', 'nonce': 3377682607}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 3377682607}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 3377682607}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6810', 'nonce': 3377682607}]}, 'public_addr': '172.21.15.204:6808/3377682607', 'cluster_addr': '172.21.15.204:6809/3377682607', 'heartbeat_back_addr': '172.21.15.204:6811/3377682607', 'heartbeat_front_addr': '172.21.15.204:6810/3377682607', 'state': ['exists']}, {'osd': 7, 'uuid': '0c4f7aa9-ff95-4192-a146-ca576eee907f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6812', 'nonce': 3511791879}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 3511791879}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 3511791879}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6814', 'nonce': 3511791879}]}, 'public_addr': '172.21.15.204:6812/3511791879', 'cluster_addr': '172.21.15.204:6813/3511791879', 'heartbeat_back_addr': '172.21.15.204:6815/3511791879', 'heartbeat_front_addr': '172.21.15.204:6814/3511791879', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:25:37.226997+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:24:57.789498+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:25:29.443786+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:25:36.450533+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:25:07.585348+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:25:26.538402+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:23:21.545352+0000', 'dead_epoch': 94}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:25:08.449074+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/1745574083': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/3866220552': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:6800/3760775432': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:0/1746797014': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:0/1029045895': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:6800/3310949126': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/2874160765': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/2620179974': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/421194352': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:6800/1077377902': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/3428757300': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/1759116360': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/2564333506': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/717120167': '2024-09-11T18:22:08.678985+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T18:25:54.295 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-10T18:25:54.296 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:54.321 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:54.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:54.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:54.423 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:54.445 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:54.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:54.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:54.461 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:25:54.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:54.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:54.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:54.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:54.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:54.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:54.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:54.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:54.894 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:54.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:54.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:55.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:55.031 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:55.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:55.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:55.173 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:55.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:55.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:55.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:54 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:55.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:54 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:55.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:55.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:55.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:55.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:54 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:25:55.418 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:55.440 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:55.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:55.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:55.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:55.585 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:55.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:55.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:55.686 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:55.713 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:55.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:55.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:55.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:55.857 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:55.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:55.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:55.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:55.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:55.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:56.100 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:56.123 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:56.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:56.178 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:55 smithi204 ceph-mon[32357]: pgmap v170: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 667 op/s; 30169/290253 objects degraded (10.394%) 2024-09-10T18:25:56.178 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:55 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 30169/290253 objects degraded (10.394%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:56.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:56.226 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:25:56.226 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:25:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:55 smithi028 ceph-mon[39220]: pgmap v170: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 667 op/s; 30169/290253 objects degraded (10.394%) 2024-09-10T18:25:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:55 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 30169/290253 objects degraded (10.394%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:55 smithi028 ceph-mon[29291]: pgmap v170: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 667 op/s; 30169/290253 objects degraded (10.394%) 2024-09-10T18:25:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:55 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 30169/290253 objects degraded (10.394%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:25:56.259 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:56.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:56.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:56.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:56.398 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:56.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:56.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:56.500 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:56.524 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:56.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:56.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:56.625 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:56.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:56.649 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:56.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:56.750 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:56.773 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:56.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:56.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:56.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:56.922 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:56.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:56.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:56.936 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:25:56.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-10T18:25:57.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:56 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/46918337' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:25:57.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:56 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/46918337' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:25:57.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:57.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:57.071 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:57.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:57.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:57.212 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:57.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:57.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:57.256 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:25:57.313 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:57.337 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:57.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:56 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/46918337' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:25:57.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:57.487 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:57.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:57.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:57.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:57.721 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:57.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:57.822 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:57.846 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:57.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:57.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:57.948 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:58.000 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:58.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:58.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:58.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:58.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:57 smithi028 ceph-mon[39220]: pgmap v171: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 972 op/s; 28845/277302 objects degraded (10.402%) 2024-09-10T18:25:58.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:57 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:25:58.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:57 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/46918337' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:25:58.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:57 smithi028 ceph-mon[39220]: osdmap e130: 8 total, 7 up, 8 in 2024-09-10T18:25:58.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:57 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:25:58.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:57 smithi028 ceph-mon[39220]: 6.19 scrub starts 2024-09-10T18:25:58.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:57 smithi028 ceph-mon[39220]: 6.19 scrub ok 2024-09-10T18:25:58.149 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:25:58.149 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:25:58.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:57 smithi028 ceph-mon[29291]: pgmap v171: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 972 op/s; 28845/277302 objects degraded (10.402%) 2024-09-10T18:25:58.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:57 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:25:58.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:57 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/46918337' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:25:58.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:57 smithi028 ceph-mon[29291]: osdmap e130: 8 total, 7 up, 8 in 2024-09-10T18:25:58.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:57 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:25:58.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:57 smithi028 ceph-mon[29291]: 6.19 scrub starts 2024-09-10T18:25:58.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:57 smithi028 ceph-mon[29291]: 6.19 scrub ok 2024-09-10T18:25:58.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:58.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:58.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:58.186 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:25:58.186 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:25:58.226 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:25:58.226 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:25:58.262 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:25:58.262 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:25:58.271 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:25:58.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:58.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:58.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:25:58.295 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:25:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:25:58.327 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:25:58.328 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:25:58.329 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-10T18:25:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:57 smithi204 ceph-mon[32357]: pgmap v171: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 972 op/s; 28845/277302 objects degraded (10.402%) 2024-09-10T18:25:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:57 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:25:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:57 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/46918337' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:25:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:57 smithi204 ceph-mon[32357]: osdmap e130: 8 total, 7 up, 8 in 2024-09-10T18:25:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:57 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:25:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:57 smithi204 ceph-mon[32357]: 6.19 scrub starts 2024-09-10T18:25:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:57 smithi204 ceph-mon[32357]: 6.19 scrub ok 2024-09-10T18:25:58.361 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:25:58.361 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:25:58.395 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:58.417 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:58.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:58.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:58.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:58.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:58.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:58.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:25:58.702 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:58.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:58.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:25:58.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:58.845 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:58.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:58.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:58.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:25:58.986 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:58.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:58.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:25:59.053 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:58 smithi204 ceph-mon[32357]: 5.12 scrub starts 2024-09-10T18:25:59.054 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:58 smithi204 ceph-mon[32357]: 5.12 scrub ok 2024-09-10T18:25:59.054 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:58 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3361947321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:25:59.054 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:58 smithi204 ceph-mon[32357]: 5.a scrub starts 2024-09-10T18:25:59.054 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:58 smithi204 ceph-mon[32357]: 5.a scrub ok 2024-09-10T18:25:59.054 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:58 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:25:59.054 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:58 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:25:59.054 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:58 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:25:59.083 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T18:25:59.083 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T18:25:59.088 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:59.116 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:59.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:59.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:59.171 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:25:59.172 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:25:59.172 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-10T18:25:59.172 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-10T18:25:59.172 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:25:59.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:59.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:58 smithi028 ceph-mon[29291]: 5.12 scrub starts 2024-09-10T18:25:59.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:58 smithi028 ceph-mon[29291]: 5.12 scrub ok 2024-09-10T18:25:59.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:58 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3361947321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:25:59.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:58 smithi028 ceph-mon[29291]: 5.a scrub starts 2024-09-10T18:25:59.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:58 smithi028 ceph-mon[29291]: 5.a scrub ok 2024-09-10T18:25:59.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:58 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:25:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:58 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:25:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:58 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:25:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:58 smithi028 ceph-mon[39220]: 5.12 scrub starts 2024-09-10T18:25:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:58 smithi028 ceph-mon[39220]: 5.12 scrub ok 2024-09-10T18:25:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:58 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3361947321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:25:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:58 smithi028 ceph-mon[39220]: 5.a scrub starts 2024-09-10T18:25:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:58 smithi028 ceph-mon[39220]: 5.a scrub ok 2024-09-10T18:25:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:58 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:25:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:58 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:25:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:58 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:25:59.265 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:59.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:59.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:25:59.296 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-09-10T18:25:59.296 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:25:59.297 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-10T18:25:59.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:25:59.368 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:25:59.390 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:59.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:59.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:25:59.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:59.542 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:59.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:59.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:59.644 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:25:59.667 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:25:59.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:59.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:25:59.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:25:59.775 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:25:59.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:59.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:25:59.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:25:59.908 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-10T18:25:59.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:25:59.980 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:25:59.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:25:59.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:00.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:00.131 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:00.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:00.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:00.202 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:26:00.233 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:00.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:59 smithi028 ceph-mon[29291]: pgmap v173: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 623 B/s rd, 0 B/s wr, 651 op/s; 28845/277302 objects degraded (10.402%) 2024-09-10T18:26:00.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:59 smithi028 ceph-mon[29291]: 3.15 scrub starts 2024-09-10T18:26:00.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:25:59 smithi028 ceph-mon[29291]: 3.15 scrub ok 2024-09-10T18:26:00.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:59 smithi028 ceph-mon[39220]: pgmap v173: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 623 B/s rd, 0 B/s wr, 651 op/s; 28845/277302 objects degraded (10.402%) 2024-09-10T18:26:00.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:59 smithi028 ceph-mon[39220]: 3.15 scrub starts 2024-09-10T18:26:00.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:25:59 smithi028 ceph-mon[39220]: 3.15 scrub ok 2024-09-10T18:26:00.261 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:00.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:00.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:00.327 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:59 smithi204 ceph-mon[32357]: pgmap v173: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 623 B/s rd, 0 B/s wr, 651 op/s; 28845/277302 objects degraded (10.402%) 2024-09-10T18:26:00.327 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:59 smithi204 ceph-mon[32357]: 3.15 scrub starts 2024-09-10T18:26:00.327 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:25:59 smithi204 ceph-mon[32357]: 3.15 scrub ok 2024-09-10T18:26:00.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:00.402 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:00.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:00.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:00.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:00.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:00.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:00.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:00.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:00.684 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:00.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:00.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:00.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:00.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:00.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:00.979 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:00.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:00.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:01.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:00 smithi028 ceph-mon[39220]: 4.1b scrub starts 2024-09-10T18:26:01.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:00 smithi028 ceph-mon[39220]: 4.1b scrub ok 2024-09-10T18:26:01.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:00 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 28845/277302 objects degraded (10.402%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:01.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:00 smithi028 ceph-mon[39220]: osdmap e131: 8 total, 7 up, 8 in 2024-09-10T18:26:01.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:00 smithi028 ceph-mon[29291]: 4.1b scrub starts 2024-09-10T18:26:01.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:00 smithi028 ceph-mon[29291]: 4.1b scrub ok 2024-09-10T18:26:01.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:00 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 28845/277302 objects degraded (10.402%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:01.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:00 smithi028 ceph-mon[29291]: osdmap e131: 8 total, 7 up, 8 in 2024-09-10T18:26:01.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:01.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:26:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:01.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:01.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:01.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:01.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:01.281 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:01.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:01.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:00 smithi204 ceph-mon[32357]: 4.1b scrub starts 2024-09-10T18:26:01.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:00 smithi204 ceph-mon[32357]: 4.1b scrub ok 2024-09-10T18:26:01.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:00 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 28845/277302 objects degraded (10.402%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:01.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:00 smithi204 ceph-mon[32357]: osdmap e131: 8 total, 7 up, 8 in 2024-09-10T18:26:01.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:01.434 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:01.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:01.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:26:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:26:01] "GET /metrics HTTP/1.1" 200 223772 "" "Prometheus/2.43.0" 2024-09-10T18:26:01.535 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:01.558 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:01.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:01.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:01.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:01.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:01.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:01.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:01.841 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:01.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:01.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:01.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:01.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:01.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:01.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:02.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:01 smithi028 ceph-mon[39220]: 4.f scrub starts 2024-09-10T18:26:02.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:01 smithi028 ceph-mon[39220]: 4.f scrub ok 2024-09-10T18:26:02.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:01 smithi028 ceph-mon[39220]: pgmap v175: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 27204/261219 objects degraded (10.414%) 2024-09-10T18:26:02.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:01 smithi028 ceph-mon[39220]: 4.1c scrub starts 2024-09-10T18:26:02.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:01 smithi028 ceph-mon[29291]: 4.f scrub starts 2024-09-10T18:26:02.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:01 smithi028 ceph-mon[29291]: 4.f scrub ok 2024-09-10T18:26:02.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:01 smithi028 ceph-mon[29291]: pgmap v175: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 27204/261219 objects degraded (10.414%) 2024-09-10T18:26:02.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:01 smithi028 ceph-mon[29291]: 4.1c scrub starts 2024-09-10T18:26:02.085 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:02.111 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:02.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:02.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:02.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:02.235 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:02.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:02.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:02.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:02.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:26:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:26:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:26:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:01 smithi204 ceph-mon[32357]: 4.f scrub starts 2024-09-10T18:26:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:01 smithi204 ceph-mon[32357]: 4.f scrub ok 2024-09-10T18:26:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:01 smithi204 ceph-mon[32357]: pgmap v175: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 27204/261219 objects degraded (10.414%) 2024-09-10T18:26:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:01 smithi204 ceph-mon[32357]: 4.1c scrub starts 2024-09-10T18:26:02.378 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:02.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:02.479 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:02.501 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:02.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:02.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:02.627 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:02.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:02.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:02.728 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:02.752 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:02.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:02.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:02.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:02.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:02.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:02.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:02.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:03.027 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:03.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:03.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:03.128 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:03.151 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:03.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:03.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:02 smithi028 ceph-mon[39220]: 4.1c scrub ok 2024-09-10T18:26:03.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:02 smithi028 ceph-mon[39220]: 6.8 scrub starts 2024-09-10T18:26:03.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:02 smithi028 ceph-mon[39220]: 6.8 scrub ok 2024-09-10T18:26:03.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:02 smithi028 ceph-mon[29291]: 4.1c scrub ok 2024-09-10T18:26:03.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:02 smithi028 ceph-mon[29291]: 6.8 scrub starts 2024-09-10T18:26:03.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:02 smithi028 ceph-mon[29291]: 6.8 scrub ok 2024-09-10T18:26:03.208 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:02 smithi204 ceph-mon[32357]: 4.1c scrub ok 2024-09-10T18:26:03.208 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:02 smithi204 ceph-mon[32357]: 6.8 scrub starts 2024-09-10T18:26:03.208 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:02 smithi204 ceph-mon[32357]: 6.8 scrub ok 2024-09-10T18:26:03.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:03.289 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:03.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:03.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:03.391 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:03.418 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:03.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:03.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:03.455 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:26:03.455 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-10T18:26:03.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-10T18:26:03.455 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:26:03.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.455 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.34334.0:568 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94344803770144 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-10T18:26:03.455 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:09.060228+0000", 2024-09-10T18:26:03.455 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 174.39072635700001, 2024-09-10T18:26:03.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0146112759999999, 2024-09-10T18:26:03.455 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.455 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:26:03.456 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.456 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.34334", 2024-09-10T18:26:03.456 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/601695668", 2024-09-10T18:26:03.456 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 568 2024-09-10T18:26:03.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.456 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.060228+0000", 2024-09-10T18:26:03.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.060228+0000", 2024-09-10T18:26:03.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.060224+0000", 2024-09-10T18:26:03.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999962 2024-09-10T18:26:03.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.060238+0000", 2024-09-10T18:26:03.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4086e-05 2024-09-10T18:26:03.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.060260+0000", 2024-09-10T18:26:03.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1081999999999999e-05 2024-09-10T18:26:03.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.060270+0000", 2024-09-10T18:26:03.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0057e-05 2024-09-10T18:26:03.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.080528+0000", 2024-09-10T18:26:03.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020258911000000001 2024-09-10T18:26:03.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:26:03.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.080534+0000", 2024-09-10T18:26:03.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5489999999999997e-06 2024-09-10T18:26:03.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.067258+0000", 2024-09-10T18:26:03.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98672402199999998 2024-09-10T18:26:03.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for missing object", 2024-09-10T18:26:03.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.067399+0000", 2024-09-10T18:26:03.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014092 2024-09-10T18:26:03.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.070170+0000", 2024-09-10T18:26:03.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027705450000000001 2024-09-10T18:26:03.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-10T18:26:03.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.070293+0000", 2024-09-10T18:26:03.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012330800000000001 2024-09-10T18:26:03.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.072302+0000", 2024-09-10T18:26:03.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020088089999999999 2024-09-10T18:26:03.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.461 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.072366+0000", 2024-09-10T18:26:03.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4388999999999996e-05 2024-09-10T18:26:03.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.461 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.074839+0000", 2024-09-10T18:26:03.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024734029999999999 2024-09-10T18:26:03.461 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.461 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.461 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.462 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.34334.0:575 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch ping cookie 94344803770144 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-10T18:26:03.462 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:09.771277+0000", 2024-09-10T18:26:03.462 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 173.679677349, 2024-09-10T18:26:03.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.30179671000000002, 2024-09-10T18:26:03.462 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.462 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:26:03.462 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.462 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.34334", 2024-09-10T18:26:03.462 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/601695668", 2024-09-10T18:26:03.462 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 575 2024-09-10T18:26:03.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.462 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.771277+0000", 2024-09-10T18:26:03.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.771277+0000", 2024-09-10T18:26:03.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.771276+0000", 2024-09-10T18:26:03.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:26:03.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.771284+0000", 2024-09-10T18:26:03.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9179999999999994e-06 2024-09-10T18:26:03.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.771292+0000", 2024-09-10T18:26:03.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5220000000000001e-06 2024-09-10T18:26:03.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.771296+0000", 2024-09-10T18:26:03.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3100000000000002e-06 2024-09-10T18:26:03.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.771350+0000", 2024-09-10T18:26:03.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3996999999999998e-05 2024-09-10T18:26:03.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:26:03.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.771357+0000", 2024-09-10T18:26:03.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0260000000000002e-06 2024-09-10T18:26:03.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.067428+0000", 2024-09-10T18:26:03.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29607112099999999 2024-09-10T18:26:03.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for missing object", 2024-09-10T18:26:03.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.067474+0000", 2024-09-10T18:26:03.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6084999999999997e-05 2024-09-10T18:26:03.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.070328+0000", 2024-09-10T18:26:03.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028540940000000002 2024-09-10T18:26:03.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-10T18:26:03.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.070421+0000", 2024-09-10T18:26:03.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2822e-05 2024-09-10T18:26:03.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.072747+0000", 2024-09-10T18:26:03.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023261639999999999 2024-09-10T18:26:03.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.072849+0000", 2024-09-10T18:26:03.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010183099999999999 2024-09-10T18:26:03.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.073074+0000", 2024-09-10T18:26:03.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022475899999999999 2024-09-10T18:26:03.468 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.468 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.468 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.468 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:28268 7.0 7:03e36e58:::benchmark_data_smithi028_77856_object28267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:26:03.469 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:55.402609+0000", 2024-09-10T18:26:03.469 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 128.04834590799999, 2024-09-10T18:26:03.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.129438948, 2024-09-10T18:26:03.469 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.469 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.469 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.469 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.469 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.469 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 28268 2024-09-10T18:26:03.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.470 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.402609+0000", 2024-09-10T18:26:03.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.402609+0000", 2024-09-10T18:26:03.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.402605+0000", 2024-09-10T18:26:03.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999967 2024-09-10T18:26:03.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.402652+0000", 2024-09-10T18:26:03.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6505000000000003e-05 2024-09-10T18:26:03.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.402674+0000", 2024-09-10T18:26:03.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1889e-05 2024-09-10T18:26:03.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.402682+0000", 2024-09-10T18:26:03.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0129999999999993e-06 2024-09-10T18:26:03.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.402742+0000", 2024-09-10T18:26:03.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0185000000000001e-05 2024-09-10T18:26:03.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.402956+0000", 2024-09-10T18:26:03.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021390999999999999 2024-09-10T18:26:03.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.403124+0000", 2024-09-10T18:26:03.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000168775 2024-09-10T18:26:03.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.405090+0000", 2024-09-10T18:26:03.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001965381 2024-09-10T18:26:03.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.405435+0000", 2024-09-10T18:26:03.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00034498100000000001 2024-09-10T18:26:03.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.531950+0000", 2024-09-10T18:26:03.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.126515715 2024-09-10T18:26:03.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.532005+0000", 2024-09-10T18:26:03.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4333000000000003e-05 2024-09-10T18:26:03.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.532048+0000", 2024-09-10T18:26:03.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2728000000000002e-05 2024-09-10T18:26:03.474 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.475 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.475 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.475 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:28269 7.0 7:077ce142:::benchmark_data_smithi028_77856_object28268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:26:03.475 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:55.403527+0000", 2024-09-10T18:26:03.475 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 128.047427228, 2024-09-10T18:26:03.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.12895774700000001, 2024-09-10T18:26:03.475 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.475 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.475 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.475 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.475 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.475 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 28269 2024-09-10T18:26:03.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.476 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.403527+0000", 2024-09-10T18:26:03.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.403527+0000", 2024-09-10T18:26:03.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.403526+0000", 2024-09-10T18:26:03.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:26:03.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.403593+0000", 2024-09-10T18:26:03.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7466999999999996e-05 2024-09-10T18:26:03.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.403608+0000", 2024-09-10T18:26:03.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4786999999999999e-05 2024-09-10T18:26:03.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.403614+0000", 2024-09-10T18:26:03.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7049999999999996e-06 2024-09-10T18:26:03.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.404139+0000", 2024-09-10T18:26:03.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00052514200000000003 2024-09-10T18:26:03.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.404265+0000", 2024-09-10T18:26:03.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012630900000000001 2024-09-10T18:26:03.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.404407+0000", 2024-09-10T18:26:03.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014217200000000001 2024-09-10T18:26:03.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.405237+0000", 2024-09-10T18:26:03.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00082998100000000003 2024-09-10T18:26:03.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.405538+0000", 2024-09-10T18:26:03.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00030039900000000002 2024-09-10T18:26:03.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.532432+0000", 2024-09-10T18:26:03.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.126894384 2024-09-10T18:26:03.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.532462+0000", 2024-09-10T18:26:03.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9771e-05 2024-09-10T18:26:03.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.532485+0000", 2024-09-10T18:26:03.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3269e-05 2024-09-10T18:26:03.481 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.481 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.481 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.481 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:28635 7.0 7:067af013:::benchmark_data_smithi028_77856_object28634:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:26:03.481 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:55.726823+0000", 2024-09-10T18:26:03.481 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 127.72413114, 2024-09-10T18:26:03.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13670795599999999, 2024-09-10T18:26:03.481 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.481 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.481 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.482 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.482 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.482 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 28635 2024-09-10T18:26:03.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.482 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726823+0000", 2024-09-10T18:26:03.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726823+0000", 2024-09-10T18:26:03.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726821+0000", 2024-09-10T18:26:03.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:26:03.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726888+0000", 2024-09-10T18:26:03.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7714999999999998e-05 2024-09-10T18:26:03.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726910+0000", 2024-09-10T18:26:03.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1688e-05 2024-09-10T18:26:03.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726917+0000", 2024-09-10T18:26:03.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.934e-06 2024-09-10T18:26:03.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726967+0000", 2024-09-10T18:26:03.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0111000000000002e-05 2024-09-10T18:26:03.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.727088+0000", 2024-09-10T18:26:03.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012124 2024-09-10T18:26:03.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.727225+0000", 2024-09-10T18:26:03.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000136778 2024-09-10T18:26:03.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.728383+0000", 2024-09-10T18:26:03.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011583089999999999 2024-09-10T18:26:03.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.728694+0000", 2024-09-10T18:26:03.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00031094 2024-09-10T18:26:03.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.863440+0000", 2024-09-10T18:26:03.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13474614500000001 2024-09-10T18:26:03.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.863489+0000", 2024-09-10T18:26:03.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8908e-05 2024-09-10T18:26:03.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.863531+0000", 2024-09-10T18:26:03.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2027000000000003e-05 2024-09-10T18:26:03.487 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.487 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.487 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.487 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:34163 7.0 7:0f36bf7d:::benchmark_data_smithi028_77856_object34162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:26:03.487 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:59.285980+0000", 2024-09-10T18:26:03.487 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 124.164974543, 2024-09-10T18:26:03.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.10179368, 2024-09-10T18:26:03.488 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.488 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.488 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.488 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.488 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.488 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 34163 2024-09-10T18:26:03.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.488 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.285980+0000", 2024-09-10T18:26:03.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.285980+0000", 2024-09-10T18:26:03.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.285978+0000", 2024-09-10T18:26:03.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:26:03.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.286030+0000", 2024-09-10T18:26:03.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1989e-05 2024-09-10T18:26:03.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.298958+0000", 2024-09-10T18:26:03.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012928873 2024-09-10T18:26:03.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.298971+0000", 2024-09-10T18:26:03.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2529e-05 2024-09-10T18:26:03.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.299196+0000", 2024-09-10T18:26:03.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000225493 2024-09-10T18:26:03.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.299321+0000", 2024-09-10T18:26:03.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000124871 2024-09-10T18:26:03.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.299457+0000", 2024-09-10T18:26:03.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013559299999999999 2024-09-10T18:26:03.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.301952+0000", 2024-09-10T18:26:03.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024954729999999998 2024-09-10T18:26:03.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.301987+0000", 2024-09-10T18:26:03.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5036999999999997e-05 2024-09-10T18:26:03.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.387705+0000", 2024-09-10T18:26:03.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.085717481999999998 2024-09-10T18:26:03.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.387739+0000", 2024-09-10T18:26:03.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.362e-05 2024-09-10T18:26:03.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.387774+0000", 2024-09-10T18:26:03.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5111000000000003e-05 2024-09-10T18:26:03.493 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.493 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.493 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.493 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:34170 7.0 7:0862d140:::benchmark_data_smithi028_77856_object34169:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:26:03.494 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:59.298997+0000", 2024-09-10T18:26:03.494 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 124.151957502, 2024-09-10T18:26:03.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.088863384000000004, 2024-09-10T18:26:03.494 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.494 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.494 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.494 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.494 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.494 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 34170 2024-09-10T18:26:03.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.494 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.298997+0000", 2024-09-10T18:26:03.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.298997+0000", 2024-09-10T18:26:03.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.298995+0000", 2024-09-10T18:26:03.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:26:03.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.299036+0000", 2024-09-10T18:26:03.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1223999999999997e-05 2024-09-10T18:26:03.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.299045+0000", 2024-09-10T18:26:03.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8610000000000004e-06 2024-09-10T18:26:03.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.299050+0000", 2024-09-10T18:26:03.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5229999999999997e-06 2024-09-10T18:26:03.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.300724+0000", 2024-09-10T18:26:03.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001674306 2024-09-10T18:26:03.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.300838+0000", 2024-09-10T18:26:03.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114033 2024-09-10T18:26:03.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.300939+0000", 2024-09-10T18:26:03.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000100967 2024-09-10T18:26:03.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.302100+0000", 2024-09-10T18:26:03.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011612949999999999 2024-09-10T18:26:03.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.302231+0000", 2024-09-10T18:26:03.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000130648 2024-09-10T18:26:03.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.387806+0000", 2024-09-10T18:26:03.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.085575021000000001 2024-09-10T18:26:03.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.387838+0000", 2024-09-10T18:26:03.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1990999999999999e-05 2024-09-10T18:26:03.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.387860+0000", 2024-09-10T18:26:03.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2385e-05 2024-09-10T18:26:03.500 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.500 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.500 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.500 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:34192 7.0 7:04307710:::benchmark_data_smithi028_77856_object34191:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:26:03.500 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:59.303363+0000", 2024-09-10T18:26:03.501 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 124.14759195800001, 2024-09-10T18:26:03.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.087319685999999994, 2024-09-10T18:26:03.501 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.501 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.501 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.501 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.501 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.501 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 34192 2024-09-10T18:26:03.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.501 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303363+0000", 2024-09-10T18:26:03.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303363+0000", 2024-09-10T18:26:03.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303360+0000", 2024-09-10T18:26:03.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:26:03.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303422+0000", 2024-09-10T18:26:03.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1478999999999999e-05 2024-09-10T18:26:03.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.503 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303433+0000", 2024-09-10T18:26:03.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1136e-05 2024-09-10T18:26:03.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.503 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303438+0000", 2024-09-10T18:26:03.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2979999999999999e-06 2024-09-10T18:26:03.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303469+0000", 2024-09-10T18:26:03.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1127000000000002e-05 2024-09-10T18:26:03.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303545+0000", 2024-09-10T18:26:03.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5278999999999995e-05 2024-09-10T18:26:03.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303621+0000", 2024-09-10T18:26:03.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6248999999999995e-05 2024-09-10T18:26:03.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.304592+0000", 2024-09-10T18:26:03.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00097132200000000005 2024-09-10T18:26:03.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.304991+0000", 2024-09-10T18:26:03.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00039831900000000001 2024-09-10T18:26:03.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.390596+0000", 2024-09-10T18:26:03.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.085605590999999995 2024-09-10T18:26:03.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.390647+0000", 2024-09-10T18:26:03.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0890000000000002e-05 2024-09-10T18:26:03.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.390682+0000", 2024-09-10T18:26:03.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5182000000000003e-05 2024-09-10T18:26:03.506 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.506 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.506 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.507 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:36611 7.0 7:030438c2:::benchmark_data_smithi028_77856_object36610:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-10T18:26:03.507 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:24:00.925382+0000", 2024-09-10T18:26:03.507 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 122.525572306, 2024-09-10T18:26:03.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.082486538999999998, 2024-09-10T18:26:03.507 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.507 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.507 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.507 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.507 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.507 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 36611 2024-09-10T18:26:03.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.507 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:00.925382+0000", 2024-09-10T18:26:03.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:00.925382+0000", 2024-09-10T18:26:03.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:00.925380+0000", 2024-09-10T18:26:03.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:26:03.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:00.925449+0000", 2024-09-10T18:26:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9263000000000004e-05 2024-09-10T18:26:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:00.925461+0000", 2024-09-10T18:26:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1449e-05 2024-09-10T18:26:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:00.925466+0000", 2024-09-10T18:26:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2680000000000002e-06 2024-09-10T18:26:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:00.925507+0000", 2024-09-10T18:26:03.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1457e-05 2024-09-10T18:26:03.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:00.925625+0000", 2024-09-10T18:26:03.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000117543 2024-09-10T18:26:03.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:00.925763+0000", 2024-09-10T18:26:03.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013795100000000001 2024-09-10T18:26:03.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:00.927304+0000", 2024-09-10T18:26:03.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0015408360000000001 2024-09-10T18:26:03.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:00.927915+0000", 2024-09-10T18:26:03.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00061077600000000005 2024-09-10T18:26:03.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.007815+0000", 2024-09-10T18:26:03.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.079900461000000006 2024-09-10T18:26:03.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.007845+0000", 2024-09-10T18:26:03.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0020999999999999e-05 2024-09-10T18:26:03.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.007869+0000", 2024-09-10T18:26:03.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.366e-05 2024-09-10T18:26:03.514 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.514 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.514 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.514 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:37019 7.0 7:0101e7cb:::benchmark_data_smithi028_77856_object37018:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-10T18:26:03.514 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:24:01.213014+0000", 2024-09-10T18:26:03.514 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 122.237940743, 2024-09-10T18:26:03.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.088142556999999996, 2024-09-10T18:26:03.514 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.514 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.514 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.514 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.515 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.515 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 37019 2024-09-10T18:26:03.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.515 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213014+0000", 2024-09-10T18:26:03.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213014+0000", 2024-09-10T18:26:03.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213012+0000", 2024-09-10T18:26:03.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:26:03.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213069+0000", 2024-09-10T18:26:03.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7249e-05 2024-09-10T18:26:03.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213082+0000", 2024-09-10T18:26:03.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2611e-05 2024-09-10T18:26:03.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213087+0000", 2024-09-10T18:26:03.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7359999999999998e-06 2024-09-10T18:26:03.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213127+0000", 2024-09-10T18:26:03.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9477999999999997e-05 2024-09-10T18:26:03.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213242+0000", 2024-09-10T18:26:03.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114975 2024-09-10T18:26:03.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213368+0000", 2024-09-10T18:26:03.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012595799999999999 2024-09-10T18:26:03.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.215333+0000", 2024-09-10T18:26:03.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019656859999999999 2024-09-10T18:26:03.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.217238+0000", 2024-09-10T18:26:03.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001904406 2024-09-10T18:26:03.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.301068+0000", 2024-09-10T18:26:03.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.083830528000000001 2024-09-10T18:26:03.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.301119+0000", 2024-09-10T18:26:03.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0800000000000002e-05 2024-09-10T18:26:03.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.301156+0000", 2024-09-10T18:26:03.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7094000000000001e-05 2024-09-10T18:26:03.520 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.520 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.520 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.520 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:37020 7.0 7:06b186a2:::benchmark_data_smithi028_77856_object37019:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-10T18:26:03.520 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:24:01.213097+0000", 2024-09-10T18:26:03.520 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 122.237857336, 2024-09-10T18:26:03.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.088148546999999994, 2024-09-10T18:26:03.520 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.521 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.521 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.521 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.521 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.521 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 37020 2024-09-10T18:26:03.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.521 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213097+0000", 2024-09-10T18:26:03.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213097+0000", 2024-09-10T18:26:03.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213096+0000", 2024-09-10T18:26:03.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:26:03.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213132+0000", 2024-09-10T18:26:03.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5602000000000002e-05 2024-09-10T18:26:03.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213137+0000", 2024-09-10T18:26:03.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6770000000000003e-06 2024-09-10T18:26:03.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213140+0000", 2024-09-10T18:26:03.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1250000000000001e-06 2024-09-10T18:26:03.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.214360+0000", 2024-09-10T18:26:03.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0012207539999999999 2024-09-10T18:26:03.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.214483+0000", 2024-09-10T18:26:03.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012256499999999999 2024-09-10T18:26:03.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.214628+0000", 2024-09-10T18:26:03.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000144602 2024-09-10T18:26:03.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.215466+0000", 2024-09-10T18:26:03.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000838586 2024-09-10T18:26:03.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.217618+0000", 2024-09-10T18:26:03.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002152284 2024-09-10T18:26:03.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.301194+0000", 2024-09-10T18:26:03.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.083575682999999998 2024-09-10T18:26:03.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.301225+0000", 2024-09-10T18:26:03.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0471e-05 2024-09-10T18:26:03.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.526 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.526 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.301246+0000", 2024-09-10T18:26:03.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.109e-05 2024-09-10T18:26:03.526 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.526 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.526 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.526 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:38187 7.0 7:0bd0cd47:::benchmark_data_smithi028_77856_object38186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-10T18:26:03.526 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:24:02.016947+0000", 2024-09-10T18:26:03.526 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 121.434007987, 2024-09-10T18:26:03.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.064508947999999997, 2024-09-10T18:26:03.526 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.526 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.527 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.527 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.527 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.527 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 38187 2024-09-10T18:26:03.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.527 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:02.016947+0000", 2024-09-10T18:26:03.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:02.016947+0000", 2024-09-10T18:26:03.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:02.016944+0000", 2024-09-10T18:26:03.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:26:03.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:02.016989+0000", 2024-09-10T18:26:03.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4617000000000003e-05 2024-09-10T18:26:03.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:02.017000+0000", 2024-09-10T18:26:03.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1299e-05 2024-09-10T18:26:03.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:02.017006+0000", 2024-09-10T18:26:03.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6439999999999997e-06 2024-09-10T18:26:03.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:02.017042+0000", 2024-09-10T18:26:03.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6128999999999999e-05 2024-09-10T18:26:03.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:02.017141+0000", 2024-09-10T18:26:03.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9075999999999998e-05 2024-09-10T18:26:03.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:02.017237+0000", 2024-09-10T18:26:03.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5429999999999999e-05 2024-09-10T18:26:03.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:02.018309+0000", 2024-09-10T18:26:03.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001072459 2024-09-10T18:26:03.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:02.018613+0000", 2024-09-10T18:26:03.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00030392899999999998 2024-09-10T18:26:03.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:02.081373+0000", 2024-09-10T18:26:03.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.062759542000000001 2024-09-10T18:26:03.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:02.081415+0000", 2024-09-10T18:26:03.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2169000000000002e-05 2024-09-10T18:26:03.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:02.081455+0000", 2024-09-10T18:26:03.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0731e-05 2024-09-10T18:26:03.532 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.532 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.532 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.532 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:81102 7.0 7:030c6891:::benchmark_data_smithi028_77856_object81101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-10T18:26:03.532 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:24:34.410675+0000", 2024-09-10T18:26:03.532 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 89.040279393999995, 2024-09-10T18:26:03.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.100851274, 2024-09-10T18:26:03.532 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.532 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.533 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.533 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.533 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.533 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 81102 2024-09-10T18:26:03.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.533 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.410675+0000", 2024-09-10T18:26:03.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.410675+0000", 2024-09-10T18:26:03.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.410672+0000", 2024-09-10T18:26:03.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:26:03.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.410768+0000", 2024-09-10T18:26:03.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5235000000000006e-05 2024-09-10T18:26:03.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.410780+0000", 2024-09-10T18:26:03.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2145e-05 2024-09-10T18:26:03.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.410785+0000", 2024-09-10T18:26:03.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3739999999999997e-06 2024-09-10T18:26:03.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.412516+0000", 2024-09-10T18:26:03.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0017309490000000001 2024-09-10T18:26:03.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.412632+0000", 2024-09-10T18:26:03.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011565699999999999 2024-09-10T18:26:03.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.412731+0000", 2024-09-10T18:26:03.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9159000000000004e-05 2024-09-10T18:26:03.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.415836+0000", 2024-09-10T18:26:03.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0031046379999999998 2024-09-10T18:26:03.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.422925+0000", 2024-09-10T18:26:03.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0070893240000000001 2024-09-10T18:26:03.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.511458+0000", 2024-09-10T18:26:03.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.088532936000000007 2024-09-10T18:26:03.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.538 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.538 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.511492+0000", 2024-09-10T18:26:03.538 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4279999999999997e-05 2024-09-10T18:26:03.538 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.538 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.538 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.511526+0000", 2024-09-10T18:26:03.538 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4341000000000002e-05 2024-09-10T18:26:03.538 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.538 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.538 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.538 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.539 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.539 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:81105 7.0 7:021bcade:::benchmark_data_smithi028_77856_object81104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-10T18:26:03.539 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:24:34.411681+0000", 2024-09-10T18:26:03.539 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 89.039273899999998, 2024-09-10T18:26:03.539 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.102490836, 2024-09-10T18:26:03.539 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.539 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.539 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.539 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.539 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.539 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 81105 2024-09-10T18:26:03.539 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.539 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.540 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.540 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.540 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.411681+0000", 2024-09-10T18:26:03.540 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.540 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.540 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.540 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.540 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.411681+0000", 2024-09-10T18:26:03.540 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.540 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.540 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.540 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.540 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.411679+0000", 2024-09-10T18:26:03.541 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:26:03.541 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.541 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.541 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.541 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.411748+0000", 2024-09-10T18:26:03.541 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9499999999999995e-05 2024-09-10T18:26:03.541 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.541 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.541 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.541 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.411761+0000", 2024-09-10T18:26:03.541 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2789000000000001e-05 2024-09-10T18:26:03.541 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.541 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.411766+0000", 2024-09-10T18:26:03.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.57e-06 2024-09-10T18:26:03.542 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.542 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.414466+0000", 2024-09-10T18:26:03.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002699786 2024-09-10T18:26:03.542 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.542 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.414542+0000", 2024-09-10T18:26:03.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5507999999999996e-05 2024-09-10T18:26:03.543 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.543 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.543 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.543 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.414624+0000", 2024-09-10T18:26:03.543 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2033999999999994e-05 2024-09-10T18:26:03.543 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.543 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.543 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.543 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.415986+0000", 2024-09-10T18:26:03.543 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013624729999999999 2024-09-10T18:26:03.543 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.543 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.543 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.543 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.422959+0000", 2024-09-10T18:26:03.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0069727890000000001 2024-09-10T18:26:03.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.544 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.514112+0000", 2024-09-10T18:26:03.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.091152601 2024-09-10T18:26:03.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.544 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.514149+0000", 2024-09-10T18:26:03.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6931000000000003e-05 2024-09-10T18:26:03.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.544 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.514171+0000", 2024-09-10T18:26:03.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2881999999999999e-05 2024-09-10T18:26:03.545 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.545 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.545 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.545 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.545 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.545 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:82446 7.0 7:04ae1c1f:::benchmark_data_smithi028_77856_object82445:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-10T18:26:03.545 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:24:35.849328+0000", 2024-09-10T18:26:03.545 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 87.601626893000002, 2024-09-10T18:26:03.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.079980908000000003, 2024-09-10T18:26:03.545 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.545 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.545 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.545 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.546 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.546 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 82446 2024-09-10T18:26:03.546 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.546 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.546 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.546 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.546 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.849328+0000", 2024-09-10T18:26:03.547 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.547 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.547 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.547 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.547 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.849328+0000", 2024-09-10T18:26:03.548 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.548 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.548 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.548 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.548 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.849325+0000", 2024-09-10T18:26:03.548 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:26:03.548 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.548 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.548 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.548 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.849370+0000", 2024-09-10T18:26:03.548 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5148000000000002e-05 2024-09-10T18:26:03.548 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.548 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.548 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.548 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.849386+0000", 2024-09-10T18:26:03.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5420000000000001e-05 2024-09-10T18:26:03.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.549 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.549 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.849393+0000", 2024-09-10T18:26:03.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9e-06 2024-09-10T18:26:03.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.549 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.549 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.849456+0000", 2024-09-10T18:26:03.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2972000000000006e-05 2024-09-10T18:26:03.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.549 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.849628+0000", 2024-09-10T18:26:03.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017222 2024-09-10T18:26:03.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.849807+0000", 2024-09-10T18:26:03.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017890500000000001 2024-09-10T18:26:03.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.852561+0000", 2024-09-10T18:26:03.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027540749999999999 2024-09-10T18:26:03.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.857384+0000", 2024-09-10T18:26:03.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0048230369999999996 2024-09-10T18:26:03.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.929219+0000", 2024-09-10T18:26:03.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.071835449999999995 2024-09-10T18:26:03.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.929270+0000", 2024-09-10T18:26:03.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0485000000000002e-05 2024-09-10T18:26:03.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.552 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.552 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.552 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.929309+0000", 2024-09-10T18:26:03.552 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8835999999999999e-05 2024-09-10T18:26:03.552 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.552 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.552 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.552 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.552 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.552 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:82449 7.0 7:0b9db2cc:::benchmark_data_smithi028_77856_object82448:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-10T18:26:03.552 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:24:35.853794+0000", 2024-09-10T18:26:03.552 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 87.597160242000001, 2024-09-10T18:26:03.552 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.075799596999999996, 2024-09-10T18:26:03.552 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.553 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.553 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.553 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.553 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.553 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 82449 2024-09-10T18:26:03.553 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.553 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.553 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.553 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.553 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.853794+0000", 2024-09-10T18:26:03.553 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.553 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.553 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.553 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.554 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.853794+0000", 2024-09-10T18:26:03.554 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.554 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.554 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.554 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.554 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.853792+0000", 2024-09-10T18:26:03.554 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:26:03.554 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.554 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.554 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.554 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.853840+0000", 2024-09-10T18:26:03.554 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7250000000000003e-05 2024-09-10T18:26:03.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.853853+0000", 2024-09-10T18:26:03.555 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2853e-05 2024-09-10T18:26:03.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.853858+0000", 2024-09-10T18:26:03.555 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2569999999999998e-06 2024-09-10T18:26:03.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.853903+0000", 2024-09-10T18:26:03.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4783999999999997e-05 2024-09-10T18:26:03.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.853992+0000", 2024-09-10T18:26:03.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9761999999999999e-05 2024-09-10T18:26:03.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.854078+0000", 2024-09-10T18:26:03.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5978999999999998e-05 2024-09-10T18:26:03.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.856332+0000", 2024-09-10T18:26:03.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022539320000000002 2024-09-10T18:26:03.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.857521+0000", 2024-09-10T18:26:03.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001188239 2024-09-10T18:26:03.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.929528+0000", 2024-09-10T18:26:03.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.072007979999999999 2024-09-10T18:26:03.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.929559+0000", 2024-09-10T18:26:03.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0201000000000001e-05 2024-09-10T18:26:03.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:35.929594+0000", 2024-09-10T18:26:03.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5169999999999997e-05 2024-09-10T18:26:03.558 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.558 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.558 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.558 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:115304 7.0 7:0bd9f7a8:::benchmark_data_smithi028_77856_object115303:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T18:26:03.559 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:25:08.135786+0000", 2024-09-10T18:26:03.559 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 55.315168124000003, 2024-09-10T18:26:03.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.069302721999999997, 2024-09-10T18:26:03.559 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.559 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.559 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.559 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.559 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.559 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 115304 2024-09-10T18:26:03.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.559 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.559 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.559 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:08.135786+0000", 2024-09-10T18:26:03.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.560 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:08.135786+0000", 2024-09-10T18:26:03.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.560 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:08.135785+0000", 2024-09-10T18:26:03.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:26:03.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:08.135834+0000", 2024-09-10T18:26:03.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9270000000000001e-05 2024-09-10T18:26:03.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:08.135843+0000", 2024-09-10T18:26:03.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4369999999999996e-06 2024-09-10T18:26:03.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:08.135847+0000", 2024-09-10T18:26:03.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8920000000000002e-06 2024-09-10T18:26:03.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:08.135888+0000", 2024-09-10T18:26:03.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1069000000000003e-05 2024-09-10T18:26:03.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:08.136021+0000", 2024-09-10T18:26:03.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013228599999999999 2024-09-10T18:26:03.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:08.136174+0000", 2024-09-10T18:26:03.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000153688 2024-09-10T18:26:03.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:08.138426+0000", 2024-09-10T18:26:03.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002251215 2024-09-10T18:26:03.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:08.139651+0000", 2024-09-10T18:26:03.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0012249559999999999 2024-09-10T18:26:03.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:08.205002+0000", 2024-09-10T18:26:03.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.065351139000000003 2024-09-10T18:26:03.564 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.564 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.564 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.564 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:08.205052+0000", 2024-09-10T18:26:03.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0421999999999999e-05 2024-09-10T18:26:03.564 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.564 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.564 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.564 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:08.205089+0000", 2024-09-10T18:26:03.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6956000000000003e-05 2024-09-10T18:26:03.564 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.564 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.564 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.564 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.564 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.564 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:121400 7.0 7:0b67f93f:::benchmark_data_smithi028_77856_object300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:26:03.565 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:25:14.170420+0000", 2024-09-10T18:26:03.565 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.280534236999998, 2024-09-10T18:26:03.565 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.098332630000000004, 2024-09-10T18:26:03.565 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.565 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.565 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.565 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.565 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.565 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 121400 2024-09-10T18:26:03.565 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.565 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.565 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.565 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.565 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170420+0000", 2024-09-10T18:26:03.566 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.566 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.566 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.566 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170420+0000", 2024-09-10T18:26:03.566 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.566 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.566 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.566 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170419+0000", 2024-09-10T18:26:03.566 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:26:03.566 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.566 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.566 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170424+0000", 2024-09-10T18:26:03.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4879999999999998e-06 2024-09-10T18:26:03.567 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.567 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.567 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.567 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170432+0000", 2024-09-10T18:26:03.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.559e-06 2024-09-10T18:26:03.567 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.567 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.567 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.567 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170435+0000", 2024-09-10T18:26:03.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3620000000000002e-06 2024-09-10T18:26:03.567 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.567 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.567 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.568 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170478+0000", 2024-09-10T18:26:03.568 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2814e-05 2024-09-10T18:26:03.568 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.568 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.568 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170608+0000", 2024-09-10T18:26:03.568 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012954500000000001 2024-09-10T18:26:03.568 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.568 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.568 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170733+0000", 2024-09-10T18:26:03.568 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000124855 2024-09-10T18:26:03.568 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.569 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.174781+0000", 2024-09-10T18:26:03.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040485110000000003 2024-09-10T18:26:03.569 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.569 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.569 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.178138+0000", 2024-09-10T18:26:03.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003357077 2024-09-10T18:26:03.569 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.569 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.569 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.268672+0000", 2024-09-10T18:26:03.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.090533708000000004 2024-09-10T18:26:03.570 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.570 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.570 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.570 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.268726+0000", 2024-09-10T18:26:03.570 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4499000000000001e-05 2024-09-10T18:26:03.570 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.570 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.570 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.570 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.268753+0000", 2024-09-10T18:26:03.570 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6543e-05 2024-09-10T18:26:03.570 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.570 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.570 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.570 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.571 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:121401 7.0 7:06878fa9:::benchmark_data_smithi028_77856_object301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:26:03.571 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:25:14.170563+0000", 2024-09-10T18:26:03.571 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.280391559000002, 2024-09-10T18:26:03.571 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.10107751800000001, 2024-09-10T18:26:03.571 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.571 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.571 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.571 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.571 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.571 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 121401 2024-09-10T18:26:03.571 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.571 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.572 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.572 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170563+0000", 2024-09-10T18:26:03.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.572 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.572 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.572 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.572 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170563+0000", 2024-09-10T18:26:03.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.572 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.572 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.572 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.572 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170562+0000", 2024-09-10T18:26:03.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999995 2024-09-10T18:26:03.573 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170565+0000", 2024-09-10T18:26:03.573 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.52e-06 2024-09-10T18:26:03.573 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170571+0000", 2024-09-10T18:26:03.573 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6829999999999996e-06 2024-09-10T18:26:03.573 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170573+0000", 2024-09-10T18:26:03.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6000000000000001e-06 2024-09-10T18:26:03.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.172895+0000", 2024-09-10T18:26:03.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023221980000000001 2024-09-10T18:26:03.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.172991+0000", 2024-09-10T18:26:03.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5458000000000007e-05 2024-09-10T18:26:03.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.173075+0000", 2024-09-10T18:26:03.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3683000000000005e-05 2024-09-10T18:26:03.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.575 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.575 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.174905+0000", 2024-09-10T18:26:03.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001829991 2024-09-10T18:26:03.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.575 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.575 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.178179+0000", 2024-09-10T18:26:03.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032743680000000002 2024-09-10T18:26:03.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.575 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.575 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.271586+0000", 2024-09-10T18:26:03.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.093406792000000002 2024-09-10T18:26:03.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.271617+0000", 2024-09-10T18:26:03.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1347999999999999e-05 2024-09-10T18:26:03.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.271640+0000", 2024-09-10T18:26:03.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3400999999999998e-05 2024-09-10T18:26:03.576 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.576 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.576 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.577 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:121429 7.0 7:0f8b2bfb:::benchmark_data_smithi028_77856_object329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:26:03.577 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:25:14.203906+0000", 2024-09-10T18:26:03.577 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.247048436, 2024-09-10T18:26:03.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.073414558000000005, 2024-09-10T18:26:03.577 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:03.577 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:03.577 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:03.577 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:03.577 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:03.577 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 121429 2024-09-10T18:26:03.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.577 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:03.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:03.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.203906+0000", 2024-09-10T18:26:03.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:03.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.203906+0000", 2024-09-10T18:26:03.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:03.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:03.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.203904+0000", 2024-09-10T18:26:03.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:26:03.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:03.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.203911+0000", 2024-09-10T18:26:03.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0620000000000003e-06 2024-09-10T18:26:03.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:03.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.203922+0000", 2024-09-10T18:26:03.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.022e-05 2024-09-10T18:26:03.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:03.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.203927+0000", 2024-09-10T18:26:03.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4750000000000001e-06 2024-09-10T18:26:03.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:03.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.203963+0000", 2024-09-10T18:26:03.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6148e-05 2024-09-10T18:26:03.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:03.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.204046+0000", 2024-09-10T18:26:03.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2793000000000005e-05 2024-09-10T18:26:03.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:26:03.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.204118+0000", 2024-09-10T18:26:03.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2176999999999997e-05 2024-09-10T18:26:03.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:03.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.206564+0000", 2024-09-10T18:26:03.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002445563 2024-09-10T18:26:03.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.206735+0000", 2024-09-10T18:26:03.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017068300000000001 2024-09-10T18:26:03.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:03.582 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.277265+0000", 2024-09-10T18:26:03.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.070530213999999994 2024-09-10T18:26:03.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:26:03.582 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.277297+0000", 2024-09-10T18:26:03.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2129999999999999e-05 2024-09-10T18:26:03.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:03.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:03.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:26:03.583 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.277321+0000", 2024-09-10T18:26:03.583 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3762999999999999e-05 2024-09-10T18:26:03.583 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.583 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.583 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.583 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:03.583 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:03.583 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:26:03.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:03.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:03.625 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:03.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:03.726 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:03.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:03.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:03.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:03.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:03.895 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:03.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:03.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:03.997 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:04.021 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:04.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:04.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:04.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:04.163 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:26:04.190 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:04.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:04.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:04.224 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:26:04.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:03 smithi028 ceph-mon[29291]: pgmap v176: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s; 27204/261219 objects degraded (10.414%) 2024-09-10T18:26:04.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:03 smithi028 ceph-mon[29291]: 7.2 scrub starts 2024-09-10T18:26:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:03 smithi028 ceph-mon[39220]: pgmap v176: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s; 27204/261219 objects degraded (10.414%) 2024-09-10T18:26:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:03 smithi028 ceph-mon[39220]: 7.2 scrub starts 2024-09-10T18:26:04.292 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:04.315 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:04.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:04.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:03 smithi204 ceph-mon[32357]: pgmap v176: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s; 27204/261219 objects degraded (10.414%) 2024-09-10T18:26:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:03 smithi204 ceph-mon[32357]: 7.2 scrub starts 2024-09-10T18:26:04.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:26:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:26:04.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:04.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:04.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:04.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:04.620 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:04.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:04.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:04.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:04.764 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:04.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:04.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:04.913 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:04.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:05.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:05.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:05.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:05.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-10T18:26:05.159 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:05.205 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:05.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:05.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:05.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:04 smithi028 ceph-mon[39220]: 3.17 scrub starts 2024-09-10T18:26:05.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:04 smithi028 ceph-mon[39220]: 3.17 scrub ok 2024-09-10T18:26:05.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:04 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:05.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:04 smithi028 ceph-mon[39220]: 7.a scrub starts 2024-09-10T18:26:05.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:04 smithi028 ceph-mon[39220]: 5.6 scrub starts 2024-09-10T18:26:05.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:04 smithi028 ceph-mon[39220]: 5.6 scrub ok 2024-09-10T18:26:05.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:04 smithi028 ceph-mon[29291]: 3.17 scrub starts 2024-09-10T18:26:05.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:04 smithi028 ceph-mon[29291]: 3.17 scrub ok 2024-09-10T18:26:05.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:04 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:05.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:04 smithi028 ceph-mon[29291]: 7.a scrub starts 2024-09-10T18:26:05.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:04 smithi028 ceph-mon[29291]: 5.6 scrub starts 2024-09-10T18:26:05.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:04 smithi028 ceph-mon[29291]: 5.6 scrub ok 2024-09-10T18:26:05.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:05.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:05.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:04 smithi204 ceph-mon[32357]: 3.17 scrub starts 2024-09-10T18:26:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:04 smithi204 ceph-mon[32357]: 3.17 scrub ok 2024-09-10T18:26:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:04 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:04 smithi204 ceph-mon[32357]: 7.a scrub starts 2024-09-10T18:26:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:04 smithi204 ceph-mon[32357]: 5.6 scrub starts 2024-09-10T18:26:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:04 smithi204 ceph-mon[32357]: 5.6 scrub ok 2024-09-10T18:26:05.437 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:26:05.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:05.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:05.478 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:05.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:05.579 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:05.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:05.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:05.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:05.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:05.743 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:05.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:05.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:05.845 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:05.875 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:05.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:05.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:05.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:06.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:06.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:06.083 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:05 smithi028 ceph-mon[39220]: 7.6 scrub starts 2024-09-10T18:26:06.083 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:05 smithi028 ceph-mon[39220]: pgmap v177: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 794 op/s; 26883/258255 objects degraded (10.409%) 2024-09-10T18:26:06.083 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:05 smithi028 ceph-mon[39220]: 5.16 scrub starts 2024-09-10T18:26:06.083 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:05 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 26883/258255 objects degraded (10.409%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:06.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:05 smithi028 ceph-mon[29291]: 7.6 scrub starts 2024-09-10T18:26:06.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:05 smithi028 ceph-mon[29291]: pgmap v177: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 794 op/s; 26883/258255 objects degraded (10.409%) 2024-09-10T18:26:06.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:05 smithi028 ceph-mon[29291]: 5.16 scrub starts 2024-09-10T18:26:06.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:05 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 26883/258255 objects degraded (10.409%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:05 smithi204 ceph-mon[32357]: 7.6 scrub starts 2024-09-10T18:26:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:05 smithi204 ceph-mon[32357]: pgmap v177: 148 pgs: 24 active+undersized, 24 active+undersized+degraded, 100 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 794 op/s; 26883/258255 objects degraded (10.409%) 2024-09-10T18:26:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:05 smithi204 ceph-mon[32357]: 5.16 scrub starts 2024-09-10T18:26:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:05 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 26883/258255 objects degraded (10.409%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:06.125 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:06.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:06.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:06.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:06.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:06.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:06.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:06.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:06.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:06.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:06.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:06.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:06.564 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:06.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:06.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:06.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:06.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:06.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:06.836 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:06.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:06.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:06.938 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:06.962 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:06.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:06.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:07.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:06 smithi028 ceph-mon[29291]: 3.4 scrub starts 2024-09-10T18:26:07.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:06 smithi028 ceph-mon[29291]: 3.4 scrub ok 2024-09-10T18:26:07.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:06 smithi028 ceph-mon[29291]: 2.1 scrub starts 2024-09-10T18:26:07.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:06 smithi028 ceph-mon[39220]: 3.4 scrub starts 2024-09-10T18:26:07.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:06 smithi028 ceph-mon[39220]: 3.4 scrub ok 2024-09-10T18:26:07.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:06 smithi028 ceph-mon[39220]: 2.1 scrub starts 2024-09-10T18:26:07.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:07.108 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:07.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:07.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:26:07.142 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:26:07.164 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:06 smithi204 ceph-mon[32357]: 3.4 scrub starts 2024-09-10T18:26:07.164 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:06 smithi204 ceph-mon[32357]: 3.4 scrub ok 2024-09-10T18:26:07.164 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:06 smithi204 ceph-mon[32357]: 2.1 scrub starts 2024-09-10T18:26:07.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:07.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:07.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:07.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:07.417 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:07.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:07.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:07.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:07.564 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:07.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:07.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:07.614 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:26:07.614 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:26:07.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.614 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:163038 7.5 7:a6594e7d:::benchmark_data_smithi028_77856_object41938:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T18:26:07.615 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:07.567297+0000", 2024-09-10T18:26:07.615 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0086469960000000005, 2024-09-10T18:26:07.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086689149999999993, 2024-09-10T18:26:07.615 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:07.615 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:26:07.615 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:07.615 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:07.615 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:07.615 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 163038 2024-09-10T18:26:07.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.615 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:07.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:07.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.567297+0000", 2024-09-10T18:26:07.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:07.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:07.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.567297+0000", 2024-09-10T18:26:07.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:07.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:07.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.567295+0000", 2024-09-10T18:26:07.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:26:07.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:07.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.567304+0000", 2024-09-10T18:26:07.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3890000000000005e-06 2024-09-10T18:26:07.617 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:07.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.567314+0000", 2024-09-10T18:26:07.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0317e-05 2024-09-10T18:26:07.617 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:07.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.567318+0000", 2024-09-10T18:26:07.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2799999999999997e-06 2024-09-10T18:26:07.617 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.618 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:07.618 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.567352+0000", 2024-09-10T18:26:07.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4028e-05 2024-09-10T18:26:07.618 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.618 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.618 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:07.618 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.567425+0000", 2024-09-10T18:26:07.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2774999999999994e-05 2024-09-10T18:26:07.618 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.618 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.618 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2", 2024-09-10T18:26:07.618 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.567492+0000", 2024-09-10T18:26:07.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7288999999999999e-05 2024-09-10T18:26:07.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.619 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:07.619 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.569680+0000", 2024-09-10T18:26:07.619 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021881299999999999 2024-09-10T18:26:07.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.619 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:07.619 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.573267+0000", 2024-09-10T18:26:07.619 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035860250000000001 2024-09-10T18:26:07.619 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:07.619 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:07.619 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:07.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.620 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:163041 7.5 7:ae8e1c7c:::benchmark_data_smithi028_77856_object41941:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T18:26:07.620 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:07.569796+0000", 2024-09-10T18:26:07.620 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.006148232, 2024-09-10T18:26:07.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0062240419999999999, 2024-09-10T18:26:07.620 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:07.620 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:26:07.620 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:07.620 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:07.620 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:07.620 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 163041 2024-09-10T18:26:07.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.620 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:07.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:07.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.569796+0000", 2024-09-10T18:26:07.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:07.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:07.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.569796+0000", 2024-09-10T18:26:07.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:07.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:07.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.569794+0000", 2024-09-10T18:26:07.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:26:07.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:07.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.569800+0000", 2024-09-10T18:26:07.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8170000000000004e-06 2024-09-10T18:26:07.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:07.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.569812+0000", 2024-09-10T18:26:07.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1649e-05 2024-09-10T18:26:07.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:07.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.569818+0000", 2024-09-10T18:26:07.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9359999999999996e-06 2024-09-10T18:26:07.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:07.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.569852+0000", 2024-09-10T18:26:07.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4099e-05 2024-09-10T18:26:07.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:07.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.569964+0000", 2024-09-10T18:26:07.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011188099999999999 2024-09-10T18:26:07.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2", 2024-09-10T18:26:07.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.570073+0000", 2024-09-10T18:26:07.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000109409 2024-09-10T18:26:07.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.624 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.624 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:26:07.624 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.572065+0000", 2024-09-10T18:26:07.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019914960000000002 2024-09-10T18:26:07.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.624 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.624 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:07.624 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.574458+0000", 2024-09-10T18:26:07.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023930549999999998 2024-09-10T18:26:07.624 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:07.624 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:07.624 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:07.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.625 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:163048 7.5 7:a3c48c9a:::benchmark_data_smithi028_77856_object41948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T18:26:07.625 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:07.574908+0000", 2024-09-10T18:26:07.625 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0010355589999999999, 2024-09-10T18:26:07.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001157005, 2024-09-10T18:26:07.625 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:26:07.625 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:26:07.625 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:26:07.625 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:26:07.625 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:07.625 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 163048 2024-09-10T18:26:07.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.625 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:26:07.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:26:07.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.574908+0000", 2024-09-10T18:26:07.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:07.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:26:07.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.574908+0000", 2024-09-10T18:26:07.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:26:07.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:26:07.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.574906+0000", 2024-09-10T18:26:07.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:26:07.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:26:07.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.574913+0000", 2024-09-10T18:26:07.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6359999999999997e-06 2024-09-10T18:26:07.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:26:07.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.574927+0000", 2024-09-10T18:26:07.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3686000000000001e-05 2024-09-10T18:26:07.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:26:07.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.574933+0000", 2024-09-10T18:26:07.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1369999999999999e-06 2024-09-10T18:26:07.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:26:07.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.574980+0000", 2024-09-10T18:26:07.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6916999999999998e-05 2024-09-10T18:26:07.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:26:07.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.575109+0000", 2024-09-10T18:26:07.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000129541 2024-09-10T18:26:07.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:26:07.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:26:07.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2", 2024-09-10T18:26:07.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:07.575243+0000", 2024-09-10T18:26:07.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000133891 2024-09-10T18:26:07.629 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:07.629 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:26:07.629 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:07.629 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:26:07.629 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-09-10T18:26:07.629 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 3 2024-09-10T18:26:07.629 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:26:07.666 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:07.691 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:07.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:07.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:07.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:07.831 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:07.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:07.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:07.933 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:07.958 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:07.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:07.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:08.059 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:08.082 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:08.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:08.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:08.102 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-10T18:26:06.815135+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5353904372,"num_objects":81882,"num_object_clones":0,"num_object_copies":245646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81882,"num_whiteouts":0,"num_read":1112,"num_read_kb":969,"num_write":161053,"num_write_kb":7751880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1181094,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87163,"ondisk_log_size":87163,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":439,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16574712,"kb_used_data":14141560,"kb_used_omap":13,"kb_used_meta":2432818,"kb_avail":733386504,"statfs":{"total":767960285184,"available":750987780096,"internally_reserved":0,"allocated":14480957440,"data_stored":14477074746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2491206336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,8,6,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-691535872,"num_objects":-10552,"num_object_clones":0,"num_object_copies":-31656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10552,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10552,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.884417"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":91,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.944247+0000","last_change":"2024-09-10T18:23:54.884619+0000","last_active":"2024-09-10T18:25:56.944247+0000","last_peered":"2024-09-10T18:25:56.944247+0000","last_clean":"2024-09-10T18:25:56.944247+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:25:56.944247+0000","last_undegraded":"2024-09-10T18:25:56.944247+0000","last_fullsized":"2024-09-10T18:25:56.944247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:54.884551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:23:54.884551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067550040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":112,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:56.933689+0000","last_change":"2024-09-10T18:23:28.805926+0000","last_active":"2024-09-10T18:25:56.933689+0000","last_peered":"2024-09-10T18:25:56.933689+0000","last_clean":"2024-09-10T18:23:26.785602+0000","last_became_active":"2024-09-10T18:23:28.805926+0000","last_became_peered":"2024-09-10T18:23:28.805926+0000","last_unstale":"2024-09-10T18:25:56.933689+0000","last_undegraded":"2024-09-10T18:23:28.798987+0000","last_fullsized":"2024-09-10T18:23:28.798888+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:23:26.523317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:23:26.523317+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001877617,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":96,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.934667+0000","last_change":"2024-09-10T18:25:00.127275+0000","last_active":"2024-09-10T18:25:56.934667+0000","last_peered":"2024-09-10T18:25:56.934667+0000","last_clean":"2024-09-10T18:25:56.934667+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:25:56.934667+0000","last_undegraded":"2024-09-10T18:25:56.934667+0000","last_fullsized":"2024-09-10T18:25:56.934667+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:00.127195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:25:00.127195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:00.650428+0000","scrub_duration":0.0089727910000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":95,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.933791+0000","last_change":"2024-09-10T18:25:03.904400+0000","last_active":"2024-09-10T18:25:56.933791+0000","last_peered":"2024-09-10T18:25:56.933791+0000","last_clean":"2024-09-10T18:25:56.933791+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:25:56.933791+0000","last_undegraded":"2024-09-10T18:25:56.933791+0000","last_fullsized":"2024-09-10T18:25:56.933791+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:03.904322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:25:03.904322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:14.199169+0000","scrub_duration":0.0047095419999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":93,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.946381+0000","last_change":"2024-09-10T18:23:28.797260+0000","last_active":"2024-09-10T18:25:56.946381+0000","last_peered":"2024-09-10T18:25:56.946381+0000","last_clean":"2024-09-10T18:23:26.786856+0000","last_became_active":"2024-09-10T18:23:28.797260+0000","last_became_peered":"2024-09-10T18:23:28.797260+0000","last_unstale":"2024-09-10T18:25:56.946381+0000","last_undegraded":"2024-09-10T18:25:56.946381+0000","last_fullsized":"2024-09-10T18:23:28.794737+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:13.246242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:23:13.246242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001386579,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":99,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.947116+0000","last_change":"2024-09-10T18:23:28.801171+0000","last_active":"2024-09-10T18:25:56.947116+0000","last_peered":"2024-09-10T18:25:56.947116+0000","last_clean":"2024-09-10T18:23:26.787861+0000","last_became_active":"2024-09-10T18:23:28.801171+0000","last_became_peered":"2024-09-10T18:23:28.801171+0000","last_unstale":"2024-09-10T18:25:56.947116+0000","last_undegraded":"2024-09-10T18:25:56.947116+0000","last_fullsized":"2024-09-10T18:23:28.796365+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:06.424889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:23:06.424889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001806062,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"123'22","reported_seq":132,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.948985+0000","last_change":"2024-09-10T18:25:13.118573+0000","last_active":"2024-09-10T18:25:56.948985+0000","last_peered":"2024-09-10T18:25:56.948985+0000","last_clean":"2024-09-10T18:25:56.948985+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:25:56.948985+0000","last_undegraded":"2024-09-10T18:25:56.948985+0000","last_fullsized":"2024-09-10T18:25:56.948985+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'22","last_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:25:13.118335+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:39.185807+0000","scrub_duration":0.023175792000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":94,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.933618+0000","last_change":"2024-09-10T18:23:28.805308+0000","last_active":"2024-09-10T18:25:56.933618+0000","last_peered":"2024-09-10T18:25:56.933618+0000","last_clean":"2024-09-10T18:23:26.785604+0000","last_became_active":"2024-09-10T18:23:28.805308+0000","last_became_peered":"2024-09-10T18:23:28.805308+0000","last_unstale":"2024-09-10T18:25:56.933618+0000","last_undegraded":"2024-09-10T18:25:56.933618+0000","last_fullsized":"2024-09-10T18:23:28.798562+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:23:04.633647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012299979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":98,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.943708+0000","last_change":"2024-09-10T18:23:28.804349+0000","last_active":"2024-09-10T18:25:56.943708+0000","last_peered":"2024-09-10T18:25:56.943708+0000","last_clean":"2024-09-10T18:23:26.792477+0000","last_became_active":"2024-09-10T18:23:28.804349+0000","last_became_peered":"2024-09-10T18:23:28.804349+0000","last_unstale":"2024-09-10T18:25:56.943708+0000","last_undegraded":"2024-09-10T18:25:56.943708+0000","last_fullsized":"2024-09-10T18:23:28.799958+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:26.455933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:23:26.455933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014302690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":115,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.933939+0000","last_change":"2024-09-10T18:24:21.679101+0000","last_active":"2024-09-10T18:25:56.933939+0000","last_peered":"2024-09-10T18:25:56.933939+0000","last_clean":"2024-09-10T18:25:56.933939+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:25:56.933939+0000","last_undegraded":"2024-09-10T18:25:56.933939+0000","last_fullsized":"2024-09-10T18:25:56.933939+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:24:21.679046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:24:21.679046+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099387759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":97,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.944674+0000","last_change":"2024-09-10T18:24:23.393847+0000","last_active":"2024-09-10T18:25:56.944674+0000","last_peered":"2024-09-10T18:25:56.944674+0000","last_clean":"2024-09-10T18:25:56.944674+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:25:56.944674+0000","last_undegraded":"2024-09-10T18:25:56.944674+0000","last_fullsized":"2024-09-10T18:25:56.944674+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:23.393786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:24:23.393786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051571780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":95,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.942260+0000","last_change":"2024-09-10T18:24:01.044631+0000","last_active":"2024-09-10T18:25:56.942260+0000","last_peered":"2024-09-10T18:25:56.942260+0000","last_clean":"2024-09-10T18:25:56.942260+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:25:56.942260+0000","last_undegraded":"2024-09-10T18:25:56.942260+0000","last_fullsized":"2024-09-10T18:25:56.942260+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:01.044532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:24:01.044532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052733060000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":101,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.943104+0000","last_change":"2024-09-10T18:24:10.567100+0000","last_active":"2024-09-10T18:25:56.943104+0000","last_peered":"2024-09-10T18:25:56.943104+0000","last_clean":"2024-09-10T18:25:56.943104+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:25:56.943104+0000","last_undegraded":"2024-09-10T18:25:56.943104+0000","last_fullsized":"2024-09-10T18:25:56.943104+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:10.567051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:24:10.567051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055138289999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"85'7","reported_seq":127,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:59.870140+0000","last_change":"2024-09-10T18:25:59.870084+0000","last_active":"2024-09-10T18:25:59.870140+0000","last_peered":"2024-09-10T18:25:59.870140+0000","last_clean":"2024-09-10T18:25:59.870140+0000","last_became_active":"2024-09-10T18:21:33.615429+0000","last_became_peered":"2024-09-10T18:21:33.615429+0000","last_unstale":"2024-09-10T18:25:59.870140+0000","last_undegraded":"2024-09-10T18:25:59.870140+0000","last_fullsized":"2024-09-10T18:25:59.870140+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'7","last_scrub_stamp":"2024-09-10T18:25:59.869847+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:25:59.869847+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:27:09.580064+0000","scrub_duration":0.013127822000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":97,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.943261+0000","last_change":"2024-09-10T18:24:59.842362+0000","last_active":"2024-09-10T18:25:56.943261+0000","last_peered":"2024-09-10T18:25:56.943261+0000","last_clean":"2024-09-10T18:25:56.943261+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:25:56.943261+0000","last_undegraded":"2024-09-10T18:25:56.943261+0000","last_fullsized":"2024-09-10T18:25:56.943261+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:59.842312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:24:59.842312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:11.513265+0000","scrub_duration":0.0018424279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":98,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:57.571563+0000","last_change":"2024-09-10T18:25:57.571498+0000","last_active":"2024-09-10T18:25:57.571563+0000","last_peered":"2024-09-10T18:25:57.571563+0000","last_clean":"2024-09-10T18:25:57.571563+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:25:57.571563+0000","last_undegraded":"2024-09-10T18:25:57.571563+0000","last_fullsized":"2024-09-10T18:25:57.571563+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:57.571422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:18.504364+0000","last_clean_scrub_stamp":"2024-09-10T18:25:57.571422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:27:06.468976+0000","scrub_duration":0.0033045639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":92,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.934423+0000","last_change":"2024-09-10T18:23:28.801995+0000","last_active":"2024-09-10T18:25:56.934423+0000","last_peered":"2024-09-10T18:25:56.934423+0000","last_clean":"2024-09-10T18:23:26.786526+0000","last_became_active":"2024-09-10T18:23:28.801995+0000","last_became_peered":"2024-09-10T18:23:28.801995+0000","last_unstale":"2024-09-10T18:25:56.934423+0000","last_undegraded":"2024-09-10T18:25:56.934423+0000","last_fullsized":"2024-09-10T18:23:28.797275+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:10.495800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:23:10.495800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001201398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":91,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.943638+0000","last_change":"2024-09-10T18:23:53.901181+0000","last_active":"2024-09-10T18:25:56.943638+0000","last_peered":"2024-09-10T18:25:56.943638+0000","last_clean":"2024-09-10T18:25:56.943638+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:25:56.943638+0000","last_undegraded":"2024-09-10T18:25:56.943638+0000","last_fullsized":"2024-09-10T18:25:56.943638+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:53.901072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:23:53.901072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054409159999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":134,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-10T18:26:01.284367+0000","last_change":"2024-09-10T18:26:01.284314+0000","last_active":"2024-09-10T18:26:01.284367+0000","last_peered":"2024-09-10T18:26:01.284367+0000","last_clean":"2024-09-10T18:26:01.284367+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:26:01.284367+0000","last_undegraded":"2024-09-10T18:26:01.284367+0000","last_fullsized":"2024-09-10T18:26:01.284367+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-10T18:26:01.284214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:01.284214+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:27:06.408310+0000","scrub_duration":0.016537554999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":96,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.935698+0000","last_change":"2024-09-10T18:24:10.105507+0000","last_active":"2024-09-10T18:25:56.935698+0000","last_peered":"2024-09-10T18:25:56.935698+0000","last_clean":"2024-09-10T18:25:56.935698+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:25:56.935698+0000","last_undegraded":"2024-09-10T18:25:56.935698+0000","last_fullsized":"2024-09-10T18:25:56.935698+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:10.105431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:24:10.105431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005163529,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":93,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.947856+0000","last_change":"2024-09-10T18:23:28.807662+0000","last_active":"2024-09-10T18:25:56.947856+0000","last_peered":"2024-09-10T18:25:56.947856+0000","last_clean":"2024-09-10T18:23:26.786042+0000","last_became_active":"2024-09-10T18:23:28.807662+0000","last_became_peered":"2024-09-10T18:23:28.807662+0000","last_unstale":"2024-09-10T18:25:56.947856+0000","last_undegraded":"2024-09-10T18:25:56.947856+0000","last_fullsized":"2024-09-10T18:23:28.798523+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:04.413172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:23:04.413172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001699489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":101,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.934011+0000","last_change":"2024-09-10T18:24:19.654382+0000","last_active":"2024-09-10T18:25:56.934011+0000","last_peered":"2024-09-10T18:25:56.934011+0000","last_clean":"2024-09-10T18:25:56.934011+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:25:56.934011+0000","last_undegraded":"2024-09-10T18:25:56.934011+0000","last_fullsized":"2024-09-10T18:25:56.934011+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:19.654311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:24:19.654311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01201637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"88'17","reported_seq":128,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.946047+0000","last_change":"2024-09-10T18:25:00.147826+0000","last_active":"2024-09-10T18:25:56.946047+0000","last_peered":"2024-09-10T18:25:56.946047+0000","last_clean":"2024-09-10T18:25:56.946047+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:25:56.946047+0000","last_undegraded":"2024-09-10T18:25:56.946047+0000","last_fullsized":"2024-09-10T18:25:56.946047+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-09-10T18:25:00.147710+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:25:00.147710+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:27.292655+0000","scrub_duration":0.0072101989999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":97,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.935045+0000","last_change":"2024-09-10T18:24:10.653546+0000","last_active":"2024-09-10T18:25:56.935045+0000","last_peered":"2024-09-10T18:25:56.935045+0000","last_clean":"2024-09-10T18:25:56.935045+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:25:56.935045+0000","last_undegraded":"2024-09-10T18:25:56.935045+0000","last_fullsized":"2024-09-10T18:25:56.935045+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:10.653472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:24:10.653472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006191879,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":97,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.942990+0000","last_change":"2024-09-10T18:24:58.806296+0000","last_active":"2024-09-10T18:25:56.942990+0000","last_peered":"2024-09-10T18:25:56.942990+0000","last_clean":"2024-09-10T18:25:56.942990+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:25:56.942990+0000","last_undegraded":"2024-09-10T18:25:56.942990+0000","last_fullsized":"2024-09-10T18:25:56.942990+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:24:58.806208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:24:58.806208+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:22.822282+0000","scrub_duration":0.0075855569999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":100,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.934788+0000","last_change":"2024-09-10T18:24:13.149579+0000","last_active":"2024-09-10T18:25:56.934788+0000","last_peered":"2024-09-10T18:25:56.934788+0000","last_clean":"2024-09-10T18:25:56.934788+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:25:56.934788+0000","last_undegraded":"2024-09-10T18:25:56.934788+0000","last_fullsized":"2024-09-10T18:25:56.934788+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:13.149473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:24:13.149473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007086958,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"86'15","reported_seq":119,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.949845+0000","last_change":"2024-09-10T18:24:23.267965+0000","last_active":"2024-09-10T18:25:56.949845+0000","last_peered":"2024-09-10T18:25:56.949845+0000","last_clean":"2024-09-10T18:25:56.949845+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:25:56.949845+0000","last_undegraded":"2024-09-10T18:25:56.949845+0000","last_fullsized":"2024-09-10T18:25:56.949845+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-09-10T18:24:23.267714+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:24:23.267714+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.020876362999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"96'9","reported_seq":99,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:59.805557+0000","last_change":"2024-09-10T18:23:28.801186+0000","last_active":"2024-09-10T18:25:59.805557+0000","last_peered":"2024-09-10T18:25:59.805557+0000","last_clean":"2024-09-10T18:21:38.779119+0000","last_became_active":"2024-09-10T18:23:28.801186+0000","last_became_peered":"2024-09-10T18:23:28.801186+0000","last_unstale":"2024-09-10T18:25:59.805557+0000","last_undegraded":"2024-09-10T18:23:28.798601+0000","last_fullsized":"2024-09-10T18:23:28.798499+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:22:48.795158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:48.795158+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"131'9921","reported_seq":9966,"reported_epoch":131,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:26:05.054452+0000","last_change":"2024-09-10T18:23:43.900563+0000","last_active":"2024-09-10T18:26:05.054452+0000","last_peered":"2024-09-10T18:26:05.054452+0000","last_clean":"2024-09-10T18:23:25.774520+0000","last_became_active":"2024-09-10T18:23:28.804457+0000","last_became_peered":"2024-09-10T18:23:28.804457+0000","last_unstale":"2024-09-10T18:26:05.054452+0000","last_undegraded":"2024-09-10T18:23:43.883715+0000","last_fullsized":"2024-09-10T18:23:28.795582+0000","mapping_epoch":95,"log_start":"115'5300","ondisk_log_start":"115'5300","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_clean_scrub_stamp":"2024-09-10T18:23:25.774520+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337313792,"num_objects":5147,"num_object_clones":0,"num_object_copies":15441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9921,"num_write_kb":482176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":5147}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"83'22","reported_seq":143,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:26:00.574691+0000","last_change":"2024-09-10T18:26:00.574623+0000","last_active":"2024-09-10T18:26:00.574691+0000","last_peered":"2024-09-10T18:26:00.574691+0000","last_clean":"2024-09-10T18:26:00.574691+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:26:00.574691+0000","last_undegraded":"2024-09-10T18:26:00.574691+0000","last_fullsized":"2024-09-10T18:26:00.574691+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-09-10T18:26:00.574534+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:26:00.574534+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:27:19.164202+0000","scrub_duration":0.0070793189999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":102,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.946529+0000","last_change":"2024-09-10T18:24:57.222017+0000","last_active":"2024-09-10T18:25:56.946529+0000","last_peered":"2024-09-10T18:25:56.946529+0000","last_clean":"2024-09-10T18:25:56.946529+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:25:56.946529+0000","last_undegraded":"2024-09-10T18:25:56.946529+0000","last_fullsized":"2024-09-10T18:25:56.946529+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:57.221924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:24:57.221924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:02.518460+0000","scrub_duration":0.011341357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":162,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:59.805575+0000","last_change":"2024-09-10T18:25:09.932129+0000","last_active":"2024-09-10T18:25:59.805575+0000","last_peered":"2024-09-10T18:25:59.805575+0000","last_clean":"2024-09-10T18:25:59.805575+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:25:59.805575+0000","last_undegraded":"2024-09-10T18:25:59.805575+0000","last_fullsized":"2024-09-10T18:25:59.805575+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:25:09.932039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:25:09.932039+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:15.205858+0000","scrub_duration":0.0033245879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":95,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.943369+0000","last_change":"2024-09-10T18:25:01.375917+0000","last_active":"2024-09-10T18:25:56.943369+0000","last_peered":"2024-09-10T18:25:56.943369+0000","last_clean":"2024-09-10T18:25:56.943369+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:25:56.943369+0000","last_undegraded":"2024-09-10T18:25:56.943369+0000","last_fullsized":"2024-09-10T18:25:56.943369+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:01.375849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:25:01.375849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:01.679282+0000","scrub_duration":0.012674467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"131'10005","reported_seq":10051,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-10T18:26:05.031418+0000","last_change":"2024-09-10T18:23:26.795283+0000","last_active":"2024-09-10T18:26:05.031418+0000","last_peered":"2024-09-10T18:26:05.031418+0000","last_clean":"2024-09-10T18:26:05.031418+0000","last_became_active":"2024-09-10T18:23:26.795002+0000","last_became_peered":"2024-09-10T18:23:26.795002+0000","last_unstale":"2024-09-10T18:26:05.031418+0000","last_undegraded":"2024-09-10T18:26:05.031418+0000","last_fullsized":"2024-09-10T18:26:05.031418+0000","mapping_epoch":93,"log_start":"115'5300","ondisk_log_start":"115'5300","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_clean_scrub_stamp":"2024-09-10T18:23:25.774520+0000","objects_scrubbed":0,"log_size":4705,"log_dups_size":0,"ondisk_log_size":4705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339542016,"num_objects":5181,"num_object_clones":0,"num_object_copies":15543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10005,"num_write_kb":485952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":132,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.946322+0000","last_change":"2024-09-10T18:25:12.136511+0000","last_active":"2024-09-10T18:25:56.946322+0000","last_peered":"2024-09-10T18:25:56.946322+0000","last_clean":"2024-09-10T18:25:56.946322+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:25:56.946322+0000","last_undegraded":"2024-09-10T18:25:56.946322+0000","last_fullsized":"2024-09-10T18:25:56.946322+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-10T18:25:12.136409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:25:12.136409+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:22.484341+0000","scrub_duration":0.0047902860000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":102,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.948335+0000","last_change":"2024-09-10T18:25:04.166966+0000","last_active":"2024-09-10T18:25:56.948335+0000","last_peered":"2024-09-10T18:25:56.948335+0000","last_clean":"2024-09-10T18:25:56.948335+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:25:56.948335+0000","last_undegraded":"2024-09-10T18:25:56.948335+0000","last_fullsized":"2024-09-10T18:25:56.948335+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:04.166734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:25:04.166734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:22.641066+0000","scrub_duration":0.015597358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":97,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.933999+0000","last_change":"2024-09-10T18:24:02.729168+0000","last_active":"2024-09-10T18:25:56.933999+0000","last_peered":"2024-09-10T18:25:56.933999+0000","last_clean":"2024-09-10T18:25:56.933999+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:25:56.933999+0000","last_undegraded":"2024-09-10T18:25:56.933999+0000","last_fullsized":"2024-09-10T18:25:56.933999+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:02.729115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:24:02.729115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004997619,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":95,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.933909+0000","last_change":"2024-09-10T18:25:08.950736+0000","last_active":"2024-09-10T18:25:56.933909+0000","last_peered":"2024-09-10T18:25:56.933909+0000","last_clean":"2024-09-10T18:25:56.933909+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:25:56.933909+0000","last_undegraded":"2024-09-10T18:25:56.933909+0000","last_fullsized":"2024-09-10T18:25:56.933909+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:08.950677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:25:08.950677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:16.921549+0000","scrub_duration":0.010251718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"131'13459","reported_seq":13798,"reported_epoch":131,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T18:26:05.058738+0000","last_change":"2024-09-10T18:26:03.114641+0000","last_active":"2024-09-10T18:26:05.058738+0000","last_peered":"2024-09-10T18:26:05.058738+0000","last_clean":"2024-09-10T18:26:05.058738+0000","last_became_active":"2024-09-10T18:23:26.799779+0000","last_became_peered":"2024-09-10T18:23:26.799779+0000","last_unstale":"2024-09-10T18:26:05.058738+0000","last_undegraded":"2024-09-10T18:26:05.058738+0000","last_fullsized":"2024-09-10T18:26:05.058738+0000","mapping_epoch":93,"log_start":"123'8800","ondisk_log_start":"123'8800","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_clean_scrub_stamp":"2024-09-10T18:23:25.774520+0000","objects_scrubbed":3161,"log_size":4659,"log_dups_size":0,"ondisk_log_size":4659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335544320,"num_objects":5120,"num_object_clones":0,"num_object_copies":15360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10042,"num_write_kb":485184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":117,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:56.934021+0000","last_change":"2024-09-10T18:23:28.803009+0000","last_active":"2024-09-10T18:25:56.934021+0000","last_peered":"2024-09-10T18:25:56.934021+0000","last_clean":"2024-09-10T18:23:26.786331+0000","last_became_active":"2024-09-10T18:23:28.803009+0000","last_became_peered":"2024-09-10T18:23:28.803009+0000","last_unstale":"2024-09-10T18:25:56.934021+0000","last_undegraded":"2024-09-10T18:23:28.797578+0000","last_fullsized":"2024-09-10T18:23:28.797480+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-10T18:22:37.322094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:37.322094+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001716817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":101,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.950065+0000","last_change":"2024-09-10T18:25:28.438845+0000","last_active":"2024-09-10T18:25:56.950065+0000","last_peered":"2024-09-10T18:25:56.950065+0000","last_clean":"2024-09-10T18:25:56.950065+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:25:56.950065+0000","last_undegraded":"2024-09-10T18:25:56.950065+0000","last_fullsized":"2024-09-10T18:25:56.950065+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:28.438755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:25:28.438755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:39.338235+0000","scrub_duration":0.019825987,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":162,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-10T18:26:00.609887+0000","last_change":"2024-09-10T18:25:11.152465+0000","last_active":"2024-09-10T18:26:00.609887+0000","last_peered":"2024-09-10T18:26:00.609887+0000","last_clean":"2024-09-10T18:26:00.609887+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:26:00.609887+0000","last_undegraded":"2024-09-10T18:26:00.609887+0000","last_fullsized":"2024-09-10T18:26:00.609887+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:25:11.152368+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:25:11.152368+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:25.632166+0000","scrub_duration":0.011232113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":92,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.943494+0000","last_change":"2024-09-10T18:23:28.800616+0000","last_active":"2024-09-10T18:25:56.943494+0000","last_peered":"2024-09-10T18:25:56.943494+0000","last_clean":"2024-09-10T18:23:26.789612+0000","last_became_active":"2024-09-10T18:23:28.800616+0000","last_became_peered":"2024-09-10T18:23:28.800616+0000","last_unstale":"2024-09-10T18:25:56.943494+0000","last_undegraded":"2024-09-10T18:25:56.943494+0000","last_fullsized":"2024-09-10T18:23:28.794650+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:02.011610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:02.011610+0000","last_clean_scrub_stamp":"2024-09-10T18:23:02.011610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017054329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"131'9901","reported_seq":9954,"reported_epoch":131,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:26:04.613991+0000","last_change":"2024-09-10T18:23:43.888545+0000","last_active":"2024-09-10T18:26:04.613991+0000","last_peered":"2024-09-10T18:26:04.613991+0000","last_clean":"2024-09-10T18:23:26.792780+0000","last_became_active":"2024-09-10T18:23:28.801622+0000","last_became_peered":"2024-09-10T18:23:28.801622+0000","last_unstale":"2024-09-10T18:26:04.613991+0000","last_undegraded":"2024-09-10T18:23:43.888362+0000","last_fullsized":"2024-09-10T18:23:28.796996+0000","mapping_epoch":95,"log_start":"115'5100","ondisk_log_start":"115'5100","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_clean_scrub_stamp":"2024-09-10T18:23:25.774520+0000","objects_scrubbed":0,"log_size":4801,"log_dups_size":0,"ondisk_log_size":4801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339017728,"num_objects":5173,"num_object_clones":0,"num_object_copies":15519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9901,"num_write_kb":482368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":5173}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"123'15","reported_seq":128,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.934817+0000","last_change":"2024-09-10T18:25:18.356319+0000","last_active":"2024-09-10T18:25:56.934817+0000","last_peered":"2024-09-10T18:25:56.934817+0000","last_clean":"2024-09-10T18:25:56.934817+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:25:56.934817+0000","last_undegraded":"2024-09-10T18:25:56.934817+0000","last_fullsized":"2024-09-10T18:25:56.934817+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'15","last_scrub_stamp":"2024-09-10T18:25:18.356218+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:25:18.356218+0000","last_clean_scrub_stamp":"2024-09-10T18:25:18.356218+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:21.718455+0000","scrub_duration":0.056607008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":101,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.943797+0000","last_change":"2024-09-10T18:25:13.598068+0000","last_active":"2024-09-10T18:25:56.943797+0000","last_peered":"2024-09-10T18:25:56.943797+0000","last_clean":"2024-09-10T18:25:56.943797+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:25:56.943797+0000","last_undegraded":"2024-09-10T18:25:56.943797+0000","last_fullsized":"2024-09-10T18:25:56.943797+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:13.598014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:25:13.598014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:32.667159+0000","scrub_duration":0.0056627420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"96'9","reported_seq":148,"reported_epoch":131,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:26:00.616361+0000","last_change":"2024-09-10T18:23:28.805206+0000","last_active":"2024-09-10T18:26:00.616361+0000","last_peered":"2024-09-10T18:26:00.616361+0000","last_clean":"2024-09-10T18:23:26.791072+0000","last_became_active":"2024-09-10T18:23:28.805206+0000","last_became_peered":"2024-09-10T18:23:28.805206+0000","last_unstale":"2024-09-10T18:26:00.616361+0000","last_undegraded":"2024-09-10T18:23:28.802978+0000","last_fullsized":"2024-09-10T18:23:28.802879+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:21:34.598351+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":95,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.942456+0000","last_change":"2024-09-10T18:23:56.902487+0000","last_active":"2024-09-10T18:25:56.942456+0000","last_peered":"2024-09-10T18:25:56.942456+0000","last_clean":"2024-09-10T18:25:56.942456+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:25:56.942456+0000","last_undegraded":"2024-09-10T18:25:56.942456+0000","last_fullsized":"2024-09-10T18:25:56.942456+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:56.902407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:23:56.902407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00728891,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"131'10172","reported_seq":10527,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-10T18:26:04.702688+0000","last_change":"2024-09-10T18:25:17.747403+0000","last_active":"2024-09-10T18:26:04.702688+0000","last_peered":"2024-09-10T18:26:04.702688+0000","last_clean":"2024-09-10T18:26:04.702688+0000","last_became_active":"2024-09-10T18:23:26.799625+0000","last_became_peered":"2024-09-10T18:23:26.799625+0000","last_unstale":"2024-09-10T18:26:04.702688+0000","last_undegraded":"2024-09-10T18:26:04.702688+0000","last_fullsized":"2024-09-10T18:26:04.702688+0000","mapping_epoch":93,"log_start":"109'4400","ondisk_log_start":"109'4400","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'7812","last_scrub_stamp":"2024-09-10T18:25:17.747314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_clean_scrub_stamp":"2024-09-10T18:25:17.747314+0000","objects_scrubbed":7449,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:23.586247+0000","scrub_duration":2.1877917949999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329515008,"num_objects":5028,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10172,"num_write_kb":486400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":135,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.943430+0000","last_change":"2024-09-10T18:25:05.446500+0000","last_active":"2024-09-10T18:25:56.943430+0000","last_peered":"2024-09-10T18:25:56.943430+0000","last_clean":"2024-09-10T18:25:56.943430+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:25:56.943430+0000","last_undegraded":"2024-09-10T18:25:56.943430+0000","last_fullsized":"2024-09-10T18:25:56.943430+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-10T18:25:05.446411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:25:05.446411+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:15.371393+0000","scrub_duration":0.016906469,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":90,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.938837+0000","last_change":"2024-09-10T18:23:59.130237+0000","last_active":"2024-09-10T18:25:56.938837+0000","last_peered":"2024-09-10T18:25:56.938837+0000","last_clean":"2024-09-10T18:25:56.938837+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:25:56.938837+0000","last_undegraded":"2024-09-10T18:25:56.938837+0000","last_fullsized":"2024-09-10T18:25:56.938837+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:59.130141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:23:59.130141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006733714,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":45,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.942861+0000","last_change":"2024-09-10T18:23:28.805342+0000","last_active":"2024-09-10T18:25:56.942861+0000","last_peered":"2024-09-10T18:25:56.942861+0000","last_clean":"2024-09-10T18:21:34.598351+0000","last_became_active":"2024-09-10T18:23:28.805342+0000","last_became_peered":"2024-09-10T18:23:28.805342+0000","last_unstale":"2024-09-10T18:25:56.942861+0000","last_undegraded":"2024-09-10T18:25:56.942861+0000","last_fullsized":"2024-09-10T18:23:28.801625+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:38.853839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:38.853839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":81,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.934562+0000","last_change":"2024-09-10T18:23:28.805722+0000","last_active":"2024-09-10T18:25:56.934562+0000","last_peered":"2024-09-10T18:25:56.934562+0000","last_clean":"2024-09-10T18:23:26.789916+0000","last_became_active":"2024-09-10T18:23:28.805722+0000","last_became_peered":"2024-09-10T18:23:28.805722+0000","last_unstale":"2024-09-10T18:25:56.934562+0000","last_undegraded":"2024-09-10T18:25:56.934562+0000","last_fullsized":"2024-09-10T18:23:28.799176+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"131'10139","reported_seq":10184,"reported_epoch":131,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:26:05.136406+0000","last_change":"2024-09-10T18:23:43.885244+0000","last_active":"2024-09-10T18:26:05.136406+0000","last_peered":"2024-09-10T18:26:05.136406+0000","last_clean":"2024-09-10T18:23:25.774520+0000","last_became_active":"2024-09-10T18:23:28.807807+0000","last_became_peered":"2024-09-10T18:23:28.807807+0000","last_unstale":"2024-09-10T18:26:05.136406+0000","last_undegraded":"2024-09-10T18:23:43.880096+0000","last_fullsized":"2024-09-10T18:23:28.803547+0000","mapping_epoch":95,"log_start":"111'4500","ondisk_log_start":"111'4500","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_clean_scrub_stamp":"2024-09-10T18:23:25.774520+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337182720,"num_objects":5145,"num_object_clones":0,"num_object_copies":15435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10139,"num_write_kb":489088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":5145}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"90'29","reported_seq":119,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:59.805512+0000","last_change":"2024-09-10T18:25:00.413166+0000","last_active":"2024-09-10T18:25:59.805512+0000","last_peered":"2024-09-10T18:25:59.805512+0000","last_clean":"2024-09-10T18:25:59.805512+0000","last_became_active":"2024-09-10T18:23:10.067049+0000","last_became_peered":"2024-09-10T18:23:10.067049+0000","last_unstale":"2024-09-10T18:25:59.805512+0000","last_undegraded":"2024-09-10T18:25:59.805512+0000","last_fullsized":"2024-09-10T18:25:59.805512+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'29","last_scrub_stamp":"2024-09-10T18:25:00.413073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:25:00.413073+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:01.147119+0000","scrub_duration":0.006016258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":98,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.934289+0000","last_change":"2024-09-10T18:23:28.804769+0000","last_active":"2024-09-10T18:25:56.934289+0000","last_peered":"2024-09-10T18:25:56.934289+0000","last_clean":"2024-09-10T18:23:26.786105+0000","last_became_active":"2024-09-10T18:23:28.804769+0000","last_became_peered":"2024-09-10T18:23:28.804769+0000","last_unstale":"2024-09-10T18:25:56.934289+0000","last_undegraded":"2024-09-10T18:25:56.934289+0000","last_fullsized":"2024-09-10T18:23:28.798330+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:42.335152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:42.335152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013614720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"62'2","reported_seq":105,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:56.944395+0000","last_change":"2024-09-10T18:23:28.806943+0000","last_active":"2024-09-10T18:25:56.944395+0000","last_peered":"2024-09-10T18:25:56.944395+0000","last_clean":"2024-09-10T18:23:26.790792+0000","last_became_active":"2024-09-10T18:23:28.806943+0000","last_became_peered":"2024-09-10T18:23:28.806943+0000","last_unstale":"2024-09-10T18:25:56.944395+0000","last_undegraded":"2024-09-10T18:23:28.797896+0000","last_fullsized":"2024-09-10T18:23:28.797782+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-10T18:23:10.591298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:23:10.591298+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001471563,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":97,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.945104+0000","last_change":"2024-09-10T18:24:22.274437+0000","last_active":"2024-09-10T18:25:56.945104+0000","last_peered":"2024-09-10T18:25:56.945104+0000","last_clean":"2024-09-10T18:25:56.945104+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:25:56.945104+0000","last_undegraded":"2024-09-10T18:25:56.945104+0000","last_fullsized":"2024-09-10T18:25:56.945104+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:22.272678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:24:22.272678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013576644000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":96,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.946861+0000","last_change":"2024-09-10T18:25:03.137073+0000","last_active":"2024-09-10T18:25:56.946861+0000","last_peered":"2024-09-10T18:25:56.946861+0000","last_clean":"2024-09-10T18:25:56.946861+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:25:56.946861+0000","last_undegraded":"2024-09-10T18:25:56.946861+0000","last_fullsized":"2024-09-10T18:25:56.946861+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:03.136987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:25:03.136987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:24.306126+0000","scrub_duration":0.002943471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"131'10109","reported_seq":10464,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-10T18:26:05.136992+0000","last_change":"2024-09-10T18:25:18.263573+0000","last_active":"2024-09-10T18:26:05.136992+0000","last_peered":"2024-09-10T18:26:05.136992+0000","last_clean":"2024-09-10T18:26:05.136992+0000","last_became_active":"2024-09-10T18:23:26.796388+0000","last_became_peered":"2024-09-10T18:23:26.796388+0000","last_unstale":"2024-09-10T18:26:05.136992+0000","last_undegraded":"2024-09-10T18:26:05.136992+0000","last_fullsized":"2024-09-10T18:26:05.136992+0000","mapping_epoch":93,"log_start":"111'4500","ondisk_log_start":"111'4500","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'7837","last_scrub_stamp":"2024-09-10T18:25:18.263314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_clean_scrub_stamp":"2024-09-10T18:25:18.263314+0000","objects_scrubbed":7482,"log_size":5609,"log_dups_size":0,"ondisk_log_size":5609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:37.550028+0000","scrub_duration":2.9687582959999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336789504,"num_objects":5139,"num_object_clones":0,"num_object_copies":15417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10109,"num_write_kb":487936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":125,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.937051+0000","last_change":"2024-09-10T18:24:20.133536+0000","last_active":"2024-09-10T18:25:56.937051+0000","last_peered":"2024-09-10T18:25:56.937051+0000","last_clean":"2024-09-10T18:25:56.937051+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:25:56.937051+0000","last_undegraded":"2024-09-10T18:25:56.937051+0000","last_fullsized":"2024-09-10T18:25:56.937051+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-10T18:24:20.133443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:24:20.133443+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014537605,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":97,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.934474+0000","last_change":"2024-09-10T18:23:28.806509+0000","last_active":"2024-09-10T18:25:56.934474+0000","last_peered":"2024-09-10T18:25:56.934474+0000","last_clean":"2024-09-10T18:23:26.792066+0000","last_became_active":"2024-09-10T18:23:28.806509+0000","last_became_peered":"2024-09-10T18:23:28.806509+0000","last_unstale":"2024-09-10T18:25:56.934474+0000","last_undegraded":"2024-09-10T18:25:56.934474+0000","last_fullsized":"2024-09-10T18:23:28.799973+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:52.160732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:52.160732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015264289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":107,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.937387+0000","last_change":"2024-09-10T18:24:19.355071+0000","last_active":"2024-09-10T18:25:56.937387+0000","last_peered":"2024-09-10T18:25:56.937387+0000","last_clean":"2024-09-10T18:25:56.937387+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:25:56.937387+0000","last_undegraded":"2024-09-10T18:25:56.937387+0000","last_fullsized":"2024-09-10T18:25:56.937387+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:24:19.354984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:24:19.354984+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005402451,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":107,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-10T18:26:04.278227+0000","last_change":"2024-09-10T18:26:04.278188+0000","last_active":"2024-09-10T18:26:04.278227+0000","last_peered":"2024-09-10T18:26:04.278227+0000","last_clean":"2024-09-10T18:26:04.278227+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:26:04.278227+0000","last_undegraded":"2024-09-10T18:26:04.278227+0000","last_fullsized":"2024-09-10T18:26:04.278227+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:04.278111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:26:04.278111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:27:31.276815+0000","scrub_duration":0.0036508310000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":92,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.944694+0000","last_change":"2024-09-10T18:23:28.801185+0000","last_active":"2024-09-10T18:25:56.944694+0000","last_peered":"2024-09-10T18:25:56.944694+0000","last_clean":"2024-09-10T18:23:26.791143+0000","last_became_active":"2024-09-10T18:23:28.801185+0000","last_became_peered":"2024-09-10T18:23:28.801185+0000","last_unstale":"2024-09-10T18:25:56.944694+0000","last_undegraded":"2024-09-10T18:25:56.944694+0000","last_fullsized":"2024-09-10T18:23:28.795286+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:26.019003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:23:26.019003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015113240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"131'10029","reported_seq":10366,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-10T18:26:04.714799+0000","last_change":"2024-09-10T18:25:06.548444+0000","last_active":"2024-09-10T18:26:04.714799+0000","last_peered":"2024-09-10T18:26:04.714799+0000","last_clean":"2024-09-10T18:26:04.714799+0000","last_became_active":"2024-09-10T18:23:26.796190+0000","last_became_peered":"2024-09-10T18:23:26.796190+0000","last_unstale":"2024-09-10T18:26:04.714799+0000","last_undegraded":"2024-09-10T18:26:04.714799+0000","last_fullsized":"2024-09-10T18:26:04.714799+0000","mapping_epoch":93,"log_start":"112'4800","ondisk_log_start":"112'4800","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'7143","last_scrub_stamp":"2024-09-10T18:25:06.548322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_clean_scrub_stamp":"2024-09-10T18:25:06.548322+0000","objects_scrubbed":7070,"log_size":5229,"log_dups_size":0,"ondisk_log_size":5229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:09.094503+0000","scrub_duration":2.7807513039999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336003072,"num_objects":5127,"num_object_clones":0,"num_object_copies":15381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10029,"num_write_kb":484992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":117,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.949930+0000","last_change":"2024-09-10T18:24:58.387371+0000","last_active":"2024-09-10T18:25:56.949930+0000","last_peered":"2024-09-10T18:25:56.949930+0000","last_clean":"2024-09-10T18:25:56.949930+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:25:56.949930+0000","last_undegraded":"2024-09-10T18:25:56.949930+0000","last_fullsized":"2024-09-10T18:25:56.949930+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:24:58.387169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:24:58.387169+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:18.454509+0000","scrub_duration":0.045705452000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":101,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.942568+0000","last_change":"2024-09-10T18:24:18.269110+0000","last_active":"2024-09-10T18:25:56.942568+0000","last_peered":"2024-09-10T18:25:56.942568+0000","last_clean":"2024-09-10T18:25:56.942568+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:25:56.942568+0000","last_undegraded":"2024-09-10T18:25:56.942568+0000","last_fullsized":"2024-09-10T18:25:56.942568+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:24:18.269034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012120479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":385,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-10T18:26:03.127381+0000","last_change":"2024-09-10T18:25:13.845468+0000","last_active":"2024-09-10T18:26:03.127381+0000","last_peered":"2024-09-10T18:26:03.127381+0000","last_clean":"2024-09-10T18:26:03.127381+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:26:03.127381+0000","last_undegraded":"2024-09-10T18:26:03.127381+0000","last_fullsized":"2024-09-10T18:26:03.127381+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:25:13.845382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:25:13.845382+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:26.900007+0000","scrub_duration":0.0051969970000000001,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":266,"num_read_kb":261,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":100,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.944415+0000","last_change":"2024-09-10T18:25:28.781713+0000","last_active":"2024-09-10T18:25:56.944415+0000","last_peered":"2024-09-10T18:25:56.944415+0000","last_clean":"2024-09-10T18:25:56.944415+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:25:56.944415+0000","last_undegraded":"2024-09-10T18:25:56.944415+0000","last_fullsized":"2024-09-10T18:25:56.944415+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:28.781660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:25:28.781660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:38.508571+0000","scrub_duration":0.069585199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":94,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.936816+0000","last_change":"2024-09-10T18:25:09.189940+0000","last_active":"2024-09-10T18:25:56.936816+0000","last_peered":"2024-09-10T18:25:56.936816+0000","last_clean":"2024-09-10T18:25:56.936816+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:25:56.936816+0000","last_undegraded":"2024-09-10T18:25:56.936816+0000","last_fullsized":"2024-09-10T18:25:56.936816+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:09.189571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:25:09.189571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:23.217656+0000","scrub_duration":0.0086882009999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"131'10052","reported_seq":10139,"reported_epoch":131,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T18:26:05.157836+0000","last_change":"2024-09-10T18:26:04.710083+0000","last_active":"2024-09-10T18:26:05.157836+0000","last_peered":"2024-09-10T18:26:05.157836+0000","last_clean":"2024-09-10T18:26:05.157836+0000","last_became_active":"2024-09-10T18:23:26.800391+0000","last_became_peered":"2024-09-10T18:23:26.800391+0000","last_unstale":"2024-09-10T18:26:05.157836+0000","last_undegraded":"2024-09-10T18:26:05.157836+0000","last_fullsized":"2024-09-10T18:26:05.157836+0000","mapping_epoch":93,"log_start":"111'4400","ondisk_log_start":"111'4400","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_clean_scrub_stamp":"2024-09-10T18:23:25.774520+0000","objects_scrubbed":849,"log_size":5652,"log_dups_size":0,"ondisk_log_size":5652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330432512,"num_objects":5042,"num_object_clones":0,"num_object_copies":15126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10052,"num_write_kb":483008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":79,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:56.934438+0000","last_change":"2024-09-10T18:23:28.800312+0000","last_active":"2024-09-10T18:25:56.934438+0000","last_peered":"2024-09-10T18:25:56.934438+0000","last_clean":"2024-09-10T18:21:38.957059+0000","last_became_active":"2024-09-10T18:23:28.800312+0000","last_became_peered":"2024-09-10T18:23:28.800312+0000","last_unstale":"2024-09-10T18:25:56.934438+0000","last_undegraded":"2024-09-10T18:23:28.797181+0000","last_fullsized":"2024-09-10T18:23:28.797075+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-10T18:23:04.868685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:23:04.868685+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":99,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.947202+0000","last_change":"2024-09-10T18:23:28.800830+0000","last_active":"2024-09-10T18:25:56.947202+0000","last_peered":"2024-09-10T18:25:56.947202+0000","last_clean":"2024-09-10T18:23:26.786263+0000","last_became_active":"2024-09-10T18:23:28.800830+0000","last_became_peered":"2024-09-10T18:23:28.800830+0000","last_unstale":"2024-09-10T18:25:56.947202+0000","last_undegraded":"2024-09-10T18:25:56.947202+0000","last_fullsized":"2024-09-10T18:23:28.794997+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:45.383768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:45.383768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001512844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":132,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:56.943669+0000","last_change":"2024-09-10T18:23:28.803565+0000","last_active":"2024-09-10T18:25:56.943669+0000","last_peered":"2024-09-10T18:25:56.943669+0000","last_clean":"2024-09-10T18:23:26.790061+0000","last_became_active":"2024-09-10T18:23:28.803565+0000","last_became_peered":"2024-09-10T18:23:28.803565+0000","last_unstale":"2024-09-10T18:25:56.943669+0000","last_undegraded":"2024-09-10T18:23:28.795373+0000","last_fullsized":"2024-09-10T18:23:28.795273+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:22:17.024553+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003148245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":97,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.943105+0000","last_change":"2024-09-10T18:25:07.810708+0000","last_active":"2024-09-10T18:25:56.943105+0000","last_peered":"2024-09-10T18:25:56.943105+0000","last_clean":"2024-09-10T18:25:56.943105+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:25:56.943105+0000","last_undegraded":"2024-09-10T18:25:56.943105+0000","last_fullsized":"2024-09-10T18:25:56.943105+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:07.810638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:25:07.810638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:10.637181+0000","scrub_duration":0.009172319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":88,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.944211+0000","last_change":"2024-09-10T18:25:23.725039+0000","last_active":"2024-09-10T18:25:56.944211+0000","last_peered":"2024-09-10T18:25:56.944211+0000","last_clean":"2024-09-10T18:25:56.944211+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:25:56.944211+0000","last_undegraded":"2024-09-10T18:25:56.944211+0000","last_fullsized":"2024-09-10T18:25:56.944211+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:54.576654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:23:54.576654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001978573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"131'9891","reported_seq":9937,"reported_epoch":131,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:26:04.765345+0000","last_change":"2024-09-10T18:23:43.885054+0000","last_active":"2024-09-10T18:26:04.765345+0000","last_peered":"2024-09-10T18:26:04.765345+0000","last_clean":"2024-09-10T18:23:25.774520+0000","last_became_active":"2024-09-10T18:23:28.806665+0000","last_became_peered":"2024-09-10T18:23:28.806665+0000","last_unstale":"2024-09-10T18:26:04.765345+0000","last_undegraded":"2024-09-10T18:23:43.882492+0000","last_fullsized":"2024-09-10T18:23:28.800077+0000","mapping_epoch":95,"log_start":"113'4800","ondisk_log_start":"113'4800","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_clean_scrub_stamp":"2024-09-10T18:23:25.774520+0000","objects_scrubbed":0,"log_size":5091,"log_dups_size":0,"ondisk_log_size":5091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327548952,"num_objects":4999,"num_object_clones":0,"num_object_copies":14997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4999,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9891,"num_write_kb":476417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":4999}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":115,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:56.934576+0000","last_change":"2024-09-10T18:23:28.805647+0000","last_active":"2024-09-10T18:25:56.934576+0000","last_peered":"2024-09-10T18:25:56.934576+0000","last_clean":"2024-09-10T18:23:26.786589+0000","last_became_active":"2024-09-10T18:23:28.805647+0000","last_became_peered":"2024-09-10T18:23:28.805647+0000","last_unstale":"2024-09-10T18:25:56.934576+0000","last_undegraded":"2024-09-10T18:23:28.797701+0000","last_fullsized":"2024-09-10T18:23:28.797608+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:23:18.461181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:23:18.461181+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020669920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":101,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.934216+0000","last_change":"2024-09-10T18:25:13.996941+0000","last_active":"2024-09-10T18:25:56.934216+0000","last_peered":"2024-09-10T18:25:56.934216+0000","last_clean":"2024-09-10T18:25:56.934216+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:25:56.934216+0000","last_undegraded":"2024-09-10T18:25:56.934216+0000","last_fullsized":"2024-09-10T18:25:56.934216+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:13.996848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:25:13.996848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:23.927780+0000","scrub_duration":0.0015007390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":94,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.944025+0000","last_change":"2024-09-10T18:23:28.806919+0000","last_active":"2024-09-10T18:25:56.944025+0000","last_peered":"2024-09-10T18:25:56.944025+0000","last_clean":"2024-09-10T18:23:26.792304+0000","last_became_active":"2024-09-10T18:23:28.806919+0000","last_became_peered":"2024-09-10T18:23:28.806919+0000","last_unstale":"2024-09-10T18:25:56.944025+0000","last_undegraded":"2024-09-10T18:25:56.944025+0000","last_fullsized":"2024-09-10T18:23:28.799345+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:22.519160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:23:22.519160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001283574,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":93,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.945957+0000","last_change":"2024-09-10T18:23:28.800816+0000","last_active":"2024-09-10T18:25:56.945957+0000","last_peered":"2024-09-10T18:25:56.945957+0000","last_clean":"2024-09-10T18:23:26.786960+0000","last_became_active":"2024-09-10T18:23:28.800816+0000","last_became_peered":"2024-09-10T18:23:28.800816+0000","last_unstale":"2024-09-10T18:25:56.945957+0000","last_undegraded":"2024-09-10T18:25:56.945957+0000","last_fullsized":"2024-09-10T18:23:28.795366+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:50.298320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:50.298320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021610420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"131'10012","reported_seq":10349,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-10T18:26:04.829228+0000","last_change":"2024-09-10T18:25:08.084796+0000","last_active":"2024-09-10T18:26:04.829228+0000","last_peered":"2024-09-10T18:26:04.829228+0000","last_clean":"2024-09-10T18:26:04.829228+0000","last_became_active":"2024-09-10T18:23:26.798070+0000","last_became_peered":"2024-09-10T18:23:26.798070+0000","last_unstale":"2024-09-10T18:26:04.829228+0000","last_undegraded":"2024-09-10T18:26:04.829228+0000","last_fullsized":"2024-09-10T18:26:04.829228+0000","mapping_epoch":93,"log_start":"104'3100","ondisk_log_start":"104'3100","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'7172","last_scrub_stamp":"2024-09-10T18:25:08.084572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_clean_scrub_stamp":"2024-09-10T18:25:08.084572+0000","objects_scrubbed":7025,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:12.454064+0000","scrub_duration":4.6862829770000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330039296,"num_objects":5036,"num_object_clones":0,"num_object_copies":15108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10012,"num_write_kb":481536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":127,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.934511+0000","last_change":"2024-09-10T18:25:06.888396+0000","last_active":"2024-09-10T18:25:56.934511+0000","last_peered":"2024-09-10T18:25:56.934511+0000","last_clean":"2024-09-10T18:25:56.934511+0000","last_became_active":"2024-09-10T18:21:33.611203+0000","last_became_peered":"2024-09-10T18:21:33.611203+0000","last_unstale":"2024-09-10T18:25:56.934511+0000","last_undegraded":"2024-09-10T18:25:56.934511+0000","last_fullsized":"2024-09-10T18:25:56.934511+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-10T18:25:06.888305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:25:06.888305+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:11.608700+0000","scrub_duration":0.0034891200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"58'1","reported_seq":53,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:56.933999+0000","last_change":"2024-09-10T18:23:28.805906+0000","last_active":"2024-09-10T18:25:56.933999+0000","last_peered":"2024-09-10T18:25:56.933999+0000","last_clean":"2024-09-10T18:21:31.529643+0000","last_became_active":"2024-09-10T18:23:28.805906+0000","last_became_peered":"2024-09-10T18:23:28.805906+0000","last_unstale":"2024-09-10T18:25:56.933999+0000","last_undegraded":"2024-09-10T18:23:28.799800+0000","last_fullsized":"2024-09-10T18:23:28.799702+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-10T18:23:09.877514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:23:09.877514+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":96,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.936914+0000","last_change":"2024-09-10T18:24:58.149252+0000","last_active":"2024-09-10T18:25:56.936914+0000","last_peered":"2024-09-10T18:25:56.936914+0000","last_clean":"2024-09-10T18:25:56.936914+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:25:56.936914+0000","last_undegraded":"2024-09-10T18:25:56.936914+0000","last_fullsized":"2024-09-10T18:25:56.936914+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:58.148936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:24:58.148936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:12.154200+0000","scrub_duration":0.01063065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":105,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.938069+0000","last_change":"2024-09-10T18:25:10.356187+0000","last_active":"2024-09-10T18:25:56.938069+0000","last_peered":"2024-09-10T18:25:56.938069+0000","last_clean":"2024-09-10T18:25:56.938069+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:25:56.938069+0000","last_undegraded":"2024-09-10T18:25:56.938069+0000","last_fullsized":"2024-09-10T18:25:56.938069+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:10.356102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:25:10.356102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:15.857881+0000","scrub_duration":0.0083002449999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"131'14551","reported_seq":14808,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-10T18:26:05.159255+0000","last_change":"2024-09-10T18:25:46.972618+0000","last_active":"2024-09-10T18:26:05.159255+0000","last_peered":"2024-09-10T18:26:05.159255+0000","last_clean":"2024-09-10T18:26:05.159255+0000","last_became_active":"2024-09-10T18:23:26.801284+0000","last_became_peered":"2024-09-10T18:23:26.801284+0000","last_unstale":"2024-09-10T18:26:05.159255+0000","last_undegraded":"2024-09-10T18:26:05.159255+0000","last_fullsized":"2024-09-10T18:26:05.159255+0000","mapping_epoch":93,"log_start":"123'8903","ondisk_log_start":"123'8903","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_clean_scrub_stamp":"2024-09-10T18:23:25.774520+0000","objects_scrubbed":5072,"log_size":5648,"log_dups_size":0,"ondisk_log_size":5648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337444864,"num_objects":5149,"num_object_clones":0,"num_object_copies":15447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10029,"num_write_kb":485696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"92'15","reported_seq":116,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:56.935088+0000","last_change":"2024-09-10T18:23:28.806564+0000","last_active":"2024-09-10T18:25:56.935088+0000","last_peered":"2024-09-10T18:25:56.935088+0000","last_clean":"2024-09-10T18:23:26.785943+0000","last_became_active":"2024-09-10T18:23:28.806564+0000","last_became_peered":"2024-09-10T18:23:28.806564+0000","last_unstale":"2024-09-10T18:25:56.935088+0000","last_undegraded":"2024-09-10T18:23:28.801239+0000","last_fullsized":"2024-09-10T18:23:28.801130+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'15","last_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:23:23.566732+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035214249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":102,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.946571+0000","last_change":"2024-09-10T18:25:15.101521+0000","last_active":"2024-09-10T18:25:56.946571+0000","last_peered":"2024-09-10T18:25:56.946571+0000","last_clean":"2024-09-10T18:25:56.946571+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:25:56.946571+0000","last_undegraded":"2024-09-10T18:25:56.946571+0000","last_fullsized":"2024-09-10T18:25:56.946571+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:15.101285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:25:15.101285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:36.541593+0000","scrub_duration":0.0096093979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":170,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-10T18:26:00.619654+0000","last_change":"2024-09-10T18:25:12.409404+0000","last_active":"2024-09-10T18:26:00.619654+0000","last_peered":"2024-09-10T18:26:00.619654+0000","last_clean":"2024-09-10T18:26:00.619654+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:26:00.619654+0000","last_undegraded":"2024-09-10T18:26:00.619654+0000","last_fullsized":"2024-09-10T18:26:00.619654+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:25:12.409316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:25:12.409316+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:16.202225+0000","scrub_duration":0.0043000269999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":98,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.934337+0000","last_change":"2024-09-10T18:25:24.076699+0000","last_active":"2024-09-10T18:25:56.934337+0000","last_peered":"2024-09-10T18:25:56.934337+0000","last_clean":"2024-09-10T18:25:56.934337+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:25:56.934337+0000","last_undegraded":"2024-09-10T18:25:56.934337+0000","last_fullsized":"2024-09-10T18:25:56.934337+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:24.076647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:25:24.076647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:53.910090+0000","scrub_duration":0.001738355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"131'10015","reported_seq":10354,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-10T18:26:04.823535+0000","last_change":"2024-09-10T18:25:07.057909+0000","last_active":"2024-09-10T18:26:04.823535+0000","last_peered":"2024-09-10T18:26:04.823535+0000","last_clean":"2024-09-10T18:26:04.823535+0000","last_became_active":"2024-09-10T18:23:26.794276+0000","last_became_peered":"2024-09-10T18:23:26.794276+0000","last_unstale":"2024-09-10T18:26:04.823535+0000","last_undegraded":"2024-09-10T18:26:04.823535+0000","last_fullsized":"2024-09-10T18:26:04.823535+0000","mapping_epoch":93,"log_start":"105'3100","ondisk_log_start":"105'3100","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'7143","last_scrub_stamp":"2024-09-10T18:25:07.057681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_clean_scrub_stamp":"2024-09-10T18:25:07.057681+0000","objects_scrubbed":7075,"log_size":6915,"log_dups_size":0,"ondisk_log_size":6915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:10.493152+0000","scrub_duration":2.676696561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333774848,"num_objects":5093,"num_object_clones":0,"num_object_copies":15279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10015,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":105,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.943361+0000","last_change":"2024-09-10T18:23:27.610460+0000","last_active":"2024-09-10T18:25:56.943361+0000","last_peered":"2024-09-10T18:25:56.943361+0000","last_clean":"2024-09-10T18:25:56.943361+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:25:56.943361+0000","last_undegraded":"2024-09-10T18:25:56.943361+0000","last_fullsized":"2024-09-10T18:25:56.943361+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:23:27.610402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:23:27.610402+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018542929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"58'1","reported_seq":107,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:56.943224+0000","last_change":"2024-09-10T18:23:28.803549+0000","last_active":"2024-09-10T18:25:56.943224+0000","last_peered":"2024-09-10T18:25:56.943224+0000","last_clean":"2024-09-10T18:23:26.792082+0000","last_became_active":"2024-09-10T18:23:28.803549+0000","last_became_peered":"2024-09-10T18:23:28.803549+0000","last_unstale":"2024-09-10T18:25:56.943224+0000","last_undegraded":"2024-09-10T18:23:28.799302+0000","last_fullsized":"2024-09-10T18:23:28.797834+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-10T18:23:04.541238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:23:04.541238+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017464519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":113,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:58.279758+0000","last_change":"2024-09-10T18:25:58.279391+0000","last_active":"2024-09-10T18:25:58.279758+0000","last_peered":"2024-09-10T18:25:58.279758+0000","last_clean":"2024-09-10T18:25:58.279758+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:25:58.279758+0000","last_undegraded":"2024-09-10T18:25:58.279758+0000","last_fullsized":"2024-09-10T18:25:58.279758+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:58.278520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:25:58.278520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:27:16.181274+0000","scrub_duration":0.021510333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":95,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.942497+0000","last_change":"2024-09-10T18:25:14.252363+0000","last_active":"2024-09-10T18:25:56.942497+0000","last_peered":"2024-09-10T18:25:56.942497+0000","last_clean":"2024-09-10T18:25:56.942497+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:25:56.942497+0000","last_undegraded":"2024-09-10T18:25:56.942497+0000","last_fullsized":"2024-09-10T18:25:56.942497+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:14.252294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:25:14.252294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:19.723046+0000","scrub_duration":0.00144524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"131'10203","reported_seq":10548,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-10T18:26:05.142702+0000","last_change":"2024-09-10T18:25:25.514993+0000","last_active":"2024-09-10T18:26:05.142702+0000","last_peered":"2024-09-10T18:26:05.142702+0000","last_clean":"2024-09-10T18:26:05.142702+0000","last_became_active":"2024-09-10T18:23:26.795924+0000","last_became_peered":"2024-09-10T18:23:26.795924+0000","last_unstale":"2024-09-10T18:26:05.142702+0000","last_undegraded":"2024-09-10T18:26:05.142702+0000","last_fullsized":"2024-09-10T18:26:05.142702+0000","mapping_epoch":93,"log_start":"111'4600","ondisk_log_start":"111'4600","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'8192","last_scrub_stamp":"2024-09-10T18:25:25.514679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_clean_scrub_stamp":"2024-09-10T18:25:25.514679+0000","objects_scrubbed":7225,"log_size":5603,"log_dups_size":0,"ondisk_log_size":5603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:41.055671+0000","scrub_duration":2.2062421200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339410944,"num_objects":5179,"num_object_clones":0,"num_object_copies":15537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10203,"num_write_kb":492224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"86'28","reported_seq":134,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:56.938764+0000","last_change":"2024-09-10T18:23:28.803097+0000","last_active":"2024-09-10T18:25:56.938764+0000","last_peered":"2024-09-10T18:25:56.938764+0000","last_clean":"2024-09-10T18:23:00.604569+0000","last_became_active":"2024-09-10T18:23:28.803097+0000","last_became_peered":"2024-09-10T18:23:28.803097+0000","last_unstale":"2024-09-10T18:25:56.938764+0000","last_undegraded":"2024-09-10T18:23:28.798316+0000","last_fullsized":"2024-09-10T18:23:28.798202+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'28","last_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:22:54.863323+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053712059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":9}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":98,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.944917+0000","last_change":"2024-09-10T18:23:28.805830+0000","last_active":"2024-09-10T18:25:56.944917+0000","last_peered":"2024-09-10T18:25:56.944917+0000","last_clean":"2024-09-10T18:23:26.791493+0000","last_became_active":"2024-09-10T18:23:28.805830+0000","last_became_peered":"2024-09-10T18:23:28.805830+0000","last_unstale":"2024-09-10T18:25:56.944917+0000","last_undegraded":"2024-09-10T18:25:56.944917+0000","last_fullsized":"2024-09-10T18:23:28.795780+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:32.137600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:32.137600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00151761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":107,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.943802+0000","last_change":"2024-09-10T18:25:07.355341+0000","last_active":"2024-09-10T18:25:56.943802+0000","last_peered":"2024-09-10T18:25:56.943802+0000","last_clean":"2024-09-10T18:25:56.943802+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:25:56.943802+0000","last_undegraded":"2024-09-10T18:25:56.943802+0000","last_fullsized":"2024-09-10T18:25:56.943802+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:07.355252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:25:07.355252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:18.325829+0000","scrub_duration":0.0098698840000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":108,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-10T18:26:02.005081+0000","last_change":"2024-09-10T18:26:02.005000+0000","last_active":"2024-09-10T18:26:02.005081+0000","last_peered":"2024-09-10T18:26:02.005081+0000","last_clean":"2024-09-10T18:26:02.005081+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:26:02.005081+0000","last_undegraded":"2024-09-10T18:26:02.005081+0000","last_fullsized":"2024-09-10T18:26:02.005081+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:02.004922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:26:02.004922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:27:10.558834+0000","scrub_duration":0.0035028479999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"131'13777","reported_seq":14061,"reported_epoch":131,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T18:26:05.058877+0000","last_change":"2024-09-10T18:26:04.074213+0000","last_active":"2024-09-10T18:26:05.058877+0000","last_peered":"2024-09-10T18:26:05.058877+0000","last_clean":"2024-09-10T18:26:05.058877+0000","last_became_active":"2024-09-10T18:23:26.794368+0000","last_became_peered":"2024-09-10T18:23:26.794368+0000","last_unstale":"2024-09-10T18:26:05.058877+0000","last_undegraded":"2024-09-10T18:26:05.058877+0000","last_fullsized":"2024-09-10T18:26:05.058877+0000","mapping_epoch":93,"log_start":"123'9102","ondisk_log_start":"123'9102","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_clean_scrub_stamp":"2024-09-10T18:23:25.774520+0000","objects_scrubbed":1361,"log_size":4675,"log_dups_size":0,"ondisk_log_size":4675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332136448,"num_objects":5068,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10058,"num_write_kb":484032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":125,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.933892+0000","last_change":"2024-09-10T18:25:19.993608+0000","last_active":"2024-09-10T18:25:56.933892+0000","last_peered":"2024-09-10T18:25:56.933892+0000","last_clean":"2024-09-10T18:25:56.933892+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:25:56.933892+0000","last_undegraded":"2024-09-10T18:25:56.933892+0000","last_fullsized":"2024-09-10T18:25:56.933892+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:24:07.660983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:24:07.660983+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011636207000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":101,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.944220+0000","last_change":"2024-09-10T18:25:04.751223+0000","last_active":"2024-09-10T18:25:56.944220+0000","last_peered":"2024-09-10T18:25:56.944220+0000","last_clean":"2024-09-10T18:25:56.944220+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:25:56.944220+0000","last_undegraded":"2024-09-10T18:25:56.944220+0000","last_fullsized":"2024-09-10T18:25:56.944220+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:04.751136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:25:04.751136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:06.291636+0000","scrub_duration":0.003829135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":107,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.940341+0000","last_change":"2024-09-10T18:25:08.350606+0000","last_active":"2024-09-10T18:25:56.940341+0000","last_peered":"2024-09-10T18:25:56.940341+0000","last_clean":"2024-09-10T18:25:56.940341+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:25:56.940341+0000","last_undegraded":"2024-09-10T18:25:56.940341+0000","last_fullsized":"2024-09-10T18:25:56.940341+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:08.350307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:25:08.350307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:38.136223+0000","scrub_duration":0.0046995140000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":86,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.946337+0000","last_change":"2024-09-10T18:23:55.187698+0000","last_active":"2024-09-10T18:25:56.946337+0000","last_peered":"2024-09-10T18:25:56.946337+0000","last_clean":"2024-09-10T18:25:56.946337+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:25:56.946337+0000","last_undegraded":"2024-09-10T18:25:56.946337+0000","last_fullsized":"2024-09-10T18:25:56.946337+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:55.187514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:23:55.187514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030691469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"131'9932","reported_seq":9985,"reported_epoch":131,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:26:04.805596+0000","last_change":"2024-09-10T18:23:43.908569+0000","last_active":"2024-09-10T18:26:04.805596+0000","last_peered":"2024-09-10T18:26:04.805596+0000","last_clean":"2024-09-10T18:23:26.799781+0000","last_became_active":"2024-09-10T18:23:28.805074+0000","last_became_peered":"2024-09-10T18:23:28.805074+0000","last_unstale":"2024-09-10T18:26:04.805596+0000","last_undegraded":"2024-09-10T18:23:43.907114+0000","last_fullsized":"2024-09-10T18:23:28.800033+0000","mapping_epoch":95,"log_start":"113'4900","ondisk_log_start":"113'4900","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:25.774520+0000","last_clean_scrub_stamp":"2024-09-10T18:23:25.774520+0000","objects_scrubbed":0,"log_size":5032,"log_dups_size":0,"ondisk_log_size":5032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331612160,"num_objects":5060,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9932,"num_write_kb":479744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5060}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":121,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:56.942643+0000","last_change":"2024-09-10T18:23:28.805224+0000","last_active":"2024-09-10T18:25:56.942643+0000","last_peered":"2024-09-10T18:25:56.942643+0000","last_clean":"2024-09-10T18:23:26.790956+0000","last_became_active":"2024-09-10T18:23:28.805224+0000","last_became_peered":"2024-09-10T18:23:28.805224+0000","last_unstale":"2024-09-10T18:25:56.942643+0000","last_undegraded":"2024-09-10T18:23:28.799199+0000","last_fullsized":"2024-09-10T18:23:28.799098+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-10T18:22:51.599559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:51.599559+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019152609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":118,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.944231+0000","last_change":"2024-09-10T18:25:02.820320+0000","last_active":"2024-09-10T18:25:56.944231+0000","last_peered":"2024-09-10T18:25:56.944231+0000","last_clean":"2024-09-10T18:25:56.944231+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:25:56.944231+0000","last_undegraded":"2024-09-10T18:25:56.944231+0000","last_fullsized":"2024-09-10T18:25:56.944231+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:25:02.820228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:25:02.820228+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:28.180605+0000","scrub_duration":0.011143505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"96'9","reported_seq":157,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:59.805879+0000","last_change":"2024-09-10T18:23:28.799268+0000","last_active":"2024-09-10T18:25:59.805879+0000","last_peered":"2024-09-10T18:25:59.805879+0000","last_clean":"2024-09-10T18:23:26.789428+0000","last_became_active":"2024-09-10T18:23:28.799268+0000","last_became_peered":"2024-09-10T18:23:28.799268+0000","last_unstale":"2024-09-10T18:25:59.805879+0000","last_undegraded":"2024-09-10T18:23:28.794667+0000","last_fullsized":"2024-09-10T18:23:28.794501+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:23:24.042767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:23:24.042767+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014271819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":82,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.943940+0000","last_change":"2024-09-10T18:23:28.808031+0000","last_active":"2024-09-10T18:25:56.943940+0000","last_peered":"2024-09-10T18:25:56.943940+0000","last_clean":"2024-09-10T18:23:26.790504+0000","last_became_active":"2024-09-10T18:23:28.808031+0000","last_became_peered":"2024-09-10T18:23:28.808031+0000","last_unstale":"2024-09-10T18:25:56.943940+0000","last_undegraded":"2024-09-10T18:25:56.943940+0000","last_fullsized":"2024-09-10T18:23:28.797184+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:21:36.612407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":45,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.942546+0000","last_change":"2024-09-10T18:23:28.805204+0000","last_active":"2024-09-10T18:25:56.942546+0000","last_peered":"2024-09-10T18:25:56.942546+0000","last_clean":"2024-09-10T18:21:30.506537+0000","last_became_active":"2024-09-10T18:23:28.805204+0000","last_became_peered":"2024-09-10T18:23:28.805204+0000","last_unstale":"2024-09-10T18:25:56.942546+0000","last_undegraded":"2024-09-10T18:25:56.942546+0000","last_fullsized":"2024-09-10T18:23:28.802259+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:23:08.899949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":109,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.948082+0000","last_change":"2024-09-10T18:25:08.220646+0000","last_active":"2024-09-10T18:25:56.948082+0000","last_peered":"2024-09-10T18:25:56.948082+0000","last_clean":"2024-09-10T18:25:56.948082+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:25:56.948082+0000","last_undegraded":"2024-09-10T18:25:56.948082+0000","last_fullsized":"2024-09-10T18:25:56.948082+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-10T18:25:08.219610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:25:08.219610+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:09.034537+0000","scrub_duration":0.016635921000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":90,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.945271+0000","last_change":"2024-09-10T18:25:21.679378+0000","last_active":"2024-09-10T18:25:56.945271+0000","last_peered":"2024-09-10T18:25:56.945271+0000","last_clean":"2024-09-10T18:25:56.945271+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:25:56.945271+0000","last_undegraded":"2024-09-10T18:25:56.945271+0000","last_fullsized":"2024-09-10T18:25:56.945271+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:20.508172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:23:20.508172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001469052,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":92,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.944557+0000","last_change":"2024-09-10T18:23:28.802324+0000","last_active":"2024-09-10T18:25:56.944557+0000","last_peered":"2024-09-10T18:25:56.944557+0000","last_clean":"2024-09-10T18:23:26.791064+0000","last_became_active":"2024-09-10T18:23:28.802324+0000","last_became_peered":"2024-09-10T18:23:28.802324+0000","last_unstale":"2024-09-10T18:25:56.944557+0000","last_undegraded":"2024-09-10T18:25:56.944557+0000","last_fullsized":"2024-09-10T18:23:28.794989+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:09.102383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:23:09.102383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001332179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":108,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.950001+0000","last_change":"2024-09-10T18:25:22.306378+0000","last_active":"2024-09-10T18:25:56.950001+0000","last_peered":"2024-09-10T18:25:56.950001+0000","last_clean":"2024-09-10T18:25:56.950001+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:25:56.950001+0000","last_undegraded":"2024-09-10T18:25:56.950001+0000","last_fullsized":"2024-09-10T18:25:56.950001+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:23:20.549080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:23:20.549080+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017928729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":98,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.943041+0000","last_change":"2024-09-10T18:23:28.798928+0000","last_active":"2024-09-10T18:25:56.943041+0000","last_peered":"2024-09-10T18:25:56.943041+0000","last_clean":"2024-09-10T18:23:26.789569+0000","last_became_active":"2024-09-10T18:23:28.798928+0000","last_became_peered":"2024-09-10T18:23:28.798928+0000","last_unstale":"2024-09-10T18:25:56.943041+0000","last_undegraded":"2024-09-10T18:25:56.943041+0000","last_fullsized":"2024-09-10T18:23:28.794733+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:22:56.100012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001414212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":98,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.947965+0000","last_change":"2024-09-10T18:24:22.229003+0000","last_active":"2024-09-10T18:25:56.947965+0000","last_peered":"2024-09-10T18:25:56.947965+0000","last_clean":"2024-09-10T18:25:56.947965+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:25:56.947965+0000","last_undegraded":"2024-09-10T18:25:56.947965+0000","last_fullsized":"2024-09-10T18:25:56.947965+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:22.228913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:24:22.228913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013892619,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":95,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.937541+0000","last_change":"2024-09-10T18:25:02.410841+0000","last_active":"2024-09-10T18:25:56.937541+0000","last_peered":"2024-09-10T18:25:56.937541+0000","last_clean":"2024-09-10T18:25:56.937541+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:25:56.937541+0000","last_undegraded":"2024-09-10T18:25:56.937541+0000","last_fullsized":"2024-09-10T18:25:56.937541+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:02.410769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:25:02.410769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:06.114963+0000","scrub_duration":0.0033376999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":115,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.943637+0000","last_change":"2024-09-10T18:25:12.556610+0000","last_active":"2024-09-10T18:25:56.943637+0000","last_peered":"2024-09-10T18:25:56.943637+0000","last_clean":"2024-09-10T18:25:56.943637+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:25:56.943637+0000","last_undegraded":"2024-09-10T18:25:56.943637+0000","last_fullsized":"2024-09-10T18:25:56.943637+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:25:12.556539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:25:12.556539+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:25.192723+0000","scrub_duration":0.0025899959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":101,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.943014+0000","last_change":"2024-09-10T18:24:05.105125+0000","last_active":"2024-09-10T18:25:56.943014+0000","last_peered":"2024-09-10T18:25:56.943014+0000","last_clean":"2024-09-10T18:25:56.943014+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:25:56.943014+0000","last_undegraded":"2024-09-10T18:25:56.943014+0000","last_fullsized":"2024-09-10T18:25:56.943014+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:05.105023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:24:05.105023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089269280000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":161,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-10T18:26:00.610239+0000","last_change":"2024-09-10T18:24:04.179763+0000","last_active":"2024-09-10T18:26:00.610239+0000","last_peered":"2024-09-10T18:26:00.610239+0000","last_clean":"2024-09-10T18:26:00.610239+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:26:00.610239+0000","last_undegraded":"2024-09-10T18:26:00.610239+0000","last_fullsized":"2024-09-10T18:26:00.610239+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:24:04.179679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:24:04.179679+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095452430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":95,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.934635+0000","last_change":"2024-09-10T18:24:57.844336+0000","last_active":"2024-09-10T18:25:56.934635+0000","last_peered":"2024-09-10T18:25:56.934635+0000","last_clean":"2024-09-10T18:25:56.934635+0000","last_became_active":"2024-09-10T18:21:37.639706+0000","last_became_peered":"2024-09-10T18:21:37.639706+0000","last_unstale":"2024-09-10T18:25:56.934635+0000","last_undegraded":"2024-09-10T18:25:56.934635+0000","last_fullsized":"2024-09-10T18:25:56.934635+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:57.844284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:24:57.844284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:15.118695+0000","scrub_duration":0.0014439209999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":115,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.948338+0000","last_change":"2024-09-10T18:24:02.185609+0000","last_active":"2024-09-10T18:25:56.948338+0000","last_peered":"2024-09-10T18:25:56.948338+0000","last_clean":"2024-09-10T18:25:56.948338+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:25:56.948338+0000","last_undegraded":"2024-09-10T18:25:56.948338+0000","last_fullsized":"2024-09-10T18:25:56.948338+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-10T18:24:02.185242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:24:02.185242+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096732180000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":101,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.943801+0000","last_change":"2024-09-10T18:25:01.828168+0000","last_active":"2024-09-10T18:25:56.943801+0000","last_peered":"2024-09-10T18:25:56.943801+0000","last_clean":"2024-09-10T18:25:56.943801+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:25:56.943801+0000","last_undegraded":"2024-09-10T18:25:56.943801+0000","last_fullsized":"2024-09-10T18:25:56.943801+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:01.828116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:25:01.828116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:14.845932+0000","scrub_duration":0.0051578020000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":159,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:59.805500+0000","last_change":"2024-09-10T18:24:16.526822+0000","last_active":"2024-09-10T18:25:59.805500+0000","last_peered":"2024-09-10T18:25:59.805500+0000","last_clean":"2024-09-10T18:25:59.805500+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:25:59.805500+0000","last_undegraded":"2024-09-10T18:25:59.805500+0000","last_fullsized":"2024-09-10T18:25:59.805500+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:24:16.526732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:24:16.526732+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091942020000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":95,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.949502+0000","last_change":"2024-09-10T18:25:04.394820+0000","last_active":"2024-09-10T18:25:56.949502+0000","last_peered":"2024-09-10T18:25:56.949502+0000","last_clean":"2024-09-10T18:25:56.949502+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:25:56.949502+0000","last_undegraded":"2024-09-10T18:25:56.949502+0000","last_fullsized":"2024-09-10T18:25:56.949502+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:04.394579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:25:04.394579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:26.784307+0000","scrub_duration":0.014640459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":115,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:56.933899+0000","last_change":"2024-09-10T18:23:28.805289+0000","last_active":"2024-09-10T18:25:56.933899+0000","last_peered":"2024-09-10T18:25:56.933899+0000","last_clean":"2024-09-10T18:23:26.785809+0000","last_became_active":"2024-09-10T18:23:28.805289+0000","last_became_peered":"2024-09-10T18:23:28.805289+0000","last_unstale":"2024-09-10T18:25:56.933899+0000","last_undegraded":"2024-09-10T18:23:28.798269+0000","last_fullsized":"2024-09-10T18:23:28.798159+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:23:15.465128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:23:15.465128+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002207695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":98,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.935216+0000","last_change":"2024-09-10T18:23:28.804865+0000","last_active":"2024-09-10T18:25:56.935216+0000","last_peered":"2024-09-10T18:25:56.935216+0000","last_clean":"2024-09-10T18:23:26.786141+0000","last_became_active":"2024-09-10T18:23:28.804865+0000","last_became_peered":"2024-09-10T18:23:28.804865+0000","last_unstale":"2024-09-10T18:25:56.935216+0000","last_undegraded":"2024-09-10T18:25:56.935216+0000","last_fullsized":"2024-09-10T18:23:28.800654+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:53.433012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:22:53.433012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013676739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":109,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:57.599643+0000","last_change":"2024-09-10T18:25:57.599468+0000","last_active":"2024-09-10T18:25:57.599643+0000","last_peered":"2024-09-10T18:25:57.599643+0000","last_clean":"2024-09-10T18:25:57.599643+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:25:57.599643+0000","last_undegraded":"2024-09-10T18:25:57.599643+0000","last_fullsized":"2024-09-10T18:25:57.599643+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:57.599382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:25:57.599382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:27:13.985685+0000","scrub_duration":0.010921520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":86,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.948584+0000","last_change":"2024-09-10T18:24:00.129408+0000","last_active":"2024-09-10T18:25:56.948584+0000","last_peered":"2024-09-10T18:25:56.948584+0000","last_clean":"2024-09-10T18:25:56.948584+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:25:56.948584+0000","last_undegraded":"2024-09-10T18:25:56.948584+0000","last_fullsized":"2024-09-10T18:25:56.948584+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:00.128986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:24:00.128986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074512320000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":111,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:56.934920+0000","last_change":"2024-09-10T18:23:28.805174+0000","last_active":"2024-09-10T18:25:56.934920+0000","last_peered":"2024-09-10T18:25:56.934920+0000","last_clean":"2024-09-10T18:23:26.790880+0000","last_became_active":"2024-09-10T18:23:28.805174+0000","last_became_peered":"2024-09-10T18:23:28.805174+0000","last_unstale":"2024-09-10T18:25:56.934920+0000","last_undegraded":"2024-09-10T18:23:28.800304+0000","last_fullsized":"2024-09-10T18:23:28.800181+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:22:42.015292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:22:42.015292+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001986785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":114,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:59.037668+0000","last_change":"2024-09-10T18:25:59.037584+0000","last_active":"2024-09-10T18:25:59.037668+0000","last_peered":"2024-09-10T18:25:59.037668+0000","last_clean":"2024-09-10T18:25:59.037668+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:25:59.037668+0000","last_undegraded":"2024-09-10T18:25:59.037668+0000","last_fullsized":"2024-09-10T18:25:59.037668+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:59.037508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:25:59.037508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:27:05.331944+0000","scrub_duration":0.0032188809999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":88,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.946408+0000","last_change":"2024-09-10T18:23:58.140765+0000","last_active":"2024-09-10T18:25:56.946408+0000","last_peered":"2024-09-10T18:25:56.946408+0000","last_clean":"2024-09-10T18:25:56.946408+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:25:56.946408+0000","last_undegraded":"2024-09-10T18:25:56.946408+0000","last_fullsized":"2024-09-10T18:25:56.946408+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:58.140686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:23:58.140686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013234302999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":95,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.942137+0000","last_change":"2024-09-10T18:24:59.324787+0000","last_active":"2024-09-10T18:25:56.942137+0000","last_peered":"2024-09-10T18:25:56.942137+0000","last_clean":"2024-09-10T18:25:56.942137+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:25:56.942137+0000","last_undegraded":"2024-09-10T18:25:56.942137+0000","last_fullsized":"2024-09-10T18:25:56.942137+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:59.324712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:24:59.324712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:12.874545+0000","scrub_duration":0.0099064889999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":116,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:56.948720+0000","last_change":"2024-09-10T18:23:28.799376+0000","last_active":"2024-09-10T18:25:56.948720+0000","last_peered":"2024-09-10T18:25:56.948720+0000","last_clean":"2024-09-10T18:23:26.786048+0000","last_became_active":"2024-09-10T18:23:28.799376+0000","last_became_peered":"2024-09-10T18:23:28.799376+0000","last_unstale":"2024-09-10T18:25:56.948720+0000","last_undegraded":"2024-09-10T18:23:28.794892+0000","last_fullsized":"2024-09-10T18:23:28.794783+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:23:05.391853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:23:05.391853+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018201140000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":111,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.945260+0000","last_change":"2024-09-10T18:25:25.752612+0000","last_active":"2024-09-10T18:25:56.945260+0000","last_peered":"2024-09-10T18:25:56.945260+0000","last_clean":"2024-09-10T18:25:56.945260+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:25:56.945260+0000","last_undegraded":"2024-09-10T18:25:56.945260+0000","last_fullsized":"2024-09-10T18:25:56.945260+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:25.752560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:25:25.752560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:31.605503+0000","scrub_duration":0.002076353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":94,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.942751+0000","last_change":"2024-09-10T18:23:28.803239+0000","last_active":"2024-09-10T18:25:56.942751+0000","last_peered":"2024-09-10T18:25:56.942751+0000","last_clean":"2024-09-10T18:23:26.789096+0000","last_became_active":"2024-09-10T18:23:28.803239+0000","last_became_peered":"2024-09-10T18:23:28.803239+0000","last_unstale":"2024-09-10T18:25:56.942751+0000","last_undegraded":"2024-09-10T18:25:56.942751+0000","last_fullsized":"2024-09-10T18:23:28.794290+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:23:15.028129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:23:15.028129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012012749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":93,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.946598+0000","last_change":"2024-09-10T18:23:28.806037+0000","last_active":"2024-09-10T18:25:56.946598+0000","last_peered":"2024-09-10T18:25:56.946598+0000","last_clean":"2024-09-10T18:23:26.787094+0000","last_became_active":"2024-09-10T18:23:28.806037+0000","last_became_peered":"2024-09-10T18:23:28.806037+0000","last_unstale":"2024-09-10T18:25:56.946598+0000","last_undegraded":"2024-09-10T18:25:56.946598+0000","last_fullsized":"2024-09-10T18:23:28.796205+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:46.343137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:22:46.343137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016021189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":103,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:56.945862+0000","last_change":"2024-09-10T18:23:28.802387+0000","last_active":"2024-09-10T18:25:56.945862+0000","last_peered":"2024-09-10T18:25:56.945862+0000","last_clean":"2024-09-10T18:23:26.787056+0000","last_became_active":"2024-09-10T18:23:28.802387+0000","last_became_peered":"2024-09-10T18:23:28.802387+0000","last_unstale":"2024-09-10T18:25:56.945862+0000","last_undegraded":"2024-09-10T18:23:28.794718+0000","last_fullsized":"2024-09-10T18:23:28.794591+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-09-10T18:23:26.303471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:23:26.303471+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019958559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":104,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-10T18:26:03.706401+0000","last_change":"2024-09-10T18:26:03.705930+0000","last_active":"2024-09-10T18:26:03.706401+0000","last_peered":"2024-09-10T18:26:03.706401+0000","last_clean":"2024-09-10T18:26:03.706401+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:26:03.706401+0000","last_undegraded":"2024-09-10T18:26:03.706401+0000","last_fullsized":"2024-09-10T18:26:03.706401+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:03.705840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:26:03.705840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:27:12.410069+0000","scrub_duration":0.026713131000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":45,"reported_epoch":130,"state":"active+undersized","last_fresh":"2024-09-10T18:25:56.933768+0000","last_change":"2024-09-10T18:23:28.801328+0000","last_active":"2024-09-10T18:25:56.933768+0000","last_peered":"2024-09-10T18:25:56.933768+0000","last_clean":"2024-09-10T18:21:34.598351+0000","last_became_active":"2024-09-10T18:23:28.801328+0000","last_became_peered":"2024-09-10T18:23:28.801328+0000","last_unstale":"2024-09-10T18:25:56.933768+0000","last_undegraded":"2024-09-10T18:25:56.933768+0000","last_fullsized":"2024-09-10T18:23:28.796710+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:22:49.757435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:22:49.757435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":95,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.943198+0000","last_change":"2024-09-10T18:25:10.831530+0000","last_active":"2024-09-10T18:25:56.943198+0000","last_peered":"2024-09-10T18:25:56.943198+0000","last_clean":"2024-09-10T18:25:56.943198+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:25:56.943198+0000","last_undegraded":"2024-09-10T18:25:56.943198+0000","last_fullsized":"2024-09-10T18:25:56.943198+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:10.831426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:25:10.831426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:24.734816+0000","scrub_duration":0.0023264380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":104,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.935447+0000","last_change":"2024-09-10T18:25:10.232573+0000","last_active":"2024-09-10T18:25:56.935447+0000","last_peered":"2024-09-10T18:25:56.935447+0000","last_clean":"2024-09-10T18:25:56.935447+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:25:56.935447+0000","last_undegraded":"2024-09-10T18:25:56.935447+0000","last_fullsized":"2024-09-10T18:25:56.935447+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:10.232493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:25:10.232493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:17.367771+0000","scrub_duration":0.0018856280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":102,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.946459+0000","last_change":"2024-09-10T18:25:18.030959+0000","last_active":"2024-09-10T18:25:56.946459+0000","last_peered":"2024-09-10T18:25:56.946459+0000","last_clean":"2024-09-10T18:25:56.946459+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:25:56.946459+0000","last_undegraded":"2024-09-10T18:25:56.946459+0000","last_fullsized":"2024-09-10T18:25:56.946459+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:25:18.030878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:26:46.739705+0000","scrub_duration":0.026365884999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":76,"reported_epoch":130,"state":"active+undersized+degraded","last_fresh":"2024-09-10T18:25:56.944300+0000","last_change":"2024-09-10T18:23:28.806629+0000","last_active":"2024-09-10T18:25:56.944300+0000","last_peered":"2024-09-10T18:25:56.944300+0000","last_clean":"2024-09-10T18:21:38.969682+0000","last_became_active":"2024-09-10T18:23:28.806629+0000","last_became_peered":"2024-09-10T18:23:28.806629+0000","last_unstale":"2024-09-10T18:25:56.944300+0000","last_undegraded":"2024-09-10T18:23:28.800010+0000","last_fullsized":"2024-09-10T18:23:28.799893+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:23:00.864408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:23:00.864408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":97,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-10T18:25:56.949777+0000","last_change":"2024-09-10T18:24:08.169433+0000","last_active":"2024-09-10T18:25:56.949777+0000","last_peered":"2024-09-10T18:25:56.949777+0000","last_clean":"2024-09-10T18:25:56.949777+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:25:56.949777+0000","last_undegraded":"2024-09-10T18:25:56.949777+0000","last_fullsized":"2024-09-10T18:25:56.949777+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:24:08.169230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:24:08.169230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011716268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5353308184,"num_objects":81686,"num_object_clones":0,"num_object_copies":245058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81686,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160510,"num_write_kb":7750209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14397939712,"data_stored":14397931568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86563,"ondisk_log_size":86563,"up":43,"acting":43,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":86,"acting":86,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":88,"acting":88,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":65,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":696,"num_read_kb":453,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":437,"ondisk_log_size":437,"up":84,"acting":84,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":85,"acting":85,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":266,"num_read_kb":261,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234042,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1958432,"kb_used_data":1656920,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91786720,"statfs":{"total":95995035648,"available":93989601280,"internally_reserved":0,"allocated":1696686080,"data_stored":1696182025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:25:26 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":0.77700000000000002,"15min":0.60899999999999999},"min":{"1min":0.223,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":10.132999999999999,"5min":10.132999999999999,"15min":10.132999999999999},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.742,"5min":0.84699999999999998,"15min":0.627},"min":{"1min":0.246,"5min":0.222,"15min":0.222},"max":{"1min":9.7449999999999992,"5min":9.7449999999999992,"15min":9.7449999999999992},"last":0.29899999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:25:26 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":0.81399999999999995,"15min":0.61599999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.7370000000000001,"5min":8.9700000000000006,"15min":8.9700000000000006},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.246,"5min":0.84699999999999998,"15min":0.65900000000000003},"min":{"1min":0.30399999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.7450000000000001,"5min":8.9550000000000001,"15min":8.9550000000000001},"last":0.42699999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:25:26 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.70599999999999996,"15min":0.58899999999999997},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.109,"5min":4.109,"15min":4.109},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.68700000000000006,"15min":0.59699999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":3.9039999999999999,"5min":3.9119999999999999,"15min":3.9119999999999999},"last":1.0589999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:25:26 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.2,"15min":0.75800000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":5.3319999999999999,"5min":46.581000000000003,"15min":46.581000000000003},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2090000000000001,"15min":0.77100000000000002},"min":{"1min":0.29999999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":9.0890000000000004,"5min":46.725999999999999,"15min":46.725999999999999},"last":0.47799999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:25:26 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.39800000000000002,"15min":0.42399999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.47099999999999997,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.40300000000000002,"15min":0.42899999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.47899999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.316}]},{"osd":5,"last update":"Tue Sep 10 18:25:26 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.40799999999999997,"15min":0.41199999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.56000000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.442,"15min":0.46700000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.59799999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.35499999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.498,"15min":0.49199999999999999},"min":{"1min":0.35099999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.48599999999999999,"15min":0.47699999999999998},"min":{"1min":0.36799999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.42999999999999999}]}]},{"osd":6,"up_from":46,"seq":197568495648,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298588,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446564,"statfs":{"total":95995035648,"available":95689281536,"internally_reserved":0,"allocated":1265664,"data_stored":909579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55100000000000005,"15min":0.56000000000000005},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71299999999999997,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53900000000000003,"15min":0.54000000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.747,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.38600000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.55400000000000005,"15min":0.56000000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.65400000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.434},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54200000000000004,"15min":0.54600000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74099999999999999,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.47899999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.71199999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54900000000000004,"15min":0.55200000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73999999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.46200000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.68100000000000005,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.497},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.70899999999999996,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.56399999999999995}]},{"osd":4,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.42299999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.438,"15min":0.434},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.54900000000000004}]},{"osd":5,"last update":"Tue Sep 10 18:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.443,"15min":0.435},"min":{"1min":0.317,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.44600000000000001,"15min":0.439},"min":{"1min":0.315,"5min":0.308,"15min":0.308},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.57999999999999996}]},{"osd":7,"last update":"Tue Sep 10 18:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.627},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.61099999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724613,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1607328,"kb_used_data":1297496,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92137824,"statfs":{"total":95995035648,"available":94349131776,"internally_reserved":0,"allocated":1328635904,"data_stored":1328131811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:25:47 2024","interfaces":[{"interface":"back","average":{"1min":3.4689999999999999,"5min":1.2010000000000001,"15min":0.76500000000000001},"min":{"1min":0.23899999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":16.811,"5min":16.811,"15min":16.811},"last":0.36799999999999999},{"interface":"front","average":{"1min":1.04,"5min":0.73799999999999999,"15min":0.59199999999999997},"min":{"1min":0.24399999999999999,"5min":0.216,"15min":0.216},"max":{"1min":4.8940000000000001,"5min":4.8940000000000001,"15min":4.8940000000000001},"last":2.0859999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.63200000000000001,"15min":0.58299999999999996},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":2.3530000000000002,"5min":3.0049999999999999,"15min":3.0049999999999999},"last":1.8839999999999999},{"interface":"front","average":{"1min":2.4350000000000001,"5min":0.94999999999999996,"15min":0.68600000000000005},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":17.661000000000001,"5min":17.661000000000001,"15min":17.661000000000001},"last":1.8979999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:25:47 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":0.71299999999999997,"15min":0.60599999999999998},"min":{"1min":0.29599999999999999,"5min":0.25,"15min":0.25},"max":{"1min":9.8719999999999999,"5min":9.8719999999999999,"15min":9.8719999999999999},"last":3.254},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.70199999999999996,"15min":0.59799999999999998},"min":{"1min":0.26100000000000001,"5min":0.255,"15min":0.255},"max":{"1min":2.7370000000000001,"5min":2.7370000000000001,"15min":2.7370000000000001},"last":0.51000000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:25:47 2024","interfaces":[{"interface":"back","average":{"1min":2.3199999999999998,"5min":1.008,"15min":0.69599999999999995},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":9.657,"5min":9.657,"15min":9.657},"last":0.40200000000000002},{"interface":"front","average":{"1min":1.468,"5min":0.78200000000000003,"15min":0.62},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":7.4649999999999999,"5min":7.4649999999999999,"15min":7.4649999999999999},"last":1.6659999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.35999999999999999,"15min":0.35799999999999998},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.44800000000000001,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.38800000000000001,"15min":0.38400000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68899999999999995,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.42999999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:22:58 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.42399999999999999,"15min":0.40999999999999998},"min":{"1min":0.36599999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.44800000000000001,"15min":0.43099999999999999},"min":{"1min":0.34100000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.52400000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.41899999999999998,"15min":0.442},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.46500000000000002,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.42399999999999999,"15min":0.432},"min":{"1min":0.27200000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.54300000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.41499999999999998}]}]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1974940,"kb_used_data":1673428,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91770212,"statfs":{"total":95995035648,"available":93972697088,"internally_reserved":0,"allocated":1713590272,"data_stored":1713089580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:25:18 2024","interfaces":[{"interface":"back","average":{"1min":2.2010000000000001,"5min":0.86199999999999999,"15min":0.67400000000000004},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":31.585000000000001,"5min":31.585000000000001,"15min":31.585000000000001},"last":6.9870000000000001},{"interface":"front","average":{"1min":2.4990000000000001,"5min":1.0629999999999999,"15min":0.745},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":29.802,"5min":29.802,"15min":29.802},"last":2.165}]},{"osd":1,"last update":"Tue Sep 10 18:25:18 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.65700000000000003,"15min":0.61099999999999999},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.5129999999999999,"5min":4.7729999999999997,"15min":4.7729999999999997},"last":0.25},{"interface":"front","average":{"1min":0.999,"5min":0.67600000000000005,"15min":0.61099999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":4.3899999999999997,"5min":4.3899999999999997,"15min":4.3899999999999997},"last":2.6339999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:25:18 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.68600000000000005,"15min":0.60599999999999998},"min":{"1min":0.28499999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.7519999999999998,"5min":4.8040000000000003,"15min":4.8040000000000003},"last":2.282},{"interface":"front","average":{"1min":1.208,"5min":0.68600000000000005,"15min":0.622},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.1429999999999998,"5min":4.1429999999999998,"15min":4.1429999999999998},"last":2.0459999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:25:18 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":1.048,"15min":0.73799999999999999},"min":{"1min":0.314,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":30.734999999999999,"5min":30.734999999999999,"15min":30.734999999999999},"last":4.1609999999999996},{"interface":"front","average":{"1min":2.4790000000000001,"5min":1.028,"15min":0.74199999999999999},"min":{"1min":0.29799999999999999,"5min":0.255,"15min":0.255},"max":{"1min":29.952999999999999,"5min":29.952999999999999,"15min":29.952999999999999},"last":2.254}]},{"osd":5,"last update":"Tue Sep 10 18:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.214,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58099999999999996,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.38300000000000001,"15min":0.40999999999999998},"min":{"1min":0.255,"5min":0.247,"15min":0.247},"max":{"1min":0.49399999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.90000000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:22:59 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40400000000000003},"min":{"1min":0.29299999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.61599999999999999,"5min":0.629,"15min":0.629},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.46999999999999997,"15min":0.46600000000000003},"min":{"1min":0.36299999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64100000000000001,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.313}]},{"osd":7,"last update":"Tue Sep 10 18:25:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.443,"15min":0.502},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.61399999999999999,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.443,"15min":0.48899999999999999},"min":{"1min":0.27800000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.44700000000000001,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.379}]}]},{"osd":3,"up_from":27,"seq":115964117071,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3250088,"kb_used_data":2948576,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":90495064,"statfs":{"total":95995035648,"available":92666945536,"internally_reserved":0,"allocated":3019341824,"data_stored":3018829688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:25:57 2024","interfaces":[{"interface":"back","average":{"1min":2.5939999999999999,"5min":1.3400000000000001,"15min":0.69199999999999995},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":6.5099999999999998,"5min":8.5530000000000008,"15min":8.5530000000000008},"last":7.0899999999999999},{"interface":"front","average":{"1min":3.1640000000000001,"5min":1.3089999999999999,"15min":0.69599999999999995},"min":{"1min":0.36499999999999999,"5min":0.222,"15min":0.222},"max":{"1min":11.638,"5min":11.638,"15min":11.638},"last":3.4340000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:25:57 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":1.1779999999999999,"15min":0.68200000000000005},"min":{"1min":0.33300000000000002,"5min":0.247,"15min":0.247},"max":{"1min":11.606,"5min":11.606,"15min":11.606},"last":3.46},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.006,"15min":0.59799999999999998},"min":{"1min":0.33900000000000002,"5min":0.19,"15min":0.19},"max":{"1min":6.5709999999999997,"5min":8.5229999999999997,"15min":8.5229999999999997},"last":7.1210000000000004}]},{"osd":2,"last update":"Tue Sep 10 18:25:57 2024","interfaces":[{"interface":"back","average":{"1min":1.982,"5min":1.149,"15min":0.67300000000000004},"min":{"1min":0.34699999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":6.548,"5min":7.7960000000000003,"15min":7.7960000000000003},"last":7.0439999999999996},{"interface":"front","average":{"1min":2.448,"5min":1.1240000000000001,"15min":0.63400000000000001},"min":{"1min":0.35399999999999998,"5min":0.252,"15min":0.252},"max":{"1min":10.113,"5min":10.113,"15min":10.113},"last":2.3599999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:25:24 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.006,"15min":0.68999999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":3.863,"5min":23.684999999999999,"15min":23.684999999999999},"last":2.3130000000000002},{"interface":"front","average":{"1min":2.024,"5min":1.25,"15min":0.78400000000000003},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":6.0609999999999999,"5min":23.649000000000001,"15min":23.649000000000001},"last":0.86299999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:25:42 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.3009999999999999,"15min":0.82099999999999995},"min":{"1min":0.373,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":5.29,"5min":23.719999999999999,"15min":23.719999999999999},"last":2.2770000000000001},{"interface":"front","average":{"1min":2.2330000000000001,"5min":1.429,"15min":0.877},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":11.523,"5min":23.707000000000001,"15min":23.707000000000001},"last":0.89200000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.63500000000000001,"15min":0.629},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.64400000000000002,"15min":0.64600000000000002},"min":{"1min":0.41299999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.79800000000000004}]},{"osd":7,"last update":"Tue Sep 10 18:25:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.0289999999999999,"15min":0.76400000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":3.907,"5min":23.748999999999999,"15min":23.748999999999999},"last":2.3370000000000002},{"interface":"front","average":{"1min":2.0600000000000001,"5min":1.2789999999999999,"15min":0.86899999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":6.1130000000000004,"5min":25.058,"15min":25.058},"last":0.91200000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2278476,"kb_used_data":1976964,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91466676,"statfs":{"total":95995035648,"available":93661876224,"internally_reserved":0,"allocated":2024411136,"data_stored":2023926300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:25:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4420000000000002,"5min":1.2669999999999999,"15min":0.63600000000000001},"min":{"1min":0.217,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":12.06,"5min":17.135000000000002,"15min":17.135000000000002},"last":0.218},{"interface":"front","average":{"1min":2.8919999999999999,"5min":1.2829999999999999,"15min":0.65300000000000002},"min":{"1min":0.51700000000000002,"5min":0.224,"15min":0.224},"max":{"1min":7.1210000000000004,"5min":17.146000000000001,"15min":17.146000000000001},"last":0.16900000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:25:33 2024","interfaces":[{"interface":"back","average":{"1min":3.1520000000000001,"5min":1.2150000000000001,"15min":0.625},"min":{"1min":0.23499999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":10.077999999999999,"5min":10.077999999999999,"15min":10.077999999999999},"last":0.22900000000000001},{"interface":"front","average":{"1min":2.1880000000000002,"5min":0.96999999999999997,"15min":0.54800000000000004},"min":{"1min":0.27900000000000003,"5min":0.215,"15min":0.215},"max":{"1min":12.532999999999999,"5min":12.532999999999999,"15min":12.532999999999999},"last":0.19900000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.5760000000000001,"5min":1.6970000000000001,"15min":0.874},"min":{"1min":0.29299999999999998,"5min":0.251,"15min":0.251},"max":{"1min":12.407,"5min":60.786999999999999,"15min":60.786999999999999},"last":5.0199999999999996},{"interface":"front","average":{"1min":3.024,"5min":1.7929999999999999,"15min":0.88500000000000001},"min":{"1min":0.38100000000000001,"5min":0.25,"15min":0.25},"max":{"1min":8.5259999999999998,"5min":60.116999999999997,"15min":60.116999999999997},"last":5.0869999999999997}]},{"osd":4,"last update":"Tue Sep 10 18:25:18 2024","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":0.82999999999999996,"15min":0.65200000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":5.5460000000000003,"5min":5.5460000000000003,"15min":5.5460000000000003},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.6739999999999999,"5min":0.86199999999999999,"15min":0.70099999999999996},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":12.130000000000001,"5min":12.130000000000001,"15min":12.130000000000001},"last":0.35399999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:25:42 2024","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":0.91600000000000004,"15min":0.69399999999999995},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":12.106,"5min":12.106,"15min":12.106},"last":0.27500000000000002},{"interface":"front","average":{"1min":1.665,"5min":0.93300000000000005,"15min":0.71699999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":12.154,"5min":12.154,"15min":12.154},"last":0.31900000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:23:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.88300000000000001,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.625,"15min":0.628},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.69799999999999995}]},{"osd":7,"last update":"Tue Sep 10 18:25:33 2024","interfaces":[{"interface":"back","average":{"1min":1.8740000000000001,"5min":0.95899999999999996,"15min":0.74099999999999999},"min":{"1min":0.39000000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":5.6539999999999999,"5min":7.1639999999999997,"15min":7.1639999999999997},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.873,"5min":0.94799999999999995,"15min":0.72399999999999998},"min":{"1min":0.34399999999999997,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":6.9669999999999996,"5min":6.9669999999999996,"15min":6.9669999999999996},"last":0.307}]}]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2273324,"kb_used_data":1963364,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91471828,"statfs":{"total":95995035648,"available":93667151872,"internally_reserved":0,"allocated":2010484736,"data_stored":2009968286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:25:09 2024","interfaces":[{"interface":"back","average":{"1min":5.2389999999999999,"5min":1.4199999999999999,"15min":0.69899999999999995},"min":{"1min":0.187,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":80.456999999999994,"5min":80.456999999999994,"15min":80.456999999999994},"last":1.361},{"interface":"front","average":{"1min":5.7169999999999996,"5min":1.5309999999999999,"15min":0.753},"min":{"1min":0.33300000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":80.578000000000003,"5min":80.578000000000003,"15min":80.578000000000003},"last":2.0870000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:25:27 2024","interfaces":[{"interface":"back","average":{"1min":1.9850000000000001,"5min":1.0049999999999999,"15min":0.58099999999999996},"min":{"1min":0.27700000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":6.7389999999999999,"5min":13.035,"15min":13.035},"last":2.0579999999999998},{"interface":"front","average":{"1min":1.4770000000000001,"5min":0.76300000000000001,"15min":0.49099999999999999},"min":{"1min":0.248,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.8179999999999996,"5min":4.8179999999999996,"15min":4.8179999999999996},"last":2.1219999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:25:58 2024","interfaces":[{"interface":"back","average":{"1min":4.2199999999999998,"5min":1.746,"15min":0.85999999999999999},"min":{"1min":0.28199999999999997,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":36.029000000000003,"5min":36.029000000000003,"15min":36.029000000000003},"last":0.66200000000000003},{"interface":"front","average":{"1min":4.0960000000000001,"5min":1.8029999999999999,"15min":0.91500000000000004},"min":{"1min":0.40699999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":36.136000000000003,"5min":36.136000000000003,"15min":36.136000000000003},"last":2.4769999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:25:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":0.751,"15min":0.64100000000000001},"min":{"1min":0.32900000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":7.1500000000000004,"5min":7.1500000000000004,"15min":7.1500000000000004},"last":0.38100000000000001},{"interface":"front","average":{"1min":1.123,"5min":0.92900000000000005,"15min":0.69599999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.165,"5min":19.393000000000001,"15min":19.393000000000001},"last":0.51100000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:25:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":0.93600000000000005,"15min":0.67800000000000005},"min":{"1min":0.32700000000000001,"5min":0.311,"15min":0.311},"max":{"1min":4.2119999999999997,"5min":15.48,"15min":15.48},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.79400000000000004,"15min":0.66800000000000004},"min":{"1min":0.36199999999999999,"5min":0.308,"15min":0.308},"max":{"1min":2.7759999999999998,"5min":4.7690000000000001,"15min":4.7690000000000001},"last":0.44800000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:23:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.60899999999999999,"15min":0.61099999999999999},"min":{"1min":0.435,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78700000000000003,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84199999999999997,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.70499999999999996}]},{"osd":7,"last update":"Tue Sep 10 18:25:27 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":0.91100000000000003,"15min":0.70199999999999996},"min":{"1min":0.36199999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":15.545999999999999,"5min":15.545999999999999,"15min":15.545999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.2829999999999999,"5min":0.80200000000000005,"15min":0.65600000000000003},"min":{"1min":0.33400000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":7.1799999999999997,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":0.41499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2933536,"kb_used_data":2623576,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90811616,"statfs":{"total":95995035648,"available":92991094784,"internally_reserved":0,"allocated":2686541824,"data_stored":2686037477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:25:08 2024","interfaces":[{"interface":"back","average":{"1min":2.6549999999999998,"5min":0.878,"15min":0.52200000000000002},"min":{"1min":0.434,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":12.106999999999999,"5min":12.106999999999999,"15min":12.106999999999999},"last":5.1459999999999999},{"interface":"front","average":{"1min":2.5939999999999999,"5min":1.03,"15min":0.57199999999999995},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":13.436,"5min":13.436,"15min":13.436},"last":1.518}]},{"osd":2,"last update":"Tue Sep 10 18:25:32 2024","interfaces":[{"interface":"back","average":{"1min":2.2429999999999999,"5min":1.0940000000000001,"15min":0.63500000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":13.332000000000001,"5min":13.332000000000001,"15min":13.332000000000001},"last":1.337},{"interface":"front","average":{"1min":2.508,"5min":1.0569999999999999,"15min":0.61899999999999999},"min":{"1min":0.26600000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":12.226000000000001,"5min":12.226000000000001,"15min":12.226000000000001},"last":1.494}]},{"osd":3,"last update":"Tue Sep 10 18:25:57 2024","interfaces":[{"interface":"back","average":{"1min":3.2879999999999998,"5min":1.5920000000000001,"15min":0.83399999999999996},"min":{"1min":0.34599999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":23.143999999999998,"5min":23.143999999999998,"15min":23.143999999999998},"last":4.3600000000000003},{"interface":"front","average":{"1min":2.8879999999999999,"5min":1.5609999999999999,"15min":0.82099999999999995},"min":{"1min":0.36099999999999999,"5min":0.248,"15min":0.248},"max":{"1min":23.035,"5min":23.035,"15min":23.035},"last":4.9269999999999996}]},{"osd":4,"last update":"Tue Sep 10 18:25:22 2024","interfaces":[{"interface":"back","average":{"1min":3.1469999999999998,"5min":1.1399999999999999,"15min":0.80300000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":17.151,"5min":17.151,"15min":17.151},"last":4.8899999999999997},{"interface":"front","average":{"1min":1.99,"5min":1.1279999999999999,"15min":0.81999999999999995},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":13.651,"5min":13.651,"15min":13.651},"last":1.1140000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:25:43 2024","interfaces":[{"interface":"back","average":{"1min":2.165,"5min":1.381,"15min":0.86499999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":17.195,"5min":17.195,"15min":17.195},"last":1.083},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.131,"15min":0.78300000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":7.6520000000000001,"5min":14.991,"15min":14.991},"last":0.39900000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:23:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60899999999999999,"15min":0.61299999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.61899999999999999,"15min":0.624},"min":{"1min":0.46000000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.76800000000000002,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57899999999999996}]},{"osd":7,"last update":"Tue Sep 10 18:25:25 2024","interfaces":[{"interface":"back","average":{"1min":2.1720000000000002,"5min":1.1599999999999999,"15min":0.77000000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":13.711,"5min":13.711,"15min":13.711},"last":1.135},{"interface":"front","average":{"1min":2.335,"5min":1.081,"15min":0.74099999999999999},"min":{"1min":0.36599999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":15.032,"5min":15.032,"15min":15.032},"last":0.433}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2669940736,"data_stored":2669936664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2003505152,"data_stored":2003501080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2008743936,"data_stored":2008743936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3017015296,"data_stored":3017015296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1693515776,"data_stored":1693515776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1325006848,"data_stored":1325006848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1680211968,"data_stored":1680211968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:26:08.110 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-10T18:26:08.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-10T18:26:08.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:08.237 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:08.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:08.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:08.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[29291]: 7.2 scrub ok 2024-09-10T18:26:08.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[29291]: 3.19 scrub starts 2024-09-10T18:26:08.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[29291]: 3.19 scrub ok 2024-09-10T18:26:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[29291]: 4.16 scrub starts 2024-09-10T18:26:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[29291]: 4.16 scrub ok 2024-09-10T18:26:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[29291]: 4.9 scrub starts 2024-09-10T18:26:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[29291]: 4.9 scrub ok 2024-09-10T18:26:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[29291]: pgmap v178: 148 pgs: 3 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s; 25597/245646 objects degraded (10.420%) 2024-09-10T18:26:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[29291]: from='client.14946 v1:172.21.15.28:0/3094098683' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:26:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[29291]: 7.a scrub ok 2024-09-10T18:26:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[29291]: 6.7 scrub starts 2024-09-10T18:26:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[29291]: 6.7 scrub ok 2024-09-10T18:26:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[29291]: 7.6 scrub ok 2024-09-10T18:26:08.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[39220]: 7.2 scrub ok 2024-09-10T18:26:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[39220]: 3.19 scrub starts 2024-09-10T18:26:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[39220]: 3.19 scrub ok 2024-09-10T18:26:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[39220]: 4.16 scrub starts 2024-09-10T18:26:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[39220]: 4.16 scrub ok 2024-09-10T18:26:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[39220]: 4.9 scrub starts 2024-09-10T18:26:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[39220]: 4.9 scrub ok 2024-09-10T18:26:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[39220]: pgmap v178: 148 pgs: 3 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s; 25597/245646 objects degraded (10.420%) 2024-09-10T18:26:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[39220]: from='client.14946 v1:172.21.15.28:0/3094098683' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:26:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[39220]: 7.a scrub ok 2024-09-10T18:26:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[39220]: 6.7 scrub starts 2024-09-10T18:26:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[39220]: 6.7 scrub ok 2024-09-10T18:26:08.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:07 smithi028 ceph-mon[39220]: 7.6 scrub ok 2024-09-10T18:26:08.339 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:07 smithi204 ceph-mon[32357]: 7.2 scrub ok 2024-09-10T18:26:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:07 smithi204 ceph-mon[32357]: 3.19 scrub starts 2024-09-10T18:26:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:07 smithi204 ceph-mon[32357]: 3.19 scrub ok 2024-09-10T18:26:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:07 smithi204 ceph-mon[32357]: 4.16 scrub starts 2024-09-10T18:26:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:07 smithi204 ceph-mon[32357]: 4.16 scrub ok 2024-09-10T18:26:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:07 smithi204 ceph-mon[32357]: 4.9 scrub starts 2024-09-10T18:26:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:07 smithi204 ceph-mon[32357]: 4.9 scrub ok 2024-09-10T18:26:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:07 smithi204 ceph-mon[32357]: pgmap v178: 148 pgs: 3 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s; 25597/245646 objects degraded (10.420%) 2024-09-10T18:26:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:07 smithi204 ceph-mon[32357]: from='client.14946 v1:172.21.15.28:0/3094098683' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:26:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:07 smithi204 ceph-mon[32357]: 7.a scrub ok 2024-09-10T18:26:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:07 smithi204 ceph-mon[32357]: 6.7 scrub starts 2024-09-10T18:26:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:07 smithi204 ceph-mon[32357]: 6.7 scrub ok 2024-09-10T18:26:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:07 smithi204 ceph-mon[32357]: 7.6 scrub ok 2024-09-10T18:26:08.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:08.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:08.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:08.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:08.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:08.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:08.599 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-10T18:26:08.617 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:08.669 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:08.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:08.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:08.671 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:26:08.671 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:26:08.705 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:26:08.705 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:26:08.741 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:26:08.741 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:26:08.771 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:08.773 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:26:08.773 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:26:08.794 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:08.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:08.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:08.804 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:26:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:26:08.844 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:26:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:26:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-10T18:26:08.886 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:26:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:26:08.894 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:26:08.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:08.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:09.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:09.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:09.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:09.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:09.214 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:09.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:09.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:09.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:08 smithi028 ceph-mon[39220]: 7.f scrub starts 2024-09-10T18:26:09.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:08 smithi028 ceph-mon[39220]: 5.1 scrub starts 2024-09-10T18:26:09.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:08 smithi028 ceph-mon[39220]: 5.1 scrub ok 2024-09-10T18:26:09.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:08 smithi028 ceph-mon[39220]: 7.3 scrub starts 2024-09-10T18:26:09.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:08 smithi028 ceph-mon[39220]: 6.14 scrub starts 2024-09-10T18:26:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:08 smithi028 ceph-mon[39220]: 6.14 scrub ok 2024-09-10T18:26:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:08 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/297102757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:26:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:08 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:26:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:08 smithi028 ceph-mon[29291]: 7.f scrub starts 2024-09-10T18:26:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:08 smithi028 ceph-mon[29291]: 5.1 scrub starts 2024-09-10T18:26:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:08 smithi028 ceph-mon[29291]: 5.1 scrub ok 2024-09-10T18:26:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:08 smithi028 ceph-mon[29291]: 7.3 scrub starts 2024-09-10T18:26:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:08 smithi028 ceph-mon[29291]: 6.14 scrub starts 2024-09-10T18:26:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:08 smithi028 ceph-mon[29291]: 6.14 scrub ok 2024-09-10T18:26:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:08 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/297102757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:26:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:08 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:26:09.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:08 smithi204 ceph-mon[32357]: 7.f scrub starts 2024-09-10T18:26:09.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:08 smithi204 ceph-mon[32357]: 5.1 scrub starts 2024-09-10T18:26:09.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:08 smithi204 ceph-mon[32357]: 5.1 scrub ok 2024-09-10T18:26:09.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:08 smithi204 ceph-mon[32357]: 7.3 scrub starts 2024-09-10T18:26:09.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:08 smithi204 ceph-mon[32357]: 6.14 scrub starts 2024-09-10T18:26:09.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:08 smithi204 ceph-mon[32357]: 6.14 scrub ok 2024-09-10T18:26:09.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:08 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/297102757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:26:09.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:08 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:26:09.316 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:09.339 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:09.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:09.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:09.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:09.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:09.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:09.585 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:09.609 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:09.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:09.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:09.711 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:09.734 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:09.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:09.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:09.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:09.859 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:09.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:09.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:09.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:09.983 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:09.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:09.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:09.998 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:26:10.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:10.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:10.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:10.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T18:26:10.173 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T18:26:10.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:10.250 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:10.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:10.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:10 smithi028 ceph-mon[39220]: pgmap v179: 148 pgs: 3 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 25597/245646 objects degraded (10.420%) 2024-09-10T18:26:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:10 smithi028 ceph-mon[39220]: 3.e scrub starts 2024-09-10T18:26:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:10 smithi028 ceph-mon[39220]: 3.e scrub ok 2024-09-10T18:26:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:10 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2132409054' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:26:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:10 smithi028 ceph-mon[39220]: 5.1b scrub starts 2024-09-10T18:26:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:10 smithi028 ceph-mon[39220]: 5.1b scrub ok 2024-09-10T18:26:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:10 smithi028 ceph-mon[29291]: pgmap v179: 148 pgs: 3 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 25597/245646 objects degraded (10.420%) 2024-09-10T18:26:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:10 smithi028 ceph-mon[29291]: 3.e scrub starts 2024-09-10T18:26:10.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:10 smithi028 ceph-mon[29291]: 3.e scrub ok 2024-09-10T18:26:10.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:10 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2132409054' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:26:10.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:10 smithi028 ceph-mon[29291]: 5.1b scrub starts 2024-09-10T18:26:10.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:10 smithi028 ceph-mon[29291]: 5.1b scrub ok 2024-09-10T18:26:10.316 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:10 smithi204 ceph-mon[32357]: pgmap v179: 148 pgs: 3 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 97 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 25597/245646 objects degraded (10.420%) 2024-09-10T18:26:10.317 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:10 smithi204 ceph-mon[32357]: 3.e scrub starts 2024-09-10T18:26:10.317 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:10 smithi204 ceph-mon[32357]: 3.e scrub ok 2024-09-10T18:26:10.317 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:10 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2132409054' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:26:10.317 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:10 smithi204 ceph-mon[32357]: 5.1b scrub starts 2024-09-10T18:26:10.317 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:10 smithi204 ceph-mon[32357]: 5.1b scrub ok 2024-09-10T18:26:10.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:10.362 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:26:10.375 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:10.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:10.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:10.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:10.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:10.525 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:10.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:10.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:10.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:10.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:10.733 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:26:10.733 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:26:10.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:10.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:10.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:10.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:10.910 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:10.934 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:10.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:10.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:11.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:11.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:11.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:11.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[39220]: 4.b scrub starts 2024-09-10T18:26:11.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[39220]: 4.b scrub ok 2024-09-10T18:26:11.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[39220]: 5.1c scrub starts 2024-09-10T18:26:11.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[39220]: 5.1c scrub ok 2024-09-10T18:26:11.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:26:11.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2132409054' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:26:11.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[39220]: osdmap e132: 8 total, 7 up, 8 in 2024-09-10T18:26:11.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[39220]: 5.13 deep-scrub starts 2024-09-10T18:26:11.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[39220]: 5.13 deep-scrub ok 2024-09-10T18:26:11.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[39220]: 2.1 scrub starts 2024-09-10T18:26:11.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[39220]: 2.1 scrub ok 2024-09-10T18:26:11.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 25597/245646 objects degraded (10.420%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:11.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[39220]: osdmap e133: 8 total, 7 up, 8 in 2024-09-10T18:26:11.102 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[29291]: 4.b scrub starts 2024-09-10T18:26:11.102 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[29291]: 4.b scrub ok 2024-09-10T18:26:11.102 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[29291]: 5.1c scrub starts 2024-09-10T18:26:11.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[29291]: 5.1c scrub ok 2024-09-10T18:26:11.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:26:11.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2132409054' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:26:11.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[29291]: osdmap e132: 8 total, 7 up, 8 in 2024-09-10T18:26:11.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[29291]: 5.13 deep-scrub starts 2024-09-10T18:26:11.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[29291]: 5.13 deep-scrub ok 2024-09-10T18:26:11.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[29291]: 2.1 scrub starts 2024-09-10T18:26:11.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[29291]: 2.1 scrub ok 2024-09-10T18:26:11.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 25597/245646 objects degraded (10.420%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:11.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-mon[29291]: osdmap e133: 8 total, 7 up, 8 in 2024-09-10T18:26:11.175 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:11.198 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:11.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:11.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:11.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:11.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:11.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:11 smithi204 ceph-mon[32357]: 4.b scrub starts 2024-09-10T18:26:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:11 smithi204 ceph-mon[32357]: 4.b scrub ok 2024-09-10T18:26:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:11 smithi204 ceph-mon[32357]: 5.1c scrub starts 2024-09-10T18:26:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:11 smithi204 ceph-mon[32357]: 5.1c scrub ok 2024-09-10T18:26:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:11 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:26:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:11 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2132409054' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:26:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:11 smithi204 ceph-mon[32357]: osdmap e132: 8 total, 7 up, 8 in 2024-09-10T18:26:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:11 smithi204 ceph-mon[32357]: 5.13 deep-scrub starts 2024-09-10T18:26:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:11 smithi204 ceph-mon[32357]: 5.13 deep-scrub ok 2024-09-10T18:26:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:11 smithi204 ceph-mon[32357]: 2.1 scrub starts 2024-09-10T18:26:11.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:11 smithi204 ceph-mon[32357]: 2.1 scrub ok 2024-09-10T18:26:11.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:11 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 25597/245646 objects degraded (10.420%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:11.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:11 smithi204 ceph-mon[32357]: osdmap e133: 8 total, 7 up, 8 in 2024-09-10T18:26:11.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:11.480 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:11.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:11.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:11.487 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-10T18:26:11.503 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:26:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:26:11] "GET /metrics HTTP/1.1" 200 223869 "" "Prometheus/2.43.0" 2024-09-10T18:26:11.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:11.625 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:11.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:11.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:11.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:11.766 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:11.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:11.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:11.800 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:26:11.868 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:11.893 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:11.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:11.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:11.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:12.037 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:12.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:12.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:12.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:12.163 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:12.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:12.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:12.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:26:12.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:12 smithi028 ceph-mon[29291]: 7.3 scrub ok 2024-09-10T18:26:12.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:12 smithi028 ceph-mon[29291]: pgmap v182: 148 pgs: 2 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 24130/231489 objects degraded (10.424%) 2024-09-10T18:26:12.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:12 smithi028 ceph-mon[29291]: 7.f scrub ok 2024-09-10T18:26:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:12 smithi028 ceph-mon[39220]: 7.3 scrub ok 2024-09-10T18:26:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:12 smithi028 ceph-mon[39220]: pgmap v182: 148 pgs: 2 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 24130/231489 objects degraded (10.424%) 2024-09-10T18:26:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:12 smithi028 ceph-mon[39220]: 7.f scrub ok 2024-09-10T18:26:12.262 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:12 smithi204 ceph-mon[32357]: 7.3 scrub ok 2024-09-10T18:26:12.262 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:12 smithi204 ceph-mon[32357]: pgmap v182: 148 pgs: 2 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 24130/231489 objects degraded (10.424%) 2024-09-10T18:26:12.262 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:12 smithi204 ceph-mon[32357]: 7.f scrub ok 2024-09-10T18:26:12.262 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:26:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:26:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:26:12.265 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:12.288 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:12.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:12.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:12.460 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:12.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:12.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:12.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:12.615 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:12.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:12.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:12.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:12.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:12.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:26:12.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:26:12.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:12.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:12.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:13.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:13.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:13.110 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:13.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:13.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:13.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:13 smithi028 ceph-mon[39220]: 4.1a scrub starts 2024-09-10T18:26:13.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:13 smithi028 ceph-mon[39220]: 4.1a scrub ok 2024-09-10T18:26:13.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:13 smithi028 ceph-mon[39220]: 6.b deep-scrub starts 2024-09-10T18:26:13.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:13 smithi028 ceph-mon[39220]: 6.b deep-scrub ok 2024-09-10T18:26:13.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:13 smithi028 ceph-mon[39220]: 7.8 scrub starts 2024-09-10T18:26:13.255 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:13.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:13 smithi028 ceph-mon[29291]: 4.1a scrub starts 2024-09-10T18:26:13.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:13 smithi028 ceph-mon[29291]: 4.1a scrub ok 2024-09-10T18:26:13.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:13 smithi028 ceph-mon[29291]: 6.b deep-scrub starts 2024-09-10T18:26:13.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:13 smithi028 ceph-mon[29291]: 6.b deep-scrub ok 2024-09-10T18:26:13.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:13 smithi028 ceph-mon[29291]: 7.8 scrub starts 2024-09-10T18:26:13.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:13.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:13.285 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:13 smithi204 ceph-mon[32357]: 4.1a scrub starts 2024-09-10T18:26:13.286 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:13 smithi204 ceph-mon[32357]: 4.1a scrub ok 2024-09-10T18:26:13.286 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:13 smithi204 ceph-mon[32357]: 6.b deep-scrub starts 2024-09-10T18:26:13.286 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:13 smithi204 ceph-mon[32357]: 6.b deep-scrub ok 2024-09-10T18:26:13.286 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:13 smithi204 ceph-mon[32357]: 7.8 scrub starts 2024-09-10T18:26:13.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:13.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:13.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:13.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:13.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:13.525 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:13.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:13.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:13.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:13.666 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:13.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:13.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:13.671 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:26:13.671 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:26:13.768 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:13.793 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:13.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:13.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:13.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:13.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:13.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:14.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:14.054 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 1 pg_num to 11 2024-09-10T18:26:14.058 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:14.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:14.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:14.160 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:14.185 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:14.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:14.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:14.287 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:14.310 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:14.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:14.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:14.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:26:14 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:26:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:14 smithi204 ceph-mon[32357]: pgmap v183: 148 pgs: 2 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s; 24130/231489 objects degraded (10.424%) 2024-09-10T18:26:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:14 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:26:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:14 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:26:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:14 smithi204 ceph-mon[32357]: 7.5 scrub starts 2024-09-10T18:26:14.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:14 smithi204 ceph-mon[32357]: osdmap e134: 8 total, 7 up, 8 in 2024-09-10T18:26:14.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:14 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/260469379' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T18:26:14.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:14 smithi204 ceph-mon[32357]: from='client.34847 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T18:26:14.412 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:14.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T18:26:14.439 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:14.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:14.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:14.494 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:14 smithi028 ceph-mon[39220]: pgmap v183: 148 pgs: 2 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s; 24130/231489 objects degraded (10.424%) 2024-09-10T18:26:14.494 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:14 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:26:14.494 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:14 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:26:14.494 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:14 smithi028 ceph-mon[39220]: 7.5 scrub starts 2024-09-10T18:26:14.494 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:14 smithi028 ceph-mon[39220]: osdmap e134: 8 total, 7 up, 8 in 2024-09-10T18:26:14.494 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:14 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/260469379' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T18:26:14.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:14 smithi028 ceph-mon[39220]: from='client.34847 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T18:26:14.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:14 smithi028 ceph-mon[29291]: pgmap v183: 148 pgs: 2 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s; 24130/231489 objects degraded (10.424%) 2024-09-10T18:26:14.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:14 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:26:14.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:14 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:26:14.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:14 smithi028 ceph-mon[29291]: 7.5 scrub starts 2024-09-10T18:26:14.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:14 smithi028 ceph-mon[29291]: osdmap e134: 8 total, 7 up, 8 in 2024-09-10T18:26:14.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:14 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/260469379' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T18:26:14.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:14 smithi028 ceph-mon[29291]: from='client.34847 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T18:26:14.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:14.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:14.586 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:14.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:14.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:14.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:14.742 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:14.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:14.842 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:14.868 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:14.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:14.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:14.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:15.018 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:15.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:15.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:15.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:15.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:15.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[29291]: 3.1a deep-scrub starts 2024-09-10T18:26:15.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[29291]: 3.1a deep-scrub ok 2024-09-10T18:26:15.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[29291]: 5.4 scrub starts 2024-09-10T18:26:15.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[29291]: 5.4 scrub ok 2024-09-10T18:26:15.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[29291]: 7.8 scrub ok 2024-09-10T18:26:15.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[29291]: from='client.34847 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-10T18:26:15.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[29291]: osdmap e135: 8 total, 7 up, 8 in 2024-09-10T18:26:15.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:15.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[29291]: 7.e scrub starts 2024-09-10T18:26:15.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[29291]: 5.15 scrub starts 2024-09-10T18:26:15.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[29291]: 5.15 scrub ok 2024-09-10T18:26:15.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[29291]: 3.1b scrub starts 2024-09-10T18:26:15.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[29291]: 3.1b scrub ok 2024-09-10T18:26:15.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T18:26:15.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T18:26:15.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[39220]: 3.1a deep-scrub starts 2024-09-10T18:26:15.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[39220]: 3.1a deep-scrub ok 2024-09-10T18:26:15.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[39220]: 5.4 scrub starts 2024-09-10T18:26:15.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[39220]: 5.4 scrub ok 2024-09-10T18:26:15.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[39220]: 7.8 scrub ok 2024-09-10T18:26:15.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[39220]: from='client.34847 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-10T18:26:15.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[39220]: osdmap e135: 8 total, 7 up, 8 in 2024-09-10T18:26:15.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:15.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[39220]: 7.e scrub starts 2024-09-10T18:26:15.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[39220]: 5.15 scrub starts 2024-09-10T18:26:15.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[39220]: 5.15 scrub ok 2024-09-10T18:26:15.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[39220]: 3.1b scrub starts 2024-09-10T18:26:15.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[39220]: 3.1b scrub ok 2024-09-10T18:26:15.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T18:26:15.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:15 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T18:26:15.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:15.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:15.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:15 smithi204 ceph-mon[32357]: 3.1a deep-scrub starts 2024-09-10T18:26:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:15 smithi204 ceph-mon[32357]: 3.1a deep-scrub ok 2024-09-10T18:26:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:15 smithi204 ceph-mon[32357]: 5.4 scrub starts 2024-09-10T18:26:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:15 smithi204 ceph-mon[32357]: 5.4 scrub ok 2024-09-10T18:26:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:15 smithi204 ceph-mon[32357]: 7.8 scrub ok 2024-09-10T18:26:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:15 smithi204 ceph-mon[32357]: from='client.34847 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-10T18:26:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:15 smithi204 ceph-mon[32357]: osdmap e135: 8 total, 7 up, 8 in 2024-09-10T18:26:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:15 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:15 smithi204 ceph-mon[32357]: 7.e scrub starts 2024-09-10T18:26:15.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:15 smithi204 ceph-mon[32357]: 5.15 scrub starts 2024-09-10T18:26:15.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:15 smithi204 ceph-mon[32357]: 5.15 scrub ok 2024-09-10T18:26:15.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:15 smithi204 ceph-mon[32357]: 3.1b scrub starts 2024-09-10T18:26:15.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:15 smithi204 ceph-mon[32357]: 3.1b scrub ok 2024-09-10T18:26:15.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:15 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T18:26:15.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:15 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T18:26:15.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:15.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:15.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:15.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:15.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:15.717 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:15.740 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:15.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:15.842 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:15.866 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:15.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:15.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:26:15.967 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:26:15.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:16.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:16.022 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:16.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:16.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:16.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:16.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:16.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:16.219 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:26:16.219 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:26:16.219 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[39220]: 7.5 scrub ok 2024-09-10T18:26:16.219 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:26:16.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:26:16.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[39220]: pgmap v186: 148 pgs: 2 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 732 op/s; 23822/228816 objects degraded (10.411%) 2024-09-10T18:26:16.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-10T18:26:16.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[39220]: osdmap e136: 8 total, 7 up, 8 in 2024-09-10T18:26:16.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 23822/228816 objects degraded (10.411%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:16.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[39220]: osdmap e137: 8 total, 7 up, 8 in 2024-09-10T18:26:16.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:26:16.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:26:16.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[29291]: 7.5 scrub ok 2024-09-10T18:26:16.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:26:16.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:26:16.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[29291]: pgmap v186: 148 pgs: 2 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 732 op/s; 23822/228816 objects degraded (10.411%) 2024-09-10T18:26:16.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-10T18:26:16.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[29291]: osdmap e136: 8 total, 7 up, 8 in 2024-09-10T18:26:16.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 23822/228816 objects degraded (10.411%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:16.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:16 smithi028 ceph-mon[29291]: osdmap e137: 8 total, 7 up, 8 in 2024-09-10T18:26:16.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:16.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:16.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:16 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:26:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:16 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:26:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:16 smithi204 ceph-mon[32357]: 7.5 scrub ok 2024-09-10T18:26:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:16 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:26:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:16 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:26:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:16 smithi204 ceph-mon[32357]: pgmap v186: 148 pgs: 2 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 732 op/s; 23822/228816 objects degraded (10.411%) 2024-09-10T18:26:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:16 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-10T18:26:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:16 smithi204 ceph-mon[32357]: osdmap e136: 8 total, 7 up, 8 in 2024-09-10T18:26:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:16 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 23822/228816 objects degraded (10.411%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:16.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:16 smithi204 ceph-mon[32357]: osdmap e137: 8 total, 7 up, 8 in 2024-09-10T18:26:16.415 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:16.439 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:16.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:16.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:16.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:16.564 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:16.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:16.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:16.665 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:16.689 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:16.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:16.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:16.791 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:16.814 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:16.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:16.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:16.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:16.962 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:16.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:16.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:17.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:17.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:17.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:17.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:17.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:17 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T18:26:17.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:17 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T18:26:17.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:17 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T18:26:17.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:17 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T18:26:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:17.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:17.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:17 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T18:26:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:17 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T18:26:17.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:17.394 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:17.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:17.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:17.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:17.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:17.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:17.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:17.669 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:17.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:17.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:17.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:17.781 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:26:17.782 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:26:17.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:17.817 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:26:17.818 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:26:17.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:17.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:17.857 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:26:17.857 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:26:17.893 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:26:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:26:17.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:17.926 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:26:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:26:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:17.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:17.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:17.959 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:26:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:26:17.960 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-10T18:26:17.994 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:26:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:26:18.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:18.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:18.146 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:18.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:18.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:18.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[29291]: 7.e scrub ok 2024-09-10T18:26:18.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[29291]: 4.d scrub starts 2024-09-10T18:26:18.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[29291]: 4.d scrub ok 2024-09-10T18:26:18.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[29291]: pgmap v189: 158 pgs: 10 unknown, 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 22666/217245 objects degraded (10.433%) 2024-09-10T18:26:18.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-10T18:26:18.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[29291]: osdmap e138: 8 total, 7 up, 8 in 2024-09-10T18:26:18.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[29291]: 6.f scrub starts 2024-09-10T18:26:18.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[29291]: 3.1c deep-scrub starts 2024-09-10T18:26:18.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[29291]: 3.1c deep-scrub ok 2024-09-10T18:26:18.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[29291]: 6.0 scrub starts 2024-09-10T18:26:18.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/357184155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:26:18.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[39220]: 7.e scrub ok 2024-09-10T18:26:18.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[39220]: 4.d scrub starts 2024-09-10T18:26:18.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[39220]: 4.d scrub ok 2024-09-10T18:26:18.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[39220]: pgmap v189: 158 pgs: 10 unknown, 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 22666/217245 objects degraded (10.433%) 2024-09-10T18:26:18.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-10T18:26:18.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[39220]: osdmap e138: 8 total, 7 up, 8 in 2024-09-10T18:26:18.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[39220]: 6.f scrub starts 2024-09-10T18:26:18.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[39220]: 3.1c deep-scrub starts 2024-09-10T18:26:18.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[39220]: 3.1c deep-scrub ok 2024-09-10T18:26:18.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[39220]: 6.0 scrub starts 2024-09-10T18:26:18.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:18 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/357184155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:26:18.291 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:18.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:18.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:18 smithi204 ceph-mon[32357]: 7.e scrub ok 2024-09-10T18:26:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:18 smithi204 ceph-mon[32357]: 4.d scrub starts 2024-09-10T18:26:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:18 smithi204 ceph-mon[32357]: 4.d scrub ok 2024-09-10T18:26:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:18 smithi204 ceph-mon[32357]: pgmap v189: 158 pgs: 10 unknown, 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 22666/217245 objects degraded (10.433%) 2024-09-10T18:26:18.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:18 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-10T18:26:18.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:18 smithi204 ceph-mon[32357]: osdmap e138: 8 total, 7 up, 8 in 2024-09-10T18:26:18.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:18 smithi204 ceph-mon[32357]: 6.f scrub starts 2024-09-10T18:26:18.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:18 smithi204 ceph-mon[32357]: 3.1c deep-scrub starts 2024-09-10T18:26:18.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:18 smithi204 ceph-mon[32357]: 3.1c deep-scrub ok 2024-09-10T18:26:18.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:18 smithi204 ceph-mon[32357]: 6.0 scrub starts 2024-09-10T18:26:18.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:18 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/357184155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:26:18.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:18.437 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:18.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:18.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:18.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:18.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:18.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:18.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:18.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:18.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:18.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:18.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:18.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:18.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:18.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:18.960 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:18.982 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:18.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:18.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:19.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:19.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:19.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:19.279 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:19.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:19.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:19 smithi204 ceph-mon[32357]: 6.f scrub ok 2024-09-10T18:26:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:19 smithi204 ceph-mon[32357]: 5.1d scrub starts 2024-09-10T18:26:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:19 smithi204 ceph-mon[32357]: 5.1d scrub ok 2024-09-10T18:26:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:19 smithi204 ceph-mon[32357]: 6.0 scrub ok 2024-09-10T18:26:19.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:19 smithi204 ceph-mon[32357]: 6.11 scrub starts 2024-09-10T18:26:19.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:19 smithi204 ceph-mon[32357]: 6.11 scrub ok 2024-09-10T18:26:19.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:19 smithi204 ceph-mon[32357]: osdmap e139: 8 total, 7 up, 8 in 2024-09-10T18:26:19.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:19 smithi028 ceph-mon[29291]: 6.f scrub ok 2024-09-10T18:26:19.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:19 smithi028 ceph-mon[29291]: 5.1d scrub starts 2024-09-10T18:26:19.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:19 smithi028 ceph-mon[29291]: 5.1d scrub ok 2024-09-10T18:26:19.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:19 smithi028 ceph-mon[29291]: 6.0 scrub ok 2024-09-10T18:26:19.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:19 smithi028 ceph-mon[29291]: 6.11 scrub starts 2024-09-10T18:26:19.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:19 smithi028 ceph-mon[29291]: 6.11 scrub ok 2024-09-10T18:26:19.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:19 smithi028 ceph-mon[29291]: osdmap e139: 8 total, 7 up, 8 in 2024-09-10T18:26:19.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:19 smithi028 ceph-mon[39220]: 6.f scrub ok 2024-09-10T18:26:19.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:19 smithi028 ceph-mon[39220]: 5.1d scrub starts 2024-09-10T18:26:19.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:19 smithi028 ceph-mon[39220]: 5.1d scrub ok 2024-09-10T18:26:19.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:19 smithi028 ceph-mon[39220]: 6.0 scrub ok 2024-09-10T18:26:19.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:19 smithi028 ceph-mon[39220]: 6.11 scrub starts 2024-09-10T18:26:19.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:19 smithi028 ceph-mon[39220]: 6.11 scrub ok 2024-09-10T18:26:19.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:19 smithi028 ceph-mon[39220]: osdmap e139: 8 total, 7 up, 8 in 2024-09-10T18:26:19.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:19.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:19.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:19.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:19.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:19.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:19.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:19.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:19.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:19.716 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:19.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:19.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:19.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:19.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:19.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:19.939 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-09-10T18:26:19.939 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-09-10T18:26:19.939 INFO:tasks.cephadm.osd.6:Restarting osd.6 (starting--it wasn't running)... 2024-09-10T18:26:19.939 DEBUG:teuthology.orchestra.run.smithi204:osd.6> sudo journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.6.service 2024-09-10T18:26:19.941 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl start ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.6 2024-09-10T18:26:19.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:20.016 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:20.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:20.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:20.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:20.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:20.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:20.259 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:20.263 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:20 smithi204 ceph-mon[32357]: 6.1a deep-scrub starts 2024-09-10T18:26:20.263 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:20 smithi204 ceph-mon[32357]: 6.1a deep-scrub ok 2024-09-10T18:26:20.263 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:20 smithi204 ceph-mon[32357]: pgmap v192: 158 pgs: 10 unknown, 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 22666/217245 objects degraded (10.433%) 2024-09-10T18:26:20.263 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:20 smithi204 ceph-mon[32357]: 5.1a scrub starts 2024-09-10T18:26:20.264 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:20 smithi204 ceph-mon[32357]: 5.1a scrub ok 2024-09-10T18:26:20.264 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:20 smithi204 ceph-mon[32357]: 4.14 scrub starts 2024-09-10T18:26:20.264 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:20 smithi204 ceph-mon[32357]: osdmap e140: 8 total, 7 up, 8 in 2024-09-10T18:26:20.264 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:20 smithi204 ceph-mon[32357]: 4.14 scrub ok 2024-09-10T18:26:20.264 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:20 smithi204 ceph-mon[32357]: 4.4 scrub starts 2024-09-10T18:26:20.264 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:20 smithi204 ceph-mon[32357]: 4.4 scrub ok 2024-09-10T18:26:20.264 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 systemd[1]: Starting Ceph osd.6 for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:26:20.286 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:20.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:20.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:20.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[39220]: 6.1a deep-scrub starts 2024-09-10T18:26:20.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[39220]: 6.1a deep-scrub ok 2024-09-10T18:26:20.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[39220]: pgmap v192: 158 pgs: 10 unknown, 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 22666/217245 objects degraded (10.433%) 2024-09-10T18:26:20.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[39220]: 5.1a scrub starts 2024-09-10T18:26:20.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[39220]: 5.1a scrub ok 2024-09-10T18:26:20.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[39220]: 4.14 scrub starts 2024-09-10T18:26:20.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[39220]: osdmap e140: 8 total, 7 up, 8 in 2024-09-10T18:26:20.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[39220]: 4.14 scrub ok 2024-09-10T18:26:20.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[39220]: 4.4 scrub starts 2024-09-10T18:26:20.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[39220]: 4.4 scrub ok 2024-09-10T18:26:20.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[29291]: 6.1a deep-scrub starts 2024-09-10T18:26:20.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[29291]: 6.1a deep-scrub ok 2024-09-10T18:26:20.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[29291]: pgmap v192: 158 pgs: 10 unknown, 1 active+clean+scrubbing, 24 active+undersized, 24 active+undersized+degraded, 99 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 22666/217245 objects degraded (10.433%) 2024-09-10T18:26:20.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[29291]: 5.1a scrub starts 2024-09-10T18:26:20.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[29291]: 5.1a scrub ok 2024-09-10T18:26:20.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[29291]: 4.14 scrub starts 2024-09-10T18:26:20.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[29291]: osdmap e140: 8 total, 7 up, 8 in 2024-09-10T18:26:20.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[29291]: 4.14 scrub ok 2024-09-10T18:26:20.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[29291]: 4.4 scrub starts 2024-09-10T18:26:20.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:20 smithi028 ceph-mon[29291]: 4.4 scrub ok 2024-09-10T18:26:20.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T18:26:20.401 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T18:26:20.414 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:20.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:20.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:20.516 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:20.541 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:20.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:20.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:20.604 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 podman[78623]: 2024-09-10 18:26:20.165369393 +0000 UTC m=+0.019684975 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T18:26:20.604 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 podman[78623]: 2024-09-10 18:26:20.271006371 +0000 UTC m=+0.125321940 container create 61edc98e19e97be83f1f3f214cf8ec02cbd0a4d556d196f221c5ff3f22f8044d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T18:26:20.604 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 podman[78623]: 2024-09-10 18:26:20.506460734 +0000 UTC m=+0.360776299 container init 61edc98e19e97be83f1f3f214cf8ec02cbd0a4d556d196f221c5ff3f22f8044d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS) 2024-09-10T18:26:20.604 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 podman[78623]: 2024-09-10 18:26:20.511215762 +0000 UTC m=+0.365531327 container start 61edc98e19e97be83f1f3f214cf8ec02cbd0a4d556d196f221c5ff3f22f8044d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-activate, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-10T18:26:20.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:20.672 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:20.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:20.776 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:20.803 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:20.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:20.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:20.868 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 podman[78623]: 2024-09-10 18:26:20.606959028 +0000 UTC m=+0.461274602 container attach 61edc98e19e97be83f1f3f214cf8ec02cbd0a4d556d196f221c5ff3f22f8044d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-activate, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T18:26:20.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:20.951 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:20.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:21.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:21.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:21.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:21.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:21 smithi028 ceph-mon[39220]: 5.d scrub starts 2024-09-10T18:26:21.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:21 smithi028 ceph-mon[39220]: 5.d scrub ok 2024-09-10T18:26:21.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:21 smithi028 ceph-mon[39220]: 5.b scrub starts 2024-09-10T18:26:21.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:21 smithi028 ceph-mon[39220]: 5.b scrub ok 2024-09-10T18:26:21.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:21 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 22666/217245 objects degraded (10.433%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:21.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:21 smithi028 ceph-mon[39220]: osdmap e141: 8 total, 7 up, 8 in 2024-09-10T18:26:21.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:21 smithi028 ceph-mon[39220]: 6.18 scrub starts 2024-09-10T18:26:21.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:21 smithi028 ceph-mon[29291]: 5.d scrub starts 2024-09-10T18:26:21.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:21 smithi028 ceph-mon[29291]: 5.d scrub ok 2024-09-10T18:26:21.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:21 smithi028 ceph-mon[29291]: 5.b scrub starts 2024-09-10T18:26:21.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:21 smithi028 ceph-mon[29291]: 5.b scrub ok 2024-09-10T18:26:21.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:21 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 22666/217245 objects degraded (10.433%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:21.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:21 smithi028 ceph-mon[29291]: osdmap e141: 8 total, 7 up, 8 in 2024-09-10T18:26:21.136 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:21 smithi028 ceph-mon[29291]: 6.18 scrub starts 2024-09-10T18:26:21.202 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:21.224 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:21 smithi204 ceph-mon[32357]: 5.d scrub starts 2024-09-10T18:26:21.224 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:21 smithi204 ceph-mon[32357]: 5.d scrub ok 2024-09-10T18:26:21.224 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:21 smithi204 ceph-mon[32357]: 5.b scrub starts 2024-09-10T18:26:21.224 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:21 smithi204 ceph-mon[32357]: 5.b scrub ok 2024-09-10T18:26:21.224 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:21 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 22666/217245 objects degraded (10.433%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:21.225 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:21 smithi204 ceph-mon[32357]: osdmap e141: 8 total, 7 up, 8 in 2024-09-10T18:26:21.225 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:21 smithi204 ceph-mon[32357]: 6.18 scrub starts 2024-09-10T18:26:21.225 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-activate[78655]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-10T18:26:21.225 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 bash[78623]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-10T18:26:21.225 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-activate[78655]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-10T18:26:21.225 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 bash[78623]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-10T18:26:21.225 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-activate[78655]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-10T18:26:21.225 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 bash[78623]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-10T18:26:21.226 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-activate[78655]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-10T18:26:21.226 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 bash[78623]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-10T18:26:21.226 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-activate[78655]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-10T18:26:21.226 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 bash[78623]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-10T18:26:21.226 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-activate[78655]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-10T18:26:21.226 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 bash[78623]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-10T18:26:21.226 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-activate[78655]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-10T18:26:21.226 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 bash[78623]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-10T18:26:21.226 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:20 smithi204 podman[78623]: 2024-09-10 18:26:20.931564947 +0000 UTC m=+0.785880522 container died 61edc98e19e97be83f1f3f214cf8ec02cbd0a4d556d196f221c5ff3f22f8044d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-10T18:26:21.229 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:21.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:21.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:21.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:21.361 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:26:21.361 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:26:21.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.361 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:176986 7.e e141/93 7:73958461:::benchmark_data_smithi028_77856_object55886:head v 141'11059, mlcod=141'11056)", 2024-09-10T18:26:21.361 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:21.117555+0000", 2024-09-10T18:26:21.361 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.012928342000000001, 2024-09-10T18:26:21.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01295663, 2024-09-10T18:26:21.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:21.361 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:26:21.361 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:21.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:21.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.117555+0000", 2024-09-10T18:26:21.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:21.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:21.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.117555+0000", 2024-09-10T18:26:21.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:21.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:21.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.117553+0000", 2024-09-10T18:26:21.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:26:21.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:21.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.117562+0000", 2024-09-10T18:26:21.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7299999999999994e-06 2024-09-10T18:26:21.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:21.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.117578+0000", 2024-09-10T18:26:21.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5797000000000001e-05 2024-09-10T18:26:21.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:21.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.117585+0000", 2024-09-10T18:26:21.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2439999999999996e-06 2024-09-10T18:26:21.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:21.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.124373+0000", 2024-09-10T18:26:21.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0067880309999999999 2024-09-10T18:26:21.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:21.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.124864+0000", 2024-09-10T18:26:21.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049136199999999996 2024-09-10T18:26:21.365 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:21.365 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:21.365 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:21.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.365 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:176993 7.6 7:653088e6:::benchmark_data_smithi028_77856_object55893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-09-10T18:26:21.365 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:21.125036+0000", 2024-09-10T18:26:21.365 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0054470559999999996, 2024-09-10T18:26:21.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055452410000000002, 2024-09-10T18:26:21.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:21.365 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:26:21.365 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:21.365 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:21.365 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:21.366 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 176993 2024-09-10T18:26:21.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.366 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:21.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:21.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.125036+0000", 2024-09-10T18:26:21.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:21.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:21.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.125036+0000", 2024-09-10T18:26:21.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:21.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:21.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.125034+0000", 2024-09-10T18:26:21.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:26:21.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:21.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.125041+0000", 2024-09-10T18:26:21.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0650000000000001e-06 2024-09-10T18:26:21.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:21.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.125059+0000", 2024-09-10T18:26:21.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7736000000000001e-05 2024-09-10T18:26:21.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:21.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.125071+0000", 2024-09-10T18:26:21.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2508e-05 2024-09-10T18:26:21.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:21.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.127043+0000", 2024-09-10T18:26:21.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001971549 2024-09-10T18:26:21.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:21.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.129726+0000", 2024-09-10T18:26:21.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026835409999999998 2024-09-10T18:26:21.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:26:21.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.130010+0000", 2024-09-10T18:26:21.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000283369 2024-09-10T18:26:21.369 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:21.369 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:21.369 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:21.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.369 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:176991 7.d e141/95)", 2024-09-10T18:26:21.369 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:21.128557+0000", 2024-09-10T18:26:21.369 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.00192555, 2024-09-10T18:26:21.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002093812, 2024-09-10T18:26:21.370 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:21.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-09-10T18:26:21.370 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:21.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:21.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.128557+0000", 2024-09-10T18:26:21.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:21.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:21.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.128557+0000", 2024-09-10T18:26:21.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:21.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:21.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.128556+0000", 2024-09-10T18:26:21.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:26:21.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:21.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.128566+0000", 2024-09-10T18:26:21.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0202e-05 2024-09-10T18:26:21.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:21.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.129561+0000", 2024-09-10T18:26:21.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00099479299999999989 2024-09-10T18:26:21.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:21.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.129569+0000", 2024-09-10T18:26:21.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2099999999999993e-06 2024-09-10T18:26:21.372 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:21.372 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:21.372 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:21.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.372 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:176972 7.5 e141/93 7:ac35c650:::benchmark_data_smithi028_77856_object55872:head v 141'11038, mlcod=141'11035)", 2024-09-10T18:26:21.372 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:21.097673+0000", 2024-09-10T18:26:21.372 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.032810086000000002, 2024-09-10T18:26:21.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.033024426000000003, 2024-09-10T18:26:21.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:21.372 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:26:21.372 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:21.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:21.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.097673+0000", 2024-09-10T18:26:21.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:21.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:21.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.097673+0000", 2024-09-10T18:26:21.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:21.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:21.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.097670+0000", 2024-09-10T18:26:21.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-10T18:26:21.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:21.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.097684+0000", 2024-09-10T18:26:21.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4377e-05 2024-09-10T18:26:21.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:21.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.097704+0000", 2024-09-10T18:26:21.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9303000000000001e-05 2024-09-10T18:26:21.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:21.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.097711+0000", 2024-09-10T18:26:21.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.576e-06 2024-09-10T18:26:21.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:21.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.100891+0000", 2024-09-10T18:26:21.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031793189999999999 2024-09-10T18:26:21.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:21.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.100964+0000", 2024-09-10T18:26:21.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2965000000000004e-05 2024-09-10T18:26:21.375 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:21.375 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:21.375 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:21.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.376 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:176966 7.d e141/95 7:becf21ca:::benchmark_data_smithi028_77856_object55866:head v 141'10887, mlcod=141'10884)", 2024-09-10T18:26:21.376 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:21.098273+0000", 2024-09-10T18:26:21.376 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.032209693999999997, 2024-09-10T18:26:21.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032485342, 2024-09-10T18:26:21.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:21.376 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:26:21.376 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:21.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:21.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.098273+0000", 2024-09-10T18:26:21.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:21.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:21.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.098273+0000", 2024-09-10T18:26:21.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:21.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:21.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.098272+0000", 2024-09-10T18:26:21.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:26:21.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:21.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.098281+0000", 2024-09-10T18:26:21.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0820000000000005e-06 2024-09-10T18:26:21.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:21.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.098294+0000", 2024-09-10T18:26:21.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3281e-05 2024-09-10T18:26:21.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:21.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.098300+0000", 2024-09-10T18:26:21.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4469999999999997e-06 2024-09-10T18:26:21.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:21.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.100931+0000", 2024-09-10T18:26:21.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026313360000000002 2024-09-10T18:26:21.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:21.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.101023+0000", 2024-09-10T18:26:21.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1640999999999999e-05 2024-09-10T18:26:21.379 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:21.379 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:21.379 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:21.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.379 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:176967 7.d e141/95 7:bbeeba08:::benchmark_data_smithi028_77856_object55867:head v 141'10888, mlcod=141'10884)", 2024-09-10T18:26:21.379 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:21.102275+0000", 2024-09-10T18:26:21.379 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.028207624000000001, 2024-09-10T18:26:21.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.02854334, 2024-09-10T18:26:21.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:21.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:26:21.380 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:21.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:21.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.102275+0000", 2024-09-10T18:26:21.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:21.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:21.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.102275+0000", 2024-09-10T18:26:21.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:21.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:21.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.102274+0000", 2024-09-10T18:26:21.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:26:21.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:21.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.102283+0000", 2024-09-10T18:26:21.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9160000000000007e-06 2024-09-10T18:26:21.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:21.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.102294+0000", 2024-09-10T18:26:21.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0732e-05 2024-09-10T18:26:21.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:21.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.102299+0000", 2024-09-10T18:26:21.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4909999999999996e-06 2024-09-10T18:26:21.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:21.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.121069+0000", 2024-09-10T18:26:21.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018770190999999999 2024-09-10T18:26:21.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:21.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.121085+0000", 2024-09-10T18:26:21.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5143000000000001e-05 2024-09-10T18:26:21.383 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:21.383 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:21.383 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:21.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.383 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:176975 7.d e141/95 7:b80c5825:::benchmark_data_smithi028_77856_object55875:head v 141'10889, mlcod=141'10884)", 2024-09-10T18:26:21.383 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:21.106123+0000", 2024-09-10T18:26:21.383 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.024360386000000001, 2024-09-10T18:26:21.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024760694999999999, 2024-09-10T18:26:21.383 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:21.383 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:26:21.383 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:21.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:21.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.106123+0000", 2024-09-10T18:26:21.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:21.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:21.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.106123+0000", 2024-09-10T18:26:21.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:21.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:21.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.106121+0000", 2024-09-10T18:26:21.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:26:21.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:21.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.106131+0000", 2024-09-10T18:26:21.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0101999999999999e-05 2024-09-10T18:26:21.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:21.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.106145+0000", 2024-09-10T18:26:21.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.433e-05 2024-09-10T18:26:21.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:21.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.106152+0000", 2024-09-10T18:26:21.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6669999999999999e-06 2024-09-10T18:26:21.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:21.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.123931+0000", 2024-09-10T18:26:21.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017778664999999999 2024-09-10T18:26:21.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:21.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.123986+0000", 2024-09-10T18:26:21.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5174999999999999e-05 2024-09-10T18:26:21.386 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:21.386 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:21.386 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:21.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.387 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:176981 7.d e141/95 7:b684a6f2:::benchmark_data_smithi028_77856_object55881:head v 141'10890, mlcod=141'10884)", 2024-09-10T18:26:21.387 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:21.113159+0000", 2024-09-10T18:26:21.387 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.017324464000000001, 2024-09-10T18:26:21.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017782986000000001, 2024-09-10T18:26:21.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:21.387 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:26:21.387 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:21.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:21.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.113159+0000", 2024-09-10T18:26:21.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:21.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:21.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.113159+0000", 2024-09-10T18:26:21.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:21.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:21.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.113158+0000", 2024-09-10T18:26:21.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:26:21.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:21.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.113164+0000", 2024-09-10T18:26:21.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7229999999999998e-06 2024-09-10T18:26:21.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:21.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.113172+0000", 2024-09-10T18:26:21.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2660000000000004e-06 2024-09-10T18:26:21.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:21.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.113176+0000", 2024-09-10T18:26:21.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0670000000000002e-06 2024-09-10T18:26:21.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:21.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.129981+0000", 2024-09-10T18:26:21.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016804817 2024-09-10T18:26:21.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:21.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:21.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:21.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:21.130122+0000", 2024-09-10T18:26:21.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014176099999999999 2024-09-10T18:26:21.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:21.390 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:21.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:21.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:21.390 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:26:21.390 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 8 2024-09-10T18:26:21.390 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:26:21.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:21.391 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:21.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:21.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:21.503 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:26:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:26:21] "GET /metrics HTTP/1.1" 200 223869 "" "Prometheus/2.43.0" 2024-09-10T18:26:21.534 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:21.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:21.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:21.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:21.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:21.665 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:21.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:21.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:21.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:21.801 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:21.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:21.902 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:21.928 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:21.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:21.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:21.949 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:21 smithi204 podman[78849]: 2024-09-10 18:26:21.646515945 +0000 UTC m=+0.705460616 container remove 61edc98e19e97be83f1f3f214cf8ec02cbd0a4d556d196f221c5ff3f22f8044d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-activate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-10T18:26:21.949 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:21 smithi204 podman[78954]: 2024-09-10 18:26:21.798619222 +0000 UTC m=+0.019592600 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T18:26:21.949 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:21 smithi204 podman[78954]: 2024-09-10 18:26:21.947707168 +0000 UTC m=+0.168680542 container create b69a2462c2d6225f7d03e831b050b6dd43fd5ae6724fbb70a7d6710b12df4053 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-10T18:26:22.030 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:22.052 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:22.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:22.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:26:22.153 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:22.179 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:22.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:22.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:22.200 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:26:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:26:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:26:22.200 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:22 smithi204 podman[78954]: 2024-09-10 18:26:22.193109853 +0000 UTC m=+0.414083229 container init b69a2462c2d6225f7d03e831b050b6dd43fd5ae6724fbb70a7d6710b12df4053 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T18:26:22.200 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:22 smithi204 ceph-mon[32357]: 6.18 scrub ok 2024-09-10T18:26:22.200 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:22 smithi204 ceph-mon[32357]: pgmap v195: 158 pgs: 5 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 105 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.43k op/s; 20812/200088 objects degraded (10.401%); 144 KiB/s, 0 objects/s recovering 2024-09-10T18:26:22.200 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:22 smithi204 ceph-mon[32357]: 6.d deep-scrub starts 2024-09-10T18:26:22.201 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:22 smithi204 ceph-mon[32357]: 6.d deep-scrub ok 2024-09-10T18:26:22.201 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:22 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2146163929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:26:22.201 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:22 smithi204 ceph-mon[32357]: from='client.24886 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:26:22.201 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:22 smithi204 ceph-mon[32357]: 3.1f deep-scrub starts 2024-09-10T18:26:22.201 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:22 smithi204 ceph-mon[32357]: 3.1f deep-scrub ok 2024-09-10T18:26:22.282 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:22.306 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:22.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:22.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:22.316 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T18:26:22.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:26:22.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:22.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:22 smithi028 ceph-mon[39220]: 6.18 scrub ok 2024-09-10T18:26:22.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:22 smithi028 ceph-mon[39220]: pgmap v195: 158 pgs: 5 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 105 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.43k op/s; 20812/200088 objects degraded (10.401%); 144 KiB/s, 0 objects/s recovering 2024-09-10T18:26:22.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:22 smithi028 ceph-mon[39220]: 6.d deep-scrub starts 2024-09-10T18:26:22.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:22 smithi028 ceph-mon[39220]: 6.d deep-scrub ok 2024-09-10T18:26:22.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:22 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2146163929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:26:22.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:22 smithi028 ceph-mon[39220]: from='client.24886 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:26:22.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:22 smithi028 ceph-mon[39220]: 3.1f deep-scrub starts 2024-09-10T18:26:22.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:22 smithi028 ceph-mon[39220]: 3.1f deep-scrub ok 2024-09-10T18:26:22.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:22 smithi028 ceph-mon[29291]: 6.18 scrub ok 2024-09-10T18:26:22.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:22 smithi028 ceph-mon[29291]: pgmap v195: 158 pgs: 5 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 105 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.43k op/s; 20812/200088 objects degraded (10.401%); 144 KiB/s, 0 objects/s recovering 2024-09-10T18:26:22.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:22 smithi028 ceph-mon[29291]: 6.d deep-scrub starts 2024-09-10T18:26:22.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:22 smithi028 ceph-mon[29291]: 6.d deep-scrub ok 2024-09-10T18:26:22.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:22 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2146163929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:26:22.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:22 smithi028 ceph-mon[29291]: from='client.24886 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:26:22.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:22 smithi028 ceph-mon[29291]: 3.1f deep-scrub starts 2024-09-10T18:26:22.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:22 smithi028 ceph-mon[29291]: 3.1f deep-scrub ok 2024-09-10T18:26:22.485 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:22.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:22.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:22.587 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:22.604 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:22 smithi204 podman[78954]: 2024-09-10 18:26:22.198674633 +0000 UTC m=+0.419648005 container start b69a2462c2d6225f7d03e831b050b6dd43fd5ae6724fbb70a7d6710b12df4053 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-09-10T18:26:22.604 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:22 smithi204 bash[78954]: b69a2462c2d6225f7d03e831b050b6dd43fd5ae6724fbb70a7d6710b12df4053 2024-09-10T18:26:22.604 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:22 smithi204 systemd[1]: Started Ceph osd.6 for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:26:22.637 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:22.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:22.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:22.650 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:26:22.739 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:22.766 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:22.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:22.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:22.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:22.900 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:22.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:22.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:23.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:23.040 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:23.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:23.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:23.066 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:26:22.736+0000 7fc929098440 -1 Falling back to public interface 2024-09-10T18:26:23.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:23.171 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:23.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:23.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:23.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:23.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:23.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:23 smithi204 ceph-mon[32357]: 6.17 scrub starts 2024-09-10T18:26:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:23 smithi204 ceph-mon[32357]: 6.17 scrub ok 2024-09-10T18:26:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:23 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:26:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:23 smithi204 ceph-mon[32357]: from='client.24886 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:26:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:23 smithi204 ceph-mon[32357]: osdmap e142: 8 total, 7 up, 8 in 2024-09-10T18:26:23.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:23 smithi028 ceph-mon[29291]: 6.17 scrub starts 2024-09-10T18:26:23.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:23 smithi028 ceph-mon[29291]: 6.17 scrub ok 2024-09-10T18:26:23.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:23 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:26:23.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:23 smithi028 ceph-mon[29291]: from='client.24886 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:26:23.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:23 smithi028 ceph-mon[29291]: osdmap e142: 8 total, 7 up, 8 in 2024-09-10T18:26:23.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:23 smithi028 ceph-mon[39220]: 6.17 scrub starts 2024-09-10T18:26:23.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:23 smithi028 ceph-mon[39220]: 6.17 scrub ok 2024-09-10T18:26:23.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:23 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:26:23.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:23 smithi028 ceph-mon[39220]: from='client.24886 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:26:23.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:23 smithi028 ceph-mon[39220]: osdmap e142: 8 total, 7 up, 8 in 2024-09-10T18:26:23.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:23.459 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:23.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:23.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:23.560 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:23.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:23.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:23.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:23.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:23.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:23.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:23.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:23.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:23.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:23.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:23.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:23.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:23.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-10T18:26:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:24.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:24.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:24.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:24.148 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:24.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:24.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:24.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:24.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:24 smithi028 ceph-mon[29291]: pgmap v197: 158 pgs: 5 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 105 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 20812/200088 objects degraded (10.401%); 121 KiB/s, 0 objects/s recovering 2024-09-10T18:26:24.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:26:24.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:24 smithi028 ceph-mon[39220]: pgmap v197: 158 pgs: 5 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 105 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 20812/200088 objects degraded (10.401%); 121 KiB/s, 0 objects/s recovering 2024-09-10T18:26:24.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:26:24.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:24.309 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:24.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:24 smithi204 ceph-mon[32357]: pgmap v197: 158 pgs: 5 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 105 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 20812/200088 objects degraded (10.401%); 121 KiB/s, 0 objects/s recovering 2024-09-10T18:26:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:26:24.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:26:24 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:26:24.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:24.456 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:24.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:24.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:24.472 INFO:teuthology.orchestra.run.smithi204.stderr:no valid command found; 10 closest matches: 2024-09-10T18:26:24.472 INFO:teuthology.orchestra.run.smithi204.stderr:0 2024-09-10T18:26:24.472 INFO:teuthology.orchestra.run.smithi204.stderr:1 2024-09-10T18:26:24.472 INFO:teuthology.orchestra.run.smithi204.stderr:2 2024-09-10T18:26:24.472 INFO:teuthology.orchestra.run.smithi204.stderr:abort 2024-09-10T18:26:24.472 INFO:teuthology.orchestra.run.smithi204.stderr:assert 2024-09-10T18:26:24.472 INFO:teuthology.orchestra.run.smithi204.stderr:config diff 2024-09-10T18:26:24.472 INFO:teuthology.orchestra.run.smithi204.stderr:config diff get 2024-09-10T18:26:24.472 INFO:teuthology.orchestra.run.smithi204.stderr:config get 2024-09-10T18:26:24.473 INFO:teuthology.orchestra.run.smithi204.stderr:config help [] 2024-09-10T18:26:24.473 INFO:teuthology.orchestra.run.smithi204.stderr:config set ... 2024-09-10T18:26:24.473 INFO:teuthology.orchestra.run.smithi204.stderr:admin_socket: invalid command 2024-09-10T18:26:24.558 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:24.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:24.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:24.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:24.686 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:24.710 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:24.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:24.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:24.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:24.835 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:24.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:24.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:24.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:24.978 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:24.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:24.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:25.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:25.101 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:25.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:25.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:25.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:25.241 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:25.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:25.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:25.330 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-10T18:26:25.331 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-09-10T18:26:25.343 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:25.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:25.370 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:25.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:25.459 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:25 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:25.471 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:25.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:25.493 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:25.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:25.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:25 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:25.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:25 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:25.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:25.620 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:25.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:25.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:25.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:25.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:25.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:25.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:25.854 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:25 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:26:25.456+0000 7fc929098440 -1 osd.6 94 log_to_monitors true 2024-09-10T18:26:25.854 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:26:25 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:26:25.670+0000 7fc91de3e640 -1 osd.6 143 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T18:26:25.860 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:25.883 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:25.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:25.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:25.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:26.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:26.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:26.152 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:26.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:26.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:26.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:26.292 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:26.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:26.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:26.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:26.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:26.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:26 smithi028 ceph-mon[39220]: pgmap v198: 158 pgs: 5 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 105 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 20493/197190 objects degraded (10.393%); 96 KiB/s, 0 objects/s recovering 2024-09-10T18:26:26.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:26 smithi028 ceph-mon[39220]: from='osd.6 v1:172.21.15.204:6808/3479311452' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:26:26.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:26 smithi028 ceph-mon[39220]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:26:26.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:26 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 20493/197190 objects degraded (10.393%), 23 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:26.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:26 smithi028 ceph-mon[39220]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-10T18:26:26.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:26 smithi028 ceph-mon[39220]: osdmap e143: 8 total, 7 up, 8 in 2024-09-10T18:26:26.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:26 smithi028 ceph-mon[39220]: from='osd.6 v1:172.21.15.204:6808/3479311452' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:26:26.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:26 smithi028 ceph-mon[39220]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:26:26.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:26 smithi028 ceph-mon[29291]: pgmap v198: 158 pgs: 5 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 105 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 20493/197190 objects degraded (10.393%); 96 KiB/s, 0 objects/s recovering 2024-09-10T18:26:26.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:26 smithi028 ceph-mon[29291]: from='osd.6 v1:172.21.15.204:6808/3479311452' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:26:26.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:26 smithi028 ceph-mon[29291]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:26:26.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:26 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 20493/197190 objects degraded (10.393%), 23 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:26.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:26 smithi028 ceph-mon[29291]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-10T18:26:26.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:26 smithi028 ceph-mon[29291]: osdmap e143: 8 total, 7 up, 8 in 2024-09-10T18:26:26.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:26 smithi028 ceph-mon[29291]: from='osd.6 v1:172.21.15.204:6808/3479311452' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:26:26.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:26 smithi028 ceph-mon[29291]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:26:26.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:26.558 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:26.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:26.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:26 smithi204 ceph-mon[32357]: pgmap v198: 158 pgs: 5 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 105 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 20493/197190 objects degraded (10.393%); 96 KiB/s, 0 objects/s recovering 2024-09-10T18:26:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:26 smithi204 ceph-mon[32357]: from='osd.6 v1:172.21.15.204:6808/3479311452' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:26:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:26 smithi204 ceph-mon[32357]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:26:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:26 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 20493/197190 objects degraded (10.393%), 23 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:26 smithi204 ceph-mon[32357]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-10T18:26:26.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:26 smithi204 ceph-mon[32357]: osdmap e143: 8 total, 7 up, 8 in 2024-09-10T18:26:26.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:26 smithi204 ceph-mon[32357]: from='osd.6 v1:172.21.15.204:6808/3479311452' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:26:26.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:26 smithi204 ceph-mon[32357]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-10T18:26:26.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:26.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:26.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:26.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:26.806 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:26.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:26.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:26.908 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:26.930 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:26.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:26.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:27.004 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:26:27.032 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:27.053 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:27.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:27.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:27.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:27.177 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:27.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:27.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:27.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:27 smithi028 ceph-mon[29291]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-10T18:26:27.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:27 smithi028 ceph-mon[29291]: osd.6 v1:172.21.15.204:6808/3479311452 boot 2024-09-10T18:26:27.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:27 smithi028 ceph-mon[29291]: osdmap e144: 8 total, 8 up, 8 in 2024-09-10T18:26:27.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:27 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:26:27.238 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:27 smithi028 ceph-mon[39220]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-10T18:26:27.238 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:27 smithi028 ceph-mon[39220]: osd.6 v1:172.21.15.204:6808/3479311452 boot 2024-09-10T18:26:27.238 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:27 smithi028 ceph-mon[39220]: osdmap e144: 8 total, 8 up, 8 in 2024-09-10T18:26:27.238 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:27 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:26:27.279 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:27.301 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:27.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:27.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:27.403 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:27.425 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:27.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:27.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:27.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:27.551 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:27.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:27.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:27 smithi204 ceph-mon[32357]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-10T18:26:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:27 smithi204 ceph-mon[32357]: osd.6 v1:172.21.15.204:6808/3479311452 boot 2024-09-10T18:26:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:27 smithi204 ceph-mon[32357]: osdmap e144: 8 total, 8 up, 8 in 2024-09-10T18:26:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:27 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:26:27.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:27.676 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:27.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:27.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:27.799 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:27.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:27.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:27.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:27.924 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:27.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:27.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:28.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:28.047 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:28.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:28.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:28.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:28.172 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:28.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:28.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:28.274 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:28.297 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:28.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:28.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:28.380 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:28 smithi028 ceph-mon[39220]: pgmap v201: 158 pgs: 5 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 105 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 893 op/s; 19135/184014 objects degraded (10.399%) 2024-09-10T18:26:28.381 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:28 smithi028 ceph-mon[39220]: osdmap e145: 8 total, 8 up, 8 in 2024-09-10T18:26:28.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:28 smithi028 ceph-mon[29291]: pgmap v201: 158 pgs: 5 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 105 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 893 op/s; 19135/184014 objects degraded (10.399%) 2024-09-10T18:26:28.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:28 smithi028 ceph-mon[29291]: osdmap e145: 8 total, 8 up, 8 in 2024-09-10T18:26:28.398 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:28.421 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:28.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:28.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:28.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:28.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:28.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:28 smithi204 ceph-mon[32357]: pgmap v201: 158 pgs: 5 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 105 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 893 op/s; 19135/184014 objects degraded (10.399%) 2024-09-10T18:26:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:28 smithi204 ceph-mon[32357]: osdmap e145: 8 total, 8 up, 8 in 2024-09-10T18:26:28.618 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:26:28.618 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:26:28.622 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:26:28.623 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:26:28.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:28.649 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:26:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:26:28.666 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:26:28.666 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:26:28.682 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:28.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:28.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:26:28.684 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:26:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:26:28.702 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:26:28.702 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:26:28.720 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:26:28.720 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:26:28.753 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:26:28.754 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:26:28.784 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:28.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:28.814 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:28.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:28.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:28.967 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:28.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:29.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:26:29.069 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:29.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:29.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:29.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:29.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:29.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:29.229 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:29.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:29.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:29 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:26:29.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:29 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:26:29.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:29 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1676400685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:26:29.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:29 smithi028 ceph-mon[39220]: osdmap e146: 8 total, 8 up, 8 in 2024-09-10T18:26:29.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:29 smithi028 ceph-mon[39220]: 7.d starting backfill to osd.6 from (0'0,0'0] MAX to 143'11230 2024-09-10T18:26:29.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:29 smithi028 ceph-mon[39220]: 7.7 starting backfill to osd.6 from (0'0,0'0] MAX to 143'11567 2024-09-10T18:26:29.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:29 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:26:29.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:29 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:26:29.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:29 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1676400685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:26:29.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:29 smithi028 ceph-mon[29291]: osdmap e146: 8 total, 8 up, 8 in 2024-09-10T18:26:29.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:29 smithi028 ceph-mon[29291]: 7.d starting backfill to osd.6 from (0'0,0'0] MAX to 143'11230 2024-09-10T18:26:29.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:29 smithi028 ceph-mon[29291]: 7.7 starting backfill to osd.6 from (0'0,0'0] MAX to 143'11567 2024-09-10T18:26:29.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:29.386 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:29.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:29.488 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:29.516 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:29.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:29 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:26:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:29 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:26:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:29 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1676400685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:26:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:29 smithi204 ceph-mon[32357]: osdmap e146: 8 total, 8 up, 8 in 2024-09-10T18:26:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:29 smithi204 ceph-mon[32357]: 7.d starting backfill to osd.6 from (0'0,0'0] MAX to 143'11230 2024-09-10T18:26:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:29 smithi204 ceph-mon[32357]: 7.7 starting backfill to osd.6 from (0'0,0'0] MAX to 143'11567 2024-09-10T18:26:29.618 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:29.646 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:29.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:29.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:29.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:29.775 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:29.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:29.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:29.876 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:29.906 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:29.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:29.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:30.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:30.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:30.036 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:30.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:30.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:30.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:30.175 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:30.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:30.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:30.305 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:30.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:30.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:30.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:30 smithi028 ceph-mon[29291]: 7.1 starting backfill to osd.6 from (0'0,0'0] MAX to 143'11284 2024-09-10T18:26:30.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:30 smithi028 ceph-mon[29291]: 7.b starting backfill to osd.6 from (0'0,0'0] MAX to 143'11302 2024-09-10T18:26:30.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:30 smithi028 ceph-mon[29291]: 7.c starting backfill to osd.6 from (0'0,0'0] MAX to 143'11374 2024-09-10T18:26:30.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:30 smithi028 ceph-mon[29291]: pgmap v204: 158 pgs: 5 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 105 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 19135/184014 objects degraded (10.399%) 2024-09-10T18:26:30.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:30 smithi028 ceph-mon[39220]: 7.1 starting backfill to osd.6 from (0'0,0'0] MAX to 143'11284 2024-09-10T18:26:30.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:30 smithi028 ceph-mon[39220]: 7.b starting backfill to osd.6 from (0'0,0'0] MAX to 143'11302 2024-09-10T18:26:30.332 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:30 smithi028 ceph-mon[39220]: 7.c starting backfill to osd.6 from (0'0,0'0] MAX to 143'11374 2024-09-10T18:26:30.332 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:30 smithi028 ceph-mon[39220]: pgmap v204: 158 pgs: 5 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 105 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 19135/184014 objects degraded (10.399%) 2024-09-10T18:26:30.332 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T18:26:30.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:30.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:30.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:30.548 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:30.578 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:30.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:30.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:30 smithi204 ceph-mon[32357]: 7.1 starting backfill to osd.6 from (0'0,0'0] MAX to 143'11284 2024-09-10T18:26:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:30 smithi204 ceph-mon[32357]: 7.b starting backfill to osd.6 from (0'0,0'0] MAX to 143'11302 2024-09-10T18:26:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:30 smithi204 ceph-mon[32357]: 7.c starting backfill to osd.6 from (0'0,0'0] MAX to 143'11374 2024-09-10T18:26:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:30 smithi204 ceph-mon[32357]: pgmap v204: 158 pgs: 5 active+clean+remapped, 25 active+undersized, 23 active+undersized+degraded, 105 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 19135/184014 objects degraded (10.399%) 2024-09-10T18:26:30.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:30.716 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:26:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:30.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:30.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:30.821 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:30.849 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:30.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:30.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:30.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:30.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:30.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:31.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:31.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:31.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:31.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:31.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:31.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:31.262 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:31.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:31.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:31.404 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:31.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:31.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:26:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:26:31] "GET /metrics HTTP/1.1" 200 223911 "" "Prometheus/2.43.0" 2024-09-10T18:26:31.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:31 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 19135/184014 objects degraded (10.399%), 23 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:31.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:31 smithi028 ceph-mon[39220]: osdmap e147: 8 total, 8 up, 8 in 2024-09-10T18:26:31.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:31 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 19135/184014 objects degraded (10.399%), 23 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:31.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:31 smithi028 ceph-mon[29291]: osdmap e147: 8 total, 8 up, 8 in 2024-09-10T18:26:31.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:31.539 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:31.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:31.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:31.553 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:31 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 19135/184014 objects degraded (10.399%), 23 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2024-09-10T18:26:31.553 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:31 smithi204 ceph-mon[32357]: osdmap e147: 8 total, 8 up, 8 in 2024-09-10T18:26:31.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:31.685 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:31.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:31.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:31.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:26:31.809 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:31.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:31.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:31.911 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:31.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:31.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:31.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:32.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:32.063 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:32.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:32.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:32.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:32.213 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:32.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:32.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:32.287 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:32 smithi204 ceph-mon[32357]: pgmap v206: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 706 op/s; 17532/175095 objects degraded (10.013%); 9.0 MiB/s, 144 objects/s recovering 2024-09-10T18:26:32.288 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:32 smithi204 ceph-mon[32357]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T18:26:32.288 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:26:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:26:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:26:32.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:32.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:32.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:32.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:32.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:32 smithi028 ceph-mon[39220]: pgmap v206: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 706 op/s; 17532/175095 objects degraded (10.013%); 9.0 MiB/s, 144 objects/s recovering 2024-09-10T18:26:32.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:32 smithi028 ceph-mon[39220]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T18:26:32.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:32 smithi028 ceph-mon[29291]: pgmap v206: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 706 op/s; 17532/175095 objects degraded (10.013%); 9.0 MiB/s, 144 objects/s recovering 2024-09-10T18:26:32.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:32 smithi028 ceph-mon[29291]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T18:26:32.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:32.511 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:32.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:32.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:32.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:32.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:32.763 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:26:32.763 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:26:32.787 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:26:32.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:32.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:32.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:32.930 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:32.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:32.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:33.032 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:33.060 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:33.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:33.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:33.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:33.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:33.354 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:33.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:33.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:33.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:33.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:33.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-10T18:26:33.600 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:33.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:33.629 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:33.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:33.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:33.759 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:33.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:33.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:33.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:33.917 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:33.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:33.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:34.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:34.048 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:34.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:34.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:34.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:34.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:26:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:26:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:34.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:34.218 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:26:34.300 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:34.329 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:34.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:34.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:34.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:34.468 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:34.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:34.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:34.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:34 smithi028 ceph-mon[39220]: pgmap v207: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 496 op/s; 17532/175095 objects degraded (10.013%); 6.3 MiB/s, 101 objects/s recovering 2024-09-10T18:26:34.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:34 smithi028 ceph-mon[29291]: pgmap v207: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 496 op/s; 17532/175095 objects degraded (10.013%); 6.3 MiB/s, 101 objects/s recovering 2024-09-10T18:26:34.570 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:34.598 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:34.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:34.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:34.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:34 smithi204 ceph-mon[32357]: pgmap v207: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 496 op/s; 17532/175095 objects degraded (10.013%); 6.3 MiB/s, 101 objects/s recovering 2024-09-10T18:26:34.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:34.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:34.745 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:34.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:34.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:34.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:34.890 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:34.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:34.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:35.038 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:35.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:35.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:35.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:35.142 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:26:35.143 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:26:35.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:35.188 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:35.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:35.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:35 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:26:35.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:35 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:26:35.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:35 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:35.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:35 smithi028 ceph-mon[29291]: osdmap e148: 8 total, 8 up, 8 in 2024-09-10T18:26:35.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:35 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:26:35.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:35 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:26:35.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:35 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:35.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:35 smithi028 ceph-mon[39220]: osdmap e148: 8 total, 8 up, 8 in 2024-09-10T18:26:35.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:35.330 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:35.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:35.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:35 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:26:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:35 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:26:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:35 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:35 smithi204 ceph-mon[32357]: osdmap e148: 8 total, 8 up, 8 in 2024-09-10T18:26:35.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:35.471 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:35.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:35.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:35.573 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:35.602 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:35.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:35.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:35.704 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:35.732 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:35.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:35.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:35.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:35.870 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:35.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:35.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:35.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-10T18:26:35.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:36.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:36.032 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:36.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:36.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:36.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:36.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:36.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:36 smithi028 ceph-mon[39220]: pgmap v209: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 639 op/s; 15370/172122 objects degraded (8.930%); 26 MiB/s, 413 objects/s recovering 2024-09-10T18:26:36.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:36 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 15370/172122 objects degraded (8.930%), 5 pgs degraded (PG_DEGRADED) 2024-09-10T18:26:36.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:36 smithi028 ceph-mon[39220]: osdmap e149: 8 total, 8 up, 8 in 2024-09-10T18:26:36.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:36 smithi028 ceph-mon[29291]: pgmap v209: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 639 op/s; 15370/172122 objects degraded (8.930%); 26 MiB/s, 413 objects/s recovering 2024-09-10T18:26:36.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:36 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 15370/172122 objects degraded (8.930%), 5 pgs degraded (PG_DEGRADED) 2024-09-10T18:26:36.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:36 smithi028 ceph-mon[29291]: osdmap e149: 8 total, 8 up, 8 in 2024-09-10T18:26:36.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:36.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:36.339 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:36.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:36.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:36.466 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:36.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:36.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:36.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:36 smithi204 ceph-mon[32357]: pgmap v209: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 639 op/s; 15370/172122 objects degraded (8.930%); 26 MiB/s, 413 objects/s recovering 2024-09-10T18:26:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:36 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 15370/172122 objects degraded (8.930%), 5 pgs degraded (PG_DEGRADED) 2024-09-10T18:26:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:36 smithi204 ceph-mon[32357]: osdmap e149: 8 total, 8 up, 8 in 2024-09-10T18:26:36.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:36.618 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:36.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:26:36.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:36.774 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:36.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:36.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:36.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:36.930 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:36.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:36.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:37.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:37.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:37.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:37.175 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:37.184 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:26:37.203 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:37.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:37.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:37.304 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:37.331 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:37.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:37.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:37.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:37.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:37.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:37.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:37 smithi028 ceph-mon[29291]: pgmap v211: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 987 B/s rd, 0 B/s wr, 827 op/s; 13435/159663 objects degraded (8.415%); 27 MiB/s, 431 objects/s recovering 2024-09-10T18:26:37.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:37 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2964755794' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:26:37.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:37 smithi028 ceph-mon[39220]: pgmap v211: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 987 B/s rd, 0 B/s wr, 827 op/s; 13435/159663 objects degraded (8.415%); 27 MiB/s, 431 objects/s recovering 2024-09-10T18:26:37.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:37 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2964755794' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:26:37.571 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:37.594 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:37.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:37.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:37 smithi204 ceph-mon[32357]: pgmap v211: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 987 B/s rd, 0 B/s wr, 827 op/s; 13435/159663 objects degraded (8.415%); 27 MiB/s, 431 objects/s recovering 2024-09-10T18:26:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:37 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2964755794' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:26:37.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:37.744 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:37.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:37.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:37.784 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:26:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:26:37.818 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:26:37.818 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:26:37.847 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:37.852 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:26:37.852 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:26:37.870 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:37.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:37.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:37.891 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:26:37.891 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:26:37.930 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:26:37.930 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:26:37.962 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:26:37.962 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:26:37.972 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:37.996 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:37.997 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:26:37.997 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:26:37.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:37.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:38.031 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:26:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:26:38.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:38.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:38.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:38.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:38.249 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:38.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:38.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:38.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:38.376 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:38.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:38.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:38.434 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:38 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:26:38.434 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:38 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:26:38.434 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:38 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:26:38.434 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:38 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2964755794' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:26:38.434 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:38 smithi028 ceph-mon[39220]: osdmap e150: 8 total, 8 up, 8 in 2024-09-10T18:26:38.434 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:38 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1911770260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:26:38.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:38 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:26:38.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:38 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:26:38.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:38 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:26:38.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:38 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2964755794' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:26:38.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:38 smithi028 ceph-mon[29291]: osdmap e150: 8 total, 8 up, 8 in 2024-09-10T18:26:38.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:38 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1911770260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:26:38.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:38.508 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:38.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:38.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:38 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:26:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:38 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:26:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:38 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:26:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:38 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2964755794' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:26:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:38 smithi204 ceph-mon[32357]: osdmap e150: 8 total, 8 up, 8 in 2024-09-10T18:26:38.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:38 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1911770260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:26:38.610 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:38.631 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:38.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:38.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:38.733 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:38.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:38.755 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:38.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:38.856 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:38.878 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:38.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:38.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:38.979 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:39.001 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:39.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:39.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:39.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:39.128 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:39.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:39.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:39.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:39.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:39.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:39.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:39.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:39.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:39 smithi028 ceph-mon[29291]: osdmap e151: 8 total, 8 up, 8 in 2024-09-10T18:26:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:39 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:26:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:26:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:39 smithi028 ceph-mon[29291]: pgmap v214: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.10k op/s; 13435/159663 objects degraded (8.415%); 36 MiB/s, 575 objects/s recovering 2024-09-10T18:26:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:39 smithi028 ceph-mon[29291]: osdmap e152: 8 total, 8 up, 8 in 2024-09-10T18:26:39.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:39 smithi028 ceph-mon[39220]: osdmap e151: 8 total, 8 up, 8 in 2024-09-10T18:26:39.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:39 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:26:39.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:26:39.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:39 smithi028 ceph-mon[39220]: pgmap v214: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.10k op/s; 13435/159663 objects degraded (8.415%); 36 MiB/s, 575 objects/s recovering 2024-09-10T18:26:39.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:39 smithi028 ceph-mon[39220]: osdmap e152: 8 total, 8 up, 8 in 2024-09-10T18:26:39.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:39.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:39.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:39 smithi204 ceph-mon[32357]: osdmap e151: 8 total, 8 up, 8 in 2024-09-10T18:26:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:39 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:26:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:26:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:39 smithi204 ceph-mon[32357]: pgmap v214: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.10k op/s; 13435/159663 objects degraded (8.415%); 36 MiB/s, 575 objects/s recovering 2024-09-10T18:26:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:39 smithi204 ceph-mon[32357]: osdmap e152: 8 total, 8 up, 8 in 2024-09-10T18:26:39.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:39.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:39.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:26:39.797 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:39.823 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:39.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:39.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:39.924 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:39.949 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:39.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:39.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:40.051 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:40.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:40.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:40.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:40.243 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:40.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:40.345 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:40.372 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:40.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:40.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:40.474 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:40.498 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:40.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:40.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:40.600 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:40.622 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:40.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:40.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:40.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:40.765 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:26:40.765 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-10T18:26:40.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-10T18:26:40.765 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:26:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:28633 7.f 7:ffdaa3af:::benchmark_data_smithi028_77856_object28632:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:26:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:55.726456+0000", 2024-09-10T18:26:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 165.03211931600001, 2024-09-10T18:26:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14808337999999999, 2024-09-10T18:26:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 28633 2024-09-10T18:26:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726456+0000", 2024-09-10T18:26:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726456+0000", 2024-09-10T18:26:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726455+0000", 2024-09-10T18:26:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:26:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726476+0000", 2024-09-10T18:26:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1577e-05 2024-09-10T18:26:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726485+0000", 2024-09-10T18:26:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8540000000000003e-06 2024-09-10T18:26:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.860732+0000", 2024-09-10T18:26:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.134246369 2024-09-10T18:26:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.862493+0000", 2024-09-10T18:26:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017610779999999999 2024-09-10T18:26:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.862607+0000", 2024-09-10T18:26:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011408000000000001 2024-09-10T18:26:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:26:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.862777+0000", 2024-09-10T18:26:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000170161 2024-09-10T18:26:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:26:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.864620+0000", 2024-09-10T18:26:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018431400000000001 2024-09-10T18:26:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.870301+0000", 2024-09-10T18:26:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0056804409999999996 2024-09-10T18:26:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.874449+0000", 2024-09-10T18:26:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041486459999999998 2024-09-10T18:26:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.874499+0000", 2024-09-10T18:26:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9891000000000001e-05 2024-09-10T18:26:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.874539+0000", 2024-09-10T18:26:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0159e-05 2024-09-10T18:26:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31736 7.6 7:6f8bbc73:::benchmark_data_smithi028_77856_object31735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:26:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.829509+0000", 2024-09-10T18:26:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 162.92906592099999, 2024-09-10T18:26:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.148056353, 2024-09-10T18:26:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31736 2024-09-10T18:26:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829509+0000", 2024-09-10T18:26:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829509+0000", 2024-09-10T18:26:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829508+0000", 2024-09-10T18:26:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:26:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829534+0000", 2024-09-10T18:26:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6038999999999999e-05 2024-09-10T18:26:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829544+0000", 2024-09-10T18:26:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6029999999999994e-06 2024-09-10T18:26:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829549+0000", 2024-09-10T18:26:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2970000000000003e-06 2024-09-10T18:26:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829601+0000", 2024-09-10T18:26:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1629999999999999e-05 2024-09-10T18:26:40.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.776 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829725+0000", 2024-09-10T18:26:40.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012429300000000001 2024-09-10T18:26:40.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:26:40.776 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829865+0000", 2024-09-10T18:26:40.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000139493 2024-09-10T18:26:40.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.777 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.834740+0000", 2024-09-10T18:26:40.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048757419999999997 2024-09-10T18:26:40.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.777 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:26:40.777 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836134+0000", 2024-09-10T18:26:40.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013935079999999999 2024-09-10T18:26:40.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.777 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.777 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838041+0000", 2024-09-10T18:26:40.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001907583 2024-09-10T18:26:40.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838084+0000", 2024-09-10T18:26:40.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3000000000000002e-05 2024-09-10T18:26:40.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977566+0000", 2024-09-10T18:26:40.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13948112900000001 2024-09-10T18:26:40.778 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.778 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.778 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.779 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31737 7.6 7:67a3034c:::benchmark_data_smithi028_77856_object31736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:26:40.779 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.829562+0000", 2024-09-10T18:26:40.779 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 162.92901274499999, 2024-09-10T18:26:40.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14825862000000001, 2024-09-10T18:26:40.779 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.779 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.779 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:40.779 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:40.779 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:40.779 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31737 2024-09-10T18:26:40.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.779 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829562+0000", 2024-09-10T18:26:40.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829562+0000", 2024-09-10T18:26:40.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829562+0000", 2024-09-10T18:26:40.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:26:40.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.781 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829583+0000", 2024-09-10T18:26:40.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1251999999999999e-05 2024-09-10T18:26:40.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.781 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829591+0000", 2024-09-10T18:26:40.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5890000000000003e-06 2024-09-10T18:26:40.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829597+0000", 2024-09-10T18:26:40.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-06 2024-09-10T18:26:40.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.831307+0000", 2024-09-10T18:26:40.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017102700000000001 2024-09-10T18:26:40.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.831417+0000", 2024-09-10T18:26:40.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110029 2024-09-10T18:26:40.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:26:40.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.831543+0000", 2024-09-10T18:26:40.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012614200000000001 2024-09-10T18:26:40.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.834769+0000", 2024-09-10T18:26:40.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032261270000000001 2024-09-10T18:26:40.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:26:40.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836140+0000", 2024-09-10T18:26:40.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013711909999999999 2024-09-10T18:26:40.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977739+0000", 2024-09-10T18:26:40.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.141598956 2024-09-10T18:26:40.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977783+0000", 2024-09-10T18:26:40.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3890999999999997e-05 2024-09-10T18:26:40.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977821+0000", 2024-09-10T18:26:40.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7901000000000002e-05 2024-09-10T18:26:40.785 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.786 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.786 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.786 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31740 7.f 7:f269a3c7:::benchmark_data_smithi028_77856_object31739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:26:40.786 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.829970+0000", 2024-09-10T18:26:40.786 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 162.928605584, 2024-09-10T18:26:40.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.147845861, 2024-09-10T18:26:40.786 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.786 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.786 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:40.786 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:40.786 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:40.786 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31740 2024-09-10T18:26:40.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.787 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.787 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.787 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829970+0000", 2024-09-10T18:26:40.787 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.787 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.787 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829970+0000", 2024-09-10T18:26:40.787 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.787 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829968+0000", 2024-09-10T18:26:40.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:26:40.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829993+0000", 2024-09-10T18:26:40.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4621e-05 2024-09-10T18:26:40.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.830002+0000", 2024-09-10T18:26:40.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0860000000000008e-06 2024-09-10T18:26:40.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.830007+0000", 2024-09-10T18:26:40.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3979999999999998e-06 2024-09-10T18:26:40.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.832120+0000", 2024-09-10T18:26:40.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021130070000000001 2024-09-10T18:26:40.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.832231+0000", 2024-09-10T18:26:40.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110889 2024-09-10T18:26:40.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:26:40.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.832455+0000", 2024-09-10T18:26:40.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000224284 2024-09-10T18:26:40.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.834618+0000", 2024-09-10T18:26:40.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002162509 2024-09-10T18:26:40.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:26:40.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.835983+0000", 2024-09-10T18:26:40.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013655060000000001 2024-09-10T18:26:40.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977732+0000", 2024-09-10T18:26:40.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14174862399999999 2024-09-10T18:26:40.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977776+0000", 2024-09-10T18:26:40.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3828000000000001e-05 2024-09-10T18:26:40.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977815+0000", 2024-09-10T18:26:40.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.947e-05 2024-09-10T18:26:40.792 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.792 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.792 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.792 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31753 7.7 7:e80563d5:::benchmark_data_smithi028_77856_object31752:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:26:40.792 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.836698+0000", 2024-09-10T18:26:40.792 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 162.921876854, 2024-09-10T18:26:40.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14739100099999999, 2024-09-10T18:26:40.792 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.792 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.792 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:40.792 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:40.792 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:40.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31753 2024-09-10T18:26:40.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.793 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836698+0000", 2024-09-10T18:26:40.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836698+0000", 2024-09-10T18:26:40.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836697+0000", 2024-09-10T18:26:40.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:26:40.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:40.795 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:40.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:40.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836726+0000", 2024-09-10T18:26:40.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9164999999999999e-05 2024-09-10T18:26:40.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836736+0000", 2024-09-10T18:26:40.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0232e-05 2024-09-10T18:26:40.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836742+0000", 2024-09-10T18:26:40.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3569999999999997e-06 2024-09-10T18:26:40.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836885+0000", 2024-09-10T18:26:40.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014372000000000001 2024-09-10T18:26:40.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836989+0000", 2024-09-10T18:26:40.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010346800000000001 2024-09-10T18:26:40.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-10T18:26:40.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837108+0000", 2024-09-10T18:26:40.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011928400000000001 2024-09-10T18:26:40.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.978097+0000", 2024-09-10T18:26:40.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.140988426 2024-09-10T18:26:40.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:26:40.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.982298+0000", 2024-09-10T18:26:40.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042009559999999996 2024-09-10T18:26:40.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.983963+0000", 2024-09-10T18:26:40.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016655089999999999 2024-09-10T18:26:40.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.984089+0000", 2024-09-10T18:26:40.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012626599999999999 2024-09-10T18:26:40.800 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.800 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.800 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.800 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31754 7.7 7:e51855d4:::benchmark_data_smithi028_77856_object31753:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:26:40.800 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.837301+0000", 2024-09-10T18:26:40.800 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 162.92127438700001, 2024-09-10T18:26:40.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.151838735, 2024-09-10T18:26:40.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.800 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.800 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:40.801 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:40.801 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:40.801 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31754 2024-09-10T18:26:40.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.801 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837301+0000", 2024-09-10T18:26:40.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837301+0000", 2024-09-10T18:26:40.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837300+0000", 2024-09-10T18:26:40.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:26:40.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837322+0000", 2024-09-10T18:26:40.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1506999999999999e-05 2024-09-10T18:26:40.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837329+0000", 2024-09-10T18:26:40.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.515e-06 2024-09-10T18:26:40.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837334+0000", 2024-09-10T18:26:40.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8749999999999999e-06 2024-09-10T18:26:40.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.978333+0000", 2024-09-10T18:26:40.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.140998866 2024-09-10T18:26:40.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.978442+0000", 2024-09-10T18:26:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108992 2024-09-10T18:26:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-10T18:26:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.978578+0000", 2024-09-10T18:26:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013659699999999999 2024-09-10T18:26:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:26:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.984104+0000", 2024-09-10T18:26:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055253580000000002 2024-09-10T18:26:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.989014+0000", 2024-09-10T18:26:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049104839999999997 2024-09-10T18:26:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.989062+0000", 2024-09-10T18:26:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7441000000000002e-05 2024-09-10T18:26:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.989139+0000", 2024-09-10T18:26:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7847000000000002e-05 2024-09-10T18:26:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31755 7.4 7:261feab9:::benchmark_data_smithi028_77856_object31754:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:26:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.837773+0000", 2024-09-10T18:26:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 162.92080201900001, 2024-09-10T18:26:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.146545749, 2024-09-10T18:26:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:40.807 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31755 2024-09-10T18:26:40.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.807 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837773+0000", 2024-09-10T18:26:40.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837773+0000", 2024-09-10T18:26:40.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837772+0000", 2024-09-10T18:26:40.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:26:40.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837793+0000", 2024-09-10T18:26:40.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1016999999999999e-05 2024-09-10T18:26:40.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837803+0000", 2024-09-10T18:26:40.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9839999999999996e-06 2024-09-10T18:26:40.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837808+0000", 2024-09-10T18:26:40.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2460000000000003e-06 2024-09-10T18:26:40.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837942+0000", 2024-09-10T18:26:40.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013420000000000001 2024-09-10T18:26:40.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838059+0000", 2024-09-10T18:26:40.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116656 2024-09-10T18:26:40.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:26:40.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977722+0000", 2024-09-10T18:26:40.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13966282899999999 2024-09-10T18:26:40.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:26:40.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.980883+0000", 2024-09-10T18:26:40.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031608460000000001 2024-09-10T18:26:40.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.984034+0000", 2024-09-10T18:26:40.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031513270000000002 2024-09-10T18:26:40.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.984192+0000", 2024-09-10T18:26:40.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015838500000000001 2024-09-10T18:26:40.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.984285+0000", 2024-09-10T18:26:40.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3158999999999994e-05 2024-09-10T18:26:40.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.984319+0000", 2024-09-10T18:26:40.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3457999999999997e-05 2024-09-10T18:26:40.812 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.812 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.812 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.812 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31758 7.6 7:6c6ad8ac:::benchmark_data_smithi028_77856_object31757:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:26:40.812 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.837963+0000", 2024-09-10T18:26:40.812 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 162.920612619, 2024-09-10T18:26:40.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15004868099999999, 2024-09-10T18:26:40.812 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.812 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.813 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:40.813 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:40.813 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:40.813 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31758 2024-09-10T18:26:40.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.813 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837963+0000", 2024-09-10T18:26:40.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837963+0000", 2024-09-10T18:26:40.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837961+0000", 2024-09-10T18:26:40.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:26:40.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837982+0000", 2024-09-10T18:26:40.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1183e-05 2024-09-10T18:26:40.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837991+0000", 2024-09-10T18:26:40.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9570000000000008e-06 2024-09-10T18:26:40.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837996+0000", 2024-09-10T18:26:40.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6330000000000004e-06 2024-09-10T18:26:40.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977854+0000", 2024-09-10T18:26:40.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13985836200000001 2024-09-10T18:26:40.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.978294+0000", 2024-09-10T18:26:40.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043966200000000001 2024-09-10T18:26:40.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:26:40.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.978419+0000", 2024-09-10T18:26:40.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012486499999999999 2024-09-10T18:26:40.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.983938+0000", 2024-09-10T18:26:40.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055189319999999998 2024-09-10T18:26:40.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:26:40.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.983945+0000", 2024-09-10T18:26:40.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7680000000000003e-06 2024-09-10T18:26:40.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.987897+0000", 2024-09-10T18:26:40.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039524770000000002 2024-09-10T18:26:40.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.987973+0000", 2024-09-10T18:26:40.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6062000000000005e-05 2024-09-10T18:26:40.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.988011+0000", 2024-09-10T18:26:40.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8003999999999997e-05 2024-09-10T18:26:40.818 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.818 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.818 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.819 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:139702 7.f 7:f2f55ab9:::benchmark_data_smithi028_77856_object18602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:26:40.819 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:43.935577+0000", 2024-09-10T18:26:40.819 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 56.822998556000002, 2024-09-10T18:26:40.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15684685600000001, 2024-09-10T18:26:40.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.819 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.819 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:40.819 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:40.819 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:40.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 139702 2024-09-10T18:26:40.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.819 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935577+0000", 2024-09-10T18:26:40.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935577+0000", 2024-09-10T18:26:40.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935576+0000", 2024-09-10T18:26:40.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:26:40.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935580+0000", 2024-09-10T18:26:40.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6879999999999998e-06 2024-09-10T18:26:40.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935588+0000", 2024-09-10T18:26:40.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9999999999999996e-06 2024-09-10T18:26:40.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935594+0000", 2024-09-10T18:26:40.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2820000000000004e-06 2024-09-10T18:26:40.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.982138+0000", 2024-09-10T18:26:40.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.046544461000000002 2024-09-10T18:26:40.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.982248+0000", 2024-09-10T18:26:40.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110157 2024-09-10T18:26:40.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:26:40.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.982382+0000", 2024-09-10T18:26:40.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000133895 2024-09-10T18:26:40.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:26:40.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.989604+0000", 2024-09-10T18:26:40.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007222366 2024-09-10T18:26:40.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.992745+0000", 2024-09-10T18:26:40.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031404810000000001 2024-09-10T18:26:40.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:44.092292+0000", 2024-09-10T18:26:40.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.099547449999999996 2024-09-10T18:26:40.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:44.092339+0000", 2024-09-10T18:26:40.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6990000000000002e-05 2024-09-10T18:26:40.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:44.092423+0000", 2024-09-10T18:26:40.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4138e-05 2024-09-10T18:26:40.825 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.825 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.825 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.825 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:140670 7.f 7:ff7f13ba:::benchmark_data_smithi028_77856_object19570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:26:40.825 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:45.605622+0000", 2024-09-10T18:26:40.825 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.152952878999997, 2024-09-10T18:26:40.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.17139099099999999, 2024-09-10T18:26:40.826 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.826 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.826 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:40.826 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:40.826 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:40.826 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 140670 2024-09-10T18:26:40.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.826 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605622+0000", 2024-09-10T18:26:40.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605622+0000", 2024-09-10T18:26:40.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605620+0000", 2024-09-10T18:26:40.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:26:40.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605628+0000", 2024-09-10T18:26:40.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7510000000000008e-06 2024-09-10T18:26:40.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605641+0000", 2024-09-10T18:26:40.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2692e-05 2024-09-10T18:26:40.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605647+0000", 2024-09-10T18:26:40.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4169999999999997e-06 2024-09-10T18:26:40.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.612161+0000", 2024-09-10T18:26:40.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0065135159999999996 2024-09-10T18:26:40.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.612264+0000", 2024-09-10T18:26:40.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103127 2024-09-10T18:26:40.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:26:40.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.612383+0000", 2024-09-10T18:26:40.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000119157 2024-09-10T18:26:40.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:26:40.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.618786+0000", 2024-09-10T18:26:40.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006402942 2024-09-10T18:26:40.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.620538+0000", 2024-09-10T18:26:40.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001752368 2024-09-10T18:26:40.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.776944+0000", 2024-09-10T18:26:40.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.156405708 2024-09-10T18:26:40.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.776979+0000", 2024-09-10T18:26:40.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4829999999999997e-05 2024-09-10T18:26:40.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.777013+0000", 2024-09-10T18:26:40.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4496999999999999e-05 2024-09-10T18:26:40.834 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.834 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.834 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.835 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:140692 7.f 7:f303c2e4:::benchmark_data_smithi028_77856_object19592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:26:40.835 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:45.628709+0000", 2024-09-10T18:26:40.835 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.129865993000003, 2024-09-10T18:26:40.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14840287099999999, 2024-09-10T18:26:40.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.835 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.835 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:40.835 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:40.835 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:40.835 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 140692 2024-09-10T18:26:40.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.835 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628709+0000", 2024-09-10T18:26:40.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628709+0000", 2024-09-10T18:26:40.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628707+0000", 2024-09-10T18:26:40.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:26:40.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628714+0000", 2024-09-10T18:26:40.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4229999999999999e-06 2024-09-10T18:26:40.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628730+0000", 2024-09-10T18:26:40.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5991999999999999e-05 2024-09-10T18:26:40.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628738+0000", 2024-09-10T18:26:40.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0469999999999996e-06 2024-09-10T18:26:40.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628829+0000", 2024-09-10T18:26:40.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1197000000000001e-05 2024-09-10T18:26:40.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628955+0000", 2024-09-10T18:26:40.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012585100000000001 2024-09-10T18:26:40.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:26:40.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.629089+0000", 2024-09-10T18:26:40.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013479200000000001 2024-09-10T18:26:40.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.634532+0000", 2024-09-10T18:26:40.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054427750000000004 2024-09-10T18:26:40.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:26:40.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.642486+0000", 2024-09-10T18:26:40.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007953669 2024-09-10T18:26:40.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.777048+0000", 2024-09-10T18:26:40.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13456268499999999 2024-09-10T18:26:40.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.777082+0000", 2024-09-10T18:26:40.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3025000000000003e-05 2024-09-10T18:26:40.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.777112+0000", 2024-09-10T18:26:40.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0516999999999999e-05 2024-09-10T18:26:40.840 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.841 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.841 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.842 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:141317 7.f 7:f85eab8a:::benchmark_data_smithi028_77856_object20217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:26:40.842 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:46.778960+0000", 2024-09-10T18:26:40.842 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 53.979615074999998, 2024-09-10T18:26:40.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15742622000000001, 2024-09-10T18:26:40.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.842 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.842 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:40.842 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:40.842 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:40.842 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 141317 2024-09-10T18:26:40.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.842 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778960+0000", 2024-09-10T18:26:40.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778960+0000", 2024-09-10T18:26:40.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778958+0000", 2024-09-10T18:26:40.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:26:40.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778965+0000", 2024-09-10T18:26:40.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7599999999999997e-06 2024-09-10T18:26:40.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778973+0000", 2024-09-10T18:26:40.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7770000000000002e-06 2024-09-10T18:26:40.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778977+0000", 2024-09-10T18:26:40.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5229999999999997e-06 2024-09-10T18:26:40.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.789488+0000", 2024-09-10T18:26:40.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010510472999999999 2024-09-10T18:26:40.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.789609+0000", 2024-09-10T18:26:40.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121301 2024-09-10T18:26:40.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:26:40.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.789743+0000", 2024-09-10T18:26:40.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013393199999999999 2024-09-10T18:26:40.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.793571+0000", 2024-09-10T18:26:40.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0038282289999999998 2024-09-10T18:26:40.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:26:40.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.810048+0000", 2024-09-10T18:26:40.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0164768 2024-09-10T18:26:40.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.936295+0000", 2024-09-10T18:26:40.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.126246568 2024-09-10T18:26:40.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.936346+0000", 2024-09-10T18:26:40.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1708e-05 2024-09-10T18:26:40.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.936386+0000", 2024-09-10T18:26:40.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0021000000000002e-05 2024-09-10T18:26:40.847 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.847 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.848 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.848 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181160 7.7 7:e2311557:::benchmark_data_smithi028_77856_object60060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:26:40.848 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.037797+0000", 2024-09-10T18:26:40.848 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 15.720778412, 2024-09-10T18:26:40.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.155207649, 2024-09-10T18:26:40.848 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.848 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.848 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:40.848 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:40.848 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:40.848 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181160 2024-09-10T18:26:40.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.849 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037797+0000", 2024-09-10T18:26:40.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037797+0000", 2024-09-10T18:26:40.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037795+0000", 2024-09-10T18:26:40.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:26:40.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037810+0000", 2024-09-10T18:26:40.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4810000000000001e-05 2024-09-10T18:26:40.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037822+0000", 2024-09-10T18:26:40.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2568000000000001e-05 2024-09-10T18:26:40.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037829+0000", 2024-09-10T18:26:40.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.562e-06 2024-09-10T18:26:40.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.038647+0000", 2024-09-10T18:26:40.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081780299999999998 2024-09-10T18:26:40.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.038755+0000", 2024-09-10T18:26:40.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108941 2024-09-10T18:26:40.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-10T18:26:40.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.038876+0000", 2024-09-10T18:26:40.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000120259 2024-09-10T18:26:40.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:26:40.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041587+0000", 2024-09-10T18:26:40.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027112709999999999 2024-09-10T18:26:40.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191165+0000", 2024-09-10T18:26:40.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14957848300000001 2024-09-10T18:26:40.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.192642+0000", 2024-09-10T18:26:40.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001476148 2024-09-10T18:26:40.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.193004+0000", 2024-09-10T18:26:40.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00036274700000000002 2024-09-10T18:26:40.853 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.853 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.853 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.854 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181165 7.6 7:69ba312f:::benchmark_data_smithi028_77856_object60065:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:26:40.854 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.041719+0000", 2024-09-10T18:26:40.854 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 15.716856135, 2024-09-10T18:26:40.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14957322100000001, 2024-09-10T18:26:40.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.854 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.854 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:40.854 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:40.854 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:40.854 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181165 2024-09-10T18:26:40.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.854 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041719+0000", 2024-09-10T18:26:40.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041719+0000", 2024-09-10T18:26:40.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041718+0000", 2024-09-10T18:26:40.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:26:40.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041723+0000", 2024-09-10T18:26:40.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.48e-06 2024-09-10T18:26:40.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041733+0000", 2024-09-10T18:26:40.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6520000000000001e-06 2024-09-10T18:26:40.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041738+0000", 2024-09-10T18:26:40.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.553e-06 2024-09-10T18:26:40.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044567+0000", 2024-09-10T18:26:40.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028282770000000001 2024-09-10T18:26:40.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044684+0000", 2024-09-10T18:26:40.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117491 2024-09-10T18:26:40.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:26:40.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044768+0000", 2024-09-10T18:26:40.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4078000000000004e-05 2024-09-10T18:26:40.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.049844+0000", 2024-09-10T18:26:40.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050756799999999999 2024-09-10T18:26:40.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:26:40.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.050265+0000", 2024-09-10T18:26:40.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00042087299999999999 2024-09-10T18:26:40.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191082+0000", 2024-09-10T18:26:40.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14081718500000001 2024-09-10T18:26:40.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191174+0000", 2024-09-10T18:26:40.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1875999999999999e-05 2024-09-10T18:26:40.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191292+0000", 2024-09-10T18:26:40.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000118331 2024-09-10T18:26:40.860 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.860 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.860 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.860 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181169 7.6 7:60f36a9a:::benchmark_data_smithi028_77856_object60069:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:26:40.860 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.044506+0000", 2024-09-10T18:26:40.860 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 15.714069134000001, 2024-09-10T18:26:40.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14689063499999999, 2024-09-10T18:26:40.860 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.860 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.860 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:40.860 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:40.860 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:40.861 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181169 2024-09-10T18:26:40.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.861 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044506+0000", 2024-09-10T18:26:40.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044506+0000", 2024-09-10T18:26:40.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044504+0000", 2024-09-10T18:26:40.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:26:40.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044510+0000", 2024-09-10T18:26:40.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2729999999999999e-06 2024-09-10T18:26:40.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044520+0000", 2024-09-10T18:26:40.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8470000000000008e-06 2024-09-10T18:26:40.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044525+0000", 2024-09-10T18:26:40.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1499999999999998e-06 2024-09-10T18:26:40.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.046791+0000", 2024-09-10T18:26:40.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022659580000000002 2024-09-10T18:26:40.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.046905+0000", 2024-09-10T18:26:40.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011319699999999999 2024-09-10T18:26:40.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:26:40.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.047013+0000", 2024-09-10T18:26:40.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108261 2024-09-10T18:26:40.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.050258+0000", 2024-09-10T18:26:40.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032447309999999998 2024-09-10T18:26:40.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:26:40.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.050273+0000", 2024-09-10T18:26:40.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4898e-05 2024-09-10T18:26:40.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191325+0000", 2024-09-10T18:26:40.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.141052908 2024-09-10T18:26:40.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191363+0000", 2024-09-10T18:26:40.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7694000000000002e-05 2024-09-10T18:26:40.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191397+0000", 2024-09-10T18:26:40.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3528999999999997e-05 2024-09-10T18:26:40.866 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.866 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.866 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.866 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181171 7.f 7:fefa0a2a:::benchmark_data_smithi028_77856_object60071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:26:40.866 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.044900+0000", 2024-09-10T18:26:40.866 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 15.71367542, 2024-09-10T18:26:40.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14605806099999999, 2024-09-10T18:26:40.866 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.866 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.866 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:40.866 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:40.866 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:40.867 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181171 2024-09-10T18:26:40.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.867 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044900+0000", 2024-09-10T18:26:40.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044900+0000", 2024-09-10T18:26:40.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044898+0000", 2024-09-10T18:26:40.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:26:40.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044904+0000", 2024-09-10T18:26:40.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9050000000000002e-06 2024-09-10T18:26:40.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044914+0000", 2024-09-10T18:26:40.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0164e-05 2024-09-10T18:26:40.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044920+0000", 2024-09-10T18:26:40.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5539999999999997e-06 2024-09-10T18:26:40.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.045532+0000", 2024-09-10T18:26:40.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00061150500000000003 2024-09-10T18:26:40.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.045628+0000", 2024-09-10T18:26:40.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6620999999999995e-05 2024-09-10T18:26:40.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:26:40.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.045727+0000", 2024-09-10T18:26:40.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8969999999999996e-05 2024-09-10T18:26:40.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048704+0000", 2024-09-10T18:26:40.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029767560000000001 2024-09-10T18:26:40.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:26:40.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048783+0000", 2024-09-10T18:26:40.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9275000000000002e-05 2024-09-10T18:26:40.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.190858+0000", 2024-09-10T18:26:40.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14207499100000001 2024-09-10T18:26:40.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.190919+0000", 2024-09-10T18:26:40.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0489000000000003e-05 2024-09-10T18:26:40.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.190958+0000", 2024-09-10T18:26:40.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9171999999999997e-05 2024-09-10T18:26:40.871 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.871 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.872 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.872 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:181174 7.5 e142/93 7:a54ae781:::benchmark_data_smithi028_77856_object60074:head v 142'11303, mlcod=142'11301)", 2024-09-10T18:26:40.872 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.048963+0000", 2024-09-10T18:26:40.872 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 15.709611827, 2024-09-10T18:26:40.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14980299699999999, 2024-09-10T18:26:40.872 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.872 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.872 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048963+0000", 2024-09-10T18:26:40.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048963+0000", 2024-09-10T18:26:40.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048962+0000", 2024-09-10T18:26:40.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:26:40.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048969+0000", 2024-09-10T18:26:40.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.08e-06 2024-09-10T18:26:40.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048979+0000", 2024-09-10T18:26:40.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.006e-05 2024-09-10T18:26:40.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048985+0000", 2024-09-10T18:26:40.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8780000000000003e-06 2024-09-10T18:26:40.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.049055+0000", 2024-09-10T18:26:40.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9560000000000005e-05 2024-09-10T18:26:40.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.049072+0000", 2024-09-10T18:26:40.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7476e-05 2024-09-10T18:26:40.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198732+0000", 2024-09-10T18:26:40.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14965962499999999 2024-09-10T18:26:40.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198766+0000", 2024-09-10T18:26:40.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4514000000000002e-05 2024-09-10T18:26:40.875 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.875 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.875 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.876 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:181175 7.5 e142/93 7:a07e4061:::benchmark_data_smithi028_77856_object60075:head v 142'11304, mlcod=142'11301)", 2024-09-10T18:26:40.876 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.051631+0000", 2024-09-10T18:26:40.876 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 15.706943904999999, 2024-09-10T18:26:40.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15340213799999999, 2024-09-10T18:26:40.876 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.876 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051631+0000", 2024-09-10T18:26:40.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051631+0000", 2024-09-10T18:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051630+0000", 2024-09-10T18:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051639+0000", 2024-09-10T18:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8189999999999998e-06 2024-09-10T18:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051649+0000", 2024-09-10T18:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0101999999999999e-05 2024-09-10T18:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051655+0000", 2024-09-10T18:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9490000000000001e-06 2024-09-10T18:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051705+0000", 2024-09-10T18:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0328000000000003e-05 2024-09-10T18:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051722+0000", 2024-09-10T18:26:40.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6617999999999999e-05 2024-09-10T18:26:40.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205001+0000", 2024-09-10T18:26:40.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15327945300000001 2024-09-10T18:26:40.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205033+0000", 2024-09-10T18:26:40.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2257e-05 2024-09-10T18:26:40.879 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.879 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.879 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.880 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181177 7.f 7:f9ebf4ca:::benchmark_data_smithi028_77856_object60077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:26:40.880 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.053108+0000", 2024-09-10T18:26:40.880 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 15.705466812000001, 2024-09-10T18:26:40.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15150366200000001, 2024-09-10T18:26:40.880 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.880 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.880 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:26:40.880 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:26:40.880 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:26:40.880 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181177 2024-09-10T18:26:40.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.880 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.053108+0000", 2024-09-10T18:26:40.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.053108+0000", 2024-09-10T18:26:40.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.053106+0000", 2024-09-10T18:26:40.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:26:40.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.053113+0000", 2024-09-10T18:26:40.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0520000000000004e-06 2024-09-10T18:26:40.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.190879+0000", 2024-09-10T18:26:40.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13776606999999999 2024-09-10T18:26:40.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.190891+0000", 2024-09-10T18:26:40.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1102e-05 2024-09-10T18:26:40.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.190985+0000", 2024-09-10T18:26:40.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4087999999999998e-05 2024-09-10T18:26:40.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191099+0000", 2024-09-10T18:26:40.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114473 2024-09-10T18:26:40.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:26:40.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191299+0000", 2024-09-10T18:26:40.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019984899999999999 2024-09-10T18:26:40.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.197973+0000", 2024-09-10T18:26:40.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066738869999999999 2024-09-10T18:26:40.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:26:40.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198142+0000", 2024-09-10T18:26:40.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000169006 2024-09-10T18:26:40.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:26:40.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.204473+0000", 2024-09-10T18:26:40.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063307729999999996 2024-09-10T18:26:40.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.204527+0000", 2024-09-10T18:26:40.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4588e-05 2024-09-10T18:26:40.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.204612+0000", 2024-09-10T18:26:40.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.475e-05 2024-09-10T18:26:40.886 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.886 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.886 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.886 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:181183 7.d e142/95 7:b1361f07:::benchmark_data_smithi028_77856_object60083:head v 142'11153, mlcod=142'11151)", 2024-09-10T18:26:40.886 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.055525+0000", 2024-09-10T18:26:40.886 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 15.703050449999999, 2024-09-10T18:26:40.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14994769499999999, 2024-09-10T18:26:40.886 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:26:40.886 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:26:40.886 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:26:40.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:26:40.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055525+0000", 2024-09-10T18:26:40.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:26:40.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055525+0000", 2024-09-10T18:26:40.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:26:40.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:26:40.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055523+0000", 2024-09-10T18:26:40.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:26:40.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:26:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055532+0000", 2024-09-10T18:26:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0159999999999997e-06 2024-09-10T18:26:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:26:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191429+0000", 2024-09-10T18:26:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13589642099999999 2024-09-10T18:26:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:26:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191435+0000", 2024-09-10T18:26:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9569999999999999e-06 2024-09-10T18:26:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:26:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198700+0000", 2024-09-10T18:26:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0072655549999999999 2024-09-10T18:26:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:26:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198722+0000", 2024-09-10T18:26:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1307000000000001e-05 2024-09-10T18:26:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:26:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205091+0000", 2024-09-10T18:26:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063695779999999999 2024-09-10T18:26:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:26:40.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:26:40.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:26:40.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205472+0000", 2024-09-10T18:26:40.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00038129400000000001 2024-09-10T18:26:40.890 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.890 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.890 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.890 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:26:40.890 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:26:40.890 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:26:40.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:40.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:40.934 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:40.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:41.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:40 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 13435/159663 objects degraded (8.415%), 5 pgs degraded (PG_DEGRADED) 2024-09-10T18:26:41.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:40 smithi028 ceph-mon[39220]: osdmap e153: 8 total, 8 up, 8 in 2024-09-10T18:26:41.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:40 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 13435/159663 objects degraded (8.415%), 5 pgs degraded (PG_DEGRADED) 2024-09-10T18:26:41.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:40 smithi028 ceph-mon[29291]: osdmap e153: 8 total, 8 up, 8 in 2024-09-10T18:26:41.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:41.059 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:41.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:41.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:40 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 13435/159663 objects degraded (8.415%), 5 pgs degraded (PG_DEGRADED) 2024-09-10T18:26:41.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:40 smithi204 ceph-mon[32357]: osdmap e153: 8 total, 8 up, 8 in 2024-09-10T18:26:41.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:41.185 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:41.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:41.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:41.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:26:41.287 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:41.312 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:41.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:41.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:41.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:41.463 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:41.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:41.564 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:26:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:26:41] "GET /metrics HTTP/1.1" 200 224028 "" "Prometheus/2.43.0" 2024-09-10T18:26:41.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:41.587 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:41.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:41.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:41.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:41.737 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:41.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:41.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:41.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:41.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:41.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:41.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:42.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:41 smithi028 ceph-mon[39220]: pgmap v217: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 10002/146352 objects degraded (6.834%); 38 MiB/s, 601 objects/s recovering 2024-09-10T18:26:42.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:41 smithi028 ceph-mon[29291]: pgmap v217: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 10002/146352 objects degraded (6.834%); 38 MiB/s, 601 objects/s recovering 2024-09-10T18:26:42.025 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:42.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:42.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:41 smithi204 ceph-mon[32357]: pgmap v217: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 10002/146352 objects degraded (6.834%); 38 MiB/s, 601 objects/s recovering 2024-09-10T18:26:42.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:42.168 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:42.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:42.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:42.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:26:42.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:42.316 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:42.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:42.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:42.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:42.457 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:42.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:42.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:42.559 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:42.586 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:42.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:42.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:42.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:26:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:26:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:26:42.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T18:26:42.687 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:42.710 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:42.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:42.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:42.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:42.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:42.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:42.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:42.964 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:42.989 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:42.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:42.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:43.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:43.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:43.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:43.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:43.232 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:26:43.254 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:43.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:43.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:43.265 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:26:43.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:43.418 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:43.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:43.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:43.563 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:26:43.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:43.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:43.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:43.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:43.730 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:43.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:43.831 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:43.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:43.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:43.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:43.983 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:43.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:43.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:44.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:43 smithi028 ceph-mon[39220]: pgmap v218: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 904 B/s rd, 0 B/s wr, 784 op/s; 10002/146352 objects degraded (6.834%); 27 MiB/s, 424 objects/s recovering 2024-09-10T18:26:44.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:43 smithi028 ceph-mon[29291]: pgmap v218: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 904 B/s rd, 0 B/s wr, 784 op/s; 10002/146352 objects degraded (6.834%); 27 MiB/s, 424 objects/s recovering 2024-09-10T18:26:44.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:44.129 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:44.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:44.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:44.268 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:44.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:44.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:43 smithi204 ceph-mon[32357]: pgmap v218: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 904 B/s rd, 0 B/s wr, 784 op/s; 10002/146352 objects degraded (6.834%); 27 MiB/s, 424 objects/s recovering 2024-09-10T18:26:44.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:26:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:26:44.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:44.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:44.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:44.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:44.562 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:44.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:44.664 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:44.692 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:44.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:44.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:44.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:44.833 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:44.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:44.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:44.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:44.980 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:44.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:44.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:45.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:44 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:45.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:44 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:45.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:45.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:45.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:45.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:45.289 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:45.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:45.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:44 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:45.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:45.415 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:45.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:45.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:45.516 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:45.542 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:45.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:45.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:45.644 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:45.667 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:45.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:45.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:45.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:45.793 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:45.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:45.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:45.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.0 to 1 (10000) 2024-09-10T18:26:45.937 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:45.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:45 smithi028 ceph-mon[29291]: pgmap v219: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 803 op/s; 10002/143640 objects degraded (6.963%); 23 MiB/s, 361 objects/s recovering 2024-09-10T18:26:45.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:45 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 10002/143640 objects degraded (6.963%), 4 pgs degraded (PG_DEGRADED) 2024-09-10T18:26:45.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:45 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1487306237' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:26:45.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:45 smithi028 ceph-mon[29291]: from='client.34889 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:26:45.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:45.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:46.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:46.085 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:46.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:46.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:46.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:46.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:45 smithi028 ceph-mon[39220]: pgmap v219: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 803 op/s; 10002/143640 objects degraded (6.963%); 23 MiB/s, 361 objects/s recovering 2024-09-10T18:26:46.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:45 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 10002/143640 objects degraded (6.963%), 4 pgs degraded (PG_DEGRADED) 2024-09-10T18:26:46.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:45 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1487306237' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:26:46.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:45 smithi028 ceph-mon[39220]: from='client.34889 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:26:46.229 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:46.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:46.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:46.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:45 smithi204 ceph-mon[32357]: pgmap v219: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 803 op/s; 10002/143640 objects degraded (6.963%); 23 MiB/s, 361 objects/s recovering 2024-09-10T18:26:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:45 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 10002/143640 objects degraded (6.963%), 4 pgs degraded (PG_DEGRADED) 2024-09-10T18:26:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:45 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1487306237' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:26:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:45 smithi204 ceph-mon[32357]: from='client.34889 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:26:46.366 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:46.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:46.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:46.468 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:46.495 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:46.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:46.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:46.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:46.643 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:46.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:46.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:46.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:46.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:46.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:46.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:46.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:46.922 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:46.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:46.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:46.945 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:26:47.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:47.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:47.052 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:47.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:47.153 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:47.174 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:47.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:47.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:47.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:46 smithi028 ceph-mon[29291]: from='client.34889 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T18:26:47.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:46 smithi028 ceph-mon[29291]: osdmap e154: 8 total, 8 up, 8 in 2024-09-10T18:26:47.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:46 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4016329326' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:26:47.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:46 smithi028 ceph-mon[39220]: from='client.34889 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T18:26:47.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:46 smithi028 ceph-mon[39220]: osdmap e154: 8 total, 8 up, 8 in 2024-09-10T18:26:47.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:46 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4016329326' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:26:47.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:47.299 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:47.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:46 smithi204 ceph-mon[32357]: from='client.34889 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T18:26:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:46 smithi204 ceph-mon[32357]: osdmap e154: 8 total, 8 up, 8 in 2024-09-10T18:26:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:46 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4016329326' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:26:47.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:47.423 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:47.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:47.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:47.525 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:47.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:47.546 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:47.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:47.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:47.668 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:47.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:47.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:47.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:47.791 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:47.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:47.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:47.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:47.914 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:47.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:47.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:48.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:48.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:48.051 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:48.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:48.152 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:48.179 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:48.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:48.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:48.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:47 smithi028 ceph-mon[29291]: pgmap v221: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s; 8328/131589 objects degraded (6.329%); 25 MiB/s, 404 objects/s recovering 2024-09-10T18:26:48.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:47 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:26:48.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:47 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4016329326' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:26:48.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:47 smithi028 ceph-mon[29291]: osdmap e155: 8 total, 8 up, 8 in 2024-09-10T18:26:48.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:47 smithi028 ceph-mon[29291]: 3.3 scrub starts 2024-09-10T18:26:48.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:47 smithi028 ceph-mon[29291]: 3.3 scrub ok 2024-09-10T18:26:48.204 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:47 smithi028 ceph-mon[39220]: pgmap v221: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s; 8328/131589 objects degraded (6.329%); 25 MiB/s, 404 objects/s recovering 2024-09-10T18:26:48.204 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:47 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:26:48.204 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:47 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4016329326' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:26:48.204 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:47 smithi028 ceph-mon[39220]: osdmap e155: 8 total, 8 up, 8 in 2024-09-10T18:26:48.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:47 smithi028 ceph-mon[39220]: 3.3 scrub starts 2024-09-10T18:26:48.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:47 smithi028 ceph-mon[39220]: 3.3 scrub ok 2024-09-10T18:26:48.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:48.307 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:48.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:48.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:47 smithi204 ceph-mon[32357]: pgmap v221: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s; 8328/131589 objects degraded (6.329%); 25 MiB/s, 404 objects/s recovering 2024-09-10T18:26:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:47 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:26:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:47 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4016329326' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:26:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:47 smithi204 ceph-mon[32357]: osdmap e155: 8 total, 8 up, 8 in 2024-09-10T18:26:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:47 smithi204 ceph-mon[32357]: 3.3 scrub starts 2024-09-10T18:26:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:47 smithi204 ceph-mon[32357]: 3.3 scrub ok 2024-09-10T18:26:48.408 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:48.434 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:48.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:48.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:48.536 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:48.560 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:48.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:48.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:48.662 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:48.686 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:48.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:48.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:48.814 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:48.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:48.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:48.915 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:48.940 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:48.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:49.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:49.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:49.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:49.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:49.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:48 smithi028 ceph-mon[39220]: 5.3 scrub starts 2024-09-10T18:26:49.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:48 smithi028 ceph-mon[39220]: 5.3 scrub ok 2024-09-10T18:26:49.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:48 smithi028 ceph-mon[39220]: osdmap e156: 8 total, 8 up, 8 in 2024-09-10T18:26:49.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:48 smithi028 ceph-mon[39220]: 5.8 scrub starts 2024-09-10T18:26:49.155 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:48 smithi028 ceph-mon[39220]: 5.8 scrub ok 2024-09-10T18:26:49.155 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:48 smithi028 ceph-mon[29291]: 5.3 scrub starts 2024-09-10T18:26:49.155 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:48 smithi028 ceph-mon[29291]: 5.3 scrub ok 2024-09-10T18:26:49.155 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:48 smithi028 ceph-mon[29291]: osdmap e156: 8 total, 8 up, 8 in 2024-09-10T18:26:49.155 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:48 smithi028 ceph-mon[29291]: 5.8 scrub starts 2024-09-10T18:26:49.155 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:48 smithi028 ceph-mon[29291]: 5.8 scrub ok 2024-09-10T18:26:49.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:49.192 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:49.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:49.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:49.294 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:49.318 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:49.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:49.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:48 smithi204 ceph-mon[32357]: 5.3 scrub starts 2024-09-10T18:26:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:48 smithi204 ceph-mon[32357]: 5.3 scrub ok 2024-09-10T18:26:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:48 smithi204 ceph-mon[32357]: osdmap e156: 8 total, 8 up, 8 in 2024-09-10T18:26:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:48 smithi204 ceph-mon[32357]: 5.8 scrub starts 2024-09-10T18:26:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:48 smithi204 ceph-mon[32357]: 5.8 scrub ok 2024-09-10T18:26:49.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:49.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:49.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:49.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:49.564 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:49.590 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:49.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:49.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:49.716 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:49.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:49.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:49.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:49.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:49.862 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:49.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:49.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:49.989 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:49.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:49.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:50.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:50.127 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:50.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:50.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:50.229 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:50.250 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:50.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:50.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:49 smithi028 ceph-mon[39220]: pgmap v224: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 820 op/s; 8328/131589 objects degraded (6.329%); 7.2 MiB/s, 115 objects/s recovering 2024-09-10T18:26:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:49 smithi028 ceph-mon[39220]: 4.1 scrub starts 2024-09-10T18:26:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:49 smithi028 ceph-mon[39220]: 4.1 scrub ok 2024-09-10T18:26:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:49 smithi028 ceph-mon[39220]: 3.d scrub starts 2024-09-10T18:26:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:49 smithi028 ceph-mon[39220]: 3.d scrub ok 2024-09-10T18:26:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:49 smithi028 ceph-mon[39220]: 5.2 scrub starts 2024-09-10T18:26:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:49 smithi028 ceph-mon[39220]: 5.2 scrub ok 2024-09-10T18:26:50.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:49 smithi028 ceph-mon[29291]: pgmap v224: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 820 op/s; 8328/131589 objects degraded (6.329%); 7.2 MiB/s, 115 objects/s recovering 2024-09-10T18:26:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:49 smithi028 ceph-mon[29291]: 4.1 scrub starts 2024-09-10T18:26:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:49 smithi028 ceph-mon[29291]: 4.1 scrub ok 2024-09-10T18:26:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:49 smithi028 ceph-mon[29291]: 3.d scrub starts 2024-09-10T18:26:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:49 smithi028 ceph-mon[29291]: 3.d scrub ok 2024-09-10T18:26:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:49 smithi028 ceph-mon[29291]: 5.2 scrub starts 2024-09-10T18:26:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:49 smithi028 ceph-mon[29291]: 5.2 scrub ok 2024-09-10T18:26:50.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:49 smithi204 ceph-mon[32357]: pgmap v224: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 154 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 820 op/s; 8328/131589 objects degraded (6.329%); 7.2 MiB/s, 115 objects/s recovering 2024-09-10T18:26:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:49 smithi204 ceph-mon[32357]: 4.1 scrub starts 2024-09-10T18:26:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:49 smithi204 ceph-mon[32357]: 4.1 scrub ok 2024-09-10T18:26:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:49 smithi204 ceph-mon[32357]: 3.d scrub starts 2024-09-10T18:26:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:49 smithi204 ceph-mon[32357]: 3.d scrub ok 2024-09-10T18:26:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:49 smithi204 ceph-mon[32357]: 5.2 scrub starts 2024-09-10T18:26:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:49 smithi204 ceph-mon[32357]: 5.2 scrub ok 2024-09-10T18:26:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:50.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:50.497 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:50.535 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:50.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:50.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:50.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:26:50.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:50.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:50.692 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:50.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:50.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:50.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:50.852 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:50.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:50.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:50.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:50 smithi028 ceph-mon[29291]: 5.11 scrub starts 2024-09-10T18:26:50.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:50 smithi028 ceph-mon[29291]: 5.11 scrub ok 2024-09-10T18:26:50.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:50 smithi028 ceph-mon[29291]: 6.a scrub starts 2024-09-10T18:26:50.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:50 smithi028 ceph-mon[29291]: 6.a scrub ok 2024-09-10T18:26:50.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:50 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 8328/131589 objects degraded (6.329%), 4 pgs degraded (PG_DEGRADED) 2024-09-10T18:26:50.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:50 smithi028 ceph-mon[29291]: osdmap e157: 8 total, 8 up, 8 in 2024-09-10T18:26:50.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:50 smithi028 ceph-mon[39220]: 5.11 scrub starts 2024-09-10T18:26:50.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:50 smithi028 ceph-mon[39220]: 5.11 scrub ok 2024-09-10T18:26:50.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:50 smithi028 ceph-mon[39220]: 6.a scrub starts 2024-09-10T18:26:50.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:50 smithi028 ceph-mon[39220]: 6.a scrub ok 2024-09-10T18:26:50.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:50 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 8328/131589 objects degraded (6.329%), 4 pgs degraded (PG_DEGRADED) 2024-09-10T18:26:50.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:50 smithi028 ceph-mon[39220]: osdmap e157: 8 total, 8 up, 8 in 2024-09-10T18:26:51.005 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:51.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:51.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:51.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:51.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:51.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:51.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:51.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:51.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:50 smithi204 ceph-mon[32357]: 5.11 scrub starts 2024-09-10T18:26:51.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:50 smithi204 ceph-mon[32357]: 5.11 scrub ok 2024-09-10T18:26:51.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:50 smithi204 ceph-mon[32357]: 6.a scrub starts 2024-09-10T18:26:51.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:50 smithi204 ceph-mon[32357]: 6.a scrub ok 2024-09-10T18:26:51.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:50 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 8328/131589 objects degraded (6.329%), 4 pgs degraded (PG_DEGRADED) 2024-09-10T18:26:51.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:50 smithi204 ceph-mon[32357]: osdmap e157: 8 total, 8 up, 8 in 2024-09-10T18:26:51.271 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:51.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:51.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:51.373 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:51.400 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:51.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:51.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:51.403 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:26:51.403 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:26:51.403 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:26:51.403 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:26:51.403 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:26:51.403 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:26:51.403 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:26:51.403 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:26:51.403 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:26:51.404 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:26:51.404 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:26:51.404 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:26:51.404 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:26:51.404 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:26:51.404 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:26:51.404 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:26:51.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:51.505 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:26:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:26:51] "GET /metrics HTTP/1.1" 200 224028 "" "Prometheus/2.43.0" 2024-09-10T18:26:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:51.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:51.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:51.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:51.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:51.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:51.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:51.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:51.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:51.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:51.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:51.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:51.993 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:51 smithi204 ceph-mon[32357]: pgmap v226: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 915 op/s; 7334/118062 objects degraded (6.212%); 2.3 MiB/s, 36 objects/s recovering 2024-09-10T18:26:51.993 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:51 smithi204 ceph-mon[32357]: 3.11 scrub starts 2024-09-10T18:26:51.993 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:51 smithi204 ceph-mon[32357]: 3.11 scrub ok 2024-09-10T18:26:51.993 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:51 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4207546862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:26:51.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:51 smithi204 ceph-mon[32357]: 4.8 scrub starts 2024-09-10T18:26:51.994 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:51 smithi204 ceph-mon[32357]: 4.8 scrub ok 2024-09-10T18:26:52.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:51 smithi028 ceph-mon[39220]: pgmap v226: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 915 op/s; 7334/118062 objects degraded (6.212%); 2.3 MiB/s, 36 objects/s recovering 2024-09-10T18:26:52.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:51 smithi028 ceph-mon[39220]: 3.11 scrub starts 2024-09-10T18:26:52.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:51 smithi028 ceph-mon[39220]: 3.11 scrub ok 2024-09-10T18:26:52.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:51 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4207546862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:26:52.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:51 smithi028 ceph-mon[39220]: 4.8 scrub starts 2024-09-10T18:26:52.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:51 smithi028 ceph-mon[39220]: 4.8 scrub ok 2024-09-10T18:26:52.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:51 smithi028 ceph-mon[29291]: pgmap v226: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 915 op/s; 7334/118062 objects degraded (6.212%); 2.3 MiB/s, 36 objects/s recovering 2024-09-10T18:26:52.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:51 smithi028 ceph-mon[29291]: 3.11 scrub starts 2024-09-10T18:26:52.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:51 smithi028 ceph-mon[29291]: 3.11 scrub ok 2024-09-10T18:26:52.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:51 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4207546862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:26:52.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:51 smithi028 ceph-mon[29291]: 4.8 scrub starts 2024-09-10T18:26:52.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:51 smithi028 ceph-mon[29291]: 4.8 scrub ok 2024-09-10T18:26:52.088 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:52.121 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:52.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:52.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:52.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:52.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:52.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:52.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:26:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:26:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:26:52.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:52.404 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:52.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:52.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:52.537 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:52.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:52.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:52.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:52.675 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:52.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:52.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:52.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:52.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-10T18:26:52.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:52.862 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:52.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:52.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:52.992 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:52.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:52.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:53.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:53.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:53.139 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:53.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:53.239 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:52 smithi028 ceph-mon[29291]: 6.1 scrub starts 2024-09-10T18:26:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:52 smithi028 ceph-mon[29291]: 6.1 scrub ok 2024-09-10T18:26:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:52 smithi028 ceph-mon[29291]: 5.1f scrub starts 2024-09-10T18:26:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:52 smithi028 ceph-mon[29291]: 5.1f scrub ok 2024-09-10T18:26:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:52 smithi028 ceph-mon[29291]: 6.13 scrub starts 2024-09-10T18:26:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:52 smithi028 ceph-mon[29291]: 6.13 scrub ok 2024-09-10T18:26:53.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:52 smithi028 ceph-mon[39220]: 6.1 scrub starts 2024-09-10T18:26:53.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:52 smithi028 ceph-mon[39220]: 6.1 scrub ok 2024-09-10T18:26:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:52 smithi028 ceph-mon[39220]: 5.1f scrub starts 2024-09-10T18:26:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:52 smithi028 ceph-mon[39220]: 5.1f scrub ok 2024-09-10T18:26:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:52 smithi028 ceph-mon[39220]: 6.13 scrub starts 2024-09-10T18:26:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:52 smithi028 ceph-mon[39220]: 6.13 scrub ok 2024-09-10T18:26:53.266 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:53.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:53.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:52 smithi204 ceph-mon[32357]: 6.1 scrub starts 2024-09-10T18:26:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:52 smithi204 ceph-mon[32357]: 6.1 scrub ok 2024-09-10T18:26:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:52 smithi204 ceph-mon[32357]: 5.1f scrub starts 2024-09-10T18:26:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:52 smithi204 ceph-mon[32357]: 5.1f scrub ok 2024-09-10T18:26:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:52 smithi204 ceph-mon[32357]: 6.13 scrub starts 2024-09-10T18:26:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:52 smithi204 ceph-mon[32357]: 6.13 scrub ok 2024-09-10T18:26:53.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:53.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:53.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:53.554 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:53.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:53.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:53.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:53.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:53.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:53.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:53.843 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:53.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:53.945 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:53.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:53.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:54.073 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:54.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:54.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:54.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:54.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:54.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:54.239 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:54.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:53 smithi028 ceph-mon[39220]: 4.12 scrub starts 2024-09-10T18:26:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:53 smithi028 ceph-mon[39220]: 4.12 scrub ok 2024-09-10T18:26:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:53 smithi028 ceph-mon[39220]: 3.12 scrub starts 2024-09-10T18:26:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:53 smithi028 ceph-mon[39220]: 3.12 scrub ok 2024-09-10T18:26:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:53 smithi028 ceph-mon[39220]: pgmap v227: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 751 op/s; 7334/118062 objects degraded (6.212%); 1.9 MiB/s, 29 objects/s recovering 2024-09-10T18:26:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:53 smithi028 ceph-mon[39220]: 3.c scrub starts 2024-09-10T18:26:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:53 smithi028 ceph-mon[39220]: 3.c scrub ok 2024-09-10T18:26:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:53 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:26:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:53 smithi028 ceph-mon[29291]: 4.12 scrub starts 2024-09-10T18:26:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:53 smithi028 ceph-mon[29291]: 4.12 scrub ok 2024-09-10T18:26:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:53 smithi028 ceph-mon[29291]: 3.12 scrub starts 2024-09-10T18:26:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:53 smithi028 ceph-mon[29291]: 3.12 scrub ok 2024-09-10T18:26:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:53 smithi028 ceph-mon[29291]: pgmap v227: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 751 op/s; 7334/118062 objects degraded (6.212%); 1.9 MiB/s, 29 objects/s recovering 2024-09-10T18:26:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:53 smithi028 ceph-mon[29291]: 3.c scrub starts 2024-09-10T18:26:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:53 smithi028 ceph-mon[29291]: 3.c scrub ok 2024-09-10T18:26:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:53 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:26:54.340 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:54.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:26:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:26:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:53 smithi204 ceph-mon[32357]: 4.12 scrub starts 2024-09-10T18:26:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:53 smithi204 ceph-mon[32357]: 4.12 scrub ok 2024-09-10T18:26:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:53 smithi204 ceph-mon[32357]: 3.12 scrub starts 2024-09-10T18:26:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:53 smithi204 ceph-mon[32357]: 3.12 scrub ok 2024-09-10T18:26:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:53 smithi204 ceph-mon[32357]: pgmap v227: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 751 op/s; 7334/118062 objects degraded (6.212%); 1.9 MiB/s, 29 objects/s recovering 2024-09-10T18:26:54.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:53 smithi204 ceph-mon[32357]: 3.c scrub starts 2024-09-10T18:26:54.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:53 smithi204 ceph-mon[32357]: 3.c scrub ok 2024-09-10T18:26:54.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:53 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:26:54.365 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:54.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:54.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:54.467 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:54.492 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:54.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:54.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:54.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:54.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:54.622 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:54.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:54.723 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:54.744 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:54.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:54.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:54.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:54.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:26:54.883 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:54.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:54.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:54.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:55.025 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:55.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:55.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:55.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:55.177 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:55.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:55.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:55.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:54 smithi028 ceph-mon[29291]: 6.15 scrub starts 2024-09-10T18:26:55.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:54 smithi028 ceph-mon[29291]: 6.15 scrub ok 2024-09-10T18:26:55.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:54 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:55.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:54 smithi028 ceph-mon[29291]: 2.2 scrub starts 2024-09-10T18:26:55.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:54 smithi028 ceph-mon[29291]: 2.2 scrub ok 2024-09-10T18:26:55.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:54 smithi028 ceph-mon[39220]: 6.15 scrub starts 2024-09-10T18:26:55.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:54 smithi028 ceph-mon[39220]: 6.15 scrub ok 2024-09-10T18:26:55.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:54 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:55.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:54 smithi028 ceph-mon[39220]: 2.2 scrub starts 2024-09-10T18:26:55.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:54 smithi028 ceph-mon[39220]: 2.2 scrub ok 2024-09-10T18:26:55.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:55.329 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:55.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:55.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:55.357 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:54 smithi204 ceph-mon[32357]: 6.15 scrub starts 2024-09-10T18:26:55.357 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:54 smithi204 ceph-mon[32357]: 6.15 scrub ok 2024-09-10T18:26:55.357 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:54 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:26:55.357 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:54 smithi204 ceph-mon[32357]: 2.2 scrub starts 2024-09-10T18:26:55.357 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:54 smithi204 ceph-mon[32357]: 2.2 scrub ok 2024-09-10T18:26:55.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:55.458 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:55.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:55.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:55.560 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:55.586 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:55.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:55.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:55.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:55.733 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:55.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:55.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:55.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:55.874 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:55.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:55.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:55.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:56.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:56.020 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:56.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:56.121 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:56.144 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:56.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:56.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:56.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:55 smithi028 ceph-mon[29291]: 3.0 scrub starts 2024-09-10T18:26:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:55 smithi028 ceph-mon[29291]: 3.0 scrub ok 2024-09-10T18:26:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:55 smithi028 ceph-mon[29291]: pgmap v228: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 2.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 691 op/s; 7334/115212 objects degraded (6.366%); 1.4 MiB/s, 22 objects/s recovering 2024-09-10T18:26:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:55 smithi028 ceph-mon[29291]: 4.5 scrub starts 2024-09-10T18:26:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:55 smithi028 ceph-mon[29291]: 4.5 scrub ok 2024-09-10T18:26:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:55 smithi028 ceph-mon[29291]: 3.2 scrub starts 2024-09-10T18:26:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:55 smithi028 ceph-mon[29291]: 3.2 scrub ok 2024-09-10T18:26:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:55 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 7334/115212 objects degraded (6.366%), 3 pgs degraded (PG_DEGRADED) 2024-09-10T18:26:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:55 smithi028 ceph-mon[39220]: 3.0 scrub starts 2024-09-10T18:26:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:55 smithi028 ceph-mon[39220]: 3.0 scrub ok 2024-09-10T18:26:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:55 smithi028 ceph-mon[39220]: pgmap v228: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 2.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 691 op/s; 7334/115212 objects degraded (6.366%); 1.4 MiB/s, 22 objects/s recovering 2024-09-10T18:26:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:55 smithi028 ceph-mon[39220]: 4.5 scrub starts 2024-09-10T18:26:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:55 smithi028 ceph-mon[39220]: 4.5 scrub ok 2024-09-10T18:26:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:55 smithi028 ceph-mon[39220]: 3.2 scrub starts 2024-09-10T18:26:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:55 smithi028 ceph-mon[39220]: 3.2 scrub ok 2024-09-10T18:26:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:55 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 7334/115212 objects degraded (6.366%), 3 pgs degraded (PG_DEGRADED) 2024-09-10T18:26:56.290 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:56.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:55 smithi204 ceph-mon[32357]: 3.0 scrub starts 2024-09-10T18:26:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:55 smithi204 ceph-mon[32357]: 3.0 scrub ok 2024-09-10T18:26:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:55 smithi204 ceph-mon[32357]: pgmap v228: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 2.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 691 op/s; 7334/115212 objects degraded (6.366%); 1.4 MiB/s, 22 objects/s recovering 2024-09-10T18:26:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:55 smithi204 ceph-mon[32357]: 4.5 scrub starts 2024-09-10T18:26:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:55 smithi204 ceph-mon[32357]: 4.5 scrub ok 2024-09-10T18:26:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:55 smithi204 ceph-mon[32357]: 3.2 scrub starts 2024-09-10T18:26:56.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:55 smithi204 ceph-mon[32357]: 3.2 scrub ok 2024-09-10T18:26:56.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:55 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 7334/115212 objects degraded (6.366%), 3 pgs degraded (PG_DEGRADED) 2024-09-10T18:26:56.391 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:56.417 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:56.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:56.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:56.557 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:56.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:56.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:56.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:56.683 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:56.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:56.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:56.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:56.832 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:56.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:56.860 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:26:56.904 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:26:56.904 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:26:56.933 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:56.957 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:56.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:56.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:57.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:57.108 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:57.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:57.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:57.210 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:57.236 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:57.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:57.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:57.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:57.363 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:57.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:57.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:57.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:57.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:57.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:57.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:57.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:57.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:57.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:57.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:57.796 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:57.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:57.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:57.898 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:26:57.924 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:57.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:57.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:26:58.013 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:26:58.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:58.050 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:58.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:58.151 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:58.177 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:58.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:58.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:58.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:57 smithi028 ceph-mon[39220]: 4.7 scrub starts 2024-09-10T18:26:58.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:57 smithi028 ceph-mon[39220]: 4.7 scrub ok 2024-09-10T18:26:58.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:57 smithi028 ceph-mon[39220]: pgmap v229: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s; 6315/102411 objects degraded (6.166%); 1.3 MiB/s, 20 objects/s recovering 2024-09-10T18:26:58.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:57 smithi028 ceph-mon[29291]: 4.7 scrub starts 2024-09-10T18:26:58.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:57 smithi028 ceph-mon[29291]: 4.7 scrub ok 2024-09-10T18:26:58.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:57 smithi028 ceph-mon[29291]: pgmap v229: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s; 6315/102411 objects degraded (6.166%); 1.3 MiB/s, 20 objects/s recovering 2024-09-10T18:26:58.279 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:58.303 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:58.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:58.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:26:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:57 smithi204 ceph-mon[32357]: 4.7 scrub starts 2024-09-10T18:26:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:57 smithi204 ceph-mon[32357]: 4.7 scrub ok 2024-09-10T18:26:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:57 smithi204 ceph-mon[32357]: pgmap v229: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s; 6315/102411 objects degraded (6.166%); 1.3 MiB/s, 20 objects/s recovering 2024-09-10T18:26:58.406 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:58.430 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:58.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:58.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:26:58.572 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:58.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:58.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:26:58.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:58.721 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:58.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:58.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:58.823 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:26:58.850 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:58.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:58.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:26:58.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:59.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:59.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:59.200 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:59.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:59.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:59.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:26:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:59 smithi204 ceph-mon[32357]: 5.19 scrub starts 2024-09-10T18:26:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:59 smithi204 ceph-mon[32357]: 5.19 scrub ok 2024-09-10T18:26:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:59 smithi204 ceph-mon[32357]: 6.c scrub starts 2024-09-10T18:26:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:59 smithi204 ceph-mon[32357]: 6.c scrub ok 2024-09-10T18:26:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:59 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:26:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:59 smithi204 ceph-mon[32357]: 5.f scrub starts 2024-09-10T18:26:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:26:59 smithi204 ceph-mon[32357]: 5.f scrub ok 2024-09-10T18:26:59.365 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:59.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:59.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:26:59.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:59 smithi028 ceph-mon[29291]: 5.19 scrub starts 2024-09-10T18:26:59.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:59 smithi028 ceph-mon[29291]: 5.19 scrub ok 2024-09-10T18:26:59.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:59 smithi028 ceph-mon[29291]: 6.c scrub starts 2024-09-10T18:26:59.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:59 smithi028 ceph-mon[29291]: 6.c scrub ok 2024-09-10T18:26:59.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:59 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:26:59.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:59 smithi028 ceph-mon[29291]: 5.f scrub starts 2024-09-10T18:26:59.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:26:59 smithi028 ceph-mon[29291]: 5.f scrub ok 2024-09-10T18:26:59.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:59 smithi028 ceph-mon[39220]: 5.19 scrub starts 2024-09-10T18:26:59.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:59 smithi028 ceph-mon[39220]: 5.19 scrub ok 2024-09-10T18:26:59.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:59 smithi028 ceph-mon[39220]: 6.c scrub starts 2024-09-10T18:26:59.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:59 smithi028 ceph-mon[39220]: 6.c scrub ok 2024-09-10T18:26:59.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:59 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:26:59.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:59 smithi028 ceph-mon[39220]: 5.f scrub starts 2024-09-10T18:26:59.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:26:59 smithi028 ceph-mon[39220]: 5.f scrub ok 2024-09-10T18:26:59.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:26:59.516 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:59.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:59.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:26:59.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:26:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:59.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:59.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:26:59.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:26:59.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:59.802 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:26:59.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:26:59.903 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:26:59.932 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:26:59.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:26:59.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:00.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:00.071 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:00.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:00.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:00.173 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:00.198 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:00.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:00.300 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:00.328 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:00.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:00.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:00 smithi204 ceph-mon[32357]: pgmap v230: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 973 op/s; 6315/102411 objects degraded (6.166%); 1.1 MiB/s, 17 objects/s recovering 2024-09-10T18:27:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:00 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:27:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:00 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:27:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:00 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:27:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:00 smithi204 ceph-mon[32357]: 5.16 scrub starts 2024-09-10T18:27:00.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:00.474 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:00.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:00.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:00 smithi028 ceph-mon[39220]: pgmap v230: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 973 op/s; 6315/102411 objects degraded (6.166%); 1.1 MiB/s, 17 objects/s recovering 2024-09-10T18:27:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:00 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:27:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:00 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:27:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:00 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:27:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:00 smithi028 ceph-mon[39220]: 5.16 scrub starts 2024-09-10T18:27:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:00 smithi028 ceph-mon[29291]: pgmap v230: 158 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 973 op/s; 6315/102411 objects degraded (6.166%); 1.1 MiB/s, 17 objects/s recovering 2024-09-10T18:27:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:00 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:27:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:00 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:27:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:00 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:27:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:00 smithi028 ceph-mon[29291]: 5.16 scrub starts 2024-09-10T18:27:00.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:00.602 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:00.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:00.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:00.651 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:00.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:00.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:00.770 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:00.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:00.871 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:00.897 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:00.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:00.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:01.053 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:01.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:01.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:01.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:01.206 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:01.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:01.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:01.244 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:01 smithi028 ceph-mon[39220]: 6.3 scrub starts 2024-09-10T18:27:01.244 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:01 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 6315/102411 objects degraded (6.166%), 3 pgs degraded (PG_DEGRADED) 2024-09-10T18:27:01.244 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:01 smithi028 ceph-mon[29291]: 6.3 scrub starts 2024-09-10T18:27:01.244 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:01 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 6315/102411 objects degraded (6.166%), 3 pgs degraded (PG_DEGRADED) 2024-09-10T18:27:01.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:01 smithi204 ceph-mon[32357]: 6.3 scrub starts 2024-09-10T18:27:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:01 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 6315/102411 objects degraded (6.166%), 3 pgs degraded (PG_DEGRADED) 2024-09-10T18:27:01.373 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:01.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:01.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:01.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:01.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:27:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:27:01] "GET /metrics HTTP/1.1" 200 224064 "" "Prometheus/2.43.0" 2024-09-10T18:27:01.519 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:01.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:01.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:01.621 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:01.648 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:01.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:01.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:01.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-10T18:27:01.749 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:01.773 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:01.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:01.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:01.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:01.963 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:01.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:01.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:02.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:02.069 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:27:02.069 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:27:02.108 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:27:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:27:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:02.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:02.157 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:27:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:27:02.200 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:27:02.201 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:27:02.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:02.235 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:27:02.235 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:27:02.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:02 smithi028 ceph-mon[29291]: pgmap v231: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 1.8 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1002 B/s rd, 0 B/s wr, 1.05k op/s; 4780/86631 objects degraded (5.518%); 4.5 MiB/s, 72 objects/s recovering 2024-09-10T18:27:02.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:02 smithi028 ceph-mon[29291]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-09-10T18:27:02.236 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:02 smithi028 ceph-mon[39220]: pgmap v231: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 1.8 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1002 B/s rd, 0 B/s wr, 1.05k op/s; 4780/86631 objects degraded (5.518%); 4.5 MiB/s, 72 objects/s recovering 2024-09-10T18:27:02.236 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:02 smithi028 ceph-mon[39220]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-09-10T18:27:02.267 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:27:02.267 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:27:02.274 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:02.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:02.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:02.301 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:27:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:27:02.338 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:27:02.338 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:27:02.343 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:02 smithi204 ceph-mon[32357]: pgmap v231: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 1.8 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1002 B/s rd, 0 B/s wr, 1.05k op/s; 4780/86631 objects degraded (5.518%); 4.5 MiB/s, 72 objects/s recovering 2024-09-10T18:27:02.343 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:02 smithi204 ceph-mon[32357]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-09-10T18:27:02.343 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:27:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:27:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:27:02.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:02.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:02.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:02.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:02.572 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:02.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:02.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:02.674 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:02.701 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:02.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:02.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:02.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:02.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:02.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:02.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:02.955 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:02.979 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:02.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:03.081 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:03.106 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:03.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:03.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:03.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:03.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:03.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:03.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:03 smithi204 ceph-mon[32357]: 5.18 scrub starts 2024-09-10T18:27:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:03 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3404118566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:27:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:03 smithi204 ceph-mon[32357]: 6.5 scrub starts 2024-09-10T18:27:03.388 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:03.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:03.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:03.490 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:03.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:03 smithi028 ceph-mon[39220]: 5.18 scrub starts 2024-09-10T18:27:03.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:03 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3404118566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:27:03.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:03 smithi028 ceph-mon[39220]: 6.5 scrub starts 2024-09-10T18:27:03.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:03 smithi028 ceph-mon[29291]: 5.18 scrub starts 2024-09-10T18:27:03.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:03 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3404118566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:27:03.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:03 smithi028 ceph-mon[29291]: 6.5 scrub starts 2024-09-10T18:27:03.517 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:03.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:03.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:03.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:03.658 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:03.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:03.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:03.760 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:03.785 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:03.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:03.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:03.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:03.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:03.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:04.028 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:04.054 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:04.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:04.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:27:04.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:04.185 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:04.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:04.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:04.334 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:04.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:04.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:04.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:27:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:27:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:04 smithi204 ceph-mon[32357]: pgmap v232: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 1.8 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 873 op/s; 4780/86631 objects degraded (5.518%); 2.9 MiB/s, 46 objects/s recovering 2024-09-10T18:27:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:04 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:27:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:04 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:27:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:04 smithi204 ceph-mon[32357]: 3.14 scrub starts 2024-09-10T18:27:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:04 smithi204 ceph-mon[32357]: osdmap e158: 8 total, 8 up, 8 in 2024-09-10T18:27:04.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:04 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1844858715' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:27:04.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:04 smithi028 ceph-mon[39220]: pgmap v232: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 1.8 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 873 op/s; 4780/86631 objects degraded (5.518%); 2.9 MiB/s, 46 objects/s recovering 2024-09-10T18:27:04.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:04 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:27:04.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:04 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:27:04.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:04 smithi028 ceph-mon[39220]: 3.14 scrub starts 2024-09-10T18:27:04.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:04 smithi028 ceph-mon[39220]: osdmap e158: 8 total, 8 up, 8 in 2024-09-10T18:27:04.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:04 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1844858715' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:27:04.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:04 smithi028 ceph-mon[29291]: pgmap v232: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 1.8 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 873 op/s; 4780/86631 objects degraded (5.518%); 2.9 MiB/s, 46 objects/s recovering 2024-09-10T18:27:04.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:04 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:27:04.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:04 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:27:04.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:04 smithi028 ceph-mon[29291]: 3.14 scrub starts 2024-09-10T18:27:04.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:04 smithi028 ceph-mon[29291]: osdmap e158: 8 total, 8 up, 8 in 2024-09-10T18:27:04.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:04 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1844858715' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:27:04.435 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:04.462 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:04.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:04.564 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:04.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:04.591 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:04.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:04.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:04.734 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:04.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:04.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:04.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:04.880 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:04.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:04.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:04.981 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:05.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:05.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:05.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:05.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:05.131 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:05.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:05.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:05.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:05.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:05.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:05.290 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:05 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:27:05.290 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:05 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1844858715' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:27:05.290 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:05 smithi028 ceph-mon[39220]: osdmap e159: 8 total, 8 up, 8 in 2024-09-10T18:27:05.290 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:05 smithi028 ceph-mon[39220]: 5.10 scrub starts 2024-09-10T18:27:05.290 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:05 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:05.290 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:05 smithi028 ceph-mon[39220]: 3.1d scrub starts 2024-09-10T18:27:05.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:05 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:27:05.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:05 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1844858715' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:27:05.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:05 smithi028 ceph-mon[29291]: osdmap e159: 8 total, 8 up, 8 in 2024-09-10T18:27:05.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:05 smithi028 ceph-mon[29291]: 5.10 scrub starts 2024-09-10T18:27:05.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:05 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:05.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:05 smithi028 ceph-mon[29291]: 3.1d scrub starts 2024-09-10T18:27:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:05 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:27:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:05 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1844858715' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:27:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:05 smithi204 ceph-mon[32357]: osdmap e159: 8 total, 8 up, 8 in 2024-09-10T18:27:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:05 smithi204 ceph-mon[32357]: 5.10 scrub starts 2024-09-10T18:27:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:05 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:05 smithi204 ceph-mon[32357]: 3.1d scrub starts 2024-09-10T18:27:05.376 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:05.403 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:05.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:05.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:05.505 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:05.531 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:05.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:05.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:05.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:05.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:27:05.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:05.685 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:05.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:05.713 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:05.786 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:05.810 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:05.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:05.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:05.912 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:05.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:05.937 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:05.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:06.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:06.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:06.104 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:06.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:06.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:06 smithi028 ceph-mon[39220]: pgmap v235: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 781 op/s; 4780/83679 objects degraded (5.712%); 4.4 MiB/s, 69 objects/s recovering 2024-09-10T18:27:06.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:06 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 4780/83679 objects degraded (5.712%), 3 pgs degraded (PG_DEGRADED) 2024-09-10T18:27:06.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:06 smithi028 ceph-mon[29291]: pgmap v235: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 781 op/s; 4780/83679 objects degraded (5.712%); 4.4 MiB/s, 69 objects/s recovering 2024-09-10T18:27:06.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:06 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 4780/83679 objects degraded (5.712%), 3 pgs degraded (PG_DEGRADED) 2024-09-10T18:27:06.205 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:06.231 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:06.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:06.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:06.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:06 smithi204 ceph-mon[32357]: pgmap v235: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 781 op/s; 4780/83679 objects degraded (5.712%); 4.4 MiB/s, 69 objects/s recovering 2024-09-10T18:27:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:06 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 4780/83679 objects degraded (5.712%), 3 pgs degraded (PG_DEGRADED) 2024-09-10T18:27:06.378 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:06.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:06.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:06.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:06.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:06.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:06.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:06.619 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:06.647 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:06.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:06.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:06.749 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:06.776 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:06.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:06.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:06.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:06.902 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:06.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:06.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:07.004 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:07.029 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:07.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:07.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.1 to 1 (10000) 2024-09-10T18:27:07.131 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:07.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:27:07.156 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:07.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:07.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:07.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:07.280 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:07.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:07.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:07 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:27:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:07 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:27:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:07 smithi204 ceph-mon[32357]: osdmap e160: 8 total, 8 up, 8 in 2024-09-10T18:27:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:07 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/26263751' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:27:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:07 smithi204 ceph-mon[32357]: from='client.24943 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:27:07.382 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:07.406 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:07.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:07.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:07 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:27:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:07 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:27:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:07 smithi028 ceph-mon[29291]: osdmap e160: 8 total, 8 up, 8 in 2024-09-10T18:27:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:07 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/26263751' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:27:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:07 smithi028 ceph-mon[29291]: from='client.24943 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:27:07.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:07 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:27:07.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:07 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:27:07.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:07 smithi028 ceph-mon[39220]: osdmap e160: 8 total, 8 up, 8 in 2024-09-10T18:27:07.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:07 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/26263751' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:27:07.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:07 smithi028 ceph-mon[39220]: from='client.24943 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:27:07.508 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:07.530 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:07.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:07.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:07.632 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:07.663 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:07.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:07.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:07.765 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:07.791 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:07.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:07.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:07.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:07.928 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:07.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:07.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:08.030 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:08.055 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:08.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:08.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:08.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:08.182 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:08.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:08.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:08.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:08.316 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:08.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:08.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:08 smithi204 ceph-mon[32357]: pgmap v237: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 836 op/s; 1805/71580 objects degraded (2.522%); 24 MiB/s, 391 objects/s recovering 2024-09-10T18:27:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:08 smithi204 ceph-mon[32357]: 6.1e scrub starts 2024-09-10T18:27:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:08 smithi204 ceph-mon[32357]: from='client.24943 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T18:27:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:08 smithi204 ceph-mon[32357]: osdmap e161: 8 total, 8 up, 8 in 2024-09-10T18:27:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:08 smithi204 ceph-mon[32357]: 7.0 scrub starts 2024-09-10T18:27:08.418 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:08.443 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:08.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:08.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:08 smithi028 ceph-mon[29291]: pgmap v237: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 836 op/s; 1805/71580 objects degraded (2.522%); 24 MiB/s, 391 objects/s recovering 2024-09-10T18:27:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:08 smithi028 ceph-mon[29291]: 6.1e scrub starts 2024-09-10T18:27:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:08 smithi028 ceph-mon[29291]: from='client.24943 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T18:27:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:08 smithi028 ceph-mon[29291]: osdmap e161: 8 total, 8 up, 8 in 2024-09-10T18:27:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:08 smithi028 ceph-mon[29291]: 7.0 scrub starts 2024-09-10T18:27:08.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:08 smithi028 ceph-mon[39220]: pgmap v237: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 836 op/s; 1805/71580 objects degraded (2.522%); 24 MiB/s, 391 objects/s recovering 2024-09-10T18:27:08.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:08 smithi028 ceph-mon[39220]: 6.1e scrub starts 2024-09-10T18:27:08.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:08 smithi028 ceph-mon[39220]: from='client.24943 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T18:27:08.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:08 smithi028 ceph-mon[39220]: osdmap e161: 8 total, 8 up, 8 in 2024-09-10T18:27:08.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:08 smithi028 ceph-mon[39220]: 7.0 scrub starts 2024-09-10T18:27:08.545 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:08.566 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:08.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:08.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:08.668 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:08.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:08.691 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:08.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:08.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:08.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:08.815 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:08.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:08.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:08.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:08.939 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:08.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:09.041 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:09.063 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:09.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:09.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:09.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:09.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:09.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:09.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:09.326 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:09.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:09 smithi204 ceph-mon[32357]: 4.e scrub starts 2024-09-10T18:27:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:09 smithi204 ceph-mon[32357]: osdmap e162: 8 total, 8 up, 8 in 2024-09-10T18:27:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:27:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:27:09.428 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:09.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:09.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:09.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:09 smithi028 ceph-mon[29291]: 4.e scrub starts 2024-09-10T18:27:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:09 smithi028 ceph-mon[29291]: osdmap e162: 8 total, 8 up, 8 in 2024-09-10T18:27:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:27:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:27:09.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:09 smithi028 ceph-mon[39220]: 4.e scrub starts 2024-09-10T18:27:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:09 smithi028 ceph-mon[39220]: osdmap e162: 8 total, 8 up, 8 in 2024-09-10T18:27:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:27:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:27:09.551 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:09.572 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:09.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:09.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:09.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:09.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:09.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:09.801 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:09.824 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:09.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:09.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:09.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-10T18:27:09.926 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:09.949 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:09.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:09.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:10.050 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:10.072 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:10.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:10.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:10.210 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:10.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:10.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:10.312 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:10.336 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:10.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:10.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:10 smithi204 ceph-mon[32357]: pgmap v240: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s; 1805/71580 objects degraded (2.522%); 31 MiB/s, 490 objects/s recovering 2024-09-10T18:27:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:10 smithi204 ceph-mon[32357]: 4.13 scrub starts 2024-09-10T18:27:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:10 smithi204 ceph-mon[32357]: 4.13 scrub ok 2024-09-10T18:27:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:10 smithi204 ceph-mon[32357]: osdmap e163: 8 total, 8 up, 8 in 2024-09-10T18:27:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:10 smithi204 ceph-mon[32357]: 4.1e scrub starts 2024-09-10T18:27:10.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:10 smithi204 ceph-mon[32357]: 4.1e scrub ok 2024-09-10T18:27:10.391 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:10 smithi028 ceph-mon[39220]: pgmap v240: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s; 1805/71580 objects degraded (2.522%); 31 MiB/s, 490 objects/s recovering 2024-09-10T18:27:10.391 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:10 smithi028 ceph-mon[39220]: 4.13 scrub starts 2024-09-10T18:27:10.391 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:10 smithi028 ceph-mon[39220]: 4.13 scrub ok 2024-09-10T18:27:10.391 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:10 smithi028 ceph-mon[39220]: osdmap e163: 8 total, 8 up, 8 in 2024-09-10T18:27:10.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:10 smithi028 ceph-mon[39220]: 4.1e scrub starts 2024-09-10T18:27:10.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:10 smithi028 ceph-mon[39220]: 4.1e scrub ok 2024-09-10T18:27:10.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:10 smithi028 ceph-mon[29291]: pgmap v240: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 155 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s; 1805/71580 objects degraded (2.522%); 31 MiB/s, 490 objects/s recovering 2024-09-10T18:27:10.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:10 smithi028 ceph-mon[29291]: 4.13 scrub starts 2024-09-10T18:27:10.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:10 smithi028 ceph-mon[29291]: 4.13 scrub ok 2024-09-10T18:27:10.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:10 smithi028 ceph-mon[29291]: osdmap e163: 8 total, 8 up, 8 in 2024-09-10T18:27:10.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:10 smithi028 ceph-mon[29291]: 4.1e scrub starts 2024-09-10T18:27:10.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:10 smithi028 ceph-mon[29291]: 4.1e scrub ok 2024-09-10T18:27:10.439 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:10.465 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:10.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:10.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:10.544 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:27:10.544 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-10T18:27:10.544 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-10T18:27:10.544 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-10T18:27:10.544 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:27:10.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:10.593 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:10.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:10.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:10.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:10.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:10.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:10.876 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:10.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:10.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:11.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:11.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:11.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:11.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:11.132 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:11.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:11.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:11.234 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:11 smithi028 ceph-mon[39220]: osdmap e164: 8 total, 8 up, 8 in 2024-09-10T18:27:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:11 smithi028 ceph-mon[39220]: 6.1b scrub starts 2024-09-10T18:27:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:11 smithi028 ceph-mon[39220]: 6.1b scrub ok 2024-09-10T18:27:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:11 smithi028 ceph-mon[39220]: Health check update: Degraded data redundancy: 1805/71580 objects degraded (2.522%), 3 pgs degraded (PG_DEGRADED) 2024-09-10T18:27:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:11 smithi028 ceph-mon[39220]: osdmap e165: 8 total, 8 up, 8 in 2024-09-10T18:27:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:11 smithi028 ceph-mon[39220]: 5.9 scrub starts 2024-09-10T18:27:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:11 smithi028 ceph-mon[39220]: 5.9 scrub ok 2024-09-10T18:27:11.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:11 smithi028 ceph-mon[29291]: osdmap e164: 8 total, 8 up, 8 in 2024-09-10T18:27:11.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:11 smithi028 ceph-mon[29291]: 6.1b scrub starts 2024-09-10T18:27:11.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:11 smithi028 ceph-mon[29291]: 6.1b scrub ok 2024-09-10T18:27:11.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:11 smithi028 ceph-mon[29291]: Health check update: Degraded data redundancy: 1805/71580 objects degraded (2.522%), 3 pgs degraded (PG_DEGRADED) 2024-09-10T18:27:11.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:11 smithi028 ceph-mon[29291]: osdmap e165: 8 total, 8 up, 8 in 2024-09-10T18:27:11.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:11 smithi028 ceph-mon[29291]: 5.9 scrub starts 2024-09-10T18:27:11.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:11 smithi028 ceph-mon[29291]: 5.9 scrub ok 2024-09-10T18:27:11.261 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:11.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:11.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:11.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:11.391 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:11.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:11.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:11.401 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-10T18:27:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:11.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:11.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:11.571 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:11.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:11.603 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:27:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:27:11] "GET /metrics HTTP/1.1" 200 224104 "" "Prometheus/2.43.0" 2024-09-10T18:27:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:11 smithi204 ceph-mon[32357]: osdmap e164: 8 total, 8 up, 8 in 2024-09-10T18:27:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:11 smithi204 ceph-mon[32357]: 6.1b scrub starts 2024-09-10T18:27:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:11 smithi204 ceph-mon[32357]: 6.1b scrub ok 2024-09-10T18:27:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:11 smithi204 ceph-mon[32357]: Health check update: Degraded data redundancy: 1805/71580 objects degraded (2.522%), 3 pgs degraded (PG_DEGRADED) 2024-09-10T18:27:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:11 smithi204 ceph-mon[32357]: osdmap e165: 8 total, 8 up, 8 in 2024-09-10T18:27:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:11 smithi204 ceph-mon[32357]: 5.9 scrub starts 2024-09-10T18:27:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:11 smithi204 ceph-mon[32357]: 5.9 scrub ok 2024-09-10T18:27:11.673 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:11.699 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:11.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:11.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:11.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:11.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:11.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:11.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:12.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:12.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:12.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:12.132 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:12.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:12.233 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[39220]: 7.b scrub starts 2024-09-10T18:27:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[39220]: pgmap v244: 158 pgs: 158 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 915 op/s; 27 MiB/s, 438 objects/s recovering 2024-09-10T18:27:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[39220]: 7.b scrub ok 2024-09-10T18:27:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[39220]: 4.1c scrub starts 2024-09-10T18:27:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[39220]: 4.1c scrub ok 2024-09-10T18:27:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[39220]: 5.1e scrub starts 2024-09-10T18:27:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[39220]: 5.1e scrub ok 2024-09-10T18:27:12.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[39220]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1805/71580 objects degraded (2.522%), 3 pgs degraded) 2024-09-10T18:27:12.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:27:12.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[39220]: 3.13 scrub starts 2024-09-10T18:27:12.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[39220]: 3.13 scrub ok 2024-09-10T18:27:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[29291]: 7.b scrub starts 2024-09-10T18:27:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[29291]: pgmap v244: 158 pgs: 158 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 915 op/s; 27 MiB/s, 438 objects/s recovering 2024-09-10T18:27:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[29291]: 7.b scrub ok 2024-09-10T18:27:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[29291]: 4.1c scrub starts 2024-09-10T18:27:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[29291]: 4.1c scrub ok 2024-09-10T18:27:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[29291]: 5.1e scrub starts 2024-09-10T18:27:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[29291]: 5.1e scrub ok 2024-09-10T18:27:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[29291]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1805/71580 objects degraded (2.522%), 3 pgs degraded) 2024-09-10T18:27:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:27:12.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[29291]: 3.13 scrub starts 2024-09-10T18:27:12.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:12 smithi028 ceph-mon[29291]: 3.13 scrub ok 2024-09-10T18:27:12.260 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:12.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:12.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:12.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:12.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:12.420 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:12.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:12.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:12.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:12.561 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:12.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:12.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:27:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:27:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:27:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:12 smithi204 ceph-mon[32357]: 7.b scrub starts 2024-09-10T18:27:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:12 smithi204 ceph-mon[32357]: pgmap v244: 158 pgs: 158 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 915 op/s; 27 MiB/s, 438 objects/s recovering 2024-09-10T18:27:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:12 smithi204 ceph-mon[32357]: 7.b scrub ok 2024-09-10T18:27:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:12 smithi204 ceph-mon[32357]: 4.1c scrub starts 2024-09-10T18:27:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:12 smithi204 ceph-mon[32357]: 4.1c scrub ok 2024-09-10T18:27:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:12 smithi204 ceph-mon[32357]: 5.1e scrub starts 2024-09-10T18:27:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:12 smithi204 ceph-mon[32357]: 5.1e scrub ok 2024-09-10T18:27:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:12 smithi204 ceph-mon[32357]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1805/71580 objects degraded (2.522%), 3 pgs degraded) 2024-09-10T18:27:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:12 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:27:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:12 smithi204 ceph-mon[32357]: 3.13 scrub starts 2024-09-10T18:27:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:12 smithi204 ceph-mon[32357]: 3.13 scrub ok 2024-09-10T18:27:12.662 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:12.685 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:12.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:12.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:12.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:12.830 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:12.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:12.931 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:12.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:12.958 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:12.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:13.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:13.103 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:13.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:13.205 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:13.228 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:13.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:13.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:13.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:13.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:13.366 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:13.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:13.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:13.496 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:13.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:13.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:13 smithi028 ceph-mon[29291]: 4.18 scrub starts 2024-09-10T18:27:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:13 smithi028 ceph-mon[29291]: 4.18 scrub ok 2024-09-10T18:27:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:13 smithi028 ceph-mon[29291]: 5.5 scrub starts 2024-09-10T18:27:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:13 smithi028 ceph-mon[29291]: 5.5 scrub ok 2024-09-10T18:27:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:13 smithi028 ceph-mon[39220]: 4.18 scrub starts 2024-09-10T18:27:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:13 smithi028 ceph-mon[39220]: 4.18 scrub ok 2024-09-10T18:27:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:13 smithi028 ceph-mon[39220]: 5.5 scrub starts 2024-09-10T18:27:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:13 smithi028 ceph-mon[39220]: 5.5 scrub ok 2024-09-10T18:27:13.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:13 smithi204 ceph-mon[32357]: 4.18 scrub starts 2024-09-10T18:27:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:13 smithi204 ceph-mon[32357]: 4.18 scrub ok 2024-09-10T18:27:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:13 smithi204 ceph-mon[32357]: 5.5 scrub starts 2024-09-10T18:27:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:13 smithi204 ceph-mon[32357]: 5.5 scrub ok 2024-09-10T18:27:13.635 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:13.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:13.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:13.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:13.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:13.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:13.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:13.877 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:13.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:13.906 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:13.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:14.007 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:14.033 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:14.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:14.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:14.135 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:14.161 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:14.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:14.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:14.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:14.304 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:14.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:14.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:14.432 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:14.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:14.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:14 smithi028 ceph-mon[29291]: 4.a scrub starts 2024-09-10T18:27:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:14 smithi028 ceph-mon[29291]: 4.a scrub ok 2024-09-10T18:27:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:14 smithi028 ceph-mon[29291]: 5.c scrub starts 2024-09-10T18:27:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:14 smithi028 ceph-mon[29291]: 5.c scrub ok 2024-09-10T18:27:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:14 smithi028 ceph-mon[29291]: pgmap v245: 158 pgs: 158 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 726 op/s; 22 MiB/s, 348 objects/s recovering 2024-09-10T18:27:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:14 smithi028 ceph-mon[29291]: 4.c scrub starts 2024-09-10T18:27:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:14 smithi028 ceph-mon[29291]: 4.c scrub ok 2024-09-10T18:27:14.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:14 smithi028 ceph-mon[39220]: 4.a scrub starts 2024-09-10T18:27:14.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:14 smithi028 ceph-mon[39220]: 4.a scrub ok 2024-09-10T18:27:14.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:14 smithi028 ceph-mon[39220]: 5.c scrub starts 2024-09-10T18:27:14.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:14 smithi028 ceph-mon[39220]: 5.c scrub ok 2024-09-10T18:27:14.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:14 smithi028 ceph-mon[39220]: pgmap v245: 158 pgs: 158 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 726 op/s; 22 MiB/s, 348 objects/s recovering 2024-09-10T18:27:14.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:14 smithi028 ceph-mon[39220]: 4.c scrub starts 2024-09-10T18:27:14.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:14 smithi028 ceph-mon[39220]: 4.c scrub ok 2024-09-10T18:27:14.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:14.570 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:14.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:14.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:14.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:27:14 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:27:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:14 smithi204 ceph-mon[32357]: 4.a scrub starts 2024-09-10T18:27:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:14 smithi204 ceph-mon[32357]: 4.a scrub ok 2024-09-10T18:27:14.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:14 smithi204 ceph-mon[32357]: 5.c scrub starts 2024-09-10T18:27:14.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:14 smithi204 ceph-mon[32357]: 5.c scrub ok 2024-09-10T18:27:14.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:14 smithi204 ceph-mon[32357]: pgmap v245: 158 pgs: 158 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 726 op/s; 22 MiB/s, 348 objects/s recovering 2024-09-10T18:27:14.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:14 smithi204 ceph-mon[32357]: 4.c scrub starts 2024-09-10T18:27:14.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:14 smithi204 ceph-mon[32357]: 4.c scrub ok 2024-09-10T18:27:14.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:14.715 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:14.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:14.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:14.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:14.855 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:14.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:14.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:14.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:14.981 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:14.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:14.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:15.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:15.139 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:15.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:15.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:15.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:15 smithi028 ceph-mon[29291]: 6.1d scrub starts 2024-09-10T18:27:15.203 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:15 smithi028 ceph-mon[29291]: 6.1d scrub ok 2024-09-10T18:27:15.203 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:15 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:15.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:15 smithi028 ceph-mon[39220]: 6.1d scrub starts 2024-09-10T18:27:15.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:15 smithi028 ceph-mon[39220]: 6.1d scrub ok 2024-09-10T18:27:15.203 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:15 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:15.241 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:15.269 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:15.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:15.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:15.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:15.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:15.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:15.536 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:15.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:15.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:15 smithi204 ceph-mon[32357]: 6.1d scrub starts 2024-09-10T18:27:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:15 smithi204 ceph-mon[32357]: 6.1d scrub ok 2024-09-10T18:27:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:15 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:15.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:15.680 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:15.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:15.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:15.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:15.826 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:15.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:15.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:15.928 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:15.953 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:15.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:16.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:16.091 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:16.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:16.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:16.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:16.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:16.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:16.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:16.377 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:16.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:16.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:16.479 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:16.502 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:16.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:16.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:16.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:16 smithi028 ceph-mon[29291]: 3.1e deep-scrub starts 2024-09-10T18:27:16.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:16 smithi028 ceph-mon[29291]: 3.1e deep-scrub ok 2024-09-10T18:27:16.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:16 smithi028 ceph-mon[29291]: 4.2 scrub starts 2024-09-10T18:27:16.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:16 smithi028 ceph-mon[29291]: 4.2 scrub ok 2024-09-10T18:27:16.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:16 smithi028 ceph-mon[29291]: pgmap v246: 158 pgs: 158 active+clean; 1.2 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 728 op/s; 17 MiB/s, 275 objects/s recovering 2024-09-10T18:27:16.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:16 smithi028 ceph-mon[29291]: 6.1f scrub starts 2024-09-10T18:27:16.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:16 smithi028 ceph-mon[29291]: 6.1f scrub ok 2024-09-10T18:27:16.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:16 smithi028 ceph-mon[39220]: 3.1e deep-scrub starts 2024-09-10T18:27:16.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:16 smithi028 ceph-mon[39220]: 3.1e deep-scrub ok 2024-09-10T18:27:16.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:16 smithi028 ceph-mon[39220]: 4.2 scrub starts 2024-09-10T18:27:16.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:16 smithi028 ceph-mon[39220]: 4.2 scrub ok 2024-09-10T18:27:16.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:16 smithi028 ceph-mon[39220]: pgmap v246: 158 pgs: 158 active+clean; 1.2 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 728 op/s; 17 MiB/s, 275 objects/s recovering 2024-09-10T18:27:16.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:16 smithi028 ceph-mon[39220]: 6.1f scrub starts 2024-09-10T18:27:16.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:16 smithi028 ceph-mon[39220]: 6.1f scrub ok 2024-09-10T18:27:16.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:16 smithi204 ceph-mon[32357]: 3.1e deep-scrub starts 2024-09-10T18:27:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:16 smithi204 ceph-mon[32357]: 3.1e deep-scrub ok 2024-09-10T18:27:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:16 smithi204 ceph-mon[32357]: 4.2 scrub starts 2024-09-10T18:27:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:16 smithi204 ceph-mon[32357]: 4.2 scrub ok 2024-09-10T18:27:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:16 smithi204 ceph-mon[32357]: pgmap v246: 158 pgs: 158 active+clean; 1.2 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 728 op/s; 17 MiB/s, 275 objects/s recovering 2024-09-10T18:27:16.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:16 smithi204 ceph-mon[32357]: 6.1f scrub starts 2024-09-10T18:27:16.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:16 smithi204 ceph-mon[32357]: 6.1f scrub ok 2024-09-10T18:27:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:16.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:16.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:16.747 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:16.774 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:16.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:16.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:16.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:16.920 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:16.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:16.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:17.022 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:17.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:17.049 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:17.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:17.150 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:17.176 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:17.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:17.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:17.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:17.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:17.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:17.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:17 smithi028 ceph-mon[39220]: 4.1f scrub starts 2024-09-10T18:27:17.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:17 smithi028 ceph-mon[39220]: 4.1f scrub ok 2024-09-10T18:27:17.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:17 smithi028 ceph-mon[39220]: 3.1 scrub starts 2024-09-10T18:27:17.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:17 smithi028 ceph-mon[39220]: 3.1 scrub ok 2024-09-10T18:27:17.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:17 smithi028 ceph-mon[29291]: 4.1f scrub starts 2024-09-10T18:27:17.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:17 smithi028 ceph-mon[29291]: 4.1f scrub ok 2024-09-10T18:27:17.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:17 smithi028 ceph-mon[29291]: 3.1 scrub starts 2024-09-10T18:27:17.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:17 smithi028 ceph-mon[29291]: 3.1 scrub ok 2024-09-10T18:27:17.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:17.457 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:27:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:27:17.460 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:17.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:17.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:17.493 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:27:17.494 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:27:17.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:27:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:27:17.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:17.563 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:27:17.563 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:27:17.598 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:27:17.598 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:27:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:17 smithi204 ceph-mon[32357]: 4.1f scrub starts 2024-09-10T18:27:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:17 smithi204 ceph-mon[32357]: 4.1f scrub ok 2024-09-10T18:27:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:17 smithi204 ceph-mon[32357]: 3.1 scrub starts 2024-09-10T18:27:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:17 smithi204 ceph-mon[32357]: 3.1 scrub ok 2024-09-10T18:27:17.606 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:17.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:17.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:17.629 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:27:17.629 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:27:17.661 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:27:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:27:17.691 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:27:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:27:17.708 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:17.733 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:17.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:17.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:17.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:17.877 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:17.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:17.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:18.017 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:18.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:18.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:18.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:18.166 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:18.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:18.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:18.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:18.293 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:18.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:18.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:18.344 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:18 smithi028 ceph-mon[39220]: pgmap v247: 158 pgs: 158 active+clean; 945 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.10k op/s; 13 MiB/s, 213 objects/s recovering 2024-09-10T18:27:18.345 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:18 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2509273252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:27:18.345 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:18 smithi028 ceph-mon[39220]: 4.15 deep-scrub starts 2024-09-10T18:27:18.345 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:18 smithi028 ceph-mon[39220]: 4.15 deep-scrub ok 2024-09-10T18:27:18.345 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:18 smithi028 ceph-mon[39220]: 6.1c scrub starts 2024-09-10T18:27:18.345 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:18 smithi028 ceph-mon[39220]: 6.1c scrub ok 2024-09-10T18:27:18.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:18 smithi028 ceph-mon[29291]: pgmap v247: 158 pgs: 158 active+clean; 945 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.10k op/s; 13 MiB/s, 213 objects/s recovering 2024-09-10T18:27:18.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:18 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2509273252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:27:18.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:18 smithi028 ceph-mon[29291]: 4.15 deep-scrub starts 2024-09-10T18:27:18.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:18 smithi028 ceph-mon[29291]: 4.15 deep-scrub ok 2024-09-10T18:27:18.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:18 smithi028 ceph-mon[29291]: 6.1c scrub starts 2024-09-10T18:27:18.346 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:18 smithi028 ceph-mon[29291]: 6.1c scrub ok 2024-09-10T18:27:18.394 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:18.416 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:18.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:18.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:18.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:18.559 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:18.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:18 smithi204 ceph-mon[32357]: pgmap v247: 158 pgs: 158 active+clean; 945 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.10k op/s; 13 MiB/s, 213 objects/s recovering 2024-09-10T18:27:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:18 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2509273252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:27:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:18 smithi204 ceph-mon[32357]: 4.15 deep-scrub starts 2024-09-10T18:27:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:18 smithi204 ceph-mon[32357]: 4.15 deep-scrub ok 2024-09-10T18:27:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:18 smithi204 ceph-mon[32357]: 6.1c scrub starts 2024-09-10T18:27:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:18 smithi204 ceph-mon[32357]: 6.1c scrub ok 2024-09-10T18:27:18.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:18.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:18.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:18.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:18.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:18.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:18.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:18.972 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:18.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:18.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:19.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:19.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:19.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:19.167 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:27:19.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:19.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:19.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:19.357 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:19.380 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:19.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:19.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:19.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[39220]: 3.a deep-scrub starts 2024-09-10T18:27:19.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[39220]: 3.a deep-scrub ok 2024-09-10T18:27:19.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[39220]: 6.10 scrub starts 2024-09-10T18:27:19.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[39220]: 6.10 scrub ok 2024-09-10T18:27:19.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[39220]: 2.1 scrub starts 2024-09-10T18:27:19.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[39220]: 2.1 scrub ok 2024-09-10T18:27:19.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/862525119' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:27:19.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[39220]: from='client.34925 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:27:19.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[39220]: 6.12 scrub starts 2024-09-10T18:27:19.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[39220]: 6.12 scrub ok 2024-09-10T18:27:19.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[29291]: 3.a deep-scrub starts 2024-09-10T18:27:19.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[29291]: 3.a deep-scrub ok 2024-09-10T18:27:19.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[29291]: 6.10 scrub starts 2024-09-10T18:27:19.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[29291]: 6.10 scrub ok 2024-09-10T18:27:19.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[29291]: 2.1 scrub starts 2024-09-10T18:27:19.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[29291]: 2.1 scrub ok 2024-09-10T18:27:19.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/862525119' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:27:19.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[29291]: from='client.34925 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:27:19.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[29291]: 6.12 scrub starts 2024-09-10T18:27:19.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:19 smithi028 ceph-mon[29291]: 6.12 scrub ok 2024-09-10T18:27:19.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:19.525 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:19.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:19 smithi204 ceph-mon[32357]: 3.a deep-scrub starts 2024-09-10T18:27:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:19 smithi204 ceph-mon[32357]: 3.a deep-scrub ok 2024-09-10T18:27:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:19 smithi204 ceph-mon[32357]: 6.10 scrub starts 2024-09-10T18:27:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:19 smithi204 ceph-mon[32357]: 6.10 scrub ok 2024-09-10T18:27:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:19 smithi204 ceph-mon[32357]: 2.1 scrub starts 2024-09-10T18:27:19.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:19 smithi204 ceph-mon[32357]: 2.1 scrub ok 2024-09-10T18:27:19.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:19 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/862525119' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:27:19.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:19 smithi204 ceph-mon[32357]: from='client.34925 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:27:19.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:19 smithi204 ceph-mon[32357]: 6.12 scrub starts 2024-09-10T18:27:19.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:19 smithi204 ceph-mon[32357]: 6.12 scrub ok 2024-09-10T18:27:19.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:19.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:19.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:19.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:19.803 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:19.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:19.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:19.905 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:19.932 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:19.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:19.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:20.034 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:20.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:20.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:20.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:20.160 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:20.186 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:20.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:20.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:20.260 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:20.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:20.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:20 smithi028 ceph-mon[29291]: 4.3 scrub starts 2024-09-10T18:27:20.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:20 smithi028 ceph-mon[29291]: 4.3 scrub ok 2024-09-10T18:27:20.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:20 smithi028 ceph-mon[29291]: pgmap v248: 158 pgs: 158 active+clean; 945 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 585 B/s rd, 0 B/s wr, 828 op/s; 8.3 MiB/s, 133 objects/s recovering 2024-09-10T18:27:20.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:20 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:27:20.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:20 smithi028 ceph-mon[29291]: from='client.34925 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:27:20.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:20 smithi028 ceph-mon[29291]: osdmap e166: 8 total, 8 up, 8 in 2024-09-10T18:27:20.308 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:20 smithi028 ceph-mon[39220]: 4.3 scrub starts 2024-09-10T18:27:20.308 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:20 smithi028 ceph-mon[39220]: 4.3 scrub ok 2024-09-10T18:27:20.308 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:20 smithi028 ceph-mon[39220]: pgmap v248: 158 pgs: 158 active+clean; 945 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 585 B/s rd, 0 B/s wr, 828 op/s; 8.3 MiB/s, 133 objects/s recovering 2024-09-10T18:27:20.308 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:20 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:27:20.308 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:20 smithi028 ceph-mon[39220]: from='client.34925 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:27:20.308 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:20 smithi028 ceph-mon[39220]: osdmap e166: 8 total, 8 up, 8 in 2024-09-10T18:27:20.314 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:20.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:20.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:20.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:20.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:20.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:20.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:20.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:20.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:20.580 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:20.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:20 smithi204 ceph-mon[32357]: 4.3 scrub starts 2024-09-10T18:27:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:20 smithi204 ceph-mon[32357]: 4.3 scrub ok 2024-09-10T18:27:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:20 smithi204 ceph-mon[32357]: pgmap v248: 158 pgs: 158 active+clean; 945 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 585 B/s rd, 0 B/s wr, 828 op/s; 8.3 MiB/s, 133 objects/s recovering 2024-09-10T18:27:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:20 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:27:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:20 smithi204 ceph-mon[32357]: from='client.34925 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:27:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:20 smithi204 ceph-mon[32357]: osdmap e166: 8 total, 8 up, 8 in 2024-09-10T18:27:20.680 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:20.705 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:20.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:20.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:20.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:20.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:20.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:20.953 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:20.978 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:20.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:20.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:21.079 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:21.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:21.105 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:21.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:21.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:21.231 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:21.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:21.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:21.372 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:21.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:21.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:21.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:21.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:21.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:21.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:21.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:21.640 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:27:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:27:21] "GET /metrics HTTP/1.1" 200 224104 "" "Prometheus/2.43.0" 2024-09-10T18:27:21.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:21.643 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:21.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:21.743 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:21.772 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:21.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:21.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:21.877 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:21.904 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:21.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:21.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:22.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:21 smithi028 ceph-mon[29291]: osdmap e167: 8 total, 8 up, 8 in 2024-09-10T18:27:22.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:21 smithi028 ceph-mon[29291]: pgmap v251: 158 pgs: 158 active+clean; 622 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T18:27:22.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:21 smithi028 ceph-mon[39220]: osdmap e167: 8 total, 8 up, 8 in 2024-09-10T18:27:22.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:21 smithi028 ceph-mon[39220]: pgmap v251: 158 pgs: 158 active+clean; 622 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T18:27:22.006 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:22.032 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:22.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:22.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:22.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:27:22.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:21 smithi204 ceph-mon[32357]: osdmap e167: 8 total, 8 up, 8 in 2024-09-10T18:27:22.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:21 smithi204 ceph-mon[32357]: pgmap v251: 158 pgs: 158 active+clean; 622 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T18:27:22.134 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:22.161 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:22.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:22.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:22.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:22.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:22.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:22.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:27:22.406 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:22.430 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:22.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:22.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:22.438 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:27:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:27:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:27:22.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:22.578 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:22.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:22.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:22.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:22.707 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:22.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:22.810 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:22.835 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:22.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:22.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:22.981 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:22.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:22.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:23.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:23.127 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:23.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:23.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:23.186 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:27:23.187 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:27:23.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:23.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:23.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:23.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:23.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:23.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:23.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:23.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:23.532 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:23.560 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:23.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:23.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:23.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:23.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:23.704 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:23.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:23.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:23.842 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:23.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:23.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:23.923 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.2 to 1 (10000) 2024-09-10T18:27:23.944 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:23.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:23.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:23.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:24.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:24.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-10T18:27:24.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:23 smithi204 ceph-mon[32357]: pgmap v252: 158 pgs: 158 active+clean; 622 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-10T18:27:24.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:23 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1633048708' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:27:24.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:23 smithi204 ceph-mon[32357]: from='client.24958 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:27:24.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:23 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:27:24.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:23 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:27:24.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:24.124 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:24.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:24.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:24.251 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:24.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:24.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:23 smithi028 ceph-mon[29291]: pgmap v252: 158 pgs: 158 active+clean; 622 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-10T18:27:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:23 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1633048708' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:27:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:23 smithi028 ceph-mon[29291]: from='client.24958 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:27:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:23 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:27:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:23 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:27:24.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:23 smithi028 ceph-mon[39220]: pgmap v252: 158 pgs: 158 active+clean; 622 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-10T18:27:24.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:23 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1633048708' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:27:24.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:23 smithi028 ceph-mon[39220]: from='client.24958 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:27:24.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:23 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:27:24.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:23 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:27:24.353 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:24.377 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:24.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:24.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:24.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:24.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:24.527 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:24.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:24.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:27:24 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:27:24.629 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:24.654 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:24.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:24.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:24.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:24.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-10T18:27:24.801 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:24.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:24.901 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:24.902 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:24.902 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:24.924 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:24.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:24.972 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:25.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:24 smithi028 ceph-mon[29291]: from='client.24958 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T18:27:25.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:24 smithi028 ceph-mon[29291]: osdmap e168: 8 total, 8 up, 8 in 2024-09-10T18:27:25.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:24 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:25.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:24 smithi028 ceph-mon[39220]: from='client.24958 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T18:27:25.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:24 smithi028 ceph-mon[39220]: osdmap e168: 8 total, 8 up, 8 in 2024-09-10T18:27:25.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:24 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:25.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:25.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:25.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:25.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:25.214 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:25.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:25.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:25.316 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:25.340 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:25.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:25.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:24 smithi204 ceph-mon[32357]: from='client.24958 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T18:27:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:24 smithi204 ceph-mon[32357]: osdmap e168: 8 total, 8 up, 8 in 2024-09-10T18:27:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:24 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:25.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:25.467 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:25.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:25.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:25.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:25.607 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:25.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:25.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:25.709 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:25.730 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:25.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:25.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:25.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:25.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:25.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:26.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:26.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:26.141 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:26.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:26.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:26.242 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:26.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:25 smithi028 ceph-mon[39220]: pgmap v254: 158 pgs: 158 active+clean; 557 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:27:26.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:25 smithi028 ceph-mon[39220]: osdmap e169: 8 total, 8 up, 8 in 2024-09-10T18:27:26.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:25 smithi028 ceph-mon[29291]: pgmap v254: 158 pgs: 158 active+clean; 557 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:27:26.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:25 smithi028 ceph-mon[29291]: osdmap e169: 8 total, 8 up, 8 in 2024-09-10T18:27:26.268 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:26.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:26.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:25 smithi204 ceph-mon[32357]: pgmap v254: 158 pgs: 158 active+clean; 557 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:27:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:25 smithi204 ceph-mon[32357]: osdmap e169: 8 total, 8 up, 8 in 2024-09-10T18:27:26.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:26.407 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:26.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:26.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:26.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:26.533 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:26.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:26.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:26.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:26.678 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:26.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:26.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:26.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:26.817 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:26.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:26.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:26.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:26.958 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:26.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:26.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:27.060 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:27.084 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:27.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:27.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:27.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:27.231 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:27.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:27.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:27.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:27.367 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:27.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:27.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:27.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:27.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:27.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:27.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:27.658 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:27.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:27.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:27.760 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:27.785 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:27.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:27.886 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:27.912 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:27.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:27.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:28.014 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:28.039 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:28.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:28.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:28.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:28.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:28.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:28.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:27 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:27:28.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:27 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:27:28.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:27 smithi028 ceph-mon[39220]: pgmap v256: 158 pgs: 158 active+clean; 287 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 823 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-10T18:27:28.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:27 smithi028 ceph-mon[39220]: osdmap e170: 8 total, 8 up, 8 in 2024-09-10T18:27:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:27 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:27:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:27 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:27:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:27 smithi028 ceph-mon[29291]: pgmap v256: 158 pgs: 158 active+clean; 287 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 823 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-10T18:27:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:27 smithi028 ceph-mon[29291]: osdmap e170: 8 total, 8 up, 8 in 2024-09-10T18:27:28.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:28.321 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:28.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:28.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:27 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:27:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:27 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:27:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:27 smithi204 ceph-mon[32357]: pgmap v256: 158 pgs: 158 active+clean; 287 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 823 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-10T18:27:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:27 smithi204 ceph-mon[32357]: osdmap e170: 8 total, 8 up, 8 in 2024-09-10T18:27:28.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:28.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:28.568 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:28.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:28.593 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:28.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:28.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:28.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:28.746 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:28.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:28.847 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:28.872 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:28.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:28.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:29.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:29.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:29.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:29.165 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:29.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:29.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:29.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:29.298 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:29.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:29.314 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Removed 121097 objects 2024-09-10T18:27:29.314 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Clean up completed and total clean up time :135.411 2024-09-10T18:27:29.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:29.421 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:29.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:29.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:29.522 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:29.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:29.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:29.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:29.608 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-10T18:27:29.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-10T18:27:29.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:29.674 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:29.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:29.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:29.776 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:29.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:29.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:29.806 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:29.906 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:29.928 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:29.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:29.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:30.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:29 smithi028 ceph-mon[29291]: pgmap v258: 158 pgs: 158 active+clean; 287 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 893 op/s 2024-09-10T18:27:30.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:29 smithi028 ceph-mon[39220]: pgmap v258: 158 pgs: 158 active+clean; 287 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 893 op/s 2024-09-10T18:27:30.029 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:30.051 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:30.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:30.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:30.177 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:30.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:30.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:30.279 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:30.302 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:30.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:30.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:30.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:29 smithi204 ceph-mon[32357]: pgmap v258: 158 pgs: 158 active+clean; 287 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 893 op/s 2024-09-10T18:27:30.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:30.426 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:30.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:30.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:30.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:30.551 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:30.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:30.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:30.652 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:30.676 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:30.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:30.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:30.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:30.799 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:30.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:30.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:30.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:30.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:30.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:31.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:31.051 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:31.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:31.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:31.085 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:27:31.085 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:27:31.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:31.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:31.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:31.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:31.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:31.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:31.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:31.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:31.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:31.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:31.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:27:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:27:31] "GET /metrics HTTP/1.1" 200 224066 "" "Prometheus/2.43.0" 2024-09-10T18:27:31.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:31.557 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:31.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:31.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:31.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:31.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:31.685 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:31.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:31.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:31.811 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:31.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:31.900 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:31 smithi028 ceph-mon[39220]: osdmap e171: 8 total, 8 up, 8 in 2024-09-10T18:27:31.900 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:31 smithi028 ceph-mon[39220]: pgmap v260: 158 pgs: 158 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.48k op/s 2024-09-10T18:27:31.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:31 smithi028 ceph-mon[29291]: osdmap e171: 8 total, 8 up, 8 in 2024-09-10T18:27:31.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:31 smithi028 ceph-mon[29291]: pgmap v260: 158 pgs: 158 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.48k op/s 2024-09-10T18:27:31.914 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:31.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:31.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:31.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:32.041 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:32.067 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:32.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:32.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:32.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:31 smithi204 ceph-mon[32357]: osdmap e171: 8 total, 8 up, 8 in 2024-09-10T18:27:32.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:31 smithi204 ceph-mon[32357]: pgmap v260: 158 pgs: 158 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.48k op/s 2024-09-10T18:27:32.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:32.195 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:32.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:32.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:32.218 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:27:32.218 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:27:32.232 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:27:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:27:32.247 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:27:32.247 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:27:32.262 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:27:32.263 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:27:32.278 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:27:32.279 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:27:32.294 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:27:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:27:32.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:32.310 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:27:32.310 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:27:32.322 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:32.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:32.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:32.326 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:27:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:27:32.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:32.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:32.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:32.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:32.551 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:32.577 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:32.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:32.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:32.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:27:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:27:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:27:32.680 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:32.705 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:32.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:32.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:32.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:32.832 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:32.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:32.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:32.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:32 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/662346512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:27:32.854 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:32 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/662346512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:27:32.935 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:32.960 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:32.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:32.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:33.038 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:32 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/662346512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:27:33.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:33.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:33.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:33.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:33.218 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:33.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:33.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:33.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:33.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:33.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:33.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:33.465 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:33.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:33.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:33.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:33.590 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:33.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:33.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:33.659 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:27:33.691 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:33.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:33.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:33.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:33.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:33 smithi028 ceph-mon[29291]: pgmap v261: 158 pgs: 158 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 710 B/s rd, 0 B/s wr, 637 op/s 2024-09-10T18:27:33.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:33 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/944065742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:27:33.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:33 smithi028 ceph-mon[29291]: from='client.34940 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:27:33.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:33 smithi028 ceph-mon[39220]: pgmap v261: 158 pgs: 158 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 710 B/s rd, 0 B/s wr, 637 op/s 2024-09-10T18:27:33.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:33 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/944065742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:27:33.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:33 smithi028 ceph-mon[39220]: from='client.34940 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:27:33.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:33.838 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:33.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:33.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:33.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:33.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:33.963 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:33.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:34.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:34.086 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:34.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:34.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:34.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:33 smithi204 ceph-mon[32357]: pgmap v261: 158 pgs: 158 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 710 B/s rd, 0 B/s wr, 637 op/s 2024-09-10T18:27:34.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:33 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/944065742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:27:34.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:33 smithi204 ceph-mon[32357]: from='client.34940 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:27:34.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:34.210 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:34.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:34.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:34.312 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:34.338 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:34.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:34.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:34.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:34.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:34.465 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:34.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:34.566 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:34.592 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:34.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:34.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:34.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:27:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:27:34.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:34.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:34.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:34.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:34.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:34 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:27:34.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:34 smithi028 ceph-mon[39220]: from='client.34940 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:27:34.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:34 smithi028 ceph-mon[39220]: osdmap e172: 8 total, 8 up, 8 in 2024-09-10T18:27:34.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:34 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:34.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:34 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:27:34.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:34 smithi028 ceph-mon[29291]: from='client.34940 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:27:34.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:34 smithi028 ceph-mon[29291]: osdmap e172: 8 total, 8 up, 8 in 2024-09-10T18:27:34.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:34 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:34.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:34.842 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:34.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:34.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:34.944 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:34.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:34.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:34.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:35.072 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:35.097 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:35.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:35.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:35.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:34 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:27:35.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:34 smithi204 ceph-mon[32357]: from='client.34940 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:27:35.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:34 smithi204 ceph-mon[32357]: osdmap e172: 8 total, 8 up, 8 in 2024-09-10T18:27:35.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:34 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:35.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:35.224 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:35.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:35.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:35.327 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:35.352 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:35.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:35.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:35.454 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:35.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:35.481 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:35.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:35.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:35.605 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:35.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:35.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:35.630 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:35.709 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:35.734 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:35.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:35.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:35.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:35.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:35.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:35.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:35.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:35.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:36.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:35 smithi028 ceph-mon[29291]: pgmap v263: 158 pgs: 158 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 582 op/s 2024-09-10T18:27:36.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:35 smithi028 ceph-mon[29291]: osdmap e173: 8 total, 8 up, 8 in 2024-09-10T18:27:36.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:35 smithi028 ceph-mon[39220]: pgmap v263: 158 pgs: 158 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 582 op/s 2024-09-10T18:27:36.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:35 smithi028 ceph-mon[39220]: osdmap e173: 8 total, 8 up, 8 in 2024-09-10T18:27:36.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:35 smithi204 ceph-mon[32357]: pgmap v263: 158 pgs: 158 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 582 op/s 2024-09-10T18:27:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:35 smithi204 ceph-mon[32357]: osdmap e173: 8 total, 8 up, 8 in 2024-09-10T18:27:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:36.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:36.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:36.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:36.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:36.231 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:36.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:36.332 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:36.358 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:36.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:36.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:36.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:36.486 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:36.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:36.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:36.588 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:36.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:36.615 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:36.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.3 to 1 (10000) 2024-09-10T18:27:36.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:36.741 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:36.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:36.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:36.844 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:36.869 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:36.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:36.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:36.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:36.997 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:36.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:36.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:37.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:36 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2763729135' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:27:37.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:36 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2763729135' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:27:37.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:37.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:36 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2763729135' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:27:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:37.147 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:37.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:37.248 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:37.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:37.275 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:37.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:37.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:27:37.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:37.428 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:37.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:37.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:37.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-10T18:27:37.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:37.556 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:37.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:37.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:37.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:37.683 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:37.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:37.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:37.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:37.806 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:37.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:37.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:37.908 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:37.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:37.935 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:37.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:37.980 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:37 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2763729135' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T18:27:37.980 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:37 smithi028 ceph-mon[39220]: osdmap e174: 8 total, 8 up, 8 in 2024-09-10T18:27:37.980 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:37 smithi028 ceph-mon[39220]: pgmap v266: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:27:37.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:37 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2763729135' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T18:27:37.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:37 smithi028 ceph-mon[29291]: osdmap e174: 8 total, 8 up, 8 in 2024-09-10T18:27:37.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:37 smithi028 ceph-mon[29291]: pgmap v266: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:27:38.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:38.061 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:38.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:38.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:38.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:37 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2763729135' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T18:27:38.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:37 smithi204 ceph-mon[32357]: osdmap e174: 8 total, 8 up, 8 in 2024-09-10T18:27:38.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:37 smithi204 ceph-mon[32357]: pgmap v266: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:27:38.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:38.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:38.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:38.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:38.313 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:38.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:38.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:38.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:38.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:38.440 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:38.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:38.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:38.567 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:38.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:38.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:38.668 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:38.693 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:38.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:38.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:38.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:38.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:38.820 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:38.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:38.921 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:38.947 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:38.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:38.985 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:38 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:27:38.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:38 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:27:39.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-10T18:27:39.049 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:39.075 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:39.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:39.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:39.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:27:39.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:38 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:27:39.176 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:39.202 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:39.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:39.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:39.304 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:39.330 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:39.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:39.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:39.432 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:39.457 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:39.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:39.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:39.559 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:39.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:39.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:39.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:39.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:39.691 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:39.695 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:39.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:39.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:39.749 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:39.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_0' removed 2024-09-10T18:27:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:39.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:39.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:39.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:39 smithi028 ceph-mon[39220]: pgmap v267: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:27:39.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:39 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3450907363' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:27:39.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:39 smithi028 ceph-mon[39220]: from='client.34952 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:27:39.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:39 smithi028 ceph-mon[29291]: pgmap v267: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:27:39.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:39 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3450907363' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:27:39.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:39 smithi028 ceph-mon[29291]: from='client.34952 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:27:39.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:39.976 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:39.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:39.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:40.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:40.103 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:40.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:40.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:40.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:39 smithi204 ceph-mon[32357]: pgmap v267: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:27:40.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:39 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3450907363' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:27:40.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:39 smithi204 ceph-mon[32357]: from='client.34952 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:27:40.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:40.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:40.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:40.329 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:40.351 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:40.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:40.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:40.453 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:40.475 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:40.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:40.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:40.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:40.599 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:40.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:40.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:40.726 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:40.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:40.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:40.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:40.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:40.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:40.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:40.975 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:40.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:40.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:41.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:40 smithi028 ceph-mon[29291]: from='client.34952 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:27:41.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:40 smithi028 ceph-mon[29291]: osdmap e175: 8 total, 8 up, 8 in 2024-09-10T18:27:41.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:40 smithi028 ceph-mon[39220]: from='client.34952 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:27:41.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:40 smithi028 ceph-mon[39220]: osdmap e175: 8 total, 8 up, 8 in 2024-09-10T18:27:41.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:41.102 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:41.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:41.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:41.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:40 smithi204 ceph-mon[32357]: from='client.34952 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:27:41.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:40 smithi204 ceph-mon[32357]: osdmap e175: 8 total, 8 up, 8 in 2024-09-10T18:27:41.203 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:41.229 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:41.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:41.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:41.356 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:41.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:41.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:41.454 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:27:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:27:41] "GET /metrics HTTP/1.1" 200 223966 "" "Prometheus/2.43.0" 2024-09-10T18:27:41.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:41.480 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:41.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:41.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:41.581 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:41.607 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:41.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:41.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:41.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:41.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:41.817 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-10T18:27:41.818 INFO:tasks.radosbench:Beginning radosbench... 2024-09-10T18:27:41.819 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-10T18:27:41.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-09-10T18:27:41.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:41.860 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:41.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:41.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:41.962 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:41.984 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:41.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:41.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:42.052 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:41 smithi028 ceph-mon[29291]: pgmap v269: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T18:27:42.052 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:41 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:27:42.052 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:41 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:27:42.053 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:41 smithi028 ceph-mon[39220]: pgmap v269: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T18:27:42.053 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:41 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:27:42.053 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:41 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:27:42.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:41 smithi204 ceph-mon[32357]: pgmap v269: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T18:27:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:41 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:27:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:41 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:27:42.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:42.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:42.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:42.210 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:42.236 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:42.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:42.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:42.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:42.360 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:42.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:42.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:42.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:42.484 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:42.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:42.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:42.586 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:42.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:27:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:27:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:27:42.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:42.614 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:42.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:42.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:42.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:42.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:42.840 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:42.866 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:42.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:42.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:42.967 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:42.992 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:42.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:42.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:43.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:42 smithi028 ceph-mon[29291]: osdmap e176: 8 total, 8 up, 8 in 2024-09-10T18:27:43.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:42 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:27:43.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:42 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:27:43.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:42 smithi028 ceph-mon[39220]: osdmap e176: 8 total, 8 up, 8 in 2024-09-10T18:27:43.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:42 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:27:43.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:42 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:27:43.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:43.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:42 smithi204 ceph-mon[32357]: osdmap e176: 8 total, 8 up, 8 in 2024-09-10T18:27:43.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:42 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:27:43.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:42 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:27:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:43.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:43.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:43.243 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:43.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:43.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:43.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:43.367 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:43.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:43.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:43.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:43.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:43.493 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:43.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:43.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:43.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:43.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:43.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:43.745 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:43.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:43.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:43.847 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:43.872 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:43.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:43.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:43.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:43.996 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:43.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:43.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:44.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:43 smithi028 ceph-mon[29291]: osdmap e177: 8 total, 8 up, 8 in 2024-09-10T18:27:44.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:43 smithi028 ceph-mon[29291]: pgmap v272: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T18:27:44.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:43 smithi028 ceph-mon[39220]: osdmap e177: 8 total, 8 up, 8 in 2024-09-10T18:27:44.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:43 smithi028 ceph-mon[39220]: pgmap v272: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T18:27:44.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:43 smithi204 ceph-mon[32357]: osdmap e177: 8 total, 8 up, 8 in 2024-09-10T18:27:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:43 smithi204 ceph-mon[32357]: pgmap v272: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T18:27:44.120 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:44.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:44.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:44.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:44.245 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:44.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:44.347 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:44.373 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:44.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:44.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:44.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:44.500 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:44.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:44.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:44.602 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:44.605 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:27:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:27:44.626 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:44.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:44.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:44.728 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:44.753 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:44.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:44.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:44.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:44.880 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:44.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:44.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:44.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:45.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:45.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:45.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:44 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:45.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:45.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:45.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:45.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:44 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:44 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:45.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:45.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:45.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:45.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:45.380 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:45.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:45.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:45.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:45.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:45.600 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:27:45.600 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:27:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:27:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:27:45.601 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:27:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:27:45.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:45.619 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:27:45.619 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:27:45.630 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:45.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:45.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:45.636 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:27:45.636 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:27:45.640 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:45.656 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:27:45.656 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:27:45.673 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:27:45.674 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:27:45.689 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:27:45.690 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:27:45.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:45.753 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:45.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:45.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:45.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:45.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:45.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:45.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:45.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:46.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:46.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:46.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:45 smithi204 ceph-mon[32357]: pgmap v273: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T18:27:46.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:45 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1407478836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:27:46.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:46.112 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:45 smithi028 ceph-mon[39220]: pgmap v273: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T18:27:46.112 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:45 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1407478836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:27:46.112 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:45 smithi028 ceph-mon[29291]: pgmap v273: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T18:27:46.112 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:45 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1407478836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:27:46.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:46.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:46.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:46.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:46.254 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:46.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:46.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:46.356 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:46.381 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:46.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:46.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:46.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:46.506 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:46.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:46.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:46.608 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:46.634 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:46.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:46.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:46.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:46.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:46.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:46.861 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:46.887 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:46.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:46.989 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:47.015 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:47.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:47.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:47.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:47.142 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:47.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:47.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:47.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:47.270 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:47.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:47.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:47.372 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:47.397 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:47.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:47.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:47.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:47.523 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:47.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:47.581 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:27:47.581 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:27:47.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:47.652 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:47.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:47.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:47.754 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:47.778 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:47.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:47.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:47.905 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:47.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:47.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:48.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:48.029 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:48.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:48.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:48.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:48.153 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:48.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:47 smithi028 ceph-mon[29291]: pgmap v274: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 727 B/s rd, 0 op/s 2024-09-10T18:27:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:47 smithi028 ceph-mon[39220]: pgmap v274: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 727 B/s rd, 0 op/s 2024-09-10T18:27:48.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:48.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:48.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:48.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-10T18:27:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:47 smithi204 ceph-mon[32357]: pgmap v274: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 727 B/s rd, 0 op/s 2024-09-10T18:27:48.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:48.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:48.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:48.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:48.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:48.552 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:48.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:48.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:48.676 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:48.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:48.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:48.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:27:48.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:48.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:48.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:48.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:48.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:48.939 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:48.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:48.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:49.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:49.066 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:49.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:49.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:49.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:49.194 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:49.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:49.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:49.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:49.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:49.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:49.421 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:49.447 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:49.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:49.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:49.550 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:49.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:49.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:49.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:49.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:49.699 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:49.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:49.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:49.801 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:49.827 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:49.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:49.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:49.924 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.4 to 1 (10000) 2024-09-10T18:27:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:27:49.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:49.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:49.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:50.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:50.058 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:49 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:27:50.058 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:49 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:27:50.059 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:49 smithi028 ceph-mon[39220]: pgmap v275: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:27:50.059 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:49 smithi028 ceph-mon[39220]: osdmap e178: 8 total, 8 up, 8 in 2024-09-10T18:27:50.059 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:49 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2871572816' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:27:50.059 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:49 smithi028 ceph-mon[39220]: from='client.34964 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:27:50.059 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:49 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/539921639' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:27:50.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:49 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:27:50.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:49 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:27:50.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:49 smithi028 ceph-mon[29291]: pgmap v275: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:27:50.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:49 smithi028 ceph-mon[29291]: osdmap e178: 8 total, 8 up, 8 in 2024-09-10T18:27:50.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:49 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2871572816' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:27:50.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:49 smithi028 ceph-mon[29291]: from='client.34964 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:27:50.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:49 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/539921639' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:27:50.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:50.082 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:50.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:50.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:50.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:50.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:50.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:50.330 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:50.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:49 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:27:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:49 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:27:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:49 smithi204 ceph-mon[32357]: pgmap v275: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:27:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:49 smithi204 ceph-mon[32357]: osdmap e178: 8 total, 8 up, 8 in 2024-09-10T18:27:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:49 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2871572816' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:27:50.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:49 smithi204 ceph-mon[32357]: from='client.34964 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:27:50.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:49 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/539921639' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:27:50.432 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:50.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:50.461 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:50.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:50.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:50.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:50.687 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:50.712 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:50.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:50.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:50.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:50.837 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:50.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:50.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:50.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:50.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:50.963 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:50.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:51.065 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:51.090 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:51.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:51.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:51.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:51.217 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:51.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:51.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:51.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:50 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:27:51.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:50 smithi028 ceph-mon[39220]: from='client.34964 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T18:27:51.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:50 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/539921639' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:27:51.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:50 smithi028 ceph-mon[39220]: osdmap e179: 8 total, 8 up, 8 in 2024-09-10T18:27:51.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:50 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:27:51.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:50 smithi028 ceph-mon[29291]: from='client.34964 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T18:27:51.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:50 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/539921639' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:27:51.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:50 smithi028 ceph-mon[29291]: osdmap e179: 8 total, 8 up, 8 in 2024-09-10T18:27:51.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:51.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:51.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:51.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:50 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:27:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:50 smithi204 ceph-mon[32357]: from='client.34964 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T18:27:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:50 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/539921639' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:27:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:50 smithi204 ceph-mon[32357]: osdmap e179: 8 total, 8 up, 8 in 2024-09-10T18:27:51.444 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:51.470 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:51.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:51.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:51.572 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:51.598 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:51.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:51.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:51.699 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:51.724 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:51.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:51.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:51.749 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:27:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:27:51] "GET /metrics HTTP/1.1" 200 223966 "" "Prometheus/2.43.0" 2024-09-10T18:27:51.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:51.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:51.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:51.949 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:51.974 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:51.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:51.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:52.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:51 smithi028 ceph-mon[29291]: pgmap v278: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:27:52.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:51 smithi028 ceph-mon[39220]: pgmap v278: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:27:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_1' created 2024-09-10T18:27:52.076 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:52.100 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:52.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:52.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:52.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:52.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:52.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:51 smithi204 ceph-mon[32357]: pgmap v278: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:27:52.255 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:27:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:27:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:27:52.327 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:52.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:52.352 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:52.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:52.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:52.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:52.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:52.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-10T18:27:52.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:52.603 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:52.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:52.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:52.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:52.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:52.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:52.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:52.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:27:52.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:52.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:52.954 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:52.980 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:52.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:52.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:53.082 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:53.096 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[29291]: 4.6 scrub starts 2024-09-10T18:27:53.096 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[29291]: 4.6 scrub ok 2024-09-10T18:27:53.096 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[29291]: 4.11 scrub starts 2024-09-10T18:27:53.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[29291]: 4.11 scrub ok 2024-09-10T18:27:53.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1626538875' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-10T18:27:53.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[29291]: 6.9 scrub starts 2024-09-10T18:27:53.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[29291]: 6.9 scrub ok 2024-09-10T18:27:53.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1626538875' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-10T18:27:53.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[29291]: osdmap e180: 8 total, 8 up, 8 in 2024-09-10T18:27:53.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[29291]: 5.a scrub starts 2024-09-10T18:27:53.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[29291]: 5.a scrub ok 2024-09-10T18:27:53.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[29291]: 4.10 scrub starts 2024-09-10T18:27:53.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[29291]: 4.10 scrub ok 2024-09-10T18:27:53.097 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[39220]: 4.6 scrub starts 2024-09-10T18:27:53.097 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[39220]: 4.6 scrub ok 2024-09-10T18:27:53.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[39220]: 4.11 scrub starts 2024-09-10T18:27:53.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[39220]: 4.11 scrub ok 2024-09-10T18:27:53.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1626538875' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-10T18:27:53.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[39220]: 6.9 scrub starts 2024-09-10T18:27:53.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[39220]: 6.9 scrub ok 2024-09-10T18:27:53.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1626538875' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-10T18:27:53.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[39220]: osdmap e180: 8 total, 8 up, 8 in 2024-09-10T18:27:53.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[39220]: 5.a scrub starts 2024-09-10T18:27:53.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[39220]: 5.a scrub ok 2024-09-10T18:27:53.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[39220]: 4.10 scrub starts 2024-09-10T18:27:53.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:52 smithi028 ceph-mon[39220]: 4.10 scrub ok 2024-09-10T18:27:53.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:53.107 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:53.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:53.115 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:53.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-10T18:27:53.208 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:53.231 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:53.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:53.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:53.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:52 smithi204 ceph-mon[32357]: 4.6 scrub starts 2024-09-10T18:27:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:52 smithi204 ceph-mon[32357]: 4.6 scrub ok 2024-09-10T18:27:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:52 smithi204 ceph-mon[32357]: 4.11 scrub starts 2024-09-10T18:27:53.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:52 smithi204 ceph-mon[32357]: 4.11 scrub ok 2024-09-10T18:27:53.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:52 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1626538875' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-10T18:27:53.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:52 smithi204 ceph-mon[32357]: 6.9 scrub starts 2024-09-10T18:27:53.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:52 smithi204 ceph-mon[32357]: 6.9 scrub ok 2024-09-10T18:27:53.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:52 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1626538875' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-10T18:27:53.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:52 smithi204 ceph-mon[32357]: osdmap e180: 8 total, 8 up, 8 in 2024-09-10T18:27:53.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:52 smithi204 ceph-mon[32357]: 5.a scrub starts 2024-09-10T18:27:53.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:52 smithi204 ceph-mon[32357]: 5.a scrub ok 2024-09-10T18:27:53.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:52 smithi204 ceph-mon[32357]: 4.10 scrub starts 2024-09-10T18:27:53.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:52 smithi204 ceph-mon[32357]: 4.10 scrub ok 2024-09-10T18:27:53.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:53.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:53.457 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:53.482 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:53.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:53.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:53.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:53.606 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:53.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:53.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:53.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:53.730 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:53.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:53.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:53.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:53.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:53.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:53.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:53.956 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:53.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:53.984 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:53.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:54.085 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:54.109 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:54.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:54.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:54.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:54.235 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:54.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[29291]: 6.3 scrub starts 2024-09-10T18:27:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[29291]: 6.3 scrub ok 2024-09-10T18:27:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[29291]: 5.18 scrub starts 2024-09-10T18:27:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[29291]: 5.18 scrub ok 2024-09-10T18:27:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[29291]: 3.10 scrub starts 2024-09-10T18:27:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[29291]: 3.10 scrub ok 2024-09-10T18:27:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[29291]: 5.16 scrub starts 2024-09-10T18:27:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[29291]: 5.16 scrub ok 2024-09-10T18:27:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[29291]: pgmap v280: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:27:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[29291]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:27:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[29291]: osdmap e181: 8 total, 8 up, 8 in 2024-09-10T18:27:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[29291]: 5.14 scrub starts 2024-09-10T18:27:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[29291]: 5.14 scrub ok 2024-09-10T18:27:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:27:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:27:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[39220]: 6.3 scrub starts 2024-09-10T18:27:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[39220]: 6.3 scrub ok 2024-09-10T18:27:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[39220]: 5.18 scrub starts 2024-09-10T18:27:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[39220]: 5.18 scrub ok 2024-09-10T18:27:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[39220]: 3.10 scrub starts 2024-09-10T18:27:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[39220]: 3.10 scrub ok 2024-09-10T18:27:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[39220]: 5.16 scrub starts 2024-09-10T18:27:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[39220]: 5.16 scrub ok 2024-09-10T18:27:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[39220]: pgmap v280: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:27:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[39220]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:27:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[39220]: osdmap e181: 8 total, 8 up, 8 in 2024-09-10T18:27:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[39220]: 5.14 scrub starts 2024-09-10T18:27:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[39220]: 5.14 scrub ok 2024-09-10T18:27:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:27:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:53 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:27:54.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:54.357 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:27:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:27:54.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:53 smithi204 ceph-mon[32357]: 6.3 scrub starts 2024-09-10T18:27:54.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:53 smithi204 ceph-mon[32357]: 6.3 scrub ok 2024-09-10T18:27:54.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:53 smithi204 ceph-mon[32357]: 5.18 scrub starts 2024-09-10T18:27:54.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:53 smithi204 ceph-mon[32357]: 5.18 scrub ok 2024-09-10T18:27:54.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:53 smithi204 ceph-mon[32357]: 3.10 scrub starts 2024-09-10T18:27:54.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:53 smithi204 ceph-mon[32357]: 3.10 scrub ok 2024-09-10T18:27:54.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:53 smithi204 ceph-mon[32357]: 5.16 scrub starts 2024-09-10T18:27:54.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:53 smithi204 ceph-mon[32357]: 5.16 scrub ok 2024-09-10T18:27:54.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:53 smithi204 ceph-mon[32357]: pgmap v280: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:27:54.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:53 smithi204 ceph-mon[32357]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:27:54.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:53 smithi204 ceph-mon[32357]: osdmap e181: 8 total, 8 up, 8 in 2024-09-10T18:27:54.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:53 smithi204 ceph-mon[32357]: 5.14 scrub starts 2024-09-10T18:27:54.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:53 smithi204 ceph-mon[32357]: 5.14 scrub ok 2024-09-10T18:27:54.359 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:53 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:27:54.359 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:53 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:27:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:54.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:54.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:54.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:54.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:54.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:54.586 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:54.613 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:54.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:54.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:54.714 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:54.739 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:54.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:54.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:54.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:54.864 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:54.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:54.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:54.966 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:54.992 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:54.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:54.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:55.094 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:55.119 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:55.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:55.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:55.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:55.245 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:55.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:55.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:55.276 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:27:55.276 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:27:55.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:55 smithi028 ceph-mon[39220]: 1.6 scrub starts 2024-09-10T18:27:55.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:55 smithi028 ceph-mon[39220]: 1.6 scrub ok 2024-09-10T18:27:55.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:55 smithi028 ceph-mon[39220]: 3.1d scrub starts 2024-09-10T18:27:55.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:55 smithi028 ceph-mon[39220]: 3.1d scrub ok 2024-09-10T18:27:55.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:55 smithi028 ceph-mon[39220]: osdmap e182: 8 total, 8 up, 8 in 2024-09-10T18:27:55.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:55 smithi028 ceph-mon[39220]: 6.16 scrub starts 2024-09-10T18:27:55.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:55 smithi028 ceph-mon[39220]: 6.16 scrub ok 2024-09-10T18:27:55.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:55 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:55 smithi028 ceph-mon[29291]: 1.6 scrub starts 2024-09-10T18:27:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:55 smithi028 ceph-mon[29291]: 1.6 scrub ok 2024-09-10T18:27:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:55 smithi028 ceph-mon[29291]: 3.1d scrub starts 2024-09-10T18:27:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:55 smithi028 ceph-mon[29291]: 3.1d scrub ok 2024-09-10T18:27:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:55 smithi028 ceph-mon[29291]: osdmap e182: 8 total, 8 up, 8 in 2024-09-10T18:27:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:55 smithi028 ceph-mon[29291]: 6.16 scrub starts 2024-09-10T18:27:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:55 smithi028 ceph-mon[29291]: 6.16 scrub ok 2024-09-10T18:27:55.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:55 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:55.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:55 smithi204 ceph-mon[32357]: 1.6 scrub starts 2024-09-10T18:27:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:55 smithi204 ceph-mon[32357]: 1.6 scrub ok 2024-09-10T18:27:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:55 smithi204 ceph-mon[32357]: 3.1d scrub starts 2024-09-10T18:27:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:55 smithi204 ceph-mon[32357]: 3.1d scrub ok 2024-09-10T18:27:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:55 smithi204 ceph-mon[32357]: osdmap e182: 8 total, 8 up, 8 in 2024-09-10T18:27:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:55 smithi204 ceph-mon[32357]: 6.16 scrub starts 2024-09-10T18:27:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:55 smithi204 ceph-mon[32357]: 6.16 scrub ok 2024-09-10T18:27:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:55 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:27:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:55.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:55.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:55.472 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:55.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:55.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:55.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:55.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:55.622 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:55.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:55.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:55.724 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:55.749 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:55.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:55.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:55.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:55.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:55.975 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:56.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-10T18:27:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:56.002 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:56.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:56.034 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:56.103 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:56.126 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:56.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:56.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:56.251 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:56.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:56.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:56 smithi028 ceph-mon[29291]: 3.14 scrub starts 2024-09-10T18:27:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:56 smithi028 ceph-mon[29291]: 3.14 scrub ok 2024-09-10T18:27:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:56 smithi028 ceph-mon[29291]: 1.1 scrub starts 2024-09-10T18:27:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:56 smithi028 ceph-mon[29291]: 1.1 scrub ok 2024-09-10T18:27:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:56 smithi028 ceph-mon[29291]: pgmap v283: 158 pgs: 12 unknown, 146 active+clean; 530 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:27:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:56 smithi028 ceph-mon[29291]: osdmap e183: 8 total, 8 up, 8 in 2024-09-10T18:27:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:56 smithi028 ceph-mon[39220]: 3.14 scrub starts 2024-09-10T18:27:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:56 smithi028 ceph-mon[39220]: 3.14 scrub ok 2024-09-10T18:27:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:56 smithi028 ceph-mon[39220]: 1.1 scrub starts 2024-09-10T18:27:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:56 smithi028 ceph-mon[39220]: 1.1 scrub ok 2024-09-10T18:27:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:56 smithi028 ceph-mon[39220]: pgmap v283: 158 pgs: 12 unknown, 146 active+clean; 530 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:27:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:56 smithi028 ceph-mon[39220]: osdmap e183: 8 total, 8 up, 8 in 2024-09-10T18:27:56.353 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:56 smithi204 ceph-mon[32357]: 3.14 scrub starts 2024-09-10T18:27:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:56 smithi204 ceph-mon[32357]: 3.14 scrub ok 2024-09-10T18:27:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:56 smithi204 ceph-mon[32357]: 1.1 scrub starts 2024-09-10T18:27:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:56 smithi204 ceph-mon[32357]: 1.1 scrub ok 2024-09-10T18:27:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:56 smithi204 ceph-mon[32357]: pgmap v283: 158 pgs: 12 unknown, 146 active+clean; 530 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:27:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:56 smithi204 ceph-mon[32357]: osdmap e183: 8 total, 8 up, 8 in 2024-09-10T18:27:56.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:56.378 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:56.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:56.479 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:27:56.501 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:56.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:56.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:27:56.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:56.626 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:56.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:56.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:56.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:56.753 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:56.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:56.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:56.856 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:56.880 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:56.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:56.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:56.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:57.005 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:57.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:57.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:57.129 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:57.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:57.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:57.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:57.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:57.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:57.294 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:27:57.368 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:27:57.368 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:27:57.368 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:27:57.369 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:27:57.369 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:27:57.369 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:27:57.369 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:27:57.369 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:27:57.369 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:27:57.369 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:27:57.369 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:27:57.370 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:27:57.370 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:27:57.370 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:27:57.370 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:27:57.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[39220]: 6.5 scrub starts 2024-09-10T18:27:57.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[39220]: 6.5 scrub ok 2024-09-10T18:27:57.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[39220]: 6.1e scrub starts 2024-09-10T18:27:57.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[39220]: 6.1e scrub ok 2024-09-10T18:27:57.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[39220]: 1.8 scrub starts 2024-09-10T18:27:57.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[39220]: 1.8 scrub ok 2024-09-10T18:27:57.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:27:57.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:27:57.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[39220]: 5.17 scrub starts 2024-09-10T18:27:57.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[39220]: 5.17 scrub ok 2024-09-10T18:27:57.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/700415186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:27:57.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[29291]: 6.5 scrub starts 2024-09-10T18:27:57.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[29291]: 6.5 scrub ok 2024-09-10T18:27:57.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[29291]: 6.1e scrub starts 2024-09-10T18:27:57.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[29291]: 6.1e scrub ok 2024-09-10T18:27:57.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[29291]: 1.8 scrub starts 2024-09-10T18:27:57.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[29291]: 1.8 scrub ok 2024-09-10T18:27:57.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:27:57.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:27:57.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[29291]: 5.17 scrub starts 2024-09-10T18:27:57.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[29291]: 5.17 scrub ok 2024-09-10T18:27:57.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:57 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/700415186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:27:57.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:57.373 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:57 smithi204 ceph-mon[32357]: 6.5 scrub starts 2024-09-10T18:27:57.373 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:57 smithi204 ceph-mon[32357]: 6.5 scrub ok 2024-09-10T18:27:57.373 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:57 smithi204 ceph-mon[32357]: 6.1e scrub starts 2024-09-10T18:27:57.373 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:57 smithi204 ceph-mon[32357]: 6.1e scrub ok 2024-09-10T18:27:57.373 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:57 smithi204 ceph-mon[32357]: 1.8 scrub starts 2024-09-10T18:27:57.373 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:57 smithi204 ceph-mon[32357]: 1.8 scrub ok 2024-09-10T18:27:57.373 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:57 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:27:57.373 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:57 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:27:57.373 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:57 smithi204 ceph-mon[32357]: 5.17 scrub starts 2024-09-10T18:27:57.373 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:57 smithi204 ceph-mon[32357]: 5.17 scrub ok 2024-09-10T18:27:57.373 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:57 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/700415186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:27:57.396 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:57.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:57.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:57.498 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:57.523 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:57.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:57.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:27:57.625 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:57.648 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:57.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:57.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:57.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:57.773 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:57.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:57.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:57.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:57.897 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:57.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:57.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:27:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:58.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:58.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:27:58.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:58.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:58.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:58.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:58.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:58.273 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:58.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:58.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:58 smithi204 ceph-mon[32357]: 6.4 scrub starts 2024-09-10T18:27:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:58 smithi204 ceph-mon[32357]: 6.4 scrub ok 2024-09-10T18:27:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:58 smithi204 ceph-mon[32357]: pgmap v285: 158 pgs: 158 active+clean; 477 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:27:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:58 smithi204 ceph-mon[32357]: osdmap e184: 8 total, 8 up, 8 in 2024-09-10T18:27:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:58 smithi204 ceph-mon[32357]: 3.6 scrub starts 2024-09-10T18:27:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:58 smithi204 ceph-mon[32357]: 3.6 scrub ok 2024-09-10T18:27:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:58 smithi204 ceph-mon[32357]: 4.e scrub starts 2024-09-10T18:27:58.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:58 smithi204 ceph-mon[32357]: 4.e scrub ok 2024-09-10T18:27:58.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:58.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:58.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:58.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:58.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:27:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:58 smithi028 ceph-mon[39220]: 6.4 scrub starts 2024-09-10T18:27:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:58 smithi028 ceph-mon[39220]: 6.4 scrub ok 2024-09-10T18:27:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:58 smithi028 ceph-mon[39220]: pgmap v285: 158 pgs: 158 active+clean; 477 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:27:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:58 smithi028 ceph-mon[39220]: osdmap e184: 8 total, 8 up, 8 in 2024-09-10T18:27:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:58 smithi028 ceph-mon[39220]: 3.6 scrub starts 2024-09-10T18:27:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:58 smithi028 ceph-mon[39220]: 3.6 scrub ok 2024-09-10T18:27:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:58 smithi028 ceph-mon[39220]: 4.e scrub starts 2024-09-10T18:27:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:58 smithi028 ceph-mon[39220]: 4.e scrub ok 2024-09-10T18:27:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:58 smithi028 ceph-mon[29291]: 6.4 scrub starts 2024-09-10T18:27:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:58 smithi028 ceph-mon[29291]: 6.4 scrub ok 2024-09-10T18:27:58.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:58 smithi028 ceph-mon[29291]: pgmap v285: 158 pgs: 158 active+clean; 477 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:27:58.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:58 smithi028 ceph-mon[29291]: osdmap e184: 8 total, 8 up, 8 in 2024-09-10T18:27:58.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:58 smithi028 ceph-mon[29291]: 3.6 scrub starts 2024-09-10T18:27:58.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:58 smithi028 ceph-mon[29291]: 3.6 scrub ok 2024-09-10T18:27:58.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:58 smithi028 ceph-mon[29291]: 4.e scrub starts 2024-09-10T18:27:58.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:58 smithi028 ceph-mon[29291]: 4.e scrub ok 2024-09-10T18:27:58.525 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:58.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:58.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:27:58.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:58.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:58.652 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:58.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:58.667 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:58.668 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:27:58.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:58.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:58.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:58.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:58.900 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:58.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:58.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:59.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:59.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:59.025 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:59.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:59.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:59.148 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:59.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:59.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:59.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:59 smithi028 ceph-mon[39220]: 5.10 scrub starts 2024-09-10T18:27:59.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:59 smithi028 ceph-mon[39220]: 5.10 scrub ok 2024-09-10T18:27:59.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:59 smithi028 ceph-mon[39220]: 1.3 scrub starts 2024-09-10T18:27:59.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:27:59 smithi028 ceph-mon[39220]: 1.3 scrub ok 2024-09-10T18:27:59.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:59 smithi028 ceph-mon[29291]: 5.10 scrub starts 2024-09-10T18:27:59.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:59 smithi028 ceph-mon[29291]: 5.10 scrub ok 2024-09-10T18:27:59.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:59 smithi028 ceph-mon[29291]: 1.3 scrub starts 2024-09-10T18:27:59.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:27:59 smithi028 ceph-mon[29291]: 1.3 scrub ok 2024-09-10T18:27:59.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:27:59.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:59.273 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:59.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:27:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:59 smithi204 ceph-mon[32357]: 5.10 scrub starts 2024-09-10T18:27:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:59 smithi204 ceph-mon[32357]: 5.10 scrub ok 2024-09-10T18:27:59.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:59 smithi204 ceph-mon[32357]: 1.3 scrub starts 2024-09-10T18:27:59.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:27:59 smithi204 ceph-mon[32357]: 1.3 scrub ok 2024-09-10T18:27:59.374 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:59.400 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:59.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:59.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:59.502 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:27:59.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:59.529 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:59.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:27:59.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:27:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:59.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:59.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:27:59.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:27:59.777 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:27:59.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:59.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:27:59.880 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:27:59.902 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:27:59.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:27:59.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:00.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:00.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:00.028 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:00.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:00.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:28:00.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:00.151 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:00.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:00.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:00.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:00.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: 5.e scrub starts 2024-09-10T18:28:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: 5.e scrub ok 2024-09-10T18:28:00.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: 1.9 scrub starts 2024-09-10T18:28:00.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: 1.9 scrub ok 2024-09-10T18:28:00.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: 5.12 scrub starts 2024-09-10T18:28:00.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: 5.12 scrub ok 2024-09-10T18:28:00.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: 1.a scrub starts 2024-09-10T18:28:00.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: 1.a scrub ok 2024-09-10T18:28:00.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:28:00.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:28:00.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: pgmap v287: 158 pgs: 158 active+clean; 477 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:00.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: 6.14 scrub starts 2024-09-10T18:28:00.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: 6.14 scrub ok 2024-09-10T18:28:00.357 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: osdmap e185: 8 total, 8 up, 8 in 2024-09-10T18:28:00.357 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:28:00.357 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/430934464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:28:00.357 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:00.357 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:00 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:00.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:00.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:00.404 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:00.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: 5.e scrub starts 2024-09-10T18:28:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: 5.e scrub ok 2024-09-10T18:28:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: 1.9 scrub starts 2024-09-10T18:28:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: 1.9 scrub ok 2024-09-10T18:28:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: 5.12 scrub starts 2024-09-10T18:28:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: 5.12 scrub ok 2024-09-10T18:28:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: 1.a scrub starts 2024-09-10T18:28:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: 1.a scrub ok 2024-09-10T18:28:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:28:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:28:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: pgmap v287: 158 pgs: 158 active+clean; 477 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: 6.14 scrub starts 2024-09-10T18:28:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: 6.14 scrub ok 2024-09-10T18:28:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: osdmap e185: 8 total, 8 up, 8 in 2024-09-10T18:28:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:28:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/430934464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:28:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:00.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: 5.e scrub starts 2024-09-10T18:28:00.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: 5.e scrub ok 2024-09-10T18:28:00.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: 1.9 scrub starts 2024-09-10T18:28:00.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: 1.9 scrub ok 2024-09-10T18:28:00.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: 5.12 scrub starts 2024-09-10T18:28:00.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: 5.12 scrub ok 2024-09-10T18:28:00.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: 1.a scrub starts 2024-09-10T18:28:00.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: 1.a scrub ok 2024-09-10T18:28:00.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:28:00.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:28:00.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: pgmap v287: 158 pgs: 158 active+clean; 477 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:00.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: 6.14 scrub starts 2024-09-10T18:28:00.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: 6.14 scrub ok 2024-09-10T18:28:00.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: osdmap e185: 8 total, 8 up, 8 in 2024-09-10T18:28:00.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:28:00.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/430934464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:28:00.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:00.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:00 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:00.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:00.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:00.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:00.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:00.655 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:00.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:00.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:00.756 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:00.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:00.780 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:00.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:00.880 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:00.903 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:00.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:01.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:01.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:01.089 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:28:01.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:01.153 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:01.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:01.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[29291]: 1.2 scrub starts 2024-09-10T18:28:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[29291]: 1.2 scrub ok 2024-09-10T18:28:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[29291]: 4.f scrub starts 2024-09-10T18:28:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[29291]: 4.f scrub ok 2024-09-10T18:28:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:28:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/430934464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:28:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[29291]: osdmap e186: 8 total, 8 up, 8 in 2024-09-10T18:28:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[29291]: 3.8 scrub starts 2024-09-10T18:28:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[29291]: 3.8 scrub ok 2024-09-10T18:28:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[29291]: 1.7 scrub starts 2024-09-10T18:28:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[29291]: 1.7 scrub ok 2024-09-10T18:28:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[29291]: osdmap e187: 8 total, 8 up, 8 in 2024-09-10T18:28:01.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:01.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[39220]: 1.2 scrub starts 2024-09-10T18:28:01.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[39220]: 1.2 scrub ok 2024-09-10T18:28:01.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[39220]: 4.f scrub starts 2024-09-10T18:28:01.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[39220]: 4.f scrub ok 2024-09-10T18:28:01.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:28:01.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/430934464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:28:01.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[39220]: osdmap e186: 8 total, 8 up, 8 in 2024-09-10T18:28:01.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[39220]: 3.8 scrub starts 2024-09-10T18:28:01.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[39220]: 3.8 scrub ok 2024-09-10T18:28:01.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[39220]: 1.7 scrub starts 2024-09-10T18:28:01.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[39220]: 1.7 scrub ok 2024-09-10T18:28:01.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:01.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:01.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-mon[39220]: osdmap e187: 8 total, 8 up, 8 in 2024-09-10T18:28:01.278 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:01.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:01 smithi204 ceph-mon[32357]: 1.2 scrub starts 2024-09-10T18:28:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:01 smithi204 ceph-mon[32357]: 1.2 scrub ok 2024-09-10T18:28:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:01 smithi204 ceph-mon[32357]: 4.f scrub starts 2024-09-10T18:28:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:01 smithi204 ceph-mon[32357]: 4.f scrub ok 2024-09-10T18:28:01.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:01 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:28:01.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:01 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/430934464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:28:01.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:01 smithi204 ceph-mon[32357]: osdmap e186: 8 total, 8 up, 8 in 2024-09-10T18:28:01.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:01 smithi204 ceph-mon[32357]: 3.8 scrub starts 2024-09-10T18:28:01.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:01 smithi204 ceph-mon[32357]: 3.8 scrub ok 2024-09-10T18:28:01.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:01 smithi204 ceph-mon[32357]: 1.7 scrub starts 2024-09-10T18:28:01.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:01 smithi204 ceph-mon[32357]: 1.7 scrub ok 2024-09-10T18:28:01.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:01 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:01.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:01 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:01.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:01 smithi204 ceph-mon[32357]: osdmap e187: 8 total, 8 up, 8 in 2024-09-10T18:28:01.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:01.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:01.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:01.505 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:01.529 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:01.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:01.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:01.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:01.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:01.655 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:01.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:01.694 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:28:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:28:01] "GET /metrics HTTP/1.1" 200 223885 "" "Prometheus/2.43.0" 2024-09-10T18:28:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:28:01.756 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:01.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:01.781 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:01.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:01.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:01.905 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:01.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:01.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:02.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:02.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:02.029 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:02.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:02.130 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:02.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:02.154 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:02.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:02.255 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:02.277 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:02.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:02.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:02 smithi204 ceph-mon[32357]: pgmap v291: 158 pgs: 158 active+clean; 419 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:02 smithi204 ceph-mon[32357]: 4.17 scrub starts 2024-09-10T18:28:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:02 smithi204 ceph-mon[32357]: 4.17 scrub ok 2024-09-10T18:28:02.355 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:28:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:28:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:28:02.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:02 smithi028 ceph-mon[29291]: pgmap v291: 158 pgs: 158 active+clean; 419 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:02.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:02 smithi028 ceph-mon[29291]: 4.17 scrub starts 2024-09-10T18:28:02.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:02 smithi028 ceph-mon[29291]: 4.17 scrub ok 2024-09-10T18:28:02.375 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:02 smithi028 ceph-mon[39220]: pgmap v291: 158 pgs: 158 active+clean; 419 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:02.375 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:02 smithi028 ceph-mon[39220]: 4.17 scrub starts 2024-09-10T18:28:02.375 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:02 smithi028 ceph-mon[39220]: 4.17 scrub ok 2024-09-10T18:28:02.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:02.402 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:02.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:02.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:02.527 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:02.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:02.628 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:02.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:02.652 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:02.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:02.753 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:02.776 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:02.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:02.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:02.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:02.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:02.901 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:02.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:03.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:03.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:03.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:03.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:03.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:03.151 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:03.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:03.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:03.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:03.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:03.316 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.5 to 1 (10000) 2024-09-10T18:28:03.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[29291]: 5.1d scrub starts 2024-09-10T18:28:03.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[29291]: 5.1d scrub ok 2024-09-10T18:28:03.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[29291]: 3.16 scrub starts 2024-09-10T18:28:03.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[29291]: 3.16 scrub ok 2024-09-10T18:28:03.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:28:03.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:28:03.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:03.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/677075418' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:28:03.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[29291]: from='client.34994 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:28:03.317 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[39220]: 5.1d scrub starts 2024-09-10T18:28:03.317 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[39220]: 5.1d scrub ok 2024-09-10T18:28:03.317 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[39220]: 3.16 scrub starts 2024-09-10T18:28:03.318 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[39220]: 3.16 scrub ok 2024-09-10T18:28:03.318 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:28:03.318 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:28:03.318 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:03.318 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/677075418' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:28:03.318 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:03 smithi028 ceph-mon[39220]: from='client.34994 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:28:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:03 smithi204 ceph-mon[32357]: 5.1d scrub starts 2024-09-10T18:28:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:03 smithi204 ceph-mon[32357]: 5.1d scrub ok 2024-09-10T18:28:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:03 smithi204 ceph-mon[32357]: 3.16 scrub starts 2024-09-10T18:28:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:03 smithi204 ceph-mon[32357]: 3.16 scrub ok 2024-09-10T18:28:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:03 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:28:03.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:03 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:28:03.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:03 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:03.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:03 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/677075418' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:28:03.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:03 smithi204 ceph-mon[32357]: from='client.34994 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:28:03.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:03.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:03.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:03.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:03.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:03.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:03.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:03.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:03.647 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:03.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:03.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:28:03.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:03.788 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:03.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:03.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:03.915 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:03.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:03.974 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:28:04.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:04.039 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:04.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:04.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:04.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:04.164 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:04.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:04.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:04.265 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:04.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:04.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:04.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:04.322 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:04 smithi028 ceph-mon[39220]: pgmap v292: 158 pgs: 158 active+clean; 419 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:04.322 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:04 smithi028 ceph-mon[39220]: 6.f scrub starts 2024-09-10T18:28:04.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:04 smithi028 ceph-mon[39220]: 6.f scrub ok 2024-09-10T18:28:04.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:04 smithi028 ceph-mon[39220]: 4.0 scrub starts 2024-09-10T18:28:04.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:04 smithi028 ceph-mon[39220]: 4.0 scrub ok 2024-09-10T18:28:04.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:04 smithi028 ceph-mon[39220]: from='client.34994 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T18:28:04.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:04 smithi028 ceph-mon[39220]: osdmap e188: 8 total, 8 up, 8 in 2024-09-10T18:28:04.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:04 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3678209963' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:28:04.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:04 smithi028 ceph-mon[29291]: pgmap v292: 158 pgs: 158 active+clean; 419 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:04.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:04 smithi028 ceph-mon[29291]: 6.f scrub starts 2024-09-10T18:28:04.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:04 smithi028 ceph-mon[29291]: 6.f scrub ok 2024-09-10T18:28:04.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:04 smithi028 ceph-mon[29291]: 4.0 scrub starts 2024-09-10T18:28:04.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:04 smithi028 ceph-mon[29291]: 4.0 scrub ok 2024-09-10T18:28:04.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:04 smithi028 ceph-mon[29291]: from='client.34994 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T18:28:04.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:04 smithi028 ceph-mon[29291]: osdmap e188: 8 total, 8 up, 8 in 2024-09-10T18:28:04.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:04 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3678209963' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:28:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-10T18:28:04.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:28:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:28:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:04 smithi204 ceph-mon[32357]: pgmap v292: 158 pgs: 158 active+clean; 419 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:04 smithi204 ceph-mon[32357]: 6.f scrub starts 2024-09-10T18:28:04.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:04 smithi204 ceph-mon[32357]: 6.f scrub ok 2024-09-10T18:28:04.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:04 smithi204 ceph-mon[32357]: 4.0 scrub starts 2024-09-10T18:28:04.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:04 smithi204 ceph-mon[32357]: 4.0 scrub ok 2024-09-10T18:28:04.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:04 smithi204 ceph-mon[32357]: from='client.34994 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T18:28:04.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:04 smithi204 ceph-mon[32357]: osdmap e188: 8 total, 8 up, 8 in 2024-09-10T18:28:04.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:04 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3678209963' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:28:04.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:04.419 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:04.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:04.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:04.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:04.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:04.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:04.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:04.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-10T18:28:04.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:04.670 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:04.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:04.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:04.794 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:04.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:04.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:04.919 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:04.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:04.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:05.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:05.043 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:05.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:05.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:05.145 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:05.171 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:05.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:05.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:05.273 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:05.298 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:05.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:05.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:05.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:05 smithi028 ceph-mon[39220]: 1.0 scrub starts 2024-09-10T18:28:05.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:05 smithi028 ceph-mon[39220]: 1.0 scrub ok 2024-09-10T18:28:05.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:05 smithi028 ceph-mon[39220]: 3.5 deep-scrub starts 2024-09-10T18:28:05.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:05 smithi028 ceph-mon[39220]: 3.5 deep-scrub ok 2024-09-10T18:28:05.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:05 smithi028 ceph-mon[39220]: 6.2 scrub starts 2024-09-10T18:28:05.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:05 smithi028 ceph-mon[39220]: 6.2 scrub ok 2024-09-10T18:28:05.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:05 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3678209963' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:28:05.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:05 smithi028 ceph-mon[39220]: osdmap e189: 8 total, 8 up, 8 in 2024-09-10T18:28:05.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:05 smithi028 ceph-mon[29291]: 1.0 scrub starts 2024-09-10T18:28:05.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:05 smithi028 ceph-mon[29291]: 1.0 scrub ok 2024-09-10T18:28:05.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:05 smithi028 ceph-mon[29291]: 3.5 deep-scrub starts 2024-09-10T18:28:05.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:05 smithi028 ceph-mon[29291]: 3.5 deep-scrub ok 2024-09-10T18:28:05.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:05 smithi028 ceph-mon[29291]: 6.2 scrub starts 2024-09-10T18:28:05.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:05 smithi028 ceph-mon[29291]: 6.2 scrub ok 2024-09-10T18:28:05.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:05 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3678209963' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:28:05.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:05 smithi028 ceph-mon[29291]: osdmap e189: 8 total, 8 up, 8 in 2024-09-10T18:28:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:05 smithi204 ceph-mon[32357]: 1.0 scrub starts 2024-09-10T18:28:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:05 smithi204 ceph-mon[32357]: 1.0 scrub ok 2024-09-10T18:28:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:05 smithi204 ceph-mon[32357]: 3.5 deep-scrub starts 2024-09-10T18:28:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:05 smithi204 ceph-mon[32357]: 3.5 deep-scrub ok 2024-09-10T18:28:05.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:05 smithi204 ceph-mon[32357]: 6.2 scrub starts 2024-09-10T18:28:05.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:05 smithi204 ceph-mon[32357]: 6.2 scrub ok 2024-09-10T18:28:05.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:05 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3678209963' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:28:05.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:05 smithi204 ceph-mon[32357]: osdmap e189: 8 total, 8 up, 8 in 2024-09-10T18:28:05.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:05.424 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:05.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:05.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:05.551 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:05.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:05.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:05.653 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:05.678 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:05.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:05.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:05.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:05.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:05.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:05.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:05.926 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:05.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:05.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:06.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:06.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:06.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:06.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:06.178 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:06.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:06.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:06.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:06.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:06.309 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:06.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:06 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:06 smithi204 ceph-mon[32357]: 2.0 scrub starts 2024-09-10T18:28:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:06 smithi204 ceph-mon[32357]: 2.0 scrub ok 2024-09-10T18:28:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:06 smithi204 ceph-mon[32357]: pgmap v295: 158 pgs: 158 active+clean; 367 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:06 smithi204 ceph-mon[32357]: 3.7 deep-scrub starts 2024-09-10T18:28:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:06 smithi204 ceph-mon[32357]: 3.7 deep-scrub ok 2024-09-10T18:28:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:06 smithi204 ceph-mon[32357]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:28:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:06 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:28:06.410 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:06.435 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:06.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:06.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:06.494 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:28:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:06 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:06 smithi028 ceph-mon[39220]: 2.0 scrub starts 2024-09-10T18:28:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:06 smithi028 ceph-mon[39220]: 2.0 scrub ok 2024-09-10T18:28:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:06 smithi028 ceph-mon[39220]: pgmap v295: 158 pgs: 158 active+clean; 367 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:06 smithi028 ceph-mon[39220]: 3.7 deep-scrub starts 2024-09-10T18:28:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:06 smithi028 ceph-mon[39220]: 3.7 deep-scrub ok 2024-09-10T18:28:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:06 smithi028 ceph-mon[39220]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:28:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:06 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:28:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:06 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:06 smithi028 ceph-mon[29291]: 2.0 scrub starts 2024-09-10T18:28:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:06 smithi028 ceph-mon[29291]: 2.0 scrub ok 2024-09-10T18:28:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:06 smithi028 ceph-mon[29291]: pgmap v295: 158 pgs: 158 active+clean; 367 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:06 smithi028 ceph-mon[29291]: 3.7 deep-scrub starts 2024-09-10T18:28:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:06 smithi028 ceph-mon[29291]: 3.7 deep-scrub ok 2024-09-10T18:28:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:06 smithi028 ceph-mon[29291]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:28:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:06 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:28:06.506 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:28:06.539 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:06.562 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:06.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:06.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:06.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:06.687 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:06.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:06.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:06.789 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:06.815 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:06.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:06.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:06.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:06.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:06.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:06.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:07.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:07.055 DEBUG:teuthology.orchestra.run.smithi028:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-10T18:28:07.057 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-10T18:28:07.058 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-10T18:28:07.067 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:07.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:07.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:07.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:hints = 1 2024-09-10T18:28:07.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-10T18:28:07.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object prefix: benchmark_data_smithi028_109242 2024-09-10T18:28:07.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:28:07.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 0 0 0 0 0 0 - 0 2024-09-10T18:28:07.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:07.212 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:07.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:07.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:07.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:07 smithi204 ceph-mon[32357]: 3.19 scrub starts 2024-09-10T18:28:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:07 smithi204 ceph-mon[32357]: 3.19 scrub ok 2024-09-10T18:28:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:07 smithi204 ceph-mon[32357]: 5.7 deep-scrub starts 2024-09-10T18:28:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:07 smithi204 ceph-mon[32357]: 5.7 deep-scrub ok 2024-09-10T18:28:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:07 smithi204 ceph-mon[32357]: 3.9 scrub starts 2024-09-10T18:28:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:07 smithi204 ceph-mon[32357]: 3.9 scrub ok 2024-09-10T18:28:07.356 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:07.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:07.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:07.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:07 smithi028 ceph-mon[29291]: 3.19 scrub starts 2024-09-10T18:28:07.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:07 smithi028 ceph-mon[29291]: 3.19 scrub ok 2024-09-10T18:28:07.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:07 smithi028 ceph-mon[29291]: 5.7 deep-scrub starts 2024-09-10T18:28:07.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:07 smithi028 ceph-mon[29291]: 5.7 deep-scrub ok 2024-09-10T18:28:07.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:07 smithi028 ceph-mon[29291]: 3.9 scrub starts 2024-09-10T18:28:07.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:07 smithi028 ceph-mon[29291]: 3.9 scrub ok 2024-09-10T18:28:07.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:07 smithi028 ceph-mon[39220]: 3.19 scrub starts 2024-09-10T18:28:07.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:07 smithi028 ceph-mon[39220]: 3.19 scrub ok 2024-09-10T18:28:07.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:07 smithi028 ceph-mon[39220]: 5.7 deep-scrub starts 2024-09-10T18:28:07.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:07 smithi028 ceph-mon[39220]: 5.7 deep-scrub ok 2024-09-10T18:28:07.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:07 smithi028 ceph-mon[39220]: 3.9 scrub starts 2024-09-10T18:28:07.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:07 smithi028 ceph-mon[39220]: 3.9 scrub ok 2024-09-10T18:28:07.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:07.489 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:07.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:07.581 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:28:07.581 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:28:07.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15117.0:1743 8.7 e189/180 8:ed8f737c:::benchmark_data_smithi028_109242_object1742:head v 189'114, mlcod=189'112)", 2024-09-10T18:28:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:28:07.576002+0000", 2024-09-10T18:28:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0021886129999999998, 2024-09-10T18:28:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002215547, 2024-09-10T18:28:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:28:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.576002+0000", 2024-09-10T18:28:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:07.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.576002+0000", 2024-09-10T18:28:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.576001+0000", 2024-09-10T18:28:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:28:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.576025+0000", 2024-09-10T18:28:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4374000000000001e-05 2024-09-10T18:28:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.576044+0000", 2024-09-10T18:28:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8987999999999998e-05 2024-09-10T18:28:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.576050+0000", 2024-09-10T18:28:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2389999999999999e-06 2024-09-10T18:28:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.576786+0000", 2024-09-10T18:28:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00073569 2024-09-10T18:28:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.576810+0000", 2024-09-10T18:28:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3586000000000001e-05 2024-09-10T18:28:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15117.0:1750 8.8 e189/180 8:15cb38aa:::benchmark_data_smithi028_109242_object1749:head v 189'110, mlcod=189'108)", 2024-09-10T18:28:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:28:07.578132+0000", 2024-09-10T18:28:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.8044000000000002e-05, 2024-09-10T18:28:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015313699999999999, 2024-09-10T18:28:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:28:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578132+0000", 2024-09-10T18:28:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578132+0000", 2024-09-10T18:28:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578131+0000", 2024-09-10T18:28:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:28:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578169+0000", 2024-09-10T18:28:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8050999999999999e-05 2024-09-10T18:28:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578180+0000", 2024-09-10T18:28:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0540000000000001e-05 2024-09-10T18:28:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578186+0000", 2024-09-10T18:28:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0719999999999996e-06 2024-09-10T18:28:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578225+0000", 2024-09-10T18:28:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9446000000000001e-05 2024-09-10T18:28:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578246+0000", 2024-09-10T18:28:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.048e-05 2024-09-10T18:28:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15117.0:1747 8.7 e189/180 8:e8329bed:::benchmark_data_smithi028_109242_object1746:head v 189'115, mlcod=189'112)", 2024-09-10T18:28:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:28:07.578208+0000", 2024-09-10T18:28:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4294967295.9999819, 2024-09-10T18:28:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014145, 2024-09-10T18:28:07.589 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:28:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578208+0000", 2024-09-10T18:28:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578208+0000", 2024-09-10T18:28:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578207+0000", 2024-09-10T18:28:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:28:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578232+0000", 2024-09-10T18:28:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4454e-05 2024-09-10T18:28:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578239+0000", 2024-09-10T18:28:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2139999999999999e-06 2024-09-10T18:28:07.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578243+0000", 2024-09-10T18:28:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3819999999999997e-06 2024-09-10T18:28:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578283+0000", 2024-09-10T18:28:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9458000000000001e-05 2024-09-10T18:28:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:07.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578302+0000", 2024-09-10T18:28:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9400999999999999e-05 2024-09-10T18:28:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15117.0:1745 8.f 8:fa471d7f:::benchmark_data_smithi028_109242_object1744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-10T18:28:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:28:07.574516+0000", 2024-09-10T18:28:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.003674886, 2024-09-10T18:28:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003896418, 2024-09-10T18:28:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:28:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:07.593 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15117", 2024-09-10T18:28:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/121353060", 2024-09-10T18:28:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1745 2024-09-10T18:28:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.574516+0000", 2024-09-10T18:28:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.574516+0000", 2024-09-10T18:28:07.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.574514+0000", 2024-09-10T18:28:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:28:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.574539+0000", 2024-09-10T18:28:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5503999999999998e-05 2024-09-10T18:28:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:07.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.574555+0000", 2024-09-10T18:28:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5443000000000001e-05 2024-09-10T18:28:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.574562+0000", 2024-09-10T18:28:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1779999999999997e-06 2024-09-10T18:28:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.574639+0000", 2024-09-10T18:28:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6772000000000003e-05 2024-09-10T18:28:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.574765+0000", 2024-09-10T18:28:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000126253 2024-09-10T18:28:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:28:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.574913+0000", 2024-09-10T18:28:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014780800000000001 2024-09-10T18:28:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.575492+0000", 2024-09-10T18:28:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057889400000000004 2024-09-10T18:28:07.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578319+0000", 2024-09-10T18:28:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028277620000000002 2024-09-10T18:28:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15117.0:1746 8.b 8:d41e075e:::benchmark_data_smithi028_109242_object1745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-10T18:28:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:28:07.575994+0000", 2024-09-10T18:28:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0021969730000000001, 2024-09-10T18:28:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025103299999999999, 2024-09-10T18:28:07.598 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:28:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15117", 2024-09-10T18:28:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/121353060", 2024-09-10T18:28:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1746 2024-09-10T18:28:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.575994+0000", 2024-09-10T18:28:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.575994+0000", 2024-09-10T18:28:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.575992+0000", 2024-09-10T18:28:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:28:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.576021+0000", 2024-09-10T18:28:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9513999999999999e-05 2024-09-10T18:28:07.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.576039+0000", 2024-09-10T18:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7246000000000001e-05 2024-09-10T18:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.576046+0000", 2024-09-10T18:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1969999999999999e-06 2024-09-10T18:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.576088+0000", 2024-09-10T18:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2518999999999997e-05 2024-09-10T18:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.576208+0000", 2024-09-10T18:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011980399999999999 2024-09-10T18:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.576343+0000", 2024-09-10T18:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013484900000000001 2024-09-10T18:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.577767+0000", 2024-09-10T18:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0014237010000000001 2024-09-10T18:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:28:07.578055+0000", 2024-09-10T18:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028863100000000002 2024-09-10T18:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:07.604 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:07.604 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:07.604 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:28:07.604 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 5 2024-09-10T18:28:07.604 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:28:07.604 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:07.632 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:07.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:07.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:07.734 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:07.766 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:07.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:07.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:07.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:07.907 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:07.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:07.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:08.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:08.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:08.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:08.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1 16 3695 3679 229.925 229.938 0.00798486 0.00432608 2024-09-10T18:28:08.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:08.182 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:08.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:08.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:08.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:08.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:08.327 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:08.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:08 smithi204 ceph-mon[32357]: 3.f scrub starts 2024-09-10T18:28:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:08 smithi204 ceph-mon[32357]: 3.f scrub ok 2024-09-10T18:28:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:08 smithi204 ceph-mon[32357]: pgmap v296: 158 pgs: 158 active+clean; 367 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:08 smithi204 ceph-mon[32357]: 3.17 scrub starts 2024-09-10T18:28:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:08 smithi204 ceph-mon[32357]: 3.17 scrub ok 2024-09-10T18:28:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:08 smithi204 ceph-mon[32357]: 6.6 scrub starts 2024-09-10T18:28:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:08 smithi204 ceph-mon[32357]: 6.6 scrub ok 2024-09-10T18:28:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:08 smithi204 ceph-mon[32357]: 1.5 scrub starts 2024-09-10T18:28:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:08 smithi204 ceph-mon[32357]: 1.5 scrub ok 2024-09-10T18:28:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:08 smithi204 ceph-mon[32357]: 5.11 scrub starts 2024-09-10T18:28:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:08 smithi204 ceph-mon[32357]: 5.11 scrub ok 2024-09-10T18:28:08.414 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[39220]: 3.f scrub starts 2024-09-10T18:28:08.414 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[39220]: 3.f scrub ok 2024-09-10T18:28:08.414 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[39220]: pgmap v296: 158 pgs: 158 active+clean; 367 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:08.414 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[39220]: 3.17 scrub starts 2024-09-10T18:28:08.414 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[39220]: 3.17 scrub ok 2024-09-10T18:28:08.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[39220]: 6.6 scrub starts 2024-09-10T18:28:08.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[39220]: 6.6 scrub ok 2024-09-10T18:28:08.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[39220]: 1.5 scrub starts 2024-09-10T18:28:08.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[39220]: 1.5 scrub ok 2024-09-10T18:28:08.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[39220]: 5.11 scrub starts 2024-09-10T18:28:08.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[39220]: 5.11 scrub ok 2024-09-10T18:28:08.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[29291]: 3.f scrub starts 2024-09-10T18:28:08.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[29291]: 3.f scrub ok 2024-09-10T18:28:08.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[29291]: pgmap v296: 158 pgs: 158 active+clean; 367 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:08.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[29291]: 3.17 scrub starts 2024-09-10T18:28:08.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[29291]: 3.17 scrub ok 2024-09-10T18:28:08.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[29291]: 6.6 scrub starts 2024-09-10T18:28:08.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[29291]: 6.6 scrub ok 2024-09-10T18:28:08.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[29291]: 1.5 scrub starts 2024-09-10T18:28:08.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[29291]: 1.5 scrub ok 2024-09-10T18:28:08.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[29291]: 5.11 scrub starts 2024-09-10T18:28:08.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:08 smithi028 ceph-mon[29291]: 5.11 scrub ok 2024-09-10T18:28:08.428 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:08.458 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:08.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:08.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:08.560 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:08.591 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:08.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:08.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:08.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:08.733 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:08.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:08.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:08.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:08.872 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:08.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:08.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:08.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:09.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:09.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:09.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-10T18:28:09.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2 16 6871 6855 214.19 198.5 0.00401954 0.0046589 2024-09-10T18:28:09.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:09.146 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:09.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:09.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:09.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:09.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:09.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[29291]: 3.15 scrub starts 2024-09-10T18:28:09.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[29291]: 3.15 scrub ok 2024-09-10T18:28:09.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[29291]: 6.19 deep-scrub starts 2024-09-10T18:28:09.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[29291]: 6.19 deep-scrub ok 2024-09-10T18:28:09.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[29291]: 5.0 scrub starts 2024-09-10T18:28:09.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[29291]: 5.0 scrub ok 2024-09-10T18:28:09.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3815154239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:28:09.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:09.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:28:09.329 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[39220]: 3.15 scrub starts 2024-09-10T18:28:09.329 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[39220]: 3.15 scrub ok 2024-09-10T18:28:09.329 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[39220]: 6.19 deep-scrub starts 2024-09-10T18:28:09.329 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[39220]: 6.19 deep-scrub ok 2024-09-10T18:28:09.329 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[39220]: 5.0 scrub starts 2024-09-10T18:28:09.329 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[39220]: 5.0 scrub ok 2024-09-10T18:28:09.329 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3815154239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:28:09.329 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:09.329 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:28:09.329 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:28:09.330 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:28:09.330 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:28:09.330 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:28:09.330 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:28:09.330 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:28:09.330 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:28:09.330 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:28:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:28:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:28:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:28:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:28:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:28:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:28:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:28:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:28:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:09 smithi204 ceph-mon[32357]: 3.15 scrub starts 2024-09-10T18:28:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:09 smithi204 ceph-mon[32357]: 3.15 scrub ok 2024-09-10T18:28:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:09 smithi204 ceph-mon[32357]: 6.19 deep-scrub starts 2024-09-10T18:28:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:09 smithi204 ceph-mon[32357]: 6.19 deep-scrub ok 2024-09-10T18:28:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:09 smithi204 ceph-mon[32357]: 5.0 scrub starts 2024-09-10T18:28:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:09 smithi204 ceph-mon[32357]: 5.0 scrub ok 2024-09-10T18:28:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:09 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3815154239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:28:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:28:09.396 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:09.427 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:09.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:09.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:09.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:09.559 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:09.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:09.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:09.661 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:09.684 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:09.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:09.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:09.785 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:09.814 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:09.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:09.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:09.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:09.957 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:09.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:09.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:10.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:10.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:10.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:10.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3 15 9809 9794 204.017 183.688 0.00516621 0.00489241 2024-09-10T18:28:10.198 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:10.230 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:10.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:10.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:10.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:10 smithi204 ceph-mon[32357]: 6.8 scrub starts 2024-09-10T18:28:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:10 smithi204 ceph-mon[32357]: 6.8 scrub ok 2024-09-10T18:28:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:10 smithi204 ceph-mon[32357]: pgmap v297: 158 pgs: 158 active+clean; 367 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:10 smithi204 ceph-mon[32357]: 4.16 deep-scrub starts 2024-09-10T18:28:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:10 smithi204 ceph-mon[32357]: 4.16 deep-scrub ok 2024-09-10T18:28:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:10 smithi204 ceph-mon[32357]: 5.6 scrub starts 2024-09-10T18:28:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:10 smithi204 ceph-mon[32357]: 5.6 scrub ok 2024-09-10T18:28:10.371 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:10.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:10.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:10.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:10.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:10 smithi028 ceph-mon[29291]: 6.8 scrub starts 2024-09-10T18:28:10.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:10 smithi028 ceph-mon[29291]: 6.8 scrub ok 2024-09-10T18:28:10.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:10 smithi028 ceph-mon[29291]: pgmap v297: 158 pgs: 158 active+clean; 367 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:10.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:10 smithi028 ceph-mon[29291]: 4.16 deep-scrub starts 2024-09-10T18:28:10.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:10 smithi028 ceph-mon[29291]: 4.16 deep-scrub ok 2024-09-10T18:28:10.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:10 smithi028 ceph-mon[29291]: 5.6 scrub starts 2024-09-10T18:28:10.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:10 smithi028 ceph-mon[29291]: 5.6 scrub ok 2024-09-10T18:28:10.478 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:10 smithi028 ceph-mon[39220]: 6.8 scrub starts 2024-09-10T18:28:10.478 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:10 smithi028 ceph-mon[39220]: 6.8 scrub ok 2024-09-10T18:28:10.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:10 smithi028 ceph-mon[39220]: pgmap v297: 158 pgs: 158 active+clean; 367 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:28:10.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:10 smithi028 ceph-mon[39220]: 4.16 deep-scrub starts 2024-09-10T18:28:10.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:10 smithi028 ceph-mon[39220]: 4.16 deep-scrub ok 2024-09-10T18:28:10.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:10 smithi028 ceph-mon[39220]: 5.6 scrub starts 2024-09-10T18:28:10.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:10 smithi028 ceph-mon[39220]: 5.6 scrub ok 2024-09-10T18:28:10.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:10.518 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:10.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:10.619 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:10.648 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:10.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:10.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:10.750 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:10.779 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:10.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:10.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:10.881 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:10.910 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:10.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:10.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:11.012 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:11.044 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:11.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:11.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:11.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4 16 12939 12923 201.897 195.562 0.00352442 0.00494535 2024-09-10T18:28:11.146 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:11.177 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:11.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:11.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:11.279 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:11.309 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:11.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:11.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:11.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:11 smithi028 ceph-mon[39220]: 3.e scrub starts 2024-09-10T18:28:11.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:11 smithi028 ceph-mon[39220]: 3.e scrub ok 2024-09-10T18:28:11.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:11 smithi028 ceph-mon[39220]: 6.7 deep-scrub starts 2024-09-10T18:28:11.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:11 smithi028 ceph-mon[39220]: 6.7 deep-scrub ok 2024-09-10T18:28:11.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:11 smithi028 ceph-mon[39220]: 5.b scrub starts 2024-09-10T18:28:11.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:11 smithi028 ceph-mon[39220]: 5.b scrub ok 2024-09-10T18:28:11.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:11 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:28:11.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:11 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:28:11.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:11 smithi028 ceph-mon[29291]: 3.e scrub starts 2024-09-10T18:28:11.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:11 smithi028 ceph-mon[29291]: 3.e scrub ok 2024-09-10T18:28:11.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:11 smithi028 ceph-mon[29291]: 6.7 deep-scrub starts 2024-09-10T18:28:11.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:11 smithi028 ceph-mon[29291]: 6.7 deep-scrub ok 2024-09-10T18:28:11.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:11 smithi028 ceph-mon[29291]: 5.b scrub starts 2024-09-10T18:28:11.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:11 smithi028 ceph-mon[29291]: 5.b scrub ok 2024-09-10T18:28:11.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:11 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:28:11.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:11 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:28:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:11 smithi204 ceph-mon[32357]: 3.e scrub starts 2024-09-10T18:28:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:11 smithi204 ceph-mon[32357]: 3.e scrub ok 2024-09-10T18:28:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:11 smithi204 ceph-mon[32357]: 6.7 deep-scrub starts 2024-09-10T18:28:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:11 smithi204 ceph-mon[32357]: 6.7 deep-scrub ok 2024-09-10T18:28:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:11 smithi204 ceph-mon[32357]: 5.b scrub starts 2024-09-10T18:28:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:11 smithi204 ceph-mon[32357]: 5.b scrub ok 2024-09-10T18:28:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:11 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:28:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:11 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:28:11.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:11.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:11.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:11.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:11.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:11.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:11.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:11.629 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:28:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:28:11] "GET /metrics HTTP/1.1" 200 223904 "" "Prometheus/2.43.0" 2024-09-10T18:28:11.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:11.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:11.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:11.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:11.862 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:11.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:11.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:11.964 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:11.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:11.995 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:11.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:12.097 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:12.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5 16 15990 15974 199.652 190.688 0.00288697 0.00500207 2024-09-10T18:28:12.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:12.132 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:12.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:12.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:12.271 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:12.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:12.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:12.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:28:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:28:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:28:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:12 smithi204 ceph-mon[32357]: 5.1a scrub starts 2024-09-10T18:28:12.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:12 smithi204 ceph-mon[32357]: 5.1a scrub ok 2024-09-10T18:28:12.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:12 smithi204 ceph-mon[32357]: pgmap v298: 158 pgs: 158 active+clean; 614 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 77 MiB/s wr, 1.22k op/s 2024-09-10T18:28:12.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:12 smithi204 ceph-mon[32357]: 5.d deep-scrub starts 2024-09-10T18:28:12.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:12 smithi204 ceph-mon[32357]: 5.d deep-scrub ok 2024-09-10T18:28:12.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:12 smithi204 ceph-mon[32357]: 6.18 scrub starts 2024-09-10T18:28:12.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:12 smithi204 ceph-mon[32357]: 6.18 scrub ok 2024-09-10T18:28:12.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:12 smithi204 ceph-mon[32357]: osdmap e190: 8 total, 8 up, 8 in 2024-09-10T18:28:12.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:12.411 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:12.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:12.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:12 smithi028 ceph-mon[39220]: 5.1a scrub starts 2024-09-10T18:28:12.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:12 smithi028 ceph-mon[39220]: 5.1a scrub ok 2024-09-10T18:28:12.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:12 smithi028 ceph-mon[39220]: pgmap v298: 158 pgs: 158 active+clean; 614 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 77 MiB/s wr, 1.22k op/s 2024-09-10T18:28:12.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:12 smithi028 ceph-mon[39220]: 5.d deep-scrub starts 2024-09-10T18:28:12.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:12 smithi028 ceph-mon[39220]: 5.d deep-scrub ok 2024-09-10T18:28:12.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:12 smithi028 ceph-mon[39220]: 6.18 scrub starts 2024-09-10T18:28:12.420 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:12 smithi028 ceph-mon[39220]: 6.18 scrub ok 2024-09-10T18:28:12.420 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:12 smithi028 ceph-mon[39220]: osdmap e190: 8 total, 8 up, 8 in 2024-09-10T18:28:12.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:12 smithi028 ceph-mon[29291]: 5.1a scrub starts 2024-09-10T18:28:12.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:12 smithi028 ceph-mon[29291]: 5.1a scrub ok 2024-09-10T18:28:12.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:12 smithi028 ceph-mon[29291]: pgmap v298: 158 pgs: 158 active+clean; 614 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 77 MiB/s wr, 1.22k op/s 2024-09-10T18:28:12.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:12 smithi028 ceph-mon[29291]: 5.d deep-scrub starts 2024-09-10T18:28:12.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:12 smithi028 ceph-mon[29291]: 5.d deep-scrub ok 2024-09-10T18:28:12.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:12 smithi028 ceph-mon[29291]: 6.18 scrub starts 2024-09-10T18:28:12.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:12 smithi028 ceph-mon[29291]: 6.18 scrub ok 2024-09-10T18:28:12.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:12 smithi028 ceph-mon[29291]: osdmap e190: 8 total, 8 up, 8 in 2024-09-10T18:28:12.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:12.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:28:12.570 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:12.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:12.671 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:12.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:12.704 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:12.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:12.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:12.848 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:12.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:12.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:12.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:12.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:13.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:13.103 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 6 16 18656 18640 194.145 166.625 0.00405144 0.00514374 2024-09-10T18:28:13.125 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:13.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:13.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:13.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:13.257 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:13.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:13.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:13.280 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:28:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:13 smithi204 ceph-mon[32357]: 5.4 scrub starts 2024-09-10T18:28:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:13 smithi204 ceph-mon[32357]: 5.4 scrub ok 2024-09-10T18:28:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:13 smithi204 ceph-mon[32357]: 5.8 scrub starts 2024-09-10T18:28:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:13 smithi204 ceph-mon[32357]: 5.8 scrub ok 2024-09-10T18:28:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:13 smithi204 ceph-mon[32357]: 1.4 scrub starts 2024-09-10T18:28:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:13 smithi204 ceph-mon[32357]: 1.4 scrub ok 2024-09-10T18:28:13.359 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:13.389 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:13.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:13.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:13.491 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:13 smithi028 ceph-mon[29291]: 5.4 scrub starts 2024-09-10T18:28:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:13 smithi028 ceph-mon[29291]: 5.4 scrub ok 2024-09-10T18:28:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:13 smithi028 ceph-mon[29291]: 5.8 scrub starts 2024-09-10T18:28:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:13 smithi028 ceph-mon[29291]: 5.8 scrub ok 2024-09-10T18:28:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:13 smithi028 ceph-mon[29291]: 1.4 scrub starts 2024-09-10T18:28:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:13 smithi028 ceph-mon[29291]: 1.4 scrub ok 2024-09-10T18:28:13.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:13 smithi028 ceph-mon[39220]: 5.4 scrub starts 2024-09-10T18:28:13.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:13 smithi028 ceph-mon[39220]: 5.4 scrub ok 2024-09-10T18:28:13.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:13 smithi028 ceph-mon[39220]: 5.8 scrub starts 2024-09-10T18:28:13.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:13 smithi028 ceph-mon[39220]: 5.8 scrub ok 2024-09-10T18:28:13.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:13 smithi028 ceph-mon[39220]: 1.4 scrub starts 2024-09-10T18:28:13.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:13 smithi028 ceph-mon[39220]: 1.4 scrub ok 2024-09-10T18:28:13.521 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:13.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:13.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:13.622 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:13.653 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:13.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:13.755 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:13.787 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:13.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:13.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:13.920 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:13.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:13.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:14.022 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:14.054 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:14.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:14.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:14.104 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 7 16 21152 21136 188.694 156 0.0127863 0.00529071 2024-09-10T18:28:14.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:14.179 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:14.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:14.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:14.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:14 smithi028 ceph-mon[39220]: pgmap v300: 158 pgs: 158 active+clean; 614 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-10T18:28:14.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:14 smithi028 ceph-mon[39220]: 4.1d scrub starts 2024-09-10T18:28:14.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:14 smithi028 ceph-mon[39220]: 4.1d scrub ok 2024-09-10T18:28:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:14 smithi028 ceph-mon[29291]: pgmap v300: 158 pgs: 158 active+clean; 614 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-10T18:28:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:14 smithi028 ceph-mon[29291]: 4.1d scrub starts 2024-09-10T18:28:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:14 smithi028 ceph-mon[29291]: 4.1d scrub ok 2024-09-10T18:28:14.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:14.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:14.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:14.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:14 smithi204 ceph-mon[32357]: pgmap v300: 158 pgs: 158 active+clean; 614 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-10T18:28:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:14 smithi204 ceph-mon[32357]: 4.1d scrub starts 2024-09-10T18:28:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:14 smithi204 ceph-mon[32357]: 4.1d scrub ok 2024-09-10T18:28:14.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:28:14 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:28:14.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:14.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:14.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:14.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:14.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:14.591 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:14.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:14.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:14.693 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:14.723 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:14.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:14.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:14.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:14.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:14.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:14.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:14.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:15.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:15.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:15.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:15.104 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 8 16 23410 23394 182.746 141.125 0.00686394 0.00546605 2024-09-10T18:28:15.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:15.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:15.134 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:15.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:15.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[29291]: 3.d scrub starts 2024-09-10T18:28:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[29291]: 3.d scrub ok 2024-09-10T18:28:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[29291]: 5.1b scrub starts 2024-09-10T18:28:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[29291]: 5.1b scrub ok 2024-09-10T18:28:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:28:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:28:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[29291]: 3.1f scrub starts 2024-09-10T18:28:15.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[29291]: 3.1f scrub ok 2024-09-10T18:28:15.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[29291]: osdmap e191: 8 total, 8 up, 8 in 2024-09-10T18:28:15.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[29291]: 3.b scrub starts 2024-09-10T18:28:15.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[29291]: 3.b scrub ok 2024-09-10T18:28:15.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[29291]: 5.2 scrub starts 2024-09-10T18:28:15.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[29291]: 5.2 scrub ok 2024-09-10T18:28:15.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4077661433' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:28:15.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[29291]: from='client.25042 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:28:15.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[39220]: 3.d scrub starts 2024-09-10T18:28:15.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[39220]: 3.d scrub ok 2024-09-10T18:28:15.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[39220]: 5.1b scrub starts 2024-09-10T18:28:15.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[39220]: 5.1b scrub ok 2024-09-10T18:28:15.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:28:15.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:28:15.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[39220]: 3.1f scrub starts 2024-09-10T18:28:15.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[39220]: 3.1f scrub ok 2024-09-10T18:28:15.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[39220]: osdmap e191: 8 total, 8 up, 8 in 2024-09-10T18:28:15.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[39220]: 3.b scrub starts 2024-09-10T18:28:15.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[39220]: 3.b scrub ok 2024-09-10T18:28:15.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[39220]: 5.2 scrub starts 2024-09-10T18:28:15.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[39220]: 5.2 scrub ok 2024-09-10T18:28:15.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4077661433' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:28:15.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:15 smithi028 ceph-mon[39220]: from='client.25042 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:28:15.274 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:15.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:15.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:15.376 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:15.407 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:15.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:15.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:15.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:15.536 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:15.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:15.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:15 smithi204 ceph-mon[32357]: 3.d scrub starts 2024-09-10T18:28:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:15 smithi204 ceph-mon[32357]: 3.d scrub ok 2024-09-10T18:28:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:15 smithi204 ceph-mon[32357]: 5.1b scrub starts 2024-09-10T18:28:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:15 smithi204 ceph-mon[32357]: 5.1b scrub ok 2024-09-10T18:28:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:15 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:28:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:15 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:28:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:15 smithi204 ceph-mon[32357]: 3.1f scrub starts 2024-09-10T18:28:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:15 smithi204 ceph-mon[32357]: 3.1f scrub ok 2024-09-10T18:28:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:15 smithi204 ceph-mon[32357]: osdmap e191: 8 total, 8 up, 8 in 2024-09-10T18:28:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:15 smithi204 ceph-mon[32357]: 3.b scrub starts 2024-09-10T18:28:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:15 smithi204 ceph-mon[32357]: 3.b scrub ok 2024-09-10T18:28:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:15 smithi204 ceph-mon[32357]: 5.2 scrub starts 2024-09-10T18:28:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:15 smithi204 ceph-mon[32357]: 5.2 scrub ok 2024-09-10T18:28:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:15 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4077661433' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:28:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:15 smithi204 ceph-mon[32357]: from='client.25042 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:28:15.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:15.668 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:15.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:15.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:15.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:15.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:15.802 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:15.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:15.845 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.6 to 1 (10000) 2024-09-10T18:28:15.902 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:15.932 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:15.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:15.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:16.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:16.074 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:16.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:16.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:16.104 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 9 16 25769 25753 178.822 147.438 0.00604545 0.0055867 2024-09-10T18:28:16.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:16.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:16.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:16.321 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:16.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:16.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:16.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:16.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:16.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[39220]: 3.4 scrub starts 2024-09-10T18:28:16.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[39220]: 3.4 scrub ok 2024-09-10T18:28:16.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[39220]: pgmap v302: 158 pgs: 158 active+clean; 830 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-10T18:28:16.393 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[39220]: from='client.25042 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T18:28:16.393 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[39220]: osdmap e192: 8 total, 8 up, 8 in 2024-09-10T18:28:16.393 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[39220]: 4.19 scrub starts 2024-09-10T18:28:16.393 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[39220]: 4.19 scrub ok 2024-09-10T18:28:16.393 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[39220]: osdmap e193: 8 total, 8 up, 8 in 2024-09-10T18:28:16.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:16.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[29291]: 3.4 scrub starts 2024-09-10T18:28:16.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[29291]: 3.4 scrub ok 2024-09-10T18:28:16.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[29291]: pgmap v302: 158 pgs: 158 active+clean; 830 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-10T18:28:16.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[29291]: from='client.25042 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T18:28:16.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[29291]: osdmap e192: 8 total, 8 up, 8 in 2024-09-10T18:28:16.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[29291]: 4.19 scrub starts 2024-09-10T18:28:16.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[29291]: 4.19 scrub ok 2024-09-10T18:28:16.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:16 smithi028 ceph-mon[29291]: osdmap e193: 8 total, 8 up, 8 in 2024-09-10T18:28:16.451 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:16.482 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:16.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:16.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:16.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:16 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:16 smithi204 ceph-mon[32357]: 3.4 scrub starts 2024-09-10T18:28:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:16 smithi204 ceph-mon[32357]: 3.4 scrub ok 2024-09-10T18:28:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:16 smithi204 ceph-mon[32357]: pgmap v302: 158 pgs: 158 active+clean; 830 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-10T18:28:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:16 smithi204 ceph-mon[32357]: from='client.25042 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T18:28:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:16 smithi204 ceph-mon[32357]: osdmap e192: 8 total, 8 up, 8 in 2024-09-10T18:28:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:16 smithi204 ceph-mon[32357]: 4.19 scrub starts 2024-09-10T18:28:16.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:16 smithi204 ceph-mon[32357]: 4.19 scrub ok 2024-09-10T18:28:16.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:16 smithi204 ceph-mon[32357]: osdmap e193: 8 total, 8 up, 8 in 2024-09-10T18:28:16.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:16.628 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:16.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:16.729 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:16.757 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:16.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:16.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:16.859 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:16.890 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:16.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:16.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:16.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:17.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:17.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:17.104 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 10 16 28047 28031 175.176 142.375 0.00503933 0.00570283 2024-09-10T18:28:17.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:17.141 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:28:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:17.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:17.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:17.271 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:17.301 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:17.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:17.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:17.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:17 smithi028 ceph-mon[29291]: 6.1 scrub starts 2024-09-10T18:28:17.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:17 smithi028 ceph-mon[29291]: 6.1 scrub ok 2024-09-10T18:28:17.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:17 smithi028 ceph-mon[29291]: 5.15 scrub starts 2024-09-10T18:28:17.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:17 smithi028 ceph-mon[29291]: 5.15 scrub ok 2024-09-10T18:28:17.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:17 smithi028 ceph-mon[29291]: 6.e deep-scrub starts 2024-09-10T18:28:17.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:17 smithi028 ceph-mon[29291]: 6.e deep-scrub ok 2024-09-10T18:28:17.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:17 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2012825191' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:28:17.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:17 smithi028 ceph-mon[39220]: 6.1 scrub starts 2024-09-10T18:28:17.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:17 smithi028 ceph-mon[39220]: 6.1 scrub ok 2024-09-10T18:28:17.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:17 smithi028 ceph-mon[39220]: 5.15 scrub starts 2024-09-10T18:28:17.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:17 smithi028 ceph-mon[39220]: 5.15 scrub ok 2024-09-10T18:28:17.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:17 smithi028 ceph-mon[39220]: 6.e deep-scrub starts 2024-09-10T18:28:17.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:17 smithi028 ceph-mon[39220]: 6.e deep-scrub ok 2024-09-10T18:28:17.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:17 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2012825191' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:28:17.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:17.451 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:17.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:17.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:17.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:17.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:17.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:17 smithi204 ceph-mon[32357]: 6.1 scrub starts 2024-09-10T18:28:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:17 smithi204 ceph-mon[32357]: 6.1 scrub ok 2024-09-10T18:28:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:17 smithi204 ceph-mon[32357]: 5.15 scrub starts 2024-09-10T18:28:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:17 smithi204 ceph-mon[32357]: 5.15 scrub ok 2024-09-10T18:28:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:17 smithi204 ceph-mon[32357]: 6.e deep-scrub starts 2024-09-10T18:28:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:17 smithi204 ceph-mon[32357]: 6.e deep-scrub ok 2024-09-10T18:28:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:17 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2012825191' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:28:17.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:17.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:17.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:17.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:17.876 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:17.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:17.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:18.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:18.023 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:18.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:18.104 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 11 16 30023 30007 170.477 123.5 0.00264707 0.00585693 2024-09-10T18:28:18.125 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:18.153 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:18.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:18.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:18.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:18.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:18.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:18.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:18.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:18.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:18.436 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:18.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:18.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:18 smithi028 ceph-mon[39220]: 3.0 scrub starts 2024-09-10T18:28:18.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:18 smithi028 ceph-mon[39220]: 3.0 scrub ok 2024-09-10T18:28:18.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:18 smithi028 ceph-mon[39220]: pgmap v305: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-09-10T18:28:18.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:18 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:28:18.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:18 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2012825191' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:28:18.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:18 smithi028 ceph-mon[39220]: osdmap e194: 8 total, 8 up, 8 in 2024-09-10T18:28:18.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:18 smithi028 ceph-mon[29291]: 3.0 scrub starts 2024-09-10T18:28:18.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:18 smithi028 ceph-mon[29291]: 3.0 scrub ok 2024-09-10T18:28:18.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:18 smithi028 ceph-mon[29291]: pgmap v305: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-09-10T18:28:18.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:18 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:28:18.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:18 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2012825191' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:28:18.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:18 smithi028 ceph-mon[29291]: osdmap e194: 8 total, 8 up, 8 in 2024-09-10T18:28:18.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:18.558 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:18.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:18 smithi204 ceph-mon[32357]: 3.0 scrub starts 2024-09-10T18:28:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:18 smithi204 ceph-mon[32357]: 3.0 scrub ok 2024-09-10T18:28:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:18 smithi204 ceph-mon[32357]: pgmap v305: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-09-10T18:28:18.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:18 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:28:18.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:18 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2012825191' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:28:18.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:18 smithi204 ceph-mon[32357]: osdmap e194: 8 total, 8 up, 8 in 2024-09-10T18:28:18.661 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:18.691 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:18.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:18.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:18.793 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:18.822 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:18.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:18.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:18.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:18.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:18.953 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:18.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:19.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:19.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:19.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:19.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:19.104 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 12 16 31764 31748 165.338 108.812 0.00495545 0.00604292 2024-09-10T18:28:19.200 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:19.220 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:19.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:19.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:19.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:19.360 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:19.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:19.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:19.462 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:19.495 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:19.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:19.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:19.597 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:19.618 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:19.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:19.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:19.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:19.763 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:19.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:19.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:19.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:19.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:19.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:20.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:20.042 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:20.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:20.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:20.104 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 13 15 33489 33474 160.917 107.875 0.00330883 0.00620883 2024-09-10T18:28:20.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:28:20.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:20.184 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:20.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:20.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:20.197 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:28:20.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-10T18:28:20.286 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:20.313 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:20.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:20.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:20.415 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:20.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:20.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:20.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:20.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:20 smithi028 ceph-mon[39220]: pgmap v307: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 178 MiB/s wr, 2.84k op/s 2024-09-10T18:28:20.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:20 smithi028 ceph-mon[29291]: pgmap v307: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 178 MiB/s wr, 2.84k op/s 2024-09-10T18:28:20.543 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:20.573 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:20.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:20.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:20 smithi204 ceph-mon[32357]: pgmap v307: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 178 MiB/s wr, 2.84k op/s 2024-09-10T18:28:20.674 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:20.706 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:20.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:20.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:20.808 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:20.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:20.837 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:20.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:20.938 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:20.968 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:20.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:20.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:21.070 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:21.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:21.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:21.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:21.104 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 14 16 35243 35227 157.248 109.562 0.00683964 0.00635488 2024-09-10T18:28:21.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:21.226 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:21.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:21.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:21.327 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:21.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:21.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:21.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:21.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:21.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:21.493 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:21.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:21.579 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:28:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:28:21] "GET /metrics HTTP/1.1" 200 223904 "" "Prometheus/2.43.0" 2024-09-10T18:28:21.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:21.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:21.630 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:21.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:21.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:21.772 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:21.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:21.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:21.874 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:21.898 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:21 smithi028 ceph-mon[29291]: osdmap e195: 8 total, 8 up, 8 in 2024-09-10T18:28:21.898 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:21 smithi028 ceph-mon[29291]: pgmap v309: 158 pgs: 158 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-09-10T18:28:21.898 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:21 smithi028 ceph-mon[39220]: osdmap e195: 8 total, 8 up, 8 in 2024-09-10T18:28:21.898 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:21 smithi028 ceph-mon[39220]: pgmap v309: 158 pgs: 158 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-09-10T18:28:21.905 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:21.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:21.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:22.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:22.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:22.042 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:22.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:22.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:21 smithi204 ceph-mon[32357]: osdmap e195: 8 total, 8 up, 8 in 2024-09-10T18:28:22.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:21 smithi204 ceph-mon[32357]: pgmap v309: 158 pgs: 158 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-09-10T18:28:22.105 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 15 16 37212 37196 154.966 123.062 0.00471497 0.00644813 2024-09-10T18:28:22.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:22.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:22.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:22.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:22.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:22.426 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:22.457 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:22.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:22.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:22.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:22.603 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:22.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:28:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:28:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:28:22.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:22.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:22.706 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:22.735 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:22.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:22.837 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:22.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:22.871 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:22.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:22.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:23.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:23.105 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 16 16 39213 39197 153.097 125.062 0.00392933 0.00652705 2024-09-10T18:28:23.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:23.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:23.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:23.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:23.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:23.300 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:23.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:23.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:23.435 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:23.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:23.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:23.575 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:23.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:23.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:23.689 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:28:23.689 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:28:23.735 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:23.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:23.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:23.838 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:23.873 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:23.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:23.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:23.975 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:24.007 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:24.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:24.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:24.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:23 smithi028 ceph-mon[29291]: pgmap v310: 158 pgs: 158 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 709 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-10T18:28:24.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:23 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:24.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:23 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:28:24.068 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:23 smithi028 ceph-mon[39220]: pgmap v310: 158 pgs: 158 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 709 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-10T18:28:24.069 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:23 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:24.069 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:23 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:28:24.105 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 17 16 41127 41111 151.127 119.625 0.0103255 0.00661223 2024-09-10T18:28:24.109 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:24.139 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:24.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:24.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:24.201 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:23 smithi204 ceph-mon[32357]: pgmap v310: 158 pgs: 158 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 709 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-10T18:28:24.202 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:23 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:28:24.202 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:23 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:28:24.241 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:24.273 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:24.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:24.375 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:24.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:24.408 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:24.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:24.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:24.549 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:24.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:24.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:24.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:28:24 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:28:24.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:24.690 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:24.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:24.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:24.791 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:24.820 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:24.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:24.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:24.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-10T18:28:24.922 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:24.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:24.970 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:24.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:24.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:24 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3957873943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:28:24.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:24 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3957873943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:28:25.039 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:28:25.039 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:28:25.039 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:28:25.039 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:28:25.040 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:28:25.040 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:28:25.040 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:28:25.040 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:28:25.044 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:28:25.044 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:28:25.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:25.077 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:28:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:28:25.095 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:25.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:25.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:28:25.105 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 18 16 42876 42860 148.803 109.312 0.00833269 0.0067149 2024-09-10T18:28:25.106 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:28:25.106 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:28:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:28:25.140 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:28:25.140 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:28:25.197 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:25.224 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:25.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:25.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:25.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:24 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3957873943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:28:25.366 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:25.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:25.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:25.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:25.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:25.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:25.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:25.645 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:25.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:25.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:25.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:25.787 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:25.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:25.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:25.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:25.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:25.935 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:25.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:26.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:26.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:26.085 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:26.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:26.106 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 19 16 44687 44671 146.917 113.188 0.00389352 0.00680198 2024-09-10T18:28:26.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:26.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:25 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:26.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:25 smithi028 ceph-mon[39220]: pgmap v311: 158 pgs: 158 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-10T18:28:26.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:25 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:26.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:25 smithi028 ceph-mon[29291]: pgmap v311: 158 pgs: 158 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-10T18:28:26.215 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:26.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:26.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:26.317 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:26.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:26.347 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:26.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:25 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:25 smithi204 ceph-mon[32357]: pgmap v311: 158 pgs: 158 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-10T18:28:26.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:26.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:26.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:26.593 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:26.623 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:26.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:26.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:26.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:26.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:26.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:26.866 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:26.902 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:26.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:26.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:27.004 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:27.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:27.042 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:27.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:27.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:28:27.105842+0000 min lat: 0.00125336 max lat: 0.114589 avg lat: 0.00686173 2024-09-10T18:28:27.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:28:27.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 20 16 46624 46608 145.624 121.062 0.00403659 0.00686173 2024-09-10T18:28:27.142 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:27.170 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:27.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:27.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:27.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:27.307 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:27.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:27.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:27.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:27.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:27.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:27.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:27.594 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:27.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:27.695 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:27.727 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:27.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:27.829 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:27.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:27.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:27.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:27.962 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:27.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:27.999 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:28.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:28.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:28:28.100 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:28.148 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 21 16 48497 48481 144.264 117.062 0.0101964 0.00692578 2024-09-10T18:28:28.174 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:28.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:28.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:28.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:27 smithi028 ceph-mon[29291]: pgmap v312: 158 pgs: 158 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-10T18:28:28.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:27 smithi028 ceph-mon[39220]: pgmap v312: 158 pgs: 158 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-10T18:28:28.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:28.444 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:27 smithi204 ceph-mon[32357]: pgmap v312: 158 pgs: 158 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-10T18:28:28.490 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:28.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:28.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:28.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:28.624 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:28.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:28.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:28.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:28.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:28:28.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:28.960 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:28.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:28.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:29.062 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:29.092 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:29.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:29.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:29.106 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 22 16 50132 50116 142.351 102.188 0.010151 0.00701975 2024-09-10T18:28:29.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:29.251 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:29.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:29.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:29.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:29.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:29.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:29.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:29.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:29.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:29.642 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:29.689 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:29.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:29.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:29.790 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:29.815 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:29.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:29.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:29.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:29.958 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:29.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:29.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:30.060 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:30.093 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:30.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:30.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 23 16 51709 51693 140.447 98.5625 0.00363613 0.0071156 2024-09-10T18:28:30.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:29 smithi028 ceph-mon[29291]: pgmap v313: 158 pgs: 158 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-10T18:28:30.169 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:29 smithi028 ceph-mon[39220]: pgmap v313: 158 pgs: 158 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-10T18:28:30.194 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:30.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:30.228 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:30.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:30.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:30.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:29 smithi204 ceph-mon[32357]: pgmap v313: 158 pgs: 158 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-10T18:28:30.373 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:30.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:30.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:30.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:30.520 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:30.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:30.622 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:30.653 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:30.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:30.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:30.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:30.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:30.798 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:30.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:30.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:30.929 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:30.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:30.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:30.963 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:28:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.7 to 1 (10000) 2024-09-10T18:28:31.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:31.077 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:31.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:31.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:31.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 24 16 53294 53278 138.722 99.0625 0.0105227 0.00720399 2024-09-10T18:28:31.179 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:31.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:31.209 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:31.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:31.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:30 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3020485505' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:28:31.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:30 smithi028 ceph-mon[39220]: from='client.35039 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:28:31.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:30 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2899222654' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:28:31.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:30 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3020485505' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:28:31.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:30 smithi028 ceph-mon[29291]: from='client.35039 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:28:31.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:30 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2899222654' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:28:31.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:31.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:31.344 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:31.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:30 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3020485505' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:28:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:30 smithi204 ceph-mon[32357]: from='client.35039 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:28:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:30 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2899222654' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:28:31.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:31.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:31.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:31.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:31.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:31.619 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:28:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:28:31] "GET /metrics HTTP/1.1" 200 224144 "" "Prometheus/2.43.0" 2024-09-10T18:28:31.620 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:31.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:31.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:31.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:31.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:31.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:31.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:31.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:31.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:32.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:32.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:32.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:32.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 25 16 54992 54976 137.418 106.125 0.00432792 0.00727203 2024-09-10T18:28:32.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:32.198 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:32.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:32.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:31 smithi028 ceph-mon[39220]: pgmap v314: 158 pgs: 158 active+clean; 3.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1002 B/s rd, 112 MiB/s wr, 1.78k op/s 2024-09-10T18:28:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:31 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:28:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:31 smithi028 ceph-mon[39220]: from='client.35039 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T18:28:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:31 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2899222654' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:28:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:31 smithi028 ceph-mon[39220]: osdmap e196: 8 total, 8 up, 8 in 2024-09-10T18:28:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:31 smithi028 ceph-mon[29291]: pgmap v314: 158 pgs: 158 active+clean; 3.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1002 B/s rd, 112 MiB/s wr, 1.78k op/s 2024-09-10T18:28:32.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:31 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:28:32.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:31 smithi028 ceph-mon[29291]: from='client.35039 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T18:28:32.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:31 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2899222654' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:28:32.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:31 smithi028 ceph-mon[29291]: osdmap e196: 8 total, 8 up, 8 in 2024-09-10T18:28:32.300 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:32.332 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:32.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:32.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:32.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:28:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:28:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:28:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:31 smithi204 ceph-mon[32357]: pgmap v314: 158 pgs: 158 active+clean; 3.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1002 B/s rd, 112 MiB/s wr, 1.78k op/s 2024-09-10T18:28:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:31 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:28:32.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:31 smithi204 ceph-mon[32357]: from='client.35039 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T18:28:32.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:31 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2899222654' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:28:32.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:31 smithi204 ceph-mon[32357]: osdmap e196: 8 total, 8 up, 8 in 2024-09-10T18:28:32.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:32.461 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-10T18:28:32.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:28:32.478 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:32.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:32.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:32.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:32.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:32.630 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:32.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:32.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:32.773 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:32.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:32.875 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:32.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:32.907 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:32.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:33.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:33.040 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:33.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:33.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:33.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 26 15 56605 56590 136.012 100.875 0.00804933 0.00734742 2024-09-10T18:28:33.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:33.188 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:33.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:33.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:28:33.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:32 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:28:33.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:32 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:28:33.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:32 smithi028 ceph-mon[29291]: osdmap e197: 8 total, 8 up, 8 in 2024-09-10T18:28:33.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:32 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:28:33.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:32 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:28:33.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:32 smithi028 ceph-mon[39220]: osdmap e197: 8 total, 8 up, 8 in 2024-09-10T18:28:33.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:33.340 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:33.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:33.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:32 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:28:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:32 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:28:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:32 smithi204 ceph-mon[32357]: osdmap e197: 8 total, 8 up, 8 in 2024-09-10T18:28:33.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:33.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:33.486 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:33.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:33.587 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:33.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:33.636 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:33.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:33.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:33.778 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:33.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:33.880 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:33.905 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:33.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:33.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:34.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:34.051 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:34.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:34.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:34.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 27 16 58100 58084 134.433 93.375 0.0172303 0.0074334 2024-09-10T18:28:34.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:34.194 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:34.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:34.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:33 smithi028 ceph-mon[39220]: pgmap v317: 158 pgs: 158 active+clean; 3.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 124 MiB/s wr, 1.98k op/s 2024-09-10T18:28:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:33 smithi028 ceph-mon[29291]: pgmap v317: 158 pgs: 158 active+clean; 3.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 124 MiB/s wr, 1.98k op/s 2024-09-10T18:28:34.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:34.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:34.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:33 smithi204 ceph-mon[32357]: pgmap v317: 158 pgs: 158 active+clean; 3.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 124 MiB/s wr, 1.98k op/s 2024-09-10T18:28:34.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:28:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:28:34.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:34.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:34.473 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:34.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:34.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:34.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:34.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:34.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:34.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:34.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:34.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:34.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:34.901 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:34.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:35.002 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:35.033 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:35.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:35.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:35.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 28 16 59626 59610 133.038 95.375 0.00958563 0.00750951 2024-09-10T18:28:35.135 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:35.166 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:35.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:35.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:28:35.239 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:28:35.240 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-10T18:28:35.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-10T18:28:35.240 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:28:35.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.240 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29831 7.a 7:5aa0b980:::benchmark_data_smithi028_77856_object29830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:28:35.240 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.545597+0000", 2024-09-10T18:28:35.240 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 277.41404467199999, 2024-09-10T18:28:35.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24445233, 2024-09-10T18:28:35.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.241 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.241 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.241 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.241 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29831 2024-09-10T18:28:35.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.241 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545597+0000", 2024-09-10T18:28:35.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545597+0000", 2024-09-10T18:28:35.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545595+0000", 2024-09-10T18:28:35.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:28:35.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545618+0000", 2024-09-10T18:28:35.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2614000000000001e-05 2024-09-10T18:28:35.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545633+0000", 2024-09-10T18:28:35.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4814000000000001e-05 2024-09-10T18:28:35.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545639+0000", 2024-09-10T18:28:35.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9560000000000002e-06 2024-09-10T18:28:35.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545683+0000", 2024-09-10T18:28:35.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3893000000000001e-05 2024-09-10T18:28:35.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545798+0000", 2024-09-10T18:28:35.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000115179 2024-09-10T18:28:35.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:28:35.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545952+0000", 2024-09-10T18:28:35.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015383399999999999 2024-09-10T18:28:35.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548658+0000", 2024-09-10T18:28:35.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027061089999999999 2024-09-10T18:28:35.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548665+0000", 2024-09-10T18:28:35.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9929999999999998e-06 2024-09-10T18:28:35.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789955+0000", 2024-09-10T18:28:35.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.241290227 2024-09-10T18:28:35.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790008+0000", 2024-09-10T18:28:35.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3362999999999997e-05 2024-09-10T18:28:35.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790049+0000", 2024-09-10T18:28:35.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1037e-05 2024-09-10T18:28:35.246 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.246 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.246 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.247 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29839 7.a 7:508240a2:::benchmark_data_smithi028_77856_object29838:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:28:35.247 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.549223+0000", 2024-09-10T18:28:35.247 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 277.41041853399997, 2024-09-10T18:28:35.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.244164943, 2024-09-10T18:28:35.247 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.247 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.247 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.247 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.247 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.247 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29839 2024-09-10T18:28:35.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.248 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549223+0000", 2024-09-10T18:28:35.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549223+0000", 2024-09-10T18:28:35.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549221+0000", 2024-09-10T18:28:35.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:28:35.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549254+0000", 2024-09-10T18:28:35.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2920999999999999e-05 2024-09-10T18:28:35.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549268+0000", 2024-09-10T18:28:35.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4418e-05 2024-09-10T18:28:35.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549273+0000", 2024-09-10T18:28:35.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.22e-06 2024-09-10T18:28:35.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549316+0000", 2024-09-10T18:28:35.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2519999999999999e-05 2024-09-10T18:28:35.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549937+0000", 2024-09-10T18:28:35.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00062122000000000002 2024-09-10T18:28:35.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:28:35.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.550171+0000", 2024-09-10T18:28:35.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023401499999999999 2024-09-10T18:28:35.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.551601+0000", 2024-09-10T18:28:35.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0014293909999999999 2024-09-10T18:28:35.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.551669+0000", 2024-09-10T18:28:35.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8378000000000002e-05 2024-09-10T18:28:35.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793314+0000", 2024-09-10T18:28:35.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24164548 2024-09-10T18:28:35.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793355+0000", 2024-09-10T18:28:35.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0368999999999999e-05 2024-09-10T18:28:35.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793388+0000", 2024-09-10T18:28:35.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3241999999999998e-05 2024-09-10T18:28:35.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.253 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.253 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29852 7.2 7:40708469:::benchmark_data_smithi028_77856_object29851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:28:35.253 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.555115+0000", 2024-09-10T18:28:35.253 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 277.40452644599998, 2024-09-10T18:28:35.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.238455156, 2024-09-10T18:28:35.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.254 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.254 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.254 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.254 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29852 2024-09-10T18:28:35.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.254 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555115+0000", 2024-09-10T18:28:35.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555115+0000", 2024-09-10T18:28:35.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555113+0000", 2024-09-10T18:28:35.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:28:35.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555141+0000", 2024-09-10T18:28:35.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8665000000000001e-05 2024-09-10T18:28:35.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555157+0000", 2024-09-10T18:28:35.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5668999999999999e-05 2024-09-10T18:28:35.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555163+0000", 2024-09-10T18:28:35.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5490000000000003e-06 2024-09-10T18:28:35.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555251+0000", 2024-09-10T18:28:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7559999999999995e-05 2024-09-10T18:28:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555369+0000", 2024-09-10T18:28:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011843 2024-09-10T18:28:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:28:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555505+0000", 2024-09-10T18:28:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000135115 2024-09-10T18:28:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.557478+0000", 2024-09-10T18:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019733139999999999 2024-09-10T18:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.557928+0000", 2024-09-10T18:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00044986800000000002 2024-09-10T18:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793508+0000", 2024-09-10T18:28:35.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23558014999999999 2024-09-10T18:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793542+0000", 2024-09-10T18:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4235e-05 2024-09-10T18:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793570+0000", 2024-09-10T18:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8206999999999999e-05 2024-09-10T18:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29853 7.3 7:c42b9ada:::benchmark_data_smithi028_77856_object29852:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.555209+0000", 2024-09-10T18:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 277.40443288799997, 2024-09-10T18:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23772209899999999, 2024-09-10T18:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29853 2024-09-10T18:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.260 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555209+0000", 2024-09-10T18:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555209+0000", 2024-09-10T18:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555208+0000", 2024-09-10T18:28:35.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555230+0000", 2024-09-10T18:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2101999999999999e-05 2024-09-10T18:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555240+0000", 2024-09-10T18:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8439999999999993e-06 2024-09-10T18:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555245+0000", 2024-09-10T18:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4190000000000001e-06 2024-09-10T18:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555294+0000", 2024-09-10T18:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8812e-05 2024-09-10T18:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555386+0000", 2024-09-10T18:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2119000000000004e-05 2024-09-10T18:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,4", 2024-09-10T18:28:35.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555520+0000", 2024-09-10T18:28:35.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013431399999999999 2024-09-10T18:28:35.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.557268+0000", 2024-09-10T18:28:35.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017474999999999999 2024-09-10T18:28:35.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.557731+0000", 2024-09-10T18:28:35.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00046338800000000002 2024-09-10T18:28:35.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.792820+0000", 2024-09-10T18:28:35.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23508900299999999 2024-09-10T18:28:35.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.792864+0000", 2024-09-10T18:28:35.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3778e-05 2024-09-10T18:28:35.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.792931+0000", 2024-09-10T18:28:35.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6889000000000003e-05 2024-09-10T18:28:35.265 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.266 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.266 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.266 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29854 7.a 7:5609dd1f:::benchmark_data_smithi028_77856_object29853:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:28:35.266 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.555316+0000", 2024-09-10T18:28:35.266 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 277.40432572999998, 2024-09-10T18:28:35.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.238163033, 2024-09-10T18:28:35.266 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.266 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.266 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.266 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.266 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.266 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29854 2024-09-10T18:28:35.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.267 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555316+0000", 2024-09-10T18:28:35.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555316+0000", 2024-09-10T18:28:35.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555314+0000", 2024-09-10T18:28:35.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:28:35.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555341+0000", 2024-09-10T18:28:35.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6951e-05 2024-09-10T18:28:35.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555353+0000", 2024-09-10T18:28:35.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1398e-05 2024-09-10T18:28:35.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555360+0000", 2024-09-10T18:28:35.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2010000000000002e-06 2024-09-10T18:28:35.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555397+0000", 2024-09-10T18:28:35.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7163000000000003e-05 2024-09-10T18:28:35.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555506+0000", 2024-09-10T18:28:35.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109188 2024-09-10T18:28:35.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:28:35.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.555633+0000", 2024-09-10T18:28:35.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000126391 2024-09-10T18:28:35.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.557918+0000", 2024-09-10T18:28:35.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00228583 2024-09-10T18:28:35.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.557935+0000", 2024-09-10T18:28:35.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6439e-05 2024-09-10T18:28:35.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793418+0000", 2024-09-10T18:28:35.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23548359199999999 2024-09-10T18:28:35.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793451+0000", 2024-09-10T18:28:35.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2656e-05 2024-09-10T18:28:35.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793479+0000", 2024-09-10T18:28:35.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7838e-05 2024-09-10T18:28:35.272 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.272 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.272 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.272 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29858 7.a 7:56ca4896:::benchmark_data_smithi028_77856_object29857:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:28:35.272 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.559039+0000", 2024-09-10T18:28:35.272 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 277.40060310199999, 2024-09-10T18:28:35.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23462707999999999, 2024-09-10T18:28:35.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.273 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.273 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.273 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.273 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.273 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29858 2024-09-10T18:28:35.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.273 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559039+0000", 2024-09-10T18:28:35.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559039+0000", 2024-09-10T18:28:35.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559036+0000", 2024-09-10T18:28:35.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-10T18:28:35.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559060+0000", 2024-09-10T18:28:35.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3879999999999998e-05 2024-09-10T18:28:35.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559073+0000", 2024-09-10T18:28:35.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3339000000000001e-05 2024-09-10T18:28:35.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559079+0000", 2024-09-10T18:28:35.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.088e-06 2024-09-10T18:28:35.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559130+0000", 2024-09-10T18:28:35.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0688e-05 2024-09-10T18:28:35.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.276 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559243+0000", 2024-09-10T18:28:35.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011288199999999999 2024-09-10T18:28:35.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:28:35.276 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559381+0000", 2024-09-10T18:28:35.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013798799999999999 2024-09-10T18:28:35.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.561064+0000", 2024-09-10T18:28:35.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001683323 2024-09-10T18:28:35.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.561094+0000", 2024-09-10T18:28:35.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0114e-05 2024-09-10T18:28:35.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793598+0000", 2024-09-10T18:28:35.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.232503821 2024-09-10T18:28:35.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.278 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793638+0000", 2024-09-10T18:28:35.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0005000000000001e-05 2024-09-10T18:28:35.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.278 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.793666+0000", 2024-09-10T18:28:35.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7633000000000001e-05 2024-09-10T18:28:35.278 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.278 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.278 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.278 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29859 7.a 7:5b51003a:::benchmark_data_smithi028_77856_object29858:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:28:35.279 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.559489+0000", 2024-09-10T18:28:35.279 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 277.40015223199998, 2024-09-10T18:28:35.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23708596400000001, 2024-09-10T18:28:35.279 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.279 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.279 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.279 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.279 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.279 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29859 2024-09-10T18:28:35.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.279 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559489+0000", 2024-09-10T18:28:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559489+0000", 2024-09-10T18:28:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559488+0000", 2024-09-10T18:28:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:28:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559514+0000", 2024-09-10T18:28:35.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5272000000000001e-05 2024-09-10T18:28:35.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559526+0000", 2024-09-10T18:28:35.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.237e-05 2024-09-10T18:28:35.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.559532+0000", 2024-09-10T18:28:35.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7790000000000001e-06 2024-09-10T18:28:35.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.560346+0000", 2024-09-10T18:28:35.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081443200000000002 2024-09-10T18:28:35.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.560426+0000", 2024-09-10T18:28:35.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9412000000000005e-05 2024-09-10T18:28:35.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:28:35.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.560503+0000", 2024-09-10T18:28:35.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7825999999999997e-05 2024-09-10T18:28:35.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.283 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.283 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.561194+0000", 2024-09-10T18:28:35.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00069056 2024-09-10T18:28:35.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.283 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.283 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.561443+0000", 2024-09-10T18:28:35.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024952600000000002 2024-09-10T18:28:35.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.283 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.283 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.796494+0000", 2024-09-10T18:28:35.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23505084200000001 2024-09-10T18:28:35.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.284 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.796537+0000", 2024-09-10T18:28:35.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3089e-05 2024-09-10T18:28:35.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.284 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.796575+0000", 2024-09-10T18:28:35.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7976999999999999e-05 2024-09-10T18:28:35.284 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.284 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.284 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.285 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:121682 7.a 7:5cc2bbec:::benchmark_data_smithi028_77856_object582:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:28:35.285 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:14.512524+0000", 2024-09-10T18:28:35.285 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 199.44711726200001, 2024-09-10T18:28:35.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16768606599999999, 2024-09-10T18:28:35.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.285 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.285 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.285 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.285 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.285 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 121682 2024-09-10T18:28:35.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.286 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512524+0000", 2024-09-10T18:28:35.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512524+0000", 2024-09-10T18:28:35.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512524+0000", 2024-09-10T18:28:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:28:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512527+0000", 2024-09-10T18:28:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.467e-06 2024-09-10T18:28:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512533+0000", 2024-09-10T18:28:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5679999999999999e-06 2024-09-10T18:28:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512538+0000", 2024-09-10T18:28:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7119999999999999e-06 2024-09-10T18:28:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521276+0000", 2024-09-10T18:28:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087382520000000002 2024-09-10T18:28:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521386+0000", 2024-09-10T18:28:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110188 2024-09-10T18:28:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:28:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521505+0000", 2024-09-10T18:28:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000119021 2024-09-10T18:28:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.530749+0000", 2024-09-10T18:28:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.009244169 2024-09-10T18:28:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.530781+0000", 2024-09-10T18:28:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2119999999999997e-05 2024-09-10T18:28:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680127+0000", 2024-09-10T18:28:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14934560899999999 2024-09-10T18:28:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680182+0000", 2024-09-10T18:28:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5398000000000001e-05 2024-09-10T18:28:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680210+0000", 2024-09-10T18:28:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8191000000000002e-05 2024-09-10T18:28:35.291 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.291 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.291 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.291 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.291 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:121685 7.a 7:556e904c:::benchmark_data_smithi028_77856_object585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:28:35.291 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:14.516970+0000", 2024-09-10T18:28:35.291 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 199.44267153199999, 2024-09-10T18:28:35.291 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16365090800000001, 2024-09-10T18:28:35.291 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.291 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.291 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.291 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.291 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.292 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 121685 2024-09-10T18:28:35.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.292 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.292 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516970+0000", 2024-09-10T18:28:35.292 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.292 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516970+0000", 2024-09-10T18:28:35.292 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516968+0000", 2024-09-10T18:28:35.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:28:35.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516977+0000", 2024-09-10T18:28:35.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1600000000000004e-06 2024-09-10T18:28:35.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516989+0000", 2024-09-10T18:28:35.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2094e-05 2024-09-10T18:28:35.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516995+0000", 2024-09-10T18:28:35.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1199999999999999e-06 2024-09-10T18:28:35.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.524202+0000", 2024-09-10T18:28:35.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0072076129999999999 2024-09-10T18:28:35.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.524301+0000", 2024-09-10T18:28:35.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8858999999999997e-05 2024-09-10T18:28:35.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:28:35.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.524413+0000", 2024-09-10T18:28:35.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011134899999999999 2024-09-10T18:28:35.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.530754+0000", 2024-09-10T18:28:35.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006341218 2024-09-10T18:28:35.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.533574+0000", 2024-09-10T18:28:35.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002820473 2024-09-10T18:28:35.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680527+0000", 2024-09-10T18:28:35.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.146952895 2024-09-10T18:28:35.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680579+0000", 2024-09-10T18:28:35.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1777999999999998e-05 2024-09-10T18:28:35.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.297 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.297 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680621+0000", 2024-09-10T18:28:35.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1953999999999998e-05 2024-09-10T18:28:35.297 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.297 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.297 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.297 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:121686 7.2 7:471809bb:::benchmark_data_smithi028_77856_object586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:28:35.297 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:14.518358+0000", 2024-09-10T18:28:35.297 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 199.441283911, 2024-09-10T18:28:35.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16205149299999999, 2024-09-10T18:28:35.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.298 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.298 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.298 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.298 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.298 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 121686 2024-09-10T18:28:35.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.298 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518358+0000", 2024-09-10T18:28:35.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518358+0000", 2024-09-10T18:28:35.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518356+0000", 2024-09-10T18:28:35.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:28:35.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518362+0000", 2024-09-10T18:28:35.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8889999999999999e-06 2024-09-10T18:28:35.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518373+0000", 2024-09-10T18:28:35.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0375e-05 2024-09-10T18:28:35.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518380+0000", 2024-09-10T18:28:35.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.092e-06 2024-09-10T18:28:35.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.526805+0000", 2024-09-10T18:28:35.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0084254840000000004 2024-09-10T18:28:35.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.526929+0000", 2024-09-10T18:28:35.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012368700000000001 2024-09-10T18:28:35.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:28:35.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.527045+0000", 2024-09-10T18:28:35.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116277 2024-09-10T18:28:35.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.533582+0000", 2024-09-10T18:28:35.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0065365359999999999 2024-09-10T18:28:35.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.542247+0000", 2024-09-10T18:28:35.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0086657750000000006 2024-09-10T18:28:35.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680327+0000", 2024-09-10T18:28:35.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13807940899999999 2024-09-10T18:28:35.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680372+0000", 2024-09-10T18:28:35.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5173000000000003e-05 2024-09-10T18:28:35.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680409+0000", 2024-09-10T18:28:35.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7194000000000003e-05 2024-09-10T18:28:35.303 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.303 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.303 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.303 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:121693 7.2 7:4f645b06:::benchmark_data_smithi028_77856_object593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:28:35.303 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:14.521071+0000", 2024-09-10T18:28:35.303 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 199.43857093700001, 2024-09-10T18:28:35.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.159677503, 2024-09-10T18:28:35.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.304 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.304 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.304 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.304 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.304 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 121693 2024-09-10T18:28:35.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.304 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.304 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.304 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521071+0000", 2024-09-10T18:28:35.304 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.305 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521071+0000", 2024-09-10T18:28:35.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.305 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521070+0000", 2024-09-10T18:28:35.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:28:35.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.305 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521075+0000", 2024-09-10T18:28:35.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2220000000000001e-06 2024-09-10T18:28:35.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521083+0000", 2024-09-10T18:28:35.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6370000000000005e-06 2024-09-10T18:28:35.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521088+0000", 2024-09-10T18:28:35.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8930000000000004e-06 2024-09-10T18:28:35.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.530803+0000", 2024-09-10T18:28:35.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0097147029999999999 2024-09-10T18:28:35.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.530925+0000", 2024-09-10T18:28:35.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122376 2024-09-10T18:28:35.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:28:35.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.531026+0000", 2024-09-10T18:28:35.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010061 2024-09-10T18:28:35.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.534573+0000", 2024-09-10T18:28:35.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035470860000000001 2024-09-10T18:28:35.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.542273+0000", 2024-09-10T18:28:35.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076997999999999997 2024-09-10T18:28:35.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680669+0000", 2024-09-10T18:28:35.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13839658699999999 2024-09-10T18:28:35.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680712+0000", 2024-09-10T18:28:35.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2979999999999998e-05 2024-09-10T18:28:35.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.311 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.311 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680748+0000", 2024-09-10T18:28:35.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5775999999999998e-05 2024-09-10T18:28:35.311 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.311 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.311 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.311 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:124429 7.2 7:4838b573:::benchmark_data_smithi028_77856_object3329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:28:35.311 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:18.054003+0000", 2024-09-10T18:28:35.311 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 195.905638692, 2024-09-10T18:28:35.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.166994589, 2024-09-10T18:28:35.312 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.312 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.312 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.312 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.312 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.312 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 124429 2024-09-10T18:28:35.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.312 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054003+0000", 2024-09-10T18:28:35.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054003+0000", 2024-09-10T18:28:35.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054002+0000", 2024-09-10T18:28:35.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:28:35.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054005+0000", 2024-09-10T18:28:35.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4559999999999999e-06 2024-09-10T18:28:35.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054009+0000", 2024-09-10T18:28:35.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3669999999999999e-06 2024-09-10T18:28:35.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054013+0000", 2024-09-10T18:28:35.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.343e-06 2024-09-10T18:28:35.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.126700+0000", 2024-09-10T18:28:35.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.072687286000000004 2024-09-10T18:28:35.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.315 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.126814+0000", 2024-09-10T18:28:35.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114442 2024-09-10T18:28:35.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.315 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:28:35.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.126963+0000", 2024-09-10T18:28:35.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014888499999999999 2024-09-10T18:28:35.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.315 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.140882+0000", 2024-09-10T18:28:35.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013919124 2024-09-10T18:28:35.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.141158+0000", 2024-09-10T18:28:35.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000276042 2024-09-10T18:28:35.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.220924+0000", 2024-09-10T18:28:35.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.079766112 2024-09-10T18:28:35.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.220962+0000", 2024-09-10T18:28:35.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7707999999999999e-05 2024-09-10T18:28:35.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.220998+0000", 2024-09-10T18:28:35.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5349999999999999e-05 2024-09-10T18:28:35.317 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.317 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.317 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.317 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:139058 7.2 7:4504e7d4:::benchmark_data_smithi028_77856_object17958:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:28:35.318 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:42.798430+0000", 2024-09-10T18:28:35.318 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 171.16121118699999, 2024-09-10T18:28:35.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.21785926999999999, 2024-09-10T18:28:35.318 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.318 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.318 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.318 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.318 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.318 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 139058 2024-09-10T18:28:35.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.318 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.318 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.798430+0000", 2024-09-10T18:28:35.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.319 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.319 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.798430+0000", 2024-09-10T18:28:35.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.319 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.319 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.798429+0000", 2024-09-10T18:28:35.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:28:35.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.319 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.798435+0000", 2024-09-10T18:28:35.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4330000000000001e-06 2024-09-10T18:28:35.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.798449+0000", 2024-09-10T18:28:35.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3360000000000001e-05 2024-09-10T18:28:35.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.798455+0000", 2024-09-10T18:28:35.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1600000000000003e-06 2024-09-10T18:28:35.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.887408+0000", 2024-09-10T18:28:35.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.088953303999999997 2024-09-10T18:28:35.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.887522+0000", 2024-09-10T18:28:35.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113528 2024-09-10T18:28:35.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:28:35.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.887728+0000", 2024-09-10T18:28:35.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020583900000000001 2024-09-10T18:28:35.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.896982+0000", 2024-09-10T18:28:35.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092544399999999992 2024-09-10T18:28:35.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.896990+0000", 2024-09-10T18:28:35.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2800000000000003e-06 2024-09-10T18:28:35.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.016206+0000", 2024-09-10T18:28:35.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.119215716 2024-09-10T18:28:35.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.016249+0000", 2024-09-10T18:28:35.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3247000000000001e-05 2024-09-10T18:28:35.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.016290+0000", 2024-09-10T18:28:35.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0467999999999999e-05 2024-09-10T18:28:35.323 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.323 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.323 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.324 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:139084 7.2 7:4d9fefb8:::benchmark_data_smithi028_77856_object17984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:28:35.324 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:42.851594+0000", 2024-09-10T18:28:35.324 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 171.10804748199999, 2024-09-10T18:28:35.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.164888488, 2024-09-10T18:28:35.324 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.324 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.324 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.324 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.324 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.324 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 139084 2024-09-10T18:28:35.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.324 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.851594+0000", 2024-09-10T18:28:35.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.851594+0000", 2024-09-10T18:28:35.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.851593+0000", 2024-09-10T18:28:35.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:28:35.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.851599+0000", 2024-09-10T18:28:35.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1299999999999998e-06 2024-09-10T18:28:35.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.851610+0000", 2024-09-10T18:28:35.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.098e-05 2024-09-10T18:28:35.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.851616+0000", 2024-09-10T18:28:35.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0750000000000003e-06 2024-09-10T18:28:35.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.890507+0000", 2024-09-10T18:28:35.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.038891336999999998 2024-09-10T18:28:35.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.890635+0000", 2024-09-10T18:28:35.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012825400000000001 2024-09-10T18:28:35.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:28:35.328 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.890768+0000", 2024-09-10T18:28:35.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013264400000000001 2024-09-10T18:28:35.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.328 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.328 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.896997+0000", 2024-09-10T18:28:35.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062283959999999998 2024-09-10T18:28:35.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.328 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.328 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.897071+0000", 2024-09-10T18:28:35.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4501999999999999e-05 2024-09-10T18:28:35.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.016410+0000", 2024-09-10T18:28:35.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.11933859099999999 2024-09-10T18:28:35.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.016450+0000", 2024-09-10T18:28:35.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0340999999999999e-05 2024-09-10T18:28:35.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.016483+0000", 2024-09-10T18:28:35.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2687000000000001e-05 2024-09-10T18:28:35.330 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.330 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.330 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.330 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:139472 7.2 7:4dfe09d1:::benchmark_data_smithi028_77856_object18372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:28:35.330 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:43.522091+0000", 2024-09-10T18:28:35.330 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 170.437550405, 2024-09-10T18:28:35.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.22258198400000001, 2024-09-10T18:28:35.330 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.330 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.330 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.330 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.330 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 139472 2024-09-10T18:28:35.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.331 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.331 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.331 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.522091+0000", 2024-09-10T18:28:35.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.331 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.331 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.522091+0000", 2024-09-10T18:28:35.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.522089+0000", 2024-09-10T18:28:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:28:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.522096+0000", 2024-09-10T18:28:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0550000000000002e-06 2024-09-10T18:28:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.522107+0000", 2024-09-10T18:28:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0299000000000001e-05 2024-09-10T18:28:35.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.522113+0000", 2024-09-10T18:28:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9739999999999999e-06 2024-09-10T18:28:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.600209+0000", 2024-09-10T18:28:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.078096439000000004 2024-09-10T18:28:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.600329+0000", 2024-09-10T18:28:35.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000119535 2024-09-10T18:28:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:28:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.600449+0000", 2024-09-10T18:28:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000120189 2024-09-10T18:28:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.606584+0000", 2024-09-10T18:28:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061352220000000001 2024-09-10T18:28:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.610809+0000", 2024-09-10T18:28:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042246059999999997 2024-09-10T18:28:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744597+0000", 2024-09-10T18:28:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.133788834 2024-09-10T18:28:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744636+0000", 2024-09-10T18:28:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8461999999999999e-05 2024-09-10T18:28:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.336 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744673+0000", 2024-09-10T18:28:35.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7283999999999997e-05 2024-09-10T18:28:35.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.336 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.337 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:139485 7.a 7:5828730e:::benchmark_data_smithi028_77856_object18385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:28:35.337 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:43.554329+0000", 2024-09-10T18:28:35.337 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 170.405312326, 2024-09-10T18:28:35.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.190223159, 2024-09-10T18:28:35.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.337 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.337 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.337 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 139485 2024-09-10T18:28:35.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.337 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.337 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.554329+0000", 2024-09-10T18:28:35.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.554329+0000", 2024-09-10T18:28:35.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.554327+0000", 2024-09-10T18:28:35.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:28:35.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.554335+0000", 2024-09-10T18:28:35.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4039999999999998e-06 2024-09-10T18:28:35.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.554345+0000", 2024-09-10T18:28:35.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0973e-05 2024-09-10T18:28:35.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.554351+0000", 2024-09-10T18:28:35.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9120000000000003e-06 2024-09-10T18:28:35.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.597534+0000", 2024-09-10T18:28:35.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.043182458 2024-09-10T18:28:35.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.597642+0000", 2024-09-10T18:28:35.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010785099999999999 2024-09-10T18:28:35.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:28:35.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.597766+0000", 2024-09-10T18:28:35.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012438799999999999 2024-09-10T18:28:35.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.341 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.602961+0000", 2024-09-10T18:28:35.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.005195224 2024-09-10T18:28:35.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.341 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.606579+0000", 2024-09-10T18:28:35.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003617529 2024-09-10T18:28:35.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744464+0000", 2024-09-10T18:28:35.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.137885326 2024-09-10T18:28:35.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744513+0000", 2024-09-10T18:28:35.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8442000000000001e-05 2024-09-10T18:28:35.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744552+0000", 2024-09-10T18:28:35.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9870000000000003e-05 2024-09-10T18:28:35.342 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.343 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.343 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.343 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:139490 7.2 7:469cd9c4:::benchmark_data_smithi028_77856_object18390:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:28:35.343 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:43.578701+0000", 2024-09-10T18:28:35.343 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 170.38094062499999, 2024-09-10T18:28:35.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.166214733, 2024-09-10T18:28:35.343 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.343 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.343 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.343 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.343 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 139490 2024-09-10T18:28:35.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.344 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.578701+0000", 2024-09-10T18:28:35.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.578701+0000", 2024-09-10T18:28:35.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.578699+0000", 2024-09-10T18:28:35.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:28:35.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.578707+0000", 2024-09-10T18:28:35.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5809999999999999e-06 2024-09-10T18:28:35.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.578720+0000", 2024-09-10T18:28:35.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3888e-05 2024-09-10T18:28:35.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.578726+0000", 2024-09-10T18:28:35.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7740000000000001e-06 2024-09-10T18:28:35.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.602984+0000", 2024-09-10T18:28:35.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024257595999999999 2024-09-10T18:28:35.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.603082+0000", 2024-09-10T18:28:35.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8620000000000001e-05 2024-09-10T18:28:35.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:28:35.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.603476+0000", 2024-09-10T18:28:35.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00039365500000000002 2024-09-10T18:28:35.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.607531+0000", 2024-09-10T18:28:35.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004055042 2024-09-10T18:28:35.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.610833+0000", 2024-09-10T18:28:35.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003302049 2024-09-10T18:28:35.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744821+0000", 2024-09-10T18:28:35.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.133988041 2024-09-10T18:28:35.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744877+0000", 2024-09-10T18:28:35.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6140999999999997e-05 2024-09-10T18:28:35.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744916+0000", 2024-09-10T18:28:35.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8346000000000002e-05 2024-09-10T18:28:35.349 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.349 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.349 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.349 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:141043 7.2 7:45059280:::benchmark_data_smithi028_77856_object19943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:28:35.349 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:46.309105+0000", 2024-09-10T18:28:35.349 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 167.650537095, 2024-09-10T18:28:35.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.163191158, 2024-09-10T18:28:35.349 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.350 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.350 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.350 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.350 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 141043 2024-09-10T18:28:35.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.350 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.309105+0000", 2024-09-10T18:28:35.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.309105+0000", 2024-09-10T18:28:35.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.309103+0000", 2024-09-10T18:28:35.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:28:35.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.309110+0000", 2024-09-10T18:28:35.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6029999999999999e-06 2024-09-10T18:28:35.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.309122+0000", 2024-09-10T18:28:35.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1989e-05 2024-09-10T18:28:35.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.309128+0000", 2024-09-10T18:28:35.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0630000000000002e-06 2024-09-10T18:28:35.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.379768+0000", 2024-09-10T18:28:35.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.070639561000000003 2024-09-10T18:28:35.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.379881+0000", 2024-09-10T18:28:35.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113529 2024-09-10T18:28:35.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:28:35.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.380004+0000", 2024-09-10T18:28:35.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012301400000000001 2024-09-10T18:28:35.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.386775+0000", 2024-09-10T18:28:35.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006770496 2024-09-10T18:28:35.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.394129+0000", 2024-09-10T18:28:35.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073538070000000004 2024-09-10T18:28:35.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.472166+0000", 2024-09-10T18:28:35.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.078037168000000004 2024-09-10T18:28:35.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.355 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.472256+0000", 2024-09-10T18:28:35.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9802000000000005e-05 2024-09-10T18:28:35.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.355 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.472296+0000", 2024-09-10T18:28:35.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0132000000000001e-05 2024-09-10T18:28:35.355 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.355 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.355 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.355 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:182362 7.3 7:c3249eb8:::benchmark_data_smithi028_77856_object61262:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-10T18:28:35.356 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:26.337733+0000", 2024-09-10T18:28:35.356 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 127.621908791, 2024-09-10T18:28:35.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.166142067, 2024-09-10T18:28:35.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.356 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.356 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.356 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.356 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 182362 2024-09-10T18:28:35.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.356 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337733+0000", 2024-09-10T18:28:35.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337733+0000", 2024-09-10T18:28:35.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337731+0000", 2024-09-10T18:28:35.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:28:35.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337738+0000", 2024-09-10T18:28:35.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3520000000000003e-06 2024-09-10T18:28:35.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337749+0000", 2024-09-10T18:28:35.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1096e-05 2024-09-10T18:28:35.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337753+0000", 2024-09-10T18:28:35.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5480000000000003e-06 2024-09-10T18:28:35.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337791+0000", 2024-09-10T18:28:35.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7648999999999998e-05 2024-09-10T18:28:35.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337908+0000", 2024-09-10T18:28:35.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011726 2024-09-10T18:28:35.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,4", 2024-09-10T18:28:35.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.338054+0000", 2024-09-10T18:28:35.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014590299999999999 2024-09-10T18:28:35.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.360 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.341103+0000", 2024-09-10T18:28:35.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030488400000000001 2024-09-10T18:28:35.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.360 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.341345+0000", 2024-09-10T18:28:35.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024254600000000001 2024-09-10T18:28:35.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.503780+0000", 2024-09-10T18:28:35.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16243438299999999 2024-09-10T18:28:35.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.503826+0000", 2024-09-10T18:28:35.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6640000000000001e-05 2024-09-10T18:28:35.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.503875+0000", 2024-09-10T18:28:35.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8497e-05 2024-09-10T18:28:35.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.361 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.362 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.362 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:182397 7.3 7:c3805c23:::benchmark_data_smithi028_77856_object61297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-10T18:28:35.362 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:26.362893+0000", 2024-09-10T18:28:35.362 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 127.59674909500001, 2024-09-10T18:28:35.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16023626699999999, 2024-09-10T18:28:35.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:28:35.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:35.362 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:28:35.362 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:28:35.362 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:35.362 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 182397 2024-09-10T18:28:35.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.363 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:28:35.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:28:35.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.362893+0000", 2024-09-10T18:28:35.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:28:35.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.362893+0000", 2024-09-10T18:28:35.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:28:35.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:28:35.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.362891+0000", 2024-09-10T18:28:35.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:28:35.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:28:35.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.362898+0000", 2024-09-10T18:28:35.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7000000000000002e-06 2024-09-10T18:28:35.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:28:35.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.362910+0000", 2024-09-10T18:28:35.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2819999999999999e-05 2024-09-10T18:28:35.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:28:35.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.362916+0000", 2024-09-10T18:28:35.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9039999999999997e-06 2024-09-10T18:28:35.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:28:35.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.362968+0000", 2024-09-10T18:28:35.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1521999999999999e-05 2024-09-10T18:28:35.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:28:35.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.363080+0000", 2024-09-10T18:28:35.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011167699999999999 2024-09-10T18:28:35.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,4", 2024-09-10T18:28:35.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.363185+0000", 2024-09-10T18:28:35.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010522399999999999 2024-09-10T18:28:35.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.367164+0000", 2024-09-10T18:28:35.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039788100000000002 2024-09-10T18:28:35.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:28:35.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.367171+0000", 2024-09-10T18:28:35.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0990000000000001e-06 2024-09-10T18:28:35.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:35.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.523007+0000", 2024-09-10T18:28:35.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15583649299999999 2024-09-10T18:28:35.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:28:35.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.523093+0000", 2024-09-10T18:28:35.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6192000000000004e-05 2024-09-10T18:28:35.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:28:35.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:28:35.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:28:35.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.523129+0000", 2024-09-10T18:28:35.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5426000000000003e-05 2024-09-10T18:28:35.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.368 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:28:35.368 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:28:35.368 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:28:35.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:35.411 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:35.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:35.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:35.513 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:35.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:35.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:35.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:35.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:35.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:35.696 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:35.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:35.797 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:35.830 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:35.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:35.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:35.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:35.969 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:35.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:35.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:36.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:36.103 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:36.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:36.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:36.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 29 16 61175 61159 131.788 96.8125 0.00787657 0.00758377 2024-09-10T18:28:36.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:36.250 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-10T18:28:36.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:36 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:36.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:36 smithi028 ceph-mon[39220]: pgmap v318: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-10T18:28:36.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:35 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:36.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:35 smithi028 ceph-mon[29291]: pgmap v318: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-10T18:28:36.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:36.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:36.281 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:35 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:36.281 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:35 smithi204 ceph-mon[32357]: pgmap v318: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-10T18:28:36.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:36.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:36.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:36.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:36.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:36.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:28:36.627 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:28:36.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:36.667 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:36.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:36.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:36.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:36.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:36.821 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:36.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:36.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:36.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:36.948 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:36.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:37.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:37.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:37.097 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:37.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:37.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:37 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/782901322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:28:37.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 30 16 62545 62529 130.249 85.625 0.00548604 0.00767243 2024-09-10T18:28:37.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:37.226 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:37.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:37.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:37.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:37.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:37.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:37.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:37.454 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:37.483 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:37.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:37.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:37.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:37 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/782901322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:28:37.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:37 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/782901322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:28:37.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:37.621 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:37.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:37.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:37.723 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:37.755 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:37.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:37.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:37.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:37.887 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:37.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:37.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:37.989 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:38.020 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:38.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:38.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 31 16 63796 63780 128.57 78.1875 0.00720545 0.00777272 2024-09-10T18:28:38.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:38.155 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:38.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:38.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:38 smithi028 ceph-mon[29291]: pgmap v319: 158 pgs: 158 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.09k op/s 2024-09-10T18:28:38.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:38 smithi028 ceph-mon[39220]: pgmap v319: 158 pgs: 158 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.09k op/s 2024-09-10T18:28:38.257 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:38.290 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:38.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:38.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:38 smithi204 ceph-mon[32357]: pgmap v319: 158 pgs: 158 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.09k op/s 2024-09-10T18:28:38.391 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:38.432 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:38.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:38.534 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:38.572 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:38.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:38.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:38.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:38.714 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:38.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:38.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:38.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:38.865 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:38.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:38.966 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:39.000 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:39.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:39.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:39.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:39.107 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 32 16 65112 65096 127.122 82.25 0.00962524 0.00786156 2024-09-10T18:28:39.134 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:39.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:39.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:39.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:39.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:28:39.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:28:39.258 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:39.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:39.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:28:39.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:39.401 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:39.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:39.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:39.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:39.541 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:39.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:39.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:39.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:39.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:39.675 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:39.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:39.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:39.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:39.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:39.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:39.915 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:39.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:39.948 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:39.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:40.049 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:40.084 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:40.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:40.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:40.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 33 16 66519 66503 125.935 87.9375 0.0112435 0.00793587 2024-09-10T18:28:40.157 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:28:40.157 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:28:40.157 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:28:40.157 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:28:40.157 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:28:40.157 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:28:40.157 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:28:40.157 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:28:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:28:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:28:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:28:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:28:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:28:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:28:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:28:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:28:40.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:40.224 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:40.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:40.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:40.293 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:40 smithi204 ceph-mon[32357]: pgmap v320: 158 pgs: 158 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 990 op/s 2024-09-10T18:28:40.327 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:40.360 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:40.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:40.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:40.446 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:28:40.446 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:28:40.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:40.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:40.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:40.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:40.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:40 smithi028 ceph-mon[29291]: pgmap v320: 158 pgs: 158 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 990 op/s 2024-09-10T18:28:40.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:40 smithi028 ceph-mon[39220]: pgmap v320: 158 pgs: 158 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 990 op/s 2024-09-10T18:28:40.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:40.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:40.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:40.739 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:40.780 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:40.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:40.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:40.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:40.912 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:40.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:40.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:41.013 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:41.051 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:41.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:41.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:41.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 34 16 67949 67933 124.859 89.375 0.00684735 0.00800342 2024-09-10T18:28:41.147 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-10T18:28:41.153 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:41.241 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:41.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:41.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:41.343 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:41.382 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:41.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:41.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:41.518 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:41.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:41.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:41.598 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:28:41.619 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:41.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:41.657 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:41.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:41.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:41 smithi028 ceph-mon[29291]: pgmap v321: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-10T18:28:41.683 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:28:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:28:41] "GET /metrics HTTP/1.1" 200 224195 "" "Prometheus/2.43.0" 2024-09-10T18:28:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:28:41.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:41.805 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:41.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:41.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:41.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:41.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:41.948 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:41.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:41.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:41 smithi028 ceph-mon[39220]: pgmap v321: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-10T18:28:42.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:42.083 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:42.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:41 smithi204 ceph-mon[32357]: pgmap v321: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-10T18:28:42.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 35 16 69202 69186 123.529 78.3125 0.0139425 0.00809039 2024-09-10T18:28:42.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:42.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:42.217 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:42.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:42.318 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:42.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:42.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:42.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:42.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:42.489 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:28:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:28:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:28:42.493 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:42.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:42.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:42.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:42.633 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:42.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:42.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:42.735 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:42.764 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:42.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:42.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:28:42.817 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:28:42.866 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:42.899 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:42.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:42.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:28:43.001 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:43.037 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:43.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:43.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 36 16 70437 70421 122.242 77.1875 0.00308009 0.00816705 2024-09-10T18:28:43.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:43.162 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:43.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:43.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:43.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:43.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:43.312 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:43.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:43.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:43.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:43.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:43.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:43.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:43.582 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:43.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:43.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:43.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:43.715 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:43.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:43.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:43.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:43.860 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:43.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:43.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:43.962 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:43.993 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:43.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:43.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:44.094 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:44.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 37 16 71675 71659 121.029 77.375 0.0110486 0.00825739 2024-09-10T18:28:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:44.134 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:44.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:44.212 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:43 smithi204 ceph-mon[32357]: from='client.25075 v1:172.21.15.28:0/3066800585' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:28:44.212 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:43 smithi204 ceph-mon[32357]: pgmap v322: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 938 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-10T18:28:44.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:44.236 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:28:44.236 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:28:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:43 smithi028 ceph-mon[29291]: from='client.25075 v1:172.21.15.28:0/3066800585' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:28:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:43 smithi028 ceph-mon[29291]: pgmap v322: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 938 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-10T18:28:44.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:43 smithi028 ceph-mon[39220]: from='client.25075 v1:172.21.15.28:0/3066800585' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:28:44.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:43 smithi028 ceph-mon[39220]: pgmap v322: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 938 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-10T18:28:44.272 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:44.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:44.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:44.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:44.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:44.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:44.516 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:44.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:44.548 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:44.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:44.567 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:28:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:28:44.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:44.683 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:44.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:44.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:44.785 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:44.815 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:44.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:44.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:44.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:44.946 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:44.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:45.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-10T18:28:45.047 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:45.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 38 16 73010 72994 120.04 83.4375 0.0144871 0.00832607 2024-09-10T18:28:45.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:45.115 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:45.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:45.216 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:45.259 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:45.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:45.360 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:45.401 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:45.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:45.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:45.442 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:28:45.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:45.541 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:45.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:45.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:45.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:45.978 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:45.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:45.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:46.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:46.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 39 16 74411 74395 119.207 87.5625 0.0123757 0.00838432 2024-09-10T18:28:46.113 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:45 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:46.113 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:45 smithi204 ceph-mon[32357]: pgmap v323: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T18:28:46.113 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:45 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:28:46.113 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:45 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:28:46.115 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:46.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:46.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:46.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:45 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:45 smithi028 ceph-mon[29291]: pgmap v323: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T18:28:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:45 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:28:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:45 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:28:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:45 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:45 smithi028 ceph-mon[39220]: pgmap v323: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T18:28:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:45 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:28:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:45 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:28:46.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:46.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:46.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:46.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:46.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:46.515 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:46.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:46.550 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:46.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:46.651 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:46.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:46.685 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:46.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:46.786 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:46.812 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:46.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:46.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:46.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:46.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:46.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:47.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:47.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:47.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:47.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:47.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:28:47.107669+0000 min lat: 0.00125336 max lat: 0.12193 avg lat: 0.00846307 2024-09-10T18:28:47.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:28:47.108 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 40 16 75599 75583 118.083 74.25 0.0122173 0.00846307 2024-09-10T18:28:47.148 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:46 smithi204 ceph-mon[32357]: osdmap e198: 8 total, 8 up, 8 in 2024-09-10T18:28:47.192 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-09-10T18:28:40.879057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4362082524,"num_objects":66747,"num_object_clones":0,"num_object_copies":200241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66747,"num_whiteouts":0,"num_read":1141,"num_read_kb":893,"num_write":66981,"num_write_kb":4259307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67974,"ondisk_log_size":67974,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15088392,"kb_used_data":12650248,"kb_used_omap":19,"kb_used_meta":2438060,"kb_avail":734872824,"statfs":{"total":767960285184,"available":752509771776,"internally_reserved":0,"allocated":12953853952,"data_stored":12948794446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20033,"internal_metadata":2496573887},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":971046912,"num_objects":14817,"num_object_clones":0,"num_object_copies":44451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14817,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":14817,"num_write_kb":948288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.918414"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":177,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.971078+0000","last_change":"2024-09-10T18:28:14.076324+0000","last_active":"2024-09-10T18:28:30.971078+0000","last_peered":"2024-09-10T18:28:30.971078+0000","last_clean":"2024-09-10T18:28:30.971078+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:28:30.971078+0000","last_undegraded":"2024-09-10T18:28:30.971078+0000","last_fullsized":"2024-09-10T18:28:30.971078+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:14.076156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:28:14.076156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:43.937320+0000","scrub_duration":0.003669664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":197,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.961374+0000","last_change":"2024-09-10T18:27:11.977624+0000","last_active":"2024-09-10T18:28:30.961374+0000","last_peered":"2024-09-10T18:28:30.961374+0000","last_clean":"2024-09-10T18:28:30.961374+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:28:30.961374+0000","last_undegraded":"2024-09-10T18:28:30.961374+0000","last_fullsized":"2024-09-10T18:28:30.961374+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:27:11.977562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:27:11.977562+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012415305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":172,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.968543+0000","last_change":"2024-09-10T18:26:57.717081+0000","last_active":"2024-09-10T18:28:30.968543+0000","last_peered":"2024-09-10T18:28:30.968543+0000","last_clean":"2024-09-10T18:28:30.968543+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:28:30.968543+0000","last_undegraded":"2024-09-10T18:28:30.968543+0000","last_fullsized":"2024-09-10T18:28:30.968543+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:57.716697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:26:57.716697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004885314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":170,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.961927+0000","last_change":"2024-09-10T18:26:18.749617+0000","last_active":"2024-09-10T18:28:30.961927+0000","last_peered":"2024-09-10T18:28:30.961927+0000","last_clean":"2024-09-10T18:28:30.961927+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:28:30.961927+0000","last_undegraded":"2024-09-10T18:28:30.961927+0000","last_fullsized":"2024-09-10T18:28:30.961927+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:26:18.749558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090834959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":179,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962290+0000","last_change":"2024-09-10T18:27:10.424485+0000","last_active":"2024-09-10T18:28:30.962290+0000","last_peered":"2024-09-10T18:28:30.962290+0000","last_clean":"2024-09-10T18:28:30.962290+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:28:30.962290+0000","last_undegraded":"2024-09-10T18:28:30.962290+0000","last_fullsized":"2024-09-10T18:28:30.962290+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:10.424386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:10.424386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:34.328097+0000","scrub_duration":0.023425607000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":185,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.963111+0000","last_change":"2024-09-10T18:27:14.539312+0000","last_active":"2024-09-10T18:28:30.963111+0000","last_peered":"2024-09-10T18:28:30.963111+0000","last_clean":"2024-09-10T18:28:30.963111+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:28:30.963111+0000","last_undegraded":"2024-09-10T18:28:30.963111+0000","last_fullsized":"2024-09-10T18:28:30.963111+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:27:14.532784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063752030000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"123'22","reported_seq":208,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.961957+0000","last_change":"2024-09-10T18:28:15.221269+0000","last_active":"2024-09-10T18:28:30.961957+0000","last_peered":"2024-09-10T18:28:30.961957+0000","last_clean":"2024-09-10T18:28:30.961957+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:28:30.961957+0000","last_undegraded":"2024-09-10T18:28:30.961957+0000","last_fullsized":"2024-09-10T18:28:30.961957+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'22","last_scrub_stamp":"2024-09-10T18:28:15.221190+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:28:15.221190+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:25.690656+0000","scrub_duration":0.062041365000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":182,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.961337+0000","last_change":"2024-09-10T18:27:52.257483+0000","last_active":"2024-09-10T18:28:30.961337+0000","last_peered":"2024-09-10T18:28:30.961337+0000","last_clean":"2024-09-10T18:28:30.961337+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:28:30.961337+0000","last_undegraded":"2024-09-10T18:28:30.961337+0000","last_fullsized":"2024-09-10T18:28:30.961337+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.257417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.257417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:56.580125+0000","scrub_duration":0.001302342,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":187,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.965329+0000","last_change":"2024-09-10T18:27:53.668356+0000","last_active":"2024-09-10T18:28:30.965329+0000","last_peered":"2024-09-10T18:28:30.965329+0000","last_clean":"2024-09-10T18:28:30.965329+0000","last_became_active":"2024-09-10T18:26:27.644961+0000","last_became_peered":"2024-09-10T18:26:27.644961+0000","last_unstale":"2024-09-10T18:28:30.965329+0000","last_undegraded":"2024-09-10T18:28:30.965329+0000","last_fullsized":"2024-09-10T18:28:30.965329+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:53.668292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:27:53.668292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:03.826593+0000","scrub_duration":0.001492155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":190,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962181+0000","last_change":"2024-09-10T18:26:11.787888+0000","last_active":"2024-09-10T18:28:30.962181+0000","last_peered":"2024-09-10T18:28:30.962181+0000","last_clean":"2024-09-10T18:28:30.962181+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:28:30.962181+0000","last_undegraded":"2024-09-10T18:28:30.962181+0000","last_fullsized":"2024-09-10T18:28:30.962181+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:26:11.787799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:11.787799+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081486470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":183,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.966323+0000","last_change":"2024-09-10T18:28:13.706380+0000","last_active":"2024-09-10T18:28:30.966323+0000","last_peered":"2024-09-10T18:28:30.966323+0000","last_clean":"2024-09-10T18:28:30.966323+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:28:30.966323+0000","last_undegraded":"2024-09-10T18:28:30.966323+0000","last_fullsized":"2024-09-10T18:28:30.966323+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:13.706259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:13.706259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:17.655828+0000","scrub_duration":0.039021994999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":181,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.977214+0000","last_change":"2024-09-10T18:28:11.043011+0000","last_active":"2024-09-10T18:28:30.977214+0000","last_peered":"2024-09-10T18:28:30.977214+0000","last_clean":"2024-09-10T18:28:30.977214+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:28:30.977214+0000","last_undegraded":"2024-09-10T18:28:30.977214+0000","last_fullsized":"2024-09-10T18:28:30.977214+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:11.042929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:28:11.042929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:30.888869+0000","scrub_duration":0.0032674520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":177,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.966716+0000","last_change":"2024-09-10T18:26:17.464263+0000","last_active":"2024-09-10T18:28:30.966716+0000","last_peered":"2024-09-10T18:28:30.966716+0000","last_clean":"2024-09-10T18:28:30.966716+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:28:30.966716+0000","last_undegraded":"2024-09-10T18:28:30.966716+0000","last_fullsized":"2024-09-10T18:28:30.966716+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:17.464205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:17.464205+0000","last_clean_scrub_stamp":"2024-09-10T18:26:17.464205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024019000000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"85'7","reported_seq":192,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962066+0000","last_change":"2024-09-10T18:25:59.870084+0000","last_active":"2024-09-10T18:28:30.962066+0000","last_peered":"2024-09-10T18:28:30.962066+0000","last_clean":"2024-09-10T18:28:30.962066+0000","last_became_active":"2024-09-10T18:21:33.615429+0000","last_became_peered":"2024-09-10T18:21:33.615429+0000","last_unstale":"2024-09-10T18:28:30.962066+0000","last_undegraded":"2024-09-10T18:28:30.962066+0000","last_fullsized":"2024-09-10T18:28:30.962066+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'7","last_scrub_stamp":"2024-09-10T18:25:59.869847+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:25:59.869847+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013127822000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":182,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.969084+0000","last_change":"2024-09-10T18:28:10.328366+0000","last_active":"2024-09-10T18:28:30.969084+0000","last_peered":"2024-09-10T18:28:30.969084+0000","last_clean":"2024-09-10T18:28:30.969084+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:28:30.969084+0000","last_undegraded":"2024-09-10T18:28:30.969084+0000","last_fullsized":"2024-09-10T18:28:30.969084+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:10.328277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:10.328277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:24.423649+0000","scrub_duration":0.002745838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":174,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.965083+0000","last_change":"2024-09-10T18:28:07.786204+0000","last_active":"2024-09-10T18:28:30.965083+0000","last_peered":"2024-09-10T18:28:30.965083+0000","last_clean":"2024-09-10T18:28:30.965083+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:28:30.965083+0000","last_undegraded":"2024-09-10T18:28:30.965083+0000","last_fullsized":"2024-09-10T18:28:30.965083+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:28:07.786118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:13.460051+0000","scrub_duration":0.0053539800000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":180,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962531+0000","last_change":"2024-09-10T18:27:55.242815+0000","last_active":"2024-09-10T18:28:30.962531+0000","last_peered":"2024-09-10T18:28:30.962531+0000","last_clean":"2024-09-10T18:28:30.962531+0000","last_became_active":"2024-09-10T18:26:27.650774+0000","last_became_peered":"2024-09-10T18:26:27.650774+0000","last_unstale":"2024-09-10T18:28:30.962531+0000","last_undegraded":"2024-09-10T18:28:30.962531+0000","last_fullsized":"2024-09-10T18:28:30.962531+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:55.242751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:55.242751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:02.100445+0000","scrub_duration":0.001486242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":167,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.975167+0000","last_change":"2024-09-10T18:26:14.600755+0000","last_active":"2024-09-10T18:28:30.975167+0000","last_peered":"2024-09-10T18:28:30.975167+0000","last_clean":"2024-09-10T18:28:30.975167+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:28:30.975167+0000","last_undegraded":"2024-09-10T18:28:30.975167+0000","last_fullsized":"2024-09-10T18:28:30.975167+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:14.600680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:26:14.600680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050859140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":209,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.976953+0000","last_change":"2024-09-10T18:27:11.173886+0000","last_active":"2024-09-10T18:28:30.976953+0000","last_peered":"2024-09-10T18:28:30.976953+0000","last_clean":"2024-09-10T18:28:30.976953+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:28:30.976953+0000","last_undegraded":"2024-09-10T18:28:30.976953+0000","last_fullsized":"2024-09-10T18:28:30.976953+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-10T18:27:11.173783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:27:11.173783+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057202850000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":182,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.967918+0000","last_change":"2024-09-10T18:28:01.515411+0000","last_active":"2024-09-10T18:28:30.967918+0000","last_peered":"2024-09-10T18:28:30.967918+0000","last_clean":"2024-09-10T18:28:30.967918+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:28:30.967918+0000","last_undegraded":"2024-09-10T18:28:30.967918+0000","last_fullsized":"2024-09-10T18:28:30.967918+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:01.515358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:01.515358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:20.752464+0000","scrub_duration":0.0019375149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":179,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.963571+0000","last_change":"2024-09-10T18:27:15.496208+0000","last_active":"2024-09-10T18:28:30.963571+0000","last_peered":"2024-09-10T18:28:30.963571+0000","last_clean":"2024-09-10T18:28:30.963571+0000","last_became_active":"2024-09-10T18:26:27.640035+0000","last_became_peered":"2024-09-10T18:26:27.640035+0000","last_unstale":"2024-09-10T18:28:30.963571+0000","last_undegraded":"2024-09-10T18:28:30.963571+0000","last_fullsized":"2024-09-10T18:28:30.963571+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:15.496116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:15.496116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058500619999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":176,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962316+0000","last_change":"2024-09-10T18:26:13.774886+0000","last_active":"2024-09-10T18:28:30.962316+0000","last_peered":"2024-09-10T18:28:30.962316+0000","last_clean":"2024-09-10T18:28:30.962316+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:28:30.962316+0000","last_undegraded":"2024-09-10T18:28:30.962316+0000","last_fullsized":"2024-09-10T18:28:30.962316+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:26:13.774791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006579774,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"88'17","reported_seq":204,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.961922+0000","last_change":"2024-09-10T18:28:13.123489+0000","last_active":"2024-09-10T18:28:30.961922+0000","last_peered":"2024-09-10T18:28:30.961922+0000","last_clean":"2024-09-10T18:28:30.961922+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:28:30.961922+0000","last_undegraded":"2024-09-10T18:28:30.961922+0000","last_fullsized":"2024-09-10T18:28:30.961922+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-09-10T18:28:13.123366+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:28:13.123366+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:39.954200+0000","scrub_duration":0.0060192420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":172,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962437+0000","last_change":"2024-09-10T18:26:09.777629+0000","last_active":"2024-09-10T18:28:30.962437+0000","last_peered":"2024-09-10T18:28:30.962437+0000","last_clean":"2024-09-10T18:28:30.962437+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:28:30.962437+0000","last_undegraded":"2024-09-10T18:28:30.962437+0000","last_fullsized":"2024-09-10T18:28:30.962437+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:09.777548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:26:09.777548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031831369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":172,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.970169+0000","last_change":"2024-09-10T18:27:17.955653+0000","last_active":"2024-09-10T18:28:30.970169+0000","last_peered":"2024-09-10T18:28:30.970169+0000","last_clean":"2024-09-10T18:28:30.970169+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:28:30.970169+0000","last_undegraded":"2024-09-10T18:28:30.970169+0000","last_fullsized":"2024-09-10T18:28:30.970169+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:27:17.955591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:17.955591+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:33.349856+0000","scrub_duration":0.0035877499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":186,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.968669+0000","last_change":"2024-09-10T18:28:05.532634+0000","last_active":"2024-09-10T18:28:30.968669+0000","last_peered":"2024-09-10T18:28:30.968669+0000","last_clean":"2024-09-10T18:28:30.968669+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:28:30.968669+0000","last_undegraded":"2024-09-10T18:28:30.968669+0000","last_fullsized":"2024-09-10T18:28:30.968669+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:05.532581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:05.532581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:15.324758+0000","scrub_duration":0.001369901,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"86'15","reported_seq":195,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.971386+0000","last_change":"2024-09-10T18:27:09.623828+0000","last_active":"2024-09-10T18:28:30.971386+0000","last_peered":"2024-09-10T18:28:30.971386+0000","last_clean":"2024-09-10T18:28:30.971386+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:28:30.971386+0000","last_undegraded":"2024-09-10T18:28:30.971386+0000","last_fullsized":"2024-09-10T18:28:30.971386+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-09-10T18:27:09.623769+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:27:09.623769+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001906717,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"146'10","reported_seq":168,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:34.838547+0000","last_change":"2024-09-10T18:26:52.215856+0000","last_active":"2024-09-10T18:28:34.838547+0000","last_peered":"2024-09-10T18:28:34.838547+0000","last_clean":"2024-09-10T18:28:34.838547+0000","last_became_active":"2024-09-10T18:26:27.652489+0000","last_became_peered":"2024-09-10T18:26:27.652489+0000","last_unstale":"2024-09-10T18:28:34.838547+0000","last_undegraded":"2024-09-10T18:28:34.838547+0000","last_fullsized":"2024-09-10T18:28:34.838547+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'10","last_scrub_stamp":"2024-09-10T18:26:52.215776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:26:52.215776+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017419169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"197'4175","reported_seq":4200,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:39.906608+0000","last_change":"2024-09-10T18:27:53.036521+0000","last_active":"2024-09-10T18:28:39.906608+0000","last_peered":"2024-09-10T18:28:39.906608+0000","last_clean":"2024-09-10T18:28:39.906608+0000","last_became_active":"2024-09-10T18:27:53.036244+0000","last_became_peered":"2024-09-10T18:27:53.036244+0000","last_unstale":"2024-09-10T18:28:39.906608+0000","last_undegraded":"2024-09-10T18:28:39.906608+0000","last_fullsized":"2024-09-10T18:28:39.906608+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:59.426828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273612800,"num_objects":4175,"num_object_clones":0,"num_object_copies":12525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4175,"num_write_kb":267200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":93,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.971420+0000","last_change":"2024-09-10T18:27:58.699433+0000","last_active":"2024-09-10T18:28:30.971420+0000","last_peered":"2024-09-10T18:28:30.971420+0000","last_clean":"2024-09-10T18:28:30.971420+0000","last_became_active":"2024-09-10T18:26:28.635249+0000","last_became_peered":"2024-09-10T18:26:28.635249+0000","last_unstale":"2024-09-10T18:28:30.971420+0000","last_undegraded":"2024-09-10T18:28:30.971420+0000","last_fullsized":"2024-09-10T18:28:30.971420+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:27:58.699362+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:27:58.699362+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:13.603324+0000","scrub_duration":0.0015816230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"83'22","reported_seq":219,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.968360+0000","last_change":"2024-09-10T18:27:59.479143+0000","last_active":"2024-09-10T18:28:30.968360+0000","last_peered":"2024-09-10T18:28:30.968360+0000","last_clean":"2024-09-10T18:28:30.968360+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:28:30.968360+0000","last_undegraded":"2024-09-10T18:28:30.968360+0000","last_fullsized":"2024-09-10T18:28:30.968360+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-09-10T18:27:59.479087+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:27:59.479087+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:17.236778+0000","scrub_duration":0.0049225479999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":178,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962433+0000","last_change":"2024-09-10T18:28:00.151515+0000","last_active":"2024-09-10T18:28:30.962433+0000","last_peered":"2024-09-10T18:28:30.962433+0000","last_clean":"2024-09-10T18:28:30.962433+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:28:30.962433+0000","last_undegraded":"2024-09-10T18:28:30.962433+0000","last_fullsized":"2024-09-10T18:28:30.962433+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:00.151462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:00.151462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:11.741967+0000","scrub_duration":0.0032292530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":268,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:31.975024+0000","last_change":"2024-09-10T18:27:58.231688+0000","last_active":"2024-09-10T18:28:31.975024+0000","last_peered":"2024-09-10T18:28:31.975024+0000","last_clean":"2024-09-10T18:28:31.975024+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:28:31.975024+0000","last_undegraded":"2024-09-10T18:28:31.975024+0000","last_fullsized":"2024-09-10T18:28:31.975024+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:27:58.231620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:58.231620+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:13.570474+0000","scrub_duration":0.0016723319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":171,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.965011+0000","last_change":"2024-09-10T18:26:21.511753+0000","last_active":"2024-09-10T18:28:30.965011+0000","last_peered":"2024-09-10T18:28:30.965011+0000","last_clean":"2024-09-10T18:28:30.965011+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:28:30.965011+0000","last_undegraded":"2024-09-10T18:28:30.965011+0000","last_fullsized":"2024-09-10T18:28:30.965011+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:26:21.511676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096344640000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"197'4007","reported_seq":4032,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:40.379563+0000","last_change":"2024-09-10T18:27:53.059901+0000","last_active":"2024-09-10T18:28:40.379563+0000","last_peered":"2024-09-10T18:28:40.379563+0000","last_clean":"2024-09-10T18:28:40.379563+0000","last_became_active":"2024-09-10T18:27:53.059554+0000","last_became_peered":"2024-09-10T18:27:53.059554+0000","last_unstale":"2024-09-10T18:28:40.379563+0000","last_undegraded":"2024-09-10T18:28:40.379563+0000","last_fullsized":"2024-09-10T18:28:40.379563+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:55.250255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262602752,"num_objects":4007,"num_object_clones":0,"num_object_copies":12021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4007,"num_write_kb":256448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":92,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.963503+0000","last_change":"2024-09-10T18:28:07.183739+0000","last_active":"2024-09-10T18:28:30.963503+0000","last_peered":"2024-09-10T18:28:30.963503+0000","last_clean":"2024-09-10T18:28:30.963503+0000","last_became_active":"2024-09-10T18:26:28.632744+0000","last_became_peered":"2024-09-10T18:26:28.632744+0000","last_unstale":"2024-09-10T18:28:30.963503+0000","last_undegraded":"2024-09-10T18:28:30.963503+0000","last_fullsized":"2024-09-10T18:28:30.963503+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:28:07.183646+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:28:07.183646+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:30.511324+0000","scrub_duration":0.001501517,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":208,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962211+0000","last_change":"2024-09-10T18:28:03.152360+0000","last_active":"2024-09-10T18:28:30.962211+0000","last_peered":"2024-09-10T18:28:30.962211+0000","last_clean":"2024-09-10T18:28:30.962211+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:28:30.962211+0000","last_undegraded":"2024-09-10T18:28:30.962211+0000","last_fullsized":"2024-09-10T18:28:30.962211+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-10T18:28:03.152304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:28:03.152304+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:27.847090+0000","scrub_duration":0.002028069,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":178,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.964039+0000","last_change":"2024-09-10T18:28:05.141025+0000","last_active":"2024-09-10T18:28:30.964039+0000","last_peered":"2024-09-10T18:28:30.964039+0000","last_clean":"2024-09-10T18:28:30.964039+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:28:30.964039+0000","last_undegraded":"2024-09-10T18:28:30.964039+0000","last_fullsized":"2024-09-10T18:28:30.964039+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:05.140922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.140922+0000","last_clean_scrub_stamp":"2024-09-10T18:28:05.140922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:32.018170+0000","scrub_duration":0.0017057179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":172,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.961669+0000","last_change":"2024-09-10T18:26:07.791694+0000","last_active":"2024-09-10T18:28:30.961669+0000","last_peered":"2024-09-10T18:28:30.961669+0000","last_clean":"2024-09-10T18:28:30.961669+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:28:30.961669+0000","last_undegraded":"2024-09-10T18:28:30.961669+0000","last_fullsized":"2024-09-10T18:28:30.961669+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:07.791645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:26:07.791645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003302878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":170,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962105+0000","last_change":"2024-09-10T18:28:04.199803+0000","last_active":"2024-09-10T18:28:30.962105+0000","last_peered":"2024-09-10T18:28:30.962105+0000","last_clean":"2024-09-10T18:28:30.962105+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:28:30.962105+0000","last_undegraded":"2024-09-10T18:28:30.962105+0000","last_fullsized":"2024-09-10T18:28:30.962105+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:04.199730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:28:04.199730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:30.291668+0000","scrub_duration":0.001636628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"197'4248","reported_seq":4273,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:39.751619+0000","last_change":"2024-09-10T18:27:53.061607+0000","last_active":"2024-09-10T18:28:39.751619+0000","last_peered":"2024-09-10T18:28:39.751619+0000","last_clean":"2024-09-10T18:28:39.751619+0000","last_became_active":"2024-09-10T18:27:53.061195+0000","last_became_peered":"2024-09-10T18:27:53.061195+0000","last_unstale":"2024-09-10T18:28:39.751619+0000","last_undegraded":"2024-09-10T18:28:39.751619+0000","last_fullsized":"2024-09-10T18:28:39.751619+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4248,"log_dups_size":0,"ondisk_log_size":4248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:04.794992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278396928,"num_objects":4248,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4248,"num_write_kb":271872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":83,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962119+0000","last_change":"2024-09-10T18:28:12.086525+0000","last_active":"2024-09-10T18:28:30.962119+0000","last_peered":"2024-09-10T18:28:30.962119+0000","last_clean":"2024-09-10T18:28:30.962119+0000","last_became_active":"2024-09-10T18:26:18.091896+0000","last_became_peered":"2024-09-10T18:26:18.091896+0000","last_unstale":"2024-09-10T18:28:30.962119+0000","last_undegraded":"2024-09-10T18:28:30.962119+0000","last_fullsized":"2024-09-10T18:28:30.962119+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:28:12.086437+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:28:12.086437+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:27.212845+0000","scrub_duration":0.0048662799999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":202,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.961456+0000","last_change":"2024-09-10T18:26:49.014982+0000","last_active":"2024-09-10T18:28:30.961456+0000","last_peered":"2024-09-10T18:28:30.961456+0000","last_clean":"2024-09-10T18:28:30.961456+0000","last_became_active":"2024-09-10T18:26:27.651364+0000","last_became_peered":"2024-09-10T18:26:27.651364+0000","last_unstale":"2024-09-10T18:28:30.961456+0000","last_undegraded":"2024-09-10T18:28:30.961456+0000","last_fullsized":"2024-09-10T18:28:30.961456+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-10T18:26:49.014922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:49.014922+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040639739999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":177,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.971586+0000","last_change":"2024-09-10T18:27:57.056931+0000","last_active":"2024-09-10T18:28:30.971586+0000","last_peered":"2024-09-10T18:28:30.971586+0000","last_clean":"2024-09-10T18:28:30.971586+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:28:30.971586+0000","last_undegraded":"2024-09-10T18:28:30.971586+0000","last_fullsized":"2024-09-10T18:28:30.971586+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:57.056876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:27:57.056876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:10.364135+0000","scrub_duration":0.0022605429999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":269,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:34.837815+0000","last_change":"2024-09-10T18:28:08.161477+0000","last_active":"2024-09-10T18:28:34.837815+0000","last_peered":"2024-09-10T18:28:34.837815+0000","last_clean":"2024-09-10T18:28:34.837815+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:28:34.837815+0000","last_undegraded":"2024-09-10T18:28:34.837815+0000","last_fullsized":"2024-09-10T18:28:34.837815+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:28:08.161384+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:28:08.161384+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:31.405936+0000","scrub_duration":0.044251862000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":180,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.968387+0000","last_change":"2024-09-10T18:27:52.101815+0000","last_active":"2024-09-10T18:28:30.968387+0000","last_peered":"2024-09-10T18:28:30.968387+0000","last_clean":"2024-09-10T18:28:30.968387+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:28:30.968387+0000","last_undegraded":"2024-09-10T18:28:30.968387+0000","last_fullsized":"2024-09-10T18:28:30.968387+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.101751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:02.011610+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.101751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:08.114132+0000","scrub_duration":0.00178638,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"197'4196","reported_seq":4221,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:40.375315+0000","last_change":"2024-09-10T18:27:53.034877+0000","last_active":"2024-09-10T18:28:40.375315+0000","last_peered":"2024-09-10T18:28:40.375315+0000","last_clean":"2024-09-10T18:28:40.375315+0000","last_became_active":"2024-09-10T18:27:53.034171+0000","last_became_peered":"2024-09-10T18:27:53.034171+0000","last_unstale":"2024-09-10T18:28:40.375315+0000","last_undegraded":"2024-09-10T18:28:40.375315+0000","last_fullsized":"2024-09-10T18:28:40.375315+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4196,"log_dups_size":0,"ondisk_log_size":4196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:57.932544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274989056,"num_objects":4196,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4196,"num_write_kb":268544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":132,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.966227+0000","last_change":"2024-09-10T18:28:00.351507+0000","last_active":"2024-09-10T18:28:30.966227+0000","last_peered":"2024-09-10T18:28:30.966227+0000","last_clean":"2024-09-10T18:28:30.966227+0000","last_became_active":"2024-09-10T18:26:28.632849+0000","last_became_peered":"2024-09-10T18:26:28.632849+0000","last_unstale":"2024-09-10T18:28:30.966227+0000","last_undegraded":"2024-09-10T18:28:30.966227+0000","last_fullsized":"2024-09-10T18:28:30.966227+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:28:00.351448+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:28:00.351448+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:27.185208+0000","scrub_duration":0.0013475130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"123'15","reported_seq":204,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.973551+0000","last_change":"2024-09-10T18:27:14.653748+0000","last_active":"2024-09-10T18:28:30.973551+0000","last_peered":"2024-09-10T18:28:30.973551+0000","last_clean":"2024-09-10T18:28:30.973551+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:28:30.973551+0000","last_undegraded":"2024-09-10T18:28:30.973551+0000","last_fullsized":"2024-09-10T18:28:30.973551+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'15","last_scrub_stamp":"2024-09-10T18:27:14.653658+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:25:18.356218+0000","last_clean_scrub_stamp":"2024-09-10T18:27:14.653658+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062664110000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":177,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.965401+0000","last_change":"2024-09-10T18:28:03.772302+0000","last_active":"2024-09-10T18:28:30.965401+0000","last_peered":"2024-09-10T18:28:30.965401+0000","last_clean":"2024-09-10T18:28:30.965401+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:28:30.965401+0000","last_undegraded":"2024-09-10T18:28:30.965401+0000","last_fullsized":"2024-09-10T18:28:30.965401+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:28:03.772241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:20.727151+0000","scrub_duration":0.0014939619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"146'10","reported_seq":266,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:34.838352+0000","last_change":"2024-09-10T18:26:47.858607+0000","last_active":"2024-09-10T18:28:34.838352+0000","last_peered":"2024-09-10T18:28:34.838352+0000","last_clean":"2024-09-10T18:28:34.838352+0000","last_became_active":"2024-09-10T18:26:27.639032+0000","last_became_peered":"2024-09-10T18:26:27.639032+0000","last_unstale":"2024-09-10T18:28:34.838352+0000","last_undegraded":"2024-09-10T18:28:34.838352+0000","last_fullsized":"2024-09-10T18:28:34.838352+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'10","last_scrub_stamp":"2024-09-10T18:26:47.858542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:26:47.858542+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001506688,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":171,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.977413+0000","last_change":"2024-09-10T18:26:17.657969+0000","last_active":"2024-09-10T18:28:30.977413+0000","last_peered":"2024-09-10T18:28:30.977413+0000","last_clean":"2024-09-10T18:28:30.977413+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:28:30.977413+0000","last_undegraded":"2024-09-10T18:28:30.977413+0000","last_fullsized":"2024-09-10T18:28:30.977413+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:17.657880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:26:17.657880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018451012999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"197'4177","reported_seq":4202,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:40.299923+0000","last_change":"2024-09-10T18:27:53.037694+0000","last_active":"2024-09-10T18:28:40.299923+0000","last_peered":"2024-09-10T18:28:40.299923+0000","last_clean":"2024-09-10T18:28:40.299923+0000","last_became_active":"2024-09-10T18:27:53.037210+0000","last_became_peered":"2024-09-10T18:27:53.037210+0000","last_unstale":"2024-09-10T18:28:40.299923+0000","last_undegraded":"2024-09-10T18:28:40.299923+0000","last_fullsized":"2024-09-10T18:28:40.299923+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4177,"log_dups_size":0,"ondisk_log_size":4177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:54.052884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273743872,"num_objects":4177,"num_object_clones":0,"num_object_copies":12531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4177,"num_write_kb":267328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":132,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.967484+0000","last_change":"2024-09-10T18:27:53.479157+0000","last_active":"2024-09-10T18:28:30.967484+0000","last_peered":"2024-09-10T18:28:30.967484+0000","last_clean":"2024-09-10T18:28:30.967484+0000","last_became_active":"2024-09-10T18:26:28.635542+0000","last_became_peered":"2024-09-10T18:26:28.635542+0000","last_unstale":"2024-09-10T18:28:30.967484+0000","last_undegraded":"2024-09-10T18:28:30.967484+0000","last_fullsized":"2024-09-10T18:28:30.967484+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:27:53.479083+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:27:53.479083+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:11.027081+0000","scrub_duration":0.0017291470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":211,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.974754+0000","last_change":"2024-09-10T18:27:18.798491+0000","last_active":"2024-09-10T18:28:30.974754+0000","last_peered":"2024-09-10T18:28:30.974754+0000","last_clean":"2024-09-10T18:28:30.974754+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:28:30.974754+0000","last_undegraded":"2024-09-10T18:28:30.974754+0000","last_fullsized":"2024-09-10T18:28:30.974754+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-10T18:27:18.798402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:27:18.798402+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.020618556999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":176,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.967561+0000","last_change":"2024-09-10T18:28:14.418351+0000","last_active":"2024-09-10T18:28:30.967561+0000","last_peered":"2024-09-10T18:28:30.967561+0000","last_clean":"2024-09-10T18:28:30.967561+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:28:30.967561+0000","last_undegraded":"2024-09-10T18:28:30.967561+0000","last_fullsized":"2024-09-10T18:28:30.967561+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:14.418260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:14.418260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:19.624601+0000","scrub_duration":0.0049147540000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":122,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.965496+0000","last_change":"2024-09-10T18:28:14.399029+0000","last_active":"2024-09-10T18:28:30.965496+0000","last_peered":"2024-09-10T18:28:30.965496+0000","last_clean":"2024-09-10T18:28:30.965496+0000","last_became_active":"2024-09-10T18:26:27.642063+0000","last_became_peered":"2024-09-10T18:26:27.642063+0000","last_unstale":"2024-09-10T18:28:30.965496+0000","last_undegraded":"2024-09-10T18:28:30.965496+0000","last_fullsized":"2024-09-10T18:28:30.965496+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:14.398929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:14.398929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:38.005890+0000","scrub_duration":0.0024148870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":177,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.968476+0000","last_change":"2024-09-10T18:28:15.471450+0000","last_active":"2024-09-10T18:28:30.968476+0000","last_peered":"2024-09-10T18:28:30.968476+0000","last_clean":"2024-09-10T18:28:30.968476+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:28:30.968476+0000","last_undegraded":"2024-09-10T18:28:30.968476+0000","last_fullsized":"2024-09-10T18:28:30.968476+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:15.471241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:28:15.471241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:19.536572+0000","scrub_duration":0.0086976859999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"197'4061","reported_seq":4086,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:39.756237+0000","last_change":"2024-09-10T18:27:53.062042+0000","last_active":"2024-09-10T18:28:39.756237+0000","last_peered":"2024-09-10T18:28:39.756237+0000","last_clean":"2024-09-10T18:28:39.756237+0000","last_became_active":"2024-09-10T18:27:53.061113+0000","last_became_peered":"2024-09-10T18:27:53.061113+0000","last_unstale":"2024-09-10T18:28:39.756237+0000","last_undegraded":"2024-09-10T18:28:39.756237+0000","last_fullsized":"2024-09-10T18:28:39.756237+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4061,"log_dups_size":0,"ondisk_log_size":4061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:58.198558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266141696,"num_objects":4061,"num_object_clones":0,"num_object_copies":12183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4061,"num_write_kb":259904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":78,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.964898+0000","last_change":"2024-09-10T18:27:54.717632+0000","last_active":"2024-09-10T18:28:30.964898+0000","last_peered":"2024-09-10T18:28:30.964898+0000","last_clean":"2024-09-10T18:28:30.964898+0000","last_became_active":"2024-09-10T18:26:18.094747+0000","last_became_peered":"2024-09-10T18:26:18.094747+0000","last_unstale":"2024-09-10T18:28:30.964898+0000","last_undegraded":"2024-09-10T18:28:30.964898+0000","last_fullsized":"2024-09-10T18:28:30.964898+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:27:54.717560+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:27:54.717560+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:15.547845+0000","scrub_duration":0.0015444320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"90'29","reported_seq":226,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:31.979977+0000","last_change":"2024-09-10T18:26:19.509647+0000","last_active":"2024-09-10T18:28:31.979977+0000","last_peered":"2024-09-10T18:28:31.979977+0000","last_clean":"2024-09-10T18:28:31.979977+0000","last_became_active":"2024-09-10T18:23:10.067049+0000","last_became_peered":"2024-09-10T18:23:10.067049+0000","last_unstale":"2024-09-10T18:28:31.979977+0000","last_undegraded":"2024-09-10T18:28:31.979977+0000","last_fullsized":"2024-09-10T18:28:31.979977+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'29","last_scrub_stamp":"2024-09-10T18:26:19.509580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:19.509580+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063679319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":183,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962478+0000","last_change":"2024-09-10T18:26:47.093062+0000","last_active":"2024-09-10T18:28:30.962478+0000","last_peered":"2024-09-10T18:28:30.962478+0000","last_clean":"2024-09-10T18:28:30.962478+0000","last_became_active":"2024-09-10T18:26:27.648690+0000","last_became_peered":"2024-09-10T18:26:27.648690+0000","last_unstale":"2024-09-10T18:28:30.962478+0000","last_undegraded":"2024-09-10T18:28:30.962478+0000","last_fullsized":"2024-09-10T18:28:30.962478+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:47.093004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:26:47.093004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014500660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"62'2","reported_seq":191,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.971936+0000","last_change":"2024-09-10T18:26:54.562292+0000","last_active":"2024-09-10T18:28:30.971936+0000","last_peered":"2024-09-10T18:28:30.971936+0000","last_clean":"2024-09-10T18:28:30.971936+0000","last_became_active":"2024-09-10T18:26:27.647576+0000","last_became_peered":"2024-09-10T18:26:27.647576+0000","last_unstale":"2024-09-10T18:28:30.971936+0000","last_undegraded":"2024-09-10T18:28:30.971936+0000","last_fullsized":"2024-09-10T18:28:30.971936+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-10T18:26:54.562210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:26:54.562210+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023324999999999999,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":173,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.981933+0000","last_change":"2024-09-10T18:27:12.683381+0000","last_active":"2024-09-10T18:28:30.981933+0000","last_peered":"2024-09-10T18:28:30.981933+0000","last_clean":"2024-09-10T18:28:30.981933+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:28:30.981933+0000","last_undegraded":"2024-09-10T18:28:30.981933+0000","last_fullsized":"2024-09-10T18:28:30.981933+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:12.683294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:12.683294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:34.792552+0000","scrub_duration":0.0042722790000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":172,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962911+0000","last_change":"2024-09-10T18:28:07.121517+0000","last_active":"2024-09-10T18:28:30.962911+0000","last_peered":"2024-09-10T18:28:30.962911+0000","last_clean":"2024-09-10T18:28:30.962911+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:28:30.962911+0000","last_undegraded":"2024-09-10T18:28:30.962911+0000","last_fullsized":"2024-09-10T18:28:30.962911+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:07.121433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:28:07.121433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:34.298348+0000","scrub_duration":0.0044980100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"197'4136","reported_seq":4161,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:40.291616+0000","last_change":"2024-09-10T18:27:53.037955+0000","last_active":"2024-09-10T18:28:40.291616+0000","last_peered":"2024-09-10T18:28:40.291616+0000","last_clean":"2024-09-10T18:28:40.291616+0000","last_became_active":"2024-09-10T18:27:53.037246+0000","last_became_peered":"2024-09-10T18:27:53.037246+0000","last_unstale":"2024-09-10T18:28:40.291616+0000","last_undegraded":"2024-09-10T18:28:40.291616+0000","last_fullsized":"2024-09-10T18:28:40.291616+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4136,"log_dups_size":0,"ondisk_log_size":4136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:20.497361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271056896,"num_objects":4136,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4136,"num_write_kb":264704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":224,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.968674+0000","last_change":"2024-09-10T18:27:59.226683+0000","last_active":"2024-09-10T18:28:30.968674+0000","last_peered":"2024-09-10T18:28:30.968674+0000","last_clean":"2024-09-10T18:28:30.968674+0000","last_became_active":"2024-09-10T18:26:18.095568+0000","last_became_peered":"2024-09-10T18:26:18.095568+0000","last_unstale":"2024-09-10T18:28:30.968674+0000","last_undegraded":"2024-09-10T18:28:30.968674+0000","last_fullsized":"2024-09-10T18:28:30.968674+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:27:59.226607+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:27:59.226607+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:22.629088+0000","scrub_duration":0.0014964799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":201,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.968445+0000","last_change":"2024-09-10T18:26:56.714007+0000","last_active":"2024-09-10T18:28:30.968445+0000","last_peered":"2024-09-10T18:28:30.968445+0000","last_clean":"2024-09-10T18:28:30.968445+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:28:30.968445+0000","last_undegraded":"2024-09-10T18:28:30.968445+0000","last_fullsized":"2024-09-10T18:28:30.968445+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-10T18:26:56.713924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:56.713924+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018614987,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":193,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.969903+0000","last_change":"2024-09-10T18:28:16.493397+0000","last_active":"2024-09-10T18:28:30.969903+0000","last_peered":"2024-09-10T18:28:30.969903+0000","last_clean":"2024-09-10T18:28:30.969903+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:28:30.969903+0000","last_undegraded":"2024-09-10T18:28:30.969903+0000","last_fullsized":"2024-09-10T18:28:30.969903+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:16.493304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:16.493304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:43.855805+0000","scrub_duration":0.0045544119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":195,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.978022+0000","last_change":"2024-09-10T18:27:18.122094+0000","last_active":"2024-09-10T18:28:30.978022+0000","last_peered":"2024-09-10T18:28:30.978022+0000","last_clean":"2024-09-10T18:28:30.978022+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:28:30.978022+0000","last_undegraded":"2024-09-10T18:28:30.978022+0000","last_fullsized":"2024-09-10T18:28:30.978022+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:27:18.121985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:27:18.121985+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018606608,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":182,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.980635+0000","last_change":"2024-09-10T18:28:09.069190+0000","last_active":"2024-09-10T18:28:30.980635+0000","last_peered":"2024-09-10T18:28:30.980635+0000","last_clean":"2024-09-10T18:28:30.980635+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:28:30.980635+0000","last_undegraded":"2024-09-10T18:28:30.980635+0000","last_fullsized":"2024-09-10T18:28:30.980635+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:09.069093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:28:09.069093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:26.067294+0000","scrub_duration":0.0055535080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":180,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.972717+0000","last_change":"2024-09-10T18:27:55.152565+0000","last_active":"2024-09-10T18:28:30.972717+0000","last_peered":"2024-09-10T18:28:30.972717+0000","last_clean":"2024-09-10T18:28:30.972717+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:28:30.972717+0000","last_undegraded":"2024-09-10T18:28:30.972717+0000","last_fullsized":"2024-09-10T18:28:30.972717+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:55.152501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:55.152501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:16.874950+0000","scrub_duration":0.001407609,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"197'4103","reported_seq":4128,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:39.883004+0000","last_change":"2024-09-10T18:27:53.034603+0000","last_active":"2024-09-10T18:28:39.883004+0000","last_peered":"2024-09-10T18:28:39.883004+0000","last_clean":"2024-09-10T18:28:39.883004+0000","last_became_active":"2024-09-10T18:27:53.034315+0000","last_became_peered":"2024-09-10T18:27:53.034315+0000","last_unstale":"2024-09-10T18:28:39.883004+0000","last_undegraded":"2024-09-10T18:28:39.883004+0000","last_fullsized":"2024-09-10T18:28:39.883004+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4103,"log_dups_size":0,"ondisk_log_size":4103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:03.080369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268894208,"num_objects":4103,"num_object_clones":0,"num_object_copies":12309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4103,"num_write_kb":262592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":93,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.968030+0000","last_change":"2024-09-10T18:27:57.493055+0000","last_active":"2024-09-10T18:28:30.968030+0000","last_peered":"2024-09-10T18:28:30.968030+0000","last_clean":"2024-09-10T18:28:30.968030+0000","last_became_active":"2024-09-10T18:26:28.641791+0000","last_became_peered":"2024-09-10T18:26:28.641791+0000","last_unstale":"2024-09-10T18:28:30.968030+0000","last_undegraded":"2024-09-10T18:28:30.968030+0000","last_fullsized":"2024-09-10T18:28:30.968030+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:27:57.492998+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:27:57.492998+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:01.205814+0000","scrub_duration":0.0019993160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":193,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.971454+0000","last_change":"2024-09-10T18:27:50.951815+0000","last_active":"2024-09-10T18:28:30.971454+0000","last_peered":"2024-09-10T18:28:30.971454+0000","last_clean":"2024-09-10T18:28:30.971454+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:28:30.971454+0000","last_undegraded":"2024-09-10T18:28:30.971454+0000","last_fullsized":"2024-09-10T18:28:30.971454+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:27:50.951749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:27:50.951749+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:57.289936+0000","scrub_duration":0.003075983,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":177,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.983245+0000","last_change":"2024-09-10T18:27:15.809458+0000","last_active":"2024-09-10T18:28:30.983245+0000","last_peered":"2024-09-10T18:28:30.983245+0000","last_clean":"2024-09-10T18:28:30.983245+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:28:30.983245+0000","last_undegraded":"2024-09-10T18:28:30.983245+0000","last_fullsized":"2024-09-10T18:28:30.983245+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:15.809380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:27:15.809380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086899360000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":615,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:38.308197+0000","last_change":"2024-09-10T18:28:04.343237+0000","last_active":"2024-09-10T18:28:38.308197+0000","last_peered":"2024-09-10T18:28:38.308197+0000","last_clean":"2024-09-10T18:28:38.308197+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:28:38.308197+0000","last_undegraded":"2024-09-10T18:28:38.308197+0000","last_fullsized":"2024-09-10T18:28:38.308197+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:28:04.343138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:04.343138+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:08.341406+0000","scrub_duration":0.0024865109999999998,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":421,"num_read_kb":416,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":176,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.967240+0000","last_change":"2024-09-10T18:28:05.771485+0000","last_active":"2024-09-10T18:28:30.967240+0000","last_peered":"2024-09-10T18:28:30.967240+0000","last_clean":"2024-09-10T18:28:30.967240+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:28:30.967240+0000","last_undegraded":"2024-09-10T18:28:30.967240+0000","last_fullsized":"2024-09-10T18:28:30.967240+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:05.771415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.771415+0000","last_clean_scrub_stamp":"2024-09-10T18:28:05.771415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:23.559543+0000","scrub_duration":0.001873523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":170,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.967266+0000","last_change":"2024-09-10T18:27:56.527307+0000","last_active":"2024-09-10T18:28:30.967266+0000","last_peered":"2024-09-10T18:28:30.967266+0000","last_clean":"2024-09-10T18:28:30.967266+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:28:30.967266+0000","last_undegraded":"2024-09-10T18:28:30.967266+0000","last_fullsized":"2024-09-10T18:28:30.967266+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:56.527234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:56.527234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:19.892724+0000","scrub_duration":0.0018055840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"197'4352","reported_seq":4376,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:40.610778+0000","last_change":"2024-09-10T18:27:53.035571+0000","last_active":"2024-09-10T18:28:40.610778+0000","last_peered":"2024-09-10T18:28:40.610778+0000","last_clean":"2024-09-10T18:28:40.610778+0000","last_became_active":"2024-09-10T18:27:53.035261+0000","last_became_peered":"2024-09-10T18:27:53.035261+0000","last_unstale":"2024-09-10T18:28:40.610778+0000","last_undegraded":"2024-09-10T18:28:40.610778+0000","last_fullsized":"2024-09-10T18:28:40.610778+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:11.723017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285212672,"num_objects":4352,"num_object_clones":0,"num_object_copies":13056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4352,"num_write_kb":278528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":141,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.966862+0000","last_change":"2024-09-10T18:26:55.263707+0000","last_active":"2024-09-10T18:28:30.966862+0000","last_peered":"2024-09-10T18:28:30.966862+0000","last_clean":"2024-09-10T18:28:30.966862+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:28:30.966862+0000","last_undegraded":"2024-09-10T18:28:30.966862+0000","last_fullsized":"2024-09-10T18:28:30.966862+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-10T18:26:55.263623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:55.263623+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003225992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":185,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962063+0000","last_change":"2024-09-10T18:26:55.420188+0000","last_active":"2024-09-10T18:28:30.962063+0000","last_peered":"2024-09-10T18:28:30.962063+0000","last_clean":"2024-09-10T18:28:30.962063+0000","last_became_active":"2024-09-10T18:26:27.639129+0000","last_became_peered":"2024-09-10T18:26:27.639129+0000","last_unstale":"2024-09-10T18:28:30.962063+0000","last_undegraded":"2024-09-10T18:28:30.962063+0000","last_fullsized":"2024-09-10T18:28:30.962063+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:55.420103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:26:55.420103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.055370536999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":226,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.968804+0000","last_change":"2024-09-10T18:28:03.297772+0000","last_active":"2024-09-10T18:28:30.968804+0000","last_peered":"2024-09-10T18:28:30.968804+0000","last_clean":"2024-09-10T18:28:30.968804+0000","last_became_active":"2024-09-10T18:26:27.646658+0000","last_became_peered":"2024-09-10T18:26:27.646658+0000","last_unstale":"2024-09-10T18:28:30.968804+0000","last_undegraded":"2024-09-10T18:28:30.968804+0000","last_fullsized":"2024-09-10T18:28:30.968804+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:28:03.297713+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:28:03.297713+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:05.130955+0000","scrub_duration":0.0015114589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":182,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.969637+0000","last_change":"2024-09-10T18:28:11.367715+0000","last_active":"2024-09-10T18:28:30.969637+0000","last_peered":"2024-09-10T18:28:30.969637+0000","last_clean":"2024-09-10T18:28:30.969637+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:28:30.969637+0000","last_undegraded":"2024-09-10T18:28:30.969637+0000","last_fullsized":"2024-09-10T18:28:30.969637+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:11.367574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:11.367574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:16.471529+0000","scrub_duration":0.0025623360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":174,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.965183+0000","last_change":"2024-09-10T18:28:09.749710+0000","last_active":"2024-09-10T18:28:30.965183+0000","last_peered":"2024-09-10T18:28:30.965183+0000","last_clean":"2024-09-10T18:28:30.965183+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:28:30.965183+0000","last_undegraded":"2024-09-10T18:28:30.965183+0000","last_fullsized":"2024-09-10T18:28:30.965183+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:28:09.749627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:27.848036+0000","scrub_duration":0.0063185680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"197'3998","reported_seq":4023,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:39.736949+0000","last_change":"2024-09-10T18:27:53.034084+0000","last_active":"2024-09-10T18:28:39.736949+0000","last_peered":"2024-09-10T18:28:39.736949+0000","last_clean":"2024-09-10T18:28:39.736949+0000","last_became_active":"2024-09-10T18:27:53.033757+0000","last_became_peered":"2024-09-10T18:27:53.033757+0000","last_unstale":"2024-09-10T18:28:39.736949+0000","last_undegraded":"2024-09-10T18:28:39.736949+0000","last_fullsized":"2024-09-10T18:28:39.736949+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:20.241697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262012928,"num_objects":3998,"num_object_clones":0,"num_object_copies":11994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":255872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":203,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.964186+0000","last_change":"2024-09-10T18:27:57.211522+0000","last_active":"2024-09-10T18:28:30.964186+0000","last_peered":"2024-09-10T18:28:30.964186+0000","last_clean":"2024-09-10T18:28:30.964186+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:28:30.964186+0000","last_undegraded":"2024-09-10T18:28:30.964186+0000","last_fullsized":"2024-09-10T18:28:30.964186+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:27:57.211451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:27:57.211451+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:07.130222+0000","scrub_duration":0.0053779049999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":176,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.961601+0000","last_change":"2024-09-10T18:28:06.191180+0000","last_active":"2024-09-10T18:28:30.961601+0000","last_peered":"2024-09-10T18:28:30.961601+0000","last_clean":"2024-09-10T18:28:30.961601+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:28:30.961601+0000","last_undegraded":"2024-09-10T18:28:30.961601+0000","last_fullsized":"2024-09-10T18:28:30.961601+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:06.191117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:06.191117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:17.909998+0000","scrub_duration":0.0015003600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":180,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.965057+0000","last_change":"2024-09-10T18:26:58.532067+0000","last_active":"2024-09-10T18:28:30.965057+0000","last_peered":"2024-09-10T18:28:30.965057+0000","last_clean":"2024-09-10T18:28:30.965057+0000","last_became_active":"2024-09-10T18:26:27.645885+0000","last_became_peered":"2024-09-10T18:26:27.645885+0000","last_unstale":"2024-09-10T18:28:30.965057+0000","last_undegraded":"2024-09-10T18:28:30.965057+0000","last_fullsized":"2024-09-10T18:28:30.965057+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:58.532019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:26:58.532019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001143645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":179,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962203+0000","last_change":"2024-09-10T18:26:58.414225+0000","last_active":"2024-09-10T18:28:30.962203+0000","last_peered":"2024-09-10T18:28:30.962203+0000","last_clean":"2024-09-10T18:28:30.962203+0000","last_became_active":"2024-09-10T18:26:27.632071+0000","last_became_peered":"2024-09-10T18:26:27.632071+0000","last_unstale":"2024-09-10T18:28:30.962203+0000","last_undegraded":"2024-09-10T18:28:30.962203+0000","last_fullsized":"2024-09-10T18:28:30.962203+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:58.414024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:26:58.414024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.020948053000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"197'4084","reported_seq":4109,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:39.913177+0000","last_change":"2024-09-10T18:27:53.035721+0000","last_active":"2024-09-10T18:28:39.913177+0000","last_peered":"2024-09-10T18:28:39.913177+0000","last_clean":"2024-09-10T18:28:39.913177+0000","last_became_active":"2024-09-10T18:27:53.035419+0000","last_became_peered":"2024-09-10T18:27:53.035419+0000","last_unstale":"2024-09-10T18:28:39.913177+0000","last_undegraded":"2024-09-10T18:28:39.913177+0000","last_fullsized":"2024-09-10T18:28:39.913177+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4084,"log_dups_size":0,"ondisk_log_size":4084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:20.993490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267649024,"num_objects":4084,"num_object_clones":0,"num_object_copies":12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4084,"num_write_kb":261376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":85,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.969127+0000","last_change":"2024-09-10T18:27:55.747977+0000","last_active":"2024-09-10T18:28:30.969127+0000","last_peered":"2024-09-10T18:28:30.969127+0000","last_clean":"2024-09-10T18:28:30.969127+0000","last_became_active":"2024-09-10T18:26:18.095088+0000","last_became_peered":"2024-09-10T18:26:18.095088+0000","last_unstale":"2024-09-10T18:28:30.969127+0000","last_undegraded":"2024-09-10T18:28:30.969127+0000","last_fullsized":"2024-09-10T18:28:30.969127+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:27:55.747909+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:27:55.747909+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:00.862668+0000","scrub_duration":0.0026038910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":202,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.963335+0000","last_change":"2024-09-10T18:26:16.736677+0000","last_active":"2024-09-10T18:28:30.963335+0000","last_peered":"2024-09-10T18:28:30.963335+0000","last_clean":"2024-09-10T18:28:30.963335+0000","last_became_active":"2024-09-10T18:21:33.611203+0000","last_became_peered":"2024-09-10T18:21:33.611203+0000","last_unstale":"2024-09-10T18:28:30.963335+0000","last_undegraded":"2024-09-10T18:28:30.963335+0000","last_fullsized":"2024-09-10T18:28:30.963335+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-10T18:26:16.736598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:16.736598+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039285580000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":130,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.965747+0000","last_change":"2024-09-10T18:27:17.554261+0000","last_active":"2024-09-10T18:28:30.965747+0000","last_peered":"2024-09-10T18:28:30.965747+0000","last_clean":"2024-09-10T18:28:30.965747+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:28:30.965747+0000","last_undegraded":"2024-09-10T18:28:30.965747+0000","last_fullsized":"2024-09-10T18:28:30.965747+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:27:17.554184+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089718639999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":172,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.967341+0000","last_change":"2024-09-10T18:27:12.675527+0000","last_active":"2024-09-10T18:28:30.967341+0000","last_peered":"2024-09-10T18:28:30.967341+0000","last_clean":"2024-09-10T18:28:30.967341+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:28:30.967341+0000","last_undegraded":"2024-09-10T18:28:30.967341+0000","last_fullsized":"2024-09-10T18:28:30.967341+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:12.675265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:12.675265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.096140301999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":190,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.976469+0000","last_change":"2024-09-10T18:28:02.975548+0000","last_active":"2024-09-10T18:28:30.976469+0000","last_peered":"2024-09-10T18:28:30.976469+0000","last_clean":"2024-09-10T18:28:30.976469+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:28:30.976469+0000","last_undegraded":"2024-09-10T18:28:30.976469+0000","last_fullsized":"2024-09-10T18:28:30.976469+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:02.975493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:28:02.975493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:22.369213+0000","scrub_duration":0.019176189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"197'4059","reported_seq":4084,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:40.311523+0000","last_change":"2024-09-10T18:27:53.037543+0000","last_active":"2024-09-10T18:28:40.311523+0000","last_peered":"2024-09-10T18:28:40.311523+0000","last_clean":"2024-09-10T18:28:40.311523+0000","last_became_active":"2024-09-10T18:27:53.037235+0000","last_became_peered":"2024-09-10T18:27:53.037235+0000","last_unstale":"2024-09-10T18:28:40.311523+0000","last_undegraded":"2024-09-10T18:28:40.311523+0000","last_fullsized":"2024-09-10T18:28:40.311523+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4059,"log_dups_size":0,"ondisk_log_size":4059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:17.490638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266010624,"num_objects":4059,"num_object_clones":0,"num_object_copies":12177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4059,"num_write_kb":259776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":224,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.968556+0000","last_change":"2024-09-10T18:27:58.257714+0000","last_active":"2024-09-10T18:28:30.968556+0000","last_peered":"2024-09-10T18:28:30.968556+0000","last_clean":"2024-09-10T18:28:30.968556+0000","last_became_active":"2024-09-10T18:26:18.092014+0000","last_became_peered":"2024-09-10T18:26:18.092014+0000","last_unstale":"2024-09-10T18:28:30.968556+0000","last_undegraded":"2024-09-10T18:28:30.968556+0000","last_fullsized":"2024-09-10T18:28:30.968556+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:27:58.257644+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:27:58.257644+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:14.849409+0000","scrub_duration":0.0015187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"92'15","reported_seq":201,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962511+0000","last_change":"2024-09-10T18:27:12.992305+0000","last_active":"2024-09-10T18:28:30.962511+0000","last_peered":"2024-09-10T18:28:30.962511+0000","last_clean":"2024-09-10T18:28:30.962511+0000","last_became_active":"2024-09-10T18:26:27.649406+0000","last_became_peered":"2024-09-10T18:26:27.649406+0000","last_unstale":"2024-09-10T18:28:30.962511+0000","last_undegraded":"2024-09-10T18:28:30.962511+0000","last_fullsized":"2024-09-10T18:28:30.962511+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'15","last_scrub_stamp":"2024-09-10T18:27:12.992221+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:27:12.992221+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:32.180165+0000","scrub_duration":0.0020362739999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":178,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962636+0000","last_change":"2024-09-10T18:28:14.145492+0000","last_active":"2024-09-10T18:28:30.962636+0000","last_peered":"2024-09-10T18:28:30.962636+0000","last_clean":"2024-09-10T18:28:30.962636+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:28:30.962636+0000","last_undegraded":"2024-09-10T18:28:30.962636+0000","last_fullsized":"2024-09-10T18:28:30.962636+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:14.145364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:14.145364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:31.467386+0000","scrub_duration":0.003651989,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"189'9","reported_seq":287,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:34.840800+0000","last_change":"2024-09-10T18:28:11.017989+0000","last_active":"2024-09-10T18:28:34.840800+0000","last_peered":"2024-09-10T18:28:34.840800+0000","last_clean":"2024-09-10T18:28:34.840800+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:28:34.840800+0000","last_undegraded":"2024-09-10T18:28:34.840800+0000","last_fullsized":"2024-09-10T18:28:34.840800+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'9","last_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:28:11.017903+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:36.538530+0000","scrub_duration":0.007246756,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":173,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962462+0000","last_change":"2024-09-10T18:28:16.088804+0000","last_active":"2024-09-10T18:28:30.962462+0000","last_peered":"2024-09-10T18:28:30.962462+0000","last_clean":"2024-09-10T18:28:30.962462+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:28:30.962462+0000","last_undegraded":"2024-09-10T18:28:30.962462+0000","last_fullsized":"2024-09-10T18:28:30.962462+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_clean_scrub_stamp":"2024-09-10T18:28:16.088718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:33.515907+0000","scrub_duration":0.0030710199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"197'4203","reported_seq":4228,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:39.921875+0000","last_change":"2024-09-10T18:27:53.037684+0000","last_active":"2024-09-10T18:28:39.921875+0000","last_peered":"2024-09-10T18:28:39.921875+0000","last_clean":"2024-09-10T18:28:39.921875+0000","last_became_active":"2024-09-10T18:27:53.037361+0000","last_became_peered":"2024-09-10T18:27:53.037361+0000","last_unstale":"2024-09-10T18:28:39.921875+0000","last_undegraded":"2024-09-10T18:28:39.921875+0000","last_fullsized":"2024-09-10T18:28:39.921875+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4203,"log_dups_size":0,"ondisk_log_size":4203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:12.821097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275447808,"num_objects":4203,"num_object_clones":0,"num_object_copies":12609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4203,"num_write_kb":268992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":181,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.974674+0000","last_change":"2024-09-10T18:26:09.669513+0000","last_active":"2024-09-10T18:28:30.974674+0000","last_peered":"2024-09-10T18:28:30.974674+0000","last_clean":"2024-09-10T18:28:30.974674+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:28:30.974674+0000","last_undegraded":"2024-09-10T18:28:30.974674+0000","last_fullsized":"2024-09-10T18:28:30.974674+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:26:09.669423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:09.669423+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010591546,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"58'1","reported_seq":193,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.966869+0000","last_change":"2024-09-10T18:26:53.613266+0000","last_active":"2024-09-10T18:28:30.966869+0000","last_peered":"2024-09-10T18:28:30.966869+0000","last_clean":"2024-09-10T18:28:30.966869+0000","last_became_active":"2024-09-10T18:26:27.644997+0000","last_became_peered":"2024-09-10T18:26:27.644997+0000","last_unstale":"2024-09-10T18:28:30.966869+0000","last_undegraded":"2024-09-10T18:28:30.966869+0000","last_fullsized":"2024-09-10T18:28:30.966869+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-10T18:26:53.613208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:26:53.613208+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042600270000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":188,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.977482+0000","last_change":"2024-09-10T18:27:52.027471+0000","last_active":"2024-09-10T18:28:30.977482+0000","last_peered":"2024-09-10T18:28:30.977482+0000","last_clean":"2024-09-10T18:28:30.977482+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:28:30.977482+0000","last_undegraded":"2024-09-10T18:28:30.977482+0000","last_fullsized":"2024-09-10T18:28:30.977482+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.027381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.027381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:04.516674+0000","scrub_duration":0.0050644849999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":171,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.983176+0000","last_change":"2024-09-10T18:27:51.963263+0000","last_active":"2024-09-10T18:28:30.983176+0000","last_peered":"2024-09-10T18:28:30.983176+0000","last_clean":"2024-09-10T18:28:30.983176+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:28:30.983176+0000","last_undegraded":"2024-09-10T18:28:30.983176+0000","last_fullsized":"2024-09-10T18:28:30.983176+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:51.963197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:51.963197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:53.251200+0000","scrub_duration":0.001346552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"197'4159","reported_seq":4183,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:40.352631+0000","last_change":"2024-09-10T18:27:53.060239+0000","last_active":"2024-09-10T18:28:40.352631+0000","last_peered":"2024-09-10T18:28:40.352631+0000","last_clean":"2024-09-10T18:28:40.352631+0000","last_became_active":"2024-09-10T18:27:53.059866+0000","last_became_peered":"2024-09-10T18:27:53.059866+0000","last_unstale":"2024-09-10T18:28:40.352631+0000","last_undegraded":"2024-09-10T18:28:40.352631+0000","last_fullsized":"2024-09-10T18:28:40.352631+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:57.273048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272564224,"num_objects":4159,"num_object_clones":0,"num_object_copies":12477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4159,"num_write_kb":266176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"86'28","reported_seq":193,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.965580+0000","last_change":"2024-09-10T18:27:12.513379+0000","last_active":"2024-09-10T18:28:30.965580+0000","last_peered":"2024-09-10T18:28:30.965580+0000","last_clean":"2024-09-10T18:28:30.965580+0000","last_became_active":"2024-09-10T18:26:27.637277+0000","last_became_peered":"2024-09-10T18:26:27.637277+0000","last_unstale":"2024-09-10T18:28:30.965580+0000","last_undegraded":"2024-09-10T18:28:30.965580+0000","last_fullsized":"2024-09-10T18:28:30.965580+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'28","last_scrub_stamp":"2024-09-10T18:27:12.513323+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:27:12.513323+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052920959999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":193,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.972910+0000","last_change":"2024-09-10T18:28:13.368579+0000","last_active":"2024-09-10T18:28:30.972910+0000","last_peered":"2024-09-10T18:28:30.972910+0000","last_clean":"2024-09-10T18:28:30.972910+0000","last_became_active":"2024-09-10T18:26:27.638408+0000","last_became_peered":"2024-09-10T18:26:27.638408+0000","last_unstale":"2024-09-10T18:28:30.972910+0000","last_undegraded":"2024-09-10T18:28:30.972910+0000","last_fullsized":"2024-09-10T18:28:30.972910+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:13.368506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:13.368506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:26.991208+0000","scrub_duration":0.0025418530000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":192,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.976359+0000","last_change":"2024-09-10T18:28:10.038296+0000","last_active":"2024-09-10T18:28:30.976359+0000","last_peered":"2024-09-10T18:28:30.976359+0000","last_clean":"2024-09-10T18:28:30.976359+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:28:30.976359+0000","last_undegraded":"2024-09-10T18:28:30.976359+0000","last_fullsized":"2024-09-10T18:28:30.976359+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:10.038205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:10.038205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:26.623064+0000","scrub_duration":0.0032197340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":183,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.968667+0000","last_change":"2024-09-10T18:28:08.336391+0000","last_active":"2024-09-10T18:28:30.968667+0000","last_peered":"2024-09-10T18:28:30.968667+0000","last_clean":"2024-09-10T18:28:30.968667+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:28:30.968667+0000","last_undegraded":"2024-09-10T18:28:30.968667+0000","last_fullsized":"2024-09-10T18:28:30.968667+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:08.336308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:28:08.336308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:08.529256+0000","scrub_duration":0.0027473250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"197'4242","reported_seq":4266,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:40.648842+0000","last_change":"2024-09-10T18:27:53.061218+0000","last_active":"2024-09-10T18:28:40.648842+0000","last_peered":"2024-09-10T18:28:40.648842+0000","last_clean":"2024-09-10T18:28:40.648842+0000","last_became_active":"2024-09-10T18:27:53.060902+0000","last_became_peered":"2024-09-10T18:27:53.060902+0000","last_unstale":"2024-09-10T18:28:40.648842+0000","last_undegraded":"2024-09-10T18:28:40.648842+0000","last_fullsized":"2024-09-10T18:28:40.648842+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:11.224263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278003712,"num_objects":4242,"num_object_clones":0,"num_object_copies":12726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4242,"num_write_kb":271488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":200,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.961330+0000","last_change":"2024-09-10T18:26:06.804053+0000","last_active":"2024-09-10T18:28:30.961330+0000","last_peered":"2024-09-10T18:28:30.961330+0000","last_clean":"2024-09-10T18:28:30.961330+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:28:30.961330+0000","last_undegraded":"2024-09-10T18:28:30.961330+0000","last_fullsized":"2024-09-10T18:28:30.961330+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:26:06.804001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:06.804001+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013860801000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":186,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.970639+0000","last_change":"2024-09-10T18:28:09.316857+0000","last_active":"2024-09-10T18:28:30.970639+0000","last_peered":"2024-09-10T18:28:30.970639+0000","last_clean":"2024-09-10T18:28:30.970639+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:28:30.970639+0000","last_undegraded":"2024-09-10T18:28:30.970639+0000","last_fullsized":"2024-09-10T18:28:30.970639+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:09.316773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:09.316773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:11.869771+0000","scrub_duration":0.002163477,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":192,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.975804+0000","last_change":"2024-09-10T18:28:12.038263+0000","last_active":"2024-09-10T18:28:30.975804+0000","last_peered":"2024-09-10T18:28:30.975804+0000","last_clean":"2024-09-10T18:28:30.975804+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:28:30.975804+0000","last_undegraded":"2024-09-10T18:28:30.975804+0000","last_fullsized":"2024-09-10T18:28:30.975804+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:12.038177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:12.038177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:28.361971+0000","scrub_duration":0.0042416900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":162,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962369+0000","last_change":"2024-09-10T18:26:12.014835+0000","last_active":"2024-09-10T18:28:30.962369+0000","last_peered":"2024-09-10T18:28:30.962369+0000","last_clean":"2024-09-10T18:28:30.962369+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:28:30.962369+0000","last_undegraded":"2024-09-10T18:28:30.962369+0000","last_fullsized":"2024-09-10T18:28:30.962369+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:12.014757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:12.014757+0000","last_clean_scrub_stamp":"2024-09-10T18:26:12.014757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082019740000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"197'4351","reported_seq":4376,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:39.744671+0000","last_change":"2024-09-10T18:27:53.032783+0000","last_active":"2024-09-10T18:28:39.744671+0000","last_peered":"2024-09-10T18:28:39.744671+0000","last_clean":"2024-09-10T18:28:39.744671+0000","last_became_active":"2024-09-10T18:27:53.032159+0000","last_became_peered":"2024-09-10T18:27:53.032159+0000","last_unstale":"2024-09-10T18:28:39.744671+0000","last_undegraded":"2024-09-10T18:28:39.744671+0000","last_fullsized":"2024-09-10T18:28:39.744671+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4351,"log_dups_size":0,"ondisk_log_size":4351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:10.012526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285147136,"num_objects":4351,"num_object_clones":0,"num_object_copies":13053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4351,"num_write_kb":278464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":207,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.969255+0000","last_change":"2024-09-10T18:26:51.556115+0000","last_active":"2024-09-10T18:28:30.969255+0000","last_peered":"2024-09-10T18:28:30.969255+0000","last_clean":"2024-09-10T18:28:30.969255+0000","last_became_active":"2024-09-10T18:26:27.645932+0000","last_became_peered":"2024-09-10T18:26:27.645932+0000","last_unstale":"2024-09-10T18:28:30.969255+0000","last_undegraded":"2024-09-10T18:28:30.969255+0000","last_fullsized":"2024-09-10T18:28:30.969255+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-10T18:26:51.556026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:51.556026+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002446429,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":193,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.969455+0000","last_change":"2024-09-10T18:28:06.329960+0000","last_active":"2024-09-10T18:28:30.969455+0000","last_peered":"2024-09-10T18:28:30.969455+0000","last_clean":"2024-09-10T18:28:30.969455+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:28:30.969455+0000","last_undegraded":"2024-09-10T18:28:30.969455+0000","last_fullsized":"2024-09-10T18:28:30.969455+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:28:06.329891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:06.329891+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:36.083657+0000","scrub_duration":0.001979864,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"146'10","reported_seq":274,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:31.978131+0000","last_change":"2024-09-10T18:27:11.013671+0000","last_active":"2024-09-10T18:28:31.978131+0000","last_peered":"2024-09-10T18:28:31.978131+0000","last_clean":"2024-09-10T18:28:31.978131+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:28:31.978131+0000","last_undegraded":"2024-09-10T18:28:31.978131+0000","last_fullsized":"2024-09-10T18:28:31.978131+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'10","last_scrub_stamp":"2024-09-10T18:27:11.013616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:11.013616+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:36.058913+0000","scrub_duration":0.0022297150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":168,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.971595+0000","last_change":"2024-09-10T18:26:50.594047+0000","last_active":"2024-09-10T18:28:30.971595+0000","last_peered":"2024-09-10T18:28:30.971595+0000","last_clean":"2024-09-10T18:28:30.971595+0000","last_became_active":"2024-09-10T18:26:27.647511+0000","last_became_peered":"2024-09-10T18:26:27.647511+0000","last_unstale":"2024-09-10T18:28:30.971595+0000","last_undegraded":"2024-09-10T18:28:30.971595+0000","last_fullsized":"2024-09-10T18:28:30.971595+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:50.593965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:26:50.593965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096006660000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":121,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.965074+0000","last_change":"2024-09-10T18:27:52.527922+0000","last_active":"2024-09-10T18:28:30.965074+0000","last_peered":"2024-09-10T18:28:30.965074+0000","last_clean":"2024-09-10T18:28:30.965074+0000","last_became_active":"2024-09-10T18:26:27.640922+0000","last_became_peered":"2024-09-10T18:26:27.640922+0000","last_unstale":"2024-09-10T18:28:30.965074+0000","last_undegraded":"2024-09-10T18:28:30.965074+0000","last_fullsized":"2024-09-10T18:28:30.965074+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.527847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.527847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:56.233308+0000","scrub_duration":0.0016791950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":185,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.963808+0000","last_change":"2024-09-10T18:28:01.176967+0000","last_active":"2024-09-10T18:28:30.963808+0000","last_peered":"2024-09-10T18:28:30.963808+0000","last_clean":"2024-09-10T18:28:30.963808+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:28:30.963808+0000","last_undegraded":"2024-09-10T18:28:30.963808+0000","last_fullsized":"2024-09-10T18:28:30.963808+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-10T18:28:01.176843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:28:01.176843+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:27.123011+0000","scrub_duration":0.0029137500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":172,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.967072+0000","last_change":"2024-09-10T18:27:52.665248+0000","last_active":"2024-09-10T18:28:30.967072+0000","last_peered":"2024-09-10T18:28:30.967072+0000","last_clean":"2024-09-10T18:28:30.967072+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:28:30.967072+0000","last_undegraded":"2024-09-10T18:28:30.967072+0000","last_fullsized":"2024-09-10T18:28:30.967072+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.664666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.664666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:55.953894+0000","scrub_duration":0.001910155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":177,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.972693+0000","last_change":"2024-09-10T18:26:54.055557+0000","last_active":"2024-09-10T18:28:30.972693+0000","last_peered":"2024-09-10T18:28:30.972693+0000","last_clean":"2024-09-10T18:28:30.972693+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:28:30.972693+0000","last_undegraded":"2024-09-10T18:28:30.972693+0000","last_fullsized":"2024-09-10T18:28:30.972693+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:54.055483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:26:54.055483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002405113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"189'13","reported_seq":198,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.971523+0000","last_change":"2024-09-10T18:28:08.976270+0000","last_active":"2024-09-10T18:28:30.971523+0000","last_peered":"2024-09-10T18:28:30.971523+0000","last_clean":"2024-09-10T18:28:30.971523+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:28:30.971523+0000","last_undegraded":"2024-09-10T18:28:30.971523+0000","last_fullsized":"2024-09-10T18:28:30.971523+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'13","last_scrub_stamp":"2024-09-10T18:28:08.976162+0000","last_deep_scrub":"189'13","last_deep_scrub_stamp":"2024-09-10T18:28:08.976162+0000","last_clean_scrub_stamp":"2024-09-10T18:28:08.976162+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:38.695222+0000","scrub_duration":0.010982957999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":183,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.968741+0000","last_change":"2024-09-10T18:26:51.013771+0000","last_active":"2024-09-10T18:28:30.968741+0000","last_peered":"2024-09-10T18:28:30.968741+0000","last_clean":"2024-09-10T18:28:30.968741+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:28:30.968741+0000","last_undegraded":"2024-09-10T18:28:30.968741+0000","last_fullsized":"2024-09-10T18:28:30.968741+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:51.013696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:26:51.013696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001333568,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":174,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.963634+0000","last_change":"2024-09-10T18:27:56.215744+0000","last_active":"2024-09-10T18:28:30.963634+0000","last_peered":"2024-09-10T18:28:30.963634+0000","last_clean":"2024-09-10T18:28:30.963634+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:28:30.963634+0000","last_undegraded":"2024-09-10T18:28:30.963634+0000","last_fullsized":"2024-09-10T18:28:30.963634+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:56.215691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:56.215691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:12.955476+0000","scrub_duration":0.0034563770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":180,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.976603+0000","last_change":"2024-09-10T18:27:59.000777+0000","last_active":"2024-09-10T18:28:30.976603+0000","last_peered":"2024-09-10T18:28:30.976603+0000","last_clean":"2024-09-10T18:28:30.976603+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:28:30.976603+0000","last_undegraded":"2024-09-10T18:28:30.976603+0000","last_fullsized":"2024-09-10T18:28:30.976603+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:59.000723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:59.000723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:23.665168+0000","scrub_duration":0.002141666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"165'13","reported_seq":195,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.965253+0000","last_change":"2024-09-10T18:27:17.468672+0000","last_active":"2024-09-10T18:28:30.965253+0000","last_peered":"2024-09-10T18:28:30.965253+0000","last_clean":"2024-09-10T18:28:30.965253+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:28:30.965253+0000","last_undegraded":"2024-09-10T18:28:30.965253+0000","last_fullsized":"2024-09-10T18:28:30.965253+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'13","last_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:27:17.468619+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010200137999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":177,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.970957+0000","last_change":"2024-09-10T18:26:52.832466+0000","last_active":"2024-09-10T18:28:30.970957+0000","last_peered":"2024-09-10T18:28:30.970957+0000","last_clean":"2024-09-10T18:28:30.970957+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:28:30.970957+0000","last_undegraded":"2024-09-10T18:28:30.970957+0000","last_fullsized":"2024-09-10T18:28:30.970957+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:52.832368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:26:52.832368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019356278000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":268,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:34.839301+0000","last_change":"2024-09-10T18:27:53.176983+0000","last_active":"2024-09-10T18:28:34.839301+0000","last_peered":"2024-09-10T18:28:34.839301+0000","last_clean":"2024-09-10T18:28:34.839301+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:28:34.839301+0000","last_undegraded":"2024-09-10T18:28:34.839301+0000","last_fullsized":"2024-09-10T18:28:34.839301+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:27:53.176916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:53.176916+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:10.909920+0000","scrub_duration":0.0024011380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":170,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.963145+0000","last_change":"2024-09-10T18:26:21.808801+0000","last_active":"2024-09-10T18:28:30.963145+0000","last_peered":"2024-09-10T18:28:30.963145+0000","last_clean":"2024-09-10T18:28:30.963145+0000","last_became_active":"2024-09-10T18:21:37.639706+0000","last_became_peered":"2024-09-10T18:21:37.639706+0000","last_unstale":"2024-09-10T18:28:30.963145+0000","last_undegraded":"2024-09-10T18:28:30.963145+0000","last_fullsized":"2024-09-10T18:28:30.963145+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:21.808720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:26:21.808720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018030489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":191,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962638+0000","last_change":"2024-09-10T18:26:19.097137+0000","last_active":"2024-09-10T18:28:30.962638+0000","last_peered":"2024-09-10T18:28:30.962638+0000","last_clean":"2024-09-10T18:28:30.962638+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:28:30.962638+0000","last_undegraded":"2024-09-10T18:28:30.962638+0000","last_fullsized":"2024-09-10T18:28:30.962638+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-10T18:26:19.097047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:19.097047+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090288920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":176,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.968760+0000","last_change":"2024-09-10T18:27:12.052535+0000","last_active":"2024-09-10T18:28:30.968760+0000","last_peered":"2024-09-10T18:28:30.968760+0000","last_clean":"2024-09-10T18:28:30.968760+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:28:30.968760+0000","last_undegraded":"2024-09-10T18:28:30.968760+0000","last_fullsized":"2024-09-10T18:28:30.968760+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:12.052482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:27:12.052482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:40.437008+0000","scrub_duration":0.001572876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":276,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-09-10T18:28:31.978576+0000","last_change":"2024-09-10T18:28:15.692262+0000","last_active":"2024-09-10T18:28:31.978576+0000","last_peered":"2024-09-10T18:28:31.978576+0000","last_clean":"2024-09-10T18:28:31.978576+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:28:31.978576+0000","last_undegraded":"2024-09-10T18:28:31.978576+0000","last_fullsized":"2024-09-10T18:28:31.978576+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:28:15.692182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:15.692182+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:34.932763+0000","scrub_duration":0.0057467769999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":171,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.971653+0000","last_change":"2024-09-10T18:27:54.043813+0000","last_active":"2024-09-10T18:28:30.971653+0000","last_peered":"2024-09-10T18:28:30.971653+0000","last_clean":"2024-09-10T18:28:30.971653+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:28:30.971653+0000","last_undegraded":"2024-09-10T18:28:30.971653+0000","last_fullsized":"2024-09-10T18:28:30.971653+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:54.043726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:54.043726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:00.550467+0000","scrub_duration":0.001550514,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":200,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962137+0000","last_change":"2024-09-10T18:27:08.970903+0000","last_active":"2024-09-10T18:28:30.962137+0000","last_peered":"2024-09-10T18:28:30.962137+0000","last_clean":"2024-09-10T18:28:30.962137+0000","last_became_active":"2024-09-10T18:26:27.649202+0000","last_became_peered":"2024-09-10T18:26:27.649202+0000","last_unstale":"2024-09-10T18:28:30.962137+0000","last_undegraded":"2024-09-10T18:28:30.962137+0000","last_fullsized":"2024-09-10T18:28:30.962137+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:27:08.970839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:27:08.970839+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017431300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":186,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962854+0000","last_change":"2024-09-10T18:27:54.278241+0000","last_active":"2024-09-10T18:28:30.962854+0000","last_peered":"2024-09-10T18:28:30.962854+0000","last_clean":"2024-09-10T18:28:30.962854+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:28:30.962854+0000","last_undegraded":"2024-09-10T18:28:30.962854+0000","last_fullsized":"2024-09-10T18:28:30.962854+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:54.278176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:27:54.278176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:03.767304+0000","scrub_duration":0.001425756,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":185,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.973489+0000","last_change":"2024-09-10T18:27:58.515917+0000","last_active":"2024-09-10T18:28:30.973489+0000","last_peered":"2024-09-10T18:28:30.973489+0000","last_clean":"2024-09-10T18:28:30.973489+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:28:30.973489+0000","last_undegraded":"2024-09-10T18:28:30.973489+0000","last_fullsized":"2024-09-10T18:28:30.973489+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:58.515852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:58.515852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:59.695982+0000","scrub_duration":0.0014537980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":162,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.961498+0000","last_change":"2024-09-10T18:26:18.076688+0000","last_active":"2024-09-10T18:28:30.961498+0000","last_peered":"2024-09-10T18:28:30.961498+0000","last_clean":"2024-09-10T18:28:30.961498+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:28:30.961498+0000","last_undegraded":"2024-09-10T18:28:30.961498+0000","last_fullsized":"2024-09-10T18:28:30.961498+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:18.076617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:26:18.076617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055521420000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":197,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.973565+0000","last_change":"2024-09-10T18:26:52.667217+0000","last_active":"2024-09-10T18:28:30.973565+0000","last_peered":"2024-09-10T18:28:30.973565+0000","last_clean":"2024-09-10T18:28:30.973565+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:28:30.973565+0000","last_undegraded":"2024-09-10T18:28:30.973565+0000","last_fullsized":"2024-09-10T18:28:30.973565+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:26:52.667140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:52.667140+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068280379999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":189,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.972816+0000","last_change":"2024-09-10T18:28:07.373735+0000","last_active":"2024-09-10T18:28:30.972816+0000","last_peered":"2024-09-10T18:28:30.972816+0000","last_clean":"2024-09-10T18:28:30.972816+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:28:30.972816+0000","last_undegraded":"2024-09-10T18:28:30.972816+0000","last_fullsized":"2024-09-10T18:28:30.972816+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:07.373653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:28:07.373653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:37.151211+0000","scrub_duration":0.0021799559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":164,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962443+0000","last_change":"2024-09-10T18:26:10.052392+0000","last_active":"2024-09-10T18:28:30.962443+0000","last_peered":"2024-09-10T18:28:30.962443+0000","last_clean":"2024-09-10T18:28:30.962443+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:28:30.962443+0000","last_undegraded":"2024-09-10T18:28:30.962443+0000","last_fullsized":"2024-09-10T18:28:30.962443+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:26:10.052296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019588034000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":171,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.977069+0000","last_change":"2024-09-10T18:27:17.780318+0000","last_active":"2024-09-10T18:28:30.977069+0000","last_peered":"2024-09-10T18:28:30.977069+0000","last_clean":"2024-09-10T18:28:30.977069+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:28:30.977069+0000","last_undegraded":"2024-09-10T18:28:30.977069+0000","last_fullsized":"2024-09-10T18:28:30.977069+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:17.780238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:17.780238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:47.213160+0000","scrub_duration":0.0094000860000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":202,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.961717+0000","last_change":"2024-09-10T18:27:51.212061+0000","last_active":"2024-09-10T18:28:30.961717+0000","last_peered":"2024-09-10T18:28:30.961717+0000","last_clean":"2024-09-10T18:28:30.961717+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:28:30.961717+0000","last_undegraded":"2024-09-10T18:28:30.961717+0000","last_fullsized":"2024-09-10T18:28:30.961717+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:27:51.211994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:27:51.211994+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:58.142346+0000","scrub_duration":0.0024425879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":187,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.967131+0000","last_change":"2024-09-10T18:28:01.748224+0000","last_active":"2024-09-10T18:28:30.967131+0000","last_peered":"2024-09-10T18:28:30.967131+0000","last_clean":"2024-09-10T18:28:30.967131+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:28:30.967131+0000","last_undegraded":"2024-09-10T18:28:30.967131+0000","last_fullsized":"2024-09-10T18:28:30.967131+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:01.748169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:28:01.748169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:31.247689+0000","scrub_duration":0.0025848580000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":182,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.968361+0000","last_change":"2024-09-10T18:27:57.209856+0000","last_active":"2024-09-10T18:28:30.968361+0000","last_peered":"2024-09-10T18:28:30.968361+0000","last_clean":"2024-09-10T18:28:30.968361+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:28:30.968361+0000","last_undegraded":"2024-09-10T18:28:30.968361+0000","last_fullsized":"2024-09-10T18:28:30.968361+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:57.209756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:57.209756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:09.616768+0000","scrub_duration":0.0033515160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":179,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962574+0000","last_change":"2024-09-10T18:26:52.309818+0000","last_active":"2024-09-10T18:28:30.962574+0000","last_peered":"2024-09-10T18:28:30.962574+0000","last_clean":"2024-09-10T18:28:30.962574+0000","last_became_active":"2024-09-10T18:26:27.636996+0000","last_became_peered":"2024-09-10T18:26:27.636996+0000","last_unstale":"2024-09-10T18:28:30.962574+0000","last_undegraded":"2024-09-10T18:28:30.962574+0000","last_fullsized":"2024-09-10T18:28:30.962574+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:52.309588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:26:52.309588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053342499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":189,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962123+0000","last_change":"2024-09-10T18:27:52.206324+0000","last_active":"2024-09-10T18:28:30.962123+0000","last_peered":"2024-09-10T18:28:30.962123+0000","last_clean":"2024-09-10T18:28:30.962123+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:28:30.962123+0000","last_undegraded":"2024-09-10T18:28:30.962123+0000","last_fullsized":"2024-09-10T18:28:30.962123+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-09-10T18:27:52.205769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.205769+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:21.294714+0000","scrub_duration":0.0020401389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":180,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.971189+0000","last_change":"2024-09-10T18:28:06.991059+0000","last_active":"2024-09-10T18:28:30.971189+0000","last_peered":"2024-09-10T18:28:30.971189+0000","last_clean":"2024-09-10T18:28:30.971189+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:28:30.971189+0000","last_undegraded":"2024-09-10T18:28:30.971189+0000","last_fullsized":"2024-09-10T18:28:30.971189+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:06.991004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:06.991004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:29.268952+0000","scrub_duration":0.0015824980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":125,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.966109+0000","last_change":"2024-09-10T18:28:07.362740+0000","last_active":"2024-09-10T18:28:30.966109+0000","last_peered":"2024-09-10T18:28:30.966109+0000","last_clean":"2024-09-10T18:28:30.966109+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:28:30.966109+0000","last_undegraded":"2024-09-10T18:28:30.966109+0000","last_fullsized":"2024-09-10T18:28:30.966109+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:07.362688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:07.362688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:22.798236+0000","scrub_duration":0.001454197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":170,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.968794+0000","last_change":"2024-09-10T18:27:19.002667+0000","last_active":"2024-09-10T18:28:30.968794+0000","last_peered":"2024-09-10T18:28:30.968794+0000","last_clean":"2024-09-10T18:28:30.968794+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:28:30.968794+0000","last_undegraded":"2024-09-10T18:28:30.968794+0000","last_fullsized":"2024-09-10T18:28:30.968794+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:19.002583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:19.002583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013828224,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":180,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.967660+0000","last_change":"2024-09-10T18:27:13.620794+0000","last_active":"2024-09-10T18:28:30.967660+0000","last_peered":"2024-09-10T18:28:30.967660+0000","last_clean":"2024-09-10T18:28:30.967660+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:28:30.967660+0000","last_undegraded":"2024-09-10T18:28:30.967660+0000","last_fullsized":"2024-09-10T18:28:30.967660+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:13.620403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:13.620403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:33.504075+0000","scrub_duration":0.011110345000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":168,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.962355+0000","last_change":"2024-09-10T18:25:18.030959+0000","last_active":"2024-09-10T18:28:30.962355+0000","last_peered":"2024-09-10T18:28:30.962355+0000","last_clean":"2024-09-10T18:28:30.962355+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:28:30.962355+0000","last_undegraded":"2024-09-10T18:28:30.962355+0000","last_fullsized":"2024-09-10T18:28:30.962355+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:25:18.030878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.026365884999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":164,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.966297+0000","last_change":"2024-09-10T18:27:15.545071+0000","last_active":"2024-09-10T18:28:30.966297+0000","last_peered":"2024-09-10T18:28:30.966297+0000","last_clean":"2024-09-10T18:28:30.966297+0000","last_became_active":"2024-09-10T18:26:27.648112+0000","last_became_peered":"2024-09-10T18:26:27.648112+0000","last_unstale":"2024-09-10T18:28:30.966297+0000","last_undegraded":"2024-09-10T18:28:30.966297+0000","last_fullsized":"2024-09-10T18:28:30.966297+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:27:15.544986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:27:15.544986+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018039950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":173,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-10T18:28:30.971318+0000","last_change":"2024-09-10T18:27:11.603305+0000","last_active":"2024-09-10T18:28:30.971318+0000","last_peered":"2024-09-10T18:28:30.971318+0000","last_clean":"2024-09-10T18:28:30.971318+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:28:30.971318+0000","last_undegraded":"2024-09-10T18:28:30.971318+0000","last_fullsized":"2024-09-10T18:28:30.971318+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:11.603148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:11.603148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011808801000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4361486336,"num_objects":66551,"num_object_clones":0,"num_object_copies":199653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66551,"num_write_kb":4259264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13059096576,"data_stored":13059096576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66551,"ondisk_log_size":66551,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":421,"num_read_kb":416,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":891,"ondisk_log_size":891,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234073,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1605424,"kb_used_data":1299752,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92139728,"statfs":{"total":95995035648,"available":94351081472,"internally_reserved":0,"allocated":1330946048,"data_stored":1330314179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":2.2410000000000001,"15min":1.095},"min":{"1min":0.26400000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.3239999999999998,"5min":85.012,"15min":85.012},"last":1.169},{"interface":"front","average":{"1min":0.61099999999999999,"5min":2.202,"15min":1.077},"min":{"1min":0.312,"5min":0.222,"15min":0.222},"max":{"1min":1.421,"5min":84.665999999999997,"15min":84.665999999999997},"last":1.107}]},{"osd":1,"last update":"Tue Sep 10 18:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.3149999999999999,"15min":0.78200000000000003},"min":{"1min":0.40799999999999997,"5min":0.25,"15min":0.25},"max":{"1min":3.2189999999999999,"5min":12.583,"15min":12.583},"last":1.8580000000000001},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.2969999999999999,"15min":0.80700000000000005},"min":{"1min":0.39800000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.6720000000000002,"5min":12.601000000000001,"15min":12.601000000000001},"last":2.069}]},{"osd":2,"last update":"Tue Sep 10 18:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.90800000000000003,"15min":0.66400000000000003},"min":{"1min":0.27600000000000002,"5min":0.249,"15min":0.249},"max":{"1min":2.6850000000000001,"5min":5.3150000000000004,"15min":5.3150000000000004},"last":1.911},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.97499999999999998,"15min":0.69999999999999996},"min":{"1min":0.35099999999999998,"5min":0.254,"15min":0.254},"max":{"1min":3.2959999999999998,"5min":9.9450000000000003,"15min":9.9450000000000003},"last":0.46999999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":2.4009999999999998,"15min":1.1639999999999999},"min":{"1min":0.36599999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.865,"5min":86.305999999999997,"15min":86.305999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":2.4359999999999999,"15min":1.1799999999999999},"min":{"1min":0.33300000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.504,"5min":88.364000000000004,"15min":88.364000000000004},"last":0.71499999999999997}]},{"osd":4,"last update":"Tue Sep 10 18:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.40100000000000002,"15min":0.42499999999999999},"min":{"1min":0.29099999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.853,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.39500000000000002,"15min":0.42799999999999999},"min":{"1min":0.309,"5min":0.223,"15min":0.223},"max":{"1min":1.4850000000000001,"5min":1.925,"15min":1.925},"last":0.33600000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.46000000000000002,"15min":0.434},"min":{"1min":0.375,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.2480000000000002,"5min":2.2480000000000002,"15min":2.2480000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.436,"15min":0.46800000000000003},"min":{"1min":0.34100000000000003,"5min":0.219,"15min":0.219},"max":{"1min":1.8819999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":0.46000000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:28:26 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.48299999999999998,"15min":0.438},"min":{"1min":0.32300000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.46100000000000002,"15min":0.438},"min":{"1min":0.35499999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":0.95399999999999996}]}]},{"osd":6,"up_from":144,"seq":618475290652,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1654692,"kb_used_data":1347932,"kb_used_omap":7,"kb_used_meta":306744,"kb_avail":92090460,"statfs":{"total":95995035648,"available":94300631040,"internally_reserved":0,"allocated":1380282368,"data_stored":1379664441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":314105943},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":1.925,"5min":2.0739999999999998,"15min":2.0990000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":30.629999999999999,"5min":30.629999999999999,"15min":30.629999999999999},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.9359999999999999,"5min":1.6220000000000001,"15min":1.5700000000000001},"min":{"1min":0.23999999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":30.597000000000001,"5min":30.597000000000001,"15min":30.597000000000001},"last":0.253}]},{"osd":1,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":1.9850000000000001,"5min":0.97299999999999998,"15min":0.80500000000000005},"min":{"1min":0.308,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":30.984000000000002,"5min":30.984000000000002,"15min":30.984000000000002},"last":0.72699999999999998},{"interface":"front","average":{"1min":2.02,"5min":1.208,"15min":1.073},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":30.571000000000002,"5min":30.571000000000002,"15min":30.571000000000002},"last":0.51400000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":1.0229999999999999,"15min":1.0800000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.724,"5min":4.8259999999999996,"15min":4.8259999999999996},"last":1.1240000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.752,"15min":0.76600000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.8159999999999998,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.46400000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":2.4249999999999998,"15min":2.2629999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":61.703000000000003,"5min":61.703000000000003,"15min":61.703000000000003},"last":0.30399999999999999},{"interface":"front","average":{"1min":3.4900000000000002,"5min":2.4249999999999998,"15min":2.2469999999999999},"min":{"1min":0.39400000000000002,"5min":0.318,"15min":0.318},"max":{"1min":61.814,"5min":61.814,"15min":61.814},"last":0.378}]},{"osd":4,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.51300000000000001,"15min":0.495},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.66600000000000004,"15min":0.65100000000000002},"min":{"1min":0.27100000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":4.4359999999999999,"5min":4.4359999999999999,"15min":4.4359999999999999},"last":0.53600000000000003}]},{"osd":5,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.47599999999999998,"15min":0.46800000000000003},"min":{"1min":0.26900000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.7509999999999999,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.45800000000000002,"15min":0.42199999999999999},"min":{"1min":0.32200000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":1.1799999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.504,"15min":0.48799999999999999},"min":{"1min":0.28999999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.202,"5min":2.202,"15min":2.202},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.505,"15min":0.48499999999999999},"min":{"1min":0.29299999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.0950000000000002,"5min":3.0950000000000002,"15min":3.0950000000000002},"last":0.69799999999999995}]}]},{"osd":5,"up_from":39,"seq":167503724644,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2391288,"kb_used_data":2089776,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91353864,"statfs":{"total":95995035648,"available":93546356736,"internally_reserved":0,"allocated":2139930624,"data_stored":2139291613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:27:54 2024","interfaces":[{"interface":"back","average":{"1min":3.5369999999999999,"5min":2.359,"15min":1.1519999999999999},"min":{"1min":0.247,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":31.172000000000001,"5min":52.033999999999999,"15min":52.033999999999999},"last":2.0939999999999999},{"interface":"front","average":{"1min":3.835,"5min":2.0110000000000001,"15min":1.0169999999999999},"min":{"1min":0.245,"5min":0.216,"15min":0.216},"max":{"1min":32.246000000000002,"5min":52.167999999999999,"15min":52.167999999999999},"last":2.3799999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.73299999999999998,"15min":0.61899999999999999},"min":{"1min":0.32900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.097,"5min":4.2400000000000002,"15min":4.2400000000000002},"last":1.452},{"interface":"front","average":{"1min":0.64100000000000001,"5min":1.0589999999999999,"15min":0.72299999999999998},"min":{"1min":0.30099999999999999,"5min":0.247,"15min":0.247},"max":{"1min":2.169,"5min":17.661000000000001,"15min":17.661000000000001},"last":2.4060000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:27:54 2024","interfaces":[{"interface":"back","average":{"1min":2.3500000000000001,"5min":1.2490000000000001,"15min":0.78400000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":36.853000000000002,"5min":36.853000000000002,"15min":36.853000000000002},"last":2.7480000000000002},{"interface":"front","average":{"1min":2.1899999999999999,"5min":1.169,"15min":0.753},"min":{"1min":0.27400000000000002,"5min":0.26100000000000001,"15min":0.255},"max":{"1min":31.132000000000001,"5min":31.132000000000001,"15min":31.132000000000001},"last":1.96}]},{"osd":3,"last update":"Tue Sep 10 18:27:54 2024","interfaces":[{"interface":"back","average":{"1min":3.6539999999999999,"5min":1.756,"15min":0.94799999999999995},"min":{"1min":0.35499999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":31.207000000000001,"5min":31.207000000000001,"15min":31.207000000000001},"last":2.1949999999999998},{"interface":"front","average":{"1min":3.415,"5min":1.5069999999999999,"15min":0.85899999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":31.199000000000002,"5min":31.199000000000002,"15min":31.199000000000002},"last":1.4319999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.38400000000000001,"15min":0.36899999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.2749999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":2.3450000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.41499999999999998,"15min":0.39600000000000002},"min":{"1min":0.26300000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":2.161}]},{"osd":6,"last update":"Tue Sep 10 18:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.57799999999999996,"15min":0.56599999999999995},"min":{"1min":0.34499999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.613,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":2.1259999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.56200000000000006,"15min":0.54600000000000004},"min":{"1min":0.32600000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.0880000000000001,"5min":2.153,"15min":2.153},"last":1.405}]},{"osd":7,"last update":"Tue Sep 10 18:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.46000000000000002,"15min":0.45800000000000002},"min":{"1min":0.38400000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.2350000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":1.472},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.45100000000000001,"15min":0.44400000000000001},"min":{"1min":0.33900000000000002,"5min":0.27200000000000002,"15min":0.24299999999999999},"max":{"1min":1.258,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":2.4289999999999998}]}]},{"osd":4,"up_from":33,"seq":141733920872,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1616816,"kb_used_data":1311144,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92128336,"statfs":{"total":95995035648,"available":94339416064,"internally_reserved":0,"allocated":1342611456,"data_stored":1341979398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:28:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.3240000000000001,"15min":0.81699999999999995},"min":{"1min":0.29799999999999999,"5min":0.188,"15min":0.188},"max":{"1min":16.236999999999998,"5min":31.585000000000001,"15min":31.585000000000001},"last":5.8849999999999998},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.3720000000000001,"15min":0.85399999999999998},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":16.280999999999999,"5min":29.802,"15min":29.802},"last":1.0329999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.80600000000000005,"15min":0.66200000000000003},"min":{"1min":0.248,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.1080000000000001,"5min":6.0069999999999997,"15min":6.0069999999999997},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.84399999999999997,"15min":0.66000000000000003},"min":{"1min":0.39300000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.3730000000000002,"5min":4.3899999999999997,"15min":4.3899999999999997},"last":0.83599999999999997}]},{"osd":2,"last update":"Tue Sep 10 18:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":1.0029999999999999,"15min":0.72399999999999998},"min":{"1min":0.29499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.407,"5min":13.898999999999999,"15min":13.898999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":1.1040000000000001,"15min":0.751},"min":{"1min":0.28399999999999997,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.5840000000000001,"5min":14.51,"15min":14.51},"last":1.4179999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:28:21 2024","interfaces":[{"interface":"back","average":{"1min":2.2290000000000001,"5min":1.6140000000000001,"15min":0.91700000000000004},"min":{"1min":0.30199999999999999,"5min":0.249,"15min":0.249},"max":{"1min":37.351999999999997,"5min":37.351999999999997,"15min":37.351999999999997},"last":3.5699999999999998},{"interface":"front","average":{"1min":2.2480000000000002,"5min":1.675,"15min":0.94899999999999995},"min":{"1min":0.28000000000000003,"5min":0.255,"15min":0.255},"max":{"1min":37.366,"5min":37.366,"15min":37.366},"last":0.86799999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:27:47 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.35699999999999998,"15min":0.376},"min":{"1min":0.26200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.70999999999999996,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.39800000000000002,"15min":0.41499999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.901,"5min":1.901,"15min":1.901},"last":0.44800000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.41999999999999998,"15min":0.39800000000000002},"min":{"1min":0.29199999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.47899999999999998,"15min":0.45500000000000002},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":0.49199999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:28:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.42699999999999999,"15min":0.49299999999999999},"min":{"1min":0.32500000000000001,"5min":0.255,"15min":0.255},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.46600000000000003,"15min":0.496},"min":{"1min":0.34899999999999998,"5min":0.247,"15min":0.247},"max":{"1min":1.462,"5min":2.5230000000000001,"15min":2.5230000000000001},"last":0.47599999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117102,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1889408,"kb_used_data":1583736,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91855744,"statfs":{"total":95995035648,"available":94060281856,"internally_reserved":0,"allocated":1621745664,"data_stored":1621098610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.8740000000000001,"15min":0.88},"min":{"1min":0.32300000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.0060000000000002,"5min":8.5530000000000008,"15min":8.5530000000000008},"last":0.997},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.948,"15min":0.91600000000000004},"min":{"1min":0.24099999999999999,"5min":0.222,"15min":0.222},"max":{"1min":5.2729999999999997,"5min":13.659000000000001,"15min":13.659000000000001},"last":0.40799999999999997}]},{"osd":1,"last update":"Tue Sep 10 18:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":1.5780000000000001,"15min":0.81599999999999995},"min":{"1min":0.29099999999999998,"5min":0.247,"15min":0.247},"max":{"1min":4.1539999999999999,"5min":11.606,"15min":11.606},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.4059999999999999,"15min":0.73799999999999999},"min":{"1min":0.26600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":4.4770000000000003,"5min":8.5229999999999997,"15min":8.5229999999999997},"last":0.42899999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.5449999999999999,"15min":0.81000000000000005},"min":{"1min":0.29899999999999999,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":5.9829999999999997,"5min":7.7960000000000003,"15min":7.7960000000000003},"last":0.376},{"interface":"front","average":{"1min":1.468,"5min":1.6160000000000001,"15min":0.82499999999999996},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.252},"max":{"1min":7.7830000000000004,"5min":10.113,"15min":10.113},"last":1.496}]},{"osd":4,"last update":"Tue Sep 10 18:28:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.738,"15min":0.94299999999999995},"min":{"1min":0.40500000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.4710000000000001,"5min":23.684999999999999,"15min":23.684999999999999},"last":1.518},{"interface":"front","average":{"1min":0.89100000000000001,"5min":1.613,"15min":0.91200000000000003},"min":{"1min":0.33500000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.3879999999999999,"5min":23.649000000000001,"15min":23.649000000000001},"last":1.155}]},{"osd":5,"last update":"Tue Sep 10 18:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.583,"15min":0.96799999999999997},"min":{"1min":0.38,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":6.0430000000000001,"5min":23.719999999999999,"15min":23.719999999999999},"last":1.8879999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":1.633,"15min":0.95099999999999996},"min":{"1min":0.35599999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.657,"5min":23.707000000000001,"15min":23.707000000000001},"last":1.6479999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:28:33 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.89100000000000001,"15min":0.88300000000000001},"min":{"1min":0.38700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":4.4690000000000003,"5min":4.4690000000000003,"15min":4.4690000000000003},"last":1.109},{"interface":"front","average":{"1min":0.58199999999999996,"5min":1.1819999999999999,"15min":1.282},"min":{"1min":0.39400000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.76400000000000001,"5min":5.8710000000000004,"15min":5.8710000000000004},"last":1.2709999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:28:33 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.6399999999999999,"15min":0.96499999999999997},"min":{"1min":0.45100000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.8719999999999999,"5min":23.748999999999999,"15min":23.748999999999999},"last":1.466},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.7350000000000001,"15min":1.018},"min":{"1min":0.38100000000000001,"5min":0.307,"15min":0.307},"max":{"1min":4.4180000000000001,"5min":25.058,"15min":25.058},"last":0.44600000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411442,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1914212,"kb_used_data":1608540,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91830940,"statfs":{"total":95995035648,"available":94034882560,"internally_reserved":0,"allocated":1647144960,"data_stored":1646525206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":1.8779999999999999,"15min":0.85699999999999998},"min":{"1min":0.214,"5min":0.185,"15min":0.185},"max":{"1min":2.5350000000000001,"5min":17.135000000000002,"15min":17.135000000000002},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.056,"5min":2.1259999999999999,"15min":0.94599999999999995},"min":{"1min":0.252,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.3500000000000001,"5min":17.146000000000001,"15min":17.146000000000001},"last":1.6599999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:28:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.798,"15min":0.82799999999999996},"min":{"1min":0.27100000000000002,"5min":0.193,"15min":0.193},"max":{"1min":3.4319999999999999,"5min":10.077999999999999,"15min":10.077999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.302,"15min":0.67900000000000005},"min":{"1min":0.23499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.569,"5min":12.532999999999999,"15min":12.532999999999999},"last":0.435}]},{"osd":3,"last update":"Tue Sep 10 18:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":2.2000000000000002,"15min":1.0489999999999999},"min":{"1min":0.29699999999999999,"5min":0.251,"15min":0.251},"max":{"1min":5.8479999999999999,"5min":60.786999999999999,"15min":60.786999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.2649999999999999,"5min":2.407,"15min":1.093},"min":{"1min":0.308,"5min":0.25900000000000001,"15min":0.25},"max":{"1min":7.9109999999999996,"5min":60.116999999999997,"15min":60.116999999999997},"last":0.58599999999999997}]},{"osd":4,"last update":"Tue Sep 10 18:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.1890000000000001,"15min":0.77500000000000002},"min":{"1min":0.31900000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.2090000000000001,"5min":6.9169999999999998,"15min":6.9169999999999998},"last":1.036},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.98099999999999998,"15min":0.74399999999999999},"min":{"1min":0.32900000000000001,"5min":0.317,"15min":0.317},"max":{"1min":1.3220000000000001,"5min":12.130000000000001,"15min":12.130000000000001},"last":0.50900000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.98499999999999999,"15min":0.72799999999999998},"min":{"1min":0.35099999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.5600000000000001,"5min":12.106,"15min":12.106},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":1.0129999999999999,"15min":0.751},"min":{"1min":0.33300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.5860000000000001,"5min":12.154,"15min":12.154},"last":0.54000000000000004}]},{"osd":6,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":1.02,"15min":1.0860000000000001},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.49,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":1.123},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.85299999999999998,"15min":0.84799999999999998},"min":{"1min":0.38300000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":3.9969999999999999,"5min":3.9969999999999999,"15min":3.9969999999999999},"last":0.48399999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:28:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.264,"15min":0.84099999999999997},"min":{"1min":0.41799999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":5.0039999999999996,"5min":7.1639999999999997,"15min":7.1639999999999997},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.75700000000000001,"5min":1.232,"15min":0.81799999999999995},"min":{"1min":0.40799999999999997,"5min":0.307,"15min":0.307},"max":{"1min":2.5819999999999999,"5min":6.9669999999999996,"15min":6.9669999999999996},"last":1.0880000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574967,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2138108,"kb_used_data":1832436,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91607044,"statfs":{"total":95995035648,"available":93805613056,"internally_reserved":0,"allocated":1876414464,"data_stored":1875770720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":2.6749999999999998,"15min":1.125},"min":{"1min":0.217,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":1.6220000000000001,"5min":80.456999999999994,"15min":80.456999999999994},"last":6.4240000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":2.8559999999999999,"15min":1.204},"min":{"1min":0.22700000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.3999999999999999,"5min":80.578000000000003,"15min":80.578000000000003},"last":0.75600000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":1.6240000000000001,"15min":0.79900000000000004},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.4809999999999999,"5min":13.035,"15min":13.035},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.5349999999999999,"15min":0.75700000000000001},"min":{"1min":0.24199999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.9700000000000002,"5min":6.593,"15min":6.593},"last":0.78500000000000003}]},{"osd":3,"last update":"Tue Sep 10 18:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":2.6160000000000001,"15min":1.1479999999999999},"min":{"1min":0.29999999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.8769999999999998,"5min":43.076000000000001,"15min":43.076000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.006,"5min":2.7080000000000002,"15min":1.21},"min":{"1min":0.29699999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.5,"5min":41.93,"15min":41.93},"last":0.69199999999999995}]},{"osd":4,"last update":"Tue Sep 10 18:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":1.0760000000000001,"15min":0.75},"min":{"1min":0.32100000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.794,"5min":7.1500000000000004,"15min":7.1500000000000004},"last":3.4279999999999999},{"interface":"front","average":{"1min":0.96299999999999997,"5min":1.3009999999999999,"15min":0.879},"min":{"1min":0.38400000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.984,"5min":6.1600000000000001,"15min":19.393000000000001},"last":1.337}]},{"osd":5,"last update":"Tue Sep 10 18:27:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.258,"15min":0.78700000000000003},"min":{"1min":0.43099999999999999,"5min":0.311,"15min":0.311},"max":{"1min":6.2069999999999999,"5min":15.48,"15min":15.48},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.88,"5min":1.0580000000000001,"15min":0.755},"min":{"1min":0.36299999999999999,"5min":0.308,"15min":0.308},"max":{"1min":4.4509999999999996,"5min":6.1699999999999999,"15min":6.1699999999999999},"last":3.3780000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.2450000000000001,"15min":1.3169999999999999},"min":{"1min":0.40400000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.1499999999999999,"5min":7.0170000000000003,"15min":7.0170000000000003},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":1.2629999999999999,"15min":1.335},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.4470000000000001,"5min":4.4770000000000003,"15min":4.4770000000000003},"last":0.71599999999999997}]},{"osd":7,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.321,"15min":0.83799999999999997},"min":{"1min":0.47699999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.1829999999999998,"5min":15.545999999999999,"15min":15.545999999999999},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":1.1899999999999999,"15min":0.78600000000000003},"min":{"1min":0.438,"5min":0.313,"15min":0.313},"max":{"1min":2.8500000000000001,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":0.73699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738492,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1878444,"kb_used_data":1576932,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91866708,"statfs":{"total":95995035648,"available":94071508992,"internally_reserved":0,"allocated":1614778368,"data_stored":1614150279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:28:12 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.6259999999999999,"15min":0.77600000000000002},"min":{"1min":0.23799999999999999,"5min":0.193,"15min":0.193},"max":{"1min":8.4280000000000008,"5min":12.106999999999999,"15min":12.106999999999999},"last":1.4159999999999999},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.8340000000000001,"15min":0.85099999999999998},"min":{"1min":0.27400000000000002,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":3.6680000000000001,"5min":13.436,"15min":13.436},"last":1.1140000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:28:39 2024","interfaces":[{"interface":"back","average":{"1min":2.2970000000000002,"5min":2.1709999999999998,"15min":0.99199999999999999},"min":{"1min":0.22,"5min":0.189,"15min":0.189},"max":{"1min":30.219999999999999,"5min":30.219999999999999,"15min":30.219999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":2.23,"5min":2.198,"15min":0.998},"min":{"1min":0.24099999999999999,"5min":0.22,"15min":0.22},"max":{"1min":30.129999999999999,"5min":30.129999999999999,"15min":30.129999999999999},"last":2.7109999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:28:00 2024","interfaces":[{"interface":"back","average":{"1min":1.758,"5min":2.1850000000000001,"15min":1.028},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.1159999999999997,"5min":23.143999999999998,"15min":23.143999999999998},"last":2.7440000000000002},{"interface":"front","average":{"1min":1.159,"5min":2.012,"15min":0.96499999999999997},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.8109999999999999,"5min":23.035,"15min":23.035},"last":1.339}]},{"osd":4,"last update":"Tue Sep 10 18:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.556,"15min":0.93400000000000005},"min":{"1min":0.48699999999999999,"5min":0.246,"15min":0.246},"max":{"1min":2.1219999999999999,"5min":17.151,"15min":17.151},"last":1.3660000000000001},{"interface":"front","average":{"1min":0.89900000000000002,"5min":1.6140000000000001,"15min":0.97099999999999997},"min":{"1min":0.32200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.9910000000000001,"5min":18.728999999999999,"15min":18.728999999999999},"last":0.46700000000000003}]},{"osd":5,"last update":"Tue Sep 10 18:27:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2280000000000002,"5min":1.845,"15min":1.0149999999999999},"min":{"1min":0.33300000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":18.77,"5min":18.77,"15min":18.77},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.47,"5min":1.5189999999999999,"15min":0.90800000000000003},"min":{"1min":0.38400000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.8170000000000002,"5min":14.991,"15min":14.991},"last":2.6859999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.2090000000000001,"15min":1.274},"min":{"1min":0.45700000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.1499999999999999,"5min":4.7830000000000004,"15min":4.7830000000000004},"last":1.3939999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":1.321,"15min":1.385},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":2.9540000000000002,"5min":3.77,"15min":3.77},"last":0.44800000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.55,"15min":0.89800000000000002},"min":{"1min":0.376,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.0179999999999998,"5min":13.711,"15min":13.711},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":1.4910000000000001,"15min":0.876},"min":{"1min":0.435,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":2.4220000000000002,"5min":15.032,"15min":15.032},"last":2.6110000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1630273536,"data_stored":1630273536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1889861632,"data_stored":1889861632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1645740032,"data_stored":1645740032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1620901888,"data_stored":1620901888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1357643776,"data_stored":1357643776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2174550016,"data_stored":2174550016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1395130368,"data_stored":1395130368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1344995328,"data_stored":1344995328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:28:47.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:28:47.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:47.229 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:28:47.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:46 smithi028 ceph-mon[39220]: osdmap e198: 8 total, 8 up, 8 in 2024-09-10T18:28:47.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:46 smithi028 ceph-mon[29291]: osdmap e198: 8 total, 8 up, 8 in 2024-09-10T18:28:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:47.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:47.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:47.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:47.417 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:47.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:47.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:47.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:47.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:47.587 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:47.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:47.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:47.731 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:47.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:47.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:47.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:47.849 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:28:47.849 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:28:47.874 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:47.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:47.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:47.976 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:48.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:48.007 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:48.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:48.108 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:48.109 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 41 16 76606 76590 116.737 62.9375 0.0156963 0.00856101 2024-09-10T18:28:48.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:47 smithi028 ceph-mon[39220]: pgmap v325: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-10T18:28:48.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:47 smithi028 ceph-mon[29291]: pgmap v325: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-10T18:28:48.140 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:48.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:48.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:48.234 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:47 smithi204 ceph-mon[32357]: pgmap v325: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-10T18:28:48.242 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:48.273 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:48.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:48.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:48.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:48.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:48.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:48.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:48.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:48.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:48.666 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:48.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-10T18:28:48.700 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:48.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:48.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:48.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:48.839 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:48.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:48.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:48.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:48.975 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:28:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:48.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:48.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:49.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:49.085 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:28:49.109 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 42 16 77809 77793 115.745 75.1875 0.010895 0.00863426 2024-09-10T18:28:49.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:49.140 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:49.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:49.242 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:49.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:48 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1911963121' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:28:49.253 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:48 smithi204 ceph-mon[32357]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:28:49.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:48 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1911963121' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:28:49.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:48 smithi028 ceph-mon[29291]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:28:49.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:48 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1911963121' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:28:49.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:48 smithi028 ceph-mon[39220]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:28:49.284 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:49.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:49.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:49.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:49.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:49.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:49.525 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:49.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:49.566 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:49.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:49.667 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:49.700 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:49.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:49.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:49.802 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:49.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:49.834 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:49.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:49.936 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:49.967 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:49.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:49.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:50.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:49 smithi028 ceph-mon[39220]: pgmap v326: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-10T18:28:50.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:49 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:28:50.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:49 smithi028 ceph-mon[39220]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:28:50.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:49 smithi028 ceph-mon[39220]: osdmap e199: 8 total, 8 up, 8 in 2024-09-10T18:28:50.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:49 smithi028 ceph-mon[39220]: 4.1c scrub starts 2024-09-10T18:28:50.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:49 smithi028 ceph-mon[39220]: 4.1c scrub ok 2024-09-10T18:28:50.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:49 smithi028 ceph-mon[39220]: 4.1b scrub starts 2024-09-10T18:28:50.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:49 smithi028 ceph-mon[39220]: 4.1b scrub ok 2024-09-10T18:28:50.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:49 smithi028 ceph-mon[29291]: pgmap v326: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-10T18:28:50.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:49 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:28:50.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:49 smithi028 ceph-mon[29291]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:28:50.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:49 smithi028 ceph-mon[29291]: osdmap e199: 8 total, 8 up, 8 in 2024-09-10T18:28:50.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:49 smithi028 ceph-mon[29291]: 4.1c scrub starts 2024-09-10T18:28:50.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:49 smithi028 ceph-mon[29291]: 4.1c scrub ok 2024-09-10T18:28:50.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:49 smithi028 ceph-mon[29291]: 4.1b scrub starts 2024-09-10T18:28:50.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:49 smithi028 ceph-mon[29291]: 4.1b scrub ok 2024-09-10T18:28:50.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:50.104 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:50.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:50.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:50.109 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 43 16 79089 79073 114.914 80 0.0138991 0.00869714 2024-09-10T18:28:50.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:50.255 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:50.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:50.256 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:49 smithi204 ceph-mon[32357]: pgmap v326: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-10T18:28:50.257 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:49 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:28:50.257 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:49 smithi204 ceph-mon[32357]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:28:50.257 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:49 smithi204 ceph-mon[32357]: osdmap e199: 8 total, 8 up, 8 in 2024-09-10T18:28:50.257 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:49 smithi204 ceph-mon[32357]: 4.1c scrub starts 2024-09-10T18:28:50.257 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:49 smithi204 ceph-mon[32357]: 4.1c scrub ok 2024-09-10T18:28:50.257 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:49 smithi204 ceph-mon[32357]: 4.1b scrub starts 2024-09-10T18:28:50.257 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:49 smithi204 ceph-mon[32357]: 4.1b scrub ok 2024-09-10T18:28:50.357 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:50.368 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:28:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:28:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:50.386 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:50.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:50.403 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:28:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:28:50.438 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:28:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:28:50.475 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:28:50.476 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:28:50.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:28:50.508 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:28:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:28:50.532 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:50.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:50.548 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:28:50.549 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:28:50.578 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:28:50.578 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:28:50.614 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:28:50.615 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:28:50.635 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:50.669 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:50.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:50.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:50.771 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:50.804 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:50.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:50.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:50.906 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:50.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:50.934 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:50.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:51.035 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:51.077 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:51.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:51.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:51.110 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 44 16 80328 80312 114.062 77.4375 0.00837786 0.00876213 2024-09-10T18:28:51.115 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:28:51.115 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:28:51.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.116 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15117.0:80324 8.8 8:12f14d81:::benchmark_data_smithi028_109242_object80323:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-10T18:28:51.116 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:28:51.103735+0000", 2024-09-10T18:28:51.116 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0087700429999999999, 2024-09-10T18:28:51.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088015869999999996, 2024-09-10T18:28:51.116 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:51.116 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:28:51.117 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:51.117 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15117", 2024-09-10T18:28:51.117 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/121353060", 2024-09-10T18:28:51.117 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 80324 2024-09-10T18:28:51.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.117 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:51.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:51.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.103735+0000", 2024-09-10T18:28:51.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:51.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:51.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.103735+0000", 2024-09-10T18:28:51.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:51.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:51.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.103733+0000", 2024-09-10T18:28:51.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:28:51.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:51.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.103766+0000", 2024-09-10T18:28:51.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3420000000000002e-05 2024-09-10T18:28:51.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:51.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.103777+0000", 2024-09-10T18:28:51.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1474999999999999e-05 2024-09-10T18:28:51.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:51.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.103784+0000", 2024-09-10T18:28:51.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7170000000000002e-06 2024-09-10T18:28:51.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:51.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.103832+0000", 2024-09-10T18:28:51.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7882e-05 2024-09-10T18:28:51.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:51.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.103986+0000", 2024-09-10T18:28:51.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000153542 2024-09-10T18:28:51.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:28:51.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.104133+0000", 2024-09-10T18:28:51.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000147408 2024-09-10T18:28:51.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:51.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.106779+0000", 2024-09-10T18:28:51.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0026462370000000001 2024-09-10T18:28:51.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:51.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.108879+0000", 2024-09-10T18:28:51.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021003039999999999 2024-09-10T18:28:51.122 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:51.123 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:51.123 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:51.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.123 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.15117.0:80328 8.a e199/180 8:5642aeb0:::benchmark_data_smithi028_109242_object80327:head v 199'4988, mlcod=199'4986)", 2024-09-10T18:28:51.123 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:28:51.110949+0000", 2024-09-10T18:28:51.123 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0015553990000000001, 2024-09-10T18:28:51.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00167019, 2024-09-10T18:28:51.123 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:51.123 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:28:51.124 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:51.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:51.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.110949+0000", 2024-09-10T18:28:51.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:51.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:51.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.110949+0000", 2024-09-10T18:28:51.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:51.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:51.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.110948+0000", 2024-09-10T18:28:51.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:28:51.126 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.126 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:51.127 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.110996+0000", 2024-09-10T18:28:51.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7225999999999998e-05 2024-09-10T18:28:51.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:51.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.111005+0000", 2024-09-10T18:28:51.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6180000000000001e-06 2024-09-10T18:28:51.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:51.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.111011+0000", 2024-09-10T18:28:51.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3839999999999996e-06 2024-09-10T18:28:51.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:51.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.111054+0000", 2024-09-10T18:28:51.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3778e-05 2024-09-10T18:28:51.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:51.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.111072+0000", 2024-09-10T18:28:51.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7178000000000001e-05 2024-09-10T18:28:51.129 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:51.129 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:51.129 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:51.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.130 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.15117.0:80333 8.2 e199/180 8:45af627c:::benchmark_data_smithi028_109242_object80332:head v 199'4888, mlcod=199'4886)", 2024-09-10T18:28:51.130 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:28:51.111410+0000", 2024-09-10T18:28:51.130 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.001095083, 2024-09-10T18:28:51.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0012628439999999999, 2024-09-10T18:28:51.130 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:51.130 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:28:51.130 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:51.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:51.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.111410+0000", 2024-09-10T18:28:51.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:51.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:51.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.111410+0000", 2024-09-10T18:28:51.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:51.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:51.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.111408+0000", 2024-09-10T18:28:51.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:28:51.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:51.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.111431+0000", 2024-09-10T18:28:51.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2390000000000001e-05 2024-09-10T18:28:51.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:51.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.111443+0000", 2024-09-10T18:28:51.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1678999999999999e-05 2024-09-10T18:28:51.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:51.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.111448+0000", 2024-09-10T18:28:51.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1370000000000001e-06 2024-09-10T18:28:51.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:51.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.111500+0000", 2024-09-10T18:28:51.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1876999999999998e-05 2024-09-10T18:28:51.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:51.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:51.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:51.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:28:51.111518+0000", 2024-09-10T18:28:51.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.88e-05 2024-09-10T18:28:51.133 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:51.134 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:51.134 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:51.134 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:51.134 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-09-10T18:28:51.134 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 3 2024-09-10T18:28:51.134 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:28:51.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:51.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:51.221 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:51.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:51.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:50 smithi028 ceph-mon[29291]: 4.9 scrub starts 2024-09-10T18:28:51.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:50 smithi028 ceph-mon[29291]: 4.9 scrub ok 2024-09-10T18:28:51.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:50 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1612454545' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:28:51.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:50 smithi028 ceph-mon[39220]: 4.9 scrub starts 2024-09-10T18:28:51.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:50 smithi028 ceph-mon[39220]: 4.9 scrub ok 2024-09-10T18:28:51.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:50 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1612454545' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:28:51.257 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:50 smithi204 ceph-mon[32357]: 4.9 scrub starts 2024-09-10T18:28:51.257 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:50 smithi204 ceph-mon[32357]: 4.9 scrub ok 2024-09-10T18:28:51.257 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:50 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1612454545' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:28:51.322 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:51.364 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:51.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:51.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:51.466 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:51.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:51.504 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:51.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:51.605 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:51.615 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:28:51] "GET /metrics HTTP/1.1" 200 224195 "" "Prometheus/2.43.0" 2024-09-10T18:28:51.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:51.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:51.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:51.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:51.774 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:51.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:51.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:51.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:28:51.876 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:51.908 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:51.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:51.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:51.960 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-10T18:28:52.013 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:52.045 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:52.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:52.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:52.110 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 45 16 81617 81601 113.316 80.5625 0.00554611 0.00881972 2024-09-10T18:28:52.147 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:52.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[29291]: 4.4 scrub starts 2024-09-10T18:28:52.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[29291]: 4.4 scrub ok 2024-09-10T18:28:52.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[29291]: pgmap v328: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-10T18:28:52.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[29291]: 3.1a scrub starts 2024-09-10T18:28:52.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[29291]: 3.1a scrub ok 2024-09-10T18:28:52.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[29291]: 5.1f scrub starts 2024-09-10T18:28:52.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[29291]: 3.18 scrub starts 2024-09-10T18:28:52.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[29291]: 3.18 scrub ok 2024-09-10T18:28:52.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[29291]: 5.1f scrub ok 2024-09-10T18:28:52.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[39220]: 4.4 scrub starts 2024-09-10T18:28:52.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[39220]: 4.4 scrub ok 2024-09-10T18:28:52.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[39220]: pgmap v328: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-10T18:28:52.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[39220]: 3.1a scrub starts 2024-09-10T18:28:52.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[39220]: 3.1a scrub ok 2024-09-10T18:28:52.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[39220]: 5.1f scrub starts 2024-09-10T18:28:52.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[39220]: 3.18 scrub starts 2024-09-10T18:28:52.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[39220]: 3.18 scrub ok 2024-09-10T18:28:52.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:51 smithi028 ceph-mon[39220]: 5.1f scrub ok 2024-09-10T18:28:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:52.178 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:52.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:52.279 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:52.334 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:52.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:52.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:52.336 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:28:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:28:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:28:52.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:51 smithi204 ceph-mon[32357]: 4.4 scrub starts 2024-09-10T18:28:52.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:51 smithi204 ceph-mon[32357]: 4.4 scrub ok 2024-09-10T18:28:52.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:51 smithi204 ceph-mon[32357]: pgmap v328: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-10T18:28:52.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:51 smithi204 ceph-mon[32357]: 3.1a scrub starts 2024-09-10T18:28:52.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:51 smithi204 ceph-mon[32357]: 3.1a scrub ok 2024-09-10T18:28:52.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:51 smithi204 ceph-mon[32357]: 5.1f scrub starts 2024-09-10T18:28:52.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:51 smithi204 ceph-mon[32357]: 3.18 scrub starts 2024-09-10T18:28:52.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:51 smithi204 ceph-mon[32357]: 3.18 scrub ok 2024-09-10T18:28:52.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:51 smithi204 ceph-mon[32357]: 5.1f scrub ok 2024-09-10T18:28:52.350 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:28:52.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:52.466 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:52.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:52.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:52.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:52.608 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:52.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:52.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:52.710 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:52.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:52.742 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:52.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:52.844 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:52.872 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:52.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:52.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:52.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:28:53.042 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:53.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:53.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:53.111 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 46 16 82799 82783 112.458 73.875 0.014991 0.00888718 2024-09-10T18:28:53.144 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:53.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:53.186 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:53.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:53.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:52 smithi028 ceph-mon[39220]: 3.11 scrub starts 2024-09-10T18:28:53.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:52 smithi028 ceph-mon[39220]: 3.11 scrub ok 2024-09-10T18:28:53.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:52 smithi028 ceph-mon[39220]: 3.1c scrub starts 2024-09-10T18:28:53.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:52 smithi028 ceph-mon[39220]: 3.1c scrub ok 2024-09-10T18:28:53.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:52 smithi028 ceph-mon[39220]: 4.1a scrub starts 2024-09-10T18:28:53.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:52 smithi028 ceph-mon[39220]: 4.1a scrub ok 2024-09-10T18:28:53.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:52 smithi028 ceph-mon[39220]: 4.5 scrub starts 2024-09-10T18:28:53.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:52 smithi028 ceph-mon[39220]: 4.5 scrub ok 2024-09-10T18:28:53.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:52 smithi028 ceph-mon[29291]: 3.11 scrub starts 2024-09-10T18:28:53.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:52 smithi028 ceph-mon[29291]: 3.11 scrub ok 2024-09-10T18:28:53.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:52 smithi028 ceph-mon[29291]: 3.1c scrub starts 2024-09-10T18:28:53.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:52 smithi028 ceph-mon[29291]: 3.1c scrub ok 2024-09-10T18:28:53.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:52 smithi028 ceph-mon[29291]: 4.1a scrub starts 2024-09-10T18:28:53.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:52 smithi028 ceph-mon[29291]: 4.1a scrub ok 2024-09-10T18:28:53.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:52 smithi028 ceph-mon[29291]: 4.5 scrub starts 2024-09-10T18:28:53.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:52 smithi028 ceph-mon[29291]: 4.5 scrub ok 2024-09-10T18:28:53.286 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:53.322 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:53.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:53.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:52 smithi204 ceph-mon[32357]: 3.11 scrub starts 2024-09-10T18:28:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:52 smithi204 ceph-mon[32357]: 3.11 scrub ok 2024-09-10T18:28:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:52 smithi204 ceph-mon[32357]: 3.1c scrub starts 2024-09-10T18:28:53.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:52 smithi204 ceph-mon[32357]: 3.1c scrub ok 2024-09-10T18:28:53.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:52 smithi204 ceph-mon[32357]: 4.1a scrub starts 2024-09-10T18:28:53.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:52 smithi204 ceph-mon[32357]: 4.1a scrub ok 2024-09-10T18:28:53.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:52 smithi204 ceph-mon[32357]: 4.5 scrub starts 2024-09-10T18:28:53.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:52 smithi204 ceph-mon[32357]: 4.5 scrub ok 2024-09-10T18:28:53.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:53.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:53.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:53.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:53.604 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:53.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:53.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:53.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:28:53.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:53.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:53.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:53.870 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:53.897 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:53.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:53.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:54.009 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:28:54.013 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:28:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:54.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:54.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:54.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:54 smithi028 ceph-mon[29291]: pgmap v329: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T18:28:54.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:54 smithi028 ceph-mon[29291]: 6.1a scrub starts 2024-09-10T18:28:54.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:54 smithi028 ceph-mon[29291]: 6.1a scrub ok 2024-09-10T18:28:54.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:54 smithi028 ceph-mon[29291]: 6.b scrub starts 2024-09-10T18:28:54.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:54 smithi028 ceph-mon[29291]: 6.b scrub ok 2024-09-10T18:28:54.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:54 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:28:54.092 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:54 smithi028 ceph-mon[39220]: pgmap v329: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T18:28:54.092 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:54 smithi028 ceph-mon[39220]: 6.1a scrub starts 2024-09-10T18:28:54.092 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:54 smithi028 ceph-mon[39220]: 6.1a scrub ok 2024-09-10T18:28:54.092 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:54 smithi028 ceph-mon[39220]: 6.b scrub starts 2024-09-10T18:28:54.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:54 smithi028 ceph-mon[39220]: 6.b scrub ok 2024-09-10T18:28:54.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:54 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:28:54.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:54 smithi204 ceph-mon[32357]: pgmap v329: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T18:28:54.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:54 smithi204 ceph-mon[32357]: 6.1a scrub starts 2024-09-10T18:28:54.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:54 smithi204 ceph-mon[32357]: 6.1a scrub ok 2024-09-10T18:28:54.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:54 smithi204 ceph-mon[32357]: 6.b scrub starts 2024-09-10T18:28:54.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:54 smithi204 ceph-mon[32357]: 6.b scrub ok 2024-09-10T18:28:54.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:54 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:28:54.111 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 47 16 83875 83859 111.496 67.25 0.0200653 0.00896381 2024-09-10T18:28:54.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:54.187 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:54.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:54.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:54.281 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:28:54.281 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:28:54.281 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-10T18:28:54.281 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-10T18:28:54.282 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:28:54.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:54.324 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:54.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:54.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:54.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:54.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:54.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:54.564 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:54.600 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:54.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:54.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:54.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:28:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:28:54.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:54.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:54.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:54.847 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:54.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:54.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:54.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:54.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:55.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:55.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:55.111 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 48 16 85144 85128 110.826 79.3125 0.0111901 0.00901848 2024-09-10T18:28:55.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:55.142 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-10T18:28:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:55.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:55.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:55.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:55 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:28:55.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:55 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:28:55.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:55 smithi028 ceph-mon[39220]: 4.b scrub starts 2024-09-10T18:28:55.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:55 smithi028 ceph-mon[39220]: 4.b scrub ok 2024-09-10T18:28:55.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:55 smithi028 ceph-mon[39220]: from='client.15174 v1:172.21.15.28:0/1440977705' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:28:55.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:55 smithi028 ceph-mon[39220]: osdmap e200: 8 total, 8 up, 8 in 2024-09-10T18:28:55.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:55 smithi028 ceph-mon[39220]: 5.13 scrub starts 2024-09-10T18:28:55.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:55 smithi028 ceph-mon[39220]: 5.13 scrub ok 2024-09-10T18:28:55.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:55 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:28:55.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:55 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:28:55.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:55 smithi028 ceph-mon[29291]: 4.b scrub starts 2024-09-10T18:28:55.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:55 smithi028 ceph-mon[29291]: 4.b scrub ok 2024-09-10T18:28:55.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:55 smithi028 ceph-mon[29291]: from='client.15174 v1:172.21.15.28:0/1440977705' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:28:55.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:55 smithi028 ceph-mon[29291]: osdmap e200: 8 total, 8 up, 8 in 2024-09-10T18:28:55.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:55 smithi028 ceph-mon[29291]: 5.13 scrub starts 2024-09-10T18:28:55.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:55 smithi028 ceph-mon[29291]: 5.13 scrub ok 2024-09-10T18:28:55.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:55 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:28:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:55 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:28:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:55 smithi204 ceph-mon[32357]: 4.b scrub starts 2024-09-10T18:28:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:55 smithi204 ceph-mon[32357]: 4.b scrub ok 2024-09-10T18:28:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:55 smithi204 ceph-mon[32357]: from='client.15174 v1:172.21.15.28:0/1440977705' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:28:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:55 smithi204 ceph-mon[32357]: osdmap e200: 8 total, 8 up, 8 in 2024-09-10T18:28:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:55 smithi204 ceph-mon[32357]: 5.13 scrub starts 2024-09-10T18:28:55.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:55 smithi204 ceph-mon[32357]: 5.13 scrub ok 2024-09-10T18:28:55.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:55.356 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:55.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:55.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:55.496 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:55.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:55.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:55.511 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:28:55.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:55.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:55.640 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:55.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:55.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:55.786 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-09-10T18:28:52.884940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5186853084,"num_objects":79332,"num_object_clones":0,"num_object_copies":237996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79332,"num_whiteouts":0,"num_read":1151,"num_read_kb":903,"num_write":79566,"num_write_kb":5064747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78959,"ondisk_log_size":78959,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17527712,"kb_used_data":15085384,"kb_used_omap":19,"kb_used_meta":2442220,"kb_avail":732433504,"statfs":{"total":767960285184,"available":750011908096,"internally_reserved":0,"allocated":15447433216,"data_stored":15442286494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20033,"internal_metadata":2500833727},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":717815808,"num_objects":10953,"num_object_clones":0,"num_object_copies":32859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10953,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":10953,"num_write_kb":700992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004349"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":179,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.948858+0000","last_change":"2024-09-10T18:28:14.076324+0000","last_active":"2024-09-10T18:28:45.948858+0000","last_peered":"2024-09-10T18:28:45.948858+0000","last_clean":"2024-09-10T18:28:45.948858+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:28:45.948858+0000","last_undegraded":"2024-09-10T18:28:45.948858+0000","last_fullsized":"2024-09-10T18:28:45.948858+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:14.076156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:28:14.076156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:43.937320+0000","scrub_duration":0.003669664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":199,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.944078+0000","last_change":"2024-09-10T18:27:11.977624+0000","last_active":"2024-09-10T18:28:45.944078+0000","last_peered":"2024-09-10T18:28:45.944078+0000","last_clean":"2024-09-10T18:28:45.944078+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:28:45.944078+0000","last_undegraded":"2024-09-10T18:28:45.944078+0000","last_fullsized":"2024-09-10T18:28:45.944078+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:27:11.977562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:27:11.977562+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012415305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":174,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.948016+0000","last_change":"2024-09-10T18:26:57.717081+0000","last_active":"2024-09-10T18:28:45.948016+0000","last_peered":"2024-09-10T18:28:45.948016+0000","last_clean":"2024-09-10T18:28:45.948016+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:28:45.948016+0000","last_undegraded":"2024-09-10T18:28:45.948016+0000","last_fullsized":"2024-09-10T18:28:45.948016+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:57.716697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:26:57.716697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004885314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":172,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.945714+0000","last_change":"2024-09-10T18:26:18.749617+0000","last_active":"2024-09-10T18:28:45.945714+0000","last_peered":"2024-09-10T18:28:45.945714+0000","last_clean":"2024-09-10T18:28:45.945714+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:28:45.945714+0000","last_undegraded":"2024-09-10T18:28:45.945714+0000","last_fullsized":"2024-09-10T18:28:45.945714+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:26:18.749558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090834959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":181,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951227+0000","last_change":"2024-09-10T18:27:10.424485+0000","last_active":"2024-09-10T18:28:45.951227+0000","last_peered":"2024-09-10T18:28:45.951227+0000","last_clean":"2024-09-10T18:28:45.951227+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:28:45.951227+0000","last_undegraded":"2024-09-10T18:28:45.951227+0000","last_fullsized":"2024-09-10T18:28:45.951227+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:10.424386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:10.424386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023425607000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":187,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952781+0000","last_change":"2024-09-10T18:27:14.539312+0000","last_active":"2024-09-10T18:28:45.952781+0000","last_peered":"2024-09-10T18:28:45.952781+0000","last_clean":"2024-09-10T18:28:45.952781+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:28:45.952781+0000","last_undegraded":"2024-09-10T18:28:45.952781+0000","last_fullsized":"2024-09-10T18:28:45.952781+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:27:14.532784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063752030000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"123'22","reported_seq":210,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951661+0000","last_change":"2024-09-10T18:28:15.221269+0000","last_active":"2024-09-10T18:28:45.951661+0000","last_peered":"2024-09-10T18:28:45.951661+0000","last_clean":"2024-09-10T18:28:45.951661+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:28:45.951661+0000","last_undegraded":"2024-09-10T18:28:45.951661+0000","last_fullsized":"2024-09-10T18:28:45.951661+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'22","last_scrub_stamp":"2024-09-10T18:28:15.221190+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:28:15.221190+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:25.690656+0000","scrub_duration":0.062041365000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":184,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.943996+0000","last_change":"2024-09-10T18:27:52.257483+0000","last_active":"2024-09-10T18:28:45.943996+0000","last_peered":"2024-09-10T18:28:45.943996+0000","last_clean":"2024-09-10T18:28:45.943996+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:28:45.943996+0000","last_undegraded":"2024-09-10T18:28:45.943996+0000","last_fullsized":"2024-09-10T18:28:45.943996+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.257417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.257417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:56.580125+0000","scrub_duration":0.001302342,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":189,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951052+0000","last_change":"2024-09-10T18:27:53.668356+0000","last_active":"2024-09-10T18:28:45.951052+0000","last_peered":"2024-09-10T18:28:45.951052+0000","last_clean":"2024-09-10T18:28:45.951052+0000","last_became_active":"2024-09-10T18:26:27.644961+0000","last_became_peered":"2024-09-10T18:26:27.644961+0000","last_unstale":"2024-09-10T18:28:45.951052+0000","last_undegraded":"2024-09-10T18:28:45.951052+0000","last_fullsized":"2024-09-10T18:28:45.951052+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:53.668292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:27:53.668292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:03.826593+0000","scrub_duration":0.001492155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":192,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.946015+0000","last_change":"2024-09-10T18:26:11.787888+0000","last_active":"2024-09-10T18:28:45.946015+0000","last_peered":"2024-09-10T18:28:45.946015+0000","last_clean":"2024-09-10T18:28:45.946015+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:28:45.946015+0000","last_undegraded":"2024-09-10T18:28:45.946015+0000","last_fullsized":"2024-09-10T18:28:45.946015+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:26:11.787799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:11.787799+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081486470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":185,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.950620+0000","last_change":"2024-09-10T18:28:13.706380+0000","last_active":"2024-09-10T18:28:45.950620+0000","last_peered":"2024-09-10T18:28:45.950620+0000","last_clean":"2024-09-10T18:28:45.950620+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:28:45.950620+0000","last_undegraded":"2024-09-10T18:28:45.950620+0000","last_fullsized":"2024-09-10T18:28:45.950620+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:13.706259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:13.706259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:17.655828+0000","scrub_duration":0.039021994999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":183,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.947811+0000","last_change":"2024-09-10T18:28:11.043011+0000","last_active":"2024-09-10T18:28:45.947811+0000","last_peered":"2024-09-10T18:28:45.947811+0000","last_clean":"2024-09-10T18:28:45.947811+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:28:45.947811+0000","last_undegraded":"2024-09-10T18:28:45.947811+0000","last_fullsized":"2024-09-10T18:28:45.947811+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:11.042929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:28:11.042929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:30.888869+0000","scrub_duration":0.0032674520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":179,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952377+0000","last_change":"2024-09-10T18:26:17.464263+0000","last_active":"2024-09-10T18:28:45.952377+0000","last_peered":"2024-09-10T18:28:45.952377+0000","last_clean":"2024-09-10T18:28:45.952377+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:28:45.952377+0000","last_undegraded":"2024-09-10T18:28:45.952377+0000","last_fullsized":"2024-09-10T18:28:45.952377+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:17.464205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:17.464205+0000","last_clean_scrub_stamp":"2024-09-10T18:26:17.464205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024019000000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"85'7","reported_seq":204,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:49.205765+0000","last_change":"2024-09-10T18:28:49.205664+0000","last_active":"2024-09-10T18:28:49.205765+0000","last_peered":"2024-09-10T18:28:49.205765+0000","last_clean":"2024-09-10T18:28:49.205765+0000","last_became_active":"2024-09-10T18:21:33.615429+0000","last_became_peered":"2024-09-10T18:21:33.615429+0000","last_unstale":"2024-09-10T18:28:49.205765+0000","last_undegraded":"2024-09-10T18:28:49.205765+0000","last_fullsized":"2024-09-10T18:28:49.205765+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'7","last_scrub_stamp":"2024-09-10T18:28:49.205558+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:28:49.205558+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:30:16.578092+0000","scrub_duration":0.0074042140000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":184,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952040+0000","last_change":"2024-09-10T18:28:10.328366+0000","last_active":"2024-09-10T18:28:45.952040+0000","last_peered":"2024-09-10T18:28:45.952040+0000","last_clean":"2024-09-10T18:28:45.952040+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:28:45.952040+0000","last_undegraded":"2024-09-10T18:28:45.952040+0000","last_fullsized":"2024-09-10T18:28:45.952040+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:10.328277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:10.328277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:24.423649+0000","scrub_duration":0.002745838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":176,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951911+0000","last_change":"2024-09-10T18:28:07.786204+0000","last_active":"2024-09-10T18:28:45.951911+0000","last_peered":"2024-09-10T18:28:45.951911+0000","last_clean":"2024-09-10T18:28:45.951911+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:28:45.951911+0000","last_undegraded":"2024-09-10T18:28:45.951911+0000","last_fullsized":"2024-09-10T18:28:45.951911+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:28:07.786118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:13.460051+0000","scrub_duration":0.0053539800000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":182,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.945119+0000","last_change":"2024-09-10T18:27:55.242815+0000","last_active":"2024-09-10T18:28:45.945119+0000","last_peered":"2024-09-10T18:28:45.945119+0000","last_clean":"2024-09-10T18:28:45.945119+0000","last_became_active":"2024-09-10T18:26:27.650774+0000","last_became_peered":"2024-09-10T18:26:27.650774+0000","last_unstale":"2024-09-10T18:28:45.945119+0000","last_undegraded":"2024-09-10T18:28:45.945119+0000","last_fullsized":"2024-09-10T18:28:45.945119+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:55.242751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:55.242751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:02.100445+0000","scrub_duration":0.001486242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":169,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.948715+0000","last_change":"2024-09-10T18:26:14.600755+0000","last_active":"2024-09-10T18:28:45.948715+0000","last_peered":"2024-09-10T18:28:45.948715+0000","last_clean":"2024-09-10T18:28:45.948715+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:28:45.948715+0000","last_undegraded":"2024-09-10T18:28:45.948715+0000","last_fullsized":"2024-09-10T18:28:45.948715+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:14.600680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:26:14.600680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050859140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":221,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:49.116340+0000","last_change":"2024-09-10T18:28:49.116283+0000","last_active":"2024-09-10T18:28:49.116340+0000","last_peered":"2024-09-10T18:28:49.116340+0000","last_clean":"2024-09-10T18:28:49.116340+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:28:49.116340+0000","last_undegraded":"2024-09-10T18:28:49.116340+0000","last_fullsized":"2024-09-10T18:28:49.116340+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-10T18:28:49.108112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:28:49.108112+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:30:15.989558+0000","scrub_duration":0.0091345969999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":184,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.945497+0000","last_change":"2024-09-10T18:28:01.515411+0000","last_active":"2024-09-10T18:28:45.945497+0000","last_peered":"2024-09-10T18:28:45.945497+0000","last_clean":"2024-09-10T18:28:45.945497+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:28:45.945497+0000","last_undegraded":"2024-09-10T18:28:45.945497+0000","last_fullsized":"2024-09-10T18:28:45.945497+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:01.515358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:01.515358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:20.752464+0000","scrub_duration":0.0019375149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":181,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952606+0000","last_change":"2024-09-10T18:27:15.496208+0000","last_active":"2024-09-10T18:28:45.952606+0000","last_peered":"2024-09-10T18:28:45.952606+0000","last_clean":"2024-09-10T18:28:45.952606+0000","last_became_active":"2024-09-10T18:26:27.640035+0000","last_became_peered":"2024-09-10T18:26:27.640035+0000","last_unstale":"2024-09-10T18:28:45.952606+0000","last_undegraded":"2024-09-10T18:28:45.952606+0000","last_fullsized":"2024-09-10T18:28:45.952606+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:15.496116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:15.496116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058500619999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":178,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.946171+0000","last_change":"2024-09-10T18:26:13.774886+0000","last_active":"2024-09-10T18:28:45.946171+0000","last_peered":"2024-09-10T18:28:45.946171+0000","last_clean":"2024-09-10T18:28:45.946171+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:28:45.946171+0000","last_undegraded":"2024-09-10T18:28:45.946171+0000","last_fullsized":"2024-09-10T18:28:45.946171+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:26:13.774791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006579774,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"88'17","reported_seq":206,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952549+0000","last_change":"2024-09-10T18:28:13.123489+0000","last_active":"2024-09-10T18:28:45.952549+0000","last_peered":"2024-09-10T18:28:45.952549+0000","last_clean":"2024-09-10T18:28:45.952549+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:28:45.952549+0000","last_undegraded":"2024-09-10T18:28:45.952549+0000","last_fullsized":"2024-09-10T18:28:45.952549+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-09-10T18:28:13.123366+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:28:13.123366+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:39.954200+0000","scrub_duration":0.0060192420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":174,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.944865+0000","last_change":"2024-09-10T18:26:09.777629+0000","last_active":"2024-09-10T18:28:45.944865+0000","last_peered":"2024-09-10T18:28:45.944865+0000","last_clean":"2024-09-10T18:28:45.944865+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:28:45.944865+0000","last_undegraded":"2024-09-10T18:28:45.944865+0000","last_fullsized":"2024-09-10T18:28:45.944865+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:09.777548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:26:09.777548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031831369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":174,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.953750+0000","last_change":"2024-09-10T18:27:17.955653+0000","last_active":"2024-09-10T18:28:45.953750+0000","last_peered":"2024-09-10T18:28:45.953750+0000","last_clean":"2024-09-10T18:28:45.953750+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:28:45.953750+0000","last_undegraded":"2024-09-10T18:28:45.953750+0000","last_fullsized":"2024-09-10T18:28:45.953750+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:27:17.955591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:17.955591+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035877499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":188,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.948200+0000","last_change":"2024-09-10T18:28:05.532634+0000","last_active":"2024-09-10T18:28:45.948200+0000","last_peered":"2024-09-10T18:28:45.948200+0000","last_clean":"2024-09-10T18:28:45.948200+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:28:45.948200+0000","last_undegraded":"2024-09-10T18:28:45.948200+0000","last_fullsized":"2024-09-10T18:28:45.948200+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:05.532581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:05.532581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:15.324758+0000","scrub_duration":0.001369901,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"86'15","reported_seq":197,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.949160+0000","last_change":"2024-09-10T18:27:09.623828+0000","last_active":"2024-09-10T18:28:45.949160+0000","last_peered":"2024-09-10T18:28:45.949160+0000","last_clean":"2024-09-10T18:28:45.949160+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:28:45.949160+0000","last_undegraded":"2024-09-10T18:28:45.949160+0000","last_fullsized":"2024-09-10T18:28:45.949160+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-09-10T18:27:09.623769+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:27:09.623769+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001906717,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"146'10","reported_seq":172,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:48.990045+0000","last_change":"2024-09-10T18:26:52.215856+0000","last_active":"2024-09-10T18:28:48.990045+0000","last_peered":"2024-09-10T18:28:48.990045+0000","last_clean":"2024-09-10T18:28:48.990045+0000","last_became_active":"2024-09-10T18:26:27.652489+0000","last_became_peered":"2024-09-10T18:26:27.652489+0000","last_unstale":"2024-09-10T18:28:48.990045+0000","last_undegraded":"2024-09-10T18:28:48.990045+0000","last_fullsized":"2024-09-10T18:28:48.990045+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'10","last_scrub_stamp":"2024-09-10T18:26:52.215776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:26:52.215776+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017419169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"199'4968","reported_seq":4995,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:49.885823+0000","last_change":"2024-09-10T18:27:53.036521+0000","last_active":"2024-09-10T18:28:49.885823+0000","last_peered":"2024-09-10T18:28:49.885823+0000","last_clean":"2024-09-10T18:28:49.885823+0000","last_became_active":"2024-09-10T18:27:53.036244+0000","last_became_peered":"2024-09-10T18:27:53.036244+0000","last_unstale":"2024-09-10T18:28:49.885823+0000","last_undegraded":"2024-09-10T18:28:49.885823+0000","last_fullsized":"2024-09-10T18:28:49.885823+0000","mapping_epoch":180,"log_start":"189'100","ondisk_log_start":"189'100","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4868,"log_dups_size":0,"ondisk_log_size":4868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:59.426828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325582848,"num_objects":4968,"num_object_clones":0,"num_object_copies":14904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4968,"num_write_kb":317952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":95,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951136+0000","last_change":"2024-09-10T18:27:58.699433+0000","last_active":"2024-09-10T18:28:45.951136+0000","last_peered":"2024-09-10T18:28:45.951136+0000","last_clean":"2024-09-10T18:28:45.951136+0000","last_became_active":"2024-09-10T18:26:28.635249+0000","last_became_peered":"2024-09-10T18:26:28.635249+0000","last_unstale":"2024-09-10T18:28:45.951136+0000","last_undegraded":"2024-09-10T18:28:45.951136+0000","last_fullsized":"2024-09-10T18:28:45.951136+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:27:58.699362+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:27:58.699362+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:13.603324+0000","scrub_duration":0.0015816230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"83'22","reported_seq":221,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.948861+0000","last_change":"2024-09-10T18:27:59.479143+0000","last_active":"2024-09-10T18:28:45.948861+0000","last_peered":"2024-09-10T18:28:45.948861+0000","last_clean":"2024-09-10T18:28:45.948861+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:28:45.948861+0000","last_undegraded":"2024-09-10T18:28:45.948861+0000","last_fullsized":"2024-09-10T18:28:45.948861+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-09-10T18:27:59.479087+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:27:59.479087+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:17.236778+0000","scrub_duration":0.0049225479999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":180,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952362+0000","last_change":"2024-09-10T18:28:00.151515+0000","last_active":"2024-09-10T18:28:45.952362+0000","last_peered":"2024-09-10T18:28:45.952362+0000","last_clean":"2024-09-10T18:28:45.952362+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:28:45.952362+0000","last_undegraded":"2024-09-10T18:28:45.952362+0000","last_fullsized":"2024-09-10T18:28:45.952362+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:00.151462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:00.151462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:11.741967+0000","scrub_duration":0.0032292530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":272,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.945188+0000","last_change":"2024-09-10T18:27:58.231688+0000","last_active":"2024-09-10T18:28:45.945188+0000","last_peered":"2024-09-10T18:28:45.945188+0000","last_clean":"2024-09-10T18:28:45.945188+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:28:45.945188+0000","last_undegraded":"2024-09-10T18:28:45.945188+0000","last_fullsized":"2024-09-10T18:28:45.945188+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:27:58.231620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:58.231620+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:13.570474+0000","scrub_duration":0.0016723319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":173,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.950718+0000","last_change":"2024-09-10T18:26:21.511753+0000","last_active":"2024-09-10T18:28:45.950718+0000","last_peered":"2024-09-10T18:28:45.950718+0000","last_clean":"2024-09-10T18:28:45.950718+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:28:45.950718+0000","last_undegraded":"2024-09-10T18:28:45.950718+0000","last_fullsized":"2024-09-10T18:28:45.950718+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:26:21.511676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096344640000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"199'4871","reported_seq":4898,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:50.404958+0000","last_change":"2024-09-10T18:27:53.059901+0000","last_active":"2024-09-10T18:28:50.404958+0000","last_peered":"2024-09-10T18:28:50.404958+0000","last_clean":"2024-09-10T18:28:50.404958+0000","last_became_active":"2024-09-10T18:27:53.059554+0000","last_became_peered":"2024-09-10T18:27:53.059554+0000","last_unstale":"2024-09-10T18:28:50.404958+0000","last_undegraded":"2024-09-10T18:28:50.404958+0000","last_fullsized":"2024-09-10T18:28:50.404958+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4871,"log_dups_size":0,"ondisk_log_size":4871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:55.250255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319225856,"num_objects":4871,"num_object_clones":0,"num_object_copies":14613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4871,"num_write_kb":311744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":94,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.945289+0000","last_change":"2024-09-10T18:28:07.183739+0000","last_active":"2024-09-10T18:28:45.945289+0000","last_peered":"2024-09-10T18:28:45.945289+0000","last_clean":"2024-09-10T18:28:45.945289+0000","last_became_active":"2024-09-10T18:26:28.632744+0000","last_became_peered":"2024-09-10T18:26:28.632744+0000","last_unstale":"2024-09-10T18:28:45.945289+0000","last_undegraded":"2024-09-10T18:28:45.945289+0000","last_fullsized":"2024-09-10T18:28:45.945289+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:28:07.183646+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:28:07.183646+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:30.511324+0000","scrub_duration":0.001501517,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":210,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951730+0000","last_change":"2024-09-10T18:28:03.152360+0000","last_active":"2024-09-10T18:28:45.951730+0000","last_peered":"2024-09-10T18:28:45.951730+0000","last_clean":"2024-09-10T18:28:45.951730+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:28:45.951730+0000","last_undegraded":"2024-09-10T18:28:45.951730+0000","last_fullsized":"2024-09-10T18:28:45.951730+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-10T18:28:03.152304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:28:03.152304+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:27.847090+0000","scrub_duration":0.002028069,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":180,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.953077+0000","last_change":"2024-09-10T18:28:05.141025+0000","last_active":"2024-09-10T18:28:45.953077+0000","last_peered":"2024-09-10T18:28:45.953077+0000","last_clean":"2024-09-10T18:28:45.953077+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:28:45.953077+0000","last_undegraded":"2024-09-10T18:28:45.953077+0000","last_fullsized":"2024-09-10T18:28:45.953077+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:05.140922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.140922+0000","last_clean_scrub_stamp":"2024-09-10T18:28:05.140922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:32.018170+0000","scrub_duration":0.0017057179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":174,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.944256+0000","last_change":"2024-09-10T18:26:07.791694+0000","last_active":"2024-09-10T18:28:45.944256+0000","last_peered":"2024-09-10T18:28:45.944256+0000","last_clean":"2024-09-10T18:28:45.944256+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:28:45.944256+0000","last_undegraded":"2024-09-10T18:28:45.944256+0000","last_fullsized":"2024-09-10T18:28:45.944256+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:07.791645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:26:07.791645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003302878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":172,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.945938+0000","last_change":"2024-09-10T18:28:04.199803+0000","last_active":"2024-09-10T18:28:45.945938+0000","last_peered":"2024-09-10T18:28:45.945938+0000","last_clean":"2024-09-10T18:28:45.945938+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:28:45.945938+0000","last_undegraded":"2024-09-10T18:28:45.945938+0000","last_fullsized":"2024-09-10T18:28:45.945938+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:04.199730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:28:04.199730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:30.291668+0000","scrub_duration":0.001636628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"199'5076","reported_seq":5103,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:49.756880+0000","last_change":"2024-09-10T18:27:53.061607+0000","last_active":"2024-09-10T18:28:49.756880+0000","last_peered":"2024-09-10T18:28:49.756880+0000","last_clean":"2024-09-10T18:28:49.756880+0000","last_became_active":"2024-09-10T18:27:53.061195+0000","last_became_peered":"2024-09-10T18:27:53.061195+0000","last_unstale":"2024-09-10T18:28:49.756880+0000","last_undegraded":"2024-09-10T18:28:49.756880+0000","last_fullsized":"2024-09-10T18:28:49.756880+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":5076,"log_dups_size":0,"ondisk_log_size":5076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:04.794992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332660736,"num_objects":5076,"num_object_clones":0,"num_object_copies":15228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5076,"num_write_kb":324864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":85,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951111+0000","last_change":"2024-09-10T18:28:12.086525+0000","last_active":"2024-09-10T18:28:45.951111+0000","last_peered":"2024-09-10T18:28:45.951111+0000","last_clean":"2024-09-10T18:28:45.951111+0000","last_became_active":"2024-09-10T18:26:18.091896+0000","last_became_peered":"2024-09-10T18:26:18.091896+0000","last_unstale":"2024-09-10T18:28:45.951111+0000","last_undegraded":"2024-09-10T18:28:45.951111+0000","last_fullsized":"2024-09-10T18:28:45.951111+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:28:12.086437+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:28:12.086437+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:27.212845+0000","scrub_duration":0.0048662799999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":204,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.944459+0000","last_change":"2024-09-10T18:26:49.014982+0000","last_active":"2024-09-10T18:28:45.944459+0000","last_peered":"2024-09-10T18:28:45.944459+0000","last_clean":"2024-09-10T18:28:45.944459+0000","last_became_active":"2024-09-10T18:26:27.651364+0000","last_became_peered":"2024-09-10T18:26:27.651364+0000","last_unstale":"2024-09-10T18:28:45.944459+0000","last_undegraded":"2024-09-10T18:28:45.944459+0000","last_fullsized":"2024-09-10T18:28:45.944459+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-10T18:26:49.014922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:49.014922+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040639739999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":179,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.949367+0000","last_change":"2024-09-10T18:27:57.056931+0000","last_active":"2024-09-10T18:28:45.949367+0000","last_peered":"2024-09-10T18:28:45.949367+0000","last_clean":"2024-09-10T18:28:45.949367+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:28:45.949367+0000","last_undegraded":"2024-09-10T18:28:45.949367+0000","last_fullsized":"2024-09-10T18:28:45.949367+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:57.056876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:27:57.056876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:10.364135+0000","scrub_duration":0.0022605429999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":273,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:48.985322+0000","last_change":"2024-09-10T18:28:08.161477+0000","last_active":"2024-09-10T18:28:48.985322+0000","last_peered":"2024-09-10T18:28:48.985322+0000","last_clean":"2024-09-10T18:28:48.985322+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:28:48.985322+0000","last_undegraded":"2024-09-10T18:28:48.985322+0000","last_fullsized":"2024-09-10T18:28:48.985322+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:28:08.161384+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:28:08.161384+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:31.405936+0000","scrub_duration":0.044251862000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":182,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952418+0000","last_change":"2024-09-10T18:27:52.101815+0000","last_active":"2024-09-10T18:28:45.952418+0000","last_peered":"2024-09-10T18:28:45.952418+0000","last_clean":"2024-09-10T18:28:45.952418+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:28:45.952418+0000","last_undegraded":"2024-09-10T18:28:45.952418+0000","last_fullsized":"2024-09-10T18:28:45.952418+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.101751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:02.011610+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.101751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:08.114132+0000","scrub_duration":0.00178638,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"199'5005","reported_seq":5032,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:50.389048+0000","last_change":"2024-09-10T18:27:53.034877+0000","last_active":"2024-09-10T18:28:50.389048+0000","last_peered":"2024-09-10T18:28:50.389048+0000","last_clean":"2024-09-10T18:28:50.389048+0000","last_became_active":"2024-09-10T18:27:53.034171+0000","last_became_peered":"2024-09-10T18:27:53.034171+0000","last_unstale":"2024-09-10T18:28:50.389048+0000","last_undegraded":"2024-09-10T18:28:50.389048+0000","last_fullsized":"2024-09-10T18:28:50.389048+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":5005,"log_dups_size":0,"ondisk_log_size":5005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:57.932544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328007680,"num_objects":5005,"num_object_clones":0,"num_object_copies":15015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":320320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":134,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.947839+0000","last_change":"2024-09-10T18:28:00.351507+0000","last_active":"2024-09-10T18:28:45.947839+0000","last_peered":"2024-09-10T18:28:45.947839+0000","last_clean":"2024-09-10T18:28:45.947839+0000","last_became_active":"2024-09-10T18:26:28.632849+0000","last_became_peered":"2024-09-10T18:26:28.632849+0000","last_unstale":"2024-09-10T18:28:45.947839+0000","last_undegraded":"2024-09-10T18:28:45.947839+0000","last_fullsized":"2024-09-10T18:28:45.947839+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:28:00.351448+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:28:00.351448+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:27.185208+0000","scrub_duration":0.0013475130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"123'15","reported_seq":206,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.946062+0000","last_change":"2024-09-10T18:27:14.653748+0000","last_active":"2024-09-10T18:28:45.946062+0000","last_peered":"2024-09-10T18:28:45.946062+0000","last_clean":"2024-09-10T18:28:45.946062+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:28:45.946062+0000","last_undegraded":"2024-09-10T18:28:45.946062+0000","last_fullsized":"2024-09-10T18:28:45.946062+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'15","last_scrub_stamp":"2024-09-10T18:27:14.653658+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:25:18.356218+0000","last_clean_scrub_stamp":"2024-09-10T18:27:14.653658+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062664110000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":179,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951119+0000","last_change":"2024-09-10T18:28:03.772302+0000","last_active":"2024-09-10T18:28:45.951119+0000","last_peered":"2024-09-10T18:28:45.951119+0000","last_clean":"2024-09-10T18:28:45.951119+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:28:45.951119+0000","last_undegraded":"2024-09-10T18:28:45.951119+0000","last_fullsized":"2024-09-10T18:28:45.951119+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:28:03.772241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:20.727151+0000","scrub_duration":0.0014939619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"146'10","reported_seq":270,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:48.979103+0000","last_change":"2024-09-10T18:26:47.858607+0000","last_active":"2024-09-10T18:28:48.979103+0000","last_peered":"2024-09-10T18:28:48.979103+0000","last_clean":"2024-09-10T18:28:48.979103+0000","last_became_active":"2024-09-10T18:26:27.639032+0000","last_became_peered":"2024-09-10T18:26:27.639032+0000","last_unstale":"2024-09-10T18:28:48.979103+0000","last_undegraded":"2024-09-10T18:28:48.979103+0000","last_fullsized":"2024-09-10T18:28:48.979103+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'10","last_scrub_stamp":"2024-09-10T18:26:47.858542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:26:47.858542+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001506688,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":173,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.948029+0000","last_change":"2024-09-10T18:26:17.657969+0000","last_active":"2024-09-10T18:28:45.948029+0000","last_peered":"2024-09-10T18:28:45.948029+0000","last_clean":"2024-09-10T18:28:45.948029+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:28:45.948029+0000","last_undegraded":"2024-09-10T18:28:45.948029+0000","last_fullsized":"2024-09-10T18:28:45.948029+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:17.657880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:26:17.657880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018451012999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"199'4949","reported_seq":4975,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:50.282181+0000","last_change":"2024-09-10T18:27:53.037694+0000","last_active":"2024-09-10T18:28:50.282181+0000","last_peered":"2024-09-10T18:28:50.282181+0000","last_clean":"2024-09-10T18:28:50.282181+0000","last_became_active":"2024-09-10T18:27:53.037210+0000","last_became_peered":"2024-09-10T18:27:53.037210+0000","last_unstale":"2024-09-10T18:28:50.282181+0000","last_undegraded":"2024-09-10T18:28:50.282181+0000","last_fullsized":"2024-09-10T18:28:50.282181+0000","mapping_epoch":180,"log_start":"189'400","ondisk_log_start":"189'400","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4549,"log_dups_size":0,"ondisk_log_size":4549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:54.052884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324337664,"num_objects":4949,"num_object_clones":0,"num_object_copies":14847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4949,"num_write_kb":316736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":134,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.948676+0000","last_change":"2024-09-10T18:27:53.479157+0000","last_active":"2024-09-10T18:28:45.948676+0000","last_peered":"2024-09-10T18:28:45.948676+0000","last_clean":"2024-09-10T18:28:45.948676+0000","last_became_active":"2024-09-10T18:26:28.635542+0000","last_became_peered":"2024-09-10T18:26:28.635542+0000","last_unstale":"2024-09-10T18:28:45.948676+0000","last_undegraded":"2024-09-10T18:28:45.948676+0000","last_fullsized":"2024-09-10T18:28:45.948676+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:27:53.479083+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:27:53.479083+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:11.027081+0000","scrub_duration":0.0017291470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":213,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.948586+0000","last_change":"2024-09-10T18:27:18.798491+0000","last_active":"2024-09-10T18:28:45.948586+0000","last_peered":"2024-09-10T18:28:45.948586+0000","last_clean":"2024-09-10T18:28:45.948586+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:28:45.948586+0000","last_undegraded":"2024-09-10T18:28:45.948586+0000","last_fullsized":"2024-09-10T18:28:45.948586+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-10T18:27:18.798402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:27:18.798402+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.020618556999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":178,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.945138+0000","last_change":"2024-09-10T18:28:14.418351+0000","last_active":"2024-09-10T18:28:45.945138+0000","last_peered":"2024-09-10T18:28:45.945138+0000","last_clean":"2024-09-10T18:28:45.945138+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:28:45.945138+0000","last_undegraded":"2024-09-10T18:28:45.945138+0000","last_fullsized":"2024-09-10T18:28:45.945138+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:14.418260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:14.418260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:19.624601+0000","scrub_duration":0.0049147540000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":124,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.948298+0000","last_change":"2024-09-10T18:28:14.399029+0000","last_active":"2024-09-10T18:28:45.948298+0000","last_peered":"2024-09-10T18:28:45.948298+0000","last_clean":"2024-09-10T18:28:45.948298+0000","last_became_active":"2024-09-10T18:26:27.642063+0000","last_became_peered":"2024-09-10T18:26:27.642063+0000","last_unstale":"2024-09-10T18:28:45.948298+0000","last_undegraded":"2024-09-10T18:28:45.948298+0000","last_fullsized":"2024-09-10T18:28:45.948298+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:14.398929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:14.398929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:38.005890+0000","scrub_duration":0.0024148870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":179,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.947941+0000","last_change":"2024-09-10T18:28:15.471450+0000","last_active":"2024-09-10T18:28:45.947941+0000","last_peered":"2024-09-10T18:28:45.947941+0000","last_clean":"2024-09-10T18:28:45.947941+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:28:45.947941+0000","last_undegraded":"2024-09-10T18:28:45.947941+0000","last_fullsized":"2024-09-10T18:28:45.947941+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:15.471241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:28:15.471241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:19.536572+0000","scrub_duration":0.0086976859999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"199'4807","reported_seq":4834,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:49.746112+0000","last_change":"2024-09-10T18:27:53.062042+0000","last_active":"2024-09-10T18:28:49.746112+0000","last_peered":"2024-09-10T18:28:49.746112+0000","last_clean":"2024-09-10T18:28:49.746112+0000","last_became_active":"2024-09-10T18:27:53.061113+0000","last_became_peered":"2024-09-10T18:27:53.061113+0000","last_unstale":"2024-09-10T18:28:49.746112+0000","last_undegraded":"2024-09-10T18:28:49.746112+0000","last_fullsized":"2024-09-10T18:28:49.746112+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4807,"log_dups_size":0,"ondisk_log_size":4807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:58.198558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315031552,"num_objects":4807,"num_object_clones":0,"num_object_copies":14421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4807,"num_write_kb":307648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":80,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951719+0000","last_change":"2024-09-10T18:27:54.717632+0000","last_active":"2024-09-10T18:28:45.951719+0000","last_peered":"2024-09-10T18:28:45.951719+0000","last_clean":"2024-09-10T18:28:45.951719+0000","last_became_active":"2024-09-10T18:26:18.094747+0000","last_became_peered":"2024-09-10T18:26:18.094747+0000","last_unstale":"2024-09-10T18:28:45.951719+0000","last_undegraded":"2024-09-10T18:28:45.951719+0000","last_fullsized":"2024-09-10T18:28:45.951719+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:27:54.717560+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:27:54.717560+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:15.547845+0000","scrub_duration":0.0015444320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"90'29","reported_seq":230,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952118+0000","last_change":"2024-09-10T18:26:19.509647+0000","last_active":"2024-09-10T18:28:45.952118+0000","last_peered":"2024-09-10T18:28:45.952118+0000","last_clean":"2024-09-10T18:28:45.952118+0000","last_became_active":"2024-09-10T18:23:10.067049+0000","last_became_peered":"2024-09-10T18:23:10.067049+0000","last_unstale":"2024-09-10T18:28:45.952118+0000","last_undegraded":"2024-09-10T18:28:45.952118+0000","last_fullsized":"2024-09-10T18:28:45.952118+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'29","last_scrub_stamp":"2024-09-10T18:26:19.509580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:19.509580+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063679319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":185,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.944578+0000","last_change":"2024-09-10T18:26:47.093062+0000","last_active":"2024-09-10T18:28:45.944578+0000","last_peered":"2024-09-10T18:28:45.944578+0000","last_clean":"2024-09-10T18:28:45.944578+0000","last_became_active":"2024-09-10T18:26:27.648690+0000","last_became_peered":"2024-09-10T18:26:27.648690+0000","last_unstale":"2024-09-10T18:28:45.944578+0000","last_undegraded":"2024-09-10T18:28:45.944578+0000","last_fullsized":"2024-09-10T18:28:45.944578+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:47.093004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:26:47.093004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014500660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"62'2","reported_seq":193,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952324+0000","last_change":"2024-09-10T18:26:54.562292+0000","last_active":"2024-09-10T18:28:45.952324+0000","last_peered":"2024-09-10T18:28:45.952324+0000","last_clean":"2024-09-10T18:28:45.952324+0000","last_became_active":"2024-09-10T18:26:27.647576+0000","last_became_peered":"2024-09-10T18:26:27.647576+0000","last_unstale":"2024-09-10T18:28:45.952324+0000","last_undegraded":"2024-09-10T18:28:45.952324+0000","last_fullsized":"2024-09-10T18:28:45.952324+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-10T18:26:54.562210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:26:54.562210+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023324999999999999,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":175,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.949104+0000","last_change":"2024-09-10T18:27:12.683381+0000","last_active":"2024-09-10T18:28:45.949104+0000","last_peered":"2024-09-10T18:28:45.949104+0000","last_clean":"2024-09-10T18:28:45.949104+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:28:45.949104+0000","last_undegraded":"2024-09-10T18:28:45.949104+0000","last_fullsized":"2024-09-10T18:28:45.949104+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:12.683294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:12.683294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042722790000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":174,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952559+0000","last_change":"2024-09-10T18:28:07.121517+0000","last_active":"2024-09-10T18:28:45.952559+0000","last_peered":"2024-09-10T18:28:45.952559+0000","last_clean":"2024-09-10T18:28:45.952559+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:28:45.952559+0000","last_undegraded":"2024-09-10T18:28:45.952559+0000","last_fullsized":"2024-09-10T18:28:45.952559+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:07.121433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:28:07.121433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:34.298348+0000","scrub_duration":0.0044980100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"199'4887","reported_seq":4914,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:50.272716+0000","last_change":"2024-09-10T18:27:53.037955+0000","last_active":"2024-09-10T18:28:50.272716+0000","last_peered":"2024-09-10T18:28:50.272716+0000","last_clean":"2024-09-10T18:28:50.272716+0000","last_became_active":"2024-09-10T18:27:53.037246+0000","last_became_peered":"2024-09-10T18:27:53.037246+0000","last_unstale":"2024-09-10T18:28:50.272716+0000","last_undegraded":"2024-09-10T18:28:50.272716+0000","last_fullsized":"2024-09-10T18:28:50.272716+0000","mapping_epoch":180,"log_start":"189'400","ondisk_log_start":"189'400","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4487,"log_dups_size":0,"ondisk_log_size":4487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:20.497361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320274432,"num_objects":4887,"num_object_clones":0,"num_object_copies":14661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4887,"num_write_kb":312768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":226,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951895+0000","last_change":"2024-09-10T18:27:59.226683+0000","last_active":"2024-09-10T18:28:45.951895+0000","last_peered":"2024-09-10T18:28:45.951895+0000","last_clean":"2024-09-10T18:28:45.951895+0000","last_became_active":"2024-09-10T18:26:18.095568+0000","last_became_peered":"2024-09-10T18:26:18.095568+0000","last_unstale":"2024-09-10T18:28:45.951895+0000","last_undegraded":"2024-09-10T18:28:45.951895+0000","last_fullsized":"2024-09-10T18:28:45.951895+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:27:59.226607+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:27:59.226607+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:22.629088+0000","scrub_duration":0.0014964799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":203,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.948956+0000","last_change":"2024-09-10T18:26:56.714007+0000","last_active":"2024-09-10T18:28:45.948956+0000","last_peered":"2024-09-10T18:28:45.948956+0000","last_clean":"2024-09-10T18:28:45.948956+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:28:45.948956+0000","last_undegraded":"2024-09-10T18:28:45.948956+0000","last_fullsized":"2024-09-10T18:28:45.948956+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-10T18:26:56.713924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:56.713924+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018614987,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":195,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.946351+0000","last_change":"2024-09-10T18:28:16.493397+0000","last_active":"2024-09-10T18:28:45.946351+0000","last_peered":"2024-09-10T18:28:45.946351+0000","last_clean":"2024-09-10T18:28:45.946351+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:28:45.946351+0000","last_undegraded":"2024-09-10T18:28:45.946351+0000","last_fullsized":"2024-09-10T18:28:45.946351+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:16.493304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:16.493304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:43.855805+0000","scrub_duration":0.0045544119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":197,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.956483+0000","last_change":"2024-09-10T18:27:18.122094+0000","last_active":"2024-09-10T18:28:45.956483+0000","last_peered":"2024-09-10T18:28:45.956483+0000","last_clean":"2024-09-10T18:28:45.956483+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:28:45.956483+0000","last_undegraded":"2024-09-10T18:28:45.956483+0000","last_fullsized":"2024-09-10T18:28:45.956483+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:27:18.121985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:27:18.121985+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018606608,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":184,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.956049+0000","last_change":"2024-09-10T18:28:09.069190+0000","last_active":"2024-09-10T18:28:45.956049+0000","last_peered":"2024-09-10T18:28:45.956049+0000","last_clean":"2024-09-10T18:28:45.956049+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:28:45.956049+0000","last_undegraded":"2024-09-10T18:28:45.956049+0000","last_fullsized":"2024-09-10T18:28:45.956049+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:09.069093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:28:09.069093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:26.067294+0000","scrub_duration":0.0055535080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":182,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.953300+0000","last_change":"2024-09-10T18:27:55.152565+0000","last_active":"2024-09-10T18:28:45.953300+0000","last_peered":"2024-09-10T18:28:45.953300+0000","last_clean":"2024-09-10T18:28:45.953300+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:28:45.953300+0000","last_undegraded":"2024-09-10T18:28:45.953300+0000","last_fullsized":"2024-09-10T18:28:45.953300+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:55.152501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:55.152501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:16.874950+0000","scrub_duration":0.001407609,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"199'4912","reported_seq":4939,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:49.938951+0000","last_change":"2024-09-10T18:27:53.034603+0000","last_active":"2024-09-10T18:28:49.938951+0000","last_peered":"2024-09-10T18:28:49.938951+0000","last_clean":"2024-09-10T18:28:49.938951+0000","last_became_active":"2024-09-10T18:27:53.034315+0000","last_became_peered":"2024-09-10T18:27:53.034315+0000","last_unstale":"2024-09-10T18:28:49.938951+0000","last_undegraded":"2024-09-10T18:28:49.938951+0000","last_fullsized":"2024-09-10T18:28:49.938951+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4912,"log_dups_size":0,"ondisk_log_size":4912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:03.080369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321912832,"num_objects":4912,"num_object_clones":0,"num_object_copies":14736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4912,"num_write_kb":314368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":95,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.946775+0000","last_change":"2024-09-10T18:27:57.493055+0000","last_active":"2024-09-10T18:28:45.946775+0000","last_peered":"2024-09-10T18:28:45.946775+0000","last_clean":"2024-09-10T18:28:45.946775+0000","last_became_active":"2024-09-10T18:26:28.641791+0000","last_became_peered":"2024-09-10T18:26:28.641791+0000","last_unstale":"2024-09-10T18:28:45.946775+0000","last_undegraded":"2024-09-10T18:28:45.946775+0000","last_fullsized":"2024-09-10T18:28:45.946775+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:27:57.492998+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:27:57.492998+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:01.205814+0000","scrub_duration":0.0019993160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":195,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.949232+0000","last_change":"2024-09-10T18:27:50.951815+0000","last_active":"2024-09-10T18:28:45.949232+0000","last_peered":"2024-09-10T18:28:45.949232+0000","last_clean":"2024-09-10T18:28:45.949232+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:28:45.949232+0000","last_undegraded":"2024-09-10T18:28:45.949232+0000","last_fullsized":"2024-09-10T18:28:45.949232+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:27:50.951749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:27:50.951749+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:57.289936+0000","scrub_duration":0.003075983,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":179,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.953515+0000","last_change":"2024-09-10T18:27:15.809458+0000","last_active":"2024-09-10T18:28:45.953515+0000","last_peered":"2024-09-10T18:28:45.953515+0000","last_clean":"2024-09-10T18:28:45.953515+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:28:45.953515+0000","last_undegraded":"2024-09-10T18:28:45.953515+0000","last_fullsized":"2024-09-10T18:28:45.953515+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:15.809380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:27:15.809380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086899360000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":627,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:48.980347+0000","last_change":"2024-09-10T18:28:04.343237+0000","last_active":"2024-09-10T18:28:48.980347+0000","last_peered":"2024-09-10T18:28:48.980347+0000","last_clean":"2024-09-10T18:28:48.980347+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:28:48.980347+0000","last_undegraded":"2024-09-10T18:28:48.980347+0000","last_fullsized":"2024-09-10T18:28:48.980347+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:28:04.343138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:04.343138+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:08.341406+0000","scrub_duration":0.0024865109999999998,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":431,"num_read_kb":426,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":178,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951131+0000","last_change":"2024-09-10T18:28:05.771485+0000","last_active":"2024-09-10T18:28:45.951131+0000","last_peered":"2024-09-10T18:28:45.951131+0000","last_clean":"2024-09-10T18:28:45.951131+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:28:45.951131+0000","last_undegraded":"2024-09-10T18:28:45.951131+0000","last_fullsized":"2024-09-10T18:28:45.951131+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:05.771415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.771415+0000","last_clean_scrub_stamp":"2024-09-10T18:28:05.771415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:23.559543+0000","scrub_duration":0.001873523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":172,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.944815+0000","last_change":"2024-09-10T18:27:56.527307+0000","last_active":"2024-09-10T18:28:45.944815+0000","last_peered":"2024-09-10T18:28:45.944815+0000","last_clean":"2024-09-10T18:28:45.944815+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:28:45.944815+0000","last_undegraded":"2024-09-10T18:28:45.944815+0000","last_fullsized":"2024-09-10T18:28:45.944815+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:56.527234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:56.527234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:19.892724+0000","scrub_duration":0.0018055840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"199'5079","reported_seq":5105,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:50.643970+0000","last_change":"2024-09-10T18:27:53.035571+0000","last_active":"2024-09-10T18:28:50.643970+0000","last_peered":"2024-09-10T18:28:50.643970+0000","last_clean":"2024-09-10T18:28:50.643970+0000","last_became_active":"2024-09-10T18:27:53.035261+0000","last_became_peered":"2024-09-10T18:27:53.035261+0000","last_unstale":"2024-09-10T18:28:50.643970+0000","last_undegraded":"2024-09-10T18:28:50.643970+0000","last_fullsized":"2024-09-10T18:28:50.643970+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:11.723017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332857344,"num_objects":5079,"num_object_clones":0,"num_object_copies":15237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5079,"num_write_kb":325056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":143,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.947990+0000","last_change":"2024-09-10T18:26:55.263707+0000","last_active":"2024-09-10T18:28:45.947990+0000","last_peered":"2024-09-10T18:28:45.947990+0000","last_clean":"2024-09-10T18:28:45.947990+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:28:45.947990+0000","last_undegraded":"2024-09-10T18:28:45.947990+0000","last_fullsized":"2024-09-10T18:28:45.947990+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-10T18:26:55.263623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:55.263623+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003225992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":187,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952125+0000","last_change":"2024-09-10T18:26:55.420188+0000","last_active":"2024-09-10T18:28:45.952125+0000","last_peered":"2024-09-10T18:28:45.952125+0000","last_clean":"2024-09-10T18:28:45.952125+0000","last_became_active":"2024-09-10T18:26:27.639129+0000","last_became_peered":"2024-09-10T18:26:27.639129+0000","last_unstale":"2024-09-10T18:28:45.952125+0000","last_undegraded":"2024-09-10T18:28:45.952125+0000","last_fullsized":"2024-09-10T18:28:45.952125+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:55.420103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:26:55.420103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.055370536999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":228,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952094+0000","last_change":"2024-09-10T18:28:03.297772+0000","last_active":"2024-09-10T18:28:45.952094+0000","last_peered":"2024-09-10T18:28:45.952094+0000","last_clean":"2024-09-10T18:28:45.952094+0000","last_became_active":"2024-09-10T18:26:27.646658+0000","last_became_peered":"2024-09-10T18:26:27.646658+0000","last_unstale":"2024-09-10T18:28:45.952094+0000","last_undegraded":"2024-09-10T18:28:45.952094+0000","last_fullsized":"2024-09-10T18:28:45.952094+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:28:03.297713+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:28:03.297713+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:05.130955+0000","scrub_duration":0.0015114589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":184,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.953870+0000","last_change":"2024-09-10T18:28:11.367715+0000","last_active":"2024-09-10T18:28:45.953870+0000","last_peered":"2024-09-10T18:28:45.953870+0000","last_clean":"2024-09-10T18:28:45.953870+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:28:45.953870+0000","last_undegraded":"2024-09-10T18:28:45.953870+0000","last_fullsized":"2024-09-10T18:28:45.953870+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:11.367574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:11.367574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:16.471529+0000","scrub_duration":0.0025623360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":176,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951056+0000","last_change":"2024-09-10T18:28:09.749710+0000","last_active":"2024-09-10T18:28:45.951056+0000","last_peered":"2024-09-10T18:28:45.951056+0000","last_clean":"2024-09-10T18:28:45.951056+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:28:45.951056+0000","last_undegraded":"2024-09-10T18:28:45.951056+0000","last_fullsized":"2024-09-10T18:28:45.951056+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:28:09.749627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:27.848036+0000","scrub_duration":0.0063185680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"199'4798","reported_seq":4825,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:49.740805+0000","last_change":"2024-09-10T18:27:53.034084+0000","last_active":"2024-09-10T18:28:49.740805+0000","last_peered":"2024-09-10T18:28:49.740805+0000","last_clean":"2024-09-10T18:28:49.740805+0000","last_became_active":"2024-09-10T18:27:53.033757+0000","last_became_peered":"2024-09-10T18:27:53.033757+0000","last_unstale":"2024-09-10T18:28:49.740805+0000","last_undegraded":"2024-09-10T18:28:49.740805+0000","last_fullsized":"2024-09-10T18:28:49.740805+0000","mapping_epoch":180,"log_start":"189'200","ondisk_log_start":"189'200","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4598,"log_dups_size":0,"ondisk_log_size":4598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:20.241697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314441728,"num_objects":4798,"num_object_clones":0,"num_object_copies":14394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4798,"num_write_kb":307072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":205,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.945262+0000","last_change":"2024-09-10T18:27:57.211522+0000","last_active":"2024-09-10T18:28:45.945262+0000","last_peered":"2024-09-10T18:28:45.945262+0000","last_clean":"2024-09-10T18:28:45.945262+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:28:45.945262+0000","last_undegraded":"2024-09-10T18:28:45.945262+0000","last_fullsized":"2024-09-10T18:28:45.945262+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:27:57.211451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:27:57.211451+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:07.130222+0000","scrub_duration":0.0053779049999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":178,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.944659+0000","last_change":"2024-09-10T18:28:06.191180+0000","last_active":"2024-09-10T18:28:45.944659+0000","last_peered":"2024-09-10T18:28:45.944659+0000","last_clean":"2024-09-10T18:28:45.944659+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:28:45.944659+0000","last_undegraded":"2024-09-10T18:28:45.944659+0000","last_fullsized":"2024-09-10T18:28:45.944659+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:06.191117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:06.191117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:17.909998+0000","scrub_duration":0.0015003600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":182,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.950931+0000","last_change":"2024-09-10T18:26:58.532067+0000","last_active":"2024-09-10T18:28:45.950931+0000","last_peered":"2024-09-10T18:28:45.950931+0000","last_clean":"2024-09-10T18:28:45.950931+0000","last_became_active":"2024-09-10T18:26:27.645885+0000","last_became_peered":"2024-09-10T18:26:27.645885+0000","last_unstale":"2024-09-10T18:28:45.950931+0000","last_undegraded":"2024-09-10T18:28:45.950931+0000","last_fullsized":"2024-09-10T18:28:45.950931+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:58.532019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:26:58.532019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001143645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":181,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951205+0000","last_change":"2024-09-10T18:26:58.414225+0000","last_active":"2024-09-10T18:28:45.951205+0000","last_peered":"2024-09-10T18:28:45.951205+0000","last_clean":"2024-09-10T18:28:45.951205+0000","last_became_active":"2024-09-10T18:26:27.632071+0000","last_became_peered":"2024-09-10T18:26:27.632071+0000","last_unstale":"2024-09-10T18:28:45.951205+0000","last_undegraded":"2024-09-10T18:28:45.951205+0000","last_fullsized":"2024-09-10T18:28:45.951205+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:58.414024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:26:58.414024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.020948053000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"199'4866","reported_seq":4893,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:49.948569+0000","last_change":"2024-09-10T18:27:53.035721+0000","last_active":"2024-09-10T18:28:49.948569+0000","last_peered":"2024-09-10T18:28:49.948569+0000","last_clean":"2024-09-10T18:28:49.948569+0000","last_became_active":"2024-09-10T18:27:53.035419+0000","last_became_peered":"2024-09-10T18:27:53.035419+0000","last_unstale":"2024-09-10T18:28:49.948569+0000","last_undegraded":"2024-09-10T18:28:49.948569+0000","last_fullsized":"2024-09-10T18:28:49.948569+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:20.993490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318898176,"num_objects":4866,"num_object_clones":0,"num_object_copies":14598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4866,"num_write_kb":311424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":87,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952422+0000","last_change":"2024-09-10T18:27:55.747977+0000","last_active":"2024-09-10T18:28:45.952422+0000","last_peered":"2024-09-10T18:28:45.952422+0000","last_clean":"2024-09-10T18:28:45.952422+0000","last_became_active":"2024-09-10T18:26:18.095088+0000","last_became_peered":"2024-09-10T18:26:18.095088+0000","last_unstale":"2024-09-10T18:28:45.952422+0000","last_undegraded":"2024-09-10T18:28:45.952422+0000","last_fullsized":"2024-09-10T18:28:45.952422+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:27:55.747909+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:27:55.747909+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:00.862668+0000","scrub_duration":0.0026038910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":204,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.945150+0000","last_change":"2024-09-10T18:26:16.736677+0000","last_active":"2024-09-10T18:28:45.945150+0000","last_peered":"2024-09-10T18:28:45.945150+0000","last_clean":"2024-09-10T18:28:45.945150+0000","last_became_active":"2024-09-10T18:21:33.611203+0000","last_became_peered":"2024-09-10T18:21:33.611203+0000","last_unstale":"2024-09-10T18:28:45.945150+0000","last_undegraded":"2024-09-10T18:28:45.945150+0000","last_fullsized":"2024-09-10T18:28:45.945150+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-10T18:26:16.736598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:16.736598+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039285580000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":132,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.948589+0000","last_change":"2024-09-10T18:27:17.554261+0000","last_active":"2024-09-10T18:28:45.948589+0000","last_peered":"2024-09-10T18:28:45.948589+0000","last_clean":"2024-09-10T18:28:45.948589+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:28:45.948589+0000","last_undegraded":"2024-09-10T18:28:45.948589+0000","last_fullsized":"2024-09-10T18:28:45.948589+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:27:17.554184+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089718639999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":174,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.944899+0000","last_change":"2024-09-10T18:27:12.675527+0000","last_active":"2024-09-10T18:28:45.944899+0000","last_peered":"2024-09-10T18:28:45.944899+0000","last_clean":"2024-09-10T18:28:45.944899+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:28:45.944899+0000","last_undegraded":"2024-09-10T18:28:45.944899+0000","last_fullsized":"2024-09-10T18:28:45.944899+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:12.675265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:12.675265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.096140301999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":192,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.956087+0000","last_change":"2024-09-10T18:28:02.975548+0000","last_active":"2024-09-10T18:28:45.956087+0000","last_peered":"2024-09-10T18:28:45.956087+0000","last_clean":"2024-09-10T18:28:45.956087+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:28:45.956087+0000","last_undegraded":"2024-09-10T18:28:45.956087+0000","last_fullsized":"2024-09-10T18:28:45.956087+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:02.975493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:28:02.975493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:22.369213+0000","scrub_duration":0.019176189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"199'4896","reported_seq":4923,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:50.315548+0000","last_change":"2024-09-10T18:27:53.037543+0000","last_active":"2024-09-10T18:28:50.315548+0000","last_peered":"2024-09-10T18:28:50.315548+0000","last_clean":"2024-09-10T18:28:50.315548+0000","last_became_active":"2024-09-10T18:27:53.037235+0000","last_became_peered":"2024-09-10T18:27:53.037235+0000","last_unstale":"2024-09-10T18:28:50.315548+0000","last_undegraded":"2024-09-10T18:28:50.315548+0000","last_fullsized":"2024-09-10T18:28:50.315548+0000","mapping_epoch":180,"log_start":"189'400","ondisk_log_start":"189'400","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4496,"log_dups_size":0,"ondisk_log_size":4496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:17.490638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320864256,"num_objects":4896,"num_object_clones":0,"num_object_copies":14688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4896,"num_write_kb":313344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":226,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952584+0000","last_change":"2024-09-10T18:27:58.257714+0000","last_active":"2024-09-10T18:28:45.952584+0000","last_peered":"2024-09-10T18:28:45.952584+0000","last_clean":"2024-09-10T18:28:45.952584+0000","last_became_active":"2024-09-10T18:26:18.092014+0000","last_became_peered":"2024-09-10T18:26:18.092014+0000","last_unstale":"2024-09-10T18:28:45.952584+0000","last_undegraded":"2024-09-10T18:28:45.952584+0000","last_fullsized":"2024-09-10T18:28:45.952584+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:27:58.257644+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:27:58.257644+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:14.849409+0000","scrub_duration":0.0015187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"92'15","reported_seq":203,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.944948+0000","last_change":"2024-09-10T18:27:12.992305+0000","last_active":"2024-09-10T18:28:45.944948+0000","last_peered":"2024-09-10T18:28:45.944948+0000","last_clean":"2024-09-10T18:28:45.944948+0000","last_became_active":"2024-09-10T18:26:27.649406+0000","last_became_peered":"2024-09-10T18:26:27.649406+0000","last_unstale":"2024-09-10T18:28:45.944948+0000","last_undegraded":"2024-09-10T18:28:45.944948+0000","last_fullsized":"2024-09-10T18:28:45.944948+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'15","last_scrub_stamp":"2024-09-10T18:27:12.992221+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:27:12.992221+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020362739999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":180,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951810+0000","last_change":"2024-09-10T18:28:14.145492+0000","last_active":"2024-09-10T18:28:45.951810+0000","last_peered":"2024-09-10T18:28:45.951810+0000","last_clean":"2024-09-10T18:28:45.951810+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:28:45.951810+0000","last_undegraded":"2024-09-10T18:28:45.951810+0000","last_fullsized":"2024-09-10T18:28:45.951810+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:14.145364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:14.145364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:31.467386+0000","scrub_duration":0.003651989,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"189'9","reported_seq":291,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:48.983414+0000","last_change":"2024-09-10T18:28:11.017989+0000","last_active":"2024-09-10T18:28:48.983414+0000","last_peered":"2024-09-10T18:28:48.983414+0000","last_clean":"2024-09-10T18:28:48.983414+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:28:48.983414+0000","last_undegraded":"2024-09-10T18:28:48.983414+0000","last_fullsized":"2024-09-10T18:28:48.983414+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'9","last_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:28:11.017903+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:36.538530+0000","scrub_duration":0.007246756,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":175,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.945051+0000","last_change":"2024-09-10T18:28:16.088804+0000","last_active":"2024-09-10T18:28:45.945051+0000","last_peered":"2024-09-10T18:28:45.945051+0000","last_clean":"2024-09-10T18:28:45.945051+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:28:45.945051+0000","last_undegraded":"2024-09-10T18:28:45.945051+0000","last_fullsized":"2024-09-10T18:28:45.945051+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_clean_scrub_stamp":"2024-09-10T18:28:16.088718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:33.515907+0000","scrub_duration":0.0030710199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"199'4989","reported_seq":5016,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:49.930888+0000","last_change":"2024-09-10T18:27:53.037684+0000","last_active":"2024-09-10T18:28:49.930888+0000","last_peered":"2024-09-10T18:28:49.930888+0000","last_clean":"2024-09-10T18:28:49.930888+0000","last_became_active":"2024-09-10T18:27:53.037361+0000","last_became_peered":"2024-09-10T18:27:53.037361+0000","last_unstale":"2024-09-10T18:28:49.930888+0000","last_undegraded":"2024-09-10T18:28:49.930888+0000","last_fullsized":"2024-09-10T18:28:49.930888+0000","mapping_epoch":180,"log_start":"189'100","ondisk_log_start":"189'100","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4889,"log_dups_size":0,"ondisk_log_size":4889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:12.821097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326959104,"num_objects":4989,"num_object_clones":0,"num_object_copies":14967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4989,"num_write_kb":319296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":183,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.948503+0000","last_change":"2024-09-10T18:26:09.669513+0000","last_active":"2024-09-10T18:28:45.948503+0000","last_peered":"2024-09-10T18:28:45.948503+0000","last_clean":"2024-09-10T18:28:45.948503+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:28:45.948503+0000","last_undegraded":"2024-09-10T18:28:45.948503+0000","last_fullsized":"2024-09-10T18:28:45.948503+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:26:09.669423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:09.669423+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010591546,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"58'1","reported_seq":195,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952504+0000","last_change":"2024-09-10T18:26:53.613266+0000","last_active":"2024-09-10T18:28:45.952504+0000","last_peered":"2024-09-10T18:28:45.952504+0000","last_clean":"2024-09-10T18:28:45.952504+0000","last_became_active":"2024-09-10T18:26:27.644997+0000","last_became_peered":"2024-09-10T18:26:27.644997+0000","last_unstale":"2024-09-10T18:28:45.952504+0000","last_undegraded":"2024-09-10T18:28:45.952504+0000","last_fullsized":"2024-09-10T18:28:45.952504+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-10T18:26:53.613208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:26:53.613208+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042600270000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":190,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.956801+0000","last_change":"2024-09-10T18:27:52.027471+0000","last_active":"2024-09-10T18:28:45.956801+0000","last_peered":"2024-09-10T18:28:45.956801+0000","last_clean":"2024-09-10T18:28:45.956801+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:28:45.956801+0000","last_undegraded":"2024-09-10T18:28:45.956801+0000","last_fullsized":"2024-09-10T18:28:45.956801+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.027381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.027381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:04.516674+0000","scrub_duration":0.0050644849999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":173,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.953453+0000","last_change":"2024-09-10T18:27:51.963263+0000","last_active":"2024-09-10T18:28:45.953453+0000","last_peered":"2024-09-10T18:28:45.953453+0000","last_clean":"2024-09-10T18:28:45.953453+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:28:45.953453+0000","last_undegraded":"2024-09-10T18:28:45.953453+0000","last_fullsized":"2024-09-10T18:28:45.953453+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:51.963197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:51.963197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:53.251200+0000","scrub_duration":0.001346552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"199'4930","reported_seq":4957,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:50.356125+0000","last_change":"2024-09-10T18:27:53.060239+0000","last_active":"2024-09-10T18:28:50.356125+0000","last_peered":"2024-09-10T18:28:50.356125+0000","last_clean":"2024-09-10T18:28:50.356125+0000","last_became_active":"2024-09-10T18:27:53.059866+0000","last_became_peered":"2024-09-10T18:27:53.059866+0000","last_unstale":"2024-09-10T18:28:50.356125+0000","last_undegraded":"2024-09-10T18:28:50.356125+0000","last_fullsized":"2024-09-10T18:28:50.356125+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:57.273048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323092480,"num_objects":4930,"num_object_clones":0,"num_object_copies":14790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4930,"num_write_kb":315520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"86'28","reported_seq":195,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.948371+0000","last_change":"2024-09-10T18:27:12.513379+0000","last_active":"2024-09-10T18:28:45.948371+0000","last_peered":"2024-09-10T18:28:45.948371+0000","last_clean":"2024-09-10T18:28:45.948371+0000","last_became_active":"2024-09-10T18:26:27.637277+0000","last_became_peered":"2024-09-10T18:26:27.637277+0000","last_unstale":"2024-09-10T18:28:45.948371+0000","last_undegraded":"2024-09-10T18:28:45.948371+0000","last_fullsized":"2024-09-10T18:28:45.948371+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'28","last_scrub_stamp":"2024-09-10T18:27:12.513323+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:27:12.513323+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052920959999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":195,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.953489+0000","last_change":"2024-09-10T18:28:13.368579+0000","last_active":"2024-09-10T18:28:45.953489+0000","last_peered":"2024-09-10T18:28:45.953489+0000","last_clean":"2024-09-10T18:28:45.953489+0000","last_became_active":"2024-09-10T18:26:27.638408+0000","last_became_peered":"2024-09-10T18:26:27.638408+0000","last_unstale":"2024-09-10T18:28:45.953489+0000","last_undegraded":"2024-09-10T18:28:45.953489+0000","last_fullsized":"2024-09-10T18:28:45.953489+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:13.368506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:13.368506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:26.991208+0000","scrub_duration":0.0025418530000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":194,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.955566+0000","last_change":"2024-09-10T18:28:10.038296+0000","last_active":"2024-09-10T18:28:45.955566+0000","last_peered":"2024-09-10T18:28:45.955566+0000","last_clean":"2024-09-10T18:28:45.955566+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:28:45.955566+0000","last_undegraded":"2024-09-10T18:28:45.955566+0000","last_fullsized":"2024-09-10T18:28:45.955566+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:10.038205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:10.038205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:26.623064+0000","scrub_duration":0.0032197340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":185,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952026+0000","last_change":"2024-09-10T18:28:08.336391+0000","last_active":"2024-09-10T18:28:45.952026+0000","last_peered":"2024-09-10T18:28:45.952026+0000","last_clean":"2024-09-10T18:28:45.952026+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:28:45.952026+0000","last_undegraded":"2024-09-10T18:28:45.952026+0000","last_fullsized":"2024-09-10T18:28:45.952026+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:08.336308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:28:08.336308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:08.529256+0000","scrub_duration":0.0027473250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"199'5008","reported_seq":5034,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:50.655593+0000","last_change":"2024-09-10T18:27:53.061218+0000","last_active":"2024-09-10T18:28:50.655593+0000","last_peered":"2024-09-10T18:28:50.655593+0000","last_clean":"2024-09-10T18:28:50.655593+0000","last_became_active":"2024-09-10T18:27:53.060902+0000","last_became_peered":"2024-09-10T18:27:53.060902+0000","last_unstale":"2024-09-10T18:28:50.655593+0000","last_undegraded":"2024-09-10T18:28:50.655593+0000","last_fullsized":"2024-09-10T18:28:50.655593+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:11.224263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328204288,"num_objects":5008,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5008,"num_write_kb":320512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":202,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.944332+0000","last_change":"2024-09-10T18:26:06.804053+0000","last_active":"2024-09-10T18:28:45.944332+0000","last_peered":"2024-09-10T18:28:45.944332+0000","last_clean":"2024-09-10T18:28:45.944332+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:28:45.944332+0000","last_undegraded":"2024-09-10T18:28:45.944332+0000","last_fullsized":"2024-09-10T18:28:45.944332+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:26:06.804001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:06.804001+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013860801000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":188,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.953672+0000","last_change":"2024-09-10T18:28:09.316857+0000","last_active":"2024-09-10T18:28:45.953672+0000","last_peered":"2024-09-10T18:28:45.953672+0000","last_clean":"2024-09-10T18:28:45.953672+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:28:45.953672+0000","last_undegraded":"2024-09-10T18:28:45.953672+0000","last_fullsized":"2024-09-10T18:28:45.953672+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:09.316773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:09.316773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:11.869771+0000","scrub_duration":0.002163477,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":194,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.955569+0000","last_change":"2024-09-10T18:28:12.038263+0000","last_active":"2024-09-10T18:28:45.955569+0000","last_peered":"2024-09-10T18:28:45.955569+0000","last_clean":"2024-09-10T18:28:45.955569+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:28:45.955569+0000","last_undegraded":"2024-09-10T18:28:45.955569+0000","last_fullsized":"2024-09-10T18:28:45.955569+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:12.038177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:12.038177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:28.361971+0000","scrub_duration":0.0042416900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":164,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951309+0000","last_change":"2024-09-10T18:26:12.014835+0000","last_active":"2024-09-10T18:28:45.951309+0000","last_peered":"2024-09-10T18:28:45.951309+0000","last_clean":"2024-09-10T18:28:45.951309+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:28:45.951309+0000","last_undegraded":"2024-09-10T18:28:45.951309+0000","last_fullsized":"2024-09-10T18:28:45.951309+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:12.014757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:12.014757+0000","last_clean_scrub_stamp":"2024-09-10T18:26:12.014757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082019740000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"199'5095","reported_seq":5122,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:49.769340+0000","last_change":"2024-09-10T18:27:53.032783+0000","last_active":"2024-09-10T18:28:49.769340+0000","last_peered":"2024-09-10T18:28:49.769340+0000","last_clean":"2024-09-10T18:28:49.769340+0000","last_became_active":"2024-09-10T18:27:53.032159+0000","last_became_peered":"2024-09-10T18:27:53.032159+0000","last_unstale":"2024-09-10T18:28:49.769340+0000","last_undegraded":"2024-09-10T18:28:49.769340+0000","last_fullsized":"2024-09-10T18:28:49.769340+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.016948+0000","objects_scrubbed":0,"log_size":5095,"log_dups_size":0,"ondisk_log_size":5095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:10.012526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333905920,"num_objects":5095,"num_object_clones":0,"num_object_copies":15285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5095,"num_write_kb":326080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":209,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952554+0000","last_change":"2024-09-10T18:26:51.556115+0000","last_active":"2024-09-10T18:28:45.952554+0000","last_peered":"2024-09-10T18:28:45.952554+0000","last_clean":"2024-09-10T18:28:45.952554+0000","last_became_active":"2024-09-10T18:26:27.645932+0000","last_became_peered":"2024-09-10T18:26:27.645932+0000","last_unstale":"2024-09-10T18:28:45.952554+0000","last_undegraded":"2024-09-10T18:28:45.952554+0000","last_fullsized":"2024-09-10T18:28:45.952554+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-10T18:26:51.556026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:51.556026+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002446429,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":195,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.953573+0000","last_change":"2024-09-10T18:28:06.329960+0000","last_active":"2024-09-10T18:28:45.953573+0000","last_peered":"2024-09-10T18:28:45.953573+0000","last_clean":"2024-09-10T18:28:45.953573+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:28:45.953573+0000","last_undegraded":"2024-09-10T18:28:45.953573+0000","last_fullsized":"2024-09-10T18:28:45.953573+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:28:06.329891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:06.329891+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:36.083657+0000","scrub_duration":0.001979864,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"146'10","reported_seq":278,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952505+0000","last_change":"2024-09-10T18:27:11.013671+0000","last_active":"2024-09-10T18:28:45.952505+0000","last_peered":"2024-09-10T18:28:45.952505+0000","last_clean":"2024-09-10T18:28:45.952505+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:28:45.952505+0000","last_undegraded":"2024-09-10T18:28:45.952505+0000","last_fullsized":"2024-09-10T18:28:45.952505+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'10","last_scrub_stamp":"2024-09-10T18:27:11.013616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:11.013616+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022297150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":170,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951311+0000","last_change":"2024-09-10T18:26:50.594047+0000","last_active":"2024-09-10T18:28:45.951311+0000","last_peered":"2024-09-10T18:28:45.951311+0000","last_clean":"2024-09-10T18:28:45.951311+0000","last_became_active":"2024-09-10T18:26:27.647511+0000","last_became_peered":"2024-09-10T18:26:27.647511+0000","last_unstale":"2024-09-10T18:28:45.951311+0000","last_undegraded":"2024-09-10T18:28:45.951311+0000","last_fullsized":"2024-09-10T18:28:45.951311+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:50.593965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:26:50.593965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096006660000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":123,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.947509+0000","last_change":"2024-09-10T18:27:52.527922+0000","last_active":"2024-09-10T18:28:45.947509+0000","last_peered":"2024-09-10T18:28:45.947509+0000","last_clean":"2024-09-10T18:28:45.947509+0000","last_became_active":"2024-09-10T18:26:27.640922+0000","last_became_peered":"2024-09-10T18:26:27.640922+0000","last_unstale":"2024-09-10T18:28:45.947509+0000","last_undegraded":"2024-09-10T18:28:45.947509+0000","last_fullsized":"2024-09-10T18:28:45.947509+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.527847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.527847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:56.233308+0000","scrub_duration":0.0016791950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'6","reported_seq":187,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952859+0000","last_change":"2024-09-10T18:28:01.176967+0000","last_active":"2024-09-10T18:28:45.952859+0000","last_peered":"2024-09-10T18:28:45.952859+0000","last_clean":"2024-09-10T18:28:45.952859+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:28:45.952859+0000","last_undegraded":"2024-09-10T18:28:45.952859+0000","last_fullsized":"2024-09-10T18:28:45.952859+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-10T18:28:01.176843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:28:01.176843+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:27.123011+0000","scrub_duration":0.0029137500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":174,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.950693+0000","last_change":"2024-09-10T18:27:52.665248+0000","last_active":"2024-09-10T18:28:45.950693+0000","last_peered":"2024-09-10T18:28:45.950693+0000","last_clean":"2024-09-10T18:28:45.950693+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:28:45.950693+0000","last_undegraded":"2024-09-10T18:28:45.950693+0000","last_fullsized":"2024-09-10T18:28:45.950693+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:52.664666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.664666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:55.953894+0000","scrub_duration":0.001910155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":179,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.953252+0000","last_change":"2024-09-10T18:26:54.055557+0000","last_active":"2024-09-10T18:28:45.953252+0000","last_peered":"2024-09-10T18:28:45.953252+0000","last_clean":"2024-09-10T18:28:45.953252+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:28:45.953252+0000","last_undegraded":"2024-09-10T18:28:45.953252+0000","last_fullsized":"2024-09-10T18:28:45.953252+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:54.055483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:26:54.055483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002405113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"189'13","reported_seq":200,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.949303+0000","last_change":"2024-09-10T18:28:08.976270+0000","last_active":"2024-09-10T18:28:45.949303+0000","last_peered":"2024-09-10T18:28:45.949303+0000","last_clean":"2024-09-10T18:28:45.949303+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:28:45.949303+0000","last_undegraded":"2024-09-10T18:28:45.949303+0000","last_fullsized":"2024-09-10T18:28:45.949303+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'13","last_scrub_stamp":"2024-09-10T18:28:08.976162+0000","last_deep_scrub":"189'13","last_deep_scrub_stamp":"2024-09-10T18:28:08.976162+0000","last_clean_scrub_stamp":"2024-09-10T18:28:08.976162+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:38.695222+0000","scrub_duration":0.010982957999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":185,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952776+0000","last_change":"2024-09-10T18:26:51.013771+0000","last_active":"2024-09-10T18:28:45.952776+0000","last_peered":"2024-09-10T18:28:45.952776+0000","last_clean":"2024-09-10T18:28:45.952776+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:28:45.952776+0000","last_undegraded":"2024-09-10T18:28:45.952776+0000","last_fullsized":"2024-09-10T18:28:45.952776+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:51.013696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:26:51.013696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001333568,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":176,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952678+0000","last_change":"2024-09-10T18:27:56.215744+0000","last_active":"2024-09-10T18:28:45.952678+0000","last_peered":"2024-09-10T18:28:45.952678+0000","last_clean":"2024-09-10T18:28:45.952678+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:28:45.952678+0000","last_undegraded":"2024-09-10T18:28:45.952678+0000","last_fullsized":"2024-09-10T18:28:45.952678+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:56.215691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:56.215691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:12.955476+0000","scrub_duration":0.0034563770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":182,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.955733+0000","last_change":"2024-09-10T18:27:59.000777+0000","last_active":"2024-09-10T18:28:45.955733+0000","last_peered":"2024-09-10T18:28:45.955733+0000","last_clean":"2024-09-10T18:28:45.955733+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:28:45.955733+0000","last_undegraded":"2024-09-10T18:28:45.955733+0000","last_fullsized":"2024-09-10T18:28:45.955733+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:59.000723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:59.000723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:23.665168+0000","scrub_duration":0.002141666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"165'13","reported_seq":197,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.950982+0000","last_change":"2024-09-10T18:27:17.468672+0000","last_active":"2024-09-10T18:28:45.950982+0000","last_peered":"2024-09-10T18:28:45.950982+0000","last_clean":"2024-09-10T18:28:45.950982+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:28:45.950982+0000","last_undegraded":"2024-09-10T18:28:45.950982+0000","last_fullsized":"2024-09-10T18:28:45.950982+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'13","last_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:27:17.468619+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010200137999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":179,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951961+0000","last_change":"2024-09-10T18:26:52.832466+0000","last_active":"2024-09-10T18:28:45.951961+0000","last_peered":"2024-09-10T18:28:45.951961+0000","last_clean":"2024-09-10T18:28:45.951961+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:28:45.951961+0000","last_undegraded":"2024-09-10T18:28:45.951961+0000","last_fullsized":"2024-09-10T18:28:45.951961+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:52.832368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:26:52.832368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019356278000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":272,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-10T18:28:48.987777+0000","last_change":"2024-09-10T18:27:53.176983+0000","last_active":"2024-09-10T18:28:48.987777+0000","last_peered":"2024-09-10T18:28:48.987777+0000","last_clean":"2024-09-10T18:28:48.987777+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:28:48.987777+0000","last_undegraded":"2024-09-10T18:28:48.987777+0000","last_fullsized":"2024-09-10T18:28:48.987777+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:27:53.176916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:53.176916+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:10.909920+0000","scrub_duration":0.0024011380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":172,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.944511+0000","last_change":"2024-09-10T18:26:21.808801+0000","last_active":"2024-09-10T18:28:45.944511+0000","last_peered":"2024-09-10T18:28:45.944511+0000","last_clean":"2024-09-10T18:28:45.944511+0000","last_became_active":"2024-09-10T18:21:37.639706+0000","last_became_peered":"2024-09-10T18:21:37.639706+0000","last_unstale":"2024-09-10T18:28:45.944511+0000","last_undegraded":"2024-09-10T18:28:45.944511+0000","last_fullsized":"2024-09-10T18:28:45.944511+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:21.808720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:26:21.808720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018030489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":193,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.953403+0000","last_change":"2024-09-10T18:26:19.097137+0000","last_active":"2024-09-10T18:28:45.953403+0000","last_peered":"2024-09-10T18:28:45.953403+0000","last_clean":"2024-09-10T18:28:45.953403+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:28:45.953403+0000","last_undegraded":"2024-09-10T18:28:45.953403+0000","last_fullsized":"2024-09-10T18:28:45.953403+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-10T18:26:19.097047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:19.097047+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090288920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":178,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.952667+0000","last_change":"2024-09-10T18:27:12.052535+0000","last_active":"2024-09-10T18:28:45.952667+0000","last_peered":"2024-09-10T18:28:45.952667+0000","last_clean":"2024-09-10T18:28:45.952667+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:28:45.952667+0000","last_undegraded":"2024-09-10T18:28:45.952667+0000","last_fullsized":"2024-09-10T18:28:45.952667+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:12.052482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:27:12.052482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001572876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":280,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.950900+0000","last_change":"2024-09-10T18:28:15.692262+0000","last_active":"2024-09-10T18:28:45.950900+0000","last_peered":"2024-09-10T18:28:45.950900+0000","last_clean":"2024-09-10T18:28:45.950900+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:28:45.950900+0000","last_undegraded":"2024-09-10T18:28:45.950900+0000","last_fullsized":"2024-09-10T18:28:45.950900+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:28:15.692182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:15.692182+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:34.932763+0000","scrub_duration":0.0057467769999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":173,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.948936+0000","last_change":"2024-09-10T18:27:54.043813+0000","last_active":"2024-09-10T18:28:45.948936+0000","last_peered":"2024-09-10T18:28:45.948936+0000","last_clean":"2024-09-10T18:28:45.948936+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:28:45.948936+0000","last_undegraded":"2024-09-10T18:28:45.948936+0000","last_fullsized":"2024-09-10T18:28:45.948936+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:54.043726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:54.043726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:00.550467+0000","scrub_duration":0.001550514,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":202,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.944273+0000","last_change":"2024-09-10T18:27:08.970903+0000","last_active":"2024-09-10T18:28:45.944273+0000","last_peered":"2024-09-10T18:28:45.944273+0000","last_clean":"2024-09-10T18:28:45.944273+0000","last_became_active":"2024-09-10T18:26:27.649202+0000","last_became_peered":"2024-09-10T18:26:27.649202+0000","last_unstale":"2024-09-10T18:28:45.944273+0000","last_undegraded":"2024-09-10T18:28:45.944273+0000","last_fullsized":"2024-09-10T18:28:45.944273+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:27:08.970839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:27:08.970839+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017431300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":188,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.945114+0000","last_change":"2024-09-10T18:27:54.278241+0000","last_active":"2024-09-10T18:28:45.945114+0000","last_peered":"2024-09-10T18:28:45.945114+0000","last_clean":"2024-09-10T18:28:45.945114+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:28:45.945114+0000","last_undegraded":"2024-09-10T18:28:45.945114+0000","last_fullsized":"2024-09-10T18:28:45.945114+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:54.278176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:27:54.278176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:03.767304+0000","scrub_duration":0.001425756,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":187,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.945987+0000","last_change":"2024-09-10T18:27:58.515917+0000","last_active":"2024-09-10T18:28:45.945987+0000","last_peered":"2024-09-10T18:28:45.945987+0000","last_clean":"2024-09-10T18:28:45.945987+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:28:45.945987+0000","last_undegraded":"2024-09-10T18:28:45.945987+0000","last_fullsized":"2024-09-10T18:28:45.945987+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:58.515852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:58.515852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:59.695982+0000","scrub_duration":0.0014537980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":164,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951170+0000","last_change":"2024-09-10T18:26:18.076688+0000","last_active":"2024-09-10T18:28:45.951170+0000","last_peered":"2024-09-10T18:28:45.951170+0000","last_clean":"2024-09-10T18:28:45.951170+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:28:45.951170+0000","last_undegraded":"2024-09-10T18:28:45.951170+0000","last_fullsized":"2024-09-10T18:28:45.951170+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:18.076617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:26:18.076617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055521420000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":199,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.946140+0000","last_change":"2024-09-10T18:26:52.667217+0000","last_active":"2024-09-10T18:28:45.946140+0000","last_peered":"2024-09-10T18:28:45.946140+0000","last_clean":"2024-09-10T18:28:45.946140+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:28:45.946140+0000","last_undegraded":"2024-09-10T18:28:45.946140+0000","last_fullsized":"2024-09-10T18:28:45.946140+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:26:52.667140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:26:52.667140+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068280379999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":191,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.953420+0000","last_change":"2024-09-10T18:28:07.373735+0000","last_active":"2024-09-10T18:28:45.953420+0000","last_peered":"2024-09-10T18:28:45.953420+0000","last_clean":"2024-09-10T18:28:45.953420+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:28:45.953420+0000","last_undegraded":"2024-09-10T18:28:45.953420+0000","last_fullsized":"2024-09-10T18:28:45.953420+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:07.373653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:28:07.373653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:37.151211+0000","scrub_duration":0.0021799559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":166,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951381+0000","last_change":"2024-09-10T18:26:10.052392+0000","last_active":"2024-09-10T18:28:45.951381+0000","last_peered":"2024-09-10T18:28:45.951381+0000","last_clean":"2024-09-10T18:28:45.951381+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:28:45.951381+0000","last_undegraded":"2024-09-10T18:28:45.951381+0000","last_fullsized":"2024-09-10T18:28:45.951381+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:26:10.052296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019588034000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":173,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.947658+0000","last_change":"2024-09-10T18:27:17.780318+0000","last_active":"2024-09-10T18:28:45.947658+0000","last_peered":"2024-09-10T18:28:45.947658+0000","last_clean":"2024-09-10T18:28:45.947658+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:28:45.947658+0000","last_undegraded":"2024-09-10T18:28:45.947658+0000","last_fullsized":"2024-09-10T18:28:45.947658+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:17.780238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:17.780238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094000860000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":204,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951356+0000","last_change":"2024-09-10T18:27:51.212061+0000","last_active":"2024-09-10T18:28:45.951356+0000","last_peered":"2024-09-10T18:28:45.951356+0000","last_clean":"2024-09-10T18:28:45.951356+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:28:45.951356+0000","last_undegraded":"2024-09-10T18:28:45.951356+0000","last_fullsized":"2024-09-10T18:28:45.951356+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:27:51.211994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:27:51.211994+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:28:58.142346+0000","scrub_duration":0.0024425879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":189,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.950859+0000","last_change":"2024-09-10T18:28:01.748224+0000","last_active":"2024-09-10T18:28:45.950859+0000","last_peered":"2024-09-10T18:28:45.950859+0000","last_clean":"2024-09-10T18:28:45.950859+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:28:45.950859+0000","last_undegraded":"2024-09-10T18:28:45.950859+0000","last_fullsized":"2024-09-10T18:28:45.950859+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:01.748169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:28:01.748169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:31.247689+0000","scrub_duration":0.0025848580000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":184,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951651+0000","last_change":"2024-09-10T18:27:57.209856+0000","last_active":"2024-09-10T18:28:45.951651+0000","last_peered":"2024-09-10T18:28:45.951651+0000","last_clean":"2024-09-10T18:28:45.951651+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:28:45.951651+0000","last_undegraded":"2024-09-10T18:28:45.951651+0000","last_fullsized":"2024-09-10T18:28:45.951651+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:57.209756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:57.209756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:09.616768+0000","scrub_duration":0.0033515160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":181,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951699+0000","last_change":"2024-09-10T18:26:52.309818+0000","last_active":"2024-09-10T18:28:45.951699+0000","last_peered":"2024-09-10T18:28:45.951699+0000","last_clean":"2024-09-10T18:28:45.951699+0000","last_became_active":"2024-09-10T18:26:27.636996+0000","last_became_peered":"2024-09-10T18:26:27.636996+0000","last_unstale":"2024-09-10T18:28:45.951699+0000","last_undegraded":"2024-09-10T18:28:45.951699+0000","last_fullsized":"2024-09-10T18:28:45.951699+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:26:52.309588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:26:52.309588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053342499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":191,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951647+0000","last_change":"2024-09-10T18:27:52.206324+0000","last_active":"2024-09-10T18:28:45.951647+0000","last_peered":"2024-09-10T18:28:45.951647+0000","last_clean":"2024-09-10T18:28:45.951647+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:28:45.951647+0000","last_undegraded":"2024-09-10T18:28:45.951647+0000","last_fullsized":"2024-09-10T18:28:45.951647+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-09-10T18:27:52.205769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:27:52.205769+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:21.294714+0000","scrub_duration":0.0020401389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":182,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.948956+0000","last_change":"2024-09-10T18:28:06.991059+0000","last_active":"2024-09-10T18:28:45.948956+0000","last_peered":"2024-09-10T18:28:45.948956+0000","last_clean":"2024-09-10T18:28:45.948956+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:28:45.948956+0000","last_undegraded":"2024-09-10T18:28:45.948956+0000","last_fullsized":"2024-09-10T18:28:45.948956+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:06.991004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:06.991004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:29.268952+0000","scrub_duration":0.0015824980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":127,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.949026+0000","last_change":"2024-09-10T18:28:07.362740+0000","last_active":"2024-09-10T18:28:45.949026+0000","last_peered":"2024-09-10T18:28:45.949026+0000","last_clean":"2024-09-10T18:28:45.949026+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:28:45.949026+0000","last_undegraded":"2024-09-10T18:28:45.949026+0000","last_fullsized":"2024-09-10T18:28:45.949026+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:07.362688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:28:07.362688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:29:22.798236+0000","scrub_duration":0.001454197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":172,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.951973+0000","last_change":"2024-09-10T18:27:19.002667+0000","last_active":"2024-09-10T18:28:45.951973+0000","last_peered":"2024-09-10T18:28:45.951973+0000","last_clean":"2024-09-10T18:28:45.951973+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:28:45.951973+0000","last_undegraded":"2024-09-10T18:28:45.951973+0000","last_fullsized":"2024-09-10T18:28:45.951973+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:19.002583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:19.002583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013828224,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":182,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.945232+0000","last_change":"2024-09-10T18:27:13.620794+0000","last_active":"2024-09-10T18:28:45.945232+0000","last_peered":"2024-09-10T18:28:45.945232+0000","last_clean":"2024-09-10T18:28:45.945232+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:28:45.945232+0000","last_undegraded":"2024-09-10T18:28:45.945232+0000","last_fullsized":"2024-09-10T18:28:45.945232+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:13.620403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:27:13.620403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011110345000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":170,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.953643+0000","last_change":"2024-09-10T18:25:18.030959+0000","last_active":"2024-09-10T18:28:45.953643+0000","last_peered":"2024-09-10T18:28:45.953643+0000","last_clean":"2024-09-10T18:28:45.953643+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:28:45.953643+0000","last_undegraded":"2024-09-10T18:28:45.953643+0000","last_fullsized":"2024-09-10T18:28:45.953643+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:25:18.030878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.026365884999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":166,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.947918+0000","last_change":"2024-09-10T18:27:15.545071+0000","last_active":"2024-09-10T18:28:45.947918+0000","last_peered":"2024-09-10T18:28:45.947918+0000","last_clean":"2024-09-10T18:28:45.947918+0000","last_became_active":"2024-09-10T18:26:27.648112+0000","last_became_peered":"2024-09-10T18:26:27.648112+0000","last_unstale":"2024-09-10T18:28:45.947918+0000","last_undegraded":"2024-09-10T18:28:45.947918+0000","last_fullsized":"2024-09-10T18:28:45.947918+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:27:15.544986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:27:15.544986+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018039950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":175,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-10T18:28:45.949090+0000","last_change":"2024-09-10T18:27:11.603305+0000","last_active":"2024-09-10T18:28:45.949090+0000","last_peered":"2024-09-10T18:28:45.949090+0000","last_clean":"2024-09-10T18:28:45.949090+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:28:45.949090+0000","last_undegraded":"2024-09-10T18:28:45.949090+0000","last_fullsized":"2024-09-10T18:28:45.949090+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:27:11.603148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:27:11.603148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011808801000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5186256896,"num_objects":79136,"num_object_clones":0,"num_object_copies":237408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79136,"num_write_kb":5064704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15541927936,"data_stored":15541927936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77536,"ondisk_log_size":77536,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":431,"num_read_kb":426,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":891,"ondisk_log_size":891,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234075,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1860876,"kb_used_data":1555200,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91884276,"statfs":{"total":95995035648,"available":94089498624,"internally_reserved":0,"allocated":1592524800,"data_stored":1591882029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":2.2410000000000001,"15min":1.095},"min":{"1min":0.26400000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.3239999999999998,"5min":85.012,"15min":85.012},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":2.202,"15min":1.077},"min":{"1min":0.312,"5min":0.222,"15min":0.222},"max":{"1min":1.421,"5min":84.665999999999997,"15min":84.665999999999997},"last":0.44600000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.3149999999999999,"15min":0.78200000000000003},"min":{"1min":0.40799999999999997,"5min":0.25,"15min":0.25},"max":{"1min":3.2189999999999999,"5min":12.583,"15min":12.583},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.2969999999999999,"15min":0.80700000000000005},"min":{"1min":0.39800000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.6720000000000002,"5min":12.601000000000001,"15min":12.601000000000001},"last":0.56399999999999995}]},{"osd":2,"last update":"Tue Sep 10 18:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.90800000000000003,"15min":0.66400000000000003},"min":{"1min":0.27600000000000002,"5min":0.249,"15min":0.249},"max":{"1min":2.6850000000000001,"5min":5.3150000000000004,"15min":5.3150000000000004},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.97499999999999998,"15min":0.69999999999999996},"min":{"1min":0.35099999999999998,"5min":0.254,"15min":0.254},"max":{"1min":3.2959999999999998,"5min":9.9450000000000003,"15min":9.9450000000000003},"last":1.1539999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":2.4009999999999998,"15min":1.1639999999999999},"min":{"1min":0.36599999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.865,"5min":86.305999999999997,"15min":86.305999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":2.4359999999999999,"15min":1.1799999999999999},"min":{"1min":0.33300000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.504,"5min":88.364000000000004,"15min":88.364000000000004},"last":1.6779999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.40100000000000002,"15min":0.42499999999999999},"min":{"1min":0.29099999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.853,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.39500000000000002,"15min":0.42799999999999999},"min":{"1min":0.309,"5min":0.223,"15min":0.223},"max":{"1min":1.4850000000000001,"5min":1.925,"15min":1.925},"last":0.627}]},{"osd":5,"last update":"Tue Sep 10 18:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.46000000000000002,"15min":0.434},"min":{"1min":0.375,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.2480000000000002,"5min":2.2480000000000002,"15min":2.2480000000000002},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.436,"15min":0.46800000000000003},"min":{"1min":0.34100000000000003,"5min":0.219,"15min":0.219},"max":{"1min":1.8819999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":0.50900000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:28:26 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.48299999999999998,"15min":0.438},"min":{"1min":0.32300000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.46100000000000002,"15min":0.438},"min":{"1min":0.35499999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":0.54100000000000004}]}]},{"osd":6,"up_from":144,"seq":618475290654,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1899132,"kb_used_data":1596532,"kb_used_omap":7,"kb_used_meta":302584,"kb_avail":91846020,"statfs":{"total":95995035648,"available":94050324480,"internally_reserved":0,"allocated":1634848768,"data_stored":1634219939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":309846103},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":1.925,"5min":2.0739999999999998,"15min":2.0990000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":30.629999999999999,"5min":30.629999999999999,"15min":30.629999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":1.9359999999999999,"5min":1.6220000000000001,"15min":1.5700000000000001},"min":{"1min":0.23999999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":30.597000000000001,"5min":30.597000000000001,"15min":30.597000000000001},"last":0.27400000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":1.9850000000000001,"5min":0.97299999999999998,"15min":0.80500000000000005},"min":{"1min":0.308,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":30.984000000000002,"5min":30.984000000000002,"15min":30.984000000000002},"last":1.7230000000000001},{"interface":"front","average":{"1min":2.02,"5min":1.208,"15min":1.073},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":30.571000000000002,"5min":30.571000000000002,"15min":30.571000000000002},"last":0.80500000000000005}]},{"osd":2,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":1.0229999999999999,"15min":1.0800000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.724,"5min":4.8259999999999996,"15min":4.8259999999999996},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.752,"15min":0.76600000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.8159999999999998,"5min":2.9860000000000002,"15min":2.9860000000000002},"last":0.82599999999999996}]},{"osd":3,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":3.3940000000000001,"5min":2.4249999999999998,"15min":2.2629999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":61.703000000000003,"5min":61.703000000000003,"15min":61.703000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":3.4900000000000002,"5min":2.4249999999999998,"15min":2.2469999999999999},"min":{"1min":0.39400000000000002,"5min":0.318,"15min":0.318},"max":{"1min":61.814,"5min":61.814,"15min":61.814},"last":0.308}]},{"osd":4,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.51300000000000001,"15min":0.495},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.66600000000000004,"15min":0.65100000000000002},"min":{"1min":0.27100000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":4.4359999999999999,"5min":4.4359999999999999,"15min":4.4359999999999999},"last":0.32200000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.47599999999999998,"15min":0.46800000000000003},"min":{"1min":0.26900000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.7509999999999999,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.45800000000000002,"15min":0.42199999999999999},"min":{"1min":0.32200000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.505}]},{"osd":7,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.504,"15min":0.48799999999999999},"min":{"1min":0.28999999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.202,"5min":2.202,"15min":2.202},"last":0.441},{"interface":"front","average":{"1min":0.625,"5min":0.505,"15min":0.48499999999999999},"min":{"1min":0.29299999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.0950000000000002,"5min":3.0950000000000002,"15min":3.0950000000000002},"last":0.54700000000000004}]}]},{"osd":5,"up_from":39,"seq":167503724646,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2836436,"kb_used_data":2526600,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90908716,"statfs":{"total":95995035648,"available":93090525184,"internally_reserved":0,"allocated":2587238400,"data_stored":2586588487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:27:54 2024","interfaces":[{"interface":"back","average":{"1min":3.5369999999999999,"5min":2.359,"15min":1.1519999999999999},"min":{"1min":0.247,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":31.172000000000001,"5min":52.033999999999999,"15min":52.033999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":3.835,"5min":2.0110000000000001,"15min":1.0169999999999999},"min":{"1min":0.245,"5min":0.216,"15min":0.216},"max":{"1min":32.246000000000002,"5min":52.167999999999999,"15min":52.167999999999999},"last":0.39900000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.73299999999999998,"15min":0.61899999999999999},"min":{"1min":0.32900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.097,"5min":4.2400000000000002,"15min":4.2400000000000002},"last":1.9850000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":1.0589999999999999,"15min":0.72299999999999998},"min":{"1min":0.30099999999999999,"5min":0.247,"15min":0.247},"max":{"1min":2.169,"5min":17.661000000000001,"15min":17.661000000000001},"last":1.9770000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:27:54 2024","interfaces":[{"interface":"back","average":{"1min":2.3500000000000001,"5min":1.2490000000000001,"15min":0.78400000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":36.853000000000002,"5min":36.853000000000002,"15min":36.853000000000002},"last":1.7470000000000001},{"interface":"front","average":{"1min":2.1899999999999999,"5min":1.169,"15min":0.753},"min":{"1min":0.27400000000000002,"5min":0.26100000000000001,"15min":0.255},"max":{"1min":31.132000000000001,"5min":31.132000000000001,"15min":31.132000000000001},"last":0.499}]},{"osd":3,"last update":"Tue Sep 10 18:27:54 2024","interfaces":[{"interface":"back","average":{"1min":3.6539999999999999,"5min":1.756,"15min":0.94799999999999995},"min":{"1min":0.35499999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":31.207000000000001,"5min":31.207000000000001,"15min":31.207000000000001},"last":1.014},{"interface":"front","average":{"1min":3.415,"5min":1.5069999999999999,"15min":0.85899999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":31.199000000000002,"5min":31.199000000000002,"15min":31.199000000000002},"last":1.6419999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.38400000000000001,"15min":0.36899999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.2749999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.41499999999999998,"15min":0.39600000000000002},"min":{"1min":0.26300000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":0.435}]},{"osd":6,"last update":"Tue Sep 10 18:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.57799999999999996,"15min":0.56599999999999995},"min":{"1min":0.34499999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.613,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.56200000000000006,"15min":0.54600000000000004},"min":{"1min":0.32600000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.0880000000000001,"5min":2.153,"15min":2.153},"last":0.46800000000000003}]},{"osd":7,"last update":"Tue Sep 10 18:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.46000000000000002,"15min":0.45800000000000002},"min":{"1min":0.38400000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.2350000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.45100000000000001,"15min":0.44400000000000001},"min":{"1min":0.33900000000000002,"5min":0.27200000000000002,"15min":0.24299999999999999},"max":{"1min":1.258,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":0.55400000000000005}]}]},{"osd":4,"up_from":33,"seq":141733920874,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1869516,"kb_used_data":1563840,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91875636,"statfs":{"total":95995035648,"available":94080651264,"internally_reserved":0,"allocated":1601372160,"data_stored":1600729200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:28:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.3240000000000001,"15min":0.81699999999999995},"min":{"1min":0.29799999999999999,"5min":0.188,"15min":0.188},"max":{"1min":16.236999999999998,"5min":31.585000000000001,"15min":31.585000000000001},"last":2.532},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.3720000000000001,"15min":0.85399999999999998},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":16.280999999999999,"5min":29.802,"15min":29.802},"last":3.8319999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.80600000000000005,"15min":0.66200000000000003},"min":{"1min":0.248,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.1080000000000001,"5min":6.0069999999999997,"15min":6.0069999999999997},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.84399999999999997,"15min":0.66000000000000003},"min":{"1min":0.39300000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.3730000000000002,"5min":4.3899999999999997,"15min":4.3899999999999997},"last":0.56299999999999994}]},{"osd":2,"last update":"Tue Sep 10 18:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":1.0029999999999999,"15min":0.72399999999999998},"min":{"1min":0.29499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.407,"5min":13.898999999999999,"15min":13.898999999999999},"last":1.6240000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":1.1040000000000001,"15min":0.751},"min":{"1min":0.28399999999999997,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.5840000000000001,"5min":14.51,"15min":14.51},"last":0.34000000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:28:21 2024","interfaces":[{"interface":"back","average":{"1min":2.2290000000000001,"5min":1.6140000000000001,"15min":0.91700000000000004},"min":{"1min":0.30199999999999999,"5min":0.249,"15min":0.249},"max":{"1min":37.351999999999997,"5min":37.351999999999997,"15min":37.351999999999997},"last":2.5489999999999999},{"interface":"front","average":{"1min":2.2480000000000002,"5min":1.675,"15min":0.94899999999999995},"min":{"1min":0.28000000000000003,"5min":0.255,"15min":0.255},"max":{"1min":37.366,"5min":37.366,"15min":37.366},"last":2.073}]},{"osd":5,"last update":"Tue Sep 10 18:28:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.41199999999999998,"15min":0.39500000000000002},"min":{"1min":0.25800000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.46400000000000002,"15min":0.436},"min":{"1min":0.30499999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.8220000000000001,"5min":2.8220000000000001,"15min":2.8220000000000001},"last":0.46600000000000003}]},{"osd":6,"last update":"Tue Sep 10 18:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.41999999999999998,"15min":0.39800000000000002},"min":{"1min":0.29199999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.47899999999999998,"15min":0.45500000000000002},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":0.52500000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:28:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.42699999999999999,"15min":0.49299999999999999},"min":{"1min":0.32500000000000001,"5min":0.255,"15min":0.255},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.46600000000000003,"15min":0.496},"min":{"1min":0.34899999999999998,"5min":0.247,"15min":0.247},"max":{"1min":1.462,"5min":2.5230000000000001,"15min":2.5230000000000001},"last":0.496}]}]},{"osd":3,"up_from":27,"seq":115964117104,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2189848,"kb_used_data":1884176,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91555304,"statfs":{"total":95995035648,"available":93752631296,"internally_reserved":0,"allocated":1929396224,"data_stored":1928738268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.8740000000000001,"15min":0.88},"min":{"1min":0.32300000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.0060000000000002,"5min":8.5530000000000008,"15min":8.5530000000000008},"last":2.0910000000000002},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.948,"15min":0.91600000000000004},"min":{"1min":0.24099999999999999,"5min":0.222,"15min":0.222},"max":{"1min":5.2729999999999997,"5min":13.659000000000001,"15min":13.659000000000001},"last":5.8029999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":1.5780000000000001,"15min":0.81599999999999995},"min":{"1min":0.29099999999999998,"5min":0.247,"15min":0.247},"max":{"1min":4.1539999999999999,"5min":11.606,"15min":11.606},"last":6.242},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.4059999999999999,"15min":0.73799999999999999},"min":{"1min":0.26600000000000001,"5min":0.19,"15min":0.19},"max":{"1min":4.4770000000000003,"5min":8.5229999999999997,"15min":8.5229999999999997},"last":0.20399999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:28:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.5449999999999999,"15min":0.81000000000000005},"min":{"1min":0.29899999999999999,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":5.9829999999999997,"5min":7.7960000000000003,"15min":7.7960000000000003},"last":2.1179999999999999},{"interface":"front","average":{"1min":1.468,"5min":1.6160000000000001,"15min":0.82499999999999996},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.252},"max":{"1min":7.7830000000000004,"5min":10.113,"15min":10.113},"last":4.306}]},{"osd":4,"last update":"Tue Sep 10 18:28:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.738,"15min":0.94299999999999995},"min":{"1min":0.40500000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.4710000000000001,"5min":23.684999999999999,"15min":23.684999999999999},"last":2.4849999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":1.613,"15min":0.91200000000000003},"min":{"1min":0.33500000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.3879999999999999,"5min":23.649000000000001,"15min":23.649000000000001},"last":2.4289999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:28:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.776,"15min":1.0309999999999999},"min":{"1min":0.45000000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":5.1779999999999999,"5min":23.719999999999999,"15min":23.719999999999999},"last":2.52},{"interface":"front","average":{"1min":1.232,"5min":1.766,"15min":0.99299999999999999},"min":{"1min":0.35399999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":4.4450000000000003,"5min":23.707000000000001,"15min":23.707000000000001},"last":2.4990000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:28:33 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.89100000000000001,"15min":0.88300000000000001},"min":{"1min":0.38700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":4.4690000000000003,"5min":4.4690000000000003,"15min":4.4690000000000003},"last":2.4620000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":1.1819999999999999,"15min":1.282},"min":{"1min":0.39400000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.76400000000000001,"5min":5.8710000000000004,"15min":5.8710000000000004},"last":0.40000000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:28:33 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.6399999999999999,"15min":0.96499999999999997},"min":{"1min":0.45100000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.8719999999999999,"5min":23.748999999999999,"15min":23.748999999999999},"last":2.5529999999999999},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.7350000000000001,"15min":1.018},"min":{"1min":0.38100000000000001,"5min":0.307,"15min":0.307},"max":{"1min":4.4180000000000001,"5min":25.058,"15min":25.058},"last":2.5680000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411444,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2191808,"kb_used_data":1886132,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91553344,"statfs":{"total":95995035648,"available":93750624256,"internally_reserved":0,"allocated":1931399168,"data_stored":1930768512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":1.8779999999999999,"15min":0.85699999999999998},"min":{"1min":0.214,"5min":0.185,"15min":0.185},"max":{"1min":2.5350000000000001,"5min":17.135000000000002,"15min":17.135000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.056,"5min":2.1259999999999999,"15min":0.94599999999999995},"min":{"1min":0.252,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.3500000000000001,"5min":17.146000000000001,"15min":17.146000000000001},"last":0.76900000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:28:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.798,"15min":0.82799999999999996},"min":{"1min":0.27100000000000002,"5min":0.193,"15min":0.193},"max":{"1min":3.4319999999999999,"5min":10.077999999999999,"15min":10.077999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.302,"15min":0.67900000000000005},"min":{"1min":0.23499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.569,"5min":12.532999999999999,"15min":12.532999999999999},"last":0.36899999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":2.2000000000000002,"15min":1.0489999999999999},"min":{"1min":0.29699999999999999,"5min":0.251,"15min":0.251},"max":{"1min":5.8479999999999999,"5min":60.786999999999999,"15min":60.786999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.2649999999999999,"5min":2.407,"15min":1.093},"min":{"1min":0.308,"5min":0.25900000000000001,"15min":0.25},"max":{"1min":7.9109999999999996,"5min":60.116999999999997,"15min":60.116999999999997},"last":0.85699999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.1890000000000001,"15min":0.77500000000000002},"min":{"1min":0.31900000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.2090000000000001,"5min":6.9169999999999998,"15min":6.9169999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.98099999999999998,"15min":0.74399999999999999},"min":{"1min":0.32900000000000001,"5min":0.317,"15min":0.317},"max":{"1min":1.3220000000000001,"5min":12.130000000000001,"15min":12.130000000000001},"last":0.42399999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.98499999999999999,"15min":0.72799999999999998},"min":{"1min":0.35099999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.5600000000000001,"5min":12.106,"15min":12.106},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.70599999999999996,"5min":1.0129999999999999,"15min":0.751},"min":{"1min":0.33300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.5860000000000001,"5min":12.154,"15min":12.154},"last":0.83499999999999996}]},{"osd":6,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":1.02,"15min":1.0860000000000001},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.49,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.85299999999999998,"15min":0.84799999999999998},"min":{"1min":0.38300000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":3.9969999999999999,"5min":3.9969999999999999,"15min":3.9969999999999999},"last":0.45200000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:28:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.264,"15min":0.84099999999999997},"min":{"1min":0.41799999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":5.0039999999999996,"5min":7.1639999999999997,"15min":7.1639999999999997},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.75700000000000001,"5min":1.232,"15min":0.81799999999999995},"min":{"1min":0.40799999999999997,"5min":0.307,"15min":0.307},"max":{"1min":2.5819999999999999,"5min":6.9669999999999996,"15min":6.9669999999999996},"last":1.2829999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574969,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2493336,"kb_used_data":2187660,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91251816,"statfs":{"total":95995035648,"available":93441859584,"internally_reserved":0,"allocated":2240163840,"data_stored":2239509194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":2.6749999999999998,"15min":1.125},"min":{"1min":0.217,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":1.6220000000000001,"5min":80.456999999999994,"15min":80.456999999999994},"last":1.5},{"interface":"front","average":{"1min":0.73499999999999999,"5min":2.8559999999999999,"15min":1.204},"min":{"1min":0.22700000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.3999999999999999,"5min":80.578000000000003,"15min":80.578000000000003},"last":0.42799999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":1.6240000000000001,"15min":0.79900000000000004},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.4809999999999999,"5min":13.035,"15min":13.035},"last":2.8439999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.5349999999999999,"15min":0.75700000000000001},"min":{"1min":0.24199999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.9700000000000002,"5min":6.593,"15min":6.593},"last":3.4249999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":2.6160000000000001,"15min":1.1479999999999999},"min":{"1min":0.29999999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.8769999999999998,"5min":43.076000000000001,"15min":43.076000000000001},"last":2.8540000000000001},{"interface":"front","average":{"1min":1.006,"5min":2.7080000000000002,"15min":1.21},"min":{"1min":0.29699999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.5,"5min":41.93,"15min":41.93},"last":0.45500000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":1.0760000000000001,"15min":0.75},"min":{"1min":0.32100000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.794,"5min":7.1500000000000004,"15min":7.1500000000000004},"last":2.8119999999999998},{"interface":"front","average":{"1min":0.96299999999999997,"5min":1.3009999999999999,"15min":0.879},"min":{"1min":0.38400000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.984,"5min":6.1600000000000001,"15min":19.393000000000001},"last":0.58599999999999997}]},{"osd":5,"last update":"Tue Sep 10 18:28:46 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.405,"15min":0.83499999999999996},"min":{"1min":0.34999999999999998,"5min":0.311,"15min":0.311},"max":{"1min":4.0170000000000003,"5min":15.48,"15min":15.48},"last":2.1509999999999998},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.157,"15min":0.78600000000000003},"min":{"1min":0.38,"5min":0.308,"15min":0.308},"max":{"1min":3.3780000000000001,"5min":6.1699999999999999,"15min":6.1699999999999999},"last":0.99399999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.2450000000000001,"15min":1.3169999999999999},"min":{"1min":0.40400000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.1499999999999999,"5min":7.0170000000000003,"15min":7.0170000000000003},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.82999999999999996,"5min":1.2629999999999999,"15min":1.335},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.4470000000000001,"5min":4.4770000000000003,"15min":4.4770000000000003},"last":0.47699999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.321,"15min":0.83799999999999997},"min":{"1min":0.47699999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.1829999999999998,"5min":15.545999999999999,"15min":15.545999999999999},"last":2.585},{"interface":"front","average":{"1min":0.83399999999999996,"5min":1.1899999999999999,"15min":0.78600000000000003},"min":{"1min":0.438,"5min":0.313,"15min":0.313},"max":{"1min":2.8500000000000001,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":0.503}]}]},{"osd":0,"up_from":8,"seq":34359738494,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2186760,"kb_used_data":1885244,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91558392,"statfs":{"total":95995035648,"available":93755793408,"internally_reserved":0,"allocated":1930489856,"data_stored":1929850865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:28:12 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.6259999999999999,"15min":0.77600000000000002},"min":{"1min":0.23799999999999999,"5min":0.193,"15min":0.193},"max":{"1min":8.4280000000000008,"5min":12.106999999999999,"15min":12.106999999999999},"last":0.315},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.8340000000000001,"15min":0.85099999999999998},"min":{"1min":0.27400000000000002,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":3.6680000000000001,"5min":13.436,"15min":13.436},"last":0.55800000000000005}]},{"osd":2,"last update":"Tue Sep 10 18:28:39 2024","interfaces":[{"interface":"back","average":{"1min":2.2970000000000002,"5min":2.1709999999999998,"15min":0.99199999999999999},"min":{"1min":0.22,"5min":0.189,"15min":0.189},"max":{"1min":30.219999999999999,"5min":30.219999999999999,"15min":30.219999999999999},"last":1.101},{"interface":"front","average":{"1min":2.23,"5min":2.198,"15min":0.998},"min":{"1min":0.24099999999999999,"5min":0.22,"15min":0.22},"max":{"1min":30.129999999999999,"5min":30.129999999999999,"15min":30.129999999999999},"last":1.034}]},{"osd":3,"last update":"Tue Sep 10 18:28:00 2024","interfaces":[{"interface":"back","average":{"1min":1.758,"5min":2.1850000000000001,"15min":1.028},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.1159999999999997,"5min":23.143999999999998,"15min":23.143999999999998},"last":1.4299999999999999},{"interface":"front","average":{"1min":1.159,"5min":2.012,"15min":0.96499999999999997},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.8109999999999999,"5min":23.035,"15min":23.035},"last":0.69199999999999995}]},{"osd":4,"last update":"Tue Sep 10 18:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.556,"15min":0.93400000000000005},"min":{"1min":0.48699999999999999,"5min":0.246,"15min":0.246},"max":{"1min":2.1219999999999999,"5min":17.151,"15min":17.151},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.89900000000000002,"5min":1.6140000000000001,"15min":0.97099999999999997},"min":{"1min":0.32200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.9910000000000001,"5min":18.728999999999999,"15min":18.728999999999999},"last":0.64300000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:27:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2280000000000002,"5min":1.845,"15min":1.0149999999999999},"min":{"1min":0.33300000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":18.77,"5min":18.77,"15min":18.77},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.47,"5min":1.5189999999999999,"15min":0.90800000000000003},"min":{"1min":0.38400000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.8170000000000002,"5min":14.991,"15min":14.991},"last":0.48199999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.2090000000000001,"15min":1.274},"min":{"1min":0.45700000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.1499999999999999,"5min":4.7830000000000004,"15min":4.7830000000000004},"last":0.371},{"interface":"front","average":{"1min":0.93700000000000006,"5min":1.321,"15min":1.385},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":2.9540000000000002,"5min":3.77,"15min":3.77},"last":0.61699999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.55,"15min":0.89800000000000002},"min":{"1min":0.376,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.0179999999999998,"5min":13.711,"15min":13.711},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.76000000000000001,"5min":1.4910000000000001,"15min":0.876},"min":{"1min":0.435,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":2.4220000000000002,"5min":15.032,"15min":15.032},"last":0.51500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1941897216,"data_stored":1941897216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2253586432,"data_stored":2253586432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1957691392,"data_stored":1957691392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1931935744,"data_stored":1931935744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1615134720,"data_stored":1615134720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2588409856,"data_stored":2588409856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1645805568,"data_stored":1645805568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1607467008,"data_stored":1607467008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:28:55.794 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-10T18:28:55.794 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:28:55.795 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-10T18:28:55.795 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-10T18:28:55.795 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-10T18:28:55.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-10T18:28:55.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:55.796 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:55.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:55.897 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:55.927 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:55.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:55.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:56.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:56.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:56 smithi028 ceph-mon[29291]: 6.15 scrub starts 2024-09-10T18:28:56.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:56 smithi028 ceph-mon[29291]: 6.15 scrub ok 2024-09-10T18:28:56.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:56 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:56.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:56 smithi028 ceph-mon[29291]: pgmap v331: 158 pgs: 158 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 943 op/s 2024-09-10T18:28:56.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:56 smithi028 ceph-mon[29291]: 2.1 scrub starts 2024-09-10T18:28:56.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:56 smithi028 ceph-mon[29291]: 2.1 scrub ok 2024-09-10T18:28:56.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:56 smithi028 ceph-mon[29291]: osdmap e201: 8 total, 8 up, 8 in 2024-09-10T18:28:56.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:56.080 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:56.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:56.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 49 16 86334 86318 110.066 74.375 0.0124223 0.00908067 2024-09-10T18:28:56.181 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:56.213 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:56.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:56.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:56.315 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:56.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:56.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:56 smithi204 ceph-mon[32357]: 6.15 scrub starts 2024-09-10T18:28:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:56 smithi204 ceph-mon[32357]: 6.15 scrub ok 2024-09-10T18:28:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:56 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:56 smithi204 ceph-mon[32357]: pgmap v331: 158 pgs: 158 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 943 op/s 2024-09-10T18:28:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:56 smithi204 ceph-mon[32357]: 2.1 scrub starts 2024-09-10T18:28:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:56 smithi204 ceph-mon[32357]: 2.1 scrub ok 2024-09-10T18:28:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:56 smithi204 ceph-mon[32357]: osdmap e201: 8 total, 8 up, 8 in 2024-09-10T18:28:56.375 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:56 smithi028 ceph-mon[39220]: 6.15 scrub starts 2024-09-10T18:28:56.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:56 smithi028 ceph-mon[39220]: 6.15 scrub ok 2024-09-10T18:28:56.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:56 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:28:56.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:56 smithi028 ceph-mon[39220]: pgmap v331: 158 pgs: 158 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 943 op/s 2024-09-10T18:28:56.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:56 smithi028 ceph-mon[39220]: 2.1 scrub starts 2024-09-10T18:28:56.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:56 smithi028 ceph-mon[39220]: 2.1 scrub ok 2024-09-10T18:28:56.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:56 smithi028 ceph-mon[39220]: osdmap e201: 8 total, 8 up, 8 in 2024-09-10T18:28:56.386 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:28:56.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:56.468 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:28:56.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:56.516 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:56.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:56.617 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:56.650 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:56.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:56.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:56.752 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:56.780 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:56.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:56.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:56.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:56.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:57.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:57.052 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:28:57.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:57.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:57.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:57.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 50 16 87460 87444 109.272 70.375 0.018942 0.00914623 2024-09-10T18:28:57.160 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:57.201 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:57.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:57.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:57.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[29291]: 6.12 scrub starts 2024-09-10T18:28:57.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[29291]: 6.12 scrub ok 2024-09-10T18:28:57.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[29291]: 4.12 scrub starts 2024-09-10T18:28:57.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[29291]: 4.12 scrub ok 2024-09-10T18:28:57.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[29291]: 6.d scrub starts 2024-09-10T18:28:57.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[29291]: 6.d scrub ok 2024-09-10T18:28:57.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[29291]: 6.0 scrub starts 2024-09-10T18:28:57.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[29291]: 6.0 scrub ok 2024-09-10T18:28:57.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/904340359' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:28:57.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[29291]: from='client.35081 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:28:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[39220]: 6.12 scrub starts 2024-09-10T18:28:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[39220]: 6.12 scrub ok 2024-09-10T18:28:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[39220]: 4.12 scrub starts 2024-09-10T18:28:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[39220]: 4.12 scrub ok 2024-09-10T18:28:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[39220]: 6.d scrub starts 2024-09-10T18:28:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[39220]: 6.d scrub ok 2024-09-10T18:28:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[39220]: 6.0 scrub starts 2024-09-10T18:28:57.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[39220]: 6.0 scrub ok 2024-09-10T18:28:57.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/904340359' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:28:57.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:57 smithi028 ceph-mon[39220]: from='client.35081 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:28:57.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:57.311 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:57 smithi204 ceph-mon[32357]: 6.12 scrub starts 2024-09-10T18:28:57.311 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:57 smithi204 ceph-mon[32357]: 6.12 scrub ok 2024-09-10T18:28:57.311 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:57 smithi204 ceph-mon[32357]: 4.12 scrub starts 2024-09-10T18:28:57.311 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:57 smithi204 ceph-mon[32357]: 4.12 scrub ok 2024-09-10T18:28:57.311 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:57 smithi204 ceph-mon[32357]: 6.d scrub starts 2024-09-10T18:28:57.311 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:57 smithi204 ceph-mon[32357]: 6.d scrub ok 2024-09-10T18:28:57.311 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:57 smithi204 ceph-mon[32357]: 6.0 scrub starts 2024-09-10T18:28:57.311 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:57 smithi204 ceph-mon[32357]: 6.0 scrub ok 2024-09-10T18:28:57.311 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:57 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/904340359' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:28:57.311 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:57 smithi204 ceph-mon[32357]: from='client.35081 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:28:57.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:57.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:57.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:57.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:28:57.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:57.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:57.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:28:57.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:57.640 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:57.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:57.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:57.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:57.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:57.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:57.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:57.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:57.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:57.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:58.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:28:58.034 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:28:58.034 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-10T18:28:58.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-10T18:28:58.034 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:28:58.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.035 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.34334.0:568 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94344803770144 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-10T18:28:58.035 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:09.060228+0000", 2024-09-10T18:28:58.035 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 348.96776155399999, 2024-09-10T18:28:58.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0146112759999999, 2024-09-10T18:28:58.035 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.035 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:28:58.035 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.035 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.34334", 2024-09-10T18:28:58.035 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/601695668", 2024-09-10T18:28:58.035 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 568 2024-09-10T18:28:58.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.036 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.060228+0000", 2024-09-10T18:28:58.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.060228+0000", 2024-09-10T18:28:58.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.060224+0000", 2024-09-10T18:28:58.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999962 2024-09-10T18:28:58.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.060238+0000", 2024-09-10T18:28:58.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4086e-05 2024-09-10T18:28:58.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.060260+0000", 2024-09-10T18:28:58.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1081999999999999e-05 2024-09-10T18:28:58.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.060270+0000", 2024-09-10T18:28:58.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0057e-05 2024-09-10T18:28:58.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.080528+0000", 2024-09-10T18:28:58.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020258911000000001 2024-09-10T18:28:58.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:28:58.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.080534+0000", 2024-09-10T18:28:58.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5489999999999997e-06 2024-09-10T18:28:58.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.067258+0000", 2024-09-10T18:28:58.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98672402199999998 2024-09-10T18:28:58.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for missing object", 2024-09-10T18:28:58.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.067399+0000", 2024-09-10T18:28:58.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014092 2024-09-10T18:28:58.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.070170+0000", 2024-09-10T18:28:58.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027705450000000001 2024-09-10T18:28:58.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-10T18:28:58.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.070293+0000", 2024-09-10T18:28:58.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012330800000000001 2024-09-10T18:28:58.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.072302+0000", 2024-09-10T18:28:58.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020088089999999999 2024-09-10T18:28:58.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.072366+0000", 2024-09-10T18:28:58.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4388999999999996e-05 2024-09-10T18:28:58.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.074839+0000", 2024-09-10T18:28:58.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024734029999999999 2024-09-10T18:28:58.049 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.050 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.050 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.050 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.34334.0:575 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch ping cookie 94344803770144 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-10T18:28:58.050 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:09.771277+0000", 2024-09-10T18:28:58.051 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 348.25671254600002, 2024-09-10T18:28:58.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.30179671000000002, 2024-09-10T18:28:58.051 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.051 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:28:58.051 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.051 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.34334", 2024-09-10T18:28:58.052 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/601695668", 2024-09-10T18:28:58.052 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 575 2024-09-10T18:28:58.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.052 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.053 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.771277+0000", 2024-09-10T18:28:58.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.053 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.771277+0000", 2024-09-10T18:28:58.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.771276+0000", 2024-09-10T18:28:58.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:28:58.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.771284+0000", 2024-09-10T18:28:58.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9179999999999994e-06 2024-09-10T18:28:58.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.771292+0000", 2024-09-10T18:28:58.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5220000000000001e-06 2024-09-10T18:28:58.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.771296+0000", 2024-09-10T18:28:58.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3100000000000002e-06 2024-09-10T18:28:58.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.771350+0000", 2024-09-10T18:28:58.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3996999999999998e-05 2024-09-10T18:28:58.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:28:58.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:09.771357+0000", 2024-09-10T18:28:58.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0260000000000002e-06 2024-09-10T18:28:58.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.067428+0000", 2024-09-10T18:28:58.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29607112099999999 2024-09-10T18:28:58.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for missing object", 2024-09-10T18:28:58.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.067474+0000", 2024-09-10T18:28:58.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6084999999999997e-05 2024-09-10T18:28:58.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.070328+0000", 2024-09-10T18:28:58.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028540940000000002 2024-09-10T18:28:58.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-10T18:28:58.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.070421+0000", 2024-09-10T18:28:58.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2822e-05 2024-09-10T18:28:58.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.072747+0000", 2024-09-10T18:28:58.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023261639999999999 2024-09-10T18:28:58.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.072849+0000", 2024-09-10T18:28:58.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010183099999999999 2024-09-10T18:28:58.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:10.073074+0000", 2024-09-10T18:28:58.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022475899999999999 2024-09-10T18:28:58.060 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.060 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.060 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.060 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:28268 7.0 7:03e36e58:::benchmark_data_smithi028_77856_object28267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:28:58.061 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:55.402609+0000", 2024-09-10T18:28:58.061 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 302.62538110499997, 2024-09-10T18:28:58.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.129438948, 2024-09-10T18:28:58.061 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.062 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.062 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.062 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.062 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.062 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 28268 2024-09-10T18:28:58.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.062 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.402609+0000", 2024-09-10T18:28:58.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.402609+0000", 2024-09-10T18:28:58.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.402605+0000", 2024-09-10T18:28:58.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999967 2024-09-10T18:28:58.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.402652+0000", 2024-09-10T18:28:58.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6505000000000003e-05 2024-09-10T18:28:58.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.402674+0000", 2024-09-10T18:28:58.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1889e-05 2024-09-10T18:28:58.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.402682+0000", 2024-09-10T18:28:58.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0129999999999993e-06 2024-09-10T18:28:58.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.402742+0000", 2024-09-10T18:28:58.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0185000000000001e-05 2024-09-10T18:28:58.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.402956+0000", 2024-09-10T18:28:58.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021390999999999999 2024-09-10T18:28:58.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.403124+0000", 2024-09-10T18:28:58.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000168775 2024-09-10T18:28:58.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.405090+0000", 2024-09-10T18:28:58.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001965381 2024-09-10T18:28:58.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.405435+0000", 2024-09-10T18:28:58.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00034498100000000001 2024-09-10T18:28:58.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.531950+0000", 2024-09-10T18:28:58.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.126515715 2024-09-10T18:28:58.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.532005+0000", 2024-09-10T18:28:58.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4333000000000003e-05 2024-09-10T18:28:58.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.532048+0000", 2024-09-10T18:28:58.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2728000000000002e-05 2024-09-10T18:28:58.068 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.068 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.068 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.068 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:28269 7.0 7:077ce142:::benchmark_data_smithi028_77856_object28268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:28:58.068 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:55.403527+0000", 2024-09-10T18:28:58.069 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 302.62446242499999, 2024-09-10T18:28:58.069 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.12895774700000001, 2024-09-10T18:28:58.069 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.069 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.069 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.069 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.069 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.069 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 28269 2024-09-10T18:28:58.069 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.069 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.069 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.069 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.069 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.403527+0000", 2024-09-10T18:28:58.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.403527+0000", 2024-09-10T18:28:58.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.403526+0000", 2024-09-10T18:28:58.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:28:58.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.403593+0000", 2024-09-10T18:28:58.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7466999999999996e-05 2024-09-10T18:28:58.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.403608+0000", 2024-09-10T18:28:58.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4786999999999999e-05 2024-09-10T18:28:58.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.403614+0000", 2024-09-10T18:28:58.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7049999999999996e-06 2024-09-10T18:28:58.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.404139+0000", 2024-09-10T18:28:58.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00052514200000000003 2024-09-10T18:28:58.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.404265+0000", 2024-09-10T18:28:58.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012630900000000001 2024-09-10T18:28:58.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.404407+0000", 2024-09-10T18:28:58.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014217200000000001 2024-09-10T18:28:58.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.073 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.405237+0000", 2024-09-10T18:28:58.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00082998100000000003 2024-09-10T18:28:58.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.073 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.405538+0000", 2024-09-10T18:28:58.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00030039900000000002 2024-09-10T18:28:58.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.073 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.532432+0000", 2024-09-10T18:28:58.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.126894384 2024-09-10T18:28:58.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.532462+0000", 2024-09-10T18:28:58.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9771e-05 2024-09-10T18:28:58.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.532485+0000", 2024-09-10T18:28:58.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3269e-05 2024-09-10T18:28:58.074 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.074 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.075 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.075 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.075 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:28635 7.0 7:067af013:::benchmark_data_smithi028_77856_object28634:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:28:58.075 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:55.726823+0000", 2024-09-10T18:28:58.075 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 302.30116633699998, 2024-09-10T18:28:58.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13670795599999999, 2024-09-10T18:28:58.075 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.075 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.075 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.075 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.075 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.075 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 28635 2024-09-10T18:28:58.076 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.076 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.076 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.076 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.076 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726823+0000", 2024-09-10T18:28:58.076 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.076 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.076 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.076 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.076 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726823+0000", 2024-09-10T18:28:58.076 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.076 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.076 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.076 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726821+0000", 2024-09-10T18:28:58.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:28:58.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726888+0000", 2024-09-10T18:28:58.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7714999999999998e-05 2024-09-10T18:28:58.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726910+0000", 2024-09-10T18:28:58.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1688e-05 2024-09-10T18:28:58.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726917+0000", 2024-09-10T18:28:58.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.934e-06 2024-09-10T18:28:58.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726967+0000", 2024-09-10T18:28:58.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0111000000000002e-05 2024-09-10T18:28:58.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.727088+0000", 2024-09-10T18:28:58.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012124 2024-09-10T18:28:58.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.727225+0000", 2024-09-10T18:28:58.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000136778 2024-09-10T18:28:58.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.728383+0000", 2024-09-10T18:28:58.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011583089999999999 2024-09-10T18:28:58.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.728694+0000", 2024-09-10T18:28:58.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00031094 2024-09-10T18:28:58.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.863440+0000", 2024-09-10T18:28:58.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13474614500000001 2024-09-10T18:28:58.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.863489+0000", 2024-09-10T18:28:58.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8908e-05 2024-09-10T18:28:58.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.863531+0000", 2024-09-10T18:28:58.081 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2027000000000003e-05 2024-09-10T18:28:58.081 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.081 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.081 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.081 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.081 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.081 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:34163 7.0 7:0f36bf7d:::benchmark_data_smithi028_77856_object34162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:28:58.081 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:59.285980+0000", 2024-09-10T18:28:58.081 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 298.74200974000001, 2024-09-10T18:28:58.081 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.10179368, 2024-09-10T18:28:58.081 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.081 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.081 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.081 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.082 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.082 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 34163 2024-09-10T18:28:58.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.082 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.082 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.082 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.285980+0000", 2024-09-10T18:28:58.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.082 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.082 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.285980+0000", 2024-09-10T18:28:58.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.285978+0000", 2024-09-10T18:28:58.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:28:58.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.286030+0000", 2024-09-10T18:28:58.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1989e-05 2024-09-10T18:28:58.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.298958+0000", 2024-09-10T18:28:58.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012928873 2024-09-10T18:28:58.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.298971+0000", 2024-09-10T18:28:58.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2529e-05 2024-09-10T18:28:58.084 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.299196+0000", 2024-09-10T18:28:58.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000225493 2024-09-10T18:28:58.084 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.299321+0000", 2024-09-10T18:28:58.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000124871 2024-09-10T18:28:58.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.085 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.085 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.299457+0000", 2024-09-10T18:28:58.085 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013559299999999999 2024-09-10T18:28:58.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.085 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.085 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.301952+0000", 2024-09-10T18:28:58.085 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024954729999999998 2024-09-10T18:28:58.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.085 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.085 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.301987+0000", 2024-09-10T18:28:58.086 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5036999999999997e-05 2024-09-10T18:28:58.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.086 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.086 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.086 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.387705+0000", 2024-09-10T18:28:58.086 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.085717481999999998 2024-09-10T18:28:58.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.086 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.086 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.086 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.387739+0000", 2024-09-10T18:28:58.086 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.362e-05 2024-09-10T18:28:58.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.086 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.086 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.086 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.387774+0000", 2024-09-10T18:28:58.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5111000000000003e-05 2024-09-10T18:28:58.087 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.087 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.087 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.087 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.087 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:34170 7.0 7:0862d140:::benchmark_data_smithi028_77856_object34169:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:28:58.087 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:59.298997+0000", 2024-09-10T18:28:58.087 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 298.728992699, 2024-09-10T18:28:58.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.088863384000000004, 2024-09-10T18:28:58.087 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.087 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.087 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.087 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.087 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.088 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 34170 2024-09-10T18:28:58.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.088 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.298997+0000", 2024-09-10T18:28:58.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.298997+0000", 2024-09-10T18:28:58.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.298995+0000", 2024-09-10T18:28:58.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:28:58.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.299036+0000", 2024-09-10T18:28:58.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1223999999999997e-05 2024-09-10T18:28:58.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.299045+0000", 2024-09-10T18:28:58.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8610000000000004e-06 2024-09-10T18:28:58.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.299050+0000", 2024-09-10T18:28:58.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5229999999999997e-06 2024-09-10T18:28:58.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.300724+0000", 2024-09-10T18:28:58.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001674306 2024-09-10T18:28:58.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.300838+0000", 2024-09-10T18:28:58.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114033 2024-09-10T18:28:58.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.300939+0000", 2024-09-10T18:28:58.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000100967 2024-09-10T18:28:58.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.302100+0000", 2024-09-10T18:28:58.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011612949999999999 2024-09-10T18:28:58.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.302231+0000", 2024-09-10T18:28:58.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000130648 2024-09-10T18:28:58.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.387806+0000", 2024-09-10T18:28:58.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.085575021000000001 2024-09-10T18:28:58.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.387838+0000", 2024-09-10T18:28:58.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1990999999999999e-05 2024-09-10T18:28:58.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.093 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.093 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.387860+0000", 2024-09-10T18:28:58.093 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2385e-05 2024-09-10T18:28:58.093 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.093 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.093 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.093 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.093 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.093 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:34192 7.0 7:04307710:::benchmark_data_smithi028_77856_object34191:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:28:58.093 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:59.303363+0000", 2024-09-10T18:28:58.093 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 298.72462715500001, 2024-09-10T18:28:58.093 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.087319685999999994, 2024-09-10T18:28:58.093 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.093 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.094 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.094 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.094 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.094 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 34192 2024-09-10T18:28:58.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.094 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303363+0000", 2024-09-10T18:28:58.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303363+0000", 2024-09-10T18:28:58.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303360+0000", 2024-09-10T18:28:58.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:28:58.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303422+0000", 2024-09-10T18:28:58.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1478999999999999e-05 2024-09-10T18:28:58.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303433+0000", 2024-09-10T18:28:58.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1136e-05 2024-09-10T18:28:58.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303438+0000", 2024-09-10T18:28:58.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2979999999999999e-06 2024-09-10T18:28:58.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303469+0000", 2024-09-10T18:28:58.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1127000000000002e-05 2024-09-10T18:28:58.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303545+0000", 2024-09-10T18:28:58.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5278999999999995e-05 2024-09-10T18:28:58.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.303621+0000", 2024-09-10T18:28:58.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6248999999999995e-05 2024-09-10T18:28:58.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.304592+0000", 2024-09-10T18:28:58.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00097132200000000005 2024-09-10T18:28:58.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.304991+0000", 2024-09-10T18:28:58.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00039831900000000001 2024-09-10T18:28:58.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.390596+0000", 2024-09-10T18:28:58.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.085605590999999995 2024-09-10T18:28:58.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.390647+0000", 2024-09-10T18:28:58.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0890000000000002e-05 2024-09-10T18:28:58.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:59.390682+0000", 2024-09-10T18:28:58.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5182000000000003e-05 2024-09-10T18:28:58.099 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.099 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.099 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.099 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:37019 7.0 7:0101e7cb:::benchmark_data_smithi028_77856_object37018:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-10T18:28:58.099 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:24:01.213014+0000", 2024-09-10T18:28:58.099 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 296.81497594000001, 2024-09-10T18:28:58.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.088142556999999996, 2024-09-10T18:28:58.099 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.100 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.100 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.100 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.100 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.100 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 37019 2024-09-10T18:28:58.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.100 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213014+0000", 2024-09-10T18:28:58.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213014+0000", 2024-09-10T18:28:58.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213012+0000", 2024-09-10T18:28:58.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:28:58.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213069+0000", 2024-09-10T18:28:58.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7249e-05 2024-09-10T18:28:58.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213082+0000", 2024-09-10T18:28:58.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2611e-05 2024-09-10T18:28:58.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213087+0000", 2024-09-10T18:28:58.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7359999999999998e-06 2024-09-10T18:28:58.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213127+0000", 2024-09-10T18:28:58.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9477999999999997e-05 2024-09-10T18:28:58.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213242+0000", 2024-09-10T18:28:58.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114975 2024-09-10T18:28:58.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213368+0000", 2024-09-10T18:28:58.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012595799999999999 2024-09-10T18:28:58.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.215333+0000", 2024-09-10T18:28:58.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019656859999999999 2024-09-10T18:28:58.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.217238+0000", 2024-09-10T18:28:58.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001904406 2024-09-10T18:28:58.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.301068+0000", 2024-09-10T18:28:58.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.083830528000000001 2024-09-10T18:28:58.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.301119+0000", 2024-09-10T18:28:58.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0800000000000002e-05 2024-09-10T18:28:58.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.301156+0000", 2024-09-10T18:28:58.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7094000000000001e-05 2024-09-10T18:28:58.105 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.106 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.106 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.106 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:37020 7.0 7:06b186a2:::benchmark_data_smithi028_77856_object37019:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-10T18:28:58.106 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:24:01.213097+0000", 2024-09-10T18:28:58.106 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 296.81489253299998, 2024-09-10T18:28:58.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.088148546999999994, 2024-09-10T18:28:58.106 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.106 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.106 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.106 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.106 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.106 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 37020 2024-09-10T18:28:58.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.107 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.107 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.107 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213097+0000", 2024-09-10T18:28:58.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.107 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.107 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213097+0000", 2024-09-10T18:28:58.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.107 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213096+0000", 2024-09-10T18:28:58.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:28:58.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213132+0000", 2024-09-10T18:28:58.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5602000000000002e-05 2024-09-10T18:28:58.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213137+0000", 2024-09-10T18:28:58.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6770000000000003e-06 2024-09-10T18:28:58.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.213140+0000", 2024-09-10T18:28:58.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1250000000000001e-06 2024-09-10T18:28:58.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.214360+0000", 2024-09-10T18:28:58.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0012207539999999999 2024-09-10T18:28:58.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.214483+0000", 2024-09-10T18:28:58.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012256499999999999 2024-09-10T18:28:58.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.214628+0000", 2024-09-10T18:28:58.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000144602 2024-09-10T18:28:58.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.215466+0000", 2024-09-10T18:28:58.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000838586 2024-09-10T18:28:58.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.217618+0000", 2024-09-10T18:28:58.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002152284 2024-09-10T18:28:58.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.301194+0000", 2024-09-10T18:28:58.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.083575682999999998 2024-09-10T18:28:58.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.301225+0000", 2024-09-10T18:28:58.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0471e-05 2024-09-10T18:28:58.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:01.301246+0000", 2024-09-10T18:28:58.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.109e-05 2024-09-10T18:28:58.112 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.112 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.112 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.112 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:81102 7.0 7:030c6891:::benchmark_data_smithi028_77856_object81101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-10T18:28:58.112 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:24:34.410675+0000", 2024-09-10T18:28:58.112 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 263.61731459100002, 2024-09-10T18:28:58.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.100851274, 2024-09-10T18:28:58.112 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.112 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.112 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.113 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.113 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.113 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 81102 2024-09-10T18:28:58.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.113 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.410675+0000", 2024-09-10T18:28:58.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.410675+0000", 2024-09-10T18:28:58.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.410672+0000", 2024-09-10T18:28:58.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:28:58.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.410768+0000", 2024-09-10T18:28:58.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5235000000000006e-05 2024-09-10T18:28:58.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.410780+0000", 2024-09-10T18:28:58.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2145e-05 2024-09-10T18:28:58.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.410785+0000", 2024-09-10T18:28:58.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3739999999999997e-06 2024-09-10T18:28:58.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.412516+0000", 2024-09-10T18:28:58.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0017309490000000001 2024-09-10T18:28:58.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.412632+0000", 2024-09-10T18:28:58.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011565699999999999 2024-09-10T18:28:58.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.412731+0000", 2024-09-10T18:28:58.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9159000000000004e-05 2024-09-10T18:28:58.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.415836+0000", 2024-09-10T18:28:58.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0031046379999999998 2024-09-10T18:28:58.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.422925+0000", 2024-09-10T18:28:58.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0070893240000000001 2024-09-10T18:28:58.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:58.118 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:58.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:28:58.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.511458+0000", 2024-09-10T18:28:58.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.088532936000000007 2024-09-10T18:28:58.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.511492+0000", 2024-09-10T18:28:58.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4279999999999997e-05 2024-09-10T18:28:58.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.511526+0000", 2024-09-10T18:28:58.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4341000000000002e-05 2024-09-10T18:28:58.120 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.120 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.120 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.120 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:81105 7.0 7:021bcade:::benchmark_data_smithi028_77856_object81104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-10T18:28:58.120 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:24:34.411681+0000", 2024-09-10T18:28:58.120 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 263.616309097, 2024-09-10T18:28:58.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.102490836, 2024-09-10T18:28:58.121 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.121 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.121 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.121 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.121 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.121 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 81105 2024-09-10T18:28:58.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.121 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.411681+0000", 2024-09-10T18:28:58.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.411681+0000", 2024-09-10T18:28:58.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.411679+0000", 2024-09-10T18:28:58.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:28:58.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.411748+0000", 2024-09-10T18:28:58.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9499999999999995e-05 2024-09-10T18:28:58.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.411761+0000", 2024-09-10T18:28:58.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2789000000000001e-05 2024-09-10T18:28:58.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.411766+0000", 2024-09-10T18:28:58.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.57e-06 2024-09-10T18:28:58.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.414466+0000", 2024-09-10T18:28:58.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002699786 2024-09-10T18:28:58.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.414542+0000", 2024-09-10T18:28:58.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5507999999999996e-05 2024-09-10T18:28:58.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.414624+0000", 2024-09-10T18:28:58.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2033999999999994e-05 2024-09-10T18:28:58.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.415986+0000", 2024-09-10T18:28:58.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013624729999999999 2024-09-10T18:28:58.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.422959+0000", 2024-09-10T18:28:58.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0069727890000000001 2024-09-10T18:28:58.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.514112+0000", 2024-09-10T18:28:58.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.091152601 2024-09-10T18:28:58.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.126 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.126 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.514149+0000", 2024-09-10T18:28:58.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6931000000000003e-05 2024-09-10T18:28:58.126 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.126 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.126 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:24:34.514171+0000", 2024-09-10T18:28:58.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2881999999999999e-05 2024-09-10T18:28:58.126 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.126 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.126 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.126 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.127 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:121400 7.0 7:0b67f93f:::benchmark_data_smithi028_77856_object300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:28:58.127 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:25:14.170420+0000", 2024-09-10T18:28:58.127 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 223.857569434, 2024-09-10T18:28:58.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.098332630000000004, 2024-09-10T18:28:58.127 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.127 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.127 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.127 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.127 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.127 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 121400 2024-09-10T18:28:58.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.127 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170420+0000", 2024-09-10T18:28:58.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170420+0000", 2024-09-10T18:28:58.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170419+0000", 2024-09-10T18:28:58.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:28:58.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170424+0000", 2024-09-10T18:28:58.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4879999999999998e-06 2024-09-10T18:28:58.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170432+0000", 2024-09-10T18:28:58.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.559e-06 2024-09-10T18:28:58.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170435+0000", 2024-09-10T18:28:58.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3620000000000002e-06 2024-09-10T18:28:58.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170478+0000", 2024-09-10T18:28:58.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2814e-05 2024-09-10T18:28:58.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170608+0000", 2024-09-10T18:28:58.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012954500000000001 2024-09-10T18:28:58.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170733+0000", 2024-09-10T18:28:58.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000124855 2024-09-10T18:28:58.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.174781+0000", 2024-09-10T18:28:58.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040485110000000003 2024-09-10T18:28:58.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.178138+0000", 2024-09-10T18:28:58.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003357077 2024-09-10T18:28:58.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.268672+0000", 2024-09-10T18:28:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.090533708000000004 2024-09-10T18:28:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.268726+0000", 2024-09-10T18:28:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4499000000000001e-05 2024-09-10T18:28:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.268753+0000", 2024-09-10T18:28:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6543e-05 2024-09-10T18:28:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.132 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:121401 7.0 7:06878fa9:::benchmark_data_smithi028_77856_object301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:28:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:25:14.170563+0000", 2024-09-10T18:28:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 223.857426756, 2024-09-10T18:28:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.10107751800000001, 2024-09-10T18:28:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 121401 2024-09-10T18:28:58.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170563+0000", 2024-09-10T18:28:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170563+0000", 2024-09-10T18:28:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170562+0000", 2024-09-10T18:28:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999995 2024-09-10T18:28:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170565+0000", 2024-09-10T18:28:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.52e-06 2024-09-10T18:28:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170571+0000", 2024-09-10T18:28:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6829999999999996e-06 2024-09-10T18:28:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.170573+0000", 2024-09-10T18:28:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6000000000000001e-06 2024-09-10T18:28:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.172895+0000", 2024-09-10T18:28:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023221980000000001 2024-09-10T18:28:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.172991+0000", 2024-09-10T18:28:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5458000000000007e-05 2024-09-10T18:28:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.173075+0000", 2024-09-10T18:28:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3683000000000005e-05 2024-09-10T18:28:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.174905+0000", 2024-09-10T18:28:58.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001829991 2024-09-10T18:28:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.178179+0000", 2024-09-10T18:28:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032743680000000002 2024-09-10T18:28:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.271586+0000", 2024-09-10T18:28:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.093406792000000002 2024-09-10T18:28:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.271617+0000", 2024-09-10T18:28:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1347999999999999e-05 2024-09-10T18:28:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:25:14.271640+0000", 2024-09-10T18:28:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3400999999999998e-05 2024-09-10T18:28:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:177912 7.0 7:02191cc8:::benchmark_data_smithi028_77856_object56812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-09-10T18:28:58.139 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:22.002506+0000", 2024-09-10T18:28:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 156.02548375800001, 2024-09-10T18:28:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.095129684000000006, 2024-09-10T18:28:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 177912 2024-09-10T18:28:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:22.002506+0000", 2024-09-10T18:28:58.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:22.002506+0000", 2024-09-10T18:28:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:22.002504+0000", 2024-09-10T18:28:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:28:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:22.002511+0000", 2024-09-10T18:28:58.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5939999999999996e-06 2024-09-10T18:28:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:22.002522+0000", 2024-09-10T18:28:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0489000000000001e-05 2024-09-10T18:28:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:22.002527+0000", 2024-09-10T18:28:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2480000000000004e-06 2024-09-10T18:28:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:22.002572+0000", 2024-09-10T18:28:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4947000000000002e-05 2024-09-10T18:28:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:22.002691+0000", 2024-09-10T18:28:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011857 2024-09-10T18:28:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:22.002809+0000", 2024-09-10T18:28:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000118467 2024-09-10T18:28:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:22.005253+0000", 2024-09-10T18:28:58.144 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024438749999999999 2024-09-10T18:28:58.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.144 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.144 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:22.005815+0000", 2024-09-10T18:28:58.144 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00056206099999999998 2024-09-10T18:28:58.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.144 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.144 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:22.097545+0000", 2024-09-10T18:28:58.144 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.091729665000000002 2024-09-10T18:28:58.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.144 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:22.097598+0000", 2024-09-10T18:28:58.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3470000000000001e-05 2024-09-10T18:28:58.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:22.097636+0000", 2024-09-10T18:28:58.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7552000000000002e-05 2024-09-10T18:28:58.145 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.145 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.145 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.145 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:181517 7.0 7:02f1c46a:::benchmark_data_smithi028_77856_object60417:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:28:58.145 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:25.450852+0000", 2024-09-10T18:28:58.145 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 152.577137244, 2024-09-10T18:28:58.146 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.128967259, 2024-09-10T18:28:58.146 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.146 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.146 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.146 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.146 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.146 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 181517 2024-09-10T18:28:58.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.146 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.146 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.146 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.146 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.450852+0000", 2024-09-10T18:28:58.146 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.450852+0000", 2024-09-10T18:28:58.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.147 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.450850+0000", 2024-09-10T18:28:58.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:28:58.147 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.450859+0000", 2024-09-10T18:28:58.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6580000000000008e-06 2024-09-10T18:28:58.147 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.148 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.148 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.450885+0000", 2024-09-10T18:28:58.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5786e-05 2024-09-10T18:28:58.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.148 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.148 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.450891+0000", 2024-09-10T18:28:58.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5220000000000004e-06 2024-09-10T18:28:58.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.148 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.148 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.450939+0000", 2024-09-10T18:28:58.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7524000000000001e-05 2024-09-10T18:28:58.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.149 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.149 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.149 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451073+0000", 2024-09-10T18:28:58.149 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000134005 2024-09-10T18:28:58.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.149 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.149 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.149 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451189+0000", 2024-09-10T18:28:58.149 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116179 2024-09-10T18:28:58.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.149 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.149 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.149 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.454148+0000", 2024-09-10T18:28:58.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029589640000000001 2024-09-10T18:28:58.150 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.150 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.455956+0000", 2024-09-10T18:28:58.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001808353 2024-09-10T18:28:58.150 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.150 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.579725+0000", 2024-09-10T18:28:58.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.123768528 2024-09-10T18:28:58.150 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.579775+0000", 2024-09-10T18:28:58.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9988000000000003e-05 2024-09-10T18:28:58.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.579820+0000", 2024-09-10T18:28:58.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4960999999999999e-05 2024-09-10T18:28:58.151 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.151 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.151 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.151 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:220607 7.0 7:0ca55e40:::benchmark_data_smithi028_77856_object99507:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-10T18:28:58.151 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:07.230450+0000", 2024-09-10T18:28:58.152 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 110.79753934599999, 2024-09-10T18:28:58.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.090385495999999996, 2024-09-10T18:28:58.152 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.152 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.152 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.152 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.152 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.152 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 220607 2024-09-10T18:28:58.152 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.152 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.152 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.152 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.230450+0000", 2024-09-10T18:28:58.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.230450+0000", 2024-09-10T18:28:58.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.230448+0000", 2024-09-10T18:28:58.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:28:58.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.230456+0000", 2024-09-10T18:28:58.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1049999999999995e-06 2024-09-10T18:28:58.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.230473+0000", 2024-09-10T18:28:58.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7071999999999998e-05 2024-09-10T18:28:58.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.230480+0000", 2024-09-10T18:28:58.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9500000000000004e-06 2024-09-10T18:28:58.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.230537+0000", 2024-09-10T18:28:58.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6870999999999998e-05 2024-09-10T18:28:58.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.230693+0000", 2024-09-10T18:28:58.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000156133 2024-09-10T18:28:58.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.230812+0000", 2024-09-10T18:28:58.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011869 2024-09-10T18:28:58.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.156 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.233201+0000", 2024-09-10T18:28:58.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023884399999999999 2024-09-10T18:28:58.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.156 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.156 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.233912+0000", 2024-09-10T18:28:58.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000711438 2024-09-10T18:28:58.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.156 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.156 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.320757+0000", 2024-09-10T18:28:58.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.086845336999999995 2024-09-10T18:28:58.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.157 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.157 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.320797+0000", 2024-09-10T18:28:58.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9783999999999997e-05 2024-09-10T18:28:58.157 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.157 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.157 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.320836+0000", 2024-09-10T18:28:58.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8732000000000001e-05 2024-09-10T18:28:58.157 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.157 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.157 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.157 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.157 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:220776 7.0 7:00b848a0:::benchmark_data_smithi028_77856_object99676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-10T18:28:58.158 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:07.473694+0000", 2024-09-10T18:28:58.158 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 110.55429538, 2024-09-10T18:28:58.158 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.094466889999999998, 2024-09-10T18:28:58.158 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.158 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.158 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.158 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.158 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.158 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 220776 2024-09-10T18:28:58.158 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.158 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.158 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.158 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.158 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.473694+0000", 2024-09-10T18:28:58.158 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.473694+0000", 2024-09-10T18:28:58.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.473694+0000", 2024-09-10T18:28:58.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999995 2024-09-10T18:28:58.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.473697+0000", 2024-09-10T18:28:58.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3739999999999998e-06 2024-09-10T18:28:58.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.160 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.473703+0000", 2024-09-10T18:28:58.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1480000000000003e-06 2024-09-10T18:28:58.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.160 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.473707+0000", 2024-09-10T18:28:58.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5769999999999998e-06 2024-09-10T18:28:58.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.476587+0000", 2024-09-10T18:28:58.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028805670000000001 2024-09-10T18:28:58.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.476718+0000", 2024-09-10T18:28:58.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013066999999999999 2024-09-10T18:28:58.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.476838+0000", 2024-09-10T18:28:58.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119665 2024-09-10T18:28:58.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.162 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.162 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.482040+0000", 2024-09-10T18:28:58.162 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0052027640000000003 2024-09-10T18:28:58.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.162 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.162 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.525062+0000", 2024-09-10T18:28:58.162 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.043021829999999997 2024-09-10T18:28:58.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.162 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.162 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.568055+0000", 2024-09-10T18:28:58.162 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.042992520999999999 2024-09-10T18:28:58.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.163 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.163 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.568114+0000", 2024-09-10T18:28:58.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8949000000000001e-05 2024-09-10T18:28:58.163 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.163 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.163 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.163 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.568161+0000", 2024-09-10T18:28:58.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7475000000000002e-05 2024-09-10T18:28:58.163 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.163 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.163 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.163 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.163 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.163 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:220777 7.0 7:02ca24c8:::benchmark_data_smithi028_77856_object99677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-10T18:28:58.164 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:07.475244+0000", 2024-09-10T18:28:58.164 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 110.552745269, 2024-09-10T18:28:58.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.093096451999999996, 2024-09-10T18:28:58.164 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.164 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.164 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.164 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.164 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.164 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 220777 2024-09-10T18:28:58.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.164 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.475244+0000", 2024-09-10T18:28:58.165 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.165 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.165 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.165 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.475244+0000", 2024-09-10T18:28:58.165 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.165 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.165 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.165 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.475243+0000", 2024-09-10T18:28:58.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:28:58.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.166 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.166 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.166 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.475252+0000", 2024-09-10T18:28:58.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7809999999999994e-06 2024-09-10T18:28:58.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.166 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.166 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.166 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.475264+0000", 2024-09-10T18:28:58.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.19e-05 2024-09-10T18:28:58.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.475269+0000", 2024-09-10T18:28:58.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.558e-06 2024-09-10T18:28:58.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.479118+0000", 2024-09-10T18:28:58.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038493199999999998 2024-09-10T18:28:58.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.479249+0000", 2024-09-10T18:28:58.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000130829 2024-09-10T18:28:58.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.168 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.479377+0000", 2024-09-10T18:28:58.168 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012802899999999999 2024-09-10T18:28:58.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.168 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.482116+0000", 2024-09-10T18:28:58.168 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027392050000000002 2024-09-10T18:28:58.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.168 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.525104+0000", 2024-09-10T18:28:58.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.042987297000000001 2024-09-10T18:28:58.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.169 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.568209+0000", 2024-09-10T18:28:58.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.043105533000000001 2024-09-10T18:28:58.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.169 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.568287+0000", 2024-09-10T18:28:58.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7535999999999998e-05 2024-09-10T18:28:58.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.170 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.568341+0000", 2024-09-10T18:28:58.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4070999999999998e-05 2024-09-10T18:28:58.170 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.170 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.170 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.170 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.170 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:220786 7.0 7:0173fda9:::benchmark_data_smithi028_77856_object99686:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-10T18:28:58.170 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:07.484649+0000", 2024-09-10T18:28:58.170 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 110.543340493, 2024-09-10T18:28:58.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.088023694, 2024-09-10T18:28:58.170 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:28:58.170 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:28:58.170 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:28:58.171 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:28:58.171 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:28:58.171 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 220786 2024-09-10T18:28:58.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.171 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:28:58.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:28:58.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.484649+0000", 2024-09-10T18:28:58.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:28:58.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.484649+0000", 2024-09-10T18:28:58.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:28:58.172 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.172 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.172 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:28:58.172 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.484644+0000", 2024-09-10T18:28:58.172 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999943 2024-09-10T18:28:58.172 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.172 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.172 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:28:58.172 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.484654+0000", 2024-09-10T18:28:58.172 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0023e-05 2024-09-10T18:28:58.172 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.172 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.172 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:28:58.172 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.484667+0000", 2024-09-10T18:28:58.173 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2989999999999999e-05 2024-09-10T18:28:58.173 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.173 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.173 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:28:58.173 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.484673+0000", 2024-09-10T18:28:58.173 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.285e-06 2024-09-10T18:28:58.173 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.173 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.173 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:28:58.173 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.484713+0000", 2024-09-10T18:28:58.173 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0451000000000003e-05 2024-09-10T18:28:58.173 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.173 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.174 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:28:58.174 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.484840+0000", 2024-09-10T18:28:58.174 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000126391 2024-09-10T18:28:58.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.174 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.174 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:28:58.174 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.484964+0000", 2024-09-10T18:28:58.174 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012374700000000001 2024-09-10T18:28:58.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.174 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.174 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:28:58.174 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.487572+0000", 2024-09-10T18:28:58.174 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0026082570000000001 2024-09-10T18:28:58.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.528637+0000", 2024-09-10T18:28:58.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.041065108000000003 2024-09-10T18:28:58.175 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:28:58.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.572572+0000", 2024-09-10T18:28:58.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.043935506999999999 2024-09-10T18:28:58.175 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:28:58.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.572619+0000", 2024-09-10T18:28:58.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6341000000000002e-05 2024-09-10T18:28:58.176 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:28:58.176 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:28:58.176 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:28:58.176 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:07.572673+0000", 2024-09-10T18:28:58.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4141999999999998e-05 2024-09-10T18:28:58.176 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.176 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.176 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.176 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:28:58.176 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:28:58.176 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:28:58.177 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 51 16 88735 88719 108.692 79.6875 0.0128239 0.009195 2024-09-10T18:28:58.177 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:58 smithi204 ceph-mon[32357]: 6.1c scrub starts 2024-09-10T18:28:58.177 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:58 smithi204 ceph-mon[32357]: 6.1c scrub ok 2024-09-10T18:28:58.177 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:58 smithi204 ceph-mon[32357]: 5.f scrub starts 2024-09-10T18:28:58.177 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:58 smithi204 ceph-mon[32357]: 5.f scrub ok 2024-09-10T18:28:58.177 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:58 smithi204 ceph-mon[32357]: pgmap v333: 158 pgs: 158 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-10T18:28:58.177 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:58 smithi204 ceph-mon[32357]: 3.1b scrub starts 2024-09-10T18:28:58.177 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:58 smithi204 ceph-mon[32357]: 3.1b scrub ok 2024-09-10T18:28:58.178 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:58 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:28:58.178 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:58 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:28:58.178 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:58 smithi204 ceph-mon[32357]: from='client.35081 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:28:58.178 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:58 smithi204 ceph-mon[32357]: osdmap e202: 8 total, 8 up, 8 in 2024-09-10T18:28:58.178 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:58 smithi204 ceph-mon[32357]: 6.11 deep-scrub starts 2024-09-10T18:28:58.178 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:58 smithi204 ceph-mon[32357]: 6.11 deep-scrub ok 2024-09-10T18:28:58.219 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:58.248 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:58.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:58.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:58.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:58.387 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:58.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:58.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:58.489 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:28:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[39220]: 6.1c scrub starts 2024-09-10T18:28:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[39220]: 6.1c scrub ok 2024-09-10T18:28:58.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[39220]: 5.f scrub starts 2024-09-10T18:28:58.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[39220]: 5.f scrub ok 2024-09-10T18:28:58.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[39220]: pgmap v333: 158 pgs: 158 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-10T18:28:58.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[39220]: 3.1b scrub starts 2024-09-10T18:28:58.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[39220]: 3.1b scrub ok 2024-09-10T18:28:58.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:28:58.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:28:58.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[39220]: from='client.35081 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:28:58.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[39220]: osdmap e202: 8 total, 8 up, 8 in 2024-09-10T18:28:58.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[39220]: 6.11 deep-scrub starts 2024-09-10T18:28:58.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[39220]: 6.11 deep-scrub ok 2024-09-10T18:28:58.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[29291]: 6.1c scrub starts 2024-09-10T18:28:58.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[29291]: 6.1c scrub ok 2024-09-10T18:28:58.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[29291]: 5.f scrub starts 2024-09-10T18:28:58.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[29291]: 5.f scrub ok 2024-09-10T18:28:58.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[29291]: pgmap v333: 158 pgs: 158 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-10T18:28:58.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[29291]: 3.1b scrub starts 2024-09-10T18:28:58.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[29291]: 3.1b scrub ok 2024-09-10T18:28:58.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:28:58.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:28:58.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[29291]: from='client.35081 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:28:58.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[29291]: osdmap e202: 8 total, 8 up, 8 in 2024-09-10T18:28:58.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[29291]: 6.11 deep-scrub starts 2024-09-10T18:28:58.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:58 smithi028 ceph-mon[29291]: 6.11 deep-scrub ok 2024-09-10T18:28:58.517 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:58.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:58.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:28:58.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:58.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:58.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:58.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:58.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:58.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:58.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:58.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:58.950 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:58.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:58.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:59.052 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:28:59.085 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:59.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:59.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:28:59.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 52 16 90033 90017 108.162 81.125 0.0102772 0.00924103 2024-09-10T18:28:59.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:59.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:59.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:59 smithi028 ceph-mon[39220]: 5.19 scrub starts 2024-09-10T18:28:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:59 smithi028 ceph-mon[39220]: 5.19 scrub ok 2024-09-10T18:28:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:59 smithi028 ceph-mon[39220]: 3.12 scrub starts 2024-09-10T18:28:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:59 smithi028 ceph-mon[39220]: 3.12 scrub ok 2024-09-10T18:28:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:59 smithi028 ceph-mon[39220]: 4.1f scrub starts 2024-09-10T18:28:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:59 smithi028 ceph-mon[39220]: 5.1 scrub starts 2024-09-10T18:28:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:59 smithi028 ceph-mon[39220]: 4.1f scrub ok 2024-09-10T18:28:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:28:59 smithi028 ceph-mon[39220]: 5.1 scrub ok 2024-09-10T18:28:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:59 smithi028 ceph-mon[29291]: 5.19 scrub starts 2024-09-10T18:28:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:59 smithi028 ceph-mon[29291]: 5.19 scrub ok 2024-09-10T18:28:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:59 smithi028 ceph-mon[29291]: 3.12 scrub starts 2024-09-10T18:28:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:59 smithi028 ceph-mon[29291]: 3.12 scrub ok 2024-09-10T18:28:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:59 smithi028 ceph-mon[29291]: 4.1f scrub starts 2024-09-10T18:28:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:59 smithi028 ceph-mon[29291]: 5.1 scrub starts 2024-09-10T18:28:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:59 smithi028 ceph-mon[29291]: 4.1f scrub ok 2024-09-10T18:28:59.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:28:59 smithi028 ceph-mon[29291]: 5.1 scrub ok 2024-09-10T18:28:59.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:59 smithi204 ceph-mon[32357]: 5.19 scrub starts 2024-09-10T18:28:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:59 smithi204 ceph-mon[32357]: 5.19 scrub ok 2024-09-10T18:28:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:59 smithi204 ceph-mon[32357]: 3.12 scrub starts 2024-09-10T18:28:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:59 smithi204 ceph-mon[32357]: 3.12 scrub ok 2024-09-10T18:28:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:59 smithi204 ceph-mon[32357]: 4.1f scrub starts 2024-09-10T18:28:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:59 smithi204 ceph-mon[32357]: 5.1 scrub starts 2024-09-10T18:28:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:59 smithi204 ceph-mon[32357]: 4.1f scrub ok 2024-09-10T18:28:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:28:59 smithi204 ceph-mon[32357]: 5.1 scrub ok 2024-09-10T18:28:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:59.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:59.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:28:59.427 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-10T18:28:59.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:28:59.556 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:59.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:28:59.658 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:28:59.693 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:28:59.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:59.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:28:59.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:28:59.823 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:28:59.834 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:59.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:59.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:28:59.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:28:59.970 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:28:59.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:28:59.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:00.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:00.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:00.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:00.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 53 16 91252 91236 107.559 76.1875 0.0108278 0.00929251 2024-09-10T18:29:00.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:00.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:00 smithi204 ceph-mon[32357]: 5.9 scrub starts 2024-09-10T18:29:00.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:00 smithi204 ceph-mon[32357]: 5.9 scrub ok 2024-09-10T18:29:00.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:00 smithi204 ceph-mon[32357]: 4.7 scrub starts 2024-09-10T18:29:00.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:00 smithi204 ceph-mon[32357]: 4.7 scrub ok 2024-09-10T18:29:00.255 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:00 smithi204 ceph-mon[32357]: pgmap v335: 158 pgs: 158 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-10T18:29:00.259 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:00.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:00.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:00.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:00 smithi028 ceph-mon[29291]: 5.9 scrub starts 2024-09-10T18:29:00.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:00 smithi028 ceph-mon[29291]: 5.9 scrub ok 2024-09-10T18:29:00.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:00 smithi028 ceph-mon[29291]: 4.7 scrub starts 2024-09-10T18:29:00.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:00 smithi028 ceph-mon[29291]: 4.7 scrub ok 2024-09-10T18:29:00.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:00 smithi028 ceph-mon[29291]: pgmap v335: 158 pgs: 158 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-10T18:29:00.293 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:00 smithi028 ceph-mon[39220]: 5.9 scrub starts 2024-09-10T18:29:00.293 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:00 smithi028 ceph-mon[39220]: 5.9 scrub ok 2024-09-10T18:29:00.293 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:00 smithi028 ceph-mon[39220]: 4.7 scrub starts 2024-09-10T18:29:00.293 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:00 smithi028 ceph-mon[39220]: 4.7 scrub ok 2024-09-10T18:29:00.293 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:00 smithi028 ceph-mon[39220]: pgmap v335: 158 pgs: 158 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-10T18:29:00.338 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:29:00.338 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-10T18:29:00.338 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:29:00.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:00.389 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:00.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:00.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:00.490 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:00.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:00.619 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:00.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:00.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:00.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:00.760 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:00.796 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:00.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:00.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:00.898 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:00.923 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:00.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:00.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:01.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:01.065 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:01.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:01.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 54 16 92424 92408 106.923 73.25 0.0133084 0.00934806 2024-09-10T18:29:01.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:01.209 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:01.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:01.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:01.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:01 smithi204 ceph-mon[32357]: 3.13 scrub starts 2024-09-10T18:29:01.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:01 smithi204 ceph-mon[32357]: 3.13 scrub ok 2024-09-10T18:29:01.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:01 smithi204 ceph-mon[32357]: 5.3 scrub starts 2024-09-10T18:29:01.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:01 smithi204 ceph-mon[32357]: 5.3 scrub ok 2024-09-10T18:29:01.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:01 smithi204 ceph-mon[32357]: 4.d scrub starts 2024-09-10T18:29:01.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:01 smithi204 ceph-mon[32357]: 4.d scrub ok 2024-09-10T18:29:01.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:01 smithi204 ceph-mon[32357]: osdmap e203: 8 total, 8 up, 8 in 2024-09-10T18:29:01.311 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:01.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:01 smithi028 ceph-mon[29291]: 3.13 scrub starts 2024-09-10T18:29:01.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:01 smithi028 ceph-mon[29291]: 3.13 scrub ok 2024-09-10T18:29:01.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:01 smithi028 ceph-mon[29291]: 5.3 scrub starts 2024-09-10T18:29:01.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:01 smithi028 ceph-mon[29291]: 5.3 scrub ok 2024-09-10T18:29:01.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:01 smithi028 ceph-mon[29291]: 4.d scrub starts 2024-09-10T18:29:01.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:01 smithi028 ceph-mon[29291]: 4.d scrub ok 2024-09-10T18:29:01.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:01 smithi028 ceph-mon[29291]: osdmap e203: 8 total, 8 up, 8 in 2024-09-10T18:29:01.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:01 smithi028 ceph-mon[39220]: 3.13 scrub starts 2024-09-10T18:29:01.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:01 smithi028 ceph-mon[39220]: 3.13 scrub ok 2024-09-10T18:29:01.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:01 smithi028 ceph-mon[39220]: 5.3 scrub starts 2024-09-10T18:29:01.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:01 smithi028 ceph-mon[39220]: 5.3 scrub ok 2024-09-10T18:29:01.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:01 smithi028 ceph-mon[39220]: 4.d scrub starts 2024-09-10T18:29:01.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:01 smithi028 ceph-mon[39220]: 4.d scrub ok 2024-09-10T18:29:01.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:01 smithi028 ceph-mon[39220]: osdmap e203: 8 total, 8 up, 8 in 2024-09-10T18:29:01.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:29:01.340 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:01.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:01.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:01.402 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:29:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:29:01.435 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:29:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:29:01.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:01.471 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:29:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:29:01.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:01.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:01.507 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:29:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:29:01.541 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:29:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:29:01.572 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:29:01.572 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:29:01.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:01.606 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:29:01.606 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:29:01.606 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:29:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:29:01] "GET /metrics HTTP/1.1" 200 224256 "" "Prometheus/2.43.0" 2024-09-10T18:29:01.629 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:01.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:01.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:01.641 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:29:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:29:01.721 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:29:01.721 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:29:01.731 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:01.766 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:01.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:01.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:01.868 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:01.898 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:01.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:01.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:02.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:02.042 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:02.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:02.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:02.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 55 16 93589 93573 106.303 72.8125 0.0122527 0.00940218 2024-09-10T18:29:02.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:02.182 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:02.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:02.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:02.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:02 smithi028 ceph-mon[39220]: pgmap v337: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 111 MiB/s wr, 1.78k op/s 2024-09-10T18:29:02.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:02 smithi028 ceph-mon[39220]: 4.3 scrub starts 2024-09-10T18:29:02.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:02 smithi028 ceph-mon[39220]: 4.3 scrub ok 2024-09-10T18:29:02.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:02 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3996816562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:29:02.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:02 smithi028 ceph-mon[39220]: 4.14 scrub starts 2024-09-10T18:29:02.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:02 smithi028 ceph-mon[39220]: 4.14 scrub ok 2024-09-10T18:29:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:02 smithi028 ceph-mon[29291]: pgmap v337: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 111 MiB/s wr, 1.78k op/s 2024-09-10T18:29:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:02 smithi028 ceph-mon[29291]: 4.3 scrub starts 2024-09-10T18:29:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:02 smithi028 ceph-mon[29291]: 4.3 scrub ok 2024-09-10T18:29:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:02 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3996816562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:29:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:02 smithi028 ceph-mon[29291]: 4.14 scrub starts 2024-09-10T18:29:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:02 smithi028 ceph-mon[29291]: 4.14 scrub ok 2024-09-10T18:29:02.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:02.314 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:02.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:02.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:02 smithi204 ceph-mon[32357]: pgmap v337: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 111 MiB/s wr, 1.78k op/s 2024-09-10T18:29:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:02 smithi204 ceph-mon[32357]: 4.3 scrub starts 2024-09-10T18:29:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:02 smithi204 ceph-mon[32357]: 4.3 scrub ok 2024-09-10T18:29:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:02 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3996816562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:29:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:02 smithi204 ceph-mon[32357]: 4.14 scrub starts 2024-09-10T18:29:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:02 smithi204 ceph-mon[32357]: 4.14 scrub ok 2024-09-10T18:29:02.355 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:29:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:29:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:29:02.416 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:02.432 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-10T18:29:02.448 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:02.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:02.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:02.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:02.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:02.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:02.698 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:29:02.711 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:02.748 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:02.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:02.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:02.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:02.881 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:29:02.888 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:02.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:02.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:02.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:03.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:03.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:03.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 56 16 94720 94704 105.667 70.6875 0.0176174 0.0094583 2024-09-10T18:29:03.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:03.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:03.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:03.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:03 smithi028 ceph-mon[39220]: 3.1 scrub starts 2024-09-10T18:29:03.181 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:03 smithi028 ceph-mon[39220]: 3.1 scrub ok 2024-09-10T18:29:03.181 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:03 smithi028 ceph-mon[39220]: 6.13 scrub starts 2024-09-10T18:29:03.181 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:03 smithi028 ceph-mon[39220]: 6.13 scrub ok 2024-09-10T18:29:03.181 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:03 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:29:03.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:03 smithi028 ceph-mon[29291]: 3.1 scrub starts 2024-09-10T18:29:03.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:03 smithi028 ceph-mon[29291]: 3.1 scrub ok 2024-09-10T18:29:03.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:03 smithi028 ceph-mon[29291]: 6.13 scrub starts 2024-09-10T18:29:03.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:03 smithi028 ceph-mon[29291]: 6.13 scrub ok 2024-09-10T18:29:03.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:03 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:29:03.277 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:03.304 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:03.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:03 smithi204 ceph-mon[32357]: 3.1 scrub starts 2024-09-10T18:29:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:03 smithi204 ceph-mon[32357]: 3.1 scrub ok 2024-09-10T18:29:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:03 smithi204 ceph-mon[32357]: 6.13 scrub starts 2024-09-10T18:29:03.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:03 smithi204 ceph-mon[32357]: 6.13 scrub ok 2024-09-10T18:29:03.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:03 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:29:03.407 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:03.438 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:03.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:03.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:03.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:03.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:03.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:03.690 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:03.727 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:03.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:03.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:03.829 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:03.855 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:03.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:03.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:03.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:03.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:03.997 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:03.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:04.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:04.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 57 16 95900 95884 105.107 73.75 0.0113379 0.00950868 2024-09-10T18:29:04.148 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:04.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:04.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:04.247 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:04 smithi204 ceph-mon[32357]: 5.1c scrub starts 2024-09-10T18:29:04.247 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:04 smithi204 ceph-mon[32357]: 5.1c scrub ok 2024-09-10T18:29:04.247 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:04 smithi204 ceph-mon[32357]: pgmap v338: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 709 B/s rd, 53 MiB/s wr, 844 op/s 2024-09-10T18:29:04.247 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:04 smithi204 ceph-mon[32357]: 4.1e scrub starts 2024-09-10T18:29:04.247 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:04 smithi204 ceph-mon[32357]: 4.1e scrub ok 2024-09-10T18:29:04.247 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:04 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:29:04.247 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:04 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:29:04.248 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:04 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:29:04.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:04 smithi028 ceph-mon[39220]: 5.1c scrub starts 2024-09-10T18:29:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:04 smithi028 ceph-mon[39220]: 5.1c scrub ok 2024-09-10T18:29:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:04 smithi028 ceph-mon[39220]: pgmap v338: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 709 B/s rd, 53 MiB/s wr, 844 op/s 2024-09-10T18:29:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:04 smithi028 ceph-mon[39220]: 4.1e scrub starts 2024-09-10T18:29:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:04 smithi028 ceph-mon[39220]: 4.1e scrub ok 2024-09-10T18:29:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:04 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:29:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:04 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:29:04.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:04 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:29:04.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:04 smithi028 ceph-mon[29291]: 5.1c scrub starts 2024-09-10T18:29:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:04 smithi028 ceph-mon[29291]: 5.1c scrub ok 2024-09-10T18:29:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:04 smithi028 ceph-mon[29291]: pgmap v338: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 709 B/s rd, 53 MiB/s wr, 844 op/s 2024-09-10T18:29:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:04 smithi028 ceph-mon[29291]: 4.1e scrub starts 2024-09-10T18:29:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:04 smithi028 ceph-mon[29291]: 4.1e scrub ok 2024-09-10T18:29:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:04 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:29:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:04 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:29:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:04 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:29:04.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:04.273 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:04.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:04.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:04.415 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:04.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:04.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:29:04.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:04.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:04.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:04.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:29:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:29:04.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:04.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:04.747 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:04.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:04.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:04.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:04.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:04.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:29:04.976 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:05.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:05.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:05.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:05.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:05.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 58 16 96787 96771 104.251 55.4375 0.0148066 0.00958737 2024-09-10T18:29:05.146 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:05.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:05.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:05.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:05.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:05.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:05 smithi204 ceph-mon[32357]: 6.17 scrub starts 2024-09-10T18:29:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:05 smithi204 ceph-mon[32357]: 6.17 scrub ok 2024-09-10T18:29:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:05 smithi204 ceph-mon[32357]: 6.a scrub starts 2024-09-10T18:29:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:05 smithi204 ceph-mon[32357]: 6.a scrub ok 2024-09-10T18:29:05.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:05 smithi204 ceph-mon[32357]: 3.a scrub starts 2024-09-10T18:29:05.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:05 smithi204 ceph-mon[32357]: 3.a scrub ok 2024-09-10T18:29:05.359 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:05 smithi028 ceph-mon[29291]: 6.17 scrub starts 2024-09-10T18:29:05.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:05 smithi028 ceph-mon[29291]: 6.17 scrub ok 2024-09-10T18:29:05.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:05 smithi028 ceph-mon[29291]: 6.a scrub starts 2024-09-10T18:29:05.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:05 smithi028 ceph-mon[29291]: 6.a scrub ok 2024-09-10T18:29:05.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:05 smithi028 ceph-mon[29291]: 3.a scrub starts 2024-09-10T18:29:05.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:05 smithi028 ceph-mon[29291]: 3.a scrub ok 2024-09-10T18:29:05.361 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:05 smithi028 ceph-mon[39220]: 6.17 scrub starts 2024-09-10T18:29:05.361 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:05 smithi028 ceph-mon[39220]: 6.17 scrub ok 2024-09-10T18:29:05.361 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:05 smithi028 ceph-mon[39220]: 6.a scrub starts 2024-09-10T18:29:05.361 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:05 smithi028 ceph-mon[39220]: 6.a scrub ok 2024-09-10T18:29:05.361 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:05 smithi028 ceph-mon[39220]: 3.a scrub starts 2024-09-10T18:29:05.362 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:05 smithi028 ceph-mon[39220]: 3.a scrub ok 2024-09-10T18:29:05.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:05.431 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:05.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:05.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:05.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:05.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:05.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:05.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:05.716 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:05.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:05.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:05.818 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:05.847 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:05.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:05.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:05.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:05.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:05.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:06.089 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:29:06.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 59 16 97943 97927 103.708 72.25 0.00654345 0.00963745 2024-09-10T18:29:06.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:06.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:06.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:06.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:06.260 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:06.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:06.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:06 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:06 smithi204 ceph-mon[32357]: pgmap v339: 158 pgs: 158 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 938 op/s 2024-09-10T18:29:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:06 smithi204 ceph-mon[32357]: 2.2 deep-scrub starts 2024-09-10T18:29:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:06 smithi204 ceph-mon[32357]: 2.2 deep-scrub ok 2024-09-10T18:29:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:06 smithi204 ceph-mon[32357]: 5.1e deep-scrub starts 2024-09-10T18:29:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:06 smithi204 ceph-mon[32357]: 5.1e deep-scrub ok 2024-09-10T18:29:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:06 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2813788349' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:29:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:06 smithi204 ceph-mon[32357]: from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:29:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:06 smithi204 ceph-mon[32357]: 4.a scrub starts 2024-09-10T18:29:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:06 smithi204 ceph-mon[32357]: 4.a scrub ok 2024-09-10T18:29:06.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:06.392 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:06.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:06.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:06.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[39220]: pgmap v339: 158 pgs: 158 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 938 op/s 2024-09-10T18:29:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[39220]: 2.2 deep-scrub starts 2024-09-10T18:29:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[39220]: 2.2 deep-scrub ok 2024-09-10T18:29:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[39220]: 5.1e deep-scrub starts 2024-09-10T18:29:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[39220]: 5.1e deep-scrub ok 2024-09-10T18:29:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2813788349' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:29:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[39220]: from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:29:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[39220]: 4.a scrub starts 2024-09-10T18:29:06.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[39220]: 4.a scrub ok 2024-09-10T18:29:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[29291]: pgmap v339: 158 pgs: 158 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 938 op/s 2024-09-10T18:29:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[29291]: 2.2 deep-scrub starts 2024-09-10T18:29:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[29291]: 2.2 deep-scrub ok 2024-09-10T18:29:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[29291]: 5.1e deep-scrub starts 2024-09-10T18:29:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[29291]: 5.1e deep-scrub ok 2024-09-10T18:29:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2813788349' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:29:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[29291]: from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:29:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[29291]: 4.a scrub starts 2024-09-10T18:29:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:06 smithi028 ceph-mon[29291]: 4.a scrub ok 2024-09-10T18:29:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:06.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:06.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:06.630 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:06.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:06.665 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:06.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:06.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:06.788 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:06.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:06.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:06.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:06.917 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:06.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:07.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:07.041 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:29:07.041 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:29:07.042 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-10T18:29:07.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T18:29:07.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:07.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:07.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:07.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:29:07.118244+0000 min lat: 0.00125336 max lat: 0.134949 avg lat: 0.00969779 2024-09-10T18:29:07.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:29:07.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 60 16 98975 98959 103.055 64.5 0.00957896 0.00969779 2024-09-10T18:29:07.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:07.189 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:07.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:07.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:07.294 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:29:07.294 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:29:07.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:07.340 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:07.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:07.441 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:07.475 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:07.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:07.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:07 smithi028 ceph-mon[39220]: 1.0 scrub starts 2024-09-10T18:29:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:07 smithi028 ceph-mon[39220]: 1.0 scrub ok 2024-09-10T18:29:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:07 smithi028 ceph-mon[39220]: 3.c scrub starts 2024-09-10T18:29:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:07 smithi028 ceph-mon[39220]: 3.c scrub ok 2024-09-10T18:29:07.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:07 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:29:07.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:07 smithi028 ceph-mon[39220]: from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:29:07.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:07 smithi028 ceph-mon[39220]: osdmap e204: 8 total, 8 up, 8 in 2024-09-10T18:29:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:07 smithi028 ceph-mon[29291]: 1.0 scrub starts 2024-09-10T18:29:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:07 smithi028 ceph-mon[29291]: 1.0 scrub ok 2024-09-10T18:29:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:07 smithi028 ceph-mon[29291]: 3.c scrub starts 2024-09-10T18:29:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:07 smithi028 ceph-mon[29291]: 3.c scrub ok 2024-09-10T18:29:07.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:07 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:29:07.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:07 smithi028 ceph-mon[29291]: from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:29:07.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:07 smithi028 ceph-mon[29291]: osdmap e204: 8 total, 8 up, 8 in 2024-09-10T18:29:07.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:07 smithi204 ceph-mon[32357]: 1.0 scrub starts 2024-09-10T18:29:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:07 smithi204 ceph-mon[32357]: 1.0 scrub ok 2024-09-10T18:29:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:07 smithi204 ceph-mon[32357]: 3.c scrub starts 2024-09-10T18:29:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:07 smithi204 ceph-mon[32357]: 3.c scrub ok 2024-09-10T18:29:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:07 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:29:07.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:07 smithi204 ceph-mon[32357]: from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:29:07.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:07 smithi204 ceph-mon[32357]: osdmap e204: 8 total, 8 up, 8 in 2024-09-10T18:29:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:07.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:07.713 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:07.742 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:07.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:07.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:07.844 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:07.873 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:07.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:07.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:07.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:08.008 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:08.010 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-10T18:29:08.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:08.111 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:08.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 61 16 100235 100219 102.656 78.75 0.00920384 0.00973646 2024-09-10T18:29:08.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:08.146 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:08.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:08.247 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:08.279 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:08.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:08 smithi204 ceph-mon[32357]: pgmap v341: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T18:29:08.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:08.407 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:29:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:08.416 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:08.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:08.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:08 smithi028 ceph-mon[29291]: pgmap v341: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T18:29:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:08 smithi028 ceph-mon[39220]: pgmap v341: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T18:29:08.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:08.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:08.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:08.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:08.640 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:08.672 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:08.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:08.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:08.774 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:08.799 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:08.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:08.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:08.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:08.937 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:08.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:08.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:09.041 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:09.069 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:09.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:09.120 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 62 16 101336 101320 102.11 68.8125 0.0156282 0.00978861 2024-09-10T18:29:09.170 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:09.196 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:09.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:09.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:09.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:09.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:09.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:09.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:09.492 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:09.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:29:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:29:09.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:09.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:29:09.628 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:09.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:09.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:09.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:09.765 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:09.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:09.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:09.867 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:09.902 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:09.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:09.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:10.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:10.073 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:10.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:10.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:10.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 63 15 102601 102586 101.744 79.125 0.00742984 0.00982359 2024-09-10T18:29:10.176 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:10.209 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:10 smithi204 ceph-mon[32357]: pgmap v342: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 623 B/s rd, 42 MiB/s wr, 675 op/s 2024-09-10T18:29:10.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:10.216 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:10.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:10 smithi028 ceph-mon[39220]: pgmap v342: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 623 B/s rd, 42 MiB/s wr, 675 op/s 2024-09-10T18:29:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:10 smithi028 ceph-mon[29291]: pgmap v342: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 623 B/s rd, 42 MiB/s wr, 675 op/s 2024-09-10T18:29:10.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:10.345 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:29:10.345 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:29:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:10.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:10.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:10.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:10.496 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:10.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:10.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:10.531 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:29:10.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:10.606 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:29:10.606 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:29:10.645 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:29:10.645 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:29:10.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:10.652 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:10.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:10.677 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:29:10.677 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:29:10.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:10.756 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:29:10.757 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:29:10.789 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:29:10.789 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:29:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:10.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:10.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:10.822 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:29:10.823 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:29:10.857 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:29:10.857 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:29:10.888 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:29:10.889 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:29:10.892 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:10.925 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:10.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:11.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:11.056 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:11.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:11.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:11.097 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T18:29:11.120 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 64 16 103661 103645 101.188 66.1875 0.012498 0.00987792 2024-09-10T18:29:11.157 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:11.174 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:11 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3174735550' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:29:11.175 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:11 smithi204 ceph-mon[32357]: osdmap e205: 8 total, 8 up, 8 in 2024-09-10T18:29:11.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:11.218 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:11.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:11.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:11.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:11.356 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:11.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:11.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:11.492 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:11.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:11.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:11 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3174735550' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:29:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:11 smithi028 ceph-mon[39220]: osdmap e205: 8 total, 8 up, 8 in 2024-09-10T18:29:11.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:29:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:29:11] "GET /metrics HTTP/1.1" 200 224292 "" "Prometheus/2.43.0" 2024-09-10T18:29:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:11 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3174735550' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:29:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:11 smithi028 ceph-mon[29291]: osdmap e205: 8 total, 8 up, 8 in 2024-09-10T18:29:11.568 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:29:11.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:11.637 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:11.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:11.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:11.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:11.774 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:11.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:11.876 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:11.910 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:11.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:11.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:11.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:29:12.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:12.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:12.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:12.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 65 16 104826 104810 100.751 72.8125 0.010145 0.00992031 2024-09-10T18:29:12.152 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:12.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:12.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:12.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:12.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:12.320 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:12.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:12.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:12 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:29:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:12 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:29:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:12 smithi204 ceph-mon[32357]: pgmap v344: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-09-10T18:29:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:12 smithi204 ceph-mon[32357]: osdmap e206: 8 total, 8 up, 8 in 2024-09-10T18:29:12.355 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:29:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:29:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:29:12.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:12.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:12.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:12.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:12 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:29:12.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:12 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:29:12.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:12 smithi028 ceph-mon[39220]: pgmap v344: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-09-10T18:29:12.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:12 smithi028 ceph-mon[39220]: osdmap e206: 8 total, 8 up, 8 in 2024-09-10T18:29:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:12 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:29:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:12 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:29:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:12 smithi028 ceph-mon[29291]: pgmap v344: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-09-10T18:29:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:12 smithi028 ceph-mon[29291]: osdmap e206: 8 total, 8 up, 8 in 2024-09-10T18:29:12.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:12.593 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:12.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:12.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:12.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:12.732 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:12.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:12.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:12.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:12.862 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:12.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:12.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:12.997 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:12.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:12.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:13.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:13.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 66 16 105894 105878 100.236 66.75 0.00331498 0.00997036 2024-09-10T18:29:13.132 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:13.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:13.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:13.234 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:13.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:13.272 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:13.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:13.373 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:13.399 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:13.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:13.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:13.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:13.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:13.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:13.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:13.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:13.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:13.682 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:13.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:13.685 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:29:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:29:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.15117.0:106511 8.6 e207/180 8:6300124d:::benchmark_data_smithi028_109242_object106510:head v 207'6639, mlcod=207'6637)", 2024-09-10T18:29:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:29:13.679045+0000", 2024-09-10T18:29:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.00078738099999999998, 2024-09-10T18:29:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00081504400000000001, 2024-09-10T18:29:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:29:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679045+0000", 2024-09-10T18:29:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679045+0000", 2024-09-10T18:29:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679044+0000", 2024-09-10T18:29:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:29:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679081+0000", 2024-09-10T18:29:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7184000000000002e-05 2024-09-10T18:29:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679095+0000", 2024-09-10T18:29:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4396000000000001e-05 2024-09-10T18:29:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679101+0000", 2024-09-10T18:29:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5820000000000001e-06 2024-09-10T18:29:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679175+0000", 2024-09-10T18:29:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4337000000000003e-05 2024-09-10T18:29:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679193+0000", 2024-09-10T18:29:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7801000000000001e-05 2024-09-10T18:29:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15117.0:106517 8.5 8:a48afe71:::benchmark_data_smithi028_109242_object106516:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-10T18:29:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:29:13.679227+0000", 2024-09-10T18:29:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.00060576900000000003, 2024-09-10T18:29:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00072283099999999995, 2024-09-10T18:29:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:29:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15117", 2024-09-10T18:29:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/121353060", 2024-09-10T18:29:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 106517 2024-09-10T18:29:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679227+0000", 2024-09-10T18:29:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679227+0000", 2024-09-10T18:29:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679222+0000", 2024-09-10T18:29:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999952 2024-09-10T18:29:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679262+0000", 2024-09-10T18:29:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0602000000000002e-05 2024-09-10T18:29:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679273+0000", 2024-09-10T18:29:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0628999999999999e-05 2024-09-10T18:29:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679278+0000", 2024-09-10T18:29:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3299999999999998e-06 2024-09-10T18:29:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679402+0000", 2024-09-10T18:29:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012360600000000001 2024-09-10T18:29:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679625+0000", 2024-09-10T18:29:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022263299999999999 2024-09-10T18:29:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4,7", 2024-09-10T18:29:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679771+0000", 2024-09-10T18:29:13.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000146675 2024-09-10T18:29:13.695 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:13.695 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:13.695 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:13.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.695 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15117.0:106520 8.9 8:9965f917:::benchmark_data_smithi028_109242_object106519:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-10T18:29:13.695 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:29:13.679450+0000", 2024-09-10T18:29:13.695 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.00038190799999999998, 2024-09-10T18:29:13.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00057750900000000003, 2024-09-10T18:29:13.695 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:13.696 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:29:13.696 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:13.696 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15117", 2024-09-10T18:29:13.696 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/121353060", 2024-09-10T18:29:13.696 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 106520 2024-09-10T18:29:13.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.696 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:13.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:13.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679450+0000", 2024-09-10T18:29:13.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:13.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:13.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679450+0000", 2024-09-10T18:29:13.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:13.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:13.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679449+0000", 2024-09-10T18:29:13.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:29:13.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:13.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679539+0000", 2024-09-10T18:29:13.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9221999999999994e-05 2024-09-10T18:29:13.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:13.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679551+0000", 2024-09-10T18:29:13.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.22e-05 2024-09-10T18:29:13.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:13.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679559+0000", 2024-09-10T18:29:13.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7069999999999992e-06 2024-09-10T18:29:13.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:13.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679604+0000", 2024-09-10T18:29:13.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4974e-05 2024-09-10T18:29:13.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:13.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679731+0000", 2024-09-10T18:29:13.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012717300000000001 2024-09-10T18:29:13.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:13.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:13.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:29:13.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:29:13.679864+0000", 2024-09-10T18:29:13.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013312799999999999 2024-09-10T18:29:13.699 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:13.699 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:13.700 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:13.700 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:13.700 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-09-10T18:29:13.700 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 3 2024-09-10T18:29:13.700 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:29:13.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:13.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:13.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:13.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:13.960 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:13.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:13.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:14.062 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:14.092 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:14.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:14.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:14.122 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 67 16 107090 107074 99.854 74.75 0.017697 0.0100093 2024-09-10T18:29:14.195 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:14.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:14.225 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:14.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:14.327 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:14.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:14.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:14.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:14 smithi204 ceph-mon[32357]: pgmap v346: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 752 B/s rd, 53 MiB/s wr, 856 op/s 2024-09-10T18:29:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:14 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:29:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:14 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:29:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:14 smithi204 ceph-mon[32357]: osdmap e207: 8 total, 8 up, 8 in 2024-09-10T18:29:14.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:29:14 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:29:14.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:14.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-10T18:29:14.485 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:14.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:14.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:14 smithi028 ceph-mon[39220]: pgmap v346: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 752 B/s rd, 53 MiB/s wr, 856 op/s 2024-09-10T18:29:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:14 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:29:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:14 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:29:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:14 smithi028 ceph-mon[39220]: osdmap e207: 8 total, 8 up, 8 in 2024-09-10T18:29:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:14 smithi028 ceph-mon[29291]: pgmap v346: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 752 B/s rd, 53 MiB/s wr, 856 op/s 2024-09-10T18:29:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:14 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:29:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:14 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:29:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:14 smithi028 ceph-mon[29291]: osdmap e207: 8 total, 8 up, 8 in 2024-09-10T18:29:14.587 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:14.625 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:14.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:14.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:14.727 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:14.766 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:14.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:14.867 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:14.868 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:29:14.918 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:14.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:14.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:15.020 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:15.051 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:15.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:15.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:15.123 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 68 16 108329 108313 99.5238 77.4375 0.0121854 0.0100434 2024-09-10T18:29:15.152 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:15.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:15.186 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:15.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:15.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:15.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:15.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:15.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:15.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:15.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:15.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:15.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:15.551 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:15.582 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:15.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:15.593 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:29:15.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:15.720 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:15.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:15.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:15.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:15.862 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:15.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:15.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:15.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:16.015 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:16.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:16.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:16.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:29:16.118 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:16.122 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 69 16 109573 109557 99.2081 77.75 0.0105268 0.0100754 2024-09-10T18:29:16.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:16.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:16.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:16.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:16 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:16.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:16 smithi028 ceph-mon[29291]: pgmap v348: 158 pgs: 158 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-09-10T18:29:16.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:16 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:16.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:16 smithi028 ceph-mon[39220]: pgmap v348: 158 pgs: 158 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-09-10T18:29:16.252 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:16.280 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:16.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:16.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:16 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:16 smithi204 ceph-mon[32357]: pgmap v348: 158 pgs: 158 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-09-10T18:29:16.382 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:16.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:16.411 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:16.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:16.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:16.643 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:16.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:16.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:16.742 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:29:16.742 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:29:16.742 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-10T18:29:16.742 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-10T18:29:16.742 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:29:16.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:16.779 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:16.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:16.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:16.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:16.920 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:16.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:16.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:17.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:17.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:17.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:17.124 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 70 16 110792 110776 98.8792 76.1875 0.0149447 0.0101087 2024-09-10T18:29:17.162 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:17.189 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:17.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:17.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:17.333 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:17 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:29:17.333 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:17 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:29:17.333 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:17 smithi028 ceph-mon[39220]: osdmap e208: 8 total, 8 up, 8 in 2024-09-10T18:29:17.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:17 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:29:17.334 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:17 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:29:17.334 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:17 smithi028 ceph-mon[29291]: osdmap e208: 8 total, 8 up, 8 in 2024-09-10T18:29:17.337 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:17.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:17.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:17 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:29:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:17 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:29:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:17 smithi204 ceph-mon[32357]: osdmap e208: 8 total, 8 up, 8 in 2024-09-10T18:29:17.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:17.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:17.479 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:17.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:17.488 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-10T18:29:17.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:17.622 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:17.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:17.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:17.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:17.760 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:17.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:17.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:17.861 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:17.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:17.910 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:17.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:17.918 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:29:18.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:18.045 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:18.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:18.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:18.123 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 71 16 111937 111921 98.4945 71.5625 0.0174844 0.0101476 2024-09-10T18:29:18.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:18.172 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:18.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:18.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:18.274 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:29:18.274 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:18.302 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:18.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:18.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:18.312 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:29:18.312 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:29:18.312 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 208, 2024-09-10T18:29:18.312 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "5eababb2-6fa0-11ef-bcea-c7b262605968", 2024-09-10T18:29:18.313 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-10T18:16:20.680373+0000", 2024-09-10T18:29:18.313 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-10T18:29:16.141935+0000", 2024-09-10T18:29:18.313 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-09-10T18:26:26.611338+0000", 2024-09-10T18:29:18.313 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-09-10T18:21:02.114621+0000", 2024-09-10T18:29:18.313 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T18:29:18.313 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5801984, 2024-09-10T18:29:18.314 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-09-10T18:29:18.314 INFO:teuthology.orchestra.run.smithi028.stdout: "noscrub", 2024-09-10T18:29:18.314 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-09-10T18:29:18.314 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-09-10T18:29:18.314 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-09-10T18:29:18.314 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-09-10T18:29:18.314 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:29:18.315 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 19, 2024-09-10T18:29:18.315 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T18:29:18.315 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T18:29:18.315 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T18:29:18.315 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-09-10T18:29:18.315 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 8, 2024-09-10T18:29:18.316 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-09-10T18:29:18.316 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-09-10T18:29:18.316 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-09-10T18:29:18.316 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "quincy", 2024-09-10T18:29:18.316 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-09-10T18:29:18.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.316 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-09-10T18:29:18.316 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-09-10T18:29:18.316 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:19:24.013536+0000", 2024-09-10T18:29:18.317 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:29:18.317 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:29:18.317 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:29:18.317 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:29:18.317 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:29:18.317 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:29:18.317 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:29:18.317 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:29:18.317 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:29:18.317 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:29:18.317 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:29:18.317 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:29:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 11, 2024-09-10T18:29:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 11, 2024-09-10T18:29:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 11, 2024-09-10T18:29:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 11, 2024-09-10T18:29:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 11, 2024-09-10T18:29:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:29:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:29:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:29:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:29:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:29:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:29:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:29:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "138", 2024-09-10T18:29:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:29:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:29:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "136", 2024-09-10T18:29:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:29:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:29:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:29:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:29:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:29:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:29:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:29:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:29:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:29:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:29:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:29:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:29:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:29:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:29:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:29:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:29:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:29:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:29:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:29:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:29:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:29:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:29:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:29:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:29:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:29:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:29:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:29:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:29:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:29:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:29:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:29:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:29:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:29:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:29:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-10T18:29:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-09-10T18:29:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-09-10T18:29:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:29:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-09-10T18:29:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-09-10T18:29:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-09-10T18:29:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:30.065209+0000", 2024-09-10T18:29:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-09-10T18:29:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T18:29:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:29:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:29:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:29:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:29:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:29:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:29:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:29:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:29:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:29:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:29:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-09-10T18:29:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-09-10T18:29:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-09-10T18:29:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-09-10T18:29:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-09-10T18:29:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:29:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:29:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:29:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:29:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:29:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:29:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:29:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "62", 2024-09-10T18:29:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:29:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:29:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:29:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:29:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:29:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 3, 2024-09-10T18:29:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 62, 2024-09-10T18:29:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:29:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:29:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:29:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:29:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:29:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:29:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:29:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:29:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:29:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:29:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:29:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:29:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:29:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:29:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:29:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:29:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:29:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:29:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:29:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:29:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:29:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:29:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:29:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:29:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:29:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:29:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:29:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:29:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:29:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:29:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:29:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:29:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-09-10T18:29:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-09-10T18:29:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-09-10T18:29:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:30.506441+0000", 2024-09-10T18:29:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:29:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:29:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:29:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:29:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:29:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:29:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:29:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:29:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:29:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:29:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:29:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:29:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:29:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:29:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:29:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:29:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:29:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:29:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:29:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:29:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:29:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:29:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:29:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:29:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "59", 2024-09-10T18:29:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:29:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:29:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:29:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:29:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:29:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:29:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:29:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:29:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:29:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:29:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:29:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:29:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:29:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:29:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:29:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:29:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:29:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:29:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:29:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:29:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:29:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:29:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:29:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:29:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:29:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:29:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:29:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:29:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:29:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:29:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:29:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:29:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:29:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:29:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:29:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:29:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:29:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:29:18.334 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:29:18.334 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:29:18.334 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.334 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-09-10T18:29:18.334 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-09-10T18:29:18.334 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:32.008130+0000", 2024-09-10T18:29:18.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:29:18.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:29:18.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:29:18.334 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:29:18.335 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:29:18.335 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:29:18.335 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:29:18.335 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:29:18.335 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:29:18.335 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:29:18.335 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:29:18.335 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:29:18.335 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:29:18.335 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:29:18.335 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:29:18.336 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:29:18.336 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:29:18.336 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:29:18.336 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:29:18.336 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:29:18.336 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:29:18.336 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:29:18.336 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:29:18.336 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:29:18.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.336 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "61", 2024-09-10T18:29:18.337 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:29:18.337 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:29:18.337 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:29:18.337 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:29:18.337 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:29:18.337 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:29:18.337 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:29:18.337 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:29:18.337 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:29:18.337 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:29:18.337 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:29:18.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:29:18.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:29:18.338 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:29:18.338 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:29:18.338 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:29:18.338 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:29:18.338 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:29:18.338 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:29:18.338 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:29:18.338 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:29:18.338 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:29:18.338 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:29:18.338 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:29:18.338 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:29:18.339 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:29:18.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.339 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:29:18.339 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:29:18.339 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:29:18.339 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:29:18.339 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:29:18.339 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:29:18.339 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:29:18.339 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:29:18.339 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:29:18.339 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:29:18.340 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:29:18.340 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:29:18.340 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:29:18.340 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:29:18.340 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.340 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-09-10T18:29:18.340 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-09-10T18:29:18.340 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:34.060571+0000", 2024-09-10T18:29:18.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:29:18.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:29:18.341 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:29:18.341 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:29:18.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:29:18.341 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:29:18.341 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:29:18.341 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:29:18.341 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:29:18.341 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:29:18.341 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:29:18.341 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:29:18.341 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:29:18.342 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:29:18.342 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:29:18.342 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:29:18.342 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:29:18.342 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:29:18.342 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:29:18.342 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:29:18.342 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:29:18.342 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:29:18.343 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:29:18.343 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:29:18.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.343 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "63", 2024-09-10T18:29:18.343 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:29:18.343 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:29:18.343 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:29:18.343 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:29:18.343 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:29:18.343 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:29:18.343 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:29:18.343 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:29:18.343 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:29:18.343 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:29:18.344 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:29:18.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:29:18.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:29:18.344 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:29:18.344 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:29:18.344 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:29:18.344 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:29:18.344 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:29:18.344 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:29:18.344 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:29:18.344 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:29:18.344 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:29:18.345 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:29:18.345 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:29:18.345 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:29:18.345 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:29:18.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.345 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:29:18.345 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:29:18.345 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:29:18.345 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:29:18.345 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:29:18.345 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:29:18.345 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:29:18.345 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:29:18.346 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:29:18.346 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:29:18.346 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:29:18.346 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:29:18.346 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:29:18.346 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:29:18.346 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.346 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-09-10T18:29:18.346 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-09-10T18:29:18.346 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:36.127330+0000", 2024-09-10T18:29:18.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:29:18.347 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:29:18.347 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:29:18.347 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:29:18.347 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:29:18.347 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:29:18.347 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:29:18.347 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:29:18.347 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:29:18.347 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:29:18.347 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:29:18.347 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:29:18.347 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:29:18.347 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:29:18.347 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:29:18.348 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:29:18.348 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:29:18.348 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:29:18.348 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:29:18.348 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:29:18.348 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:29:18.348 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:29:18.348 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:29:18.348 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:29:18.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.348 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "65", 2024-09-10T18:29:18.348 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:29:18.348 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:29:18.349 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:29:18.349 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:29:18.349 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:29:18.349 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:29:18.349 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:29:18.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:29:18.349 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:29:18.349 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:29:18.349 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:29:18.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:29:18.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:29:18.350 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:29:18.350 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:29:18.350 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:29:18.350 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:29:18.350 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:29:18.350 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:29:18.350 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:29:18.350 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:29:18.350 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:29:18.350 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:29:18.350 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:29:18.351 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:29:18.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:29:18.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.351 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:29:18.351 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:29:18.351 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:29:18.351 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:29:18.351 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:29:18.352 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:29:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:29:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:29:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:29:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:29:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:29:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-10T18:29:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-09-10T18:29:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:18 smithi204 ceph-mon[32357]: pgmap v350: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 959 op/s 2024-09-10T18:29:18.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.354 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:29:18.354 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:29:18.354 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.355 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 8, 2024-09-10T18:29:18.355 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "unique_pool_1", 2024-09-10T18:29:18.355 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:27:51.966788+0000", 2024-09-10T18:29:18.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:29:18.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:29:18.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:29:18.355 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:29:18.355 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:29:18.355 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:29:18.355 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:29:18.356 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:29:18.356 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:29:18.356 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:29:18.356 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:29:18.356 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:29:18.356 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 16, 2024-09-10T18:29:18.356 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 16, 2024-09-10T18:29:18.356 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 16, 2024-09-10T18:29:18.356 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 16, 2024-09-10T18:29:18.356 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 16, 2024-09-10T18:29:18.357 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:29:18.357 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:29:18.357 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:29:18.357 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:29:18.357 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:29:18.357 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:29:18.357 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:29:18.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.357 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "189", 2024-09-10T18:29:18.358 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:29:18.358 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:29:18.358 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:29:18.358 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:29:18.358 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:29:18.358 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:29:18.358 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:29:18.358 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:29:18.358 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:29:18.358 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:29:18.358 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:29:18.358 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:29:18.358 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:29:18.358 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:29:18.358 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:29:18.359 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:29:18.359 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:29:18.359 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:29:18.359 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:29:18.359 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:29:18.359 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:29:18.359 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:29:18.359 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:29:18.359 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:29:18.359 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:29:18.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:29:18.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.359 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:29:18.360 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:29:18.360 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:29:18.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:29:18.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:29:18.360 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:29:18.360 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:29:18.360 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:29:18.360 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:29:18.360 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:29:18.360 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:29:18.360 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:29:18.360 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:29:18.360 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-09-10T18:29:18.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.361 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:29:18.361 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:29:18.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.361 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-10T18:29:18.361 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "f1c834d9-252a-4077-b25e-169181d70251", 2024-09-10T18:29:18.361 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:29:18.361 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:29:18.361 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:29:18.361 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:29:18.361 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:29:18.361 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:29:18.361 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-09-10T18:29:18.362 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 180, 2024-09-10T18:29:18.362 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:29:18.362 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:29:18.362 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:29:18.362 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.362 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-09-10T18:29:18.362 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:29:18.362 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.362 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.362 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:29:18.363 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.363 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-09-10T18:29:18.363 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:29:18.363 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.363 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.363 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:29:18.363 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.363 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-09-10T18:29:18.363 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:29:18.364 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.364 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.364 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:29:18.364 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.364 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.364 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-09-10T18:29:18.364 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:29:18.364 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.364 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.365 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/3188372351", 2024-09-10T18:29:18.365 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/3188372351", 2024-09-10T18:29:18.365 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/3188372351", 2024-09-10T18:29:18.365 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/3188372351", 2024-09-10T18:29:18.365 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:29:18.365 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:29:18.365 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:29:18.365 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.365 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-10T18:29:18.365 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "81f1855d-b946-4ca3-b246-38977b9e20fd", 2024-09-10T18:29:18.365 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:29:18.365 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:29:18.366 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:29:18.366 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:29:18.366 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:29:18.366 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:29:18.366 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-10T18:29:18.366 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 180, 2024-09-10T18:29:18.366 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:29:18.366 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:29:18.366 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:29:18.366 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.366 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-09-10T18:29:18.366 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:29:18.366 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.366 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.367 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:29:18.367 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.367 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-09-10T18:29:18.367 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:29:18.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.367 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.367 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:29:18.367 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.367 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-09-10T18:29:18.368 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:29:18.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.368 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.368 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:29:18.368 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.368 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-09-10T18:29:18.368 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:29:18.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.368 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.369 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1369912678", 2024-09-10T18:29:18.369 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1369912678", 2024-09-10T18:29:18.369 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1369912678", 2024-09-10T18:29:18.369 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1369912678", 2024-09-10T18:29:18.369 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:29:18.369 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:29:18.369 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:29:18.369 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.369 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-10T18:29:18.369 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "2ad55d2a-36fa-4756-bf22-58d4b049486e", 2024-09-10T18:29:18.369 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:29:18.370 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:29:18.370 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:29:18.370 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:29:18.370 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:29:18.370 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:29:18.370 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-09-10T18:29:18.370 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 95, 2024-09-10T18:29:18.370 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:29:18.370 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:29:18.370 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:29:18.370 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.370 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.370 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-09-10T18:29:18.371 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:29:18.371 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.371 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.371 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:29:18.371 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.371 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.371 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-09-10T18:29:18.371 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:29:18.371 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.371 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.371 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:29:18.371 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.372 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-09-10T18:29:18.372 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:29:18.372 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.372 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.372 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:29:18.372 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.372 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-09-10T18:29:18.372 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:29:18.373 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.373 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.373 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/1243406003", 2024-09-10T18:29:18.373 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/1243406003", 2024-09-10T18:29:18.373 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/1243406003", 2024-09-10T18:29:18.373 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1243406003", 2024-09-10T18:29:18.373 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:29:18.373 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:29:18.373 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:29:18.374 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.374 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-10T18:29:18.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:18 smithi028 ceph-mon[29291]: pgmap v350: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 959 op/s 2024-09-10T18:29:18.374 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:18 smithi028 ceph-mon[39220]: pgmap v350: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 959 op/s 2024-09-10T18:29:18.375 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "242729a5-a5e8-44d5-97c8-14d14e970bff", 2024-09-10T18:29:18.375 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:29:18.375 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:29:18.375 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:29:18.375 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:29:18.376 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:29:18.376 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:29:18.376 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-09-10T18:29:18.376 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 180, 2024-09-10T18:29:18.376 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:29:18.376 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:29:18.376 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:29:18.376 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.377 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-09-10T18:29:18.377 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:29:18.377 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.377 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.377 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:29:18.377 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.377 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-09-10T18:29:18.377 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:29:18.377 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.377 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.378 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:29:18.378 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.378 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-09-10T18:29:18.378 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:29:18.378 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.378 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.378 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:29:18.379 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.379 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-09-10T18:29:18.379 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:29:18.379 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.379 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.379 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/67703283", 2024-09-10T18:29:18.379 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/67703283", 2024-09-10T18:29:18.379 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/67703283", 2024-09-10T18:29:18.379 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/67703283", 2024-09-10T18:29:18.380 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:29:18.380 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:29:18.380 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:29:18.380 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.380 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-10T18:29:18.380 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "6e38b8ef-e6ed-493f-8eca-755354754787", 2024-09-10T18:29:18.380 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:29:18.380 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:29:18.381 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:29:18.381 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:29:18.381 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:29:18.381 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:29:18.381 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 33, 2024-09-10T18:29:18.381 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 180, 2024-09-10T18:29:18.381 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:29:18.381 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:29:18.381 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:29:18.381 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.382 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6800", 2024-09-10T18:29:18.382 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:29:18.382 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.382 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.382 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:29:18.382 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.382 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6801", 2024-09-10T18:29:18.382 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:29:18.383 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.383 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.383 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:29:18.383 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.383 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.383 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6803", 2024-09-10T18:29:18.383 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:29:18.384 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.384 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.384 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:29:18.384 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.384 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6802", 2024-09-10T18:29:18.384 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:29:18.384 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.385 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.385 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6800/1619669076", 2024-09-10T18:29:18.385 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6801/1619669076", 2024-09-10T18:29:18.385 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6803/1619669076", 2024-09-10T18:29:18.385 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6802/1619669076", 2024-09-10T18:29:18.385 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:29:18.385 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:29:18.385 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:29:18.386 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.386 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-10T18:29:18.386 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "a92e0441-66c9-498e-a0ac-6465acdbb1f5", 2024-09-10T18:29:18.386 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:29:18.386 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:29:18.386 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:29:18.386 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:29:18.386 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:29:18.387 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:29:18.387 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 39, 2024-09-10T18:29:18.387 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 180, 2024-09-10T18:29:18.387 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:29:18.387 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:29:18.387 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:29:18.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6804", 2024-09-10T18:29:18.388 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:29:18.388 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.388 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.388 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:29:18.388 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.388 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6805", 2024-09-10T18:29:18.388 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:29:18.388 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.388 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.389 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:29:18.389 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.389 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6807", 2024-09-10T18:29:18.389 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:29:18.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.389 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.389 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:29:18.389 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.390 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6806", 2024-09-10T18:29:18.390 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:29:18.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.390 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.390 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6804/2717802880", 2024-09-10T18:29:18.390 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6805/2717802880", 2024-09-10T18:29:18.390 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6807/2717802880", 2024-09-10T18:29:18.390 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6806/2717802880", 2024-09-10T18:29:18.390 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:29:18.390 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:29:18.390 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:29:18.390 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.391 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-09-10T18:29:18.391 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "12b02875-6653-45ed-8d2b-b4d33bd88525", 2024-09-10T18:29:18.391 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:29:18.391 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:29:18.391 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:29:18.391 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:29:18.391 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 46, 2024-09-10T18:29:18.391 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 94, 2024-09-10T18:29:18.391 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 144, 2024-09-10T18:29:18.391 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 180, 2024-09-10T18:29:18.392 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 95, 2024-09-10T18:29:18.392 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:29:18.392 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:29:18.392 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.392 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6808", 2024-09-10T18:29:18.392 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:29:18.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.392 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.393 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:29:18.393 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.393 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6809", 2024-09-10T18:29:18.393 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:29:18.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.393 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.393 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:29:18.393 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.394 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6811", 2024-09-10T18:29:18.394 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:29:18.394 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.394 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.394 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:29:18.394 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.394 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.394 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6810", 2024-09-10T18:29:18.394 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:29:18.394 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.394 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.395 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6808/3479311452", 2024-09-10T18:29:18.395 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6809/3479311452", 2024-09-10T18:29:18.395 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6811/3479311452", 2024-09-10T18:29:18.395 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6810/3479311452", 2024-09-10T18:29:18.395 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:29:18.395 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:29:18.395 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:29:18.395 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.395 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-09-10T18:29:18.395 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "0c4f7aa9-ff95-4192-a146-ca576eee907f", 2024-09-10T18:29:18.395 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:29:18.395 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:29:18.396 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:29:18.396 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:29:18.396 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:29:18.396 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:29:18.396 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 54, 2024-09-10T18:29:18.396 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 144, 2024-09-10T18:29:18.396 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:29:18.397 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:29:18.397 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:29:18.397 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.397 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6812", 2024-09-10T18:29:18.397 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:29:18.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.397 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.397 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:29:18.397 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.398 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6813", 2024-09-10T18:29:18.398 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:29:18.398 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.398 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.398 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:29:18.398 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.398 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6815", 2024-09-10T18:29:18.398 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:29:18.399 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.399 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.399 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:29:18.399 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:29:18.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.399 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:29:18.399 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6814", 2024-09-10T18:29:18.399 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:29:18.399 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.399 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.399 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6812/3511791879", 2024-09-10T18:29:18.399 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6813/3511791879", 2024-09-10T18:29:18.400 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6815/3511791879", 2024-09-10T18:29:18.400 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6814/3511791879", 2024-09-10T18:29:18.400 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:29:18.400 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:29:18.400 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:29:18.400 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.400 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.400 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:29:18.400 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-09-10T18:29:18.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.400 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-10T18:29:18.400 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:29:18.400 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:29:18.400 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:29:18.401 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:29:18.401 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:29:18.401 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:28:13.795615+0000", 2024-09-10T18:29:18.401 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:29:18.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.401 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-10T18:29:18.401 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:29:18.401 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:29:18.401 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:29:18.401 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:29:18.401 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:29:18.401 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:28:44.945813+0000", 2024-09-10T18:29:18.401 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:29:18.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.402 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-10T18:29:18.402 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:29:18.402 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:29:18.402 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:29:18.402 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:29:18.402 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:29:18.402 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:29:10.889906+0000", 2024-09-10T18:29:18.402 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:29:18.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.402 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-10T18:29:18.403 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:29:18.403 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:29:18.403 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:29:18.403 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:29:18.403 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:29:18.403 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:28:10.447709+0000", 2024-09-10T18:29:18.403 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:29:18.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.403 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-10T18:29:18.403 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:29:18.403 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:29:18.403 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:29:18.403 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:29:18.403 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:29:18.404 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:28:53.256166+0000", 2024-09-10T18:29:18.404 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:29:18.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.404 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-10T18:29:18.404 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:29:18.404 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:29:18.404 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:29:18.404 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:29:18.404 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:29:18.404 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:29:13.113559+0000", 2024-09-10T18:29:18.404 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:29:18.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.405 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-09-10T18:29:18.405 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:29:18.405 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:29:18.405 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:29:18.405 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:29:18.405 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:29:18.405 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:29:15.622169+0000", 2024-09-10T18:29:18.405 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 94 2024-09-10T18:29:18.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.405 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-09-10T18:29:18.405 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:29:18.405 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:29:18.406 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:29:18.406 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:29:18.406 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:29:18.406 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:28:31.162243+0000", 2024-09-10T18:29:18.406 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:29:18.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.406 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:29:18.406 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [], 2024-09-10T18:29:18.406 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-09-10T18:29:18.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.406 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.4", 2024-09-10T18:29:18.406 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:29:18.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.406 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:29:18.407 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-09-10T18:29:18.407 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.407 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.407 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.a", 2024-09-10T18:29:18.407 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:29:18.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.407 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:29:18.407 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-09-10T18:29:18.407 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.407 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.407 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1f", 2024-09-10T18:29:18.408 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:29:18.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:18.408 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:29:18.408 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-09-10T18:29:18.408 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.408 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:18.408 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.408 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:29:18.408 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-09-10T18:29:18.408 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-09-10T18:29:18.408 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-09-10T18:29:18.408 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1745574083": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:29:18.408 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3866220552": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:29:18.409 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3760775432": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:29:18.409 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1746797014": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:29:18.409 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1029045895": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:29:18.409 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3310949126": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:29:18.409 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2874160765": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:29:18.409 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2620179974": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:29:18.409 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/421194352": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:29:18.409 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/1077377902": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:29:18.409 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3428757300": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:29:18.409 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1759116360": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:29:18.409 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2564333506": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:29:18.409 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/717120167": "2024-09-11T18:22:08.678985+0000" 2024-09-10T18:29:18.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-09-10T18:29:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-09-10T18:29:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-09-10T18:29:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-09-10T18:29:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-09-10T18:29:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-09-10T18:29:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-09-10T18:29:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-09-10T18:29:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-09-10T18:29:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-09-10T18:29:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-09-10T18:29:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-09-10T18:29:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-09-10T18:29:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-09-10T18:29:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-09-10T18:29:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-09-10T18:29:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-09-10T18:29:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-09-10T18:29:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-09-10T18:29:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:18.411 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:29:18.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:18.460 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:18.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:18.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:18.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:18.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:18.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:18.748 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:18.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:18.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:18.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:18.894 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:18.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:18.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:18.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:19.031 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:19.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:19.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:19.123 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 72 16 112930 112914 97.9885 62.0625 0.0117487 0.0101998 2024-09-10T18:29:19.132 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:19.171 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:19.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:19.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:19.195 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:29:19.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:19.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:19.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:19 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1959960172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:29:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:19 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1625914547' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:29:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:19 smithi204 ceph-mon[32357]: from='client.25120 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:29:19.420 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:19.446 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:19.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:19.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:19.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:19 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1959960172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:29:19.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:19 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1625914547' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:29:19.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:19 smithi028 ceph-mon[39220]: from='client.25120 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:29:19.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:19 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1959960172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:29:19.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:19 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1625914547' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:29:19.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:19 smithi028 ceph-mon[29291]: from='client.25120 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:29:19.548 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:19.581 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:19.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:19.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:19.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:19.722 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:19.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:19.758 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:29:19.759 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-10T18:29:19.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-10T18:29:19.759 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:29:19.759 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.759 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:182560 7.7 7.fd896b7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e144)", 2024-09-10T18:29:19.759 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:26.644496+0000", 2024-09-10T18:29:19.759 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 173.108961986, 2024-09-10T18:29:19.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99046780599999995, 2024-09-10T18:29:19.759 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.759 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:29:19.759 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.759 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.760 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.760 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 182560 2024-09-10T18:29:19.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.760 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.760 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.760 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644496+0000", 2024-09-10T18:29:19.760 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.760 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.760 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644496+0000", 2024-09-10T18:29:19.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.761 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644494+0000", 2024-09-10T18:29:19.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:29:19.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.761 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644501+0000", 2024-09-10T18:29:19.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5039999999999999e-06 2024-09-10T18:29:19.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644510+0000", 2024-09-10T18:29:19.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2690000000000005e-06 2024-09-10T18:29:19.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644517+0000", 2024-09-10T18:29:19.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7240000000000003e-06 2024-09-10T18:29:19.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.647399+0000", 2024-09-10T18:29:19.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002882169 2024-09-10T18:29:19.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.647407+0000", 2024-09-10T18:29:19.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9710000000000004e-06 2024-09-10T18:29:19.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.764 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.634954+0000", 2024-09-10T18:29:19.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98754652799999998 2024-09-10T18:29:19.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.764 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.764 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.634964+0000", 2024-09-10T18:29:19.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0642000000000001e-05 2024-09-10T18:29:19.764 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.764 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.765 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.765 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:182566 7.7 7.1618317 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e144)", 2024-09-10T18:29:19.765 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:26.644910+0000", 2024-09-10T18:29:19.765 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 173.108548824, 2024-09-10T18:29:19.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99008567599999997, 2024-09-10T18:29:19.766 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.766 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:29:19.766 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.766 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.766 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.766 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 182566 2024-09-10T18:29:19.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.766 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644910+0000", 2024-09-10T18:29:19.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644910+0000", 2024-09-10T18:29:19.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644909+0000", 2024-09-10T18:29:19.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:29:19.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644914+0000", 2024-09-10T18:29:19.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3619999999999997e-06 2024-09-10T18:29:19.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644924+0000", 2024-09-10T18:29:19.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7000000000000003e-06 2024-09-10T18:29:19.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644929+0000", 2024-09-10T18:29:19.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.784e-06 2024-09-10T18:29:19.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.647429+0000", 2024-09-10T18:29:19.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024991610000000002 2024-09-10T18:29:19.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.647434+0000", 2024-09-10T18:29:19.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9290000000000003e-06 2024-09-10T18:29:19.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.770 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.634986+0000", 2024-09-10T18:29:19.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98755193299999999 2024-09-10T18:29:19.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.770 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.634995+0000", 2024-09-10T18:29:19.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8270000000000004e-06 2024-09-10T18:29:19.770 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.770 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.770 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.771 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.771 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:182570 7.c 7.cf5112fc (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e144)", 2024-09-10T18:29:19.771 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:26.644934+0000", 2024-09-10T18:29:19.771 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 173.108524437, 2024-09-10T18:29:19.771 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.981210578, 2024-09-10T18:29:19.771 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.771 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:29:19.771 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.771 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.771 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.771 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 182570 2024-09-10T18:29:19.771 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.771 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.771 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644934+0000", 2024-09-10T18:29:19.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.772 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644934+0000", 2024-09-10T18:29:19.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.772 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644933+0000", 2024-09-10T18:29:19.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999995 2024-09-10T18:29:19.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644936+0000", 2024-09-10T18:29:19.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9890000000000001e-06 2024-09-10T18:29:19.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644942+0000", 2024-09-10T18:29:19.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5079999999999996e-06 2024-09-10T18:29:19.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644946+0000", 2024-09-10T18:29:19.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2270000000000004e-06 2024-09-10T18:29:19.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.648100+0000", 2024-09-10T18:29:19.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0031542720000000001 2024-09-10T18:29:19.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.648106+0000", 2024-09-10T18:29:19.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0569999999999998e-06 2024-09-10T18:29:19.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.626135+0000", 2024-09-10T18:29:19.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97802908099999997 2024-09-10T18:29:19.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.626145+0000", 2024-09-10T18:29:19.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1390000000000001e-06 2024-09-10T18:29:19.775 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.775 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.776 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.776 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:182576 7.d 7.7a34432d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-10T18:29:19.776 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:26.646578+0000", 2024-09-10T18:29:19.776 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 173.10688007499999, 2024-09-10T18:29:19.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97905169700000005, 2024-09-10T18:29:19.776 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.776 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:29:19.776 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.776 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.777 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.777 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 182576 2024-09-10T18:29:19.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.777 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.646578+0000", 2024-09-10T18:29:19.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.646578+0000", 2024-09-10T18:29:19.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.646577+0000", 2024-09-10T18:29:19.778 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:29:19.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.646583+0000", 2024-09-10T18:29:19.778 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6510000000000004e-06 2024-09-10T18:29:19.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.646593+0000", 2024-09-10T18:29:19.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5589999999999994e-06 2024-09-10T18:29:19.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.646599+0000", 2024-09-10T18:29:19.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2380000000000002e-06 2024-09-10T18:29:19.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.646939+0000", 2024-09-10T18:29:19.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00034010600000000001 2024-09-10T18:29:19.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.646948+0000", 2024-09-10T18:29:19.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7020000000000007e-06 2024-09-10T18:29:19.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.625617+0000", 2024-09-10T18:29:19.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.978669448 2024-09-10T18:29:19.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.625630+0000", 2024-09-10T18:29:19.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2635e-05 2024-09-10T18:29:19.781 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.781 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.781 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.781 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:182579 7.c 7.428e990c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-10T18:29:19.781 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:26.649232+0000", 2024-09-10T18:29:19.781 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 173.10422595399999, 2024-09-10T18:29:19.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97693863700000005, 2024-09-10T18:29:19.781 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.781 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:29:19.782 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.782 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.782 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.782 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 182579 2024-09-10T18:29:19.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.782 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.649232+0000", 2024-09-10T18:29:19.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.649232+0000", 2024-09-10T18:29:19.783 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.649231+0000", 2024-09-10T18:29:19.784 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:29:19.784 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.784 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.649238+0000", 2024-09-10T18:29:19.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8900000000000001e-06 2024-09-10T18:29:19.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.649246+0000", 2024-09-10T18:29:19.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7700000000000007e-06 2024-09-10T18:29:19.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.649251+0000", 2024-09-10T18:29:19.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5639999999999998e-06 2024-09-10T18:29:19.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.649291+0000", 2024-09-10T18:29:19.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9996999999999997e-05 2024-09-10T18:29:19.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.649296+0000", 2024-09-10T18:29:19.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2800000000000003e-06 2024-09-10T18:29:19.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.626164+0000", 2024-09-10T18:29:19.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97686759400000001 2024-09-10T18:29:19.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.626171+0000", 2024-09-10T18:29:19.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3509999999999996e-06 2024-09-10T18:29:19.788 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.788 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.788 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.789 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:200476 7.d 7:b713bc26:::benchmark_data_smithi028_77856_object79376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-10T18:29:19.789 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:46.958115+0000", 2024-09-10T18:29:19.789 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 152.795343804, 2024-09-10T18:29:19.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.060615461, 2024-09-10T18:29:19.789 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.789 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:19.789 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.789 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.789 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.789 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 200476 2024-09-10T18:29:19.789 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.789 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.790 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.790 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.958115+0000", 2024-09-10T18:29:19.790 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.790 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.790 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.790 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.958115+0000", 2024-09-10T18:29:19.790 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.790 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.791 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.791 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.791 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.958113+0000", 2024-09-10T18:29:19.791 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:29:19.791 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.791 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.791 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.791 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.958121+0000", 2024-09-10T18:29:19.791 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9330000000000007e-06 2024-09-10T18:29:19.791 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.791 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.791 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.791 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.958134+0000", 2024-09-10T18:29:19.791 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2266000000000001e-05 2024-09-10T18:29:19.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.792 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.958139+0000", 2024-09-10T18:29:19.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7359999999999998e-06 2024-09-10T18:29:19.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.792 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.958190+0000", 2024-09-10T18:29:19.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0309999999999998e-05 2024-09-10T18:29:19.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.958199+0000", 2024-09-10T18:29:19.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8380000000000008e-06 2024-09-10T18:29:19.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.000643+0000", 2024-09-10T18:29:19.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0424444900000001 2024-09-10T18:29:19.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:19.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.000765+0000", 2024-09-10T18:29:19.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012213099999999999 2024-09-10T18:29:19.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1", 2024-09-10T18:29:19.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.000893+0000", 2024-09-10T18:29:19.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000127866 2024-09-10T18:29:19.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:29:19.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.005746+0000", 2024-09-10T18:29:19.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0048532239999999997 2024-09-10T18:29:19.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.018618+0000", 2024-09-10T18:29:19.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012871426 2024-09-10T18:29:19.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.797 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.018673+0000", 2024-09-10T18:29:19.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4979000000000003e-05 2024-09-10T18:29:19.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.797 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:29:19.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.018708+0000", 2024-09-10T18:29:19.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5252000000000001e-05 2024-09-10T18:29:19.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.798 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.798 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.018730+0000", 2024-09-10T18:29:19.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2079999999999999e-05 2024-09-10T18:29:19.798 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.798 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.798 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.799 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:200482 7.d 7:be3c80eb:::benchmark_data_smithi028_77856_object79382:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-10T18:29:19.799 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:46.959884+0000", 2024-09-10T18:29:19.799 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 152.79357403700001, 2024-09-10T18:29:19.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0592035399999999, 2024-09-10T18:29:19.799 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.799 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:19.800 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.800 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.800 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.800 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 200482 2024-09-10T18:29:19.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.800 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.959884+0000", 2024-09-10T18:29:19.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.959884+0000", 2024-09-10T18:29:19.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.959883+0000", 2024-09-10T18:29:19.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:29:19.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.959891+0000", 2024-09-10T18:29:19.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.627e-06 2024-09-10T18:29:19.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.959900+0000", 2024-09-10T18:29:19.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2890000000000003e-06 2024-09-10T18:29:19.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.959912+0000", 2024-09-10T18:29:19.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1941e-05 2024-09-10T18:29:19.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.959952+0000", 2024-09-10T18:29:19.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9854999999999997e-05 2024-09-10T18:29:19.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.806 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.959957+0000", 2024-09-10T18:29:19.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8949999999999997e-06 2024-09-10T18:29:19.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.806 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.002278+0000", 2024-09-10T18:29:19.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.042321533 2024-09-10T18:29:19.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:19.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.002379+0000", 2024-09-10T18:29:19.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000100484 2024-09-10T18:29:19.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1", 2024-09-10T18:29:19.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.002464+0000", 2024-09-10T18:29:19.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5884000000000006e-05 2024-09-10T18:29:19.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:29:19.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.005772+0000", 2024-09-10T18:29:19.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033071089999999999 2024-09-10T18:29:19.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.808 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.018770+0000", 2024-09-10T18:29:19.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012998699000000001 2024-09-10T18:29:19.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.808 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019048+0000", 2024-09-10T18:29:19.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027806599999999999 2024-09-10T18:29:19.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.808 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:29:19.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019070+0000", 2024-09-10T18:29:19.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1549e-05 2024-09-10T18:29:19.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019088+0000", 2024-09-10T18:29:19.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8063e-05 2024-09-10T18:29:19.809 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.809 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.809 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.810 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:200528 7.d 7:b7461a54:::benchmark_data_smithi028_77856_object79428:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-10T18:29:19.810 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:47.007764+0000", 2024-09-10T18:29:19.810 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 152.745694515, 2024-09-10T18:29:19.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.011397653, 2024-09-10T18:29:19.810 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.810 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:19.810 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.810 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.810 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.810 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 200528 2024-09-10T18:29:19.810 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.810 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.810 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.007764+0000", 2024-09-10T18:29:19.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.007764+0000", 2024-09-10T18:29:19.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.007761+0000", 2024-09-10T18:29:19.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:29:19.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.812 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.007769+0000", 2024-09-10T18:29:19.812 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7309999999999993e-06 2024-09-10T18:29:19.812 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.812 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.007780+0000", 2024-09-10T18:29:19.812 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0706999999999999e-05 2024-09-10T18:29:19.812 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.812 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.007785+0000", 2024-09-10T18:29:19.813 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6439999999999999e-06 2024-09-10T18:29:19.813 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.813 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.813 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.813 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.007826+0000", 2024-09-10T18:29:19.813 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1681000000000003e-05 2024-09-10T18:29:19.813 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.813 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.813 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.813 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.007836+0000", 2024-09-10T18:29:19.813 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5959999999999993e-06 2024-09-10T18:29:19.813 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.813 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.814 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.814 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.003462+0000", 2024-09-10T18:29:19.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99562593399999999 2024-09-10T18:29:19.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.814 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.814 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:19.814 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.003563+0000", 2024-09-10T18:29:19.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101389 2024-09-10T18:29:19.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.814 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.814 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1", 2024-09-10T18:29:19.814 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.003670+0000", 2024-09-10T18:29:19.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106879 2024-09-10T18:29:19.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.815 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:29:19.815 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.007949+0000", 2024-09-10T18:29:19.815 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042792109999999998 2024-09-10T18:29:19.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.815 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.815 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.018810+0000", 2024-09-10T18:29:19.815 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010860787 2024-09-10T18:29:19.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.816 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.816 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019115+0000", 2024-09-10T18:29:19.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00030468599999999998 2024-09-10T18:29:19.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.816 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:29:19.816 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019141+0000", 2024-09-10T18:29:19.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6704999999999999e-05 2024-09-10T18:29:19.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.817 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019162+0000", 2024-09-10T18:29:19.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0136000000000002e-05 2024-09-10T18:29:19.817 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.817 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.817 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.818 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:200537 7.d 7:be2a904c:::benchmark_data_smithi028_77856_object79437:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-10T18:29:19.818 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:47.014154+0000", 2024-09-10T18:29:19.818 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 152.73930455199999, 2024-09-10T18:29:19.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0050720369999999, 2024-09-10T18:29:19.818 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.818 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:19.818 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.818 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.818 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.818 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 200537 2024-09-10T18:29:19.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.818 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.014154+0000", 2024-09-10T18:29:19.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.014154+0000", 2024-09-10T18:29:19.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.014152+0000", 2024-09-10T18:29:19.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:29:19.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.014160+0000", 2024-09-10T18:29:19.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3869999999999995e-06 2024-09-10T18:29:19.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.014177+0000", 2024-09-10T18:29:19.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6708999999999999e-05 2024-09-10T18:29:19.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.014183+0000", 2024-09-10T18:29:19.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0430000000000004e-06 2024-09-10T18:29:19.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.821 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.014227+0000", 2024-09-10T18:29:19.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4503999999999999e-05 2024-09-10T18:29:19.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.821 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.014235+0000", 2024-09-10T18:29:19.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4560000000000003e-06 2024-09-10T18:29:19.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.822 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.822 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.004599+0000", 2024-09-10T18:29:19.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99036454799999996 2024-09-10T18:29:19.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.822 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:19.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.004697+0000", 2024-09-10T18:29:19.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7565999999999994e-05 2024-09-10T18:29:19.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1", 2024-09-10T18:29:19.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.004805+0000", 2024-09-10T18:29:19.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010770200000000001 2024-09-10T18:29:19.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:29:19.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.010179+0000", 2024-09-10T18:29:19.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0053740020000000001 2024-09-10T18:29:19.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.018845+0000", 2024-09-10T18:29:19.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086661759999999994 2024-09-10T18:29:19.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019188+0000", 2024-09-10T18:29:19.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00034277299999999999 2024-09-10T18:29:19.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:29:19.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019206+0000", 2024-09-10T18:29:19.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8780999999999999e-05 2024-09-10T18:29:19.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019226+0000", 2024-09-10T18:29:19.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9514999999999999e-05 2024-09-10T18:29:19.825 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.825 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.825 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.826 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:200541 7.d 7:b0ae68b6:::benchmark_data_smithi028_77856_object79441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-10T18:29:19.826 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:47.016825+0000", 2024-09-10T18:29:19.826 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 152.73663363599999, 2024-09-10T18:29:19.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.002464555, 2024-09-10T18:29:19.826 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.826 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:19.826 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.826 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.827 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.827 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 200541 2024-09-10T18:29:19.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.827 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.016825+0000", 2024-09-10T18:29:19.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.016825+0000", 2024-09-10T18:29:19.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.016823+0000", 2024-09-10T18:29:19.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:29:19.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.016829+0000", 2024-09-10T18:29:19.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1e-06 2024-09-10T18:29:19.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.016840+0000", 2024-09-10T18:29:19.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0421000000000001e-05 2024-09-10T18:29:19.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.016846+0000", 2024-09-10T18:29:19.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8259999999999998e-06 2024-09-10T18:29:19.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.016893+0000", 2024-09-10T18:29:19.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7369e-05 2024-09-10T18:29:19.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.016900+0000", 2024-09-10T18:29:19.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7309999999999996e-06 2024-09-10T18:29:19.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.005806+0000", 2024-09-10T18:29:19.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.988906532 2024-09-10T18:29:19.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:19.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.005883+0000", 2024-09-10T18:29:19.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7152999999999998e-05 2024-09-10T18:29:19.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1", 2024-09-10T18:29:19.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.005949+0000", 2024-09-10T18:29:19.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5680999999999997e-05 2024-09-10T18:29:19.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:29:19.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.010200+0000", 2024-09-10T18:29:19.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042509679999999999 2024-09-10T18:29:19.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.018860+0000", 2024-09-10T18:29:19.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086603429999999992 2024-09-10T18:29:19.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019251+0000", 2024-09-10T18:29:19.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00039090499999999999 2024-09-10T18:29:19.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:29:19.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019272+0000", 2024-09-10T18:29:19.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0701000000000001e-05 2024-09-10T18:29:19.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019289+0000", 2024-09-10T18:29:19.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7212000000000001e-05 2024-09-10T18:29:19.833 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.834 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.834 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.834 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221732 7.c 7:3f916f55:::benchmark_data_smithi028_77856_object100632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:29:19.834 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.154233+0000", 2024-09-10T18:29:19.834 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 130.59922542499999, 2024-09-10T18:29:19.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.005077531, 2024-09-10T18:29:19.835 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.835 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:19.835 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.835 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.835 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.835 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221732 2024-09-10T18:29:19.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.835 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.154233+0000", 2024-09-10T18:29:19.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.154233+0000", 2024-09-10T18:29:19.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.154232+0000", 2024-09-10T18:29:19.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:29:19.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.154237+0000", 2024-09-10T18:29:19.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7870000000000001e-06 2024-09-10T18:29:19.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.154244+0000", 2024-09-10T18:29:19.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6539999999999999e-06 2024-09-10T18:29:19.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.154248+0000", 2024-09-10T18:29:19.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5260000000000002e-06 2024-09-10T18:29:19.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.154283+0000", 2024-09-10T18:29:19.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4727999999999997e-05 2024-09-10T18:29:19.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.839 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.154288+0000", 2024-09-10T18:29:19.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8109999999999999e-06 2024-09-10T18:29:19.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.839 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.143725+0000", 2024-09-10T18:29:19.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98943635100000005 2024-09-10T18:29:19.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:19.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.143866+0000", 2024-09-10T18:29:19.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000141015 2024-09-10T18:29:19.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:19.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.144013+0000", 2024-09-10T18:29:19.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000147009 2024-09-10T18:29:19.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:29:19.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.151226+0000", 2024-09-10T18:29:19.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0072131060000000004 2024-09-10T18:29:19.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159157+0000", 2024-09-10T18:29:19.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0079311820000000002 2024-09-10T18:29:19.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159208+0000", 2024-09-10T18:29:19.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1016000000000001e-05 2024-09-10T18:29:19.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:29:19.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159277+0000", 2024-09-10T18:29:19.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9000999999999999e-05 2024-09-10T18:29:19.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159310+0000", 2024-09-10T18:29:19.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3488999999999997e-05 2024-09-10T18:29:19.842 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.842 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.842 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.843 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221744 7.7 7:ee76f855:::benchmark_data_smithi028_77856_object100644:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:29:19.843 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.164178+0000", 2024-09-10T18:29:19.843 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 130.58928019699999, 2024-09-10T18:29:19.843 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99366592799999998, 2024-09-10T18:29:19.843 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.843 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:19.843 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.844 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.844 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.844 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221744 2024-09-10T18:29:19.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.844 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.164178+0000", 2024-09-10T18:29:19.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.164178+0000", 2024-09-10T18:29:19.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.164176+0000", 2024-09-10T18:29:19.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:29:19.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.164183+0000", 2024-09-10T18:29:19.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3169999999999998e-06 2024-09-10T18:29:19.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.164193+0000", 2024-09-10T18:29:19.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0107e-05 2024-09-10T18:29:19.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.164197+0000", 2024-09-10T18:29:19.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4649999999999996e-06 2024-09-10T18:29:19.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.164233+0000", 2024-09-10T18:29:19.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5547999999999999e-05 2024-09-10T18:29:19.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.848 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.848 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.164238+0000", 2024-09-10T18:29:19.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4299999999999997e-06 2024-09-10T18:29:19.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.848 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.848 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.151176+0000", 2024-09-10T18:29:19.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98693812199999997 2024-09-10T18:29:19.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.848 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:19.848 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.151273+0000", 2024-09-10T18:29:19.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6928000000000004e-05 2024-09-10T18:29:19.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-09-10T18:29:19.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.151341+0000", 2024-09-10T18:29:19.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7798000000000004e-05 2024-09-10T18:29:19.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:29:19.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.156298+0000", 2024-09-10T18:29:19.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0049565299999999998 2024-09-10T18:29:19.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.157700+0000", 2024-09-10T18:29:19.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014025660000000001 2024-09-10T18:29:19.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.157759+0000", 2024-09-10T18:29:19.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9197999999999998e-05 2024-09-10T18:29:19.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:29:19.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.157804+0000", 2024-09-10T18:29:19.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4696e-05 2024-09-10T18:29:19.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.157844+0000", 2024-09-10T18:29:19.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9960999999999999e-05 2024-09-10T18:29:19.852 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.852 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.852 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.852 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221749 7.c 7:35f4dcda:::benchmark_data_smithi028_77856_object100649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:29:19.852 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.167805+0000", 2024-09-10T18:29:19.852 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 130.585653638, 2024-09-10T18:29:19.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99166929100000001, 2024-09-10T18:29:19.852 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.853 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:19.853 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.853 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.853 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.853 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221749 2024-09-10T18:29:19.853 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.853 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.167805+0000", 2024-09-10T18:29:19.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.167805+0000", 2024-09-10T18:29:19.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.167803+0000", 2024-09-10T18:29:19.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:29:19.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.167809+0000", 2024-09-10T18:29:19.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7610000000000004e-06 2024-09-10T18:29:19.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.167821+0000", 2024-09-10T18:29:19.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1977000000000001e-05 2024-09-10T18:29:19.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.167827+0000", 2024-09-10T18:29:19.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9819999999999996e-06 2024-09-10T18:29:19.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.167878+0000", 2024-09-10T18:29:19.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0801999999999999e-05 2024-09-10T18:29:19.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.857 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.857 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.167885+0000", 2024-09-10T18:29:19.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.396e-06 2024-09-10T18:29:19.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.857 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.857 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.145946+0000", 2024-09-10T18:29:19.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97806059300000003 2024-09-10T18:29:19.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.857 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:19.857 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.146048+0000", 2024-09-10T18:29:19.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101739 2024-09-10T18:29:19.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:19.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.146117+0000", 2024-09-10T18:29:19.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9497000000000002e-05 2024-09-10T18:29:19.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:29:19.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.151232+0000", 2024-09-10T18:29:19.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0051152749999999999 2024-09-10T18:29:19.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159361+0000", 2024-09-10T18:29:19.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0081289810000000004 2024-09-10T18:29:19.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159403+0000", 2024-09-10T18:29:19.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1100999999999998e-05 2024-09-10T18:29:19.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:29:19.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159442+0000", 2024-09-10T18:29:19.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9174000000000001e-05 2024-09-10T18:29:19.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159474+0000", 2024-09-10T18:29:19.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2327999999999999e-05 2024-09-10T18:29:19.860 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.860 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.860 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.860 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221753 7.c 7:3c815616:::benchmark_data_smithi028_77856_object100653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:29:19.860 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.172324+0000", 2024-09-10T18:29:19.860 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 130.581134053, 2024-09-10T18:29:19.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98727850699999997, 2024-09-10T18:29:19.861 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.861 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:19.861 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.861 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.861 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.861 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221753 2024-09-10T18:29:19.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.861 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.172324+0000", 2024-09-10T18:29:19.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.172324+0000", 2024-09-10T18:29:19.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.172323+0000", 2024-09-10T18:29:19.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:29:19.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.172328+0000", 2024-09-10T18:29:19.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4770000000000002e-06 2024-09-10T18:29:19.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.172338+0000", 2024-09-10T18:29:19.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7089999999999997e-06 2024-09-10T18:29:19.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.172343+0000", 2024-09-10T18:29:19.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0100000000000003e-06 2024-09-10T18:29:19.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.864 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.864 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.172388+0000", 2024-09-10T18:29:19.865 INFO:tasks.thrashosds.thrasher:j is {'epoch': 208, 'fsid': '5eababb2-6fa0-11ef-bcea-c7b262605968', 'created': '2024-09-10T18:16:20.680373+0000', 'modified': '2024-09-10T18:29:16.141935+0000', 'last_up_change': '2024-09-10T18:26:26.611338+0000', 'last_in_change': '2024-09-10T18:21:02.114621+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T18:19:24.013536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '136', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-10T18:21:30.065209+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 62, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T18:21:30.506441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T18:21:32.008130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T18:21:34.060571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T18:21:36.127330+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-10T18:27:51.966788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '189', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'f1c834d9-252a-4077-b25e-169181d70251', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 180, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 3188372351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 3188372351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 3188372351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 3188372351}]}, 'public_addr': '172.21.15.28:6801/3188372351', 'cluster_addr': '172.21.15.28:6802/3188372351', 'heartbeat_back_addr': '172.21.15.28:6804/3188372351', 'heartbeat_front_addr': '172.21.15.28:6803/3188372351', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '81f1855d-b946-4ca3-b246-38977b9e20fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 180, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1369912678}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1369912678}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1369912678}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1369912678}]}, 'public_addr': '172.21.15.28:6805/1369912678', 'cluster_addr': '172.21.15.28:6806/1369912678', 'heartbeat_back_addr': '172.21.15.28:6808/1369912678', 'heartbeat_front_addr': '172.21.15.28:6807/1369912678', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2ad55d2a-36fa-4756-bf22-58d4b049486e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 1243406003}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 1243406003}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 1243406003}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1243406003}]}, 'public_addr': '172.21.15.28:6809/1243406003', 'cluster_addr': '172.21.15.28:6810/1243406003', 'heartbeat_back_addr': '172.21.15.28:6812/1243406003', 'heartbeat_front_addr': '172.21.15.28:6811/1243406003', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '242729a5-a5e8-44d5-97c8-14d14e970bff', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 180, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 67703283}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 67703283}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 67703283}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 67703283}]}, 'public_addr': '172.21.15.28:6813/67703283', 'cluster_addr': '172.21.15.28:6814/67703283', 'heartbeat_back_addr': '172.21.15.28:6816/67703283', 'heartbeat_front_addr': '172.21.15.28:6815/67703283', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6e38b8ef-e6ed-493f-8eca-755354754787', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 180, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6800', 'nonce': 1619669076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 1619669076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 1619669076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6802', 'nonce': 1619669076}]}, 'public_addr': '172.21.15.204:6800/1619669076', 'cluster_addr': '172.21.15.204:6801/1619669076', 'heartbeat_back_addr': '172.21.15.204:6803/1619669076', 'heartbeat_front_addr': '172.21.15.204:6802/1619669076', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a92e0441-66c9-498e-a0ac-6465acdbb1f5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 180, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6804', 'nonce': 2717802880}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 2717802880}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 2717802880}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6806', 'nonce': 2717802880}]}, 'public_addr': '172.21.15.204:6804/2717802880', 'cluster_addr': '172.21.15.204:6805/2717802880', 'heartbeat_back_addr': '172.21.15.204:6807/2717802880', 'heartbeat_front_addr': '172.21.15.204:6806/2717802880', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '12b02875-6653-45ed-8d2b-b4d33bd88525', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 46, 'last_clean_end': 94, 'up_from': 144, 'up_thru': 180, 'down_at': 95, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6808', 'nonce': 3479311452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 3479311452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 3479311452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6810', 'nonce': 3479311452}]}, 'public_addr': '172.21.15.204:6808/3479311452', 'cluster_addr': '172.21.15.204:6809/3479311452', 'heartbeat_back_addr': '172.21.15.204:6811/3479311452', 'heartbeat_front_addr': '172.21.15.204:6810/3479311452', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0c4f7aa9-ff95-4192-a146-ca576eee907f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 144, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6812', 'nonce': 3511791879}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 3511791879}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 3511791879}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6814', 'nonce': 3511791879}]}, 'public_addr': '172.21.15.204:6812/3511791879', 'cluster_addr': '172.21.15.204:6813/3511791879', 'heartbeat_back_addr': '172.21.15.204:6815/3511791879', 'heartbeat_front_addr': '172.21.15.204:6814/3511791879', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:28:13.795615+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:28:44.945813+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:29:10.889906+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:28:10.447709+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:28:53.256166+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:29:13.113559+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:29:15.622169+0000', 'dead_epoch': 94}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:28:31.162243+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/1745574083': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/3866220552': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:6800/3760775432': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:0/1746797014': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:0/1029045895': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:6800/3310949126': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/2874160765': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/2620179974': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/421194352': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:6800/1077377902': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/3428757300': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/1759116360': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/2564333506': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/717120167': '2024-09-11T18:22:08.678985+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T18:29:19.865 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.a 2024-09-10T18:29:19.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd rm-pg-upmap-items 4.a 2024-09-10T18:29:19.866 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:19.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4990000000000001e-05 2024-09-10T18:29:19.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.172395+0000", 2024-09-10T18:29:19.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.934e-06 2024-09-10T18:29:19.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.147521+0000", 2024-09-10T18:29:19.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97512613999999997 2024-09-10T18:29:19.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:19.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.147632+0000", 2024-09-10T18:29:19.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000110757 2024-09-10T18:29:19.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:19.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.147758+0000", 2024-09-10T18:29:19.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125702 2024-09-10T18:29:19.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:29:19.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.153601+0000", 2024-09-10T18:29:19.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058435529999999996 2024-09-10T18:29:19.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159501+0000", 2024-09-10T18:29:19.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058994069999999997 2024-09-10T18:29:19.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.870 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159545+0000", 2024-09-10T18:29:19.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3912999999999998e-05 2024-09-10T18:29:19.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.870 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:29:19.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159571+0000", 2024-09-10T18:29:19.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6795e-05 2024-09-10T18:29:19.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159603+0000", 2024-09-10T18:29:19.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1520999999999999e-05 2024-09-10T18:29:19.871 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.871 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.871 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.871 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221754 7.c 7:337bf24c:::benchmark_data_smithi028_77856_object100654:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:29:19.871 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.173248+0000", 2024-09-10T18:29:19.871 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 130.58021001, 2024-09-10T18:29:19.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98651236499999995, 2024-09-10T18:29:19.871 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.872 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:19.872 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.872 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.872 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.872 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221754 2024-09-10T18:29:19.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.872 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.173248+0000", 2024-09-10T18:29:19.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.173248+0000", 2024-09-10T18:29:19.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.173247+0000", 2024-09-10T18:29:19.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:29:19.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.173252+0000", 2024-09-10T18:29:19.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6199999999999998e-06 2024-09-10T18:29:19.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.173261+0000", 2024-09-10T18:29:19.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6640000000000004e-06 2024-09-10T18:29:19.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.173265+0000", 2024-09-10T18:29:19.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9509999999999999e-06 2024-09-10T18:29:19.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.173301+0000", 2024-09-10T18:29:19.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6844999999999997e-05 2024-09-10T18:29:19.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.173306+0000", 2024-09-10T18:29:19.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6530000000000002e-06 2024-09-10T18:29:19.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.148983+0000", 2024-09-10T18:29:19.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97567699399999996 2024-09-10T18:29:19.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:19.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.149052+0000", 2024-09-10T18:29:19.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9177999999999994e-05 2024-09-10T18:29:19.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:19.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.149119+0000", 2024-09-10T18:29:19.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7175999999999995e-05 2024-09-10T18:29:19.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:29:19.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.153606+0000", 2024-09-10T18:29:19.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0044868299999999998 2024-09-10T18:29:19.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159644+0000", 2024-09-10T18:29:19.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0060377709999999999 2024-09-10T18:29:19.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159696+0000", 2024-09-10T18:29:19.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1711e-05 2024-09-10T18:29:19.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:29:19.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159731+0000", 2024-09-10T18:29:19.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5259000000000002e-05 2024-09-10T18:29:19.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159761+0000", 2024-09-10T18:29:19.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9757999999999999e-05 2024-09-10T18:29:19.878 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.879 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.879 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.879 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221759 7.c 7:3758a2a2:::benchmark_data_smithi028_77856_object100659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:29:19.879 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.174377+0000", 2024-09-10T18:29:19.879 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 130.57908112699999, 2024-09-10T18:29:19.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98546984100000001, 2024-09-10T18:29:19.879 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.879 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:19.879 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.879 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.879 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.880 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221759 2024-09-10T18:29:19.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.880 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.174377+0000", 2024-09-10T18:29:19.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.174377+0000", 2024-09-10T18:29:19.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.174376+0000", 2024-09-10T18:29:19.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:29:19.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.174385+0000", 2024-09-10T18:29:19.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6370000000000005e-06 2024-09-10T18:29:19.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.174396+0000", 2024-09-10T18:29:19.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1311999999999999e-05 2024-09-10T18:29:19.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.174401+0000", 2024-09-10T18:29:19.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3990000000000003e-06 2024-09-10T18:29:19.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.174431+0000", 2024-09-10T18:29:19.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9558999999999999e-05 2024-09-10T18:29:19.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.174437+0000", 2024-09-10T18:29:19.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0859999999999998e-06 2024-09-10T18:29:19.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.149913+0000", 2024-09-10T18:29:19.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97547597100000005 2024-09-10T18:29:19.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:19.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.150022+0000", 2024-09-10T18:29:19.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108928 2024-09-10T18:29:19.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:19.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.150125+0000", 2024-09-10T18:29:19.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010319600000000001 2024-09-10T18:29:19.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:29:19.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.156072+0000", 2024-09-10T18:29:19.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059466550000000003 2024-09-10T18:29:19.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159665+0000", 2024-09-10T18:29:19.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035927789999999999 2024-09-10T18:29:19.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159791+0000", 2024-09-10T18:29:19.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012606299999999999 2024-09-10T18:29:19.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:29:19.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159817+0000", 2024-09-10T18:29:19.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6684999999999999e-05 2024-09-10T18:29:19.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159847+0000", 2024-09-10T18:29:19.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9848e-05 2024-09-10T18:29:19.887 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.887 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.887 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.887 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221761 7.7 7:ed6e3b57:::benchmark_data_smithi028_77856_object100661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:29:19.887 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.178273+0000", 2024-09-10T18:29:19.887 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 130.575185465, 2024-09-10T18:29:19.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98234119399999997, 2024-09-10T18:29:19.887 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.887 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:19.887 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.887 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.888 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.888 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221761 2024-09-10T18:29:19.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.888 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.178273+0000", 2024-09-10T18:29:19.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.178273+0000", 2024-09-10T18:29:19.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.178271+0000", 2024-09-10T18:29:19.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:29:19.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.890 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.178278+0000", 2024-09-10T18:29:19.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0010000000000004e-06 2024-09-10T18:29:19.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.890 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.890 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.178289+0000", 2024-09-10T18:29:19.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0807e-05 2024-09-10T18:29:19.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.890 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.890 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.178294+0000", 2024-09-10T18:29:19.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2399999999999998e-06 2024-09-10T18:29:19.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.178338+0000", 2024-09-10T18:29:19.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3837000000000001e-05 2024-09-10T18:29:19.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.178347+0000", 2024-09-10T18:29:19.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8510000000000005e-06 2024-09-10T18:29:19.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.153274+0000", 2024-09-10T18:29:19.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97492656499999997 2024-09-10T18:29:19.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:19.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.153375+0000", 2024-09-10T18:29:19.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101545 2024-09-10T18:29:19.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-09-10T18:29:19.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.153484+0000", 2024-09-10T18:29:19.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010891300000000001 2024-09-10T18:29:19.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:29:19.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.157887+0000", 2024-09-10T18:29:19.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0044034080000000001 2024-09-10T18:29:19.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160502+0000", 2024-09-10T18:29:19.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002614487 2024-09-10T18:29:19.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160542+0000", 2024-09-10T18:29:19.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0200999999999997e-05 2024-09-10T18:29:19.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:29:19.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160582+0000", 2024-09-10T18:29:19.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9826999999999997e-05 2024-09-10T18:29:19.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160614+0000", 2024-09-10T18:29:19.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2214e-05 2024-09-10T18:29:19.894 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.894 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.894 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.895 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221766 7.c 7:3aafddb5:::benchmark_data_smithi028_77856_object100666:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:29:19.895 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.180051+0000", 2024-09-10T18:29:19.895 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 130.573407573, 2024-09-10T18:29:19.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97996098200000004, 2024-09-10T18:29:19.895 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.895 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:19.895 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.895 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.895 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.895 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221766 2024-09-10T18:29:19.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.896 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.180051+0000", 2024-09-10T18:29:19.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.180051+0000", 2024-09-10T18:29:19.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.180050+0000", 2024-09-10T18:29:19.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:29:19.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.180055+0000", 2024-09-10T18:29:19.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9400000000000001e-06 2024-09-10T18:29:19.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.180065+0000", 2024-09-10T18:29:19.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9660000000000008e-06 2024-09-10T18:29:19.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.180069+0000", 2024-09-10T18:29:19.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1869999999999999e-06 2024-09-10T18:29:19.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.180117+0000", 2024-09-10T18:29:19.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7732000000000003e-05 2024-09-10T18:29:19.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.180124+0000", 2024-09-10T18:29:19.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4399999999999999e-06 2024-09-10T18:29:19.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.151272+0000", 2024-09-10T18:29:19.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97114836199999999 2024-09-10T18:29:19.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:19.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.151375+0000", 2024-09-10T18:29:19.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010259000000000001 2024-09-10T18:29:19.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:19.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.151479+0000", 2024-09-10T18:29:19.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104035 2024-09-10T18:29:19.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:29:19.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.156079+0000", 2024-09-10T18:29:19.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0045997909999999998 2024-09-10T18:29:19.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159886+0000", 2024-09-10T18:29:19.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038077250000000001 2024-09-10T18:29:19.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159945+0000", 2024-09-10T18:29:19.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8746999999999998e-05 2024-09-10T18:29:19.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:29:19.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159982+0000", 2024-09-10T18:29:19.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6797e-05 2024-09-10T18:29:19.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160012+0000", 2024-09-10T18:29:19.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9785e-05 2024-09-10T18:29:19.904 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.904 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.904 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.905 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221769 7.c 7:3717c799:::benchmark_data_smithi028_77856_object100669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:29:19.905 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.182048+0000", 2024-09-10T18:29:19.905 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 130.57141075300001, 2024-09-10T18:29:19.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97804954, 2024-09-10T18:29:19.905 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.905 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:19.905 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.905 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.905 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.905 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221769 2024-09-10T18:29:19.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.905 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.182048+0000", 2024-09-10T18:29:19.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.182048+0000", 2024-09-10T18:29:19.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.182046+0000", 2024-09-10T18:29:19.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:29:19.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.182052+0000", 2024-09-10T18:29:19.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6749999999999999e-06 2024-09-10T18:29:19.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.182064+0000", 2024-09-10T18:29:19.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1534e-05 2024-09-10T18:29:19.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.182070+0000", 2024-09-10T18:29:19.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.742e-06 2024-09-10T18:29:19.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.182108+0000", 2024-09-10T18:29:19.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8028000000000003e-05 2024-09-10T18:29:19.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.182113+0000", 2024-09-10T18:29:19.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.062e-06 2024-09-10T18:29:19.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.152330+0000", 2024-09-10T18:29:19.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97021630999999997 2024-09-10T18:29:19.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:19.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.152427+0000", 2024-09-10T18:29:19.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7149000000000001e-05 2024-09-10T18:29:19.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:19.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.152529+0000", 2024-09-10T18:29:19.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102105 2024-09-10T18:29:19.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:29:19.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.158720+0000", 2024-09-10T18:29:19.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061913080000000004 2024-09-10T18:29:19.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159915+0000", 2024-09-10T18:29:19.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001194465 2024-09-10T18:29:19.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160046+0000", 2024-09-10T18:29:19.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013089300000000001 2024-09-10T18:29:19.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:29:19.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160068+0000", 2024-09-10T18:29:19.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1968999999999999e-05 2024-09-10T18:29:19.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160097+0000", 2024-09-10T18:29:19.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9626000000000001e-05 2024-09-10T18:29:19.912 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.913 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.913 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.913 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221770 7.7 7:e05c9776:::benchmark_data_smithi028_77856_object100670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:29:19.913 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.185332+0000", 2024-09-10T18:29:19.913 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 130.56812671899999, 2024-09-10T18:29:19.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98126123099999996, 2024-09-10T18:29:19.914 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:29:19.914 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:19.914 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:29:19.914 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:29:19.914 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:19.914 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221770 2024-09-10T18:29:19.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.914 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:29:19.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:29:19.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.185332+0000", 2024-09-10T18:29:19.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:29:19.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.185332+0000", 2024-09-10T18:29:19.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:29:19.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:29:19.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.185330+0000", 2024-09-10T18:29:19.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:29:19.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:29:19.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.185337+0000", 2024-09-10T18:29:19.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3579999999999999e-06 2024-09-10T18:29:19.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:29:19.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.185348+0000", 2024-09-10T18:29:19.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1902000000000001e-05 2024-09-10T18:29:19.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:29:19.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.185354+0000", 2024-09-10T18:29:19.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0260000000000004e-06 2024-09-10T18:29:19.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.185399+0000", 2024-09-10T18:29:19.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4299999999999999e-05 2024-09-10T18:29:19.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:29:19.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.185404+0000", 2024-09-10T18:29:19.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9230000000000001e-06 2024-09-10T18:29:19.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:29:19.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.154919+0000", 2024-09-10T18:29:19.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.96951485999999998 2024-09-10T18:29:19.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:29:19.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.155003+0000", 2024-09-10T18:29:19.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4492000000000004e-05 2024-09-10T18:29:19.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-09-10T18:29:19.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.155099+0000", 2024-09-10T18:29:19.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5878000000000006e-05 2024-09-10T18:29:19.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:29:19.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160467+0000", 2024-09-10T18:29:19.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0053681989999999997 2024-09-10T18:29:19.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160672+0000", 2024-09-10T18:29:19.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020532999999999999 2024-09-10T18:29:19.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:19.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.166507+0000", 2024-09-10T18:29:19.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058346630000000003 2024-09-10T18:29:19.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:29:19.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.166554+0000", 2024-09-10T18:29:19.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7089000000000003e-05 2024-09-10T18:29:19.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:29:19.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:29:19.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:29:19.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.166593+0000", 2024-09-10T18:29:19.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8670000000000001e-05 2024-09-10T18:29:19.920 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.921 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.921 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.921 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:29:19.921 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:29:19.921 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:29:19.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:19.957 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:19.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:20.058 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:20.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:20.121 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:20.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:20.123 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 73 16 114006 113990 97.5672 67.25 0.0240459 0.0102439 2024-09-10T18:29:20.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:20.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:20.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:20.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:20.370 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:20.398 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:20.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:20.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:20.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:20 smithi028 ceph-mon[29291]: pgmap v351: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 711 B/s rd, 50 MiB/s wr, 800 op/s 2024-09-10T18:29:20.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:20 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:29:20.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:20 smithi028 ceph-mon[29291]: from='client.25120 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:29:20.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:20 smithi028 ceph-mon[29291]: osdmap e209: 8 total, 8 up, 8 in 2024-09-10T18:29:20.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:20 smithi028 ceph-mon[39220]: pgmap v351: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 711 B/s rd, 50 MiB/s wr, 800 op/s 2024-09-10T18:29:20.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:20 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:29:20.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:20 smithi028 ceph-mon[39220]: from='client.25120 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:29:20.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:20 smithi028 ceph-mon[39220]: osdmap e209: 8 total, 8 up, 8 in 2024-09-10T18:29:20.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:20.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:20 smithi204 ceph-mon[32357]: pgmap v351: 158 pgs: 158 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 711 B/s rd, 50 MiB/s wr, 800 op/s 2024-09-10T18:29:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:20 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:29:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:20 smithi204 ceph-mon[32357]: from='client.25120 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:29:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:20 smithi204 ceph-mon[32357]: osdmap e209: 8 total, 8 up, 8 in 2024-09-10T18:29:20.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:20.691 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:20.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:20.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:20.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:20.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T18:29:20.836 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:20.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:20.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:20.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:20.988 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:20.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:20.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:21.089 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:21.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:21.123 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:21.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:21.123 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 74 16 115116 115100 97.1862 69.375 0.0163563 0.0102845 2024-09-10T18:29:21.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:21.257 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:21.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:21.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:21.359 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:21.385 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:21.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:21.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:29:21.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:21.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:21.618 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:29:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:29:21] "GET /metrics HTTP/1.1" 200 224292 "" "Prometheus/2.43.0" 2024-09-10T18:29:21.668 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:21.696 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:21.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:21.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:21.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:21.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:21.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:21.902 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:21 smithi028 ceph-mon[39220]: pgmap v353: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-10T18:29:21.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:21 smithi028 ceph-mon[29291]: pgmap v353: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-10T18:29:21.939 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:21.972 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:21.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:22.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:22.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:21 smithi204 ceph-mon[32357]: pgmap v353: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-10T18:29:22.123 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 75 16 116325 116309 96.8979 75.5625 0.0137909 0.0103153 2024-09-10T18:29:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:22.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:22.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:22.253 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:22.293 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:22.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:22.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:22.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:22.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:22.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:22.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:22.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:29:22.532 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:22.566 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:22.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:22.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:22.576 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:29:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:29:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:29:22.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:22.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:22.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:22.739 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:29:22.794 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:29:22.795 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:29:22.829 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:29:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:29:22.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:22.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:22 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3751130290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:29:22.869 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:29:22.869 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:29:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:22.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:22.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:22.901 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:29:22.901 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:29:22.932 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:29:22.932 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:29:22.965 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:29:22.966 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:29:22.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:23.000 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:29:23.000 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:29:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:23.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:23.036 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:29:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:29:23.053 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:22 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3751130290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:29:23.053 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:22 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3751130290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:29:23.123 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 76 16 117457 117441 96.5538 70.75 0.00829779 0.0103521 2024-09-10T18:29:23.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:23.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:23.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:23.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:23.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:23.314 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:23.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:23.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:23.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:23.458 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:23.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:23.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:23.600 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:23.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:23.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:23.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:23.740 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:23.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:23.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:23.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:23.890 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:23.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:23.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:23.992 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:24.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:23 smithi028 ceph-mon[39220]: pgmap v354: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-10T18:29:24.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:23 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:29:24.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:23 smithi028 ceph-mon[29291]: pgmap v354: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-10T18:29:24.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:23 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:29:24.024 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:24.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:24.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:24.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:23 smithi204 ceph-mon[32357]: pgmap v354: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-10T18:29:24.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:23 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:29:24.123 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 77 16 118671 118655 96.2852 75.875 0.00921275 0.0103813 2024-09-10T18:29:24.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:24.166 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:24.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:24.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:24.310 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:24.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:24.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:24.412 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:24.445 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:24.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:24.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:24.547 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:24.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:24.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:24.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:24.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:29:24 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:29:24.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:24.730 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:24.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:24.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:24.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:24.869 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:24.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:24.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:24.971 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:25.004 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:25.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:25.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:25.123 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 78 16 119946 119930 96.0724 79.6875 0.00661114 0.0104037 2024-09-10T18:29:25.150 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:25.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:25.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:25.252 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:25.291 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:25.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:25.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:25.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:25.424 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:25.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:25.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:25.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:25.557 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:25.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:25.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:25.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:25.687 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:25.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:25.789 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:25.816 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:25.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:25.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:25.918 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:25.952 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:25.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:25.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:25.971 INFO:teuthology.orchestra.run.smithi028.stderr:clear 4.a pg_upmap_items mapping 2024-09-10T18:29:25.971 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:25 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:25.971 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:25 smithi028 ceph-mon[39220]: pgmap v355: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 51 MiB/s wr, 814 op/s 2024-09-10T18:29:25.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:25 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2505769469' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.a"}]: dispatch 2024-09-10T18:29:25.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:25 smithi028 ceph-mon[39220]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.a"}]: dispatch 2024-09-10T18:29:25.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:25 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:25.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:25 smithi028 ceph-mon[29291]: pgmap v355: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 51 MiB/s wr, 814 op/s 2024-09-10T18:29:25.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:25 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2505769469' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.a"}]: dispatch 2024-09-10T18:29:25.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:25 smithi028 ceph-mon[29291]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.a"}]: dispatch 2024-09-10T18:29:26.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:26.083 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:26.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:26.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:26.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 79 16 121181 121165 95.8332 77.1875 0.00767696 0.0104299 2024-09-10T18:29:26.184 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:26.218 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:26.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:26.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:26.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:25 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:26.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:25 smithi204 ceph-mon[32357]: pgmap v355: 158 pgs: 158 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 51 MiB/s wr, 814 op/s 2024-09-10T18:29:26.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:25 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2505769469' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.a"}]: dispatch 2024-09-10T18:29:26.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:25 smithi204 ceph-mon[32357]: from='client.25129 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.a"}]: dispatch 2024-09-10T18:29:26.366 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:26.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:26.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:29:26.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:26.502 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:26.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:26.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:26.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:26.650 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:26.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:26.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:26.752 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:26.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:26.787 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:26.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:26.888 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:26.926 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:26.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:26.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:27.028 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:27.055 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:26 smithi028 ceph-mon[39220]: from='client.25129 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.a"}]': finished 2024-09-10T18:29:27.055 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:26 smithi028 ceph-mon[39220]: osdmap e210: 8 total, 8 up, 8 in 2024-09-10T18:29:27.057 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:26 smithi028 ceph-mon[29291]: from='client.25129 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.a"}]': finished 2024-09-10T18:29:27.057 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:26 smithi028 ceph-mon[29291]: osdmap e210: 8 total, 8 up, 8 in 2024-09-10T18:29:27.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:27.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:27.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:27.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:29:27.124004+0000 min lat: 0.00125336 max lat: 0.134949 avg lat: 0.0104579 2024-09-10T18:29:27.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:29:27.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 80 16 122386 122370 95.5756 75.3125 0.0276261 0.0104579 2024-09-10T18:29:27.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:27.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:27.197 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:27.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:27.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:27.336 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:27.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:27.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:27.349 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:29:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:26 smithi204 ceph-mon[32357]: from='client.25129 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.a"}]': finished 2024-09-10T18:29:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:26 smithi204 ceph-mon[32357]: osdmap e210: 8 total, 8 up, 8 in 2024-09-10T18:29:27.390 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:29:27.390 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:29:27.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:27.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:27.487 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:27.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:27.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:27.632 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:27.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:27.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:27.734 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:27.762 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:27.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:27.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:27.864 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:27.903 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:27.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:27.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:28.005 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:28.034 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:28.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:28.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:28.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:27 smithi028 ceph-mon[39220]: pgmap v357: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T18:29:28.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:27 smithi028 ceph-mon[39220]: osdmap e211: 8 total, 8 up, 8 in 2024-09-10T18:29:28.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:27 smithi028 ceph-mon[29291]: pgmap v357: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T18:29:28.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:27 smithi028 ceph-mon[29291]: osdmap e211: 8 total, 8 up, 8 in 2024-09-10T18:29:28.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 81 16 123618 123602 95.3463 77 0.00781969 0.0104836 2024-09-10T18:29:28.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:28.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:28.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:28.283 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:28.315 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:28.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:28.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:27 smithi204 ceph-mon[32357]: pgmap v357: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T18:29:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:27 smithi204 ceph-mon[32357]: osdmap e211: 8 total, 8 up, 8 in 2024-09-10T18:29:28.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-10T18:29:28.417 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:28.450 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:28.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:28.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:28.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:28.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:28.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:28.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:28.722 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:28.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:28.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:28.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:29:28.874 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:28.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:28.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:28.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:29.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:29.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:29.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:29.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 82 16 124839 124823 95.1141 76.3125 0.0110488 0.0105091 2024-09-10T18:29:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:29.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:29.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:29.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:29.304 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:29.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:29.406 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:29.435 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:29.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:29.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:29.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:29.584 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:29.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:29.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:29.734 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:29.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:29.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:29.835 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:29.863 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:29.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:29.964 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:29.995 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:29.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:30.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:30.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 83 16 126072 126056 94.8966 77.0625 0.00553951 0.0105332 2024-09-10T18:29:30.129 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:30.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:30.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:30.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:29 smithi028 ceph-mon[29291]: pgmap v359: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 748 op/s 2024-09-10T18:29:30.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:29 smithi028 ceph-mon[39220]: pgmap v359: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 748 op/s 2024-09-10T18:29:30.268 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:30.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:30.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:29 smithi204 ceph-mon[32357]: pgmap v359: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 748 op/s 2024-09-10T18:29:30.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:30.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:30.410 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:30.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:30.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:30.548 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:30.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:30.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:30.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:30.690 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:30.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:30.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:30.789 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:29:30.792 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:30.818 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:30.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:30.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:30.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:30.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:30.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:31.051 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:31.080 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:31.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:31.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:31.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 84 16 127292 127276 94.6745 76.25 0.0127975 0.0105569 2024-09-10T18:29:31.182 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:31.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:31.216 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:31.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:31.317 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:31.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:31.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:31.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:31.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:31.471 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:29:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:29:31] "GET /metrics HTTP/1.1" 200 224298 "" "Prometheus/2.43.0" 2024-09-10T18:29:31.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:31.482 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:31.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:31.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:31.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:31.627 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:31.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:31.728 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:31.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:31 smithi028 ceph-mon[39220]: pgmap v360: 158 pgs: 158 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.51k op/s; 13 B/s, 1 objects/s recovering 2024-09-10T18:29:31.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:31 smithi028 ceph-mon[29291]: pgmap v360: 158 pgs: 158 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.51k op/s; 13 B/s, 1 objects/s recovering 2024-09-10T18:29:31.758 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:31.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:31.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:31.848 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:29:31.848 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:29:31.849 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-10T18:29:31.849 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-10T18:29:31.849 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-10T18:29:31.849 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-10T18:29:31.860 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:31.931 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:31.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:31.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:32.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:32.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:32.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:31 smithi204 ceph-mon[32357]: pgmap v360: 158 pgs: 158 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.51k op/s; 13 B/s, 1 objects/s recovering 2024-09-10T18:29:32.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 85 16 128469 128453 94.4261 73.5625 0.0109661 0.010586 2024-09-10T18:29:32.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:32.207 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:32.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:32.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:32.251 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:29:32.309 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:32.351 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:32.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:32.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:32.388 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:29:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:29:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:29:32.453 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:32.489 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:32.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:32.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:32.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:32.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:32.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:32.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:32.725 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:29:32.770 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:32.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:32.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:32.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:32.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:32.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:32.994 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:32 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4222465178' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:29:32.994 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:32 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4222465178' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:29:33.018 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:33.045 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:33.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:33.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:33.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:32 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4222465178' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:29:33.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 86 16 129630 129614 94.1719 72.5625 0.00474996 0.0106142 2024-09-10T18:29:33.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:33.176 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:33.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:33.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:33.278 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:33.309 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:33.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:33.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:33.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:33.461 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:33.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:33.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:33.587 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:33.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:33.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:33.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:33.736 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:33.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:33.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:33.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:33.875 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:33.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:33.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:33.972 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:33 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:29:33.973 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:33 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4222465178' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:29:33.973 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:33 smithi204 ceph-mon[32357]: osdmap e212: 8 total, 8 up, 8 in 2024-09-10T18:29:33.973 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:33 smithi204 ceph-mon[32357]: pgmap v362: 158 pgs: 158 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 738 B/s rd, 55 MiB/s wr, 884 op/s; 15 B/s, 1 objects/s recovering 2024-09-10T18:29:33.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:34.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:33 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:29:34.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:33 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4222465178' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:29:34.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:33 smithi028 ceph-mon[39220]: osdmap e212: 8 total, 8 up, 8 in 2024-09-10T18:29:34.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:33 smithi028 ceph-mon[39220]: pgmap v362: 158 pgs: 158 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 738 B/s rd, 55 MiB/s wr, 884 op/s; 15 B/s, 1 objects/s recovering 2024-09-10T18:29:34.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:33 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:29:34.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:33 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4222465178' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:29:34.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:33 smithi028 ceph-mon[29291]: osdmap e212: 8 total, 8 up, 8 in 2024-09-10T18:29:34.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:33 smithi028 ceph-mon[29291]: pgmap v362: 158 pgs: 158 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 738 B/s rd, 55 MiB/s wr, 884 op/s; 15 B/s, 1 objects/s recovering 2024-09-10T18:29:34.006 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:34.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:34.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:34.108 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:34.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 87 15 130795 130780 93.927 72.875 0.0151485 0.0106421 2024-09-10T18:29:34.148 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:34.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:34.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:34.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:34.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:34.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:34.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:34.434 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:34.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:34.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:34.536 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:34.556 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:29:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:29:34.573 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:34.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:34.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:34.675 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:34.704 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:34.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:34.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:34.757 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:29:34.757 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:29:34.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:34.838 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:34 smithi204 ceph-mon[32357]: 5.5 scrub starts 2024-09-10T18:29:34.838 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:34 smithi204 ceph-mon[32357]: 5.5 scrub ok 2024-09-10T18:29:34.838 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:34 smithi204 ceph-mon[32357]: 3.2 scrub starts 2024-09-10T18:29:34.838 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:34 smithi204 ceph-mon[32357]: 3.2 scrub ok 2024-09-10T18:29:34.838 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:34 smithi204 ceph-mon[32357]: 6.3 scrub starts 2024-09-10T18:29:34.838 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:34 smithi204 ceph-mon[32357]: 6.3 scrub ok 2024-09-10T18:29:34.838 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:34 smithi204 ceph-mon[32357]: 4.1 scrub starts 2024-09-10T18:29:34.838 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:34 smithi204 ceph-mon[32357]: 4.1 scrub ok 2024-09-10T18:29:34.839 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:34 smithi204 ceph-mon[32357]: 3.10 scrub starts 2024-09-10T18:29:34.839 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:34 smithi204 ceph-mon[32357]: 3.10 scrub ok 2024-09-10T18:29:34.848 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:34.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:34.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:34.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:34.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:34.991 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:34.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:35.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[39220]: 5.5 scrub starts 2024-09-10T18:29:35.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[39220]: 5.5 scrub ok 2024-09-10T18:29:35.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[39220]: 3.2 scrub starts 2024-09-10T18:29:35.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[39220]: 3.2 scrub ok 2024-09-10T18:29:35.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[39220]: 6.3 scrub starts 2024-09-10T18:29:35.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[39220]: 6.3 scrub ok 2024-09-10T18:29:35.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[39220]: 4.1 scrub starts 2024-09-10T18:29:35.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[39220]: 4.1 scrub ok 2024-09-10T18:29:35.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[39220]: 3.10 scrub starts 2024-09-10T18:29:35.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[39220]: 3.10 scrub ok 2024-09-10T18:29:35.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[29291]: 5.5 scrub starts 2024-09-10T18:29:35.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[29291]: 5.5 scrub ok 2024-09-10T18:29:35.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[29291]: 3.2 scrub starts 2024-09-10T18:29:35.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[29291]: 3.2 scrub ok 2024-09-10T18:29:35.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[29291]: 6.3 scrub starts 2024-09-10T18:29:35.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[29291]: 6.3 scrub ok 2024-09-10T18:29:35.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[29291]: 4.1 scrub starts 2024-09-10T18:29:35.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[29291]: 4.1 scrub ok 2024-09-10T18:29:35.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[29291]: 3.10 scrub starts 2024-09-10T18:29:35.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:34 smithi028 ceph-mon[29291]: 3.10 scrub ok 2024-09-10T18:29:35.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:35.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 88 16 132025 132009 93.7325 76.8125 0.00788024 0.010664 2024-09-10T18:29:35.129 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:35.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:35.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:35.265 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:35.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:35.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:35.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:35.401 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:35.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:35.501 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:35.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:35.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:35.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:35.544 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:29:35.544 INFO:teuthology.orchestra.run.smithi204.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-10T18:29:35.544 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:29:35.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:35.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:35.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:35.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:35.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:35.801 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:35.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:35.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:35.903 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:35.932 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:35.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:35.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:36.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:36.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[29291]: 4.2 scrub starts 2024-09-10T18:29:36.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[29291]: 4.2 scrub ok 2024-09-10T18:29:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[29291]: pgmap v363: 158 pgs: 158 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 952 op/s; 13 B/s, 1 objects/s recovering 2024-09-10T18:29:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[29291]: 4.15 scrub starts 2024-09-10T18:29:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[29291]: 4.15 scrub ok 2024-09-10T18:29:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[29291]: 5.a scrub starts 2024-09-10T18:29:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[29291]: 5.a scrub ok 2024-09-10T18:29:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[29291]: 1.6 scrub starts 2024-09-10T18:29:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[29291]: 1.6 scrub ok 2024-09-10T18:29:36.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:36.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[39220]: 4.2 scrub starts 2024-09-10T18:29:36.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[39220]: 4.2 scrub ok 2024-09-10T18:29:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[39220]: pgmap v363: 158 pgs: 158 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 952 op/s; 13 B/s, 1 objects/s recovering 2024-09-10T18:29:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[39220]: 4.15 scrub starts 2024-09-10T18:29:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[39220]: 4.15 scrub ok 2024-09-10T18:29:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[39220]: 5.a scrub starts 2024-09-10T18:29:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[39220]: 5.a scrub ok 2024-09-10T18:29:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[39220]: 1.6 scrub starts 2024-09-10T18:29:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:35 smithi028 ceph-mon[39220]: 1.6 scrub ok 2024-09-10T18:29:36.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:36.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:36.069 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:36.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:35 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:35 smithi204 ceph-mon[32357]: 4.2 scrub starts 2024-09-10T18:29:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:35 smithi204 ceph-mon[32357]: 4.2 scrub ok 2024-09-10T18:29:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:35 smithi204 ceph-mon[32357]: pgmap v363: 158 pgs: 158 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 952 op/s; 13 B/s, 1 objects/s recovering 2024-09-10T18:29:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:35 smithi204 ceph-mon[32357]: 4.15 scrub starts 2024-09-10T18:29:36.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:35 smithi204 ceph-mon[32357]: 4.15 scrub ok 2024-09-10T18:29:36.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:35 smithi204 ceph-mon[32357]: 5.a scrub starts 2024-09-10T18:29:36.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:35 smithi204 ceph-mon[32357]: 5.a scrub ok 2024-09-10T18:29:36.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:35 smithi204 ceph-mon[32357]: 1.6 scrub starts 2024-09-10T18:29:36.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:35 smithi204 ceph-mon[32357]: 1.6 scrub ok 2024-09-10T18:29:36.125 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 89 16 132962 132946 93.3374 58.5625 0.0132411 0.0107079 2024-09-10T18:29:36.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:36.204 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:36.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:36.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:36.307 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:36.341 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:36.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:36.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:36.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:36.481 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:36.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:36.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:36.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:36.620 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:36.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:36.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:36.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:36.761 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:36.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:36.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:36.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:36.901 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:36.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:36.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:37.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:37.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[39220]: 3.3 scrub starts 2024-09-10T18:29:37.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[39220]: 3.3 scrub ok 2024-09-10T18:29:37.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[39220]: 5.c scrub starts 2024-09-10T18:29:37.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[39220]: 5.c scrub ok 2024-09-10T18:29:37.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:29:37.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:29:37.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[39220]: 8.5 scrub starts 2024-09-10T18:29:37.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[39220]: 4.18 scrub starts 2024-09-10T18:29:37.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[39220]: 4.18 scrub ok 2024-09-10T18:29:37.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[29291]: 3.3 scrub starts 2024-09-10T18:29:37.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[29291]: 3.3 scrub ok 2024-09-10T18:29:37.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[29291]: 5.c scrub starts 2024-09-10T18:29:37.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[29291]: 5.c scrub ok 2024-09-10T18:29:37.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:29:37.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:29:37.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[29291]: 8.5 scrub starts 2024-09-10T18:29:37.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[29291]: 4.18 scrub starts 2024-09-10T18:29:37.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:36 smithi028 ceph-mon[29291]: 4.18 scrub ok 2024-09-10T18:29:37.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:37.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:37.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:36 smithi204 ceph-mon[32357]: 3.3 scrub starts 2024-09-10T18:29:37.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:36 smithi204 ceph-mon[32357]: 3.3 scrub ok 2024-09-10T18:29:37.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:36 smithi204 ceph-mon[32357]: 5.c scrub starts 2024-09-10T18:29:37.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:36 smithi204 ceph-mon[32357]: 5.c scrub ok 2024-09-10T18:29:37.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:36 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:29:37.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:36 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:29:37.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:36 smithi204 ceph-mon[32357]: 8.5 scrub starts 2024-09-10T18:29:37.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:36 smithi204 ceph-mon[32357]: 4.18 scrub starts 2024-09-10T18:29:37.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:36 smithi204 ceph-mon[32357]: 4.18 scrub ok 2024-09-10T18:29:37.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total time run: 90.0066 2024-09-10T18:29:37.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total writes made: 134218 2024-09-10T18:29:37.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Write size: 65536 2024-09-10T18:29:37.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object size: 65536 2024-09-10T18:29:37.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Bandwidth (MB/sec): 93.2001 2024-09-10T18:29:37.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Bandwidth: 34.4592 2024-09-10T18:29:37.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max bandwidth (MB/sec): 229.938 2024-09-10T18:29:37.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min bandwidth (MB/sec): 55.4375 2024-09-10T18:29:37.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average IOPS: 1491 2024-09-10T18:29:37.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev IOPS: 551.348 2024-09-10T18:29:37.127 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max IOPS: 3679 2024-09-10T18:29:37.127 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min IOPS: 887 2024-09-10T18:29:37.127 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average Latency(s): 0.0107257 2024-09-10T18:29:37.127 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Latency(s): 0.00786456 2024-09-10T18:29:37.127 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max latency(s): 0.153086 2024-09-10T18:29:37.127 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min latency(s): 0.00125336 2024-09-10T18:29:37.132 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Cleaning up (deleting benchmark objects) 2024-09-10T18:29:37.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:37.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:37.186 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:37.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:37.287 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:37.325 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:37.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:37.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:37.427 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:37.463 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:37.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:37.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:37.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:37.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:37.598 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:37.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:37.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:37.740 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:37.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:37.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:29:37.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:37.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:37.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:37.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:37.973 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:29:37.974 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:29:37.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:37.986 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:29:37.986 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:29:38.028 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:29:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:29:38.029 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:38.029 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:37 smithi028 ceph-mon[39220]: osdmap e213: 8 total, 8 up, 8 in 2024-09-10T18:29:38.029 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:37 smithi028 ceph-mon[39220]: pgmap v365: 158 pgs: 158 active+clean; 8.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 13 B/s, 1 objects/s recovering 2024-09-10T18:29:38.029 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:37 smithi028 ceph-mon[39220]: 4.8 scrub starts 2024-09-10T18:29:38.029 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:37 smithi028 ceph-mon[39220]: 4.8 scrub ok 2024-09-10T18:29:38.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:38.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:38.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:37 smithi028 ceph-mon[29291]: osdmap e213: 8 total, 8 up, 8 in 2024-09-10T18:29:38.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:37 smithi028 ceph-mon[29291]: pgmap v365: 158 pgs: 158 active+clean; 8.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 13 B/s, 1 objects/s recovering 2024-09-10T18:29:38.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:37 smithi028 ceph-mon[29291]: 4.8 scrub starts 2024-09-10T18:29:38.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:37 smithi028 ceph-mon[29291]: 4.8 scrub ok 2024-09-10T18:29:38.062 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:29:38.063 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:29:38.097 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:29:38.097 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:29:38.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:37 smithi204 ceph-mon[32357]: osdmap e213: 8 total, 8 up, 8 in 2024-09-10T18:29:38.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:37 smithi204 ceph-mon[32357]: pgmap v365: 158 pgs: 158 active+clean; 8.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 13 B/s, 1 objects/s recovering 2024-09-10T18:29:38.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:37 smithi204 ceph-mon[32357]: 4.8 scrub starts 2024-09-10T18:29:38.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:37 smithi204 ceph-mon[32357]: 4.8 scrub ok 2024-09-10T18:29:38.129 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:29:38.129 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:29:38.131 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:38.159 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:29:38.159 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:29:38.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:38.165 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:38.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:38.195 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:29:38.195 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:29:38.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:38.302 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:38.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:38.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:38.404 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:38.436 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:38.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:38.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:38.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:38.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:38.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:38.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:38.712 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:38.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:38.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:38.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:38.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:38.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:38.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:38.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:38.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:38.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:39.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:38 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4177784340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:29:39.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:38 smithi028 ceph-mon[39220]: 5.16 scrub starts 2024-09-10T18:29:39.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:38 smithi028 ceph-mon[39220]: 5.16 scrub ok 2024-09-10T18:29:39.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:38 smithi028 ceph-mon[39220]: 8.5 scrub ok 2024-09-10T18:29:39.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:38 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4177784340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:29:39.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:38 smithi028 ceph-mon[29291]: 5.16 scrub starts 2024-09-10T18:29:39.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:38 smithi028 ceph-mon[29291]: 5.16 scrub ok 2024-09-10T18:29:39.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:38 smithi028 ceph-mon[29291]: 8.5 scrub ok 2024-09-10T18:29:39.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:39.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:38 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4177784340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:29:39.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:38 smithi204 ceph-mon[32357]: 5.16 scrub starts 2024-09-10T18:29:39.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:38 smithi204 ceph-mon[32357]: 5.16 scrub ok 2024-09-10T18:29:39.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:38 smithi204 ceph-mon[32357]: 8.5 scrub ok 2024-09-10T18:29:39.128 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:39.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:39.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:39.230 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:39.267 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:39.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:39.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:39.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:39.401 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:39.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:39.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:39.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:39.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:39.537 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:39.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:39.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:39.678 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:39.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:39.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:39.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:39.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:39.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:39.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:39.948 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:39.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:39.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:40.051 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:40.083 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:40.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:40.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:39 smithi204 ceph-mon[32357]: 2.0 scrub starts 2024-09-10T18:29:40.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:39 smithi204 ceph-mon[32357]: 2.0 scrub ok 2024-09-10T18:29:40.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:39 smithi204 ceph-mon[32357]: 6.1d scrub starts 2024-09-10T18:29:40.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:39 smithi204 ceph-mon[32357]: 6.1d scrub ok 2024-09-10T18:29:40.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:29:40.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:39 smithi204 ceph-mon[32357]: pgmap v366: 158 pgs: 158 active+clean; 8.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 733 op/s 2024-09-10T18:29:40.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:39 smithi204 ceph-mon[32357]: 8.8 scrub starts 2024-09-10T18:29:40.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:39 smithi204 ceph-mon[32357]: 6.f scrub starts 2024-09-10T18:29:40.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:39 smithi204 ceph-mon[32357]: 6.f scrub ok 2024-09-10T18:29:40.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:40.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:40.233 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:40.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[39220]: 2.0 scrub starts 2024-09-10T18:29:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[39220]: 2.0 scrub ok 2024-09-10T18:29:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[39220]: 6.1d scrub starts 2024-09-10T18:29:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[39220]: 6.1d scrub ok 2024-09-10T18:29:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:29:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[39220]: pgmap v366: 158 pgs: 158 active+clean; 8.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 733 op/s 2024-09-10T18:29:40.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[39220]: 8.8 scrub starts 2024-09-10T18:29:40.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[39220]: 6.f scrub starts 2024-09-10T18:29:40.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[39220]: 6.f scrub ok 2024-09-10T18:29:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[29291]: 2.0 scrub starts 2024-09-10T18:29:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[29291]: 2.0 scrub ok 2024-09-10T18:29:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[29291]: 6.1d scrub starts 2024-09-10T18:29:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[29291]: 6.1d scrub ok 2024-09-10T18:29:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:29:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[29291]: pgmap v366: 158 pgs: 158 active+clean; 8.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 733 op/s 2024-09-10T18:29:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[29291]: 8.8 scrub starts 2024-09-10T18:29:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[29291]: 6.f scrub starts 2024-09-10T18:29:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:39 smithi028 ceph-mon[29291]: 6.f scrub ok 2024-09-10T18:29:40.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:40.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:40.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:40.470 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:40.507 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:40.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:40.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:40.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:40.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:40.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:40.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:40.789 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:40.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:40.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:40.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-10T18:29:40.891 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:40.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:40.917 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:40.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:41.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:40 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:29:41.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:40 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:29:41.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:40 smithi028 ceph-mon[39220]: osdmap e214: 8 total, 8 up, 8 in 2024-09-10T18:29:41.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:40 smithi028 ceph-mon[39220]: 6.14 scrub starts 2024-09-10T18:29:41.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:40 smithi028 ceph-mon[39220]: 6.14 scrub ok 2024-09-10T18:29:41.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:40 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:29:41.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:40 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:29:41.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:40 smithi028 ceph-mon[29291]: osdmap e214: 8 total, 8 up, 8 in 2024-09-10T18:29:41.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:40 smithi028 ceph-mon[29291]: 6.14 scrub starts 2024-09-10T18:29:41.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:40 smithi028 ceph-mon[29291]: 6.14 scrub ok 2024-09-10T18:29:41.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:41.047 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:41.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:41.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:41.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:40 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:29:41.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:40 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:29:41.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:40 smithi204 ceph-mon[32357]: osdmap e214: 8 total, 8 up, 8 in 2024-09-10T18:29:41.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:40 smithi204 ceph-mon[32357]: 6.14 scrub starts 2024-09-10T18:29:41.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:40 smithi204 ceph-mon[32357]: 6.14 scrub ok 2024-09-10T18:29:41.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:41.183 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:41.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:41.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:41.223 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:29:41.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-10T18:29:41.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:41.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:41.332 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:41.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:41.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:41.486 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:41.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:41.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:41.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:29:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:29:41] "GET /metrics HTTP/1.1" 200 224303 "" "Prometheus/2.43.0" 2024-09-10T18:29:41.587 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:41.660 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:29:41.671 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:41.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:41.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:41.774 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:41.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:41.808 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:41.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:41.909 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:41.924 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:41 smithi028 ceph-mon[39220]: 6.c scrub starts 2024-09-10T18:29:41.924 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:41 smithi028 ceph-mon[39220]: 6.c scrub ok 2024-09-10T18:29:41.924 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:41 smithi028 ceph-mon[39220]: 8.3 scrub starts 2024-09-10T18:29:41.924 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:41 smithi028 ceph-mon[39220]: pgmap v368: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 63 MiB/s wr, 1.49k op/s 2024-09-10T18:29:41.924 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:41 smithi028 ceph-mon[39220]: 1.8 scrub starts 2024-09-10T18:29:41.924 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:41 smithi028 ceph-mon[39220]: 1.8 scrub ok 2024-09-10T18:29:41.925 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:41 smithi028 ceph-mon[29291]: 6.c scrub starts 2024-09-10T18:29:41.925 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:41 smithi028 ceph-mon[29291]: 6.c scrub ok 2024-09-10T18:29:41.925 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:41 smithi028 ceph-mon[29291]: 8.3 scrub starts 2024-09-10T18:29:41.925 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:41 smithi028 ceph-mon[29291]: pgmap v368: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 63 MiB/s wr, 1.49k op/s 2024-09-10T18:29:41.925 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:41 smithi028 ceph-mon[29291]: 1.8 scrub starts 2024-09-10T18:29:41.925 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:41 smithi028 ceph-mon[29291]: 1.8 scrub ok 2024-09-10T18:29:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:29:41.943 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:41.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:41.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:42.045 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:42.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:42.075 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:42.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:41 smithi204 ceph-mon[32357]: 6.c scrub starts 2024-09-10T18:29:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:41 smithi204 ceph-mon[32357]: 6.c scrub ok 2024-09-10T18:29:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:41 smithi204 ceph-mon[32357]: 8.3 scrub starts 2024-09-10T18:29:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:41 smithi204 ceph-mon[32357]: pgmap v368: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 63 MiB/s wr, 1.49k op/s 2024-09-10T18:29:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:41 smithi204 ceph-mon[32357]: 1.8 scrub starts 2024-09-10T18:29:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:41 smithi204 ceph-mon[32357]: 1.8 scrub ok 2024-09-10T18:29:42.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:42.229 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:42.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:42.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:42.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:42.375 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:42.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:42.477 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:42.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:42.510 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:42.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:42.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:29:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:29:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:29:42.611 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:42.649 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:42.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:42.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:42.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:42.789 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:42.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:42.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:42.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:42.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:42.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:42.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:43.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:42 smithi028 ceph-mon[29291]: 6.10 scrub starts 2024-09-10T18:29:43.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:42 smithi028 ceph-mon[29291]: 6.10 scrub ok 2024-09-10T18:29:43.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:42 smithi028 ceph-mon[29291]: 6.8 scrub starts 2024-09-10T18:29:43.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:42 smithi028 ceph-mon[29291]: 6.8 scrub ok 2024-09-10T18:29:43.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:42 smithi028 ceph-mon[29291]: 8.8 scrub ok 2024-09-10T18:29:43.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:42 smithi028 ceph-mon[39220]: 6.10 scrub starts 2024-09-10T18:29:43.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:42 smithi028 ceph-mon[39220]: 6.10 scrub ok 2024-09-10T18:29:43.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:42 smithi028 ceph-mon[39220]: 6.8 scrub starts 2024-09-10T18:29:43.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:42 smithi028 ceph-mon[39220]: 6.8 scrub ok 2024-09-10T18:29:43.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:42 smithi028 ceph-mon[39220]: 8.8 scrub ok 2024-09-10T18:29:43.031 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:43.056 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:43.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:43.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:43.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:42 smithi204 ceph-mon[32357]: 6.10 scrub starts 2024-09-10T18:29:43.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:42 smithi204 ceph-mon[32357]: 6.10 scrub ok 2024-09-10T18:29:43.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:42 smithi204 ceph-mon[32357]: 6.8 scrub starts 2024-09-10T18:29:43.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:42 smithi204 ceph-mon[32357]: 6.8 scrub ok 2024-09-10T18:29:43.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:42 smithi204 ceph-mon[32357]: 8.8 scrub ok 2024-09-10T18:29:43.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:43.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:43.193 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:43.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:43.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:43.338 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:43.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:43.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:43.367 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:29:43.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:43.477 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:43.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:43.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:43.579 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:43.608 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:43.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:43.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:43.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:29:43.710 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:43.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:43.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:43.849 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:43.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:43.878 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:43.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:43.980 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:43.998 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:43 smithi028 ceph-mon[39220]: 3.1e scrub starts 2024-09-10T18:29:43.999 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:43 smithi028 ceph-mon[39220]: 3.1e scrub ok 2024-09-10T18:29:43.999 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:43 smithi028 ceph-mon[39220]: 5.12 scrub starts 2024-09-10T18:29:43.999 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:43 smithi028 ceph-mon[39220]: 5.12 scrub ok 2024-09-10T18:29:43.999 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:43 smithi028 ceph-mon[39220]: pgmap v369: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 51 MiB/s wr, 1.30k op/s 2024-09-10T18:29:43.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:43 smithi028 ceph-mon[29291]: 3.1e scrub starts 2024-09-10T18:29:43.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:43 smithi028 ceph-mon[29291]: 3.1e scrub ok 2024-09-10T18:29:43.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:43 smithi028 ceph-mon[29291]: 5.12 scrub starts 2024-09-10T18:29:43.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:43 smithi028 ceph-mon[29291]: 5.12 scrub ok 2024-09-10T18:29:43.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:43 smithi028 ceph-mon[29291]: pgmap v369: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 51 MiB/s wr, 1.30k op/s 2024-09-10T18:29:44.011 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:44.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:44.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:43 smithi204 ceph-mon[32357]: 3.1e scrub starts 2024-09-10T18:29:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:43 smithi204 ceph-mon[32357]: 3.1e scrub ok 2024-09-10T18:29:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:43 smithi204 ceph-mon[32357]: 5.12 scrub starts 2024-09-10T18:29:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:43 smithi204 ceph-mon[32357]: 5.12 scrub ok 2024-09-10T18:29:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:43 smithi204 ceph-mon[32357]: pgmap v369: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 51 MiB/s wr, 1.30k op/s 2024-09-10T18:29:44.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:44.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:44.159 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:44.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:44.260 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:44.286 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:44.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:44.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:44.425 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:44.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:44.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:44.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:44.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:44.562 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:44.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:44.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:29:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:29:44.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:44.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:44.707 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:44.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:44.807 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:44.837 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:44.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:44.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:44.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:44.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:44.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:45.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:45.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:44 smithi204 ceph-mon[32357]: 4.13 scrub starts 2024-09-10T18:29:45.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:44 smithi204 ceph-mon[32357]: 6.1f scrub starts 2024-09-10T18:29:45.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:44 smithi204 ceph-mon[32357]: 4.13 scrub ok 2024-09-10T18:29:45.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:44 smithi204 ceph-mon[32357]: 6.1f scrub ok 2024-09-10T18:29:45.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:44 smithi204 ceph-mon[32357]: 1.3 scrub starts 2024-09-10T18:29:45.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:44 smithi204 ceph-mon[32357]: 1.3 scrub ok 2024-09-10T18:29:45.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:44 smithi204 ceph-mon[32357]: 8.3 scrub ok 2024-09-10T18:29:45.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:44 smithi204 ceph-mon[32357]: 5.6 scrub starts 2024-09-10T18:29:45.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:44 smithi204 ceph-mon[32357]: 5.6 scrub ok 2024-09-10T18:29:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:45.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:45.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:45.226 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[29291]: 4.13 scrub starts 2024-09-10T18:29:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[29291]: 6.1f scrub starts 2024-09-10T18:29:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[29291]: 4.13 scrub ok 2024-09-10T18:29:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[29291]: 6.1f scrub ok 2024-09-10T18:29:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[29291]: 1.3 scrub starts 2024-09-10T18:29:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[29291]: 1.3 scrub ok 2024-09-10T18:29:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[29291]: 8.3 scrub ok 2024-09-10T18:29:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[29291]: 5.6 scrub starts 2024-09-10T18:29:45.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[29291]: 5.6 scrub ok 2024-09-10T18:29:45.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[39220]: 4.13 scrub starts 2024-09-10T18:29:45.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[39220]: 6.1f scrub starts 2024-09-10T18:29:45.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[39220]: 4.13 scrub ok 2024-09-10T18:29:45.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[39220]: 6.1f scrub ok 2024-09-10T18:29:45.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[39220]: 1.3 scrub starts 2024-09-10T18:29:45.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[39220]: 1.3 scrub ok 2024-09-10T18:29:45.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[39220]: 8.3 scrub ok 2024-09-10T18:29:45.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[39220]: 5.6 scrub starts 2024-09-10T18:29:45.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:44 smithi028 ceph-mon[39220]: 5.6 scrub ok 2024-09-10T18:29:45.257 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:45.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:45.360 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:45.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:45.395 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:45.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:45.496 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:45.522 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:45.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:45.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:45.624 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:45.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:45.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:45.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:45.759 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:45.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:45.788 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:45.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:45.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:45.919 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:45.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:45.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:45 smithi028 ceph-mon[29291]: 6.9 scrub starts 2024-09-10T18:29:45.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:45 smithi028 ceph-mon[29291]: 6.9 scrub ok 2024-09-10T18:29:45.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:45 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:45.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:45 smithi028 ceph-mon[29291]: 8.a scrub starts 2024-09-10T18:29:45.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:45 smithi028 ceph-mon[29291]: pgmap v370: 158 pgs: 158 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 17 MiB/s wr, 932 op/s 2024-09-10T18:29:45.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:45 smithi028 ceph-mon[29291]: 8.9 scrub starts 2024-09-10T18:29:45.978 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:45 smithi028 ceph-mon[39220]: 6.9 scrub starts 2024-09-10T18:29:45.978 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:45 smithi028 ceph-mon[39220]: 6.9 scrub ok 2024-09-10T18:29:45.979 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:45 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:45.979 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:45 smithi028 ceph-mon[39220]: 8.a scrub starts 2024-09-10T18:29:45.979 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:45 smithi028 ceph-mon[39220]: pgmap v370: 158 pgs: 158 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 17 MiB/s wr, 932 op/s 2024-09-10T18:29:45.979 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:45 smithi028 ceph-mon[39220]: 8.9 scrub starts 2024-09-10T18:29:46.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:46.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:46.072 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:46.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:46.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:45 smithi204 ceph-mon[32357]: 6.9 scrub starts 2024-09-10T18:29:46.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:45 smithi204 ceph-mon[32357]: 6.9 scrub ok 2024-09-10T18:29:46.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:45 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:46.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:45 smithi204 ceph-mon[32357]: 8.a scrub starts 2024-09-10T18:29:46.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:45 smithi204 ceph-mon[32357]: pgmap v370: 158 pgs: 158 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 17 MiB/s wr, 932 op/s 2024-09-10T18:29:46.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:45 smithi204 ceph-mon[32357]: 8.9 scrub starts 2024-09-10T18:29:46.138 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:29:46.139 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-10T18:29:46.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-10T18:29:46.139 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:29:46.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.139 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:28633 7.f 7:ffdaa3af:::benchmark_data_smithi028_77856_object28632:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:29:46.139 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:55.726456+0000", 2024-09-10T18:29:46.139 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 350.406272516, 2024-09-10T18:29:46.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14808337999999999, 2024-09-10T18:29:46.139 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.139 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.139 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:29:46.139 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:29:46.140 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:46.140 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 28633 2024-09-10T18:29:46.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.140 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726456+0000", 2024-09-10T18:29:46.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726456+0000", 2024-09-10T18:29:46.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726455+0000", 2024-09-10T18:29:46.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:29:46.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726476+0000", 2024-09-10T18:29:46.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1577e-05 2024-09-10T18:29:46.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726485+0000", 2024-09-10T18:29:46.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8540000000000003e-06 2024-09-10T18:29:46.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.860732+0000", 2024-09-10T18:29:46.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.134246369 2024-09-10T18:29:46.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.862493+0000", 2024-09-10T18:29:46.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017610779999999999 2024-09-10T18:29:46.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.862607+0000", 2024-09-10T18:29:46.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011408000000000001 2024-09-10T18:29:46.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:46.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.862777+0000", 2024-09-10T18:29:46.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000170161 2024-09-10T18:29:46.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:29:46.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.864620+0000", 2024-09-10T18:29:46.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018431400000000001 2024-09-10T18:29:46.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.870301+0000", 2024-09-10T18:29:46.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0056804409999999996 2024-09-10T18:29:46.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.874449+0000", 2024-09-10T18:29:46.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041486459999999998 2024-09-10T18:29:46.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.874499+0000", 2024-09-10T18:29:46.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9891000000000001e-05 2024-09-10T18:29:46.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.874539+0000", 2024-09-10T18:29:46.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0159e-05 2024-09-10T18:29:46.146 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.146 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.146 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.146 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31736 7.6 7:6f8bbc73:::benchmark_data_smithi028_77856_object31735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:29:46.146 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.829509+0000", 2024-09-10T18:29:46.146 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 348.30321912099998, 2024-09-10T18:29:46.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.148056353, 2024-09-10T18:29:46.146 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.146 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.146 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:29:46.147 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:29:46.147 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:46.147 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31736 2024-09-10T18:29:46.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.147 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829509+0000", 2024-09-10T18:29:46.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829509+0000", 2024-09-10T18:29:46.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829508+0000", 2024-09-10T18:29:46.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:29:46.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829534+0000", 2024-09-10T18:29:46.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6038999999999999e-05 2024-09-10T18:29:46.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829544+0000", 2024-09-10T18:29:46.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6029999999999994e-06 2024-09-10T18:29:46.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829549+0000", 2024-09-10T18:29:46.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2970000000000003e-06 2024-09-10T18:29:46.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829601+0000", 2024-09-10T18:29:46.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1629999999999999e-05 2024-09-10T18:29:46.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829725+0000", 2024-09-10T18:29:46.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012429300000000001 2024-09-10T18:29:46.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:46.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829865+0000", 2024-09-10T18:29:46.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000139493 2024-09-10T18:29:46.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.834740+0000", 2024-09-10T18:29:46.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048757419999999997 2024-09-10T18:29:46.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:29:46.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836134+0000", 2024-09-10T18:29:46.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013935079999999999 2024-09-10T18:29:46.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838041+0000", 2024-09-10T18:29:46.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001907583 2024-09-10T18:29:46.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838084+0000", 2024-09-10T18:29:46.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3000000000000002e-05 2024-09-10T18:29:46.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977566+0000", 2024-09-10T18:29:46.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13948112900000001 2024-09-10T18:29:46.152 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.153 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.153 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.153 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31737 7.6 7:67a3034c:::benchmark_data_smithi028_77856_object31736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:29:46.153 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.829562+0000", 2024-09-10T18:29:46.153 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 348.30316594499999, 2024-09-10T18:29:46.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14825862000000001, 2024-09-10T18:29:46.153 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.153 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.153 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:29:46.153 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:29:46.153 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:46.154 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31737 2024-09-10T18:29:46.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.154 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829562+0000", 2024-09-10T18:29:46.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829562+0000", 2024-09-10T18:29:46.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829562+0000", 2024-09-10T18:29:46.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:29:46.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829583+0000", 2024-09-10T18:29:46.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1251999999999999e-05 2024-09-10T18:29:46.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829591+0000", 2024-09-10T18:29:46.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5890000000000003e-06 2024-09-10T18:29:46.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829597+0000", 2024-09-10T18:29:46.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-06 2024-09-10T18:29:46.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.831307+0000", 2024-09-10T18:29:46.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017102700000000001 2024-09-10T18:29:46.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.831417+0000", 2024-09-10T18:29:46.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110029 2024-09-10T18:29:46.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:46.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.831543+0000", 2024-09-10T18:29:46.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012614200000000001 2024-09-10T18:29:46.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.834769+0000", 2024-09-10T18:29:46.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032261270000000001 2024-09-10T18:29:46.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:29:46.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836140+0000", 2024-09-10T18:29:46.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013711909999999999 2024-09-10T18:29:46.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977739+0000", 2024-09-10T18:29:46.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.141598956 2024-09-10T18:29:46.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977783+0000", 2024-09-10T18:29:46.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3890999999999997e-05 2024-09-10T18:29:46.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977821+0000", 2024-09-10T18:29:46.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7901000000000002e-05 2024-09-10T18:29:46.159 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.159 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.159 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.159 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31740 7.f 7:f269a3c7:::benchmark_data_smithi028_77856_object31739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:29:46.159 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.829970+0000", 2024-09-10T18:29:46.159 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 348.30275878399999, 2024-09-10T18:29:46.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.147845861, 2024-09-10T18:29:46.160 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.160 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.160 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:29:46.160 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:29:46.160 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:46.160 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31740 2024-09-10T18:29:46.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.160 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829970+0000", 2024-09-10T18:29:46.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829970+0000", 2024-09-10T18:29:46.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829968+0000", 2024-09-10T18:29:46.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:29:46.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829993+0000", 2024-09-10T18:29:46.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4621e-05 2024-09-10T18:29:46.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.830002+0000", 2024-09-10T18:29:46.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0860000000000008e-06 2024-09-10T18:29:46.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.830007+0000", 2024-09-10T18:29:46.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3979999999999998e-06 2024-09-10T18:29:46.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.832120+0000", 2024-09-10T18:29:46.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021130070000000001 2024-09-10T18:29:46.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.832231+0000", 2024-09-10T18:29:46.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110889 2024-09-10T18:29:46.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:46.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.832455+0000", 2024-09-10T18:29:46.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000224284 2024-09-10T18:29:46.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.834618+0000", 2024-09-10T18:29:46.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002162509 2024-09-10T18:29:46.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:29:46.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.835983+0000", 2024-09-10T18:29:46.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013655060000000001 2024-09-10T18:29:46.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977732+0000", 2024-09-10T18:29:46.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14174862399999999 2024-09-10T18:29:46.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977776+0000", 2024-09-10T18:29:46.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3828000000000001e-05 2024-09-10T18:29:46.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977815+0000", 2024-09-10T18:29:46.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.947e-05 2024-09-10T18:29:46.165 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.165 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.165 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.166 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31753 7.7 7:e80563d5:::benchmark_data_smithi028_77856_object31752:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:29:46.166 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.836698+0000", 2024-09-10T18:29:46.166 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 348.29603005400003, 2024-09-10T18:29:46.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14739100099999999, 2024-09-10T18:29:46.166 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.166 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.166 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:29:46.166 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:29:46.166 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:46.166 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31753 2024-09-10T18:29:46.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.166 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836698+0000", 2024-09-10T18:29:46.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836698+0000", 2024-09-10T18:29:46.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836697+0000", 2024-09-10T18:29:46.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:29:46.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836726+0000", 2024-09-10T18:29:46.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9164999999999999e-05 2024-09-10T18:29:46.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836736+0000", 2024-09-10T18:29:46.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0232e-05 2024-09-10T18:29:46.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836742+0000", 2024-09-10T18:29:46.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3569999999999997e-06 2024-09-10T18:29:46.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836885+0000", 2024-09-10T18:29:46.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014372000000000001 2024-09-10T18:29:46.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836989+0000", 2024-09-10T18:29:46.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010346800000000001 2024-09-10T18:29:46.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-10T18:29:46.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837108+0000", 2024-09-10T18:29:46.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011928400000000001 2024-09-10T18:29:46.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.978097+0000", 2024-09-10T18:29:46.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.140988426 2024-09-10T18:29:46.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:29:46.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.982298+0000", 2024-09-10T18:29:46.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042009559999999996 2024-09-10T18:29:46.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.983963+0000", 2024-09-10T18:29:46.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016655089999999999 2024-09-10T18:29:46.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.984089+0000", 2024-09-10T18:29:46.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012626599999999999 2024-09-10T18:29:46.172 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.172 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.172 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.172 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31754 7.7 7:e51855d4:::benchmark_data_smithi028_77856_object31753:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:29:46.173 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.837301+0000", 2024-09-10T18:29:46.173 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 348.29542758700001, 2024-09-10T18:29:46.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.151838735, 2024-09-10T18:29:46.173 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.173 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.173 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:29:46.173 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:29:46.173 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:46.173 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31754 2024-09-10T18:29:46.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.173 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837301+0000", 2024-09-10T18:29:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837301+0000", 2024-09-10T18:29:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837300+0000", 2024-09-10T18:29:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:29:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837322+0000", 2024-09-10T18:29:46.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1506999999999999e-05 2024-09-10T18:29:46.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837329+0000", 2024-09-10T18:29:46.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.515e-06 2024-09-10T18:29:46.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837334+0000", 2024-09-10T18:29:46.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8749999999999999e-06 2024-09-10T18:29:46.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.978333+0000", 2024-09-10T18:29:46.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.140998866 2024-09-10T18:29:46.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.978442+0000", 2024-09-10T18:29:46.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108992 2024-09-10T18:29:46.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-10T18:29:46.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.978578+0000", 2024-09-10T18:29:46.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013659699999999999 2024-09-10T18:29:46.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:29:46.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.984104+0000", 2024-09-10T18:29:46.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055253580000000002 2024-09-10T18:29:46.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.989014+0000", 2024-09-10T18:29:46.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049104839999999997 2024-09-10T18:29:46.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.989062+0000", 2024-09-10T18:29:46.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7441000000000002e-05 2024-09-10T18:29:46.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.989139+0000", 2024-09-10T18:29:46.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7847000000000002e-05 2024-09-10T18:29:46.178 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.178 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.178 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.178 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31758 7.6 7:6c6ad8ac:::benchmark_data_smithi028_77856_object31757:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:29:46.178 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.837963+0000", 2024-09-10T18:29:46.179 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 348.29476581900002, 2024-09-10T18:29:46.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15004868099999999, 2024-09-10T18:29:46.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.179 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.179 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:29:46.179 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:29:46.179 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:46.179 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31758 2024-09-10T18:29:46.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.179 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837963+0000", 2024-09-10T18:29:46.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837963+0000", 2024-09-10T18:29:46.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837961+0000", 2024-09-10T18:29:46.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:29:46.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837982+0000", 2024-09-10T18:29:46.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1183e-05 2024-09-10T18:29:46.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837991+0000", 2024-09-10T18:29:46.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9570000000000008e-06 2024-09-10T18:29:46.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837996+0000", 2024-09-10T18:29:46.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6330000000000004e-06 2024-09-10T18:29:46.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977854+0000", 2024-09-10T18:29:46.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13985836200000001 2024-09-10T18:29:46.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.978294+0000", 2024-09-10T18:29:46.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043966200000000001 2024-09-10T18:29:46.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:46.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.978419+0000", 2024-09-10T18:29:46.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012486499999999999 2024-09-10T18:29:46.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.983938+0000", 2024-09-10T18:29:46.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055189319999999998 2024-09-10T18:29:46.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:29:46.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.983945+0000", 2024-09-10T18:29:46.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7680000000000003e-06 2024-09-10T18:29:46.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.987897+0000", 2024-09-10T18:29:46.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039524770000000002 2024-09-10T18:29:46.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.987973+0000", 2024-09-10T18:29:46.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6062000000000005e-05 2024-09-10T18:29:46.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.988011+0000", 2024-09-10T18:29:46.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8003999999999997e-05 2024-09-10T18:29:46.184 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.184 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.185 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.185 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:139702 7.f 7:f2f55ab9:::benchmark_data_smithi028_77856_object18602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:29:46.185 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:43.935577+0000", 2024-09-10T18:29:46.185 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 242.19715175600001, 2024-09-10T18:29:46.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15684685600000001, 2024-09-10T18:29:46.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.185 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.185 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:29:46.185 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:29:46.185 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:46.185 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 139702 2024-09-10T18:29:46.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.186 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935577+0000", 2024-09-10T18:29:46.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935577+0000", 2024-09-10T18:29:46.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935576+0000", 2024-09-10T18:29:46.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:29:46.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935580+0000", 2024-09-10T18:29:46.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6879999999999998e-06 2024-09-10T18:29:46.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935588+0000", 2024-09-10T18:29:46.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9999999999999996e-06 2024-09-10T18:29:46.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935594+0000", 2024-09-10T18:29:46.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2820000000000004e-06 2024-09-10T18:29:46.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.982138+0000", 2024-09-10T18:29:46.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.046544461000000002 2024-09-10T18:29:46.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.982248+0000", 2024-09-10T18:29:46.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110157 2024-09-10T18:29:46.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:46.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.982382+0000", 2024-09-10T18:29:46.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000133895 2024-09-10T18:29:46.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:29:46.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.989604+0000", 2024-09-10T18:29:46.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007222366 2024-09-10T18:29:46.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.992745+0000", 2024-09-10T18:29:46.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031404810000000001 2024-09-10T18:29:46.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:44.092292+0000", 2024-09-10T18:29:46.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.099547449999999996 2024-09-10T18:29:46.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:44.092339+0000", 2024-09-10T18:29:46.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6990000000000002e-05 2024-09-10T18:29:46.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:44.092423+0000", 2024-09-10T18:29:46.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4138e-05 2024-09-10T18:29:46.191 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.191 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.191 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.191 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:140670 7.f 7:ff7f13ba:::benchmark_data_smithi028_77856_object19570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:29:46.191 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:45.605622+0000", 2024-09-10T18:29:46.192 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 240.52710607899999, 2024-09-10T18:29:46.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.17139099099999999, 2024-09-10T18:29:46.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.192 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.192 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:29:46.192 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:29:46.192 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:46.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 140670 2024-09-10T18:29:46.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.192 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605622+0000", 2024-09-10T18:29:46.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605622+0000", 2024-09-10T18:29:46.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605620+0000", 2024-09-10T18:29:46.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:29:46.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605628+0000", 2024-09-10T18:29:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7510000000000008e-06 2024-09-10T18:29:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605641+0000", 2024-09-10T18:29:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2692e-05 2024-09-10T18:29:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605647+0000", 2024-09-10T18:29:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4169999999999997e-06 2024-09-10T18:29:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.612161+0000", 2024-09-10T18:29:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0065135159999999996 2024-09-10T18:29:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.612264+0000", 2024-09-10T18:29:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103127 2024-09-10T18:29:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.612383+0000", 2024-09-10T18:29:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000119157 2024-09-10T18:29:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:29:46.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.618786+0000", 2024-09-10T18:29:46.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006402942 2024-09-10T18:29:46.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.620538+0000", 2024-09-10T18:29:46.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001752368 2024-09-10T18:29:46.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.776944+0000", 2024-09-10T18:29:46.197 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:46.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.156405708 2024-09-10T18:29:46.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.776979+0000", 2024-09-10T18:29:46.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4829999999999997e-05 2024-09-10T18:29:46.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.777013+0000", 2024-09-10T18:29:46.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4496999999999999e-05 2024-09-10T18:29:46.199 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.199 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.199 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.199 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:140692 7.f 7:f303c2e4:::benchmark_data_smithi028_77856_object19592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:29:46.199 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:45.628709+0000", 2024-09-10T18:29:46.199 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 240.504019193, 2024-09-10T18:29:46.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14840287099999999, 2024-09-10T18:29:46.199 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.199 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.199 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:29:46.199 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:29:46.200 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:46.200 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 140692 2024-09-10T18:29:46.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.200 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628709+0000", 2024-09-10T18:29:46.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628709+0000", 2024-09-10T18:29:46.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628707+0000", 2024-09-10T18:29:46.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:29:46.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628714+0000", 2024-09-10T18:29:46.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4229999999999999e-06 2024-09-10T18:29:46.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628730+0000", 2024-09-10T18:29:46.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5991999999999999e-05 2024-09-10T18:29:46.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628738+0000", 2024-09-10T18:29:46.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0469999999999996e-06 2024-09-10T18:29:46.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628829+0000", 2024-09-10T18:29:46.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1197000000000001e-05 2024-09-10T18:29:46.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628955+0000", 2024-09-10T18:29:46.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012585100000000001 2024-09-10T18:29:46.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:46.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.629089+0000", 2024-09-10T18:29:46.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013479200000000001 2024-09-10T18:29:46.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.634532+0000", 2024-09-10T18:29:46.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054427750000000004 2024-09-10T18:29:46.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:29:46.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.642486+0000", 2024-09-10T18:29:46.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007953669 2024-09-10T18:29:46.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.777048+0000", 2024-09-10T18:29:46.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13456268499999999 2024-09-10T18:29:46.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.777082+0000", 2024-09-10T18:29:46.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3025000000000003e-05 2024-09-10T18:29:46.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.777112+0000", 2024-09-10T18:29:46.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0516999999999999e-05 2024-09-10T18:29:46.205 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.205 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.205 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.205 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:141317 7.f 7:f85eab8a:::benchmark_data_smithi028_77856_object20217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:29:46.205 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:46.778960+0000", 2024-09-10T18:29:46.205 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 239.35376827499999, 2024-09-10T18:29:46.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15742622000000001, 2024-09-10T18:29:46.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.206 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:29:46.206 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:29:46.206 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:46.206 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 141317 2024-09-10T18:29:46.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.206 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778960+0000", 2024-09-10T18:29:46.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778960+0000", 2024-09-10T18:29:46.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778958+0000", 2024-09-10T18:29:46.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:29:46.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778965+0000", 2024-09-10T18:29:46.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7599999999999997e-06 2024-09-10T18:29:46.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778973+0000", 2024-09-10T18:29:46.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7770000000000002e-06 2024-09-10T18:29:46.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778977+0000", 2024-09-10T18:29:46.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5229999999999997e-06 2024-09-10T18:29:46.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.789488+0000", 2024-09-10T18:29:46.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010510472999999999 2024-09-10T18:29:46.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.789609+0000", 2024-09-10T18:29:46.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121301 2024-09-10T18:29:46.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:46.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.789743+0000", 2024-09-10T18:29:46.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013393199999999999 2024-09-10T18:29:46.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.793571+0000", 2024-09-10T18:29:46.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0038282289999999998 2024-09-10T18:29:46.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:29:46.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.810048+0000", 2024-09-10T18:29:46.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0164768 2024-09-10T18:29:46.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.936295+0000", 2024-09-10T18:29:46.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.126246568 2024-09-10T18:29:46.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.936346+0000", 2024-09-10T18:29:46.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1708e-05 2024-09-10T18:29:46.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.936386+0000", 2024-09-10T18:29:46.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0021000000000002e-05 2024-09-10T18:29:46.211 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.211 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.211 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.211 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181160 7.7 7:e2311557:::benchmark_data_smithi028_77856_object60060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:29:46.212 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.037797+0000", 2024-09-10T18:29:46.212 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 201.09493161200001, 2024-09-10T18:29:46.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.155207649, 2024-09-10T18:29:46.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.212 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:29:46.212 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:29:46.212 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:46.212 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181160 2024-09-10T18:29:46.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.212 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037797+0000", 2024-09-10T18:29:46.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037797+0000", 2024-09-10T18:29:46.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037795+0000", 2024-09-10T18:29:46.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:29:46.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037810+0000", 2024-09-10T18:29:46.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4810000000000001e-05 2024-09-10T18:29:46.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037822+0000", 2024-09-10T18:29:46.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2568000000000001e-05 2024-09-10T18:29:46.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037829+0000", 2024-09-10T18:29:46.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.562e-06 2024-09-10T18:29:46.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.038647+0000", 2024-09-10T18:29:46.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081780299999999998 2024-09-10T18:29:46.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.038755+0000", 2024-09-10T18:29:46.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108941 2024-09-10T18:29:46.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-10T18:29:46.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.038876+0000", 2024-09-10T18:29:46.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000120259 2024-09-10T18:29:46.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:29:46.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041587+0000", 2024-09-10T18:29:46.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027112709999999999 2024-09-10T18:29:46.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191165+0000", 2024-09-10T18:29:46.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14957848300000001 2024-09-10T18:29:46.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.192642+0000", 2024-09-10T18:29:46.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001476148 2024-09-10T18:29:46.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.193004+0000", 2024-09-10T18:29:46.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00036274700000000002 2024-09-10T18:29:46.217 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.217 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.217 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.217 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181165 7.6 7:69ba312f:::benchmark_data_smithi028_77856_object60065:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:29:46.218 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.041719+0000", 2024-09-10T18:29:46.218 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 201.091009335, 2024-09-10T18:29:46.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14957322100000001, 2024-09-10T18:29:46.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.218 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:29:46.218 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:29:46.218 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:46.218 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181165 2024-09-10T18:29:46.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.218 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041719+0000", 2024-09-10T18:29:46.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041719+0000", 2024-09-10T18:29:46.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041718+0000", 2024-09-10T18:29:46.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:29:46.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041723+0000", 2024-09-10T18:29:46.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.48e-06 2024-09-10T18:29:46.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041733+0000", 2024-09-10T18:29:46.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6520000000000001e-06 2024-09-10T18:29:46.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041738+0000", 2024-09-10T18:29:46.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.553e-06 2024-09-10T18:29:46.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044567+0000", 2024-09-10T18:29:46.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028282770000000001 2024-09-10T18:29:46.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044684+0000", 2024-09-10T18:29:46.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117491 2024-09-10T18:29:46.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:46.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044768+0000", 2024-09-10T18:29:46.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4078000000000004e-05 2024-09-10T18:29:46.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.049844+0000", 2024-09-10T18:29:46.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050756799999999999 2024-09-10T18:29:46.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:29:46.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.050265+0000", 2024-09-10T18:29:46.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00042087299999999999 2024-09-10T18:29:46.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191082+0000", 2024-09-10T18:29:46.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14081718500000001 2024-09-10T18:29:46.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191174+0000", 2024-09-10T18:29:46.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1875999999999999e-05 2024-09-10T18:29:46.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191292+0000", 2024-09-10T18:29:46.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000118331 2024-09-10T18:29:46.223 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.223 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.224 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.224 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181169 7.6 7:60f36a9a:::benchmark_data_smithi028_77856_object60069:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:29:46.224 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.044506+0000", 2024-09-10T18:29:46.224 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 201.08822233399999, 2024-09-10T18:29:46.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14689063499999999, 2024-09-10T18:29:46.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.224 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.224 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:29:46.224 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:29:46.224 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:46.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181169 2024-09-10T18:29:46.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.225 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044506+0000", 2024-09-10T18:29:46.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044506+0000", 2024-09-10T18:29:46.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044504+0000", 2024-09-10T18:29:46.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:29:46.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044510+0000", 2024-09-10T18:29:46.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2729999999999999e-06 2024-09-10T18:29:46.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044520+0000", 2024-09-10T18:29:46.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8470000000000008e-06 2024-09-10T18:29:46.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044525+0000", 2024-09-10T18:29:46.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1499999999999998e-06 2024-09-10T18:29:46.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.046791+0000", 2024-09-10T18:29:46.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022659580000000002 2024-09-10T18:29:46.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.046905+0000", 2024-09-10T18:29:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011319699999999999 2024-09-10T18:29:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.047013+0000", 2024-09-10T18:29:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108261 2024-09-10T18:29:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.050258+0000", 2024-09-10T18:29:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032447309999999998 2024-09-10T18:29:46.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:29:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.050273+0000", 2024-09-10T18:29:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4898e-05 2024-09-10T18:29:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191325+0000", 2024-09-10T18:29:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.141052908 2024-09-10T18:29:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191363+0000", 2024-09-10T18:29:46.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7694000000000002e-05 2024-09-10T18:29:46.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191397+0000", 2024-09-10T18:29:46.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3528999999999997e-05 2024-09-10T18:29:46.230 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.230 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.230 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.230 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:181174 7.5 e142/93 7:a54ae781:::benchmark_data_smithi028_77856_object60074:head v 142'11303, mlcod=142'11301)", 2024-09-10T18:29:46.230 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.048963+0000", 2024-09-10T18:29:46.230 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 201.083765027, 2024-09-10T18:29:46.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14980299699999999, 2024-09-10T18:29:46.231 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.231 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.231 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048963+0000", 2024-09-10T18:29:46.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048963+0000", 2024-09-10T18:29:46.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048962+0000", 2024-09-10T18:29:46.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:29:46.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048969+0000", 2024-09-10T18:29:46.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.08e-06 2024-09-10T18:29:46.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048979+0000", 2024-09-10T18:29:46.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.006e-05 2024-09-10T18:29:46.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048985+0000", 2024-09-10T18:29:46.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8780000000000003e-06 2024-09-10T18:29:46.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.049055+0000", 2024-09-10T18:29:46.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9560000000000005e-05 2024-09-10T18:29:46.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.049072+0000", 2024-09-10T18:29:46.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7476e-05 2024-09-10T18:29:46.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198732+0000", 2024-09-10T18:29:46.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14965962499999999 2024-09-10T18:29:46.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198766+0000", 2024-09-10T18:29:46.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4514000000000002e-05 2024-09-10T18:29:46.235 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.235 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.235 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.235 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:181175 7.5 e142/93 7:a07e4061:::benchmark_data_smithi028_77856_object60075:head v 142'11304, mlcod=142'11301)", 2024-09-10T18:29:46.235 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.051631+0000", 2024-09-10T18:29:46.235 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 201.081097105, 2024-09-10T18:29:46.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15340213799999999, 2024-09-10T18:29:46.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.236 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051631+0000", 2024-09-10T18:29:46.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051631+0000", 2024-09-10T18:29:46.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051630+0000", 2024-09-10T18:29:46.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:29:46.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051639+0000", 2024-09-10T18:29:46.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8189999999999998e-06 2024-09-10T18:29:46.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051649+0000", 2024-09-10T18:29:46.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0101999999999999e-05 2024-09-10T18:29:46.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051655+0000", 2024-09-10T18:29:46.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9490000000000001e-06 2024-09-10T18:29:46.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051705+0000", 2024-09-10T18:29:46.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0328000000000003e-05 2024-09-10T18:29:46.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051722+0000", 2024-09-10T18:29:46.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6617999999999999e-05 2024-09-10T18:29:46.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205001+0000", 2024-09-10T18:29:46.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15327945300000001 2024-09-10T18:29:46.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205033+0000", 2024-09-10T18:29:46.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2257e-05 2024-09-10T18:29:46.239 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.239 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.240 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.240 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181177 7.f 7:f9ebf4ca:::benchmark_data_smithi028_77856_object60077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:29:46.240 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.053108+0000", 2024-09-10T18:29:46.240 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 201.07962001199999, 2024-09-10T18:29:46.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15150366200000001, 2024-09-10T18:29:46.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.240 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:29:46.240 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:29:46.240 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:46.240 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181177 2024-09-10T18:29:46.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.242 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.053108+0000", 2024-09-10T18:29:46.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.053108+0000", 2024-09-10T18:29:46.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.053106+0000", 2024-09-10T18:29:46.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:29:46.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.053113+0000", 2024-09-10T18:29:46.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0520000000000004e-06 2024-09-10T18:29:46.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.190879+0000", 2024-09-10T18:29:46.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13776606999999999 2024-09-10T18:29:46.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.190891+0000", 2024-09-10T18:29:46.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1102e-05 2024-09-10T18:29:46.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.190985+0000", 2024-09-10T18:29:46.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4087999999999998e-05 2024-09-10T18:29:46.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191099+0000", 2024-09-10T18:29:46.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114473 2024-09-10T18:29:46.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:29:46.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191299+0000", 2024-09-10T18:29:46.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019984899999999999 2024-09-10T18:29:46.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.197973+0000", 2024-09-10T18:29:46.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066738869999999999 2024-09-10T18:29:46.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:29:46.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198142+0000", 2024-09-10T18:29:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000169006 2024-09-10T18:29:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.204473+0000", 2024-09-10T18:29:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063307729999999996 2024-09-10T18:29:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.204527+0000", 2024-09-10T18:29:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4588e-05 2024-09-10T18:29:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.204612+0000", 2024-09-10T18:29:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.475e-05 2024-09-10T18:29:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:181183 7.d e142/95 7:b1361f07:::benchmark_data_smithi028_77856_object60083:head v 142'11153, mlcod=142'11151)", 2024-09-10T18:29:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.055525+0000", 2024-09-10T18:29:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 201.07720365, 2024-09-10T18:29:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14994769499999999, 2024-09-10T18:29:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055525+0000", 2024-09-10T18:29:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055525+0000", 2024-09-10T18:29:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055523+0000", 2024-09-10T18:29:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:29:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055532+0000", 2024-09-10T18:29:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0159999999999997e-06 2024-09-10T18:29:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191429+0000", 2024-09-10T18:29:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13589642099999999 2024-09-10T18:29:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191435+0000", 2024-09-10T18:29:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9569999999999999e-06 2024-09-10T18:29:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198700+0000", 2024-09-10T18:29:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0072655549999999999 2024-09-10T18:29:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198722+0000", 2024-09-10T18:29:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1307000000000001e-05 2024-09-10T18:29:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205091+0000", 2024-09-10T18:29:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063695779999999999 2024-09-10T18:29:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205472+0000", 2024-09-10T18:29:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00038129400000000001 2024-09-10T18:29:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:226332 7.9 7:9bbc16bb:::benchmark_data_smithi028_77856_object105232:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-10T18:29:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:14.384906+0000", 2024-09-10T18:29:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 151.74782215600001, 2024-09-10T18:29:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.147375691, 2024-09-10T18:29:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:29:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:29:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 226332 2024-09-10T18:29:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384906+0000", 2024-09-10T18:29:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384906+0000", 2024-09-10T18:29:46.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384905+0000", 2024-09-10T18:29:46.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:29:46.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384910+0000", 2024-09-10T18:29:46.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2900000000000002e-06 2024-09-10T18:29:46.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384919+0000", 2024-09-10T18:29:46.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7960000000000001e-06 2024-09-10T18:29:46.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384925+0000", 2024-09-10T18:29:46.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5720000000000002e-06 2024-09-10T18:29:46.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.389664+0000", 2024-09-10T18:29:46.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047393929999999997 2024-09-10T18:29:46.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.389762+0000", 2024-09-10T18:29:46.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7841000000000001e-05 2024-09-10T18:29:46.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:29:46.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.389860+0000", 2024-09-10T18:29:46.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7998999999999995e-05 2024-09-10T18:29:46.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:29:46.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.521530+0000", 2024-09-10T18:29:46.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13167006000000001 2024-09-10T18:29:46.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.521566+0000", 2024-09-10T18:29:46.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6106000000000003e-05 2024-09-10T18:29:46.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.532156+0000", 2024-09-10T18:29:46.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010590132 2024-09-10T18:29:46.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.532245+0000", 2024-09-10T18:29:46.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8571000000000003e-05 2024-09-10T18:29:46.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.532282+0000", 2024-09-10T18:29:46.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7049999999999999e-05 2024-09-10T18:29:46.267 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.267 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.267 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.267 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:226337 7.9 7:935b6cbf:::benchmark_data_smithi028_77856_object105237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-10T18:29:46.268 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:14.388673+0000", 2024-09-10T18:29:46.268 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 151.74405546099999, 2024-09-10T18:29:46.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14738206400000001, 2024-09-10T18:29:46.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:29:46.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:29:46.268 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:29:46.268 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:29:46.269 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:29:46.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 226337 2024-09-10T18:29:46.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.269 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:29:46.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:29:46.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.388673+0000", 2024-09-10T18:29:46.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:29:46.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.388673+0000", 2024-09-10T18:29:46.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:29:46.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:29:46.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.388672+0000", 2024-09-10T18:29:46.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:29:46.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:29:46.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.388675+0000", 2024-09-10T18:29:46.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7460000000000001e-06 2024-09-10T18:29:46.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:29:46.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.388680+0000", 2024-09-10T18:29:46.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7740000000000004e-06 2024-09-10T18:29:46.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:29:46.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.388683+0000", 2024-09-10T18:29:46.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2339999999999999e-06 2024-09-10T18:29:46.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:29:46.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.521593+0000", 2024-09-10T18:29:46.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13290959299999999 2024-09-10T18:29:46.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:29:46.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.521671+0000", 2024-09-10T18:29:46.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8456999999999998e-05 2024-09-10T18:29:46.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:29:46.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.521731+0000", 2024-09-10T18:29:46.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0307999999999999e-05 2024-09-10T18:29:46.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:29:46.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.532296+0000", 2024-09-10T18:29:46.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010564413999999999 2024-09-10T18:29:46.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.532328+0000", 2024-09-10T18:29:46.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2223e-05 2024-09-10T18:29:46.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:29:46.276 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.532359+0000", 2024-09-10T18:29:46.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1075999999999999e-05 2024-09-10T18:29:46.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:29:46.276 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.532528+0000", 2024-09-10T18:29:46.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000168567 2024-09-10T18:29:46.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:29:46.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:29:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:29:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.536055+0000", 2024-09-10T18:29:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035271740000000001 2024-09-10T18:29:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:29:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:29:46.277 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:29:46.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:46.279 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:46.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:46.380 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:46.411 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:46.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:46.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:46.513 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:46.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:46.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:46.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:46.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:46.686 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:46.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:46.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:46.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:46.930 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:46.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:46 smithi028 ceph-mon[29291]: 4.c scrub starts 2024-09-10T18:29:46.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:46 smithi028 ceph-mon[29291]: 4.c scrub ok 2024-09-10T18:29:46.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:46 smithi028 ceph-mon[29291]: 5.10 scrub starts 2024-09-10T18:29:46.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:46 smithi028 ceph-mon[29291]: 5.10 scrub ok 2024-09-10T18:29:46.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:46 smithi028 ceph-mon[29291]: 8.7 scrub starts 2024-09-10T18:29:46.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:46 smithi028 ceph-mon[29291]: 3.1d scrub starts 2024-09-10T18:29:46.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:46 smithi028 ceph-mon[29291]: 3.1d scrub ok 2024-09-10T18:29:46.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:46 smithi028 ceph-mon[39220]: 4.c scrub starts 2024-09-10T18:29:46.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:46 smithi028 ceph-mon[39220]: 4.c scrub ok 2024-09-10T18:29:46.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:46 smithi028 ceph-mon[39220]: 5.10 scrub starts 2024-09-10T18:29:46.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:46 smithi028 ceph-mon[39220]: 5.10 scrub ok 2024-09-10T18:29:46.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:46 smithi028 ceph-mon[39220]: 8.7 scrub starts 2024-09-10T18:29:46.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:46 smithi028 ceph-mon[39220]: 3.1d scrub starts 2024-09-10T18:29:46.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:46 smithi028 ceph-mon[39220]: 3.1d scrub ok 2024-09-10T18:29:46.954 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:46.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:47.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:47.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:47.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:47.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:47.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:46 smithi204 ceph-mon[32357]: 4.c scrub starts 2024-09-10T18:29:47.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:46 smithi204 ceph-mon[32357]: 4.c scrub ok 2024-09-10T18:29:47.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:46 smithi204 ceph-mon[32357]: 5.10 scrub starts 2024-09-10T18:29:47.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:46 smithi204 ceph-mon[32357]: 5.10 scrub ok 2024-09-10T18:29:47.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:46 smithi204 ceph-mon[32357]: 8.7 scrub starts 2024-09-10T18:29:47.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:46 smithi204 ceph-mon[32357]: 3.1d scrub starts 2024-09-10T18:29:47.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:46 smithi204 ceph-mon[32357]: 3.1d scrub ok 2024-09-10T18:29:47.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:47.297 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:47.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:47.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:47.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:47.439 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:47.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:47.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:47.583 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:47.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:47.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:47.685 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:47.714 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:47.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:47.816 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:47.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:47.851 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:47.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:47.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:48.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:48.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:48.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:48.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:48.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:47 smithi204 ceph-mon[32357]: 8.c scrub starts 2024-09-10T18:29:48.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:47 smithi204 ceph-mon[32357]: 5.18 scrub starts 2024-09-10T18:29:48.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:47 smithi204 ceph-mon[32357]: 5.18 scrub ok 2024-09-10T18:29:48.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:47 smithi204 ceph-mon[32357]: pgmap v371: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 14 MiB/s wr, 1.22k op/s 2024-09-10T18:29:48.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:47 smithi204 ceph-mon[32357]: 5.b scrub starts 2024-09-10T18:29:48.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:47 smithi204 ceph-mon[32357]: 5.b scrub ok 2024-09-10T18:29:48.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:47 smithi204 ceph-mon[32357]: 5.11 scrub starts 2024-09-10T18:29:48.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:47 smithi204 ceph-mon[32357]: 5.11 scrub ok 2024-09-10T18:29:48.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:48.165 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:48.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:48.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:47 smithi028 ceph-mon[39220]: 8.c scrub starts 2024-09-10T18:29:48.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:47 smithi028 ceph-mon[39220]: 5.18 scrub starts 2024-09-10T18:29:48.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:47 smithi028 ceph-mon[39220]: 5.18 scrub ok 2024-09-10T18:29:48.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:47 smithi028 ceph-mon[39220]: pgmap v371: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 14 MiB/s wr, 1.22k op/s 2024-09-10T18:29:48.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:47 smithi028 ceph-mon[39220]: 5.b scrub starts 2024-09-10T18:29:48.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:47 smithi028 ceph-mon[39220]: 5.b scrub ok 2024-09-10T18:29:48.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:47 smithi028 ceph-mon[39220]: 5.11 scrub starts 2024-09-10T18:29:48.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:47 smithi028 ceph-mon[39220]: 5.11 scrub ok 2024-09-10T18:29:48.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:47 smithi028 ceph-mon[29291]: 8.c scrub starts 2024-09-10T18:29:48.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:47 smithi028 ceph-mon[29291]: 5.18 scrub starts 2024-09-10T18:29:48.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:47 smithi028 ceph-mon[29291]: 5.18 scrub ok 2024-09-10T18:29:48.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:47 smithi028 ceph-mon[29291]: pgmap v371: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 14 MiB/s wr, 1.22k op/s 2024-09-10T18:29:48.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:47 smithi028 ceph-mon[29291]: 5.b scrub starts 2024-09-10T18:29:48.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:47 smithi028 ceph-mon[29291]: 5.b scrub ok 2024-09-10T18:29:48.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:47 smithi028 ceph-mon[29291]: 5.11 scrub starts 2024-09-10T18:29:48.180 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:47 smithi028 ceph-mon[29291]: 5.11 scrub ok 2024-09-10T18:29:48.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:48.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:48.318 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:48.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:48.419 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:48.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:48.445 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:48.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:48.546 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:48.573 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:48.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:48.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:48.675 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:48.699 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:48.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:48.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:48.801 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:48.826 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:48.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:48.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:48.919 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:29:48.928 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:48.950 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:48.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:48.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:49.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:49.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:48 smithi028 ceph-mon[29291]: 3.e scrub starts 2024-09-10T18:29:49.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:48 smithi028 ceph-mon[29291]: 3.e scrub ok 2024-09-10T18:29:49.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:48 smithi028 ceph-mon[29291]: 3.19 scrub starts 2024-09-10T18:29:49.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:48 smithi028 ceph-mon[29291]: 3.19 scrub ok 2024-09-10T18:29:49.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:48 smithi028 ceph-mon[29291]: 8.a scrub ok 2024-09-10T18:29:49.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:48 smithi028 ceph-mon[29291]: 8.9 scrub ok 2024-09-10T18:29:49.068 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:48 smithi028 ceph-mon[39220]: 3.e scrub starts 2024-09-10T18:29:49.068 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:48 smithi028 ceph-mon[39220]: 3.e scrub ok 2024-09-10T18:29:49.068 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:48 smithi028 ceph-mon[39220]: 3.19 scrub starts 2024-09-10T18:29:49.068 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:48 smithi028 ceph-mon[39220]: 3.19 scrub ok 2024-09-10T18:29:49.068 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:48 smithi028 ceph-mon[39220]: 8.a scrub ok 2024-09-10T18:29:49.068 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:48 smithi028 ceph-mon[39220]: 8.9 scrub ok 2024-09-10T18:29:49.102 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:49.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:49.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:49.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:48 smithi204 ceph-mon[32357]: 3.e scrub starts 2024-09-10T18:29:49.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:48 smithi204 ceph-mon[32357]: 3.e scrub ok 2024-09-10T18:29:49.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:48 smithi204 ceph-mon[32357]: 3.19 scrub starts 2024-09-10T18:29:49.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:48 smithi204 ceph-mon[32357]: 3.19 scrub ok 2024-09-10T18:29:49.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:48 smithi204 ceph-mon[32357]: 8.a scrub ok 2024-09-10T18:29:49.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:48 smithi204 ceph-mon[32357]: 8.9 scrub ok 2024-09-10T18:29:49.205 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:49.230 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:49.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:49.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:49.332 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:49.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:49.363 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:49.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:49.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:49.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T18:29:49.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:49.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:49.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:49.619 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:49.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:49.654 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:49.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:49.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:49.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:49.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:49.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:49.851 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:29:49.851 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.0 to 1 (10000) 2024-09-10T18:29:49.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:49.976 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:49.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:49.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:50.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:50.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:49 smithi204 ceph-mon[32357]: 6.1b scrub starts 2024-09-10T18:29:50.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:49 smithi204 ceph-mon[32357]: 6.1e scrub starts 2024-09-10T18:29:50.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:49 smithi204 ceph-mon[32357]: 6.1e scrub ok 2024-09-10T18:29:50.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:49 smithi204 ceph-mon[32357]: 6.1b scrub ok 2024-09-10T18:29:50.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:49 smithi204 ceph-mon[32357]: pgmap v372: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 14 MiB/s wr, 1.22k op/s 2024-09-10T18:29:50.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:49 smithi204 ceph-mon[32357]: 5.8 scrub starts 2024-09-10T18:29:50.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:49 smithi204 ceph-mon[32357]: 8.d scrub starts 2024-09-10T18:29:50.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:49 smithi204 ceph-mon[32357]: 5.8 scrub ok 2024-09-10T18:29:50.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:49 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2862651266' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:29:50.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:49 smithi204 ceph-mon[32357]: from='client.25147 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:29:50.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:49 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2717727460' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:29:50.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:49 smithi204 ceph-mon[32357]: from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:29:50.116 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:50.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:50.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:50.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[39220]: 6.1b scrub starts 2024-09-10T18:29:50.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[39220]: 6.1e scrub starts 2024-09-10T18:29:50.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[39220]: 6.1e scrub ok 2024-09-10T18:29:50.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[39220]: 6.1b scrub ok 2024-09-10T18:29:50.155 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[39220]: pgmap v372: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 14 MiB/s wr, 1.22k op/s 2024-09-10T18:29:50.155 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[39220]: 5.8 scrub starts 2024-09-10T18:29:50.155 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[39220]: 8.d scrub starts 2024-09-10T18:29:50.155 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[39220]: 5.8 scrub ok 2024-09-10T18:29:50.155 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2862651266' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:29:50.155 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[39220]: from='client.25147 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:29:50.155 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2717727460' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:29:50.155 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[39220]: from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:29:50.155 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[29291]: 6.1b scrub starts 2024-09-10T18:29:50.155 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[29291]: 6.1e scrub starts 2024-09-10T18:29:50.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[29291]: 6.1e scrub ok 2024-09-10T18:29:50.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[29291]: 6.1b scrub ok 2024-09-10T18:29:50.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[29291]: pgmap v372: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 14 MiB/s wr, 1.22k op/s 2024-09-10T18:29:50.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[29291]: 5.8 scrub starts 2024-09-10T18:29:50.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[29291]: 8.d scrub starts 2024-09-10T18:29:50.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[29291]: 5.8 scrub ok 2024-09-10T18:29:50.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2862651266' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:29:50.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[29291]: from='client.25147 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:29:50.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2717727460' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:29:50.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:49 smithi028 ceph-mon[29291]: from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:29:50.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:50.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:50.260 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:50.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:50.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:50.393 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:50.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:50.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:50.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:50.529 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:50.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:50.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:50.633 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:50.666 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:50.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:50.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:50.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:50.800 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:50.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:50.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[29291]: 4.f scrub starts 2024-09-10T18:29:50.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[29291]: 4.f scrub ok 2024-09-10T18:29:50.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:50.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:50.962 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:50.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:51.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:51.097 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:51.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:51.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:51.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:50 smithi204 ceph-mon[32357]: 4.f scrub starts 2024-09-10T18:29:51.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:50 smithi204 ceph-mon[32357]: 4.f scrub ok 2024-09-10T18:29:51.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:50 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:29:51.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:50 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:29:51.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:50 smithi204 ceph-mon[32357]: from='client.25147 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:29:51.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:50 smithi204 ceph-mon[32357]: from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T18:29:51.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:50 smithi204 ceph-mon[32357]: osdmap e215: 8 total, 8 up, 8 in 2024-09-10T18:29:51.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:50 smithi204 ceph-mon[32357]: 5.d scrub starts 2024-09-10T18:29:51.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:50 smithi204 ceph-mon[32357]: 5.d scrub ok 2024-09-10T18:29:51.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[39220]: 4.f scrub starts 2024-09-10T18:29:51.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[39220]: 4.f scrub ok 2024-09-10T18:29:51.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:29:51.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:29:51.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[39220]: from='client.25147 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:29:51.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[39220]: from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T18:29:51.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[39220]: osdmap e215: 8 total, 8 up, 8 in 2024-09-10T18:29:51.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[39220]: 5.d scrub starts 2024-09-10T18:29:51.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[39220]: 5.d scrub ok 2024-09-10T18:29:51.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:29:51.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:29:51.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[29291]: from='client.25147 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:29:51.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[29291]: from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T18:29:51.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[29291]: osdmap e215: 8 total, 8 up, 8 in 2024-09-10T18:29:51.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[29291]: 5.d scrub starts 2024-09-10T18:29:51.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:50 smithi028 ceph-mon[29291]: 5.d scrub ok 2024-09-10T18:29:51.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:51.240 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:51.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:51.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:51.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:51.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:51.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:51.454 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:29:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:29:51] "GET /metrics HTTP/1.1" 200 224303 "" "Prometheus/2.43.0" 2024-09-10T18:29:51.496 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:51.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-10T18:29:51.525 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:51.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:51.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:51.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:51.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:51.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:51.780 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:51.809 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:51.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:51.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:51.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:51.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:51.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:51.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:52.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:51 smithi028 ceph-mon[39220]: 8.c scrub ok 2024-09-10T18:29:52.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:51 smithi028 ceph-mon[39220]: 8.f scrub starts 2024-09-10T18:29:52.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:51 smithi028 ceph-mon[39220]: 8.7 scrub ok 2024-09-10T18:29:52.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:51 smithi028 ceph-mon[39220]: pgmap v374: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-10T18:29:52.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:51 smithi028 ceph-mon[39220]: 8.4 scrub starts 2024-09-10T18:29:52.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:51 smithi028 ceph-mon[39220]: 8.d scrub ok 2024-09-10T18:29:52.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:51 smithi028 ceph-mon[29291]: 8.c scrub ok 2024-09-10T18:29:52.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:51 smithi028 ceph-mon[29291]: 8.f scrub starts 2024-09-10T18:29:52.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:51 smithi028 ceph-mon[29291]: 8.7 scrub ok 2024-09-10T18:29:52.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:51 smithi028 ceph-mon[29291]: pgmap v374: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-10T18:29:52.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:51 smithi028 ceph-mon[29291]: 8.4 scrub starts 2024-09-10T18:29:52.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:51 smithi028 ceph-mon[29291]: 8.d scrub ok 2024-09-10T18:29:52.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:52.101 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:52.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:52.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:51 smithi204 ceph-mon[32357]: 8.c scrub ok 2024-09-10T18:29:52.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:51 smithi204 ceph-mon[32357]: 8.f scrub starts 2024-09-10T18:29:52.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:51 smithi204 ceph-mon[32357]: 8.7 scrub ok 2024-09-10T18:29:52.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:51 smithi204 ceph-mon[32357]: pgmap v374: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-10T18:29:52.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:51 smithi204 ceph-mon[32357]: 8.4 scrub starts 2024-09-10T18:29:52.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:51 smithi204 ceph-mon[32357]: 8.d scrub ok 2024-09-10T18:29:52.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:52.249 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:52.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:52.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:52.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:52.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:52.387 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:52.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:52.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:52.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:52.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:52.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:52.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:29:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:29:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:29:52.624 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:52.664 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:52.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:52.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:52.766 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:52.803 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:52.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:52.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:52.905 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:52.942 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:52.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:52.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:53.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:53.075 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:52 smithi204 ceph-mon[32357]: 6.19 scrub starts 2024-09-10T18:29:53.075 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:52 smithi204 ceph-mon[32357]: 6.19 scrub ok 2024-09-10T18:29:53.081 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:53.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:53.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:53.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:52 smithi028 ceph-mon[29291]: 6.19 scrub starts 2024-09-10T18:29:53.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:52 smithi028 ceph-mon[29291]: 6.19 scrub ok 2024-09-10T18:29:53.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:52 smithi028 ceph-mon[39220]: 6.19 scrub starts 2024-09-10T18:29:53.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:52 smithi028 ceph-mon[39220]: 6.19 scrub ok 2024-09-10T18:29:53.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:53.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:53.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:53.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:53.330 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:53.366 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:53.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:53.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:53.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:53.504 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:53.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:53.606 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:53.637 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:53.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:53.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:29:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:29:53.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:53.747 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:29:53.747 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:29:53.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:53.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:53.794 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:29:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:29:53.828 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:29:53.828 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:29:53.864 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:29:53.864 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:29:53.892 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:29:53.892 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:29:53.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:53.924 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:29:53.924 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:29:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:53.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:53.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:53.960 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:29:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:29:53.992 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:29:53.993 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:29:54.041 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:54.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:53 smithi028 ceph-mon[39220]: 1.9 scrub starts 2024-09-10T18:29:54.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:53 smithi028 ceph-mon[39220]: 1.9 scrub ok 2024-09-10T18:29:54.047 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:53 smithi028 ceph-mon[39220]: pgmap v375: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-10T18:29:54.047 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:53 smithi028 ceph-mon[39220]: 8.4 scrub ok 2024-09-10T18:29:54.047 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:53 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/515458394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:29:54.047 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:53 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:29:54.047 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:53 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:29:54.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:53 smithi028 ceph-mon[29291]: 1.9 scrub starts 2024-09-10T18:29:54.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:53 smithi028 ceph-mon[29291]: 1.9 scrub ok 2024-09-10T18:29:54.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:53 smithi028 ceph-mon[29291]: pgmap v375: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-10T18:29:54.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:53 smithi028 ceph-mon[29291]: 8.4 scrub ok 2024-09-10T18:29:54.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:53 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/515458394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:29:54.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:53 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:29:54.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:53 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:29:54.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:54.074 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:54.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:54.171 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:53 smithi204 ceph-mon[32357]: 1.9 scrub starts 2024-09-10T18:29:54.171 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:53 smithi204 ceph-mon[32357]: 1.9 scrub ok 2024-09-10T18:29:54.171 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:53 smithi204 ceph-mon[32357]: pgmap v375: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-09-10T18:29:54.171 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:53 smithi204 ceph-mon[32357]: 8.4 scrub ok 2024-09-10T18:29:54.171 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:53 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/515458394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:29:54.171 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:53 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:29:54.171 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:53 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:29:54.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:54.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:54.215 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:54.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:54.316 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:54.343 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:54.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:54.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:54.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:54.482 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:54.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:54.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:54.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:54.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:29:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:29:54.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:54.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:54.723 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:54.752 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:54.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:54.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:54.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:54.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:54.884 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:54.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:54.985 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:55.016 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:55.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:55.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:55.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:54 smithi028 ceph-mon[29291]: 8.f scrub ok 2024-09-10T18:29:55.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:54 smithi028 ceph-mon[29291]: 1.7 scrub starts 2024-09-10T18:29:55.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:54 smithi028 ceph-mon[29291]: 1.7 scrub ok 2024-09-10T18:29:55.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:54 smithi028 ceph-mon[39220]: 8.f scrub ok 2024-09-10T18:29:55.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:54 smithi028 ceph-mon[39220]: 1.7 scrub starts 2024-09-10T18:29:55.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:54 smithi028 ceph-mon[39220]: 1.7 scrub ok 2024-09-10T18:29:55.118 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:55.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:29:55.155 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:55.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:55.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:55.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:55.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:55.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:55.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:54 smithi204 ceph-mon[32357]: 8.f scrub ok 2024-09-10T18:29:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:54 smithi204 ceph-mon[32357]: 1.7 scrub starts 2024-09-10T18:29:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:54 smithi204 ceph-mon[32357]: 1.7 scrub ok 2024-09-10T18:29:55.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:55.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:55.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:55.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:55.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:55.578 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:55.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:55.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:55.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:55.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:55.815 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:55.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:55.848 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:55.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:55.949 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:55.979 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:55.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:55.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:56.081 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:56.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:56.113 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:56.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:56.214 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:56.247 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:56.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:56.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:29:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:29:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[39220]: 4.11 scrub starts 2024-09-10T18:29:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[39220]: 4.11 scrub ok 2024-09-10T18:29:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:29:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:29:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[39220]: 5.4 scrub starts 2024-09-10T18:29:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[39220]: 5.4 scrub ok 2024-09-10T18:29:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[39220]: osdmap e216: 8 total, 8 up, 8 in 2024-09-10T18:29:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[39220]: pgmap v377: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 805 op/s 2024-09-10T18:29:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:29:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:29:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[29291]: 4.11 scrub starts 2024-09-10T18:29:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[29291]: 4.11 scrub ok 2024-09-10T18:29:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:29:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:29:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[29291]: 5.4 scrub starts 2024-09-10T18:29:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[29291]: 5.4 scrub ok 2024-09-10T18:29:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[29291]: osdmap e216: 8 total, 8 up, 8 in 2024-09-10T18:29:56.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:55 smithi028 ceph-mon[29291]: pgmap v377: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 805 op/s 2024-09-10T18:29:56.349 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:55 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:29:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:55 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:29:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:55 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:29:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:55 smithi204 ceph-mon[32357]: 4.11 scrub starts 2024-09-10T18:29:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:55 smithi204 ceph-mon[32357]: 4.11 scrub ok 2024-09-10T18:29:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:55 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:29:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:55 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:29:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:55 smithi204 ceph-mon[32357]: 5.4 scrub starts 2024-09-10T18:29:56.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:55 smithi204 ceph-mon[32357]: 5.4 scrub ok 2024-09-10T18:29:56.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:55 smithi204 ceph-mon[32357]: osdmap e216: 8 total, 8 up, 8 in 2024-09-10T18:29:56.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:55 smithi204 ceph-mon[32357]: pgmap v377: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 805 op/s 2024-09-10T18:29:56.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:56.377 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:56.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:56.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:56.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:56.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:56.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:56.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:56.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:56.659 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:56.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:56.760 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:56.794 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:56.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:56.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:56.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:29:56.897 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:56.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:56.926 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:56.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:57.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:56 smithi028 ceph-mon[29291]: 8.6 deep-scrub starts 2024-09-10T18:29:57.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:56 smithi028 ceph-mon[29291]: 6.1 deep-scrub starts 2024-09-10T18:29:57.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:56 smithi028 ceph-mon[29291]: 6.5 scrub starts 2024-09-10T18:29:57.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:56 smithi028 ceph-mon[29291]: 6.5 scrub ok 2024-09-10T18:29:57.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:56 smithi028 ceph-mon[29291]: 6.1 deep-scrub ok 2024-09-10T18:29:57.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:56 smithi028 ceph-mon[39220]: 8.6 deep-scrub starts 2024-09-10T18:29:57.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:56 smithi028 ceph-mon[39220]: 6.1 deep-scrub starts 2024-09-10T18:29:57.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:56 smithi028 ceph-mon[39220]: 6.5 scrub starts 2024-09-10T18:29:57.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:56 smithi028 ceph-mon[39220]: 6.5 scrub ok 2024-09-10T18:29:57.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:56 smithi028 ceph-mon[39220]: 6.1 deep-scrub ok 2024-09-10T18:29:57.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:57.055 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:57.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:57.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:57.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:57.207 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:57.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:57.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:57.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:57.348 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:57.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:56 smithi204 ceph-mon[32357]: 8.6 deep-scrub starts 2024-09-10T18:29:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:56 smithi204 ceph-mon[32357]: 6.1 deep-scrub starts 2024-09-10T18:29:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:56 smithi204 ceph-mon[32357]: 6.5 scrub starts 2024-09-10T18:29:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:56 smithi204 ceph-mon[32357]: 6.5 scrub ok 2024-09-10T18:29:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:56 smithi204 ceph-mon[32357]: 6.1 deep-scrub ok 2024-09-10T18:29:57.449 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:57.479 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:57.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:57.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:57.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:57.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:57.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:57.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:57.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:57.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:57.750 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:57.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:57.851 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:57.876 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:57.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:57.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:57.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:58.011 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:58.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:58.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:58.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:58.147 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:58.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:58.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:58.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[39220]: 3.4 scrub starts 2024-09-10T18:29:58.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[39220]: 3.4 scrub ok 2024-09-10T18:29:58.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[39220]: 1.2 scrub starts 2024-09-10T18:29:58.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[39220]: 1.2 scrub ok 2024-09-10T18:29:58.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[39220]: pgmap v378: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T18:29:58.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[39220]: 5.2 scrub starts 2024-09-10T18:29:58.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[39220]: 5.2 scrub ok 2024-09-10T18:29:58.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[39220]: 3.14 deep-scrub starts 2024-09-10T18:29:58.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[39220]: 3.14 deep-scrub ok 2024-09-10T18:29:58.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[29291]: 3.4 scrub starts 2024-09-10T18:29:58.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[29291]: 3.4 scrub ok 2024-09-10T18:29:58.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[29291]: 1.2 scrub starts 2024-09-10T18:29:58.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[29291]: 1.2 scrub ok 2024-09-10T18:29:58.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[29291]: pgmap v378: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T18:29:58.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[29291]: 5.2 scrub starts 2024-09-10T18:29:58.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[29291]: 5.2 scrub ok 2024-09-10T18:29:58.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[29291]: 3.14 deep-scrub starts 2024-09-10T18:29:58.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:57 smithi028 ceph-mon[29291]: 3.14 deep-scrub ok 2024-09-10T18:29:58.192 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:29:58.193 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:29:58.249 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:58.280 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:58.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:58.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:57 smithi204 ceph-mon[32357]: 3.4 scrub starts 2024-09-10T18:29:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:57 smithi204 ceph-mon[32357]: 3.4 scrub ok 2024-09-10T18:29:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:57 smithi204 ceph-mon[32357]: 1.2 scrub starts 2024-09-10T18:29:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:57 smithi204 ceph-mon[32357]: 1.2 scrub ok 2024-09-10T18:29:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:57 smithi204 ceph-mon[32357]: pgmap v378: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T18:29:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:57 smithi204 ceph-mon[32357]: 5.2 scrub starts 2024-09-10T18:29:58.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:57 smithi204 ceph-mon[32357]: 5.2 scrub ok 2024-09-10T18:29:58.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:57 smithi204 ceph-mon[32357]: 3.14 deep-scrub starts 2024-09-10T18:29:58.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:57 smithi204 ceph-mon[32357]: 3.14 deep-scrub ok 2024-09-10T18:29:58.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:58.418 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:58.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:58.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:29:58.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:29:58.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:58.557 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:58.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:29:58.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:29:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:58.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:58.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:29:58.743 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:29:58.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:58.840 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:58.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:58.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:58.942 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:29:58.974 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:58.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:58.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:29:59.075 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:59.106 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:59.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:59.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:59.207 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:59.240 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:59.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:59.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:58 smithi028 ceph-mon[39220]: 4.6 scrub starts 2024-09-10T18:29:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:58 smithi028 ceph-mon[39220]: 4.6 scrub ok 2024-09-10T18:29:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:58 smithi028 ceph-mon[39220]: 5.14 scrub starts 2024-09-10T18:29:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:58 smithi028 ceph-mon[39220]: 5.14 scrub ok 2024-09-10T18:29:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:58 smithi028 ceph-mon[39220]: 6.4 scrub starts 2024-09-10T18:29:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:58 smithi028 ceph-mon[39220]: 6.4 scrub ok 2024-09-10T18:29:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:58 smithi028 ceph-mon[39220]: 4.e scrub starts 2024-09-10T18:29:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:58 smithi028 ceph-mon[39220]: 4.e scrub ok 2024-09-10T18:29:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:58 smithi028 ceph-mon[29291]: 4.6 scrub starts 2024-09-10T18:29:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:58 smithi028 ceph-mon[29291]: 4.6 scrub ok 2024-09-10T18:29:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:58 smithi028 ceph-mon[29291]: 5.14 scrub starts 2024-09-10T18:29:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:58 smithi028 ceph-mon[29291]: 5.14 scrub ok 2024-09-10T18:29:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:58 smithi028 ceph-mon[29291]: 6.4 scrub starts 2024-09-10T18:29:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:58 smithi028 ceph-mon[29291]: 6.4 scrub ok 2024-09-10T18:29:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:58 smithi028 ceph-mon[29291]: 4.e scrub starts 2024-09-10T18:29:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:58 smithi028 ceph-mon[29291]: 4.e scrub ok 2024-09-10T18:29:59.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:29:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:58 smithi204 ceph-mon[32357]: 4.6 scrub starts 2024-09-10T18:29:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:58 smithi204 ceph-mon[32357]: 4.6 scrub ok 2024-09-10T18:29:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:58 smithi204 ceph-mon[32357]: 5.14 scrub starts 2024-09-10T18:29:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:58 smithi204 ceph-mon[32357]: 5.14 scrub ok 2024-09-10T18:29:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:58 smithi204 ceph-mon[32357]: 6.4 scrub starts 2024-09-10T18:29:59.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:58 smithi204 ceph-mon[32357]: 6.4 scrub ok 2024-09-10T18:29:59.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:58 smithi204 ceph-mon[32357]: 4.e scrub starts 2024-09-10T18:29:59.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:58 smithi204 ceph-mon[32357]: 4.e scrub ok 2024-09-10T18:29:59.367 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:59.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:29:59.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:29:59.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:59.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:59.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:29:59.599 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:29:59.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:59.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:29:59.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:29:59.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:29:59.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:59.778 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:59.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:29:59.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:29:59.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:29:59.923 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:29:59.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:00.024 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:00.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:00.045 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:00.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:00.146 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:00.173 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:00.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:00.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:29:59 smithi028 ceph-mon[39220]: pgmap v379: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 564 B/s rd, 0 B/s wr, 1.08k op/s 2024-09-10T18:30:00.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:29:59 smithi028 ceph-mon[29291]: pgmap v379: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 564 B/s rd, 0 B/s wr, 1.08k op/s 2024-09-10T18:30:00.275 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:00.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:00.311 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:00.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:29:59 smithi204 ceph-mon[32357]: pgmap v379: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 564 B/s rd, 0 B/s wr, 1.08k op/s 2024-09-10T18:30:00.412 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:00.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:00.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:00.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:00.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:00.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:00.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:00.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:00.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:00.810 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:30:00.819 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:00.843 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:00.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:00.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:00.929 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.1 to 1 (10000) 2024-09-10T18:30:00.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:00.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:00.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:00.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:01.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:00 smithi028 ceph-mon[29291]: 8.e scrub starts 2024-09-10T18:30:01.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:00 smithi028 ceph-mon[29291]: overall HEALTH_OK 2024-09-10T18:30:01.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:00 smithi028 ceph-mon[29291]: 1.a scrub starts 2024-09-10T18:30:01.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:00 smithi028 ceph-mon[29291]: 1.a scrub ok 2024-09-10T18:30:01.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:00 smithi028 ceph-mon[29291]: 5.e scrub starts 2024-09-10T18:30:01.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:00 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1443255045' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:30:01.013 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:00 smithi028 ceph-mon[39220]: 8.e scrub starts 2024-09-10T18:30:01.014 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:00 smithi028 ceph-mon[39220]: overall HEALTH_OK 2024-09-10T18:30:01.014 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:00 smithi028 ceph-mon[39220]: 1.a scrub starts 2024-09-10T18:30:01.014 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:00 smithi028 ceph-mon[39220]: 1.a scrub ok 2024-09-10T18:30:01.014 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:00 smithi028 ceph-mon[39220]: 5.e scrub starts 2024-09-10T18:30:01.014 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:00 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1443255045' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:30:01.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:01.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:01.142 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:01.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:01.243 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:01.270 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:01.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:01.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:00 smithi204 ceph-mon[32357]: 8.e scrub starts 2024-09-10T18:30:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:00 smithi204 ceph-mon[32357]: overall HEALTH_OK 2024-09-10T18:30:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:00 smithi204 ceph-mon[32357]: 1.a scrub starts 2024-09-10T18:30:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:00 smithi204 ceph-mon[32357]: 1.a scrub ok 2024-09-10T18:30:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:00 smithi204 ceph-mon[32357]: 5.e scrub starts 2024-09-10T18:30:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:00 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1443255045' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:30:01.372 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:01.398 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:01.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:01.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:01.500 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:01.532 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:01.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:01.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:01.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:01.665 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:01.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:01.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:01.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:30:01] "GET /metrics HTTP/1.1" 200 224283 "" "Prometheus/2.43.0" 2024-09-10T18:30:01.767 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:01.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:01.798 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:01.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:01.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:01.934 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:01.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:01.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:02.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:02.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:02.078 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:02.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:02.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:02.200 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:02.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:02.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[29291]: 5.e scrub ok 2024-09-10T18:30:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[29291]: 5.1a scrub starts 2024-09-10T18:30:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[29291]: 5.1a scrub ok 2024-09-10T18:30:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[29291]: 5.1d scrub starts 2024-09-10T18:30:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[29291]: 5.1d scrub ok 2024-09-10T18:30:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[29291]: pgmap v380: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T18:30:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1443255045' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T18:30:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[29291]: osdmap e217: 8 total, 8 up, 8 in 2024-09-10T18:30:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[29291]: 1.1 scrub starts 2024-09-10T18:30:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[29291]: 1.1 scrub ok 2024-09-10T18:30:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[29291]: 3.9 scrub starts 2024-09-10T18:30:02.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[29291]: 3.9 scrub ok 2024-09-10T18:30:02.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[39220]: 5.e scrub ok 2024-09-10T18:30:02.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[39220]: 5.1a scrub starts 2024-09-10T18:30:02.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[39220]: 5.1a scrub ok 2024-09-10T18:30:02.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[39220]: 5.1d scrub starts 2024-09-10T18:30:02.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[39220]: 5.1d scrub ok 2024-09-10T18:30:02.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[39220]: pgmap v380: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T18:30:02.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1443255045' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T18:30:02.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[39220]: osdmap e217: 8 total, 8 up, 8 in 2024-09-10T18:30:02.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[39220]: 1.1 scrub starts 2024-09-10T18:30:02.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[39220]: 1.1 scrub ok 2024-09-10T18:30:02.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[39220]: 3.9 scrub starts 2024-09-10T18:30:02.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:01 smithi028 ceph-mon[39220]: 3.9 scrub ok 2024-09-10T18:30:02.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:02.341 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:02.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:02.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:02.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:30:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:30:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:30:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:01 smithi204 ceph-mon[32357]: 5.e scrub ok 2024-09-10T18:30:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:01 smithi204 ceph-mon[32357]: 5.1a scrub starts 2024-09-10T18:30:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:01 smithi204 ceph-mon[32357]: 5.1a scrub ok 2024-09-10T18:30:02.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:01 smithi204 ceph-mon[32357]: 5.1d scrub starts 2024-09-10T18:30:02.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:01 smithi204 ceph-mon[32357]: 5.1d scrub ok 2024-09-10T18:30:02.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:01 smithi204 ceph-mon[32357]: pgmap v380: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T18:30:02.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:01 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1443255045' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T18:30:02.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:01 smithi204 ceph-mon[32357]: osdmap e217: 8 total, 8 up, 8 in 2024-09-10T18:30:02.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:01 smithi204 ceph-mon[32357]: 1.1 scrub starts 2024-09-10T18:30:02.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:01 smithi204 ceph-mon[32357]: 1.1 scrub ok 2024-09-10T18:30:02.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:01 smithi204 ceph-mon[32357]: 3.9 scrub starts 2024-09-10T18:30:02.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:01 smithi204 ceph-mon[32357]: 3.9 scrub ok 2024-09-10T18:30:02.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:02.481 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:02.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:02.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:02.583 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:02.609 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:02.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:02.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:02.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:02.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:02.849 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:02.876 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:02.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:02.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:02.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:03.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:03.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:03.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:03.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:03.143 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:03.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:03.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:03.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:02 smithi028 ceph-mon[29291]: 6.16 scrub starts 2024-09-10T18:30:03.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:02 smithi028 ceph-mon[29291]: 6.16 scrub ok 2024-09-10T18:30:03.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:02 smithi028 ceph-mon[29291]: 3.d scrub starts 2024-09-10T18:30:03.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:02 smithi028 ceph-mon[29291]: 3.d scrub ok 2024-09-10T18:30:03.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:02 smithi028 ceph-mon[29291]: 8.1 scrub starts 2024-09-10T18:30:03.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:02 smithi028 ceph-mon[29291]: 3.8 scrub starts 2024-09-10T18:30:03.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:02 smithi028 ceph-mon[29291]: 3.8 scrub ok 2024-09-10T18:30:03.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:02 smithi028 ceph-mon[29291]: 8.2 scrub starts 2024-09-10T18:30:03.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:02 smithi028 ceph-mon[39220]: 6.16 scrub starts 2024-09-10T18:30:03.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:02 smithi028 ceph-mon[39220]: 6.16 scrub ok 2024-09-10T18:30:03.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:02 smithi028 ceph-mon[39220]: 3.d scrub starts 2024-09-10T18:30:03.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:02 smithi028 ceph-mon[39220]: 3.d scrub ok 2024-09-10T18:30:03.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:02 smithi028 ceph-mon[39220]: 8.1 scrub starts 2024-09-10T18:30:03.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:02 smithi028 ceph-mon[39220]: 3.8 scrub starts 2024-09-10T18:30:03.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:02 smithi028 ceph-mon[39220]: 3.8 scrub ok 2024-09-10T18:30:03.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:02 smithi028 ceph-mon[39220]: 8.2 scrub starts 2024-09-10T18:30:03.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-10T18:30:03.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:03.323 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:03.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:03.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:02 smithi204 ceph-mon[32357]: 6.16 scrub starts 2024-09-10T18:30:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:02 smithi204 ceph-mon[32357]: 6.16 scrub ok 2024-09-10T18:30:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:02 smithi204 ceph-mon[32357]: 3.d scrub starts 2024-09-10T18:30:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:02 smithi204 ceph-mon[32357]: 3.d scrub ok 2024-09-10T18:30:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:02 smithi204 ceph-mon[32357]: 8.1 scrub starts 2024-09-10T18:30:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:02 smithi204 ceph-mon[32357]: 3.8 scrub starts 2024-09-10T18:30:03.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:02 smithi204 ceph-mon[32357]: 3.8 scrub ok 2024-09-10T18:30:03.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:02 smithi204 ceph-mon[32357]: 8.2 scrub starts 2024-09-10T18:30:03.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:03.453 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:03.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:03.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:03.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:03.582 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:03.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:03.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:03.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:03.721 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:03.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:03.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:03.823 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:03.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:03.851 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:03.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:03.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:04.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:04.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:04.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:04.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[29291]: 3.f scrub starts 2024-09-10T18:30:04.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[29291]: 3.0 deep-scrub starts 2024-09-10T18:30:04.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[29291]: 3.0 deep-scrub ok 2024-09-10T18:30:04.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[29291]: 5.17 scrub starts 2024-09-10T18:30:04.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[29291]: 5.17 scrub ok 2024-09-10T18:30:04.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[29291]: pgmap v382: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 636 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-10T18:30:04.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[29291]: 5.1f deep-scrub starts 2024-09-10T18:30:04.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[29291]: 5.1f deep-scrub ok 2024-09-10T18:30:04.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[29291]: 6.2 scrub starts 2024-09-10T18:30:04.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[29291]: 6.2 scrub ok 2024-09-10T18:30:04.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:30:04.063 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[39220]: 3.f scrub starts 2024-09-10T18:30:04.063 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[39220]: 3.0 deep-scrub starts 2024-09-10T18:30:04.063 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[39220]: 3.0 deep-scrub ok 2024-09-10T18:30:04.064 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[39220]: 5.17 scrub starts 2024-09-10T18:30:04.064 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[39220]: 5.17 scrub ok 2024-09-10T18:30:04.064 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[39220]: pgmap v382: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 636 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-10T18:30:04.064 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[39220]: 5.1f deep-scrub starts 2024-09-10T18:30:04.064 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[39220]: 5.1f deep-scrub ok 2024-09-10T18:30:04.064 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[39220]: 6.2 scrub starts 2024-09-10T18:30:04.064 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[39220]: 6.2 scrub ok 2024-09-10T18:30:04.064 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:03 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:30:04.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:04.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:04.147 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:04.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:04.248 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:04.285 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:03 smithi204 ceph-mon[32357]: 3.f scrub starts 2024-09-10T18:30:04.285 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:03 smithi204 ceph-mon[32357]: 3.0 deep-scrub starts 2024-09-10T18:30:04.285 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:03 smithi204 ceph-mon[32357]: 3.0 deep-scrub ok 2024-09-10T18:30:04.285 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:03 smithi204 ceph-mon[32357]: 5.17 scrub starts 2024-09-10T18:30:04.285 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:03 smithi204 ceph-mon[32357]: 5.17 scrub ok 2024-09-10T18:30:04.285 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:03 smithi204 ceph-mon[32357]: pgmap v382: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 636 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-10T18:30:04.286 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:03 smithi204 ceph-mon[32357]: 5.1f deep-scrub starts 2024-09-10T18:30:04.286 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:03 smithi204 ceph-mon[32357]: 5.1f deep-scrub ok 2024-09-10T18:30:04.286 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:03 smithi204 ceph-mon[32357]: 6.2 scrub starts 2024-09-10T18:30:04.286 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:03 smithi204 ceph-mon[32357]: 6.2 scrub ok 2024-09-10T18:30:04.286 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:03 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:30:04.286 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:04.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:04.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:04.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:04.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:04.427 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:04.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:04.529 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:04.553 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:04.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:04.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:04.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:30:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:30:04.656 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:04.686 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:04.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:04.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:04.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:04.831 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:04.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:04.933 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:04.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:04.967 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:04.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:05.068 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:05.106 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:05.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:05.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:05.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:05.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:04 smithi028 ceph-mon[39220]: 4.12 scrub starts 2024-09-10T18:30:05.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:04 smithi028 ceph-mon[39220]: 4.12 scrub ok 2024-09-10T18:30:05.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:04 smithi028 ceph-mon[39220]: 8.1 scrub ok 2024-09-10T18:30:05.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:04 smithi028 ceph-mon[39220]: 5.1b scrub starts 2024-09-10T18:30:05.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:04 smithi028 ceph-mon[39220]: 5.1b scrub ok 2024-09-10T18:30:05.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:04 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:30:05.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:04 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:30:05.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:04 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:30:05.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:04 smithi028 ceph-mon[29291]: 4.12 scrub starts 2024-09-10T18:30:05.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:04 smithi028 ceph-mon[29291]: 4.12 scrub ok 2024-09-10T18:30:05.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:04 smithi028 ceph-mon[29291]: 8.1 scrub ok 2024-09-10T18:30:05.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:04 smithi028 ceph-mon[29291]: 5.1b scrub starts 2024-09-10T18:30:05.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:04 smithi028 ceph-mon[29291]: 5.1b scrub ok 2024-09-10T18:30:05.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:04 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:30:05.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:04 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:30:05.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:04 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:30:05.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:05.258 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:05.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:04 smithi204 ceph-mon[32357]: 4.12 scrub starts 2024-09-10T18:30:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:04 smithi204 ceph-mon[32357]: 4.12 scrub ok 2024-09-10T18:30:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:04 smithi204 ceph-mon[32357]: 8.1 scrub ok 2024-09-10T18:30:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:04 smithi204 ceph-mon[32357]: 5.1b scrub starts 2024-09-10T18:30:05.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:04 smithi204 ceph-mon[32357]: 5.1b scrub ok 2024-09-10T18:30:05.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:04 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:30:05.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:04 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:30:05.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:04 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:30:05.359 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:05.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:05.392 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:05.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:05.493 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:05.526 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:05.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:05.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:05.628 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:05.654 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:05.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:05.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:05.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:05.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:05.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:05.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:05.910 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:05.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:05.935 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:05.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:06.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:05 smithi028 ceph-mon[39220]: 3.6 scrub starts 2024-09-10T18:30:06.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:05 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:06.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:05 smithi028 ceph-mon[39220]: 3.6 scrub ok 2024-09-10T18:30:06.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:05 smithi028 ceph-mon[39220]: 8.0 scrub starts 2024-09-10T18:30:06.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:05 smithi028 ceph-mon[39220]: pgmap v383: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:30:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:05 smithi028 ceph-mon[29291]: 3.6 scrub starts 2024-09-10T18:30:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:05 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:05 smithi028 ceph-mon[29291]: 3.6 scrub ok 2024-09-10T18:30:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:05 smithi028 ceph-mon[29291]: 8.0 scrub starts 2024-09-10T18:30:06.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:05 smithi028 ceph-mon[29291]: pgmap v383: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:30:06.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:06.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:06.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:06.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:06.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:06.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:06.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:06.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:05 smithi204 ceph-mon[32357]: 3.6 scrub starts 2024-09-10T18:30:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:05 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:05 smithi204 ceph-mon[32357]: 3.6 scrub ok 2024-09-10T18:30:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:05 smithi204 ceph-mon[32357]: 8.0 scrub starts 2024-09-10T18:30:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:05 smithi204 ceph-mon[32357]: pgmap v383: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:30:06.357 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:06.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:06.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:06.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:06.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:06.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:06.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-10T18:30:06.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:06.645 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:06.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:06.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:06.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:06.789 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:30:06.814 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:06.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:06.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:06.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:06.966 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:06.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:07.067 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:07.096 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:07.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:07.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:07.198 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:07.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:07.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:07.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:06 smithi028 ceph-mon[39220]: 3.17 deep-scrub starts 2024-09-10T18:30:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:06 smithi028 ceph-mon[39220]: 3.17 deep-scrub ok 2024-09-10T18:30:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:06 smithi028 ceph-mon[39220]: 8.b deep-scrub starts 2024-09-10T18:30:07.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:06 smithi028 ceph-mon[29291]: 3.17 deep-scrub starts 2024-09-10T18:30:07.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:06 smithi028 ceph-mon[29291]: 3.17 deep-scrub ok 2024-09-10T18:30:07.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:06 smithi028 ceph-mon[29291]: 8.b deep-scrub starts 2024-09-10T18:30:07.334 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:06 smithi204 ceph-mon[32357]: 3.17 deep-scrub starts 2024-09-10T18:30:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:06 smithi204 ceph-mon[32357]: 3.17 deep-scrub ok 2024-09-10T18:30:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:06 smithi204 ceph-mon[32357]: 8.b deep-scrub starts 2024-09-10T18:30:07.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:07.361 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:07.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:07.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:07.490 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:07.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:07.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:07.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:07.621 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:07.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:07.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:07.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:07.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:07.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:07.859 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:07.890 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:07.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:07.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:07.956 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:30:07.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:07 smithi028 ceph-mon[29291]: 8.0 scrub ok 2024-09-10T18:30:07.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:07 smithi028 ceph-mon[29291]: pgmap v384: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 155 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:30:07.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:07 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/309313871' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:30:07.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:07 smithi028 ceph-mon[39220]: 8.0 scrub ok 2024-09-10T18:30:07.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:07 smithi028 ceph-mon[39220]: pgmap v384: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 155 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:30:07.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:08.019 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:08.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:08.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:08.121 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:08.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:08.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:08.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:08.239 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:07 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/309313871' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:30:08.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:08.301 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:08.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:08.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:07 smithi204 ceph-mon[32357]: 8.0 scrub ok 2024-09-10T18:30:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:07 smithi204 ceph-mon[32357]: pgmap v384: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 155 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:30:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:07 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/309313871' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:30:08.403 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:08.433 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:08.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:08.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:08.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:08.566 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:08.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:08.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:08.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:08.701 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:08.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:08.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:08.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:08.840 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:08.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:08.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:08.942 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:08.970 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:08.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:08.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:09.072 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:09.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:09.104 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:09.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:09.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:09.239 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:09.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:09.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:09.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:08 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:30:09.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:08 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/309313871' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:30:09.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:08 smithi028 ceph-mon[39220]: osdmap e218: 8 total, 8 up, 8 in 2024-09-10T18:30:09.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:08 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:30:09.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:08 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:30:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:08 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:30:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:08 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/309313871' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:30:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:08 smithi028 ceph-mon[29291]: osdmap e218: 8 total, 8 up, 8 in 2024-09-10T18:30:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:08 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:30:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:08 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:30:09.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:08 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:30:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:08 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/309313871' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:30:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:08 smithi204 ceph-mon[32357]: osdmap e218: 8 total, 8 up, 8 in 2024-09-10T18:30:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:08 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:30:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:08 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:30:09.363 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:09.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:09.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:09.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:09.508 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:09.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:09.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:09.610 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:09.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:09.642 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:09.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:09.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:09.784 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:09.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:09.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:09.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:09.908 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:09.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:09.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:10.010 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:10.047 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:10.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:10.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:10.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:10.186 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:10.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:10.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:09 smithi028 ceph-mon[39220]: pgmap v386: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 155 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 621 op/s 2024-09-10T18:30:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-10T18:30:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-10T18:30:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:09 smithi028 ceph-mon[29291]: pgmap v386: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 155 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 621 op/s 2024-09-10T18:30:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-10T18:30:10.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-10T18:30:10.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:10.319 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:10.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:10.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:09 smithi204 ceph-mon[32357]: pgmap v386: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 155 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 621 op/s 2024-09-10T18:30:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-10T18:30:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-10T18:30:10.385 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:30:10.385 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:30:10.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:10.421 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:30:10.422 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:30:10.452 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:30:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:30:10.461 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:10.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:10.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:10.479 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:30:10.479 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:30:10.497 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:30:10.497 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:30:10.521 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:30:10.522 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:30:10.558 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:30:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:30:10.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:10.592 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:30:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:30:10.596 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:10.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:10.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:10.722 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:10.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:10.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:10.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:10.867 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:10.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:10.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:10.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:10.999 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:11.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:11.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:11.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:11.134 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:11.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:11.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:11.236 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:11 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-10T18:30:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:11 smithi028 ceph-mon[39220]: osdmap e219: 8 total, 8 up, 8 in 2024-09-10T18:30:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:11 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/558590071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:30:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:10 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-10T18:30:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:10 smithi028 ceph-mon[29291]: osdmap e219: 8 total, 8 up, 8 in 2024-09-10T18:30:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:11 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/558590071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:30:11.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:11.269 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:11.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:10 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-10T18:30:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:10 smithi204 ceph-mon[32357]: osdmap e219: 8 total, 8 up, 8 in 2024-09-10T18:30:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:10 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/558590071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:30:11.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:11.397 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:11.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:11.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:11.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:11.529 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:11.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:11.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:11.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:11.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:11.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:11.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:30:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:30:11] "GET /metrics HTTP/1.1" 200 224306 "" "Prometheus/2.43.0" 2024-09-10T18:30:11.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:11.820 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:11.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:11.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:11.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:11.950 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:11.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:12.051 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:12.077 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:12.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:12.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:12.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:12.212 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:12.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:12.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:12.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:11 smithi028 ceph-mon[29291]: pgmap v388: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-10T18:30:12.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:11 smithi028 ceph-mon[29291]: osdmap e220: 8 total, 8 up, 8 in 2024-09-10T18:30:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:11 smithi028 ceph-mon[39220]: pgmap v388: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-10T18:30:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:11 smithi028 ceph-mon[39220]: osdmap e220: 8 total, 8 up, 8 in 2024-09-10T18:30:12.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:12.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:12.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:11 smithi204 ceph-mon[32357]: pgmap v388: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-10T18:30:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:11 smithi204 ceph-mon[32357]: osdmap e220: 8 total, 8 up, 8 in 2024-09-10T18:30:12.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:30:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:30:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:30:12.451 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:12.478 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:12.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:12.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:12.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:12.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:12.625 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:12.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:12.726 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:12.748 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:12.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:12.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:12.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:12.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:12.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:13.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:13.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:13.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:13.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:13.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:13.246 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:13.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:13.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:13.348 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:13.375 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:13.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:13.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:13.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:13.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:13.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:13.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:13.653 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:13.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:13.754 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:13.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:13.787 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:13.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:13.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:13.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:13.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:14.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:14.066 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:14.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:14.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:14.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:14.198 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:14.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:14.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:14.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:14 smithi028 ceph-mon[39220]: pgmap v390: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 696 op/s 2024-09-10T18:30:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:14 smithi028 ceph-mon[29291]: pgmap v390: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 696 op/s 2024-09-10T18:30:14.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:14.323 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:14 smithi204 ceph-mon[32357]: pgmap v390: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 696 op/s 2024-09-10T18:30:14.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:14.334 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:14.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:14.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:30:14.435 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:14.462 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:14.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:14.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:14.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:14.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:14.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:30:14 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:30:14.704 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:14.731 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:14.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:14.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:14.790 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:30:14.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:14.839 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:30:14.839 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-10T18:30:14.839 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-10T18:30:14.839 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-10T18:30:14.839 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:30:14.859 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:14.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:14.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:14.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:14.993 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:14.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:14.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:15.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:15.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:15.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:15.184 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:15 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:30:15.184 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:15 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:30:15.184 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:15 smithi028 ceph-mon[39220]: osdmap e221: 8 total, 8 up, 8 in 2024-09-10T18:30:15.184 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:15 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:30:15.184 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:15 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:30:15.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:15 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:30:15.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:15 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:30:15.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:15 smithi028 ceph-mon[29291]: osdmap e221: 8 total, 8 up, 8 in 2024-09-10T18:30:15.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:15 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:30:15.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:15 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:30:15.239 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:15.271 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:15.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:15.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:15 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:30:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:15 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:30:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:15 smithi204 ceph-mon[32357]: osdmap e221: 8 total, 8 up, 8 in 2024-09-10T18:30:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:15 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:30:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:15 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:30:15.374 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:15.406 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:15.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:15.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:15.508 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:15.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:15.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:15.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:15.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-10T18:30:15.640 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:15.664 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:15.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:15.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:15.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:15.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:15.803 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:15.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:15.905 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:15.936 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:15.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:15.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:16.038 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:16.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:16.070 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:16.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:16.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:16.208 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:16.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:16.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:16.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:16 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:16.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:16 smithi028 ceph-mon[29291]: pgmap v392: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 865 op/s; 18 B/s, 0 objects/s recovering 2024-09-10T18:30:16.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:16 smithi028 ceph-mon[29291]: osdmap e222: 8 total, 8 up, 8 in 2024-09-10T18:30:16.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:16 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:16.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:16 smithi028 ceph-mon[39220]: pgmap v392: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 865 op/s; 18 B/s, 0 objects/s recovering 2024-09-10T18:30:16.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:16 smithi028 ceph-mon[39220]: osdmap e222: 8 total, 8 up, 8 in 2024-09-10T18:30:16.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:16.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:16.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:16.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:16 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:16 smithi204 ceph-mon[32357]: pgmap v392: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 865 op/s; 18 B/s, 0 objects/s recovering 2024-09-10T18:30:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:16 smithi204 ceph-mon[32357]: osdmap e222: 8 total, 8 up, 8 in 2024-09-10T18:30:16.455 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:16.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:16.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:16.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:16.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:30:16.590 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:16.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:16.620 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:16.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:16.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:16.755 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:16.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:16.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:16.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:16.892 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:16.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:16.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:30:16.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:30:17.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:17.055 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:17.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:17.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:17.183 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:17.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:17.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:17.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:17.310 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:17.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:17.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:17.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:17.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:17.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:17.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:17.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:17.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:17.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:17.693 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:17.717 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:17.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:17.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:17.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:17.858 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:17.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:30:17.960 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:17.988 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:17.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:17.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:18.074 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:18 smithi028 ceph-mon[39220]: pgmap v394: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 699 op/s; 18 B/s, 0 objects/s recovering 2024-09-10T18:30:18.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:18 smithi028 ceph-mon[29291]: pgmap v394: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 699 op/s; 18 B/s, 0 objects/s recovering 2024-09-10T18:30:18.089 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:18.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:18.123 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:18.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:18.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:18.251 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:18.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:18.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:18.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:18.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:18 smithi204 ceph-mon[32357]: pgmap v394: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 699 op/s; 18 B/s, 0 objects/s recovering 2024-09-10T18:30:18.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:18.394 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:18.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:18.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:18.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:18.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:18.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:18.690 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:18.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:18.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:18.792 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:18.818 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:18.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:18.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:18.963 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:18.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:18.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:19.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:19.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:19.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:19.245 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:19.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:19.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:19.347 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:19.374 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:19.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:19.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:19.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:19.506 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:19.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:19.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:19.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:19.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:19.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:19.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:19.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:19.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:19.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:19.917 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:19.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:19.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:20.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:20.048 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:20.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:20.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:20.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:20.179 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:20.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:20.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:20.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:20 smithi028 ceph-mon[29291]: pgmap v395: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 529 op/s; 13 B/s, 0 objects/s recovering 2024-09-10T18:30:20.276 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:20 smithi028 ceph-mon[39220]: pgmap v395: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 529 op/s; 13 B/s, 0 objects/s recovering 2024-09-10T18:30:20.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:20.306 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:20.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:20.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:20.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:20 smithi204 ceph-mon[32357]: pgmap v395: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 529 op/s; 13 B/s, 0 objects/s recovering 2024-09-10T18:30:20.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:20.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:20.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:20.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:20.586 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:20.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:20.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:20.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:20.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:20.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:20.838 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:20.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:20.864 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:20.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:20.899 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:30:20.899 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-10T18:30:20.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-10T18:30:20.899 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:30:20.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.900 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31752 7.d 7:b2ee7856:::benchmark_data_smithi028_77856_object31751:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:30:20.900 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.836508+0000", 2024-09-10T18:30:20.900 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 383.00134208899999, 2024-09-10T18:30:20.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14737983199999999, 2024-09-10T18:30:20.900 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:20.900 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:20.900 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:20.900 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:20.900 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:20.900 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31752 2024-09-10T18:30:20.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.900 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:20.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:20.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836508+0000", 2024-09-10T18:30:20.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:20.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836508+0000", 2024-09-10T18:30:20.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:20.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836506+0000", 2024-09-10T18:30:20.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:30:20.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:20.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836529+0000", 2024-09-10T18:30:20.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2852000000000001e-05 2024-09-10T18:30:20.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:20.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836539+0000", 2024-09-10T18:30:20.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.005e-05 2024-09-10T18:30:20.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:20.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836545+0000", 2024-09-10T18:30:20.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2580000000000003e-06 2024-09-10T18:30:20.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836594+0000", 2024-09-10T18:30:20.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8971000000000003e-05 2024-09-10T18:30:20.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:20.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836701+0000", 2024-09-10T18:30:20.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107365 2024-09-10T18:30:20.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-10T18:30:20.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836834+0000", 2024-09-10T18:30:20.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013344799999999999 2024-09-10T18:30:20.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:20.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.839823+0000", 2024-09-10T18:30:20.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029885760000000002 2024-09-10T18:30:20.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.983711+0000", 2024-09-10T18:30:20.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14388826800000001 2024-09-10T18:30:20.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:20.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.983767+0000", 2024-09-10T18:30:20.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5339e-05 2024-09-10T18:30:20.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:20.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.983888+0000", 2024-09-10T18:30:20.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121433 2024-09-10T18:30:20.906 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.906 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:20.906 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.906 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31760 7.d 7:be0b06ed:::benchmark_data_smithi028_77856_object31759:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:30:20.906 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.838440+0000", 2024-09-10T18:30:20.906 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 382.99941052899999, 2024-09-10T18:30:20.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14565720700000001, 2024-09-10T18:30:20.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:20.906 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:20.906 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:20.907 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:20.907 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:20.907 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31760 2024-09-10T18:30:20.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.907 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:20.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:20.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838440+0000", 2024-09-10T18:30:20.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:20.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838440+0000", 2024-09-10T18:30:20.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:20.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838439+0000", 2024-09-10T18:30:20.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:30:20.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:20.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838463+0000", 2024-09-10T18:30:20.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4530000000000001e-05 2024-09-10T18:30:20.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:20.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838471+0000", 2024-09-10T18:30:20.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0399999999999993e-06 2024-09-10T18:30:20.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:20.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838476+0000", 2024-09-10T18:30:20.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6609999999999999e-06 2024-09-10T18:30:20.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838534+0000", 2024-09-10T18:30:20.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8017999999999999e-05 2024-09-10T18:30:20.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:20.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838664+0000", 2024-09-10T18:30:20.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129605 2024-09-10T18:30:20.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-10T18:30:20.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838791+0000", 2024-09-10T18:30:20.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012758999999999999 2024-09-10T18:30:20.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:20.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.840006+0000", 2024-09-10T18:30:20.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012147099999999999 2024-09-10T18:30:20.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.983943+0000", 2024-09-10T18:30:20.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14393716400000001 2024-09-10T18:30:20.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:20.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.984008+0000", 2024-09-10T18:30:20.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4768000000000001e-05 2024-09-10T18:30:20.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:20.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.984097+0000", 2024-09-10T18:30:20.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9123999999999996e-05 2024-09-10T18:30:20.911 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.912 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:20.912 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.912 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181183 7.d 7:b1361f07:::benchmark_data_smithi028_77856_object60083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:30:20.912 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.055163+0000", 2024-09-10T18:30:20.912 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 235.78268775199999, 2024-09-10T18:30:20.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15025240300000001, 2024-09-10T18:30:20.912 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:20.912 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:20.912 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:20.912 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:20.912 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:20.912 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181183 2024-09-10T18:30:20.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.913 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:20.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:20.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055163+0000", 2024-09-10T18:30:20.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:20.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055163+0000", 2024-09-10T18:30:20.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.914 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:20.914 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055160+0000", 2024-09-10T18:30:20.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:30:20.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.914 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:20.914 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055167+0000", 2024-09-10T18:30:20.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9469999999999997e-06 2024-09-10T18:30:20.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.914 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:20.914 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055175+0000", 2024-09-10T18:30:20.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1330000000000001e-06 2024-09-10T18:30:20.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:20.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055180+0000", 2024-09-10T18:30:20.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1699999999999996e-06 2024-09-10T18:30:20.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055225+0000", 2024-09-10T18:30:20.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5441000000000001e-05 2024-09-10T18:30:20.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:20.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055329+0000", 2024-09-10T18:30:20.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103892 2024-09-10T18:30:20.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-10T18:30:20.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055440+0000", 2024-09-10T18:30:20.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011092800000000001 2024-09-10T18:30:20.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:20.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.057624+0000", 2024-09-10T18:30:20.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021838669999999999 2024-09-10T18:30:20.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205284+0000", 2024-09-10T18:30:20.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14766003999999999 2024-09-10T18:30:20.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.917 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:20.917 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205332+0000", 2024-09-10T18:30:20.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7800000000000003e-05 2024-09-10T18:30:20.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.917 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:20.917 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205415+0000", 2024-09-10T18:30:20.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3227999999999997e-05 2024-09-10T18:30:20.917 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.917 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:20.917 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.918 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221438 7.b 7:d4df1cfe:::benchmark_data_smithi028_77856_object100338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:30:20.918 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.137682+0000", 2024-09-10T18:30:20.918 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.70016837200001, 2024-09-10T18:30:20.918 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99608778499999995, 2024-09-10T18:30:20.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:20.918 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:20.918 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:20.918 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:20.918 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:20.918 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221438 2024-09-10T18:30:20.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.919 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:20.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.919 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:20.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.137682+0000", 2024-09-10T18:30:20.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.919 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:20.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.137682+0000", 2024-09-10T18:30:20.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.919 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:20.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.137680+0000", 2024-09-10T18:30:20.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:30:20.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:20.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.137687+0000", 2024-09-10T18:30:20.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.469e-06 2024-09-10T18:30:20.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:20.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.137705+0000", 2024-09-10T18:30:20.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7915e-05 2024-09-10T18:30:20.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:20.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.137712+0000", 2024-09-10T18:30:20.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5740000000000001e-06 2024-09-10T18:30:20.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.137758+0000", 2024-09-10T18:30:20.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6362000000000001e-05 2024-09-10T18:30:20.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:30:20.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.137765+0000", 2024-09-10T18:30:20.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3209999999999999e-06 2024-09-10T18:30:20.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.129246+0000", 2024-09-10T18:30:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99148095599999997 2024-09-10T18:30:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.129370+0000", 2024-09-10T18:30:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012408100000000001 2024-09-10T18:30:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:30:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.129441+0000", 2024-09-10T18:30:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0684000000000003e-05 2024-09-10T18:30:20.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:20.923 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133644+0000", 2024-09-10T18:30:20.923 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042028459999999997 2024-09-10T18:30:20.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.923 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133680+0000", 2024-09-10T18:30:20.923 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5540999999999997e-05 2024-09-10T18:30:20.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.923 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133710+0000", 2024-09-10T18:30:20.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0059000000000001e-05 2024-09-10T18:30:20.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:20.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133747+0000", 2024-09-10T18:30:20.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7826e-05 2024-09-10T18:30:20.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:20.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133770+0000", 2024-09-10T18:30:20.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2313000000000001e-05 2024-09-10T18:30:20.924 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.924 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:20.924 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.925 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221491 7.b 7:de887b9b:::benchmark_data_smithi028_77856_object100391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:30:20.925 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.187950+0000", 2024-09-10T18:30:20.925 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.64989998900001, 2024-09-10T18:30:20.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.94855177599999996, 2024-09-10T18:30:20.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:20.925 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:20.925 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:20.925 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:20.925 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:20.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221491 2024-09-10T18:30:20.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.926 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:20.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.926 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:20.926 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.187950+0000", 2024-09-10T18:30:20.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.926 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:20.926 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.187950+0000", 2024-09-10T18:30:20.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.926 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:20.926 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.187948+0000", 2024-09-10T18:30:20.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-10T18:30:20.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.927 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:20.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.187958+0000", 2024-09-10T18:30:20.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8379999999999997e-06 2024-09-10T18:30:20.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.927 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:20.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.187973+0000", 2024-09-10T18:30:20.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.517e-05 2024-09-10T18:30:20.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:20.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.187980+0000", 2024-09-10T18:30:20.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2830000000000004e-06 2024-09-10T18:30:20.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.188030+0000", 2024-09-10T18:30:20.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0408000000000003e-05 2024-09-10T18:30:20.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:30:20.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.188039+0000", 2024-09-10T18:30:20.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8179999999999993e-06 2024-09-10T18:30:20.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.131369+0000", 2024-09-10T18:30:20.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.94332941599999998 2024-09-10T18:30:20.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:20.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.131435+0000", 2024-09-10T18:30:20.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6177e-05 2024-09-10T18:30:20.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:30:20.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.131495+0000", 2024-09-10T18:30:20.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0163e-05 2024-09-10T18:30:20.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:20.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133809+0000", 2024-09-10T18:30:20.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023145079999999998 2024-09-10T18:30:20.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133844+0000", 2024-09-10T18:30:20.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4686e-05 2024-09-10T18:30:20.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136445+0000", 2024-09-10T18:30:20.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002600799 2024-09-10T18:30:20.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:20.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136472+0000", 2024-09-10T18:30:20.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7078999999999999e-05 2024-09-10T18:30:20.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:20.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136502+0000", 2024-09-10T18:30:20.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0068000000000001e-05 2024-09-10T18:30:20.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.932 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:20.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.932 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221505 7.b 7:d0875639:::benchmark_data_smithi028_77856_object100405:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:30:20.932 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.200926+0000", 2024-09-10T18:30:20.932 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.63692389799999, 2024-09-10T18:30:20.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93569294999999997, 2024-09-10T18:30:20.932 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:20.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:20.932 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:20.932 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:20.933 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:20.933 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221505 2024-09-10T18:30:20.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.933 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:20.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:20.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.200926+0000", 2024-09-10T18:30:20.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:20.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.200926+0000", 2024-09-10T18:30:20.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:20.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.200924+0000", 2024-09-10T18:30:20.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-10T18:30:20.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:20.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.200933+0000", 2024-09-10T18:30:20.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.431e-06 2024-09-10T18:30:20.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:20.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.200983+0000", 2024-09-10T18:30:20.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0013999999999999e-05 2024-09-10T18:30:20.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:20.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.200990+0000", 2024-09-10T18:30:20.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2359999999999998e-06 2024-09-10T18:30:20.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.201059+0000", 2024-09-10T18:30:20.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8227999999999998e-05 2024-09-10T18:30:20.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:30:20.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.201067+0000", 2024-09-10T18:30:20.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9689999999999994e-06 2024-09-10T18:30:20.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.132417+0000", 2024-09-10T18:30:20.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93135020400000001 2024-09-10T18:30:20.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:20.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.132482+0000", 2024-09-10T18:30:20.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4906999999999994e-05 2024-09-10T18:30:20.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:30:20.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.132546+0000", 2024-09-10T18:30:20.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3997000000000004e-05 2024-09-10T18:30:20.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133891+0000", 2024-09-10T18:30:20.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001344837 2024-09-10T18:30:20.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:20.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133898+0000", 2024-09-10T18:30:20.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6809999999999998e-06 2024-09-10T18:30:20.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136546+0000", 2024-09-10T18:30:20.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002647974 2024-09-10T18:30:20.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:20.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136571+0000", 2024-09-10T18:30:20.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4953e-05 2024-09-10T18:30:20.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136619+0000", 2024-09-10T18:30:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8226000000000002e-05 2024-09-10T18:30:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221531 7.b 7:d7e9de4a:::benchmark_data_smithi028_77856_object100431:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:30:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.222649+0000", 2024-09-10T18:30:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.61520093199999, 2024-09-10T18:30:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.91614740400000005, 2024-09-10T18:30:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:20.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:20.940 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:20.940 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:20.940 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:20.940 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221531 2024-09-10T18:30:20.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.940 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:20.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:20.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.222649+0000", 2024-09-10T18:30:20.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:20.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.222649+0000", 2024-09-10T18:30:20.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:20.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.222647+0000", 2024-09-10T18:30:20.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:30:20.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:20.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.222656+0000", 2024-09-10T18:30:20.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6889999999999993e-06 2024-09-10T18:30:20.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:20.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.222672+0000", 2024-09-10T18:30:20.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6549999999999999e-05 2024-09-10T18:30:20.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:20.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.222679+0000", 2024-09-10T18:30:20.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1199999999999999e-06 2024-09-10T18:30:20.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.222720+0000", 2024-09-10T18:30:20.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1464999999999997e-05 2024-09-10T18:30:20.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:30:20.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.222727+0000", 2024-09-10T18:30:20.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.238e-06 2024-09-10T18:30:20.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133925+0000", 2024-09-10T18:30:20.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.91119738800000005 2024-09-10T18:30:20.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:20.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.134024+0000", 2024-09-10T18:30:20.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9692999999999996e-05 2024-09-10T18:30:20.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:30:20.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.134105+0000", 2024-09-10T18:30:20.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1087000000000005e-05 2024-09-10T18:30:20.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:20.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136631+0000", 2024-09-10T18:30:20.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025255 2024-09-10T18:30:20.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136692+0000", 2024-09-10T18:30:20.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0866999999999998e-05 2024-09-10T18:30:20.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138713+0000", 2024-09-10T18:30:20.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020208959999999999 2024-09-10T18:30:20.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:20.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138759+0000", 2024-09-10T18:30:20.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6530999999999999e-05 2024-09-10T18:30:20.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:20.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138797+0000", 2024-09-10T18:30:20.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7595000000000002e-05 2024-09-10T18:30:20.946 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.946 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:20.946 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.946 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221533 7.b 7:d065aacb:::benchmark_data_smithi028_77856_object100433:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:30:20.946 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.225307+0000", 2024-09-10T18:30:20.946 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.61254282900001, 2024-09-10T18:30:20.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.91623593000000003, 2024-09-10T18:30:20.947 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:20.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:20.947 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:20.947 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:20.947 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:20.947 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221533 2024-09-10T18:30:20.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.947 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:20.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:20.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.225307+0000", 2024-09-10T18:30:20.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:20.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.225307+0000", 2024-09-10T18:30:20.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:20.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.225305+0000", 2024-09-10T18:30:20.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:30:20.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:20.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.225313+0000", 2024-09-10T18:30:20.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4300000000000006e-06 2024-09-10T18:30:20.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:20.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.225331+0000", 2024-09-10T18:30:20.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.77e-05 2024-09-10T18:30:20.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:20.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.225363+0000", 2024-09-10T18:30:20.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2378000000000001e-05 2024-09-10T18:30:20.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.225452+0000", 2024-09-10T18:30:20.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8818000000000002e-05 2024-09-10T18:30:20.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:30:20.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.225461+0000", 2024-09-10T18:30:20.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9879999999999993e-06 2024-09-10T18:30:20.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.135296+0000", 2024-09-10T18:30:20.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.90983507799999996 2024-09-10T18:30:20.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:20.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.135380+0000", 2024-09-10T18:30:20.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3567999999999997e-05 2024-09-10T18:30:20.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:30:20.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.135448+0000", 2024-09-10T18:30:20.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8515000000000004e-05 2024-09-10T18:30:20.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:20.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138630+0000", 2024-09-10T18:30:20.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031812450000000001 2024-09-10T18:30:20.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138667+0000", 2024-09-10T18:30:20.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7876000000000001e-05 2024-09-10T18:30:20.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141456+0000", 2024-09-10T18:30:20.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002788727 2024-09-10T18:30:20.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:20.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141505+0000", 2024-09-10T18:30:20.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8600999999999998e-05 2024-09-10T18:30:20.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:20.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141543+0000", 2024-09-10T18:30:20.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8568000000000001e-05 2024-09-10T18:30:20.953 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.953 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:20.953 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.953 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221541 7.b 7:dde66f49:::benchmark_data_smithi028_77856_object100441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:30:20.953 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.229960+0000", 2024-09-10T18:30:20.953 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.607890449, 2024-09-10T18:30:20.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.91172626199999995, 2024-09-10T18:30:20.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:20.954 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:20.954 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:20.954 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:20.954 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:20.954 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221541 2024-09-10T18:30:20.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.954 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:20.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:20.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.229960+0000", 2024-09-10T18:30:20.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:20.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.229960+0000", 2024-09-10T18:30:20.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:20.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.229957+0000", 2024-09-10T18:30:20.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:30:20.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:20.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.229966+0000", 2024-09-10T18:30:20.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9160000000000007e-06 2024-09-10T18:30:20.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:20.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.229981+0000", 2024-09-10T18:30:20.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-10T18:30:20.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:20.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.229988+0000", 2024-09-10T18:30:20.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7669999999999998e-06 2024-09-10T18:30:20.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.230038+0000", 2024-09-10T18:30:20.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0146999999999999e-05 2024-09-10T18:30:20.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:30:20.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.230046+0000", 2024-09-10T18:30:20.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3909999999999998e-06 2024-09-10T18:30:20.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136710+0000", 2024-09-10T18:30:20.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.90666396400000004 2024-09-10T18:30:20.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:20.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136826+0000", 2024-09-10T18:30:20.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011557000000000001 2024-09-10T18:30:20.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:30:20.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136962+0000", 2024-09-10T18:30:20.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000135928 2024-09-10T18:30:20.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138844+0000", 2024-09-10T18:30:20.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001882365 2024-09-10T18:30:20.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:20.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138849+0000", 2024-09-10T18:30:20.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.322e-06 2024-09-10T18:30:20.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141592+0000", 2024-09-10T18:30:20.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002742155 2024-09-10T18:30:20.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:20.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141648+0000", 2024-09-10T18:30:20.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6178000000000003e-05 2024-09-10T18:30:20.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:20.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141686+0000", 2024-09-10T18:30:20.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8398000000000001e-05 2024-09-10T18:30:20.960 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.960 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:20.960 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.960 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221553 7.b 7:d1279096:::benchmark_data_smithi028_77856_object100453:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:30:20.960 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.240485+0000", 2024-09-10T18:30:20.960 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.597365094, 2024-09-10T18:30:20.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.90149367300000005, 2024-09-10T18:30:20.961 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:20.961 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:20.961 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:20.961 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:20.961 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:20.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221553 2024-09-10T18:30:20.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.961 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:20.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:20.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.240485+0000", 2024-09-10T18:30:20.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:20.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.240485+0000", 2024-09-10T18:30:20.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:20.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.240483+0000", 2024-09-10T18:30:20.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:30:20.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:20.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.240490+0000", 2024-09-10T18:30:20.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8529999999999992e-06 2024-09-10T18:30:20.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:20.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.240506+0000", 2024-09-10T18:30:20.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5596000000000001e-05 2024-09-10T18:30:20.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:20.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.240512+0000", 2024-09-10T18:30:20.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8329999999999999e-06 2024-09-10T18:30:20.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.240553+0000", 2024-09-10T18:30:20.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0689999999999998e-05 2024-09-10T18:30:20.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:30:20.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.240560+0000", 2024-09-10T18:30:20.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7859999999999997e-06 2024-09-10T18:30:20.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138881+0000", 2024-09-10T18:30:20.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.89832107500000002 2024-09-10T18:30:20.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:20.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138982+0000", 2024-09-10T18:30:20.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000100426 2024-09-10T18:30:20.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:30:20.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.139085+0000", 2024-09-10T18:30:20.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010320800000000001 2024-09-10T18:30:20.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141736+0000", 2024-09-10T18:30:20.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026505029999999998 2024-09-10T18:30:20.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141781+0000", 2024-09-10T18:30:20.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5590000000000002e-05 2024-09-10T18:30:20.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:20.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141790+0000", 2024-09-10T18:30:20.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1940000000000004e-06 2024-09-10T18:30:20.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:20.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141941+0000", 2024-09-10T18:30:20.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015065799999999999 2024-09-10T18:30:20.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:20.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141979+0000", 2024-09-10T18:30:20.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7880000000000003e-05 2024-09-10T18:30:20.967 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.967 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:20.967 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.967 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221562 7.b 7:de05e396:::benchmark_data_smithi028_77856_object100462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:30:20.967 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.247169+0000", 2024-09-10T18:30:20.967 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.59068109399999, 2024-09-10T18:30:20.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.901609196, 2024-09-10T18:30:20.968 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:20.968 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:20.968 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:20.968 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:20.968 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:20.968 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221562 2024-09-10T18:30:20.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.968 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:20.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:20.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.247169+0000", 2024-09-10T18:30:20.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:20.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.247169+0000", 2024-09-10T18:30:20.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:20.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.247167+0000", 2024-09-10T18:30:20.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:30:20.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:20.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.247175+0000", 2024-09-10T18:30:20.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9789999999999993e-06 2024-09-10T18:30:20.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:20.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.247189+0000", 2024-09-10T18:30:20.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3682999999999999e-05 2024-09-10T18:30:20.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:20.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.247195+0000", 2024-09-10T18:30:20.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2870000000000001e-06 2024-09-10T18:30:20.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.247238+0000", 2024-09-10T18:30:20.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2876000000000001e-05 2024-09-10T18:30:20.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:30:20.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.247270+0000", 2024-09-10T18:30:20.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1900999999999998e-05 2024-09-10T18:30:20.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.142018+0000", 2024-09-10T18:30:20.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.89474827999999995 2024-09-10T18:30:20.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:20.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.142119+0000", 2024-09-10T18:30:20.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101619 2024-09-10T18:30:20.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:30:20.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.142231+0000", 2024-09-10T18:30:20.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011147899999999999 2024-09-10T18:30:20.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148529+0000", 2024-09-10T18:30:20.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006298194 2024-09-10T18:30:20.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148565+0000", 2024-09-10T18:30:20.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6035999999999999e-05 2024-09-10T18:30:20.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:20.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148670+0000", 2024-09-10T18:30:20.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000104812 2024-09-10T18:30:20.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:20.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148722+0000", 2024-09-10T18:30:20.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1947999999999998e-05 2024-09-10T18:30:20.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:20.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148778+0000", 2024-09-10T18:30:20.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6458000000000001e-05 2024-09-10T18:30:20.974 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.974 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:20.974 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.974 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221645 7.b 7:df84e65d:::benchmark_data_smithi028_77856_object100545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:30:20.975 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.365336+0000", 2024-09-10T18:30:20.975 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.472514093, 2024-09-10T18:30:20.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.794312303, 2024-09-10T18:30:20.975 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:20.975 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:20.975 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:20.975 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:20.975 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:20.975 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221645 2024-09-10T18:30:20.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.975 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:20.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:20.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.365336+0000", 2024-09-10T18:30:20.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:20.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.365336+0000", 2024-09-10T18:30:20.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:20.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.365334+0000", 2024-09-10T18:30:20.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-10T18:30:20.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:20.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.365342+0000", 2024-09-10T18:30:20.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1869999999999997e-06 2024-09-10T18:30:20.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:20.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.365360+0000", 2024-09-10T18:30:20.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8111999999999999e-05 2024-09-10T18:30:20.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:20.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.365367+0000", 2024-09-10T18:30:20.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9929999999999998e-06 2024-09-10T18:30:20.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.365420+0000", 2024-09-10T18:30:20.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3321e-05 2024-09-10T18:30:20.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:30:20.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.365432+0000", 2024-09-10T18:30:20.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1884e-05 2024-09-10T18:30:20.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.145920+0000", 2024-09-10T18:30:20.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.78048761700000002 2024-09-10T18:30:20.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:20.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.146028+0000", 2024-09-10T18:30:20.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108578 2024-09-10T18:30:20.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:30:20.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.146140+0000", 2024-09-10T18:30:20.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011149 2024-09-10T18:30:20.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148592+0000", 2024-09-10T18:30:20.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002452475 2024-09-10T18:30:20.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148658+0000", 2024-09-10T18:30:20.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6162999999999997e-05 2024-09-10T18:30:20.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:20.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159494+0000", 2024-09-10T18:30:20.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010835463 2024-09-10T18:30:20.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:20.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159601+0000", 2024-09-10T18:30:20.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000106933 2024-09-10T18:30:20.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:20.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159649+0000", 2024-09-10T18:30:20.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7732999999999999e-05 2024-09-10T18:30:20.981 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.981 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:20.981 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.981 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221656 7.b 7:ddf1fcdc:::benchmark_data_smithi028_77856_object100556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:30:20.981 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.377170+0000", 2024-09-10T18:30:20.981 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.46068036400001, 2024-09-10T18:30:20.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.78274461299999998, 2024-09-10T18:30:20.982 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:20.982 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:20.982 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:20.982 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:20.982 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:20.982 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221656 2024-09-10T18:30:20.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.982 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:20.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:20.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.377170+0000", 2024-09-10T18:30:20.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:20.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.377170+0000", 2024-09-10T18:30:20.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:20.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.377168+0000", 2024-09-10T18:30:20.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:20.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:30:20.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:20.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.377176+0000", 2024-09-10T18:30:20.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7470000000000005e-06 2024-09-10T18:30:20.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:20.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.377192+0000", 2024-09-10T18:30:20.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6311e-05 2024-09-10T18:30:20.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:20.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.377198+0000", 2024-09-10T18:30:20.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4450000000000001e-06 2024-09-10T18:30:20.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.377245+0000", 2024-09-10T18:30:20.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6671000000000001e-05 2024-09-10T18:30:20.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:30:20.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.377253+0000", 2024-09-10T18:30:20.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0220000000000004e-06 2024-09-10T18:30:20.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148820+0000", 2024-09-10T18:30:20.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.771566682 2024-09-10T18:30:20.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:20.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148935+0000", 2024-09-10T18:30:20.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011509 2024-09-10T18:30:20.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:30:20.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.149007+0000", 2024-09-10T18:30:20.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2096999999999998e-05 2024-09-10T18:30:20.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.154054+0000", 2024-09-10T18:30:20.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050468689999999998 2024-09-10T18:30:20.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.154087+0000", 2024-09-10T18:30:20.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3636000000000001e-05 2024-09-10T18:30:20.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:20.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159824+0000", 2024-09-10T18:30:20.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0057362389999999997 2024-09-10T18:30:20.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:20.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159865+0000", 2024-09-10T18:30:20.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0917999999999997e-05 2024-09-10T18:30:20.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:20.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159915+0000", 2024-09-10T18:30:20.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9990000000000001e-05 2024-09-10T18:30:20.990 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.990 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:20.990 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.990 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221677 7.b 7:d8b7af7a:::benchmark_data_smithi028_77856_object100577:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:30:20.990 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.399460+0000", 2024-09-10T18:30:20.990 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.43838982, 2024-09-10T18:30:20.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.76054054100000001, 2024-09-10T18:30:20.990 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:20.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:20.990 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:20.991 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:20.991 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:20.991 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221677 2024-09-10T18:30:20.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.991 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:20.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:20.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.399460+0000", 2024-09-10T18:30:20.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:20.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.399460+0000", 2024-09-10T18:30:20.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:20.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.399458+0000", 2024-09-10T18:30:20.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:30:20.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:20.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.399466+0000", 2024-09-10T18:30:20.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2910000000000007e-06 2024-09-10T18:30:20.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:20.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.399485+0000", 2024-09-10T18:30:20.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8600000000000001e-05 2024-09-10T18:30:20.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:20.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.399493+0000", 2024-09-10T18:30:20.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7389999999999999e-06 2024-09-10T18:30:20.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.399543+0000", 2024-09-10T18:30:20.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0809000000000001e-05 2024-09-10T18:30:20.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:30:20.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.399553+0000", 2024-09-10T18:30:20.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2609999999999999e-06 2024-09-10T18:30:20.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:20.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.151399+0000", 2024-09-10T18:30:20.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.75184643399999995 2024-09-10T18:30:20.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:20.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.151518+0000", 2024-09-10T18:30:20.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000118336 2024-09-10T18:30:20.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:30:20.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.151624+0000", 2024-09-10T18:30:20.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000106786 2024-09-10T18:30:20.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159486+0000", 2024-09-10T18:30:20.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078619840000000007 2024-09-10T18:30:20.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:20.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159751+0000", 2024-09-10T18:30:20.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026467899999999998 2024-09-10T18:30:20.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:20.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159923+0000", 2024-09-10T18:30:20.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000172423 2024-09-10T18:30:20.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:20.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159966+0000", 2024-09-10T18:30:20.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2459999999999997e-05 2024-09-10T18:30:20.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:20.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.160001+0000", 2024-09-10T18:30:20.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5179999999999999e-05 2024-09-10T18:30:20.997 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.997 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:20.997 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:20.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.997 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221686 7.b 7:dfcedead:::benchmark_data_smithi028_77856_object100586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:30:20.997 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.407556+0000", 2024-09-10T18:30:20.997 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.430294532, 2024-09-10T18:30:20.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.75659945500000003, 2024-09-10T18:30:20.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:20.998 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:20.998 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:20.998 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:20.998 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:20.998 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221686 2024-09-10T18:30:20.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.998 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:20.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:20.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.407556+0000", 2024-09-10T18:30:20.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:20.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.407556+0000", 2024-09-10T18:30:20.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:20.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:20.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.407553+0000", 2024-09-10T18:30:20.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-10T18:30:20.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:20.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:20.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:20.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.407563+0000", 2024-09-10T18:30:20.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7589999999999992e-06 2024-09-10T18:30:20.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:21.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.407583+0000", 2024-09-10T18:30:21.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0536000000000001e-05 2024-09-10T18:30:21.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:21.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.407591+0000", 2024-09-10T18:30:21.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6149999999999999e-06 2024-09-10T18:30:21.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:21.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.407638+0000", 2024-09-10T18:30:21.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7107000000000001e-05 2024-09-10T18:30:21.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:30:21.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.407646+0000", 2024-09-10T18:30:21.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3059999999999998e-06 2024-09-10T18:30:21.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:21.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.154110+0000", 2024-09-10T18:30:21.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.74646322600000004 2024-09-10T18:30:21.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:21.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.154214+0000", 2024-09-10T18:30:21.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000104417 2024-09-10T18:30:21.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:30:21.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.154330+0000", 2024-09-10T18:30:21.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116496 2024-09-10T18:30:21.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:21.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159813+0000", 2024-09-10T18:30:21.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054826550000000003 2024-09-10T18:30:21.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:21.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.163716+0000", 2024-09-10T18:30:21.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039024319999999999 2024-09-10T18:30:21.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:21.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.163847+0000", 2024-09-10T18:30:21.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013136499999999999 2024-09-10T18:30:21.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:21.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.164128+0000", 2024-09-10T18:30:21.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028095600000000001 2024-09-10T18:30:21.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:21.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.164155+0000", 2024-09-10T18:30:21.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7330000000000001e-05 2024-09-10T18:30:21.004 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:21.004 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:21.005 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:21.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.005 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221693 7.b 7:dbf8bbe4:::benchmark_data_smithi028_77856_object100593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:30:21.005 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.414908+0000", 2024-09-10T18:30:21.005 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.42294270799999, 2024-09-10T18:30:21.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.74933771900000001, 2024-09-10T18:30:21.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:21.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:21.005 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:21.005 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:21.005 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:21.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221693 2024-09-10T18:30:21.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.006 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:21.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:21.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.414908+0000", 2024-09-10T18:30:21.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:21.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:21.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.414908+0000", 2024-09-10T18:30:21.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:21.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:21.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.414906+0000", 2024-09-10T18:30:21.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:30:21.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:21.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.414912+0000", 2024-09-10T18:30:21.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7309999999999996e-06 2024-09-10T18:30:21.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:21.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.414924+0000", 2024-09-10T18:30:21.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1374e-05 2024-09-10T18:30:21.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:21.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.414930+0000", 2024-09-10T18:30:21.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3890000000000002e-06 2024-09-10T18:30:21.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:21.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.414977+0000", 2024-09-10T18:30:21.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7419000000000001e-05 2024-09-10T18:30:21.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:30:21.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.414988+0000", 2024-09-10T18:30:21.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.029e-05 2024-09-10T18:30:21.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:21.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.156717+0000", 2024-09-10T18:30:21.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.74172964100000005 2024-09-10T18:30:21.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:21.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.156925+0000", 2024-09-10T18:30:21.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020772799999999999 2024-09-10T18:30:21.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:30:21.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.157029+0000", 2024-09-10T18:30:21.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103702 2024-09-10T18:30:21.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:21.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.163753+0000", 2024-09-10T18:30:21.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0067238059999999997 2024-09-10T18:30:21.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:21.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.163782+0000", 2024-09-10T18:30:21.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9431e-05 2024-09-10T18:30:21.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:21.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.164163+0000", 2024-09-10T18:30:21.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00038137700000000002 2024-09-10T18:30:21.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:21.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.164190+0000", 2024-09-10T18:30:21.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6729000000000001e-05 2024-09-10T18:30:21.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:21.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.164245+0000", 2024-09-10T18:30:21.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5163e-05 2024-09-10T18:30:21.011 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:21.012 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:21.012 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:21.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.012 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221707 7.b 7:da97deb7:::benchmark_data_smithi028_77856_object100607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:30:21.012 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.432975+0000", 2024-09-10T18:30:21.012 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.404875707, 2024-09-10T18:30:21.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.73418157100000003, 2024-09-10T18:30:21.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:21.012 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:21.012 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:21.012 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:21.012 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:21.013 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221707 2024-09-10T18:30:21.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.013 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:21.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:21.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.432975+0000", 2024-09-10T18:30:21.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:21.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:21.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.432975+0000", 2024-09-10T18:30:21.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:21.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:21.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.432972+0000", 2024-09-10T18:30:21.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:30:21.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:21.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.432981+0000", 2024-09-10T18:30:21.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3639999999999999e-06 2024-09-10T18:30:21.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:21.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.432995+0000", 2024-09-10T18:30:21.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4268e-05 2024-09-10T18:30:21.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:21.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.433002+0000", 2024-09-10T18:30:21.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4470000000000003e-06 2024-09-10T18:30:21.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:21.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.433050+0000", 2024-09-10T18:30:21.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8217000000000003e-05 2024-09-10T18:30:21.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:30:21.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.433060+0000", 2024-09-10T18:30:21.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0121e-05 2024-09-10T18:30:21.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:21.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.160035+0000", 2024-09-10T18:30:21.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.72697528600000005 2024-09-10T18:30:21.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:21.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.160158+0000", 2024-09-10T18:30:21.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012259700000000001 2024-09-10T18:30:21.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:30:21.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.160275+0000", 2024-09-10T18:30:21.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011747999999999999 2024-09-10T18:30:21.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:21.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.163809+0000", 2024-09-10T18:30:21.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035336759999999999 2024-09-10T18:30:21.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:21.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.163840+0000", 2024-09-10T18:30:21.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1059000000000002e-05 2024-09-10T18:30:21.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:21.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.166920+0000", 2024-09-10T18:30:21.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030802260000000001 2024-09-10T18:30:21.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:21.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.167081+0000", 2024-09-10T18:30:21.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016071 2024-09-10T18:30:21.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:21.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.167156+0000", 2024-09-10T18:30:21.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5258000000000003e-05 2024-09-10T18:30:21.019 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:21.019 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:21.019 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:21.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.019 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221709 7.b 7:d48ef2db:::benchmark_data_smithi028_77856_object100609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:30:21.019 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.434394+0000", 2024-09-10T18:30:21.019 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.40345632399999, 2024-09-10T18:30:21.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.73600392699999995, 2024-09-10T18:30:21.019 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:21.019 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:21.019 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:21.019 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:21.020 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:21.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221709 2024-09-10T18:30:21.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.020 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:21.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:21.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.434394+0000", 2024-09-10T18:30:21.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:21.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:21.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.434394+0000", 2024-09-10T18:30:21.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:21.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:21.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.434392+0000", 2024-09-10T18:30:21.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:30:21.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:21.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.434399+0000", 2024-09-10T18:30:21.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.314e-06 2024-09-10T18:30:21.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:21.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.434411+0000", 2024-09-10T18:30:21.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2057e-05 2024-09-10T18:30:21.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:21.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.434418+0000", 2024-09-10T18:30:21.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6379999999999998e-06 2024-09-10T18:30:21.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:21.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.434460+0000", 2024-09-10T18:30:21.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2648000000000002e-05 2024-09-10T18:30:21.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:30:21.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.434468+0000", 2024-09-10T18:30:21.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9419999999999995e-06 2024-09-10T18:30:21.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:21.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.164279+0000", 2024-09-10T18:30:21.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.72981096099999998 2024-09-10T18:30:21.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:21.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.164389+0000", 2024-09-10T18:30:21.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109459 2024-09-10T18:30:21.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:30:21.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.164481+0000", 2024-09-10T18:30:21.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2584999999999995e-05 2024-09-10T18:30:21.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:21.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.167166+0000", 2024-09-10T18:30:21.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026848509999999998 2024-09-10T18:30:21.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:21.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.169979+0000", 2024-09-10T18:30:21.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028130910000000002 2024-09-10T18:30:21.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:21.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.170269+0000", 2024-09-10T18:30:21.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029024600000000003 2024-09-10T18:30:21.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:21.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.170361+0000", 2024-09-10T18:30:21.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1576000000000005e-05 2024-09-10T18:30:21.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:21.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.170398+0000", 2024-09-10T18:30:21.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7029000000000001e-05 2024-09-10T18:30:21.026 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:21.026 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:21.026 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:21.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.027 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221712 7.b 7:d52c250d:::benchmark_data_smithi028_77856_object100612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:30:21.027 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.444124+0000", 2024-09-10T18:30:21.027 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.39372591399999, 2024-09-10T18:30:21.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.72799156099999995, 2024-09-10T18:30:21.027 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:21.027 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:21.027 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:30:21.027 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:30:21.027 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:30:21.027 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221712 2024-09-10T18:30:21.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.027 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:21.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:21.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.444124+0000", 2024-09-10T18:30:21.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:21.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:21.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.444124+0000", 2024-09-10T18:30:21.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:21.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:21.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.444123+0000", 2024-09-10T18:30:21.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:30:21.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:21.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.444129+0000", 2024-09-10T18:30:21.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2319999999999998e-06 2024-09-10T18:30:21.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:21.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.444139+0000", 2024-09-10T18:30:21.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0727000000000001e-05 2024-09-10T18:30:21.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:21.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.444145+0000", 2024-09-10T18:30:21.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3909999999999997e-06 2024-09-10T18:30:21.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:21.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.444183+0000", 2024-09-10T18:30:21.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7982999999999999e-05 2024-09-10T18:30:21.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:30:21.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.444189+0000", 2024-09-10T18:30:21.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0669999999999997e-06 2024-09-10T18:30:21.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:21.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.167274+0000", 2024-09-10T18:30:21.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.72308553099999995 2024-09-10T18:30:21.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:21.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.167374+0000", 2024-09-10T18:30:21.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0001 2024-09-10T18:30:21.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:30:21.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.167476+0000", 2024-09-10T18:30:21.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101915 2024-09-10T18:30:21.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:21.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.170438+0000", 2024-09-10T18:30:21.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029619999999999998 2024-09-10T18:30:21.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:30:21.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.170466+0000", 2024-09-10T18:30:21.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7421000000000001e-05 2024-09-10T18:30:21.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:30:21.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.171990+0000", 2024-09-10T18:30:21.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001524262 2024-09-10T18:30:21.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:21.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.172078+0000", 2024-09-10T18:30:21.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8418999999999995e-05 2024-09-10T18:30:21.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:21.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.172116+0000", 2024-09-10T18:30:21.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7475999999999998e-05 2024-09-10T18:30:21.033 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:21.034 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:21.034 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:21.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.034 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15117.0:145951 8.b e214/180 8:d5a4ad28:::benchmark_data_smithi028_109242_object11730:head v 214'9017, mlcod=214'9014)", 2024-09-10T18:30:21.034 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:29:46.822624+0000", 2024-09-10T18:30:21.034 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 34.015226218000002, 2024-09-10T18:30:21.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.119049102, 2024-09-10T18:30:21.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:30:21.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:30:21.034 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:30:21.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:30:21.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.822624+0000", 2024-09-10T18:30:21.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:21.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:30:21.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.822624+0000", 2024-09-10T18:30:21.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:30:21.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:30:21.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.822623+0000", 2024-09-10T18:30:21.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:30:21.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:30:21.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.822631+0000", 2024-09-10T18:30:21.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0609999999999996e-06 2024-09-10T18:30:21.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:30:21.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.822680+0000", 2024-09-10T18:30:21.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9212000000000002e-05 2024-09-10T18:30:21.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:30:21.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.822687+0000", 2024-09-10T18:30:21.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.883e-06 2024-09-10T18:30:21.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:30:21.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.822779+0000", 2024-09-10T18:30:21.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1821000000000001e-05 2024-09-10T18:30:21.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:30:21.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.822799+0000", 2024-09-10T18:30:21.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0562000000000001e-05 2024-09-10T18:30:21.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:30:21.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.941642+0000", 2024-09-10T18:30:21.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.11884307400000001 2024-09-10T18:30:21.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:30:21.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:30:21.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:30:21.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.941673+0000", 2024-09-10T18:30:21.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0760999999999999e-05 2024-09-10T18:30:21.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:21.038 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:21.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:21.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:30:21.038 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:30:21.038 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:30:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.2 to 1 (10000) 2024-09-10T18:30:21.082 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:21 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/74603755' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:30:21.082 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:21 smithi028 ceph-mon[39220]: from='client.25180 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:30:21.082 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:21 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/74603755' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:30:21.082 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:21 smithi028 ceph-mon[29291]: from='client.25180 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:30:21.089 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:21.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:21.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:21.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:21.225 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:21.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:21.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:21.327 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:21.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:21 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/74603755' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:30:21.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:21 smithi204 ceph-mon[32357]: from='client.25180 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:30:21.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:21.358 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:21.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:21.459 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:21.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:21.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:21.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:21.503 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:30:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:30:21] "GET /metrics HTTP/1.1" 200 224306 "" "Prometheus/2.43.0" 2024-09-10T18:30:21.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:21.631 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:21.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:21.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:21.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:21.765 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:21.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:21.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:21.896 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:21.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:21.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:21.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:22.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:22.030 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:22.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:22.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:22.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:22.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:22.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:22 smithi028 ceph-mon[39220]: pgmap v396: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 13 B/s, 0 objects/s recovering 2024-09-10T18:30:22.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:22 smithi028 ceph-mon[39220]: from='client.25180 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T18:30:22.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:22 smithi028 ceph-mon[39220]: osdmap e223: 8 total, 8 up, 8 in 2024-09-10T18:30:22.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:22 smithi028 ceph-mon[29291]: pgmap v396: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 13 B/s, 0 objects/s recovering 2024-09-10T18:30:22.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:22 smithi028 ceph-mon[29291]: from='client.25180 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T18:30:22.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:22 smithi028 ceph-mon[29291]: osdmap e223: 8 total, 8 up, 8 in 2024-09-10T18:30:22.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:22.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:22.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:22 smithi204 ceph-mon[32357]: pgmap v396: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 13 B/s, 0 objects/s recovering 2024-09-10T18:30:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:22 smithi204 ceph-mon[32357]: from='client.25180 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T18:30:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:22 smithi204 ceph-mon[32357]: osdmap e223: 8 total, 8 up, 8 in 2024-09-10T18:30:22.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:30:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:30:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:30:22.405 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:22.431 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:22.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:22.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:22.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:22.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:22.569 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:22.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:22.670 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:22.702 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:22.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:22.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:22.803 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:22.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:22.838 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:22.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:22.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-10T18:30:22.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:22.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:22.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:22.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:23.087 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:23.117 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:23.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:23.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:23.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:23 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:30:23.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:23 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:30:23.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:23 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:30:23.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:23 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:30:23.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:23 smithi028 ceph-mon[29291]: osdmap e224: 8 total, 8 up, 8 in 2024-09-10T18:30:23.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:23 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:30:23.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:23 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:30:23.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:23 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:30:23.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:23 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:30:23.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:23 smithi028 ceph-mon[39220]: osdmap e224: 8 total, 8 up, 8 in 2024-09-10T18:30:23.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:23.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:23.257 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:23.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:23 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:30:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:23 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:30:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:23 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:30:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:23 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:30:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:23 smithi204 ceph-mon[32357]: osdmap e224: 8 total, 8 up, 8 in 2024-09-10T18:30:23.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:23.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-10T18:30:23.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:23.426 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:23.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:23.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:23.552 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:23.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:23.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:23.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:23.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:23.706 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:23.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:23.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:23.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:23.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:23.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:23.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:23.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:24.074 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:24.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:24.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:24.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:24.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:24.231 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:24.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:24.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:24.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:24.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:24 smithi204 ceph-mon[32357]: pgmap v399: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 921 op/s 2024-09-10T18:30:24.334 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:30:24.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:24.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:24.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:24 smithi028 ceph-mon[29291]: pgmap v399: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 921 op/s 2024-09-10T18:30:24.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:30:24.391 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:24 smithi028 ceph-mon[39220]: pgmap v399: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 921 op/s 2024-09-10T18:30:24.391 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:30:24.400 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:30:24.465 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:30:24.465 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:30:24.486 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:24.494 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:30:24.494 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:30:24.512 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:24.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:24.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:24.517 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:30:24.517 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:30:24.548 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:30:24.548 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:30:24.581 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:30:24.581 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:30:24.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:30:24 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:30:24.611 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:30:24.611 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:30:24.614 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:24.637 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:30:24.638 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:30:24.643 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:24.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:24.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:24.666 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:30:24.666 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:30:24.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:24.777 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:24.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:24.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:24.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:24.917 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:24.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:24.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:25.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:25.045 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:25.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:25.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:25.147 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:25.173 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:25.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:25.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:25.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:25.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:25.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:25 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1430663383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:30:25.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:25.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:25.446 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:25.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:25.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:25 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1430663383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:30:25.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:25 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1430663383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:30:25.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:25.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:25.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:25.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:25.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:25.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:25.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:25.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:25.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:25.998 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:25.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:26.098 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:30:26.100 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:26.126 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:26.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:26.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:26.228 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:26.258 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:26.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:26 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:26 smithi204 ceph-mon[32357]: pgmap v400: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 645 op/s 2024-09-10T18:30:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:26 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3571391469' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:30:26.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:26.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:26.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:26.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:26.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:26 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:26.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:26 smithi028 ceph-mon[29291]: pgmap v400: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 645 op/s 2024-09-10T18:30:26.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:26 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3571391469' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:30:26.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:26 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:26.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:26 smithi028 ceph-mon[39220]: pgmap v400: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 645 op/s 2024-09-10T18:30:26.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:26 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3571391469' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:30:26.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:26.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:26.650 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:26.683 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:26.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:26.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:26.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:26.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:26.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:26.931 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:26.960 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:26.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:26.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:27.062 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:27.093 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:27.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:27.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:27.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:27 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:30:27.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:27 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3571391469' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:30:27.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:27 smithi028 ceph-mon[29291]: osdmap e225: 8 total, 8 up, 8 in 2024-09-10T18:30:27.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:27 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:30:27.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:27 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3571391469' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:30:27.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:27 smithi028 ceph-mon[39220]: osdmap e225: 8 total, 8 up, 8 in 2024-09-10T18:30:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:30:27.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:27.243 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:27.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:27.344 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:27 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:30:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:27 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3571391469' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:30:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:27 smithi204 ceph-mon[32357]: osdmap e225: 8 total, 8 up, 8 in 2024-09-10T18:30:27.376 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:27.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:27.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:27.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:27.524 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:27.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:27.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:27.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:27.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:27.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:27.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:27.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:27.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:27.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:27.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:27.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:30:27.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:27.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:27.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:28.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:28.115 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:28.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:28.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:28 smithi028 ceph-mon[29291]: pgmap v402: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 704 op/s 2024-09-10T18:30:28.181 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:28 smithi028 ceph-mon[39220]: pgmap v402: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 704 op/s 2024-09-10T18:30:28.216 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:28.243 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:28.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:28.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:28.345 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:28.374 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:28.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:28.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:28.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:28.516 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:28.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:28 smithi204 ceph-mon[32357]: pgmap v402: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 704 op/s 2024-09-10T18:30:28.616 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:28.648 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:28.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:28.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:28.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:28.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:28.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:28.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:28.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:28.935 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:28.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:28.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:29.037 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:29.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:29.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:29.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:29.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:29.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:30:29.215 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:29.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:29.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:29.317 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:29.345 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:29.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:29.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:29.447 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:29.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:29.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:29.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:29.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:29.600 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:29.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:29.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:29.702 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:29.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:29.731 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:29.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:29.832 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:29.862 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:29.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:29.964 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:29.996 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:29.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:29.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:30.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:30.139 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:30.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:30.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:30.242 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:30.275 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:30.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:30.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:30.377 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:30.396 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:30 smithi028 ceph-mon[39220]: pgmap v403: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 649 B/s rd, 0 B/s wr, 536 op/s 2024-09-10T18:30:30.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:30 smithi028 ceph-mon[29291]: pgmap v403: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 649 B/s rd, 0 B/s wr, 536 op/s 2024-09-10T18:30:30.405 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:30.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:30.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:30.507 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:30.537 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:30.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:30 smithi204 ceph-mon[32357]: pgmap v403: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 649 B/s rd, 0 B/s wr, 536 op/s 2024-09-10T18:30:30.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:30.678 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:30.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:30.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:30.780 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:30.805 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:30.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:30.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:30.907 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:30.935 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:30.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:30.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:31.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:31.040 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:30:31.040 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:30:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:31.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:31.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:31.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:31.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:30:31.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:31.384 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:31.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:31.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:31.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:31.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:31.535 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:31.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:31.635 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:31.669 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:31.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:31.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:31.709 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:30:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:30:31] "GET /metrics HTTP/1.1" 200 224354 "" "Prometheus/2.43.0" 2024-09-10T18:30:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.3 to 1 (10000) 2024-09-10T18:30:31.771 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:31.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:31.799 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:31.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:31.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-10T18:30:31.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:31.966 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:31.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:31.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:32.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:31 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3388904136' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:30:32.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:31 smithi028 ceph-mon[39220]: pgmap v404: 158 pgs: 158 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-10T18:30:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:31 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3388904136' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:30:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:31 smithi028 ceph-mon[29291]: pgmap v404: 158 pgs: 158 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-10T18:30:32.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:32.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:31 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3388904136' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:30:32.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:31 smithi204 ceph-mon[32357]: pgmap v404: 158 pgs: 158 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-10T18:30:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:32.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:32.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:32.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:32.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:32.255 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:32.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:32.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:32.395 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:32.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:32.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:32.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:32.527 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:32.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:32.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:32.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:30:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:30:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:30:32.629 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:32.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:32.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:32.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:32.761 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:32.787 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:32.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:32.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:32.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:32.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:32.998 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:32 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3388904136' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T18:30:32.998 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:32 smithi028 ceph-mon[39220]: osdmap e226: 8 total, 8 up, 8 in 2024-09-10T18:30:32.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:32 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3388904136' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T18:30:32.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:32 smithi028 ceph-mon[29291]: osdmap e226: 8 total, 8 up, 8 in 2024-09-10T18:30:33.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:33.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-10T18:30:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:33.104 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:33.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:33.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:32 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3388904136' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T18:30:33.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:32 smithi204 ceph-mon[32357]: osdmap e226: 8 total, 8 up, 8 in 2024-09-10T18:30:33.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:33.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:33.260 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:33.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:33.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:33.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:33.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:33.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:33.510 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:33.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:33.540 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:33.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:33.642 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:33.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:33.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:33.782 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:33.787 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:30:33.787 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:30:33.808 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:33.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:33.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:33.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:33.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:33 smithi028 ceph-mon[29291]: pgmap v406: 158 pgs: 158 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T18:30:33.931 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:33 smithi028 ceph-mon[39220]: pgmap v406: 158 pgs: 158 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T18:30:33.953 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:33.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:33.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:34.056 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:34.088 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:34.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:34.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:34.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:33 smithi204 ceph-mon[32357]: pgmap v406: 158 pgs: 158 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T18:30:34.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:34.236 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:34.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:34.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:34.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:34.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:34.371 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:34.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:34.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:34.510 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:34.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:34.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:30:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:30:34.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:34.645 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:34.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:34.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:34.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:34.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:34.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:34.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:34.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:34.924 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:34.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:35.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:35.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:35.067 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:35.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:35.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:35.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:35.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:35.309 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:35.343 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:35.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:35.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:35.445 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:35.483 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:35.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:35.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:35.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:35.621 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:35.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:35.724 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:35.752 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:35.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:35.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:35.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:35.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:35.889 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:35.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:35.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:36.019 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:36.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:36.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:36.121 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:36.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:36.152 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:36.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:36.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:36.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:35 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:36.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:35 smithi028 ceph-mon[39220]: pgmap v407: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 861 op/s 2024-09-10T18:30:36.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:35 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:36.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:35 smithi028 ceph-mon[29291]: pgmap v407: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 861 op/s 2024-09-10T18:30:36.292 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:36.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:35 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:35 smithi204 ceph-mon[32357]: pgmap v407: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 861 op/s 2024-09-10T18:30:36.394 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:36.427 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:36.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:36.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:36.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:36.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:36.569 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:36.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:36.670 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:36.695 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:36.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:36.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:36.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:36.832 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:36.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:36.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:36.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:36.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:36.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:37.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:37.102 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:37.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:37.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:37.203 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:37.235 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:37.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:37.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:37.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:37.374 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:37.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:37.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:37.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:37.503 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:37.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:37.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:37.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:37.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:37.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:37.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:37.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:37.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:37.911 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:37.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:37.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:38.013 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:38.051 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:38.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:38.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:38.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:38.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:38.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:38.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:37 smithi028 ceph-mon[39220]: pgmap v408: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-10T18:30:38.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:37 smithi028 ceph-mon[29291]: pgmap v408: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-10T18:30:38.293 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:38.323 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:38.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:38.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:37 smithi204 ceph-mon[32357]: pgmap v408: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-10T18:30:38.426 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:38.451 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:38.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:38.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:38.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:38.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:38.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:38.690 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:38.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:38.713 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:38.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:38.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:38.853 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:38.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:38.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:38.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:38.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:38.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:39.090 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:39.123 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:39.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:39.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:39.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:39.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:38 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:30:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:38 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:30:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:39.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:38 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:30:39.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:39.385 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:39.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:39.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:39.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:39.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:39.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:39.629 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:39.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:39.663 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:39.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:39.764 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:39.788 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:39.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:39.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:39.890 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:39.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:39.925 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:39.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:40.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:40.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:40.056 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:40.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:40.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:40.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:40.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:40.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:40.339 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:40.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:40.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:40 smithi204 ceph-mon[32357]: pgmap v409: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-10T18:30:40.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:40.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:40.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:40 smithi028 ceph-mon[39220]: pgmap v409: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-10T18:30:40.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:40 smithi028 ceph-mon[29291]: pgmap v409: 158 pgs: 158 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-10T18:30:40.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:40.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:40.611 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:40.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:40.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:40.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:40.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:40.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:40.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:40.889 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:40.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:40.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:40.990 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:41.011 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:41.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:41.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:41.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:41.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:41.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:41.245 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:41.275 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:41.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:41.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:41.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:41.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:41.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:41.516 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:41.548 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:41.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:41.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:41.650 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:41.684 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:41.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:41.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:41.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:30:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:30:41] "GET /metrics HTTP/1.1" 200 224336 "" "Prometheus/2.43.0" 2024-09-10T18:30:41.786 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:41.824 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:41.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:41.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:41.926 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:41.956 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:41.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:42.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:42.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:42.080 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:42.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:42.181 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:42.218 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:42.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:42.319 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:42.344 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:42.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:42.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:42.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:42 smithi028 ceph-mon[29291]: pgmap v410: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-09-10T18:30:42.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:42 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:30:42.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:42 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:30:42.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:42 smithi028 ceph-mon[39220]: pgmap v410: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-09-10T18:30:42.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:42 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:30:42.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:42 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:30:42.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:30:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:30:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:30:42.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:42 smithi204 ceph-mon[32357]: pgmap v410: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-09-10T18:30:42.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:42 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:30:42.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:42 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:30:42.445 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:42.477 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:42.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:42.579 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:42.604 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:42.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:42.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:42.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:42.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:42.858 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:42.889 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:42.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:42.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:43.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:43.024 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:43.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:43.125 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:43.152 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:43.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:43.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:43.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:43.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:43 smithi028 ceph-mon[29291]: osdmap e227: 8 total, 8 up, 8 in 2024-09-10T18:30:43.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:43 smithi028 ceph-mon[39220]: osdmap e227: 8 total, 8 up, 8 in 2024-09-10T18:30:43.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:43.305 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:43.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:43 smithi204 ceph-mon[32357]: osdmap e227: 8 total, 8 up, 8 in 2024-09-10T18:30:43.406 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:43.433 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:43.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:43.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:43.535 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:43.733 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:30:43.734 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:30:43.734 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:30:43.734 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:30:43.734 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:30:43.734 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:30:43.734 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:30:43.734 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:30:43.734 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:30:43.734 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:30:43.737 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:43.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:43.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:43.762 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:30:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:30:43.792 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:30:43.793 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:30:43.824 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:30:43.824 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:30:43.839 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:43.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:43.867 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:43.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:43.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:44.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:44.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:44.048 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:30:44.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:44.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:44.155 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:44.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:44.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:44.324 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:44.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:44.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:44 smithi204 ceph-mon[32357]: pgmap v412: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-09-10T18:30:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:44 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/616609220' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:30:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:44 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1689363607' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:30:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:44 smithi204 ceph-mon[32357]: from='client.25198 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:30:44.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:44.463 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:44.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:44.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:44 smithi028 ceph-mon[29291]: pgmap v412: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-09-10T18:30:44.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:44 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/616609220' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:30:44.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:44 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1689363607' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:30:44.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:44 smithi028 ceph-mon[29291]: from='client.25198 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:30:44.471 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:44 smithi028 ceph-mon[39220]: pgmap v412: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-09-10T18:30:44.471 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:44 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/616609220' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:30:44.471 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:44 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1689363607' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:30:44.471 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:44 smithi028 ceph-mon[39220]: from='client.25198 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:30:44.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:44.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:44.596 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:44.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:44.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:44.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:44.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:44.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:30:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:30:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:44.890 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:44.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:44.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:45.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:45.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:45.173 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:45.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:45.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:45.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:45 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:30:45.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:45 smithi028 ceph-mon[29291]: from='client.25198 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:30:45.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:45 smithi028 ceph-mon[29291]: osdmap e228: 8 total, 8 up, 8 in 2024-09-10T18:30:45.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:45 smithi028 ceph-mon[29291]: 8.b scrub starts 2024-09-10T18:30:45.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:45 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:30:45.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:45 smithi028 ceph-mon[39220]: from='client.25198 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:30:45.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:45 smithi028 ceph-mon[39220]: osdmap e228: 8 total, 8 up, 8 in 2024-09-10T18:30:45.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:45 smithi028 ceph-mon[39220]: 8.b scrub starts 2024-09-10T18:30:45.275 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:45.307 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:45.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:45.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:45 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:30:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:45 smithi204 ceph-mon[32357]: from='client.25198 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:30:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:45 smithi204 ceph-mon[32357]: osdmap e228: 8 total, 8 up, 8 in 2024-09-10T18:30:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:45 smithi204 ceph-mon[32357]: 8.b scrub starts 2024-09-10T18:30:45.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:45.444 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:45.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:45.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:45.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:45.566 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:30:45.566 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-10T18:30:45.566 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-10T18:30:45.566 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-10T18:30:45.567 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:30:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:45.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:45.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:45.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:45.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:45.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:45.881 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:45.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:45.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:45.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:46.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:46.014 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:46.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:46.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:46.150 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:46.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:46.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:46.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[29291]: 8.2 scrub starts 2024-09-10T18:30:46.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:46.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[29291]: pgmap v414: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 982 op/s 2024-09-10T18:30:46.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[29291]: 2.1 scrub starts 2024-09-10T18:30:46.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[29291]: 2.1 scrub ok 2024-09-10T18:30:46.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:30:46.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:30:46.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[29291]: 4.10 scrub starts 2024-09-10T18:30:46.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[29291]: 4.10 scrub ok 2024-09-10T18:30:46.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[39220]: 8.2 scrub starts 2024-09-10T18:30:46.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:46.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[39220]: pgmap v414: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 982 op/s 2024-09-10T18:30:46.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[39220]: 2.1 scrub starts 2024-09-10T18:30:46.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[39220]: 2.1 scrub ok 2024-09-10T18:30:46.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:30:46.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:30:46.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[39220]: 4.10 scrub starts 2024-09-10T18:30:46.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:46 smithi028 ceph-mon[39220]: 4.10 scrub ok 2024-09-10T18:30:46.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-10T18:30:46.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:46.313 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:46.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:46.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:46 smithi204 ceph-mon[32357]: 8.2 scrub starts 2024-09-10T18:30:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:46 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:46 smithi204 ceph-mon[32357]: pgmap v414: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 982 op/s 2024-09-10T18:30:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:46 smithi204 ceph-mon[32357]: 2.1 scrub starts 2024-09-10T18:30:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:46 smithi204 ceph-mon[32357]: 2.1 scrub ok 2024-09-10T18:30:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:46 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:30:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:46 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:30:46.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:46 smithi204 ceph-mon[32357]: 4.10 scrub starts 2024-09-10T18:30:46.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:46 smithi204 ceph-mon[32357]: 4.10 scrub ok 2024-09-10T18:30:46.415 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:46.442 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:46.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:46.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:46.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:46.589 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:46.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:46.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:46.691 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:46.735 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:46.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:46.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:46.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:46.895 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:46.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:46.946 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:30:46.970 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:30:46.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:47.051 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:47.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:47.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:47.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[29291]: 1.5 scrub starts 2024-09-10T18:30:47.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[29291]: 1.5 scrub ok 2024-09-10T18:30:47.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[29291]: 3.f scrub starts 2024-09-10T18:30:47.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[29291]: 3.f scrub ok 2024-09-10T18:30:47.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[29291]: 8.2 scrub ok 2024-09-10T18:30:47.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[29291]: osdmap e229: 8 total, 8 up, 8 in 2024-09-10T18:30:47.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[29291]: 4.1c scrub starts 2024-09-10T18:30:47.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[29291]: 4.1c scrub ok 2024-09-10T18:30:47.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[29291]: 8.e scrub starts 2024-09-10T18:30:47.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[29291]: 3.5 scrub starts 2024-09-10T18:30:47.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[29291]: 3.5 scrub ok 2024-09-10T18:30:47.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[29291]: 8.b scrub ok 2024-09-10T18:30:47.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[39220]: 1.5 scrub starts 2024-09-10T18:30:47.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[39220]: 1.5 scrub ok 2024-09-10T18:30:47.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[39220]: 3.f scrub starts 2024-09-10T18:30:47.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[39220]: 3.f scrub ok 2024-09-10T18:30:47.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[39220]: 8.2 scrub ok 2024-09-10T18:30:47.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[39220]: osdmap e229: 8 total, 8 up, 8 in 2024-09-10T18:30:47.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[39220]: 4.1c scrub starts 2024-09-10T18:30:47.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[39220]: 4.1c scrub ok 2024-09-10T18:30:47.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[39220]: 8.e scrub starts 2024-09-10T18:30:47.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[39220]: 3.5 scrub starts 2024-09-10T18:30:47.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[39220]: 3.5 scrub ok 2024-09-10T18:30:47.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:47 smithi028 ceph-mon[39220]: 8.b scrub ok 2024-09-10T18:30:47.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:47.199 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:47.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:47.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:47.301 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:47.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:30:47.328 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:47.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:47.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:47 smithi204 ceph-mon[32357]: 1.5 scrub starts 2024-09-10T18:30:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:47 smithi204 ceph-mon[32357]: 1.5 scrub ok 2024-09-10T18:30:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:47 smithi204 ceph-mon[32357]: 3.f scrub starts 2024-09-10T18:30:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:47 smithi204 ceph-mon[32357]: 3.f scrub ok 2024-09-10T18:30:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:47 smithi204 ceph-mon[32357]: 8.2 scrub ok 2024-09-10T18:30:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:47 smithi204 ceph-mon[32357]: osdmap e229: 8 total, 8 up, 8 in 2024-09-10T18:30:47.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:47 smithi204 ceph-mon[32357]: 4.1c scrub starts 2024-09-10T18:30:47.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:47 smithi204 ceph-mon[32357]: 4.1c scrub ok 2024-09-10T18:30:47.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:47 smithi204 ceph-mon[32357]: 8.e scrub starts 2024-09-10T18:30:47.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:47 smithi204 ceph-mon[32357]: 3.5 scrub starts 2024-09-10T18:30:47.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:47 smithi204 ceph-mon[32357]: 3.5 scrub ok 2024-09-10T18:30:47.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:47 smithi204 ceph-mon[32357]: 8.b scrub ok 2024-09-10T18:30:47.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:47.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:47.500 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:47.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:47.601 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:47.628 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:47.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:47.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:47.731 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:47.758 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:47.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:47.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:47.860 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:47.888 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:47.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:47.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:47.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:48.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:48.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:48.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:48.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:48.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:48.274 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:48.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:48.309 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:48.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:48 smithi204 ceph-mon[32357]: 6.e scrub starts 2024-09-10T18:30:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:48 smithi204 ceph-mon[32357]: 6.e scrub ok 2024-09-10T18:30:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:48 smithi204 ceph-mon[32357]: pgmap v416: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-10T18:30:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:48 smithi204 ceph-mon[32357]: 4.5 scrub starts 2024-09-10T18:30:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:48 smithi204 ceph-mon[32357]: 4.5 scrub ok 2024-09-10T18:30:48.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:48 smithi204 ceph-mon[32357]: 5.7 scrub starts 2024-09-10T18:30:48.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:48 smithi204 ceph-mon[32357]: 5.7 scrub ok 2024-09-10T18:30:48.410 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:48.436 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:48.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:48.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:48.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:48 smithi028 ceph-mon[39220]: 6.e scrub starts 2024-09-10T18:30:48.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:48 smithi028 ceph-mon[39220]: 6.e scrub ok 2024-09-10T18:30:48.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:48 smithi028 ceph-mon[39220]: pgmap v416: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-10T18:30:48.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:48 smithi028 ceph-mon[39220]: 4.5 scrub starts 2024-09-10T18:30:48.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:48 smithi028 ceph-mon[39220]: 4.5 scrub ok 2024-09-10T18:30:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:48 smithi028 ceph-mon[39220]: 5.7 scrub starts 2024-09-10T18:30:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:48 smithi028 ceph-mon[39220]: 5.7 scrub ok 2024-09-10T18:30:48.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:48 smithi028 ceph-mon[29291]: 6.e scrub starts 2024-09-10T18:30:48.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:48 smithi028 ceph-mon[29291]: 6.e scrub ok 2024-09-10T18:30:48.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:48 smithi028 ceph-mon[29291]: pgmap v416: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-10T18:30:48.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:48 smithi028 ceph-mon[29291]: 4.5 scrub starts 2024-09-10T18:30:48.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:48 smithi028 ceph-mon[29291]: 4.5 scrub ok 2024-09-10T18:30:48.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:48 smithi028 ceph-mon[29291]: 5.7 scrub starts 2024-09-10T18:30:48.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:48 smithi028 ceph-mon[29291]: 5.7 scrub ok 2024-09-10T18:30:48.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:48.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:48.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:48.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:48.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:48.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:48.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:48.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:48.880 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:30:48.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:49.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:49.012 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:49.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:49.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:49.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:49.173 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:49.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:49.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:49 smithi028 ceph-mon[39220]: 3.15 scrub starts 2024-09-10T18:30:49.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:49 smithi028 ceph-mon[39220]: 3.15 scrub ok 2024-09-10T18:30:49.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:49 smithi028 ceph-mon[39220]: 5.19 scrub starts 2024-09-10T18:30:49.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:49 smithi028 ceph-mon[39220]: 5.19 scrub ok 2024-09-10T18:30:49.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:49 smithi028 ceph-mon[39220]: 8.e scrub ok 2024-09-10T18:30:49.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:49 smithi028 ceph-mon[39220]: 4.19 scrub starts 2024-09-10T18:30:49.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:49 smithi028 ceph-mon[39220]: 4.19 scrub ok 2024-09-10T18:30:49.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:49 smithi028 ceph-mon[29291]: 3.15 scrub starts 2024-09-10T18:30:49.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:49 smithi028 ceph-mon[29291]: 3.15 scrub ok 2024-09-10T18:30:49.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:49 smithi028 ceph-mon[29291]: 5.19 scrub starts 2024-09-10T18:30:49.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:49 smithi028 ceph-mon[29291]: 5.19 scrub ok 2024-09-10T18:30:49.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:49 smithi028 ceph-mon[29291]: 8.e scrub ok 2024-09-10T18:30:49.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:49 smithi028 ceph-mon[29291]: 4.19 scrub starts 2024-09-10T18:30:49.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:49 smithi028 ceph-mon[29291]: 4.19 scrub ok 2024-09-10T18:30:49.274 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:49.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:49.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:49.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:49 smithi204 ceph-mon[32357]: 3.15 scrub starts 2024-09-10T18:30:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:49 smithi204 ceph-mon[32357]: 3.15 scrub ok 2024-09-10T18:30:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:49 smithi204 ceph-mon[32357]: 5.19 scrub starts 2024-09-10T18:30:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:49 smithi204 ceph-mon[32357]: 5.19 scrub ok 2024-09-10T18:30:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:49 smithi204 ceph-mon[32357]: 8.e scrub ok 2024-09-10T18:30:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:49 smithi204 ceph-mon[32357]: 4.19 scrub starts 2024-09-10T18:30:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:49 smithi204 ceph-mon[32357]: 4.19 scrub ok 2024-09-10T18:30:49.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:49.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:49.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:49.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:49.581 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:49.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:49.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:49.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:49.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:49.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:49.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:49.867 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:49.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:49.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:49.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:50.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:50.004 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:50.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:50.106 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:50.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:50.136 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:50.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:50.237 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:50.262 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:50.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:50.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:50.283 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:30:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:50 smithi204 ceph-mon[32357]: 6.1c scrub starts 2024-09-10T18:30:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:50 smithi204 ceph-mon[32357]: 6.1c scrub ok 2024-09-10T18:30:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:50 smithi204 ceph-mon[32357]: pgmap v417: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 738 B/s rd, 0 B/s wr, 874 op/s 2024-09-10T18:30:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:50 smithi204 ceph-mon[32357]: 6.7 scrub starts 2024-09-10T18:30:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:50 smithi204 ceph-mon[32357]: 6.7 scrub ok 2024-09-10T18:30:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:50 smithi204 ceph-mon[32357]: 4.17 scrub starts 2024-09-10T18:30:50.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:50 smithi204 ceph-mon[32357]: 4.17 scrub ok 2024-09-10T18:30:50.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:50.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:50 smithi028 ceph-mon[39220]: 6.1c scrub starts 2024-09-10T18:30:50.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:50 smithi028 ceph-mon[39220]: 6.1c scrub ok 2024-09-10T18:30:50.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:50 smithi028 ceph-mon[39220]: pgmap v417: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 738 B/s rd, 0 B/s wr, 874 op/s 2024-09-10T18:30:50.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:50 smithi028 ceph-mon[39220]: 6.7 scrub starts 2024-09-10T18:30:50.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:50 smithi028 ceph-mon[39220]: 6.7 scrub ok 2024-09-10T18:30:50.408 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:50 smithi028 ceph-mon[39220]: 4.17 scrub starts 2024-09-10T18:30:50.408 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:50 smithi028 ceph-mon[39220]: 4.17 scrub ok 2024-09-10T18:30:50.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:50 smithi028 ceph-mon[29291]: 6.1c scrub starts 2024-09-10T18:30:50.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:50 smithi028 ceph-mon[29291]: 6.1c scrub ok 2024-09-10T18:30:50.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:50 smithi028 ceph-mon[29291]: pgmap v417: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 738 B/s rd, 0 B/s wr, 874 op/s 2024-09-10T18:30:50.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:50 smithi028 ceph-mon[29291]: 6.7 scrub starts 2024-09-10T18:30:50.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:50 smithi028 ceph-mon[29291]: 6.7 scrub ok 2024-09-10T18:30:50.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:50 smithi028 ceph-mon[29291]: 4.17 scrub starts 2024-09-10T18:30:50.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:50 smithi028 ceph-mon[29291]: 4.17 scrub ok 2024-09-10T18:30:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:50.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:50.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:50.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:50.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:50.668 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:50.696 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:50.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:50.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:50.798 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:50.827 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:50.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:50.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:50.928 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:50.956 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:50.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:50.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:51.058 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:51.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:51.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:51.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:51.200 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:51.225 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:51.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:51.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:51.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:51 smithi028 ceph-mon[29291]: 4.1f scrub starts 2024-09-10T18:30:51.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:51 smithi028 ceph-mon[29291]: 4.1f scrub ok 2024-09-10T18:30:51.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:51 smithi028 ceph-mon[29291]: 3.16 scrub starts 2024-09-10T18:30:51.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:51 smithi028 ceph-mon[29291]: 3.16 scrub ok 2024-09-10T18:30:51.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:51 smithi028 ceph-mon[29291]: 1.4 scrub starts 2024-09-10T18:30:51.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:51 smithi028 ceph-mon[29291]: 1.4 scrub ok 2024-09-10T18:30:51.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:51 smithi028 ceph-mon[39220]: 4.1f scrub starts 2024-09-10T18:30:51.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:51 smithi028 ceph-mon[39220]: 4.1f scrub ok 2024-09-10T18:30:51.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:51 smithi028 ceph-mon[39220]: 3.16 scrub starts 2024-09-10T18:30:51.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:51 smithi028 ceph-mon[39220]: 3.16 scrub ok 2024-09-10T18:30:51.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:51 smithi028 ceph-mon[39220]: 1.4 scrub starts 2024-09-10T18:30:51.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:51 smithi028 ceph-mon[39220]: 1.4 scrub ok 2024-09-10T18:30:51.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:51 smithi204 ceph-mon[32357]: 4.1f scrub starts 2024-09-10T18:30:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:51 smithi204 ceph-mon[32357]: 4.1f scrub ok 2024-09-10T18:30:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:51 smithi204 ceph-mon[32357]: 3.16 scrub starts 2024-09-10T18:30:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:51 smithi204 ceph-mon[32357]: 3.16 scrub ok 2024-09-10T18:30:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:51 smithi204 ceph-mon[32357]: 1.4 scrub starts 2024-09-10T18:30:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:51 smithi204 ceph-mon[32357]: 1.4 scrub ok 2024-09-10T18:30:51.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:51.370 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:51.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:51.471 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:51.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:51.508 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:51.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:51.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:51.659 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:51.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:51.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:30:51.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:30:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:30:51] "GET /metrics HTTP/1.1" 200 224336 "" "Prometheus/2.43.0" 2024-09-10T18:30:51.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:51.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:51.801 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:51.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:51.902 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:51.938 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:51.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:51.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:52.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:52.079 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:52.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:52.110 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.4 to 1 (10000) 2024-09-10T18:30:52.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:52.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:52.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:52.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:52.360 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:52.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:52.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:52.462 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:52.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:52 smithi028 ceph-mon[39220]: pgmap v418: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T18:30:52.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:52 smithi028 ceph-mon[39220]: 8.6 scrub starts 2024-09-10T18:30:52.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:52 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3819401691' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:30:52.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:52 smithi028 ceph-mon[39220]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:30:52.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:52 smithi028 ceph-mon[29291]: pgmap v418: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T18:30:52.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:52 smithi028 ceph-mon[29291]: 8.6 scrub starts 2024-09-10T18:30:52.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:52 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3819401691' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:30:52.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:52 smithi028 ceph-mon[29291]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:30:52.493 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:52.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:52.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:52.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:52.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:30:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:30:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:30:52.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:52 smithi204 ceph-mon[32357]: pgmap v418: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T18:30:52.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:52 smithi204 ceph-mon[32357]: 8.6 scrub starts 2024-09-10T18:30:52.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:52 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3819401691' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:30:52.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:52 smithi204 ceph-mon[32357]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:30:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:52.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:52.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:52.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:52.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:52.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:52.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:52.908 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:52.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:52.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:53.009 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:30:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:30:53.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:53.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:53.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:53.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:53.053 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:30:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:30:53.092 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:30:53.092 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:30:53.121 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:30:53.122 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:30:53.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:53.153 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:30:53.153 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:30:53.185 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:30:53.185 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:30:53.192 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:53.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:53.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:53.221 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:30:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:30:53.294 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:53.306 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[39220]: 1.0 scrub starts 2024-09-10T18:30:53.307 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[39220]: 4.7 scrub starts 2024-09-10T18:30:53.307 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[39220]: 1.0 scrub ok 2024-09-10T18:30:53.307 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[39220]: 4.7 scrub ok 2024-09-10T18:30:53.307 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[39220]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T18:30:53.307 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[39220]: osdmap e230: 8 total, 8 up, 8 in 2024-09-10T18:30:53.307 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[39220]: 4.0 scrub starts 2024-09-10T18:30:53.307 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[39220]: 4.0 scrub ok 2024-09-10T18:30:53.307 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2274123753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:30:53.307 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:30:53.308 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:30:53.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[29291]: 1.0 scrub starts 2024-09-10T18:30:53.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[29291]: 4.7 scrub starts 2024-09-10T18:30:53.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[29291]: 1.0 scrub ok 2024-09-10T18:30:53.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[29291]: 4.7 scrub ok 2024-09-10T18:30:53.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[29291]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T18:30:53.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[29291]: osdmap e230: 8 total, 8 up, 8 in 2024-09-10T18:30:53.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[29291]: 4.0 scrub starts 2024-09-10T18:30:53.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[29291]: 4.0 scrub ok 2024-09-10T18:30:53.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:53 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2274123753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:30:53.323 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:53.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:53.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:53.428 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:53.454 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:53.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:53.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:53.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:53.600 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:53.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:53.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:53 smithi204 ceph-mon[32357]: 1.0 scrub starts 2024-09-10T18:30:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:53 smithi204 ceph-mon[32357]: 4.7 scrub starts 2024-09-10T18:30:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:53 smithi204 ceph-mon[32357]: 1.0 scrub ok 2024-09-10T18:30:53.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:53 smithi204 ceph-mon[32357]: 4.7 scrub ok 2024-09-10T18:30:53.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:53 smithi204 ceph-mon[32357]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T18:30:53.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:53 smithi204 ceph-mon[32357]: osdmap e230: 8 total, 8 up, 8 in 2024-09-10T18:30:53.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:53 smithi204 ceph-mon[32357]: 4.0 scrub starts 2024-09-10T18:30:53.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:53 smithi204 ceph-mon[32357]: 4.0 scrub ok 2024-09-10T18:30:53.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:53 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2274123753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:30:53.703 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:53.736 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:53.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:53.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:53.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:53.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:53.883 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:53.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:53.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:54.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:54.025 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:54.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:54.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:54.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:54.152 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:54.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:54.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:54.292 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:54.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:54.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:54.394 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:54.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:54 smithi028 ceph-mon[39220]: pgmap v420: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T18:30:54.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:54 smithi028 ceph-mon[39220]: 8.6 scrub ok 2024-09-10T18:30:54.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:54 smithi028 ceph-mon[39220]: 5.15 scrub starts 2024-09-10T18:30:54.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:54 smithi028 ceph-mon[39220]: 5.15 scrub ok 2024-09-10T18:30:54.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:54 smithi028 ceph-mon[39220]: 5.c scrub starts 2024-09-10T18:30:54.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:54 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:30:54.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:54 smithi028 ceph-mon[29291]: pgmap v420: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T18:30:54.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:54 smithi028 ceph-mon[29291]: 8.6 scrub ok 2024-09-10T18:30:54.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:54 smithi028 ceph-mon[29291]: 5.15 scrub starts 2024-09-10T18:30:54.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:54 smithi028 ceph-mon[29291]: 5.15 scrub ok 2024-09-10T18:30:54.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:54 smithi028 ceph-mon[29291]: 5.c scrub starts 2024-09-10T18:30:54.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:54 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:30:54.422 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:54.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:54.524 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:54.557 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:54.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:54.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:30:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:30:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:54 smithi204 ceph-mon[32357]: pgmap v420: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T18:30:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:54 smithi204 ceph-mon[32357]: 8.6 scrub ok 2024-09-10T18:30:54.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:54 smithi204 ceph-mon[32357]: 5.15 scrub starts 2024-09-10T18:30:54.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:54 smithi204 ceph-mon[32357]: 5.15 scrub ok 2024-09-10T18:30:54.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:54 smithi204 ceph-mon[32357]: 5.c scrub starts 2024-09-10T18:30:54.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:54 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:30:54.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:54.697 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:54.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:54.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:54.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-10T18:30:54.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:54.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:54.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:54.948 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:54.973 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:54.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:54.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:55.075 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:55.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:55.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:55.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:55.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:55.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:55.251 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:30:55.251 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:30:55.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:55.388 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:55.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:55.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:55.490 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:55.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:55 smithi028 ceph-mon[39220]: 5.c scrub ok 2024-09-10T18:30:55.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:55 smithi028 ceph-mon[39220]: 3.13 scrub starts 2024-09-10T18:30:55.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:55 smithi028 ceph-mon[39220]: 3.13 scrub ok 2024-09-10T18:30:55.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:55 smithi028 ceph-mon[29291]: 5.c scrub ok 2024-09-10T18:30:55.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:55 smithi028 ceph-mon[29291]: 3.13 scrub starts 2024-09-10T18:30:55.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:55 smithi028 ceph-mon[29291]: 3.13 scrub ok 2024-09-10T18:30:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:55.521 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:55.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:55.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:55 smithi204 ceph-mon[32357]: 5.c scrub ok 2024-09-10T18:30:55.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:55 smithi204 ceph-mon[32357]: 3.13 scrub starts 2024-09-10T18:30:55.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:55 smithi204 ceph-mon[32357]: 3.13 scrub ok 2024-09-10T18:30:55.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:55.661 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:55.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:55.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:55.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:55.806 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:55.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:55.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:55.950 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:55.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:56.052 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:56.086 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:56.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:56.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:56.187 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:56.225 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:56.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:56.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:56 smithi028 ceph-mon[39220]: 4.9 scrub starts 2024-09-10T18:30:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:56 smithi028 ceph-mon[39220]: 4.9 scrub ok 2024-09-10T18:30:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:56 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:56 smithi028 ceph-mon[39220]: pgmap v421: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 812 op/s 2024-09-10T18:30:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:56 smithi028 ceph-mon[29291]: 4.9 scrub starts 2024-09-10T18:30:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:56 smithi028 ceph-mon[29291]: 4.9 scrub ok 2024-09-10T18:30:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:56 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:56 smithi028 ceph-mon[29291]: pgmap v421: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 812 op/s 2024-09-10T18:30:56.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:56.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:56.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:56.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:56.501 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:56.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:56.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:56.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-10T18:30:56.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:56 smithi204 ceph-mon[32357]: 4.9 scrub starts 2024-09-10T18:30:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:56 smithi204 ceph-mon[32357]: 4.9 scrub ok 2024-09-10T18:30:56.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:56 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:30:56.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:56 smithi204 ceph-mon[32357]: pgmap v421: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 812 op/s 2024-09-10T18:30:56.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:56.653 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:56.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:56.754 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:56.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:56.784 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:56.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:56.886 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:56.911 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:56.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:56.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:56.964 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:30:57.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:57.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:30:57.058 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:57.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:57.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:57.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:57.195 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:57.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:57.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:57.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:57.321 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:57.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:57.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:57.423 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:57.451 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:57.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:57.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:57.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:57 smithi028 ceph-mon[29291]: 3.1c scrub starts 2024-09-10T18:30:57.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:57 smithi028 ceph-mon[29291]: 3.1c scrub ok 2024-09-10T18:30:57.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:57 smithi028 ceph-mon[29291]: 6.18 scrub starts 2024-09-10T18:30:57.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:57 smithi028 ceph-mon[29291]: 6.18 scrub ok 2024-09-10T18:30:57.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:57 smithi028 ceph-mon[29291]: 5.0 scrub starts 2024-09-10T18:30:57.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:57 smithi028 ceph-mon[29291]: 5.0 scrub ok 2024-09-10T18:30:57.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:57 smithi028 ceph-mon[39220]: 3.1c scrub starts 2024-09-10T18:30:57.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:57 smithi028 ceph-mon[39220]: 3.1c scrub ok 2024-09-10T18:30:57.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:57 smithi028 ceph-mon[39220]: 6.18 scrub starts 2024-09-10T18:30:57.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:57 smithi028 ceph-mon[39220]: 6.18 scrub ok 2024-09-10T18:30:57.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:57 smithi028 ceph-mon[39220]: 5.0 scrub starts 2024-09-10T18:30:57.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:57 smithi028 ceph-mon[39220]: 5.0 scrub ok 2024-09-10T18:30:57.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:57.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:57.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:57.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:57 smithi204 ceph-mon[32357]: 3.1c scrub starts 2024-09-10T18:30:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:57 smithi204 ceph-mon[32357]: 3.1c scrub ok 2024-09-10T18:30:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:57 smithi204 ceph-mon[32357]: 6.18 scrub starts 2024-09-10T18:30:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:57 smithi204 ceph-mon[32357]: 6.18 scrub ok 2024-09-10T18:30:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:57 smithi204 ceph-mon[32357]: 5.0 scrub starts 2024-09-10T18:30:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:57 smithi204 ceph-mon[32357]: 5.0 scrub ok 2024-09-10T18:30:57.700 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:57.728 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:57.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:57.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:57.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:57.872 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:57.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:57.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:57.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:58.001 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:58.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:58.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:58.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:58.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:58.150 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:58.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:58.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:30:58.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:58 smithi028 ceph-mon[29291]: pgmap v422: 158 pgs: 158 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-09-10T18:30:58.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:58 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:30:58.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:58 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:30:58.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:58 smithi028 ceph-mon[39220]: pgmap v422: 158 pgs: 158 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-09-10T18:30:58.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:58 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:30:58.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:58 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:30:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:58.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:58.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:30:58.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:30:58.432 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:58.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:58.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:30:58.534 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:58.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:58.566 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:58.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:58 smithi204 ceph-mon[32357]: pgmap v422: 158 pgs: 158 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-09-10T18:30:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:58 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:30:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:58 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:30:58.667 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:30:58.700 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:58.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:58.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:30:58.802 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:58.819 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:30:58.829 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:58.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:58.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:58.931 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:30:58.957 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:58.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:58.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:30:59.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:30:59.097 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:59.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:30:59.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:59.226 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:59.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:59.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[39220]: 3.a scrub starts 2024-09-10T18:30:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[39220]: 3.a scrub ok 2024-09-10T18:30:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[39220]: osdmap e231: 8 total, 8 up, 8 in 2024-09-10T18:30:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[39220]: 4.1a scrub starts 2024-09-10T18:30:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[39220]: 4.1a scrub ok 2024-09-10T18:30:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[39220]: 4.16 scrub starts 2024-09-10T18:30:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[39220]: 4.16 scrub ok 2024-09-10T18:30:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[39220]: 6.12 scrub starts 2024-09-10T18:30:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[39220]: 6.12 scrub ok 2024-09-10T18:30:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[29291]: 3.a scrub starts 2024-09-10T18:30:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[29291]: 3.a scrub ok 2024-09-10T18:30:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[29291]: osdmap e231: 8 total, 8 up, 8 in 2024-09-10T18:30:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[29291]: 4.1a scrub starts 2024-09-10T18:30:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[29291]: 4.1a scrub ok 2024-09-10T18:30:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[29291]: 4.16 scrub starts 2024-09-10T18:30:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[29291]: 4.16 scrub ok 2024-09-10T18:30:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[29291]: 6.12 scrub starts 2024-09-10T18:30:59.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:30:59 smithi028 ceph-mon[29291]: 6.12 scrub ok 2024-09-10T18:30:59.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:59.365 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:59.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:59.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:59.468 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:30:59.499 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:59.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:59.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:30:59.601 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:59.607 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:59 smithi204 ceph-mon[32357]: 3.a scrub starts 2024-09-10T18:30:59.607 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:59 smithi204 ceph-mon[32357]: 3.a scrub ok 2024-09-10T18:30:59.607 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:59 smithi204 ceph-mon[32357]: osdmap e231: 8 total, 8 up, 8 in 2024-09-10T18:30:59.607 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:59 smithi204 ceph-mon[32357]: 4.1a scrub starts 2024-09-10T18:30:59.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:59 smithi204 ceph-mon[32357]: 4.1a scrub ok 2024-09-10T18:30:59.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:59 smithi204 ceph-mon[32357]: 4.16 scrub starts 2024-09-10T18:30:59.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:59 smithi204 ceph-mon[32357]: 4.16 scrub ok 2024-09-10T18:30:59.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:59 smithi204 ceph-mon[32357]: 6.12 scrub starts 2024-09-10T18:30:59.608 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:30:59 smithi204 ceph-mon[32357]: 6.12 scrub ok 2024-09-10T18:30:59.634 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:59.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:59.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:30:59.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:30:59.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:59.774 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:30:59.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:30:59.874 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:30:59.902 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:30:59.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:30:59.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:00.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:00.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:00.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:00.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:00.180 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:31:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:00.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:00.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:00.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:00.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:00.442 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:00.469 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:00.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:00.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:00 smithi028 ceph-mon[39220]: pgmap v424: 158 pgs: 158 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 764 op/s 2024-09-10T18:31:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:00 smithi028 ceph-mon[39220]: 6.14 scrub starts 2024-09-10T18:31:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:00 smithi028 ceph-mon[39220]: 6.14 scrub ok 2024-09-10T18:31:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:00 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3517420593' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:31:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:00 smithi028 ceph-mon[39220]: 6.d scrub starts 2024-09-10T18:31:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:00 smithi028 ceph-mon[39220]: 6.d scrub ok 2024-09-10T18:31:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:00 smithi028 ceph-mon[39220]: 3.1f scrub starts 2024-09-10T18:31:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:00 smithi028 ceph-mon[39220]: 3.1f scrub ok 2024-09-10T18:31:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:00 smithi028 ceph-mon[29291]: pgmap v424: 158 pgs: 158 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 764 op/s 2024-09-10T18:31:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:00 smithi028 ceph-mon[29291]: 6.14 scrub starts 2024-09-10T18:31:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:00 smithi028 ceph-mon[29291]: 6.14 scrub ok 2024-09-10T18:31:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:00 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3517420593' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:31:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:00 smithi028 ceph-mon[29291]: 6.d scrub starts 2024-09-10T18:31:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:00 smithi028 ceph-mon[29291]: 6.d scrub ok 2024-09-10T18:31:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:00 smithi028 ceph-mon[29291]: 3.1f scrub starts 2024-09-10T18:31:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:00 smithi028 ceph-mon[29291]: 3.1f scrub ok 2024-09-10T18:31:00.571 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:00 smithi204 ceph-mon[32357]: pgmap v424: 158 pgs: 158 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 764 op/s 2024-09-10T18:31:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:00 smithi204 ceph-mon[32357]: 6.14 scrub starts 2024-09-10T18:31:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:00 smithi204 ceph-mon[32357]: 6.14 scrub ok 2024-09-10T18:31:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:00 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3517420593' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:31:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:00 smithi204 ceph-mon[32357]: 6.d scrub starts 2024-09-10T18:31:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:00 smithi204 ceph-mon[32357]: 6.d scrub ok 2024-09-10T18:31:00.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:00 smithi204 ceph-mon[32357]: 3.1f scrub starts 2024-09-10T18:31:00.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:00 smithi204 ceph-mon[32357]: 3.1f scrub ok 2024-09-10T18:31:00.606 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:00.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:00.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:00.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:00.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:00.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:00.756 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:00.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:00.904 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:00.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:01.006 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:01.032 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:01.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:01.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:01.134 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:01.162 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:01.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:01.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:01.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:01.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:01.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:01.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:01.440 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:01.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:01.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:01.451 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:31:01] "GET /metrics HTTP/1.1" 200 224385 "" "Prometheus/2.43.0" 2024-09-10T18:31:01.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[29291]: 4.1b scrub starts 2024-09-10T18:31:01.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[29291]: 4.1b scrub ok 2024-09-10T18:31:01.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[29291]: 4.a scrub starts 2024-09-10T18:31:01.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[29291]: 4.a scrub ok 2024-09-10T18:31:01.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:31:01.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:31:01.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3517420593' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:31:01.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[29291]: osdmap e232: 8 total, 8 up, 8 in 2024-09-10T18:31:01.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[29291]: 5.f deep-scrub starts 2024-09-10T18:31:01.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[29291]: 5.f deep-scrub ok 2024-09-10T18:31:01.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[29291]: 3.11 scrub starts 2024-09-10T18:31:01.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[29291]: 3.11 scrub ok 2024-09-10T18:31:01.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[29291]: 3.b scrub starts 2024-09-10T18:31:01.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[29291]: 3.b scrub ok 2024-09-10T18:31:01.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[39220]: 4.1b scrub starts 2024-09-10T18:31:01.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[39220]: 4.1b scrub ok 2024-09-10T18:31:01.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[39220]: 4.a scrub starts 2024-09-10T18:31:01.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[39220]: 4.a scrub ok 2024-09-10T18:31:01.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:31:01.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:31:01.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3517420593' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:31:01.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[39220]: osdmap e232: 8 total, 8 up, 8 in 2024-09-10T18:31:01.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[39220]: 5.f deep-scrub starts 2024-09-10T18:31:01.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[39220]: 5.f deep-scrub ok 2024-09-10T18:31:01.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[39220]: 3.11 scrub starts 2024-09-10T18:31:01.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[39220]: 3.11 scrub ok 2024-09-10T18:31:01.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[39220]: 3.b scrub starts 2024-09-10T18:31:01.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:01 smithi028 ceph-mon[39220]: 3.b scrub ok 2024-09-10T18:31:01.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:01.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:01.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:01 smithi204 ceph-mon[32357]: 4.1b scrub starts 2024-09-10T18:31:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:01 smithi204 ceph-mon[32357]: 4.1b scrub ok 2024-09-10T18:31:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:01 smithi204 ceph-mon[32357]: 4.a scrub starts 2024-09-10T18:31:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:01 smithi204 ceph-mon[32357]: 4.a scrub ok 2024-09-10T18:31:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:01 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:31:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:01 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:31:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:01 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3517420593' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:31:01.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:01 smithi204 ceph-mon[32357]: osdmap e232: 8 total, 8 up, 8 in 2024-09-10T18:31:01.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:01 smithi204 ceph-mon[32357]: 5.f deep-scrub starts 2024-09-10T18:31:01.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:01 smithi204 ceph-mon[32357]: 5.f deep-scrub ok 2024-09-10T18:31:01.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:01 smithi204 ceph-mon[32357]: 3.11 scrub starts 2024-09-10T18:31:01.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:01 smithi204 ceph-mon[32357]: 3.11 scrub ok 2024-09-10T18:31:01.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:01 smithi204 ceph-mon[32357]: 3.b scrub starts 2024-09-10T18:31:01.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:01 smithi204 ceph-mon[32357]: 3.b scrub ok 2024-09-10T18:31:01.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:01.725 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:01.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:01.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:01.827 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:01.854 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:01.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:01.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:01.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:02.003 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:02.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:02.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:02.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:02.145 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:02.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:02.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:02.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:02 smithi028 ceph-mon[39220]: 4.2 deep-scrub starts 2024-09-10T18:31:02.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:02 smithi028 ceph-mon[39220]: 4.2 deep-scrub ok 2024-09-10T18:31:02.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:02 smithi028 ceph-mon[39220]: pgmap v426: 158 pgs: 158 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-10T18:31:02.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:02 smithi028 ceph-mon[39220]: 6.15 scrub starts 2024-09-10T18:31:02.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:02 smithi028 ceph-mon[39220]: 6.15 scrub ok 2024-09-10T18:31:02.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:02 smithi028 ceph-mon[39220]: 3.7 scrub starts 2024-09-10T18:31:02.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:02 smithi028 ceph-mon[39220]: 3.7 scrub ok 2024-09-10T18:31:02.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:02 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1949661543' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:31:02.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:02 smithi028 ceph-mon[29291]: 4.2 deep-scrub starts 2024-09-10T18:31:02.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:02 smithi028 ceph-mon[29291]: 4.2 deep-scrub ok 2024-09-10T18:31:02.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:02 smithi028 ceph-mon[29291]: pgmap v426: 158 pgs: 158 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-10T18:31:02.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:02 smithi028 ceph-mon[29291]: 6.15 scrub starts 2024-09-10T18:31:02.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:02 smithi028 ceph-mon[29291]: 6.15 scrub ok 2024-09-10T18:31:02.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:02 smithi028 ceph-mon[29291]: 3.7 scrub starts 2024-09-10T18:31:02.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:02 smithi028 ceph-mon[29291]: 3.7 scrub ok 2024-09-10T18:31:02.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:02 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1949661543' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:31:02.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:02.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:02.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:02.387 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:02.396 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:31:02.396 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:31:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:02.413 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:02.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:02.514 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:02.539 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:02.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:02.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:02.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:31:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:31:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:31:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:02 smithi204 ceph-mon[32357]: 4.2 deep-scrub starts 2024-09-10T18:31:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:02 smithi204 ceph-mon[32357]: 4.2 deep-scrub ok 2024-09-10T18:31:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:02 smithi204 ceph-mon[32357]: pgmap v426: 158 pgs: 158 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-10T18:31:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:02 smithi204 ceph-mon[32357]: 6.15 scrub starts 2024-09-10T18:31:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:02 smithi204 ceph-mon[32357]: 6.15 scrub ok 2024-09-10T18:31:02.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:02 smithi204 ceph-mon[32357]: 3.7 scrub starts 2024-09-10T18:31:02.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:02 smithi204 ceph-mon[32357]: 3.7 scrub ok 2024-09-10T18:31:02.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:02 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1949661543' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:31:02.641 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:02.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:02.674 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:02.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:02.734 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.5 to 1 (10000) 2024-09-10T18:31:02.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:02.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:02.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:02.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:02.919 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:02.945 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:02.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:02.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:03.047 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:03.080 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:03.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:03.181 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:03.209 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:03.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:03.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:03.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:03.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:03.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:03.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:03.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:03.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:03 smithi028 ceph-mon[29291]: 3.10 scrub starts 2024-09-10T18:31:03.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:03 smithi028 ceph-mon[29291]: 3.10 scrub ok 2024-09-10T18:31:03.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:03 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1949661543' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T18:31:03.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:03 smithi028 ceph-mon[29291]: osdmap e233: 8 total, 8 up, 8 in 2024-09-10T18:31:03.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:03 smithi028 ceph-mon[39220]: 3.10 scrub starts 2024-09-10T18:31:03.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:03 smithi028 ceph-mon[39220]: 3.10 scrub ok 2024-09-10T18:31:03.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:03 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1949661543' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T18:31:03.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:03 smithi028 ceph-mon[39220]: osdmap e233: 8 total, 8 up, 8 in 2024-09-10T18:31:03.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:03 smithi204 ceph-mon[32357]: 3.10 scrub starts 2024-09-10T18:31:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:03 smithi204 ceph-mon[32357]: 3.10 scrub ok 2024-09-10T18:31:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:03 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1949661543' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T18:31:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:03 smithi204 ceph-mon[32357]: osdmap e233: 8 total, 8 up, 8 in 2024-09-10T18:31:03.626 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:03.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:03.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:03.729 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:03.757 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:03.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:03.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:03.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:03.899 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:03.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:04.000 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:04.028 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:04.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:04.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:04.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:04.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:04.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:04.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:04.298 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:04.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:04.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:04.447 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:04.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:04.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:04 smithi028 ceph-mon[39220]: pgmap v428: 158 pgs: 158 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:31:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:04 smithi028 ceph-mon[39220]: 1.6 scrub starts 2024-09-10T18:31:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:04 smithi028 ceph-mon[39220]: 1.6 scrub ok 2024-09-10T18:31:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:04 smithi028 ceph-mon[39220]: 5.9 deep-scrub starts 2024-09-10T18:31:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:04 smithi028 ceph-mon[39220]: 5.9 deep-scrub ok 2024-09-10T18:31:04.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:04 smithi028 ceph-mon[29291]: pgmap v428: 158 pgs: 158 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:31:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:04 smithi028 ceph-mon[29291]: 1.6 scrub starts 2024-09-10T18:31:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:04 smithi028 ceph-mon[29291]: 1.6 scrub ok 2024-09-10T18:31:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:04 smithi028 ceph-mon[29291]: 5.9 deep-scrub starts 2024-09-10T18:31:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:04 smithi028 ceph-mon[29291]: 5.9 deep-scrub ok 2024-09-10T18:31:04.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:04.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:04.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:04.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:04 smithi204 ceph-mon[32357]: pgmap v428: 158 pgs: 158 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:31:04.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:04 smithi204 ceph-mon[32357]: 1.6 scrub starts 2024-09-10T18:31:04.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:04 smithi204 ceph-mon[32357]: 1.6 scrub ok 2024-09-10T18:31:04.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:04 smithi204 ceph-mon[32357]: 5.9 deep-scrub starts 2024-09-10T18:31:04.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:04 smithi204 ceph-mon[32357]: 5.9 deep-scrub ok 2024-09-10T18:31:04.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:31:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:31:04.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:04.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:04.737 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:04.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:04.838 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:04.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:31:04.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:04.876 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:04.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:04.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:05.003 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:05.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:05.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:05.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:05.128 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:05.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:05.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:05.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:05.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-10T18:31:05.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:05.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:05.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:05.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:05.429 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:05.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:05.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[39220]: 3.1a scrub starts 2024-09-10T18:31:05.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[39220]: 3.1a scrub ok 2024-09-10T18:31:05.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[39220]: 6.f scrub starts 2024-09-10T18:31:05.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[39220]: 6.f scrub ok 2024-09-10T18:31:05.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:31:05.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[39220]: 2.2 scrub starts 2024-09-10T18:31:05.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[39220]: 2.2 scrub ok 2024-09-10T18:31:05.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[39220]: 6.6 scrub starts 2024-09-10T18:31:05.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[39220]: 6.6 scrub ok 2024-09-10T18:31:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[29291]: 3.1a scrub starts 2024-09-10T18:31:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[29291]: 3.1a scrub ok 2024-09-10T18:31:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[29291]: 6.f scrub starts 2024-09-10T18:31:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[29291]: 6.f scrub ok 2024-09-10T18:31:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:31:05.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[29291]: 2.2 scrub starts 2024-09-10T18:31:05.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[29291]: 2.2 scrub ok 2024-09-10T18:31:05.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[29291]: 6.6 scrub starts 2024-09-10T18:31:05.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:05 smithi028 ceph-mon[29291]: 6.6 scrub ok 2024-09-10T18:31:05.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:05.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:05 smithi204 ceph-mon[32357]: 3.1a scrub starts 2024-09-10T18:31:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:05 smithi204 ceph-mon[32357]: 3.1a scrub ok 2024-09-10T18:31:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:05 smithi204 ceph-mon[32357]: 6.f scrub starts 2024-09-10T18:31:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:05 smithi204 ceph-mon[32357]: 6.f scrub ok 2024-09-10T18:31:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:05 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:31:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:05 smithi204 ceph-mon[32357]: 2.2 scrub starts 2024-09-10T18:31:05.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:05 smithi204 ceph-mon[32357]: 2.2 scrub ok 2024-09-10T18:31:05.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:05 smithi204 ceph-mon[32357]: 6.6 scrub starts 2024-09-10T18:31:05.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:05 smithi204 ceph-mon[32357]: 6.6 scrub ok 2024-09-10T18:31:05.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:05.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:05.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:05.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:05.868 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:05.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:05.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:05.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:06.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:06.008 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:06.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:06.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:06.144 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:06.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:06.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:06.298 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:06.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:06.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:06.400 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:06.432 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:06.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:06.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[39220]: 5.1 deep-scrub starts 2024-09-10T18:31:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[39220]: 5.1 deep-scrub ok 2024-09-10T18:31:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[39220]: 3.19 scrub starts 2024-09-10T18:31:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[39220]: 3.19 scrub ok 2024-09-10T18:31:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[39220]: pgmap v429: 158 pgs: 158 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:31:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:31:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:31:06.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:31:06.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[39220]: 5.3 scrub starts 2024-09-10T18:31:06.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[39220]: 5.3 scrub ok 2024-09-10T18:31:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[29291]: 5.1 deep-scrub starts 2024-09-10T18:31:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[29291]: 5.1 deep-scrub ok 2024-09-10T18:31:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[29291]: 3.19 scrub starts 2024-09-10T18:31:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[29291]: 3.19 scrub ok 2024-09-10T18:31:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[29291]: pgmap v429: 158 pgs: 158 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:31:06.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:31:06.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:31:06.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:31:06.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[29291]: 5.3 scrub starts 2024-09-10T18:31:06.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:06 smithi028 ceph-mon[29291]: 5.3 scrub ok 2024-09-10T18:31:06.536 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:06.568 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:06.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:06 smithi204 ceph-mon[32357]: 5.1 deep-scrub starts 2024-09-10T18:31:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:06 smithi204 ceph-mon[32357]: 5.1 deep-scrub ok 2024-09-10T18:31:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:06 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:06 smithi204 ceph-mon[32357]: 3.19 scrub starts 2024-09-10T18:31:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:06 smithi204 ceph-mon[32357]: 3.19 scrub ok 2024-09-10T18:31:06.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:06 smithi204 ceph-mon[32357]: pgmap v429: 158 pgs: 158 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:31:06.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:06 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:31:06.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:06 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:31:06.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:06 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:31:06.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:06 smithi204 ceph-mon[32357]: 5.3 scrub starts 2024-09-10T18:31:06.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:06 smithi204 ceph-mon[32357]: 5.3 scrub ok 2024-09-10T18:31:06.668 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:06.701 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:06.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:06.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:06.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:06.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:06.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:06.935 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:06.968 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:06.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:06.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:07.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:07.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:07.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:07.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:07.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:07.244 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:07.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:07.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:07.346 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:07.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:07.378 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:07.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:07.479 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:07 smithi028 ceph-mon[39220]: 8.3 scrub starts 2024-09-10T18:31:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:07 smithi028 ceph-mon[39220]: 8.5 deep-scrub starts 2024-09-10T18:31:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:07 smithi028 ceph-mon[39220]: 6.3 deep-scrub starts 2024-09-10T18:31:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:07 smithi028 ceph-mon[39220]: 6.3 deep-scrub ok 2024-09-10T18:31:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:07 smithi028 ceph-mon[39220]: 4.1d scrub starts 2024-09-10T18:31:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:07 smithi028 ceph-mon[39220]: 4.1d scrub ok 2024-09-10T18:31:07.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:07 smithi028 ceph-mon[29291]: 8.3 scrub starts 2024-09-10T18:31:07.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:07 smithi028 ceph-mon[29291]: 8.5 deep-scrub starts 2024-09-10T18:31:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:07 smithi028 ceph-mon[29291]: 6.3 deep-scrub starts 2024-09-10T18:31:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:07 smithi028 ceph-mon[29291]: 6.3 deep-scrub ok 2024-09-10T18:31:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:07 smithi028 ceph-mon[29291]: 4.1d scrub starts 2024-09-10T18:31:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:07 smithi028 ceph-mon[29291]: 4.1d scrub ok 2024-09-10T18:31:07.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:07.508 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:07.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:07 smithi204 ceph-mon[32357]: 8.3 scrub starts 2024-09-10T18:31:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:07 smithi204 ceph-mon[32357]: 8.5 deep-scrub starts 2024-09-10T18:31:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:07 smithi204 ceph-mon[32357]: 6.3 deep-scrub starts 2024-09-10T18:31:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:07 smithi204 ceph-mon[32357]: 6.3 deep-scrub ok 2024-09-10T18:31:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:07 smithi204 ceph-mon[32357]: 4.1d scrub starts 2024-09-10T18:31:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:07 smithi204 ceph-mon[32357]: 4.1d scrub ok 2024-09-10T18:31:07.609 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:07.642 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:07.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:07.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:07.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:07.777 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:07.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:07.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:07.879 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:07.910 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:07.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:07.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:08.012 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:08.051 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:08.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:08.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:08.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:08.186 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:08.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:08.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:08.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:08.325 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:08.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:08.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-10T18:31:08.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:08 smithi028 ceph-mon[29291]: 6.17 scrub starts 2024-09-10T18:31:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:08 smithi028 ceph-mon[29291]: 6.17 scrub ok 2024-09-10T18:31:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:08 smithi028 ceph-mon[29291]: 8.3 scrub ok 2024-09-10T18:31:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:08 smithi028 ceph-mon[29291]: pgmap v430: 158 pgs: 158 active+clean; 2.3 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-09-10T18:31:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:08 smithi028 ceph-mon[39220]: 6.17 scrub starts 2024-09-10T18:31:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:08 smithi028 ceph-mon[39220]: 6.17 scrub ok 2024-09-10T18:31:08.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:08 smithi028 ceph-mon[39220]: 8.3 scrub ok 2024-09-10T18:31:08.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:08 smithi028 ceph-mon[39220]: pgmap v430: 158 pgs: 158 active+clean; 2.3 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-09-10T18:31:08.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:08.506 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:08.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:08 smithi204 ceph-mon[32357]: 6.17 scrub starts 2024-09-10T18:31:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:08 smithi204 ceph-mon[32357]: 6.17 scrub ok 2024-09-10T18:31:08.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:08 smithi204 ceph-mon[32357]: 8.3 scrub ok 2024-09-10T18:31:08.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:08 smithi204 ceph-mon[32357]: pgmap v430: 158 pgs: 158 active+clean; 2.3 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-09-10T18:31:08.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:08.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:08.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:08.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:08.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:08.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:08.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:08.887 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:08.920 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:08.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:08.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:09.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:09.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:09.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:09.149 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:09.160 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:09.189 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:09.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:09.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:09.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:09.329 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:09.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:09.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:09.332 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:09 smithi028 ceph-mon[39220]: 5.a scrub starts 2024-09-10T18:31:09.332 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:09 smithi028 ceph-mon[39220]: 5.a scrub ok 2024-09-10T18:31:09.333 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:09 smithi028 ceph-mon[39220]: 3.1 scrub starts 2024-09-10T18:31:09.333 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:09 smithi028 ceph-mon[39220]: 3.1 scrub ok 2024-09-10T18:31:09.333 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:31:09.333 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:31:09.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:09 smithi028 ceph-mon[29291]: 5.a scrub starts 2024-09-10T18:31:09.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:09 smithi028 ceph-mon[29291]: 5.a scrub ok 2024-09-10T18:31:09.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:09 smithi028 ceph-mon[29291]: 3.1 scrub starts 2024-09-10T18:31:09.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:09 smithi028 ceph-mon[29291]: 3.1 scrub ok 2024-09-10T18:31:09.334 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:31:09.334 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:31:09.432 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:09.471 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:09.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:09.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:09.573 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:09 smithi204 ceph-mon[32357]: 5.a scrub starts 2024-09-10T18:31:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:09 smithi204 ceph-mon[32357]: 5.a scrub ok 2024-09-10T18:31:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:09 smithi204 ceph-mon[32357]: 3.1 scrub starts 2024-09-10T18:31:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:09 smithi204 ceph-mon[32357]: 3.1 scrub ok 2024-09-10T18:31:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:31:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:31:09.608 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:09.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:09.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:09.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:09.742 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:09.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:09.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:09.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:09.884 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:09.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:09.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:09.927 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:31:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:31:09.973 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:31:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:31:09.987 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:10.002 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:31:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:31:10.018 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:10.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:10.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:10.031 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:31:10.032 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:31:10.065 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:31:10.065 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:31:10.097 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:31:10.098 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:31:10.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:10.125 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:31:10.126 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:31:10.159 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:10.160 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:31:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:31:10.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:10.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:10.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:10.300 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:10.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:10.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:10.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:10.446 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:10.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:10.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:10 smithi028 ceph-mon[39220]: pgmap v431: 158 pgs: 158 active+clean; 2.3 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 583 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-10T18:31:10.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:10 smithi028 ceph-mon[39220]: 5.d scrub starts 2024-09-10T18:31:10.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:10 smithi028 ceph-mon[39220]: 5.d scrub ok 2024-09-10T18:31:10.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:10 smithi028 ceph-mon[39220]: 8.9 scrub starts 2024-09-10T18:31:10.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:10 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3200087551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:31:10.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:10 smithi028 ceph-mon[29291]: pgmap v431: 158 pgs: 158 active+clean; 2.3 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 583 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-10T18:31:10.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:10 smithi028 ceph-mon[29291]: 5.d scrub starts 2024-09-10T18:31:10.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:10 smithi028 ceph-mon[29291]: 5.d scrub ok 2024-09-10T18:31:10.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:10 smithi028 ceph-mon[29291]: 8.9 scrub starts 2024-09-10T18:31:10.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:10 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3200087551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:31:10.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:10.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:10.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:10 smithi204 ceph-mon[32357]: pgmap v431: 158 pgs: 158 active+clean; 2.3 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 583 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-10T18:31:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:10 smithi204 ceph-mon[32357]: 5.d scrub starts 2024-09-10T18:31:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:10 smithi204 ceph-mon[32357]: 5.d scrub ok 2024-09-10T18:31:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:10 smithi204 ceph-mon[32357]: 8.9 scrub starts 2024-09-10T18:31:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:10 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3200087551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:31:10.690 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:10.719 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:10.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:10.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:10.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:10.862 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:10.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:10.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:11.002 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:11.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:11.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:11.104 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:11.135 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:11.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:11.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:11.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:11.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:11.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:11.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:11.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:11.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:11.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:11.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:31:11] "GET /metrics HTTP/1.1" 200 224342 "" "Prometheus/2.43.0" 2024-09-10T18:31:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[39220]: 6.1a scrub starts 2024-09-10T18:31:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[39220]: 6.1a scrub ok 2024-09-10T18:31:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[39220]: 8.9 scrub ok 2024-09-10T18:31:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[39220]: 5.11 scrub starts 2024-09-10T18:31:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[39220]: 5.11 scrub ok 2024-09-10T18:31:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[39220]: 4.3 scrub starts 2024-09-10T18:31:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[39220]: 4.3 scrub ok 2024-09-10T18:31:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[39220]: 1.9 scrub starts 2024-09-10T18:31:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[39220]: 1.9 scrub ok 2024-09-10T18:31:11.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[29291]: 6.1a scrub starts 2024-09-10T18:31:11.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[29291]: 6.1a scrub ok 2024-09-10T18:31:11.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[29291]: 8.9 scrub ok 2024-09-10T18:31:11.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[29291]: 5.11 scrub starts 2024-09-10T18:31:11.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[29291]: 5.11 scrub ok 2024-09-10T18:31:11.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[29291]: 4.3 scrub starts 2024-09-10T18:31:11.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[29291]: 4.3 scrub ok 2024-09-10T18:31:11.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[29291]: 1.9 scrub starts 2024-09-10T18:31:11.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:11 smithi028 ceph-mon[29291]: 1.9 scrub ok 2024-09-10T18:31:11.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:11.547 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:11.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:11.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:11 smithi204 ceph-mon[32357]: 6.1a scrub starts 2024-09-10T18:31:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:11 smithi204 ceph-mon[32357]: 6.1a scrub ok 2024-09-10T18:31:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:11 smithi204 ceph-mon[32357]: 8.9 scrub ok 2024-09-10T18:31:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:11 smithi204 ceph-mon[32357]: 5.11 scrub starts 2024-09-10T18:31:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:11 smithi204 ceph-mon[32357]: 5.11 scrub ok 2024-09-10T18:31:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:11 smithi204 ceph-mon[32357]: 4.3 scrub starts 2024-09-10T18:31:11.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:11 smithi204 ceph-mon[32357]: 4.3 scrub ok 2024-09-10T18:31:11.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:11 smithi204 ceph-mon[32357]: 1.9 scrub starts 2024-09-10T18:31:11.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:11 smithi204 ceph-mon[32357]: 1.9 scrub ok 2024-09-10T18:31:11.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:11.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:11.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:11.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:11.817 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:11.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:11.919 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:11.955 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:11.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:11.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:12.057 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:12.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:12.093 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:12.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:12.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:12.228 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:12.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:12.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:12.330 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:12.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:12.361 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:12.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:12.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:12.484 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:12.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:12.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[29291]: 5.6 scrub starts 2024-09-10T18:31:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[29291]: 5.6 scrub ok 2024-09-10T18:31:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[29291]: pgmap v432: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:31:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:31:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:31:12.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[29291]: 4.4 scrub starts 2024-09-10T18:31:12.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[29291]: 4.4 scrub ok 2024-09-10T18:31:12.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[29291]: 4.f scrub starts 2024-09-10T18:31:12.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[29291]: 4.f scrub ok 2024-09-10T18:31:12.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[29291]: osdmap e234: 8 total, 8 up, 8 in 2024-09-10T18:31:12.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[39220]: 5.6 scrub starts 2024-09-10T18:31:12.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[39220]: 5.6 scrub ok 2024-09-10T18:31:12.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[39220]: pgmap v432: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:31:12.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:31:12.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:31:12.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[39220]: 4.4 scrub starts 2024-09-10T18:31:12.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[39220]: 4.4 scrub ok 2024-09-10T18:31:12.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[39220]: 4.f scrub starts 2024-09-10T18:31:12.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[39220]: 4.f scrub ok 2024-09-10T18:31:12.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:12 smithi028 ceph-mon[39220]: osdmap e234: 8 total, 8 up, 8 in 2024-09-10T18:31:12.585 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:12 smithi204 ceph-mon[32357]: 5.6 scrub starts 2024-09-10T18:31:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:12 smithi204 ceph-mon[32357]: 5.6 scrub ok 2024-09-10T18:31:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:12 smithi204 ceph-mon[32357]: pgmap v432: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:31:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:12 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:31:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:12 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:31:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:12 smithi204 ceph-mon[32357]: 4.4 scrub starts 2024-09-10T18:31:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:12 smithi204 ceph-mon[32357]: 4.4 scrub ok 2024-09-10T18:31:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:12 smithi204 ceph-mon[32357]: 4.f scrub starts 2024-09-10T18:31:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:12 smithi204 ceph-mon[32357]: 4.f scrub ok 2024-09-10T18:31:12.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:12 smithi204 ceph-mon[32357]: osdmap e234: 8 total, 8 up, 8 in 2024-09-10T18:31:12.606 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:31:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:31:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:31:12.625 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:12.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:12.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:12.727 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:12.768 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:12.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:12.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:12.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:12.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:13.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:13.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:13.140 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:13.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:13.179 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:13.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:13.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:13.317 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:13.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:13.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:13.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:13.452 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:13.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:13 smithi028 ceph-mon[39220]: 8.a scrub starts 2024-09-10T18:31:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:13 smithi028 ceph-mon[39220]: 3.e deep-scrub starts 2024-09-10T18:31:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:13 smithi028 ceph-mon[39220]: 3.e deep-scrub ok 2024-09-10T18:31:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:13 smithi028 ceph-mon[39220]: 6.b deep-scrub starts 2024-09-10T18:31:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:13 smithi028 ceph-mon[39220]: 6.b deep-scrub ok 2024-09-10T18:31:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:13 smithi028 ceph-mon[29291]: 8.a scrub starts 2024-09-10T18:31:13.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:13 smithi028 ceph-mon[29291]: 3.e deep-scrub starts 2024-09-10T18:31:13.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:13 smithi028 ceph-mon[29291]: 3.e deep-scrub ok 2024-09-10T18:31:13.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:13 smithi028 ceph-mon[29291]: 6.b deep-scrub starts 2024-09-10T18:31:13.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:13 smithi028 ceph-mon[29291]: 6.b deep-scrub ok 2024-09-10T18:31:13.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:13.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:13.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:13 smithi204 ceph-mon[32357]: 8.a scrub starts 2024-09-10T18:31:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:13 smithi204 ceph-mon[32357]: 3.e deep-scrub starts 2024-09-10T18:31:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:13 smithi204 ceph-mon[32357]: 3.e deep-scrub ok 2024-09-10T18:31:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:13 smithi204 ceph-mon[32357]: 6.b deep-scrub starts 2024-09-10T18:31:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:13 smithi204 ceph-mon[32357]: 6.b deep-scrub ok 2024-09-10T18:31:13.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:13.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:13.722 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:13.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:13.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:13.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:13.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:13.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:14.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:14.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:14.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:14.137 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:14.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:14.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:14.239 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:14.269 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:14.272 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:14.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:14.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:14.374 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:14.407 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:14.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:14.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:14 smithi028 ceph-mon[39220]: 5.1c scrub starts 2024-09-10T18:31:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:14 smithi028 ceph-mon[39220]: 5.1c scrub ok 2024-09-10T18:31:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:14 smithi028 ceph-mon[39220]: 5.8 scrub starts 2024-09-10T18:31:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:14 smithi028 ceph-mon[39220]: 5.8 scrub ok 2024-09-10T18:31:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:14 smithi028 ceph-mon[39220]: pgmap v434: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:31:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:14 smithi028 ceph-mon[39220]: 8.a scrub ok 2024-09-10T18:31:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:14 smithi028 ceph-mon[39220]: 2.0 scrub starts 2024-09-10T18:31:14.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:14 smithi028 ceph-mon[39220]: 2.0 scrub ok 2024-09-10T18:31:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:14 smithi028 ceph-mon[29291]: 5.1c scrub starts 2024-09-10T18:31:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:14 smithi028 ceph-mon[29291]: 5.1c scrub ok 2024-09-10T18:31:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:14 smithi028 ceph-mon[29291]: 5.8 scrub starts 2024-09-10T18:31:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:14 smithi028 ceph-mon[29291]: 5.8 scrub ok 2024-09-10T18:31:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:14 smithi028 ceph-mon[29291]: pgmap v434: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:31:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:14 smithi028 ceph-mon[29291]: 8.a scrub ok 2024-09-10T18:31:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:14 smithi028 ceph-mon[29291]: 2.0 scrub starts 2024-09-10T18:31:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:14 smithi028 ceph-mon[29291]: 2.0 scrub ok 2024-09-10T18:31:14.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:14.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:14.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:14 smithi204 ceph-mon[32357]: 5.1c scrub starts 2024-09-10T18:31:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:14 smithi204 ceph-mon[32357]: 5.1c scrub ok 2024-09-10T18:31:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:14 smithi204 ceph-mon[32357]: 5.8 scrub starts 2024-09-10T18:31:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:14 smithi204 ceph-mon[32357]: 5.8 scrub ok 2024-09-10T18:31:14.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:14 smithi204 ceph-mon[32357]: pgmap v434: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:31:14.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:14 smithi204 ceph-mon[32357]: 8.a scrub ok 2024-09-10T18:31:14.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:14 smithi204 ceph-mon[32357]: 2.0 scrub starts 2024-09-10T18:31:14.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:14 smithi204 ceph-mon[32357]: 2.0 scrub ok 2024-09-10T18:31:14.605 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:31:14 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:31:14.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:14.691 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:14.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:14.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:14.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:14.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:14.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:14.930 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:14.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:14.966 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:14.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:15.068 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:15.101 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:15.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:15.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:15.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:15.248 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:31:15.255 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:15.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:15.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:15.357 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:15.375 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:15 smithi028 ceph-mon[39220]: 6.a scrub starts 2024-09-10T18:31:15.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:15 smithi028 ceph-mon[39220]: 6.a scrub ok 2024-09-10T18:31:15.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:15 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:15.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:15 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2135218588' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:31:15.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:15 smithi028 ceph-mon[39220]: 6.13 scrub starts 2024-09-10T18:31:15.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:15 smithi028 ceph-mon[39220]: 6.13 scrub ok 2024-09-10T18:31:15.377 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:15 smithi028 ceph-mon[39220]: pgmap v435: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.9 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:31:15.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:15 smithi028 ceph-mon[29291]: 6.a scrub starts 2024-09-10T18:31:15.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:15 smithi028 ceph-mon[29291]: 6.a scrub ok 2024-09-10T18:31:15.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:15 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:15.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:15 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2135218588' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:31:15.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:15 smithi028 ceph-mon[29291]: 6.13 scrub starts 2024-09-10T18:31:15.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:15 smithi028 ceph-mon[29291]: 6.13 scrub ok 2024-09-10T18:31:15.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:15 smithi028 ceph-mon[29291]: pgmap v435: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.9 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:31:15.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:15.399 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:15.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:15.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:15.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:15.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:15 smithi204 ceph-mon[32357]: 6.a scrub starts 2024-09-10T18:31:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:15 smithi204 ceph-mon[32357]: 6.a scrub ok 2024-09-10T18:31:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:15 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:15 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2135218588' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:31:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:15 smithi204 ceph-mon[32357]: 6.13 scrub starts 2024-09-10T18:31:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:15 smithi204 ceph-mon[32357]: 6.13 scrub ok 2024-09-10T18:31:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:15 smithi204 ceph-mon[32357]: pgmap v435: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.9 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:31:15.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:15.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:15.674 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:15.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:15.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:15.811 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:15.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:15.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:15.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:15.949 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:15.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:15.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:16.051 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:16.095 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:16.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:16.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:16.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:31:16.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:16.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:16.270 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:16.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:16.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:16.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:16.403 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:16.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:16.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:16.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:16 smithi028 ceph-mon[29291]: 5.b scrub starts 2024-09-10T18:31:16.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:16 smithi028 ceph-mon[29291]: 5.b scrub ok 2024-09-10T18:31:16.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:16 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:31:16.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:16 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2135218588' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:31:16.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:16 smithi028 ceph-mon[29291]: osdmap e235: 8 total, 8 up, 8 in 2024-09-10T18:31:16.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:16 smithi028 ceph-mon[39220]: 5.b scrub starts 2024-09-10T18:31:16.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:16 smithi028 ceph-mon[39220]: 5.b scrub ok 2024-09-10T18:31:16.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:16 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:31:16.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:16 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2135218588' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:31:16.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:16 smithi028 ceph-mon[39220]: osdmap e235: 8 total, 8 up, 8 in 2024-09-10T18:31:16.554 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:16.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:16.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:16 smithi204 ceph-mon[32357]: 5.b scrub starts 2024-09-10T18:31:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:16 smithi204 ceph-mon[32357]: 5.b scrub ok 2024-09-10T18:31:16.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:16 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:31:16.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:16 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2135218588' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:31:16.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:16 smithi204 ceph-mon[32357]: osdmap e235: 8 total, 8 up, 8 in 2024-09-10T18:31:16.656 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:16.688 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:16.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:16.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:16.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:16.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:16.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:16.878 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:31:16.879 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-10T18:31:16.879 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-10T18:31:16.879 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-10T18:31:16.879 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:31:16.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:16.969 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:16.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:16.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:17.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:17.104 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:17.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:17.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:17.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:17.242 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:17.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:17.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:17.344 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:17.374 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:17.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:17.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:17.504 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:17.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:17.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:17 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:31:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:17 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:31:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:17 smithi204 ceph-mon[32357]: osdmap e236: 8 total, 8 up, 8 in 2024-09-10T18:31:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:17 smithi204 ceph-mon[32357]: pgmap v438: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 939 op/s 2024-09-10T18:31:17.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:17.622 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:17 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:31:17.622 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:17 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:31:17.622 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:17 smithi028 ceph-mon[39220]: osdmap e236: 8 total, 8 up, 8 in 2024-09-10T18:31:17.622 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:17 smithi028 ceph-mon[39220]: pgmap v438: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 939 op/s 2024-09-10T18:31:17.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:17 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:31:17.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:17 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:31:17.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:17 smithi028 ceph-mon[29291]: osdmap e236: 8 total, 8 up, 8 in 2024-09-10T18:31:17.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:17 smithi028 ceph-mon[29291]: pgmap v438: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 939 op/s 2024-09-10T18:31:17.650 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:17.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:17.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:17.752 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:17.778 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:17.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:17.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:17.880 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:17.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:17.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:18.032 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:18.062 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:18.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:18.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:18.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:18.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:18.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:18.362 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:18.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:18.410 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:18 smithi028 ceph-mon[39220]: 8.5 deep-scrub ok 2024-09-10T18:31:18.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:18 smithi028 ceph-mon[29291]: 8.5 deep-scrub ok 2024-09-10T18:31:18.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:18.490 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:18.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:18.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:18.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:18 smithi204 ceph-mon[32357]: 8.5 deep-scrub ok 2024-09-10T18:31:18.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:18.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:18.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:18.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:31:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:18.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-10T18:31:18.739 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:18.763 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:18.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:18.865 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:18.890 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:18.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:18.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:18.993 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:19.021 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:19.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:19.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:19.123 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:19.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:19.150 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:19.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:19.251 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:19.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:19.280 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:19.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:19.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:19.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:19.444 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:19.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:19.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:19 smithi028 ceph-mon[29291]: pgmap v439: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 710 B/s rd, 0 B/s wr, 783 op/s 2024-09-10T18:31:19.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:19 smithi028 ceph-mon[39220]: pgmap v439: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 710 B/s rd, 0 B/s wr, 783 op/s 2024-09-10T18:31:19.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:19.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:19.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:19 smithi204 ceph-mon[32357]: pgmap v439: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 710 B/s rd, 0 B/s wr, 783 op/s 2024-09-10T18:31:19.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:19.729 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:19.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:19.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:19.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:19.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:19.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:19.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:20.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:20.116 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:20.144 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:20.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:20.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:20.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:20.282 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:20.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:20.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:20.348 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.6 to 1 (10000) 2024-09-10T18:31:20.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:20.434 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:20.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:20.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:20 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1708056117' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:31:20.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:20 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1708056117' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:31:20.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:20.589 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:20.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:20.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:20 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1708056117' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:31:20.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:20.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:20.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:20.835 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:20.868 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:20.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:20.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:20.970 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:21.004 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:21.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:21.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:21.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:21.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:21.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:21.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-10T18:31:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:21.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:21.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:21.389 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:21.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:21.417 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:21.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:21.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:21 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1708056117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T18:31:21.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:21 smithi028 ceph-mon[29291]: osdmap e237: 8 total, 8 up, 8 in 2024-09-10T18:31:21.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:21 smithi028 ceph-mon[29291]: pgmap v441: 158 pgs: 158 active+clean; 1.3 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.64k op/s 2024-09-10T18:31:21.488 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:31:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:31:21] "GET /metrics HTTP/1.1" 200 224342 "" "Prometheus/2.43.0" 2024-09-10T18:31:21.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:21 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1708056117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T18:31:21.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:21 smithi028 ceph-mon[39220]: osdmap e237: 8 total, 8 up, 8 in 2024-09-10T18:31:21.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:21 smithi028 ceph-mon[39220]: pgmap v441: 158 pgs: 158 active+clean; 1.3 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.64k op/s 2024-09-10T18:31:21.519 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:21.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:21.551 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:21.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:21.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:21 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1708056117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T18:31:21.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:21 smithi204 ceph-mon[32357]: osdmap e237: 8 total, 8 up, 8 in 2024-09-10T18:31:21.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:21 smithi204 ceph-mon[32357]: pgmap v441: 158 pgs: 158 active+clean; 1.3 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.64k op/s 2024-09-10T18:31:21.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:21.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:21.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:21.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:21.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:21.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:21.949 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:21.978 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:21.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:21.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:22.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:22.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:22.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:22.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:22.267 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:22.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:22.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:22.406 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:22.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:22.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:22.505 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:31:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:31:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:31:22.507 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:22.539 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:22.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:22.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:22.623 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:31:22.623 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:31:22.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:22.653 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:31:22.653 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:31:22.684 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:31:22.685 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:31:22.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:22.689 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:22.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:22.724 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:31:22.724 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:31:22.752 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:31:22.752 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:31:22.783 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:31:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:31:22.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:22.815 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:31:22.815 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:31:22.830 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:22.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:22.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:22.846 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:31:22.846 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:31:22.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:22 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1935275996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:31:22.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:22.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:23.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:22 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1935275996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:31:23.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:22 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1935275996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:31:23.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:23.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:23.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:23.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:23.241 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:23.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:23.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:23.343 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:23.379 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:23.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:23.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:23.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:23.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:23.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:23.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:23.671 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:23.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:23.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:23.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:23.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:23.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:23.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:23.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:23.953 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:23.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:23.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:24.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:23 smithi028 ceph-mon[39220]: pgmap v442: 158 pgs: 158 active+clean; 1.3 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 664 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-10T18:31:24.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:23 smithi028 ceph-mon[29291]: pgmap v442: 158 pgs: 158 active+clean; 1.3 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 664 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-10T18:31:24.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:24.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:24.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:23 smithi204 ceph-mon[32357]: pgmap v442: 158 pgs: 158 active+clean; 1.3 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 664 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-10T18:31:24.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:24.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:24.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:24.332 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:24.360 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:24.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:24.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:24.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:24.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:24.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:24.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:24.599 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:24.633 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:24.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:24.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:24.735 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:24.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:24.767 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:24.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:24.777 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:31:24 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:31:24.868 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:24.897 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:24.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:24.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:25.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:25.034 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:25.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:25.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:24 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:31:25.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:31:25.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:24 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:31:25.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:31:25.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:24 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:31:25.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:31:25.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:25.182 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:25.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:25.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:25.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:25.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:25.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:25.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:25.467 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:25.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:25.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:25.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:25.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:25.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:25.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:25.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:25.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:25.893 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:25.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:25.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:25.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:26.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:25 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:26.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:25 smithi028 ceph-mon[39220]: pgmap v443: 158 pgs: 158 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 833 op/s 2024-09-10T18:31:26.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:25 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:26.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:25 smithi028 ceph-mon[29291]: pgmap v443: 158 pgs: 158 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 833 op/s 2024-09-10T18:31:26.023 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:26.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:26.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:26.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:25 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:26.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:25 smithi204 ceph-mon[32357]: pgmap v443: 158 pgs: 158 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 833 op/s 2024-09-10T18:31:26.125 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:26.157 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:26.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:26.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:26.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:26.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:26.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:26.404 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:26.431 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:26.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:26.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:26.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:26.546 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:31:26.546 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:31:26.576 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:26.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:26.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:26.678 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:26.705 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:26.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:26.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:26.807 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:26.833 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:26.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:26.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:26.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:26.975 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:26.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:26.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:27.076 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:27.103 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:27.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:27.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:27.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:27.240 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:27.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:27.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:27.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:27.387 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:27.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:27.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:27.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:27.525 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:27.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:27.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:27.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:27.673 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:27.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:27.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:27.776 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:27.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:31:27.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:27.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:27.918 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:27.946 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:27.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:27.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:31:28.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:28.110 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:28.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:28.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:28.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:28.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:28 smithi028 ceph-mon[39220]: pgmap v444: 158 pgs: 158 active+clean; 912 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-10T18:31:28.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:28 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3933363927' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:31:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:28 smithi028 ceph-mon[29291]: pgmap v444: 158 pgs: 158 active+clean; 912 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-10T18:31:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:28 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3933363927' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:31:28.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:28.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:28.331 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-10T18:31:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:28 smithi204 ceph-mon[32357]: pgmap v444: 158 pgs: 158 active+clean; 912 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-10T18:31:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:28 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3933363927' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:31:28.360 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:28.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:28.409 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:28.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:28.510 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:28.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:28.554 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:28.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:28.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:28.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:28.696 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:28.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:28.733 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:31:28.797 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:28.829 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:28.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:28.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:28.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:28.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:28.981 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:28.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:29.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:29.127 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:29.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:29.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:29.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:29 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:31:29.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:29 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3933363927' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:31:29.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:29 smithi028 ceph-mon[39220]: osdmap e238: 8 total, 8 up, 8 in 2024-09-10T18:31:29.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:29 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:31:29.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:29 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3933363927' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:31:29.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:29 smithi028 ceph-mon[29291]: osdmap e238: 8 total, 8 up, 8 in 2024-09-10T18:31:29.178 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:29 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:31:29.178 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:29 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3933363927' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:31:29.179 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:29 smithi204 ceph-mon[32357]: osdmap e238: 8 total, 8 up, 8 in 2024-09-10T18:31:29.229 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:29.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:29.268 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:29.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:29.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:29.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:29.507 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:29.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:29.534 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:29.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:29.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:29.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:29.684 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:29.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:29.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:29.828 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:29.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:29.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:29.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:29.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:29.968 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:29.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:30.069 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:30.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:30.096 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:30.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:30.147 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:30.197 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:30.239 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:30.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:30.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:30 smithi028 ceph-mon[29291]: pgmap v446: 158 pgs: 158 active+clean; 912 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 594 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-10T18:31:30.292 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:30 smithi028 ceph-mon[39220]: pgmap v446: 158 pgs: 158 active+clean; 912 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 594 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-10T18:31:30.310 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:30 smithi204 ceph-mon[32357]: pgmap v446: 158 pgs: 158 active+clean; 912 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 594 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-10T18:31:30.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:30.379 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:30.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:30.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:30.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:30.519 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:30.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:30.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:30.662 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:30.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:30.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:30.763 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:30.801 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:30.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:30.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:30.815 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:31:30.815 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:31:30.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:30.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:30.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:31.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:31.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:31.079 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:31.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:31.180 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:31.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:31.215 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:31.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:31.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:31.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:31.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:31.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:31.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:31.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:31.487 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:31.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:31.588 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:31.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:31.620 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:31.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:31.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:31:31.679 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:31:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:31:31] "GET /metrics HTTP/1.1" 200 224343 "" "Prometheus/2.43.0" 2024-09-10T18:31:31.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-10T18:31:31.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:31.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:31.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:31.845 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:31 smithi204 ceph-mon[32357]: pgmap v447: 158 pgs: 158 active+clean; 520 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-10T18:31:31.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:31.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:31.889 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:31.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:31.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:32.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:31 smithi028 ceph-mon[29291]: pgmap v447: 158 pgs: 158 active+clean; 520 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-10T18:31:32.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:31 smithi028 ceph-mon[39220]: pgmap v447: 158 pgs: 158 active+clean; 520 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-10T18:31:32.022 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:32.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:32.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:32.070 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:31:32.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:32.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:32.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:32.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:32.293 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:32.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:32.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:32.390 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:31:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:31:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:31:32.395 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:32.433 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:32.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:32.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:32.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:32.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:32.571 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:32.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:32.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:32.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:32.709 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:32.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:32.810 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:32.844 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:32.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:32.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:32.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:32.975 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:32.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:32.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:33.010 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:32 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:31:33.010 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:32 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:31:33.017 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:32 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:31:33.017 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:32 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:31:33.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:32 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:31:33.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:32 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:31:33.077 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:33.104 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:33.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:33.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:33.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:33.238 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:33.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:33.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:33.340 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:33.366 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:33.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:33.468 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:33.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:33.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:33.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:33.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:33.633 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:33.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:33.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:33.736 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:33.768 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:33.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:33.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:33.893 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:33.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:33.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:33.994 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:34.026 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:34.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:34.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:34.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:33 smithi204 ceph-mon[32357]: osdmap e239: 8 total, 8 up, 8 in 2024-09-10T18:31:34.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:33 smithi204 ceph-mon[32357]: pgmap v449: 158 pgs: 158 active+clean; 520 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T18:31:34.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:34.152 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:34.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:34.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:34.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:33 smithi028 ceph-mon[29291]: osdmap e239: 8 total, 8 up, 8 in 2024-09-10T18:31:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:33 smithi028 ceph-mon[29291]: pgmap v449: 158 pgs: 158 active+clean; 520 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T18:31:34.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:33 smithi028 ceph-mon[39220]: osdmap e239: 8 total, 8 up, 8 in 2024-09-10T18:31:34.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:33 smithi028 ceph-mon[39220]: pgmap v449: 158 pgs: 158 active+clean; 520 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T18:31:34.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:34.277 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:34.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:34.378 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:34.406 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:34.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:34.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:34.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:34.535 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:34.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:34.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:34.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:34.665 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:34.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:34.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:34.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:34.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:34.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:34.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:31:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:31:34.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:34.953 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:34.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:34.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:35.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:35.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:35.081 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:35.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:35.182 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:35.207 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:35.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:35.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:35.309 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:35.358 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:35.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:35.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:35.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:35.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:35.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:35.600 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:35.629 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:35.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:35.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:35.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:35.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:35.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:35.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:35.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:35.918 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:35.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:35.929 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:31:35.929 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:31:36.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:36.046 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:36.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:36.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:36.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:36 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:36.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:36 smithi028 ceph-mon[29291]: pgmap v450: 158 pgs: 158 active+clean; 433 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 957 op/s 2024-09-10T18:31:36.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:36 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:36.098 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:36 smithi028 ceph-mon[39220]: pgmap v450: 158 pgs: 158 active+clean; 433 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 957 op/s 2024-09-10T18:31:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:36 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:36 smithi204 ceph-mon[32357]: pgmap v450: 158 pgs: 158 active+clean; 433 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 957 op/s 2024-09-10T18:31:36.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:36.177 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:36.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:36.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:36.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:36.315 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:36.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:36.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:36.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:36.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:36.454 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:36.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:36.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:36.585 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:36.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:36.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:36.687 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:36.714 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:36.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:36.817 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:36.848 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:36.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:36.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:36.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-10T18:31:36.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:36.951 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:31:36.951 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:31:36.951 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:31:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:31:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:31:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:31:36.952 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:31:36.953 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:31:36.953 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:31:36.953 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:31:36.953 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:31:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:31:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:31:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:31:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:31:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:31:36.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:36.988 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:36.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:37.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:37.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:37.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:37.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:37.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:37.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:37 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/847450423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:31:37.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:37 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/847450423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:31:37.273 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:37.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:37.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:37.293 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:31:37.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:37 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/847450423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:31:37.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:37.411 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:37.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:37.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:37.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:37.558 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:37.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:37.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:37.696 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:37.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:37.733 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Removed 134218 objects 2024-09-10T18:31:37.733 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Clean up completed and total clean up time :120.601 2024-09-10T18:31:37.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:37.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:37.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:37.922 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:37.945 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:37.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:37.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.7 to 1 (10000) 2024-09-10T18:31:38.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:38.069 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:38.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:38.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:38.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:38.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:38.194 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:38.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:38.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:38.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:38.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:38 smithi204 ceph-mon[32357]: pgmap v451: 158 pgs: 158 active+clean; 179 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-10T18:31:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:38 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3178048679' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:31:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:38 smithi204 ceph-mon[32357]: from='client.35231 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:31:38.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:38 smithi028 ceph-mon[39220]: pgmap v451: 158 pgs: 158 active+clean; 179 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-10T18:31:38.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:38 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3178048679' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:31:38.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:38 smithi028 ceph-mon[39220]: from='client.35231 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:31:38.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:38 smithi028 ceph-mon[29291]: pgmap v451: 158 pgs: 158 active+clean; 179 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-10T18:31:38.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:38 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3178048679' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:31:38.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:38 smithi028 ceph-mon[29291]: from='client.35231 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:31:38.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:38.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:38.442 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:38.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:38.543 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:38.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:38.567 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:38.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:38.668 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:38.690 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:38.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:38.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:38.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:38.810 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-10T18:31:38.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:31:38.814 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:38.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:38.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:38.870 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:38.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:38.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:38.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:39.035 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:39 smithi028 ceph-mon[39220]: from='client.35231 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T18:31:39.036 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:39 smithi028 ceph-mon[39220]: osdmap e240: 8 total, 8 up, 8 in 2024-09-10T18:31:39.036 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:31:39.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:39 smithi028 ceph-mon[29291]: from='client.35231 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T18:31:39.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:39 smithi028 ceph-mon[29291]: osdmap e240: 8 total, 8 up, 8 in 2024-09-10T18:31:39.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:31:39.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:39.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:39.064 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:39.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:39.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:39 smithi204 ceph-mon[32357]: from='client.35231 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T18:31:39.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:39 smithi204 ceph-mon[32357]: osdmap e240: 8 total, 8 up, 8 in 2024-09-10T18:31:39.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:31:39.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:39.190 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:39.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:39.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:39.292 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:39.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:39.314 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:39.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:39.415 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:39.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:39.438 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:39.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:39.539 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:39.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:39.562 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:39.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:39.626 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:31:39.626 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-10T18:31:39.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-10T18:31:39.627 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:31:39.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.627 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:28631 7.1 7:88578cb4:::benchmark_data_smithi028_77856_object28630:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:31:39.627 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:55.725864+0000", 2024-09-10T18:31:39.627 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 463.87856340600001, 2024-09-10T18:31:39.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13970326299999999, 2024-09-10T18:31:39.627 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.627 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.627 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.627 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.627 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.627 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 28631 2024-09-10T18:31:39.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.628 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.725864+0000", 2024-09-10T18:31:39.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.725864+0000", 2024-09-10T18:31:39.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.725862+0000", 2024-09-10T18:31:39.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:31:39.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.725910+0000", 2024-09-10T18:31:39.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8566999999999998e-05 2024-09-10T18:31:39.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.725922+0000", 2024-09-10T18:31:39.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.137e-05 2024-09-10T18:31:39.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.630 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.725927+0000", 2024-09-10T18:31:39.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0459999999999996e-06 2024-09-10T18:31:39.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.630 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.725977+0000", 2024-09-10T18:31:39.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0581000000000002e-05 2024-09-10T18:31:39.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.630 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726106+0000", 2024-09-10T18:31:39.631 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012897900000000001 2024-09-10T18:31:39.631 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.631 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-10T18:31:39.631 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726265+0000", 2024-09-10T18:31:39.631 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015849700000000001 2024-09-10T18:31:39.631 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.631 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.631 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.728278+0000", 2024-09-10T18:31:39.631 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020135869999999998 2024-09-10T18:31:39.631 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.865431+0000", 2024-09-10T18:31:39.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.137152574 2024-09-10T18:31:39.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.865484+0000", 2024-09-10T18:31:39.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2617999999999997e-05 2024-09-10T18:31:39.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.865567+0000", 2024-09-10T18:31:39.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3238000000000005e-05 2024-09-10T18:31:39.633 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.633 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.633 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.633 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:28632 7.1 7:86538a1f:::benchmark_data_smithi028_77856_object28631:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:31:39.633 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:23:55.726101+0000", 2024-09-10T18:31:39.633 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 463.87832648800003, 2024-09-10T18:31:39.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13960805300000001, 2024-09-10T18:31:39.633 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.633 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.633 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.633 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.634 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.634 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 28632 2024-09-10T18:31:39.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.634 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726101+0000", 2024-09-10T18:31:39.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726101+0000", 2024-09-10T18:31:39.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726099+0000", 2024-09-10T18:31:39.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:31:39.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726156+0000", 2024-09-10T18:31:39.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6954999999999999e-05 2024-09-10T18:31:39.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726175+0000", 2024-09-10T18:31:39.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9415e-05 2024-09-10T18:31:39.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.636 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.636 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.726182+0000", 2024-09-10T18:31:39.636 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4359999999999999e-06 2024-09-10T18:31:39.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.636 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.636 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.727186+0000", 2024-09-10T18:31:39.636 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001004741 2024-09-10T18:31:39.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.637 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.637 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.637 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.727354+0000", 2024-09-10T18:31:39.637 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016707899999999999 2024-09-10T18:31:39.637 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.637 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.637 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-10T18:31:39.637 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.727503+0000", 2024-09-10T18:31:39.637 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014936600000000001 2024-09-10T18:31:39.637 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.637 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.637 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.637 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.728384+0000", 2024-09-10T18:31:39.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00088080000000000005 2024-09-10T18:31:39.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.865595+0000", 2024-09-10T18:31:39.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13721176099999999 2024-09-10T18:31:39.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.865639+0000", 2024-09-10T18:31:39.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3596000000000001e-05 2024-09-10T18:31:39.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:23:55.865709+0000", 2024-09-10T18:31:39.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9511000000000006e-05 2024-09-10T18:31:39.639 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.639 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.639 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.639 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.639 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:181520 7.1 7:822e6410:::benchmark_data_smithi028_77856_object60420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:31:39.639 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:25.451480+0000", 2024-09-10T18:31:39.639 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 314.152946628, 2024-09-10T18:31:39.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13309078699999999, 2024-09-10T18:31:39.639 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.639 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.639 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.640 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.640 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.640 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 181520 2024-09-10T18:31:39.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.640 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.640 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.640 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451480+0000", 2024-09-10T18:31:39.640 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.640 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.640 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451480+0000", 2024-09-10T18:31:39.640 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.641 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.641 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.641 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451478+0000", 2024-09-10T18:31:39.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:31:39.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.641 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.641 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.641 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451485+0000", 2024-09-10T18:31:39.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3570000000000003e-06 2024-09-10T18:31:39.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451495+0000", 2024-09-10T18:31:39.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0385e-05 2024-09-10T18:31:39.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451500+0000", 2024-09-10T18:31:39.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8389999999999997e-06 2024-09-10T18:31:39.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451540+0000", 2024-09-10T18:31:39.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0339000000000001e-05 2024-09-10T18:31:39.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451626+0000", 2024-09-10T18:31:39.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5860999999999997e-05 2024-09-10T18:31:39.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-10T18:31:39.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451694+0000", 2024-09-10T18:31:39.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8169000000000004e-05 2024-09-10T18:31:39.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.457845+0000", 2024-09-10T18:31:39.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006151045 2024-09-10T18:31:39.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.584485+0000", 2024-09-10T18:31:39.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.126639366 2024-09-10T18:31:39.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.584517+0000", 2024-09-10T18:31:39.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2372000000000001e-05 2024-09-10T18:31:39.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.645 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.645 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.584571+0000", 2024-09-10T18:31:39.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4137e-05 2024-09-10T18:31:39.645 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.645 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.645 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.645 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:181529 7.1 7:8089af12:::benchmark_data_smithi028_77856_object60429:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:31:39.645 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:25.453132+0000", 2024-09-10T18:31:39.645 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 314.15129519300001, 2024-09-10T18:31:39.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13400395500000001, 2024-09-10T18:31:39.645 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.646 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.646 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.646 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.646 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.646 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 181529 2024-09-10T18:31:39.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.646 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453132+0000", 2024-09-10T18:31:39.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453132+0000", 2024-09-10T18:31:39.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453131+0000", 2024-09-10T18:31:39.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:31:39.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453136+0000", 2024-09-10T18:31:39.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9910000000000002e-06 2024-09-10T18:31:39.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453144+0000", 2024-09-10T18:31:39.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7280000000000001e-06 2024-09-10T18:31:39.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453149+0000", 2024-09-10T18:31:39.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4479999999999996e-06 2024-09-10T18:31:39.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.456233+0000", 2024-09-10T18:31:39.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030846509999999999 2024-09-10T18:31:39.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.456312+0000", 2024-09-10T18:31:39.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8944999999999997e-05 2024-09-10T18:31:39.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-10T18:31:39.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.456378+0000", 2024-09-10T18:31:39.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5303999999999997e-05 2024-09-10T18:31:39.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.458107+0000", 2024-09-10T18:31:39.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0017296569999999999 2024-09-10T18:31:39.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.587061+0000", 2024-09-10T18:31:39.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.12895343500000001 2024-09-10T18:31:39.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.587089+0000", 2024-09-10T18:31:39.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7996000000000001e-05 2024-09-10T18:31:39.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.651 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.587136+0000", 2024-09-10T18:31:39.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6949000000000001e-05 2024-09-10T18:31:39.651 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.651 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.651 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.651 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192360 7.1 7:8a845e96:::benchmark_data_smithi028_77856_object71260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-10T18:31:39.651 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.176590+0000", 2024-09-10T18:31:39.651 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 301.42783661700003, 2024-09-10T18:31:39.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0444927079999999, 2024-09-10T18:31:39.652 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.652 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:31:39.652 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.652 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.652 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.652 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192360 2024-09-10T18:31:39.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.652 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176590+0000", 2024-09-10T18:31:39.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176590+0000", 2024-09-10T18:31:39.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176588+0000", 2024-09-10T18:31:39.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:31:39.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176595+0000", 2024-09-10T18:31:39.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7190000000000004e-06 2024-09-10T18:31:39.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176607+0000", 2024-09-10T18:31:39.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1936000000000001e-05 2024-09-10T18:31:39.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176612+0000", 2024-09-10T18:31:39.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2299999999999999e-06 2024-09-10T18:31:39.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176659+0000", 2024-09-10T18:31:39.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6782e-05 2024-09-10T18:31:39.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176790+0000", 2024-09-10T18:31:39.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013042800000000001 2024-09-10T18:31:39.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:31:39.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176921+0000", 2024-09-10T18:31:39.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000131552 2024-09-10T18:31:39.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.179313+0000", 2024-09-10T18:31:39.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023916839999999998 2024-09-10T18:31:39.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.180160+0000", 2024-09-10T18:31:39.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00084720399999999995 2024-09-10T18:31:39.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.185626+0000", 2024-09-10T18:31:39.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054663239999999998 2024-09-10T18:31:39.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:31:39.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.185637+0000", 2024-09-10T18:31:39.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.112e-05 2024-09-10T18:31:39.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.220975+0000", 2024-09-10T18:31:39.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0353376249999999 2024-09-10T18:31:39.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.221083+0000", 2024-09-10T18:31:39.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108022 2024-09-10T18:31:39.658 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.658 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.658 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.658 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192414 7.1 7:8900b3fc:::benchmark_data_smithi028_77856_object71314:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:31:39.658 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.234714+0000", 2024-09-10T18:31:39.658 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 301.369712795, 2024-09-10T18:31:39.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0099985309999999, 2024-09-10T18:31:39.658 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.658 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.658 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.659 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.659 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.659 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192414 2024-09-10T18:31:39.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.659 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.659 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.234714+0000", 2024-09-10T18:31:39.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.659 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.234714+0000", 2024-09-10T18:31:39.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.660 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.660 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.660 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.660 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.234712+0000", 2024-09-10T18:31:39.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:31:39.660 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.660 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.660 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.660 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.234720+0000", 2024-09-10T18:31:39.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8809999999999996e-06 2024-09-10T18:31:39.660 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.660 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.660 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.234731+0000", 2024-09-10T18:31:39.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1804000000000001e-05 2024-09-10T18:31:39.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.234737+0000", 2024-09-10T18:31:39.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3340000000000001e-06 2024-09-10T18:31:39.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.234776+0000", 2024-09-10T18:31:39.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9490999999999998e-05 2024-09-10T18:31:39.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:31:39.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.234786+0000", 2024-09-10T18:31:39.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5510000000000005e-06 2024-09-10T18:31:39.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.221104+0000", 2024-09-10T18:31:39.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98631840199999998 2024-09-10T18:31:39.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.221216+0000", 2024-09-10T18:31:39.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011218300000000001 2024-09-10T18:31:39.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:31:39.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.221336+0000", 2024-09-10T18:31:39.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011998900000000001 2024-09-10T18:31:39.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.228488+0000", 2024-09-10T18:31:39.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007151704 2024-09-10T18:31:39.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244439+0000", 2024-09-10T18:31:39.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015951402999999999 2024-09-10T18:31:39.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244633+0000", 2024-09-10T18:31:39.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019376699999999999 2024-09-10T18:31:39.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244679+0000", 2024-09-10T18:31:39.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5878999999999999e-05 2024-09-10T18:31:39.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244713+0000", 2024-09-10T18:31:39.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3683000000000003e-05 2024-09-10T18:31:39.665 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.665 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.665 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.666 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192418 7.1 7:82bfeb99:::benchmark_data_smithi028_77856_object71318:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:31:39.666 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.235688+0000", 2024-09-10T18:31:39.666 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 301.368739205, 2024-09-10T18:31:39.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.009211074, 2024-09-10T18:31:39.666 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.666 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.666 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.666 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.666 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.666 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192418 2024-09-10T18:31:39.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.666 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.235688+0000", 2024-09-10T18:31:39.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.235688+0000", 2024-09-10T18:31:39.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.235686+0000", 2024-09-10T18:31:39.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:31:39.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.235693+0000", 2024-09-10T18:31:39.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.905e-06 2024-09-10T18:31:39.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.235707+0000", 2024-09-10T18:31:39.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4069e-05 2024-09-10T18:31:39.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.235714+0000", 2024-09-10T18:31:39.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4189999999999999e-06 2024-09-10T18:31:39.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.235760+0000", 2024-09-10T18:31:39.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6245999999999997e-05 2024-09-10T18:31:39.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:31:39.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.235766+0000", 2024-09-10T18:31:39.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4740000000000002e-06 2024-09-10T18:31:39.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.223084+0000", 2024-09-10T18:31:39.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98731729499999998 2024-09-10T18:31:39.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.223188+0000", 2024-09-10T18:31:39.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0001043 2024-09-10T18:31:39.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:31:39.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.223302+0000", 2024-09-10T18:31:39.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000113992 2024-09-10T18:31:39.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.228494+0000", 2024-09-10T18:31:39.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005192543 2024-09-10T18:31:39.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244498+0000", 2024-09-10T18:31:39.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016003567999999999 2024-09-10T18:31:39.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244841+0000", 2024-09-10T18:31:39.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000342963 2024-09-10T18:31:39.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244868+0000", 2024-09-10T18:31:39.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7489999999999999e-05 2024-09-10T18:31:39.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244899+0000", 2024-09-10T18:31:39.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0375999999999999e-05 2024-09-10T18:31:39.672 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.672 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.672 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.672 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192458 7.1 7:825d7094:::benchmark_data_smithi028_77856_object71358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:31:39.673 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.267613+0000", 2024-09-10T18:31:39.673 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 301.33681432899999, 2024-09-10T18:31:39.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97748335900000005, 2024-09-10T18:31:39.673 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.673 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.673 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.673 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.673 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.673 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192458 2024-09-10T18:31:39.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.673 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.267613+0000", 2024-09-10T18:31:39.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.267613+0000", 2024-09-10T18:31:39.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.267611+0000", 2024-09-10T18:31:39.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:31:39.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.267618+0000", 2024-09-10T18:31:39.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1720000000000001e-06 2024-09-10T18:31:39.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.267628+0000", 2024-09-10T18:31:39.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9000000000000001e-06 2024-09-10T18:31:39.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.267633+0000", 2024-09-10T18:31:39.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4410000000000001e-06 2024-09-10T18:31:39.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.267677+0000", 2024-09-10T18:31:39.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3467999999999998e-05 2024-09-10T18:31:39.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:31:39.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.267683+0000", 2024-09-10T18:31:39.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6250000000000001e-06 2024-09-10T18:31:39.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.224848+0000", 2024-09-10T18:31:39.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95716446499999996 2024-09-10T18:31:39.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.224967+0000", 2024-09-10T18:31:39.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0001192 2024-09-10T18:31:39.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:31:39.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.225087+0000", 2024-09-10T18:31:39.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000120241 2024-09-10T18:31:39.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.233973+0000", 2024-09-10T18:31:39.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088862950000000007 2024-09-10T18:31:39.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244535+0000", 2024-09-10T18:31:39.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010561305 2024-09-10T18:31:39.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245035+0000", 2024-09-10T18:31:39.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00050065799999999996 2024-09-10T18:31:39.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245063+0000", 2024-09-10T18:31:39.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7438999999999999e-05 2024-09-10T18:31:39.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245096+0000", 2024-09-10T18:31:39.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3263999999999999e-05 2024-09-10T18:31:39.680 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.680 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.680 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.680 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192468 7.1 7:8dcd11d8:::benchmark_data_smithi028_77856_object71368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:31:39.680 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.280719+0000", 2024-09-10T18:31:39.680 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 301.32370824100002, 2024-09-10T18:31:39.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.96446010999999998, 2024-09-10T18:31:39.680 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.680 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.681 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.681 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.681 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.681 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192468 2024-09-10T18:31:39.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.681 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.280719+0000", 2024-09-10T18:31:39.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.280719+0000", 2024-09-10T18:31:39.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.280716+0000", 2024-09-10T18:31:39.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:31:39.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.280724+0000", 2024-09-10T18:31:39.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2220000000000002e-06 2024-09-10T18:31:39.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.280739+0000", 2024-09-10T18:31:39.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4776e-05 2024-09-10T18:31:39.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.280745+0000", 2024-09-10T18:31:39.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0580000000000003e-06 2024-09-10T18:31:39.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.280794+0000", 2024-09-10T18:31:39.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8605e-05 2024-09-10T18:31:39.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:31:39.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.280804+0000", 2024-09-10T18:31:39.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0166000000000001e-05 2024-09-10T18:31:39.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.226133+0000", 2024-09-10T18:31:39.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.94532860699999999 2024-09-10T18:31:39.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.226220+0000", 2024-09-10T18:31:39.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7226999999999997e-05 2024-09-10T18:31:39.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:31:39.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.226289+0000", 2024-09-10T18:31:39.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8881999999999995e-05 2024-09-10T18:31:39.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.233980+0000", 2024-09-10T18:31:39.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0076915480000000003 2024-09-10T18:31:39.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244574+0000", 2024-09-10T18:31:39.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010593981000000001 2024-09-10T18:31:39.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245133+0000", 2024-09-10T18:31:39.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00055907199999999998 2024-09-10T18:31:39.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245159+0000", 2024-09-10T18:31:39.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5573999999999999e-05 2024-09-10T18:31:39.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245179+0000", 2024-09-10T18:31:39.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9879999999999999e-05 2024-09-10T18:31:39.687 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.687 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.687 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.687 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192490 7.1 7:85c9b87e:::benchmark_data_smithi028_77856_object71390:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:31:39.687 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.297655+0000", 2024-09-10T18:31:39.687 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 301.30677186399998, 2024-09-10T18:31:39.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.947659696, 2024-09-10T18:31:39.687 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.687 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.687 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.688 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.688 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.688 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192490 2024-09-10T18:31:39.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.688 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.297655+0000", 2024-09-10T18:31:39.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.297655+0000", 2024-09-10T18:31:39.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.297653+0000", 2024-09-10T18:31:39.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:31:39.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.297661+0000", 2024-09-10T18:31:39.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3289999999999994e-06 2024-09-10T18:31:39.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.297676+0000", 2024-09-10T18:31:39.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4829e-05 2024-09-10T18:31:39.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.297682+0000", 2024-09-10T18:31:39.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5740000000000001e-06 2024-09-10T18:31:39.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.297729+0000", 2024-09-10T18:31:39.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6752999999999998e-05 2024-09-10T18:31:39.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:31:39.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.297738+0000", 2024-09-10T18:31:39.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5720000000000003e-06 2024-09-10T18:31:39.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.227394+0000", 2024-09-10T18:31:39.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92965577099999996 2024-09-10T18:31:39.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.227483+0000", 2024-09-10T18:31:39.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9287000000000001e-05 2024-09-10T18:31:39.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:31:39.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.227548+0000", 2024-09-10T18:31:39.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5151e-05 2024-09-10T18:31:39.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.238564+0000", 2024-09-10T18:31:39.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011016336999999999 2024-09-10T18:31:39.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244614+0000", 2024-09-10T18:31:39.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0060498039999999998 2024-09-10T18:31:39.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.693 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245259+0000", 2024-09-10T18:31:39.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00064463900000000002 2024-09-10T18:31:39.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.693 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245284+0000", 2024-09-10T18:31:39.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5096000000000002e-05 2024-09-10T18:31:39.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.693 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245315+0000", 2024-09-10T18:31:39.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0988999999999998e-05 2024-09-10T18:31:39.694 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.694 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.694 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.694 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192546 7.1 7:8c0c27ca:::benchmark_data_smithi028_77856_object71446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:31:39.694 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.351141+0000", 2024-09-10T18:31:39.694 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 301.25328622500001, 2024-09-10T18:31:39.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.89439569500000005, 2024-09-10T18:31:39.694 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.694 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.694 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.694 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.695 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.695 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192546 2024-09-10T18:31:39.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.695 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.695 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.695 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.351141+0000", 2024-09-10T18:31:39.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.695 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.695 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.351141+0000", 2024-09-10T18:31:39.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.351139+0000", 2024-09-10T18:31:39.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:31:39.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.351149+0000", 2024-09-10T18:31:39.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0575e-05 2024-09-10T18:31:39.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.351160+0000", 2024-09-10T18:31:39.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.094e-05 2024-09-10T18:31:39.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.351166+0000", 2024-09-10T18:31:39.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7810000000000002e-06 2024-09-10T18:31:39.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.351206+0000", 2024-09-10T18:31:39.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9694000000000003e-05 2024-09-10T18:31:39.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:31:39.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.351215+0000", 2024-09-10T18:31:39.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2189999999999993e-06 2024-09-10T18:31:39.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.228532+0000", 2024-09-10T18:31:39.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.87731700499999998 2024-09-10T18:31:39.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.228612+0000", 2024-09-10T18:31:39.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0439999999999996e-05 2024-09-10T18:31:39.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:31:39.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.228725+0000", 2024-09-10T18:31:39.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011289 2024-09-10T18:31:39.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.238572+0000", 2024-09-10T18:31:39.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098468070000000008 2024-09-10T18:31:39.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244766+0000", 2024-09-10T18:31:39.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061940500000000004 2024-09-10T18:31:39.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.700 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.700 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245469+0000", 2024-09-10T18:31:39.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00070266000000000003 2024-09-10T18:31:39.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.700 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.700 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245502+0000", 2024-09-10T18:31:39.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3154999999999997e-05 2024-09-10T18:31:39.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.700 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.701 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245536+0000", 2024-09-10T18:31:39.701 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4672999999999998e-05 2024-09-10T18:31:39.701 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.701 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.701 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.701 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.701 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.701 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192567 7.1 7:827122b7:::benchmark_data_smithi028_77856_object71467:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:31:39.701 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.367207+0000", 2024-09-10T18:31:39.701 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 301.23722014800001, 2024-09-10T18:31:39.701 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.87840781999999995, 2024-09-10T18:31:39.701 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.701 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.701 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.702 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.702 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.702 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192567 2024-09-10T18:31:39.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.702 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.367207+0000", 2024-09-10T18:31:39.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.367207+0000", 2024-09-10T18:31:39.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.367204+0000", 2024-09-10T18:31:39.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:31:39.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.367214+0000", 2024-09-10T18:31:39.705 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0013000000000001e-05 2024-09-10T18:31:39.705 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.705 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.705 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.705 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.367230+0000", 2024-09-10T18:31:39.705 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6504e-05 2024-09-10T18:31:39.705 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.705 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.705 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.705 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.367238+0000", 2024-09-10T18:31:39.705 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6599999999999995e-06 2024-09-10T18:31:39.705 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.705 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.705 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.706 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.367294+0000", 2024-09-10T18:31:39.706 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5603999999999998e-05 2024-09-10T18:31:39.706 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.706 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.706 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:31:39.706 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.367301+0000", 2024-09-10T18:31:39.706 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9650000000000002e-06 2024-09-10T18:31:39.706 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.706 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.706 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.706 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.230432+0000", 2024-09-10T18:31:39.706 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.86313160700000002 2024-09-10T18:31:39.706 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.706 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.707 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.707 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.230540+0000", 2024-09-10T18:31:39.707 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107299 2024-09-10T18:31:39.707 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.707 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.707 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:31:39.707 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.230656+0000", 2024-09-10T18:31:39.707 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116549 2024-09-10T18:31:39.707 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.707 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.707 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.707 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.238579+0000", 2024-09-10T18:31:39.707 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0079227970000000005 2024-09-10T18:31:39.708 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.708 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.708 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.708 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244792+0000", 2024-09-10T18:31:39.708 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0062134 2024-09-10T18:31:39.708 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.708 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.708 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.708 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245567+0000", 2024-09-10T18:31:39.708 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00077483000000000001 2024-09-10T18:31:39.709 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.709 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.709 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.709 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245593+0000", 2024-09-10T18:31:39.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6061e-05 2024-09-10T18:31:39.709 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.709 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.709 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.709 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245615+0000", 2024-09-10T18:31:39.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1498e-05 2024-09-10T18:31:39.709 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.709 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.709 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.710 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.710 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192573 7.1 7:84aa733b:::benchmark_data_smithi028_77856_object71473:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:31:39.710 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.372364+0000", 2024-09-10T18:31:39.710 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 301.23206308900001, 2024-09-10T18:31:39.710 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.87491753800000005, 2024-09-10T18:31:39.710 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.710 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.710 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.710 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.710 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.710 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192573 2024-09-10T18:31:39.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.711 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.711 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.711 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.711 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.372364+0000", 2024-09-10T18:31:39.711 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.711 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.711 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.711 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.711 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.372364+0000", 2024-09-10T18:31:39.711 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.711 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.711 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.711 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.372362+0000", 2024-09-10T18:31:39.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:31:39.712 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.712 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.712 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.372370+0000", 2024-09-10T18:31:39.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6400000000000003e-06 2024-09-10T18:31:39.712 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.712 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.712 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.372385+0000", 2024-09-10T18:31:39.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4629e-05 2024-09-10T18:31:39.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.713 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.713 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.372391+0000", 2024-09-10T18:31:39.713 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.28e-06 2024-09-10T18:31:39.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.713 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.713 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.372440+0000", 2024-09-10T18:31:39.713 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9190000000000002e-05 2024-09-10T18:31:39.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:39.715 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.715 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:31:39.715 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.372448+0000", 2024-09-10T18:31:39.715 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2640000000000002e-06 2024-09-10T18:31:39.715 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.715 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.715 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.716 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.232611+0000", 2024-09-10T18:31:39.716 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.86016320099999999 2024-09-10T18:31:39.716 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.716 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.716 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.716 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.232689+0000", 2024-09-10T18:31:39.716 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8687000000000001e-05 2024-09-10T18:31:39.716 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.716 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.716 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:31:39.716 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.232757+0000", 2024-09-10T18:31:39.716 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7353000000000004e-05 2024-09-10T18:31:39.716 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.717 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.717 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.717 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.241251+0000", 2024-09-10T18:31:39.717 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0084944689999999993 2024-09-10T18:31:39.717 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.717 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.717 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.717 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244818+0000", 2024-09-10T18:31:39.717 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035666539999999998 2024-09-10T18:31:39.717 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.717 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.717 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.717 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.247229+0000", 2024-09-10T18:31:39.717 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024109600000000002 2024-09-10T18:31:39.718 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.718 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.718 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.718 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.247261+0000", 2024-09-10T18:31:39.718 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2400000000000001e-05 2024-09-10T18:31:39.718 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.718 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.718 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.718 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.247281+0000", 2024-09-10T18:31:39.718 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0166999999999999e-05 2024-09-10T18:31:39.718 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.718 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.718 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.718 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.719 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.719 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192579 7.1 7:86f625e9:::benchmark_data_smithi028_77856_object71479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:31:39.719 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.377253+0000", 2024-09-10T18:31:39.719 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 301.22717443599998, 2024-09-10T18:31:39.719 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.87009756199999999, 2024-09-10T18:31:39.719 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.719 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.719 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.719 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.719 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.719 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192579 2024-09-10T18:31:39.720 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.720 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.720 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.720 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.720 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.377253+0000", 2024-09-10T18:31:39.720 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.720 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.720 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.720 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.720 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.377253+0000", 2024-09-10T18:31:39.720 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.720 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.720 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.721 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.721 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.377250+0000", 2024-09-10T18:31:39.721 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:31:39.721 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.721 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.721 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.721 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.377261+0000", 2024-09-10T18:31:39.721 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0648999999999999e-05 2024-09-10T18:31:39.721 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.721 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.721 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.721 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.377277+0000", 2024-09-10T18:31:39.721 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6158e-05 2024-09-10T18:31:39.721 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.722 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.722 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.722 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.377283+0000", 2024-09-10T18:31:39.722 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7370000000000003e-06 2024-09-10T18:31:39.722 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.722 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.722 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.722 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.377362+0000", 2024-09-10T18:31:39.722 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9010000000000004e-05 2024-09-10T18:31:39.722 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.722 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.722 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:31:39.722 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.377370+0000", 2024-09-10T18:31:39.722 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.79e-06 2024-09-10T18:31:39.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.723 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.723 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.723 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.234019+0000", 2024-09-10T18:31:39.723 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.85664904799999997 2024-09-10T18:31:39.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.723 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.723 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.723 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.234086+0000", 2024-09-10T18:31:39.723 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7403000000000005e-05 2024-09-10T18:31:39.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.723 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.723 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:31:39.723 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.234153+0000", 2024-09-10T18:31:39.724 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6556000000000004e-05 2024-09-10T18:31:39.724 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.724 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.724 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.724 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.241258+0000", 2024-09-10T18:31:39.724 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007105218 2024-09-10T18:31:39.724 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.724 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.724 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.724 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244949+0000", 2024-09-10T18:31:39.724 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003690652 2024-09-10T18:31:39.724 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.724 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.724 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.725 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.247309+0000", 2024-09-10T18:31:39.725 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023603790000000001 2024-09-10T18:31:39.725 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.725 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.725 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.725 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.247332+0000", 2024-09-10T18:31:39.725 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3147999999999999e-05 2024-09-10T18:31:39.725 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.725 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.725 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.725 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.247350+0000", 2024-09-10T18:31:39.725 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.808e-05 2024-09-10T18:31:39.725 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.725 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.726 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.726 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.726 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.726 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192599 7.1 7:83571491:::benchmark_data_smithi028_77856_object71499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:31:39.726 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.395289+0000", 2024-09-10T18:31:39.726 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 301.20913782999997, 2024-09-10T18:31:39.726 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.85726006200000004, 2024-09-10T18:31:39.726 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.726 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.726 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.726 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.726 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.726 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192599 2024-09-10T18:31:39.726 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.727 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.727 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.727 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.727 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.395289+0000", 2024-09-10T18:31:39.727 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.727 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.727 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.727 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.727 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.395289+0000", 2024-09-10T18:31:39.727 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.727 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.727 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.727 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.727 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.395287+0000", 2024-09-10T18:31:39.728 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:31:39.728 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.728 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.728 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.728 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.395295+0000", 2024-09-10T18:31:39.728 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3599999999999996e-06 2024-09-10T18:31:39.728 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.728 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.728 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.728 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.395308+0000", 2024-09-10T18:31:39.728 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3115999999999999e-05 2024-09-10T18:31:39.728 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.728 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.728 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.729 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.395315+0000", 2024-09-10T18:31:39.729 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5849999999999996e-06 2024-09-10T18:31:39.729 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.729 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.729 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.729 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.395386+0000", 2024-09-10T18:31:39.729 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1477999999999996e-05 2024-09-10T18:31:39.729 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.729 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.729 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:31:39.729 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.395395+0000", 2024-09-10T18:31:39.729 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2719999999999997e-06 2024-09-10T18:31:39.729 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.730 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.730 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.235362+0000", 2024-09-10T18:31:39.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.83996730600000002 2024-09-10T18:31:39.730 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.730 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.730 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.235439+0000", 2024-09-10T18:31:39.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6870000000000001e-05 2024-09-10T18:31:39.730 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.730 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:31:39.730 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.235501+0000", 2024-09-10T18:31:39.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2680000000000003e-05 2024-09-10T18:31:39.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.731 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.731 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.241264+0000", 2024-09-10T18:31:39.731 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0057622319999999999 2024-09-10T18:31:39.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.731 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.731 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244990+0000", 2024-09-10T18:31:39.731 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0037264389999999998 2024-09-10T18:31:39.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.731 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.731 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252467+0000", 2024-09-10T18:31:39.732 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0074771150000000003 2024-09-10T18:31:39.732 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.732 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.732 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.732 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252511+0000", 2024-09-10T18:31:39.732 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3251999999999999e-05 2024-09-10T18:31:39.732 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.732 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.732 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.732 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252549+0000", 2024-09-10T18:31:39.732 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8717000000000002e-05 2024-09-10T18:31:39.732 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.732 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.733 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.733 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.733 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.733 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192616 7.1 7:842d8045:::benchmark_data_smithi028_77856_object71516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:31:39.733 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.412146+0000", 2024-09-10T18:31:39.733 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 301.19228119600001, 2024-09-10T18:31:39.733 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.84049511600000004, 2024-09-10T18:31:39.733 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.733 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.733 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.733 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.733 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.733 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192616 2024-09-10T18:31:39.733 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.734 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.734 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.734 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.412146+0000", 2024-09-10T18:31:39.734 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.734 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.734 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.734 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.412146+0000", 2024-09-10T18:31:39.734 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.734 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.734 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.412143+0000", 2024-09-10T18:31:39.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:31:39.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.412151+0000", 2024-09-10T18:31:39.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7749999999999993e-06 2024-09-10T18:31:39.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.412166+0000", 2024-09-10T18:31:39.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4347e-05 2024-09-10T18:31:39.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.736 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.736 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.412171+0000", 2024-09-10T18:31:39.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7599999999999999e-06 2024-09-10T18:31:39.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.736 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.736 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.412207+0000", 2024-09-10T18:31:39.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5488000000000003e-05 2024-09-10T18:31:39.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.736 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:31:39.736 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.412214+0000", 2024-09-10T18:31:39.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3010000000000001e-06 2024-09-10T18:31:39.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.737 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.737 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.737 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.236816+0000", 2024-09-10T18:31:39.737 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.82460156100000004 2024-09-10T18:31:39.737 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.737 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.737 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.737 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.236904+0000", 2024-09-10T18:31:39.737 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8072000000000007e-05 2024-09-10T18:31:39.737 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.737 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.737 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:31:39.737 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.236968+0000", 2024-09-10T18:31:39.737 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4579999999999995e-05 2024-09-10T18:31:39.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.738 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.738 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.738 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244125+0000", 2024-09-10T18:31:39.738 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0071561769999999997 2024-09-10T18:31:39.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.738 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.738 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.738 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245213+0000", 2024-09-10T18:31:39.738 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001088971 2024-09-10T18:31:39.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.738 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.738 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.738 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252590+0000", 2024-09-10T18:31:39.739 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0073765030000000004 2024-09-10T18:31:39.739 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.739 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.739 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.739 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252618+0000", 2024-09-10T18:31:39.739 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8408e-05 2024-09-10T18:31:39.739 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.739 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.739 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.739 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252641+0000", 2024-09-10T18:31:39.739 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2498999999999999e-05 2024-09-10T18:31:39.739 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.739 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.739 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.740 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.740 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.740 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192633 7.1 7:869757c6:::benchmark_data_smithi028_77856_object71533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:31:39.740 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.436867+0000", 2024-09-10T18:31:39.740 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 301.16755966400001, 2024-09-10T18:31:39.740 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.815892744, 2024-09-10T18:31:39.740 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.740 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.740 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.740 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.740 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.740 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192633 2024-09-10T18:31:39.740 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.741 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.436867+0000", 2024-09-10T18:31:39.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.436867+0000", 2024-09-10T18:31:39.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.436865+0000", 2024-09-10T18:31:39.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:31:39.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.436878+0000", 2024-09-10T18:31:39.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.237e-05 2024-09-10T18:31:39.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.436890+0000", 2024-09-10T18:31:39.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1688999999999999e-05 2024-09-10T18:31:39.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.743 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.436894+0000", 2024-09-10T18:31:39.743 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5390000000000001e-06 2024-09-10T18:31:39.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.743 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.743 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.436930+0000", 2024-09-10T18:31:39.743 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6236999999999999e-05 2024-09-10T18:31:39.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.743 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:31:39.743 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.436937+0000", 2024-09-10T18:31:39.743 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7689999999999999e-06 2024-09-10T18:31:39.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.744 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.744 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.744 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.238619+0000", 2024-09-10T18:31:39.744 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80168163400000003 2024-09-10T18:31:39.744 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.744 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.744 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.744 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.238725+0000", 2024-09-10T18:31:39.744 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010587 2024-09-10T18:31:39.744 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.744 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.744 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:31:39.744 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.238846+0000", 2024-09-10T18:31:39.745 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000121402 2024-09-10T18:31:39.745 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.745 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.745 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.745 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244131+0000", 2024-09-10T18:31:39.745 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005285334 2024-09-10T18:31:39.745 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.745 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.745 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.745 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245237+0000", 2024-09-10T18:31:39.745 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011056600000000001 2024-09-10T18:31:39.745 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.745 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.745 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252719+0000", 2024-09-10T18:31:39.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007482201 2024-09-10T18:31:39.746 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.746 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252743+0000", 2024-09-10T18:31:39.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4221999999999999e-05 2024-09-10T18:31:39.746 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.746 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252760+0000", 2024-09-10T18:31:39.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6670000000000001e-05 2024-09-10T18:31:39.746 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.747 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.747 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.747 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.747 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192640 7.1 7:8b08cdd5:::benchmark_data_smithi028_77856_object71540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:31:39.747 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.443619+0000", 2024-09-10T18:31:39.747 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 301.16080806999997, 2024-09-10T18:31:39.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.82085115399999997, 2024-09-10T18:31:39.747 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.747 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.747 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.747 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.747 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.747 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192640 2024-09-10T18:31:39.748 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.748 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.748 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.748 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.443619+0000", 2024-09-10T18:31:39.748 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.748 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.748 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.748 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.443619+0000", 2024-09-10T18:31:39.748 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.748 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.443616+0000", 2024-09-10T18:31:39.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:31:39.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.443625+0000", 2024-09-10T18:31:39.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8899999999999996e-06 2024-09-10T18:31:39.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.443641+0000", 2024-09-10T18:31:39.750 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6076999999999999e-05 2024-09-10T18:31:39.750 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.750 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.750 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.750 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.443648+0000", 2024-09-10T18:31:39.750 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9519999999999997e-06 2024-09-10T18:31:39.750 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.750 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.750 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.750 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.443687+0000", 2024-09-10T18:31:39.750 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9224000000000003e-05 2024-09-10T18:31:39.750 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.750 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.751 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:31:39.751 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.443695+0000", 2024-09-10T18:31:39.751 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4669999999999998e-06 2024-09-10T18:31:39.751 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.751 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.751 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.751 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.240084+0000", 2024-09-10T18:31:39.751 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.79638984999999995 2024-09-10T18:31:39.751 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.751 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.751 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.751 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.240166+0000", 2024-09-10T18:31:39.751 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1206999999999997e-05 2024-09-10T18:31:39.751 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.752 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.752 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:31:39.752 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.240231+0000", 2024-09-10T18:31:39.752 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5543000000000006e-05 2024-09-10T18:31:39.752 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.752 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.752 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.752 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244229+0000", 2024-09-10T18:31:39.752 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039977229999999999 2024-09-10T18:31:39.752 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.752 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.752 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.752 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245370+0000", 2024-09-10T18:31:39.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011409199999999999 2024-09-10T18:31:39.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.753 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.753 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264392+0000", 2024-09-10T18:31:39.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01902216 2024-09-10T18:31:39.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.753 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.753 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264429+0000", 2024-09-10T18:31:39.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6761000000000003e-05 2024-09-10T18:31:39.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.754 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264470+0000", 2024-09-10T18:31:39.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1295999999999999e-05 2024-09-10T18:31:39.754 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.754 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.754 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.754 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.754 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192653 7.1 7:8f4ef615:::benchmark_data_smithi028_77856_object71553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:31:39.754 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.468961+0000", 2024-09-10T18:31:39.754 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 301.13546581999998, 2024-09-10T18:31:39.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.79562034500000001, 2024-09-10T18:31:39.754 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.754 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.755 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.755 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.755 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.755 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192653 2024-09-10T18:31:39.755 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.755 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.755 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.755 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.755 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.468961+0000", 2024-09-10T18:31:39.755 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.755 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.755 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.755 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.755 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.468961+0000", 2024-09-10T18:31:39.756 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.756 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.468958+0000", 2024-09-10T18:31:39.756 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:31:39.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.756 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.468968+0000", 2024-09-10T18:31:39.756 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0129000000000001e-05 2024-09-10T18:31:39.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.757 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.468984+0000", 2024-09-10T18:31:39.757 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5835999999999999e-05 2024-09-10T18:31:39.757 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.757 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.757 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.757 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.468992+0000", 2024-09-10T18:31:39.757 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7319999999999998e-06 2024-09-10T18:31:39.757 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.757 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.757 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.757 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.469044+0000", 2024-09-10T18:31:39.757 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2101999999999997e-05 2024-09-10T18:31:39.757 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.758 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.758 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:31:39.758 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.469055+0000", 2024-09-10T18:31:39.758 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0757999999999999e-05 2024-09-10T18:31:39.758 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.758 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.758 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.758 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.241306+0000", 2024-09-10T18:31:39.758 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.77225105699999996 2024-09-10T18:31:39.758 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.758 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.758 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.758 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.241417+0000", 2024-09-10T18:31:39.758 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011148499999999999 2024-09-10T18:31:39.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.759 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.759 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:31:39.759 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.241494+0000", 2024-09-10T18:31:39.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6364000000000003e-05 2024-09-10T18:31:39.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.759 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.759 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.759 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244234+0000", 2024-09-10T18:31:39.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027403029999999999 2024-09-10T18:31:39.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.759 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.759 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.760 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245402+0000", 2024-09-10T18:31:39.760 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001168475 2024-09-10T18:31:39.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.760 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.760 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264519+0000", 2024-09-10T18:31:39.760 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019116528000000001 2024-09-10T18:31:39.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.760 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.760 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264560+0000", 2024-09-10T18:31:39.760 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1043e-05 2024-09-10T18:31:39.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.761 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264582+0000", 2024-09-10T18:31:39.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1518999999999998e-05 2024-09-10T18:31:39.761 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.761 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.761 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.761 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192692 7.1 7:844b92c8:::benchmark_data_smithi028_77856_object71592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:31:39.761 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.589431+0000", 2024-09-10T18:31:39.761 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 301.01499556700003, 2024-09-10T18:31:39.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.67523672300000004, 2024-09-10T18:31:39.761 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:31:39.762 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:31:39.762 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:31:39.762 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:31:39.762 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:31:39.762 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192692 2024-09-10T18:31:39.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.762 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:31:39.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:31:39.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.589431+0000", 2024-09-10T18:31:39.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:31:39.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.589431+0000", 2024-09-10T18:31:39.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:31:39.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:31:39.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.589430+0000", 2024-09-10T18:31:39.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:31:39.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:31:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.589436+0000", 2024-09-10T18:31:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7690000000000002e-06 2024-09-10T18:31:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:31:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.589446+0000", 2024-09-10T18:31:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0214000000000001e-05 2024-09-10T18:31:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:31:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.589450+0000", 2024-09-10T18:31:39.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2209999999999999e-06 2024-09-10T18:31:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.589485+0000", 2024-09-10T18:31:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5308999999999997e-05 2024-09-10T18:31:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:31:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.589491+0000", 2024-09-10T18:31:39.765 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7320000000000003e-06 2024-09-10T18:31:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:31:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.242569+0000", 2024-09-10T18:31:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.65307782199999997 2024-09-10T18:31:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:31:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.242658+0000", 2024-09-10T18:31:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8823e-05 2024-09-10T18:31:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:31:39.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.242757+0000", 2024-09-10T18:31:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8987000000000006e-05 2024-09-10T18:31:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:31:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244448+0000", 2024-09-10T18:31:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016910569999999999 2024-09-10T18:31:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245435+0000", 2024-09-10T18:31:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00098748499999999997 2024-09-10T18:31:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:31:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264619+0000", 2024-09-10T18:31:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019183861 2024-09-10T18:31:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:31:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264645+0000", 2024-09-10T18:31:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6288999999999999e-05 2024-09-10T18:31:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:31:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:31:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:31:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264668+0000", 2024-09-10T18:31:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2674e-05 2024-09-10T18:31:39.768 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:31:39.769 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:31:39.769 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:31:39.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:39.792 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:39.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:39.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:39.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:39.917 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:39.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:40.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:40.041 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:40.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:40.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:31:40.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:40.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:40.168 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:40.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:40.269 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:40.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:40.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:40.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:40 smithi204 ceph-mon[32357]: pgmap v453: 158 pgs: 158 active+clean; 179 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 682 op/s 2024-09-10T18:31:40.375 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:40 smithi028 ceph-mon[39220]: pgmap v453: 158 pgs: 158 active+clean; 179 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 682 op/s 2024-09-10T18:31:40.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:40 smithi028 ceph-mon[29291]: pgmap v453: 158 pgs: 158 active+clean; 179 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 682 op/s 2024-09-10T18:31:40.376 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:40.394 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:40.417 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:40.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:40.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:40.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:40.542 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:40.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:40.606 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-10T18:31:40.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-10T18:31:40.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:40.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:40.674 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:40.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:40.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:40.799 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:40.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:40.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:40.901 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:40.924 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:40.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:41.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-10T18:31:41.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:41.040 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:31:41.079 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:41.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:41.181 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:41.206 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:41.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:41.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:41.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:41 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:31:41.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:41 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:31:41.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:41 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4043721822' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:31:41.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:41 smithi028 ceph-mon[39220]: from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:31:41.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:41 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:31:41.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:41 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:31:41.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:41 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4043721822' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:31:41.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:41 smithi028 ceph-mon[29291]: from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:31:41.269 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:31:41.309 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:41.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:41.332 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:41.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:41 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:31:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:41 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:31:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:41 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4043721822' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:31:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:41 smithi204 ceph-mon[32357]: from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:31:41.435 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:41.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:41.459 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:41.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:41.557 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:31:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:31:41] "GET /metrics HTTP/1.1" 200 224321 "" "Prometheus/2.43.0" 2024-09-10T18:31:41.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:41.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:41.590 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:41.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:41.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:41.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:41.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:41.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:41.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:41.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:41.974 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:41.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:42.021 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:42.021 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:42.077 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:42.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:42.103 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:42.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:42.205 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:42.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:42.228 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:42.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:42 smithi028 ceph-mon[29291]: pgmap v454: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-09-10T18:31:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:42 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:31:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:42 smithi028 ceph-mon[29291]: from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:31:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:42 smithi028 ceph-mon[29291]: osdmap e241: 8 total, 8 up, 8 in 2024-09-10T18:31:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:42 smithi028 ceph-mon[29291]: 3.c scrub starts 2024-09-10T18:31:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:42 smithi028 ceph-mon[29291]: 3.c scrub ok 2024-09-10T18:31:42.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:42 smithi028 ceph-mon[39220]: pgmap v454: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-09-10T18:31:42.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:42 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:31:42.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:42 smithi028 ceph-mon[39220]: from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:31:42.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:42 smithi028 ceph-mon[39220]: osdmap e241: 8 total, 8 up, 8 in 2024-09-10T18:31:42.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:42 smithi028 ceph-mon[39220]: 3.c scrub starts 2024-09-10T18:31:42.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:42 smithi028 ceph-mon[39220]: 3.c scrub ok 2024-09-10T18:31:42.329 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:42.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:42.352 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:42.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:42.357 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:31:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:31:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:31:42.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:42 smithi204 ceph-mon[32357]: pgmap v454: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-09-10T18:31:42.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:42 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:31:42.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:42 smithi204 ceph-mon[32357]: from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:31:42.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:42 smithi204 ceph-mon[32357]: osdmap e241: 8 total, 8 up, 8 in 2024-09-10T18:31:42.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:42 smithi204 ceph-mon[32357]: 3.c scrub starts 2024-09-10T18:31:42.358 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:42 smithi204 ceph-mon[32357]: 3.c scrub ok 2024-09-10T18:31:42.462 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:42.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:42.485 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:42.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:42.586 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:42.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:42.611 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:42.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:42.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:42.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:42.734 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:42.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:42.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:42.858 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:42.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:42.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:42.939 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:31:42.939 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:31:42.939 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-10T18:31:42.940 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:31:42.960 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:42.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:42.983 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:42.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:43.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:43.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:43.107 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:43.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:43.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:43.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:43.231 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:43.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:43.332 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:43.344 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:43 smithi028 ceph-mon[39220]: 4.d scrub starts 2024-09-10T18:31:43.344 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:43 smithi028 ceph-mon[39220]: 4.d scrub ok 2024-09-10T18:31:43.344 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:43 smithi028 ceph-mon[39220]: 1.7 scrub starts 2024-09-10T18:31:43.344 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:43 smithi028 ceph-mon[39220]: 1.7 scrub ok 2024-09-10T18:31:43.344 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:43 smithi028 ceph-mon[29291]: 4.d scrub starts 2024-09-10T18:31:43.344 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:43 smithi028 ceph-mon[29291]: 4.d scrub ok 2024-09-10T18:31:43.344 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:43 smithi028 ceph-mon[29291]: 1.7 scrub starts 2024-09-10T18:31:43.344 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:43 smithi028 ceph-mon[29291]: 1.7 scrub ok 2024-09-10T18:31:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:43 smithi204 ceph-mon[32357]: 4.d scrub starts 2024-09-10T18:31:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:43 smithi204 ceph-mon[32357]: 4.d scrub ok 2024-09-10T18:31:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:43 smithi204 ceph-mon[32357]: 1.7 scrub starts 2024-09-10T18:31:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:43 smithi204 ceph-mon[32357]: 1.7 scrub ok 2024-09-10T18:31:43.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:43.355 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:43.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:43.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:43.478 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:43.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:43.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:43.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:43.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:43.605 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:43.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:43.630 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-10T18:31:43.707 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:43.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:43.731 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:43.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:43.832 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:43.854 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:43.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:43.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:43.883 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:31:43.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:43.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:31:43.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:43.980 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:43.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:44.081 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:44.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:44.104 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:44.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:44.205 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:44.226 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:44.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:44.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:44 smithi028 ceph-mon[29291]: 4.18 scrub starts 2024-09-10T18:31:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:44 smithi028 ceph-mon[29291]: 4.18 scrub ok 2024-09-10T18:31:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:44 smithi028 ceph-mon[29291]: pgmap v456: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 954 op/s 2024-09-10T18:31:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:44 smithi028 ceph-mon[29291]: 8.8 scrub starts 2024-09-10T18:31:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:44 smithi028 ceph-mon[29291]: 8.8 scrub ok 2024-09-10T18:31:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:44 smithi028 ceph-mon[29291]: 3.18 scrub starts 2024-09-10T18:31:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:44 smithi028 ceph-mon[29291]: 3.18 scrub ok 2024-09-10T18:31:44.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:44 smithi028 ceph-mon[39220]: 4.18 scrub starts 2024-09-10T18:31:44.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:44 smithi028 ceph-mon[39220]: 4.18 scrub ok 2024-09-10T18:31:44.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:44 smithi028 ceph-mon[39220]: pgmap v456: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 954 op/s 2024-09-10T18:31:44.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:44 smithi028 ceph-mon[39220]: 8.8 scrub starts 2024-09-10T18:31:44.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:44 smithi028 ceph-mon[39220]: 8.8 scrub ok 2024-09-10T18:31:44.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:44 smithi028 ceph-mon[39220]: 3.18 scrub starts 2024-09-10T18:31:44.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:44 smithi028 ceph-mon[39220]: 3.18 scrub ok 2024-09-10T18:31:44.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:44.351 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:44 smithi204 ceph-mon[32357]: 4.18 scrub starts 2024-09-10T18:31:44.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:44 smithi204 ceph-mon[32357]: 4.18 scrub ok 2024-09-10T18:31:44.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:44 smithi204 ceph-mon[32357]: pgmap v456: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 954 op/s 2024-09-10T18:31:44.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:44 smithi204 ceph-mon[32357]: 8.8 scrub starts 2024-09-10T18:31:44.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:44 smithi204 ceph-mon[32357]: 8.8 scrub ok 2024-09-10T18:31:44.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:44 smithi204 ceph-mon[32357]: 3.18 scrub starts 2024-09-10T18:31:44.352 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:44 smithi204 ceph-mon[32357]: 3.18 scrub ok 2024-09-10T18:31:44.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:44.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:44.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:44.454 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:44.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:44.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:44.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:44.558 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:31:44.561 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:31:44.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:44.598 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:44.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:44.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:44.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:31:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:31:44.701 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:44.723 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:44.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:44.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:44.825 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:44.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:44.850 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:44.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:44.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:44.975 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:44.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:44.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:45.097 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:45.119 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:45.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:45.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:45.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:45.242 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:45.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:45.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:45.323 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:45 smithi204 ceph-mon[32357]: 6.0 scrub starts 2024-09-10T18:31:45.323 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:45 smithi204 ceph-mon[32357]: 6.0 scrub ok 2024-09-10T18:31:45.323 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:45 smithi204 ceph-mon[32357]: 5.13 scrub starts 2024-09-10T18:31:45.323 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:45 smithi204 ceph-mon[32357]: 5.13 scrub ok 2024-09-10T18:31:45.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:45.368 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:45.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:45.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:45.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:45 smithi028 ceph-mon[29291]: 6.0 scrub starts 2024-09-10T18:31:45.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:45 smithi028 ceph-mon[29291]: 6.0 scrub ok 2024-09-10T18:31:45.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:45 smithi028 ceph-mon[29291]: 5.13 scrub starts 2024-09-10T18:31:45.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:45 smithi028 ceph-mon[29291]: 5.13 scrub ok 2024-09-10T18:31:45.380 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:45 smithi028 ceph-mon[39220]: 6.0 scrub starts 2024-09-10T18:31:45.380 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:45 smithi028 ceph-mon[39220]: 6.0 scrub ok 2024-09-10T18:31:45.380 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:45 smithi028 ceph-mon[39220]: 5.13 scrub starts 2024-09-10T18:31:45.380 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:45 smithi028 ceph-mon[39220]: 5.13 scrub ok 2024-09-10T18:31:45.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:45.493 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:45.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:45.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:45.528 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":456,"stamp":"2024-09-10T18:31:42.955272+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1751,"num_read_kb":1503,"num_write":269154,"num_write_kb":8589996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82609,"ondisk_log_size":82609,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2449752,"kb_used_data":7428,"kb_used_omap":19,"kb_used_meta":2442220,"kb_avail":747511464,"statfs":{"total":767960285184,"available":765451739136,"internally_reserved":0,"allocated":7606272,"data_stored":4696220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20033,"internal_metadata":2500833727},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-453574680,"num_objects":-6922,"num_object_clones":0,"num_object_copies":-20766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6922,"num_whiteouts":0,"num_read":434,"num_read_kb":434,"num_write":7208,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002488"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":230,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781611+0000","last_change":"2024-09-10T18:30:59.501814+0000","last_active":"2024-09-10T18:31:32.781611+0000","last_peered":"2024-09-10T18:31:32.781611+0000","last_clean":"2024-09-10T18:31:32.781611+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:31:32.781611+0000","last_undegraded":"2024-09-10T18:31:32.781611+0000","last_fullsized":"2024-09-10T18:31:32.781611+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:59.501735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:30:59.501735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:24.012243+0000","scrub_duration":0.010428556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":259,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.055027+0000","last_change":"2024-09-10T18:29:36.235460+0000","last_active":"2024-09-10T18:31:39.055027+0000","last_peered":"2024-09-10T18:31:39.055027+0000","last_clean":"2024-09-10T18:31:39.055027+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:31:39.055027+0000","last_undegraded":"2024-09-10T18:31:39.055027+0000","last_fullsized":"2024-09-10T18:31:39.055027+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:29:36.235377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:29:36.235377+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052752670000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":235,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.774176+0000","last_change":"2024-09-10T18:30:48.481468+0000","last_active":"2024-09-10T18:31:32.774176+0000","last_peered":"2024-09-10T18:31:32.774176+0000","last_clean":"2024-09-10T18:31:32.774176+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:31:32.774176+0000","last_undegraded":"2024-09-10T18:31:32.774176+0000","last_fullsized":"2024-09-10T18:31:32.774176+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:48.481378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:48.481378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:59.396857+0000","scrub_duration":0.0060252910000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":233,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.772905+0000","last_change":"2024-09-10T18:31:09.486730+0000","last_active":"2024-09-10T18:31:32.772905+0000","last_peered":"2024-09-10T18:31:32.772905+0000","last_clean":"2024-09-10T18:31:32.772905+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:31:32.772905+0000","last_undegraded":"2024-09-10T18:31:32.772905+0000","last_fullsized":"2024-09-10T18:31:32.772905+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:09.486379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:31:09.486379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:30.435919+0000","scrub_duration":0.010976669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":232,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.783282+0000","last_change":"2024-09-10T18:29:48.283119+0000","last_active":"2024-09-10T18:31:32.783282+0000","last_peered":"2024-09-10T18:31:32.783282+0000","last_clean":"2024-09-10T18:31:32.783282+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:31:32.783282+0000","last_undegraded":"2024-09-10T18:31:32.783282+0000","last_fullsized":"2024-09-10T18:31:32.783282+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:48.282880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:48.282880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095569450000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":238,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.784251+0000","last_change":"2024-09-10T18:29:42.297141+0000","last_active":"2024-09-10T18:31:32.784251+0000","last_peered":"2024-09-10T18:31:32.784251+0000","last_clean":"2024-09-10T18:31:32.784251+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:31:32.784251+0000","last_undegraded":"2024-09-10T18:31:32.784251+0000","last_fullsized":"2024-09-10T18:31:32.784251+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:42.297045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:29:42.297045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047534680000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'34","reported_seq":279,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.295106+0000","last_change":"2024-09-10T18:30:48.720386+0000","last_active":"2024-09-10T18:31:39.295106+0000","last_peered":"2024-09-10T18:31:39.295106+0000","last_clean":"2024-09-10T18:31:39.295106+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:31:39.295106+0000","last_undegraded":"2024-09-10T18:31:39.295106+0000","last_fullsized":"2024-09-10T18:31:39.295106+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'22","last_scrub_stamp":"2024-09-10T18:30:48.720296+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:30:48.720296+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:08.914191+0000","scrub_duration":0.0037509370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":235,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.770252+0000","last_change":"2024-09-10T18:29:46.293370+0000","last_active":"2024-09-10T18:31:32.770252+0000","last_peered":"2024-09-10T18:31:32.770252+0000","last_clean":"2024-09-10T18:31:32.770252+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:31:32.770252+0000","last_undegraded":"2024-09-10T18:31:32.770252+0000","last_fullsized":"2024-09-10T18:31:32.770252+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:46.293285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:29:46.293285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071945330000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":240,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781628+0000","last_change":"2024-09-10T18:29:46.069209+0000","last_active":"2024-09-10T18:31:32.781628+0000","last_peered":"2024-09-10T18:31:32.781628+0000","last_clean":"2024-09-10T18:31:32.781628+0000","last_became_active":"2024-09-10T18:26:27.644961+0000","last_became_peered":"2024-09-10T18:26:27.644961+0000","last_unstale":"2024-09-10T18:31:32.781628+0000","last_undegraded":"2024-09-10T18:31:32.781628+0000","last_fullsized":"2024-09-10T18:31:32.781628+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:46.069158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:29:46.069158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014897689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'15","reported_seq":262,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.205425+0000","last_change":"2024-09-10T18:30:58.249859+0000","last_active":"2024-09-10T18:31:39.205425+0000","last_peered":"2024-09-10T18:31:39.205425+0000","last_clean":"2024-09-10T18:31:39.205425+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:31:39.205425+0000","last_undegraded":"2024-09-10T18:31:39.205425+0000","last_fullsized":"2024-09-10T18:31:39.205425+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:30:58.249769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:58.249769+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:58.368543+0000","scrub_duration":0.0056195170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":236,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781163+0000","last_change":"2024-09-10T18:30:04.067809+0000","last_active":"2024-09-10T18:31:32.781163+0000","last_peered":"2024-09-10T18:31:32.781163+0000","last_clean":"2024-09-10T18:31:32.781163+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:31:32.781163+0000","last_undegraded":"2024-09-10T18:31:32.781163+0000","last_fullsized":"2024-09-10T18:31:32.781163+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:04.067676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:04.067676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054560930000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":234,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.779281+0000","last_change":"2024-09-10T18:30:56.456172+0000","last_active":"2024-09-10T18:31:32.779281+0000","last_peered":"2024-09-10T18:31:32.779281+0000","last_clean":"2024-09-10T18:31:32.779281+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:31:32.779281+0000","last_undegraded":"2024-09-10T18:31:32.779281+0000","last_fullsized":"2024-09-10T18:31:32.779281+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:56.456087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:56.456087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:16.486677+0000","scrub_duration":0.0081476819999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":240,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781914+0000","last_change":"2024-09-10T18:30:56.338770+0000","last_active":"2024-09-10T18:31:32.781914+0000","last_peered":"2024-09-10T18:31:32.781914+0000","last_clean":"2024-09-10T18:31:32.781914+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:31:32.781914+0000","last_undegraded":"2024-09-10T18:31:32.781914+0000","last_fullsized":"2024-09-10T18:31:32.781914+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:56.338722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:17.464205+0000","last_clean_scrub_stamp":"2024-09-10T18:30:56.338722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:00.099480+0000","scrub_duration":0.0038549069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":274,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.127508+0000","last_change":"2024-09-10T18:30:59.295764+0000","last_active":"2024-09-10T18:31:39.127508+0000","last_peered":"2024-09-10T18:31:39.127508+0000","last_clean":"2024-09-10T18:31:39.127508+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:31:39.127508+0000","last_undegraded":"2024-09-10T18:31:39.127508+0000","last_fullsized":"2024-09-10T18:31:39.127508+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'7","last_scrub_stamp":"2024-09-10T18:30:59.295710+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:30:59.295710+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:00.454829+0000","scrub_duration":0.0032984920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":235,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.779779+0000","last_change":"2024-09-10T18:30:00.485717+0000","last_active":"2024-09-10T18:31:32.779779+0000","last_peered":"2024-09-10T18:31:32.779779+0000","last_clean":"2024-09-10T18:31:32.779779+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:31:32.779779+0000","last_undegraded":"2024-09-10T18:31:32.779779+0000","last_fullsized":"2024-09-10T18:31:32.779779+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:00.485621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:00.485621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032987590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":227,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781305+0000","last_change":"2024-09-10T18:29:52.039654+0000","last_active":"2024-09-10T18:31:32.781305+0000","last_peered":"2024-09-10T18:31:32.781305+0000","last_clean":"2024-09-10T18:31:32.781305+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:31:32.781305+0000","last_undegraded":"2024-09-10T18:31:32.781305+0000","last_fullsized":"2024-09-10T18:31:32.781305+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:52.039580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:29:52.039580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032991380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":233,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.770542+0000","last_change":"2024-09-10T18:29:48.278074+0000","last_active":"2024-09-10T18:31:32.770542+0000","last_peered":"2024-09-10T18:31:32.770542+0000","last_clean":"2024-09-10T18:31:32.770542+0000","last_became_active":"2024-09-10T18:26:27.650774+0000","last_became_peered":"2024-09-10T18:26:27.650774+0000","last_unstale":"2024-09-10T18:31:32.770542+0000","last_undegraded":"2024-09-10T18:31:32.770542+0000","last_fullsized":"2024-09-10T18:31:32.770542+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:48.277982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:48.277982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018607579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":220,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780197+0000","last_change":"2024-09-10T18:28:56.989664+0000","last_active":"2024-09-10T18:31:32.780197+0000","last_peered":"2024-09-10T18:31:32.780197+0000","last_clean":"2024-09-10T18:31:32.780197+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:31:32.780197+0000","last_undegraded":"2024-09-10T18:31:32.780197+0000","last_fullsized":"2024-09-10T18:31:32.780197+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:56.989590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:56.989590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038444830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"240'27","reported_seq":290,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.235136+0000","last_change":"2024-09-10T18:30:46.078403+0000","last_active":"2024-09-10T18:31:39.235136+0000","last_peered":"2024-09-10T18:31:39.235136+0000","last_clean":"2024-09-10T18:31:39.235136+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:31:39.235136+0000","last_undegraded":"2024-09-10T18:31:39.235136+0000","last_fullsized":"2024-09-10T18:31:39.235136+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-10T18:30:46.078313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:46.078313+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:51.777750+0000","scrub_duration":0.011671104999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":235,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.773823+0000","last_change":"2024-09-10T18:30:00.569368+0000","last_active":"2024-09-10T18:31:32.773823+0000","last_peered":"2024-09-10T18:31:32.773823+0000","last_clean":"2024-09-10T18:31:32.773823+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:31:32.773823+0000","last_undegraded":"2024-09-10T18:31:32.773823+0000","last_fullsized":"2024-09-10T18:31:32.773823+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:00.569288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:00.569288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001687004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":232,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.783152+0000","last_change":"2024-09-10T18:29:43.320896+0000","last_active":"2024-09-10T18:31:32.783152+0000","last_peered":"2024-09-10T18:31:32.783152+0000","last_clean":"2024-09-10T18:31:32.783152+0000","last_became_active":"2024-09-10T18:26:27.640035+0000","last_became_peered":"2024-09-10T18:26:27.640035+0000","last_unstale":"2024-09-10T18:31:32.783152+0000","last_undegraded":"2024-09-10T18:31:32.783152+0000","last_fullsized":"2024-09-10T18:31:32.783152+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:43.320795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:43.320795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015436353999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":239,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.773229+0000","last_change":"2024-09-10T18:31:03.350685+0000","last_active":"2024-09-10T18:31:32.773229+0000","last_peered":"2024-09-10T18:31:32.773229+0000","last_clean":"2024-09-10T18:31:32.773229+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:31:32.773229+0000","last_undegraded":"2024-09-10T18:31:32.773229+0000","last_fullsized":"2024-09-10T18:31:32.773229+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:03.350631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:31:03.350631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:18.256534+0000","scrub_duration":0.016656491999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'23","reported_seq":266,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.297477+0000","last_change":"2024-09-10T18:31:06.712809+0000","last_active":"2024-09-10T18:31:39.297477+0000","last_peered":"2024-09-10T18:31:39.297477+0000","last_clean":"2024-09-10T18:31:39.297477+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:31:39.297477+0000","last_undegraded":"2024-09-10T18:31:39.297477+0000","last_fullsized":"2024-09-10T18:31:39.297477+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-09-10T18:31:06.712724+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:31:06.712724+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:07.264277+0000","scrub_duration":0.017994977999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":235,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.770477+0000","last_change":"2024-09-10T18:31:12.515598+0000","last_active":"2024-09-10T18:31:32.770477+0000","last_peered":"2024-09-10T18:31:32.770477+0000","last_clean":"2024-09-10T18:31:32.770477+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:31:32.770477+0000","last_undegraded":"2024-09-10T18:31:32.770477+0000","last_fullsized":"2024-09-10T18:31:32.770477+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:12.515540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:12.515540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:28.271069+0000","scrub_duration":0.0011808949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":235,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780159+0000","last_change":"2024-09-10T18:30:48.487103+0000","last_active":"2024-09-10T18:31:32.780159+0000","last_peered":"2024-09-10T18:31:32.780159+0000","last_clean":"2024-09-10T18:31:32.780159+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:31:32.780159+0000","last_undegraded":"2024-09-10T18:31:32.780159+0000","last_fullsized":"2024-09-10T18:31:32.780159+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:30:48.487005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:48.487005+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:58.811476+0000","scrub_duration":0.0023474720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":249,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.774380+0000","last_change":"2024-09-10T18:31:04.526514+0000","last_active":"2024-09-10T18:31:32.774380+0000","last_peered":"2024-09-10T18:31:32.774380+0000","last_clean":"2024-09-10T18:31:32.774380+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:31:32.774380+0000","last_undegraded":"2024-09-10T18:31:32.774380+0000","last_fullsized":"2024-09-10T18:31:32.774380+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:04.526424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:04.526424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:13.474553+0000","scrub_duration":0.013541001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'25","reported_seq":263,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.284179+0000","last_change":"2024-09-10T18:29:03.110471+0000","last_active":"2024-09-10T18:31:39.284179+0000","last_peered":"2024-09-10T18:31:39.284179+0000","last_clean":"2024-09-10T18:31:39.284179+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:31:39.284179+0000","last_undegraded":"2024-09-10T18:31:39.284179+0000","last_fullsized":"2024-09-10T18:31:39.284179+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-09-10T18:29:03.110142+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:29:03.110142+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.025364746000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"217'11","reported_seq":268,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:38.741152+0000","last_change":"2024-09-10T18:30:03.162072+0000","last_active":"2024-09-10T18:31:38.741152+0000","last_peered":"2024-09-10T18:31:38.741152+0000","last_clean":"2024-09-10T18:31:38.741152+0000","last_became_active":"2024-09-10T18:26:27.652489+0000","last_became_peered":"2024-09-10T18:26:27.652489+0000","last_unstale":"2024-09-10T18:31:38.741152+0000","last_undegraded":"2024-09-10T18:31:38.741152+0000","last_fullsized":"2024-09-10T18:31:38.741152+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11","last_scrub_stamp":"2024-09-10T18:30:03.161985+0000","last_deep_scrub":"217'11","last_deep_scrub_stamp":"2024-09-10T18:30:03.161985+0000","last_clean_scrub_stamp":"2024-09-10T18:30:03.161985+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064398809999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"239'16986","reported_seq":17494,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:37.728775+0000","last_change":"2024-09-10T18:31:06.545419+0000","last_active":"2024-09-10T18:31:37.728775+0000","last_peered":"2024-09-10T18:31:37.728775+0000","last_clean":"2024-09-10T18:31:37.728775+0000","last_became_active":"2024-09-10T18:27:53.036244+0000","last_became_peered":"2024-09-10T18:27:53.036244+0000","last_unstale":"2024-09-10T18:31:37.728775+0000","last_undegraded":"2024-09-10T18:31:37.728775+0000","last_fullsized":"2024-09-10T18:31:37.728775+0000","mapping_epoch":180,"log_start":"226'12100","ondisk_log_start":"226'12100","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'14696","last_scrub_stamp":"2024-09-10T18:31:06.545331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:31:06.545331+0000","objects_scrubbed":2324,"log_size":4886,"log_dups_size":0,"ondisk_log_size":4886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:14.375257+0000","scrub_duration":1.028754875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16986,"num_write_kb":543552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":146,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781471+0000","last_change":"2024-09-10T18:30:00.070274+0000","last_active":"2024-09-10T18:31:32.781471+0000","last_peered":"2024-09-10T18:31:32.781471+0000","last_clean":"2024-09-10T18:31:32.781471+0000","last_became_active":"2024-09-10T18:26:28.635249+0000","last_became_peered":"2024-09-10T18:26:28.635249+0000","last_unstale":"2024-09-10T18:31:32.781471+0000","last_undegraded":"2024-09-10T18:31:32.781471+0000","last_fullsized":"2024-09-10T18:31:32.781471+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:30:00.070176+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:30:00.070176+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046094300000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"240'34","reported_seq":300,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.304352+0000","last_change":"2024-09-10T18:31:11.374516+0000","last_active":"2024-09-10T18:31:39.304352+0000","last_peered":"2024-09-10T18:31:39.304352+0000","last_clean":"2024-09-10T18:31:39.304352+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:31:39.304352+0000","last_undegraded":"2024-09-10T18:31:39.304352+0000","last_fullsized":"2024-09-10T18:31:39.304352+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-09-10T18:31:11.373949+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:31:11.373949+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:28.044226+0000","scrub_duration":0.0054884399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":231,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.787534+0000","last_change":"2024-09-10T18:30:01.544293+0000","last_active":"2024-09-10T18:31:32.787534+0000","last_peered":"2024-09-10T18:31:32.787534+0000","last_clean":"2024-09-10T18:31:32.787534+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:31:32.787534+0000","last_undegraded":"2024-09-10T18:31:32.787534+0000","last_fullsized":"2024-09-10T18:31:32.787534+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:01.544196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:30:01.544196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011854096,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":357,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:34.884399+0000","last_change":"2024-09-10T18:30:00.293382+0000","last_active":"2024-09-10T18:31:34.884399+0000","last_peered":"2024-09-10T18:31:34.884399+0000","last_clean":"2024-09-10T18:31:34.884399+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:31:34.884399+0000","last_undegraded":"2024-09-10T18:31:34.884399+0000","last_fullsized":"2024-09-10T18:31:34.884399+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:30:00.293289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:00.293289+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.105069817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":234,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781260+0000","last_change":"2024-09-10T18:30:59.381476+0000","last_active":"2024-09-10T18:31:32.781260+0000","last_peered":"2024-09-10T18:31:32.781260+0000","last_clean":"2024-09-10T18:31:32.781260+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:31:32.781260+0000","last_undegraded":"2024-09-10T18:31:32.781260+0000","last_fullsized":"2024-09-10T18:31:32.781260+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:59.381396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:30:59.381396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:12.427795+0000","scrub_duration":0.011130997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"239'16594","reported_seq":16973,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:37.710420+0000","last_change":"2024-09-10T18:29:50.011661+0000","last_active":"2024-09-10T18:31:37.710420+0000","last_peered":"2024-09-10T18:31:37.710420+0000","last_clean":"2024-09-10T18:31:37.710420+0000","last_became_active":"2024-09-10T18:27:53.059554+0000","last_became_peered":"2024-09-10T18:27:53.059554+0000","last_unstale":"2024-09-10T18:31:37.710420+0000","last_undegraded":"2024-09-10T18:31:37.710420+0000","last_fullsized":"2024-09-10T18:31:37.710420+0000","mapping_epoch":180,"log_start":"224'11100","ondisk_log_start":"224'11100","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'9197","last_scrub_stamp":"2024-09-10T18:29:50.011576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:29:50.011576+0000","objects_scrubbed":7509,"log_size":5494,"log_dups_size":0,"ondisk_log_size":5494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.8603947789999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16594,"num_write_kb":531008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":145,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.775315+0000","last_change":"2024-09-10T18:30:45.366763+0000","last_active":"2024-09-10T18:31:32.775315+0000","last_peered":"2024-09-10T18:31:32.775315+0000","last_clean":"2024-09-10T18:31:32.775315+0000","last_became_active":"2024-09-10T18:26:28.632744+0000","last_became_peered":"2024-09-10T18:26:28.632744+0000","last_unstale":"2024-09-10T18:31:32.775315+0000","last_undegraded":"2024-09-10T18:31:32.775315+0000","last_fullsized":"2024-09-10T18:31:32.775315+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:30:45.366615+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:30:45.366615+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:14.087487+0000","scrub_duration":0.013315966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":279,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.258187+0000","last_change":"2024-09-10T18:30:52.742276+0000","last_active":"2024-09-10T18:31:39.258187+0000","last_peered":"2024-09-10T18:31:39.258187+0000","last_clean":"2024-09-10T18:31:39.258187+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:31:39.258187+0000","last_undegraded":"2024-09-10T18:31:39.258187+0000","last_fullsized":"2024-09-10T18:31:39.258187+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-10T18:30:52.742191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:52.742191+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:19.019211+0000","scrub_duration":0.015070948000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":231,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.783672+0000","last_change":"2024-09-10T18:31:01.695940+0000","last_active":"2024-09-10T18:31:32.783672+0000","last_peered":"2024-09-10T18:31:32.783672+0000","last_clean":"2024-09-10T18:31:32.783672+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:31:32.783672+0000","last_undegraded":"2024-09-10T18:31:32.783672+0000","last_fullsized":"2024-09-10T18:31:32.783672+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:01.695706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.140922+0000","last_clean_scrub_stamp":"2024-09-10T18:31:01.695706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:05.639726+0000","scrub_duration":0.0022117759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":235,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.770328+0000","last_change":"2024-09-10T18:31:04.334432+0000","last_active":"2024-09-10T18:31:32.770328+0000","last_peered":"2024-09-10T18:31:32.770328+0000","last_clean":"2024-09-10T18:31:32.770328+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:31:32.770328+0000","last_undegraded":"2024-09-10T18:31:32.770328+0000","last_fullsized":"2024-09-10T18:31:32.770328+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:31:04.334337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:05.418495+0000","scrub_duration":0.0012632089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":223,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.773054+0000","last_change":"2024-09-10T18:30:03.186000+0000","last_active":"2024-09-10T18:31:32.773054+0000","last_peered":"2024-09-10T18:31:32.773054+0000","last_clean":"2024-09-10T18:31:32.773054+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:31:32.773054+0000","last_undegraded":"2024-09-10T18:31:32.773054+0000","last_fullsized":"2024-09-10T18:31:32.773054+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:03.185938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:03.185938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007381528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"239'17264","reported_seq":17651,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:37.734860+0000","last_change":"2024-09-10T18:29:51.277252+0000","last_active":"2024-09-10T18:31:37.734860+0000","last_peered":"2024-09-10T18:31:37.734860+0000","last_clean":"2024-09-10T18:31:37.734860+0000","last_became_active":"2024-09-10T18:27:53.061195+0000","last_became_peered":"2024-09-10T18:27:53.061195+0000","last_unstale":"2024-09-10T18:31:37.734860+0000","last_undegraded":"2024-09-10T18:31:37.734860+0000","last_fullsized":"2024-09-10T18:31:37.734860+0000","mapping_epoch":180,"log_start":"225'12200","ondisk_log_start":"225'12200","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'9680","last_scrub_stamp":"2024-09-10T18:29:51.277190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:29:51.277190+0000","objects_scrubbed":7656,"log_size":5064,"log_dups_size":0,"ondisk_log_size":5064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.2581719800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17264,"num_write_kb":552385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":136,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780693+0000","last_change":"2024-09-10T18:30:50.806220+0000","last_active":"2024-09-10T18:31:32.780693+0000","last_peered":"2024-09-10T18:31:32.780693+0000","last_clean":"2024-09-10T18:31:32.780693+0000","last_became_active":"2024-09-10T18:26:18.091896+0000","last_became_peered":"2024-09-10T18:26:18.091896+0000","last_unstale":"2024-09-10T18:31:32.780693+0000","last_undegraded":"2024-09-10T18:31:32.780693+0000","last_fullsized":"2024-09-10T18:31:32.780693+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:30:50.805972+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:30:50.805972+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:20.779313+0000","scrub_duration":0.0087755490000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"240'28","reported_seq":276,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.286382+0000","last_change":"2024-09-10T18:29:34.225862+0000","last_active":"2024-09-10T18:31:39.286382+0000","last_peered":"2024-09-10T18:31:39.286382+0000","last_clean":"2024-09-10T18:31:39.286382+0000","last_became_active":"2024-09-10T18:26:27.651364+0000","last_became_peered":"2024-09-10T18:26:27.651364+0000","last_unstale":"2024-09-10T18:31:39.286382+0000","last_undegraded":"2024-09-10T18:31:39.286382+0000","last_fullsized":"2024-09-10T18:31:39.286382+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-10T18:29:34.225782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:29:34.225782+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021929990000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":230,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781376+0000","last_change":"2024-09-10T18:30:04.443950+0000","last_active":"2024-09-10T18:31:32.781376+0000","last_peered":"2024-09-10T18:31:32.781376+0000","last_clean":"2024-09-10T18:31:32.781376+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:31:32.781376+0000","last_undegraded":"2024-09-10T18:31:32.781376+0000","last_fullsized":"2024-09-10T18:31:32.781376+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:04.443859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:30:04.443859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.167471064,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":358,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:38.735266+0000","last_change":"2024-09-10T18:30:56.702968+0000","last_active":"2024-09-10T18:31:38.735266+0000","last_peered":"2024-09-10T18:31:38.735266+0000","last_clean":"2024-09-10T18:31:38.735266+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:31:38.735266+0000","last_undegraded":"2024-09-10T18:31:38.735266+0000","last_fullsized":"2024-09-10T18:31:38.735266+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:30:56.702886+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:30:56.702886+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:21.421713+0000","scrub_duration":0.0084155429999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":243,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780016+0000","last_change":"2024-09-10T18:31:06.527936+0000","last_active":"2024-09-10T18:31:32.780016+0000","last_peered":"2024-09-10T18:31:32.780016+0000","last_clean":"2024-09-10T18:31:32.780016+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:31:32.780016+0000","last_undegraded":"2024-09-10T18:31:32.780016+0000","last_fullsized":"2024-09-10T18:31:32.780016+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:31:06.527847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:08.659894+0000","scrub_duration":0.0088842569999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"239'16924","reported_seq":17292,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:37.728680+0000","last_change":"2024-09-10T18:30:48.591412+0000","last_active":"2024-09-10T18:31:37.728680+0000","last_peered":"2024-09-10T18:31:37.728680+0000","last_clean":"2024-09-10T18:31:37.728680+0000","last_became_active":"2024-09-10T18:27:53.034171+0000","last_became_peered":"2024-09-10T18:27:53.034171+0000","last_unstale":"2024-09-10T18:31:37.728680+0000","last_undegraded":"2024-09-10T18:31:37.728680+0000","last_fullsized":"2024-09-10T18:31:37.728680+0000","mapping_epoch":180,"log_start":"224'11400","ondisk_log_start":"224'11400","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'13270","last_scrub_stamp":"2024-09-10T18:30:48.591316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:30:48.591316+0000","objects_scrubbed":3744,"log_size":5524,"log_dups_size":0,"ondisk_log_size":5524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:50.983523+0000","scrub_duration":2.2895402169999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16924,"num_write_kb":541568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":185,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.775103+0000","last_change":"2024-09-10T18:29:54.182630+0000","last_active":"2024-09-10T18:31:32.775103+0000","last_peered":"2024-09-10T18:31:32.775103+0000","last_clean":"2024-09-10T18:31:32.775103+0000","last_became_active":"2024-09-10T18:26:28.632849+0000","last_became_peered":"2024-09-10T18:26:28.632849+0000","last_unstale":"2024-09-10T18:31:32.775103+0000","last_undegraded":"2024-09-10T18:31:32.775103+0000","last_fullsized":"2024-09-10T18:31:32.775103+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:29:54.182470+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:29:54.182470+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024114499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":282,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.274268+0000","last_change":"2024-09-10T18:31:00.546611+0000","last_active":"2024-09-10T18:31:39.274268+0000","last_peered":"2024-09-10T18:31:39.274268+0000","last_clean":"2024-09-10T18:31:39.274268+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:31:39.274268+0000","last_undegraded":"2024-09-10T18:31:39.274268+0000","last_fullsized":"2024-09-10T18:31:39.274268+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'15","last_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:31:00.545612+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:20.427793+0000","scrub_duration":0.0072351619999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":230,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781703+0000","last_change":"2024-09-10T18:30:46.332919+0000","last_active":"2024-09-10T18:31:32.781703+0000","last_peered":"2024-09-10T18:31:32.781703+0000","last_clean":"2024-09-10T18:31:32.781703+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:31:32.781703+0000","last_undegraded":"2024-09-10T18:31:32.781703+0000","last_fullsized":"2024-09-10T18:31:32.781703+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:46.332861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:30:46.332861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:08.986192+0000","scrub_duration":0.0061624269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"146'10","reported_seq":365,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:38.023783+0000","last_change":"2024-09-10T18:31:05.540491+0000","last_active":"2024-09-10T18:31:38.023783+0000","last_peered":"2024-09-10T18:31:38.023783+0000","last_clean":"2024-09-10T18:31:38.023783+0000","last_became_active":"2024-09-10T18:26:27.639032+0000","last_became_peered":"2024-09-10T18:26:27.639032+0000","last_unstale":"2024-09-10T18:31:38.023783+0000","last_undegraded":"2024-09-10T18:31:38.023783+0000","last_fullsized":"2024-09-10T18:31:38.023783+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'10","last_scrub_stamp":"2024-09-10T18:31:05.540403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:05.540403+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:18.590346+0000","scrub_duration":0.003060657,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":224,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.779477+0000","last_change":"2024-09-10T18:28:56.061899+0000","last_active":"2024-09-10T18:31:32.779477+0000","last_peered":"2024-09-10T18:31:32.779477+0000","last_clean":"2024-09-10T18:31:32.779477+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:31:32.779477+0000","last_undegraded":"2024-09-10T18:31:32.779477+0000","last_fullsized":"2024-09-10T18:31:32.779477+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:56.061785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:28:56.061785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.038637143999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"239'16796","reported_seq":17168,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:37.726159+0000","last_change":"2024-09-10T18:29:53.667412+0000","last_active":"2024-09-10T18:31:37.726159+0000","last_peered":"2024-09-10T18:31:37.726159+0000","last_clean":"2024-09-10T18:31:37.726159+0000","last_became_active":"2024-09-10T18:27:53.037210+0000","last_became_peered":"2024-09-10T18:27:53.037210+0000","last_unstale":"2024-09-10T18:31:37.726159+0000","last_undegraded":"2024-09-10T18:31:37.726159+0000","last_fullsized":"2024-09-10T18:31:37.726159+0000","mapping_epoch":180,"log_start":"226'12300","ondisk_log_start":"226'12300","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'9619","last_scrub_stamp":"2024-09-10T18:29:53.667320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:29:53.667320+0000","objects_scrubbed":7323,"log_size":4496,"log_dups_size":0,"ondisk_log_size":4496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.3498647269999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16796,"num_write_kb":537472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":195,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.774942+0000","last_change":"2024-09-10T18:31:03.017561+0000","last_active":"2024-09-10T18:31:32.774942+0000","last_peered":"2024-09-10T18:31:32.774942+0000","last_clean":"2024-09-10T18:31:32.774942+0000","last_became_active":"2024-09-10T18:26:28.635542+0000","last_became_peered":"2024-09-10T18:26:28.635542+0000","last_unstale":"2024-09-10T18:31:32.774942+0000","last_undegraded":"2024-09-10T18:31:32.774942+0000","last_fullsized":"2024-09-10T18:31:32.774942+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:03.017452+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:03.017452+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:30.278613+0000","scrub_duration":0.0066732069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"240'29","reported_seq":289,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.268204+0000","last_change":"2024-09-10T18:31:10.474539+0000","last_active":"2024-09-10T18:31:39.268204+0000","last_peered":"2024-09-10T18:31:39.268204+0000","last_clean":"2024-09-10T18:31:39.268204+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:31:39.268204+0000","last_undegraded":"2024-09-10T18:31:39.268204+0000","last_fullsized":"2024-09-10T18:31:39.268204+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-10T18:31:10.474434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:10.474434+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:37.841342+0000","scrub_duration":0.0058438149999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":229,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.773673+0000","last_change":"2024-09-10T18:29:56.487743+0000","last_active":"2024-09-10T18:31:32.773673+0000","last_peered":"2024-09-10T18:31:32.773673+0000","last_clean":"2024-09-10T18:31:32.773673+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:31:32.773673+0000","last_undegraded":"2024-09-10T18:31:32.773673+0000","last_fullsized":"2024-09-10T18:31:32.773673+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:56.487689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:29:56.487689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025845439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":175,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.774298+0000","last_change":"2024-09-10T18:29:57.122438+0000","last_active":"2024-09-10T18:31:32.774298+0000","last_peered":"2024-09-10T18:31:32.774298+0000","last_clean":"2024-09-10T18:31:32.774298+0000","last_became_active":"2024-09-10T18:26:27.642063+0000","last_became_peered":"2024-09-10T18:26:27.642063+0000","last_unstale":"2024-09-10T18:31:32.774298+0000","last_undegraded":"2024-09-10T18:31:32.774298+0000","last_fullsized":"2024-09-10T18:31:32.774298+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:57.122360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:29:57.122360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023484949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":230,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.774076+0000","last_change":"2024-09-10T18:29:55.520215+0000","last_active":"2024-09-10T18:31:32.774076+0000","last_peered":"2024-09-10T18:31:32.774076+0000","last_clean":"2024-09-10T18:31:32.774076+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:31:32.774076+0000","last_undegraded":"2024-09-10T18:31:32.774076+0000","last_fullsized":"2024-09-10T18:31:32.774076+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:29:55.520136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095486129999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"239'16440","reported_seq":16836,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:37.729393+0000","last_change":"2024-09-10T18:29:41.912536+0000","last_active":"2024-09-10T18:31:37.729393+0000","last_peered":"2024-09-10T18:31:37.729393+0000","last_clean":"2024-09-10T18:31:37.729393+0000","last_became_active":"2024-09-10T18:27:53.061113+0000","last_became_peered":"2024-09-10T18:27:53.061113+0000","last_unstale":"2024-09-10T18:31:37.729393+0000","last_undegraded":"2024-09-10T18:31:37.729393+0000","last_fullsized":"2024-09-10T18:31:37.729393+0000","mapping_epoch":180,"log_start":"225'11400","ondisk_log_start":"225'11400","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'8591","last_scrub_stamp":"2024-09-10T18:29:41.912457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:29:41.912457+0000","objects_scrubbed":7942,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.9546324749999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16440,"num_write_kb":526080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":131,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780794+0000","last_change":"2024-09-10T18:30:01.069834+0000","last_active":"2024-09-10T18:31:32.780794+0000","last_peered":"2024-09-10T18:31:32.780794+0000","last_clean":"2024-09-10T18:31:32.780794+0000","last_became_active":"2024-09-10T18:26:18.094747+0000","last_became_peered":"2024-09-10T18:26:18.094747+0000","last_unstale":"2024-09-10T18:31:32.780794+0000","last_undegraded":"2024-09-10T18:31:32.780794+0000","last_fullsized":"2024-09-10T18:31:32.780794+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:30:01.069779+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:30:01.069779+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002552529,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"240'41","reported_seq":344,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.276917+0000","last_change":"2024-09-10T18:31:11.323534+0000","last_active":"2024-09-10T18:31:39.276917+0000","last_peered":"2024-09-10T18:31:39.276917+0000","last_clean":"2024-09-10T18:31:39.276917+0000","last_became_active":"2024-09-10T18:23:10.067049+0000","last_became_peered":"2024-09-10T18:23:10.067049+0000","last_unstale":"2024-09-10T18:31:39.276917+0000","last_undegraded":"2024-09-10T18:31:39.276917+0000","last_fullsized":"2024-09-10T18:31:39.276917+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'29","last_scrub_stamp":"2024-09-10T18:31:11.323461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:11.323461+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:15.747737+0000","scrub_duration":0.0098705089999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":236,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.772323+0000","last_change":"2024-09-10T18:29:35.285093+0000","last_active":"2024-09-10T18:31:32.772323+0000","last_peered":"2024-09-10T18:31:32.772323+0000","last_clean":"2024-09-10T18:31:32.772323+0000","last_became_active":"2024-09-10T18:26:27.648690+0000","last_became_peered":"2024-09-10T18:26:27.648690+0000","last_unstale":"2024-09-10T18:31:32.772323+0000","last_undegraded":"2024-09-10T18:31:32.772323+0000","last_fullsized":"2024-09-10T18:31:32.772323+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:35.284705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:29:35.284705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.026955588999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":255,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781852+0000","last_change":"2024-09-10T18:31:04.391774+0000","last_active":"2024-09-10T18:31:32.781852+0000","last_peered":"2024-09-10T18:31:32.781852+0000","last_clean":"2024-09-10T18:31:32.781852+0000","last_became_active":"2024-09-10T18:26:27.647576+0000","last_became_peered":"2024-09-10T18:26:27.647576+0000","last_unstale":"2024-09-10T18:31:32.781852+0000","last_undegraded":"2024-09-10T18:31:32.781852+0000","last_fullsized":"2024-09-10T18:31:32.781852+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:31:04.391687+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:31:04.391687+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:17.192131+0000","scrub_duration":0.011656032,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":226,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780405+0000","last_change":"2024-09-10T18:29:33.023742+0000","last_active":"2024-09-10T18:31:32.780405+0000","last_peered":"2024-09-10T18:31:32.780405+0000","last_clean":"2024-09-10T18:31:32.780405+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:31:32.780405+0000","last_undegraded":"2024-09-10T18:31:32.780405+0000","last_fullsized":"2024-09-10T18:31:32.780405+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:33.023658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:29:33.023658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024546590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":225,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781448+0000","last_change":"2024-09-10T18:31:04.714213+0000","last_active":"2024-09-10T18:31:32.781448+0000","last_peered":"2024-09-10T18:31:32.781448+0000","last_clean":"2024-09-10T18:31:32.781448+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:31:32.781448+0000","last_undegraded":"2024-09-10T18:31:32.781448+0000","last_fullsized":"2024-09-10T18:31:32.781448+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:04.714112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:04.714112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:31.376995+0000","scrub_duration":0.0085368679999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"239'21009","reported_seq":21422,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:37.729024+0000","last_change":"2024-09-10T18:30:46.448052+0000","last_active":"2024-09-10T18:31:37.729024+0000","last_peered":"2024-09-10T18:31:37.729024+0000","last_clean":"2024-09-10T18:31:37.729024+0000","last_became_active":"2024-09-10T18:27:53.037246+0000","last_became_peered":"2024-09-10T18:27:53.037246+0000","last_unstale":"2024-09-10T18:31:37.729024+0000","last_undegraded":"2024-09-10T18:31:37.729024+0000","last_fullsized":"2024-09-10T18:31:37.729024+0000","mapping_epoch":180,"log_start":"226'16504","ondisk_log_start":"226'16504","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'17268","last_scrub_stamp":"2024-09-10T18:30:46.447757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:30:46.447757+0000","objects_scrubbed":3789,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:52.411404+0000","scrub_duration":1.7370367789999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16602,"num_write_kb":531264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":277,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.779644+0000","last_change":"2024-09-10T18:29:56.539912+0000","last_active":"2024-09-10T18:31:32.779644+0000","last_peered":"2024-09-10T18:31:32.779644+0000","last_clean":"2024-09-10T18:31:32.779644+0000","last_became_active":"2024-09-10T18:26:18.095568+0000","last_became_peered":"2024-09-10T18:26:18.095568+0000","last_unstale":"2024-09-10T18:31:32.779644+0000","last_undegraded":"2024-09-10T18:31:32.779644+0000","last_fullsized":"2024-09-10T18:31:32.779644+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:29:56.539815+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:29:56.539815+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016440160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"240'17","reported_seq":270,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.223037+0000","last_change":"2024-09-10T18:30:51.522994+0000","last_active":"2024-09-10T18:31:39.223037+0000","last_peered":"2024-09-10T18:31:39.223037+0000","last_clean":"2024-09-10T18:31:39.223037+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:31:39.223037+0000","last_undegraded":"2024-09-10T18:31:39.223037+0000","last_fullsized":"2024-09-10T18:31:39.223037+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-10T18:30:51.522555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:51.522555+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:59.965878+0000","scrub_duration":0.0030428550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":246,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.773764+0000","last_change":"2024-09-10T18:30:02.567170+0000","last_active":"2024-09-10T18:31:32.773764+0000","last_peered":"2024-09-10T18:31:32.773764+0000","last_clean":"2024-09-10T18:31:32.773764+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:31:32.773764+0000","last_undegraded":"2024-09-10T18:31:32.773764+0000","last_fullsized":"2024-09-10T18:31:32.773764+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:02.567084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:02.567084+0000","last_clean_scrub_stamp":"2024-09-10T18:30:02.567084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030254050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":258,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.768511+0000","last_change":"2024-09-10T18:30:45.042033+0000","last_active":"2024-09-10T18:31:32.768511+0000","last_peered":"2024-09-10T18:31:32.768511+0000","last_clean":"2024-09-10T18:31:32.768511+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:31:32.768511+0000","last_undegraded":"2024-09-10T18:31:32.768511+0000","last_fullsized":"2024-09-10T18:31:32.768511+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:30:45.041941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:30:45.041941+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:49.744342+0000","scrub_duration":0.016163502999999999,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":245,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.768258+0000","last_change":"2024-09-10T18:31:10.933803+0000","last_active":"2024-09-10T18:31:32.768258+0000","last_peered":"2024-09-10T18:31:32.768258+0000","last_clean":"2024-09-10T18:31:32.768258+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:31:32.768258+0000","last_undegraded":"2024-09-10T18:31:32.768258+0000","last_fullsized":"2024-09-10T18:31:32.768258+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:10.933715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:31:10.933715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:38.397405+0000","scrub_duration":0.0061095990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":233,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780685+0000","last_change":"2024-09-10T18:29:55.513631+0000","last_active":"2024-09-10T18:31:32.780685+0000","last_peered":"2024-09-10T18:31:32.780685+0000","last_clean":"2024-09-10T18:31:32.780685+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:31:32.780685+0000","last_undegraded":"2024-09-10T18:31:32.780685+0000","last_fullsized":"2024-09-10T18:31:32.780685+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:55.513549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:55.513549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024107299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"239'16638","reported_seq":17101,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:37.725858+0000","last_change":"2024-09-10T18:31:13.034453+0000","last_active":"2024-09-10T18:31:37.725858+0000","last_peered":"2024-09-10T18:31:37.725858+0000","last_clean":"2024-09-10T18:31:37.725858+0000","last_became_active":"2024-09-10T18:27:53.034315+0000","last_became_peered":"2024-09-10T18:27:53.034315+0000","last_unstale":"2024-09-10T18:31:37.725858+0000","last_undegraded":"2024-09-10T18:31:37.725858+0000","last_fullsized":"2024-09-10T18:31:37.725858+0000","mapping_epoch":180,"log_start":"225'11715","ondisk_log_start":"225'11715","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'14864","last_scrub_stamp":"2024-09-10T18:31:13.034383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:31:13.034383+0000","objects_scrubbed":1789,"log_size":4923,"log_dups_size":0,"ondisk_log_size":4923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:38.107591+0000","scrub_duration":0.67597978599999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16638,"num_write_kb":532416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":146,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.774364+0000","last_change":"2024-09-10T18:29:43.699835+0000","last_active":"2024-09-10T18:31:32.774364+0000","last_peered":"2024-09-10T18:31:32.774364+0000","last_clean":"2024-09-10T18:31:32.774364+0000","last_became_active":"2024-09-10T18:26:28.641791+0000","last_became_peered":"2024-09-10T18:26:28.641791+0000","last_unstale":"2024-09-10T18:31:32.774364+0000","last_undegraded":"2024-09-10T18:31:32.774364+0000","last_fullsized":"2024-09-10T18:31:32.774364+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:29:43.699736+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:29:43.699736+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016715837000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":264,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.233239+0000","last_change":"2024-09-10T18:29:57.326700+0000","last_active":"2024-09-10T18:31:39.233239+0000","last_peered":"2024-09-10T18:31:39.233239+0000","last_clean":"2024-09-10T18:31:39.233239+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:31:39.233239+0000","last_undegraded":"2024-09-10T18:31:39.233239+0000","last_fullsized":"2024-09-10T18:31:39.233239+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:29:57.326613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:29:57.326613+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.138665703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":240,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781385+0000","last_change":"2024-09-10T18:31:08.529508+0000","last_active":"2024-09-10T18:31:32.781385+0000","last_peered":"2024-09-10T18:31:32.781385+0000","last_clean":"2024-09-10T18:31:32.781385+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:31:32.781385+0000","last_undegraded":"2024-09-10T18:31:32.781385+0000","last_fullsized":"2024-09-10T18:31:32.781385+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:08.529424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:31:08.529424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:08.560187+0000","scrub_duration":0.0078874119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":858,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:38.580405+0000","last_change":"2024-09-10T18:31:13.536758+0000","last_active":"2024-09-10T18:31:38.580405+0000","last_peered":"2024-09-10T18:31:38.580405+0000","last_clean":"2024-09-10T18:31:38.580405+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:31:38.580405+0000","last_undegraded":"2024-09-10T18:31:38.580405+0000","last_fullsized":"2024-09-10T18:31:38.580405+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:31:13.536419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:13.536419+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:34.617675+0000","scrub_duration":0.0043736720000000003,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":601,"num_read_kb":596,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":229,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.782308+0000","last_change":"2024-09-10T18:30:47.337000+0000","last_active":"2024-09-10T18:31:32.782308+0000","last_peered":"2024-09-10T18:31:32.782308+0000","last_clean":"2024-09-10T18:31:32.782308+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:31:32.782308+0000","last_undegraded":"2024-09-10T18:31:32.782308+0000","last_fullsized":"2024-09-10T18:31:32.782308+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:47.336845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.771415+0000","last_clean_scrub_stamp":"2024-09-10T18:30:47.336845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:16.049963+0000","scrub_duration":0.0051314819999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":223,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.773446+0000","last_change":"2024-09-10T18:29:57.534601+0000","last_active":"2024-09-10T18:31:32.773446+0000","last_peered":"2024-09-10T18:31:32.773446+0000","last_clean":"2024-09-10T18:31:32.773446+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:31:32.773446+0000","last_undegraded":"2024-09-10T18:31:32.773446+0000","last_fullsized":"2024-09-10T18:31:32.773446+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:57.534528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:57.534528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011484388999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"239'16804","reported_seq":17276,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:37.717069+0000","last_change":"2024-09-10T18:31:09.711045+0000","last_active":"2024-09-10T18:31:37.717069+0000","last_peered":"2024-09-10T18:31:37.717069+0000","last_clean":"2024-09-10T18:31:37.717069+0000","last_became_active":"2024-09-10T18:27:53.035261+0000","last_became_peered":"2024-09-10T18:27:53.035261+0000","last_unstale":"2024-09-10T18:31:37.717069+0000","last_undegraded":"2024-09-10T18:31:37.717069+0000","last_fullsized":"2024-09-10T18:31:37.717069+0000","mapping_epoch":180,"log_start":"222'11100","ondisk_log_start":"222'11100","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'14823","last_scrub_stamp":"2024-09-10T18:31:09.710964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:31:09.710964+0000","objects_scrubbed":2001,"log_size":5704,"log_dups_size":0,"ondisk_log_size":5704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:36.211088+0000","scrub_duration":0.65510596700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16804,"num_write_kb":537728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"240'26","reported_seq":222,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.278074+0000","last_change":"2024-09-10T18:30:47.151480+0000","last_active":"2024-09-10T18:31:39.278074+0000","last_peered":"2024-09-10T18:31:39.278074+0000","last_clean":"2024-09-10T18:31:39.278074+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:31:39.278074+0000","last_undegraded":"2024-09-10T18:31:39.278074+0000","last_fullsized":"2024-09-10T18:31:39.278074+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-10T18:30:47.151322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:47.151322+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:10.861641+0000","scrub_duration":0.0022400229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":238,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780506+0000","last_change":"2024-09-10T18:29:33.216704+0000","last_active":"2024-09-10T18:31:32.780506+0000","last_peered":"2024-09-10T18:31:32.780506+0000","last_clean":"2024-09-10T18:31:32.780506+0000","last_became_active":"2024-09-10T18:26:27.639129+0000","last_became_peered":"2024-09-10T18:26:27.639129+0000","last_unstale":"2024-09-10T18:31:32.780506+0000","last_undegraded":"2024-09-10T18:31:32.780506+0000","last_fullsized":"2024-09-10T18:31:32.780506+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:33.216613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:29:33.216613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013069801000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":289,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.779457+0000","last_change":"2024-09-10T18:30:51.519828+0000","last_active":"2024-09-10T18:31:32.779457+0000","last_peered":"2024-09-10T18:31:32.779457+0000","last_clean":"2024-09-10T18:31:32.779457+0000","last_became_active":"2024-09-10T18:26:27.646658+0000","last_became_peered":"2024-09-10T18:26:27.646658+0000","last_unstale":"2024-09-10T18:31:32.779457+0000","last_undegraded":"2024-09-10T18:31:32.779457+0000","last_fullsized":"2024-09-10T18:31:32.779457+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:30:51.519744+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:30:51.519744+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:59.960685+0000","scrub_duration":0.0040833290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":235,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780345+0000","last_change":"2024-09-10T18:29:54.545893+0000","last_active":"2024-09-10T18:31:32.780345+0000","last_peered":"2024-09-10T18:31:32.780345+0000","last_clean":"2024-09-10T18:31:32.780345+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:31:32.780345+0000","last_undegraded":"2024-09-10T18:31:32.780345+0000","last_fullsized":"2024-09-10T18:31:32.780345+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:54.545809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:29:54.545809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041017759999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":227,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.782238+0000","last_change":"2024-09-10T18:30:49.395000+0000","last_active":"2024-09-10T18:31:32.782238+0000","last_peered":"2024-09-10T18:31:32.782238+0000","last_clean":"2024-09-10T18:31:32.782238+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:31:32.782238+0000","last_undegraded":"2024-09-10T18:31:32.782238+0000","last_fullsized":"2024-09-10T18:31:32.782238+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:49.394941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:30:49.394941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:09.069344+0000","scrub_duration":0.0017872470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"239'16486","reported_seq":16722,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:37.725477+0000","last_change":"2024-09-10T18:30:45.455710+0000","last_active":"2024-09-10T18:31:37.725477+0000","last_peered":"2024-09-10T18:31:37.725477+0000","last_clean":"2024-09-10T18:31:37.725477+0000","last_became_active":"2024-09-10T18:27:53.033757+0000","last_became_peered":"2024-09-10T18:27:53.033757+0000","last_unstale":"2024-09-10T18:31:37.725477+0000","last_undegraded":"2024-09-10T18:31:37.725477+0000","last_fullsized":"2024-09-10T18:31:37.725477+0000","mapping_epoch":180,"log_start":"226'11900","ondisk_log_start":"226'11900","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'12692","last_scrub_stamp":"2024-09-10T18:30:45.455623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:30:45.455623+0000","objects_scrubbed":3837,"log_size":4586,"log_dups_size":0,"ondisk_log_size":4586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:06.033806+0000","scrub_duration":1.1074504000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16486,"num_write_kb":527552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"240'19","reported_seq":268,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.281954+0000","last_change":"2024-09-10T18:29:58.236626+0000","last_active":"2024-09-10T18:31:39.281954+0000","last_peered":"2024-09-10T18:31:39.281954+0000","last_clean":"2024-09-10T18:31:39.281954+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:31:39.281954+0000","last_undegraded":"2024-09-10T18:31:39.281954+0000","last_fullsized":"2024-09-10T18:31:39.281954+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:29:58.236548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:29:58.236548+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091212359999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":229,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.770680+0000","last_change":"2024-09-10T18:30:01.224897+0000","last_active":"2024-09-10T18:31:32.770680+0000","last_peered":"2024-09-10T18:31:32.770680+0000","last_clean":"2024-09-10T18:31:32.770680+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:31:32.770680+0000","last_undegraded":"2024-09-10T18:31:32.770680+0000","last_fullsized":"2024-09-10T18:31:32.770680+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:01.224811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:30:01.224811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011944340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":243,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.782105+0000","last_change":"2024-09-10T18:31:00.387970+0000","last_active":"2024-09-10T18:31:32.782105+0000","last_peered":"2024-09-10T18:31:32.782105+0000","last_clean":"2024-09-10T18:31:32.782105+0000","last_became_active":"2024-09-10T18:26:27.645885+0000","last_became_peered":"2024-09-10T18:26:27.645885+0000","last_unstale":"2024-09-10T18:31:32.782105+0000","last_undegraded":"2024-09-10T18:31:32.782105+0000","last_fullsized":"2024-09-10T18:31:32.782105+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:31:00.387887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:25.925387+0000","scrub_duration":0.001951775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":232,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780771+0000","last_change":"2024-09-10T18:29:40.267823+0000","last_active":"2024-09-10T18:31:32.780771+0000","last_peered":"2024-09-10T18:31:32.780771+0000","last_clean":"2024-09-10T18:31:32.780771+0000","last_became_active":"2024-09-10T18:26:27.632071+0000","last_became_peered":"2024-09-10T18:26:27.632071+0000","last_unstale":"2024-09-10T18:31:32.780771+0000","last_undegraded":"2024-09-10T18:31:32.780771+0000","last_fullsized":"2024-09-10T18:31:32.780771+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:40.267731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:40.267731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037925329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"239'16860","reported_seq":17202,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:37.726167+0000","last_change":"2024-09-10T18:30:03.941778+0000","last_active":"2024-09-10T18:31:37.726167+0000","last_peered":"2024-09-10T18:31:37.726167+0000","last_clean":"2024-09-10T18:31:37.726167+0000","last_became_active":"2024-09-10T18:27:53.035419+0000","last_became_peered":"2024-09-10T18:27:53.035419+0000","last_unstale":"2024-09-10T18:31:37.726167+0000","last_undegraded":"2024-09-10T18:31:37.726167+0000","last_fullsized":"2024-09-10T18:31:37.726167+0000","mapping_epoch":180,"log_start":"226'12000","ondisk_log_start":"226'12000","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'10327","last_scrub_stamp":"2024-09-10T18:30:03.941534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:30:03.941534+0000","objects_scrubbed":6614,"log_size":4860,"log_dups_size":0,"ondisk_log_size":4860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.4155082019999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16860,"num_write_kb":539520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":138,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780885+0000","last_change":"2024-09-10T18:29:40.943571+0000","last_active":"2024-09-10T18:31:32.780885+0000","last_peered":"2024-09-10T18:31:32.780885+0000","last_clean":"2024-09-10T18:31:32.780885+0000","last_became_active":"2024-09-10T18:26:18.095088+0000","last_became_peered":"2024-09-10T18:26:18.095088+0000","last_unstale":"2024-09-10T18:31:32.780885+0000","last_undegraded":"2024-09-10T18:31:32.780885+0000","last_fullsized":"2024-09-10T18:31:32.780885+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:29:40.943514+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:29:40.943514+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016048987000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":276,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.280034+0000","last_change":"2024-09-10T18:29:00.230613+0000","last_active":"2024-09-10T18:31:39.280034+0000","last_peered":"2024-09-10T18:31:39.280034+0000","last_clean":"2024-09-10T18:31:39.280034+0000","last_became_active":"2024-09-10T18:21:33.611203+0000","last_became_peered":"2024-09-10T18:21:33.611203+0000","last_unstale":"2024-09-10T18:31:39.280034+0000","last_undegraded":"2024-09-10T18:31:39.280034+0000","last_fullsized":"2024-09-10T18:31:39.280034+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-09-10T18:29:00.230525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:29:00.230525+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054835300000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":193,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.774537+0000","last_change":"2024-09-10T18:30:58.072260+0000","last_active":"2024-09-10T18:31:32.774537+0000","last_peered":"2024-09-10T18:31:32.774537+0000","last_clean":"2024-09-10T18:31:32.774537+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:31:32.774537+0000","last_undegraded":"2024-09-10T18:31:32.774537+0000","last_fullsized":"2024-09-10T18:31:32.774537+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:30:58.072177+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:30:58.072177+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:14.158109+0000","scrub_duration":0.003192453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":235,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.773518+0000","last_change":"2024-09-10T18:30:53.506075+0000","last_active":"2024-09-10T18:31:32.773518+0000","last_peered":"2024-09-10T18:31:32.773518+0000","last_clean":"2024-09-10T18:31:32.773518+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:31:32.773518+0000","last_undegraded":"2024-09-10T18:31:32.773518+0000","last_fullsized":"2024-09-10T18:31:32.773518+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:53.506000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:53.506000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:05.835328+0000","scrub_duration":0.011216421000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":253,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.767964+0000","last_change":"2024-09-10T18:31:04.059884+0000","last_active":"2024-09-10T18:31:32.767964+0000","last_peered":"2024-09-10T18:31:32.767964+0000","last_clean":"2024-09-10T18:31:32.767964+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:31:32.767964+0000","last_undegraded":"2024-09-10T18:31:32.767964+0000","last_fullsized":"2024-09-10T18:31:32.767964+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:04.059795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:04.059795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:06.431870+0000","scrub_duration":0.010230702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"239'16574","reported_seq":16907,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:37.732098+0000","last_change":"2024-09-10T18:30:06.516566+0000","last_active":"2024-09-10T18:31:37.732098+0000","last_peered":"2024-09-10T18:31:37.732098+0000","last_clean":"2024-09-10T18:31:37.732098+0000","last_became_active":"2024-09-10T18:27:53.037235+0000","last_became_peered":"2024-09-10T18:27:53.037235+0000","last_unstale":"2024-09-10T18:31:37.732098+0000","last_undegraded":"2024-09-10T18:31:37.732098+0000","last_fullsized":"2024-09-10T18:31:37.732098+0000","mapping_epoch":180,"log_start":"226'12000","ondisk_log_start":"226'12000","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'10249","last_scrub_stamp":"2024-09-10T18:30:06.516475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:30:06.516475+0000","objects_scrubbed":6370,"log_size":4574,"log_dups_size":0,"ondisk_log_size":4574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1.9308149449999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16574,"num_write_kb":530368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":287,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.779564+0000","last_change":"2024-09-10T18:31:10.502369+0000","last_active":"2024-09-10T18:31:32.779564+0000","last_peered":"2024-09-10T18:31:32.779564+0000","last_clean":"2024-09-10T18:31:32.779564+0000","last_became_active":"2024-09-10T18:26:18.092014+0000","last_became_peered":"2024-09-10T18:26:18.092014+0000","last_unstale":"2024-09-10T18:31:32.779564+0000","last_undegraded":"2024-09-10T18:31:32.779564+0000","last_fullsized":"2024-09-10T18:31:32.779564+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:10.502290+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:10.502290+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:38.681535+0000","scrub_duration":0.0082798439999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"240'25","reported_seq":269,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.243213+0000","last_change":"2024-09-10T18:29:45.292442+0000","last_active":"2024-09-10T18:31:39.243213+0000","last_peered":"2024-09-10T18:31:39.243213+0000","last_clean":"2024-09-10T18:31:39.243213+0000","last_became_active":"2024-09-10T18:26:27.649406+0000","last_became_peered":"2024-09-10T18:26:27.649406+0000","last_unstale":"2024-09-10T18:31:39.243213+0000","last_undegraded":"2024-09-10T18:31:39.243213+0000","last_fullsized":"2024-09-10T18:31:39.243213+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'15","last_scrub_stamp":"2024-09-10T18:29:45.292368+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:29:45.292368+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033922850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":231,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.783681+0000","last_change":"2024-09-10T18:31:00.667182+0000","last_active":"2024-09-10T18:31:32.783681+0000","last_peered":"2024-09-10T18:31:32.783681+0000","last_clean":"2024-09-10T18:31:32.783681+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:31:32.783681+0000","last_undegraded":"2024-09-10T18:31:32.783681+0000","last_fullsized":"2024-09-10T18:31:32.783681+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:00.667103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:00.667103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:22.994719+0000","scrub_duration":0.0017606639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"189'9","reported_seq":386,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:38.661784+0000","last_change":"2024-09-10T18:31:09.044554+0000","last_active":"2024-09-10T18:31:38.661784+0000","last_peered":"2024-09-10T18:31:38.661784+0000","last_clean":"2024-09-10T18:31:38.661784+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:31:38.661784+0000","last_undegraded":"2024-09-10T18:31:38.661784+0000","last_fullsized":"2024-09-10T18:31:38.661784+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'9","last_scrub_stamp":"2024-09-10T18:31:09.044476+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:31:09.044476+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:10.212254+0000","scrub_duration":0.059079303999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":226,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.770457+0000","last_change":"2024-09-10T18:30:46.397016+0000","last_active":"2024-09-10T18:31:32.770457+0000","last_peered":"2024-09-10T18:31:32.770457+0000","last_clean":"2024-09-10T18:31:32.770457+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:31:32.770457+0000","last_undegraded":"2024-09-10T18:31:32.770457+0000","last_fullsized":"2024-09-10T18:31:32.770457+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:46.396958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_clean_scrub_stamp":"2024-09-10T18:30:46.396958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:59.092367+0000","scrub_duration":0.013383155000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"239'16860","reported_seq":17244,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:37.705457+0000","last_change":"2024-09-10T18:29:50.519885+0000","last_active":"2024-09-10T18:31:37.705457+0000","last_peered":"2024-09-10T18:31:37.705457+0000","last_clean":"2024-09-10T18:31:37.705457+0000","last_became_active":"2024-09-10T18:27:53.037361+0000","last_became_peered":"2024-09-10T18:27:53.037361+0000","last_unstale":"2024-09-10T18:31:37.705457+0000","last_undegraded":"2024-09-10T18:31:37.705457+0000","last_fullsized":"2024-09-10T18:31:37.705457+0000","mapping_epoch":180,"log_start":"225'11927","ondisk_log_start":"225'11927","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'9361","last_scrub_stamp":"2024-09-10T18:29:50.519772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:29:50.519772+0000","objects_scrubbed":7639,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"queued for scrub","scrub_duration":4.8806252880000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16860,"num_write_kb":539520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"240'15","reported_seq":243,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.264158+0000","last_change":"2024-09-10T18:28:53.964677+0000","last_active":"2024-09-10T18:31:39.264158+0000","last_peered":"2024-09-10T18:31:39.264158+0000","last_clean":"2024-09-10T18:31:39.264158+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:31:39.264158+0000","last_undegraded":"2024-09-10T18:31:39.264158+0000","last_fullsized":"2024-09-10T18:31:39.264158+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:28:53.964578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:28:53.964578+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056047789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"58'1","reported_seq":246,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.782045+0000","last_change":"2024-09-10T18:29:05.902842+0000","last_active":"2024-09-10T18:31:32.782045+0000","last_peered":"2024-09-10T18:31:32.782045+0000","last_clean":"2024-09-10T18:31:32.782045+0000","last_became_active":"2024-09-10T18:26:27.644997+0000","last_became_peered":"2024-09-10T18:26:27.644997+0000","last_unstale":"2024-09-10T18:31:32.782045+0000","last_undegraded":"2024-09-10T18:31:32.782045+0000","last_fullsized":"2024-09-10T18:31:32.782045+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-10T18:29:05.902369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:29:05.902369+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051854029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":251,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.768101+0000","last_change":"2024-09-10T18:31:08.038810+0000","last_active":"2024-09-10T18:31:32.768101+0000","last_peered":"2024-09-10T18:31:32.768101+0000","last_clean":"2024-09-10T18:31:32.768101+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:31:32.768101+0000","last_undegraded":"2024-09-10T18:31:32.768101+0000","last_fullsized":"2024-09-10T18:31:32.768101+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:08.038615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:08.038615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:29.821394+0000","scrub_duration":0.0064931390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":224,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781316+0000","last_change":"2024-09-10T18:29:44.098384+0000","last_active":"2024-09-10T18:31:32.781316+0000","last_peered":"2024-09-10T18:31:32.781316+0000","last_clean":"2024-09-10T18:31:32.781316+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:31:32.781316+0000","last_undegraded":"2024-09-10T18:31:32.781316+0000","last_fullsized":"2024-09-10T18:31:32.781316+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:44.098280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:44.098280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055883269999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"239'21527","reported_seq":21944,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:37.716137+0000","last_change":"2024-09-10T18:30:53.024268+0000","last_active":"2024-09-10T18:31:37.716137+0000","last_peered":"2024-09-10T18:31:37.716137+0000","last_clean":"2024-09-10T18:31:37.716137+0000","last_became_active":"2024-09-10T18:27:53.059866+0000","last_became_peered":"2024-09-10T18:27:53.059866+0000","last_unstale":"2024-09-10T18:31:37.716137+0000","last_undegraded":"2024-09-10T18:31:37.716137+0000","last_fullsized":"2024-09-10T18:31:37.716137+0000","mapping_epoch":180,"log_start":"224'16004","ondisk_log_start":"224'16004","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'18230","last_scrub_stamp":"2024-09-10T18:30:53.024177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:30:53.024177+0000","objects_scrubbed":3365,"log_size":5523,"log_dups_size":0,"ondisk_log_size":5523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:12.287378+0000","scrub_duration":1.699362767,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16752,"num_write_kb":536064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"240'44","reported_seq":310,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.301857+0000","last_change":"2024-09-10T18:31:00.043959+0000","last_active":"2024-09-10T18:31:39.301857+0000","last_peered":"2024-09-10T18:31:39.301857+0000","last_clean":"2024-09-10T18:31:39.301857+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:31:39.301857+0000","last_undegraded":"2024-09-10T18:31:39.301857+0000","last_fullsized":"2024-09-10T18:31:39.301857+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'28","last_scrub_stamp":"2024-09-10T18:31:00.043812+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:31:00.043812+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:03.642411+0000","scrub_duration":0.004131656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":246,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.782459+0000","last_change":"2024-09-10T18:30:01.485412+0000","last_active":"2024-09-10T18:31:32.782459+0000","last_peered":"2024-09-10T18:31:32.782459+0000","last_clean":"2024-09-10T18:31:32.782459+0000","last_became_active":"2024-09-10T18:26:27.638408+0000","last_became_peered":"2024-09-10T18:26:27.638408+0000","last_unstale":"2024-09-10T18:31:32.782459+0000","last_undegraded":"2024-09-10T18:31:32.782459+0000","last_fullsized":"2024-09-10T18:31:32.782459+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:01.485332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:30:01.485332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048583630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":255,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.767883+0000","last_change":"2024-09-10T18:31:14.899762+0000","last_active":"2024-09-10T18:31:32.767883+0000","last_peered":"2024-09-10T18:31:32.767883+0000","last_clean":"2024-09-10T18:31:32.767883+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:31:32.767883+0000","last_undegraded":"2024-09-10T18:31:32.767883+0000","last_fullsized":"2024-09-10T18:31:32.767883+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:14.899683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:14.899683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:24.241674+0000","scrub_duration":0.0054936819999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":236,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.779391+0000","last_change":"2024-09-10T18:29:41.447464+0000","last_active":"2024-09-10T18:31:32.779391+0000","last_peered":"2024-09-10T18:31:32.779391+0000","last_clean":"2024-09-10T18:31:32.779391+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:31:32.779391+0000","last_undegraded":"2024-09-10T18:31:32.779391+0000","last_fullsized":"2024-09-10T18:31:32.779391+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:41.447227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:41.447227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012629659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"239'18689","reported_seq":19188,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:37.717528+0000","last_change":"2024-09-10T18:31:16.902340+0000","last_active":"2024-09-10T18:31:37.717528+0000","last_peered":"2024-09-10T18:31:37.717528+0000","last_clean":"2024-09-10T18:31:37.717528+0000","last_became_active":"2024-09-10T18:27:53.060902+0000","last_became_peered":"2024-09-10T18:27:53.060902+0000","last_unstale":"2024-09-10T18:31:37.717528+0000","last_undegraded":"2024-09-10T18:31:37.717528+0000","last_fullsized":"2024-09-10T18:31:37.717528+0000","mapping_epoch":180,"log_start":"229'13000","ondisk_log_start":"229'13000","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'17099","last_scrub_stamp":"2024-09-10T18:31:16.902287+0000","last_deep_scrub":"236'17099","last_deep_scrub_stamp":"2024-09-10T18:31:16.902287+0000","last_clean_scrub_stamp":"2024-09-10T18:31:16.902287+0000","objects_scrubbed":1927,"log_size":5689,"log_dups_size":0,"ondisk_log_size":5689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:22.813809+0000","scrub_duration":10.85097494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16762,"num_write_kb":536384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"240'18","reported_seq":272,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.071315+0000","last_change":"2024-09-10T18:30:54.285825+0000","last_active":"2024-09-10T18:31:39.071315+0000","last_peered":"2024-09-10T18:31:39.071315+0000","last_clean":"2024-09-10T18:31:39.071315+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:31:39.071315+0000","last_undegraded":"2024-09-10T18:31:39.071315+0000","last_fullsized":"2024-09-10T18:31:39.071315+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:30:54.285738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:54.285738+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:13.017862+0000","scrub_duration":0.0045401779999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":249,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780689+0000","last_change":"2024-09-10T18:31:12.489636+0000","last_active":"2024-09-10T18:31:32.780689+0000","last_peered":"2024-09-10T18:31:32.780689+0000","last_clean":"2024-09-10T18:31:32.780689+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:31:32.780689+0000","last_undegraded":"2024-09-10T18:31:32.780689+0000","last_fullsized":"2024-09-10T18:31:32.780689+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:31:12.489588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:18.281160+0000","scrub_duration":0.00146783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":255,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.767704+0000","last_change":"2024-09-10T18:31:12.882344+0000","last_active":"2024-09-10T18:31:32.767704+0000","last_peered":"2024-09-10T18:31:32.767704+0000","last_clean":"2024-09-10T18:31:32.767704+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:31:32.767704+0000","last_undegraded":"2024-09-10T18:31:32.767704+0000","last_fullsized":"2024-09-10T18:31:32.767704+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:12.882248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:12.882248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:24.267881+0000","scrub_duration":0.0020040589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":225,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.783368+0000","last_change":"2024-09-10T18:31:12.803420+0000","last_active":"2024-09-10T18:31:32.783368+0000","last_peered":"2024-09-10T18:31:32.783368+0000","last_clean":"2024-09-10T18:31:32.783368+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:31:32.783368+0000","last_undegraded":"2024-09-10T18:31:32.783368+0000","last_fullsized":"2024-09-10T18:31:32.783368+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:12.803236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.803236+0000","last_clean_scrub_stamp":"2024-09-10T18:31:12.803236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:14.528857+0000","scrub_duration":0.0041535599999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"239'17096","reported_seq":17470,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:37.727477+0000","last_change":"2024-09-10T18:29:53.310710+0000","last_active":"2024-09-10T18:31:37.727477+0000","last_peered":"2024-09-10T18:31:37.727477+0000","last_clean":"2024-09-10T18:31:37.727477+0000","last_became_active":"2024-09-10T18:27:53.032159+0000","last_became_peered":"2024-09-10T18:27:53.032159+0000","last_unstale":"2024-09-10T18:31:37.727477+0000","last_undegraded":"2024-09-10T18:31:37.727477+0000","last_fullsized":"2024-09-10T18:31:37.727477+0000","mapping_epoch":180,"log_start":"225'12000","ondisk_log_start":"225'12000","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'9794","last_scrub_stamp":"2024-09-10T18:29:53.310629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:29:53.310629+0000","objects_scrubbed":7406,"log_size":5096,"log_dups_size":0,"ondisk_log_size":5096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.2945158860000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17096,"num_write_kb":547072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":278,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.241148+0000","last_change":"2024-09-10T18:29:36.972493+0000","last_active":"2024-09-10T18:31:39.241148+0000","last_peered":"2024-09-10T18:31:39.241148+0000","last_clean":"2024-09-10T18:31:39.241148+0000","last_became_active":"2024-09-10T18:26:27.645932+0000","last_became_peered":"2024-09-10T18:26:27.645932+0000","last_unstale":"2024-09-10T18:31:39.241148+0000","last_undegraded":"2024-09-10T18:31:39.241148+0000","last_fullsized":"2024-09-10T18:31:39.241148+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-10T18:29:36.972409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:29:36.972409+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012032055999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":249,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780640+0000","last_change":"2024-09-10T18:30:45.438521+0000","last_active":"2024-09-10T18:31:32.780640+0000","last_peered":"2024-09-10T18:31:32.780640+0000","last_clean":"2024-09-10T18:31:32.780640+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:31:32.780640+0000","last_undegraded":"2024-09-10T18:31:32.780640+0000","last_fullsized":"2024-09-10T18:31:32.780640+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:30:45.438437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:30:45.438437+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:07.661643+0000","scrub_duration":0.0026276839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":374,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:34.882691+0000","last_change":"2024-09-10T18:31:03.524139+0000","last_active":"2024-09-10T18:31:34.882691+0000","last_peered":"2024-09-10T18:31:34.882691+0000","last_clean":"2024-09-10T18:31:34.882691+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:31:34.882691+0000","last_undegraded":"2024-09-10T18:31:34.882691+0000","last_fullsized":"2024-09-10T18:31:34.882691+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:31:03.524066+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:25.302495+0000","scrub_duration":0.0040323920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":231,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781701+0000","last_change":"2024-09-10T18:31:14.326966+0000","last_active":"2024-09-10T18:31:32.781701+0000","last_peered":"2024-09-10T18:31:32.781701+0000","last_clean":"2024-09-10T18:31:32.781701+0000","last_became_active":"2024-09-10T18:26:27.647511+0000","last_became_peered":"2024-09-10T18:26:27.647511+0000","last_unstale":"2024-09-10T18:31:32.781701+0000","last_undegraded":"2024-09-10T18:31:32.781701+0000","last_fullsized":"2024-09-10T18:31:32.781701+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:14.326705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:14.326705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:18.962305+0000","scrub_duration":0.0018698650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":184,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.774012+0000","last_change":"2024-09-10T18:31:02.067834+0000","last_active":"2024-09-10T18:31:32.774012+0000","last_peered":"2024-09-10T18:31:32.774012+0000","last_clean":"2024-09-10T18:31:32.774012+0000","last_became_active":"2024-09-10T18:26:27.640922+0000","last_became_peered":"2024-09-10T18:26:27.640922+0000","last_unstale":"2024-09-10T18:31:32.774012+0000","last_undegraded":"2024-09-10T18:31:32.774012+0000","last_fullsized":"2024-09-10T18:31:32.774012+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:02.067753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:31:02.067753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:30.111230+0000","scrub_duration":0.011667588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"240'12","reported_seq":247,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.254594+0000","last_change":"2024-09-10T18:30:49.755553+0000","last_active":"2024-09-10T18:31:39.254594+0000","last_peered":"2024-09-10T18:31:39.254594+0000","last_clean":"2024-09-10T18:31:39.254594+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:31:39.254594+0000","last_undegraded":"2024-09-10T18:31:39.254594+0000","last_fullsized":"2024-09-10T18:31:39.254594+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-10T18:30:49.755466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:49.755466+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:09.195216+0000","scrub_duration":0.0040815260000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":225,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781283+0000","last_change":"2024-09-10T18:29:37.932209+0000","last_active":"2024-09-10T18:31:32.781283+0000","last_peered":"2024-09-10T18:31:32.781283+0000","last_clean":"2024-09-10T18:31:32.781283+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:31:32.781283+0000","last_undegraded":"2024-09-10T18:31:32.781283+0000","last_fullsized":"2024-09-10T18:31:32.781283+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:37.932136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:29:37.932136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053209679999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":240,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780619+0000","last_change":"2024-09-10T18:31:01.523924+0000","last_active":"2024-09-10T18:31:32.780619+0000","last_peered":"2024-09-10T18:31:32.780619+0000","last_clean":"2024-09-10T18:31:32.780619+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:31:32.780619+0000","last_undegraded":"2024-09-10T18:31:32.780619+0000","last_fullsized":"2024-09-10T18:31:32.780619+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:01.523839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:01.523839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:28.978824+0000","scrub_duration":0.0016753880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"240'19","reported_seq":260,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.083235+0000","last_change":"2024-09-10T18:30:58.459335+0000","last_active":"2024-09-10T18:31:39.083235+0000","last_peered":"2024-09-10T18:31:39.083235+0000","last_clean":"2024-09-10T18:31:39.083235+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:31:39.083235+0000","last_undegraded":"2024-09-10T18:31:39.083235+0000","last_fullsized":"2024-09-10T18:31:39.083235+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'13","last_scrub_stamp":"2024-09-10T18:30:58.459246+0000","last_deep_scrub":"189'13","last_deep_scrub_stamp":"2024-09-10T18:28:08.976162+0000","last_clean_scrub_stamp":"2024-09-10T18:30:58.459246+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:25.581872+0000","scrub_duration":0.0093093749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":246,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.779726+0000","last_change":"2024-09-10T18:31:00.524076+0000","last_active":"2024-09-10T18:31:32.779726+0000","last_peered":"2024-09-10T18:31:32.779726+0000","last_clean":"2024-09-10T18:31:32.779726+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:31:32.779726+0000","last_undegraded":"2024-09-10T18:31:32.779726+0000","last_fullsized":"2024-09-10T18:31:32.779726+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:00.523985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:31:00.523985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:28.606520+0000","scrub_duration":0.0018784559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":227,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.783755+0000","last_change":"2024-09-10T18:30:02.587567+0000","last_active":"2024-09-10T18:31:32.783755+0000","last_peered":"2024-09-10T18:31:32.783755+0000","last_clean":"2024-09-10T18:31:32.783755+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:31:32.783755+0000","last_undegraded":"2024-09-10T18:31:32.783755+0000","last_fullsized":"2024-09-10T18:31:32.783755+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:02.587404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:02.587404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006571632,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":243,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.768612+0000","last_change":"2024-09-10T18:30:59.094192+0000","last_active":"2024-09-10T18:31:32.768612+0000","last_peered":"2024-09-10T18:31:32.768612+0000","last_clean":"2024-09-10T18:31:32.768612+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:31:32.768612+0000","last_undegraded":"2024-09-10T18:31:32.768612+0000","last_fullsized":"2024-09-10T18:31:32.768612+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:59.094103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:59.094103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:13.830457+0000","scrub_duration":0.0095530630000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"240'19","reported_seq":257,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.272314+0000","last_change":"2024-09-10T18:29:34.995159+0000","last_active":"2024-09-10T18:31:39.272314+0000","last_peered":"2024-09-10T18:31:39.272314+0000","last_clean":"2024-09-10T18:31:39.272314+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:31:39.272314+0000","last_undegraded":"2024-09-10T18:31:39.272314+0000","last_fullsized":"2024-09-10T18:31:39.272314+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'13","last_scrub_stamp":"2024-09-10T18:29:34.995105+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:29:34.995105+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053157309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":230,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.779390+0000","last_change":"2024-09-10T18:28:57.977326+0000","last_active":"2024-09-10T18:31:32.779390+0000","last_peered":"2024-09-10T18:31:32.779390+0000","last_clean":"2024-09-10T18:31:32.779390+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:31:32.779390+0000","last_undegraded":"2024-09-10T18:31:32.779390+0000","last_fullsized":"2024-09-10T18:31:32.779390+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:57.977273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:57.977273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019164130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":357,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:38.735947+0000","last_change":"2024-09-10T18:29:57.457944+0000","last_active":"2024-09-10T18:31:38.735947+0000","last_peered":"2024-09-10T18:31:38.735947+0000","last_clean":"2024-09-10T18:31:38.735947+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:31:38.735947+0000","last_undegraded":"2024-09-10T18:31:38.735947+0000","last_fullsized":"2024-09-10T18:31:38.735947+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:29:57.457607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:29:57.457607+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033795549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":233,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.770943+0000","last_change":"2024-09-10T18:31:06.427465+0000","last_active":"2024-09-10T18:31:32.770943+0000","last_peered":"2024-09-10T18:31:32.770943+0000","last_clean":"2024-09-10T18:31:32.770943+0000","last_became_active":"2024-09-10T18:21:37.639706+0000","last_became_peered":"2024-09-10T18:21:37.639706+0000","last_unstale":"2024-09-10T18:31:32.770943+0000","last_undegraded":"2024-09-10T18:31:32.770943+0000","last_fullsized":"2024-09-10T18:31:32.770943+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:06.427389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:06.427389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:31.517707+0000","scrub_duration":0.0019673400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":259,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.270428+0000","last_change":"2024-09-10T18:29:01.377785+0000","last_active":"2024-09-10T18:31:39.270428+0000","last_peered":"2024-09-10T18:31:39.270428+0000","last_clean":"2024-09-10T18:31:39.270428+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:31:39.270428+0000","last_undegraded":"2024-09-10T18:31:39.270428+0000","last_fullsized":"2024-09-10T18:31:39.270428+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-09-10T18:29:01.377503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:29:01.377503+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0093114570000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":239,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780290+0000","last_change":"2024-09-10T18:30:53.519511+0000","last_active":"2024-09-10T18:31:32.780290+0000","last_peered":"2024-09-10T18:31:32.780290+0000","last_clean":"2024-09-10T18:31:32.780290+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:31:32.780290+0000","last_undegraded":"2024-09-10T18:31:32.780290+0000","last_fullsized":"2024-09-10T18:31:32.780290+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:53.519424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:30:53.519424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:58.732708+0000","scrub_duration":0.002185144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":366,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:38.500878+0000","last_change":"2024-09-10T18:30:53.452921+0000","last_active":"2024-09-10T18:31:38.500878+0000","last_peered":"2024-09-10T18:31:38.500878+0000","last_clean":"2024-09-10T18:31:38.500878+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:31:38.500878+0000","last_undegraded":"2024-09-10T18:31:38.500878+0000","last_fullsized":"2024-09-10T18:31:38.500878+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:30:53.452719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:53.452719+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:04.758158+0000","scrub_duration":0.0035769230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":224,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780888+0000","last_change":"2024-09-10T18:30:01.300052+0000","last_active":"2024-09-10T18:31:32.780888+0000","last_peered":"2024-09-10T18:31:32.780888+0000","last_clean":"2024-09-10T18:31:32.780888+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:31:32.780888+0000","last_undegraded":"2024-09-10T18:31:32.780888+0000","last_fullsized":"2024-09-10T18:31:32.780888+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:01.298802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:01.298802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.024539056,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":265,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.262106+0000","last_change":"2024-09-10T18:29:43.305438+0000","last_active":"2024-09-10T18:31:39.262106+0000","last_peered":"2024-09-10T18:31:39.262106+0000","last_clean":"2024-09-10T18:31:39.262106+0000","last_became_active":"2024-09-10T18:26:27.649202+0000","last_became_peered":"2024-09-10T18:26:27.649202+0000","last_unstale":"2024-09-10T18:31:39.262106+0000","last_undegraded":"2024-09-10T18:31:39.262106+0000","last_fullsized":"2024-09-10T18:31:39.262106+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:29:43.305328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:29:43.305328+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075088339999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":239,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.770714+0000","last_change":"2024-09-10T18:29:57.246363+0000","last_active":"2024-09-10T18:31:32.770714+0000","last_peered":"2024-09-10T18:31:32.770714+0000","last_clean":"2024-09-10T18:31:32.770714+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:31:32.770714+0000","last_undegraded":"2024-09-10T18:31:32.770714+0000","last_fullsized":"2024-09-10T18:31:32.770714+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:29:57.246282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019570709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":238,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.773448+0000","last_change":"2024-09-10T18:29:42.658563+0000","last_active":"2024-09-10T18:31:32.773448+0000","last_peered":"2024-09-10T18:31:32.773448+0000","last_clean":"2024-09-10T18:31:32.773448+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:31:32.773448+0000","last_undegraded":"2024-09-10T18:31:32.773448+0000","last_fullsized":"2024-09-10T18:31:32.773448+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:42.658490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:29:42.658490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059542680000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":215,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780515+0000","last_change":"2024-09-10T18:28:57.310828+0000","last_active":"2024-09-10T18:31:32.780515+0000","last_peered":"2024-09-10T18:31:32.780515+0000","last_clean":"2024-09-10T18:31:32.780515+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:31:32.780515+0000","last_undegraded":"2024-09-10T18:31:32.780515+0000","last_fullsized":"2024-09-10T18:31:32.780515+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:28:57.310730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054119199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"240'15","reported_seq":269,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.199786+0000","last_change":"2024-09-10T18:30:03.639358+0000","last_active":"2024-09-10T18:31:39.199786+0000","last_peered":"2024-09-10T18:31:39.199786+0000","last_clean":"2024-09-10T18:31:39.199786+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:31:39.199786+0000","last_undegraded":"2024-09-10T18:31:39.199786+0000","last_fullsized":"2024-09-10T18:31:39.199786+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:30:03.639256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:03.639256+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010855609,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":242,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.780841+0000","last_change":"2024-09-10T18:30:47.525989+0000","last_active":"2024-09-10T18:31:32.780841+0000","last_peered":"2024-09-10T18:31:32.780841+0000","last_clean":"2024-09-10T18:31:32.780841+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:31:32.780841+0000","last_undegraded":"2024-09-10T18:31:32.780841+0000","last_fullsized":"2024-09-10T18:31:32.780841+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:47.525912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:30:47.525912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:03.043559+0000","scrub_duration":0.0075901459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":217,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.783449+0000","last_change":"2024-09-10T18:28:54.267580+0000","last_active":"2024-09-10T18:31:32.783449+0000","last_peered":"2024-09-10T18:31:32.783449+0000","last_clean":"2024-09-10T18:31:32.783449+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:31:32.783449+0000","last_undegraded":"2024-09-10T18:31:32.783449+0000","last_fullsized":"2024-09-10T18:31:32.783449+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:54.267489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:28:54.267489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067110889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":224,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.779149+0000","last_change":"2024-09-10T18:29:41.038597+0000","last_active":"2024-09-10T18:31:32.779149+0000","last_peered":"2024-09-10T18:31:32.779149+0000","last_clean":"2024-09-10T18:31:32.779149+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:31:32.779149+0000","last_undegraded":"2024-09-10T18:31:32.779149+0000","last_fullsized":"2024-09-10T18:31:32.779149+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:41.038513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:41.038513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01221999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":267,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.288569+0000","last_change":"2024-09-10T18:29:54.385449+0000","last_active":"2024-09-10T18:31:39.288569+0000","last_peered":"2024-09-10T18:31:39.288569+0000","last_clean":"2024-09-10T18:31:39.288569+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:31:39.288569+0000","last_undegraded":"2024-09-10T18:31:39.288569+0000","last_fullsized":"2024-09-10T18:31:39.288569+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:29:54.385389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:29:54.385389+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002949199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":240,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781356+0000","last_change":"2024-09-10T18:30:50.439256+0000","last_active":"2024-09-10T18:31:32.781356+0000","last_peered":"2024-09-10T18:31:32.781356+0000","last_clean":"2024-09-10T18:31:32.781356+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:31:32.781356+0000","last_undegraded":"2024-09-10T18:31:32.781356+0000","last_fullsized":"2024-09-10T18:31:32.781356+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:50.439179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:30:50.439179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:00.808451+0000","scrub_duration":0.0056857770000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":235,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.779218+0000","last_change":"2024-09-10T18:29:45.432826+0000","last_active":"2024-09-10T18:31:32.779218+0000","last_peered":"2024-09-10T18:31:32.779218+0000","last_clean":"2024-09-10T18:31:32.779218+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:31:32.779218+0000","last_undegraded":"2024-09-10T18:31:32.779218+0000","last_fullsized":"2024-09-10T18:31:32.779218+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:45.432776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:29:45.432776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001236942,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":242,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.783590+0000","last_change":"2024-09-10T18:31:14.775776+0000","last_active":"2024-09-10T18:31:32.783590+0000","last_peered":"2024-09-10T18:31:32.783590+0000","last_clean":"2024-09-10T18:31:32.783590+0000","last_became_active":"2024-09-10T18:26:27.636996+0000","last_became_peered":"2024-09-10T18:26:27.636996+0000","last_unstale":"2024-09-10T18:31:32.783590+0000","last_undegraded":"2024-09-10T18:31:32.783590+0000","last_fullsized":"2024-09-10T18:31:32.783590+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:14.775693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:14.775693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:29.982163+0000","scrub_duration":0.00163416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":248,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.073238+0000","last_change":"2024-09-10T18:30:45.710780+0000","last_active":"2024-09-10T18:31:39.073238+0000","last_peered":"2024-09-10T18:31:39.073238+0000","last_clean":"2024-09-10T18:31:39.073238+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:31:39.073238+0000","last_undegraded":"2024-09-10T18:31:39.073238+0000","last_fullsized":"2024-09-10T18:31:39.073238+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-09-10T18:30:45.710665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:45.710665+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:53.558994+0000","scrub_duration":0.0078962489999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":233,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781672+0000","last_change":"2024-09-10T18:30:05.204555+0000","last_active":"2024-09-10T18:31:32.781672+0000","last_peered":"2024-09-10T18:31:32.781672+0000","last_clean":"2024-09-10T18:31:32.781672+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:31:32.781672+0000","last_undegraded":"2024-09-10T18:31:32.781672+0000","last_fullsized":"2024-09-10T18:31:32.781672+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:30:05.204467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021240970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":188,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.774095+0000","last_change":"2024-09-10T18:31:10.050895+0000","last_active":"2024-09-10T18:31:32.774095+0000","last_peered":"2024-09-10T18:31:32.774095+0000","last_clean":"2024-09-10T18:31:32.774095+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:31:32.774095+0000","last_undegraded":"2024-09-10T18:31:32.774095+0000","last_fullsized":"2024-09-10T18:31:32.774095+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:10.050810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:10.050810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:32.282596+0000","scrub_duration":0.0052628850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":233,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.779714+0000","last_change":"2024-09-10T18:30:58.517218+0000","last_active":"2024-09-10T18:31:32.779714+0000","last_peered":"2024-09-10T18:31:32.779714+0000","last_clean":"2024-09-10T18:31:32.779714+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:31:32.779714+0000","last_undegraded":"2024-09-10T18:31:32.779714+0000","last_fullsized":"2024-09-10T18:31:32.779714+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:58.517160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:58.517160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:14.392419+0000","scrub_duration":0.0070680939999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":233,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.773621+0000","last_change":"2024-09-10T18:29:38.649747+0000","last_active":"2024-09-10T18:31:32.773621+0000","last_peered":"2024-09-10T18:31:32.773621+0000","last_clean":"2024-09-10T18:31:32.773621+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:31:32.773621+0000","last_undegraded":"2024-09-10T18:31:32.773621+0000","last_fullsized":"2024-09-10T18:31:32.773621+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:38.649655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:38.649655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035279059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":221,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.787460+0000","last_change":"2024-09-10T18:28:51.230961+0000","last_active":"2024-09-10T18:31:32.787460+0000","last_peered":"2024-09-10T18:31:32.787460+0000","last_clean":"2024-09-10T18:31:32.787460+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:31:32.787460+0000","last_undegraded":"2024-09-10T18:31:32.787460+0000","last_fullsized":"2024-09-10T18:31:32.787460+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:51.230858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:28:51.230858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049907440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":239,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-10T18:31:39.290653+0000","last_change":"2024-09-10T18:30:50.074300+0000","last_active":"2024-09-10T18:31:39.290653+0000","last_peered":"2024-09-10T18:31:39.290653+0000","last_clean":"2024-09-10T18:31:39.290653+0000","last_became_active":"2024-09-10T18:26:27.648112+0000","last_became_peered":"2024-09-10T18:26:27.648112+0000","last_unstale":"2024-09-10T18:31:39.290653+0000","last_undegraded":"2024-09-10T18:31:39.290653+0000","last_fullsized":"2024-09-10T18:31:39.290653+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:30:50.074214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:50.074214+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:53.216359+0000","scrub_duration":0.00651967,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":226,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-10T18:31:32.781085+0000","last_change":"2024-09-10T18:29:05.092825+0000","last_active":"2024-09-10T18:31:32.781085+0000","last_peered":"2024-09-10T18:31:32.781085+0000","last_clean":"2024-09-10T18:31:32.781085+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:31:32.781085+0000","last_undegraded":"2024-09-10T18:31:32.781085+0000","last_fullsized":"2024-09-10T18:31:32.781085+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:29:05.092434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021443608999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":268438,"num_write_kb":8589953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80897,"ondisk_log_size":80897,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1126,"num_read_kb":883,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":731,"ondisk_log_size":731,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":601,"num_read_kb":596,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":891,"ondisk_log_size":891,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234109,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298384,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446768,"statfs":{"total":95995035648,"available":95689490432,"internally_reserved":0,"allocated":1056768,"data_stored":693254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:30:39 2024","interfaces":[{"interface":"back","average":{"1min":5.1440000000000001,"5min":3.2639999999999998,"15min":1.522},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20799999999999999},"max":{"1min":75.506,"5min":85.012,"15min":85.012},"last":0.497},{"interface":"front","average":{"1min":5.4290000000000003,"5min":3.173,"15min":1.512},"min":{"1min":0.33500000000000002,"5min":0.22800000000000001,"15min":0.222},"max":{"1min":75.710999999999999,"5min":84.665999999999997,"15min":84.665999999999997},"last":0.40999999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:30:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.9650000000000001,"15min":1.0980000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.335,"5min":49.317999999999998,"15min":49.317999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.151,"5min":1.849,"15min":1.087},"min":{"1min":0.28000000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.4929999999999999,"5min":49.353000000000002,"15min":49.353000000000002},"last":0.55000000000000004}]},{"osd":2,"last update":"Tue Sep 10 18:30:39 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":0.97499999999999998,"15min":0.73699999999999999},"min":{"1min":0.33800000000000002,"5min":0.249,"15min":0.249},"max":{"1min":5.008,"5min":5.3150000000000004,"15min":5.3150000000000004},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1579999999999999,"15min":0.79900000000000004},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.254},"max":{"1min":5.0460000000000003,"5min":9.9450000000000003,"15min":9.9450000000000003},"last":0.46800000000000003}]},{"osd":3,"last update":"Tue Sep 10 18:30:39 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.956,"15min":1.2310000000000001},"min":{"1min":0.28699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.3700000000000001,"5min":86.305999999999997,"15min":86.305999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.085,"15min":1.282},"min":{"1min":0.34899999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.1779999999999999,"5min":88.364000000000004,"15min":88.364000000000004},"last":0.56599999999999995}]},{"osd":4,"last update":"Tue Sep 10 18:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.53300000000000003,"15min":0.45600000000000002},"min":{"1min":0.249,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.4670000000000001,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.57699999999999996,"15min":0.47399999999999998},"min":{"1min":0.311,"5min":0.25,"15min":0.223},"max":{"1min":1.796,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":0.30199999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.52900000000000003},"min":{"1min":0.27600000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.6599999999999999,"5min":7.3109999999999999,"15min":7.3109999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.63300000000000001,"15min":0.51800000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.219},"max":{"1min":3.4300000000000002,"5min":3.7919999999999998,"15min":3.7919999999999998},"last":0.38800000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84899999999999998,"15min":0.56000000000000005},"min":{"1min":0.28899999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.3900000000000001,"5min":7.2750000000000004,"15min":7.2750000000000004},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.143,"5min":0.73199999999999998,"15min":0.52800000000000002},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.585,"5min":4.585,"15min":4.585},"last":0.36799999999999999}]}]},{"osd":6,"up_from":144,"seq":618475290688,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311616,"kb_used_data":696,"kb_used_omap":7,"kb_used_meta":310904,"kb_avail":93433536,"statfs":{"total":95995035648,"available":95675940864,"internally_reserved":0,"allocated":712704,"data_stored":365746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":318365783},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":1.8009999999999999,"15min":2.008},"min":{"1min":0.28100000000000003,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.3949999999999996,"5min":30.629999999999999,"15min":30.629999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.591,"5min":1.3839999999999999,"15min":1.49},"min":{"1min":0.23100000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":8.4640000000000004,"5min":30.597000000000001,"15min":30.597000000000001},"last":0.443}]},{"osd":1,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.3129999999999999,"15min":0.91800000000000004},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.0329999999999995,"5min":30.984000000000002,"15min":30.984000000000002},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.6819999999999999,"5min":1.46,"15min":1.157},"min":{"1min":0.246,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.7859999999999996,"5min":30.571000000000002,"15min":30.571000000000002},"last":0.42299999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.075},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.6539999999999999,"5min":4.8259999999999996,"15min":4.8259999999999996},"last":0.495},{"interface":"front","average":{"1min":1.448,"5min":1.0409999999999999,"15min":0.86199999999999999},"min":{"1min":0.33100000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":5.415,"5min":11.164999999999999,"15min":11.164999999999999},"last":0.40000000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.9079999999999999,"15min":2.0910000000000002},"min":{"1min":0.29799999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.8520000000000003,"5min":61.703000000000003,"15min":61.703000000000003},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.909,"15min":2.0750000000000002},"min":{"1min":0.29499999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.7229999999999999,"5min":61.814,"15min":61.814},"last":0.46899999999999997}]},{"osd":4,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.66400000000000003,"15min":0.54500000000000004},"min":{"1min":0.27600000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.6219999999999999,"5min":5.4950000000000001,"15min":5.4950000000000001},"last":0.371},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.68400000000000005,"15min":0.65700000000000003},"min":{"1min":0.27800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":4.6390000000000002,"5min":4.6390000000000002,"15min":4.6390000000000002},"last":0.38}]},{"osd":5,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.64300000000000002,"15min":0.52300000000000002},"min":{"1min":0.27200000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.6619999999999999,"5min":4.6619999999999999,"15min":4.6619999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.626,"15min":0.47799999999999998},"min":{"1min":0.313,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.6219999999999999,"5min":2.6219999999999999,"15min":2.6219999999999999},"last":0.43099999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.60099999999999998,"15min":0.52000000000000002},"min":{"1min":0.33800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.74199999999999999,"15min":0.56399999999999995},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.9300000000000002,"5min":5.5309999999999997,"15min":5.5309999999999997},"last":0.34200000000000003}]}]},{"osd":5,"up_from":39,"seq":167503724680,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306972,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93438180,"statfs":{"total":95995035648,"available":95680696320,"internally_reserved":0,"allocated":1327104,"data_stored":956448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:30:59 2024","interfaces":[{"interface":"back","average":{"1min":2.8849999999999998,"5min":2.407,"15min":1.385},"min":{"1min":0.23499999999999999,"5min":0.22900000000000001,"15min":0.21099999999999999},"max":{"1min":26.288,"5min":52.033999999999999,"15min":52.033999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":2.6549999999999998,"5min":2.4049999999999998,"15min":1.2210000000000001},"min":{"1min":0.33800000000000002,"5min":0.22800000000000001,"15min":0.216},"max":{"1min":24.178000000000001,"5min":52.167999999999999,"15min":52.167999999999999},"last":0.47899999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:30:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4689999999999999,"5min":1.296,"15min":0.82899999999999996},"min":{"1min":0.33100000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":24.16,"5min":24.16,"15min":24.16},"last":0.434},{"interface":"front","average":{"1min":2.5979999999999999,"5min":1.204,"15min":0.90200000000000002},"min":{"1min":0.28499999999999998,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":24.129999999999999,"5min":24.129999999999999,"15min":24.129999999999999},"last":0.44600000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:30:59 2024","interfaces":[{"interface":"back","average":{"1min":1.6699999999999999,"5min":1.621,"15min":0.96199999999999997},"min":{"1min":0.26900000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":16.736999999999998,"5min":36.853000000000002,"15min":36.853000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.304,"5min":1.5429999999999999,"15min":0.93000000000000005},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":8.1869999999999994,"5min":31.132000000000001,"15min":31.132000000000001},"last":0.53400000000000003}]},{"osd":3,"last update":"Tue Sep 10 18:30:59 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.101,"15min":1.2170000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":30.475000000000001,"5min":31.207000000000001,"15min":31.207000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.0350000000000001,"15min":1.119},"min":{"1min":0.38700000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":30.172000000000001,"5min":31.199000000000002,"15min":31.199000000000002},"last":0.39800000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:30:59 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.59299999999999997,"15min":0.436},"min":{"1min":0.22900000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.879,"5min":2.879,"15min":2.879},"last":0.371},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.64900000000000002,"15min":0.47299999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.22700000000000001},"max":{"1min":2.2730000000000001,"5min":4.7489999999999997,"15min":4.7489999999999997},"last":0.52100000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.79200000000000004,"15min":0.63700000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.194,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.82199999999999995,"15min":0.63300000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":5.3680000000000003,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.498}]},{"osd":7,"last update":"Tue Sep 10 18:31:29 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.80700000000000005,"15min":0.55300000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.22500000000000001},"max":{"1min":5.4900000000000002,"5min":5.569,"15min":5.569},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63900000000000001,"15min":0.496},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.24299999999999999},"max":{"1min":3.173,"5min":3.1819999999999999,"15min":3.1819999999999999},"last":0.51100000000000001}]}]},{"osd":4,"up_from":33,"seq":141733920908,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298064,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447088,"statfs":{"total":95995035648,"available":95689818112,"internally_reserved":0,"allocated":729088,"data_stored":365385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:31:30 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.383,"15min":0.99099999999999999},"min":{"1min":0.26900000000000002,"5min":0.21199999999999999,"15min":0.188},"max":{"1min":4.2009999999999996,"5min":16.236999999999998,"15min":31.585000000000001},"last":0.435},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.125,"15min":0.94599999999999995},"min":{"1min":0.23000000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.3129999999999997,"5min":16.280999999999999,"15min":29.802},"last":0.39200000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:31:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.93999999999999995,"15min":0.745},"min":{"1min":0.24399999999999999,"5min":0.22600000000000001,"15min":0.21299999999999999},"max":{"1min":6.8559999999999999,"5min":6.8559999999999999,"15min":6.8559999999999999},"last":0.502},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.1060000000000001,"15min":0.79700000000000004},"min":{"1min":0.309,"5min":0.246,"15min":0.22900000000000001},"max":{"1min":4.1210000000000004,"5min":5.085,"15min":5.085},"last":0.47899999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":1.008,"15min":0.83599999999999997},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.0519999999999996,"5min":6.4180000000000001,"15min":13.898999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":1.056,"5min":0.96599999999999997,"15min":0.85199999999999998},"min":{"1min":0.29399999999999998,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":4.0460000000000003,"5min":4.1950000000000003,"15min":14.51},"last":0.44700000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:31:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.464,"15min":1.052},"min":{"1min":0.26300000000000001,"5min":0.249,"15min":0.249},"max":{"1min":3.6709999999999998,"5min":37.351999999999997,"15min":37.351999999999997},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.4470000000000001,"15min":1.0920000000000001},"min":{"1min":0.35199999999999998,"5min":0.28000000000000003,"15min":0.255},"max":{"1min":2.8639999999999999,"5min":37.366,"15min":37.366},"last":0.46899999999999997}]},{"osd":5,"last update":"Tue Sep 10 18:30:50 2024","interfaces":[{"interface":"back","average":{"1min":1.8100000000000001,"5min":0.82099999999999995,"15min":0.52100000000000002},"min":{"1min":0.28000000000000003,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":22.861000000000001,"5min":22.861000000000001,"15min":22.861000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.6459999999999999,"5min":0.84299999999999997,"15min":0.55000000000000004},"min":{"1min":0.32900000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":19.611999999999998,"5min":19.611999999999998,"15min":19.611999999999998},"last":0.35499999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58099999999999996,"15min":0.45100000000000001},"min":{"1min":0.27900000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.5310000000000001,"5min":6.7990000000000004,"15min":6.7990000000000004},"last":0.376},{"interface":"front","average":{"1min":0.628,"5min":0.66200000000000003,"15min":0.51600000000000001},"min":{"1min":0.33200000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.883,"5min":7.0300000000000002,"15min":7.0300000000000002},"last":0.45700000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.64900000000000002,"15min":0.53200000000000003},"min":{"1min":0.317,"5min":0.28799999999999998,"15min":0.255},"max":{"1min":4.8600000000000003,"5min":7.0670000000000002,"15min":7.0670000000000002},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.69599999999999995,"15min":0.54300000000000004},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.247},"max":{"1min":2.6139999999999999,"5min":6.5330000000000004,"15min":6.5330000000000004},"last":0.52300000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117138,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311140,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93434012,"statfs":{"total":95995035648,"available":95676428288,"internally_reserved":0,"allocated":1335296,"data_stored":956597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.5569999999999999,"15min":1.0880000000000001},"min":{"1min":0.33800000000000002,"5min":0.218,"15min":0.19900000000000001},"max":{"1min":3.3769999999999998,"5min":7.0899999999999999,"15min":8.5530000000000008},"last":0.24099999999999999},{"interface":"front","average":{"1min":2.0209999999999999,"5min":1.722,"15min":1.1399999999999999},"min":{"1min":0.307,"5min":0.193,"15min":0.193},"max":{"1min":6.7960000000000003,"5min":13.659000000000001,"15min":13.659000000000001},"last":0.27900000000000003}]},{"osd":1,"last update":"Tue Sep 10 18:31:08 2024","interfaces":[{"interface":"back","average":{"1min":3.8330000000000002,"5min":2.0419999999999998,"15min":1.218},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.247},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.34599999999999997},{"interface":"front","average":{"1min":4.5890000000000004,"5min":2.0579999999999998,"15min":1.1519999999999999},"min":{"1min":0.25,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":36.902000000000001,"5min":36.902000000000001,"15min":36.902000000000001},"last":0.309}]},{"osd":2,"last update":"Tue Sep 10 18:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.29,"15min":0.95799999999999996},"min":{"1min":0.25700000000000001,"5min":0.222,"15min":0.222},"max":{"1min":3.403,"5min":7.0439999999999996,"15min":7.7960000000000003},"last":0.33200000000000002},{"interface":"front","average":{"1min":1.9710000000000001,"5min":1.8380000000000001,"15min":1.117},"min":{"1min":0.29099999999999998,"5min":0.217,"15min":0.217},"max":{"1min":14.997999999999999,"5min":14.997999999999999,"15min":14.997999999999999},"last":0.317}]},{"osd":4,"last update":"Tue Sep 10 18:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.5840000000000001,"15min":1.0409999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":4.1159999999999997,"5min":17.649000000000001,"15min":23.684999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":1.04,"5min":1.2030000000000001,"15min":1.0009999999999999},"min":{"1min":0.36799999999999999,"5min":0.29699999999999999,"15min":0.27200000000000002},"max":{"1min":2.3999999999999999,"5min":4.6050000000000004,"15min":23.649000000000001},"last":0.499}]},{"osd":5,"last update":"Tue Sep 10 18:30:47 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":1.5169999999999999,"15min":1.133},"min":{"1min":0.35399999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.2250000000000001,"5min":7.3339999999999996,"15min":23.719999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.2410000000000001,"15min":1.0900000000000001},"min":{"1min":0.35699999999999998,"5min":0.35399999999999998,"15min":0.33100000000000002},"max":{"1min":2.3399999999999999,"5min":5.3440000000000003,"15min":23.707000000000001},"last":0.436}]},{"osd":6,"last update":"Tue Sep 10 18:30:40 2024","interfaces":[{"interface":"back","average":{"1min":2.7280000000000002,"5min":1.6100000000000001,"15min":1.123},"min":{"1min":0.32200000000000001,"5min":0.317,"15min":0.317},"max":{"1min":31.815999999999999,"5min":31.815999999999999,"15min":31.815999999999999},"last":0.379},{"interface":"front","average":{"1min":3.274,"5min":1.7450000000000001,"15min":1.4690000000000001},"min":{"1min":0.33400000000000002,"5min":0.315,"15min":0.315},"max":{"1min":31.742000000000001,"5min":31.742000000000001,"15min":31.742000000000001},"last":0.45400000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:30:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.5760000000000001,"15min":1.079},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.2599999999999998,"5min":7.2389999999999999,"15min":23.748999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":1.3169999999999999,"15min":1.0860000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.307},"max":{"1min":2.3660000000000001,"5min":11.571999999999999,"15min":25.058},"last":0.47899999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411478,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310792,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93434360,"statfs":{"total":95995035648,"available":95676784640,"internally_reserved":0,"allocated":978944,"data_stored":627545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:30:42 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.784,"15min":1.1240000000000001},"min":{"1min":0.216,"5min":0.185,"15min":0.185},"max":{"1min":4.7430000000000003,"5min":43.890000000000001,"15min":43.890000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":2.1549999999999998,"5min":2.355,"15min":1.325},"min":{"1min":0.33000000000000002,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":9.8209999999999997,"5min":44.331000000000003,"15min":44.331000000000003},"last":0.36799999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:30:42 2024","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":2.0179999999999998,"15min":1.1879999999999999},"min":{"1min":0.22,"5min":0.193,"15min":0.193},"max":{"1min":5.1719999999999997,"5min":44.575000000000003,"15min":44.575000000000003},"last":0.30399999999999999},{"interface":"front","average":{"1min":1.8320000000000001,"5min":1.7,"15min":1.002},"min":{"1min":0.38500000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.0330000000000004,"5min":44.368000000000002,"15min":44.368000000000002},"last":0.42899999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:31:04 2024","interfaces":[{"interface":"back","average":{"1min":1.881,"5min":1.861,"15min":1.3400000000000001},"min":{"1min":0.27000000000000002,"5min":0.26300000000000001,"15min":0.251},"max":{"1min":7.181,"5min":12.579000000000001,"15min":60.786999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.8620000000000001,"15min":1.3620000000000001},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.214},"max":{"1min":6.7999999999999998,"5min":12.712999999999999,"15min":60.116999999999997},"last":0.40699999999999997}]},{"osd":4,"last update":"Tue Sep 10 18:31:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.1739999999999999,"15min":0.93600000000000005},"min":{"1min":0.28699999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":4.641,"5min":5.9249999999999998,"15min":6.9169999999999998},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.161,"15min":0.91200000000000003},"min":{"1min":0.34200000000000003,"5min":0.32900000000000001,"15min":0.317},"max":{"1min":7.0730000000000004,"5min":7.0730000000000004,"15min":12.130000000000001},"last":0.78900000000000003}]},{"osd":5,"last update":"Tue Sep 10 18:30:52 2024","interfaces":[{"interface":"back","average":{"1min":1.982,"5min":1.1899999999999999,"15min":0.89600000000000002},"min":{"1min":0.30299999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":7.1609999999999996,"5min":7.1609999999999996,"15min":12.106},"last":0.754},{"interface":"front","average":{"1min":2.0390000000000001,"5min":1.1910000000000001,"15min":0.91100000000000003},"min":{"1min":0.41999999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":7.1109999999999998,"5min":7.1109999999999998,"15min":12.154},"last":0.63400000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:31:35 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.226,"15min":1.155},"min":{"1min":0.36799999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":4.6849999999999996,"5min":5.9989999999999997,"15min":5.9989999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.552,"5min":1.234,"15min":0.97499999999999998},"min":{"1min":0.39200000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":7.1349999999999998,"5min":7.1349999999999998,"15min":7.1349999999999998},"last":0.70299999999999996}]},{"osd":7,"last update":"Tue Sep 10 18:30:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.212,"15min":0.93400000000000005},"min":{"1min":0.35399999999999998,"5min":0.32100000000000001,"15min":0.30499999999999999},"max":{"1min":3.4420000000000002,"5min":6.2990000000000004,"15min":7.1639999999999997},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.645,"5min":1.3049999999999999,"15min":0.95499999999999996},"min":{"1min":0.39800000000000002,"5min":0.307,"15min":0.307},"max":{"1min":4.6769999999999996,"5min":5.9660000000000002,"15min":6.9669999999999996},"last":0.67400000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575003,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302240,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93442912,"statfs":{"total":95995035648,"available":95685541888,"internally_reserved":0,"allocated":741376,"data_stored":365987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:31:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.923,"15min":1.3700000000000001},"min":{"1min":0.33400000000000002,"5min":0.20399999999999999,"15min":0.16700000000000001},"max":{"1min":3.202,"5min":42.420000000000002,"15min":80.456999999999994},"last":0.377},{"interface":"front","average":{"1min":1.46,"5min":1.9710000000000001,"15min":1.454},"min":{"1min":0.38100000000000001,"5min":0.187,"15min":0.187},"max":{"1min":5.6189999999999998,"5min":42.548000000000002,"15min":80.578000000000003},"last":0.437}]},{"osd":2,"last update":"Tue Sep 10 18:31:38 2024","interfaces":[{"interface":"back","average":{"1min":1.706,"5min":1.579,"15min":1.093},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.6479999999999997,"5min":9.4700000000000006,"15min":13.035},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.5940000000000001,"15min":1.026},"min":{"1min":0.29899999999999999,"5min":0.216,"15min":0.19900000000000001},"max":{"1min":4.0670000000000002,"5min":12.222,"15min":12.222},"last":0.30299999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":2.0179999999999998,"15min":1.3939999999999999},"min":{"1min":0.35799999999999998,"5min":0.23899999999999999,"15min":0.22600000000000001},"max":{"1min":3.3450000000000002,"5min":43.076000000000001,"15min":43.076000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":1.4399999999999999,"5min":2.1560000000000001,"15min":1.476},"min":{"1min":0.32400000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.5860000000000003,"5min":41.93,"15min":41.93},"last":0.33700000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:31:26 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.345,"15min":0.94099999999999995},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.29799999999999999},"max":{"1min":3.9100000000000001,"5min":7.2859999999999996,"15min":7.2859999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.4770000000000001,"15min":1.0449999999999999},"min":{"1min":0.38100000000000001,"5min":0.30199999999999999,"15min":0.28299999999999997},"max":{"1min":4.0359999999999996,"5min":6.1600000000000001,"15min":19.393000000000001},"last":0.58599999999999997}]},{"osd":5,"last update":"Tue Sep 10 18:30:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.6639999999999999,"15min":1.05},"min":{"1min":0.35899999999999999,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":3.9990000000000001,"5min":12.206,"15min":15.48},"last":0.5},{"interface":"front","average":{"1min":1.151,"5min":1.48,"15min":0.95899999999999996},"min":{"1min":0.38600000000000001,"5min":0.34100000000000003,"15min":0.308},"max":{"1min":3.9119999999999999,"5min":9.9359999999999999,"15min":9.9359999999999999},"last":0.57399999999999995}]},{"osd":6,"last update":"Tue Sep 10 18:31:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.337,"15min":1.347},"min":{"1min":0.42599999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.7320000000000002,"5min":10.039,"15min":10.039},"last":0.51900000000000002},{"interface":"front","average":{"1min":1.353,"5min":1.4570000000000001,"15min":1.3999999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.9449999999999998,"5min":7.3369999999999997,"15min":7.3369999999999997},"last":0.60199999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:31:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.3680000000000001,"15min":1.036},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.34399999999999997},"max":{"1min":3.7530000000000001,"5min":10.005000000000001,"15min":15.545999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.538,"5min":1.393,"15min":0.999},"min":{"1min":0.38600000000000001,"5min":0.35699999999999998,"15min":0.313},"max":{"1min":5.5289999999999999,"5min":7.359,"15min":7.359},"last":0.628}]}]},{"osd":0,"up_from":8,"seq":34359738528,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310544,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93434608,"statfs":{"total":95995035648,"available":95677038592,"internally_reserved":0,"allocated":724992,"data_stored":365258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:31:16 2024","interfaces":[{"interface":"back","average":{"1min":1.8280000000000001,"5min":2.0350000000000001,"15min":1.2150000000000001},"min":{"1min":0.23400000000000001,"5min":0.193,"15min":0.193},"max":{"1min":7.2800000000000002,"5min":33.130000000000003,"15min":33.130000000000003},"last":0.34399999999999997},{"interface":"front","average":{"1min":2.5680000000000001,"5min":2.2050000000000001,"15min":1.3240000000000001},"min":{"1min":0.46899999999999997,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":7.0289999999999999,"5min":31.329999999999998,"15min":31.329999999999998},"last":0.36099999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:30:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.875,"15min":1.125},"min":{"1min":0.36199999999999999,"5min":0.22,"15min":0.189},"max":{"1min":4.2380000000000004,"5min":30.219999999999999,"15min":30.219999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.343,"5min":1.972,"15min":1.143},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":4.7670000000000003,"5min":30.129999999999999,"15min":30.129999999999999},"last":0.32800000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.7629999999999999,"5min":1.8,"15min":1.282},"min":{"1min":0.41199999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":6.5369999999999999,"5min":8.5709999999999997,"15min":23.143999999999998},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.9730000000000001,"5min":1.742,"15min":1.2509999999999999},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.1070000000000002,"5min":9.3360000000000003,"15min":23.035},"last":0.504}]},{"osd":4,"last update":"Tue Sep 10 18:31:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.349,"15min":1.093},"min":{"1min":0.34100000000000003,"5min":0.246,"15min":0.246},"max":{"1min":4.2640000000000002,"5min":8.3699999999999992,"15min":17.151},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3160000000000001,"15min":1.0780000000000001},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.25900000000000001},"max":{"1min":3.7919999999999998,"5min":18.728999999999999,"15min":18.728999999999999},"last":0.57699999999999996}]},{"osd":5,"last update":"Tue Sep 10 18:30:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":1.478,"15min":1.1559999999999999},"min":{"1min":0.35399999999999998,"5min":0.311,"15min":0.29499999999999998},"max":{"1min":3.746,"5min":18.77,"15min":18.77},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.4419999999999999,"15min":1.0609999999999999},"min":{"1min":0.377,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.9969999999999999,"5min":6.3769999999999998,"15min":14.991},"last":0.70399999999999996}]},{"osd":6,"last update":"Tue Sep 10 18:31:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":1.3100000000000001,"15min":1.3080000000000001},"min":{"1min":0.372,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.2939999999999996,"5min":8.3940000000000001,"15min":8.3940000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.375,"5min":1.3500000000000001,"15min":1.3939999999999999},"min":{"1min":0.48999999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.9079999999999999,"5min":9.7859999999999996,"15min":9.7859999999999996},"last":0.55900000000000005}]},{"osd":7,"last update":"Tue Sep 10 18:31:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.341,"15min":1.0760000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.26700000000000002},"max":{"1min":3.8660000000000001,"5min":5.0430000000000001,"15min":13.711},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.409,"5min":1.387,"15min":1.0669999999999999},"min":{"1min":0.46100000000000002,"5min":0.36199999999999999,"15min":0.33500000000000002},"max":{"1min":6.6399999999999997,"5min":9.6549999999999994,"15min":15.032},"last":0.65400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:31:45.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:31:45.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:45.617 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:45.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:45.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:45.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:45.747 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:31:45.747 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:31:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:45.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:45.760 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:31:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:31:45.776 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:31:45.777 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:31:45.791 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:31:45.791 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:31:45.808 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:31:45.808 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:31:45.823 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:31:45.823 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:31:45.838 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:31:45.838 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:31:45.848 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:31:45.849 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:31:45.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:45.855 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:31:45.855 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:31:45.877 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:45.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:45.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:45.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:46.000 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:46.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:46.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:46.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:46.124 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:46.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:46.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:46.248 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:46.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:46.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:46.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[29291]: from='client.15342 v1:172.21.15.28:0/929369877' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:31:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:31:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:31:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[29291]: pgmap v457: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 447 op/s 2024-09-10T18:31:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[29291]: osdmap e242: 8 total, 8 up, 8 in 2024-09-10T18:31:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[29291]: 4.15 scrub starts 2024-09-10T18:31:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[29291]: 4.15 scrub ok 2024-09-10T18:31:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/490577554' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:31:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[29291]: 6.11 scrub starts 2024-09-10T18:31:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[29291]: 6.11 scrub ok 2024-09-10T18:31:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[39220]: from='client.15342 v1:172.21.15.28:0/929369877' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:31:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:31:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:31:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[39220]: pgmap v457: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 447 op/s 2024-09-10T18:31:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[39220]: osdmap e242: 8 total, 8 up, 8 in 2024-09-10T18:31:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[39220]: 4.15 scrub starts 2024-09-10T18:31:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[39220]: 4.15 scrub ok 2024-09-10T18:31:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/490577554' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:31:46.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[39220]: 6.11 scrub starts 2024-09-10T18:31:46.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:46 smithi028 ceph-mon[39220]: 6.11 scrub ok 2024-09-10T18:31:46.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:46 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:46 smithi204 ceph-mon[32357]: from='client.15342 v1:172.21.15.28:0/929369877' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:31:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:46 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:31:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:46 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:31:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:46 smithi204 ceph-mon[32357]: pgmap v457: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 447 op/s 2024-09-10T18:31:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:46 smithi204 ceph-mon[32357]: osdmap e242: 8 total, 8 up, 8 in 2024-09-10T18:31:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:46 smithi204 ceph-mon[32357]: 4.15 scrub starts 2024-09-10T18:31:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:46 smithi204 ceph-mon[32357]: 4.15 scrub ok 2024-09-10T18:31:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:46 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/490577554' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:31:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:46 smithi204 ceph-mon[32357]: 6.11 scrub starts 2024-09-10T18:31:46.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:46 smithi204 ceph-mon[32357]: 6.11 scrub ok 2024-09-10T18:31:46.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:46.374 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:46.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:46.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:46.497 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:46.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:46.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:46.599 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:46.621 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:46.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:46.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:46.673 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-10T18:31:46.723 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:46.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:46.746 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:46.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:46.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:46.871 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:46.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:46.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:46.920 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:31:46.972 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:46.994 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:46.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:46.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:47.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:47.124 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:47.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:47.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:47.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:47.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:47.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:47.318 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:47 smithi204 ceph-mon[32357]: 3.d scrub starts 2024-09-10T18:31:47.318 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:47 smithi204 ceph-mon[32357]: 3.d scrub ok 2024-09-10T18:31:47.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:47.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:47.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:47.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:47 smithi028 ceph-mon[29291]: 3.d scrub starts 2024-09-10T18:31:47.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:47 smithi028 ceph-mon[29291]: 3.d scrub ok 2024-09-10T18:31:47.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:47 smithi028 ceph-mon[39220]: 3.d scrub starts 2024-09-10T18:31:47.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:47 smithi028 ceph-mon[39220]: 3.d scrub ok 2024-09-10T18:31:47.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:47.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:47.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:47.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:47.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:47.622 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:47.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:47.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:47.667 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:47.724 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:47.745 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:47.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:47.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:47.870 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:47.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:47.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:47.940 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:47.972 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:47.993 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:47.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:47.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:48.095 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:48.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:48.118 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:48.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:48.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:48.243 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:48.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:48.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:48.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:48 smithi028 ceph-mon[39220]: 4.1 scrub starts 2024-09-10T18:31:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:48 smithi028 ceph-mon[39220]: 4.1 scrub ok 2024-09-10T18:31:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:48 smithi028 ceph-mon[39220]: pgmap v459: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 447 op/s 2024-09-10T18:31:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:48 smithi028 ceph-mon[39220]: 4.8 scrub starts 2024-09-10T18:31:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:48 smithi028 ceph-mon[39220]: 4.8 scrub ok 2024-09-10T18:31:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:48 smithi028 ceph-mon[29291]: 4.1 scrub starts 2024-09-10T18:31:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:48 smithi028 ceph-mon[29291]: 4.1 scrub ok 2024-09-10T18:31:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:48 smithi028 ceph-mon[29291]: pgmap v459: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 447 op/s 2024-09-10T18:31:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:48 smithi028 ceph-mon[29291]: 4.8 scrub starts 2024-09-10T18:31:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:48 smithi028 ceph-mon[29291]: 4.8 scrub ok 2024-09-10T18:31:48.345 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:48.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:48 smithi204 ceph-mon[32357]: 4.1 scrub starts 2024-09-10T18:31:48.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:48 smithi204 ceph-mon[32357]: 4.1 scrub ok 2024-09-10T18:31:48.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:48 smithi204 ceph-mon[32357]: pgmap v459: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 447 op/s 2024-09-10T18:31:48.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:48 smithi204 ceph-mon[32357]: 4.8 scrub starts 2024-09-10T18:31:48.346 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:48 smithi204 ceph-mon[32357]: 4.8 scrub ok 2024-09-10T18:31:48.387 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:48.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:48.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:48.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:48.511 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:48.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:48.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:48.568 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:31:48.569 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:31:48.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:48.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:48.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:48.739 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:48.761 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:48.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:48.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:48.863 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:48.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:48.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:48.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:48.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:49.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:49.009 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:49.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:49.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:31:49.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:49.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:49.167 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:49.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:49.229 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-10T18:31:49.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:49.291 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:49.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:49.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:49.347 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:49 smithi028 ceph-mon[39220]: 4.13 scrub starts 2024-09-10T18:31:49.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:49 smithi028 ceph-mon[39220]: 4.13 scrub ok 2024-09-10T18:31:49.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:49 smithi028 ceph-mon[39220]: 5.2 scrub starts 2024-09-10T18:31:49.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:49 smithi028 ceph-mon[39220]: 5.2 scrub ok 2024-09-10T18:31:49.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:49 smithi028 ceph-mon[39220]: 1.3 scrub starts 2024-09-10T18:31:49.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:49 smithi028 ceph-mon[39220]: 1.3 scrub ok 2024-09-10T18:31:49.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:49 smithi028 ceph-mon[39220]: 4.14 scrub starts 2024-09-10T18:31:49.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:49 smithi028 ceph-mon[39220]: 4.14 scrub ok 2024-09-10T18:31:49.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:49 smithi028 ceph-mon[29291]: 4.13 scrub starts 2024-09-10T18:31:49.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:49 smithi028 ceph-mon[29291]: 4.13 scrub ok 2024-09-10T18:31:49.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:49 smithi028 ceph-mon[29291]: 5.2 scrub starts 2024-09-10T18:31:49.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:49 smithi028 ceph-mon[29291]: 5.2 scrub ok 2024-09-10T18:31:49.349 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:49 smithi028 ceph-mon[29291]: 1.3 scrub starts 2024-09-10T18:31:49.349 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:49 smithi028 ceph-mon[29291]: 1.3 scrub ok 2024-09-10T18:31:49.349 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:49 smithi028 ceph-mon[29291]: 4.14 scrub starts 2024-09-10T18:31:49.349 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:49 smithi028 ceph-mon[29291]: 4.14 scrub ok 2024-09-10T18:31:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:49 smithi204 ceph-mon[32357]: 4.13 scrub starts 2024-09-10T18:31:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:49 smithi204 ceph-mon[32357]: 4.13 scrub ok 2024-09-10T18:31:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:49 smithi204 ceph-mon[32357]: 5.2 scrub starts 2024-09-10T18:31:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:49 smithi204 ceph-mon[32357]: 5.2 scrub ok 2024-09-10T18:31:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:49 smithi204 ceph-mon[32357]: 1.3 scrub starts 2024-09-10T18:31:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:49 smithi204 ceph-mon[32357]: 1.3 scrub ok 2024-09-10T18:31:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:49 smithi204 ceph-mon[32357]: 4.14 scrub starts 2024-09-10T18:31:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:49 smithi204 ceph-mon[32357]: 4.14 scrub ok 2024-09-10T18:31:49.394 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:49.416 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:49.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:49.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:49.475 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:31:49.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:49.541 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:49.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:49.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:49.642 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:49.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:49.666 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:49.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:49.767 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:49.788 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:49.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:49.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:49.890 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:49.913 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:49.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:50.014 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:50.036 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:50.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:50.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:50.161 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:50.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:50 smithi028 ceph-mon[39220]: 5.18 scrub starts 2024-09-10T18:31:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:50 smithi028 ceph-mon[39220]: 5.18 scrub ok 2024-09-10T18:31:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:50 smithi028 ceph-mon[39220]: pgmap v460: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:31:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:50 smithi028 ceph-mon[39220]: 5.1f scrub starts 2024-09-10T18:31:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:50 smithi028 ceph-mon[39220]: 5.1f scrub ok 2024-09-10T18:31:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:50 smithi028 ceph-mon[39220]: 6.1d scrub starts 2024-09-10T18:31:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:50 smithi028 ceph-mon[39220]: 6.1d scrub ok 2024-09-10T18:31:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:50 smithi028 ceph-mon[29291]: 5.18 scrub starts 2024-09-10T18:31:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:50 smithi028 ceph-mon[29291]: 5.18 scrub ok 2024-09-10T18:31:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:50 smithi028 ceph-mon[29291]: pgmap v460: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:31:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:50 smithi028 ceph-mon[29291]: 5.1f scrub starts 2024-09-10T18:31:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:50 smithi028 ceph-mon[29291]: 5.1f scrub ok 2024-09-10T18:31:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:50 smithi028 ceph-mon[29291]: 6.1d scrub starts 2024-09-10T18:31:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:50 smithi028 ceph-mon[29291]: 6.1d scrub ok 2024-09-10T18:31:50.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:50.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:50.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:50.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:50 smithi204 ceph-mon[32357]: 5.18 scrub starts 2024-09-10T18:31:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:50 smithi204 ceph-mon[32357]: 5.18 scrub ok 2024-09-10T18:31:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:50 smithi204 ceph-mon[32357]: pgmap v460: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:31:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:50 smithi204 ceph-mon[32357]: 5.1f scrub starts 2024-09-10T18:31:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:50 smithi204 ceph-mon[32357]: 5.1f scrub ok 2024-09-10T18:31:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:50 smithi204 ceph-mon[32357]: 6.1d scrub starts 2024-09-10T18:31:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:50 smithi204 ceph-mon[32357]: 6.1d scrub ok 2024-09-10T18:31:50.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:50.412 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:50.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:50.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:50.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:50.536 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:50.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:50.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:50.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:50.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:50.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:50.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:50.765 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:50.787 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:50.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:50.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:50.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:50.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:50.911 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:50.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:51.012 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:51.033 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:51.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:51.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:51.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:51.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:51 smithi028 ceph-mon[39220]: 3.3 scrub starts 2024-09-10T18:31:51.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:51 smithi028 ceph-mon[39220]: 3.3 scrub ok 2024-09-10T18:31:51.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:51 smithi028 ceph-mon[39220]: 1.8 scrub starts 2024-09-10T18:31:51.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:51 smithi028 ceph-mon[39220]: 1.8 scrub ok 2024-09-10T18:31:51.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:51 smithi028 ceph-mon[39220]: 6.c scrub starts 2024-09-10T18:31:51.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:51 smithi028 ceph-mon[39220]: 6.c scrub ok 2024-09-10T18:31:51.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:51 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3237762469' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:31:51.153 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:51 smithi028 ceph-mon[39220]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:31:51.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:51 smithi028 ceph-mon[29291]: 3.3 scrub starts 2024-09-10T18:31:51.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:51 smithi028 ceph-mon[29291]: 3.3 scrub ok 2024-09-10T18:31:51.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:51 smithi028 ceph-mon[29291]: 1.8 scrub starts 2024-09-10T18:31:51.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:51 smithi028 ceph-mon[29291]: 1.8 scrub ok 2024-09-10T18:31:51.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:51 smithi028 ceph-mon[29291]: 6.c scrub starts 2024-09-10T18:31:51.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:51 smithi028 ceph-mon[29291]: 6.c scrub ok 2024-09-10T18:31:51.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:51 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3237762469' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:31:51.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:51 smithi028 ceph-mon[29291]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:31:51.158 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:51.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:51.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:51.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:51 smithi204 ceph-mon[32357]: 3.3 scrub starts 2024-09-10T18:31:51.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:51 smithi204 ceph-mon[32357]: 3.3 scrub ok 2024-09-10T18:31:51.220 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:51 smithi204 ceph-mon[32357]: 1.8 scrub starts 2024-09-10T18:31:51.220 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:51 smithi204 ceph-mon[32357]: 1.8 scrub ok 2024-09-10T18:31:51.220 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:51 smithi204 ceph-mon[32357]: 6.c scrub starts 2024-09-10T18:31:51.220 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:51 smithi204 ceph-mon[32357]: 6.c scrub ok 2024-09-10T18:31:51.220 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:51 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3237762469' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:31:51.220 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:51 smithi204 ceph-mon[32357]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:31:51.225 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:31:51.226 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:31:51.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:51.282 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:51.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:51.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:51.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:51.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:51.407 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:51.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:51.487 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:31:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:31:51] "GET /metrics HTTP/1.1" 200 224321 "" "Prometheus/2.43.0" 2024-09-10T18:31:51.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:51.537 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:31:51.540 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:31:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:51.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:51.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:51.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_1' removed 2024-09-10T18:31:51.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:51.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:51.797 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:51.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:51.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:51.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:51.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:51.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:52.028 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-10T18:31:52.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:52.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:52.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:52.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:52.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:52.179 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:52.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:52.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:52.273 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:31:52.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:52.304 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:52.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:52.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:52.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:31:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:31:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:31:52.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:52 smithi204 ceph-mon[32357]: 4.b scrub starts 2024-09-10T18:31:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:52 smithi204 ceph-mon[32357]: 4.b scrub ok 2024-09-10T18:31:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:52 smithi204 ceph-mon[32357]: 5.10 scrub starts 2024-09-10T18:31:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:52 smithi204 ceph-mon[32357]: 5.10 scrub ok 2024-09-10T18:31:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:52 smithi204 ceph-mon[32357]: 4.e scrub starts 2024-09-10T18:31:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:52 smithi204 ceph-mon[32357]: 4.e scrub ok 2024-09-10T18:31:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:52 smithi204 ceph-mon[32357]: pgmap v461: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T18:31:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:52 smithi204 ceph-mon[32357]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:31:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:52 smithi204 ceph-mon[32357]: osdmap e243: 8 total, 8 up, 8 in 2024-09-10T18:31:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:52 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3122887931' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:31:52.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:52.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:52.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[29291]: 4.b scrub starts 2024-09-10T18:31:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[29291]: 4.b scrub ok 2024-09-10T18:31:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[29291]: 5.10 scrub starts 2024-09-10T18:31:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[29291]: 5.10 scrub ok 2024-09-10T18:31:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[29291]: 4.e scrub starts 2024-09-10T18:31:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[29291]: 4.e scrub ok 2024-09-10T18:31:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[29291]: pgmap v461: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T18:31:52.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[29291]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:31:52.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[29291]: osdmap e243: 8 total, 8 up, 8 in 2024-09-10T18:31:52.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3122887931' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:31:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[39220]: 4.b scrub starts 2024-09-10T18:31:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[39220]: 4.b scrub ok 2024-09-10T18:31:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[39220]: 5.10 scrub starts 2024-09-10T18:31:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[39220]: 5.10 scrub ok 2024-09-10T18:31:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[39220]: 4.e scrub starts 2024-09-10T18:31:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[39220]: 4.e scrub ok 2024-09-10T18:31:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[39220]: pgmap v461: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T18:31:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[39220]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:31:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[39220]: osdmap e243: 8 total, 8 up, 8 in 2024-09-10T18:31:52.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:52 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3122887931' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:31:52.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:52.553 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:52.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:52.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:52.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:52.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:52.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:52.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:52.702 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-09-10T18:31:50.957049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1761,"num_read_kb":1513,"num_write":269154,"num_write_kb":8589996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82609,"ondisk_log_size":82609,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2449928,"kb_used_data":7620,"kb_used_omap":19,"kb_used_meta":2442220,"kb_avail":747511288,"statfs":{"total":767960285184,"available":765451558912,"internally_reserved":0,"allocated":7802880,"data_stored":4806012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20033,"internal_metadata":2500833727},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.921681"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":232,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252535+0000","last_change":"2024-09-10T18:30:59.501814+0000","last_active":"2024-09-10T18:31:41.252535+0000","last_peered":"2024-09-10T18:31:41.252535+0000","last_clean":"2024-09-10T18:31:41.252535+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:31:41.252535+0000","last_undegraded":"2024-09-10T18:31:41.252535+0000","last_fullsized":"2024-09-10T18:31:41.252535+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:59.501735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:30:59.501735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:24.012243+0000","scrub_duration":0.010428556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":271,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:42.556037+0000","last_change":"2024-09-10T18:31:42.555975+0000","last_active":"2024-09-10T18:31:42.556037+0000","last_peered":"2024-09-10T18:31:42.556037+0000","last_clean":"2024-09-10T18:31:42.556037+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:31:42.556037+0000","last_undegraded":"2024-09-10T18:31:42.556037+0000","last_fullsized":"2024-09-10T18:31:42.556037+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:31:42.555913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:42.555913+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:44.837983+0000","scrub_duration":0.0021762769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":237,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.036427+0000","last_change":"2024-09-10T18:30:48.481468+0000","last_active":"2024-09-10T18:31:41.036427+0000","last_peered":"2024-09-10T18:31:41.036427+0000","last_clean":"2024-09-10T18:31:41.036427+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:31:41.036427+0000","last_undegraded":"2024-09-10T18:31:41.036427+0000","last_fullsized":"2024-09-10T18:31:41.036427+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:48.481378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:48.481378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:59.396857+0000","scrub_duration":0.0060252910000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":235,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.038300+0000","last_change":"2024-09-10T18:31:09.486730+0000","last_active":"2024-09-10T18:31:41.038300+0000","last_peered":"2024-09-10T18:31:41.038300+0000","last_clean":"2024-09-10T18:31:41.038300+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:31:41.038300+0000","last_undegraded":"2024-09-10T18:31:41.038300+0000","last_fullsized":"2024-09-10T18:31:41.038300+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:09.486379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:31:09.486379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:30.435919+0000","scrub_duration":0.010976669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":234,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251317+0000","last_change":"2024-09-10T18:29:48.283119+0000","last_active":"2024-09-10T18:31:41.251317+0000","last_peered":"2024-09-10T18:31:41.251317+0000","last_clean":"2024-09-10T18:31:41.251317+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:31:41.251317+0000","last_undegraded":"2024-09-10T18:31:41.251317+0000","last_fullsized":"2024-09-10T18:31:41.251317+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:48.282880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:48.282880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095569450000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":240,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251947+0000","last_change":"2024-09-10T18:29:42.297141+0000","last_active":"2024-09-10T18:31:41.251947+0000","last_peered":"2024-09-10T18:31:41.251947+0000","last_clean":"2024-09-10T18:31:41.251947+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:31:41.251947+0000","last_undegraded":"2024-09-10T18:31:41.251947+0000","last_fullsized":"2024-09-10T18:31:41.251947+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:42.297045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:29:42.297045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047534680000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'34","reported_seq":281,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252388+0000","last_change":"2024-09-10T18:30:48.720386+0000","last_active":"2024-09-10T18:31:41.252388+0000","last_peered":"2024-09-10T18:31:41.252388+0000","last_clean":"2024-09-10T18:31:41.252388+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:31:41.252388+0000","last_undegraded":"2024-09-10T18:31:41.252388+0000","last_fullsized":"2024-09-10T18:31:41.252388+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'22","last_scrub_stamp":"2024-09-10T18:30:48.720296+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:30:48.720296+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:08.914191+0000","scrub_duration":0.0037509370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":247,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:48.634565+0000","last_change":"2024-09-10T18:31:48.634526+0000","last_active":"2024-09-10T18:31:48.634565+0000","last_peered":"2024-09-10T18:31:48.634565+0000","last_clean":"2024-09-10T18:31:48.634565+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:31:48.634565+0000","last_undegraded":"2024-09-10T18:31:48.634565+0000","last_fullsized":"2024-09-10T18:31:48.634565+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:48.634465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.634465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:14.831892+0000","scrub_duration":0.0012201320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":242,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252832+0000","last_change":"2024-09-10T18:29:46.069209+0000","last_active":"2024-09-10T18:31:41.252832+0000","last_peered":"2024-09-10T18:31:41.252832+0000","last_clean":"2024-09-10T18:31:41.252832+0000","last_became_active":"2024-09-10T18:26:27.644961+0000","last_became_peered":"2024-09-10T18:26:27.644961+0000","last_unstale":"2024-09-10T18:31:41.252832+0000","last_undegraded":"2024-09-10T18:31:41.252832+0000","last_fullsized":"2024-09-10T18:31:41.252832+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:46.069158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:29:46.069158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014897689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'15","reported_seq":264,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.038530+0000","last_change":"2024-09-10T18:30:58.249859+0000","last_active":"2024-09-10T18:31:41.038530+0000","last_peered":"2024-09-10T18:31:41.038530+0000","last_clean":"2024-09-10T18:31:41.038530+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:31:41.038530+0000","last_undegraded":"2024-09-10T18:31:41.038530+0000","last_fullsized":"2024-09-10T18:31:41.038530+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:30:58.249769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:58.249769+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:58.368543+0000","scrub_duration":0.0056195170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":238,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251783+0000","last_change":"2024-09-10T18:30:04.067809+0000","last_active":"2024-09-10T18:31:41.251783+0000","last_peered":"2024-09-10T18:31:41.251783+0000","last_clean":"2024-09-10T18:31:41.251783+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:31:41.251783+0000","last_undegraded":"2024-09-10T18:31:41.251783+0000","last_fullsized":"2024-09-10T18:31:41.251783+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:04.067676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:04.067676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054560930000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":236,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250251+0000","last_change":"2024-09-10T18:30:56.456172+0000","last_active":"2024-09-10T18:31:41.250251+0000","last_peered":"2024-09-10T18:31:41.250251+0000","last_clean":"2024-09-10T18:31:41.250251+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:31:41.250251+0000","last_undegraded":"2024-09-10T18:31:41.250251+0000","last_fullsized":"2024-09-10T18:31:41.250251+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:56.456087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:56.456087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:16.486677+0000","scrub_duration":0.0081476819999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":242,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252327+0000","last_change":"2024-09-10T18:30:56.338770+0000","last_active":"2024-09-10T18:31:41.252327+0000","last_peered":"2024-09-10T18:31:41.252327+0000","last_clean":"2024-09-10T18:31:41.252327+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:31:41.252327+0000","last_undegraded":"2024-09-10T18:31:41.252327+0000","last_fullsized":"2024-09-10T18:31:41.252327+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:56.338722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:17.464205+0000","last_clean_scrub_stamp":"2024-09-10T18:30:56.338722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:00.099480+0000","scrub_duration":0.0038549069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":276,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.038695+0000","last_change":"2024-09-10T18:30:59.295764+0000","last_active":"2024-09-10T18:31:41.038695+0000","last_peered":"2024-09-10T18:31:41.038695+0000","last_clean":"2024-09-10T18:31:41.038695+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:31:41.038695+0000","last_undegraded":"2024-09-10T18:31:41.038695+0000","last_fullsized":"2024-09-10T18:31:41.038695+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'7","last_scrub_stamp":"2024-09-10T18:30:59.295710+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:30:59.295710+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:00.454829+0000","scrub_duration":0.0032984920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":237,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251394+0000","last_change":"2024-09-10T18:30:00.485717+0000","last_active":"2024-09-10T18:31:41.251394+0000","last_peered":"2024-09-10T18:31:41.251394+0000","last_clean":"2024-09-10T18:31:41.251394+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:31:41.251394+0000","last_undegraded":"2024-09-10T18:31:41.251394+0000","last_fullsized":"2024-09-10T18:31:41.251394+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:00.485621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:00.485621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032987590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":229,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250738+0000","last_change":"2024-09-10T18:29:52.039654+0000","last_active":"2024-09-10T18:31:41.250738+0000","last_peered":"2024-09-10T18:31:41.250738+0000","last_clean":"2024-09-10T18:31:41.250738+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:31:41.250738+0000","last_undegraded":"2024-09-10T18:31:41.250738+0000","last_fullsized":"2024-09-10T18:31:41.250738+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:52.039580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:29:52.039580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032991380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":235,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.039082+0000","last_change":"2024-09-10T18:29:48.278074+0000","last_active":"2024-09-10T18:31:41.039082+0000","last_peered":"2024-09-10T18:31:41.039082+0000","last_clean":"2024-09-10T18:31:41.039082+0000","last_became_active":"2024-09-10T18:26:27.650774+0000","last_became_peered":"2024-09-10T18:26:27.650774+0000","last_unstale":"2024-09-10T18:31:41.039082+0000","last_undegraded":"2024-09-10T18:31:41.039082+0000","last_fullsized":"2024-09-10T18:31:41.039082+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:48.277982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:48.277982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018607579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":222,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251361+0000","last_change":"2024-09-10T18:28:56.989664+0000","last_active":"2024-09-10T18:31:41.251361+0000","last_peered":"2024-09-10T18:31:41.251361+0000","last_clean":"2024-09-10T18:31:41.251361+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:31:41.251361+0000","last_undegraded":"2024-09-10T18:31:41.251361+0000","last_fullsized":"2024-09-10T18:31:41.251361+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:56.989590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:56.989590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038444830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"240'27","reported_seq":292,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251600+0000","last_change":"2024-09-10T18:30:46.078403+0000","last_active":"2024-09-10T18:31:41.251600+0000","last_peered":"2024-09-10T18:31:41.251600+0000","last_clean":"2024-09-10T18:31:41.251600+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:31:41.251600+0000","last_undegraded":"2024-09-10T18:31:41.251600+0000","last_fullsized":"2024-09-10T18:31:41.251600+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-09-10T18:30:46.078313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:46.078313+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:51.777750+0000","scrub_duration":0.011671104999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":237,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.037083+0000","last_change":"2024-09-10T18:30:00.569368+0000","last_active":"2024-09-10T18:31:41.037083+0000","last_peered":"2024-09-10T18:31:41.037083+0000","last_clean":"2024-09-10T18:31:41.037083+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:31:41.037083+0000","last_undegraded":"2024-09-10T18:31:41.037083+0000","last_fullsized":"2024-09-10T18:31:41.037083+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:00.569288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:00.569288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001687004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":234,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252048+0000","last_change":"2024-09-10T18:29:43.320896+0000","last_active":"2024-09-10T18:31:41.252048+0000","last_peered":"2024-09-10T18:31:41.252048+0000","last_clean":"2024-09-10T18:31:41.252048+0000","last_became_active":"2024-09-10T18:26:27.640035+0000","last_became_peered":"2024-09-10T18:26:27.640035+0000","last_unstale":"2024-09-10T18:31:41.252048+0000","last_undegraded":"2024-09-10T18:31:41.252048+0000","last_fullsized":"2024-09-10T18:31:41.252048+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:43.320795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:43.320795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015436353999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":241,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.038606+0000","last_change":"2024-09-10T18:31:03.350685+0000","last_active":"2024-09-10T18:31:41.038606+0000","last_peered":"2024-09-10T18:31:41.038606+0000","last_clean":"2024-09-10T18:31:41.038606+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:31:41.038606+0000","last_undegraded":"2024-09-10T18:31:41.038606+0000","last_fullsized":"2024-09-10T18:31:41.038606+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:03.350631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:31:03.350631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:18.256534+0000","scrub_duration":0.016656491999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'23","reported_seq":268,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251046+0000","last_change":"2024-09-10T18:31:06.712809+0000","last_active":"2024-09-10T18:31:41.251046+0000","last_peered":"2024-09-10T18:31:41.251046+0000","last_clean":"2024-09-10T18:31:41.251046+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:31:41.251046+0000","last_undegraded":"2024-09-10T18:31:41.251046+0000","last_fullsized":"2024-09-10T18:31:41.251046+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-09-10T18:31:06.712724+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:31:06.712724+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:07.264277+0000","scrub_duration":0.017994977999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":237,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.038767+0000","last_change":"2024-09-10T18:31:12.515598+0000","last_active":"2024-09-10T18:31:41.038767+0000","last_peered":"2024-09-10T18:31:41.038767+0000","last_clean":"2024-09-10T18:31:41.038767+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:31:41.038767+0000","last_undegraded":"2024-09-10T18:31:41.038767+0000","last_fullsized":"2024-09-10T18:31:41.038767+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:12.515540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:12.515540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:28.271069+0000","scrub_duration":0.0011808949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":237,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252760+0000","last_change":"2024-09-10T18:30:48.487103+0000","last_active":"2024-09-10T18:31:41.252760+0000","last_peered":"2024-09-10T18:31:41.252760+0000","last_clean":"2024-09-10T18:31:41.252760+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:31:41.252760+0000","last_undegraded":"2024-09-10T18:31:41.252760+0000","last_fullsized":"2024-09-10T18:31:41.252760+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:30:48.487005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:48.487005+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:58.811476+0000","scrub_duration":0.0023474720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":251,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.036501+0000","last_change":"2024-09-10T18:31:04.526514+0000","last_active":"2024-09-10T18:31:41.036501+0000","last_peered":"2024-09-10T18:31:41.036501+0000","last_clean":"2024-09-10T18:31:41.036501+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:31:41.036501+0000","last_undegraded":"2024-09-10T18:31:41.036501+0000","last_fullsized":"2024-09-10T18:31:41.036501+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:04.526424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:04.526424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:13.474553+0000","scrub_duration":0.013541001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'25","reported_seq":265,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252501+0000","last_change":"2024-09-10T18:29:03.110471+0000","last_active":"2024-09-10T18:31:41.252501+0000","last_peered":"2024-09-10T18:31:41.252501+0000","last_clean":"2024-09-10T18:31:41.252501+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:31:41.252501+0000","last_undegraded":"2024-09-10T18:31:41.252501+0000","last_fullsized":"2024-09-10T18:31:41.252501+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-09-10T18:29:03.110142+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:29:03.110142+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.025364746000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"217'11","reported_seq":282,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:49.104846+0000","last_change":"2024-09-10T18:31:49.104801+0000","last_active":"2024-09-10T18:31:49.104846+0000","last_peered":"2024-09-10T18:31:49.104846+0000","last_clean":"2024-09-10T18:31:49.104846+0000","last_became_active":"2024-09-10T18:26:27.652489+0000","last_became_peered":"2024-09-10T18:26:27.652489+0000","last_unstale":"2024-09-10T18:31:49.104846+0000","last_undegraded":"2024-09-10T18:31:49.104846+0000","last_fullsized":"2024-09-10T18:31:49.104846+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11","last_scrub_stamp":"2024-09-10T18:31:49.104746+0000","last_deep_scrub":"217'11","last_deep_scrub_stamp":"2024-09-10T18:30:03.161985+0000","last_clean_scrub_stamp":"2024-09-10T18:31:49.104746+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:51.048090+0000","scrub_duration":0.0013830120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"239'16986","reported_seq":17496,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.036719+0000","last_change":"2024-09-10T18:31:06.545419+0000","last_active":"2024-09-10T18:31:41.036719+0000","last_peered":"2024-09-10T18:31:41.036719+0000","last_clean":"2024-09-10T18:31:41.036719+0000","last_became_active":"2024-09-10T18:27:53.036244+0000","last_became_peered":"2024-09-10T18:27:53.036244+0000","last_unstale":"2024-09-10T18:31:41.036719+0000","last_undegraded":"2024-09-10T18:31:41.036719+0000","last_fullsized":"2024-09-10T18:31:41.036719+0000","mapping_epoch":180,"log_start":"226'12100","ondisk_log_start":"226'12100","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'14696","last_scrub_stamp":"2024-09-10T18:31:06.545331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:31:06.545331+0000","objects_scrubbed":2324,"log_size":4886,"log_dups_size":0,"ondisk_log_size":4886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:14.375257+0000","scrub_duration":1.028754875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16986,"num_write_kb":543552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":148,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250491+0000","last_change":"2024-09-10T18:30:00.070274+0000","last_active":"2024-09-10T18:31:41.250491+0000","last_peered":"2024-09-10T18:31:41.250491+0000","last_clean":"2024-09-10T18:31:41.250491+0000","last_became_active":"2024-09-10T18:26:28.635249+0000","last_became_peered":"2024-09-10T18:26:28.635249+0000","last_unstale":"2024-09-10T18:31:41.250491+0000","last_undegraded":"2024-09-10T18:31:41.250491+0000","last_fullsized":"2024-09-10T18:31:41.250491+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:30:00.070176+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:30:00.070176+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046094300000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"240'34","reported_seq":302,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.036983+0000","last_change":"2024-09-10T18:31:11.374516+0000","last_active":"2024-09-10T18:31:41.036983+0000","last_peered":"2024-09-10T18:31:41.036983+0000","last_clean":"2024-09-10T18:31:41.036983+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:31:41.036983+0000","last_undegraded":"2024-09-10T18:31:41.036983+0000","last_fullsized":"2024-09-10T18:31:41.036983+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-09-10T18:31:11.373949+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:31:11.373949+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:28.044226+0000","scrub_duration":0.0054884399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":233,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251058+0000","last_change":"2024-09-10T18:30:01.544293+0000","last_active":"2024-09-10T18:31:41.251058+0000","last_peered":"2024-09-10T18:31:41.251058+0000","last_clean":"2024-09-10T18:31:41.251058+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:31:41.251058+0000","last_undegraded":"2024-09-10T18:31:41.251058+0000","last_fullsized":"2024-09-10T18:31:41.251058+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:01.544196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:30:01.544196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011854096,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":361,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:44.883892+0000","last_change":"2024-09-10T18:30:00.293382+0000","last_active":"2024-09-10T18:31:44.883892+0000","last_peered":"2024-09-10T18:31:44.883892+0000","last_clean":"2024-09-10T18:31:44.883892+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:31:44.883892+0000","last_undegraded":"2024-09-10T18:31:44.883892+0000","last_fullsized":"2024-09-10T18:31:44.883892+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:30:00.293289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:00.293289+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.105069817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":236,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252612+0000","last_change":"2024-09-10T18:30:59.381476+0000","last_active":"2024-09-10T18:31:41.252612+0000","last_peered":"2024-09-10T18:31:41.252612+0000","last_clean":"2024-09-10T18:31:41.252612+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:31:41.252612+0000","last_undegraded":"2024-09-10T18:31:41.252612+0000","last_fullsized":"2024-09-10T18:31:41.252612+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:59.381396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:30:59.381396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:12.427795+0000","scrub_duration":0.011130997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"239'16594","reported_seq":16975,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251834+0000","last_change":"2024-09-10T18:29:50.011661+0000","last_active":"2024-09-10T18:31:41.251834+0000","last_peered":"2024-09-10T18:31:41.251834+0000","last_clean":"2024-09-10T18:31:41.251834+0000","last_became_active":"2024-09-10T18:27:53.059554+0000","last_became_peered":"2024-09-10T18:27:53.059554+0000","last_unstale":"2024-09-10T18:31:41.251834+0000","last_undegraded":"2024-09-10T18:31:41.251834+0000","last_fullsized":"2024-09-10T18:31:41.251834+0000","mapping_epoch":180,"log_start":"224'11100","ondisk_log_start":"224'11100","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'9197","last_scrub_stamp":"2024-09-10T18:29:50.011576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:29:50.011576+0000","objects_scrubbed":7509,"log_size":5494,"log_dups_size":0,"ondisk_log_size":5494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.8603947789999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16594,"num_write_kb":531008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":147,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.039538+0000","last_change":"2024-09-10T18:30:45.366763+0000","last_active":"2024-09-10T18:31:41.039538+0000","last_peered":"2024-09-10T18:31:41.039538+0000","last_clean":"2024-09-10T18:31:41.039538+0000","last_became_active":"2024-09-10T18:26:28.632744+0000","last_became_peered":"2024-09-10T18:26:28.632744+0000","last_unstale":"2024-09-10T18:31:41.039538+0000","last_undegraded":"2024-09-10T18:31:41.039538+0000","last_fullsized":"2024-09-10T18:31:41.039538+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:30:45.366615+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:30:45.366615+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:14.087487+0000","scrub_duration":0.013315966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":281,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250834+0000","last_change":"2024-09-10T18:30:52.742276+0000","last_active":"2024-09-10T18:31:41.250834+0000","last_peered":"2024-09-10T18:31:41.250834+0000","last_clean":"2024-09-10T18:31:41.250834+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:31:41.250834+0000","last_undegraded":"2024-09-10T18:31:41.250834+0000","last_fullsized":"2024-09-10T18:31:41.250834+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-10T18:30:52.742191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:52.742191+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:19.019211+0000","scrub_duration":0.015070948000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":233,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252355+0000","last_change":"2024-09-10T18:31:01.695940+0000","last_active":"2024-09-10T18:31:41.252355+0000","last_peered":"2024-09-10T18:31:41.252355+0000","last_clean":"2024-09-10T18:31:41.252355+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:31:41.252355+0000","last_undegraded":"2024-09-10T18:31:41.252355+0000","last_fullsized":"2024-09-10T18:31:41.252355+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:01.695706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.140922+0000","last_clean_scrub_stamp":"2024-09-10T18:31:01.695706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:05.639726+0000","scrub_duration":0.0022117759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":237,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.038181+0000","last_change":"2024-09-10T18:31:04.334432+0000","last_active":"2024-09-10T18:31:41.038181+0000","last_peered":"2024-09-10T18:31:41.038181+0000","last_clean":"2024-09-10T18:31:41.038181+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:31:41.038181+0000","last_undegraded":"2024-09-10T18:31:41.038181+0000","last_fullsized":"2024-09-10T18:31:41.038181+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:31:04.334337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:05.418495+0000","scrub_duration":0.0012632089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":225,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.038511+0000","last_change":"2024-09-10T18:30:03.186000+0000","last_active":"2024-09-10T18:31:41.038511+0000","last_peered":"2024-09-10T18:31:41.038511+0000","last_clean":"2024-09-10T18:31:41.038511+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:31:41.038511+0000","last_undegraded":"2024-09-10T18:31:41.038511+0000","last_fullsized":"2024-09-10T18:31:41.038511+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:03.185938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:03.185938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007381528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"239'17264","reported_seq":17653,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252540+0000","last_change":"2024-09-10T18:29:51.277252+0000","last_active":"2024-09-10T18:31:41.252540+0000","last_peered":"2024-09-10T18:31:41.252540+0000","last_clean":"2024-09-10T18:31:41.252540+0000","last_became_active":"2024-09-10T18:27:53.061195+0000","last_became_peered":"2024-09-10T18:27:53.061195+0000","last_unstale":"2024-09-10T18:31:41.252540+0000","last_undegraded":"2024-09-10T18:31:41.252540+0000","last_fullsized":"2024-09-10T18:31:41.252540+0000","mapping_epoch":180,"log_start":"225'12200","ondisk_log_start":"225'12200","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'9680","last_scrub_stamp":"2024-09-10T18:29:51.277190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:29:51.277190+0000","objects_scrubbed":7656,"log_size":5064,"log_dups_size":0,"ondisk_log_size":5064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.2581719800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17264,"num_write_kb":552385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":138,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251140+0000","last_change":"2024-09-10T18:30:50.806220+0000","last_active":"2024-09-10T18:31:41.251140+0000","last_peered":"2024-09-10T18:31:41.251140+0000","last_clean":"2024-09-10T18:31:41.251140+0000","last_became_active":"2024-09-10T18:26:18.091896+0000","last_became_peered":"2024-09-10T18:26:18.091896+0000","last_unstale":"2024-09-10T18:31:41.251140+0000","last_undegraded":"2024-09-10T18:31:41.251140+0000","last_fullsized":"2024-09-10T18:31:41.251140+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:30:50.805972+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:30:50.805972+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:20.779313+0000","scrub_duration":0.0087755490000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"240'28","reported_seq":288,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:46.592544+0000","last_change":"2024-09-10T18:31:46.592515+0000","last_active":"2024-09-10T18:31:46.592544+0000","last_peered":"2024-09-10T18:31:46.592544+0000","last_clean":"2024-09-10T18:31:46.592544+0000","last_became_active":"2024-09-10T18:26:27.651364+0000","last_became_peered":"2024-09-10T18:26:27.651364+0000","last_unstale":"2024-09-10T18:31:46.592544+0000","last_undegraded":"2024-09-10T18:31:46.592544+0000","last_fullsized":"2024-09-10T18:31:46.592544+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'28","last_scrub_stamp":"2024-09-10T18:31:46.592456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:46.592456+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:58.738883+0000","scrub_duration":0.0019972359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":232,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252618+0000","last_change":"2024-09-10T18:30:04.443950+0000","last_active":"2024-09-10T18:31:41.252618+0000","last_peered":"2024-09-10T18:31:41.252618+0000","last_clean":"2024-09-10T18:31:41.252618+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:31:41.252618+0000","last_undegraded":"2024-09-10T18:31:41.252618+0000","last_fullsized":"2024-09-10T18:31:41.252618+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:04.443859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:30:04.443859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.167471064,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":362,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:45.046399+0000","last_change":"2024-09-10T18:30:56.702968+0000","last_active":"2024-09-10T18:31:45.046399+0000","last_peered":"2024-09-10T18:31:45.046399+0000","last_clean":"2024-09-10T18:31:45.046399+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:31:45.046399+0000","last_undegraded":"2024-09-10T18:31:45.046399+0000","last_fullsized":"2024-09-10T18:31:45.046399+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:30:56.702886+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:30:56.702886+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:21.421713+0000","scrub_duration":0.0084155429999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":245,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250914+0000","last_change":"2024-09-10T18:31:06.527936+0000","last_active":"2024-09-10T18:31:41.250914+0000","last_peered":"2024-09-10T18:31:41.250914+0000","last_clean":"2024-09-10T18:31:41.250914+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:31:41.250914+0000","last_undegraded":"2024-09-10T18:31:41.250914+0000","last_fullsized":"2024-09-10T18:31:41.250914+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:31:06.527847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:08.659894+0000","scrub_duration":0.0088842569999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"239'16924","reported_seq":17294,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251251+0000","last_change":"2024-09-10T18:30:48.591412+0000","last_active":"2024-09-10T18:31:41.251251+0000","last_peered":"2024-09-10T18:31:41.251251+0000","last_clean":"2024-09-10T18:31:41.251251+0000","last_became_active":"2024-09-10T18:27:53.034171+0000","last_became_peered":"2024-09-10T18:27:53.034171+0000","last_unstale":"2024-09-10T18:31:41.251251+0000","last_undegraded":"2024-09-10T18:31:41.251251+0000","last_fullsized":"2024-09-10T18:31:41.251251+0000","mapping_epoch":180,"log_start":"224'11400","ondisk_log_start":"224'11400","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'13270","last_scrub_stamp":"2024-09-10T18:30:48.591316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:30:48.591316+0000","objects_scrubbed":3744,"log_size":5524,"log_dups_size":0,"ondisk_log_size":5524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:50.983523+0000","scrub_duration":2.2895402169999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16924,"num_write_kb":541568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":197,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:42.153416+0000","last_change":"2024-09-10T18:31:42.153381+0000","last_active":"2024-09-10T18:31:42.153416+0000","last_peered":"2024-09-10T18:31:42.153416+0000","last_clean":"2024-09-10T18:31:42.153416+0000","last_became_active":"2024-09-10T18:26:28.632849+0000","last_became_peered":"2024-09-10T18:26:28.632849+0000","last_unstale":"2024-09-10T18:31:42.153416+0000","last_undegraded":"2024-09-10T18:31:42.153416+0000","last_fullsized":"2024-09-10T18:31:42.153416+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:42.153325+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:42.153325+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:49.020619+0000","scrub_duration":0.001252305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":284,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.036726+0000","last_change":"2024-09-10T18:31:00.546611+0000","last_active":"2024-09-10T18:31:41.036726+0000","last_peered":"2024-09-10T18:31:41.036726+0000","last_clean":"2024-09-10T18:31:41.036726+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:31:41.036726+0000","last_undegraded":"2024-09-10T18:31:41.036726+0000","last_fullsized":"2024-09-10T18:31:41.036726+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'15","last_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:31:00.545612+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:20.427793+0000","scrub_duration":0.0072351619999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":232,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252870+0000","last_change":"2024-09-10T18:30:46.332919+0000","last_active":"2024-09-10T18:31:41.252870+0000","last_peered":"2024-09-10T18:31:41.252870+0000","last_clean":"2024-09-10T18:31:41.252870+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:31:41.252870+0000","last_undegraded":"2024-09-10T18:31:41.252870+0000","last_fullsized":"2024-09-10T18:31:41.252870+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:46.332861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:30:46.332861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:08.986192+0000","scrub_duration":0.0061624269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"146'10","reported_seq":369,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:45.353405+0000","last_change":"2024-09-10T18:31:05.540491+0000","last_active":"2024-09-10T18:31:45.353405+0000","last_peered":"2024-09-10T18:31:45.353405+0000","last_clean":"2024-09-10T18:31:45.353405+0000","last_became_active":"2024-09-10T18:26:27.639032+0000","last_became_peered":"2024-09-10T18:26:27.639032+0000","last_unstale":"2024-09-10T18:31:45.353405+0000","last_undegraded":"2024-09-10T18:31:45.353405+0000","last_fullsized":"2024-09-10T18:31:45.353405+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'10","last_scrub_stamp":"2024-09-10T18:31:05.540403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:05.540403+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:18.590346+0000","scrub_duration":0.003060657,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":236,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:43.338933+0000","last_change":"2024-09-10T18:31:43.338883+0000","last_active":"2024-09-10T18:31:43.338933+0000","last_peered":"2024-09-10T18:31:43.338933+0000","last_clean":"2024-09-10T18:31:43.338933+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:31:43.338933+0000","last_undegraded":"2024-09-10T18:31:43.338933+0000","last_fullsized":"2024-09-10T18:31:43.338933+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:43.338816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:43.338816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:08.069598+0000","scrub_duration":0.0011905869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"239'16796","reported_seq":17170,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252223+0000","last_change":"2024-09-10T18:29:53.667412+0000","last_active":"2024-09-10T18:31:41.252223+0000","last_peered":"2024-09-10T18:31:41.252223+0000","last_clean":"2024-09-10T18:31:41.252223+0000","last_became_active":"2024-09-10T18:27:53.037210+0000","last_became_peered":"2024-09-10T18:27:53.037210+0000","last_unstale":"2024-09-10T18:31:41.252223+0000","last_undegraded":"2024-09-10T18:31:41.252223+0000","last_fullsized":"2024-09-10T18:31:41.252223+0000","mapping_epoch":180,"log_start":"226'12300","ondisk_log_start":"226'12300","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'9619","last_scrub_stamp":"2024-09-10T18:29:53.667320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:29:53.667320+0000","objects_scrubbed":7323,"log_size":4496,"log_dups_size":0,"ondisk_log_size":4496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.3498647269999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16796,"num_write_kb":537472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":197,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252547+0000","last_change":"2024-09-10T18:31:03.017561+0000","last_active":"2024-09-10T18:31:41.252547+0000","last_peered":"2024-09-10T18:31:41.252547+0000","last_clean":"2024-09-10T18:31:41.252547+0000","last_became_active":"2024-09-10T18:26:28.635542+0000","last_became_peered":"2024-09-10T18:26:28.635542+0000","last_unstale":"2024-09-10T18:31:41.252547+0000","last_undegraded":"2024-09-10T18:31:41.252547+0000","last_fullsized":"2024-09-10T18:31:41.252547+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:03.017452+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:03.017452+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:30.278613+0000","scrub_duration":0.0066732069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"240'29","reported_seq":291,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251229+0000","last_change":"2024-09-10T18:31:10.474539+0000","last_active":"2024-09-10T18:31:41.251229+0000","last_peered":"2024-09-10T18:31:41.251229+0000","last_clean":"2024-09-10T18:31:41.251229+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:31:41.251229+0000","last_undegraded":"2024-09-10T18:31:41.251229+0000","last_fullsized":"2024-09-10T18:31:41.251229+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-10T18:31:10.474434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:10.474434+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:37.841342+0000","scrub_duration":0.0058438149999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":231,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.037442+0000","last_change":"2024-09-10T18:29:56.487743+0000","last_active":"2024-09-10T18:31:41.037442+0000","last_peered":"2024-09-10T18:31:41.037442+0000","last_clean":"2024-09-10T18:31:41.037442+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:31:41.037442+0000","last_undegraded":"2024-09-10T18:31:41.037442+0000","last_fullsized":"2024-09-10T18:31:41.037442+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:56.487689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:29:56.487689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025845439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":187,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:48.149460+0000","last_change":"2024-09-10T18:31:48.149427+0000","last_active":"2024-09-10T18:31:48.149460+0000","last_peered":"2024-09-10T18:31:48.149460+0000","last_clean":"2024-09-10T18:31:48.149460+0000","last_became_active":"2024-09-10T18:26:27.642063+0000","last_became_peered":"2024-09-10T18:26:27.642063+0000","last_unstale":"2024-09-10T18:31:48.149460+0000","last_undegraded":"2024-09-10T18:31:48.149460+0000","last_fullsized":"2024-09-10T18:31:48.149460+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:48.149372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.149372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:54.678387+0000","scrub_duration":0.001174172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":232,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.036389+0000","last_change":"2024-09-10T18:29:55.520215+0000","last_active":"2024-09-10T18:31:41.036389+0000","last_peered":"2024-09-10T18:31:41.036389+0000","last_clean":"2024-09-10T18:31:41.036389+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:31:41.036389+0000","last_undegraded":"2024-09-10T18:31:41.036389+0000","last_fullsized":"2024-09-10T18:31:41.036389+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:29:55.520136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095486129999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"239'16440","reported_seq":16848,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:43.568735+0000","last_change":"2024-09-10T18:31:43.568670+0000","last_active":"2024-09-10T18:31:43.568735+0000","last_peered":"2024-09-10T18:31:43.568735+0000","last_clean":"2024-09-10T18:31:43.568735+0000","last_became_active":"2024-09-10T18:27:53.061113+0000","last_became_peered":"2024-09-10T18:27:53.061113+0000","last_unstale":"2024-09-10T18:31:43.568735+0000","last_undegraded":"2024-09-10T18:31:43.568735+0000","last_fullsized":"2024-09-10T18:31:43.568735+0000","mapping_epoch":180,"log_start":"225'11400","ondisk_log_start":"225'11400","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'16440","last_scrub_stamp":"2024-09-10T18:31:43.568576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:31:43.568576+0000","objects_scrubbed":0,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:55.708082+0000","scrub_duration":0.0082008340000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16440,"num_write_kb":526080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":133,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250555+0000","last_change":"2024-09-10T18:30:01.069834+0000","last_active":"2024-09-10T18:31:41.250555+0000","last_peered":"2024-09-10T18:31:41.250555+0000","last_clean":"2024-09-10T18:31:41.250555+0000","last_became_active":"2024-09-10T18:26:18.094747+0000","last_became_peered":"2024-09-10T18:26:18.094747+0000","last_unstale":"2024-09-10T18:31:41.250555+0000","last_undegraded":"2024-09-10T18:31:41.250555+0000","last_fullsized":"2024-09-10T18:31:41.250555+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:30:01.069779+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:30:01.069779+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002552529,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"240'41","reported_seq":348,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:45.498168+0000","last_change":"2024-09-10T18:31:11.323534+0000","last_active":"2024-09-10T18:31:45.498168+0000","last_peered":"2024-09-10T18:31:45.498168+0000","last_clean":"2024-09-10T18:31:45.498168+0000","last_became_active":"2024-09-10T18:23:10.067049+0000","last_became_peered":"2024-09-10T18:23:10.067049+0000","last_unstale":"2024-09-10T18:31:45.498168+0000","last_undegraded":"2024-09-10T18:31:45.498168+0000","last_fullsized":"2024-09-10T18:31:45.498168+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'29","last_scrub_stamp":"2024-09-10T18:31:11.323461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:11.323461+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:15.747737+0000","scrub_duration":0.0098705089999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":248,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:49.684915+0000","last_change":"2024-09-10T18:31:49.684883+0000","last_active":"2024-09-10T18:31:49.684915+0000","last_peered":"2024-09-10T18:31:49.684915+0000","last_clean":"2024-09-10T18:31:49.684915+0000","last_became_active":"2024-09-10T18:26:27.648690+0000","last_became_peered":"2024-09-10T18:26:27.648690+0000","last_unstale":"2024-09-10T18:31:49.684915+0000","last_undegraded":"2024-09-10T18:31:49.684915+0000","last_fullsized":"2024-09-10T18:31:49.684915+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:49.684824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:49.684824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:19.116090+0000","scrub_duration":0.0012508370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":257,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251828+0000","last_change":"2024-09-10T18:31:04.391774+0000","last_active":"2024-09-10T18:31:41.251828+0000","last_peered":"2024-09-10T18:31:41.251828+0000","last_clean":"2024-09-10T18:31:41.251828+0000","last_became_active":"2024-09-10T18:26:27.647576+0000","last_became_peered":"2024-09-10T18:26:27.647576+0000","last_unstale":"2024-09-10T18:31:41.251828+0000","last_undegraded":"2024-09-10T18:31:41.251828+0000","last_fullsized":"2024-09-10T18:31:41.251828+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:31:04.391687+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:31:04.391687+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:17.192131+0000","scrub_duration":0.011656032,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":228,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251606+0000","last_change":"2024-09-10T18:29:33.023742+0000","last_active":"2024-09-10T18:31:41.251606+0000","last_peered":"2024-09-10T18:31:41.251606+0000","last_clean":"2024-09-10T18:31:41.251606+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:31:41.251606+0000","last_undegraded":"2024-09-10T18:31:41.251606+0000","last_fullsized":"2024-09-10T18:31:41.251606+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:33.023658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:29:33.023658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024546590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":227,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251668+0000","last_change":"2024-09-10T18:31:04.714213+0000","last_active":"2024-09-10T18:31:41.251668+0000","last_peered":"2024-09-10T18:31:41.251668+0000","last_clean":"2024-09-10T18:31:41.251668+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:31:41.251668+0000","last_undegraded":"2024-09-10T18:31:41.251668+0000","last_fullsized":"2024-09-10T18:31:41.251668+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:04.714112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:04.714112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:31.376995+0000","scrub_duration":0.0085368679999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"239'21009","reported_seq":21424,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251239+0000","last_change":"2024-09-10T18:30:46.448052+0000","last_active":"2024-09-10T18:31:41.251239+0000","last_peered":"2024-09-10T18:31:41.251239+0000","last_clean":"2024-09-10T18:31:41.251239+0000","last_became_active":"2024-09-10T18:27:53.037246+0000","last_became_peered":"2024-09-10T18:27:53.037246+0000","last_unstale":"2024-09-10T18:31:41.251239+0000","last_undegraded":"2024-09-10T18:31:41.251239+0000","last_fullsized":"2024-09-10T18:31:41.251239+0000","mapping_epoch":180,"log_start":"226'16504","ondisk_log_start":"226'16504","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'17268","last_scrub_stamp":"2024-09-10T18:30:46.447757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:30:46.447757+0000","objects_scrubbed":3789,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:52.411404+0000","scrub_duration":1.7370367789999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16602,"num_write_kb":531264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":279,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251260+0000","last_change":"2024-09-10T18:29:56.539912+0000","last_active":"2024-09-10T18:31:41.251260+0000","last_peered":"2024-09-10T18:31:41.251260+0000","last_clean":"2024-09-10T18:31:41.251260+0000","last_became_active":"2024-09-10T18:26:18.095568+0000","last_became_peered":"2024-09-10T18:26:18.095568+0000","last_unstale":"2024-09-10T18:31:41.251260+0000","last_undegraded":"2024-09-10T18:31:41.251260+0000","last_fullsized":"2024-09-10T18:31:41.251260+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:29:56.539815+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:29:56.539815+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016440160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"240'17","reported_seq":272,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.037018+0000","last_change":"2024-09-10T18:30:51.522994+0000","last_active":"2024-09-10T18:31:41.037018+0000","last_peered":"2024-09-10T18:31:41.037018+0000","last_clean":"2024-09-10T18:31:41.037018+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:31:41.037018+0000","last_undegraded":"2024-09-10T18:31:41.037018+0000","last_fullsized":"2024-09-10T18:31:41.037018+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-10T18:30:51.522555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:51.522555+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:59.965878+0000","scrub_duration":0.0030428550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":248,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.036537+0000","last_change":"2024-09-10T18:30:02.567170+0000","last_active":"2024-09-10T18:31:41.036537+0000","last_peered":"2024-09-10T18:31:41.036537+0000","last_clean":"2024-09-10T18:31:41.036537+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:31:41.036537+0000","last_undegraded":"2024-09-10T18:31:41.036537+0000","last_fullsized":"2024-09-10T18:31:41.036537+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:02.567084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:02.567084+0000","last_clean_scrub_stamp":"2024-09-10T18:30:02.567084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030254050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":260,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252305+0000","last_change":"2024-09-10T18:30:45.042033+0000","last_active":"2024-09-10T18:31:41.252305+0000","last_peered":"2024-09-10T18:31:41.252305+0000","last_clean":"2024-09-10T18:31:41.252305+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:31:41.252305+0000","last_undegraded":"2024-09-10T18:31:41.252305+0000","last_fullsized":"2024-09-10T18:31:41.252305+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:30:45.041941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:30:45.041941+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:49.744342+0000","scrub_duration":0.016163502999999999,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":247,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251215+0000","last_change":"2024-09-10T18:31:10.933803+0000","last_active":"2024-09-10T18:31:41.251215+0000","last_peered":"2024-09-10T18:31:41.251215+0000","last_clean":"2024-09-10T18:31:41.251215+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:31:41.251215+0000","last_undegraded":"2024-09-10T18:31:41.251215+0000","last_fullsized":"2024-09-10T18:31:41.251215+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:10.933715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:31:10.933715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:38.397405+0000","scrub_duration":0.0061095990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":235,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252188+0000","last_change":"2024-09-10T18:29:55.513631+0000","last_active":"2024-09-10T18:31:41.252188+0000","last_peered":"2024-09-10T18:31:41.252188+0000","last_clean":"2024-09-10T18:31:41.252188+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:31:41.252188+0000","last_undegraded":"2024-09-10T18:31:41.252188+0000","last_fullsized":"2024-09-10T18:31:41.252188+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:55.513549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:55.513549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024107299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"239'16638","reported_seq":17103,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.036582+0000","last_change":"2024-09-10T18:31:13.034453+0000","last_active":"2024-09-10T18:31:41.036582+0000","last_peered":"2024-09-10T18:31:41.036582+0000","last_clean":"2024-09-10T18:31:41.036582+0000","last_became_active":"2024-09-10T18:27:53.034315+0000","last_became_peered":"2024-09-10T18:27:53.034315+0000","last_unstale":"2024-09-10T18:31:41.036582+0000","last_undegraded":"2024-09-10T18:31:41.036582+0000","last_fullsized":"2024-09-10T18:31:41.036582+0000","mapping_epoch":180,"log_start":"225'11715","ondisk_log_start":"225'11715","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'14864","last_scrub_stamp":"2024-09-10T18:31:13.034383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:31:13.034383+0000","objects_scrubbed":1789,"log_size":4923,"log_dups_size":0,"ondisk_log_size":4923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:38.107591+0000","scrub_duration":0.67597978599999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16638,"num_write_kb":532416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":158,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:48.466735+0000","last_change":"2024-09-10T18:31:48.466710+0000","last_active":"2024-09-10T18:31:48.466735+0000","last_peered":"2024-09-10T18:31:48.466735+0000","last_clean":"2024-09-10T18:31:48.466735+0000","last_became_active":"2024-09-10T18:26:28.641791+0000","last_became_peered":"2024-09-10T18:26:28.641791+0000","last_unstale":"2024-09-10T18:31:48.466735+0000","last_undegraded":"2024-09-10T18:31:48.466735+0000","last_fullsized":"2024-09-10T18:31:48.466735+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:48.466654+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.466654+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:15.898353+0000","scrub_duration":0.001538871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":266,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252546+0000","last_change":"2024-09-10T18:29:57.326700+0000","last_active":"2024-09-10T18:31:41.252546+0000","last_peered":"2024-09-10T18:31:41.252546+0000","last_clean":"2024-09-10T18:31:41.252546+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:31:41.252546+0000","last_undegraded":"2024-09-10T18:31:41.252546+0000","last_fullsized":"2024-09-10T18:31:41.252546+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:29:57.326613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:29:57.326613+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.138665703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":242,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251800+0000","last_change":"2024-09-10T18:31:08.529508+0000","last_active":"2024-09-10T18:31:41.251800+0000","last_peered":"2024-09-10T18:31:41.251800+0000","last_clean":"2024-09-10T18:31:41.251800+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:31:41.251800+0000","last_undegraded":"2024-09-10T18:31:41.251800+0000","last_fullsized":"2024-09-10T18:31:41.251800+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:08.529424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:31:08.529424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:08.560187+0000","scrub_duration":0.0078874119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":870,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:48.591218+0000","last_change":"2024-09-10T18:31:13.536758+0000","last_active":"2024-09-10T18:31:48.591218+0000","last_peered":"2024-09-10T18:31:48.591218+0000","last_clean":"2024-09-10T18:31:48.591218+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:31:48.591218+0000","last_undegraded":"2024-09-10T18:31:48.591218+0000","last_fullsized":"2024-09-10T18:31:48.591218+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:31:13.536419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:13.536419+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:34.617675+0000","scrub_duration":0.0043736720000000003,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":611,"num_read_kb":606,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":231,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252243+0000","last_change":"2024-09-10T18:30:47.337000+0000","last_active":"2024-09-10T18:31:41.252243+0000","last_peered":"2024-09-10T18:31:41.252243+0000","last_clean":"2024-09-10T18:31:41.252243+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:31:41.252243+0000","last_undegraded":"2024-09-10T18:31:41.252243+0000","last_fullsized":"2024-09-10T18:31:41.252243+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:47.336845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.771415+0000","last_clean_scrub_stamp":"2024-09-10T18:30:47.336845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:16.049963+0000","scrub_duration":0.0051314819999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":225,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.036629+0000","last_change":"2024-09-10T18:29:57.534601+0000","last_active":"2024-09-10T18:31:41.036629+0000","last_peered":"2024-09-10T18:31:41.036629+0000","last_clean":"2024-09-10T18:31:41.036629+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:31:41.036629+0000","last_undegraded":"2024-09-10T18:31:41.036629+0000","last_fullsized":"2024-09-10T18:31:41.036629+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:57.534528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:57.534528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011484388999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"239'16804","reported_seq":17278,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250729+0000","last_change":"2024-09-10T18:31:09.711045+0000","last_active":"2024-09-10T18:31:41.250729+0000","last_peered":"2024-09-10T18:31:41.250729+0000","last_clean":"2024-09-10T18:31:41.250729+0000","last_became_active":"2024-09-10T18:27:53.035261+0000","last_became_peered":"2024-09-10T18:27:53.035261+0000","last_unstale":"2024-09-10T18:31:41.250729+0000","last_undegraded":"2024-09-10T18:31:41.250729+0000","last_fullsized":"2024-09-10T18:31:41.250729+0000","mapping_epoch":180,"log_start":"222'11100","ondisk_log_start":"222'11100","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'14823","last_scrub_stamp":"2024-09-10T18:31:09.710964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:31:09.710964+0000","objects_scrubbed":2001,"log_size":5704,"log_dups_size":0,"ondisk_log_size":5704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:36.211088+0000","scrub_duration":0.65510596700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16804,"num_write_kb":537728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"240'26","reported_seq":224,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252291+0000","last_change":"2024-09-10T18:30:47.151480+0000","last_active":"2024-09-10T18:31:41.252291+0000","last_peered":"2024-09-10T18:31:41.252291+0000","last_clean":"2024-09-10T18:31:41.252291+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:31:41.252291+0000","last_undegraded":"2024-09-10T18:31:41.252291+0000","last_fullsized":"2024-09-10T18:31:41.252291+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-10T18:30:47.151322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:47.151322+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:10.861641+0000","scrub_duration":0.0022400229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":240,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250828+0000","last_change":"2024-09-10T18:29:33.216704+0000","last_active":"2024-09-10T18:31:41.250828+0000","last_peered":"2024-09-10T18:31:41.250828+0000","last_clean":"2024-09-10T18:31:41.250828+0000","last_became_active":"2024-09-10T18:26:27.639129+0000","last_became_peered":"2024-09-10T18:26:27.639129+0000","last_unstale":"2024-09-10T18:31:41.250828+0000","last_undegraded":"2024-09-10T18:31:41.250828+0000","last_fullsized":"2024-09-10T18:31:41.250828+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:33.216613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:29:33.216613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013069801000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":291,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251282+0000","last_change":"2024-09-10T18:30:51.519828+0000","last_active":"2024-09-10T18:31:41.251282+0000","last_peered":"2024-09-10T18:31:41.251282+0000","last_clean":"2024-09-10T18:31:41.251282+0000","last_became_active":"2024-09-10T18:26:27.646658+0000","last_became_peered":"2024-09-10T18:26:27.646658+0000","last_unstale":"2024-09-10T18:31:41.251282+0000","last_undegraded":"2024-09-10T18:31:41.251282+0000","last_fullsized":"2024-09-10T18:31:41.251282+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:30:51.519744+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:30:51.519744+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:59.960685+0000","scrub_duration":0.0040833290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":237,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252857+0000","last_change":"2024-09-10T18:29:54.545893+0000","last_active":"2024-09-10T18:31:41.252857+0000","last_peered":"2024-09-10T18:31:41.252857+0000","last_clean":"2024-09-10T18:31:41.252857+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:31:41.252857+0000","last_undegraded":"2024-09-10T18:31:41.252857+0000","last_fullsized":"2024-09-10T18:31:41.252857+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:54.545809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:29:54.545809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041017759999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":229,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252130+0000","last_change":"2024-09-10T18:30:49.395000+0000","last_active":"2024-09-10T18:31:41.252130+0000","last_peered":"2024-09-10T18:31:41.252130+0000","last_clean":"2024-09-10T18:31:41.252130+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:31:41.252130+0000","last_undegraded":"2024-09-10T18:31:41.252130+0000","last_fullsized":"2024-09-10T18:31:41.252130+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:49.394941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:30:49.394941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:09.069344+0000","scrub_duration":0.0017872470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"239'16486","reported_seq":16724,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.038057+0000","last_change":"2024-09-10T18:30:45.455710+0000","last_active":"2024-09-10T18:31:41.038057+0000","last_peered":"2024-09-10T18:31:41.038057+0000","last_clean":"2024-09-10T18:31:41.038057+0000","last_became_active":"2024-09-10T18:27:53.033757+0000","last_became_peered":"2024-09-10T18:27:53.033757+0000","last_unstale":"2024-09-10T18:31:41.038057+0000","last_undegraded":"2024-09-10T18:31:41.038057+0000","last_fullsized":"2024-09-10T18:31:41.038057+0000","mapping_epoch":180,"log_start":"226'11900","ondisk_log_start":"226'11900","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'12692","last_scrub_stamp":"2024-09-10T18:30:45.455623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:30:45.455623+0000","objects_scrubbed":3837,"log_size":4586,"log_dups_size":0,"ondisk_log_size":4586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:06.033806+0000","scrub_duration":1.1074504000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16486,"num_write_kb":527552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"240'19","reported_seq":270,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.039294+0000","last_change":"2024-09-10T18:29:58.236626+0000","last_active":"2024-09-10T18:31:41.039294+0000","last_peered":"2024-09-10T18:31:41.039294+0000","last_clean":"2024-09-10T18:31:41.039294+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:31:41.039294+0000","last_undegraded":"2024-09-10T18:31:41.039294+0000","last_fullsized":"2024-09-10T18:31:41.039294+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:29:58.236548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:29:58.236548+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091212359999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":231,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.038342+0000","last_change":"2024-09-10T18:30:01.224897+0000","last_active":"2024-09-10T18:31:41.038342+0000","last_peered":"2024-09-10T18:31:41.038342+0000","last_clean":"2024-09-10T18:31:41.038342+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:31:41.038342+0000","last_undegraded":"2024-09-10T18:31:41.038342+0000","last_fullsized":"2024-09-10T18:31:41.038342+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:01.224811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:30:01.224811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011944340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":245,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252011+0000","last_change":"2024-09-10T18:31:00.387970+0000","last_active":"2024-09-10T18:31:41.252011+0000","last_peered":"2024-09-10T18:31:41.252011+0000","last_clean":"2024-09-10T18:31:41.252011+0000","last_became_active":"2024-09-10T18:26:27.645885+0000","last_became_peered":"2024-09-10T18:26:27.645885+0000","last_unstale":"2024-09-10T18:31:41.252011+0000","last_undegraded":"2024-09-10T18:31:41.252011+0000","last_fullsized":"2024-09-10T18:31:41.252011+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:31:00.387887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:25.925387+0000","scrub_duration":0.001951775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":234,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251472+0000","last_change":"2024-09-10T18:29:40.267823+0000","last_active":"2024-09-10T18:31:41.251472+0000","last_peered":"2024-09-10T18:31:41.251472+0000","last_clean":"2024-09-10T18:31:41.251472+0000","last_became_active":"2024-09-10T18:26:27.632071+0000","last_became_peered":"2024-09-10T18:26:27.632071+0000","last_unstale":"2024-09-10T18:31:41.251472+0000","last_undegraded":"2024-09-10T18:31:41.251472+0000","last_fullsized":"2024-09-10T18:31:41.251472+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:40.267731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:40.267731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037925329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"239'16860","reported_seq":17204,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.036334+0000","last_change":"2024-09-10T18:30:03.941778+0000","last_active":"2024-09-10T18:31:41.036334+0000","last_peered":"2024-09-10T18:31:41.036334+0000","last_clean":"2024-09-10T18:31:41.036334+0000","last_became_active":"2024-09-10T18:27:53.035419+0000","last_became_peered":"2024-09-10T18:27:53.035419+0000","last_unstale":"2024-09-10T18:31:41.036334+0000","last_undegraded":"2024-09-10T18:31:41.036334+0000","last_fullsized":"2024-09-10T18:31:41.036334+0000","mapping_epoch":180,"log_start":"226'12000","ondisk_log_start":"226'12000","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'10327","last_scrub_stamp":"2024-09-10T18:30:03.941534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:30:03.941534+0000","objects_scrubbed":6614,"log_size":4860,"log_dups_size":0,"ondisk_log_size":4860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.4155082019999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16860,"num_write_kb":539520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":140,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250594+0000","last_change":"2024-09-10T18:29:40.943571+0000","last_active":"2024-09-10T18:31:41.250594+0000","last_peered":"2024-09-10T18:31:41.250594+0000","last_clean":"2024-09-10T18:31:41.250594+0000","last_became_active":"2024-09-10T18:26:18.095088+0000","last_became_peered":"2024-09-10T18:26:18.095088+0000","last_unstale":"2024-09-10T18:31:41.250594+0000","last_undegraded":"2024-09-10T18:31:41.250594+0000","last_fullsized":"2024-09-10T18:31:41.250594+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:29:40.943514+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:29:40.943514+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016048987000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":288,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.540583+0000","last_change":"2024-09-10T18:31:41.540517+0000","last_active":"2024-09-10T18:31:41.540583+0000","last_peered":"2024-09-10T18:31:41.540583+0000","last_clean":"2024-09-10T18:31:41.540583+0000","last_became_active":"2024-09-10T18:21:33.611203+0000","last_became_peered":"2024-09-10T18:21:33.611203+0000","last_unstale":"2024-09-10T18:31:41.540583+0000","last_undegraded":"2024-09-10T18:31:41.540583+0000","last_fullsized":"2024-09-10T18:31:41.540583+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-09-10T18:31:41.540420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:41.540420+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:02.083864+0000","scrub_duration":0.0028551750000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":195,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251972+0000","last_change":"2024-09-10T18:30:58.072260+0000","last_active":"2024-09-10T18:31:41.251972+0000","last_peered":"2024-09-10T18:31:41.251972+0000","last_clean":"2024-09-10T18:31:41.251972+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:31:41.251972+0000","last_undegraded":"2024-09-10T18:31:41.251972+0000","last_fullsized":"2024-09-10T18:31:41.251972+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:30:58.072177+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:30:58.072177+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:14.158109+0000","scrub_duration":0.003192453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":237,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.036711+0000","last_change":"2024-09-10T18:30:53.506075+0000","last_active":"2024-09-10T18:31:41.036711+0000","last_peered":"2024-09-10T18:31:41.036711+0000","last_clean":"2024-09-10T18:31:41.036711+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:31:41.036711+0000","last_undegraded":"2024-09-10T18:31:41.036711+0000","last_fullsized":"2024-09-10T18:31:41.036711+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:53.506000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:53.506000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:05.835328+0000","scrub_duration":0.011216421000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":255,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250885+0000","last_change":"2024-09-10T18:31:04.059884+0000","last_active":"2024-09-10T18:31:41.250885+0000","last_peered":"2024-09-10T18:31:41.250885+0000","last_clean":"2024-09-10T18:31:41.250885+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:31:41.250885+0000","last_undegraded":"2024-09-10T18:31:41.250885+0000","last_fullsized":"2024-09-10T18:31:41.250885+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:04.059795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:04.059795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:06.431870+0000","scrub_duration":0.010230702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"239'16574","reported_seq":16909,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250426+0000","last_change":"2024-09-10T18:30:06.516566+0000","last_active":"2024-09-10T18:31:41.250426+0000","last_peered":"2024-09-10T18:31:41.250426+0000","last_clean":"2024-09-10T18:31:41.250426+0000","last_became_active":"2024-09-10T18:27:53.037235+0000","last_became_peered":"2024-09-10T18:27:53.037235+0000","last_unstale":"2024-09-10T18:31:41.250426+0000","last_undegraded":"2024-09-10T18:31:41.250426+0000","last_fullsized":"2024-09-10T18:31:41.250426+0000","mapping_epoch":180,"log_start":"226'12000","ondisk_log_start":"226'12000","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'10249","last_scrub_stamp":"2024-09-10T18:30:06.516475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:30:06.516475+0000","objects_scrubbed":6370,"log_size":4574,"log_dups_size":0,"ondisk_log_size":4574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1.9308149449999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16574,"num_write_kb":530368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":289,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251569+0000","last_change":"2024-09-10T18:31:10.502369+0000","last_active":"2024-09-10T18:31:41.251569+0000","last_peered":"2024-09-10T18:31:41.251569+0000","last_clean":"2024-09-10T18:31:41.251569+0000","last_became_active":"2024-09-10T18:26:18.092014+0000","last_became_peered":"2024-09-10T18:26:18.092014+0000","last_unstale":"2024-09-10T18:31:41.251569+0000","last_undegraded":"2024-09-10T18:31:41.251569+0000","last_fullsized":"2024-09-10T18:31:41.251569+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:10.502290+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:10.502290+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:38.681535+0000","scrub_duration":0.0082798439999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"240'25","reported_seq":271,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.038847+0000","last_change":"2024-09-10T18:29:45.292442+0000","last_active":"2024-09-10T18:31:41.038847+0000","last_peered":"2024-09-10T18:31:41.038847+0000","last_clean":"2024-09-10T18:31:41.038847+0000","last_became_active":"2024-09-10T18:26:27.649406+0000","last_became_peered":"2024-09-10T18:26:27.649406+0000","last_unstale":"2024-09-10T18:31:41.038847+0000","last_undegraded":"2024-09-10T18:31:41.038847+0000","last_fullsized":"2024-09-10T18:31:41.038847+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'15","last_scrub_stamp":"2024-09-10T18:29:45.292368+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:29:45.292368+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033922850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":233,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.253945+0000","last_change":"2024-09-10T18:31:00.667182+0000","last_active":"2024-09-10T18:31:41.253945+0000","last_peered":"2024-09-10T18:31:41.253945+0000","last_clean":"2024-09-10T18:31:41.253945+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:31:41.253945+0000","last_undegraded":"2024-09-10T18:31:41.253945+0000","last_fullsized":"2024-09-10T18:31:41.253945+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:00.667103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:00.667103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:22.994719+0000","scrub_duration":0.0017606639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"189'9","reported_seq":390,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:45.796214+0000","last_change":"2024-09-10T18:31:09.044554+0000","last_active":"2024-09-10T18:31:45.796214+0000","last_peered":"2024-09-10T18:31:45.796214+0000","last_clean":"2024-09-10T18:31:45.796214+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:31:45.796214+0000","last_undegraded":"2024-09-10T18:31:45.796214+0000","last_fullsized":"2024-09-10T18:31:45.796214+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'9","last_scrub_stamp":"2024-09-10T18:31:09.044476+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:31:09.044476+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:10.212254+0000","scrub_duration":0.059079303999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":228,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.039004+0000","last_change":"2024-09-10T18:30:46.397016+0000","last_active":"2024-09-10T18:31:41.039004+0000","last_peered":"2024-09-10T18:31:41.039004+0000","last_clean":"2024-09-10T18:31:41.039004+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:31:41.039004+0000","last_undegraded":"2024-09-10T18:31:41.039004+0000","last_fullsized":"2024-09-10T18:31:41.039004+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:46.396958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_clean_scrub_stamp":"2024-09-10T18:30:46.396958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:59.092367+0000","scrub_duration":0.013383155000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"239'16860","reported_seq":17246,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.036776+0000","last_change":"2024-09-10T18:29:50.519885+0000","last_active":"2024-09-10T18:31:41.036776+0000","last_peered":"2024-09-10T18:31:41.036776+0000","last_clean":"2024-09-10T18:31:41.036776+0000","last_became_active":"2024-09-10T18:27:53.037361+0000","last_became_peered":"2024-09-10T18:27:53.037361+0000","last_unstale":"2024-09-10T18:31:41.036776+0000","last_undegraded":"2024-09-10T18:31:41.036776+0000","last_fullsized":"2024-09-10T18:31:41.036776+0000","mapping_epoch":180,"log_start":"225'11927","ondisk_log_start":"225'11927","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'9361","last_scrub_stamp":"2024-09-10T18:29:50.519772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:29:50.519772+0000","objects_scrubbed":7639,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"queued for scrub","scrub_duration":4.8806252880000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16860,"num_write_kb":539520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"240'15","reported_seq":255,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:50.320718+0000","last_change":"2024-09-10T18:31:50.320682+0000","last_active":"2024-09-10T18:31:50.320718+0000","last_peered":"2024-09-10T18:31:50.320718+0000","last_clean":"2024-09-10T18:31:50.320718+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:31:50.320718+0000","last_undegraded":"2024-09-10T18:31:50.320718+0000","last_fullsized":"2024-09-10T18:31:50.320718+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:31:50.320624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:50.320624+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:07.995041+0000","scrub_duration":0.002110139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"58'1","reported_seq":258,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.486160+0000","last_change":"2024-09-10T18:31:41.486097+0000","last_active":"2024-09-10T18:31:41.486160+0000","last_peered":"2024-09-10T18:31:41.486160+0000","last_clean":"2024-09-10T18:31:41.486160+0000","last_became_active":"2024-09-10T18:26:27.644997+0000","last_became_peered":"2024-09-10T18:26:27.644997+0000","last_unstale":"2024-09-10T18:31:41.486160+0000","last_undegraded":"2024-09-10T18:31:41.486160+0000","last_fullsized":"2024-09-10T18:31:41.486160+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-10T18:31:41.486036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:41.486036+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:52.403197+0000","scrub_duration":0.0019908389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":253,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250939+0000","last_change":"2024-09-10T18:31:08.038810+0000","last_active":"2024-09-10T18:31:41.250939+0000","last_peered":"2024-09-10T18:31:41.250939+0000","last_clean":"2024-09-10T18:31:41.250939+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:31:41.250939+0000","last_undegraded":"2024-09-10T18:31:41.250939+0000","last_fullsized":"2024-09-10T18:31:41.250939+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:08.038615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:08.038615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:29.821394+0000","scrub_duration":0.0064931390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":226,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251723+0000","last_change":"2024-09-10T18:29:44.098384+0000","last_active":"2024-09-10T18:31:41.251723+0000","last_peered":"2024-09-10T18:31:41.251723+0000","last_clean":"2024-09-10T18:31:41.251723+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:31:41.251723+0000","last_undegraded":"2024-09-10T18:31:41.251723+0000","last_fullsized":"2024-09-10T18:31:41.251723+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:44.098280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:44.098280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055883269999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"239'21527","reported_seq":21946,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252374+0000","last_change":"2024-09-10T18:30:53.024268+0000","last_active":"2024-09-10T18:31:41.252374+0000","last_peered":"2024-09-10T18:31:41.252374+0000","last_clean":"2024-09-10T18:31:41.252374+0000","last_became_active":"2024-09-10T18:27:53.059866+0000","last_became_peered":"2024-09-10T18:27:53.059866+0000","last_unstale":"2024-09-10T18:31:41.252374+0000","last_undegraded":"2024-09-10T18:31:41.252374+0000","last_fullsized":"2024-09-10T18:31:41.252374+0000","mapping_epoch":180,"log_start":"224'16004","ondisk_log_start":"224'16004","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'18230","last_scrub_stamp":"2024-09-10T18:30:53.024177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:30:53.024177+0000","objects_scrubbed":3365,"log_size":5523,"log_dups_size":0,"ondisk_log_size":5523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:12.287378+0000","scrub_duration":1.699362767,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16752,"num_write_kb":536064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"240'44","reported_seq":312,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251784+0000","last_change":"2024-09-10T18:31:00.043959+0000","last_active":"2024-09-10T18:31:41.251784+0000","last_peered":"2024-09-10T18:31:41.251784+0000","last_clean":"2024-09-10T18:31:41.251784+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:31:41.251784+0000","last_undegraded":"2024-09-10T18:31:41.251784+0000","last_fullsized":"2024-09-10T18:31:41.251784+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'28","last_scrub_stamp":"2024-09-10T18:31:00.043812+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:31:00.043812+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:03.642411+0000","scrub_duration":0.004131656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":258,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:45.496384+0000","last_change":"2024-09-10T18:31:45.496272+0000","last_active":"2024-09-10T18:31:45.496384+0000","last_peered":"2024-09-10T18:31:45.496384+0000","last_clean":"2024-09-10T18:31:45.496384+0000","last_became_active":"2024-09-10T18:26:27.638408+0000","last_became_peered":"2024-09-10T18:26:27.638408+0000","last_unstale":"2024-09-10T18:31:45.496384+0000","last_undegraded":"2024-09-10T18:31:45.496384+0000","last_fullsized":"2024-09-10T18:31:45.496384+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:45.496183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:45.496183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:49.972930+0000","scrub_duration":0.0015755630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":257,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250657+0000","last_change":"2024-09-10T18:31:14.899762+0000","last_active":"2024-09-10T18:31:41.250657+0000","last_peered":"2024-09-10T18:31:41.250657+0000","last_clean":"2024-09-10T18:31:41.250657+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:31:41.250657+0000","last_undegraded":"2024-09-10T18:31:41.250657+0000","last_fullsized":"2024-09-10T18:31:41.250657+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:14.899683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:14.899683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:24.241674+0000","scrub_duration":0.0054936819999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":238,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251216+0000","last_change":"2024-09-10T18:29:41.447464+0000","last_active":"2024-09-10T18:31:41.251216+0000","last_peered":"2024-09-10T18:31:41.251216+0000","last_clean":"2024-09-10T18:31:41.251216+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:31:41.251216+0000","last_undegraded":"2024-09-10T18:31:41.251216+0000","last_fullsized":"2024-09-10T18:31:41.251216+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:41.447227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:41.447227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012629659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"239'18689","reported_seq":19190,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252204+0000","last_change":"2024-09-10T18:31:16.902340+0000","last_active":"2024-09-10T18:31:41.252204+0000","last_peered":"2024-09-10T18:31:41.252204+0000","last_clean":"2024-09-10T18:31:41.252204+0000","last_became_active":"2024-09-10T18:27:53.060902+0000","last_became_peered":"2024-09-10T18:27:53.060902+0000","last_unstale":"2024-09-10T18:31:41.252204+0000","last_undegraded":"2024-09-10T18:31:41.252204+0000","last_fullsized":"2024-09-10T18:31:41.252204+0000","mapping_epoch":180,"log_start":"229'13000","ondisk_log_start":"229'13000","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'17099","last_scrub_stamp":"2024-09-10T18:31:16.902287+0000","last_deep_scrub":"236'17099","last_deep_scrub_stamp":"2024-09-10T18:31:16.902287+0000","last_clean_scrub_stamp":"2024-09-10T18:31:16.902287+0000","objects_scrubbed":1927,"log_size":5689,"log_dups_size":0,"ondisk_log_size":5689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:22.813809+0000","scrub_duration":10.85097494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16762,"num_write_kb":536384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"240'18","reported_seq":274,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.038164+0000","last_change":"2024-09-10T18:30:54.285825+0000","last_active":"2024-09-10T18:31:41.038164+0000","last_peered":"2024-09-10T18:31:41.038164+0000","last_clean":"2024-09-10T18:31:41.038164+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:31:41.038164+0000","last_undegraded":"2024-09-10T18:31:41.038164+0000","last_fullsized":"2024-09-10T18:31:41.038164+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:30:54.285738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:54.285738+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:13.017862+0000","scrub_duration":0.0045401779999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":251,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252661+0000","last_change":"2024-09-10T18:31:12.489636+0000","last_active":"2024-09-10T18:31:41.252661+0000","last_peered":"2024-09-10T18:31:41.252661+0000","last_clean":"2024-09-10T18:31:41.252661+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:31:41.252661+0000","last_undegraded":"2024-09-10T18:31:41.252661+0000","last_fullsized":"2024-09-10T18:31:41.252661+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:31:12.489588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:18.281160+0000","scrub_duration":0.00146783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":257,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250443+0000","last_change":"2024-09-10T18:31:12.882344+0000","last_active":"2024-09-10T18:31:41.250443+0000","last_peered":"2024-09-10T18:31:41.250443+0000","last_clean":"2024-09-10T18:31:41.250443+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:31:41.250443+0000","last_undegraded":"2024-09-10T18:31:41.250443+0000","last_fullsized":"2024-09-10T18:31:41.250443+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:12.882248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:12.882248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:24.267881+0000","scrub_duration":0.0020040589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":227,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251388+0000","last_change":"2024-09-10T18:31:12.803420+0000","last_active":"2024-09-10T18:31:41.251388+0000","last_peered":"2024-09-10T18:31:41.251388+0000","last_clean":"2024-09-10T18:31:41.251388+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:31:41.251388+0000","last_undegraded":"2024-09-10T18:31:41.251388+0000","last_fullsized":"2024-09-10T18:31:41.251388+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:12.803236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.803236+0000","last_clean_scrub_stamp":"2024-09-10T18:31:12.803236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:14.528857+0000","scrub_duration":0.0041535599999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"239'17096","reported_seq":17472,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251947+0000","last_change":"2024-09-10T18:29:53.310710+0000","last_active":"2024-09-10T18:31:41.251947+0000","last_peered":"2024-09-10T18:31:41.251947+0000","last_clean":"2024-09-10T18:31:41.251947+0000","last_became_active":"2024-09-10T18:27:53.032159+0000","last_became_peered":"2024-09-10T18:27:53.032159+0000","last_unstale":"2024-09-10T18:31:41.251947+0000","last_undegraded":"2024-09-10T18:31:41.251947+0000","last_fullsized":"2024-09-10T18:31:41.251947+0000","mapping_epoch":180,"log_start":"225'12000","ondisk_log_start":"225'12000","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'9794","last_scrub_stamp":"2024-09-10T18:29:53.310629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:52.016948+0000","last_clean_scrub_stamp":"2024-09-10T18:29:53.310629+0000","objects_scrubbed":7406,"log_size":5096,"log_dups_size":0,"ondisk_log_size":5096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.2945158860000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17096,"num_write_kb":547072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":290,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:47.535579+0000","last_change":"2024-09-10T18:31:47.535552+0000","last_active":"2024-09-10T18:31:47.535579+0000","last_peered":"2024-09-10T18:31:47.535579+0000","last_clean":"2024-09-10T18:31:47.535579+0000","last_became_active":"2024-09-10T18:26:27.645932+0000","last_became_peered":"2024-09-10T18:26:27.645932+0000","last_unstale":"2024-09-10T18:31:47.535579+0000","last_undegraded":"2024-09-10T18:31:47.535579+0000","last_fullsized":"2024-09-10T18:31:47.535579+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:31:47.535496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:47.535496+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:05.510739+0000","scrub_duration":0.0017425819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":251,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252178+0000","last_change":"2024-09-10T18:30:45.438521+0000","last_active":"2024-09-10T18:31:41.252178+0000","last_peered":"2024-09-10T18:31:41.252178+0000","last_clean":"2024-09-10T18:31:41.252178+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:31:41.252178+0000","last_undegraded":"2024-09-10T18:31:41.252178+0000","last_fullsized":"2024-09-10T18:31:41.252178+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:30:45.438437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:30:45.438437+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:07.661643+0000","scrub_duration":0.0026276839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":378,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:44.883865+0000","last_change":"2024-09-10T18:31:03.524139+0000","last_active":"2024-09-10T18:31:44.883865+0000","last_peered":"2024-09-10T18:31:44.883865+0000","last_clean":"2024-09-10T18:31:44.883865+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:31:44.883865+0000","last_undegraded":"2024-09-10T18:31:44.883865+0000","last_fullsized":"2024-09-10T18:31:44.883865+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:31:03.524066+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:25.302495+0000","scrub_duration":0.0040323920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":233,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251329+0000","last_change":"2024-09-10T18:31:14.326966+0000","last_active":"2024-09-10T18:31:41.251329+0000","last_peered":"2024-09-10T18:31:41.251329+0000","last_clean":"2024-09-10T18:31:41.251329+0000","last_became_active":"2024-09-10T18:26:27.647511+0000","last_became_peered":"2024-09-10T18:26:27.647511+0000","last_unstale":"2024-09-10T18:31:41.251329+0000","last_undegraded":"2024-09-10T18:31:41.251329+0000","last_fullsized":"2024-09-10T18:31:41.251329+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:14.326705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:14.326705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:18.962305+0000","scrub_duration":0.0018698650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":186,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251176+0000","last_change":"2024-09-10T18:31:02.067834+0000","last_active":"2024-09-10T18:31:41.251176+0000","last_peered":"2024-09-10T18:31:41.251176+0000","last_clean":"2024-09-10T18:31:41.251176+0000","last_became_active":"2024-09-10T18:26:27.640922+0000","last_became_peered":"2024-09-10T18:26:27.640922+0000","last_unstale":"2024-09-10T18:31:41.251176+0000","last_undegraded":"2024-09-10T18:31:41.251176+0000","last_fullsized":"2024-09-10T18:31:41.251176+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:02.067753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:31:02.067753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:30.111230+0000","scrub_duration":0.011667588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"240'12","reported_seq":249,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252680+0000","last_change":"2024-09-10T18:30:49.755553+0000","last_active":"2024-09-10T18:31:41.252680+0000","last_peered":"2024-09-10T18:31:41.252680+0000","last_clean":"2024-09-10T18:31:41.252680+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:31:41.252680+0000","last_undegraded":"2024-09-10T18:31:41.252680+0000","last_fullsized":"2024-09-10T18:31:41.252680+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-10T18:30:49.755466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:49.755466+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:09.195216+0000","scrub_duration":0.0040815260000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":227,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252551+0000","last_change":"2024-09-10T18:29:37.932209+0000","last_active":"2024-09-10T18:31:41.252551+0000","last_peered":"2024-09-10T18:31:41.252551+0000","last_clean":"2024-09-10T18:31:41.252551+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:31:41.252551+0000","last_undegraded":"2024-09-10T18:31:41.252551+0000","last_fullsized":"2024-09-10T18:31:41.252551+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:37.932136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:29:37.932136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053209679999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":242,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252121+0000","last_change":"2024-09-10T18:31:01.523924+0000","last_active":"2024-09-10T18:31:41.252121+0000","last_peered":"2024-09-10T18:31:41.252121+0000","last_clean":"2024-09-10T18:31:41.252121+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:31:41.252121+0000","last_undegraded":"2024-09-10T18:31:41.252121+0000","last_fullsized":"2024-09-10T18:31:41.252121+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:01.523839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:01.523839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:28.978824+0000","scrub_duration":0.0016753880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"240'19","reported_seq":262,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252585+0000","last_change":"2024-09-10T18:30:58.459335+0000","last_active":"2024-09-10T18:31:41.252585+0000","last_peered":"2024-09-10T18:31:41.252585+0000","last_clean":"2024-09-10T18:31:41.252585+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:31:41.252585+0000","last_undegraded":"2024-09-10T18:31:41.252585+0000","last_fullsized":"2024-09-10T18:31:41.252585+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'13","last_scrub_stamp":"2024-09-10T18:30:58.459246+0000","last_deep_scrub":"189'13","last_deep_scrub_stamp":"2024-09-10T18:28:08.976162+0000","last_clean_scrub_stamp":"2024-09-10T18:30:58.459246+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:25.581872+0000","scrub_duration":0.0093093749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":248,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251693+0000","last_change":"2024-09-10T18:31:00.524076+0000","last_active":"2024-09-10T18:31:41.251693+0000","last_peered":"2024-09-10T18:31:41.251693+0000","last_clean":"2024-09-10T18:31:41.251693+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:31:41.251693+0000","last_undegraded":"2024-09-10T18:31:41.251693+0000","last_fullsized":"2024-09-10T18:31:41.251693+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:00.523985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:31:00.523985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:28.606520+0000","scrub_duration":0.0018784559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":229,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252065+0000","last_change":"2024-09-10T18:30:02.587567+0000","last_active":"2024-09-10T18:31:41.252065+0000","last_peered":"2024-09-10T18:31:41.252065+0000","last_clean":"2024-09-10T18:31:41.252065+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:31:41.252065+0000","last_undegraded":"2024-09-10T18:31:41.252065+0000","last_fullsized":"2024-09-10T18:31:41.252065+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:02.587404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:02.587404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006571632,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":245,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251724+0000","last_change":"2024-09-10T18:30:59.094192+0000","last_active":"2024-09-10T18:31:41.251724+0000","last_peered":"2024-09-10T18:31:41.251724+0000","last_clean":"2024-09-10T18:31:41.251724+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:31:41.251724+0000","last_undegraded":"2024-09-10T18:31:41.251724+0000","last_fullsized":"2024-09-10T18:31:41.251724+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:59.094103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:59.094103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:13.830457+0000","scrub_duration":0.0095530630000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"240'19","reported_seq":269,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:45.544071+0000","last_change":"2024-09-10T18:31:45.544047+0000","last_active":"2024-09-10T18:31:45.544071+0000","last_peered":"2024-09-10T18:31:45.544071+0000","last_clean":"2024-09-10T18:31:45.544071+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:31:45.544071+0000","last_undegraded":"2024-09-10T18:31:45.544071+0000","last_fullsized":"2024-09-10T18:31:45.544071+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:31:45.543996+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:31:45.543996+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:53.615668+0000","scrub_duration":0.0016860379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":232,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250789+0000","last_change":"2024-09-10T18:28:57.977326+0000","last_active":"2024-09-10T18:31:41.250789+0000","last_peered":"2024-09-10T18:31:41.250789+0000","last_clean":"2024-09-10T18:31:41.250789+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:31:41.250789+0000","last_undegraded":"2024-09-10T18:31:41.250789+0000","last_fullsized":"2024-09-10T18:31:41.250789+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:28:57.977273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:28:57.977273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019164130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":361,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:45.047187+0000","last_change":"2024-09-10T18:29:57.457944+0000","last_active":"2024-09-10T18:31:45.047187+0000","last_peered":"2024-09-10T18:31:45.047187+0000","last_clean":"2024-09-10T18:31:45.047187+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:31:45.047187+0000","last_undegraded":"2024-09-10T18:31:45.047187+0000","last_fullsized":"2024-09-10T18:31:45.047187+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:29:57.457607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:29:57.457607+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033795549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":235,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.039004+0000","last_change":"2024-09-10T18:31:06.427465+0000","last_active":"2024-09-10T18:31:41.039004+0000","last_peered":"2024-09-10T18:31:41.039004+0000","last_clean":"2024-09-10T18:31:41.039004+0000","last_became_active":"2024-09-10T18:21:37.639706+0000","last_became_peered":"2024-09-10T18:21:37.639706+0000","last_unstale":"2024-09-10T18:31:41.039004+0000","last_undegraded":"2024-09-10T18:31:41.039004+0000","last_fullsized":"2024-09-10T18:31:41.039004+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:06.427389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:06.427389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:31.517707+0000","scrub_duration":0.0019673400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":271,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:48.766329+0000","last_change":"2024-09-10T18:31:48.766301+0000","last_active":"2024-09-10T18:31:48.766329+0000","last_peered":"2024-09-10T18:31:48.766329+0000","last_clean":"2024-09-10T18:31:48.766329+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:31:48.766329+0000","last_undegraded":"2024-09-10T18:31:48.766329+0000","last_fullsized":"2024-09-10T18:31:48.766329+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-10T18:31:48.766245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.766245+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:49.312976+0000","scrub_duration":0.001995817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":241,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251185+0000","last_change":"2024-09-10T18:30:53.519511+0000","last_active":"2024-09-10T18:31:41.251185+0000","last_peered":"2024-09-10T18:31:41.251185+0000","last_clean":"2024-09-10T18:31:41.251185+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:31:41.251185+0000","last_undegraded":"2024-09-10T18:31:41.251185+0000","last_fullsized":"2024-09-10T18:31:41.251185+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:53.519424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:30:53.519424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:58.732708+0000","scrub_duration":0.002185144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":370,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:45.498351+0000","last_change":"2024-09-10T18:30:53.452921+0000","last_active":"2024-09-10T18:31:45.498351+0000","last_peered":"2024-09-10T18:31:45.498351+0000","last_clean":"2024-09-10T18:31:45.498351+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:31:45.498351+0000","last_undegraded":"2024-09-10T18:31:45.498351+0000","last_fullsized":"2024-09-10T18:31:45.498351+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:30:53.452719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:53.452719+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:04.758158+0000","scrub_duration":0.0035769230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":226,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252153+0000","last_change":"2024-09-10T18:30:01.300052+0000","last_active":"2024-09-10T18:31:41.252153+0000","last_peered":"2024-09-10T18:31:41.252153+0000","last_clean":"2024-09-10T18:31:41.252153+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:31:41.252153+0000","last_undegraded":"2024-09-10T18:31:41.252153+0000","last_fullsized":"2024-09-10T18:31:41.252153+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:01.298802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:01.298802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.024539056,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":277,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:47.604064+0000","last_change":"2024-09-10T18:31:47.604036+0000","last_active":"2024-09-10T18:31:47.604064+0000","last_peered":"2024-09-10T18:31:47.604064+0000","last_clean":"2024-09-10T18:31:47.604064+0000","last_became_active":"2024-09-10T18:26:27.649202+0000","last_became_peered":"2024-09-10T18:26:27.649202+0000","last_unstale":"2024-09-10T18:31:47.604064+0000","last_undegraded":"2024-09-10T18:31:47.604064+0000","last_fullsized":"2024-09-10T18:31:47.604064+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:31:47.603984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:47.603984+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:54.830645+0000","scrub_duration":0.0018863860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":241,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.039064+0000","last_change":"2024-09-10T18:29:57.246363+0000","last_active":"2024-09-10T18:31:41.039064+0000","last_peered":"2024-09-10T18:31:41.039064+0000","last_clean":"2024-09-10T18:31:41.039064+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:31:41.039064+0000","last_undegraded":"2024-09-10T18:31:41.039064+0000","last_fullsized":"2024-09-10T18:31:41.039064+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:29:57.246282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019570709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":240,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.036688+0000","last_change":"2024-09-10T18:29:42.658563+0000","last_active":"2024-09-10T18:31:41.036688+0000","last_peered":"2024-09-10T18:31:41.036688+0000","last_clean":"2024-09-10T18:31:41.036688+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:31:41.036688+0000","last_undegraded":"2024-09-10T18:31:41.036688+0000","last_fullsized":"2024-09-10T18:31:41.036688+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:42.658490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:29:42.658490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059542680000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":227,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:45.798013+0000","last_change":"2024-09-10T18:31:45.797979+0000","last_active":"2024-09-10T18:31:45.798013+0000","last_peered":"2024-09-10T18:31:45.798013+0000","last_clean":"2024-09-10T18:31:45.798013+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:31:45.798013+0000","last_undegraded":"2024-09-10T18:31:45.798013+0000","last_fullsized":"2024-09-10T18:31:45.798013+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:45.797924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:31:45.797924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:05.950165+0000","scrub_duration":0.001612093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"240'15","reported_seq":271,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.036779+0000","last_change":"2024-09-10T18:30:03.639358+0000","last_active":"2024-09-10T18:31:41.036779+0000","last_peered":"2024-09-10T18:31:41.036779+0000","last_clean":"2024-09-10T18:31:41.036779+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:31:41.036779+0000","last_undegraded":"2024-09-10T18:31:41.036779+0000","last_fullsized":"2024-09-10T18:31:41.036779+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:30:03.639256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:03.639256+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010855609,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":244,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252353+0000","last_change":"2024-09-10T18:30:47.525989+0000","last_active":"2024-09-10T18:31:41.252353+0000","last_peered":"2024-09-10T18:31:41.252353+0000","last_clean":"2024-09-10T18:31:41.252353+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:31:41.252353+0000","last_undegraded":"2024-09-10T18:31:41.252353+0000","last_fullsized":"2024-09-10T18:31:41.252353+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:47.525912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:30:47.525912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:03.043559+0000","scrub_duration":0.0075901459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":229,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:44.838275+0000","last_change":"2024-09-10T18:31:44.838204+0000","last_active":"2024-09-10T18:31:44.838275+0000","last_peered":"2024-09-10T18:31:44.838275+0000","last_clean":"2024-09-10T18:31:44.838275+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:31:44.838275+0000","last_undegraded":"2024-09-10T18:31:44.838275+0000","last_fullsized":"2024-09-10T18:31:44.838275+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:44.838141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:31:44.838141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:04.036018+0000","scrub_duration":0.0011071239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":226,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250180+0000","last_change":"2024-09-10T18:29:41.038597+0000","last_active":"2024-09-10T18:31:41.250180+0000","last_peered":"2024-09-10T18:31:41.250180+0000","last_clean":"2024-09-10T18:31:41.250180+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:31:41.250180+0000","last_undegraded":"2024-09-10T18:31:41.250180+0000","last_fullsized":"2024-09-10T18:31:41.250180+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:41.038513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:41.038513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01221999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":269,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252190+0000","last_change":"2024-09-10T18:29:54.385449+0000","last_active":"2024-09-10T18:31:41.252190+0000","last_peered":"2024-09-10T18:31:41.252190+0000","last_clean":"2024-09-10T18:31:41.252190+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:31:41.252190+0000","last_undegraded":"2024-09-10T18:31:41.252190+0000","last_fullsized":"2024-09-10T18:31:41.252190+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:29:54.385389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:29:54.385389+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002949199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":242,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252613+0000","last_change":"2024-09-10T18:30:50.439256+0000","last_active":"2024-09-10T18:31:41.252613+0000","last_peered":"2024-09-10T18:31:41.252613+0000","last_clean":"2024-09-10T18:31:41.252613+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:31:41.252613+0000","last_undegraded":"2024-09-10T18:31:41.252613+0000","last_fullsized":"2024-09-10T18:31:41.252613+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:50.439179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:30:50.439179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:00.808451+0000","scrub_duration":0.0056857770000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":237,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250918+0000","last_change":"2024-09-10T18:29:45.432826+0000","last_active":"2024-09-10T18:31:41.250918+0000","last_peered":"2024-09-10T18:31:41.250918+0000","last_clean":"2024-09-10T18:31:41.250918+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:31:41.250918+0000","last_undegraded":"2024-09-10T18:31:41.250918+0000","last_fullsized":"2024-09-10T18:31:41.250918+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:45.432776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:29:45.432776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001236942,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":244,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.253558+0000","last_change":"2024-09-10T18:31:14.775776+0000","last_active":"2024-09-10T18:31:41.253558+0000","last_peered":"2024-09-10T18:31:41.253558+0000","last_clean":"2024-09-10T18:31:41.253558+0000","last_became_active":"2024-09-10T18:26:27.636996+0000","last_became_peered":"2024-09-10T18:26:27.636996+0000","last_unstale":"2024-09-10T18:31:41.253558+0000","last_undegraded":"2024-09-10T18:31:41.253558+0000","last_fullsized":"2024-09-10T18:31:41.253558+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:14.775693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:14.775693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:29.982163+0000","scrub_duration":0.00163416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":250,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.250749+0000","last_change":"2024-09-10T18:30:45.710780+0000","last_active":"2024-09-10T18:31:41.250749+0000","last_peered":"2024-09-10T18:31:41.250749+0000","last_clean":"2024-09-10T18:31:41.250749+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:31:41.250749+0000","last_undegraded":"2024-09-10T18:31:41.250749+0000","last_fullsized":"2024-09-10T18:31:41.250749+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-09-10T18:30:45.710665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:45.710665+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:53.558994+0000","scrub_duration":0.0078962489999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":235,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252580+0000","last_change":"2024-09-10T18:30:05.204555+0000","last_active":"2024-09-10T18:31:41.252580+0000","last_peered":"2024-09-10T18:31:41.252580+0000","last_clean":"2024-09-10T18:31:41.252580+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:31:41.252580+0000","last_undegraded":"2024-09-10T18:31:41.252580+0000","last_fullsized":"2024-09-10T18:31:41.252580+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:30:05.204467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021240970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":190,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251038+0000","last_change":"2024-09-10T18:31:10.050895+0000","last_active":"2024-09-10T18:31:41.251038+0000","last_peered":"2024-09-10T18:31:41.251038+0000","last_clean":"2024-09-10T18:31:41.251038+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:31:41.251038+0000","last_undegraded":"2024-09-10T18:31:41.251038+0000","last_fullsized":"2024-09-10T18:31:41.251038+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:10.050810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:10.050810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:32.282596+0000","scrub_duration":0.0052628850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":235,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.251330+0000","last_change":"2024-09-10T18:30:58.517218+0000","last_active":"2024-09-10T18:31:41.251330+0000","last_peered":"2024-09-10T18:31:41.251330+0000","last_clean":"2024-09-10T18:31:41.251330+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:31:41.251330+0000","last_undegraded":"2024-09-10T18:31:41.251330+0000","last_fullsized":"2024-09-10T18:31:41.251330+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:58.517160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:58.517160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:14.392419+0000","scrub_duration":0.0070680939999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":245,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-10T18:31:49.495644+0000","last_change":"2024-09-10T18:31:49.495614+0000","last_active":"2024-09-10T18:31:49.495644+0000","last_peered":"2024-09-10T18:31:49.495644+0000","last_clean":"2024-09-10T18:31:49.495644+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:31:49.495644+0000","last_undegraded":"2024-09-10T18:31:49.495644+0000","last_fullsized":"2024-09-10T18:31:49.495644+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:49.495556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:49.495556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:12.834399+0000","scrub_duration":0.0012999590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":233,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:43.830621+0000","last_change":"2024-09-10T18:31:43.830584+0000","last_active":"2024-09-10T18:31:43.830621+0000","last_peered":"2024-09-10T18:31:43.830621+0000","last_clean":"2024-09-10T18:31:43.830621+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:31:43.830621+0000","last_undegraded":"2024-09-10T18:31:43.830621+0000","last_fullsized":"2024-09-10T18:31:43.830621+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:43.830528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:31:43.830528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:58.039416+0000","scrub_duration":0.0011198810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":241,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252206+0000","last_change":"2024-09-10T18:30:50.074300+0000","last_active":"2024-09-10T18:31:41.252206+0000","last_peered":"2024-09-10T18:31:41.252206+0000","last_clean":"2024-09-10T18:31:41.252206+0000","last_became_active":"2024-09-10T18:26:27.648112+0000","last_became_peered":"2024-09-10T18:26:27.648112+0000","last_unstale":"2024-09-10T18:31:41.252206+0000","last_undegraded":"2024-09-10T18:31:41.252206+0000","last_fullsized":"2024-09-10T18:31:41.252206+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:30:50.074214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:50.074214+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:31:53.216359+0000","scrub_duration":0.00651967,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":228,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-10T18:31:41.252461+0000","last_change":"2024-09-10T18:29:05.092825+0000","last_active":"2024-09-10T18:31:41.252461+0000","last_peered":"2024-09-10T18:31:41.252461+0000","last_clean":"2024-09-10T18:31:41.252461+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:31:41.252461+0000","last_undegraded":"2024-09-10T18:31:41.252461+0000","last_fullsized":"2024-09-10T18:31:41.252461+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:29:05.092434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021443608999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":268438,"num_write_kb":8589953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80897,"ondisk_log_size":80897,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1126,"num_read_kb":883,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":731,"ondisk_log_size":731,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":611,"num_read_kb":606,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":891,"ondisk_log_size":891,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234111,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298408,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446744,"statfs":{"total":95995035648,"available":95689465856,"internally_reserved":0,"allocated":1081344,"data_stored":706978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:31:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":2.2669999999999999,"15min":1.569},"min":{"1min":0.26200000000000001,"5min":0.21099999999999999,"15min":0.20799999999999999},"max":{"1min":3.8959999999999999,"5min":75.506,"15min":85.012},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":2.1429999999999998,"15min":1.5369999999999999},"min":{"1min":0.28199999999999997,"5min":0.22800000000000001,"15min":0.222},"max":{"1min":3.3250000000000002,"5min":75.710999999999999,"15min":84.665999999999997},"last":0.38800000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:31:44 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":2.0009999999999999,"15min":1.1699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":6.4530000000000003,"5min":49.317999999999998,"15min":49.317999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.127,"5min":1.7889999999999999,"15min":1.1240000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.3029999999999999,"5min":49.353000000000002,"15min":49.353000000000002},"last":0.498}]},{"osd":2,"last update":"Tue Sep 10 18:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.97999999999999998,"15min":0.75700000000000001},"min":{"1min":0.26800000000000002,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":2.6960000000000002,"5min":5.3150000000000004,"15min":5.3150000000000004},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.1559999999999999,"15min":0.83299999999999996},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.6529999999999996,"5min":9.9450000000000003,"15min":9.9450000000000003},"last":0.40899999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:31:44 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.093,"15min":1.294},"min":{"1min":0.29999999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.4160000000000004,"5min":5.718,"15min":86.305999999999997},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.1539999999999999,"15min":1.335},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.26700000000000002},"max":{"1min":4.4660000000000002,"5min":7.3339999999999996,"15min":88.364000000000004},"last":0.54100000000000004}]},{"osd":4,"last update":"Tue Sep 10 18:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.58299999999999996,"15min":0.47099999999999997},"min":{"1min":0.224,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":2.0990000000000002,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.623,"15min":0.48999999999999999},"min":{"1min":0.26300000000000001,"5min":0.25,"15min":0.223},"max":{"1min":2.2690000000000001,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":0.39800000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:31:44 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":0.93400000000000005,"15min":0.58899999999999997},"min":{"1min":0.29799999999999999,"5min":0.27600000000000002,"15min":0.23499999999999999},"max":{"1min":7.1820000000000004,"5min":7.3109999999999999,"15min":7.3109999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":1.022,"5min":0.73999999999999999,"15min":0.55400000000000005},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.219},"max":{"1min":6.3360000000000003,"5min":6.3360000000000003,"15min":6.3360000000000003},"last":0.435}]},{"osd":6,"last update":"Tue Sep 10 18:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84899999999999998,"15min":0.56000000000000005},"min":{"1min":0.28899999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.3900000000000001,"5min":7.2750000000000004,"15min":7.2750000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":1.143,"5min":0.73199999999999998,"15min":0.52800000000000002},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.585,"5min":4.585,"15min":4.585},"last":0.46800000000000003}]}]},{"osd":6,"up_from":144,"seq":618475290690,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311640,"kb_used_data":720,"kb_used_omap":7,"kb_used_meta":310904,"kb_avail":93433512,"statfs":{"total":95995035648,"available":95675916288,"internally_reserved":0,"allocated":737280,"data_stored":379470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":318365783},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":1.8009999999999999,"15min":2.008},"min":{"1min":0.28100000000000003,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.3949999999999996,"5min":30.629999999999999,"15min":30.629999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.591,"5min":1.3839999999999999,"15min":1.49},"min":{"1min":0.23100000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":8.4640000000000004,"5min":30.597000000000001,"15min":30.597000000000001},"last":0.45900000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.3129999999999999,"15min":0.91800000000000004},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.0329999999999995,"5min":30.984000000000002,"15min":30.984000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":1.6819999999999999,"5min":1.46,"15min":1.157},"min":{"1min":0.246,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.7859999999999996,"5min":30.571000000000002,"15min":30.571000000000002},"last":0.442}]},{"osd":2,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.075},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.6539999999999999,"5min":4.8259999999999996,"15min":4.8259999999999996},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.448,"5min":1.0409999999999999,"15min":0.86199999999999999},"min":{"1min":0.33100000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":5.415,"5min":11.164999999999999,"15min":11.164999999999999},"last":0.499}]},{"osd":3,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.9079999999999999,"15min":2.0910000000000002},"min":{"1min":0.29799999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.8520000000000003,"5min":61.703000000000003,"15min":61.703000000000003},"last":0.433},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.909,"15min":2.0750000000000002},"min":{"1min":0.29499999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.7229999999999999,"5min":61.814,"15min":61.814},"last":0.51400000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.66400000000000003,"15min":0.54500000000000004},"min":{"1min":0.27600000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.6219999999999999,"5min":5.4950000000000001,"15min":5.4950000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.68400000000000005,"15min":0.65700000000000003},"min":{"1min":0.27800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":4.6390000000000002,"5min":4.6390000000000002,"15min":4.6390000000000002},"last":0.33900000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.64300000000000002,"15min":0.52300000000000002},"min":{"1min":0.27200000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.6619999999999999,"5min":4.6619999999999999,"15min":4.6619999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.626,"15min":0.47799999999999998},"min":{"1min":0.313,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.6219999999999999,"5min":2.6219999999999999,"15min":2.6219999999999999},"last":0.47199999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.60099999999999998,"15min":0.52000000000000002},"min":{"1min":0.33800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.74199999999999999,"15min":0.56399999999999995},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.9300000000000002,"5min":5.5309999999999997,"15min":5.5309999999999997},"last":0.48999999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724682,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306992,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93438160,"statfs":{"total":95995035648,"available":95680675840,"internally_reserved":0,"allocated":1351680,"data_stored":970172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:30:59 2024","interfaces":[{"interface":"back","average":{"1min":2.8849999999999998,"5min":2.407,"15min":1.385},"min":{"1min":0.23499999999999999,"5min":0.22900000000000001,"15min":0.21099999999999999},"max":{"1min":26.288,"5min":52.033999999999999,"15min":52.033999999999999},"last":0.372},{"interface":"front","average":{"1min":2.6549999999999998,"5min":2.4049999999999998,"15min":1.2210000000000001},"min":{"1min":0.33800000000000002,"5min":0.22800000000000001,"15min":0.216},"max":{"1min":24.178000000000001,"5min":52.167999999999999,"15min":52.167999999999999},"last":0.40500000000000003}]},{"osd":1,"last update":"Tue Sep 10 18:30:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4689999999999999,"5min":1.296,"15min":0.82899999999999996},"min":{"1min":0.33100000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":24.16,"5min":24.16,"15min":24.16},"last":0.379},{"interface":"front","average":{"1min":2.5979999999999999,"5min":1.204,"15min":0.90200000000000002},"min":{"1min":0.28499999999999998,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":24.129999999999999,"5min":24.129999999999999,"15min":24.129999999999999},"last":0.33600000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:30:59 2024","interfaces":[{"interface":"back","average":{"1min":1.6699999999999999,"5min":1.621,"15min":0.96199999999999997},"min":{"1min":0.26900000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":16.736999999999998,"5min":36.853000000000002,"15min":36.853000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.304,"5min":1.5429999999999999,"15min":0.93000000000000005},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":8.1869999999999994,"5min":31.132000000000001,"15min":31.132000000000001},"last":0.34499999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:30:59 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.101,"15min":1.2170000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":30.475000000000001,"5min":31.207000000000001,"15min":31.207000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":3.2120000000000002,"5min":2.0350000000000001,"15min":1.119},"min":{"1min":0.38700000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":30.172000000000001,"5min":31.199000000000002,"15min":31.199000000000002},"last":0.42099999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:30:59 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.59299999999999997,"15min":0.436},"min":{"1min":0.22900000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.879,"5min":2.879,"15min":2.879},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.64900000000000002,"15min":0.47299999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.22700000000000001},"max":{"1min":2.2730000000000001,"5min":4.7489999999999997,"15min":4.7489999999999997},"last":0.32500000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.79200000000000004,"15min":0.63700000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.194,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.82199999999999995,"15min":0.63300000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":5.3680000000000003,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.27000000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:31:29 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.80700000000000005,"15min":0.55300000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.22500000000000001},"max":{"1min":5.4900000000000002,"5min":5.569,"15min":5.569},"last":0.318},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63900000000000001,"15min":0.496},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.24299999999999999},"max":{"1min":3.173,"5min":3.1819999999999999,"15min":3.1819999999999999},"last":0.36099999999999999}]}]},{"osd":4,"up_from":33,"seq":141733920910,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298088,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447064,"statfs":{"total":95995035648,"available":95689793536,"internally_reserved":0,"allocated":753664,"data_stored":379109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:31:30 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.383,"15min":0.99099999999999999},"min":{"1min":0.26900000000000002,"5min":0.21199999999999999,"15min":0.188},"max":{"1min":4.2009999999999996,"5min":16.236999999999998,"15min":31.585000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.125,"15min":0.94599999999999995},"min":{"1min":0.23000000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.3129999999999997,"5min":16.280999999999999,"15min":29.802},"last":0.48799999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:31:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.93999999999999995,"15min":0.745},"min":{"1min":0.24399999999999999,"5min":0.22600000000000001,"15min":0.21299999999999999},"max":{"1min":6.8559999999999999,"5min":6.8559999999999999,"15min":6.8559999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.1060000000000001,"15min":0.79700000000000004},"min":{"1min":0.309,"5min":0.246,"15min":0.22900000000000001},"max":{"1min":4.1210000000000004,"5min":5.085,"15min":5.085},"last":0.46300000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":1.008,"15min":0.83599999999999997},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.0519999999999996,"5min":6.4180000000000001,"15min":13.898999999999999},"last":0.377},{"interface":"front","average":{"1min":1.056,"5min":0.96599999999999997,"15min":0.85199999999999998},"min":{"1min":0.29399999999999998,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":4.0460000000000003,"5min":4.1950000000000003,"15min":14.51},"last":0.45000000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:31:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.464,"15min":1.052},"min":{"1min":0.26300000000000001,"5min":0.249,"15min":0.249},"max":{"1min":3.6709999999999998,"5min":37.351999999999997,"15min":37.351999999999997},"last":0.33100000000000002},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.4470000000000001,"15min":1.0920000000000001},"min":{"1min":0.35199999999999998,"5min":0.28000000000000003,"15min":0.255},"max":{"1min":2.8639999999999999,"5min":37.366,"15min":37.366},"last":0.502}]},{"osd":5,"last update":"Tue Sep 10 18:30:50 2024","interfaces":[{"interface":"back","average":{"1min":1.8100000000000001,"5min":0.82099999999999995,"15min":0.52100000000000002},"min":{"1min":0.28000000000000003,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":22.861000000000001,"5min":22.861000000000001,"15min":22.861000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.6459999999999999,"5min":0.84299999999999997,"15min":0.55000000000000004},"min":{"1min":0.32900000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":19.611999999999998,"5min":19.611999999999998,"15min":19.611999999999998},"last":0.39300000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58099999999999996,"15min":0.45100000000000001},"min":{"1min":0.27900000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.5310000000000001,"5min":6.7990000000000004,"15min":6.7990000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.66200000000000003,"15min":0.51600000000000001},"min":{"1min":0.33200000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.883,"5min":7.0300000000000002,"15min":7.0300000000000002},"last":0.43099999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.64900000000000002,"15min":0.53200000000000003},"min":{"1min":0.317,"5min":0.28799999999999998,"15min":0.255},"max":{"1min":4.8600000000000003,"5min":7.0670000000000002,"15min":7.0670000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.69599999999999995,"15min":0.54300000000000004},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.247},"max":{"1min":2.6139999999999999,"5min":6.5330000000000004,"15min":6.5330000000000004},"last":0.51400000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117140,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311160,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93433992,"statfs":{"total":95995035648,"available":95676407808,"internally_reserved":0,"allocated":1359872,"data_stored":970321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.5569999999999999,"15min":1.0880000000000001},"min":{"1min":0.33800000000000002,"5min":0.218,"15min":0.19900000000000001},"max":{"1min":3.3769999999999998,"5min":7.0899999999999999,"15min":8.5530000000000008},"last":0.30499999999999999},{"interface":"front","average":{"1min":2.0209999999999999,"5min":1.722,"15min":1.1399999999999999},"min":{"1min":0.307,"5min":0.193,"15min":0.193},"max":{"1min":6.7960000000000003,"5min":13.659000000000001,"15min":13.659000000000001},"last":0.33300000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:31:08 2024","interfaces":[{"interface":"back","average":{"1min":3.8330000000000002,"5min":2.0419999999999998,"15min":1.218},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.247},"max":{"1min":35.406999999999996,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.36799999999999999},{"interface":"front","average":{"1min":4.5890000000000004,"5min":2.0579999999999998,"15min":1.1519999999999999},"min":{"1min":0.25,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":36.902000000000001,"5min":36.902000000000001,"15min":36.902000000000001},"last":0.35499999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.29,"15min":0.95799999999999996},"min":{"1min":0.25700000000000001,"5min":0.222,"15min":0.222},"max":{"1min":3.403,"5min":7.0439999999999996,"15min":7.7960000000000003},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.9710000000000001,"5min":1.8380000000000001,"15min":1.117},"min":{"1min":0.29099999999999998,"5min":0.217,"15min":0.217},"max":{"1min":14.997999999999999,"5min":14.997999999999999,"15min":14.997999999999999},"last":0.317}]},{"osd":4,"last update":"Tue Sep 10 18:31:41 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.319,"15min":1.103},"min":{"1min":0.33300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.2919999999999998,"5min":6.1059999999999999,"15min":23.684999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":1.325,"5min":1.121,"15min":1.0529999999999999},"min":{"1min":0.36399999999999999,"5min":0.29899999999999999,"15min":0.27200000000000002},"max":{"1min":5.2160000000000002,"5min":5.2160000000000002,"15min":23.649000000000001},"last":0.65500000000000003}]},{"osd":5,"last update":"Tue Sep 10 18:30:47 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":1.5169999999999999,"15min":1.133},"min":{"1min":0.35399999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.2250000000000001,"5min":7.3339999999999996,"15min":23.719999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.2410000000000001,"15min":1.0900000000000001},"min":{"1min":0.35699999999999998,"5min":0.35399999999999998,"15min":0.33100000000000002},"max":{"1min":2.3399999999999999,"5min":5.3440000000000003,"15min":23.707000000000001},"last":0.622}]},{"osd":6,"last update":"Tue Sep 10 18:31:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.6950000000000001,"15min":1.151},"min":{"1min":0.379,"5min":0.317,"15min":0.317},"max":{"1min":2.9180000000000001,"5min":31.815999999999999,"15min":31.815999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.958,"5min":1.8700000000000001,"15min":1.5109999999999999},"min":{"1min":0.45400000000000001,"5min":0.315,"15min":0.315},"max":{"1min":6.7640000000000002,"5min":31.742000000000001,"15min":31.742000000000001},"last":0.66500000000000004}]},{"osd":7,"last update":"Tue Sep 10 18:31:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.448,"15min":1.137},"min":{"1min":0.38200000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.2690000000000001,"5min":6.077,"15min":23.748999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.345,"5min":1.157,"15min":1.1319999999999999},"min":{"1min":0.43099999999999999,"5min":0.35199999999999998,"15min":0.307},"max":{"1min":2.8170000000000002,"5min":4.4180000000000001,"15min":25.058},"last":0.76800000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411480,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310816,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93434336,"statfs":{"total":95995035648,"available":95676760064,"internally_reserved":0,"allocated":1003520,"data_stored":641269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:31:46 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":1.9650000000000001,"15min":1.256},"min":{"1min":0.218,"5min":0.214,"15min":0.185},"max":{"1min":15.098000000000001,"5min":43.890000000000001,"15min":43.890000000000001},"last":0.27900000000000003},{"interface":"front","average":{"1min":2.1349999999999998,"5min":2.3599999999999999,"15min":1.4450000000000001},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.16900000000000001},"max":{"1min":15.019,"5min":44.331000000000003,"15min":44.331000000000003},"last":0.49399999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:31:46 2024","interfaces":[{"interface":"back","average":{"1min":2.3290000000000002,"5min":2.1760000000000002,"15min":1.321},"min":{"1min":0.23599999999999999,"5min":0.193,"15min":0.193},"max":{"1min":15.750999999999999,"5min":44.575000000000003,"15min":44.575000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":2.3820000000000001,"5min":1.901,"15min":1.139},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.19900000000000001},"max":{"1min":14.988,"5min":44.368000000000002,"15min":44.368000000000002},"last":0.374}]},{"osd":3,"last update":"Tue Sep 10 18:31:04 2024","interfaces":[{"interface":"back","average":{"1min":1.881,"5min":1.861,"15min":1.3400000000000001},"min":{"1min":0.27000000000000002,"5min":0.26300000000000001,"15min":0.251},"max":{"1min":7.181,"5min":12.579000000000001,"15min":60.786999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.8620000000000001,"15min":1.3620000000000001},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.214},"max":{"1min":6.7999999999999998,"5min":12.712999999999999,"15min":60.116999999999997},"last":0.41899999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:31:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.1739999999999999,"15min":0.93600000000000005},"min":{"1min":0.28699999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":4.641,"5min":5.9249999999999998,"15min":6.9169999999999998},"last":0.436},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.161,"15min":0.91200000000000003},"min":{"1min":0.34200000000000003,"5min":0.32900000000000001,"15min":0.317},"max":{"1min":7.0730000000000004,"5min":7.0730000000000004,"15min":12.130000000000001},"last":0.60799999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:30:52 2024","interfaces":[{"interface":"back","average":{"1min":1.982,"5min":1.1899999999999999,"15min":0.89600000000000002},"min":{"1min":0.30299999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":7.1609999999999996,"5min":7.1609999999999996,"15min":12.106},"last":0.64100000000000001},{"interface":"front","average":{"1min":2.0390000000000001,"5min":1.1910000000000001,"15min":0.91100000000000003},"min":{"1min":0.41999999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":7.1109999999999998,"5min":7.1109999999999998,"15min":12.154},"last":0.52900000000000003}]},{"osd":6,"last update":"Tue Sep 10 18:31:35 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.226,"15min":1.155},"min":{"1min":0.36799999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":4.6849999999999996,"5min":5.9989999999999997,"15min":5.9989999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.552,"5min":1.234,"15min":0.97499999999999998},"min":{"1min":0.39200000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":7.1349999999999998,"5min":7.1349999999999998,"15min":7.1349999999999998},"last":0.57399999999999995}]},{"osd":7,"last update":"Tue Sep 10 18:31:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":0.97899999999999998},"min":{"1min":0.374,"5min":0.34200000000000003,"15min":0.30499999999999999},"max":{"1min":5.1210000000000004,"5min":6.2990000000000004,"15min":7.1639999999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.498,"5min":1.3520000000000001,"15min":1.014},"min":{"1min":0.45600000000000002,"5min":0.311,"15min":0.307},"max":{"1min":4.7169999999999996,"5min":5.9660000000000002,"15min":6.9669999999999996},"last":0.55100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575005,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302260,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93442892,"statfs":{"total":95995035648,"available":95685521408,"internally_reserved":0,"allocated":765952,"data_stored":379711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:31:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.923,"15min":1.3700000000000001},"min":{"1min":0.33400000000000002,"5min":0.20399999999999999,"15min":0.16700000000000001},"max":{"1min":3.202,"5min":42.420000000000002,"15min":80.456999999999994},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.46,"5min":1.9710000000000001,"15min":1.454},"min":{"1min":0.38100000000000001,"5min":0.187,"15min":0.187},"max":{"1min":5.6189999999999998,"5min":42.548000000000002,"15min":80.578000000000003},"last":0.45600000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:31:38 2024","interfaces":[{"interface":"back","average":{"1min":1.706,"5min":1.579,"15min":1.093},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.6479999999999997,"5min":9.4700000000000006,"15min":13.035},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.5940000000000001,"15min":1.026},"min":{"1min":0.29899999999999999,"5min":0.216,"15min":0.19900000000000001},"max":{"1min":4.0670000000000002,"5min":12.222,"15min":12.222},"last":0.436}]},{"osd":3,"last update":"Tue Sep 10 18:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":2.0179999999999998,"15min":1.3939999999999999},"min":{"1min":0.35799999999999998,"5min":0.23899999999999999,"15min":0.22600000000000001},"max":{"1min":3.3450000000000002,"5min":43.076000000000001,"15min":43.076000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.4399999999999999,"5min":2.1560000000000001,"15min":1.476},"min":{"1min":0.32400000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.5860000000000003,"5min":41.93,"15min":41.93},"last":0.53500000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:31:26 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.345,"15min":0.94099999999999995},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.29799999999999999},"max":{"1min":3.9100000000000001,"5min":7.2859999999999996,"15min":7.2859999999999996},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.4770000000000001,"15min":1.0449999999999999},"min":{"1min":0.38100000000000001,"5min":0.30199999999999999,"15min":0.28299999999999997},"max":{"1min":4.0359999999999996,"5min":6.1600000000000001,"15min":19.393000000000001},"last":0.58599999999999997}]},{"osd":5,"last update":"Tue Sep 10 18:30:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.6639999999999999,"15min":1.05},"min":{"1min":0.35899999999999999,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":3.9990000000000001,"5min":12.206,"15min":15.48},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.151,"5min":1.48,"15min":0.95899999999999996},"min":{"1min":0.38600000000000001,"5min":0.34100000000000003,"15min":0.308},"max":{"1min":3.9119999999999999,"5min":9.9359999999999999,"15min":9.9359999999999999},"last":0.54800000000000004}]},{"osd":6,"last update":"Tue Sep 10 18:31:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.337,"15min":1.347},"min":{"1min":0.42599999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.7320000000000002,"5min":10.039,"15min":10.039},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.353,"5min":1.4570000000000001,"15min":1.3999999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.9449999999999998,"5min":7.3369999999999997,"15min":7.3369999999999997},"last":0.59599999999999997}]},{"osd":7,"last update":"Tue Sep 10 18:31:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.3680000000000001,"15min":1.036},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.34399999999999997},"max":{"1min":3.7530000000000001,"5min":10.005000000000001,"15min":15.545999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.538,"5min":1.393,"15min":0.999},"min":{"1min":0.38600000000000001,"5min":0.35699999999999998,"15min":0.313},"max":{"1min":5.5289999999999999,"5min":7.359,"15min":7.359},"last":0.56899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738530,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310564,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93434588,"statfs":{"total":95995035648,"available":95677018112,"internally_reserved":0,"allocated":749568,"data_stored":378982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:31:16 2024","interfaces":[{"interface":"back","average":{"1min":1.8280000000000001,"5min":2.0350000000000001,"15min":1.2150000000000001},"min":{"1min":0.23400000000000001,"5min":0.193,"15min":0.193},"max":{"1min":7.2800000000000002,"5min":33.130000000000003,"15min":33.130000000000003},"last":0.32200000000000001},{"interface":"front","average":{"1min":2.5680000000000001,"5min":2.2050000000000001,"15min":1.3240000000000001},"min":{"1min":0.46899999999999997,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":7.0289999999999999,"5min":31.329999999999998,"15min":31.329999999999998},"last":0.34499999999999997}]},{"osd":2,"last update":"Tue Sep 10 18:31:46 2024","interfaces":[{"interface":"back","average":{"1min":1.962,"5min":1.7290000000000001,"15min":1.228},"min":{"1min":0.253,"5min":0.22,"15min":0.189},"max":{"1min":7.4329999999999998,"5min":30.219999999999999,"15min":30.219999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":2.0390000000000001,"5min":1.827,"15min":1.2529999999999999},"min":{"1min":0.23400000000000001,"5min":0.22,"15min":0.22},"max":{"1min":7.5119999999999996,"5min":30.129999999999999,"15min":30.129999999999999},"last":0.30399999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.7629999999999999,"5min":1.8,"15min":1.282},"min":{"1min":0.41199999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":6.5369999999999999,"5min":8.5709999999999997,"15min":23.143999999999998},"last":0.40300000000000002},{"interface":"front","average":{"1min":1.9730000000000001,"5min":1.742,"15min":1.2509999999999999},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.1070000000000002,"5min":9.3360000000000003,"15min":23.035},"last":0.38900000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:31:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.349,"15min":1.093},"min":{"1min":0.34100000000000003,"5min":0.246,"15min":0.246},"max":{"1min":4.2640000000000002,"5min":8.3699999999999992,"15min":17.151},"last":0.496},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3160000000000001,"15min":1.0780000000000001},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.25900000000000001},"max":{"1min":3.7919999999999998,"5min":18.728999999999999,"15min":18.728999999999999},"last":0.47499999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:30:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":1.478,"15min":1.1559999999999999},"min":{"1min":0.35399999999999998,"5min":0.311,"15min":0.29499999999999998},"max":{"1min":3.746,"5min":18.77,"15min":18.77},"last":0.442},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.4419999999999999,"15min":1.0609999999999999},"min":{"1min":0.377,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.9969999999999999,"5min":6.3769999999999998,"15min":14.991},"last":0.51200000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:31:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":1.3100000000000001,"15min":1.3080000000000001},"min":{"1min":0.372,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.2939999999999996,"5min":8.3940000000000001,"15min":8.3940000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":1.375,"5min":1.3500000000000001,"15min":1.3939999999999999},"min":{"1min":0.48999999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.9079999999999999,"5min":9.7859999999999996,"15min":9.7859999999999996},"last":0.60799999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:31:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.341,"15min":1.0760000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.26700000000000002},"max":{"1min":3.8660000000000001,"5min":5.0430000000000001,"15min":13.711},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.409,"5min":1.387,"15min":1.0669999999999999},"min":{"1min":0.46100000000000002,"5min":0.36199999999999999,"15min":0.33500000000000002},"max":{"1min":6.6399999999999997,"5min":9.6549999999999994,"15min":15.032},"last":0.56799999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:31:52.710 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-10T18:31:52.710 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:31:52.710 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-10T18:31:52.711 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-10T18:31:52.711 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-10T18:31:52.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-10T18:31:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:31:52.780 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:52.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:52.805 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:52.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:52.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:52.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:52.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:53.031 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:53.053 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:53.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:53.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:53.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:53.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:53.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:53.279 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:53.301 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:53.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:53.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:53.349 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[39220]: from='client.15360 v1:172.21.15.28:0/562669823' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:31:53.350 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[39220]: 8.7 scrub starts 2024-09-10T18:31:53.350 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[39220]: 8.7 scrub ok 2024-09-10T18:31:53.350 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:31:53.350 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:31:53.350 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3122887931' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:31:53.350 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[39220]: osdmap e244: 8 total, 8 up, 8 in 2024-09-10T18:31:53.350 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[39220]: 3.2 scrub starts 2024-09-10T18:31:53.350 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[39220]: 3.2 scrub ok 2024-09-10T18:31:53.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[29291]: from='client.15360 v1:172.21.15.28:0/562669823' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:31:53.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[29291]: 8.7 scrub starts 2024-09-10T18:31:53.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[29291]: 8.7 scrub ok 2024-09-10T18:31:53.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:31:53.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:31:53.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3122887931' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:31:53.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[29291]: osdmap e244: 8 total, 8 up, 8 in 2024-09-10T18:31:53.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[29291]: 3.2 scrub starts 2024-09-10T18:31:53.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:53 smithi028 ceph-mon[29291]: 3.2 scrub ok 2024-09-10T18:31:53.353 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:53 smithi204 ceph-mon[32357]: from='client.15360 v1:172.21.15.28:0/562669823' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:31:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:53 smithi204 ceph-mon[32357]: 8.7 scrub starts 2024-09-10T18:31:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:53 smithi204 ceph-mon[32357]: 8.7 scrub ok 2024-09-10T18:31:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:53 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:31:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:53 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:31:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:53 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3122887931' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:31:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:53 smithi204 ceph-mon[32357]: osdmap e244: 8 total, 8 up, 8 in 2024-09-10T18:31:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:53 smithi204 ceph-mon[32357]: 3.2 scrub starts 2024-09-10T18:31:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:53 smithi204 ceph-mon[32357]: 3.2 scrub ok 2024-09-10T18:31:53.377 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-10T18:31:53.378 INFO:tasks.radosbench:Beginning radosbench... 2024-09-10T18:31:53.378 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-10T18:31:53.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-09-10T18:31:53.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:53.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:53.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:53.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:53.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:53.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:53.670 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:53.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:53.693 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:53.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:53.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:53.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:53.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:53.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:53.918 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:53.941 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:53.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:53.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:54.013 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:31:54.013 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:31:54.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:54.067 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:54.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:54.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:54.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:54.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:54.191 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:54.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:54.292 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:54.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:54.314 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:54.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:54 smithi204 ceph-mon[32357]: 3.4 scrub starts 2024-09-10T18:31:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:54 smithi204 ceph-mon[32357]: 3.4 scrub ok 2024-09-10T18:31:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:54 smithi204 ceph-mon[32357]: pgmap v464: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 517 B/s rd, 0 op/s 2024-09-10T18:31:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:54 smithi204 ceph-mon[32357]: 5.16 scrub starts 2024-09-10T18:31:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:54 smithi204 ceph-mon[32357]: 5.16 scrub ok 2024-09-10T18:31:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:54 smithi204 ceph-mon[32357]: 6.1f scrub starts 2024-09-10T18:31:54.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:54 smithi204 ceph-mon[32357]: 6.1f scrub ok 2024-09-10T18:31:54.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:54 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:31:54.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:54 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:31:54.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[29291]: 3.4 scrub starts 2024-09-10T18:31:54.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[29291]: 3.4 scrub ok 2024-09-10T18:31:54.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[29291]: pgmap v464: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 517 B/s rd, 0 op/s 2024-09-10T18:31:54.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[29291]: 5.16 scrub starts 2024-09-10T18:31:54.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[29291]: 5.16 scrub ok 2024-09-10T18:31:54.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[29291]: 6.1f scrub starts 2024-09-10T18:31:54.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[29291]: 6.1f scrub ok 2024-09-10T18:31:54.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:31:54.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:31:54.374 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[39220]: 3.4 scrub starts 2024-09-10T18:31:54.374 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[39220]: 3.4 scrub ok 2024-09-10T18:31:54.374 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[39220]: pgmap v464: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 517 B/s rd, 0 op/s 2024-09-10T18:31:54.374 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[39220]: 5.16 scrub starts 2024-09-10T18:31:54.374 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[39220]: 5.16 scrub ok 2024-09-10T18:31:54.374 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[39220]: 6.1f scrub starts 2024-09-10T18:31:54.374 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[39220]: 6.1f scrub ok 2024-09-10T18:31:54.374 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:31:54.374 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:54 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:31:54.416 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:54.437 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:54.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:54.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:54.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:54.564 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:54.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:54.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:54.622 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-10T18:31:54.665 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:54.687 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:54.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:54.733 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:54.735 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:54.789 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:54.811 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:54.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:54.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:54.856 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:31:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:31:54.872 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:31:54.913 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:54.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:54.936 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:54.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:55.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:55.061 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:55.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:55.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:55.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:55.185 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:55.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[29291]: 3.1b scrub starts 2024-09-10T18:31:55.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[29291]: 3.1b scrub ok 2024-09-10T18:31:55.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[29291]: 5.4 scrub starts 2024-09-10T18:31:55.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[29291]: 5.4 scrub ok 2024-09-10T18:31:55.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[29291]: 4.1f scrub starts 2024-09-10T18:31:55.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[29291]: 4.1f scrub ok 2024-09-10T18:31:55.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[29291]: 3.1d deep-scrub starts 2024-09-10T18:31:55.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[29291]: 3.1d deep-scrub ok 2024-09-10T18:31:55.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[29291]: 6.1b scrub starts 2024-09-10T18:31:55.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[29291]: 6.1b scrub ok 2024-09-10T18:31:55.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[39220]: 3.1b scrub starts 2024-09-10T18:31:55.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[39220]: 3.1b scrub ok 2024-09-10T18:31:55.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[39220]: 5.4 scrub starts 2024-09-10T18:31:55.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[39220]: 5.4 scrub ok 2024-09-10T18:31:55.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[39220]: 4.1f scrub starts 2024-09-10T18:31:55.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[39220]: 4.1f scrub ok 2024-09-10T18:31:55.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[39220]: 3.1d deep-scrub starts 2024-09-10T18:31:55.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[39220]: 3.1d deep-scrub ok 2024-09-10T18:31:55.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[39220]: 6.1b scrub starts 2024-09-10T18:31:55.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:55 smithi028 ceph-mon[39220]: 6.1b scrub ok 2024-09-10T18:31:55.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:55.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:55.283 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:55 smithi204 ceph-mon[32357]: 3.1b scrub starts 2024-09-10T18:31:55.284 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:55 smithi204 ceph-mon[32357]: 3.1b scrub ok 2024-09-10T18:31:55.284 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:55 smithi204 ceph-mon[32357]: 5.4 scrub starts 2024-09-10T18:31:55.284 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:55 smithi204 ceph-mon[32357]: 5.4 scrub ok 2024-09-10T18:31:55.284 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:55 smithi204 ceph-mon[32357]: 4.1f scrub starts 2024-09-10T18:31:55.284 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:55 smithi204 ceph-mon[32357]: 4.1f scrub ok 2024-09-10T18:31:55.284 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:55 smithi204 ceph-mon[32357]: 3.1d deep-scrub starts 2024-09-10T18:31:55.284 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:55 smithi204 ceph-mon[32357]: 3.1d deep-scrub ok 2024-09-10T18:31:55.285 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:55 smithi204 ceph-mon[32357]: 6.1b scrub starts 2024-09-10T18:31:55.285 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:55 smithi204 ceph-mon[32357]: 6.1b scrub ok 2024-09-10T18:31:55.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:55.312 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:55.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:55.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:55.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:55.437 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:55.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:55.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:55.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:55.562 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:55.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:55.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:55.664 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:55.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:55.687 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:55.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:55.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:55.810 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:55.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:55.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:55.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:55.934 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:55.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:55.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:56.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:56.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:56.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:56.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:56.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:31:56.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:56.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:56.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:56.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:56.296 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:56.320 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:56.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:56.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:56.322 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:56 smithi204 ceph-mon[32357]: 3.12 deep-scrub starts 2024-09-10T18:31:56.322 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:56 smithi204 ceph-mon[32357]: 3.12 deep-scrub ok 2024-09-10T18:31:56.322 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:56 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:56.322 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:56 smithi204 ceph-mon[32357]: 5.1d scrub starts 2024-09-10T18:31:56.322 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:56 smithi204 ceph-mon[32357]: 5.1d scrub ok 2024-09-10T18:31:56.322 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:56 smithi204 ceph-mon[32357]: 2.1 scrub starts 2024-09-10T18:31:56.322 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:56 smithi204 ceph-mon[32357]: 2.1 scrub ok 2024-09-10T18:31:56.322 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:56 smithi204 ceph-mon[32357]: pgmap v465: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:31:56.327 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:56 smithi028 ceph-mon[39220]: 3.12 deep-scrub starts 2024-09-10T18:31:56.327 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:56 smithi028 ceph-mon[39220]: 3.12 deep-scrub ok 2024-09-10T18:31:56.327 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:56 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:56.327 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:56 smithi028 ceph-mon[39220]: 5.1d scrub starts 2024-09-10T18:31:56.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:56 smithi028 ceph-mon[39220]: 5.1d scrub ok 2024-09-10T18:31:56.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:56 smithi028 ceph-mon[39220]: 2.1 scrub starts 2024-09-10T18:31:56.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:56 smithi028 ceph-mon[39220]: 2.1 scrub ok 2024-09-10T18:31:56.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:56 smithi028 ceph-mon[39220]: pgmap v465: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:31:56.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:56 smithi028 ceph-mon[29291]: 3.12 deep-scrub starts 2024-09-10T18:31:56.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:56 smithi028 ceph-mon[29291]: 3.12 deep-scrub ok 2024-09-10T18:31:56.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:56 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:31:56.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:56 smithi028 ceph-mon[29291]: 5.1d scrub starts 2024-09-10T18:31:56.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:56 smithi028 ceph-mon[29291]: 5.1d scrub ok 2024-09-10T18:31:56.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:56 smithi028 ceph-mon[29291]: 2.1 scrub starts 2024-09-10T18:31:56.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:56 smithi028 ceph-mon[29291]: 2.1 scrub ok 2024-09-10T18:31:56.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:56 smithi028 ceph-mon[29291]: pgmap v465: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:31:56.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:31:56.443 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:56.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:56.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:31:56.545 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:56.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:56.571 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:56.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:56.578 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:31:56.578 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:31:56.673 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:56.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:56.695 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:56.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:56.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:56.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:56.820 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:56.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:56.921 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:56.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:56.945 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:56.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:57.046 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:57.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:57.068 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:57.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:57.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:57.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:57.191 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:57.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:57.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:57.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:57.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:57.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:57 smithi028 ceph-mon[39220]: 5.e scrub starts 2024-09-10T18:31:57.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:57 smithi028 ceph-mon[39220]: 5.e scrub ok 2024-09-10T18:31:57.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:57 smithi028 ceph-mon[39220]: 4.1c scrub starts 2024-09-10T18:31:57.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:57 smithi028 ceph-mon[39220]: 4.1c scrub ok 2024-09-10T18:31:57.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:57 smithi028 ceph-mon[39220]: 1.1 scrub starts 2024-09-10T18:31:57.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:57 smithi028 ceph-mon[39220]: 1.1 scrub ok 2024-09-10T18:31:57.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:57 smithi028 ceph-mon[29291]: 5.e scrub starts 2024-09-10T18:31:57.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:57 smithi028 ceph-mon[29291]: 5.e scrub ok 2024-09-10T18:31:57.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:57 smithi028 ceph-mon[29291]: 4.1c scrub starts 2024-09-10T18:31:57.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:57 smithi028 ceph-mon[29291]: 4.1c scrub ok 2024-09-10T18:31:57.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:57 smithi028 ceph-mon[29291]: 1.1 scrub starts 2024-09-10T18:31:57.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:57 smithi028 ceph-mon[29291]: 1.1 scrub ok 2024-09-10T18:31:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:57 smithi204 ceph-mon[32357]: 5.e scrub starts 2024-09-10T18:31:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:57 smithi204 ceph-mon[32357]: 5.e scrub ok 2024-09-10T18:31:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:57 smithi204 ceph-mon[32357]: 4.1c scrub starts 2024-09-10T18:31:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:57 smithi204 ceph-mon[32357]: 4.1c scrub ok 2024-09-10T18:31:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:57 smithi204 ceph-mon[32357]: 1.1 scrub starts 2024-09-10T18:31:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:57 smithi204 ceph-mon[32357]: 1.1 scrub ok 2024-09-10T18:31:57.379 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T18:31:57.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:57.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:57.447 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:57.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:31:57.525 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:31:57.542 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:31:57.542 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:31:57.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:57.548 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:31:57.559 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:31:57.559 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:31:57.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:57.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:57.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:57.575 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:31:57.575 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:31:57.592 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:31:57.593 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:31:57.608 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:31:57.608 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:31:57.623 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:31:57.624 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:31:57.630 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:31:57.639 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:31:57.640 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:31:57.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:57.699 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:57.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:57.800 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:57.822 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:57.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:57.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:57.924 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:57.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:57.947 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:57.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:58.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:58.072 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:58.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:58.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:58.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:58.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:58.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:58.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:31:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:58.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:58.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:31:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:58 smithi204 ceph-mon[32357]: 5.12 deep-scrub starts 2024-09-10T18:31:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:58 smithi204 ceph-mon[32357]: 5.12 deep-scrub ok 2024-09-10T18:31:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:58 smithi204 ceph-mon[32357]: pgmap v466: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:31:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:58 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1434045115' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:31:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:58 smithi204 ceph-mon[32357]: 6.19 scrub starts 2024-09-10T18:31:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:58 smithi204 ceph-mon[32357]: 6.19 scrub ok 2024-09-10T18:31:58.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:58.447 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:58.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:58.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:58.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:58 smithi028 ceph-mon[39220]: 5.12 deep-scrub starts 2024-09-10T18:31:58.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:58 smithi028 ceph-mon[39220]: 5.12 deep-scrub ok 2024-09-10T18:31:58.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:58 smithi028 ceph-mon[39220]: pgmap v466: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:31:58.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:58 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1434045115' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:31:58.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:58 smithi028 ceph-mon[39220]: 6.19 scrub starts 2024-09-10T18:31:58.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:58 smithi028 ceph-mon[39220]: 6.19 scrub ok 2024-09-10T18:31:58.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:58 smithi028 ceph-mon[29291]: 5.12 deep-scrub starts 2024-09-10T18:31:58.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:58 smithi028 ceph-mon[29291]: 5.12 deep-scrub ok 2024-09-10T18:31:58.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:58 smithi028 ceph-mon[29291]: pgmap v466: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:31:58.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:58 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1434045115' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:31:58.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:58 smithi028 ceph-mon[29291]: 6.19 scrub starts 2024-09-10T18:31:58.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:58 smithi028 ceph-mon[29291]: 6.19 scrub ok 2024-09-10T18:31:58.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:58.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:58.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:58.674 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:58.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:58.697 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:58.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:58.798 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:31:58.819 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:58.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:58.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:31:58.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:31:58.943 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:58.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:31:59.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:59.067 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:59.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:59.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:59.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:59.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:59.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:59.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:59.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:59 smithi204 ceph-mon[32357]: 1.2 scrub starts 2024-09-10T18:31:59.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:59 smithi204 ceph-mon[32357]: 1.2 scrub ok 2024-09-10T18:31:59.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:59 smithi204 ceph-mon[32357]: 6.1e scrub starts 2024-09-10T18:31:59.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:59 smithi204 ceph-mon[32357]: 6.1e scrub ok 2024-09-10T18:31:59.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:59 smithi204 ceph-mon[32357]: 6.4 scrub starts 2024-09-10T18:31:59.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:59 smithi204 ceph-mon[32357]: 6.4 scrub ok 2024-09-10T18:31:59.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:59 smithi204 ceph-mon[32357]: 4.1e scrub starts 2024-09-10T18:31:59.267 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:59 smithi204 ceph-mon[32357]: 4.1e scrub ok 2024-09-10T18:31:59.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:59 smithi204 ceph-mon[32357]: 5.1b deep-scrub starts 2024-09-10T18:31:59.268 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:31:59 smithi204 ceph-mon[32357]: 5.1b deep-scrub ok 2024-09-10T18:31:59.277 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:31:59.277 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:31:59.278 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-10T18:31:59.278 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:31:59.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:31:59.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:59.316 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:59.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:31:59.417 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:59.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:59.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:59.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:59.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[29291]: 1.2 scrub starts 2024-09-10T18:31:59.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[29291]: 1.2 scrub ok 2024-09-10T18:31:59.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[29291]: 6.1e scrub starts 2024-09-10T18:31:59.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[29291]: 6.1e scrub ok 2024-09-10T18:31:59.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[29291]: 6.4 scrub starts 2024-09-10T18:31:59.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[29291]: 6.4 scrub ok 2024-09-10T18:31:59.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[29291]: 4.1e scrub starts 2024-09-10T18:31:59.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[29291]: 4.1e scrub ok 2024-09-10T18:31:59.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[29291]: 5.1b deep-scrub starts 2024-09-10T18:31:59.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[29291]: 5.1b deep-scrub ok 2024-09-10T18:31:59.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[39220]: 1.2 scrub starts 2024-09-10T18:31:59.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[39220]: 1.2 scrub ok 2024-09-10T18:31:59.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[39220]: 6.1e scrub starts 2024-09-10T18:31:59.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[39220]: 6.1e scrub ok 2024-09-10T18:31:59.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[39220]: 6.4 scrub starts 2024-09-10T18:31:59.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[39220]: 6.4 scrub ok 2024-09-10T18:31:59.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[39220]: 4.1e scrub starts 2024-09-10T18:31:59.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[39220]: 4.1e scrub ok 2024-09-10T18:31:59.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[39220]: 5.1b deep-scrub starts 2024-09-10T18:31:59.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:31:59 smithi028 ceph-mon[39220]: 5.1b deep-scrub ok 2024-09-10T18:31:59.542 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:31:59.555 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:31:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-10T18:31:59.555 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:31:59.563 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:59.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:59.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:31:59.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:59.687 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:59.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:59.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:59.789 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:31:59.811 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:31:59.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:59.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:31:59.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:31:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:31:59.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:31:59.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:31:59.987 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-10T18:32:00.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:00.059 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:00.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:00.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:00.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:00.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:00.184 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:00.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:00.234 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:32:00.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:00.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:00.308 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:00.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:00 smithi204 ceph-mon[32357]: 6.8 scrub starts 2024-09-10T18:32:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:00 smithi204 ceph-mon[32357]: 6.8 scrub ok 2024-09-10T18:32:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:00 smithi204 ceph-mon[32357]: 3.0 scrub starts 2024-09-10T18:32:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:00 smithi204 ceph-mon[32357]: 3.0 scrub ok 2024-09-10T18:32:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:00 smithi204 ceph-mon[32357]: pgmap v467: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T18:32:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:00 smithi204 ceph-mon[32357]: 1.a scrub starts 2024-09-10T18:32:00.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:00 smithi204 ceph-mon[32357]: 1.a scrub ok 2024-09-10T18:32:00.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:00 smithi204 ceph-mon[32357]: 3.8 scrub starts 2024-09-10T18:32:00.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:00 smithi204 ceph-mon[32357]: 3.8 scrub ok 2024-09-10T18:32:00.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:00.432 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:00.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:00.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:00.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[39220]: 6.8 scrub starts 2024-09-10T18:32:00.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[39220]: 6.8 scrub ok 2024-09-10T18:32:00.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[39220]: 3.0 scrub starts 2024-09-10T18:32:00.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[39220]: 3.0 scrub ok 2024-09-10T18:32:00.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[39220]: pgmap v467: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T18:32:00.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[39220]: 1.a scrub starts 2024-09-10T18:32:00.465 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[39220]: 1.a scrub ok 2024-09-10T18:32:00.465 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[39220]: 3.8 scrub starts 2024-09-10T18:32:00.465 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[39220]: 3.8 scrub ok 2024-09-10T18:32:00.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[29291]: 6.8 scrub starts 2024-09-10T18:32:00.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[29291]: 6.8 scrub ok 2024-09-10T18:32:00.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[29291]: 3.0 scrub starts 2024-09-10T18:32:00.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[29291]: 3.0 scrub ok 2024-09-10T18:32:00.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[29291]: pgmap v467: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T18:32:00.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[29291]: 1.a scrub starts 2024-09-10T18:32:00.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[29291]: 1.a scrub ok 2024-09-10T18:32:00.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[29291]: 3.8 scrub starts 2024-09-10T18:32:00.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:00 smithi028 ceph-mon[29291]: 3.8 scrub ok 2024-09-10T18:32:00.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:00.556 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:00.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:00.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:00.658 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:00.680 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:00.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:00.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:00.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:00.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:00.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:00.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:00.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:00.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:00.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:01.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:01.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:01.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:01.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:01.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:01.180 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:01.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:01.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:01 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:32:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:01 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:32:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:01 smithi028 ceph-mon[29291]: 5.1e scrub starts 2024-09-10T18:32:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:01 smithi028 ceph-mon[29291]: 5.1e scrub ok 2024-09-10T18:32:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:01 smithi028 ceph-mon[29291]: 3.1c scrub starts 2024-09-10T18:32:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:01 smithi028 ceph-mon[29291]: 3.1c scrub ok 2024-09-10T18:32:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:01 smithi028 ceph-mon[29291]: 3.1e scrub starts 2024-09-10T18:32:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:01 smithi028 ceph-mon[29291]: 3.1e scrub ok 2024-09-10T18:32:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:01 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:32:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:01 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:32:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:01 smithi028 ceph-mon[39220]: 5.1e scrub starts 2024-09-10T18:32:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:01 smithi028 ceph-mon[39220]: 5.1e scrub ok 2024-09-10T18:32:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:01 smithi028 ceph-mon[39220]: 3.1c scrub starts 2024-09-10T18:32:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:01 smithi028 ceph-mon[39220]: 3.1c scrub ok 2024-09-10T18:32:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:01 smithi028 ceph-mon[39220]: 3.1e scrub starts 2024-09-10T18:32:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:01 smithi028 ceph-mon[39220]: 3.1e scrub ok 2024-09-10T18:32:01.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:01.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:01.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:01.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:01 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:32:01.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:01 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:32:01.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:01 smithi204 ceph-mon[32357]: 5.1e scrub starts 2024-09-10T18:32:01.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:01 smithi204 ceph-mon[32357]: 5.1e scrub ok 2024-09-10T18:32:01.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:01 smithi204 ceph-mon[32357]: 3.1c scrub starts 2024-09-10T18:32:01.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:01 smithi204 ceph-mon[32357]: 3.1c scrub ok 2024-09-10T18:32:01.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:01 smithi204 ceph-mon[32357]: 3.1e scrub starts 2024-09-10T18:32:01.337 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:01 smithi204 ceph-mon[32357]: 3.1e scrub ok 2024-09-10T18:32:01.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:01.430 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:01.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:01.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:01.532 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:01.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:01.554 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:01.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:01.621 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:32:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:32:01] "GET /metrics HTTP/1.1" 200 221919 "" "Prometheus/2.43.0" 2024-09-10T18:32:01.655 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:01.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:01.679 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:01.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:01.780 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:01.801 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:01.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:01.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:01.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:01.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:01.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:01.998 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:32:01.998 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:32:01.998 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-10T18:32:01.999 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-10T18:32:01.999 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:32:02.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:02.049 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:02.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:02.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:02.151 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:02.172 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:02.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:02.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:02.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:02.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:02.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:02.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:02.320 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:32:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:32:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:32:02.373 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:32:02.375 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_2' created 2024-09-10T18:32:02.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:02.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:02.421 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:02.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:02.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:32:02.522 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:02.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:02.544 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:02.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:02 smithi204 ceph-mon[32357]: 6.5 scrub starts 2024-09-10T18:32:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:02 smithi204 ceph-mon[32357]: 6.5 scrub ok 2024-09-10T18:32:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:02 smithi204 ceph-mon[32357]: 3.14 scrub starts 2024-09-10T18:32:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:02 smithi204 ceph-mon[32357]: 3.14 scrub ok 2024-09-10T18:32:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:02 smithi204 ceph-mon[32357]: pgmap v468: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:32:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:02 smithi204 ceph-mon[32357]: osdmap e245: 8 total, 8 up, 8 in 2024-09-10T18:32:02.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:02 smithi204 ceph-mon[32357]: 3.16 scrub starts 2024-09-10T18:32:02.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:02 smithi204 ceph-mon[32357]: 3.16 scrub ok 2024-09-10T18:32:02.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:02 smithi028 ceph-mon[29291]: 6.5 scrub starts 2024-09-10T18:32:02.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:02 smithi028 ceph-mon[29291]: 6.5 scrub ok 2024-09-10T18:32:02.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:02 smithi028 ceph-mon[29291]: 3.14 scrub starts 2024-09-10T18:32:02.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:02 smithi028 ceph-mon[29291]: 3.14 scrub ok 2024-09-10T18:32:02.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:02 smithi028 ceph-mon[29291]: pgmap v468: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:32:02.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:02 smithi028 ceph-mon[29291]: osdmap e245: 8 total, 8 up, 8 in 2024-09-10T18:32:02.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:02 smithi028 ceph-mon[29291]: 3.16 scrub starts 2024-09-10T18:32:02.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:02 smithi028 ceph-mon[29291]: 3.16 scrub ok 2024-09-10T18:32:02.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:02 smithi028 ceph-mon[39220]: 6.5 scrub starts 2024-09-10T18:32:02.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:02 smithi028 ceph-mon[39220]: 6.5 scrub ok 2024-09-10T18:32:02.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:02 smithi028 ceph-mon[39220]: 3.14 scrub starts 2024-09-10T18:32:02.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:02 smithi028 ceph-mon[39220]: 3.14 scrub ok 2024-09-10T18:32:02.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:02 smithi028 ceph-mon[39220]: pgmap v468: 142 pgs: 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:32:02.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:02 smithi028 ceph-mon[39220]: osdmap e245: 8 total, 8 up, 8 in 2024-09-10T18:32:02.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:02 smithi028 ceph-mon[39220]: 3.16 scrub starts 2024-09-10T18:32:02.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:02 smithi028 ceph-mon[39220]: 3.16 scrub ok 2024-09-10T18:32:02.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:02.666 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:02.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:02.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:02.693 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-10T18:32:02.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:02.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:02.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:02.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:02.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:02.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:02.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:02.978 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:32:03.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:03.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:03.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:03.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:03.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:03.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:03.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:03.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:03.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:03.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:03.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:03 smithi204 ceph-mon[32357]: 5.1a scrub starts 2024-09-10T18:32:03.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:03 smithi204 ceph-mon[32357]: 5.1a scrub ok 2024-09-10T18:32:03.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:03 smithi204 ceph-mon[32357]: 6.1 scrub starts 2024-09-10T18:32:03.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:03 smithi204 ceph-mon[32357]: 6.1 scrub ok 2024-09-10T18:32:03.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:03 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1982219740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-10T18:32:03.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:03 smithi204 ceph-mon[32357]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-10T18:32:03.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:03 smithi204 ceph-mon[32357]: 6.10 scrub starts 2024-09-10T18:32:03.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:03 smithi204 ceph-mon[32357]: 6.10 scrub ok 2024-09-10T18:32:03.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:03 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1574850182' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:32:03.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:03 smithi204 ceph-mon[32357]: from='client.35285 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:32:03.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:03 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:32:03.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:03 smithi204 ceph-mon[32357]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-10T18:32:03.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:03 smithi204 ceph-mon[32357]: from='client.35285 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:32:03.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:03 smithi204 ceph-mon[32357]: osdmap e246: 8 total, 8 up, 8 in 2024-09-10T18:32:03.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:03 smithi204 ceph-mon[32357]: pgmap v471: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:32:03.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:03.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:03.417 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:03.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:03.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-10T18:32:03.518 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:03.540 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:03.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:03.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:03.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[29291]: 5.1a scrub starts 2024-09-10T18:32:03.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[29291]: 5.1a scrub ok 2024-09-10T18:32:03.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[29291]: 6.1 scrub starts 2024-09-10T18:32:03.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[29291]: 6.1 scrub ok 2024-09-10T18:32:03.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1982219740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-10T18:32:03.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[29291]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-10T18:32:03.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[29291]: 6.10 scrub starts 2024-09-10T18:32:03.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[29291]: 6.10 scrub ok 2024-09-10T18:32:03.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1574850182' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:32:03.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[29291]: from='client.35285 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:32:03.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:32:03.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[29291]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-10T18:32:03.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[29291]: from='client.35285 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:32:03.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[29291]: osdmap e246: 8 total, 8 up, 8 in 2024-09-10T18:32:03.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[29291]: pgmap v471: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:32:03.625 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[39220]: 5.1a scrub starts 2024-09-10T18:32:03.625 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[39220]: 5.1a scrub ok 2024-09-10T18:32:03.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[39220]: 6.1 scrub starts 2024-09-10T18:32:03.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[39220]: 6.1 scrub ok 2024-09-10T18:32:03.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1982219740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-10T18:32:03.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[39220]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-10T18:32:03.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[39220]: 6.10 scrub starts 2024-09-10T18:32:03.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[39220]: 6.10 scrub ok 2024-09-10T18:32:03.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1574850182' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:32:03.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[39220]: from='client.35285 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:32:03.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:32:03.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[39220]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-10T18:32:03.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[39220]: from='client.35285 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:32:03.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[39220]: osdmap e246: 8 total, 8 up, 8 in 2024-09-10T18:32:03.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:03 smithi028 ceph-mon[39220]: pgmap v471: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:32:03.642 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:03.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:03.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:03.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:03.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:03.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:03.789 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:03.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:03.890 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:03.911 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:03.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:03.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:04.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:04.040 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:04.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:04.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:04.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:04.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:04.166 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:04.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:04.267 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:04.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:04.291 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:04.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:04.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:04.412 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:04 smithi204 ceph-mon[32357]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:32:04.412 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:04 smithi204 ceph-mon[32357]: osdmap e247: 8 total, 8 up, 8 in 2024-09-10T18:32:04.415 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:04.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:04.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:04.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:04.540 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:04.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:04.642 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:04.662 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:32:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:32:04.664 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:32:04.664 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-10T18:32:04.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-10T18:32:04.664 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:32:04.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.664 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:182560 7.7 7.fd896b7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e144)", 2024-09-10T18:32:04.665 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:26.644496+0000", 2024-09-10T18:32:04.665 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 338.01372580600002, 2024-09-10T18:32:04.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99046780599999995, 2024-09-10T18:32:04.665 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.665 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:32:04.665 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.665 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.666 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.666 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 182560 2024-09-10T18:32:04.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.666 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644496+0000", 2024-09-10T18:32:04.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644496+0000", 2024-09-10T18:32:04.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644494+0000", 2024-09-10T18:32:04.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:04.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644501+0000", 2024-09-10T18:32:04.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5039999999999999e-06 2024-09-10T18:32:04.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644510+0000", 2024-09-10T18:32:04.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2690000000000005e-06 2024-09-10T18:32:04.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644517+0000", 2024-09-10T18:32:04.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7240000000000003e-06 2024-09-10T18:32:04.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.647399+0000", 2024-09-10T18:32:04.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002882169 2024-09-10T18:32:04.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.647407+0000", 2024-09-10T18:32:04.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9710000000000004e-06 2024-09-10T18:32:04.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.634954+0000", 2024-09-10T18:32:04.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98754652799999998 2024-09-10T18:32:04.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.634964+0000", 2024-09-10T18:32:04.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0642000000000001e-05 2024-09-10T18:32:04.673 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.673 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.673 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.674 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:182566 7.7 7.1618317 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e144)", 2024-09-10T18:32:04.674 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:26.644910+0000", 2024-09-10T18:32:04.674 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 338.013312644, 2024-09-10T18:32:04.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99008567599999997, 2024-09-10T18:32:04.674 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.674 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:32:04.674 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.674 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.674 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.674 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 182566 2024-09-10T18:32:04.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.675 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644910+0000", 2024-09-10T18:32:04.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644910+0000", 2024-09-10T18:32:04.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644909+0000", 2024-09-10T18:32:04.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:32:04.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644914+0000", 2024-09-10T18:32:04.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3619999999999997e-06 2024-09-10T18:32:04.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644924+0000", 2024-09-10T18:32:04.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7000000000000003e-06 2024-09-10T18:32:04.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644929+0000", 2024-09-10T18:32:04.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.784e-06 2024-09-10T18:32:04.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.647429+0000", 2024-09-10T18:32:04.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024991610000000002 2024-09-10T18:32:04.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.647434+0000", 2024-09-10T18:32:04.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9290000000000003e-06 2024-09-10T18:32:04.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.634986+0000", 2024-09-10T18:32:04.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98755193299999999 2024-09-10T18:32:04.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.634995+0000", 2024-09-10T18:32:04.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8270000000000004e-06 2024-09-10T18:32:04.678 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.678 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.678 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.679 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:182570 7.c 7.cf5112fc (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e144)", 2024-09-10T18:32:04.679 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:26.644934+0000", 2024-09-10T18:32:04.679 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 338.013288257, 2024-09-10T18:32:04.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.981210578, 2024-09-10T18:32:04.679 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.679 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:32:04.679 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.679 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.679 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.679 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 182570 2024-09-10T18:32:04.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.679 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644934+0000", 2024-09-10T18:32:04.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644934+0000", 2024-09-10T18:32:04.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644933+0000", 2024-09-10T18:32:04.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999995 2024-09-10T18:32:04.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644936+0000", 2024-09-10T18:32:04.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9890000000000001e-06 2024-09-10T18:32:04.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644942+0000", 2024-09-10T18:32:04.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5079999999999996e-06 2024-09-10T18:32:04.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.644946+0000", 2024-09-10T18:32:04.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2270000000000004e-06 2024-09-10T18:32:04.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.648100+0000", 2024-09-10T18:32:04.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0031542720000000001 2024-09-10T18:32:04.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.648106+0000", 2024-09-10T18:32:04.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0569999999999998e-06 2024-09-10T18:32:04.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.626135+0000", 2024-09-10T18:32:04.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97802908099999997 2024-09-10T18:32:04.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.626145+0000", 2024-09-10T18:32:04.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1390000000000001e-06 2024-09-10T18:32:04.684 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.684 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.684 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.684 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:182576 7.d 7.7a34432d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-10T18:32:04.684 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:26.646578+0000", 2024-09-10T18:32:04.684 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 338.01164389500002, 2024-09-10T18:32:04.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97905169700000005, 2024-09-10T18:32:04.684 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.684 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:32:04.685 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.685 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.685 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.685 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 182576 2024-09-10T18:32:04.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.685 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.646578+0000", 2024-09-10T18:32:04.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.646578+0000", 2024-09-10T18:32:04.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.646577+0000", 2024-09-10T18:32:04.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:32:04.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.646583+0000", 2024-09-10T18:32:04.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6510000000000004e-06 2024-09-10T18:32:04.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.646593+0000", 2024-09-10T18:32:04.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5589999999999994e-06 2024-09-10T18:32:04.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.646599+0000", 2024-09-10T18:32:04.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2380000000000002e-06 2024-09-10T18:32:04.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.646939+0000", 2024-09-10T18:32:04.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00034010600000000001 2024-09-10T18:32:04.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.646948+0000", 2024-09-10T18:32:04.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7020000000000007e-06 2024-09-10T18:32:04.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.625617+0000", 2024-09-10T18:32:04.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.978669448 2024-09-10T18:32:04.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.625630+0000", 2024-09-10T18:32:04.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2635e-05 2024-09-10T18:32:04.689 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.689 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.689 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.689 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:182579 7.c 7.428e990c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-10T18:32:04.689 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:26.649232+0000", 2024-09-10T18:32:04.689 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 338.00898977399999, 2024-09-10T18:32:04.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97693863700000005, 2024-09-10T18:32:04.689 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.689 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:32:04.689 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.689 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.689 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.690 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 182579 2024-09-10T18:32:04.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.690 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.649232+0000", 2024-09-10T18:32:04.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.649232+0000", 2024-09-10T18:32:04.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.649231+0000", 2024-09-10T18:32:04.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:04.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.649238+0000", 2024-09-10T18:32:04.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8900000000000001e-06 2024-09-10T18:32:04.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.649246+0000", 2024-09-10T18:32:04.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7700000000000007e-06 2024-09-10T18:32:04.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.649251+0000", 2024-09-10T18:32:04.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5639999999999998e-06 2024-09-10T18:32:04.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.649291+0000", 2024-09-10T18:32:04.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9996999999999997e-05 2024-09-10T18:32:04.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.693 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.649296+0000", 2024-09-10T18:32:04.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2800000000000003e-06 2024-09-10T18:32:04.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.626164+0000", 2024-09-10T18:32:04.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97686759400000001 2024-09-10T18:32:04.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:27.626171+0000", 2024-09-10T18:32:04.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3509999999999996e-06 2024-09-10T18:32:04.694 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.694 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.694 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.695 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:200476 7.d 7:b713bc26:::benchmark_data_smithi028_77856_object79376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-10T18:32:04.695 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:46.958115+0000", 2024-09-10T18:32:04.695 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 317.700107624, 2024-09-10T18:32:04.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.060615461, 2024-09-10T18:32:04.695 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.695 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:04.695 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.695 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.695 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.695 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 200476 2024-09-10T18:32:04.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.695 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.695 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.958115+0000", 2024-09-10T18:32:04.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.958115+0000", 2024-09-10T18:32:04.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.958113+0000", 2024-09-10T18:32:04.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:04.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.958121+0000", 2024-09-10T18:32:04.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9330000000000007e-06 2024-09-10T18:32:04.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.958134+0000", 2024-09-10T18:32:04.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2266000000000001e-05 2024-09-10T18:32:04.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.958139+0000", 2024-09-10T18:32:04.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7359999999999998e-06 2024-09-10T18:32:04.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.958190+0000", 2024-09-10T18:32:04.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0309999999999998e-05 2024-09-10T18:32:04.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.958199+0000", 2024-09-10T18:32:04.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8380000000000008e-06 2024-09-10T18:32:04.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.000643+0000", 2024-09-10T18:32:04.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0424444900000001 2024-09-10T18:32:04.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:32:04.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.000765+0000", 2024-09-10T18:32:04.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012213099999999999 2024-09-10T18:32:04.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.700 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1", 2024-09-10T18:32:04.700 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.000893+0000", 2024-09-10T18:32:04.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000127866 2024-09-10T18:32:04.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.700 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:32:04.700 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.005746+0000", 2024-09-10T18:32:04.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0048532239999999997 2024-09-10T18:32:04.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.700 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.700 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.018618+0000", 2024-09-10T18:32:04.701 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012871426 2024-09-10T18:32:04.701 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.701 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.701 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.701 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.018673+0000", 2024-09-10T18:32:04.701 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4979000000000003e-05 2024-09-10T18:32:04.701 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.701 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.701 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:32:04.701 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.018708+0000", 2024-09-10T18:32:04.701 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5252000000000001e-05 2024-09-10T18:32:04.701 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.018730+0000", 2024-09-10T18:32:04.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2079999999999999e-05 2024-09-10T18:32:04.702 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.702 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.702 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.702 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:200482 7.d 7:be3c80eb:::benchmark_data_smithi028_77856_object79382:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-10T18:32:04.702 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:46.959884+0000", 2024-09-10T18:32:04.702 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 317.69833785700001, 2024-09-10T18:32:04.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0592035399999999, 2024-09-10T18:32:04.703 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.703 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:04.703 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.703 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.703 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.703 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 200482 2024-09-10T18:32:04.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.703 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.959884+0000", 2024-09-10T18:32:04.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.959884+0000", 2024-09-10T18:32:04.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.705 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.959883+0000", 2024-09-10T18:32:04.705 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:32:04.705 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.705 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.705 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.705 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.959891+0000", 2024-09-10T18:32:04.706 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.627e-06 2024-09-10T18:32:04.706 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.706 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.706 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.706 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.959900+0000", 2024-09-10T18:32:04.706 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2890000000000003e-06 2024-09-10T18:32:04.707 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.707 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.707 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.707 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.959912+0000", 2024-09-10T18:32:04.707 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1941e-05 2024-09-10T18:32:04.707 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.707 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.708 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.708 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.959952+0000", 2024-09-10T18:32:04.708 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9854999999999997e-05 2024-09-10T18:32:04.708 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.708 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.708 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.708 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:46.959957+0000", 2024-09-10T18:32:04.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8949999999999997e-06 2024-09-10T18:32:04.709 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.709 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.709 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.709 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.002278+0000", 2024-09-10T18:32:04.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.042321533 2024-09-10T18:32:04.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.710 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.710 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:32:04.710 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.002379+0000", 2024-09-10T18:32:04.710 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000100484 2024-09-10T18:32:04.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.710 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.711 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1", 2024-09-10T18:32:04.711 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.002464+0000", 2024-09-10T18:32:04.711 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5884000000000006e-05 2024-09-10T18:32:04.711 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.711 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.711 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:32:04.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.005772+0000", 2024-09-10T18:32:04.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033071089999999999 2024-09-10T18:32:04.712 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.712 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.712 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.018770+0000", 2024-09-10T18:32:04.713 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012998699000000001 2024-09-10T18:32:04.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.713 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.713 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019048+0000", 2024-09-10T18:32:04.713 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027806599999999999 2024-09-10T18:32:04.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.714 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:32:04.714 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019070+0000", 2024-09-10T18:32:04.714 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1549e-05 2024-09-10T18:32:04.714 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.714 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.714 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.714 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019088+0000", 2024-09-10T18:32:04.714 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8063e-05 2024-09-10T18:32:04.715 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.715 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.715 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.715 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.715 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.715 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:200528 7.d 7:b7461a54:::benchmark_data_smithi028_77856_object79428:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-10T18:32:04.715 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:47.007764+0000", 2024-09-10T18:32:04.716 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 317.650458335, 2024-09-10T18:32:04.716 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.011397653, 2024-09-10T18:32:04.716 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.716 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:04.716 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.716 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.716 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.717 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 200528 2024-09-10T18:32:04.717 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.717 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.717 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.717 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.717 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.007764+0000", 2024-09-10T18:32:04.718 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.718 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.718 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.718 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.718 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.007764+0000", 2024-09-10T18:32:04.718 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.719 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.719 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.719 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.719 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.007761+0000", 2024-09-10T18:32:04.719 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:32:04.720 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.720 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.720 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.720 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.007769+0000", 2024-09-10T18:32:04.720 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7309999999999993e-06 2024-09-10T18:32:04.720 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.721 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.721 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.721 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.007780+0000", 2024-09-10T18:32:04.721 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0706999999999999e-05 2024-09-10T18:32:04.721 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.721 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.722 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.722 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.007785+0000", 2024-09-10T18:32:04.722 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6439999999999999e-06 2024-09-10T18:32:04.722 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.722 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.722 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.722 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.007826+0000", 2024-09-10T18:32:04.722 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1681000000000003e-05 2024-09-10T18:32:04.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.723 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.723 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.723 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.007836+0000", 2024-09-10T18:32:04.723 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5959999999999993e-06 2024-09-10T18:32:04.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.723 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.723 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:04.725 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:04.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:04.726 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.003462+0000", 2024-09-10T18:32:04.727 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99562593399999999 2024-09-10T18:32:04.727 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.727 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.727 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:32:04.727 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.003563+0000", 2024-09-10T18:32:04.727 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101389 2024-09-10T18:32:04.727 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.727 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.727 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1", 2024-09-10T18:32:04.727 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.003670+0000", 2024-09-10T18:32:04.727 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106879 2024-09-10T18:32:04.727 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.727 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.728 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:32:04.728 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.007949+0000", 2024-09-10T18:32:04.728 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042792109999999998 2024-09-10T18:32:04.728 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.728 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.728 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.728 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.018810+0000", 2024-09-10T18:32:04.728 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010860787 2024-09-10T18:32:04.728 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.728 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.728 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.728 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019115+0000", 2024-09-10T18:32:04.728 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00030468599999999998 2024-09-10T18:32:04.728 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.729 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.729 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:32:04.729 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019141+0000", 2024-09-10T18:32:04.729 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6704999999999999e-05 2024-09-10T18:32:04.729 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.729 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.729 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.729 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019162+0000", 2024-09-10T18:32:04.729 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0136000000000002e-05 2024-09-10T18:32:04.729 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.729 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.729 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.729 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.729 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.730 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:200537 7.d 7:be2a904c:::benchmark_data_smithi028_77856_object79437:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-10T18:32:04.730 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:47.014154+0000", 2024-09-10T18:32:04.730 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 317.64406837199999, 2024-09-10T18:32:04.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0050720369999999, 2024-09-10T18:32:04.730 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.730 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:04.730 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.730 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.730 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.730 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 200537 2024-09-10T18:32:04.730 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.730 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.730 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.731 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.014154+0000", 2024-09-10T18:32:04.731 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.731 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.731 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.014154+0000", 2024-09-10T18:32:04.731 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.731 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.731 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.014152+0000", 2024-09-10T18:32:04.731 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:04.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.732 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.732 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.014160+0000", 2024-09-10T18:32:04.732 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3869999999999995e-06 2024-09-10T18:32:04.732 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.732 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.732 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.732 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.014177+0000", 2024-09-10T18:32:04.732 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6708999999999999e-05 2024-09-10T18:32:04.732 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.732 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.732 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.732 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.014183+0000", 2024-09-10T18:32:04.732 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0430000000000004e-06 2024-09-10T18:32:04.732 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.733 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.733 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.733 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.014227+0000", 2024-09-10T18:32:04.733 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4503999999999999e-05 2024-09-10T18:32:04.733 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.733 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.733 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.733 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.014235+0000", 2024-09-10T18:32:04.733 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4560000000000003e-06 2024-09-10T18:32:04.733 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.733 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.733 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.733 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.004599+0000", 2024-09-10T18:32:04.733 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99036454799999996 2024-09-10T18:32:04.734 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.734 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:32:04.734 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.004697+0000", 2024-09-10T18:32:04.734 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7565999999999994e-05 2024-09-10T18:32:04.734 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.734 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1", 2024-09-10T18:32:04.734 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.004805+0000", 2024-09-10T18:32:04.734 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010770200000000001 2024-09-10T18:32:04.734 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.734 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:32:04.734 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.010179+0000", 2024-09-10T18:32:04.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0053740020000000001 2024-09-10T18:32:04.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.018845+0000", 2024-09-10T18:32:04.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086661759999999994 2024-09-10T18:32:04.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019188+0000", 2024-09-10T18:32:04.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00034277299999999999 2024-09-10T18:32:04.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:32:04.736 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019206+0000", 2024-09-10T18:32:04.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8780999999999999e-05 2024-09-10T18:32:04.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.736 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.736 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019226+0000", 2024-09-10T18:32:04.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9514999999999999e-05 2024-09-10T18:32:04.736 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.736 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.736 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.736 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:200541 7.d 7:b0ae68b6:::benchmark_data_smithi028_77856_object79441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-10T18:32:04.737 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:47.016825+0000", 2024-09-10T18:32:04.737 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 317.64139745599999, 2024-09-10T18:32:04.737 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.002464555, 2024-09-10T18:32:04.737 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.737 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:04.737 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.737 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.737 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.737 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 200541 2024-09-10T18:32:04.737 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.737 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.737 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.737 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.737 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.016825+0000", 2024-09-10T18:32:04.738 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.738 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.738 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.738 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.016825+0000", 2024-09-10T18:32:04.738 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.738 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.738 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.738 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.016823+0000", 2024-09-10T18:32:04.738 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:32:04.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.738 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.738 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.739 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.016829+0000", 2024-09-10T18:32:04.739 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1e-06 2024-09-10T18:32:04.739 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.739 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.739 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.739 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.016840+0000", 2024-09-10T18:32:04.739 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0421000000000001e-05 2024-09-10T18:32:04.739 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.739 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.739 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.739 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.016846+0000", 2024-09-10T18:32:04.739 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8259999999999998e-06 2024-09-10T18:32:04.739 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.739 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.740 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.740 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.016893+0000", 2024-09-10T18:32:04.740 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7369e-05 2024-09-10T18:32:04.740 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.740 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.740 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.740 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:47.016900+0000", 2024-09-10T18:32:04.740 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7309999999999996e-06 2024-09-10T18:32:04.740 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.740 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.740 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.740 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.005806+0000", 2024-09-10T18:32:04.740 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.988906532 2024-09-10T18:32:04.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:32:04.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.005883+0000", 2024-09-10T18:32:04.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7152999999999998e-05 2024-09-10T18:32:04.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1", 2024-09-10T18:32:04.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.005949+0000", 2024-09-10T18:32:04.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5680999999999997e-05 2024-09-10T18:32:04.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:32:04.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.010200+0000", 2024-09-10T18:32:04.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042509679999999999 2024-09-10T18:32:04.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.018860+0000", 2024-09-10T18:32:04.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086603429999999992 2024-09-10T18:32:04.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019251+0000", 2024-09-10T18:32:04.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00039090499999999999 2024-09-10T18:32:04.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.743 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:32:04.743 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019272+0000", 2024-09-10T18:32:04.743 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0701000000000001e-05 2024-09-10T18:32:04.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.743 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.743 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:48.019289+0000", 2024-09-10T18:32:04.743 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7212000000000001e-05 2024-09-10T18:32:04.743 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.743 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.743 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.743 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221732 7.c 7:3f916f55:::benchmark_data_smithi028_77856_object100632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:32:04.744 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.154233+0000", 2024-09-10T18:32:04.744 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 295.50398924500001, 2024-09-10T18:32:04.744 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.005077531, 2024-09-10T18:32:04.744 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.744 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:04.744 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.744 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.744 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.744 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221732 2024-09-10T18:32:04.744 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.744 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.744 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.744 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.744 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.154233+0000", 2024-09-10T18:32:04.744 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.745 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.745 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.745 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.745 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.154233+0000", 2024-09-10T18:32:04.745 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.745 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.745 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.745 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.745 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.154232+0000", 2024-09-10T18:32:04.745 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:32:04.745 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.745 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.745 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.745 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.154237+0000", 2024-09-10T18:32:04.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7870000000000001e-06 2024-09-10T18:32:04.746 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.746 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.154244+0000", 2024-09-10T18:32:04.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6539999999999999e-06 2024-09-10T18:32:04.746 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.746 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.154248+0000", 2024-09-10T18:32:04.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5260000000000002e-06 2024-09-10T18:32:04.746 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.746 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.154283+0000", 2024-09-10T18:32:04.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4727999999999997e-05 2024-09-10T18:32:04.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.747 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.747 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.154288+0000", 2024-09-10T18:32:04.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8109999999999999e-06 2024-09-10T18:32:04.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.747 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.747 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.143725+0000", 2024-09-10T18:32:04.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98943635100000005 2024-09-10T18:32:04.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.747 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.747 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:32:04.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.143866+0000", 2024-09-10T18:32:04.748 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000141015 2024-09-10T18:32:04.748 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.748 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:32:04.748 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.144013+0000", 2024-09-10T18:32:04.748 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000147009 2024-09-10T18:32:04.748 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.748 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:32:04.748 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.151226+0000", 2024-09-10T18:32:04.748 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0072131060000000004 2024-09-10T18:32:04.748 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.748 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.748 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159157+0000", 2024-09-10T18:32:04.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0079311820000000002 2024-09-10T18:32:04.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159208+0000", 2024-09-10T18:32:04.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1016000000000001e-05 2024-09-10T18:32:04.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:32:04.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159277+0000", 2024-09-10T18:32:04.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9000999999999999e-05 2024-09-10T18:32:04.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159310+0000", 2024-09-10T18:32:04.750 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3488999999999997e-05 2024-09-10T18:32:04.750 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.750 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.750 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.750 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.750 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.750 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221744 7.7 7:ee76f855:::benchmark_data_smithi028_77856_object100644:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:32:04.750 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.164178+0000", 2024-09-10T18:32:04.750 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 295.49404401700002, 2024-09-10T18:32:04.750 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99366592799999998, 2024-09-10T18:32:04.750 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.750 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:04.750 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.750 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.751 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.751 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221744 2024-09-10T18:32:04.751 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.751 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.751 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.751 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.751 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.164178+0000", 2024-09-10T18:32:04.751 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.751 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.751 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.751 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.751 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.164178+0000", 2024-09-10T18:32:04.751 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.751 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.752 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.752 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.752 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.164176+0000", 2024-09-10T18:32:04.752 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:04.752 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.752 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.752 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.752 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.164183+0000", 2024-09-10T18:32:04.752 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3169999999999998e-06 2024-09-10T18:32:04.752 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.752 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.752 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.752 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.164193+0000", 2024-09-10T18:32:04.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0107e-05 2024-09-10T18:32:04.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.753 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.753 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.164197+0000", 2024-09-10T18:32:04.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4649999999999996e-06 2024-09-10T18:32:04.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.753 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.753 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.164233+0000", 2024-09-10T18:32:04.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5547999999999999e-05 2024-09-10T18:32:04.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.753 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.754 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.164238+0000", 2024-09-10T18:32:04.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4299999999999997e-06 2024-09-10T18:32:04.754 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.754 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.151176+0000", 2024-09-10T18:32:04.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98693812199999997 2024-09-10T18:32:04.754 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:32:04.754 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.151273+0000", 2024-09-10T18:32:04.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6928000000000004e-05 2024-09-10T18:32:04.754 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-09-10T18:32:04.755 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.151341+0000", 2024-09-10T18:32:04.755 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7798000000000004e-05 2024-09-10T18:32:04.755 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.755 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.755 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:32:04.755 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.156298+0000", 2024-09-10T18:32:04.755 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0049565299999999998 2024-09-10T18:32:04.755 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.755 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.755 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.755 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.157700+0000", 2024-09-10T18:32:04.755 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014025660000000001 2024-09-10T18:32:04.755 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.755 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.756 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.157759+0000", 2024-09-10T18:32:04.756 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9197999999999998e-05 2024-09-10T18:32:04.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:32:04.756 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.157804+0000", 2024-09-10T18:32:04.756 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4696e-05 2024-09-10T18:32:04.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.756 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.157844+0000", 2024-09-10T18:32:04.756 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9960999999999999e-05 2024-09-10T18:32:04.757 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.757 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.757 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.757 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.757 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.757 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221749 7.c 7:35f4dcda:::benchmark_data_smithi028_77856_object100649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:32:04.757 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.167805+0000", 2024-09-10T18:32:04.757 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 295.49041745800002, 2024-09-10T18:32:04.757 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99166929100000001, 2024-09-10T18:32:04.757 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.757 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:04.757 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.757 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.757 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.758 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221749 2024-09-10T18:32:04.758 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.758 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.758 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.758 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.758 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.167805+0000", 2024-09-10T18:32:04.758 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.758 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.758 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.758 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.758 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.167805+0000", 2024-09-10T18:32:04.758 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.758 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.758 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.759 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.759 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.167803+0000", 2024-09-10T18:32:04.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:32:04.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.759 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.759 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.759 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.167809+0000", 2024-09-10T18:32:04.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7610000000000004e-06 2024-09-10T18:32:04.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.759 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.759 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.759 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.167821+0000", 2024-09-10T18:32:04.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1977000000000001e-05 2024-09-10T18:32:04.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.760 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.760 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.167827+0000", 2024-09-10T18:32:04.760 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9819999999999996e-06 2024-09-10T18:32:04.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.760 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.760 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.167878+0000", 2024-09-10T18:32:04.760 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0801999999999999e-05 2024-09-10T18:32:04.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.760 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.760 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.167885+0000", 2024-09-10T18:32:04.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.396e-06 2024-09-10T18:32:04.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.761 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.145946+0000", 2024-09-10T18:32:04.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97806059300000003 2024-09-10T18:32:04.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:32:04.761 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.146048+0000", 2024-09-10T18:32:04.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101739 2024-09-10T18:32:04.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:32:04.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.146117+0000", 2024-09-10T18:32:04.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9497000000000002e-05 2024-09-10T18:32:04.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:32:04.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.151232+0000", 2024-09-10T18:32:04.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0051152749999999999 2024-09-10T18:32:04.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159361+0000", 2024-09-10T18:32:04.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0081289810000000004 2024-09-10T18:32:04.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159403+0000", 2024-09-10T18:32:04.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1100999999999998e-05 2024-09-10T18:32:04.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:32:04.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159442+0000", 2024-09-10T18:32:04.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9174000000000001e-05 2024-09-10T18:32:04.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159474+0000", 2024-09-10T18:32:04.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2327999999999999e-05 2024-09-10T18:32:04.763 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.764 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.764 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.764 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221753 7.c 7:3c815616:::benchmark_data_smithi028_77856_object100653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:32:04.764 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.172324+0000", 2024-09-10T18:32:04.764 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 295.485897873, 2024-09-10T18:32:04.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98727850699999997, 2024-09-10T18:32:04.764 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.764 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:04.764 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.764 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.764 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.764 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221753 2024-09-10T18:32:04.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.765 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.765 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.765 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.172324+0000", 2024-09-10T18:32:04.765 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.765 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.765 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.172324+0000", 2024-09-10T18:32:04.765 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.765 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.172323+0000", 2024-09-10T18:32:04.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:32:04.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.172328+0000", 2024-09-10T18:32:04.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4770000000000002e-06 2024-09-10T18:32:04.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.172338+0000", 2024-09-10T18:32:04.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7089999999999997e-06 2024-09-10T18:32:04.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.172343+0000", 2024-09-10T18:32:04.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0100000000000003e-06 2024-09-10T18:32:04.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.172388+0000", 2024-09-10T18:32:04.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4990000000000001e-05 2024-09-10T18:32:04.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.172395+0000", 2024-09-10T18:32:04.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.934e-06 2024-09-10T18:32:04.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.147521+0000", 2024-09-10T18:32:04.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97512613999999997 2024-09-10T18:32:04.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:32:04.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.147632+0000", 2024-09-10T18:32:04.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000110757 2024-09-10T18:32:04.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:32:04.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.147758+0000", 2024-09-10T18:32:04.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125702 2024-09-10T18:32:04.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:32:04.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.153601+0000", 2024-09-10T18:32:04.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058435529999999996 2024-09-10T18:32:04.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159501+0000", 2024-09-10T18:32:04.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058994069999999997 2024-09-10T18:32:04.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159545+0000", 2024-09-10T18:32:04.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3912999999999998e-05 2024-09-10T18:32:04.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.770 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:32:04.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159571+0000", 2024-09-10T18:32:04.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6795e-05 2024-09-10T18:32:04.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.770 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159603+0000", 2024-09-10T18:32:04.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1520999999999999e-05 2024-09-10T18:32:04.770 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.770 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.771 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.771 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.771 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.771 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221754 7.c 7:337bf24c:::benchmark_data_smithi028_77856_object100654:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:32:04.771 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.173248+0000", 2024-09-10T18:32:04.771 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 295.48497383, 2024-09-10T18:32:04.771 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98651236499999995, 2024-09-10T18:32:04.771 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.771 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:04.771 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.771 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.771 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.771 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221754 2024-09-10T18:32:04.771 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.772 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.173248+0000", 2024-09-10T18:32:04.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.772 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.173248+0000", 2024-09-10T18:32:04.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.772 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.173247+0000", 2024-09-10T18:32:04.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:32:04.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.173252+0000", 2024-09-10T18:32:04.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6199999999999998e-06 2024-09-10T18:32:04.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.173261+0000", 2024-09-10T18:32:04.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6640000000000004e-06 2024-09-10T18:32:04.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.173265+0000", 2024-09-10T18:32:04.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9509999999999999e-06 2024-09-10T18:32:04.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.173301+0000", 2024-09-10T18:32:04.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6844999999999997e-05 2024-09-10T18:32:04.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.173306+0000", 2024-09-10T18:32:04.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6530000000000002e-06 2024-09-10T18:32:04.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.148983+0000", 2024-09-10T18:32:04.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97567699399999996 2024-09-10T18:32:04.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:32:04.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.149052+0000", 2024-09-10T18:32:04.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9177999999999994e-05 2024-09-10T18:32:04.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:32:04.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.149119+0000", 2024-09-10T18:32:04.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7175999999999995e-05 2024-09-10T18:32:04.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:32:04.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.153606+0000", 2024-09-10T18:32:04.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0044868299999999998 2024-09-10T18:32:04.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159644+0000", 2024-09-10T18:32:04.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0060377709999999999 2024-09-10T18:32:04.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159696+0000", 2024-09-10T18:32:04.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1711e-05 2024-09-10T18:32:04.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:32:04.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159731+0000", 2024-09-10T18:32:04.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5259000000000002e-05 2024-09-10T18:32:04.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159761+0000", 2024-09-10T18:32:04.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9757999999999999e-05 2024-09-10T18:32:04.777 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.777 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.777 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.778 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221759 7.c 7:3758a2a2:::benchmark_data_smithi028_77856_object100659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:32:04.778 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.174377+0000", 2024-09-10T18:32:04.778 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 295.48384494700002, 2024-09-10T18:32:04.778 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98546984100000001, 2024-09-10T18:32:04.778 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.778 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:04.778 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.778 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.778 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.778 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221759 2024-09-10T18:32:04.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.778 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.174377+0000", 2024-09-10T18:32:04.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.174377+0000", 2024-09-10T18:32:04.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.174376+0000", 2024-09-10T18:32:04.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:32:04.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.174385+0000", 2024-09-10T18:32:04.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6370000000000005e-06 2024-09-10T18:32:04.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.174396+0000", 2024-09-10T18:32:04.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1311999999999999e-05 2024-09-10T18:32:04.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.174401+0000", 2024-09-10T18:32:04.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3990000000000003e-06 2024-09-10T18:32:04.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.174431+0000", 2024-09-10T18:32:04.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9558999999999999e-05 2024-09-10T18:32:04.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.174437+0000", 2024-09-10T18:32:04.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0859999999999998e-06 2024-09-10T18:32:04.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.149913+0000", 2024-09-10T18:32:04.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97547597100000005 2024-09-10T18:32:04.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:32:04.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.150022+0000", 2024-09-10T18:32:04.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108928 2024-09-10T18:32:04.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:32:04.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.150125+0000", 2024-09-10T18:32:04.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010319600000000001 2024-09-10T18:32:04.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:32:04.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.156072+0000", 2024-09-10T18:32:04.783 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059466550000000003 2024-09-10T18:32:04.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159665+0000", 2024-09-10T18:32:04.783 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035927789999999999 2024-09-10T18:32:04.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159791+0000", 2024-09-10T18:32:04.783 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012606299999999999 2024-09-10T18:32:04.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.784 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.784 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:32:04.784 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159817+0000", 2024-09-10T18:32:04.784 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6684999999999999e-05 2024-09-10T18:32:04.784 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.784 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.784 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.784 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159847+0000", 2024-09-10T18:32:04.784 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9848e-05 2024-09-10T18:32:04.784 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.784 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.784 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.784 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.784 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.785 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221761 7.7 7:ed6e3b57:::benchmark_data_smithi028_77856_object100661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:32:04.785 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.178273+0000", 2024-09-10T18:32:04.785 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 295.47994928499998, 2024-09-10T18:32:04.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98234119399999997, 2024-09-10T18:32:04.785 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.785 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:04.785 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.785 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.785 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.785 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221761 2024-09-10T18:32:04.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.785 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.178273+0000", 2024-09-10T18:32:04.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.178273+0000", 2024-09-10T18:32:04.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.178271+0000", 2024-09-10T18:32:04.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:04.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.178278+0000", 2024-09-10T18:32:04.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0010000000000004e-06 2024-09-10T18:32:04.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.178289+0000", 2024-09-10T18:32:04.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0807e-05 2024-09-10T18:32:04.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.178294+0000", 2024-09-10T18:32:04.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2399999999999998e-06 2024-09-10T18:32:04.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.178338+0000", 2024-09-10T18:32:04.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3837000000000001e-05 2024-09-10T18:32:04.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.178347+0000", 2024-09-10T18:32:04.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8510000000000005e-06 2024-09-10T18:32:04.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.153274+0000", 2024-09-10T18:32:04.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97492656499999997 2024-09-10T18:32:04.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.789 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.789 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:32:04.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.153375+0000", 2024-09-10T18:32:04.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101545 2024-09-10T18:32:04.789 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.789 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.789 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-09-10T18:32:04.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.153484+0000", 2024-09-10T18:32:04.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010891300000000001 2024-09-10T18:32:04.789 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.789 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.789 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:32:04.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.157887+0000", 2024-09-10T18:32:04.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0044034080000000001 2024-09-10T18:32:04.790 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.790 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.790 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160502+0000", 2024-09-10T18:32:04.790 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002614487 2024-09-10T18:32:04.790 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.790 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.790 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160542+0000", 2024-09-10T18:32:04.790 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0200999999999997e-05 2024-09-10T18:32:04.790 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.790 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:32:04.790 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160582+0000", 2024-09-10T18:32:04.791 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9826999999999997e-05 2024-09-10T18:32:04.791 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.791 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.791 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.791 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160614+0000", 2024-09-10T18:32:04.791 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2214e-05 2024-09-10T18:32:04.791 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.791 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.791 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.791 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.791 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.791 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221766 7.c 7:3aafddb5:::benchmark_data_smithi028_77856_object100666:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:32:04.791 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.180051+0000", 2024-09-10T18:32:04.791 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 295.47817139300003, 2024-09-10T18:32:04.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97996098200000004, 2024-09-10T18:32:04.792 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.792 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:04.792 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.792 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.792 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.792 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221766 2024-09-10T18:32:04.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.792 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.792 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.180051+0000", 2024-09-10T18:32:04.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.180051+0000", 2024-09-10T18:32:04.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.180050+0000", 2024-09-10T18:32:04.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:32:04.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.180055+0000", 2024-09-10T18:32:04.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9400000000000001e-06 2024-09-10T18:32:04.794 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.794 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.180065+0000", 2024-09-10T18:32:04.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9660000000000008e-06 2024-09-10T18:32:04.794 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.794 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.180069+0000", 2024-09-10T18:32:04.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1869999999999999e-06 2024-09-10T18:32:04.794 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.180117+0000", 2024-09-10T18:32:04.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7732000000000003e-05 2024-09-10T18:32:04.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.180124+0000", 2024-09-10T18:32:04.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4399999999999999e-06 2024-09-10T18:32:04.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.151272+0000", 2024-09-10T18:32:04.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97114836199999999 2024-09-10T18:32:04.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:32:04.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.151375+0000", 2024-09-10T18:32:04.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010259000000000001 2024-09-10T18:32:04.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:32:04.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.151479+0000", 2024-09-10T18:32:04.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104035 2024-09-10T18:32:04.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:32:04.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.156079+0000", 2024-09-10T18:32:04.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0045997909999999998 2024-09-10T18:32:04.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.797 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159886+0000", 2024-09-10T18:32:04.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038077250000000001 2024-09-10T18:32:04.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.797 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159945+0000", 2024-09-10T18:32:04.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8746999999999998e-05 2024-09-10T18:32:04.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.797 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:32:04.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159982+0000", 2024-09-10T18:32:04.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6797e-05 2024-09-10T18:32:04.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.798 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.798 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160012+0000", 2024-09-10T18:32:04.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9785e-05 2024-09-10T18:32:04.798 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.798 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.798 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.798 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221769 7.c 7:3717c799:::benchmark_data_smithi028_77856_object100669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:32:04.798 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.182048+0000", 2024-09-10T18:32:04.798 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 295.47617457299998, 2024-09-10T18:32:04.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97804954, 2024-09-10T18:32:04.798 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.798 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:04.799 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.799 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.799 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.799 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221769 2024-09-10T18:32:04.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.799 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.799 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.799 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.182048+0000", 2024-09-10T18:32:04.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.799 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.799 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.182048+0000", 2024-09-10T18:32:04.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.182046+0000", 2024-09-10T18:32:04.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:32:04.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.182052+0000", 2024-09-10T18:32:04.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6749999999999999e-06 2024-09-10T18:32:04.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.182064+0000", 2024-09-10T18:32:04.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1534e-05 2024-09-10T18:32:04.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.182070+0000", 2024-09-10T18:32:04.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.742e-06 2024-09-10T18:32:04.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.182108+0000", 2024-09-10T18:32:04.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8028000000000003e-05 2024-09-10T18:32:04.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.182113+0000", 2024-09-10T18:32:04.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.062e-06 2024-09-10T18:32:04.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.152330+0000", 2024-09-10T18:32:04.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97021630999999997 2024-09-10T18:32:04.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:32:04.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.152427+0000", 2024-09-10T18:32:04.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7149000000000001e-05 2024-09-10T18:32:04.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:32:04.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.152529+0000", 2024-09-10T18:32:04.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102105 2024-09-10T18:32:04.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:32:04.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.158720+0000", 2024-09-10T18:32:04.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061913080000000004 2024-09-10T18:32:04.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.159915+0000", 2024-09-10T18:32:04.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001194465 2024-09-10T18:32:04.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160046+0000", 2024-09-10T18:32:04.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013089300000000001 2024-09-10T18:32:04.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:32:04.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160068+0000", 2024-09-10T18:32:04.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1968999999999999e-05 2024-09-10T18:32:04.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160097+0000", 2024-09-10T18:32:04.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9626000000000001e-05 2024-09-10T18:32:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:221770 7.7 7:e05c9776:::benchmark_data_smithi028_77856_object100670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-10T18:32:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:27:09.185332+0000", 2024-09-10T18:32:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 295.47289053899999, 2024-09-10T18:32:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98126123099999996, 2024-09-10T18:32:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:32:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:32:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:32:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 221770 2024-09-10T18:32:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:32:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:32:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.185332+0000", 2024-09-10T18:32:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:32:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.185332+0000", 2024-09-10T18:32:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:32:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:32:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.185330+0000", 2024-09-10T18:32:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:32:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:32:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.185337+0000", 2024-09-10T18:32:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3579999999999999e-06 2024-09-10T18:32:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:32:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.185348+0000", 2024-09-10T18:32:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1902000000000001e-05 2024-09-10T18:32:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:32:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.185354+0000", 2024-09-10T18:32:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0260000000000004e-06 2024-09-10T18:32:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.185399+0000", 2024-09-10T18:32:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4299999999999999e-05 2024-09-10T18:32:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:32:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:09.185404+0000", 2024-09-10T18:32:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9230000000000001e-06 2024-09-10T18:32:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:32:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.154919+0000", 2024-09-10T18:32:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.96951485999999998 2024-09-10T18:32:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:32:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.155003+0000", 2024-09-10T18:32:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4492000000000004e-05 2024-09-10T18:32:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3", 2024-09-10T18:32:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.155099+0000", 2024-09-10T18:32:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5878000000000006e-05 2024-09-10T18:32:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:32:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160467+0000", 2024-09-10T18:32:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0053681989999999997 2024-09-10T18:32:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.160672+0000", 2024-09-10T18:32:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020532999999999999 2024-09-10T18:32:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.166507+0000", 2024-09-10T18:32:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058346630000000003 2024-09-10T18:32:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:32:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.166554+0000", 2024-09-10T18:32:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7089000000000003e-05 2024-09-10T18:32:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:32:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:32:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:32:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:27:10.166593+0000", 2024-09-10T18:32:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8670000000000001e-05 2024-09-10T18:32:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:32:04.812 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:32:04.812 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:32:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:04.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:04 smithi028 ceph-mon[29291]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:32:04.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:04 smithi028 ceph-mon[29291]: osdmap e247: 8 total, 8 up, 8 in 2024-09-10T18:32:04.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:04 smithi028 ceph-mon[39220]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:32:04.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:04 smithi028 ceph-mon[39220]: osdmap e247: 8 total, 8 up, 8 in 2024-09-10T18:32:04.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:04.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:04.849 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:04.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:04.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:04.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:04.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:04.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:05.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:05.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:05.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:05.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:05.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:05.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:05.225 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:05.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:05.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:05.351 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:05.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:05.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:05.453 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:05.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:05.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:05.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:05.595 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:05 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:32:05.595 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:05 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:32:05.595 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:05 smithi204 ceph-mon[32357]: osdmap e248: 8 total, 8 up, 8 in 2024-09-10T18:32:05.595 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:05 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:05.596 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:05 smithi204 ceph-mon[32357]: pgmap v474: 158 pgs: 9 unknown, 149 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:32:05.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T18:32:05.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:05 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:32:05.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:05 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:32:05.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:05 smithi028 ceph-mon[39220]: osdmap e248: 8 total, 8 up, 8 in 2024-09-10T18:32:05.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:05 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:05.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:05 smithi028 ceph-mon[39220]: pgmap v474: 158 pgs: 9 unknown, 149 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:32:05.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:05 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:32:05.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:05 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:32:05.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:05 smithi028 ceph-mon[29291]: osdmap e248: 8 total, 8 up, 8 in 2024-09-10T18:32:05.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:05 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:05.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:05 smithi028 ceph-mon[29291]: pgmap v474: 158 pgs: 9 unknown, 149 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:32:05.673 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:05.694 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:05.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:05.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:05.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:05.820 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:05.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:05.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:05.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:05.943 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:05.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:06.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:32:06.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:06.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:06.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:06.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:06.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:06.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:06.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:06.293 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:06.317 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:06.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:06.419 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:06.442 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:06.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:06.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:06.504 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:32:06.504 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:32:06.504 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-10T18:32:06.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T18:32:06.544 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:06.566 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:06.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:06.669 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:06.691 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:06.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:06.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:06 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:32:06.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:06 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:32:06.793 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:06.817 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:06.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:06.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:06 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:32:06.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:06.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:06.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:07.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:07.068 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:07.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:07.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:07.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:07.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:07.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:07.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:07.321 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:07.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:07.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:07.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:07.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:07.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:07.455 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:32:07.455 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:32:07.455 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:32:07.455 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:32:07.455 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:32:07.456 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:32:07.456 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:32:07.456 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:32:07.456 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:32:07.456 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:32:07.456 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:32:07.456 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:32:07.456 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:32:07.457 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:32:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:32:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:32:07.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:07 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:32:07.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:07 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:32:07.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:07 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:32:07.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:07 smithi028 ceph-mon[29291]: pgmap v475: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-09-10T18:32:07.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:07 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3415751150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:32:07.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:07 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:32:07.547 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:07 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:32:07.547 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:07 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:32:07.547 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:07 smithi028 ceph-mon[39220]: pgmap v475: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-09-10T18:32:07.547 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:07 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3415751150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:32:07.548 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:07.574 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:07.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:07.676 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:07.700 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:07.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:07.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:07.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:07.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:07.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:07.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:07.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:07 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:32:07.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:07 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:32:07.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:07 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:32:07.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:07 smithi204 ceph-mon[32357]: pgmap v475: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-09-10T18:32:07.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:07 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3415751150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:32:07.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:07.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:07.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:08.051 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:08.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:08.077 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:08.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:08.179 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:08.203 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:08.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:08.305 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:08.330 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:08.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:08.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:08.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:08.455 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:08.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:08.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:08.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:08.581 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:08.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:08.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:08.588 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:08.589 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:08.589 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:08.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:08.705 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:08.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:08.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:08.807 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:08.820 INFO:teuthology.orchestra.run.smithi028.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-10T18:32:08.833 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:08.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:08.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:08.960 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:08.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:08.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:09.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:08 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2044380242' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:32:09.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:08 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:32:09.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:08 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:32:09.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:08 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2044380242' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:32:09.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:08 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:32:09.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:08 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:32:09.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:09.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:09.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:09.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:08 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2044380242' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:32:09.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:08 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:32:09.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:08 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:32:09.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:09.212 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:09.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:09.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:09.313 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:09.339 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:09.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:09.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:09.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:09.464 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:09.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:09.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:09.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:09.589 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:09.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:09.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:09.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:09.716 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:09.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:09.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:09.818 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:09.841 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:09.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:09.943 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:09.966 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:09.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:10.068 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:10.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:32:10.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:10.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:10.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:09 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2044380242' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:32:10.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:09 smithi204 ceph-mon[32357]: osdmap e249: 8 total, 8 up, 8 in 2024-09-10T18:32:10.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:09 smithi204 ceph-mon[32357]: pgmap v477: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:32:10.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:09 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2044380242' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:32:10.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:09 smithi028 ceph-mon[29291]: osdmap e249: 8 total, 8 up, 8 in 2024-09-10T18:32:10.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:09 smithi028 ceph-mon[29291]: pgmap v477: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:32:10.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:09 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2044380242' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:32:10.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:09 smithi028 ceph-mon[39220]: osdmap e249: 8 total, 8 up, 8 in 2024-09-10T18:32:10.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:09 smithi028 ceph-mon[39220]: pgmap v477: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:32:10.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:10.218 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:10.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:10.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:10.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:10.345 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:10.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:10.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:10.447 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:10.472 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:10.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:10.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:10.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:10.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:10.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:10.699 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:10.724 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:10.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:10.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:10.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:10.850 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:10.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:10.951 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:10.976 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:10.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:10.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:11.078 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:11.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:10 smithi204 ceph-mon[32357]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:32:11.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:11.105 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:11.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:11.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:11.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:11.230 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:11.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:11.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:10 smithi028 ceph-mon[29291]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:32:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:10 smithi028 ceph-mon[39220]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:32:11.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:11.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:11.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:11.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:11.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:11.478 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:11.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:11.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:11.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:11.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:11.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:11.627 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:32:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:32:11] "GET /metrics HTTP/1.1" 200 224195 "" "Prometheus/2.43.0" 2024-09-10T18:32:11.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:11.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:11.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:11.828 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:11.850 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:11.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:11.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:11 smithi028 ceph-mon[39220]: pgmap v478: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:32:11.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:11 smithi028 ceph-mon[29291]: pgmap v478: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:32:11.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:11.976 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:11.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:11.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:12.078 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:12.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:11 smithi204 ceph-mon[32357]: pgmap v478: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:32:12.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:12.105 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:12.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:12.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:12.232 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:12.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:12.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:12.334 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:12.359 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:12.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:12.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:12.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:12.484 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:12.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:12.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:12.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:12.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:32:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:32:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:32:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:12.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:12.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:12.710 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:12.736 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:12.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:12.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:12.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:12.861 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:12.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:12.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:12.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:12.975 DEBUG:teuthology.orchestra.run.smithi028:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-10T18:32:12.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:12.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:12.987 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-10T18:32:12.987 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-10T18:32:13.032 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:hints = 1 2024-09-10T18:32:13.032 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-10T18:32:13.032 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object prefix: benchmark_data_smithi028_135703 2024-09-10T18:32:13.033 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:32:13.033 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 0 0 0 0 0 0 - 0 2024-09-10T18:32:13.087 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:13.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:13.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:13.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:13.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:13.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:13.257 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:13.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:13.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:13.379 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:32:13.379 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:32:13.402 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:13.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:13.504 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:13.536 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:13.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:13.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:13.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:13.669 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:13.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:13.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:13.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:13.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:13.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:13.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:13.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:13.950 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:13.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:14.033 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1 16 3506 3490 218.112 218.125 0.00431042 0.00456766 2024-09-10T18:32:14.051 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:14.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:14.092 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:14.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:14.186 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:32:14.186 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:32:14.186 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 249, 2024-09-10T18:32:14.186 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "5eababb2-6fa0-11ef-bcea-c7b262605968", 2024-09-10T18:32:14.186 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-10T18:16:20.680373+0000", 2024-09-10T18:32:14.186 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-10T18:32:08.814813+0000", 2024-09-10T18:32:14.186 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-09-10T18:26:26.611338+0000", 2024-09-10T18:32:14.187 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-09-10T18:21:02.114621+0000", 2024-09-10T18:32:14.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T18:32:14.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5801984, 2024-09-10T18:32:14.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-09-10T18:32:14.187 INFO:teuthology.orchestra.run.smithi028.stdout: "noscrub", 2024-09-10T18:32:14.187 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-09-10T18:32:14.187 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-09-10T18:32:14.187 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-09-10T18:32:14.187 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-09-10T18:32:14.187 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:32:14.187 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 19, 2024-09-10T18:32:14.187 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T18:32:14.187 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T18:32:14.187 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T18:32:14.188 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-09-10T18:32:14.188 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 9, 2024-09-10T18:32:14.188 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-09-10T18:32:14.188 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-09-10T18:32:14.188 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-09-10T18:32:14.188 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "quincy", 2024-09-10T18:32:14.188 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-09-10T18:32:14.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.188 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-09-10T18:32:14.188 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-09-10T18:32:14.188 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:19:24.013536+0000", 2024-09-10T18:32:14.188 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:32:14.188 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:32:14.188 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:32:14.189 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:32:14.189 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:32:14.189 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:32:14.189 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:32:14.189 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:32:14.189 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:32:14.189 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:32:14.189 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:32:14.189 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:32:14.189 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 11, 2024-09-10T18:32:14.189 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 11, 2024-09-10T18:32:14.189 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 11, 2024-09-10T18:32:14.189 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 11, 2024-09-10T18:32:14.190 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 11, 2024-09-10T18:32:14.190 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:32:14.190 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:32:14.190 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:32:14.190 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:32:14.190 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:32:14.190 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:32:14.190 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:32:14.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.190 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "138", 2024-09-10T18:32:14.190 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:32:14.190 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:32:14.190 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "136", 2024-09-10T18:32:14.191 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:32:14.191 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:32:14.191 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:32:14.191 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:32:14.191 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:32:14.191 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:32:14.191 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:32:14.191 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:32:14.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:32:14.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:32:14.191 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:32:14.191 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:32:14.191 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:32:14.191 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:32:14.192 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:32:14.192 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:32:14.192 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:32:14.192 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:32:14.192 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:32:14.192 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:32:14.192 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:32:14.192 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:32:14.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:32:14.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.192 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:32:14.192 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:32:14.192 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:32:14.193 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:32:14.193 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:32:14.193 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:32:14.193 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:32:14.193 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:32:14.193 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:32:14.193 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:32:14.193 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:32:14.193 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-10T18:32:14.193 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-09-10T18:32:14.193 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-09-10T18:32:14.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.193 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:32:14.193 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-09-10T18:32:14.194 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.194 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-09-10T18:32:14.194 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-09-10T18:32:14.194 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:30.065209+0000", 2024-09-10T18:32:14.194 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-09-10T18:32:14.194 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T18:32:14.194 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:32:14.194 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:32:14.194 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:32:14.194 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:32:14.194 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:32:14.195 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:32:14.195 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:32:14.195 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:32:14.195 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:32:14.195 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:32:14.195 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-09-10T18:32:14.195 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-09-10T18:32:14.195 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-09-10T18:32:14.195 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-09-10T18:32:14.195 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-09-10T18:32:14.195 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:32:14.195 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:32:14.195 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:32:14.195 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:32:14.196 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:32:14.196 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:32:14.196 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:32:14.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.196 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "62", 2024-09-10T18:32:14.196 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:32:14.196 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:32:14.196 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:32:14.196 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:32:14.196 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:32:14.196 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 3, 2024-09-10T18:32:14.196 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 62, 2024-09-10T18:32:14.196 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:32:14.196 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:32:14.197 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:32:14.197 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:32:14.197 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:32:14.197 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:32:14.197 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:32:14.197 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:32:14.197 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:32:14.197 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:32:14.197 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:32:14.197 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:32:14.197 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:32:14.197 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:32:14.197 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:32:14.197 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:32:14.198 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:32:14.198 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:32:14.198 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:32:14.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.198 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:32:14.198 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:32:14.198 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:32:14.198 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:32:14.198 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:32:14.198 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:32:14.198 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:32:14.198 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:32:14.198 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:32:14.198 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:32:14.199 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:32:14.199 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:32:14.199 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:32:14.199 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-09-10T18:32:14.199 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.199 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-09-10T18:32:14.199 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-09-10T18:32:14.199 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:30.506441+0000", 2024-09-10T18:32:14.199 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:32:14.199 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:32:14.199 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:32:14.199 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:32:14.199 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:32:14.200 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:32:14.200 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:32:14.200 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:32:14.200 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:32:14.200 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:32:14.200 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:32:14.200 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:32:14.200 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:32:14.200 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:32:14.200 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:32:14.200 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:32:14.200 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:32:14.200 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:32:14.200 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:32:14.201 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:32:14.201 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:32:14.201 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:32:14.201 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:32:14.201 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:32:14.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.201 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "59", 2024-09-10T18:32:14.201 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:32:14.201 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:32:14.201 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:32:14.201 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:32:14.201 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:32:14.201 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:32:14.202 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:32:14.202 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:32:14.202 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:32:14.202 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:32:14.202 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:32:14.202 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:32:14.202 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:32:14.202 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:32:14.202 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:32:14.202 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:32:14.202 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:32:14.202 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:32:14.202 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:32:14.202 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:32:14.202 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:32:14.203 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:32:14.203 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:32:14.203 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:32:14.203 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:32:14.203 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:32:14.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.203 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:32:14.203 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:32:14.203 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:32:14.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:32:14.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:32:14.203 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:32:14.203 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:32:14.204 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:32:14.204 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:32:14.204 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:32:14.204 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:32:14.204 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:32:14.204 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:32:14.204 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:32:14.204 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.204 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-09-10T18:32:14.204 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-09-10T18:32:14.204 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:32.008130+0000", 2024-09-10T18:32:14.205 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:32:14.205 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:32:14.205 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:32:14.205 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:32:14.205 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:32:14.205 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:32:14.205 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:32:14.205 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:32:14.205 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:32:14.205 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:32:14.205 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:32:14.205 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:32:14.205 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:32:14.205 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:32:14.206 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:32:14.206 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:32:14.206 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:32:14.206 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:32:14.206 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:32:14.206 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:32:14.206 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:32:14.206 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:32:14.206 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:32:14.206 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:32:14.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.206 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "61", 2024-09-10T18:32:14.206 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:32:14.207 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:32:14.207 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:32:14.207 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:32:14.207 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:32:14.207 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:32:14.207 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:32:14.207 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:32:14.207 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:32:14.207 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:32:14.207 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:32:14.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:32:14.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:32:14.207 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:32:14.207 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:32:14.208 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:32:14.208 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:32:14.208 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:32:14.208 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:32:14.208 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:32:14.208 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:32:14.208 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:32:14.208 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:32:14.208 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:32:14.208 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:32:14.208 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:32:14.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.208 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:32:14.208 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:32:14.209 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:32:14.209 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:32:14.209 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:32:14.209 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:32:14.209 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:32:14.209 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:32:14.209 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:32:14.209 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:32:14.209 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:32:14.209 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:32:14.209 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:32:14.209 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:32:14.209 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.210 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-09-10T18:32:14.210 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-09-10T18:32:14.210 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:34.060571+0000", 2024-09-10T18:32:14.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:32:14.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:32:14.210 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:32:14.210 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:32:14.210 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:32:14.210 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:32:14.210 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:32:14.210 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:32:14.210 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:32:14.211 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:32:14.211 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:32:14.211 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:32:14.211 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:32:14.211 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:32:14.211 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:32:14.211 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:32:14.211 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:32:14.211 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:32:14.211 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:32:14.211 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:32:14.211 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:32:14.211 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:32:14.211 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:32:14.212 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:32:14.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.212 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "63", 2024-09-10T18:32:14.212 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:32:14.212 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:32:14.212 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:32:14.212 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:32:14.212 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:32:14.212 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:32:14.212 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:32:14.212 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:32:14.212 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:32:14.212 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:32:14.213 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:32:14.213 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:32:14.213 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:32:14.213 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:32:14.213 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:32:14.213 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:32:14.213 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:32:14.213 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:32:14.213 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:32:14.213 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:32:14.213 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:32:14.213 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:32:14.213 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:32:14.213 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:32:14.214 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:32:14.214 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:32:14.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.214 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:32:14.214 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:32:14.214 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:32:14.214 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:32:14.214 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:32:14.214 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:32:14.214 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:32:14.214 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:32:14.214 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:32:14.215 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:32:14.215 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:32:14.215 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:32:14.215 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:32:14.215 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:32:14.215 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.215 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-09-10T18:32:14.215 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-09-10T18:32:14.215 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:36.127330+0000", 2024-09-10T18:32:14.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:32:14.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:32:14.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:32:14.216 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:32:14.216 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:32:14.216 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:32:14.216 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:32:14.216 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:32:14.216 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:32:14.216 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:32:14.216 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:32:14.216 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:32:14.216 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:32:14.216 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:32:14.216 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:32:14.216 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:32:14.216 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:32:14.217 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:32:14.217 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:32:14.217 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:32:14.217 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:32:14.217 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:32:14.217 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:32:14.217 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:32:14.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.217 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "65", 2024-09-10T18:32:14.217 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:32:14.217 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:32:14.217 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:32:14.217 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:32:14.217 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:32:14.218 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:32:14.218 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:32:14.218 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:32:14.218 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:32:14.218 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:32:14.218 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:32:14.218 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:32:14.218 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:32:14.218 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:32:14.218 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:32:14.218 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:32:14.218 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:32:14.218 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:32:14.218 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:32:14.218 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:32:14.219 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:32:14.219 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:32:14.219 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:32:14.219 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:32:14.219 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:32:14.219 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:32:14.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.219 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:32:14.219 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:32:14.219 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:32:14.219 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:32:14.219 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:32:14.219 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:32:14.219 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:32:14.219 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:32:14.220 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:32:14.220 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:32:14.220 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:32:14.220 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-10T18:32:14.220 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-09-10T18:32:14.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.220 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:32:14.220 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:32:14.220 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.220 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 9, 2024-09-10T18:32:14.220 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "unique_pool_2", 2024-09-10T18:32:14.220 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:32:02.316841+0000", 2024-09-10T18:32:14.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:32:14.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:32:14.221 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:32:14.221 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:32:14.221 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:32:14.221 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:32:14.221 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:32:14.221 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:32:14.221 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:32:14.221 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:32:14.221 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:32:14.221 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:32:14.221 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 16, 2024-09-10T18:32:14.222 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 16, 2024-09-10T18:32:14.222 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 16, 2024-09-10T18:32:14.222 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 16, 2024-09-10T18:32:14.222 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 16, 2024-09-10T18:32:14.222 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:32:14.222 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:32:14.222 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:32:14.222 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:32:14.222 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:32:14.222 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:32:14.222 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:32:14.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.222 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "249", 2024-09-10T18:32:14.222 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:32:14.223 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:32:14.223 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:32:14.223 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:32:14.223 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:32:14.223 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:32:14.223 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:32:14.223 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:32:14.223 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:32:14.223 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:32:14.223 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:32:14.223 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:32:14.223 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:32:14.223 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:32:14.223 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:32:14.224 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:32:14.224 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:32:14.224 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:32:14.224 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:32:14.224 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:32:14.224 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:32:14.224 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:32:14.224 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:32:14.224 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:32:14.224 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:32:14.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:32:14.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.224 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:32:14.224 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:32:14.224 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:32:14.225 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:32:14.225 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:32:14.225 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:32:14.225 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:32:14.225 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:32:14.225 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:32:14.225 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:32:14.225 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:32:14.225 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:32:14.225 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:32:14.225 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-09-10T18:32:14.225 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.225 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.225 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:32:14.226 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:32:14.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.226 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-10T18:32:14.226 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "f1c834d9-252a-4077-b25e-169181d70251", 2024-09-10T18:32:14.226 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:32:14.226 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:32:14.226 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:32:14.226 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:32:14.226 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:32:14.226 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:32:14.226 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-09-10T18:32:14.226 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 246, 2024-09-10T18:32:14.226 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:32:14.227 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:32:14.227 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:32:14.227 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.227 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-09-10T18:32:14.227 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:32:14.227 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.227 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.227 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:32:14.227 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.227 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-09-10T18:32:14.228 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:32:14.228 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.228 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.228 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:32:14.228 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.228 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-09-10T18:32:14.228 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:32:14.228 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.228 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.228 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:32:14.228 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.229 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.229 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-09-10T18:32:14.229 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:32:14.229 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.229 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.229 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/3188372351", 2024-09-10T18:32:14.229 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/3188372351", 2024-09-10T18:32:14.229 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/3188372351", 2024-09-10T18:32:14.229 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/3188372351", 2024-09-10T18:32:14.229 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:32:14.229 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:32:14.229 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:32:14.230 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.230 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-10T18:32:14.230 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "81f1855d-b946-4ca3-b246-38977b9e20fd", 2024-09-10T18:32:14.230 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:32:14.230 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:32:14.230 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:32:14.230 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:32:14.230 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:32:14.230 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:32:14.230 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-10T18:32:14.230 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 246, 2024-09-10T18:32:14.230 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:32:14.230 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:32:14.231 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:32:14.231 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.231 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.231 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-09-10T18:32:14.231 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:32:14.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.231 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.231 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:32:14.231 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.231 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.231 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-09-10T18:32:14.232 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:32:14.232 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.232 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.232 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:32:14.232 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.232 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.232 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-09-10T18:32:14.232 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:32:14.232 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.232 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.232 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:32:14.233 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.233 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.233 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-09-10T18:32:14.233 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:32:14.233 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.233 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.233 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1369912678", 2024-09-10T18:32:14.233 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1369912678", 2024-09-10T18:32:14.233 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1369912678", 2024-09-10T18:32:14.233 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1369912678", 2024-09-10T18:32:14.233 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:32:14.234 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:32:14.234 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:32:14.234 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.234 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-10T18:32:14.234 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "2ad55d2a-36fa-4756-bf22-58d4b049486e", 2024-09-10T18:32:14.234 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:32:14.234 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:32:14.234 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:32:14.234 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:32:14.234 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:32:14.234 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:32:14.234 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-09-10T18:32:14.235 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 95, 2024-09-10T18:32:14.235 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:32:14.235 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:32:14.235 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:32:14.235 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.235 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-09-10T18:32:14.235 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:32:14.235 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.235 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.235 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:32:14.235 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.236 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-09-10T18:32:14.236 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:32:14.236 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.236 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.236 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:32:14.236 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.236 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-09-10T18:32:14.236 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:32:14.236 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.237 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.237 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:32:14.237 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.237 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.237 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-09-10T18:32:14.237 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:32:14.237 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.237 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.237 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/1243406003", 2024-09-10T18:32:14.237 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/1243406003", 2024-09-10T18:32:14.238 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/1243406003", 2024-09-10T18:32:14.238 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1243406003", 2024-09-10T18:32:14.238 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:32:14.238 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:32:14.238 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:32:14.238 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.238 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-10T18:32:14.238 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "242729a5-a5e8-44d5-97c8-14d14e970bff", 2024-09-10T18:32:14.238 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:32:14.238 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:32:14.238 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:32:14.238 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:32:14.239 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:32:14.239 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:32:14.239 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-09-10T18:32:14.239 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 246, 2024-09-10T18:32:14.239 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:32:14.239 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:32:14.239 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:32:14.239 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.239 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.239 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-09-10T18:32:14.239 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:32:14.239 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.239 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.240 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:32:14.240 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.240 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-09-10T18:32:14.240 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:32:14.240 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.240 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.240 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:32:14.240 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.240 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-09-10T18:32:14.241 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:32:14.241 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.241 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.241 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:32:14.241 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.241 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.241 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-09-10T18:32:14.241 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:32:14.241 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.241 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.242 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/67703283", 2024-09-10T18:32:14.242 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/67703283", 2024-09-10T18:32:14.242 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/67703283", 2024-09-10T18:32:14.242 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/67703283", 2024-09-10T18:32:14.242 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:32:14.242 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:32:14.242 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:32:14.242 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.242 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-10T18:32:14.242 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "6e38b8ef-e6ed-493f-8eca-755354754787", 2024-09-10T18:32:14.242 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:32:14.243 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:32:14.243 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:32:14.243 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:32:14.243 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:32:14.243 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:32:14.243 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 33, 2024-09-10T18:32:14.243 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 219, 2024-09-10T18:32:14.243 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:32:14.243 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:32:14.243 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:32:14.243 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.244 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6800", 2024-09-10T18:32:14.244 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:32:14.244 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.244 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.244 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:32:14.244 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.244 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6801", 2024-09-10T18:32:14.244 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:32:14.244 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.244 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.245 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:32:14.245 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.245 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6803", 2024-09-10T18:32:14.245 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:32:14.245 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.245 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.246 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:32:14.246 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.246 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6802", 2024-09-10T18:32:14.246 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:32:14.246 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.246 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.246 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6800/1619669076", 2024-09-10T18:32:14.246 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6801/1619669076", 2024-09-10T18:32:14.246 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6803/1619669076", 2024-09-10T18:32:14.246 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6802/1619669076", 2024-09-10T18:32:14.247 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:32:14.247 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:32:14.247 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:32:14.247 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.247 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-10T18:32:14.247 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "a92e0441-66c9-498e-a0ac-6465acdbb1f5", 2024-09-10T18:32:14.247 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:32:14.247 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:32:14.247 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:32:14.247 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:32:14.247 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:32:14.247 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:32:14.248 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 39, 2024-09-10T18:32:14.248 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 246, 2024-09-10T18:32:14.248 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:32:14.248 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:32:14.248 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:32:14.248 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.248 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.248 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6804", 2024-09-10T18:32:14.248 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:32:14.248 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.248 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.248 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:32:14.248 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.249 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6805", 2024-09-10T18:32:14.249 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:32:14.249 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.249 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.249 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:32:14.249 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.249 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6807", 2024-09-10T18:32:14.249 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:32:14.249 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.250 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.250 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:32:14.250 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.250 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6806", 2024-09-10T18:32:14.250 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:32:14.250 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.250 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.250 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6804/2717802880", 2024-09-10T18:32:14.250 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6805/2717802880", 2024-09-10T18:32:14.250 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6807/2717802880", 2024-09-10T18:32:14.251 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6806/2717802880", 2024-09-10T18:32:14.251 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:32:14.251 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:32:14.251 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:32:14.251 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.251 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-09-10T18:32:14.251 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "12b02875-6653-45ed-8d2b-b4d33bd88525", 2024-09-10T18:32:14.251 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:32:14.251 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:32:14.251 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:32:14.251 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:32:14.251 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 46, 2024-09-10T18:32:14.252 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 94, 2024-09-10T18:32:14.252 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 144, 2024-09-10T18:32:14.252 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 210, 2024-09-10T18:32:14.252 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 95, 2024-09-10T18:32:14.252 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:32:14.252 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:32:14.252 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.252 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6808", 2024-09-10T18:32:14.252 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:32:14.252 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.252 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.253 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:32:14.253 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.253 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.253 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6809", 2024-09-10T18:32:14.253 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:32:14.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.253 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.253 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:32:14.253 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.253 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.254 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6811", 2024-09-10T18:32:14.254 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:32:14.254 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.254 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.254 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:32:14.254 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.254 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6810", 2024-09-10T18:32:14.254 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:32:14.254 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.254 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.254 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6808/3479311452", 2024-09-10T18:32:14.255 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6809/3479311452", 2024-09-10T18:32:14.255 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6811/3479311452", 2024-09-10T18:32:14.255 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6810/3479311452", 2024-09-10T18:32:14.255 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:32:14.255 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:32:14.255 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:32:14.255 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.255 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-09-10T18:32:14.255 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "0c4f7aa9-ff95-4192-a146-ca576eee907f", 2024-09-10T18:32:14.255 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:32:14.255 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:32:14.255 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:32:14.256 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:32:14.256 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:32:14.256 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:32:14.256 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 54, 2024-09-10T18:32:14.256 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 246, 2024-09-10T18:32:14.256 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:32:14.256 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:32:14.256 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:32:14.256 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.256 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6812", 2024-09-10T18:32:14.256 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:32:14.256 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.257 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.257 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:32:14.257 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.257 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.257 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6813", 2024-09-10T18:32:14.257 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:32:14.257 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.257 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.257 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:32:14.257 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.258 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6815", 2024-09-10T18:32:14.258 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:32:14.258 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.258 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.258 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:32:14.258 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:32:14.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:32:14.258 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6814", 2024-09-10T18:32:14.258 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:32:14.258 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.258 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.259 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6812/3511791879", 2024-09-10T18:32:14.259 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6813/3511791879", 2024-09-10T18:32:14.259 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6815/3511791879", 2024-09-10T18:32:14.259 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6814/3511791879", 2024-09-10T18:32:14.259 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:32:14.259 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:32:14.259 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:32:14.259 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.259 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.259 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:32:14.259 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-09-10T18:32:14.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.259 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-10T18:32:14.260 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:32:14.260 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:32:14.260 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:32:14.260 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:32:14.260 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:32:14.260 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:32:04.145891+0000", 2024-09-10T18:32:14.260 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:32:14.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.260 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-10T18:32:14.260 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:32:14.260 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:32:14.260 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:32:14.260 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:32:14.261 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:32:14.261 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:31:15.697480+0000", 2024-09-10T18:32:14.261 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:32:14.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.261 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-10T18:32:14.261 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:32:14.261 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:32:14.261 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:32:14.261 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:32:14.261 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:32:14.261 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:31:32.154984+0000", 2024-09-10T18:32:14.261 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:32:14.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.262 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-10T18:32:14.262 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:32:14.262 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:32:14.262 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:32:14.262 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:32:14.262 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:32:14.262 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:32:00.088088+0000", 2024-09-10T18:32:14.262 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:32:14.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.262 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-10T18:32:14.262 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:32:14.263 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:32:14.263 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:32:14.263 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:32:14.263 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:32:14.263 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:31:11.166816+0000", 2024-09-10T18:32:14.263 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:32:14.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.263 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-10T18:32:14.263 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:32:14.263 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:32:14.263 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:32:14.263 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:32:14.263 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:32:14.264 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:31:44.879737+0000", 2024-09-10T18:32:14.264 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:32:14.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.264 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-09-10T18:32:14.264 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:32:14.264 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:32:14.264 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:32:14.264 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:32:14.264 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:32:14.264 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:31:40.119261+0000", 2024-09-10T18:32:14.264 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 94 2024-09-10T18:32:14.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.265 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-09-10T18:32:14.265 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:32:14.265 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:32:14.265 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:32:14.265 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:32:14.265 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:32:14.265 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:30:57.187179+0000", 2024-09-10T18:32:14.265 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:32:14.265 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.265 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:32:14.265 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [], 2024-09-10T18:32:14.265 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-09-10T18:32:14.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.265 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.4", 2024-09-10T18:32:14.266 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:32:14.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.266 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:32:14.266 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-09-10T18:32:14.266 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.266 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.266 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1b", 2024-09-10T18:32:14.266 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:32:14.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.266 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:32:14.266 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-09-10T18:32:14.267 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.267 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.267 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1f", 2024-09-10T18:32:14.267 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:32:14.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:14.267 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:32:14.267 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-09-10T18:32:14.267 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.267 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:14.267 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.267 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:32:14.267 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-09-10T18:32:14.268 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-09-10T18:32:14.268 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-09-10T18:32:14.268 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1745574083": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:32:14.268 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3866220552": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:32:14.268 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3760775432": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:32:14.268 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1746797014": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:32:14.268 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1029045895": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:32:14.268 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3310949126": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:32:14.268 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2874160765": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:32:14.268 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2620179974": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:32:14.268 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/421194352": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:32:14.268 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/1077377902": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:32:14.268 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3428757300": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:32:14.268 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1759116360": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:32:14.269 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2564333506": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:32:14.269 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/717120167": "2024-09-11T18:22:08.678985+0000" 2024-09-10T18:32:14.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.269 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-09-10T18:32:14.269 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-09-10T18:32:14.269 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-09-10T18:32:14.269 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-09-10T18:32:14.269 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-09-10T18:32:14.269 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-09-10T18:32:14.269 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-09-10T18:32:14.269 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-09-10T18:32:14.269 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:14.269 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-09-10T18:32:14.270 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-09-10T18:32:14.270 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-09-10T18:32:14.270 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-09-10T18:32:14.270 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-09-10T18:32:14.270 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-09-10T18:32:14.270 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-09-10T18:32:14.270 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-09-10T18:32:14.270 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-09-10T18:32:14.270 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-09-10T18:32:14.270 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-09-10T18:32:14.270 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:14.270 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:32:14.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:14.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:14.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:14 smithi204 ceph-mon[32357]: pgmap v479: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:32:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:14 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3318774469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:32:14.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:14.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:14.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:14.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:14 smithi028 ceph-mon[39220]: pgmap v479: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:32:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:14 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3318774469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:32:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:14 smithi028 ceph-mon[29291]: pgmap v479: 158 pgs: 158 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:32:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:14 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3318774469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:32:14.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:14.583 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:14.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:14.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:14.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:14.729 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:14.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:14.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:14.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-10T18:32:14.831 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:14.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:32:14 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:32:14.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:14.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:14.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:14.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:14.993 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:14.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:14.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:15.033 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2 16 7005 6989 218.355 218.688 0.00285783 0.00456875 2024-09-10T18:32:15.095 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:15.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:15.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:15.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:15.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:15.229 INFO:tasks.thrashosds.thrasher:j is {'epoch': 249, 'fsid': '5eababb2-6fa0-11ef-bcea-c7b262605968', 'created': '2024-09-10T18:16:20.680373+0000', 'modified': '2024-09-10T18:32:08.814813+0000', 'last_up_change': '2024-09-10T18:26:26.611338+0000', 'last_in_change': '2024-09-10T18:21:02.114621+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T18:19:24.013536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '136', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-10T18:21:30.065209+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 62, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T18:21:30.506441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T18:21:32.008130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T18:21:34.060571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T18:21:36.127330+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-10T18:32:02.316841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '249', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'f1c834d9-252a-4077-b25e-169181d70251', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 246, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 3188372351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 3188372351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 3188372351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 3188372351}]}, 'public_addr': '172.21.15.28:6801/3188372351', 'cluster_addr': '172.21.15.28:6802/3188372351', 'heartbeat_back_addr': '172.21.15.28:6804/3188372351', 'heartbeat_front_addr': '172.21.15.28:6803/3188372351', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '81f1855d-b946-4ca3-b246-38977b9e20fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 246, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1369912678}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1369912678}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1369912678}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1369912678}]}, 'public_addr': '172.21.15.28:6805/1369912678', 'cluster_addr': '172.21.15.28:6806/1369912678', 'heartbeat_back_addr': '172.21.15.28:6808/1369912678', 'heartbeat_front_addr': '172.21.15.28:6807/1369912678', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2ad55d2a-36fa-4756-bf22-58d4b049486e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 1243406003}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 1243406003}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 1243406003}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1243406003}]}, 'public_addr': '172.21.15.28:6809/1243406003', 'cluster_addr': '172.21.15.28:6810/1243406003', 'heartbeat_back_addr': '172.21.15.28:6812/1243406003', 'heartbeat_front_addr': '172.21.15.28:6811/1243406003', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '242729a5-a5e8-44d5-97c8-14d14e970bff', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 246, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 67703283}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 67703283}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 67703283}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 67703283}]}, 'public_addr': '172.21.15.28:6813/67703283', 'cluster_addr': '172.21.15.28:6814/67703283', 'heartbeat_back_addr': '172.21.15.28:6816/67703283', 'heartbeat_front_addr': '172.21.15.28:6815/67703283', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6e38b8ef-e6ed-493f-8eca-755354754787', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 219, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6800', 'nonce': 1619669076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 1619669076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 1619669076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6802', 'nonce': 1619669076}]}, 'public_addr': '172.21.15.204:6800/1619669076', 'cluster_addr': '172.21.15.204:6801/1619669076', 'heartbeat_back_addr': '172.21.15.204:6803/1619669076', 'heartbeat_front_addr': '172.21.15.204:6802/1619669076', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a92e0441-66c9-498e-a0ac-6465acdbb1f5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 246, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6804', 'nonce': 2717802880}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 2717802880}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 2717802880}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6806', 'nonce': 2717802880}]}, 'public_addr': '172.21.15.204:6804/2717802880', 'cluster_addr': '172.21.15.204:6805/2717802880', 'heartbeat_back_addr': '172.21.15.204:6807/2717802880', 'heartbeat_front_addr': '172.21.15.204:6806/2717802880', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '12b02875-6653-45ed-8d2b-b4d33bd88525', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 46, 'last_clean_end': 94, 'up_from': 144, 'up_thru': 210, 'down_at': 95, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6808', 'nonce': 3479311452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 3479311452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 3479311452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6810', 'nonce': 3479311452}]}, 'public_addr': '172.21.15.204:6808/3479311452', 'cluster_addr': '172.21.15.204:6809/3479311452', 'heartbeat_back_addr': '172.21.15.204:6811/3479311452', 'heartbeat_front_addr': '172.21.15.204:6810/3479311452', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0c4f7aa9-ff95-4192-a146-ca576eee907f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 246, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6812', 'nonce': 3511791879}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 3511791879}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 3511791879}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6814', 'nonce': 3511791879}]}, 'public_addr': '172.21.15.204:6812/3511791879', 'cluster_addr': '172.21.15.204:6813/3511791879', 'heartbeat_back_addr': '172.21.15.204:6815/3511791879', 'heartbeat_front_addr': '172.21.15.204:6814/3511791879', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:32:04.145891+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:31:15.697480+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:31:32.154984+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:32:00.088088+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:31:11.166816+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:31:44.879737+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:31:40.119261+0000', 'dead_epoch': 94}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:30:57.187179+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/1745574083': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/3866220552': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:6800/3760775432': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:0/1746797014': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:0/1029045895': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:6800/3310949126': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/2874160765': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/2620179974': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/421194352': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:6800/1077377902': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/3428757300': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/1759116360': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/2564333506': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/717120167': '2024-09-11T18:22:08.678985+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T18:32:15.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:32:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:15.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:15.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:15.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:15.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:15.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:15.514 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:15.546 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:15.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:15.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:15.648 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:15.680 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:15.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:15.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:15.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:15.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:15.883 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:15.928 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:15.934 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:15.963 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:15.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:15.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:16.033 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3 16 10056 10040 209.125 190.688 0.00423617 0.0047735 2024-09-10T18:32:16.056 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:16 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:16.056 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:16 smithi028 ceph-mon[29291]: pgmap v480: 158 pgs: 158 active+clean; 175 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 17 MiB/s wr, 279 op/s 2024-09-10T18:32:16.056 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:16 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:16.057 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:16 smithi028 ceph-mon[39220]: pgmap v480: 158 pgs: 158 active+clean; 175 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 17 MiB/s wr, 279 op/s 2024-09-10T18:32:16.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:16.112 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:16.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:16.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:16.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:16.257 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:16.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:16.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:16 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:16 smithi204 ceph-mon[32357]: pgmap v480: 158 pgs: 158 active+clean; 175 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 17 MiB/s wr, 279 op/s 2024-09-10T18:32:16.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:16.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:16.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:16.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:16.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:16.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:16.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:16.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:16.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:16.682 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:16.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:16.783 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:16.816 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:16.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:16.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:16.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:16.956 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:16.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:17.033 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4 15 13257 13242 206.871 200.125 0.0050058 0.00482625 2024-09-10T18:32:17.059 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:17.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:17.093 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:17.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:17.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:17.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:17.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:17.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:17 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:32:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:17 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:32:17.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:17.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:17.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:17.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:17 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:32:17.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:17 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:32:17.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:17 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:32:17.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:17 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:32:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:17.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:17.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:17.620 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:17.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:17.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:17.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:17.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:17.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:17.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:17.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:17.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:17.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:17.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:18.033 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5 15 16085 16070 200.844 176.75 0.00554836 0.00497147 2024-09-10T18:32:18.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:18.095 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:18.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:18.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:18.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:18.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:18.238 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:18.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:18.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:18 smithi028 ceph-mon[29291]: pgmap v481: 158 pgs: 158 active+clean; 486 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 49 MiB/s wr, 777 op/s 2024-09-10T18:32:18.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:18 smithi028 ceph-mon[29291]: osdmap e250: 8 total, 8 up, 8 in 2024-09-10T18:32:18.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:18 smithi028 ceph-mon[39220]: pgmap v481: 158 pgs: 158 active+clean; 486 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 49 MiB/s wr, 777 op/s 2024-09-10T18:32:18.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:18 smithi028 ceph-mon[39220]: osdmap e250: 8 total, 8 up, 8 in 2024-09-10T18:32:18.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:18 smithi204 ceph-mon[32357]: pgmap v481: 158 pgs: 158 active+clean; 486 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 49 MiB/s wr, 777 op/s 2024-09-10T18:32:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:18 smithi204 ceph-mon[32357]: osdmap e250: 8 total, 8 up, 8 in 2024-09-10T18:32:18.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:18.381 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:18.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:18.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:18.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:18.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:18.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:18.625 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:18.656 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:18.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:18.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:18.758 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:18.788 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:18.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:18.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:18.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:18.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:18.932 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:18.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:19.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:19.034 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 6 16 18712 18696 194.722 164.125 0.00408316 0.00512738 2024-09-10T18:32:19.051 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:32:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:19.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:19.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:19.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:19.220 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:19.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:19.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:19.321 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:19 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/980885945' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:32:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:19 smithi204 ceph-mon[32357]: from='client.25324 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:32:19.355 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:19.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:19.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:19.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:19.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:19.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:19 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/980885945' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:32:19.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:19 smithi028 ceph-mon[29291]: from='client.25324 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:32:19.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:19 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/980885945' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:32:19.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:19 smithi028 ceph-mon[39220]: from='client.25324 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:32:19.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:19.629 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:19.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:19.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:19.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:19.770 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:19.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:19.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:19.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:19.912 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:19.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:19.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:20.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:20.034 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 7 16 21460 21444 191.437 171.75 0.00406263 0.00521802 2024-09-10T18:32:20.056 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:20.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:20.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:20.157 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:20.186 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:20.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:20.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:20.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:20.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:20 smithi028 ceph-mon[29291]: pgmap v483: 158 pgs: 158 active+clean; 486 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 49 MiB/s wr, 777 op/s 2024-09-10T18:32:20.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:20 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:32:20.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:20 smithi028 ceph-mon[29291]: from='client.25324 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:32:20.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:20 smithi028 ceph-mon[29291]: osdmap e251: 8 total, 8 up, 8 in 2024-09-10T18:32:20.290 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:20 smithi028 ceph-mon[39220]: pgmap v483: 158 pgs: 158 active+clean; 486 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 49 MiB/s wr, 777 op/s 2024-09-10T18:32:20.290 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:20 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:32:20.290 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:20 smithi028 ceph-mon[39220]: from='client.25324 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:32:20.290 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:20 smithi028 ceph-mon[39220]: osdmap e251: 8 total, 8 up, 8 in 2024-09-10T18:32:20.318 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:20.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:20.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:20.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:20 smithi204 ceph-mon[32357]: pgmap v483: 158 pgs: 158 active+clean; 486 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 49 MiB/s wr, 777 op/s 2024-09-10T18:32:20.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:20 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:32:20.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:20 smithi204 ceph-mon[32357]: from='client.25324 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:32:20.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:20 smithi204 ceph-mon[32357]: osdmap e251: 8 total, 8 up, 8 in 2024-09-10T18:32:20.420 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:20.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:20.453 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:20.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:20.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:20.580 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:20.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:20.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:20.682 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:20.714 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:20.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:20.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:20.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:20.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:20.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:20.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:20.960 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:20.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:20.992 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:20.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:21.034 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 8 15 23760 23745 185.483 143.812 0.00750231 0.00538481 2024-09-10T18:32:21.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:21.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:21.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:21.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:21.174 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:32:21.174 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:32:21.211 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:32:21.212 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:32:21.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:21.249 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:32:21.250 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:32:21.265 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:21.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:21.288 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:32:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:32:21.319 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:32:21.319 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:32:21.351 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:32:21.351 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:32:21.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:21.381 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:32:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:32:21.399 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:21.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:21.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:21.417 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:32:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:32:21.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:21.503 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:32:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:32:21] "GET /metrics HTTP/1.1" 200 224195 "" "Prometheus/2.43.0" 2024-09-10T18:32:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:21.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:21.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:21.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:21.683 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:21.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:21.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:21.785 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:21.816 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:21.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:21.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:21.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:21.956 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:21.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:21.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:22.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:21 smithi028 ceph-mon[39220]: pgmap v485: 158 pgs: 158 active+clean; 1.3 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 172 MiB/s wr, 2.75k op/s 2024-09-10T18:32:22.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:21 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/524397759' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:32:22.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:21 smithi028 ceph-mon[29291]: pgmap v485: 158 pgs: 158 active+clean; 1.3 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 172 MiB/s wr, 2.75k op/s 2024-09-10T18:32:22.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:21 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/524397759' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:32:22.034 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 9 15 26122 26107 181.275 147.625 0.00745321 0.00551162 2024-09-10T18:32:22.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:22.091 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:22.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:22.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:21 smithi204 ceph-mon[32357]: pgmap v485: 158 pgs: 158 active+clean; 1.3 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 172 MiB/s wr, 2.75k op/s 2024-09-10T18:32:22.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:21 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/524397759' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:32:22.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:22.230 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:22.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:22.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:22.331 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:22.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:22.369 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:22.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:22.470 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:22.502 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:22.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:22.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:22.575 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:32:22.575 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-10T18:32:22.576 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-10T18:32:22.576 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-10T18:32:22.576 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:32:22.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:32:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:32:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:32:22.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:22.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:22.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:22.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:22.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:22.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:22.885 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:22.924 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:22.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:22.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:23.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:23.034 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 10 16 28438 28422 177.615 144.688 0.00610604 0.00562452 2024-09-10T18:32:23.062 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:23.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:23.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:23.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:23.202 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:23.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:23.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:23.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:23.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:23.481 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:23.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:23.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:23.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:23.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:23.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:23.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-10T18:32:23.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:23.763 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:23.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:23.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:23.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:23.907 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:23.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:23.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:24.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:24.034 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 11 15 30499 30484 173.181 128.875 0.00755661 0.00576848 2024-09-10T18:32:24.043 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:24.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:24.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:24.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:24.186 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:24.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:24.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:24.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:24.325 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:24.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:24.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:24.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:32:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:24 smithi204 ceph-mon[32357]: pgmap v486: 158 pgs: 158 active+clean; 1.3 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 150 MiB/s wr, 2.40k op/s 2024-09-10T18:32:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:24 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:32:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:32:24.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:24 smithi028 ceph-mon[29291]: pgmap v486: 158 pgs: 158 active+clean; 1.3 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 150 MiB/s wr, 2.40k op/s 2024-09-10T18:32:24.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:24 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:32:24.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:32:24.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:24 smithi028 ceph-mon[39220]: pgmap v486: 158 pgs: 158 active+clean; 1.3 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 150 MiB/s wr, 2.40k op/s 2024-09-10T18:32:24.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:24 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:32:24.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:32:24.427 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:24.448 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:24.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:24.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:24.550 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:24.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:24.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:24.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:24.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:24.704 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:24.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:24.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:24.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:24.849 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:24.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:24.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:32:24 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:32:24.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:24.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:24.989 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:24.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:25.034 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 12 16 32370 32354 168.488 116.875 0.00862639 0.00593039 2024-09-10T18:32:25.090 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:25.126 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:25.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:25.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:25.228 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:25.259 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:25.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:25.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:25.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:25.393 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:25.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:25.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:25.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:25.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:25.534 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:25.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:25.635 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:25.668 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:25.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:25.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:25.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:25.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:25.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:25.911 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:25.947 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:25.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:25.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:26.034 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 13 16 34424 34408 165.402 128.375 0.00924786 0.00604007 2024-09-10T18:32:26.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:26.090 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:26.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:26.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:26.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:26.224 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:26.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:26.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:26.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:26 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:26.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:26 smithi028 ceph-mon[29291]: pgmap v487: 158 pgs: 158 active+clean; 1.6 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 149 MiB/s wr, 2.38k op/s 2024-09-10T18:32:26.291 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:26 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:26.291 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:26 smithi028 ceph-mon[39220]: pgmap v487: 158 pgs: 158 active+clean; 1.6 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 149 MiB/s wr, 2.38k op/s 2024-09-10T18:32:26.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:26 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:26 smithi204 ceph-mon[32357]: pgmap v487: 158 pgs: 158 active+clean; 1.6 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 149 MiB/s wr, 2.38k op/s 2024-09-10T18:32:26.365 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:26.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:26.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:26.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:26.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:26.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:26.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:26.608 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:26.630 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:26.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:26.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:26.732 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:26.767 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:26.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:26.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:26.869 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:26.902 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:26.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:26.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:27.003 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:27.033 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:27.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:27.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:27.034 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 14 16 36259 36243 161.779 114.688 0.00886156 0.00617456 2024-09-10T18:32:27.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:27.164 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:27.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:27.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:27.298 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:27.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:27.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:27 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:32:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:27 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:32:27.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:27.422 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:27.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:27.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:27 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:32:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:27 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:32:27.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:27 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:32:27.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:27 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:32:27.524 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:27.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:27.557 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:27.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:27.658 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:27.692 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:27.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:27.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:27.794 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:27.823 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:27.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:27.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:27.925 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:27.959 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:27.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:27.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:28.035 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 15 16 37597 37581 156.567 83.625 0.0019166 0.00635301 2024-09-10T18:32:28.061 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:28.082 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:28.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:28.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:28.184 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:28.218 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:28.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:28.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:28.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:28.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:28.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:28.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:28.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:28 smithi204 ceph-mon[32357]: pgmap v488: 158 pgs: 158 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 158 MiB/s wr, 2.53k op/s 2024-09-10T18:32:28.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:28 smithi204 ceph-mon[32357]: osdmap e252: 8 total, 8 up, 8 in 2024-09-10T18:32:28.445 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:28.482 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:28.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:28.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:28 smithi028 ceph-mon[29291]: pgmap v488: 158 pgs: 158 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 158 MiB/s wr, 2.53k op/s 2024-09-10T18:32:28.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:28 smithi028 ceph-mon[29291]: osdmap e252: 8 total, 8 up, 8 in 2024-09-10T18:32:28.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:28 smithi028 ceph-mon[39220]: pgmap v488: 158 pgs: 158 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 158 MiB/s wr, 2.53k op/s 2024-09-10T18:32:28.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:28 smithi028 ceph-mon[39220]: osdmap e252: 8 total, 8 up, 8 in 2024-09-10T18:32:28.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:28.623 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:28.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:28.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:28.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:28.764 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:28.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:28.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:28.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:28.906 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:28.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:28.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:29.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:29.036 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:29.037 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 16 16 39440 39424 153.963 115.188 0.00635892 0.00648888 2024-09-10T18:32:29.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:29.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:29.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:29.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:29.174 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:29.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:29.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:29.313 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:29.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:29.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:29.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:29.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:29.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:29.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:29.606 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:29.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:29.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:29.708 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:29.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:29.741 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:29.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:29.842 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:29.875 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:29.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:29.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:29.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:30.011 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:30.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:30.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:30.032 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:30.039 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 17 16 41432 41416 152.205 124.5 0.0181171 0.00656424 2024-09-10T18:32:30.113 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:30.145 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:30.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:30.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:30.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:30.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:30 smithi028 ceph-mon[29291]: pgmap v490: 158 pgs: 158 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 158 MiB/s wr, 2.54k op/s 2024-09-10T18:32:30.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:30 smithi028 ceph-mon[39220]: pgmap v490: 158 pgs: 158 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 158 MiB/s wr, 2.54k op/s 2024-09-10T18:32:30.290 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:30.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:30.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:30.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:30 smithi204 ceph-mon[32357]: pgmap v490: 158 pgs: 158 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 158 MiB/s wr, 2.54k op/s 2024-09-10T18:32:30.391 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:30.424 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:30.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:30.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:30.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:30.558 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:30.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:30.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:30.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:30.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:30.684 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:30.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:30.785 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:30.817 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:30.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:30.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:30.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:30.957 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:30.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:31.039 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 18 16 43196 43180 149.874 110.25 0.00512236 0.00666559 2024-09-10T18:32:31.059 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:31.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:32:31.087 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:31.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:31.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:31.189 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:31.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:31.220 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:31.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:31.322 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:31.323 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:32:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:32:31.356 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:31.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:31.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:31.388 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:31.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:31.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:31.506 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:31.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:31.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:31.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:31.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:31.721 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:32:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:32:31] "GET /metrics HTTP/1.1" 200 224368 "" "Prometheus/2.43.0" 2024-09-10T18:32:31.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:31.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:31.782 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:31.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:31.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:31.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:31.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:32.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:31 smithi028 ceph-mon[39220]: pgmap v491: 158 pgs: 158 active+clean; 2.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-10T18:32:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:31 smithi028 ceph-mon[29291]: pgmap v491: 158 pgs: 158 active+clean; 2.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-10T18:32:32.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:32.039 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 19 16 44983 44967 147.863 111.688 0.00731933 0.00675893 2024-09-10T18:32:32.065 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:32.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:32.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:32.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:31 smithi204 ceph-mon[32357]: pgmap v491: 158 pgs: 158 active+clean; 2.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-10T18:32:32.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:32.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:32.204 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:32.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:32.239 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":491,"stamp":"2024-09-10T18:32:30.969353+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2745964764,"num_objects":42087,"num_object_clones":0,"num_object_copies":126261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42087,"num_whiteouts":0,"num_read":1800,"num_read_kb":1552,"num_write":42607,"num_write_kb":2681067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43603,"ondisk_log_size":43603,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10352552,"kb_used_data":7893604,"kb_used_omap":19,"kb_used_meta":2458860,"kb_avail":739608664,"statfs":{"total":767960285184,"available":757359271936,"internally_reserved":0,"allocated":8083050496,"data_stored":8079586924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20033,"internal_metadata":2517873087},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,4,5,5,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1305608192,"num_objects":19922,"num_object_clones":0,"num_object_copies":59766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19922,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":19922,"num_write_kb":1275008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003417"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":242,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.056077+0000","last_change":"2024-09-10T18:30:59.501814+0000","last_active":"2024-09-10T18:32:19.056077+0000","last_peered":"2024-09-10T18:32:19.056077+0000","last_clean":"2024-09-10T18:32:19.056077+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:32:19.056077+0000","last_undegraded":"2024-09-10T18:32:19.056077+0000","last_fullsized":"2024-09-10T18:32:19.056077+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:59.501735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:30:59.501735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010428556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":281,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.043817+0000","last_change":"2024-09-10T18:31:42.555975+0000","last_active":"2024-09-10T18:32:19.043817+0000","last_peered":"2024-09-10T18:32:19.043817+0000","last_clean":"2024-09-10T18:32:19.043817+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:32:19.043817+0000","last_undegraded":"2024-09-10T18:32:19.043817+0000","last_fullsized":"2024-09-10T18:32:19.043817+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:31:42.555913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:42.555913+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:44.837983+0000","scrub_duration":0.0021762769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":247,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.046399+0000","last_change":"2024-09-10T18:30:48.481468+0000","last_active":"2024-09-10T18:32:19.046399+0000","last_peered":"2024-09-10T18:32:19.046399+0000","last_clean":"2024-09-10T18:32:19.046399+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:32:19.046399+0000","last_undegraded":"2024-09-10T18:32:19.046399+0000","last_fullsized":"2024-09-10T18:32:19.046399+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:48.481378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:48.481378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060252910000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":245,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.044331+0000","last_change":"2024-09-10T18:31:09.486730+0000","last_active":"2024-09-10T18:32:19.044331+0000","last_peered":"2024-09-10T18:32:19.044331+0000","last_clean":"2024-09-10T18:32:19.044331+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:32:19.044331+0000","last_undegraded":"2024-09-10T18:32:19.044331+0000","last_fullsized":"2024-09-10T18:32:19.044331+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:09.486379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:31:09.486379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:30.435919+0000","scrub_duration":0.010976669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":254,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.052647+0000","last_change":"2024-09-10T18:31:54.788489+0000","last_active":"2024-09-10T18:32:19.052647+0000","last_peered":"2024-09-10T18:32:19.052647+0000","last_clean":"2024-09-10T18:32:19.052647+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:32:19.052647+0000","last_undegraded":"2024-09-10T18:32:19.052647+0000","last_fullsized":"2024-09-10T18:32:19.052647+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:54.788436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:54.788436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:07.067753+0000","scrub_duration":0.0013008939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":260,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.053440+0000","last_change":"2024-09-10T18:32:00.727061+0000","last_active":"2024-09-10T18:32:19.053440+0000","last_peered":"2024-09-10T18:32:19.053440+0000","last_clean":"2024-09-10T18:32:19.053440+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:32:19.053440+0000","last_undegraded":"2024-09-10T18:32:19.053440+0000","last_fullsized":"2024-09-10T18:32:19.053440+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.727002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.727002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:29.752529+0000","scrub_duration":0.0014900709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'34","reported_seq":291,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.053089+0000","last_change":"2024-09-10T18:30:48.720386+0000","last_active":"2024-09-10T18:32:19.053089+0000","last_peered":"2024-09-10T18:32:19.053089+0000","last_clean":"2024-09-10T18:32:19.053089+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:32:19.053089+0000","last_undegraded":"2024-09-10T18:32:19.053089+0000","last_fullsized":"2024-09-10T18:32:19.053089+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'22","last_scrub_stamp":"2024-09-10T18:30:48.720296+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:30:48.720296+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037509370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":257,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.043809+0000","last_change":"2024-09-10T18:31:48.634526+0000","last_active":"2024-09-10T18:32:19.043809+0000","last_peered":"2024-09-10T18:32:19.043809+0000","last_clean":"2024-09-10T18:32:19.043809+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:32:19.043809+0000","last_undegraded":"2024-09-10T18:32:19.043809+0000","last_fullsized":"2024-09-10T18:32:19.043809+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:48.634465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.634465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:14.831892+0000","scrub_duration":0.0012201320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":262,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.050497+0000","last_change":"2024-09-10T18:31:54.606380+0000","last_active":"2024-09-10T18:32:19.050497+0000","last_peered":"2024-09-10T18:32:19.050497+0000","last_clean":"2024-09-10T18:32:19.050497+0000","last_became_active":"2024-09-10T18:26:27.644961+0000","last_became_peered":"2024-09-10T18:26:27.644961+0000","last_unstale":"2024-09-10T18:32:19.050497+0000","last_undegraded":"2024-09-10T18:32:19.050497+0000","last_fullsized":"2024-09-10T18:32:19.050497+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:54.606321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.606321+0000","last_clean_scrub_stamp":"2024-09-10T18:31:54.606321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:56.359072+0000","scrub_duration":0.0016163989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'15","reported_seq":274,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.044577+0000","last_change":"2024-09-10T18:30:58.249859+0000","last_active":"2024-09-10T18:32:19.044577+0000","last_peered":"2024-09-10T18:32:19.044577+0000","last_clean":"2024-09-10T18:32:19.044577+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:32:19.044577+0000","last_undegraded":"2024-09-10T18:32:19.044577+0000","last_fullsized":"2024-09-10T18:32:19.044577+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:30:58.249769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:58.249769+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056195170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":258,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048700+0000","last_change":"2024-09-10T18:31:58.589684+0000","last_active":"2024-09-10T18:32:19.048700+0000","last_peered":"2024-09-10T18:32:19.048700+0000","last_clean":"2024-09-10T18:32:19.048700+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:32:19.048700+0000","last_undegraded":"2024-09-10T18:32:19.048700+0000","last_fullsized":"2024-09-10T18:32:19.048700+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_clean_scrub_stamp":"2024-09-10T18:31:58.589631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:24.982578+0000","scrub_duration":0.0011763940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":246,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.053500+0000","last_change":"2024-09-10T18:30:56.456172+0000","last_active":"2024-09-10T18:32:19.053500+0000","last_peered":"2024-09-10T18:32:19.053500+0000","last_clean":"2024-09-10T18:32:19.053500+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:32:19.053500+0000","last_undegraded":"2024-09-10T18:32:19.053500+0000","last_fullsized":"2024-09-10T18:32:19.053500+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:56.456087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:56.456087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081476819999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":262,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.049260+0000","last_change":"2024-09-10T18:32:00.582011+0000","last_active":"2024-09-10T18:32:19.049260+0000","last_peered":"2024-09-10T18:32:19.049260+0000","last_clean":"2024-09-10T18:32:19.049260+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:32:19.049260+0000","last_undegraded":"2024-09-10T18:32:19.049260+0000","last_fullsized":"2024-09-10T18:32:19.049260+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.581959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:17.464205+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.581959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:29.570654+0000","scrub_duration":0.0012934120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":286,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.044314+0000","last_change":"2024-09-10T18:30:59.295764+0000","last_active":"2024-09-10T18:32:19.044314+0000","last_peered":"2024-09-10T18:32:19.044314+0000","last_clean":"2024-09-10T18:32:19.044314+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:32:19.044314+0000","last_undegraded":"2024-09-10T18:32:19.044314+0000","last_fullsized":"2024-09-10T18:32:19.044314+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'7","last_scrub_stamp":"2024-09-10T18:30:59.295710+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:30:59.295710+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032984920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":257,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048017+0000","last_change":"2024-09-10T18:32:01.525753+0000","last_active":"2024-09-10T18:32:19.048017+0000","last_peered":"2024-09-10T18:32:19.048017+0000","last_clean":"2024-09-10T18:32:19.048017+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:32:19.048017+0000","last_undegraded":"2024-09-10T18:32:19.048017+0000","last_fullsized":"2024-09-10T18:32:19.048017+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:01.525701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:01.525701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:24.525312+0000","scrub_duration":0.001223902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":249,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048425+0000","last_change":"2024-09-10T18:31:57.628816+0000","last_active":"2024-09-10T18:32:19.048425+0000","last_peered":"2024-09-10T18:32:19.048425+0000","last_clean":"2024-09-10T18:32:19.048425+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:32:19.048425+0000","last_undegraded":"2024-09-10T18:32:19.048425+0000","last_fullsized":"2024-09-10T18:32:19.048425+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:57.628758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.628758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:06.984921+0000","scrub_duration":0.0013180119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":255,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.045000+0000","last_change":"2024-09-10T18:31:57.658865+0000","last_active":"2024-09-10T18:32:19.045000+0000","last_peered":"2024-09-10T18:32:19.045000+0000","last_clean":"2024-09-10T18:32:19.045000+0000","last_became_active":"2024-09-10T18:26:27.650774+0000","last_became_peered":"2024-09-10T18:26:27.650774+0000","last_unstale":"2024-09-10T18:32:19.045000+0000","last_undegraded":"2024-09-10T18:32:19.045000+0000","last_fullsized":"2024-09-10T18:32:19.045000+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:57.658808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.658808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:12.353198+0000","scrub_duration":0.001743421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":242,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.056164+0000","last_change":"2024-09-10T18:31:53.332936+0000","last_active":"2024-09-10T18:32:19.056164+0000","last_peered":"2024-09-10T18:32:19.056164+0000","last_clean":"2024-09-10T18:32:19.056164+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:32:19.056164+0000","last_undegraded":"2024-09-10T18:32:19.056164+0000","last_fullsized":"2024-09-10T18:32:19.056164+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:53.332881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:53.332881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:08.279926+0000","scrub_duration":0.0014944929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"240'27","reported_seq":312,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.050733+0000","last_change":"2024-09-10T18:31:55.802263+0000","last_active":"2024-09-10T18:32:19.050733+0000","last_peered":"2024-09-10T18:32:19.050733+0000","last_clean":"2024-09-10T18:32:19.050733+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:32:19.050733+0000","last_undegraded":"2024-09-10T18:32:19.050733+0000","last_fullsized":"2024-09-10T18:32:19.050733+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:31:55.802206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:55.802206+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:00.946750+0000","scrub_duration":0.0019229290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":257,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.046781+0000","last_change":"2024-09-10T18:31:54.614766+0000","last_active":"2024-09-10T18:32:19.046781+0000","last_peered":"2024-09-10T18:32:19.046781+0000","last_clean":"2024-09-10T18:32:19.046781+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:32:19.046781+0000","last_undegraded":"2024-09-10T18:32:19.046781+0000","last_fullsized":"2024-09-10T18:32:19.046781+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:54.614707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:54.614707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:57.730057+0000","scrub_duration":0.0013384510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":254,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.055845+0000","last_change":"2024-09-10T18:31:53.754090+0000","last_active":"2024-09-10T18:32:19.055845+0000","last_peered":"2024-09-10T18:32:19.055845+0000","last_clean":"2024-09-10T18:32:19.055845+0000","last_became_active":"2024-09-10T18:26:27.640035+0000","last_became_peered":"2024-09-10T18:26:27.640035+0000","last_unstale":"2024-09-10T18:32:19.055845+0000","last_undegraded":"2024-09-10T18:32:19.055845+0000","last_fullsized":"2024-09-10T18:32:19.055845+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:53.754032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:53.754032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:17.934661+0000","scrub_duration":0.0016988699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":251,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.044699+0000","last_change":"2024-09-10T18:31:03.350685+0000","last_active":"2024-09-10T18:32:19.044699+0000","last_peered":"2024-09-10T18:32:19.044699+0000","last_clean":"2024-09-10T18:32:19.044699+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:32:19.044699+0000","last_undegraded":"2024-09-10T18:32:19.044699+0000","last_fullsized":"2024-09-10T18:32:19.044699+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:03.350631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:31:03.350631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016656491999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'23","reported_seq":278,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.052878+0000","last_change":"2024-09-10T18:31:06.712809+0000","last_active":"2024-09-10T18:32:19.052878+0000","last_peered":"2024-09-10T18:32:19.052878+0000","last_clean":"2024-09-10T18:32:19.052878+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:32:19.052878+0000","last_undegraded":"2024-09-10T18:32:19.052878+0000","last_fullsized":"2024-09-10T18:32:19.052878+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'17","last_scrub_stamp":"2024-09-10T18:31:06.712724+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:31:06.712724+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017994977999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":247,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.044613+0000","last_change":"2024-09-10T18:31:12.515598+0000","last_active":"2024-09-10T18:32:19.044613+0000","last_peered":"2024-09-10T18:32:19.044613+0000","last_clean":"2024-09-10T18:32:19.044613+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:32:19.044613+0000","last_undegraded":"2024-09-10T18:32:19.044613+0000","last_fullsized":"2024-09-10T18:32:19.044613+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:12.515540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:12.515540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:28.271069+0000","scrub_duration":0.0011808949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":247,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048222+0000","last_change":"2024-09-10T18:30:48.487103+0000","last_active":"2024-09-10T18:32:19.048222+0000","last_peered":"2024-09-10T18:32:19.048222+0000","last_clean":"2024-09-10T18:32:19.048222+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:32:19.048222+0000","last_undegraded":"2024-09-10T18:32:19.048222+0000","last_fullsized":"2024-09-10T18:32:19.048222+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:30:48.487005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:48.487005+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023474720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":261,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.046621+0000","last_change":"2024-09-10T18:31:04.526514+0000","last_active":"2024-09-10T18:32:19.046621+0000","last_peered":"2024-09-10T18:32:19.046621+0000","last_clean":"2024-09-10T18:32:19.046621+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:32:19.046621+0000","last_undegraded":"2024-09-10T18:32:19.046621+0000","last_fullsized":"2024-09-10T18:32:19.046621+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:04.526424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:04.526424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013541001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'25","reported_seq":285,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.055554+0000","last_change":"2024-09-10T18:31:58.319117+0000","last_active":"2024-09-10T18:32:19.055554+0000","last_peered":"2024-09-10T18:32:19.055554+0000","last_clean":"2024-09-10T18:32:19.055554+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:32:19.055554+0000","last_undegraded":"2024-09-10T18:32:19.055554+0000","last_fullsized":"2024-09-10T18:32:19.055554+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:31:58.319058+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:31:58.319058+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:07.968135+0000","scrub_duration":0.0020563980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"217'11","reported_seq":300,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:27.048166+0000","last_change":"2024-09-10T18:31:49.104801+0000","last_active":"2024-09-10T18:32:27.048166+0000","last_peered":"2024-09-10T18:32:27.048166+0000","last_clean":"2024-09-10T18:32:27.048166+0000","last_became_active":"2024-09-10T18:26:27.652489+0000","last_became_peered":"2024-09-10T18:26:27.652489+0000","last_unstale":"2024-09-10T18:32:27.048166+0000","last_undegraded":"2024-09-10T18:32:27.048166+0000","last_fullsized":"2024-09-10T18:32:27.048166+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11","last_scrub_stamp":"2024-09-10T18:31:49.104746+0000","last_deep_scrub":"217'11","last_deep_scrub_stamp":"2024-09-10T18:30:03.161985+0000","last_clean_scrub_stamp":"2024-09-10T18:31:49.104746+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:51.048090+0000","scrub_duration":0.0013830120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"252'2684","reported_seq":2698,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:30.588396+0000","last_change":"2024-09-10T18:32:03.385708+0000","last_active":"2024-09-10T18:32:30.588396+0000","last_peered":"2024-09-10T18:32:30.588396+0000","last_clean":"2024-09-10T18:32:30.588396+0000","last_became_active":"2024-09-10T18:32:03.385428+0000","last_became_peered":"2024-09-10T18:32:03.385428+0000","last_unstale":"2024-09-10T18:32:30.588396+0000","last_undegraded":"2024-09-10T18:32:30.588396+0000","last_fullsized":"2024-09-10T18:32:30.588396+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":2684,"log_dups_size":0,"ondisk_log_size":2684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:03.115976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175898624,"num_objects":2684,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2684,"num_write_kb":171776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":168,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048292+0000","last_change":"2024-09-10T18:31:59.600055+0000","last_active":"2024-09-10T18:32:19.048292+0000","last_peered":"2024-09-10T18:32:19.048292+0000","last_clean":"2024-09-10T18:32:19.048292+0000","last_became_active":"2024-09-10T18:26:28.635249+0000","last_became_peered":"2024-09-10T18:26:28.635249+0000","last_unstale":"2024-09-10T18:32:19.048292+0000","last_undegraded":"2024-09-10T18:32:19.048292+0000","last_fullsized":"2024-09-10T18:32:19.048292+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:59.599999+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:59.599999+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:00.567313+0000","scrub_duration":0.001350027,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"240'34","reported_seq":312,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.047774+0000","last_change":"2024-09-10T18:31:11.374516+0000","last_active":"2024-09-10T18:32:19.047774+0000","last_peered":"2024-09-10T18:32:19.047774+0000","last_clean":"2024-09-10T18:32:19.047774+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:32:19.047774+0000","last_undegraded":"2024-09-10T18:32:19.047774+0000","last_fullsized":"2024-09-10T18:32:19.047774+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-09-10T18:31:11.373949+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:31:11.373949+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:28.044226+0000","scrub_duration":0.0054884399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":253,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.062156+0000","last_change":"2024-09-10T18:31:59.772883+0000","last_active":"2024-09-10T18:32:19.062156+0000","last_peered":"2024-09-10T18:32:19.062156+0000","last_clean":"2024-09-10T18:32:19.062156+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:32:19.062156+0000","last_undegraded":"2024-09-10T18:32:19.062156+0000","last_fullsized":"2024-09-10T18:32:19.062156+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:59.772816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:59.772816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:16.535228+0000","scrub_duration":0.0015225810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":389,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:24.891226+0000","last_change":"2024-09-10T18:31:55.634620+0000","last_active":"2024-09-10T18:32:24.891226+0000","last_peered":"2024-09-10T18:32:24.891226+0000","last_clean":"2024-09-10T18:32:24.891226+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:32:24.891226+0000","last_undegraded":"2024-09-10T18:32:24.891226+0000","last_fullsized":"2024-09-10T18:32:24.891226+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:31:55.634564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:55.634564+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:17.377608+0000","scrub_duration":0.001562346,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":246,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.050188+0000","last_change":"2024-09-10T18:30:59.381476+0000","last_active":"2024-09-10T18:32:19.050188+0000","last_peered":"2024-09-10T18:32:19.050188+0000","last_clean":"2024-09-10T18:32:19.050188+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:32:19.050188+0000","last_undegraded":"2024-09-10T18:32:19.050188+0000","last_fullsized":"2024-09-10T18:32:19.050188+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:59.381396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:30:59.381396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011130997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"252'2529","reported_seq":2543,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:29.841054+0000","last_change":"2024-09-10T18:32:03.379554+0000","last_active":"2024-09-10T18:32:29.841054+0000","last_peered":"2024-09-10T18:32:29.841054+0000","last_clean":"2024-09-10T18:32:29.841054+0000","last_became_active":"2024-09-10T18:32:03.379257+0000","last_became_peered":"2024-09-10T18:32:03.379257+0000","last_unstale":"2024-09-10T18:32:29.841054+0000","last_undegraded":"2024-09-10T18:32:29.841054+0000","last_fullsized":"2024-09-10T18:32:29.841054+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":2529,"log_dups_size":0,"ondisk_log_size":2529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:13.652481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165740544,"num_objects":2529,"num_object_clones":0,"num_object_copies":7587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2529,"num_write_kb":161856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":157,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.045155+0000","last_change":"2024-09-10T18:30:45.366763+0000","last_active":"2024-09-10T18:32:19.045155+0000","last_peered":"2024-09-10T18:32:19.045155+0000","last_clean":"2024-09-10T18:32:19.045155+0000","last_became_active":"2024-09-10T18:26:28.632744+0000","last_became_peered":"2024-09-10T18:26:28.632744+0000","last_unstale":"2024-09-10T18:32:19.045155+0000","last_undegraded":"2024-09-10T18:32:19.045155+0000","last_fullsized":"2024-09-10T18:32:19.045155+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:30:45.366615+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:30:45.366615+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013315966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":291,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.061820+0000","last_change":"2024-09-10T18:30:52.742276+0000","last_active":"2024-09-10T18:32:19.061820+0000","last_peered":"2024-09-10T18:32:19.061820+0000","last_clean":"2024-09-10T18:32:19.061820+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:32:19.061820+0000","last_undegraded":"2024-09-10T18:32:19.061820+0000","last_fullsized":"2024-09-10T18:32:19.061820+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-09-10T18:30:52.742191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:52.742191+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015070948000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":243,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.058092+0000","last_change":"2024-09-10T18:31:01.695940+0000","last_active":"2024-09-10T18:32:19.058092+0000","last_peered":"2024-09-10T18:32:19.058092+0000","last_clean":"2024-09-10T18:32:19.058092+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:32:19.058092+0000","last_undegraded":"2024-09-10T18:32:19.058092+0000","last_fullsized":"2024-09-10T18:32:19.058092+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:01.695706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.140922+0000","last_clean_scrub_stamp":"2024-09-10T18:31:01.695706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022117759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":247,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.043912+0000","last_change":"2024-09-10T18:31:04.334432+0000","last_active":"2024-09-10T18:32:19.043912+0000","last_peered":"2024-09-10T18:32:19.043912+0000","last_clean":"2024-09-10T18:32:19.043912+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:32:19.043912+0000","last_undegraded":"2024-09-10T18:32:19.043912+0000","last_fullsized":"2024-09-10T18:32:19.043912+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:31:04.334337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012632089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":235,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.044502+0000","last_change":"2024-09-10T18:30:03.186000+0000","last_active":"2024-09-10T18:32:19.044502+0000","last_peered":"2024-09-10T18:32:19.044502+0000","last_clean":"2024-09-10T18:32:19.044502+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:32:19.044502+0000","last_undegraded":"2024-09-10T18:32:19.044502+0000","last_fullsized":"2024-09-10T18:32:19.044502+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:03.185938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:03.185938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007381528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"252'2604","reported_seq":2618,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:30.569202+0000","last_change":"2024-09-10T18:32:03.554760+0000","last_active":"2024-09-10T18:32:30.569202+0000","last_peered":"2024-09-10T18:32:30.569202+0000","last_clean":"2024-09-10T18:32:30.569202+0000","last_became_active":"2024-09-10T18:32:03.554400+0000","last_became_peered":"2024-09-10T18:32:03.554400+0000","last_unstale":"2024-09-10T18:32:30.569202+0000","last_undegraded":"2024-09-10T18:32:30.569202+0000","last_fullsized":"2024-09-10T18:32:30.569202+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":2604,"log_dups_size":0,"ondisk_log_size":2604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:30.476745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170655744,"num_objects":2604,"num_object_clones":0,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2604,"num_write_kb":166656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":148,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.052743+0000","last_change":"2024-09-10T18:30:50.806220+0000","last_active":"2024-09-10T18:32:19.052743+0000","last_peered":"2024-09-10T18:32:19.052743+0000","last_clean":"2024-09-10T18:32:19.052743+0000","last_became_active":"2024-09-10T18:26:18.091896+0000","last_became_peered":"2024-09-10T18:26:18.091896+0000","last_unstale":"2024-09-10T18:32:19.052743+0000","last_undegraded":"2024-09-10T18:32:19.052743+0000","last_fullsized":"2024-09-10T18:32:19.052743+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:30:50.805972+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:30:50.805972+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087755490000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"240'28","reported_seq":298,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.044025+0000","last_change":"2024-09-10T18:31:46.592515+0000","last_active":"2024-09-10T18:32:19.044025+0000","last_peered":"2024-09-10T18:32:19.044025+0000","last_clean":"2024-09-10T18:32:19.044025+0000","last_became_active":"2024-09-10T18:26:27.651364+0000","last_became_peered":"2024-09-10T18:26:27.651364+0000","last_unstale":"2024-09-10T18:32:19.044025+0000","last_undegraded":"2024-09-10T18:32:19.044025+0000","last_fullsized":"2024-09-10T18:32:19.044025+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'28","last_scrub_stamp":"2024-09-10T18:31:46.592456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:46.592456+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:58.738883+0000","scrub_duration":0.0019972359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":242,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.055767+0000","last_change":"2024-09-10T18:30:04.443950+0000","last_active":"2024-09-10T18:32:19.055767+0000","last_peered":"2024-09-10T18:32:19.055767+0000","last_clean":"2024-09-10T18:32:19.055767+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:32:19.055767+0000","last_undegraded":"2024-09-10T18:32:19.055767+0000","last_fullsized":"2024-09-10T18:32:19.055767+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:04.443859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:30:04.443859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.167471064,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":380,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:27.037290+0000","last_change":"2024-09-10T18:30:56.702968+0000","last_active":"2024-09-10T18:32:27.037290+0000","last_peered":"2024-09-10T18:32:27.037290+0000","last_clean":"2024-09-10T18:32:27.037290+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:32:27.037290+0000","last_undegraded":"2024-09-10T18:32:27.037290+0000","last_fullsized":"2024-09-10T18:32:27.037290+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:30:56.702886+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:30:56.702886+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084155429999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":255,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048556+0000","last_change":"2024-09-10T18:31:06.527936+0000","last_active":"2024-09-10T18:32:19.048556+0000","last_peered":"2024-09-10T18:32:19.048556+0000","last_clean":"2024-09-10T18:32:19.048556+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:32:19.048556+0000","last_undegraded":"2024-09-10T18:32:19.048556+0000","last_fullsized":"2024-09-10T18:32:19.048556+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:31:06.527847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088842569999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"252'2531","reported_seq":2545,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:29.897289+0000","last_change":"2024-09-10T18:32:03.384690+0000","last_active":"2024-09-10T18:32:29.897289+0000","last_peered":"2024-09-10T18:32:29.897289+0000","last_clean":"2024-09-10T18:32:29.897289+0000","last_became_active":"2024-09-10T18:32:03.384288+0000","last_became_peered":"2024-09-10T18:32:03.384288+0000","last_unstale":"2024-09-10T18:32:29.897289+0000","last_undegraded":"2024-09-10T18:32:29.897289+0000","last_fullsized":"2024-09-10T18:32:29.897289+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":2531,"log_dups_size":0,"ondisk_log_size":2531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:04.684802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165871616,"num_objects":2531,"num_object_clones":0,"num_object_copies":7593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2531,"num_write_kb":161984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":207,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.049884+0000","last_change":"2024-09-10T18:31:42.153381+0000","last_active":"2024-09-10T18:32:19.049884+0000","last_peered":"2024-09-10T18:32:19.049884+0000","last_clean":"2024-09-10T18:32:19.049884+0000","last_became_active":"2024-09-10T18:26:28.632849+0000","last_became_peered":"2024-09-10T18:26:28.632849+0000","last_unstale":"2024-09-10T18:32:19.049884+0000","last_undegraded":"2024-09-10T18:32:19.049884+0000","last_fullsized":"2024-09-10T18:32:19.049884+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:42.153325+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:42.153325+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:49.020619+0000","scrub_duration":0.001252305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":294,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048370+0000","last_change":"2024-09-10T18:31:00.546611+0000","last_active":"2024-09-10T18:32:19.048370+0000","last_peered":"2024-09-10T18:32:19.048370+0000","last_clean":"2024-09-10T18:32:19.048370+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:32:19.048370+0000","last_undegraded":"2024-09-10T18:32:19.048370+0000","last_fullsized":"2024-09-10T18:32:19.048370+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'15","last_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:31:00.545612+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0072351619999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":242,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.050567+0000","last_change":"2024-09-10T18:30:46.332919+0000","last_active":"2024-09-10T18:32:19.050567+0000","last_peered":"2024-09-10T18:32:19.050567+0000","last_clean":"2024-09-10T18:32:19.050567+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:32:19.050567+0000","last_undegraded":"2024-09-10T18:32:19.050567+0000","last_fullsized":"2024-09-10T18:32:19.050567+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:46.332861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:30:46.332861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061624269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"146'10","reported_seq":387,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:27.033409+0000","last_change":"2024-09-10T18:31:05.540491+0000","last_active":"2024-09-10T18:32:27.033409+0000","last_peered":"2024-09-10T18:32:27.033409+0000","last_clean":"2024-09-10T18:32:27.033409+0000","last_became_active":"2024-09-10T18:26:27.639032+0000","last_became_peered":"2024-09-10T18:26:27.639032+0000","last_unstale":"2024-09-10T18:32:27.033409+0000","last_undegraded":"2024-09-10T18:32:27.033409+0000","last_fullsized":"2024-09-10T18:32:27.033409+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'10","last_scrub_stamp":"2024-09-10T18:31:05.540403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:05.540403+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003060657,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":246,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.053687+0000","last_change":"2024-09-10T18:31:43.338883+0000","last_active":"2024-09-10T18:32:19.053687+0000","last_peered":"2024-09-10T18:32:19.053687+0000","last_clean":"2024-09-10T18:32:19.053687+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:32:19.053687+0000","last_undegraded":"2024-09-10T18:32:19.053687+0000","last_fullsized":"2024-09-10T18:32:19.053687+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:43.338816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:43.338816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:08.069598+0000","scrub_duration":0.0011905869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"252'2612","reported_seq":2626,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:29.860953+0000","last_change":"2024-09-10T18:32:03.555680+0000","last_active":"2024-09-10T18:32:29.860953+0000","last_peered":"2024-09-10T18:32:29.860953+0000","last_clean":"2024-09-10T18:32:29.860953+0000","last_became_active":"2024-09-10T18:32:03.554693+0000","last_became_peered":"2024-09-10T18:32:03.554693+0000","last_unstale":"2024-09-10T18:32:29.860953+0000","last_undegraded":"2024-09-10T18:32:29.860953+0000","last_fullsized":"2024-09-10T18:32:29.860953+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":2612,"log_dups_size":0,"ondisk_log_size":2612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:03.469270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171180032,"num_objects":2612,"num_object_clones":0,"num_object_copies":7836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2612,"num_write_kb":167168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":207,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.049728+0000","last_change":"2024-09-10T18:31:03.017561+0000","last_active":"2024-09-10T18:32:19.049728+0000","last_peered":"2024-09-10T18:32:19.049728+0000","last_clean":"2024-09-10T18:32:19.049728+0000","last_became_active":"2024-09-10T18:26:28.635542+0000","last_became_peered":"2024-09-10T18:26:28.635542+0000","last_unstale":"2024-09-10T18:32:19.049728+0000","last_undegraded":"2024-09-10T18:32:19.049728+0000","last_fullsized":"2024-09-10T18:32:19.049728+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:03.017452+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:03.017452+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:30.278613+0000","scrub_duration":0.0066732069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"240'29","reported_seq":301,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.055937+0000","last_change":"2024-09-10T18:31:10.474539+0000","last_active":"2024-09-10T18:32:19.055937+0000","last_peered":"2024-09-10T18:32:19.055937+0000","last_clean":"2024-09-10T18:32:19.055937+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:32:19.055937+0000","last_undegraded":"2024-09-10T18:32:19.055937+0000","last_fullsized":"2024-09-10T18:32:19.055937+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-09-10T18:31:10.474434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:10.474434+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:37.841342+0000","scrub_duration":0.0058438149999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":251,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.047146+0000","last_change":"2024-09-10T18:31:52.607842+0000","last_active":"2024-09-10T18:32:19.047146+0000","last_peered":"2024-09-10T18:32:19.047146+0000","last_clean":"2024-09-10T18:32:19.047146+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:32:19.047146+0000","last_undegraded":"2024-09-10T18:32:19.047146+0000","last_fullsized":"2024-09-10T18:32:19.047146+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:52.607787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:52.607787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:20.479043+0000","scrub_duration":0.0014830799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":197,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048773+0000","last_change":"2024-09-10T18:31:48.149427+0000","last_active":"2024-09-10T18:32:19.048773+0000","last_peered":"2024-09-10T18:32:19.048773+0000","last_clean":"2024-09-10T18:32:19.048773+0000","last_became_active":"2024-09-10T18:26:27.642063+0000","last_became_peered":"2024-09-10T18:26:27.642063+0000","last_unstale":"2024-09-10T18:32:19.048773+0000","last_undegraded":"2024-09-10T18:32:19.048773+0000","last_fullsized":"2024-09-10T18:32:19.048773+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:48.149372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.149372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:54.678387+0000","scrub_duration":0.001174172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":252,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.046331+0000","last_change":"2024-09-10T18:32:01.852929+0000","last_active":"2024-09-10T18:32:19.046331+0000","last_peered":"2024-09-10T18:32:19.046331+0000","last_clean":"2024-09-10T18:32:19.046331+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:32:19.046331+0000","last_undegraded":"2024-09-10T18:32:19.046331+0000","last_fullsized":"2024-09-10T18:32:19.046331+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:01.852861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:32:01.852861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:08.383658+0000","scrub_duration":0.0013400300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"252'2668","reported_seq":2682,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:30.830926+0000","last_change":"2024-09-10T18:32:03.382208+0000","last_active":"2024-09-10T18:32:30.830926+0000","last_peered":"2024-09-10T18:32:30.830926+0000","last_clean":"2024-09-10T18:32:30.830926+0000","last_became_active":"2024-09-10T18:32:03.381872+0000","last_became_peered":"2024-09-10T18:32:03.381872+0000","last_unstale":"2024-09-10T18:32:30.830926+0000","last_undegraded":"2024-09-10T18:32:30.830926+0000","last_fullsized":"2024-09-10T18:32:30.830926+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":2668,"log_dups_size":0,"ondisk_log_size":2668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:07.871489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174850048,"num_objects":2668,"num_object_clones":0,"num_object_copies":8004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2668,"num_write_kb":170752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":153,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048242+0000","last_change":"2024-09-10T18:31:56.629743+0000","last_active":"2024-09-10T18:32:19.048242+0000","last_peered":"2024-09-10T18:32:19.048242+0000","last_clean":"2024-09-10T18:32:19.048242+0000","last_became_active":"2024-09-10T18:26:18.094747+0000","last_became_peered":"2024-09-10T18:26:18.094747+0000","last_unstale":"2024-09-10T18:32:19.048242+0000","last_undegraded":"2024-09-10T18:32:19.048242+0000","last_fullsized":"2024-09-10T18:32:19.048242+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:56.629683+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:56.629683+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:22.298804+0000","scrub_duration":0.001774133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"240'41","reported_seq":366,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:27.039027+0000","last_change":"2024-09-10T18:31:11.323534+0000","last_active":"2024-09-10T18:32:27.039027+0000","last_peered":"2024-09-10T18:32:27.039027+0000","last_clean":"2024-09-10T18:32:27.039027+0000","last_became_active":"2024-09-10T18:23:10.067049+0000","last_became_peered":"2024-09-10T18:23:10.067049+0000","last_unstale":"2024-09-10T18:32:27.039027+0000","last_undegraded":"2024-09-10T18:32:27.039027+0000","last_fullsized":"2024-09-10T18:32:27.039027+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'29","last_scrub_stamp":"2024-09-10T18:31:11.323461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:11.323461+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0098705089999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":258,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.044467+0000","last_change":"2024-09-10T18:31:49.684883+0000","last_active":"2024-09-10T18:32:19.044467+0000","last_peered":"2024-09-10T18:32:19.044467+0000","last_clean":"2024-09-10T18:32:19.044467+0000","last_became_active":"2024-09-10T18:26:27.648690+0000","last_became_peered":"2024-09-10T18:26:27.648690+0000","last_unstale":"2024-09-10T18:32:19.044467+0000","last_undegraded":"2024-09-10T18:32:19.044467+0000","last_fullsized":"2024-09-10T18:32:19.044467+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:49.684824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:49.684824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:19.116090+0000","scrub_duration":0.0012508370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":267,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.049623+0000","last_change":"2024-09-10T18:31:04.391774+0000","last_active":"2024-09-10T18:32:19.049623+0000","last_peered":"2024-09-10T18:32:19.049623+0000","last_clean":"2024-09-10T18:32:19.049623+0000","last_became_active":"2024-09-10T18:26:27.647576+0000","last_became_peered":"2024-09-10T18:26:27.647576+0000","last_unstale":"2024-09-10T18:32:19.049623+0000","last_undegraded":"2024-09-10T18:32:19.049623+0000","last_fullsized":"2024-09-10T18:32:19.049623+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:31:04.391687+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:31:04.391687+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011656032,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":238,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.055146+0000","last_change":"2024-09-10T18:29:33.023742+0000","last_active":"2024-09-10T18:32:19.055146+0000","last_peered":"2024-09-10T18:32:19.055146+0000","last_clean":"2024-09-10T18:32:19.055146+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:32:19.055146+0000","last_undegraded":"2024-09-10T18:32:19.055146+0000","last_fullsized":"2024-09-10T18:32:19.055146+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:33.023658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:29:33.023658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024546590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":237,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.053199+0000","last_change":"2024-09-10T18:31:04.714213+0000","last_active":"2024-09-10T18:32:19.053199+0000","last_peered":"2024-09-10T18:32:19.053199+0000","last_clean":"2024-09-10T18:32:19.053199+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:32:19.053199+0000","last_undegraded":"2024-09-10T18:32:19.053199+0000","last_fullsized":"2024-09-10T18:32:19.053199+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:04.714112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:04.714112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:31.376995+0000","scrub_duration":0.0085368679999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"252'2599","reported_seq":2613,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:29.895930+0000","last_change":"2024-09-10T18:32:03.380691+0000","last_active":"2024-09-10T18:32:29.895930+0000","last_peered":"2024-09-10T18:32:29.895930+0000","last_clean":"2024-09-10T18:32:29.895930+0000","last_became_active":"2024-09-10T18:32:03.380391+0000","last_became_peered":"2024-09-10T18:32:03.380391+0000","last_unstale":"2024-09-10T18:32:29.895930+0000","last_undegraded":"2024-09-10T18:32:29.895930+0000","last_fullsized":"2024-09-10T18:32:29.895930+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":2599,"log_dups_size":0,"ondisk_log_size":2599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:05.417320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170328064,"num_objects":2599,"num_object_clones":0,"num_object_copies":7797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2599,"num_write_kb":166336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":299,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.047919+0000","last_change":"2024-09-10T18:31:57.474380+0000","last_active":"2024-09-10T18:32:19.047919+0000","last_peered":"2024-09-10T18:32:19.047919+0000","last_clean":"2024-09-10T18:32:19.047919+0000","last_became_active":"2024-09-10T18:26:18.095568+0000","last_became_peered":"2024-09-10T18:26:18.095568+0000","last_unstale":"2024-09-10T18:32:19.047919+0000","last_undegraded":"2024-09-10T18:32:19.047919+0000","last_fullsized":"2024-09-10T18:32:19.047919+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:57.474295+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.474295+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:12.518679+0000","scrub_duration":0.0015998690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"240'17","reported_seq":282,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.047985+0000","last_change":"2024-09-10T18:30:51.522994+0000","last_active":"2024-09-10T18:32:19.047985+0000","last_peered":"2024-09-10T18:32:19.047985+0000","last_clean":"2024-09-10T18:32:19.047985+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:32:19.047985+0000","last_undegraded":"2024-09-10T18:32:19.047985+0000","last_fullsized":"2024-09-10T18:32:19.047985+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-09-10T18:30:51.522555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:51.522555+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030428550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":268,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.054640+0000","last_change":"2024-09-10T18:31:58.763146+0000","last_active":"2024-09-10T18:32:19.054640+0000","last_peered":"2024-09-10T18:32:19.054640+0000","last_clean":"2024-09-10T18:32:19.054640+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:32:19.054640+0000","last_undegraded":"2024-09-10T18:32:19.054640+0000","last_fullsized":"2024-09-10T18:32:19.054640+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:58.763082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:02.567084+0000","last_clean_scrub_stamp":"2024-09-10T18:31:58.763082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:08.731645+0000","scrub_duration":0.0013124370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":280,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.050437+0000","last_change":"2024-09-10T18:31:54.779251+0000","last_active":"2024-09-10T18:32:19.050437+0000","last_peered":"2024-09-10T18:32:19.050437+0000","last_clean":"2024-09-10T18:32:19.050437+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:32:19.050437+0000","last_undegraded":"2024-09-10T18:32:19.050437+0000","last_fullsized":"2024-09-10T18:32:19.050437+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:31:54.779189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:54.779189+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:22.671479+0000","scrub_duration":0.0018921529999999999,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":257,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.053939+0000","last_change":"2024-09-10T18:31:10.933803+0000","last_active":"2024-09-10T18:32:19.053939+0000","last_peered":"2024-09-10T18:32:19.053939+0000","last_clean":"2024-09-10T18:32:19.053939+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:32:19.053939+0000","last_undegraded":"2024-09-10T18:32:19.053939+0000","last_fullsized":"2024-09-10T18:32:19.053939+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:10.933715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:31:10.933715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:38.397405+0000","scrub_duration":0.0061095990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":255,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048640+0000","last_change":"2024-09-10T18:32:00.501396+0000","last_active":"2024-09-10T18:32:19.048640+0000","last_peered":"2024-09-10T18:32:19.048640+0000","last_clean":"2024-09-10T18:32:19.048640+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:32:19.048640+0000","last_undegraded":"2024-09-10T18:32:19.048640+0000","last_fullsized":"2024-09-10T18:32:19.048640+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.501344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.501344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:03.841832+0000","scrub_duration":0.001367442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"252'2754","reported_seq":2768,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:30.860551+0000","last_change":"2024-09-10T18:32:03.555231+0000","last_active":"2024-09-10T18:32:30.860551+0000","last_peered":"2024-09-10T18:32:30.860551+0000","last_clean":"2024-09-10T18:32:30.860551+0000","last_became_active":"2024-09-10T18:32:03.554958+0000","last_became_peered":"2024-09-10T18:32:03.554958+0000","last_unstale":"2024-09-10T18:32:30.860551+0000","last_undegraded":"2024-09-10T18:32:30.860551+0000","last_fullsized":"2024-09-10T18:32:30.860551+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":2754,"log_dups_size":0,"ondisk_log_size":2754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:09.616039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180486144,"num_objects":2754,"num_object_clones":0,"num_object_copies":8262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2754,"num_write_kb":176256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":168,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.047167+0000","last_change":"2024-09-10T18:31:48.466710+0000","last_active":"2024-09-10T18:32:19.047167+0000","last_peered":"2024-09-10T18:32:19.047167+0000","last_clean":"2024-09-10T18:32:19.047167+0000","last_became_active":"2024-09-10T18:26:28.641791+0000","last_became_peered":"2024-09-10T18:26:28.641791+0000","last_unstale":"2024-09-10T18:32:19.047167+0000","last_undegraded":"2024-09-10T18:32:19.047167+0000","last_fullsized":"2024-09-10T18:32:19.047167+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:48.466654+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.466654+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:15.898353+0000","scrub_duration":0.001538871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":276,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.055629+0000","last_change":"2024-09-10T18:29:57.326700+0000","last_active":"2024-09-10T18:32:19.055629+0000","last_peered":"2024-09-10T18:32:19.055629+0000","last_clean":"2024-09-10T18:32:19.055629+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:32:19.055629+0000","last_undegraded":"2024-09-10T18:32:19.055629+0000","last_fullsized":"2024-09-10T18:32:19.055629+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:29:57.326613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:29:57.326613+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.138665703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":252,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.054190+0000","last_change":"2024-09-10T18:31:08.529508+0000","last_active":"2024-09-10T18:32:19.054190+0000","last_peered":"2024-09-10T18:32:19.054190+0000","last_clean":"2024-09-10T18:32:19.054190+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:32:19.054190+0000","last_undegraded":"2024-09-10T18:32:19.054190+0000","last_fullsized":"2024-09-10T18:32:19.054190+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:08.529424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:31:08.529424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0078874119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":920,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:28.644193+0000","last_change":"2024-09-10T18:31:13.536758+0000","last_active":"2024-09-10T18:32:28.644193+0000","last_peered":"2024-09-10T18:32:28.644193+0000","last_clean":"2024-09-10T18:32:28.644193+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:32:28.644193+0000","last_undegraded":"2024-09-10T18:32:28.644193+0000","last_fullsized":"2024-09-10T18:32:28.644193+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:31:13.536419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:13.536419+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:34.617675+0000","scrub_duration":0.0043736720000000003,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":651,"num_read_kb":646,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":241,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.050680+0000","last_change":"2024-09-10T18:30:47.337000+0000","last_active":"2024-09-10T18:32:19.050680+0000","last_peered":"2024-09-10T18:32:19.050680+0000","last_clean":"2024-09-10T18:32:19.050680+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:32:19.050680+0000","last_undegraded":"2024-09-10T18:32:19.050680+0000","last_fullsized":"2024-09-10T18:32:19.050680+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:47.336845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.771415+0000","last_clean_scrub_stamp":"2024-09-10T18:30:47.336845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051314819999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":245,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.046393+0000","last_change":"2024-09-10T18:31:57.719333+0000","last_active":"2024-09-10T18:32:19.046393+0000","last_peered":"2024-09-10T18:32:19.046393+0000","last_clean":"2024-09-10T18:32:19.046393+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:32:19.046393+0000","last_undegraded":"2024-09-10T18:32:19.046393+0000","last_fullsized":"2024-09-10T18:32:19.046393+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:57.719278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.719278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:25.197723+0000","scrub_duration":0.0011883760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"252'2593","reported_seq":2607,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:30.575790+0000","last_change":"2024-09-10T18:32:03.382727+0000","last_active":"2024-09-10T18:32:30.575790+0000","last_peered":"2024-09-10T18:32:30.575790+0000","last_clean":"2024-09-10T18:32:30.575790+0000","last_became_active":"2024-09-10T18:32:03.382335+0000","last_became_peered":"2024-09-10T18:32:03.382335+0000","last_unstale":"2024-09-10T18:32:30.575790+0000","last_undegraded":"2024-09-10T18:32:30.575790+0000","last_fullsized":"2024-09-10T18:32:30.575790+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":2593,"log_dups_size":0,"ondisk_log_size":2593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:14.202550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169934848,"num_objects":2593,"num_object_clones":0,"num_object_copies":7779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2593,"num_write_kb":165952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'26","reported_seq":234,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.049591+0000","last_change":"2024-09-10T18:30:47.151480+0000","last_active":"2024-09-10T18:32:19.049591+0000","last_peered":"2024-09-10T18:32:19.049591+0000","last_clean":"2024-09-10T18:32:19.049591+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:32:19.049591+0000","last_undegraded":"2024-09-10T18:32:19.049591+0000","last_fullsized":"2024-09-10T18:32:19.049591+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-09-10T18:30:47.151322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:47.151322+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022400229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":260,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.059103+0000","last_change":"2024-09-10T18:31:52.756369+0000","last_active":"2024-09-10T18:32:19.059103+0000","last_peered":"2024-09-10T18:32:19.059103+0000","last_clean":"2024-09-10T18:32:19.059103+0000","last_became_active":"2024-09-10T18:26:27.639129+0000","last_became_peered":"2024-09-10T18:26:27.639129+0000","last_unstale":"2024-09-10T18:32:19.059103+0000","last_undegraded":"2024-09-10T18:32:19.059103+0000","last_fullsized":"2024-09-10T18:32:19.059103+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:52.756258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:52.756258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:14.824560+0000","scrub_duration":0.002681269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":301,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048034+0000","last_change":"2024-09-10T18:30:51.519828+0000","last_active":"2024-09-10T18:32:19.048034+0000","last_peered":"2024-09-10T18:32:19.048034+0000","last_clean":"2024-09-10T18:32:19.048034+0000","last_became_active":"2024-09-10T18:26:27.646658+0000","last_became_peered":"2024-09-10T18:26:27.646658+0000","last_unstale":"2024-09-10T18:32:19.048034+0000","last_undegraded":"2024-09-10T18:32:19.048034+0000","last_fullsized":"2024-09-10T18:32:19.048034+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:30:51.519744+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:30:51.519744+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040833290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":257,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048286+0000","last_change":"2024-09-10T18:31:53.416989+0000","last_active":"2024-09-10T18:32:19.048286+0000","last_peered":"2024-09-10T18:32:19.048286+0000","last_clean":"2024-09-10T18:32:19.048286+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:32:19.048286+0000","last_undegraded":"2024-09-10T18:32:19.048286+0000","last_fullsized":"2024-09-10T18:32:19.048286+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:53.416936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:53.416936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:57.010695+0000","scrub_duration":0.0012036219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":239,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.050544+0000","last_change":"2024-09-10T18:30:49.395000+0000","last_active":"2024-09-10T18:32:19.050544+0000","last_peered":"2024-09-10T18:32:19.050544+0000","last_clean":"2024-09-10T18:32:19.050544+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:32:19.050544+0000","last_undegraded":"2024-09-10T18:32:19.050544+0000","last_fullsized":"2024-09-10T18:32:19.050544+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:49.394941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:30:49.394941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017872470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"252'2523","reported_seq":2537,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:29.858369+0000","last_change":"2024-09-10T18:32:03.384526+0000","last_active":"2024-09-10T18:32:29.858369+0000","last_peered":"2024-09-10T18:32:29.858369+0000","last_clean":"2024-09-10T18:32:29.858369+0000","last_became_active":"2024-09-10T18:32:03.384247+0000","last_became_peered":"2024-09-10T18:32:03.384247+0000","last_unstale":"2024-09-10T18:32:29.858369+0000","last_undegraded":"2024-09-10T18:32:29.858369+0000","last_fullsized":"2024-09-10T18:32:29.858369+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":2523,"log_dups_size":0,"ondisk_log_size":2523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:03.281934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165347328,"num_objects":2523,"num_object_clones":0,"num_object_copies":7569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2523,"num_write_kb":161472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"240'19","reported_seq":290,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.045174+0000","last_change":"2024-09-10T18:31:50.655266+0000","last_active":"2024-09-10T18:32:19.045174+0000","last_peered":"2024-09-10T18:32:19.045174+0000","last_clean":"2024-09-10T18:32:19.045174+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:32:19.045174+0000","last_undegraded":"2024-09-10T18:32:19.045174+0000","last_fullsized":"2024-09-10T18:32:19.045174+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:31:50.655208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:50.655208+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:20.517259+0000","scrub_duration":0.0021897840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":241,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.044138+0000","last_change":"2024-09-10T18:30:01.224897+0000","last_active":"2024-09-10T18:32:19.044138+0000","last_peered":"2024-09-10T18:32:19.044138+0000","last_clean":"2024-09-10T18:32:19.044138+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:32:19.044138+0000","last_undegraded":"2024-09-10T18:32:19.044138+0000","last_fullsized":"2024-09-10T18:32:19.044138+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:01.224811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:30:01.224811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011944340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":255,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.050458+0000","last_change":"2024-09-10T18:31:00.387970+0000","last_active":"2024-09-10T18:32:19.050458+0000","last_peered":"2024-09-10T18:32:19.050458+0000","last_clean":"2024-09-10T18:32:19.050458+0000","last_became_active":"2024-09-10T18:26:27.645885+0000","last_became_peered":"2024-09-10T18:26:27.645885+0000","last_unstale":"2024-09-10T18:32:19.050458+0000","last_undegraded":"2024-09-10T18:32:19.050458+0000","last_fullsized":"2024-09-10T18:32:19.050458+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:31:00.387887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001951775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":254,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.052832+0000","last_change":"2024-09-10T18:31:50.740204+0000","last_active":"2024-09-10T18:32:19.052832+0000","last_peered":"2024-09-10T18:32:19.052832+0000","last_clean":"2024-09-10T18:32:19.052832+0000","last_became_active":"2024-09-10T18:26:27.632071+0000","last_became_peered":"2024-09-10T18:26:27.632071+0000","last_unstale":"2024-09-10T18:32:19.052832+0000","last_undegraded":"2024-09-10T18:32:19.052832+0000","last_fullsized":"2024-09-10T18:32:19.052832+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:50.740104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:50.740104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:51.977586+0000","scrub_duration":0.0014989739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"252'2654","reported_seq":2668,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:30.189689+0000","last_change":"2024-09-10T18:32:03.555018+0000","last_active":"2024-09-10T18:32:30.189689+0000","last_peered":"2024-09-10T18:32:30.189689+0000","last_clean":"2024-09-10T18:32:30.189689+0000","last_became_active":"2024-09-10T18:32:03.554412+0000","last_became_peered":"2024-09-10T18:32:03.554412+0000","last_unstale":"2024-09-10T18:32:30.189689+0000","last_undegraded":"2024-09-10T18:32:30.189689+0000","last_fullsized":"2024-09-10T18:32:30.189689+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":2654,"log_dups_size":0,"ondisk_log_size":2654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:08.908305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173932544,"num_objects":2654,"num_object_clones":0,"num_object_copies":7962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2654,"num_write_kb":169856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":160,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.047986+0000","last_change":"2024-09-10T18:31:50.557439+0000","last_active":"2024-09-10T18:32:19.047986+0000","last_peered":"2024-09-10T18:32:19.047986+0000","last_clean":"2024-09-10T18:32:19.047986+0000","last_became_active":"2024-09-10T18:26:18.095088+0000","last_became_peered":"2024-09-10T18:26:18.095088+0000","last_unstale":"2024-09-10T18:32:19.047986+0000","last_undegraded":"2024-09-10T18:32:19.047986+0000","last_fullsized":"2024-09-10T18:32:19.047986+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:50.557386+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:50.557386+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:52.172646+0000","scrub_duration":0.0015687660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":298,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.045093+0000","last_change":"2024-09-10T18:31:41.540517+0000","last_active":"2024-09-10T18:32:19.045093+0000","last_peered":"2024-09-10T18:32:19.045093+0000","last_clean":"2024-09-10T18:32:19.045093+0000","last_became_active":"2024-09-10T18:21:33.611203+0000","last_became_peered":"2024-09-10T18:21:33.611203+0000","last_unstale":"2024-09-10T18:32:19.045093+0000","last_undegraded":"2024-09-10T18:32:19.045093+0000","last_fullsized":"2024-09-10T18:32:19.045093+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-09-10T18:31:41.540420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:41.540420+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:02.083864+0000","scrub_duration":0.0028551750000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":205,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.049107+0000","last_change":"2024-09-10T18:30:58.072260+0000","last_active":"2024-09-10T18:32:19.049107+0000","last_peered":"2024-09-10T18:32:19.049107+0000","last_clean":"2024-09-10T18:32:19.049107+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:32:19.049107+0000","last_undegraded":"2024-09-10T18:32:19.049107+0000","last_fullsized":"2024-09-10T18:32:19.049107+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:30:58.072177+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:30:58.072177+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003192453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":247,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.047067+0000","last_change":"2024-09-10T18:30:53.506075+0000","last_active":"2024-09-10T18:32:19.047067+0000","last_peered":"2024-09-10T18:32:19.047067+0000","last_clean":"2024-09-10T18:32:19.047067+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:32:19.047067+0000","last_undegraded":"2024-09-10T18:32:19.047067+0000","last_fullsized":"2024-09-10T18:32:19.047067+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:53.506000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:53.506000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011216421000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":265,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.053781+0000","last_change":"2024-09-10T18:31:04.059884+0000","last_active":"2024-09-10T18:32:19.053781+0000","last_peered":"2024-09-10T18:32:19.053781+0000","last_clean":"2024-09-10T18:32:19.053781+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:32:19.053781+0000","last_undegraded":"2024-09-10T18:32:19.053781+0000","last_fullsized":"2024-09-10T18:32:19.053781+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:04.059795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:04.059795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010230702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"252'2682","reported_seq":2695,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:30.575685+0000","last_change":"2024-09-10T18:32:03.555761+0000","last_active":"2024-09-10T18:32:30.575685+0000","last_peered":"2024-09-10T18:32:30.575685+0000","last_clean":"2024-09-10T18:32:30.575685+0000","last_became_active":"2024-09-10T18:32:03.554475+0000","last_became_peered":"2024-09-10T18:32:03.554475+0000","last_unstale":"2024-09-10T18:32:30.575685+0000","last_undegraded":"2024-09-10T18:32:30.575685+0000","last_fullsized":"2024-09-10T18:32:30.575685+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":2682,"log_dups_size":0,"ondisk_log_size":2682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:05.049179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175767552,"num_objects":2682,"num_object_clones":0,"num_object_copies":8046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2682,"num_write_kb":171648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":299,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.049733+0000","last_change":"2024-09-10T18:31:10.502369+0000","last_active":"2024-09-10T18:32:19.049733+0000","last_peered":"2024-09-10T18:32:19.049733+0000","last_clean":"2024-09-10T18:32:19.049733+0000","last_became_active":"2024-09-10T18:26:18.092014+0000","last_became_peered":"2024-09-10T18:26:18.092014+0000","last_unstale":"2024-09-10T18:32:19.049733+0000","last_undegraded":"2024-09-10T18:32:19.049733+0000","last_fullsized":"2024-09-10T18:32:19.049733+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:10.502290+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:10.502290+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:38.681535+0000","scrub_duration":0.0082798439999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"240'25","reported_seq":281,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.044686+0000","last_change":"2024-09-10T18:29:45.292442+0000","last_active":"2024-09-10T18:32:19.044686+0000","last_peered":"2024-09-10T18:32:19.044686+0000","last_clean":"2024-09-10T18:32:19.044686+0000","last_became_active":"2024-09-10T18:26:27.649406+0000","last_became_peered":"2024-09-10T18:26:27.649406+0000","last_unstale":"2024-09-10T18:32:19.044686+0000","last_undegraded":"2024-09-10T18:32:19.044686+0000","last_fullsized":"2024-09-10T18:32:19.044686+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'15","last_scrub_stamp":"2024-09-10T18:29:45.292368+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:29:45.292368+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033922850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":243,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.053019+0000","last_change":"2024-09-10T18:31:00.667182+0000","last_active":"2024-09-10T18:32:19.053019+0000","last_peered":"2024-09-10T18:32:19.053019+0000","last_clean":"2024-09-10T18:32:19.053019+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:32:19.053019+0000","last_undegraded":"2024-09-10T18:32:19.053019+0000","last_fullsized":"2024-09-10T18:32:19.053019+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:00.667103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:00.667103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017606639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"189'9","reported_seq":408,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:27.041470+0000","last_change":"2024-09-10T18:31:09.044554+0000","last_active":"2024-09-10T18:32:27.041470+0000","last_peered":"2024-09-10T18:32:27.041470+0000","last_clean":"2024-09-10T18:32:27.041470+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:32:27.041470+0000","last_undegraded":"2024-09-10T18:32:27.041470+0000","last_fullsized":"2024-09-10T18:32:27.041470+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'9","last_scrub_stamp":"2024-09-10T18:31:09.044476+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:31:09.044476+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.059079303999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":238,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.044864+0000","last_change":"2024-09-10T18:30:46.397016+0000","last_active":"2024-09-10T18:32:19.044864+0000","last_peered":"2024-09-10T18:32:19.044864+0000","last_clean":"2024-09-10T18:32:19.044864+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:32:19.044864+0000","last_undegraded":"2024-09-10T18:32:19.044864+0000","last_fullsized":"2024-09-10T18:32:19.044864+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:46.396958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_clean_scrub_stamp":"2024-09-10T18:30:46.396958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013383155000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"252'2630","reported_seq":2644,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:29.885727+0000","last_change":"2024-09-10T18:32:03.384707+0000","last_active":"2024-09-10T18:32:29.885727+0000","last_peered":"2024-09-10T18:32:29.885727+0000","last_clean":"2024-09-10T18:32:29.885727+0000","last_became_active":"2024-09-10T18:32:03.384417+0000","last_became_peered":"2024-09-10T18:32:03.384417+0000","last_unstale":"2024-09-10T18:32:29.885727+0000","last_undegraded":"2024-09-10T18:32:29.885727+0000","last_fullsized":"2024-09-10T18:32:29.885727+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":2630,"log_dups_size":0,"ondisk_log_size":2630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:12.032493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172359680,"num_objects":2630,"num_object_clones":0,"num_object_copies":7890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2630,"num_write_kb":168320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"240'15","reported_seq":265,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.055846+0000","last_change":"2024-09-10T18:31:50.320682+0000","last_active":"2024-09-10T18:32:19.055846+0000","last_peered":"2024-09-10T18:32:19.055846+0000","last_clean":"2024-09-10T18:32:19.055846+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:32:19.055846+0000","last_undegraded":"2024-09-10T18:32:19.055846+0000","last_fullsized":"2024-09-10T18:32:19.055846+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:31:50.320624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:50.320624+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:07.995041+0000","scrub_duration":0.002110139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"58'1","reported_seq":268,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.049382+0000","last_change":"2024-09-10T18:31:41.486097+0000","last_active":"2024-09-10T18:32:19.049382+0000","last_peered":"2024-09-10T18:32:19.049382+0000","last_clean":"2024-09-10T18:32:19.049382+0000","last_became_active":"2024-09-10T18:26:27.644997+0000","last_became_peered":"2024-09-10T18:26:27.644997+0000","last_unstale":"2024-09-10T18:32:19.049382+0000","last_undegraded":"2024-09-10T18:32:19.049382+0000","last_fullsized":"2024-09-10T18:32:19.049382+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-09-10T18:31:41.486036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:41.486036+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:52.403197+0000","scrub_duration":0.0019908389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":263,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.050968+0000","last_change":"2024-09-10T18:31:08.038810+0000","last_active":"2024-09-10T18:32:19.050968+0000","last_peered":"2024-09-10T18:32:19.050968+0000","last_clean":"2024-09-10T18:32:19.050968+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:32:19.050968+0000","last_undegraded":"2024-09-10T18:32:19.050968+0000","last_fullsized":"2024-09-10T18:32:19.050968+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:08.038615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:08.038615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:29.821394+0000","scrub_duration":0.0064931390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":236,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.054168+0000","last_change":"2024-09-10T18:29:44.098384+0000","last_active":"2024-09-10T18:32:19.054168+0000","last_peered":"2024-09-10T18:32:19.054168+0000","last_clean":"2024-09-10T18:32:19.054168+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:32:19.054168+0000","last_undegraded":"2024-09-10T18:32:19.054168+0000","last_fullsized":"2024-09-10T18:32:19.054168+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:29:44.098280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:29:44.098280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055883269999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"252'2538","reported_seq":2552,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:29.888178+0000","last_change":"2024-09-10T18:32:03.382716+0000","last_active":"2024-09-10T18:32:29.888178+0000","last_peered":"2024-09-10T18:32:29.888178+0000","last_clean":"2024-09-10T18:32:29.888178+0000","last_became_active":"2024-09-10T18:32:03.382430+0000","last_became_peered":"2024-09-10T18:32:03.382430+0000","last_unstale":"2024-09-10T18:32:29.888178+0000","last_undegraded":"2024-09-10T18:32:29.888178+0000","last_fullsized":"2024-09-10T18:32:29.888178+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":2538,"log_dups_size":0,"ondisk_log_size":2538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:27.873094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166330368,"num_objects":2538,"num_object_clones":0,"num_object_copies":7614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2538,"num_write_kb":162432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"240'44","reported_seq":322,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048864+0000","last_change":"2024-09-10T18:31:00.043959+0000","last_active":"2024-09-10T18:32:19.048864+0000","last_peered":"2024-09-10T18:32:19.048864+0000","last_clean":"2024-09-10T18:32:19.048864+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:32:19.048864+0000","last_undegraded":"2024-09-10T18:32:19.048864+0000","last_fullsized":"2024-09-10T18:32:19.048864+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'28","last_scrub_stamp":"2024-09-10T18:31:00.043812+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:31:00.043812+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004131656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":268,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.049094+0000","last_change":"2024-09-10T18:31:45.496272+0000","last_active":"2024-09-10T18:32:19.049094+0000","last_peered":"2024-09-10T18:32:19.049094+0000","last_clean":"2024-09-10T18:32:19.049094+0000","last_became_active":"2024-09-10T18:26:27.638408+0000","last_became_peered":"2024-09-10T18:26:27.638408+0000","last_unstale":"2024-09-10T18:32:19.049094+0000","last_undegraded":"2024-09-10T18:32:19.049094+0000","last_fullsized":"2024-09-10T18:32:19.049094+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:45.496183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:45.496183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:49.972930+0000","scrub_duration":0.0015755630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":267,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.050656+0000","last_change":"2024-09-10T18:31:14.899762+0000","last_active":"2024-09-10T18:32:19.050656+0000","last_peered":"2024-09-10T18:32:19.050656+0000","last_clean":"2024-09-10T18:32:19.050656+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:32:19.050656+0000","last_undegraded":"2024-09-10T18:32:19.050656+0000","last_fullsized":"2024-09-10T18:32:19.050656+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:14.899683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:14.899683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054936819999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":258,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.047942+0000","last_change":"2024-09-10T18:31:58.460588+0000","last_active":"2024-09-10T18:32:19.047942+0000","last_peered":"2024-09-10T18:32:19.047942+0000","last_clean":"2024-09-10T18:32:19.047942+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:32:19.047942+0000","last_undegraded":"2024-09-10T18:32:19.047942+0000","last_fullsized":"2024-09-10T18:32:19.047942+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:58.460533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:58.460533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:28.153691+0000","scrub_duration":0.001375914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"252'2613","reported_seq":2627,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:30.190623+0000","last_change":"2024-09-10T18:32:03.554922+0000","last_active":"2024-09-10T18:32:30.190623+0000","last_peered":"2024-09-10T18:32:30.190623+0000","last_clean":"2024-09-10T18:32:30.190623+0000","last_became_active":"2024-09-10T18:32:03.554423+0000","last_became_peered":"2024-09-10T18:32:03.554423+0000","last_unstale":"2024-09-10T18:32:30.190623+0000","last_undegraded":"2024-09-10T18:32:30.190623+0000","last_fullsized":"2024-09-10T18:32:30.190623+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":2613,"log_dups_size":0,"ondisk_log_size":2613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:13.591482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171245568,"num_objects":2613,"num_object_clones":0,"num_object_copies":7839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2613,"num_write_kb":167232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"240'18","reported_seq":284,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.043949+0000","last_change":"2024-09-10T18:30:54.285825+0000","last_active":"2024-09-10T18:32:19.043949+0000","last_peered":"2024-09-10T18:32:19.043949+0000","last_clean":"2024-09-10T18:32:19.043949+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:32:19.043949+0000","last_undegraded":"2024-09-10T18:32:19.043949+0000","last_fullsized":"2024-09-10T18:32:19.043949+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-09-10T18:30:54.285738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:54.285738+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045401779999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":261,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.049530+0000","last_change":"2024-09-10T18:31:12.489636+0000","last_active":"2024-09-10T18:32:19.049530+0000","last_peered":"2024-09-10T18:32:19.049530+0000","last_clean":"2024-09-10T18:32:19.049530+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:32:19.049530+0000","last_undegraded":"2024-09-10T18:32:19.049530+0000","last_fullsized":"2024-09-10T18:32:19.049530+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:31:12.489588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00146783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":267,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.050325+0000","last_change":"2024-09-10T18:31:12.882344+0000","last_active":"2024-09-10T18:32:19.050325+0000","last_peered":"2024-09-10T18:32:19.050325+0000","last_clean":"2024-09-10T18:32:19.050325+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:32:19.050325+0000","last_undegraded":"2024-09-10T18:32:19.050325+0000","last_fullsized":"2024-09-10T18:32:19.050325+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:12.882248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:12.882248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020040589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":237,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.059301+0000","last_change":"2024-09-10T18:31:12.803420+0000","last_active":"2024-09-10T18:32:19.059301+0000","last_peered":"2024-09-10T18:32:19.059301+0000","last_clean":"2024-09-10T18:32:19.059301+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:32:19.059301+0000","last_undegraded":"2024-09-10T18:32:19.059301+0000","last_fullsized":"2024-09-10T18:32:19.059301+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:12.803236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.803236+0000","last_clean_scrub_stamp":"2024-09-10T18:31:12.803236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041535599999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"252'2677","reported_seq":2691,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:30.874755+0000","last_change":"2024-09-10T18:32:03.383156+0000","last_active":"2024-09-10T18:32:30.874755+0000","last_peered":"2024-09-10T18:32:30.874755+0000","last_clean":"2024-09-10T18:32:30.874755+0000","last_became_active":"2024-09-10T18:32:03.382890+0000","last_became_peered":"2024-09-10T18:32:03.382890+0000","last_unstale":"2024-09-10T18:32:30.874755+0000","last_undegraded":"2024-09-10T18:32:30.874755+0000","last_fullsized":"2024-09-10T18:32:30.874755+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":2677,"log_dups_size":0,"ondisk_log_size":2677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:17.281396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175439872,"num_objects":2677,"num_object_clones":0,"num_object_copies":8031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2677,"num_write_kb":171328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":300,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048125+0000","last_change":"2024-09-10T18:31:47.535552+0000","last_active":"2024-09-10T18:32:19.048125+0000","last_peered":"2024-09-10T18:32:19.048125+0000","last_clean":"2024-09-10T18:32:19.048125+0000","last_became_active":"2024-09-10T18:26:27.645932+0000","last_became_peered":"2024-09-10T18:26:27.645932+0000","last_unstale":"2024-09-10T18:32:19.048125+0000","last_undegraded":"2024-09-10T18:32:19.048125+0000","last_fullsized":"2024-09-10T18:32:19.048125+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:31:47.535496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:47.535496+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:05.510739+0000","scrub_duration":0.0017425819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":261,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.049639+0000","last_change":"2024-09-10T18:30:45.438521+0000","last_active":"2024-09-10T18:32:19.049639+0000","last_peered":"2024-09-10T18:32:19.049639+0000","last_clean":"2024-09-10T18:32:19.049639+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:32:19.049639+0000","last_undegraded":"2024-09-10T18:32:19.049639+0000","last_fullsized":"2024-09-10T18:32:19.049639+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:30:45.438437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:30:45.438437+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026276839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":396,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:24.891818+0000","last_change":"2024-09-10T18:31:03.524139+0000","last_active":"2024-09-10T18:32:24.891818+0000","last_peered":"2024-09-10T18:32:24.891818+0000","last_clean":"2024-09-10T18:32:24.891818+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:32:24.891818+0000","last_undegraded":"2024-09-10T18:32:24.891818+0000","last_fullsized":"2024-09-10T18:32:24.891818+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:31:03.524066+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040323920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":243,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048421+0000","last_change":"2024-09-10T18:31:14.326966+0000","last_active":"2024-09-10T18:32:19.048421+0000","last_peered":"2024-09-10T18:32:19.048421+0000","last_clean":"2024-09-10T18:32:19.048421+0000","last_became_active":"2024-09-10T18:26:27.647511+0000","last_became_peered":"2024-09-10T18:26:27.647511+0000","last_unstale":"2024-09-10T18:32:19.048421+0000","last_undegraded":"2024-09-10T18:32:19.048421+0000","last_fullsized":"2024-09-10T18:32:19.048421+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:14.326705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:14.326705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018698650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":196,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048840+0000","last_change":"2024-09-10T18:31:02.067834+0000","last_active":"2024-09-10T18:32:19.048840+0000","last_peered":"2024-09-10T18:32:19.048840+0000","last_clean":"2024-09-10T18:32:19.048840+0000","last_became_active":"2024-09-10T18:26:27.640922+0000","last_became_peered":"2024-09-10T18:26:27.640922+0000","last_unstale":"2024-09-10T18:32:19.048840+0000","last_undegraded":"2024-09-10T18:32:19.048840+0000","last_fullsized":"2024-09-10T18:32:19.048840+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:02.067753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:31:02.067753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:30.111230+0000","scrub_duration":0.011667588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"240'12","reported_seq":259,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.057967+0000","last_change":"2024-09-10T18:30:49.755553+0000","last_active":"2024-09-10T18:32:19.057967+0000","last_peered":"2024-09-10T18:32:19.057967+0000","last_clean":"2024-09-10T18:32:19.057967+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:32:19.057967+0000","last_undegraded":"2024-09-10T18:32:19.057967+0000","last_fullsized":"2024-09-10T18:32:19.057967+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2024-09-10T18:30:49.755466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:49.755466+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040815260000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":247,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.050229+0000","last_change":"2024-09-10T18:31:53.634789+0000","last_active":"2024-09-10T18:32:19.050229+0000","last_peered":"2024-09-10T18:32:19.050229+0000","last_clean":"2024-09-10T18:32:19.050229+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:32:19.050229+0000","last_undegraded":"2024-09-10T18:32:19.050229+0000","last_fullsized":"2024-09-10T18:32:19.050229+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:53.634704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:53.634704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:01.998530+0000","scrub_duration":0.0014977510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":252,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048461+0000","last_change":"2024-09-10T18:31:01.523924+0000","last_active":"2024-09-10T18:32:19.048461+0000","last_peered":"2024-09-10T18:32:19.048461+0000","last_clean":"2024-09-10T18:32:19.048461+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:32:19.048461+0000","last_undegraded":"2024-09-10T18:32:19.048461+0000","last_fullsized":"2024-09-10T18:32:19.048461+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:01.523839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:01.523839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:28.978824+0000","scrub_duration":0.0016753880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"240'19","reported_seq":272,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.055703+0000","last_change":"2024-09-10T18:30:58.459335+0000","last_active":"2024-09-10T18:32:19.055703+0000","last_peered":"2024-09-10T18:32:19.055703+0000","last_clean":"2024-09-10T18:32:19.055703+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:32:19.055703+0000","last_undegraded":"2024-09-10T18:32:19.055703+0000","last_fullsized":"2024-09-10T18:32:19.055703+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'13","last_scrub_stamp":"2024-09-10T18:30:58.459246+0000","last_deep_scrub":"189'13","last_deep_scrub_stamp":"2024-09-10T18:28:08.976162+0000","last_clean_scrub_stamp":"2024-09-10T18:30:58.459246+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0093093749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":258,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.050639+0000","last_change":"2024-09-10T18:31:00.524076+0000","last_active":"2024-09-10T18:32:19.050639+0000","last_peered":"2024-09-10T18:32:19.050639+0000","last_clean":"2024-09-10T18:32:19.050639+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:32:19.050639+0000","last_undegraded":"2024-09-10T18:32:19.050639+0000","last_fullsized":"2024-09-10T18:32:19.050639+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:00.523985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:31:00.523985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:28.606520+0000","scrub_duration":0.0018784559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":239,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.055933+0000","last_change":"2024-09-10T18:30:02.587567+0000","last_active":"2024-09-10T18:32:19.055933+0000","last_peered":"2024-09-10T18:32:19.055933+0000","last_clean":"2024-09-10T18:32:19.055933+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:32:19.055933+0000","last_undegraded":"2024-09-10T18:32:19.055933+0000","last_fullsized":"2024-09-10T18:32:19.055933+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:02.587404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:02.587404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006571632,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":255,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.050353+0000","last_change":"2024-09-10T18:30:59.094192+0000","last_active":"2024-09-10T18:32:19.050353+0000","last_peered":"2024-09-10T18:32:19.050353+0000","last_clean":"2024-09-10T18:32:19.050353+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:32:19.050353+0000","last_undegraded":"2024-09-10T18:32:19.050353+0000","last_fullsized":"2024-09-10T18:32:19.050353+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:59.094103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:59.094103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095530630000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"240'19","reported_seq":279,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.050516+0000","last_change":"2024-09-10T18:31:45.544047+0000","last_active":"2024-09-10T18:32:19.050516+0000","last_peered":"2024-09-10T18:32:19.050516+0000","last_clean":"2024-09-10T18:32:19.050516+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:32:19.050516+0000","last_undegraded":"2024-09-10T18:32:19.050516+0000","last_fullsized":"2024-09-10T18:32:19.050516+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:31:45.543996+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:31:45.543996+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:53.615668+0000","scrub_duration":0.0016860379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":252,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.054551+0000","last_change":"2024-09-10T18:31:54.376624+0000","last_active":"2024-09-10T18:32:19.054551+0000","last_peered":"2024-09-10T18:32:19.054551+0000","last_clean":"2024-09-10T18:32:19.054551+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:32:19.054551+0000","last_undegraded":"2024-09-10T18:32:19.054551+0000","last_fullsized":"2024-09-10T18:32:19.054551+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_clean_scrub_stamp":"2024-09-10T18:31:54.375851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:59.542885+0000","scrub_duration":0.0023543449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":379,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:27.037392+0000","last_change":"2024-09-10T18:29:57.457944+0000","last_active":"2024-09-10T18:32:27.037392+0000","last_peered":"2024-09-10T18:32:27.037392+0000","last_clean":"2024-09-10T18:32:27.037392+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:32:27.037392+0000","last_undegraded":"2024-09-10T18:32:27.037392+0000","last_fullsized":"2024-09-10T18:32:27.037392+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:29:57.457607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:29:57.457607+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033795549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":245,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.044844+0000","last_change":"2024-09-10T18:31:06.427465+0000","last_active":"2024-09-10T18:32:19.044844+0000","last_peered":"2024-09-10T18:32:19.044844+0000","last_clean":"2024-09-10T18:32:19.044844+0000","last_became_active":"2024-09-10T18:21:37.639706+0000","last_became_peered":"2024-09-10T18:21:37.639706+0000","last_unstale":"2024-09-10T18:32:19.044844+0000","last_undegraded":"2024-09-10T18:32:19.044844+0000","last_fullsized":"2024-09-10T18:32:19.044844+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:06.427389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:06.427389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:31.517707+0000","scrub_duration":0.0019673400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":281,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.054406+0000","last_change":"2024-09-10T18:31:48.766301+0000","last_active":"2024-09-10T18:32:19.054406+0000","last_peered":"2024-09-10T18:32:19.054406+0000","last_clean":"2024-09-10T18:32:19.054406+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:32:19.054406+0000","last_undegraded":"2024-09-10T18:32:19.054406+0000","last_fullsized":"2024-09-10T18:32:19.054406+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-10T18:31:48.766245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.766245+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:49.312976+0000","scrub_duration":0.001995817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":251,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.050179+0000","last_change":"2024-09-10T18:30:53.519511+0000","last_active":"2024-09-10T18:32:19.050179+0000","last_peered":"2024-09-10T18:32:19.050179+0000","last_clean":"2024-09-10T18:32:19.050179+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:32:19.050179+0000","last_undegraded":"2024-09-10T18:32:19.050179+0000","last_fullsized":"2024-09-10T18:32:19.050179+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:53.519424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:30:53.519424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002185144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":388,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-10T18:32:27.039533+0000","last_change":"2024-09-10T18:30:53.452921+0000","last_active":"2024-09-10T18:32:27.039533+0000","last_peered":"2024-09-10T18:32:27.039533+0000","last_clean":"2024-09-10T18:32:27.039533+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:32:27.039533+0000","last_undegraded":"2024-09-10T18:32:27.039533+0000","last_fullsized":"2024-09-10T18:32:27.039533+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:30:53.452719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:30:53.452719+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035769230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":236,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.055231+0000","last_change":"2024-09-10T18:30:01.300052+0000","last_active":"2024-09-10T18:32:19.055231+0000","last_peered":"2024-09-10T18:32:19.055231+0000","last_clean":"2024-09-10T18:32:19.055231+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:32:19.055231+0000","last_undegraded":"2024-09-10T18:32:19.055231+0000","last_fullsized":"2024-09-10T18:32:19.055231+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:01.298802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:01.298802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.024539056,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":287,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.044319+0000","last_change":"2024-09-10T18:31:47.604036+0000","last_active":"2024-09-10T18:32:19.044319+0000","last_peered":"2024-09-10T18:32:19.044319+0000","last_clean":"2024-09-10T18:32:19.044319+0000","last_became_active":"2024-09-10T18:26:27.649202+0000","last_became_peered":"2024-09-10T18:26:27.649202+0000","last_unstale":"2024-09-10T18:32:19.044319+0000","last_undegraded":"2024-09-10T18:32:19.044319+0000","last_fullsized":"2024-09-10T18:32:19.044319+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:31:47.603984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:47.603984+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:54.830645+0000","scrub_duration":0.0018863860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":261,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.044962+0000","last_change":"2024-09-10T18:32:00.649943+0000","last_active":"2024-09-10T18:32:19.044962+0000","last_peered":"2024-09-10T18:32:19.044962+0000","last_clean":"2024-09-10T18:32:19.044962+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:32:19.044962+0000","last_undegraded":"2024-09-10T18:32:19.044962+0000","last_fullsized":"2024-09-10T18:32:19.044962+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.649881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.649881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:05.400883+0000","scrub_duration":0.001205547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":260,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048288+0000","last_change":"2024-09-10T18:31:56.674517+0000","last_active":"2024-09-10T18:32:19.048288+0000","last_peered":"2024-09-10T18:32:19.048288+0000","last_clean":"2024-09-10T18:32:19.048288+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:32:19.048288+0000","last_undegraded":"2024-09-10T18:32:19.048288+0000","last_fullsized":"2024-09-10T18:32:19.048288+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_clean_scrub_stamp":"2024-09-10T18:31:56.674462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:20.788769+0000","scrub_duration":0.0010991219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":237,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.052501+0000","last_change":"2024-09-10T18:31:45.797979+0000","last_active":"2024-09-10T18:32:19.052501+0000","last_peered":"2024-09-10T18:32:19.052501+0000","last_clean":"2024-09-10T18:32:19.052501+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:32:19.052501+0000","last_undegraded":"2024-09-10T18:32:19.052501+0000","last_fullsized":"2024-09-10T18:32:19.052501+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:45.797924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:31:45.797924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:05.950165+0000","scrub_duration":0.001612093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"240'15","reported_seq":281,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048454+0000","last_change":"2024-09-10T18:30:03.639358+0000","last_active":"2024-09-10T18:32:19.048454+0000","last_peered":"2024-09-10T18:32:19.048454+0000","last_clean":"2024-09-10T18:32:19.048454+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:32:19.048454+0000","last_undegraded":"2024-09-10T18:32:19.048454+0000","last_fullsized":"2024-09-10T18:32:19.048454+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-09-10T18:30:03.639256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:03.639256+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010855609,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":254,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048977+0000","last_change":"2024-09-10T18:30:47.525989+0000","last_active":"2024-09-10T18:32:19.048977+0000","last_peered":"2024-09-10T18:32:19.048977+0000","last_clean":"2024-09-10T18:32:19.048977+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:32:19.048977+0000","last_undegraded":"2024-09-10T18:32:19.048977+0000","last_fullsized":"2024-09-10T18:32:19.048977+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:47.525912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:30:47.525912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075901459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":239,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.052738+0000","last_change":"2024-09-10T18:31:44.838204+0000","last_active":"2024-09-10T18:32:19.052738+0000","last_peered":"2024-09-10T18:32:19.052738+0000","last_clean":"2024-09-10T18:32:19.052738+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:32:19.052738+0000","last_undegraded":"2024-09-10T18:32:19.052738+0000","last_fullsized":"2024-09-10T18:32:19.052738+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:44.838141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:31:44.838141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:04.036018+0000","scrub_duration":0.0011071239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":246,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.053417+0000","last_change":"2024-09-10T18:32:02.220356+0000","last_active":"2024-09-10T18:32:19.053417+0000","last_peered":"2024-09-10T18:32:19.053417+0000","last_clean":"2024-09-10T18:32:19.053417+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:32:19.053417+0000","last_undegraded":"2024-09-10T18:32:19.053417+0000","last_fullsized":"2024-09-10T18:32:19.053417+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.220305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.220305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:11.568327+0000","scrub_duration":0.0015049099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":279,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.052804+0000","last_change":"2024-09-10T18:29:54.385449+0000","last_active":"2024-09-10T18:32:19.052804+0000","last_peered":"2024-09-10T18:32:19.052804+0000","last_clean":"2024-09-10T18:32:19.052804+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:32:19.052804+0000","last_undegraded":"2024-09-10T18:32:19.052804+0000","last_fullsized":"2024-09-10T18:32:19.052804+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-09-10T18:29:54.385389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:29:54.385389+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002949199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":262,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.050306+0000","last_change":"2024-09-10T18:32:01.595970+0000","last_active":"2024-09-10T18:32:19.050306+0000","last_peered":"2024-09-10T18:32:19.050306+0000","last_clean":"2024-09-10T18:32:19.050306+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:32:19.050306+0000","last_undegraded":"2024-09-10T18:32:19.050306+0000","last_fullsized":"2024-09-10T18:32:19.050306+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:01.595917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:32:01.595917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:06.674049+0000","scrub_duration":0.0016309899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":257,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.047642+0000","last_change":"2024-09-10T18:31:50.423429+0000","last_active":"2024-09-10T18:32:19.047642+0000","last_peered":"2024-09-10T18:32:19.047642+0000","last_clean":"2024-09-10T18:32:19.047642+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:32:19.047642+0000","last_undegraded":"2024-09-10T18:32:19.047642+0000","last_fullsized":"2024-09-10T18:32:19.047642+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:50.423374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:50.423374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:14.190747+0000","scrub_duration":0.00116716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":254,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.052932+0000","last_change":"2024-09-10T18:31:14.775776+0000","last_active":"2024-09-10T18:32:19.052932+0000","last_peered":"2024-09-10T18:32:19.052932+0000","last_clean":"2024-09-10T18:32:19.052932+0000","last_became_active":"2024-09-10T18:26:27.636996+0000","last_became_peered":"2024-09-10T18:26:27.636996+0000","last_unstale":"2024-09-10T18:32:19.052932+0000","last_undegraded":"2024-09-10T18:32:19.052932+0000","last_fullsized":"2024-09-10T18:32:19.052932+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:14.775693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:14.775693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:29.982163+0000","scrub_duration":0.00163416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":260,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.061663+0000","last_change":"2024-09-10T18:30:45.710780+0000","last_active":"2024-09-10T18:32:19.061663+0000","last_peered":"2024-09-10T18:32:19.061663+0000","last_clean":"2024-09-10T18:32:19.061663+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:32:19.061663+0000","last_undegraded":"2024-09-10T18:32:19.061663+0000","last_fullsized":"2024-09-10T18:32:19.061663+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-09-10T18:30:45.710665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:30:45.710665+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0078962489999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":245,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.056145+0000","last_change":"2024-09-10T18:30:05.204555+0000","last_active":"2024-09-10T18:32:19.056145+0000","last_peered":"2024-09-10T18:32:19.056145+0000","last_clean":"2024-09-10T18:32:19.056145+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:32:19.056145+0000","last_undegraded":"2024-09-10T18:32:19.056145+0000","last_fullsized":"2024-09-10T18:32:19.056145+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:30:05.204467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021240970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":200,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.048768+0000","last_change":"2024-09-10T18:31:10.050895+0000","last_active":"2024-09-10T18:32:19.048768+0000","last_peered":"2024-09-10T18:32:19.048768+0000","last_clean":"2024-09-10T18:32:19.048768+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:32:19.048768+0000","last_undegraded":"2024-09-10T18:32:19.048768+0000","last_fullsized":"2024-09-10T18:32:19.048768+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:10.050810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:10.050810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:32.282596+0000","scrub_duration":0.0052628850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":245,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.047951+0000","last_change":"2024-09-10T18:30:58.517218+0000","last_active":"2024-09-10T18:32:19.047951+0000","last_peered":"2024-09-10T18:32:19.047951+0000","last_clean":"2024-09-10T18:32:19.047951+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:32:19.047951+0000","last_undegraded":"2024-09-10T18:32:19.047951+0000","last_fullsized":"2024-09-10T18:32:19.047951+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:30:58.517160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:30:58.517160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070680939999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":255,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.046530+0000","last_change":"2024-09-10T18:31:49.495614+0000","last_active":"2024-09-10T18:32:19.046530+0000","last_peered":"2024-09-10T18:32:19.046530+0000","last_clean":"2024-09-10T18:32:19.046530+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:32:19.046530+0000","last_undegraded":"2024-09-10T18:32:19.046530+0000","last_fullsized":"2024-09-10T18:32:19.046530+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:49.495556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:49.495556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:12.834399+0000","scrub_duration":0.0012999590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":243,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.062116+0000","last_change":"2024-09-10T18:31:43.830584+0000","last_active":"2024-09-10T18:32:19.062116+0000","last_peered":"2024-09-10T18:32:19.062116+0000","last_clean":"2024-09-10T18:32:19.062116+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:32:19.062116+0000","last_undegraded":"2024-09-10T18:32:19.062116+0000","last_fullsized":"2024-09-10T18:32:19.062116+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:43.830528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:31:43.830528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:32:58.039416+0000","scrub_duration":0.0011198810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":261,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.049910+0000","last_change":"2024-09-10T18:31:54.060506+0000","last_active":"2024-09-10T18:32:19.049910+0000","last_peered":"2024-09-10T18:32:19.049910+0000","last_clean":"2024-09-10T18:32:19.049910+0000","last_became_active":"2024-09-10T18:26:27.648112+0000","last_became_peered":"2024-09-10T18:26:27.648112+0000","last_unstale":"2024-09-10T18:32:19.049910+0000","last_undegraded":"2024-09-10T18:32:19.049910+0000","last_fullsized":"2024-09-10T18:32:19.049910+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:31:54.060444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:54.060444+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:19.181773+0000","scrub_duration":0.0019908159999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":248,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-10T18:32:19.055479+0000","last_change":"2024-09-10T18:32:00.271380+0000","last_active":"2024-09-10T18:32:19.055479+0000","last_peered":"2024-09-10T18:32:19.055479+0000","last_clean":"2024-09-10T18:32:19.055479+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:32:19.055479+0000","last_undegraded":"2024-09-10T18:32:19.055479+0000","last_fullsized":"2024-09-10T18:32:19.055479+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.271326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.271326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:12.093852+0000","scrub_duration":0.0014948439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2745368576,"num_objects":41891,"num_object_clones":0,"num_object_copies":125673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41891,"num_write_kb":2681024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8208646144,"data_stored":8208646144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41891,"ondisk_log_size":41891,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1126,"num_read_kb":883,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":731,"ondisk_log_size":731,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":651,"num_read_kb":646,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":891,"ondisk_log_size":891,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234119,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1760100,"kb_used_data":1446108,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":91985052,"statfs":{"total":95995035648,"available":94192693248,"internally_reserved":0,"allocated":1480814592,"data_stored":1480381888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,2,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:31:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":2.2669999999999999,"15min":1.569},"min":{"1min":0.26200000000000001,"5min":0.21099999999999999,"15min":0.20799999999999999},"max":{"1min":3.8959999999999999,"5min":75.506,"15min":85.012},"last":1.171},{"interface":"front","average":{"1min":0.89300000000000002,"5min":2.1429999999999998,"15min":1.5369999999999999},"min":{"1min":0.28199999999999997,"5min":0.22800000000000001,"15min":0.222},"max":{"1min":3.3250000000000002,"5min":75.710999999999999,"15min":84.665999999999997},"last":0.53500000000000003}]},{"osd":1,"last update":"Tue Sep 10 18:31:44 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":2.0009999999999999,"15min":1.1699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":6.4530000000000003,"5min":49.317999999999998,"15min":49.317999999999998},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.127,"5min":1.7889999999999999,"15min":1.1240000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.3029999999999999,"5min":49.353000000000002,"15min":49.353000000000002},"last":0.91600000000000004}]},{"osd":2,"last update":"Tue Sep 10 18:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.97999999999999998,"15min":0.75700000000000001},"min":{"1min":0.26800000000000002,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":2.6960000000000002,"5min":5.3150000000000004,"15min":5.3150000000000004},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.1559999999999999,"15min":0.83299999999999996},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.6529999999999996,"5min":9.9450000000000003,"15min":9.9450000000000003},"last":1.24}]},{"osd":3,"last update":"Tue Sep 10 18:31:44 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.093,"15min":1.294},"min":{"1min":0.29999999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.4160000000000004,"5min":5.718,"15min":86.305999999999997},"last":1.637},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.1539999999999999,"15min":1.335},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.26700000000000002},"max":{"1min":4.4660000000000002,"5min":7.3339999999999996,"15min":88.364000000000004},"last":1.2689999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.58299999999999996,"15min":0.47099999999999997},"min":{"1min":0.224,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":2.0990000000000002,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":0.495},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.623,"15min":0.48999999999999999},"min":{"1min":0.26300000000000001,"5min":0.25,"15min":0.223},"max":{"1min":2.2690000000000001,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":0.84099999999999997}]},{"osd":5,"last update":"Tue Sep 10 18:31:44 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":0.93400000000000005,"15min":0.58899999999999997},"min":{"1min":0.29799999999999999,"5min":0.27600000000000002,"15min":0.23499999999999999},"max":{"1min":7.1820000000000004,"5min":7.3109999999999999,"15min":7.3109999999999999},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.022,"5min":0.73999999999999999,"15min":0.55400000000000005},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.219},"max":{"1min":6.3360000000000003,"5min":6.3360000000000003,"15min":6.3360000000000003},"last":0.57999999999999996}]},{"osd":6,"last update":"Tue Sep 10 18:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84899999999999998,"15min":0.56000000000000005},"min":{"1min":0.28899999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.3900000000000001,"5min":7.2750000000000004,"15min":7.2750000000000004},"last":1.212},{"interface":"front","average":{"1min":1.143,"5min":0.73199999999999998,"15min":0.52800000000000002},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.585,"5min":4.585,"15min":4.585},"last":0.55700000000000005}]}]},{"osd":6,"up_from":144,"seq":618475290698,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":814808,"kb_used_data":503884,"kb_used_omap":7,"kb_used_meta":310904,"kb_avail":92930344,"statfs":{"total":95995035648,"available":95160672256,"internally_reserved":0,"allocated":515977216,"data_stored":515561068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":318365783},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":1.8009999999999999,"15min":2.008},"min":{"1min":0.28100000000000003,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.3949999999999996,"5min":30.629999999999999,"15min":30.629999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":1.591,"5min":1.3839999999999999,"15min":1.49},"min":{"1min":0.23100000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":8.4640000000000004,"5min":30.597000000000001,"15min":30.597000000000001},"last":0.27600000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.3129999999999999,"15min":0.91800000000000004},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.0329999999999995,"5min":30.984000000000002,"15min":30.984000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.6819999999999999,"5min":1.46,"15min":1.157},"min":{"1min":0.246,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.7859999999999996,"5min":30.571000000000002,"15min":30.571000000000002},"last":0.39200000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.075},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.6539999999999999,"5min":4.8259999999999996,"15min":4.8259999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.448,"5min":1.0409999999999999,"15min":0.86199999999999999},"min":{"1min":0.33100000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":5.415,"5min":11.164999999999999,"15min":11.164999999999999},"last":0.29499999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.9079999999999999,"15min":2.0910000000000002},"min":{"1min":0.29799999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.8520000000000003,"5min":61.703000000000003,"15min":61.703000000000003},"last":0.33100000000000002},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.909,"15min":2.0750000000000002},"min":{"1min":0.29499999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.7229999999999999,"5min":61.814,"15min":61.814},"last":3.141}]},{"osd":4,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.66400000000000003,"15min":0.54500000000000004},"min":{"1min":0.27600000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.6219999999999999,"5min":5.4950000000000001,"15min":5.4950000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.68400000000000005,"15min":0.65700000000000003},"min":{"1min":0.27800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":4.6390000000000002,"5min":4.6390000000000002,"15min":4.6390000000000002},"last":1.048}]},{"osd":5,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.64300000000000002,"15min":0.52300000000000002},"min":{"1min":0.27200000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.6619999999999999,"5min":4.6619999999999999,"15min":4.6619999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.626,"15min":0.47799999999999998},"min":{"1min":0.313,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.6219999999999999,"5min":2.6219999999999999,"15min":2.6219999999999999},"last":0.38200000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.60099999999999998,"15min":0.52000000000000002},"min":{"1min":0.33800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.74199999999999999,"15min":0.56399999999999995},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.9300000000000002,"5min":5.5309999999999997,"15min":5.5309999999999997},"last":0.47699999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724690,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1766956,"kb_used_data":1465444,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91978196,"statfs":{"total":95995035648,"available":94185672704,"internally_reserved":0,"allocated":1500614656,"data_stored":1500174810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:31:59 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.9239999999999999,"15min":1.411},"min":{"1min":0.24199999999999999,"5min":0.23000000000000001,"15min":0.21099999999999999},"max":{"1min":3.9300000000000002,"5min":31.172000000000001,"15min":52.033999999999999},"last":0.37},{"interface":"front","average":{"1min":0.60899999999999999,"5min":1.8129999999999999,"15min":1.2270000000000001},"min":{"1min":0.26500000000000001,"5min":0.245,"15min":0.216},"max":{"1min":2.5990000000000002,"5min":32.246000000000002,"15min":52.167999999999999},"last":0.46899999999999997}]},{"osd":1,"last update":"Tue Sep 10 18:31:59 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.385,"15min":0.88800000000000001},"min":{"1min":0.34100000000000003,"5min":0.315,"15min":0.22600000000000001},"max":{"1min":7.7679999999999998,"5min":24.16,"15min":24.16},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.383,"5min":1.2749999999999999,"15min":0.95799999999999996},"min":{"1min":0.33600000000000002,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":7.7779999999999996,"5min":24.129999999999999,"15min":24.129999999999999},"last":0.60699999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:31:59 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":1.5029999999999999,"15min":0.98099999999999998},"min":{"1min":0.245,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.7509999999999999,"5min":36.853000000000002,"15min":36.853000000000002},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":1.476,"15min":0.95399999999999996},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":5.4400000000000004,"5min":31.132000000000001,"15min":31.132000000000001},"last":0.70699999999999996}]},{"osd":3,"last update":"Tue Sep 10 18:31:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":2.101,"15min":1.2609999999999999},"min":{"1min":0.311,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":3.7839999999999998,"5min":31.207000000000001,"15min":31.207000000000001},"last":1.147},{"interface":"front","average":{"1min":1.1799999999999999,"5min":2.024,"15min":1.1619999999999999},"min":{"1min":0.39600000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":5.4640000000000004,"5min":31.199000000000002,"15min":31.199000000000002},"last":0.42799999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:31:59 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.68799999999999994,"15min":0.46899999999999997},"min":{"1min":0.26900000000000002,"5min":0.22900000000000001,"15min":0.20799999999999999},"max":{"1min":4.5199999999999996,"5min":4.5199999999999996,"15min":4.5199999999999996},"last":0.63},{"interface":"front","average":{"1min":1.131,"5min":0.79700000000000004,"15min":0.52300000000000002},"min":{"1min":0.26700000000000002,"5min":0.24199999999999999,"15min":0.22700000000000001},"max":{"1min":6.798,"5min":6.798,"15min":6.798},"last":1.1759999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:31:35 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.79200000000000004,"15min":0.63700000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.194,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.82199999999999995,"15min":0.63300000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":5.3680000000000003,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.45100000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:31:29 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.80700000000000005,"15min":0.55300000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.22500000000000001},"max":{"1min":5.4900000000000002,"5min":5.569,"15min":5.569},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63900000000000001,"15min":0.496},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.24299999999999999},"max":{"1min":3.173,"5min":3.1819999999999999,"15min":3.1819999999999999},"last":0.65400000000000003}]}]},{"osd":4,"up_from":33,"seq":141733920918,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":801636,"kb_used_data":487644,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":92943516,"statfs":{"total":95995035648,"available":95174160384,"internally_reserved":0,"allocated":499347456,"data_stored":498914563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:31:30 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.383,"15min":0.99099999999999999},"min":{"1min":0.26900000000000002,"5min":0.21199999999999999,"15min":0.188},"max":{"1min":4.2009999999999996,"5min":16.236999999999998,"15min":31.585000000000001},"last":1.518},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.125,"15min":0.94599999999999995},"min":{"1min":0.23000000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.3129999999999997,"5min":16.280999999999999,"15min":29.802},"last":0.253}]},{"osd":1,"last update":"Tue Sep 10 18:31:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.93999999999999995,"15min":0.745},"min":{"1min":0.24399999999999999,"5min":0.22600000000000001,"15min":0.21299999999999999},"max":{"1min":6.8559999999999999,"5min":6.8559999999999999,"15min":6.8559999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.1060000000000001,"15min":0.79700000000000004},"min":{"1min":0.309,"5min":0.246,"15min":0.22900000000000001},"max":{"1min":4.1210000000000004,"5min":5.085,"15min":5.085},"last":0.53500000000000003}]},{"osd":2,"last update":"Tue Sep 10 18:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":1.008,"15min":0.83599999999999997},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.0519999999999996,"5min":6.4180000000000001,"15min":13.898999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.056,"5min":0.96599999999999997,"15min":0.85199999999999998},"min":{"1min":0.29399999999999998,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":4.0460000000000003,"5min":4.1950000000000003,"15min":14.51},"last":0.28299999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:31:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.464,"15min":1.052},"min":{"1min":0.26300000000000001,"5min":0.249,"15min":0.249},"max":{"1min":3.6709999999999998,"5min":37.351999999999997,"15min":37.351999999999997},"last":0.629},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.4470000000000001,"15min":1.0920000000000001},"min":{"1min":0.35199999999999998,"5min":0.28000000000000003,"15min":0.255},"max":{"1min":2.8639999999999999,"5min":37.366,"15min":37.366},"last":0.46200000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:31:53 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.88700000000000001,"15min":0.54300000000000004},"min":{"1min":0.27300000000000002,"5min":0.25800000000000001,"15min":0.20499999999999999},"max":{"1min":3.0979999999999999,"5min":22.861000000000001,"15min":22.861000000000001},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.91800000000000004,"15min":0.57299999999999995},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":2.6400000000000001,"5min":19.611999999999998,"15min":19.611999999999998},"last":0.36599999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:31:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58099999999999996,"15min":0.45100000000000001},"min":{"1min":0.27900000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.5310000000000001,"5min":6.7990000000000004,"15min":6.7990000000000004},"last":1.486},{"interface":"front","average":{"1min":0.628,"5min":0.66200000000000003,"15min":0.51600000000000001},"min":{"1min":0.33200000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.883,"5min":7.0300000000000002,"15min":7.0300000000000002},"last":0.496}]},{"osd":7,"last update":"Tue Sep 10 18:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.64900000000000002,"15min":0.53200000000000003},"min":{"1min":0.317,"5min":0.28799999999999998,"15min":0.255},"max":{"1min":4.8600000000000003,"5min":7.0670000000000002,"15min":7.0670000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.69599999999999995,"15min":0.54300000000000004},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.247},"max":{"1min":2.6139999999999999,"5min":6.5330000000000004,"15min":6.5330000000000004},"last":1.413}]}]},{"osd":3,"up_from":27,"seq":115964117148,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1633140,"kb_used_data":1323308,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92112012,"statfs":{"total":95995035648,"available":94322700288,"internally_reserved":0,"allocated":1355067392,"data_stored":1354619503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:32:10 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1.3300000000000001,"15min":1.1259999999999999},"min":{"1min":0.24099999999999999,"5min":0.23899999999999999,"15min":0.19900000000000001},"max":{"1min":5.4560000000000004,"5min":7.0060000000000002,"15min":8.5530000000000008},"last":1.2110000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":1.262,"15min":1.167},"min":{"1min":0.23100000000000001,"5min":0.193,"15min":0.193},"max":{"1min":2.577,"5min":6.7960000000000003,"15min":13.659000000000001},"last":0.752}]},{"osd":1,"last update":"Tue Sep 10 18:32:10 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.8129999999999999,"15min":1.2509999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.4740000000000002,"5min":35.406999999999996,"15min":35.406999999999996},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.069,"5min":1.885,"15min":1.1970000000000001},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":5.3529999999999998,"5min":36.902000000000001,"15min":36.902000000000001},"last":1.843}]},{"osd":2,"last update":"Tue Sep 10 18:32:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":1.1040000000000001,"15min":0.97199999999999998},"min":{"1min":0.247,"5min":0.222,"15min":0.222},"max":{"1min":2.028,"5min":5.9829999999999997,"15min":7.7960000000000003},"last":1.754},{"interface":"front","average":{"1min":0.622,"5min":1.528,"15min":1.133},"min":{"1min":0.28499999999999998,"5min":0.217,"15min":0.217},"max":{"1min":2.4630000000000001,"5min":14.997999999999999,"15min":14.997999999999999},"last":1.1950000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:31:41 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.319,"15min":1.103},"min":{"1min":0.33300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.2919999999999998,"5min":6.1059999999999999,"15min":23.684999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.325,"5min":1.121,"15min":1.0529999999999999},"min":{"1min":0.36399999999999999,"5min":0.29899999999999999,"15min":0.27200000000000002},"max":{"1min":5.2160000000000002,"5min":5.2160000000000002,"15min":23.649000000000001},"last":0.48999999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:31:51 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.3859999999999999,"15min":1.1870000000000001},"min":{"1min":0.46500000000000002,"5min":0.35399999999999998,"15min":0.34499999999999997},"max":{"1min":3.3540000000000001,"5min":6.0430000000000001,"15min":23.719999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.157,"15min":1.1319999999999999},"min":{"1min":0.436,"5min":0.35399999999999998,"15min":0.33100000000000002},"max":{"1min":2.5859999999999999,"5min":5.3440000000000003,"15min":23.707000000000001},"last":0.442}]},{"osd":6,"last update":"Tue Sep 10 18:31:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.6950000000000001,"15min":1.151},"min":{"1min":0.379,"5min":0.317,"15min":0.317},"max":{"1min":2.9180000000000001,"5min":31.815999999999999,"15min":31.815999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.958,"5min":1.8700000000000001,"15min":1.5109999999999999},"min":{"1min":0.45400000000000001,"5min":0.315,"15min":0.315},"max":{"1min":6.7640000000000002,"5min":31.742000000000001,"15min":31.742000000000001},"last":1.6499999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:31:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.448,"15min":1.137},"min":{"1min":0.38200000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.2690000000000001,"5min":6.077,"15min":23.748999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.345,"5min":1.157,"15min":1.1319999999999999},"min":{"1min":0.43099999999999999,"5min":0.35199999999999998,"15min":0.307},"max":{"1min":2.8170000000000002,"5min":4.4180000000000001,"15min":25.058},"last":0.625}]}]},{"osd":2,"up_from":18,"seq":77309411488,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":804504,"kb_used_data":494672,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92940648,"statfs":{"total":95995035648,"available":95171223552,"internally_reserved":0,"allocated":506544128,"data_stored":506123539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:31:46 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":1.9650000000000001,"15min":1.256},"min":{"1min":0.218,"5min":0.214,"15min":0.185},"max":{"1min":15.098000000000001,"5min":43.890000000000001,"15min":43.890000000000001},"last":0.95999999999999996},{"interface":"front","average":{"1min":2.1349999999999998,"5min":2.3599999999999999,"15min":1.4450000000000001},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.16900000000000001},"max":{"1min":15.019,"5min":44.331000000000003,"15min":44.331000000000003},"last":0.92900000000000005}]},{"osd":1,"last update":"Tue Sep 10 18:31:46 2024","interfaces":[{"interface":"back","average":{"1min":2.3290000000000002,"5min":2.1760000000000002,"15min":1.321},"min":{"1min":0.23599999999999999,"5min":0.193,"15min":0.193},"max":{"1min":15.750999999999999,"5min":44.575000000000003,"15min":44.575000000000003},"last":2.5649999999999999},{"interface":"front","average":{"1min":2.3820000000000001,"5min":1.901,"15min":1.139},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.19900000000000001},"max":{"1min":14.988,"5min":44.368000000000002,"15min":44.368000000000002},"last":0.72699999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:32:06 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.637,"15min":1.4139999999999999},"min":{"1min":0.26700000000000002,"5min":0.26300000000000001,"15min":0.251},"max":{"1min":15.069000000000001,"5min":15.069000000000001,"15min":60.786999999999999},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.625,"15min":1.4359999999999999},"min":{"1min":0.27100000000000002,"5min":0.214,"15min":0.214},"max":{"1min":15.055999999999999,"5min":15.055999999999999,"15min":60.116999999999997},"last":1.288}]},{"osd":4,"last update":"Tue Sep 10 18:31:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.1739999999999999,"15min":0.93600000000000005},"min":{"1min":0.28699999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":4.641,"5min":5.9249999999999998,"15min":6.9169999999999998},"last":2.6030000000000002},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.161,"15min":0.91200000000000003},"min":{"1min":0.34200000000000003,"5min":0.32900000000000001,"15min":0.317},"max":{"1min":7.0730000000000004,"5min":7.0730000000000004,"15min":12.130000000000001},"last":0.76600000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:31:55 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.218,"15min":0.93200000000000005},"min":{"1min":0.40999999999999998,"5min":0.30299999999999999,"15min":0.27500000000000002},"max":{"1min":4.3819999999999997,"5min":7.1609999999999996,"15min":12.106},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.228,"15min":0.94999999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.31900000000000001},"max":{"1min":4.4320000000000004,"5min":7.1109999999999998,"15min":12.154},"last":0.88500000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:31:35 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.226,"15min":1.155},"min":{"1min":0.36799999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":4.6849999999999996,"5min":5.9989999999999997,"15min":5.9989999999999997},"last":2.6309999999999998},{"interface":"front","average":{"1min":1.552,"5min":1.234,"15min":0.97499999999999998},"min":{"1min":0.39200000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":7.1349999999999998,"5min":7.1349999999999998,"15min":7.1349999999999998},"last":0.84499999999999997}]},{"osd":7,"last update":"Tue Sep 10 18:31:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":0.97899999999999998},"min":{"1min":0.374,"5min":0.34200000000000003,"15min":0.30499999999999999},"max":{"1min":5.1210000000000004,"5min":6.2990000000000004,"15min":7.1639999999999997},"last":1.018},{"interface":"front","average":{"1min":1.498,"5min":1.3520000000000001,"15min":1.014},"min":{"1min":0.45600000000000002,"5min":0.311,"15min":0.307},"max":{"1min":4.7169999999999996,"5min":5.9660000000000002,"15min":6.9669999999999996},"last":2.6779999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575013,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1808368,"kb_used_data":1506856,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91936784,"statfs":{"total":95995035648,"available":94143266816,"internally_reserved":0,"allocated":1543020544,"data_stored":1542575965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:32:26 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.242,"15min":1.409},"min":{"1min":0.224,"5min":0.20399999999999999,"15min":0.16700000000000001},"max":{"1min":3.6579999999999999,"5min":14.079000000000001,"15min":80.456999999999994},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.353,"15min":1.5089999999999999},"min":{"1min":0.23799999999999999,"5min":0.187,"15min":0.187},"max":{"1min":3.3959999999999999,"5min":7.3120000000000003,"15min":80.578000000000003},"last":3.2949999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:31:38 2024","interfaces":[{"interface":"back","average":{"1min":1.706,"5min":1.579,"15min":1.093},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.6479999999999997,"5min":9.4700000000000006,"15min":13.035},"last":0.30399999999999999},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.5940000000000001,"15min":1.026},"min":{"1min":0.29899999999999999,"5min":0.216,"15min":0.19900000000000001},"max":{"1min":4.0670000000000002,"5min":12.222,"15min":12.222},"last":0.33900000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:32:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.3620000000000001,"15min":1.4379999999999999},"min":{"1min":0.29299999999999998,"5min":0.23899999999999999,"15min":0.22600000000000001},"max":{"1min":3.71,"5min":14.121,"15min":43.076000000000001},"last":0.433},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.53,"15min":1.5269999999999999},"min":{"1min":0.32800000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":4.0990000000000002,"5min":7.3799999999999999,"15min":41.93},"last":3.3759999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.2709999999999999,"15min":0.97399999999999998},"min":{"1min":0.33700000000000002,"5min":0.32100000000000001,"15min":0.29799999999999999},"max":{"1min":3.3570000000000002,"5min":7.2859999999999996,"15min":7.2859999999999996},"last":3.3290000000000002},{"interface":"front","average":{"1min":1.044,"5min":1.2470000000000001,"15min":1.0760000000000001},"min":{"1min":0.35599999999999998,"5min":0.30499999999999999,"15min":0.28299999999999997},"max":{"1min":3.9910000000000001,"5min":5.7750000000000004,"15min":19.393000000000001},"last":3.4929999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:31:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.6160000000000001,"15min":1.0920000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.0490000000000004,"5min":12.206,"15min":15.48},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.4079999999999999,"15min":1.004},"min":{"1min":0.39300000000000002,"5min":0.36299999999999999,"15min":0.308},"max":{"1min":3.6869999999999998,"5min":9.9359999999999999,"15min":9.9359999999999999},"last":0.45900000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:31:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.337,"15min":1.347},"min":{"1min":0.42599999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.7320000000000002,"5min":10.039,"15min":10.039},"last":0.39000000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.4570000000000001,"15min":1.3999999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.9449999999999998,"5min":7.3369999999999997,"15min":7.3369999999999997},"last":0.52800000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:31:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.3680000000000001,"15min":1.036},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.34399999999999997},"max":{"1min":3.7530000000000001,"5min":10.005000000000001,"15min":15.545999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.538,"5min":1.393,"15min":0.999},"min":{"1min":0.38600000000000001,"5min":0.35699999999999998,"15min":0.313},"max":{"1min":5.5289999999999999,"5min":7.359,"15min":7.359},"last":3.2309999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738538,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":963040,"kb_used_data":665688,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":92782112,"statfs":{"total":95995035648,"available":95008882688,"internally_reserved":0,"allocated":681664512,"data_stored":681235588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:32:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":1.8600000000000001,"15min":1.238},"min":{"1min":0.21199999999999999,"5min":0.20499999999999999,"15min":0.193},"max":{"1min":4.1989999999999998,"5min":33.130000000000003,"15min":33.130000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.505,"5min":1.8939999999999999,"15min":1.335},"min":{"1min":0.23100000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":1.911,"5min":31.329999999999998,"15min":31.329999999999998},"last":1.8089999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:31:46 2024","interfaces":[{"interface":"back","average":{"1min":1.962,"5min":1.7290000000000001,"15min":1.228},"min":{"1min":0.253,"5min":0.22,"15min":0.189},"max":{"1min":7.4329999999999998,"5min":30.219999999999999,"15min":30.219999999999999},"last":1.5900000000000001},{"interface":"front","average":{"1min":2.0390000000000001,"5min":1.827,"15min":1.2529999999999999},"min":{"1min":0.23400000000000001,"5min":0.22,"15min":0.22},"max":{"1min":7.5119999999999996,"5min":30.129999999999999,"15min":30.129999999999999},"last":0.33500000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.633,"15min":1.3340000000000001},"min":{"1min":0.29599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":7.0999999999999996,"5min":8.5709999999999997,"15min":23.143999999999998},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.88200000000000001,"5min":1.5349999999999999,"15min":1.28},"min":{"1min":0.28399999999999997,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.234,"5min":9.3360000000000003,"15min":23.035},"last":1.4419999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:31:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.349,"15min":1.093},"min":{"1min":0.34100000000000003,"5min":0.246,"15min":0.246},"max":{"1min":4.2640000000000002,"5min":8.3699999999999992,"15min":17.151},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3160000000000001,"15min":1.0780000000000001},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.25900000000000001},"max":{"1min":3.7919999999999998,"5min":18.728999999999999,"15min":18.728999999999999},"last":1.508}]},{"osd":5,"last update":"Tue Sep 10 18:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":1.375,"15min":1.163},"min":{"1min":0.442,"5min":0.311,"15min":0.29499999999999998},"max":{"1min":2.0699999999999998,"5min":18.77,"15min":18.77},"last":1.472},{"interface":"front","average":{"1min":1.228,"5min":1.363,"15min":1.1020000000000001},"min":{"1min":0.48599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":6.6079999999999997,"5min":6.6079999999999997,"15min":14.991},"last":0.749}]},{"osd":6,"last update":"Tue Sep 10 18:31:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":1.3100000000000001,"15min":1.3080000000000001},"min":{"1min":0.372,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.2939999999999996,"5min":8.3940000000000001,"15min":8.3940000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.375,"5min":1.3500000000000001,"15min":1.3939999999999999},"min":{"1min":0.48999999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.9079999999999999,"5min":9.7859999999999996,"15min":9.7859999999999996},"last":1.538}]},{"osd":7,"last update":"Tue Sep 10 18:31:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.341,"15min":1.0760000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.26700000000000002},"max":{"1min":3.8660000000000001,"5min":5.0430000000000001,"15min":13.711},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.409,"5min":1.387,"15min":1.0669999999999999},"min":{"1min":0.46100000000000002,"5min":0.36199999999999999,"15min":0.33500000000000002},"max":{"1min":6.6399999999999997,"5min":9.6549999999999994,"15min":15.032},"last":0.72099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":702349312,"data_stored":702349312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1543962624,"data_stored":1543962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":517865472,"data_stored":517865472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1380450304,"data_stored":1380450304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":505348096,"data_stored":505348096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1511784448,"data_stored":1511784448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":536412160,"data_stored":536412160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1510473728,"data_stored":1510473728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:32:32.247 INFO:tasks.thrashosds.thrasher:Setting 4.c to [7, 4, 2] 2024-09-10T18:32:32.247 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.c', '7', '4', '2'] 2024-09-10T18:32:32.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pg-upmap 4.c 7 4 2 2024-09-10T18:32:32.305 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:32.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:32.337 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:32.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:32.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:32.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:32.483 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:32.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:32.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:32.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:32:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:32:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:32:32.630 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:32.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:32.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:32.732 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:32.768 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:32.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:32.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:32.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:32.926 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:32.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:32.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:33.037 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:33.040 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:32:33.039847+0000 min lat: 0.00134629 max lat: 0.147491 avg lat: 0.00685001 2024-09-10T18:32:33.040 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:32:33.040 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 20 16 46713 46697 145.871 108.125 0.0106287 0.00685001 2024-09-10T18:32:33.062 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:32 smithi028 ceph-mon[39220]: from='client.25336 v1:172.21.15.28:0/2386510347' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:32:33.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:32 smithi028 ceph-mon[29291]: from='client.25336 v1:172.21.15.28:0/2386510347' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:32:33.068 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:33.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:33.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:33.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:32 smithi204 ceph-mon[32357]: from='client.25336 v1:172.21.15.28:0/2386510347' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:32:33.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:33.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:33.225 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:33.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:33.326 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:33.361 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:33.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:33.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:33.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:33.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:33.498 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:33.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:33.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:33.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:33.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:33.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:33.769 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:33.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:33.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:33.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:33.913 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:33.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:34.015 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:34.040 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 21 16 48426 48410 144.023 107.062 0.0125144 0.00693837 2024-09-10T18:32:34.053 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:34.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:34.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:34.055 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:33 smithi028 ceph-mon[39220]: pgmap v492: 158 pgs: 158 active+clean; 2.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-10T18:32:34.055 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:33 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:32:34.056 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:33 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:32:34.056 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:33 smithi028 ceph-mon[29291]: pgmap v492: 158 pgs: 158 active+clean; 2.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-10T18:32:34.056 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:33 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:32:34.056 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:33 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:32:34.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:33 smithi204 ceph-mon[32357]: pgmap v492: 158 pgs: 158 active+clean; 2.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-10T18:32:34.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:33 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:32:34.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:33 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:32:34.155 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:34.191 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:34.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:34.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:34.227 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:32:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-10T18:32:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-10T18:32:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:32:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:28633 7.f 7:ffdaa3af:::benchmark_data_smithi028_77856_object28632:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:32:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:55.726456+0000", 2024-09-10T18:32:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 518.49490763200004, 2024-09-10T18:32:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14808337999999999, 2024-09-10T18:32:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 28633 2024-09-10T18:32:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726456+0000", 2024-09-10T18:32:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726456+0000", 2024-09-10T18:32:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726455+0000", 2024-09-10T18:32:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:32:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726476+0000", 2024-09-10T18:32:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1577e-05 2024-09-10T18:32:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.726485+0000", 2024-09-10T18:32:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8540000000000003e-06 2024-09-10T18:32:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.860732+0000", 2024-09-10T18:32:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.134246369 2024-09-10T18:32:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.862493+0000", 2024-09-10T18:32:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017610779999999999 2024-09-10T18:32:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.862607+0000", 2024-09-10T18:32:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011408000000000001 2024-09-10T18:32:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:32:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.862777+0000", 2024-09-10T18:32:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000170161 2024-09-10T18:32:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.864620+0000", 2024-09-10T18:32:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018431400000000001 2024-09-10T18:32:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.870301+0000", 2024-09-10T18:32:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0056804409999999996 2024-09-10T18:32:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.874449+0000", 2024-09-10T18:32:34.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041486459999999998 2024-09-10T18:32:34.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.874499+0000", 2024-09-10T18:32:34.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9891000000000001e-05 2024-09-10T18:32:34.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:55.874539+0000", 2024-09-10T18:32:34.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0159e-05 2024-09-10T18:32:34.233 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.233 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31736 7.6 7:6f8bbc73:::benchmark_data_smithi028_77856_object31735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:32:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.829509+0000", 2024-09-10T18:32:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 516.39185423699996, 2024-09-10T18:32:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.148056353, 2024-09-10T18:32:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31736 2024-09-10T18:32:34.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.235 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829509+0000", 2024-09-10T18:32:34.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829509+0000", 2024-09-10T18:32:34.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829508+0000", 2024-09-10T18:32:34.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:32:34.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829534+0000", 2024-09-10T18:32:34.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6038999999999999e-05 2024-09-10T18:32:34.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829544+0000", 2024-09-10T18:32:34.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6029999999999994e-06 2024-09-10T18:32:34.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829549+0000", 2024-09-10T18:32:34.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2970000000000003e-06 2024-09-10T18:32:34.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829601+0000", 2024-09-10T18:32:34.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1629999999999999e-05 2024-09-10T18:32:34.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829725+0000", 2024-09-10T18:32:34.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012429300000000001 2024-09-10T18:32:34.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:32:34.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829865+0000", 2024-09-10T18:32:34.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000139493 2024-09-10T18:32:34.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.834740+0000", 2024-09-10T18:32:34.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048757419999999997 2024-09-10T18:32:34.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:34.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836134+0000", 2024-09-10T18:32:34.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013935079999999999 2024-09-10T18:32:34.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838041+0000", 2024-09-10T18:32:34.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001907583 2024-09-10T18:32:34.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838084+0000", 2024-09-10T18:32:34.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3000000000000002e-05 2024-09-10T18:32:34.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977566+0000", 2024-09-10T18:32:34.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13948112900000001 2024-09-10T18:32:34.239 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.239 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.239 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.240 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31737 7.6 7:67a3034c:::benchmark_data_smithi028_77856_object31736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:32:34.240 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.829562+0000", 2024-09-10T18:32:34.240 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 516.39180106100002, 2024-09-10T18:32:34.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14825862000000001, 2024-09-10T18:32:34.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.240 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:34.240 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:34.240 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:34.240 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31737 2024-09-10T18:32:34.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.240 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829562+0000", 2024-09-10T18:32:34.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829562+0000", 2024-09-10T18:32:34.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829562+0000", 2024-09-10T18:32:34.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:32:34.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829583+0000", 2024-09-10T18:32:34.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1251999999999999e-05 2024-09-10T18:32:34.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829591+0000", 2024-09-10T18:32:34.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5890000000000003e-06 2024-09-10T18:32:34.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.829597+0000", 2024-09-10T18:32:34.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-06 2024-09-10T18:32:34.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.831307+0000", 2024-09-10T18:32:34.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017102700000000001 2024-09-10T18:32:34.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.831417+0000", 2024-09-10T18:32:34.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110029 2024-09-10T18:32:34.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:32:34.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.831543+0000", 2024-09-10T18:32:34.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012614200000000001 2024-09-10T18:32:34.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.834769+0000", 2024-09-10T18:32:34.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032261270000000001 2024-09-10T18:32:34.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:34.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836140+0000", 2024-09-10T18:32:34.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013711909999999999 2024-09-10T18:32:34.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977739+0000", 2024-09-10T18:32:34.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.141598956 2024-09-10T18:32:34.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977783+0000", 2024-09-10T18:32:34.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3890999999999997e-05 2024-09-10T18:32:34.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977821+0000", 2024-09-10T18:32:34.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7901000000000002e-05 2024-09-10T18:32:34.246 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.246 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.246 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.246 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31754 7.7 7:e51855d4:::benchmark_data_smithi028_77856_object31753:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:32:34.246 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.837301+0000", 2024-09-10T18:32:34.246 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 516.38406270300004, 2024-09-10T18:32:34.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.151838735, 2024-09-10T18:32:34.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.246 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.247 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:34.247 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:34.247 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:34.247 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31754 2024-09-10T18:32:34.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.247 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837301+0000", 2024-09-10T18:32:34.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837301+0000", 2024-09-10T18:32:34.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837300+0000", 2024-09-10T18:32:34.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:32:34.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837322+0000", 2024-09-10T18:32:34.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1506999999999999e-05 2024-09-10T18:32:34.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837329+0000", 2024-09-10T18:32:34.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.515e-06 2024-09-10T18:32:34.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837334+0000", 2024-09-10T18:32:34.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8749999999999999e-06 2024-09-10T18:32:34.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.978333+0000", 2024-09-10T18:32:34.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.140998866 2024-09-10T18:32:34.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.978442+0000", 2024-09-10T18:32:34.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108992 2024-09-10T18:32:34.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-10T18:32:34.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.978578+0000", 2024-09-10T18:32:34.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013659699999999999 2024-09-10T18:32:34.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:34.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.984104+0000", 2024-09-10T18:32:34.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055253580000000002 2024-09-10T18:32:34.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.989014+0000", 2024-09-10T18:32:34.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049104839999999997 2024-09-10T18:32:34.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.989062+0000", 2024-09-10T18:32:34.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7441000000000002e-05 2024-09-10T18:32:34.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.989139+0000", 2024-09-10T18:32:34.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7847000000000002e-05 2024-09-10T18:32:34.252 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.252 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.252 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.252 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31758 7.6 7:6c6ad8ac:::benchmark_data_smithi028_77856_object31757:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:32:34.252 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.837963+0000", 2024-09-10T18:32:34.252 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 516.38340093500005, 2024-09-10T18:32:34.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15004868099999999, 2024-09-10T18:32:34.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.252 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.252 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:34.252 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:34.253 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:34.253 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31758 2024-09-10T18:32:34.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.253 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837963+0000", 2024-09-10T18:32:34.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837963+0000", 2024-09-10T18:32:34.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837961+0000", 2024-09-10T18:32:34.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:32:34.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837982+0000", 2024-09-10T18:32:34.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1183e-05 2024-09-10T18:32:34.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837991+0000", 2024-09-10T18:32:34.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9570000000000008e-06 2024-09-10T18:32:34.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.837996+0000", 2024-09-10T18:32:34.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6330000000000004e-06 2024-09-10T18:32:34.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.977854+0000", 2024-09-10T18:32:34.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13985836200000001 2024-09-10T18:32:34.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.978294+0000", 2024-09-10T18:32:34.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043966200000000001 2024-09-10T18:32:34.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:32:34.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.978419+0000", 2024-09-10T18:32:34.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012486499999999999 2024-09-10T18:32:34.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.983938+0000", 2024-09-10T18:32:34.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055189319999999998 2024-09-10T18:32:34.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:34.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.983945+0000", 2024-09-10T18:32:34.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7680000000000003e-06 2024-09-10T18:32:34.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.987897+0000", 2024-09-10T18:32:34.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039524770000000002 2024-09-10T18:32:34.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.987973+0000", 2024-09-10T18:32:34.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6062000000000005e-05 2024-09-10T18:32:34.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.988011+0000", 2024-09-10T18:32:34.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8003999999999997e-05 2024-09-10T18:32:34.258 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.258 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.258 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.259 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:139702 7.f 7:f2f55ab9:::benchmark_data_smithi028_77856_object18602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:32:34.259 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:43.935577+0000", 2024-09-10T18:32:34.259 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 410.28578687200002, 2024-09-10T18:32:34.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15684685600000001, 2024-09-10T18:32:34.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.259 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:34.259 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:34.259 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:34.259 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 139702 2024-09-10T18:32:34.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.259 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935577+0000", 2024-09-10T18:32:34.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935577+0000", 2024-09-10T18:32:34.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935576+0000", 2024-09-10T18:32:34.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:32:34.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935580+0000", 2024-09-10T18:32:34.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6879999999999998e-06 2024-09-10T18:32:34.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935588+0000", 2024-09-10T18:32:34.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9999999999999996e-06 2024-09-10T18:32:34.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935594+0000", 2024-09-10T18:32:34.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2820000000000004e-06 2024-09-10T18:32:34.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.982138+0000", 2024-09-10T18:32:34.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.046544461000000002 2024-09-10T18:32:34.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.982248+0000", 2024-09-10T18:32:34.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110157 2024-09-10T18:32:34.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:32:34.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.982382+0000", 2024-09-10T18:32:34.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000133895 2024-09-10T18:32:34.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:34.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.989604+0000", 2024-09-10T18:32:34.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007222366 2024-09-10T18:32:34.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.992745+0000", 2024-09-10T18:32:34.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031404810000000001 2024-09-10T18:32:34.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:44.092292+0000", 2024-09-10T18:32:34.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.099547449999999996 2024-09-10T18:32:34.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:44.092339+0000", 2024-09-10T18:32:34.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6990000000000002e-05 2024-09-10T18:32:34.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:44.092423+0000", 2024-09-10T18:32:34.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4138e-05 2024-09-10T18:32:34.264 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.264 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.264 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.265 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:140670 7.f 7:ff7f13ba:::benchmark_data_smithi028_77856_object19570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:32:34.265 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:45.605622+0000", 2024-09-10T18:32:34.265 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 408.615741195, 2024-09-10T18:32:34.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.17139099099999999, 2024-09-10T18:32:34.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.265 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.265 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:34.265 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:34.265 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:34.265 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 140670 2024-09-10T18:32:34.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.266 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605622+0000", 2024-09-10T18:32:34.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605622+0000", 2024-09-10T18:32:34.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605620+0000", 2024-09-10T18:32:34.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:34.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605628+0000", 2024-09-10T18:32:34.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7510000000000008e-06 2024-09-10T18:32:34.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605641+0000", 2024-09-10T18:32:34.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2692e-05 2024-09-10T18:32:34.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605647+0000", 2024-09-10T18:32:34.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4169999999999997e-06 2024-09-10T18:32:34.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.612161+0000", 2024-09-10T18:32:34.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0065135159999999996 2024-09-10T18:32:34.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.612264+0000", 2024-09-10T18:32:34.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103127 2024-09-10T18:32:34.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:32:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.612383+0000", 2024-09-10T18:32:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000119157 2024-09-10T18:32:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.618786+0000", 2024-09-10T18:32:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006402942 2024-09-10T18:32:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.620538+0000", 2024-09-10T18:32:34.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001752368 2024-09-10T18:32:34.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.776944+0000", 2024-09-10T18:32:34.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.156405708 2024-09-10T18:32:34.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.776979+0000", 2024-09-10T18:32:34.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4829999999999997e-05 2024-09-10T18:32:34.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.777013+0000", 2024-09-10T18:32:34.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4496999999999999e-05 2024-09-10T18:32:34.271 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.271 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.271 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.271 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:140692 7.f 7:f303c2e4:::benchmark_data_smithi028_77856_object19592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:32:34.271 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:45.628709+0000", 2024-09-10T18:32:34.272 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 408.59265430900001, 2024-09-10T18:32:34.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14840287099999999, 2024-09-10T18:32:34.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.272 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:34.272 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:34.272 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:34.272 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 140692 2024-09-10T18:32:34.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.272 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628709+0000", 2024-09-10T18:32:34.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628709+0000", 2024-09-10T18:32:34.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628707+0000", 2024-09-10T18:32:34.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:34.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628714+0000", 2024-09-10T18:32:34.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4229999999999999e-06 2024-09-10T18:32:34.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628730+0000", 2024-09-10T18:32:34.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5991999999999999e-05 2024-09-10T18:32:34.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628738+0000", 2024-09-10T18:32:34.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0469999999999996e-06 2024-09-10T18:32:34.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628829+0000", 2024-09-10T18:32:34.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1197000000000001e-05 2024-09-10T18:32:34.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628955+0000", 2024-09-10T18:32:34.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012585100000000001 2024-09-10T18:32:34.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:32:34.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.629089+0000", 2024-09-10T18:32:34.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013479200000000001 2024-09-10T18:32:34.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.276 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.634532+0000", 2024-09-10T18:32:34.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054427750000000004 2024-09-10T18:32:34.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:34.276 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.642486+0000", 2024-09-10T18:32:34.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007953669 2024-09-10T18:32:34.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.777048+0000", 2024-09-10T18:32:34.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13456268499999999 2024-09-10T18:32:34.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.777082+0000", 2024-09-10T18:32:34.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3025000000000003e-05 2024-09-10T18:32:34.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.777112+0000", 2024-09-10T18:32:34.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0516999999999999e-05 2024-09-10T18:32:34.277 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:141317 7.f 7:f85eab8a:::benchmark_data_smithi028_77856_object20217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:32:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:46.778960+0000", 2024-09-10T18:32:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 407.44240339100003, 2024-09-10T18:32:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15742622000000001, 2024-09-10T18:32:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 141317 2024-09-10T18:32:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778960+0000", 2024-09-10T18:32:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778960+0000", 2024-09-10T18:32:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778958+0000", 2024-09-10T18:32:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778965+0000", 2024-09-10T18:32:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7599999999999997e-06 2024-09-10T18:32:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778973+0000", 2024-09-10T18:32:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7770000000000002e-06 2024-09-10T18:32:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778977+0000", 2024-09-10T18:32:34.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5229999999999997e-06 2024-09-10T18:32:34.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.789488+0000", 2024-09-10T18:32:34.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010510472999999999 2024-09-10T18:32:34.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.789609+0000", 2024-09-10T18:32:34.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121301 2024-09-10T18:32:34.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:32:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.789743+0000", 2024-09-10T18:32:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013393199999999999 2024-09-10T18:32:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.793571+0000", 2024-09-10T18:32:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0038282289999999998 2024-09-10T18:32:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.810048+0000", 2024-09-10T18:32:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0164768 2024-09-10T18:32:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.283 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.283 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.936295+0000", 2024-09-10T18:32:34.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.126246568 2024-09-10T18:32:34.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.283 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.284 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.936346+0000", 2024-09-10T18:32:34.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1708e-05 2024-09-10T18:32:34.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.284 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.936386+0000", 2024-09-10T18:32:34.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0021000000000002e-05 2024-09-10T18:32:34.284 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.284 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.284 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.284 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181160 7.7 7:e2311557:::benchmark_data_smithi028_77856_object60060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:32:34.285 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.037797+0000", 2024-09-10T18:32:34.285 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 369.18356672800002, 2024-09-10T18:32:34.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.155207649, 2024-09-10T18:32:34.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.285 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.285 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:34.285 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:34.285 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:34.285 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181160 2024-09-10T18:32:34.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.285 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037797+0000", 2024-09-10T18:32:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037797+0000", 2024-09-10T18:32:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037795+0000", 2024-09-10T18:32:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037810+0000", 2024-09-10T18:32:34.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4810000000000001e-05 2024-09-10T18:32:34.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037822+0000", 2024-09-10T18:32:34.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2568000000000001e-05 2024-09-10T18:32:34.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037829+0000", 2024-09-10T18:32:34.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.562e-06 2024-09-10T18:32:34.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.038647+0000", 2024-09-10T18:32:34.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081780299999999998 2024-09-10T18:32:34.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.038755+0000", 2024-09-10T18:32:34.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108941 2024-09-10T18:32:34.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-10T18:32:34.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.038876+0000", 2024-09-10T18:32:34.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000120259 2024-09-10T18:32:34.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:34.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041587+0000", 2024-09-10T18:32:34.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027112709999999999 2024-09-10T18:32:34.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191165+0000", 2024-09-10T18:32:34.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14957848300000001 2024-09-10T18:32:34.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.192642+0000", 2024-09-10T18:32:34.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001476148 2024-09-10T18:32:34.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.193004+0000", 2024-09-10T18:32:34.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00036274700000000002 2024-09-10T18:32:34.290 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.290 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.290 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.290 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181165 7.6 7:69ba312f:::benchmark_data_smithi028_77856_object60065:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:32:34.290 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.041719+0000", 2024-09-10T18:32:34.290 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 369.179644451, 2024-09-10T18:32:34.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14957322100000001, 2024-09-10T18:32:34.290 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.290 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.291 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:34.291 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:34.291 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:34.291 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181165 2024-09-10T18:32:34.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.291 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.291 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.291 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.291 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041719+0000", 2024-09-10T18:32:34.291 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.291 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.291 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.292 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041719+0000", 2024-09-10T18:32:34.292 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.292 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041718+0000", 2024-09-10T18:32:34.292 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:32:34.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.292 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041723+0000", 2024-09-10T18:32:34.292 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.48e-06 2024-09-10T18:32:34.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041733+0000", 2024-09-10T18:32:34.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6520000000000001e-06 2024-09-10T18:32:34.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041738+0000", 2024-09-10T18:32:34.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.553e-06 2024-09-10T18:32:34.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044567+0000", 2024-09-10T18:32:34.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028282770000000001 2024-09-10T18:32:34.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044684+0000", 2024-09-10T18:32:34.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117491 2024-09-10T18:32:34.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:32:34.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044768+0000", 2024-09-10T18:32:34.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4078000000000004e-05 2024-09-10T18:32:34.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.049844+0000", 2024-09-10T18:32:34.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050756799999999999 2024-09-10T18:32:34.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:34.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.050265+0000", 2024-09-10T18:32:34.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00042087299999999999 2024-09-10T18:32:34.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191082+0000", 2024-09-10T18:32:34.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14081718500000001 2024-09-10T18:32:34.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191174+0000", 2024-09-10T18:32:34.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1875999999999999e-05 2024-09-10T18:32:34.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191292+0000", 2024-09-10T18:32:34.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000118331 2024-09-10T18:32:34.296 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.296 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.296 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.296 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:181174 7.5 e142/93 7:a54ae781:::benchmark_data_smithi028_77856_object60074:head v 142'11303, mlcod=142'11301)", 2024-09-10T18:32:34.296 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.048963+0000", 2024-09-10T18:32:34.297 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 369.172400143, 2024-09-10T18:32:34.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14980299699999999, 2024-09-10T18:32:34.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.297 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.297 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.297 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.297 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048963+0000", 2024-09-10T18:32:34.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.297 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048963+0000", 2024-09-10T18:32:34.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048962+0000", 2024-09-10T18:32:34.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:32:34.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048969+0000", 2024-09-10T18:32:34.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.08e-06 2024-09-10T18:32:34.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048979+0000", 2024-09-10T18:32:34.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.006e-05 2024-09-10T18:32:34.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048985+0000", 2024-09-10T18:32:34.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8780000000000003e-06 2024-09-10T18:32:34.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.049055+0000", 2024-09-10T18:32:34.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9560000000000005e-05 2024-09-10T18:32:34.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.049072+0000", 2024-09-10T18:32:34.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7476e-05 2024-09-10T18:32:34.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198732+0000", 2024-09-10T18:32:34.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14965962499999999 2024-09-10T18:32:34.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198766+0000", 2024-09-10T18:32:34.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4514000000000002e-05 2024-09-10T18:32:34.301 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.301 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.301 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.301 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:181175 7.5 e142/93 7:a07e4061:::benchmark_data_smithi028_77856_object60075:head v 142'11304, mlcod=142'11301)", 2024-09-10T18:32:34.301 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.051631+0000", 2024-09-10T18:32:34.301 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 369.169732221, 2024-09-10T18:32:34.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15340213799999999, 2024-09-10T18:32:34.301 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.301 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.301 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051631+0000", 2024-09-10T18:32:34.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051631+0000", 2024-09-10T18:32:34.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051630+0000", 2024-09-10T18:32:34.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:32:34.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051639+0000", 2024-09-10T18:32:34.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8189999999999998e-06 2024-09-10T18:32:34.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051649+0000", 2024-09-10T18:32:34.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0101999999999999e-05 2024-09-10T18:32:34.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051655+0000", 2024-09-10T18:32:34.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9490000000000001e-06 2024-09-10T18:32:34.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.304 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.304 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051705+0000", 2024-09-10T18:32:34.304 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0328000000000003e-05 2024-09-10T18:32:34.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.304 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.304 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051722+0000", 2024-09-10T18:32:34.304 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6617999999999999e-05 2024-09-10T18:32:34.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.304 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.304 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205001+0000", 2024-09-10T18:32:34.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15327945300000001 2024-09-10T18:32:34.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.305 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205033+0000", 2024-09-10T18:32:34.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2257e-05 2024-09-10T18:32:34.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.305 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.305 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181177 7.f 7:f9ebf4ca:::benchmark_data_smithi028_77856_object60077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:32:34.305 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.053108+0000", 2024-09-10T18:32:34.306 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 369.168255128, 2024-09-10T18:32:34.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15150366200000001, 2024-09-10T18:32:34.306 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.306 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.306 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:34.306 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:34.306 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:34.306 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181177 2024-09-10T18:32:34.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.306 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.053108+0000", 2024-09-10T18:32:34.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.053108+0000", 2024-09-10T18:32:34.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.053106+0000", 2024-09-10T18:32:34.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:34.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.053113+0000", 2024-09-10T18:32:34.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0520000000000004e-06 2024-09-10T18:32:34.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.190879+0000", 2024-09-10T18:32:34.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13776606999999999 2024-09-10T18:32:34.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.190891+0000", 2024-09-10T18:32:34.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1102e-05 2024-09-10T18:32:34.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.190985+0000", 2024-09-10T18:32:34.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4087999999999998e-05 2024-09-10T18:32:34.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191099+0000", 2024-09-10T18:32:34.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114473 2024-09-10T18:32:34.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:32:34.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191299+0000", 2024-09-10T18:32:34.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019984899999999999 2024-09-10T18:32:34.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.197973+0000", 2024-09-10T18:32:34.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066738869999999999 2024-09-10T18:32:34.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:34.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198142+0000", 2024-09-10T18:32:34.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000169006 2024-09-10T18:32:34.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.204473+0000", 2024-09-10T18:32:34.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063307729999999996 2024-09-10T18:32:34.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.311 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.311 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.204527+0000", 2024-09-10T18:32:34.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4588e-05 2024-09-10T18:32:34.311 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:34.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.204612+0000", 2024-09-10T18:32:34.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.475e-05 2024-09-10T18:32:34.313 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.313 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.313 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.313 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:181183 7.d e142/95 7:b1361f07:::benchmark_data_smithi028_77856_object60083:head v 142'11153, mlcod=142'11151)", 2024-09-10T18:32:34.313 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.055525+0000", 2024-09-10T18:32:34.313 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 369.16583876599998, 2024-09-10T18:32:34.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14994769499999999, 2024-09-10T18:32:34.314 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.314 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.314 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055525+0000", 2024-09-10T18:32:34.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055525+0000", 2024-09-10T18:32:34.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055523+0000", 2024-09-10T18:32:34.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:32:34.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.315 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055532+0000", 2024-09-10T18:32:34.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0159999999999997e-06 2024-09-10T18:32:34.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.315 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191429+0000", 2024-09-10T18:32:34.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13589642099999999 2024-09-10T18:32:34.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191435+0000", 2024-09-10T18:32:34.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9569999999999999e-06 2024-09-10T18:32:34.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198700+0000", 2024-09-10T18:32:34.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0072655549999999999 2024-09-10T18:32:34.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198722+0000", 2024-09-10T18:32:34.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1307000000000001e-05 2024-09-10T18:32:34.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205091+0000", 2024-09-10T18:32:34.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063695779999999999 2024-09-10T18:32:34.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205472+0000", 2024-09-10T18:32:34.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00038129400000000001 2024-09-10T18:32:34.317 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.317 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.317 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.318 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15117.0:170382 8.6 8:688b0bc1:::benchmark_data_smithi028_109242_object36161:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-10T18:32:34.318 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:30:10.978519+0000", 2024-09-10T18:32:34.318 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 143.24284422400001, 2024-09-10T18:32:34.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15547377100000001, 2024-09-10T18:32:34.318 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.318 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.318 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:34.318 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15117", 2024-09-10T18:32:34.318 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/121353060", 2024-09-10T18:32:34.318 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 170382 2024-09-10T18:32:34.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.318 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.319 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.319 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.978519+0000", 2024-09-10T18:32:34.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.319 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.319 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.978519+0000", 2024-09-10T18:32:34.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.319 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.319 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.978517+0000", 2024-09-10T18:32:34.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:34.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.978525+0000", 2024-09-10T18:32:34.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6359999999999994e-06 2024-09-10T18:32:34.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.978540+0000", 2024-09-10T18:32:34.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4792e-05 2024-09-10T18:32:34.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.978546+0000", 2024-09-10T18:32:34.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7960000000000001e-06 2024-09-10T18:32:34.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.983066+0000", 2024-09-10T18:32:34.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045201210000000002 2024-09-10T18:32:34.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.983187+0000", 2024-09-10T18:32:34.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121585 2024-09-10T18:32:34.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-09-10T18:32:34.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.983364+0000", 2024-09-10T18:32:34.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017670500000000001 2024-09-10T18:32:34.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:34.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.997893+0000", 2024-09-10T18:32:34.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014529179999999999 2024-09-10T18:32:34.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:11.133743+0000", 2024-09-10T18:32:34.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.135849677 2024-09-10T18:32:34.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:11.133908+0000", 2024-09-10T18:32:34.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000164919 2024-09-10T18:32:34.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:11.133951+0000", 2024-09-10T18:32:34.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3668000000000003e-05 2024-09-10T18:32:34.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.324 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.324 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:11.133993+0000", 2024-09-10T18:32:34.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1588000000000002e-05 2024-09-10T18:32:34.324 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.324 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.324 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.324 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15117.0:170388 8.6 8:67c70fd9:::benchmark_data_smithi028_109242_object36167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-10T18:32:34.324 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:30:10.980841+0000", 2024-09-10T18:32:34.324 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 143.240522328, 2024-09-10T18:32:34.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15331135000000001, 2024-09-10T18:32:34.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.325 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.325 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:34.325 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15117", 2024-09-10T18:32:34.325 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/121353060", 2024-09-10T18:32:34.325 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 170388 2024-09-10T18:32:34.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.325 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.980841+0000", 2024-09-10T18:32:34.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.980841+0000", 2024-09-10T18:32:34.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.980839+0000", 2024-09-10T18:32:34.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:32:34.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.980846+0000", 2024-09-10T18:32:34.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1969999999999999e-06 2024-09-10T18:32:34.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.980860+0000", 2024-09-10T18:32:34.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.38e-05 2024-09-10T18:32:34.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.980865+0000", 2024-09-10T18:32:34.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6210000000000001e-06 2024-09-10T18:32:34.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.989078+0000", 2024-09-10T18:32:34.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0082128789999999993 2024-09-10T18:32:34.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.328 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.328 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.989184+0000", 2024-09-10T18:32:34.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000106086 2024-09-10T18:32:34.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.328 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-09-10T18:32:34.328 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.989298+0000", 2024-09-10T18:32:34.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113318 2024-09-10T18:32:34.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.328 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:34.328 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.999946+0000", 2024-09-10T18:32:34.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010648283 2024-09-10T18:32:34.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:11.134028+0000", 2024-09-10T18:32:34.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13408160099999999 2024-09-10T18:32:34.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:11.134083+0000", 2024-09-10T18:32:34.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5206e-05 2024-09-10T18:32:34.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:11.134120+0000", 2024-09-10T18:32:34.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6808999999999999e-05 2024-09-10T18:32:34.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:11.134152+0000", 2024-09-10T18:32:34.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2808000000000002e-05 2024-09-10T18:32:34.330 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.330 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.330 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.330 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15117.0:173622 8.6 8:679bf3f0:::benchmark_data_smithi028_109242_object39401:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-09-10T18:32:34.330 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:30:14.779803+0000", 2024-09-10T18:32:34.331 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 139.441560004, 2024-09-10T18:32:34.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15561011, 2024-09-10T18:32:34.331 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.331 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.331 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:34.331 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15117", 2024-09-10T18:32:34.331 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/121353060", 2024-09-10T18:32:34.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 173622 2024-09-10T18:32:34.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.331 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.331 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.331 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.779803+0000", 2024-09-10T18:32:34.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.779803+0000", 2024-09-10T18:32:34.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.779801+0000", 2024-09-10T18:32:34.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:32:34.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.779810+0000", 2024-09-10T18:32:34.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7590000000000003e-06 2024-09-10T18:32:34.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.333 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.333 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.779827+0000", 2024-09-10T18:32:34.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7048999999999999e-05 2024-09-10T18:32:34.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.333 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.333 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.779834+0000", 2024-09-10T18:32:34.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2250000000000003e-06 2024-09-10T18:32:34.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.813829+0000", 2024-09-10T18:32:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.033995385000000003 2024-09-10T18:32:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.814039+0000", 2024-09-10T18:32:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00021000399999999999 2024-09-10T18:32:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-09-10T18:32:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.814237+0000", 2024-09-10T18:32:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019735 2024-09-10T18:32:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.818552+0000", 2024-09-10T18:32:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004315129 2024-09-10T18:32:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.822712+0000", 2024-09-10T18:32:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041595620000000003 2024-09-10T18:32:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:34.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.935346+0000", 2024-09-10T18:32:34.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.11263444 2024-09-10T18:32:34.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.336 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.935389+0000", 2024-09-10T18:32:34.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3339e-05 2024-09-10T18:32:34.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.336 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.935414+0000", 2024-09-10T18:32:34.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4267e-05 2024-09-10T18:32:34.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.337 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.337 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.337 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15117.0:177491 8.6 8:63c103fa:::benchmark_data_smithi028_109242_object43270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-10T18:32:34.337 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:30:19.568763+0000", 2024-09-10T18:32:34.337 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 134.652600858, 2024-09-10T18:32:34.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.17489908000000001, 2024-09-10T18:32:34.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.337 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:34.337 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15117", 2024-09-10T18:32:34.337 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/121353060", 2024-09-10T18:32:34.338 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 177491 2024-09-10T18:32:34.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.338 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.568763+0000", 2024-09-10T18:32:34.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.568763+0000", 2024-09-10T18:32:34.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.568761+0000", 2024-09-10T18:32:34.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:34.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.568767+0000", 2024-09-10T18:32:34.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5930000000000002e-06 2024-09-10T18:32:34.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.568777+0000", 2024-09-10T18:32:34.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2820000000000002e-06 2024-09-10T18:32:34.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.568782+0000", 2024-09-10T18:32:34.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7729999999999996e-06 2024-09-10T18:32:34.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.637153+0000", 2024-09-10T18:32:34.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.068370809000000005 2024-09-10T18:32:34.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.637452+0000", 2024-09-10T18:32:34.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029882500000000001 2024-09-10T18:32:34.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.341 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-09-10T18:32:34.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.637873+0000", 2024-09-10T18:32:34.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00042066599999999999 2024-09-10T18:32:34.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.341 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.644753+0000", 2024-09-10T18:32:34.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068805439999999997 2024-09-10T18:32:34.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:34.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.645799+0000", 2024-09-10T18:32:34.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010459169999999999 2024-09-10T18:32:34.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.743549+0000", 2024-09-10T18:32:34.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.097750291000000003 2024-09-10T18:32:34.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.743620+0000", 2024-09-10T18:32:34.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0461000000000002e-05 2024-09-10T18:32:34.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.743662+0000", 2024-09-10T18:32:34.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1730999999999997e-05 2024-09-10T18:32:34.345 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.345 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.345 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.345 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15117.0:177492 8.6 8:66cc57c8:::benchmark_data_smithi028_109242_object43271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-10T18:32:34.346 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:30:19.570479+0000", 2024-09-10T18:32:34.346 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 134.650884856, 2024-09-10T18:32:34.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.17340629799999999, 2024-09-10T18:32:34.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:34.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:34.346 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:34.346 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15117", 2024-09-10T18:32:34.346 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/121353060", 2024-09-10T18:32:34.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 177492 2024-09-10T18:32:34.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.346 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:34.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:34.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.570479+0000", 2024-09-10T18:32:34.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:34.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.570479+0000", 2024-09-10T18:32:34.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:34.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:34.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.570477+0000", 2024-09-10T18:32:34.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:34.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:34.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.570483+0000", 2024-09-10T18:32:34.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.579e-06 2024-09-10T18:32:34.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:34.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.570500+0000", 2024-09-10T18:32:34.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6141999999999999e-05 2024-09-10T18:32:34.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:34.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.570506+0000", 2024-09-10T18:32:34.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2210000000000002e-06 2024-09-10T18:32:34.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:34.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.641242+0000", 2024-09-10T18:32:34.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.07073662 2024-09-10T18:32:34.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:34.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.641352+0000", 2024-09-10T18:32:34.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109949 2024-09-10T18:32:34.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-09-10T18:32:34.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.641492+0000", 2024-09-10T18:32:34.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013988499999999999 2024-09-10T18:32:34.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.652310+0000", 2024-09-10T18:32:34.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010817614 2024-09-10T18:32:34.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:34.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.652325+0000", 2024-09-10T18:32:34.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4673e-05 2024-09-10T18:32:34.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:34.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.743825+0000", 2024-09-10T18:32:34.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.091500026999999998 2024-09-10T18:32:34.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:34.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.743852+0000", 2024-09-10T18:32:34.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7208000000000001e-05 2024-09-10T18:32:34.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:34.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:34.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:34.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.743885+0000", 2024-09-10T18:32:34.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3067e-05 2024-09-10T18:32:34.352 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.352 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.352 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.352 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:34.352 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:34.352 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:32:34.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:34.353 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:34.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:34.454 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:34.480 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:34.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:34.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:34.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:34.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:34.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:34.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:34.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:34.755 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:34.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:34.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:32:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:32:34.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:34.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:34.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:34.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:34.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:35.030 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:35.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:35.040 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 22 16 50165 50149 142.417 108.688 0.0105147 0.00701646 2024-09-10T18:32:35.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:34 smithi204 ceph-mon[32357]: osdmap e253: 8 total, 8 up, 8 in 2024-09-10T18:32:35.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:35.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:35.173 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:35.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:35.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:34 smithi028 ceph-mon[39220]: osdmap e253: 8 total, 8 up, 8 in 2024-09-10T18:32:35.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:34 smithi028 ceph-mon[29291]: osdmap e253: 8 total, 8 up, 8 in 2024-09-10T18:32:35.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:35.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:35.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:35.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:35.454 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:35.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:35.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:35.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:35.596 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:35.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:35.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:35.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:35.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T18:32:35.726 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:35.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:35.828 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:35.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:35.880 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:35.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:35.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:35 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:35 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:32:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:35 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:32:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:35 smithi028 ceph-mon[29291]: pgmap v494: 158 pgs: 158 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-10T18:32:36.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:35 smithi028 ceph-mon[29291]: osdmap e254: 8 total, 8 up, 8 in 2024-09-10T18:32:36.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:35 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:36.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:35 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:32:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:35 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:32:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:35 smithi028 ceph-mon[39220]: pgmap v494: 158 pgs: 158 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-10T18:32:36.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:35 smithi028 ceph-mon[39220]: osdmap e254: 8 total, 8 up, 8 in 2024-09-10T18:32:36.025 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:36.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:36.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:36.043 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 23 16 51833 51817 140.742 104.25 0.0052325 0.0070996 2024-09-10T18:32:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:35 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:35 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:32:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:35 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:32:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:35 smithi204 ceph-mon[32357]: pgmap v494: 158 pgs: 158 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-10T18:32:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:35 smithi204 ceph-mon[32357]: osdmap e254: 8 total, 8 up, 8 in 2024-09-10T18:32:36.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:36.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:36.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:36.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:36.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:36.381 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:36.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:36.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:36.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:36.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:36.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:36.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:36.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:36.658 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:36.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:36.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:36.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:36.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:36.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:36.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:36.950 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:36.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:36.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:37.047 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 24 16 53495 53479 139.185 103.875 0.00656788 0.00717901 2024-09-10T18:32:37.052 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:37.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:37.079 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:37.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:37.180 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:37.215 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:37.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:37.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:37.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:37.355 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:37.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:37.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:37.457 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:37.490 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:37.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:37.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:37.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:37.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:37.736 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:37.768 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:37.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:37.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:37.797 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:37.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:37.917 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:37.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:37.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:38.021 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:38.047 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 25 16 55097 55081 137.621 100.125 0.0178175 0.00726123 2024-09-10T18:32:38.050 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:38.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:38.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:38.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:38.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:38 smithi028 ceph-mon[29291]: pgmap v496: 158 pgs: 158 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.21k op/s 2024-09-10T18:32:38.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:38 smithi028 ceph-mon[39220]: pgmap v496: 158 pgs: 158 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.21k op/s 2024-09-10T18:32:38.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:38.203 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:38.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:38.304 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:38.331 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:38.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:38.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:38 smithi204 ceph-mon[32357]: pgmap v496: 158 pgs: 158 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.21k op/s 2024-09-10T18:32:38.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:38.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:38.476 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:38.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:38.488 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:32:38.488 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:32:38.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:32:38.525 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:32:38.559 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:32:38.559 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:32:38.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:38.593 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:32:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:32:38.604 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:38.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:38.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:38.628 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:32:38.628 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:32:38.660 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:32:38.661 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:32:38.690 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:32:38.690 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:32:38.706 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:38.726 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:32:38.726 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:32:38.738 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:38.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:38.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:38.840 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:38.868 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:38.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:38.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:38.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:39.012 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:39.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:39.048 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 26 16 56398 56382 135.457 81.3125 0.0148032 0.00737625 2024-09-10T18:32:39.054 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:32:39.114 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:39.148 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:39.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:39.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:39.252 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:39.285 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:39.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:39.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:39 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3600315399' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:32:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:39 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3083599375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:32:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:32:39.360 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:39 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3600315399' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:32:39.360 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:39 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3083599375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:32:39.360 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:32:39.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:39 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3600315399' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:32:39.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:39 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3083599375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:32:39.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:32:39.387 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:39.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:39.423 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:39.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:39.523 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:39.557 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:39.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:39.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:39.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:39.682 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:39.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:39.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:39.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:39.811 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:39.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:39.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:39.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:39.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:39.964 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:39.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:40.048 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 27 16 57589 57573 133.197 74.4375 0.0246116 0.00750154 2024-09-10T18:32:40.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:40.099 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:40.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:40.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:40.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:40.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:40.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:40.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:40.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:40 smithi028 ceph-mon[29291]: pgmap v497: 158 pgs: 158 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-10T18:32:40.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:40 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:32:40.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:40 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3600315399' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:32:40.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:40 smithi028 ceph-mon[29291]: osdmap e255: 8 total, 8 up, 8 in 2024-09-10T18:32:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:40 smithi028 ceph-mon[39220]: pgmap v497: 158 pgs: 158 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-10T18:32:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:40 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:32:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:40 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3600315399' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:32:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:40 smithi028 ceph-mon[39220]: osdmap e255: 8 total, 8 up, 8 in 2024-09-10T18:32:40.344 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:40 smithi204 ceph-mon[32357]: pgmap v497: 158 pgs: 158 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-10T18:32:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:40 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:32:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:40 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3600315399' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:32:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:40 smithi204 ceph-mon[32357]: osdmap e255: 8 total, 8 up, 8 in 2024-09-10T18:32:40.375 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:40.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:40.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:40.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:40.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:40.528 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:40.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:40.629 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:40.658 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:40.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:40.760 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:40.795 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:40.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:40.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:40.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:40.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:40.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:41.040 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:41.047 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 28 16 59063 59047 131.731 92.125 0.00771057 0.00757558 2024-09-10T18:32:41.061 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:41.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:41.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:41.163 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:41.197 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:41.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:41.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:41.299 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:41.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[39220]: 5.5 scrub starts 2024-09-10T18:32:41.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[39220]: 5.5 scrub ok 2024-09-10T18:32:41.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[39220]: 5.17 scrub starts 2024-09-10T18:32:41.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[39220]: 5.17 scrub ok 2024-09-10T18:32:41.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[39220]: 3.13 scrub starts 2024-09-10T18:32:41.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[39220]: 3.13 scrub ok 2024-09-10T18:32:41.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[39220]: 4.c scrub starts 2024-09-10T18:32:41.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[39220]: 4.c scrub ok 2024-09-10T18:32:41.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[39220]: 6.f scrub starts 2024-09-10T18:32:41.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[39220]: 6.f scrub ok 2024-09-10T18:32:41.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[39220]: 4.a scrub starts 2024-09-10T18:32:41.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[39220]: 4.a scrub ok 2024-09-10T18:32:41.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[39220]: osdmap e256: 8 total, 8 up, 8 in 2024-09-10T18:32:41.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[29291]: 5.5 scrub starts 2024-09-10T18:32:41.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[29291]: 5.5 scrub ok 2024-09-10T18:32:41.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[29291]: 5.17 scrub starts 2024-09-10T18:32:41.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[29291]: 5.17 scrub ok 2024-09-10T18:32:41.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[29291]: 3.13 scrub starts 2024-09-10T18:32:41.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[29291]: 3.13 scrub ok 2024-09-10T18:32:41.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[29291]: 4.c scrub starts 2024-09-10T18:32:41.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[29291]: 4.c scrub ok 2024-09-10T18:32:41.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[29291]: 6.f scrub starts 2024-09-10T18:32:41.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[29291]: 6.f scrub ok 2024-09-10T18:32:41.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[29291]: 4.a scrub starts 2024-09-10T18:32:41.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[29291]: 4.a scrub ok 2024-09-10T18:32:41.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-mon[29291]: osdmap e256: 8 total, 8 up, 8 in 2024-09-10T18:32:41.327 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:41.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:41.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:41 smithi204 ceph-mon[32357]: 5.5 scrub starts 2024-09-10T18:32:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:41 smithi204 ceph-mon[32357]: 5.5 scrub ok 2024-09-10T18:32:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:41 smithi204 ceph-mon[32357]: 5.17 scrub starts 2024-09-10T18:32:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:41 smithi204 ceph-mon[32357]: 5.17 scrub ok 2024-09-10T18:32:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:41 smithi204 ceph-mon[32357]: 3.13 scrub starts 2024-09-10T18:32:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:41 smithi204 ceph-mon[32357]: 3.13 scrub ok 2024-09-10T18:32:41.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:41 smithi204 ceph-mon[32357]: 4.c scrub starts 2024-09-10T18:32:41.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:41 smithi204 ceph-mon[32357]: 4.c scrub ok 2024-09-10T18:32:41.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:41 smithi204 ceph-mon[32357]: 6.f scrub starts 2024-09-10T18:32:41.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:41 smithi204 ceph-mon[32357]: 6.f scrub ok 2024-09-10T18:32:41.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:41 smithi204 ceph-mon[32357]: 4.a scrub starts 2024-09-10T18:32:41.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:41 smithi204 ceph-mon[32357]: 4.a scrub ok 2024-09-10T18:32:41.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:41 smithi204 ceph-mon[32357]: osdmap e256: 8 total, 8 up, 8 in 2024-09-10T18:32:41.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:41.466 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:41.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:41.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:41.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:41.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:41.610 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:41.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:41.711 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:41.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:41.750 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:41.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:41.753 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:32:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:32:41] "GET /metrics HTTP/1.1" 200 224400 "" "Prometheus/2.43.0" 2024-09-10T18:32:41.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:41.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:41.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:41.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:42.027 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:42.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:42.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:42.047 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 29 16 60651 60635 130.611 99.25 0.00739886 0.00765129 2024-09-10T18:32:42.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:42.174 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:42.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:42.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:42.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:42.308 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:42.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:42.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:42.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:42 smithi028 ceph-mon[29291]: 5.14 scrub starts 2024-09-10T18:32:42.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:42 smithi028 ceph-mon[29291]: 5.14 scrub ok 2024-09-10T18:32:42.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:42 smithi028 ceph-mon[29291]: 3.9 scrub starts 2024-09-10T18:32:42.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:42 smithi028 ceph-mon[29291]: 3.9 scrub ok 2024-09-10T18:32:42.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:42 smithi028 ceph-mon[29291]: pgmap v500: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-09-10T18:32:42.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:42 smithi028 ceph-mon[29291]: 4.12 scrub starts 2024-09-10T18:32:42.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:42 smithi028 ceph-mon[29291]: 4.12 scrub ok 2024-09-10T18:32:42.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:42 smithi028 ceph-mon[39220]: 5.14 scrub starts 2024-09-10T18:32:42.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:42 smithi028 ceph-mon[39220]: 5.14 scrub ok 2024-09-10T18:32:42.332 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:42 smithi028 ceph-mon[39220]: 3.9 scrub starts 2024-09-10T18:32:42.332 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:42 smithi028 ceph-mon[39220]: 3.9 scrub ok 2024-09-10T18:32:42.332 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:42 smithi028 ceph-mon[39220]: pgmap v500: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-09-10T18:32:42.332 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:42 smithi028 ceph-mon[39220]: 4.12 scrub starts 2024-09-10T18:32:42.332 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:42 smithi028 ceph-mon[39220]: 4.12 scrub ok 2024-09-10T18:32:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:42 smithi204 ceph-mon[32357]: 5.14 scrub starts 2024-09-10T18:32:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:42 smithi204 ceph-mon[32357]: 5.14 scrub ok 2024-09-10T18:32:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:42 smithi204 ceph-mon[32357]: 3.9 scrub starts 2024-09-10T18:32:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:42 smithi204 ceph-mon[32357]: 3.9 scrub ok 2024-09-10T18:32:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:42 smithi204 ceph-mon[32357]: pgmap v500: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-09-10T18:32:42.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:42 smithi204 ceph-mon[32357]: 4.12 scrub starts 2024-09-10T18:32:42.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:42 smithi204 ceph-mon[32357]: 4.12 scrub ok 2024-09-10T18:32:42.355 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:32:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:32:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:32:42.410 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:42.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:42.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:42.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:42.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:42.602 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:42.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:42.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:42.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:42.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:42.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:42.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:42.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:42.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:42.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:43.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:43.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:43.048 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 30 16 62243 62227 129.574 99.5 0.00540407 0.0077122 2024-09-10T18:32:43.121 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:43.152 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:43.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:43.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:43.254 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:43.290 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:43.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:43.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:43 smithi204 ceph-mon[32357]: 4.5 scrub starts 2024-09-10T18:32:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:43 smithi204 ceph-mon[32357]: 4.5 scrub ok 2024-09-10T18:32:43.392 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:43.427 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:43.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:43.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:43.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:43 smithi028 ceph-mon[29291]: 4.5 scrub starts 2024-09-10T18:32:43.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:43 smithi028 ceph-mon[29291]: 4.5 scrub ok 2024-09-10T18:32:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:43 smithi028 ceph-mon[39220]: 4.5 scrub starts 2024-09-10T18:32:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:43 smithi028 ceph-mon[39220]: 4.5 scrub ok 2024-09-10T18:32:43.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:43.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:43.568 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:43.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:43.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:43.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:43.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:43.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:43.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:43.849 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:43.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:43.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:43.987 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:43.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:43.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:44.048 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 31 16 63766 63750 128.465 95.1875 0.00374708 0.00777275 2024-09-10T18:32:44.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:44.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:44.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:44.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:44.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:44.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:32:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:44.261 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:44.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:44 smithi204 ceph-mon[32357]: 6.16 scrub starts 2024-09-10T18:32:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:44 smithi204 ceph-mon[32357]: 6.16 scrub ok 2024-09-10T18:32:44.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:44 smithi204 ceph-mon[32357]: 5.d scrub starts 2024-09-10T18:32:44.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:44 smithi204 ceph-mon[32357]: 5.d scrub ok 2024-09-10T18:32:44.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:44 smithi204 ceph-mon[32357]: pgmap v501: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 709 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-10T18:32:44.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:44 smithi204 ceph-mon[32357]: 5.19 scrub starts 2024-09-10T18:32:44.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:44 smithi204 ceph-mon[32357]: 5.19 scrub ok 2024-09-10T18:32:44.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:44.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:44.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:44.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:44 smithi028 ceph-mon[29291]: 6.16 scrub starts 2024-09-10T18:32:44.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:44 smithi028 ceph-mon[29291]: 6.16 scrub ok 2024-09-10T18:32:44.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:44 smithi028 ceph-mon[29291]: 5.d scrub starts 2024-09-10T18:32:44.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:44 smithi028 ceph-mon[29291]: 5.d scrub ok 2024-09-10T18:32:44.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:44 smithi028 ceph-mon[29291]: pgmap v501: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 709 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-10T18:32:44.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:44 smithi028 ceph-mon[29291]: 5.19 scrub starts 2024-09-10T18:32:44.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:44 smithi028 ceph-mon[29291]: 5.19 scrub ok 2024-09-10T18:32:44.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:44 smithi028 ceph-mon[39220]: 6.16 scrub starts 2024-09-10T18:32:44.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:44 smithi028 ceph-mon[39220]: 6.16 scrub ok 2024-09-10T18:32:44.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:44 smithi028 ceph-mon[39220]: 5.d scrub starts 2024-09-10T18:32:44.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:44 smithi028 ceph-mon[39220]: 5.d scrub ok 2024-09-10T18:32:44.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:44 smithi028 ceph-mon[39220]: pgmap v501: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 709 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-10T18:32:44.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:44 smithi028 ceph-mon[39220]: 5.19 scrub starts 2024-09-10T18:32:44.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:44 smithi028 ceph-mon[39220]: 5.19 scrub ok 2024-09-10T18:32:44.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:44.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:44.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:44.644 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:44.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:44.676 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:44.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:44.777 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:44.814 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:44.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:44.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:44.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:32:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:32:44.915 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:44.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:44.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:44.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:45.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:45.048 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 32 16 65036 65020 126.931 79.375 0.0132333 0.00787314 2024-09-10T18:32:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:45.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:45.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:45.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:45.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:45.220 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:45.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:45.321 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:45.353 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:45.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:45.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:45 smithi204 ceph-mon[32357]: 6.14 scrub starts 2024-09-10T18:32:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:45 smithi204 ceph-mon[32357]: 6.14 scrub ok 2024-09-10T18:32:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:45 smithi204 ceph-mon[32357]: 5.15 scrub starts 2024-09-10T18:32:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:45 smithi204 ceph-mon[32357]: 5.15 scrub ok 2024-09-10T18:32:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:45 smithi204 ceph-mon[32357]: 3.a scrub starts 2024-09-10T18:32:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:45 smithi204 ceph-mon[32357]: 3.a scrub ok 2024-09-10T18:32:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:45 smithi204 ceph-mon[32357]: 4.7 scrub starts 2024-09-10T18:32:45.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:45 smithi204 ceph-mon[32357]: 4.7 scrub ok 2024-09-10T18:32:45.454 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:45.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:45.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:45.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:45 smithi028 ceph-mon[29291]: 6.14 scrub starts 2024-09-10T18:32:45.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:45 smithi028 ceph-mon[29291]: 6.14 scrub ok 2024-09-10T18:32:45.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:45 smithi028 ceph-mon[29291]: 5.15 scrub starts 2024-09-10T18:32:45.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:45 smithi028 ceph-mon[29291]: 5.15 scrub ok 2024-09-10T18:32:45.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:45 smithi028 ceph-mon[29291]: 3.a scrub starts 2024-09-10T18:32:45.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:45 smithi028 ceph-mon[29291]: 3.a scrub ok 2024-09-10T18:32:45.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:45 smithi028 ceph-mon[29291]: 4.7 scrub starts 2024-09-10T18:32:45.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:45 smithi028 ceph-mon[29291]: 4.7 scrub ok 2024-09-10T18:32:45.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:45 smithi028 ceph-mon[39220]: 6.14 scrub starts 2024-09-10T18:32:45.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:45 smithi028 ceph-mon[39220]: 6.14 scrub ok 2024-09-10T18:32:45.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:45 smithi028 ceph-mon[39220]: 5.15 scrub starts 2024-09-10T18:32:45.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:45 smithi028 ceph-mon[39220]: 5.15 scrub ok 2024-09-10T18:32:45.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:45 smithi028 ceph-mon[39220]: 3.a scrub starts 2024-09-10T18:32:45.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:45 smithi028 ceph-mon[39220]: 3.a scrub ok 2024-09-10T18:32:45.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:45 smithi028 ceph-mon[39220]: 4.7 scrub starts 2024-09-10T18:32:45.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:45 smithi028 ceph-mon[39220]: 4.7 scrub ok 2024-09-10T18:32:45.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:45.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:45.635 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:45.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:45.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:45.777 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:45.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:45.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:45.913 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:45.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:46.015 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:46.048 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:46.049 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 33 16 66151 66135 125.197 69.6875 0.0128742 0.00798271 2024-09-10T18:32:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:46.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:46.151 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:46.183 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:46.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:46.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[39220]: 6.9 scrub starts 2024-09-10T18:32:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[39220]: 6.9 scrub ok 2024-09-10T18:32:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[39220]: 6.2 scrub starts 2024-09-10T18:32:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[39220]: 6.2 scrub ok 2024-09-10T18:32:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[39220]: pgmap v502: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-09-10T18:32:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:32:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:32:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[39220]: osdmap e257: 8 total, 8 up, 8 in 2024-09-10T18:32:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[39220]: osdmap e258: 8 total, 8 up, 8 in 2024-09-10T18:32:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[29291]: 6.9 scrub starts 2024-09-10T18:32:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[29291]: 6.9 scrub ok 2024-09-10T18:32:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[29291]: 6.2 scrub starts 2024-09-10T18:32:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[29291]: 6.2 scrub ok 2024-09-10T18:32:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[29291]: pgmap v502: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-09-10T18:32:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:32:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:32:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[29291]: osdmap e257: 8 total, 8 up, 8 in 2024-09-10T18:32:46.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:46 smithi028 ceph-mon[29291]: osdmap e258: 8 total, 8 up, 8 in 2024-09-10T18:32:46.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:46.318 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:46.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:46 smithi204 ceph-mon[32357]: 6.9 scrub starts 2024-09-10T18:32:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:46 smithi204 ceph-mon[32357]: 6.9 scrub ok 2024-09-10T18:32:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:46 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:46 smithi204 ceph-mon[32357]: 6.2 scrub starts 2024-09-10T18:32:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:46 smithi204 ceph-mon[32357]: 6.2 scrub ok 2024-09-10T18:32:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:46 smithi204 ceph-mon[32357]: pgmap v502: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-09-10T18:32:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:46 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:32:46.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:46 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:32:46.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:46 smithi204 ceph-mon[32357]: osdmap e257: 8 total, 8 up, 8 in 2024-09-10T18:32:46.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:46 smithi204 ceph-mon[32357]: osdmap e258: 8 total, 8 up, 8 in 2024-09-10T18:32:46.419 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:46.447 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:46.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:46.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:46.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:46.605 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:46.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:46.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:46.709 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:46.733 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:46.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:46.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:46.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:46.857 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:46.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:46.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:46.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:47.022 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:47.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:47.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:47.048 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 34 16 67342 67326 123.704 74.4375 0.00551588 0.00807837 2024-09-10T18:32:47.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:47.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:47.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:47.240 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:32:47.240 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:32:47.245 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:47 smithi028 ceph-mon[39220]: 4.1a scrub starts 2024-09-10T18:32:47.245 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:47 smithi028 ceph-mon[39220]: 4.1a scrub ok 2024-09-10T18:32:47.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:47 smithi028 ceph-mon[39220]: 3.5 scrub starts 2024-09-10T18:32:47.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:47 smithi028 ceph-mon[39220]: 3.5 scrub ok 2024-09-10T18:32:47.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:47 smithi028 ceph-mon[29291]: 4.1a scrub starts 2024-09-10T18:32:47.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:47 smithi028 ceph-mon[29291]: 4.1a scrub ok 2024-09-10T18:32:47.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:47 smithi028 ceph-mon[29291]: 3.5 scrub starts 2024-09-10T18:32:47.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:47 smithi028 ceph-mon[29291]: 3.5 scrub ok 2024-09-10T18:32:47.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:47.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:47.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:47 smithi204 ceph-mon[32357]: 4.1a scrub starts 2024-09-10T18:32:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:47 smithi204 ceph-mon[32357]: 4.1a scrub ok 2024-09-10T18:32:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:47 smithi204 ceph-mon[32357]: 3.5 scrub starts 2024-09-10T18:32:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:47 smithi204 ceph-mon[32357]: 3.5 scrub ok 2024-09-10T18:32:47.416 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:47.445 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:47.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:47.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:47.585 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:47.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:47.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:47.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:47.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:47.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:47.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:47.871 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:47.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:47.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:47.973 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:48.006 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:48.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:48.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:48.048 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 35 16 68767 68751 122.715 89.0625 0.00817652 0.00814442 2024-09-10T18:32:48.108 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:48.137 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:48.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:48.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:48.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-10T18:32:48.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:48.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:48.317 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:48.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:48 smithi204 ceph-mon[32357]: 4.11 scrub starts 2024-09-10T18:32:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:48 smithi204 ceph-mon[32357]: 4.11 scrub ok 2024-09-10T18:32:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:48 smithi204 ceph-mon[32357]: 6.e scrub starts 2024-09-10T18:32:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:48 smithi204 ceph-mon[32357]: 6.e scrub ok 2024-09-10T18:32:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:48 smithi204 ceph-mon[32357]: pgmap v505: 158 pgs: 158 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 822 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-10T18:32:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:48 smithi204 ceph-mon[32357]: 3.10 scrub starts 2024-09-10T18:32:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:48 smithi204 ceph-mon[32357]: 3.10 scrub ok 2024-09-10T18:32:48.418 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:48.448 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:48.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:48.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:48 smithi028 ceph-mon[29291]: 4.11 scrub starts 2024-09-10T18:32:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:48 smithi028 ceph-mon[29291]: 4.11 scrub ok 2024-09-10T18:32:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:48 smithi028 ceph-mon[29291]: 6.e scrub starts 2024-09-10T18:32:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:48 smithi028 ceph-mon[29291]: 6.e scrub ok 2024-09-10T18:32:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:48 smithi028 ceph-mon[29291]: pgmap v505: 158 pgs: 158 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 822 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-10T18:32:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:48 smithi028 ceph-mon[29291]: 3.10 scrub starts 2024-09-10T18:32:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:48 smithi028 ceph-mon[29291]: 3.10 scrub ok 2024-09-10T18:32:48.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:48 smithi028 ceph-mon[39220]: 4.11 scrub starts 2024-09-10T18:32:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:48 smithi028 ceph-mon[39220]: 4.11 scrub ok 2024-09-10T18:32:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:48 smithi028 ceph-mon[39220]: 6.e scrub starts 2024-09-10T18:32:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:48 smithi028 ceph-mon[39220]: 6.e scrub ok 2024-09-10T18:32:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:48 smithi028 ceph-mon[39220]: pgmap v505: 158 pgs: 158 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 822 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-10T18:32:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:48 smithi028 ceph-mon[39220]: 3.10 scrub starts 2024-09-10T18:32:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:48 smithi028 ceph-mon[39220]: 3.10 scrub ok 2024-09-10T18:32:48.549 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:48.580 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:48.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:48.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:48.682 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:48.707 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:48.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:48.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:48.765 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:48.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:48.880 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:48.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:48.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:48.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:48.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:32:49.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:49.026 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:49.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:49.048 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 36 16 70049 70033 121.532 80.125 0.0081375 0.00822296 2024-09-10T18:32:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:49.127 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:49.157 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:49.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:49.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:49.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:49 smithi028 ceph-mon[29291]: 4.10 scrub starts 2024-09-10T18:32:49.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:49 smithi028 ceph-mon[29291]: 4.10 scrub ok 2024-09-10T18:32:49.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:49 smithi028 ceph-mon[29291]: 1.6 scrub starts 2024-09-10T18:32:49.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:49 smithi028 ceph-mon[29291]: 1.6 scrub ok 2024-09-10T18:32:49.221 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:49 smithi028 ceph-mon[39220]: 4.10 scrub starts 2024-09-10T18:32:49.221 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:49 smithi028 ceph-mon[39220]: 4.10 scrub ok 2024-09-10T18:32:49.221 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:49 smithi028 ceph-mon[39220]: 1.6 scrub starts 2024-09-10T18:32:49.221 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:49 smithi028 ceph-mon[39220]: 1.6 scrub ok 2024-09-10T18:32:49.259 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:49.287 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:49.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:49.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:49 smithi204 ceph-mon[32357]: 4.10 scrub starts 2024-09-10T18:32:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:49 smithi204 ceph-mon[32357]: 4.10 scrub ok 2024-09-10T18:32:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:49 smithi204 ceph-mon[32357]: 1.6 scrub starts 2024-09-10T18:32:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:49 smithi204 ceph-mon[32357]: 1.6 scrub ok 2024-09-10T18:32:49.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:49.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:49.419 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:49.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:49.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:49.558 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:49.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:49.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:49.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:49.705 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:49.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:49.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:49.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:49.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:49.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:49.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:49.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:50.000 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:50.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:50.049 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 37 16 71407 71391 120.541 84.875 0.0143619 0.00829121 2024-09-10T18:32:50.102 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:set 4.c pg_upmap mapping to [7,4,2] 2024-09-10T18:32:50.138 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:50.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:50.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:50.239 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:50.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:50.277 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:50.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:50.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:50 smithi028 ceph-mon[39220]: 5.b scrub starts 2024-09-10T18:32:50.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:50 smithi028 ceph-mon[39220]: 4.1b scrub starts 2024-09-10T18:32:50.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:50 smithi028 ceph-mon[39220]: 5.b scrub ok 2024-09-10T18:32:50.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:50 smithi028 ceph-mon[39220]: 4.1b scrub ok 2024-09-10T18:32:50.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:50 smithi028 ceph-mon[39220]: pgmap v506: 158 pgs: 158 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 914 op/s 2024-09-10T18:32:50.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:50 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2794495267' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.c", "id": [7, 4, 2]}]: dispatch 2024-09-10T18:32:50.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:50 smithi028 ceph-mon[39220]: from='client.25351 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.c", "id": [7, 4, 2]}]: dispatch 2024-09-10T18:32:50.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:50 smithi028 ceph-mon[29291]: 5.b scrub starts 2024-09-10T18:32:50.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:50 smithi028 ceph-mon[29291]: 4.1b scrub starts 2024-09-10T18:32:50.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:50 smithi028 ceph-mon[29291]: 5.b scrub ok 2024-09-10T18:32:50.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:50 smithi028 ceph-mon[29291]: 4.1b scrub ok 2024-09-10T18:32:50.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:50 smithi028 ceph-mon[29291]: pgmap v506: 158 pgs: 158 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 914 op/s 2024-09-10T18:32:50.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:50 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2794495267' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.c", "id": [7, 4, 2]}]: dispatch 2024-09-10T18:32:50.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:50 smithi028 ceph-mon[29291]: from='client.25351 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.c", "id": [7, 4, 2]}]: dispatch 2024-09-10T18:32:50.378 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:50.409 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:50.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:50.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:50.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:50.541 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:50.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:50.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:50 smithi204 ceph-mon[32357]: 5.b scrub starts 2024-09-10T18:32:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:50 smithi204 ceph-mon[32357]: 4.1b scrub starts 2024-09-10T18:32:50.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:50 smithi204 ceph-mon[32357]: 5.b scrub ok 2024-09-10T18:32:50.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:50 smithi204 ceph-mon[32357]: 4.1b scrub ok 2024-09-10T18:32:50.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:50 smithi204 ceph-mon[32357]: pgmap v506: 158 pgs: 158 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 914 op/s 2024-09-10T18:32:50.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:50 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2794495267' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.c", "id": [7, 4, 2]}]: dispatch 2024-09-10T18:32:50.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:50 smithi204 ceph-mon[32357]: from='client.25351 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.c", "id": [7, 4, 2]}]: dispatch 2024-09-10T18:32:50.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:50.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:50.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:50.789 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:50.819 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:50.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:50.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:50.921 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:50.957 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:50.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:50.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:51.049 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 38 16 72816 72800 119.687 88.0625 0.00604669 0.00835001 2024-09-10T18:32:51.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:51.087 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:51.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:51.189 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:51.221 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:51.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:51.323 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:51.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:51.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:51.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:51.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:51.500 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:51.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:51.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:51.505 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:32:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:32:51] "GET /metrics HTTP/1.1" 200 224400 "" "Prometheus/2.43.0" 2024-09-10T18:32:51.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:51 smithi028 ceph-mon[39220]: 3.7 scrub starts 2024-09-10T18:32:51.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:51 smithi028 ceph-mon[39220]: 3.7 scrub ok 2024-09-10T18:32:51.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:51 smithi028 ceph-mon[39220]: 6.7 scrub starts 2024-09-10T18:32:51.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:51 smithi028 ceph-mon[39220]: 6.7 scrub ok 2024-09-10T18:32:51.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:51 smithi028 ceph-mon[39220]: from='client.25351 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.c", "id": [7, 4, 2]}]': finished 2024-09-10T18:32:51.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:51 smithi028 ceph-mon[39220]: osdmap e259: 8 total, 8 up, 8 in 2024-09-10T18:32:51.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:51 smithi028 ceph-mon[39220]: 5.c scrub starts 2024-09-10T18:32:51.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:51 smithi028 ceph-mon[39220]: 5.c scrub ok 2024-09-10T18:32:51.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:51 smithi028 ceph-mon[29291]: 3.7 scrub starts 2024-09-10T18:32:51.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:51 smithi028 ceph-mon[29291]: 3.7 scrub ok 2024-09-10T18:32:51.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:51 smithi028 ceph-mon[29291]: 6.7 scrub starts 2024-09-10T18:32:51.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:51 smithi028 ceph-mon[29291]: 6.7 scrub ok 2024-09-10T18:32:51.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:51 smithi028 ceph-mon[29291]: from='client.25351 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.c", "id": [7, 4, 2]}]': finished 2024-09-10T18:32:51.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:51 smithi028 ceph-mon[29291]: osdmap e259: 8 total, 8 up, 8 in 2024-09-10T18:32:51.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:51 smithi028 ceph-mon[29291]: 5.c scrub starts 2024-09-10T18:32:51.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:51 smithi028 ceph-mon[29291]: 5.c scrub ok 2024-09-10T18:32:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:51 smithi204 ceph-mon[32357]: 3.7 scrub starts 2024-09-10T18:32:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:51 smithi204 ceph-mon[32357]: 3.7 scrub ok 2024-09-10T18:32:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:51 smithi204 ceph-mon[32357]: 6.7 scrub starts 2024-09-10T18:32:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:51 smithi204 ceph-mon[32357]: 6.7 scrub ok 2024-09-10T18:32:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:51 smithi204 ceph-mon[32357]: from='client.25351 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.c", "id": [7, 4, 2]}]': finished 2024-09-10T18:32:51.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:51 smithi204 ceph-mon[32357]: osdmap e259: 8 total, 8 up, 8 in 2024-09-10T18:32:51.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:51 smithi204 ceph-mon[32357]: 5.c scrub starts 2024-09-10T18:32:51.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:51 smithi204 ceph-mon[32357]: 5.c scrub ok 2024-09-10T18:32:51.606 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:51.644 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:51.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:51.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:51.746 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:51.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:51.776 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:51.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:51.877 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:51.913 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:51.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:51.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:52.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:52.052 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:52.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:52.065 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 39 16 74224 74208 118.825 88 0.00290982 0.00840702 2024-09-10T18:32:52.155 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:52.186 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:52.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:52.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:52.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[29291]: 4.1d scrub starts 2024-09-10T18:32:52.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[29291]: 4.1d scrub ok 2024-09-10T18:32:52.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[29291]: 5.8 scrub starts 2024-09-10T18:32:52.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[29291]: 5.8 scrub ok 2024-09-10T18:32:52.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[29291]: 6.d scrub starts 2024-09-10T18:32:52.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[29291]: 6.d scrub ok 2024-09-10T18:32:52.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[29291]: pgmap v508: 158 pgs: 158 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 110 MiB/s wr, 1.75k op/s 2024-09-10T18:32:52.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[29291]: osdmap e260: 8 total, 8 up, 8 in 2024-09-10T18:32:52.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[29291]: 3.19 scrub starts 2024-09-10T18:32:52.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[29291]: 3.19 scrub ok 2024-09-10T18:32:52.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[39220]: 4.1d scrub starts 2024-09-10T18:32:52.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[39220]: 4.1d scrub ok 2024-09-10T18:32:52.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[39220]: 5.8 scrub starts 2024-09-10T18:32:52.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[39220]: 5.8 scrub ok 2024-09-10T18:32:52.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[39220]: 6.d scrub starts 2024-09-10T18:32:52.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[39220]: 6.d scrub ok 2024-09-10T18:32:52.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[39220]: pgmap v508: 158 pgs: 158 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 110 MiB/s wr, 1.75k op/s 2024-09-10T18:32:52.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[39220]: osdmap e260: 8 total, 8 up, 8 in 2024-09-10T18:32:52.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[39220]: 3.19 scrub starts 2024-09-10T18:32:52.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:52 smithi028 ceph-mon[39220]: 3.19 scrub ok 2024-09-10T18:32:52.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:52.327 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:52.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:52.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:52.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:52.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:52.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:52 smithi204 ceph-mon[32357]: 4.1d scrub starts 2024-09-10T18:32:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:52 smithi204 ceph-mon[32357]: 4.1d scrub ok 2024-09-10T18:32:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:52 smithi204 ceph-mon[32357]: 5.8 scrub starts 2024-09-10T18:32:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:52 smithi204 ceph-mon[32357]: 5.8 scrub ok 2024-09-10T18:32:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:52 smithi204 ceph-mon[32357]: 6.d scrub starts 2024-09-10T18:32:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:52 smithi204 ceph-mon[32357]: 6.d scrub ok 2024-09-10T18:32:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:52 smithi204 ceph-mon[32357]: pgmap v508: 158 pgs: 158 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 110 MiB/s wr, 1.75k op/s 2024-09-10T18:32:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:52 smithi204 ceph-mon[32357]: osdmap e260: 8 total, 8 up, 8 in 2024-09-10T18:32:52.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:52 smithi204 ceph-mon[32357]: 3.19 scrub starts 2024-09-10T18:32:52.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:52 smithi204 ceph-mon[32357]: 3.19 scrub ok 2024-09-10T18:32:52.506 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:32:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:32:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:32:52.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:52.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:52.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:52.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:52.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:52.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:52.755 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:52.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:52.796 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:52.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:52.883 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:52.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:52.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:52.984 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:53.018 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:53.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:53.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:53.065 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:32:53.064473+0000 min lat: 0.00134629 max lat: 0.235622 avg lat: 0.00846911 2024-09-10T18:32:53.065 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:32:53.065 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 40 16 75599 75583 118.003 85.9375 0.00508065 0.00846911 2024-09-10T18:32:53.120 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:53.148 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:53.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:53.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:53.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:53.259 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:32:53.260 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:32:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:53.293 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:53.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:53.301 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:32:53.301 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:32:53.334 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:32:53.334 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:32:53.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:32:53.365 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:32:53.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:53.399 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:32:53.399 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:32:53.430 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:32:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:32:53.437 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:53.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:53.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:53.465 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:32:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:32:53.504 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:32:53.504 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:32:53.515 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:32:53.515 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:32:53.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:53.583 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:53.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:53.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:53.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:53.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:53.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:53.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:53.859 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:53.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:53.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:53.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:53.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:53.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:53.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:54.065 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 41 16 76907 76891 117.119 81.75 0.015021 0.00853312 2024-09-10T18:32:54.099 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:54.129 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:54.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:54.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:54.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:54.274 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:54.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:54.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:54.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:54.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:54.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:54 smithi204 ceph-mon[32357]: 5.1 scrub starts 2024-09-10T18:32:54.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:54 smithi204 ceph-mon[32357]: 5.1 scrub ok 2024-09-10T18:32:54.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:54 smithi204 ceph-mon[32357]: 5.a scrub starts 2024-09-10T18:32:54.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:54 smithi204 ceph-mon[32357]: 5.a scrub ok 2024-09-10T18:32:54.502 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:54 smithi204 ceph-mon[32357]: pgmap v510: 158 pgs: 158 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 708 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-09-10T18:32:54.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:54 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1099907465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:32:54.503 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:54 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:32:54.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:54 smithi028 ceph-mon[29291]: 5.1 scrub starts 2024-09-10T18:32:54.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:54 smithi028 ceph-mon[29291]: 5.1 scrub ok 2024-09-10T18:32:54.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:54 smithi028 ceph-mon[29291]: 5.a scrub starts 2024-09-10T18:32:54.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:54 smithi028 ceph-mon[29291]: 5.a scrub ok 2024-09-10T18:32:54.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:54 smithi028 ceph-mon[29291]: pgmap v510: 158 pgs: 158 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 708 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-09-10T18:32:54.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:54 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1099907465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:32:54.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:54 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:32:54.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:54 smithi028 ceph-mon[39220]: 5.1 scrub starts 2024-09-10T18:32:54.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:54 smithi028 ceph-mon[39220]: 5.1 scrub ok 2024-09-10T18:32:54.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:54 smithi028 ceph-mon[39220]: 5.a scrub starts 2024-09-10T18:32:54.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:54 smithi028 ceph-mon[39220]: 5.a scrub ok 2024-09-10T18:32:54.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:54 smithi028 ceph-mon[39220]: pgmap v510: 158 pgs: 158 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 708 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-09-10T18:32:54.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:54 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1099907465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:32:54.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:54 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:32:54.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:54.553 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:54.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:54.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:54.655 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:54.689 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:54.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:54.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:54.828 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:54.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:54.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:32:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:32:54.929 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:54.961 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:54.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:54.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:55.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:55.065 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 42 16 78320 78304 116.433 88.3125 0.012017 0.0085836 2024-09-10T18:32:55.094 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:55.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:55.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-10T18:32:55.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:55.254 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:55.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:55.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:55.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:55.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:55.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:55.498 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:55.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:55 smithi028 ceph-mon[29291]: 4.6 scrub starts 2024-09-10T18:32:55.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:55 smithi028 ceph-mon[29291]: 4.6 scrub ok 2024-09-10T18:32:55.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:55 smithi028 ceph-mon[29291]: 5.6 scrub starts 2024-09-10T18:32:55.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:55 smithi028 ceph-mon[29291]: 5.6 scrub ok 2024-09-10T18:32:55.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:55 smithi028 ceph-mon[39220]: 4.6 scrub starts 2024-09-10T18:32:55.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:55 smithi028 ceph-mon[39220]: 4.6 scrub ok 2024-09-10T18:32:55.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:55 smithi028 ceph-mon[39220]: 5.6 scrub starts 2024-09-10T18:32:55.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:55 smithi028 ceph-mon[39220]: 5.6 scrub ok 2024-09-10T18:32:55.531 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:55.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:55.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:55.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:55 smithi204 ceph-mon[32357]: 4.6 scrub starts 2024-09-10T18:32:55.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:55 smithi204 ceph-mon[32357]: 4.6 scrub ok 2024-09-10T18:32:55.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:55 smithi204 ceph-mon[32357]: 5.6 scrub starts 2024-09-10T18:32:55.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:55 smithi204 ceph-mon[32357]: 5.6 scrub ok 2024-09-10T18:32:55.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:55.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:55.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:55.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:55.835 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:55.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:55.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:55.938 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:55.974 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:55.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:55.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:56.065 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 43 16 79631 79615 115.632 81.9375 0.00698863 0.0086435 2024-09-10T18:32:56.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:56.118 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:56.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:56.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:56.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:56.252 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:56.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:56.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:56.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:56.397 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:56.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:56.401 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:56.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:56.502 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:56.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[29291]: 3.6 scrub starts 2024-09-10T18:32:56.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[29291]: 3.6 scrub ok 2024-09-10T18:32:56.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:56.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[29291]: 4.9 scrub starts 2024-09-10T18:32:56.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[29291]: 4.9 scrub ok 2024-09-10T18:32:56.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[29291]: pgmap v511: 158 pgs: 158 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:32:56.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[29291]: 4.2 scrub starts 2024-09-10T18:32:56.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[29291]: 4.2 scrub ok 2024-09-10T18:32:56.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[29291]: osdmap e261: 8 total, 8 up, 8 in 2024-09-10T18:32:56.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[39220]: 3.6 scrub starts 2024-09-10T18:32:56.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[39220]: 3.6 scrub ok 2024-09-10T18:32:56.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:56.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[39220]: 4.9 scrub starts 2024-09-10T18:32:56.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[39220]: 4.9 scrub ok 2024-09-10T18:32:56.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[39220]: pgmap v511: 158 pgs: 158 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:32:56.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[39220]: 4.2 scrub starts 2024-09-10T18:32:56.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[39220]: 4.2 scrub ok 2024-09-10T18:32:56.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:56 smithi028 ceph-mon[39220]: osdmap e261: 8 total, 8 up, 8 in 2024-09-10T18:32:56.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:56.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:56.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:56 smithi204 ceph-mon[32357]: 3.6 scrub starts 2024-09-10T18:32:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:56 smithi204 ceph-mon[32357]: 3.6 scrub ok 2024-09-10T18:32:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:56 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:32:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:56 smithi204 ceph-mon[32357]: 4.9 scrub starts 2024-09-10T18:32:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:56 smithi204 ceph-mon[32357]: 4.9 scrub ok 2024-09-10T18:32:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:56 smithi204 ceph-mon[32357]: pgmap v511: 158 pgs: 158 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:32:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:56 smithi204 ceph-mon[32357]: 4.2 scrub starts 2024-09-10T18:32:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:56 smithi204 ceph-mon[32357]: 4.2 scrub ok 2024-09-10T18:32:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:56 smithi204 ceph-mon[32357]: osdmap e261: 8 total, 8 up, 8 in 2024-09-10T18:32:56.641 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:56.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:56.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:56.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:56.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:56.804 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:56.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:56.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:56.906 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:56.937 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:56.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:56.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:57.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:57.065 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 44 16 80970 80954 114.906 83.6875 0.0090449 0.00869846 2024-09-10T18:32:57.079 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:57.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:57.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:57.181 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:57.215 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:57.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:57.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:57.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:57.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:57.363 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:57.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:57.464 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:57.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:57 smithi028 ceph-mon[39220]: 4.19 scrub starts 2024-09-10T18:32:57.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:57 smithi028 ceph-mon[39220]: 4.19 scrub ok 2024-09-10T18:32:57.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:57 smithi028 ceph-mon[39220]: 4.4 scrub starts 2024-09-10T18:32:57.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:57 smithi028 ceph-mon[39220]: 4.4 scrub ok 2024-09-10T18:32:57.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:57 smithi028 ceph-mon[39220]: 4.f scrub starts 2024-09-10T18:32:57.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:57 smithi028 ceph-mon[39220]: 4.f scrub ok 2024-09-10T18:32:57.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:57 smithi028 ceph-mon[29291]: 4.19 scrub starts 2024-09-10T18:32:57.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:57 smithi028 ceph-mon[29291]: 4.19 scrub ok 2024-09-10T18:32:57.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:57 smithi028 ceph-mon[29291]: 4.4 scrub starts 2024-09-10T18:32:57.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:57 smithi028 ceph-mon[29291]: 4.4 scrub ok 2024-09-10T18:32:57.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:57 smithi028 ceph-mon[29291]: 4.f scrub starts 2024-09-10T18:32:57.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:57 smithi028 ceph-mon[29291]: 4.f scrub ok 2024-09-10T18:32:57.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:57.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:57.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:57.560 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:32:57.561 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:32:57.561 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-10T18:32:57.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T18:32:57.597 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:57 smithi204 ceph-mon[32357]: 4.19 scrub starts 2024-09-10T18:32:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:57 smithi204 ceph-mon[32357]: 4.19 scrub ok 2024-09-10T18:32:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:57 smithi204 ceph-mon[32357]: 4.4 scrub starts 2024-09-10T18:32:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:57 smithi204 ceph-mon[32357]: 4.4 scrub ok 2024-09-10T18:32:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:57 smithi204 ceph-mon[32357]: 4.f scrub starts 2024-09-10T18:32:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:57 smithi204 ceph-mon[32357]: 4.f scrub ok 2024-09-10T18:32:57.635 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:57.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:57.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:57.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:57.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:32:57.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:57.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:57.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:57.959 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:57.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:57.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:32:58.061 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:58.067 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 45 16 82147 82131 113.987 73.5625 0.00816013 0.00876759 2024-09-10T18:32:58.090 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:58.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:58.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:58.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:58.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:58.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:58.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:58.371 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:58.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:58.472 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:58.500 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:58.500 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:58.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:58.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:58 smithi028 ceph-mon[39220]: 4.17 scrub starts 2024-09-10T18:32:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:58 smithi028 ceph-mon[39220]: 4.17 scrub ok 2024-09-10T18:32:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:58 smithi028 ceph-mon[39220]: 5.7 scrub starts 2024-09-10T18:32:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:58 smithi028 ceph-mon[39220]: 5.7 scrub ok 2024-09-10T18:32:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:58 smithi028 ceph-mon[39220]: pgmap v513: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 594 B/s rd, 96 MiB/s wr, 1.53k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:32:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:58 smithi028 ceph-mon[29291]: 4.17 scrub starts 2024-09-10T18:32:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:58 smithi028 ceph-mon[29291]: 4.17 scrub ok 2024-09-10T18:32:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:58 smithi028 ceph-mon[29291]: 5.7 scrub starts 2024-09-10T18:32:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:58 smithi028 ceph-mon[29291]: 5.7 scrub ok 2024-09-10T18:32:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:58 smithi028 ceph-mon[29291]: pgmap v513: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 594 B/s rd, 96 MiB/s wr, 1.53k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:32:58.524 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:32:58.602 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:32:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:58 smithi204 ceph-mon[32357]: 4.17 scrub starts 2024-09-10T18:32:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:58 smithi204 ceph-mon[32357]: 4.17 scrub ok 2024-09-10T18:32:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:58 smithi204 ceph-mon[32357]: 5.7 scrub starts 2024-09-10T18:32:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:58 smithi204 ceph-mon[32357]: 5.7 scrub ok 2024-09-10T18:32:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:58 smithi204 ceph-mon[32357]: pgmap v513: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 594 B/s rd, 96 MiB/s wr, 1.53k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:32:58.647 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:58.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:58.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:32:58.749 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:58.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:58.777 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:58.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:58.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:58.912 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:58.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:58.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:59.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:32:59.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:59.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:32:59.066 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 46 16 83343 83327 113.134 74.75 0.00838999 0.00883436 2024-09-10T18:32:59.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:32:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:32:59.190 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:59.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:59.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:32:59.277 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:32:59.277 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-10T18:32:59.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-10T18:32:59.277 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:32:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31752 7.d 7:b2ee7856:::benchmark_data_smithi028_77856_object31751:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:32:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.836508+0000", 2024-09-10T18:32:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 541.43578811899999, 2024-09-10T18:32:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14737983199999999, 2024-09-10T18:32:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31752 2024-09-10T18:32:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.278 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836508+0000", 2024-09-10T18:32:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836508+0000", 2024-09-10T18:32:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836506+0000", 2024-09-10T18:32:59.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836529+0000", 2024-09-10T18:32:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2852000000000001e-05 2024-09-10T18:32:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836539+0000", 2024-09-10T18:32:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.005e-05 2024-09-10T18:32:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836545+0000", 2024-09-10T18:32:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2580000000000003e-06 2024-09-10T18:32:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836594+0000", 2024-09-10T18:32:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8971000000000003e-05 2024-09-10T18:32:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836701+0000", 2024-09-10T18:32:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107365 2024-09-10T18:32:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-10T18:32:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.836834+0000", 2024-09-10T18:32:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013344799999999999 2024-09-10T18:32:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.839823+0000", 2024-09-10T18:32:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029885760000000002 2024-09-10T18:32:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.983711+0000", 2024-09-10T18:32:59.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14388826800000001 2024-09-10T18:32:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.983767+0000", 2024-09-10T18:32:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5339e-05 2024-09-10T18:32:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.983888+0000", 2024-09-10T18:32:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121433 2024-09-10T18:32:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.283 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:31760 7.d 7:be0b06ed:::benchmark_data_smithi028_77856_object31759:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:32:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:57.838440+0000", 2024-09-10T18:32:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 541.43385655899999, 2024-09-10T18:32:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14565720700000001, 2024-09-10T18:32:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31760 2024-09-10T18:32:59.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838440+0000", 2024-09-10T18:32:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838440+0000", 2024-09-10T18:32:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838439+0000", 2024-09-10T18:32:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:32:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838463+0000", 2024-09-10T18:32:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4530000000000001e-05 2024-09-10T18:32:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838471+0000", 2024-09-10T18:32:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0399999999999993e-06 2024-09-10T18:32:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838476+0000", 2024-09-10T18:32:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6609999999999999e-06 2024-09-10T18:32:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838534+0000", 2024-09-10T18:32:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8017999999999999e-05 2024-09-10T18:32:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838664+0000", 2024-09-10T18:32:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129605 2024-09-10T18:32:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-10T18:32:59.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.838791+0000", 2024-09-10T18:32:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012758999999999999 2024-09-10T18:32:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.840006+0000", 2024-09-10T18:32:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012147099999999999 2024-09-10T18:32:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.983943+0000", 2024-09-10T18:32:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14393716400000001 2024-09-10T18:32:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.984008+0000", 2024-09-10T18:32:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4768000000000001e-05 2024-09-10T18:32:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:57.984097+0000", 2024-09-10T18:32:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9123999999999996e-05 2024-09-10T18:32:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181183 7.d 7:b1361f07:::benchmark_data_smithi028_77856_object60083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:32:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.055163+0000", 2024-09-10T18:32:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 394.21713378200002, 2024-09-10T18:32:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15025240300000001, 2024-09-10T18:32:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181183 2024-09-10T18:32:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055163+0000", 2024-09-10T18:32:59.291 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.291 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.291 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.291 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055163+0000", 2024-09-10T18:32:59.291 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:32:59.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055160+0000", 2024-09-10T18:32:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055167+0000", 2024-09-10T18:32:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9469999999999997e-06 2024-09-10T18:32:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055175+0000", 2024-09-10T18:32:59.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1330000000000001e-06 2024-09-10T18:32:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055180+0000", 2024-09-10T18:32:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1699999999999996e-06 2024-09-10T18:32:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055225+0000", 2024-09-10T18:32:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5441000000000001e-05 2024-09-10T18:32:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055329+0000", 2024-09-10T18:32:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103892 2024-09-10T18:32:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-10T18:32:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055440+0000", 2024-09-10T18:32:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011092800000000001 2024-09-10T18:32:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.057624+0000", 2024-09-10T18:32:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021838669999999999 2024-09-10T18:32:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205284+0000", 2024-09-10T18:32:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14766003999999999 2024-09-10T18:32:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205332+0000", 2024-09-10T18:32:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7800000000000003e-05 2024-09-10T18:32:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205415+0000", 2024-09-10T18:32:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3227999999999997e-05 2024-09-10T18:32:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.296 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221438 7.b 7:d4df1cfe:::benchmark_data_smithi028_77856_object100338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:32:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.137682+0000", 2024-09-10T18:32:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 351.13461440200001, 2024-09-10T18:32:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99608778499999995, 2024-09-10T18:32:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.297 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221438 2024-09-10T18:32:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.137682+0000", 2024-09-10T18:32:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.137682+0000", 2024-09-10T18:32:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.137680+0000", 2024-09-10T18:32:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:32:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.137687+0000", 2024-09-10T18:32:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.469e-06 2024-09-10T18:32:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.137705+0000", 2024-09-10T18:32:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7915e-05 2024-09-10T18:32:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.137712+0000", 2024-09-10T18:32:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5740000000000001e-06 2024-09-10T18:32:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.137758+0000", 2024-09-10T18:32:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6362000000000001e-05 2024-09-10T18:32:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:32:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.137765+0000", 2024-09-10T18:32:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3209999999999999e-06 2024-09-10T18:32:59.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.129246+0000", 2024-09-10T18:32:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99148095599999997 2024-09-10T18:32:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.129370+0000", 2024-09-10T18:32:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012408100000000001 2024-09-10T18:32:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:32:59.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.129441+0000", 2024-09-10T18:32:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0684000000000003e-05 2024-09-10T18:32:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133644+0000", 2024-09-10T18:32:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042028459999999997 2024-09-10T18:32:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133680+0000", 2024-09-10T18:32:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5540999999999997e-05 2024-09-10T18:32:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133710+0000", 2024-09-10T18:32:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0059000000000001e-05 2024-09-10T18:32:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133747+0000", 2024-09-10T18:32:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7826e-05 2024-09-10T18:32:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133770+0000", 2024-09-10T18:32:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2313000000000001e-05 2024-09-10T18:32:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.303 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221491 7.b 7:de887b9b:::benchmark_data_smithi028_77856_object100391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:32:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.187950+0000", 2024-09-10T18:32:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 351.08434601900001, 2024-09-10T18:32:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.94855177599999996, 2024-09-10T18:32:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221491 2024-09-10T18:32:59.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.187950+0000", 2024-09-10T18:32:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.187950+0000", 2024-09-10T18:32:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.187948+0000", 2024-09-10T18:32:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-10T18:32:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.187958+0000", 2024-09-10T18:32:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8379999999999997e-06 2024-09-10T18:32:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.187973+0000", 2024-09-10T18:32:59.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.517e-05 2024-09-10T18:32:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.187980+0000", 2024-09-10T18:32:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2830000000000004e-06 2024-09-10T18:32:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.188030+0000", 2024-09-10T18:32:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0408000000000003e-05 2024-09-10T18:32:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:32:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.188039+0000", 2024-09-10T18:32:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8179999999999993e-06 2024-09-10T18:32:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.131369+0000", 2024-09-10T18:32:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.94332941599999998 2024-09-10T18:32:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.131435+0000", 2024-09-10T18:32:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6177e-05 2024-09-10T18:32:59.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:32:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.131495+0000", 2024-09-10T18:32:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0163e-05 2024-09-10T18:32:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133809+0000", 2024-09-10T18:32:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023145079999999998 2024-09-10T18:32:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133844+0000", 2024-09-10T18:32:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4686e-05 2024-09-10T18:32:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136445+0000", 2024-09-10T18:32:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002600799 2024-09-10T18:32:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136472+0000", 2024-09-10T18:32:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7078999999999999e-05 2024-09-10T18:32:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136502+0000", 2024-09-10T18:32:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0068000000000001e-05 2024-09-10T18:32:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221505 7.b 7:d0875639:::benchmark_data_smithi028_77856_object100405:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:32:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.200926+0000", 2024-09-10T18:32:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 351.07136992800002, 2024-09-10T18:32:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93569294999999997, 2024-09-10T18:32:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.311 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221505 2024-09-10T18:32:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.200926+0000", 2024-09-10T18:32:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.200926+0000", 2024-09-10T18:32:59.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.200924+0000", 2024-09-10T18:32:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-10T18:32:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.200933+0000", 2024-09-10T18:32:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.431e-06 2024-09-10T18:32:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.200983+0000", 2024-09-10T18:32:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0013999999999999e-05 2024-09-10T18:32:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.200990+0000", 2024-09-10T18:32:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2359999999999998e-06 2024-09-10T18:32:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.201059+0000", 2024-09-10T18:32:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8227999999999998e-05 2024-09-10T18:32:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:32:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.201067+0000", 2024-09-10T18:32:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9689999999999994e-06 2024-09-10T18:32:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.132417+0000", 2024-09-10T18:32:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93135020400000001 2024-09-10T18:32:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.132482+0000", 2024-09-10T18:32:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4906999999999994e-05 2024-09-10T18:32:59.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:32:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.132546+0000", 2024-09-10T18:32:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3997000000000004e-05 2024-09-10T18:32:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133891+0000", 2024-09-10T18:32:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001344837 2024-09-10T18:32:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133898+0000", 2024-09-10T18:32:59.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6809999999999998e-06 2024-09-10T18:32:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136546+0000", 2024-09-10T18:32:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002647974 2024-09-10T18:32:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136571+0000", 2024-09-10T18:32:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4953e-05 2024-09-10T18:32:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136619+0000", 2024-09-10T18:32:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8226000000000002e-05 2024-09-10T18:32:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221531 7.b 7:d7e9de4a:::benchmark_data_smithi028_77856_object100431:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:32:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.222649+0000", 2024-09-10T18:32:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 351.049646962, 2024-09-10T18:32:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.91614740400000005, 2024-09-10T18:32:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.318 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221531 2024-09-10T18:32:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.222649+0000", 2024-09-10T18:32:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.222649+0000", 2024-09-10T18:32:59.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.222647+0000", 2024-09-10T18:32:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:32:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.222656+0000", 2024-09-10T18:32:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6889999999999993e-06 2024-09-10T18:32:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.222672+0000", 2024-09-10T18:32:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6549999999999999e-05 2024-09-10T18:32:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.222679+0000", 2024-09-10T18:32:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1199999999999999e-06 2024-09-10T18:32:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.222720+0000", 2024-09-10T18:32:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1464999999999997e-05 2024-09-10T18:32:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:32:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.222727+0000", 2024-09-10T18:32:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.238e-06 2024-09-10T18:32:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.133925+0000", 2024-09-10T18:32:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.91119738800000005 2024-09-10T18:32:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.134024+0000", 2024-09-10T18:32:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9692999999999996e-05 2024-09-10T18:32:59.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:32:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.134105+0000", 2024-09-10T18:32:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1087000000000005e-05 2024-09-10T18:32:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136631+0000", 2024-09-10T18:32:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025255 2024-09-10T18:32:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136692+0000", 2024-09-10T18:32:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0866999999999998e-05 2024-09-10T18:32:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138713+0000", 2024-09-10T18:32:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020208959999999999 2024-09-10T18:32:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138759+0000", 2024-09-10T18:32:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6530999999999999e-05 2024-09-10T18:32:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.324 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138797+0000", 2024-09-10T18:32:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7595000000000002e-05 2024-09-10T18:32:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221533 7.b 7:d065aacb:::benchmark_data_smithi028_77856_object100433:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:32:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.225307+0000", 2024-09-10T18:32:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 351.04698885900001, 2024-09-10T18:32:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.91623593000000003, 2024-09-10T18:32:59.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221533 2024-09-10T18:32:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.225307+0000", 2024-09-10T18:32:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.225307+0000", 2024-09-10T18:32:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.225305+0000", 2024-09-10T18:32:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:32:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.225313+0000", 2024-09-10T18:32:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4300000000000006e-06 2024-09-10T18:32:59.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.225331+0000", 2024-09-10T18:32:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.77e-05 2024-09-10T18:32:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.225363+0000", 2024-09-10T18:32:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2378000000000001e-05 2024-09-10T18:32:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.225452+0000", 2024-09-10T18:32:59.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8818000000000002e-05 2024-09-10T18:32:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:32:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.225461+0000", 2024-09-10T18:32:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9879999999999993e-06 2024-09-10T18:32:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.135296+0000", 2024-09-10T18:32:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.90983507799999996 2024-09-10T18:32:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.135380+0000", 2024-09-10T18:32:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3567999999999997e-05 2024-09-10T18:32:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:32:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.135448+0000", 2024-09-10T18:32:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8515000000000004e-05 2024-09-10T18:32:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138630+0000", 2024-09-10T18:32:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031812450000000001 2024-09-10T18:32:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138667+0000", 2024-09-10T18:32:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7876000000000001e-05 2024-09-10T18:32:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141456+0000", 2024-09-10T18:32:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002788727 2024-09-10T18:32:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141505+0000", 2024-09-10T18:32:59.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8600999999999998e-05 2024-09-10T18:32:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141543+0000", 2024-09-10T18:32:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8568000000000001e-05 2024-09-10T18:32:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221541 7.b 7:dde66f49:::benchmark_data_smithi028_77856_object100441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:32:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.229960+0000", 2024-09-10T18:32:59.332 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 351.04233647900003, 2024-09-10T18:32:59.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.91172626199999995, 2024-09-10T18:32:59.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.333 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.333 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.333 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.333 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.333 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221541 2024-09-10T18:32:59.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.333 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.333 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.333 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.229960+0000", 2024-09-10T18:32:59.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.229960+0000", 2024-09-10T18:32:59.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.229957+0000", 2024-09-10T18:32:59.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:32:59.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.229966+0000", 2024-09-10T18:32:59.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9160000000000007e-06 2024-09-10T18:32:59.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.336 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.229981+0000", 2024-09-10T18:32:59.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-10T18:32:59.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.336 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.229988+0000", 2024-09-10T18:32:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7669999999999998e-06 2024-09-10T18:32:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.230038+0000", 2024-09-10T18:32:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0146999999999999e-05 2024-09-10T18:32:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:32:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.230046+0000", 2024-09-10T18:32:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3909999999999998e-06 2024-09-10T18:32:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136710+0000", 2024-09-10T18:32:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.90666396400000004 2024-09-10T18:32:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136826+0000", 2024-09-10T18:32:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011557000000000001 2024-09-10T18:32:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:32:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.136962+0000", 2024-09-10T18:32:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000135928 2024-09-10T18:32:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138844+0000", 2024-09-10T18:32:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001882365 2024-09-10T18:32:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138849+0000", 2024-09-10T18:32:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.322e-06 2024-09-10T18:32:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141592+0000", 2024-09-10T18:32:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002742155 2024-09-10T18:32:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141648+0000", 2024-09-10T18:32:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6178000000000003e-05 2024-09-10T18:32:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141686+0000", 2024-09-10T18:32:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8398000000000001e-05 2024-09-10T18:32:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221553 7.b 7:d1279096:::benchmark_data_smithi028_77856_object100453:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:32:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.240485+0000", 2024-09-10T18:32:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 351.031811124, 2024-09-10T18:32:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.90149367300000005, 2024-09-10T18:32:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221553 2024-09-10T18:32:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.240485+0000", 2024-09-10T18:32:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.240485+0000", 2024-09-10T18:32:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.240483+0000", 2024-09-10T18:32:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:32:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.240490+0000", 2024-09-10T18:32:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8529999999999992e-06 2024-09-10T18:32:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.240506+0000", 2024-09-10T18:32:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5596000000000001e-05 2024-09-10T18:32:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.240512+0000", 2024-09-10T18:32:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8329999999999999e-06 2024-09-10T18:32:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.240553+0000", 2024-09-10T18:32:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0689999999999998e-05 2024-09-10T18:32:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:32:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.240560+0000", 2024-09-10T18:32:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7859999999999997e-06 2024-09-10T18:32:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138881+0000", 2024-09-10T18:32:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.89832107500000002 2024-09-10T18:32:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.138982+0000", 2024-09-10T18:32:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000100426 2024-09-10T18:32:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:32:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.139085+0000", 2024-09-10T18:32:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010320800000000001 2024-09-10T18:32:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141736+0000", 2024-09-10T18:32:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026505029999999998 2024-09-10T18:32:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141781+0000", 2024-09-10T18:32:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5590000000000002e-05 2024-09-10T18:32:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141790+0000", 2024-09-10T18:32:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1940000000000004e-06 2024-09-10T18:32:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141941+0000", 2024-09-10T18:32:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015065799999999999 2024-09-10T18:32:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.141979+0000", 2024-09-10T18:32:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7880000000000003e-05 2024-09-10T18:32:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221562 7.b 7:de05e396:::benchmark_data_smithi028_77856_object100462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:32:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.247169+0000", 2024-09-10T18:32:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 351.02512712399999, 2024-09-10T18:32:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.901609196, 2024-09-10T18:32:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221562 2024-09-10T18:32:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.247169+0000", 2024-09-10T18:32:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.247169+0000", 2024-09-10T18:32:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.247167+0000", 2024-09-10T18:32:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:32:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.247175+0000", 2024-09-10T18:32:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9789999999999993e-06 2024-09-10T18:32:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.247189+0000", 2024-09-10T18:32:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3682999999999999e-05 2024-09-10T18:32:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.247195+0000", 2024-09-10T18:32:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2870000000000001e-06 2024-09-10T18:32:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.247238+0000", 2024-09-10T18:32:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2876000000000001e-05 2024-09-10T18:32:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:32:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.247270+0000", 2024-09-10T18:32:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1900999999999998e-05 2024-09-10T18:32:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.142018+0000", 2024-09-10T18:32:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.89474827999999995 2024-09-10T18:32:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.142119+0000", 2024-09-10T18:32:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101619 2024-09-10T18:32:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:32:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.142231+0000", 2024-09-10T18:32:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011147899999999999 2024-09-10T18:32:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148529+0000", 2024-09-10T18:32:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006298194 2024-09-10T18:32:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148565+0000", 2024-09-10T18:32:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6035999999999999e-05 2024-09-10T18:32:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148670+0000", 2024-09-10T18:32:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000104812 2024-09-10T18:32:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148722+0000", 2024-09-10T18:32:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1947999999999998e-05 2024-09-10T18:32:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148778+0000", 2024-09-10T18:32:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6458000000000001e-05 2024-09-10T18:32:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221645 7.b 7:df84e65d:::benchmark_data_smithi028_77856_object100545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:32:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.365336+0000", 2024-09-10T18:32:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 350.90696012299998, 2024-09-10T18:32:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.794312303, 2024-09-10T18:32:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221645 2024-09-10T18:32:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.365336+0000", 2024-09-10T18:32:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.365336+0000", 2024-09-10T18:32:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.365334+0000", 2024-09-10T18:32:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-10T18:32:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.365342+0000", 2024-09-10T18:32:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1869999999999997e-06 2024-09-10T18:32:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.365360+0000", 2024-09-10T18:32:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8111999999999999e-05 2024-09-10T18:32:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.365367+0000", 2024-09-10T18:32:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9929999999999998e-06 2024-09-10T18:32:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.365420+0000", 2024-09-10T18:32:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3321e-05 2024-09-10T18:32:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:32:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.365432+0000", 2024-09-10T18:32:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1884e-05 2024-09-10T18:32:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.145920+0000", 2024-09-10T18:32:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.78048761700000002 2024-09-10T18:32:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.146028+0000", 2024-09-10T18:32:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108578 2024-09-10T18:32:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:32:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.146140+0000", 2024-09-10T18:32:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011149 2024-09-10T18:32:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148592+0000", 2024-09-10T18:32:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002452475 2024-09-10T18:32:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148658+0000", 2024-09-10T18:32:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6162999999999997e-05 2024-09-10T18:32:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159494+0000", 2024-09-10T18:32:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010835463 2024-09-10T18:32:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159601+0000", 2024-09-10T18:32:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000106933 2024-09-10T18:32:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159649+0000", 2024-09-10T18:32:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7732999999999999e-05 2024-09-10T18:32:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221656 7.b 7:ddf1fcdc:::benchmark_data_smithi028_77856_object100556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:32:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.377170+0000", 2024-09-10T18:32:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 350.89512639399999, 2024-09-10T18:32:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.78274461299999998, 2024-09-10T18:32:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221656 2024-09-10T18:32:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.377170+0000", 2024-09-10T18:32:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.377170+0000", 2024-09-10T18:32:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.377168+0000", 2024-09-10T18:32:59.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:59.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:59.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.377176+0000", 2024-09-10T18:32:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7470000000000005e-06 2024-09-10T18:32:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.377192+0000", 2024-09-10T18:32:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6311e-05 2024-09-10T18:32:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.377198+0000", 2024-09-10T18:32:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4450000000000001e-06 2024-09-10T18:32:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.377245+0000", 2024-09-10T18:32:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6671000000000001e-05 2024-09-10T18:32:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:32:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.377253+0000", 2024-09-10T18:32:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0220000000000004e-06 2024-09-10T18:32:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148820+0000", 2024-09-10T18:32:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.771566682 2024-09-10T18:32:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.148935+0000", 2024-09-10T18:32:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011509 2024-09-10T18:32:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:32:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.149007+0000", 2024-09-10T18:32:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2096999999999998e-05 2024-09-10T18:32:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.154054+0000", 2024-09-10T18:32:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050468689999999998 2024-09-10T18:32:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.154087+0000", 2024-09-10T18:32:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3636000000000001e-05 2024-09-10T18:32:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159824+0000", 2024-09-10T18:32:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0057362389999999997 2024-09-10T18:32:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159865+0000", 2024-09-10T18:32:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0917999999999997e-05 2024-09-10T18:32:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159915+0000", 2024-09-10T18:32:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9990000000000001e-05 2024-09-10T18:32:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221677 7.b 7:d8b7af7a:::benchmark_data_smithi028_77856_object100577:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:32:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.399460+0000", 2024-09-10T18:32:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 350.87283585, 2024-09-10T18:32:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.76054054100000001, 2024-09-10T18:32:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221677 2024-09-10T18:32:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.399460+0000", 2024-09-10T18:32:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.399460+0000", 2024-09-10T18:32:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.399458+0000", 2024-09-10T18:32:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:32:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.399466+0000", 2024-09-10T18:32:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2910000000000007e-06 2024-09-10T18:32:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.399485+0000", 2024-09-10T18:32:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8600000000000001e-05 2024-09-10T18:32:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.399493+0000", 2024-09-10T18:32:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7389999999999999e-06 2024-09-10T18:32:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.399543+0000", 2024-09-10T18:32:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0809000000000001e-05 2024-09-10T18:32:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:32:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.399553+0000", 2024-09-10T18:32:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2609999999999999e-06 2024-09-10T18:32:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.151399+0000", 2024-09-10T18:32:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.75184643399999995 2024-09-10T18:32:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.151518+0000", 2024-09-10T18:32:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000118336 2024-09-10T18:32:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:32:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.151624+0000", 2024-09-10T18:32:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000106786 2024-09-10T18:32:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159486+0000", 2024-09-10T18:32:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078619840000000007 2024-09-10T18:32:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159751+0000", 2024-09-10T18:32:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026467899999999998 2024-09-10T18:32:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159923+0000", 2024-09-10T18:32:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000172423 2024-09-10T18:32:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159966+0000", 2024-09-10T18:32:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2459999999999997e-05 2024-09-10T18:32:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.160001+0000", 2024-09-10T18:32:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5179999999999999e-05 2024-09-10T18:32:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221686 7.b 7:dfcedead:::benchmark_data_smithi028_77856_object100586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:32:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.407556+0000", 2024-09-10T18:32:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 350.86474056200001, 2024-09-10T18:32:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.75659945500000003, 2024-09-10T18:32:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221686 2024-09-10T18:32:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.407556+0000", 2024-09-10T18:32:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.407556+0000", 2024-09-10T18:32:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:59.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:32:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.407553+0000", 2024-09-10T18:32:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-10T18:32:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.407563+0000", 2024-09-10T18:32:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7589999999999992e-06 2024-09-10T18:32:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.407583+0000", 2024-09-10T18:32:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0536000000000001e-05 2024-09-10T18:32:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.407591+0000", 2024-09-10T18:32:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6149999999999999e-06 2024-09-10T18:32:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.407638+0000", 2024-09-10T18:32:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7107000000000001e-05 2024-09-10T18:32:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:32:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.407646+0000", 2024-09-10T18:32:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3059999999999998e-06 2024-09-10T18:32:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.154110+0000", 2024-09-10T18:32:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.74646322600000004 2024-09-10T18:32:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.154214+0000", 2024-09-10T18:32:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000104417 2024-09-10T18:32:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:32:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.154330+0000", 2024-09-10T18:32:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116496 2024-09-10T18:32:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.159813+0000", 2024-09-10T18:32:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054826550000000003 2024-09-10T18:32:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.163716+0000", 2024-09-10T18:32:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039024319999999999 2024-09-10T18:32:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.163847+0000", 2024-09-10T18:32:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013136499999999999 2024-09-10T18:32:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.164128+0000", 2024-09-10T18:32:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028095600000000001 2024-09-10T18:32:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.164155+0000", 2024-09-10T18:32:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7330000000000001e-05 2024-09-10T18:32:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221693 7.b 7:dbf8bbe4:::benchmark_data_smithi028_77856_object100593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:32:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.414908+0000", 2024-09-10T18:32:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 350.857388738, 2024-09-10T18:32:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.74933771900000001, 2024-09-10T18:32:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221693 2024-09-10T18:32:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.414908+0000", 2024-09-10T18:32:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.414908+0000", 2024-09-10T18:32:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.414906+0000", 2024-09-10T18:32:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.414912+0000", 2024-09-10T18:32:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7309999999999996e-06 2024-09-10T18:32:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.414924+0000", 2024-09-10T18:32:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1374e-05 2024-09-10T18:32:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.414930+0000", 2024-09-10T18:32:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3890000000000002e-06 2024-09-10T18:32:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.414977+0000", 2024-09-10T18:32:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7419000000000001e-05 2024-09-10T18:32:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:32:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.414988+0000", 2024-09-10T18:32:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.029e-05 2024-09-10T18:32:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.156717+0000", 2024-09-10T18:32:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.74172964100000005 2024-09-10T18:32:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.156925+0000", 2024-09-10T18:32:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020772799999999999 2024-09-10T18:32:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:32:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.157029+0000", 2024-09-10T18:32:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103702 2024-09-10T18:32:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.163753+0000", 2024-09-10T18:32:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0067238059999999997 2024-09-10T18:32:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.163782+0000", 2024-09-10T18:32:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9431e-05 2024-09-10T18:32:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.164163+0000", 2024-09-10T18:32:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00038137700000000002 2024-09-10T18:32:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.164190+0000", 2024-09-10T18:32:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6729000000000001e-05 2024-09-10T18:32:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.164245+0000", 2024-09-10T18:32:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5163e-05 2024-09-10T18:32:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221707 7.b 7:da97deb7:::benchmark_data_smithi028_77856_object100607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:32:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.432975+0000", 2024-09-10T18:32:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 350.83932173699998, 2024-09-10T18:32:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.73418157100000003, 2024-09-10T18:32:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221707 2024-09-10T18:32:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.432975+0000", 2024-09-10T18:32:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.432975+0000", 2024-09-10T18:32:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.432972+0000", 2024-09-10T18:32:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.432981+0000", 2024-09-10T18:32:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3639999999999999e-06 2024-09-10T18:32:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.432995+0000", 2024-09-10T18:32:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4268e-05 2024-09-10T18:32:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.433002+0000", 2024-09-10T18:32:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4470000000000003e-06 2024-09-10T18:32:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.433050+0000", 2024-09-10T18:32:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8217000000000003e-05 2024-09-10T18:32:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:32:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.433060+0000", 2024-09-10T18:32:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0121e-05 2024-09-10T18:32:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.160035+0000", 2024-09-10T18:32:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.72697528600000005 2024-09-10T18:32:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.160158+0000", 2024-09-10T18:32:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012259700000000001 2024-09-10T18:32:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:32:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.160275+0000", 2024-09-10T18:32:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011747999999999999 2024-09-10T18:32:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.163809+0000", 2024-09-10T18:32:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035336759999999999 2024-09-10T18:32:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.163840+0000", 2024-09-10T18:32:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1059000000000002e-05 2024-09-10T18:32:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.166920+0000", 2024-09-10T18:32:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030802260000000001 2024-09-10T18:32:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.167081+0000", 2024-09-10T18:32:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016071 2024-09-10T18:32:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.167156+0000", 2024-09-10T18:32:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5258000000000003e-05 2024-09-10T18:32:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221709 7.b 7:d48ef2db:::benchmark_data_smithi028_77856_object100609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:32:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.434394+0000", 2024-09-10T18:32:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 350.83790235399999, 2024-09-10T18:32:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.73600392699999995, 2024-09-10T18:32:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221709 2024-09-10T18:32:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.434394+0000", 2024-09-10T18:32:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.434394+0000", 2024-09-10T18:32:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.434392+0000", 2024-09-10T18:32:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:32:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.434399+0000", 2024-09-10T18:32:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.314e-06 2024-09-10T18:32:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.434411+0000", 2024-09-10T18:32:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2057e-05 2024-09-10T18:32:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.434418+0000", 2024-09-10T18:32:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6379999999999998e-06 2024-09-10T18:32:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.434460+0000", 2024-09-10T18:32:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2648000000000002e-05 2024-09-10T18:32:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:32:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.434468+0000", 2024-09-10T18:32:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9419999999999995e-06 2024-09-10T18:32:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.164279+0000", 2024-09-10T18:32:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.72981096099999998 2024-09-10T18:32:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.164389+0000", 2024-09-10T18:32:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109459 2024-09-10T18:32:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:32:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.164481+0000", 2024-09-10T18:32:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2584999999999995e-05 2024-09-10T18:32:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.167166+0000", 2024-09-10T18:32:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026848509999999998 2024-09-10T18:32:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.169979+0000", 2024-09-10T18:32:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028130910000000002 2024-09-10T18:32:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.170269+0000", 2024-09-10T18:32:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029024600000000003 2024-09-10T18:32:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.170361+0000", 2024-09-10T18:32:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1576000000000005e-05 2024-09-10T18:32:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.170398+0000", 2024-09-10T18:32:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7029000000000001e-05 2024-09-10T18:32:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:221712 7.b 7:d52c250d:::benchmark_data_smithi028_77856_object100612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-10T18:32:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:08.444124+0000", 2024-09-10T18:32:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 350.82817194400002, 2024-09-10T18:32:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.72799156099999995, 2024-09-10T18:32:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:32:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:32:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:32:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 221712 2024-09-10T18:32:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.444124+0000", 2024-09-10T18:32:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.444124+0000", 2024-09-10T18:32:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.444123+0000", 2024-09-10T18:32:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:32:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.444129+0000", 2024-09-10T18:32:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2319999999999998e-06 2024-09-10T18:32:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.444139+0000", 2024-09-10T18:32:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0727000000000001e-05 2024-09-10T18:32:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.444145+0000", 2024-09-10T18:32:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3909999999999997e-06 2024-09-10T18:32:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.444183+0000", 2024-09-10T18:32:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7982999999999999e-05 2024-09-10T18:32:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-09-10T18:32:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:08.444189+0000", 2024-09-10T18:32:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0669999999999997e-06 2024-09-10T18:32:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.167274+0000", 2024-09-10T18:32:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.72308553099999995 2024-09-10T18:32:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.167374+0000", 2024-09-10T18:32:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0001 2024-09-10T18:32:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:32:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.167476+0000", 2024-09-10T18:32:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101915 2024-09-10T18:32:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.170438+0000", 2024-09-10T18:32:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029619999999999998 2024-09-10T18:32:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:32:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.170466+0000", 2024-09-10T18:32:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7421000000000001e-05 2024-09-10T18:32:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:32:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.171990+0000", 2024-09-10T18:32:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001524262 2024-09-10T18:32:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.172078+0000", 2024-09-10T18:32:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8418999999999995e-05 2024-09-10T18:32:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:09.172116+0000", 2024-09-10T18:32:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7475999999999998e-05 2024-09-10T18:32:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15117.0:145951 8.b e214/180 8:d5a4ad28:::benchmark_data_smithi028_109242_object11730:head v 214'9017, mlcod=214'9014)", 2024-09-10T18:32:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:29:46.822624+0000", 2024-09-10T18:32:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.44967224800001, 2024-09-10T18:32:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.119049102, 2024-09-10T18:32:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:32:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:32:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:32:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:32:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.822624+0000", 2024-09-10T18:32:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:32:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.822624+0000", 2024-09-10T18:32:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:32:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:32:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.822623+0000", 2024-09-10T18:32:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:32:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:32:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.822631+0000", 2024-09-10T18:32:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0609999999999996e-06 2024-09-10T18:32:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:32:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.822680+0000", 2024-09-10T18:32:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9212000000000002e-05 2024-09-10T18:32:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:32:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.822687+0000", 2024-09-10T18:32:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.883e-06 2024-09-10T18:32:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:32:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.822779+0000", 2024-09-10T18:32:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1821000000000001e-05 2024-09-10T18:32:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:32:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.822799+0000", 2024-09-10T18:32:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0562000000000001e-05 2024-09-10T18:32:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:32:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.941642+0000", 2024-09-10T18:32:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.11884307400000001 2024-09-10T18:32:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:32:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:32:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:32:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:29:46.941673+0000", 2024-09-10T18:32:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0760999999999999e-05 2024-09-10T18:32:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:32:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:32:59.417 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:32:59.424 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[39220]: 3.17 scrub starts 2024-09-10T18:32:59.424 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[39220]: 3.17 scrub ok 2024-09-10T18:32:59.424 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[39220]: 6.1c scrub starts 2024-09-10T18:32:59.424 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[39220]: 6.1c scrub ok 2024-09-10T18:32:59.424 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2209976256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:32:59.424 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[39220]: from='client.25360 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:32:59.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[39220]: 5.1d scrub starts 2024-09-10T18:32:59.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[39220]: 5.1d scrub ok 2024-09-10T18:32:59.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[39220]: 6.b scrub starts 2024-09-10T18:32:59.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[39220]: 6.b scrub ok 2024-09-10T18:32:59.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[29291]: 3.17 scrub starts 2024-09-10T18:32:59.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[29291]: 3.17 scrub ok 2024-09-10T18:32:59.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[29291]: 6.1c scrub starts 2024-09-10T18:32:59.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[29291]: 6.1c scrub ok 2024-09-10T18:32:59.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2209976256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:32:59.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[29291]: from='client.25360 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:32:59.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[29291]: 5.1d scrub starts 2024-09-10T18:32:59.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[29291]: 5.1d scrub ok 2024-09-10T18:32:59.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[29291]: 6.b scrub starts 2024-09-10T18:32:59.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:32:59 smithi028 ceph-mon[29291]: 6.b scrub ok 2024-09-10T18:32:59.479 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:32:59.513 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:59.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:59.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:32:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:59 smithi204 ceph-mon[32357]: 3.17 scrub starts 2024-09-10T18:32:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:59 smithi204 ceph-mon[32357]: 3.17 scrub ok 2024-09-10T18:32:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:59 smithi204 ceph-mon[32357]: 6.1c scrub starts 2024-09-10T18:32:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:59 smithi204 ceph-mon[32357]: 6.1c scrub ok 2024-09-10T18:32:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:59 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2209976256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:32:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:59 smithi204 ceph-mon[32357]: from='client.25360 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:32:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:59 smithi204 ceph-mon[32357]: 5.1d scrub starts 2024-09-10T18:32:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:59 smithi204 ceph-mon[32357]: 5.1d scrub ok 2024-09-10T18:32:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:59 smithi204 ceph-mon[32357]: 6.b scrub starts 2024-09-10T18:32:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:32:59 smithi204 ceph-mon[32357]: 6.b scrub ok 2024-09-10T18:32:59.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:32:59.652 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:59.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:59.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:32:59.754 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:32:59.788 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:32:59.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:59.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:32:59.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:32:59.934 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:32:59.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:32:59.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:00.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:00.066 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 47 16 84630 84614 112.439 80.4375 0.00768645 0.00888867 2024-09-10T18:33:00.072 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:00.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:00.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:00.218 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:00.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:00.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:00.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:00.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:00.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:00.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:00.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:00.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:00.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:00.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:00.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[29291]: 1.5 scrub starts 2024-09-10T18:33:00.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[29291]: 1.5 scrub ok 2024-09-10T18:33:00.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[29291]: 1.0 scrub starts 2024-09-10T18:33:00.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[29291]: 1.0 scrub ok 2024-09-10T18:33:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[29291]: pgmap v514: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 511 B/s rd, 54 MiB/s wr, 862 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:33:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:33:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:33:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[29291]: from='client.25360 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:33:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[29291]: osdmap e262: 8 total, 8 up, 8 in 2024-09-10T18:33:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[29291]: 4.0 scrub starts 2024-09-10T18:33:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[29291]: 4.0 scrub ok 2024-09-10T18:33:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[39220]: 1.5 scrub starts 2024-09-10T18:33:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[39220]: 1.5 scrub ok 2024-09-10T18:33:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[39220]: 1.0 scrub starts 2024-09-10T18:33:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[39220]: 1.0 scrub ok 2024-09-10T18:33:00.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[39220]: pgmap v514: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 511 B/s rd, 54 MiB/s wr, 862 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:33:00.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:33:00.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:33:00.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[39220]: from='client.25360 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:33:00.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[39220]: osdmap e262: 8 total, 8 up, 8 in 2024-09-10T18:33:00.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[39220]: 4.0 scrub starts 2024-09-10T18:33:00.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:00 smithi028 ceph-mon[39220]: 4.0 scrub ok 2024-09-10T18:33:00.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:00 smithi204 ceph-mon[32357]: 1.5 scrub starts 2024-09-10T18:33:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:00 smithi204 ceph-mon[32357]: 1.5 scrub ok 2024-09-10T18:33:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:00 smithi204 ceph-mon[32357]: 1.0 scrub starts 2024-09-10T18:33:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:00 smithi204 ceph-mon[32357]: 1.0 scrub ok 2024-09-10T18:33:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:00 smithi204 ceph-mon[32357]: pgmap v514: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 511 B/s rd, 54 MiB/s wr, 862 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:33:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:00 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:33:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:00 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:33:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:00 smithi204 ceph-mon[32357]: from='client.25360 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:33:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:00 smithi204 ceph-mon[32357]: osdmap e262: 8 total, 8 up, 8 in 2024-09-10T18:33:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:00 smithi204 ceph-mon[32357]: 4.0 scrub starts 2024-09-10T18:33:00.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:00 smithi204 ceph-mon[32357]: 4.0 scrub ok 2024-09-10T18:33:00.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:00.631 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:00.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:00.732 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:00.762 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:00.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:00.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:00.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:00.907 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:00.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:00.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:01.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:01.041 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:01.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:01.066 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 48 16 85995 85979 111.874 85.3125 0.00511471 0.00893371 2024-09-10T18:33:01.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:01.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:01.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:01.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:01.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:01 smithi028 ceph-mon[29291]: 3.1 scrub starts 2024-09-10T18:33:01.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:01 smithi028 ceph-mon[29291]: 3.1 scrub ok 2024-09-10T18:33:01.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:01 smithi028 ceph-mon[29291]: 3.1a scrub starts 2024-09-10T18:33:01.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:01 smithi028 ceph-mon[29291]: 3.1a scrub ok 2024-09-10T18:33:01.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:01 smithi028 ceph-mon[29291]: 5.11 scrub starts 2024-09-10T18:33:01.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:01 smithi028 ceph-mon[29291]: 5.11 scrub ok 2024-09-10T18:33:01.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:01 smithi028 ceph-mon[29291]: osdmap e263: 8 total, 8 up, 8 in 2024-09-10T18:33:01.259 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:01 smithi028 ceph-mon[39220]: 3.1 scrub starts 2024-09-10T18:33:01.259 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:01 smithi028 ceph-mon[39220]: 3.1 scrub ok 2024-09-10T18:33:01.259 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:01 smithi028 ceph-mon[39220]: 3.1a scrub starts 2024-09-10T18:33:01.259 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:01 smithi028 ceph-mon[39220]: 3.1a scrub ok 2024-09-10T18:33:01.259 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:01 smithi028 ceph-mon[39220]: 5.11 scrub starts 2024-09-10T18:33:01.259 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:01 smithi028 ceph-mon[39220]: 5.11 scrub ok 2024-09-10T18:33:01.259 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:01 smithi028 ceph-mon[39220]: osdmap e263: 8 total, 8 up, 8 in 2024-09-10T18:33:01.279 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:01.315 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:01.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:01.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:01.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:01.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:01.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:01.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:01.595 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:01.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:01 smithi204 ceph-mon[32357]: 3.1 scrub starts 2024-09-10T18:33:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:01 smithi204 ceph-mon[32357]: 3.1 scrub ok 2024-09-10T18:33:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:01 smithi204 ceph-mon[32357]: 3.1a scrub starts 2024-09-10T18:33:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:01 smithi204 ceph-mon[32357]: 3.1a scrub ok 2024-09-10T18:33:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:01 smithi204 ceph-mon[32357]: 5.11 scrub starts 2024-09-10T18:33:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:01 smithi204 ceph-mon[32357]: 5.11 scrub ok 2024-09-10T18:33:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:01 smithi204 ceph-mon[32357]: osdmap e263: 8 total, 8 up, 8 in 2024-09-10T18:33:01.696 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:01.731 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:01.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:01.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:01.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:33:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:33:01] "GET /metrics HTTP/1.1" 200 224445 "" "Prometheus/2.43.0" 2024-09-10T18:33:01.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:01.981 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:01.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:01.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:02.066 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 49 16 87359 87343 111.331 85.25 0.00901114 0.00897764 2024-09-10T18:33:02.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:02.117 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:02.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:02.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:02.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:02.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:02.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:02.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:02.402 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:02.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[39220]: 6.18 scrub starts 2024-09-10T18:33:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[39220]: 6.18 scrub ok 2024-09-10T18:33:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[39220]: pgmap v517: 158 pgs: 158 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-10T18:33:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[39220]: 1.7 scrub starts 2024-09-10T18:33:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[39220]: 1.7 scrub ok 2024-09-10T18:33:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[39220]: 1.4 scrub starts 2024-09-10T18:33:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[39220]: 1.4 scrub ok 2024-09-10T18:33:02.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[39220]: 3.15 scrub starts 2024-09-10T18:33:02.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[39220]: 3.15 scrub ok 2024-09-10T18:33:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[29291]: 6.18 scrub starts 2024-09-10T18:33:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[29291]: 6.18 scrub ok 2024-09-10T18:33:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[29291]: pgmap v517: 158 pgs: 158 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-10T18:33:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[29291]: 1.7 scrub starts 2024-09-10T18:33:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[29291]: 1.7 scrub ok 2024-09-10T18:33:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[29291]: 1.4 scrub starts 2024-09-10T18:33:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[29291]: 1.4 scrub ok 2024-09-10T18:33:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[29291]: 3.15 scrub starts 2024-09-10T18:33:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:02 smithi028 ceph-mon[29291]: 3.15 scrub ok 2024-09-10T18:33:02.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:02.539 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:02.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:02.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:02.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:33:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:33:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:33:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:02 smithi204 ceph-mon[32357]: 6.18 scrub starts 2024-09-10T18:33:02.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:02 smithi204 ceph-mon[32357]: 6.18 scrub ok 2024-09-10T18:33:02.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:02 smithi204 ceph-mon[32357]: pgmap v517: 158 pgs: 158 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-10T18:33:02.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:02 smithi204 ceph-mon[32357]: 1.7 scrub starts 2024-09-10T18:33:02.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:02 smithi204 ceph-mon[32357]: 1.7 scrub ok 2024-09-10T18:33:02.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:02 smithi204 ceph-mon[32357]: 1.4 scrub starts 2024-09-10T18:33:02.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:02 smithi204 ceph-mon[32357]: 1.4 scrub ok 2024-09-10T18:33:02.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:02 smithi204 ceph-mon[32357]: 3.15 scrub starts 2024-09-10T18:33:02.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:02 smithi204 ceph-mon[32357]: 3.15 scrub ok 2024-09-10T18:33:02.642 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:02.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:02.676 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:02.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:02.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:33:02.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:02.836 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:02.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:02.937 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:02.954 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-10T18:33:02.965 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:02.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:02.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:03.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:03.067 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 50 16 88654 88638 110.721 80.9375 0.00632478 0.0090269 2024-09-10T18:33:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:03.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:03.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:03.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:03.287 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:03.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:03.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:03.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:03 smithi028 ceph-mon[29291]: 5.3 deep-scrub starts 2024-09-10T18:33:03.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:03 smithi028 ceph-mon[29291]: 5.3 deep-scrub ok 2024-09-10T18:33:03.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:03 smithi028 ceph-mon[39220]: 5.3 deep-scrub starts 2024-09-10T18:33:03.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:03 smithi028 ceph-mon[39220]: 5.3 deep-scrub ok 2024-09-10T18:33:03.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:03.421 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:03.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:03.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:03.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:03.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:03.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:03 smithi204 ceph-mon[32357]: 5.3 deep-scrub starts 2024-09-10T18:33:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:03 smithi204 ceph-mon[32357]: 5.3 deep-scrub ok 2024-09-10T18:33:03.646 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:03.677 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:03.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:03.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:03.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:03.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:03.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:03.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:03.918 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:03.947 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:03.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:03.969 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:33:03.971 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:33:03.971 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 263, 2024-09-10T18:33:03.972 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "5eababb2-6fa0-11ef-bcea-c7b262605968", 2024-09-10T18:33:03.972 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-10T18:16:20.680373+0000", 2024-09-10T18:33:03.972 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-10T18:33:00.753300+0000", 2024-09-10T18:33:03.972 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-09-10T18:26:26.611338+0000", 2024-09-10T18:33:03.972 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-09-10T18:21:02.114621+0000", 2024-09-10T18:33:03.972 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T18:33:03.972 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5799936, 2024-09-10T18:33:03.972 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-09-10T18:33:03.972 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-09-10T18:33:03.972 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-09-10T18:33:03.972 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-09-10T18:33:03.972 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-09-10T18:33:03.972 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:33:03.973 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 19, 2024-09-10T18:33:03.973 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T18:33:03.973 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T18:33:03.973 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T18:33:03.973 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-09-10T18:33:03.973 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 9, 2024-09-10T18:33:03.973 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-09-10T18:33:03.973 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-09-10T18:33:03.973 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-09-10T18:33:03.973 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "quincy", 2024-09-10T18:33:03.973 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-09-10T18:33:03.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:03.974 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-09-10T18:33:03.974 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-09-10T18:33:03.974 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:19:24.013536+0000", 2024-09-10T18:33:03.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:33:03.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:33:03.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:33:03.974 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:33:03.974 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:33:03.974 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:33:03.974 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:33:03.974 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:33:03.974 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:33:03.974 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:33:03.974 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:33:03.974 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:33:03.975 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 11, 2024-09-10T18:33:03.975 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 11, 2024-09-10T18:33:03.975 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 11, 2024-09-10T18:33:03.975 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 11, 2024-09-10T18:33:03.975 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 11, 2024-09-10T18:33:03.975 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:33:03.975 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:33:03.975 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:33:03.975 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:33:03.975 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:33:03.975 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:33:03.975 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:33:03.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:03.975 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "138", 2024-09-10T18:33:03.975 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:33:03.976 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:33:03.976 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "136", 2024-09-10T18:33:03.976 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:33:03.976 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:33:03.976 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:33:03.976 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:33:03.976 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:33:03.976 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:33:03.976 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:33:03.976 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:33:03.976 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:33:03.976 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:33:03.976 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:33:03.976 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:33:03.976 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:33:03.977 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:33:03.977 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:33:03.977 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:33:03.977 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:33:03.977 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:33:03.977 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:33:03.977 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:33:03.977 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:33:03.977 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:33:03.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:33:03.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:03.977 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:33:03.977 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:33:03.977 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:33:03.977 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:33:03.978 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:33:03.978 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:33:03.978 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:33:03.978 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:33:03.978 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:33:03.978 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:33:03.978 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:33:03.978 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-10T18:33:03.978 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-09-10T18:33:03.978 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-09-10T18:33:03.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:03.978 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:33:03.978 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-09-10T18:33:03.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:03.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:03.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:03.979 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-09-10T18:33:03.979 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-09-10T18:33:03.979 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:30.065209+0000", 2024-09-10T18:33:03.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-09-10T18:33:03.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T18:33:03.979 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:33:03.979 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:33:03.979 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:33:03.979 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:33:03.979 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:33:03.979 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:33:03.979 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:33:03.979 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:33:03.979 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:33:03.980 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:33:03.980 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-09-10T18:33:03.980 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-09-10T18:33:03.980 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-09-10T18:33:03.980 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-09-10T18:33:03.980 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-09-10T18:33:03.980 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:33:03.980 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:33:03.980 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:33:03.980 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:33:03.980 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:33:03.980 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:33:03.980 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:33:03.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:03.980 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "62", 2024-09-10T18:33:03.981 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:33:03.981 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:33:03.981 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:33:03.981 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:33:03.981 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:33:03.981 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 3, 2024-09-10T18:33:03.981 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 62, 2024-09-10T18:33:03.981 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:33:03.981 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:33:03.981 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:33:03.981 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:33:03.981 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:33:03.981 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:33:03.982 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:33:03.982 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:33:03.982 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:33:03.982 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:33:03.982 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:33:03.982 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:33:03.982 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:33:03.982 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:33:03.982 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:33:03.982 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:33:03.983 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:33:03.983 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:33:03.983 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:33:03.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:03.983 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:33:03.983 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:33:03.983 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:33:03.983 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:33:03.983 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:33:03.983 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:33:03.983 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:33:03.983 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:33:03.983 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:33:03.984 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:33:03.984 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:33:03.984 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:33:03.984 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:33:03.984 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-09-10T18:33:03.984 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:03.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:03.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:03.984 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-09-10T18:33:03.984 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-09-10T18:33:03.984 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:30.506441+0000", 2024-09-10T18:33:03.984 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:33:03.984 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:33:03.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:33:03.985 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:33:03.985 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:33:03.985 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:33:03.985 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:33:03.985 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:33:03.985 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:33:03.985 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:33:03.985 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:33:03.985 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:33:03.985 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:33:03.985 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:33:03.985 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:33:03.986 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:33:03.986 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:33:03.986 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:33:03.986 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:33:03.986 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:33:03.986 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:33:03.986 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:33:03.986 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:33:03.986 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:33:03.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:03.986 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "59", 2024-09-10T18:33:03.986 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:33:03.986 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:33:03.986 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:33:03.987 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:33:03.987 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:33:03.987 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:33:03.987 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:33:03.987 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:33:03.987 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:33:03.987 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:33:03.987 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:33:03.987 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:33:03.987 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:33:03.987 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:33:03.987 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:33:03.987 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:33:03.988 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:33:03.988 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:33:03.988 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:33:03.988 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:33:03.988 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:33:03.988 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:33:03.988 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:33:03.988 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:33:03.988 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:33:03.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:33:03.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:03.988 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:33:03.988 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:33:03.989 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:33:03.989 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:33:03.989 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:33:03.989 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:33:03.989 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:33:03.989 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:33:03.989 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:33:03.989 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:33:03.989 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:33:03.989 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:33:03.989 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:33:03.989 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:33:03.990 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:03.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:03.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:03.990 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-09-10T18:33:03.990 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-09-10T18:33:03.990 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:32.008130+0000", 2024-09-10T18:33:03.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:33:03.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:33:03.990 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:33:03.990 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:33:03.990 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:33:03.990 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:33:03.990 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:33:03.991 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:33:03.991 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:33:03.991 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:33:03.991 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:33:03.991 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:33:03.991 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:33:03.991 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:33:03.991 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:33:03.991 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:33:03.991 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:33:03.991 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:33:03.991 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:33:03.991 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:33:03.991 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:33:03.992 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:33:03.992 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:33:03.992 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:33:03.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:03.992 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "61", 2024-09-10T18:33:03.992 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:33:03.992 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:33:03.992 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:33:03.992 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:33:03.992 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:33:03.992 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:33:03.992 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:33:03.993 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:33:03.993 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:33:03.993 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:33:03.993 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:33:03.993 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:33:03.993 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:33:03.993 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:33:03.993 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:33:03.993 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:33:03.993 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:33:03.993 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:33:03.993 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:33:03.993 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:33:03.994 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:33:03.994 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:33:03.994 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:33:03.994 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:33:03.994 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:33:03.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:33:03.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:03.994 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:33:03.994 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:33:03.994 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:33:03.994 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:33:03.994 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:33:03.994 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:33:03.994 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:33:03.995 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:33:03.995 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:33:03.995 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:33:03.995 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:33:03.995 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:33:03.996 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:33:03.996 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:33:03.996 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:03.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:03.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:03.996 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-09-10T18:33:03.996 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-09-10T18:33:03.997 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:34.060571+0000", 2024-09-10T18:33:03.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:33:03.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:33:03.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:33:03.997 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:33:03.997 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:33:03.997 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:33:03.997 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:33:03.997 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:33:03.997 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:33:03.997 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:33:03.997 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:33:03.997 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:33:03.998 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:33:03.998 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:33:03.998 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:33:03.998 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:33:03.998 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:33:03.998 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:33:03.998 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:33:03.998 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:33:03.998 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:33:03.998 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:33:03.998 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:33:03.998 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:33:03.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:03.998 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "63", 2024-09-10T18:33:03.998 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:33:03.999 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:33:03.999 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:33:03.999 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:33:03.999 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:33:03.999 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:33:03.999 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:33:03.999 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:33:03.999 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:33:03.999 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:33:03.999 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:33:03.999 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:33:03.999 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:33:03.999 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:33:04.000 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:33:04.000 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:33:04.000 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:33:04.000 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:33:04.000 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:33:04.000 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:33:04.000 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:33:04.000 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:33:04.000 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:33:04.000 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:33:04.000 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:33:04.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:33:04.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.001 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:33:04.001 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:33:04.001 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:33:04.001 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:33:04.001 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:33:04.001 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:33:04.001 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:33:04.001 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:33:04.001 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:33:04.001 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:33:04.001 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:33:04.002 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:33:04.002 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:33:04.002 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:33:04.002 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.002 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-09-10T18:33:04.002 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-09-10T18:33:04.002 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:36.127330+0000", 2024-09-10T18:33:04.002 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:33:04.002 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:33:04.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:33:04.002 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:33:04.003 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:33:04.003 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:33:04.003 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:33:04.003 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:33:04.003 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:33:04.003 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:33:04.003 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:33:04.003 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:33:04.003 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:33:04.003 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:33:04.003 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:33:04.003 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:33:04.003 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:33:04.003 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:33:04.004 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:33:04.004 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:33:04.004 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:33:04.004 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:33:04.004 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:33:04.004 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:33:04.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.004 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "65", 2024-09-10T18:33:04.004 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:33:04.004 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:33:04.004 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:33:04.004 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:33:04.004 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:33:04.005 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:33:04.005 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:33:04.005 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:33:04.005 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:33:04.005 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:33:04.005 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:33:04.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:33:04.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:33:04.005 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:33:04.005 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:33:04.005 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:33:04.005 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:33:04.005 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:33:04.005 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:33:04.005 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:33:04.006 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:33:04.006 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:33:04.006 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:33:04.006 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:33:04.006 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:33:04.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:33:04.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.006 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:33:04.006 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:33:04.006 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:33:04.006 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:33:04.006 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:33:04.006 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:33:04.006 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:33:04.007 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:33:04.007 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:33:04.007 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:33:04.007 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:33:04.007 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-10T18:33:04.007 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-09-10T18:33:04.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.008 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:33:04.008 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:33:04.008 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.008 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 9, 2024-09-10T18:33:04.008 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "unique_pool_2", 2024-09-10T18:33:04.008 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:32:02.316841+0000", 2024-09-10T18:33:04.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:33:04.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:33:04.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:33:04.009 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:33:04.009 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:33:04.009 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:33:04.009 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:33:04.009 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:33:04.009 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:33:04.009 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:33:04.009 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:33:04.009 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:33:04.009 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 16, 2024-09-10T18:33:04.009 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 16, 2024-09-10T18:33:04.009 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 16, 2024-09-10T18:33:04.010 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 16, 2024-09-10T18:33:04.010 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 16, 2024-09-10T18:33:04.010 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:33:04.010 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:33:04.010 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:33:04.010 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:33:04.010 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:33:04.010 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:33:04.010 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:33:04.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.010 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "249", 2024-09-10T18:33:04.010 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:33:04.010 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:33:04.010 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:33:04.011 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:33:04.011 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:33:04.011 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:33:04.011 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:33:04.011 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:33:04.011 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:33:04.011 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:33:04.011 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:33:04.011 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:33:04.011 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:33:04.011 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:33:04.011 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:33:04.011 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:33:04.012 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:33:04.012 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:33:04.012 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:33:04.012 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:33:04.012 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:33:04.012 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:33:04.012 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:33:04.012 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:33:04.012 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:33:04.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:33:04.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.012 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:33:04.012 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:33:04.013 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:33:04.013 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:33:04.013 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:33:04.013 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:33:04.013 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:33:04.013 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:33:04.013 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:33:04.013 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:33:04.013 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:33:04.013 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:33:04.013 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:33:04.013 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-09-10T18:33:04.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.014 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:33:04.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:33:04.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-10T18:33:04.014 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "f1c834d9-252a-4077-b25e-169181d70251", 2024-09-10T18:33:04.014 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:33:04.014 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:33:04.014 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:33:04.014 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:33:04.014 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:33:04.014 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:33:04.014 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-09-10T18:33:04.014 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 246, 2024-09-10T18:33:04.014 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:33:04.014 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:33:04.015 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:33:04.015 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.015 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-09-10T18:33:04.015 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:33:04.015 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.015 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.015 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:33:04.015 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.015 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-09-10T18:33:04.015 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:33:04.016 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.016 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.016 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:33:04.016 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.016 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-09-10T18:33:04.016 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:33:04.016 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.016 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.016 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:33:04.017 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.017 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.017 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-09-10T18:33:04.017 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:33:04.017 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.017 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.017 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/3188372351", 2024-09-10T18:33:04.017 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/3188372351", 2024-09-10T18:33:04.017 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/3188372351", 2024-09-10T18:33:04.017 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/3188372351", 2024-09-10T18:33:04.017 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:33:04.017 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:33:04.018 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:33:04.018 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.018 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-10T18:33:04.018 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "81f1855d-b946-4ca3-b246-38977b9e20fd", 2024-09-10T18:33:04.018 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:33:04.018 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:33:04.018 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:33:04.018 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:33:04.018 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:33:04.018 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:33:04.018 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-10T18:33:04.019 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 246, 2024-09-10T18:33:04.019 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:33:04.019 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:33:04.019 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:33:04.019 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.019 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.019 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-09-10T18:33:04.019 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:33:04.019 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.019 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.019 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:33:04.019 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.020 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-09-10T18:33:04.020 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:33:04.020 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.020 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.020 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:33:04.020 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.020 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-09-10T18:33:04.021 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:33:04.021 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.021 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.021 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:33:04.021 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.021 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-09-10T18:33:04.021 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:33:04.021 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.021 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.022 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1369912678", 2024-09-10T18:33:04.022 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1369912678", 2024-09-10T18:33:04.022 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1369912678", 2024-09-10T18:33:04.022 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1369912678", 2024-09-10T18:33:04.022 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:33:04.022 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:33:04.022 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:33:04.022 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.022 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-10T18:33:04.022 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "2ad55d2a-36fa-4756-bf22-58d4b049486e", 2024-09-10T18:33:04.022 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:33:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:33:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:33:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:33:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:33:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:33:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-09-10T18:33:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 95, 2024-09-10T18:33:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:33:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:33:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:33:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-09-10T18:33:04.024 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:33:04.024 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.024 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.024 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:33:04.024 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.024 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-09-10T18:33:04.024 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:33:04.024 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.024 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.024 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:33:04.025 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.025 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.025 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-09-10T18:33:04.025 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:33:04.025 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.025 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.025 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:33:04.025 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.025 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.026 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-09-10T18:33:04.026 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:33:04.026 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.026 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.026 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/1243406003", 2024-09-10T18:33:04.026 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/1243406003", 2024-09-10T18:33:04.026 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/1243406003", 2024-09-10T18:33:04.026 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1243406003", 2024-09-10T18:33:04.026 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:33:04.026 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:33:04.026 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:33:04.026 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.027 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-10T18:33:04.027 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "242729a5-a5e8-44d5-97c8-14d14e970bff", 2024-09-10T18:33:04.027 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:33:04.027 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:33:04.027 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:33:04.027 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:33:04.027 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:33:04.027 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:33:04.027 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-09-10T18:33:04.027 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 246, 2024-09-10T18:33:04.027 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:33:04.027 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:33:04.027 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:33:04.028 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.028 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.028 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-09-10T18:33:04.028 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:33:04.028 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.028 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.028 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:33:04.028 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.028 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.028 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-09-10T18:33:04.028 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:33:04.029 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.029 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.029 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:33:04.029 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.029 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-09-10T18:33:04.029 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:33:04.029 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.029 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.029 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:33:04.029 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.030 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.030 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-09-10T18:33:04.030 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:33:04.030 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.030 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.030 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/67703283", 2024-09-10T18:33:04.030 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/67703283", 2024-09-10T18:33:04.030 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/67703283", 2024-09-10T18:33:04.030 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/67703283", 2024-09-10T18:33:04.030 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:33:04.030 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:33:04.030 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:33:04.030 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-10T18:33:04.031 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "6e38b8ef-e6ed-493f-8eca-755354754787", 2024-09-10T18:33:04.031 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:33:04.031 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:33:04.031 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:33:04.031 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:33:04.031 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:33:04.031 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:33:04.031 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 33, 2024-09-10T18:33:04.032 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 219, 2024-09-10T18:33:04.032 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:33:04.032 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:33:04.032 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:33:04.032 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.032 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.032 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6800", 2024-09-10T18:33:04.032 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:33:04.032 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.032 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.032 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:33:04.033 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.033 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6801", 2024-09-10T18:33:04.033 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:33:04.033 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.033 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.033 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:33:04.033 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.033 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6803", 2024-09-10T18:33:04.033 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:33:04.033 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.034 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.034 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:33:04.034 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.034 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6802", 2024-09-10T18:33:04.034 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:33:04.034 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.034 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.034 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6800/1619669076", 2024-09-10T18:33:04.034 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6801/1619669076", 2024-09-10T18:33:04.034 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6803/1619669076", 2024-09-10T18:33:04.034 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6802/1619669076", 2024-09-10T18:33:04.035 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:33:04.035 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:33:04.035 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:33:04.035 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.035 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-10T18:33:04.035 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "a92e0441-66c9-498e-a0ac-6465acdbb1f5", 2024-09-10T18:33:04.035 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:33:04.035 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:33:04.035 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:33:04.035 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:33:04.035 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:33:04.035 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:33:04.036 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 39, 2024-09-10T18:33:04.036 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 246, 2024-09-10T18:33:04.036 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:33:04.036 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:33:04.036 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:33:04.036 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.036 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.036 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6804", 2024-09-10T18:33:04.036 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:33:04.036 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.036 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.036 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:33:04.037 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.037 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.037 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6805", 2024-09-10T18:33:04.037 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:33:04.037 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.037 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.037 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:33:04.037 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.037 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.037 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6807", 2024-09-10T18:33:04.037 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:33:04.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.038 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.038 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:33:04.038 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.038 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6806", 2024-09-10T18:33:04.038 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:33:04.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.038 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.038 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6804/2717802880", 2024-09-10T18:33:04.038 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6805/2717802880", 2024-09-10T18:33:04.038 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6807/2717802880", 2024-09-10T18:33:04.039 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6806/2717802880", 2024-09-10T18:33:04.039 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:33:04.039 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:33:04.039 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:33:04.039 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.039 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-09-10T18:33:04.039 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "12b02875-6653-45ed-8d2b-b4d33bd88525", 2024-09-10T18:33:04.039 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:33:04.039 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:33:04.039 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:33:04.039 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:33:04.039 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 46, 2024-09-10T18:33:04.039 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 94, 2024-09-10T18:33:04.040 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 144, 2024-09-10T18:33:04.040 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 210, 2024-09-10T18:33:04.040 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 95, 2024-09-10T18:33:04.040 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:33:04.040 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:33:04.040 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.040 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6808", 2024-09-10T18:33:04.040 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:33:04.040 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.040 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.040 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:33:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6809", 2024-09-10T18:33:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:33:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:33:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6811", 2024-09-10T18:33:04.041 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:33:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:33:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6810", 2024-09-10T18:33:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:33:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6808/3479311452", 2024-09-10T18:33:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6809/3479311452", 2024-09-10T18:33:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6811/3479311452", 2024-09-10T18:33:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6810/3479311452", 2024-09-10T18:33:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:33:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:33:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:33:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-09-10T18:33:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "0c4f7aa9-ff95-4192-a146-ca576eee907f", 2024-09-10T18:33:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:33:04.043 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:33:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:33:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:33:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:33:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:33:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 54, 2024-09-10T18:33:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 259, 2024-09-10T18:33:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:33:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:33:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:33:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.044 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6812", 2024-09-10T18:33:04.045 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:33:04.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.045 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.045 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:33:04.045 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.045 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6813", 2024-09-10T18:33:04.045 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:33:04.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.045 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.045 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:33:04.046 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.046 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6815", 2024-09-10T18:33:04.046 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:33:04.046 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.046 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.046 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:33:04.046 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:33:04.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.047 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:33:04.047 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6814", 2024-09-10T18:33:04.047 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:33:04.047 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.047 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.047 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6812/3511791879", 2024-09-10T18:33:04.047 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6813/3511791879", 2024-09-10T18:33:04.047 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6815/3511791879", 2024-09-10T18:33:04.047 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6814/3511791879", 2024-09-10T18:33:04.047 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:33:04.047 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:33:04.047 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:33:04.047 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.048 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.048 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:33:04.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-09-10T18:33:04.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-10T18:33:04.048 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:33:04.048 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:33:04.048 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:33:04.048 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:33:04.048 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:33:04.048 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:32:04.145891+0000", 2024-09-10T18:33:04.048 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:33:04.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-10T18:33:04.049 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:33:04.049 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:33:04.049 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:33:04.049 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:33:04.049 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:33:04.049 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:32:26.296451+0000", 2024-09-10T18:33:04.049 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:33:04.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.049 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-10T18:33:04.049 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:33:04.049 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:33:04.049 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:33:04.050 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:33:04.050 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:33:04.050 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:32:33.256290+0000", 2024-09-10T18:33:04.050 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:33:04.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.050 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-10T18:33:04.050 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:33:04.050 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:33:04.050 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:33:04.050 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:33:04.050 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:33:04.050 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:32:00.088088+0000", 2024-09-10T18:33:04.050 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:33:04.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.051 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-10T18:33:04.051 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:33:04.051 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:33:04.051 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:33:04.051 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:33:04.051 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:33:04.051 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:32:34.957631+0000", 2024-09-10T18:33:04.051 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:33:04.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.051 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-10T18:33:04.051 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:33:04.052 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:33:04.052 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:33:04.052 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:33:04.052 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:33:04.052 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:31:44.879737+0000", 2024-09-10T18:33:04.052 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:33:04.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.052 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-09-10T18:33:04.052 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:33:04.052 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:33:04.052 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:33:04.052 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:33:04.052 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:33:04.053 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:32:44.999951+0000", 2024-09-10T18:33:04.053 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 94 2024-09-10T18:33:04.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.053 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-09-10T18:33:04.053 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:33:04.053 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:33:04.053 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:33:04.053 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:33:04.053 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:33:04.053 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:32:16.131177+0000", 2024-09-10T18:33:04.053 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:33:04.053 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.053 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:33:04.054 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-09-10T18:33:04.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.054 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.c", 2024-09-10T18:33:04.054 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:33:04.054 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-09-10T18:33:04.054 INFO:teuthology.orchestra.run.smithi028.stdout: 4, 2024-09-10T18:33:04.054 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-10T18:33:04.054 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.054 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.054 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:33:04.054 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-09-10T18:33:04.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.054 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.4", 2024-09-10T18:33:04.054 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:33:04.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.055 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:33:04.055 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-09-10T18:33:04.055 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.055 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.055 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1b", 2024-09-10T18:33:04.055 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:33:04.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.056 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:33:04.056 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-09-10T18:33:04.056 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.056 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.056 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1f", 2024-09-10T18:33:04.056 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:33:04.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:04.057 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:33:04.057 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-09-10T18:33:04.057 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.057 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:04.057 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.057 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:33:04.057 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-09-10T18:33:04.057 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-09-10T18:33:04.057 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-09-10T18:33:04.057 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1745574083": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:33:04.057 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3866220552": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:33:04.057 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3760775432": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:33:04.057 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1746797014": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:33:04.057 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1029045895": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:33:04.058 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3310949126": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:33:04.058 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2874160765": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:33:04.058 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2620179974": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:33:04.058 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/421194352": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:33:04.058 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/1077377902": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:33:04.058 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3428757300": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:33:04.058 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1759116360": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:33:04.058 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2564333506": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:33:04.058 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/717120167": "2024-09-11T18:22:08.678985+0000" 2024-09-10T18:33:04.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.058 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-09-10T18:33:04.058 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-09-10T18:33:04.058 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-09-10T18:33:04.058 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-09-10T18:33:04.058 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-09-10T18:33:04.059 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-09-10T18:33:04.059 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-09-10T18:33:04.059 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-09-10T18:33:04.059 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:04.059 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-09-10T18:33:04.059 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-09-10T18:33:04.059 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-09-10T18:33:04.059 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-09-10T18:33:04.059 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-09-10T18:33:04.059 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-09-10T18:33:04.059 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-09-10T18:33:04.059 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-09-10T18:33:04.059 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-09-10T18:33:04.059 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-09-10T18:33:04.060 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-09-10T18:33:04.060 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:04.060 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:33:04.060 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:04.067 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 51 16 89622 89606 109.737 60.5 0.0123697 0.00910759 2024-09-10T18:33:04.086 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:04.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:04.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:04.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:04.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:04.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:04.333 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:04.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:04.368 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:04.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:04.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:04.503 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:04 smithi028 ceph-mon[39220]: pgmap v518: 158 pgs: 158 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 708 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-10T18:33:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:04 smithi028 ceph-mon[39220]: 5.1f scrub starts 2024-09-10T18:33:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:04 smithi028 ceph-mon[39220]: 5.1f scrub ok 2024-09-10T18:33:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:04 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1518860635' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:33:04.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:04 smithi028 ceph-mon[29291]: pgmap v518: 158 pgs: 158 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 708 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-10T18:33:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:04 smithi028 ceph-mon[29291]: 5.1f scrub starts 2024-09-10T18:33:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:04 smithi028 ceph-mon[29291]: 5.1f scrub ok 2024-09-10T18:33:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:04 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1518860635' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:33:04.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:04.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:04.512 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:04 smithi204 ceph-mon[32357]: pgmap v518: 158 pgs: 158 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 708 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-10T18:33:04.512 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:04 smithi204 ceph-mon[32357]: 5.1f scrub starts 2024-09-10T18:33:04.513 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:04 smithi204 ceph-mon[32357]: 5.1f scrub ok 2024-09-10T18:33:04.513 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:04 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1518860635' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:33:04.608 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:04.647 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:04.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:04.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:04.749 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:04.781 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:04.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:04.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:04.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:33:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:33:04.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:04.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:04.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:05.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:05.068 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 52 16 90963 90947 109.238 83.8125 0.0118625 0.00914943 2024-09-10T18:33:05.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:05.069 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:05.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:05.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:05.213 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:05.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:05.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:05.315 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:05.351 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:05.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:05.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:05.453 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:05.475 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:05.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:05.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:05.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:05.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:05.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:05.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:05.773 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:05.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:05.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:05.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:05.918 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:05.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:05.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:06.020 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:06.052 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:06.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:06.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:06.068 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 53 16 92258 92242 108.704 80.9375 0.0148351 0.00919407 2024-09-10T18:33:06.077 INFO:tasks.thrashosds.thrasher:j is {'epoch': 263, 'fsid': '5eababb2-6fa0-11ef-bcea-c7b262605968', 'created': '2024-09-10T18:16:20.680373+0000', 'modified': '2024-09-10T18:33:00.753300+0000', 'last_up_change': '2024-09-10T18:26:26.611338+0000', 'last_in_change': '2024-09-10T18:21:02.114621+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T18:19:24.013536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '136', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-10T18:21:30.065209+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 62, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T18:21:30.506441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T18:21:32.008130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T18:21:34.060571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T18:21:36.127330+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-10T18:32:02.316841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '249', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'f1c834d9-252a-4077-b25e-169181d70251', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 246, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 3188372351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 3188372351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 3188372351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 3188372351}]}, 'public_addr': '172.21.15.28:6801/3188372351', 'cluster_addr': '172.21.15.28:6802/3188372351', 'heartbeat_back_addr': '172.21.15.28:6804/3188372351', 'heartbeat_front_addr': '172.21.15.28:6803/3188372351', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '81f1855d-b946-4ca3-b246-38977b9e20fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 246, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1369912678}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1369912678}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1369912678}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1369912678}]}, 'public_addr': '172.21.15.28:6805/1369912678', 'cluster_addr': '172.21.15.28:6806/1369912678', 'heartbeat_back_addr': '172.21.15.28:6808/1369912678', 'heartbeat_front_addr': '172.21.15.28:6807/1369912678', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2ad55d2a-36fa-4756-bf22-58d4b049486e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 1243406003}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 1243406003}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 1243406003}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1243406003}]}, 'public_addr': '172.21.15.28:6809/1243406003', 'cluster_addr': '172.21.15.28:6810/1243406003', 'heartbeat_back_addr': '172.21.15.28:6812/1243406003', 'heartbeat_front_addr': '172.21.15.28:6811/1243406003', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '242729a5-a5e8-44d5-97c8-14d14e970bff', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 246, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 67703283}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 67703283}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 67703283}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 67703283}]}, 'public_addr': '172.21.15.28:6813/67703283', 'cluster_addr': '172.21.15.28:6814/67703283', 'heartbeat_back_addr': '172.21.15.28:6816/67703283', 'heartbeat_front_addr': '172.21.15.28:6815/67703283', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6e38b8ef-e6ed-493f-8eca-755354754787', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 219, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6800', 'nonce': 1619669076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 1619669076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 1619669076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6802', 'nonce': 1619669076}]}, 'public_addr': '172.21.15.204:6800/1619669076', 'cluster_addr': '172.21.15.204:6801/1619669076', 'heartbeat_back_addr': '172.21.15.204:6803/1619669076', 'heartbeat_front_addr': '172.21.15.204:6802/1619669076', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a92e0441-66c9-498e-a0ac-6465acdbb1f5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 246, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6804', 'nonce': 2717802880}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 2717802880}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 2717802880}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6806', 'nonce': 2717802880}]}, 'public_addr': '172.21.15.204:6804/2717802880', 'cluster_addr': '172.21.15.204:6805/2717802880', 'heartbeat_back_addr': '172.21.15.204:6807/2717802880', 'heartbeat_front_addr': '172.21.15.204:6806/2717802880', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '12b02875-6653-45ed-8d2b-b4d33bd88525', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 46, 'last_clean_end': 94, 'up_from': 144, 'up_thru': 210, 'down_at': 95, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6808', 'nonce': 3479311452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 3479311452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 3479311452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6810', 'nonce': 3479311452}]}, 'public_addr': '172.21.15.204:6808/3479311452', 'cluster_addr': '172.21.15.204:6809/3479311452', 'heartbeat_back_addr': '172.21.15.204:6811/3479311452', 'heartbeat_front_addr': '172.21.15.204:6810/3479311452', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0c4f7aa9-ff95-4192-a146-ca576eee907f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 259, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6812', 'nonce': 3511791879}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 3511791879}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 3511791879}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6814', 'nonce': 3511791879}]}, 'public_addr': '172.21.15.204:6812/3511791879', 'cluster_addr': '172.21.15.204:6813/3511791879', 'heartbeat_back_addr': '172.21.15.204:6815/3511791879', 'heartbeat_front_addr': '172.21.15.204:6814/3511791879', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:32:04.145891+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:32:26.296451+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:32:33.256290+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:32:00.088088+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:32:34.957631+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:31:44.879737+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:32:44.999951+0000', 'dead_epoch': 94}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:32:16.131177+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.c', 'osds': [7, 4, 2]}], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/1745574083': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/3866220552': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:6800/3760775432': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:0/1746797014': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:0/1029045895': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:6800/3310949126': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/2874160765': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/2620179974': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/421194352': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:6800/1077377902': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/3428757300': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/1759116360': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/2564333506': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/717120167': '2024-09-11T18:22:08.678985+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T18:33:06.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:33:06.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:06.203 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:06.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:06.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:06.305 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:06.327 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:06.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:06.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:06.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:06.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[29291]: 3.1f scrub starts 2024-09-10T18:33:06.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[29291]: 3.1f scrub ok 2024-09-10T18:33:06.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:06.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[29291]: 4.1c scrub starts 2024-09-10T18:33:06.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[29291]: 4.1c scrub ok 2024-09-10T18:33:06.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[29291]: 2.2 scrub starts 2024-09-10T18:33:06.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[29291]: 2.2 scrub ok 2024-09-10T18:33:06.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[29291]: pgmap v519: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-10T18:33:06.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[29291]: 3.f scrub starts 2024-09-10T18:33:06.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[29291]: 3.f scrub ok 2024-09-10T18:33:06.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[39220]: 3.1f scrub starts 2024-09-10T18:33:06.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[39220]: 3.1f scrub ok 2024-09-10T18:33:06.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:06.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[39220]: 4.1c scrub starts 2024-09-10T18:33:06.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[39220]: 4.1c scrub ok 2024-09-10T18:33:06.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[39220]: 2.2 scrub starts 2024-09-10T18:33:06.465 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[39220]: 2.2 scrub ok 2024-09-10T18:33:06.465 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[39220]: pgmap v519: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-10T18:33:06.465 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[39220]: 3.f scrub starts 2024-09-10T18:33:06.465 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:06 smithi028 ceph-mon[39220]: 3.f scrub ok 2024-09-10T18:33:06.479 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:06.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:06.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:06.581 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:06 smithi204 ceph-mon[32357]: 3.1f scrub starts 2024-09-10T18:33:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:06 smithi204 ceph-mon[32357]: 3.1f scrub ok 2024-09-10T18:33:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:06 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:06 smithi204 ceph-mon[32357]: 4.1c scrub starts 2024-09-10T18:33:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:06 smithi204 ceph-mon[32357]: 4.1c scrub ok 2024-09-10T18:33:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:06 smithi204 ceph-mon[32357]: 2.2 scrub starts 2024-09-10T18:33:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:06 smithi204 ceph-mon[32357]: 2.2 scrub ok 2024-09-10T18:33:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:06 smithi204 ceph-mon[32357]: pgmap v519: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-10T18:33:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:06 smithi204 ceph-mon[32357]: 3.f scrub starts 2024-09-10T18:33:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:06 smithi204 ceph-mon[32357]: 3.f scrub ok 2024-09-10T18:33:06.614 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:06.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:06.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:06.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:06.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:06.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:06.861 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:06.908 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:06.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:06.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:06.973 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:33:06.974 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:33:07.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:07.015 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:33:07.022 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:33:07.022 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:33:07.022 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:33:07.034 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:07.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:07.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:07.061 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:33:07.061 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:33:07.068 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 54 16 93347 93331 107.951 68.0625 0.00627451 0.00925746 2024-09-10T18:33:07.100 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:33:07.109 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:33:07.109 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:33:07.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:07.141 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:33:07.141 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:33:07.175 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:33:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:33:07.208 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:33:07.208 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:33:07.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:07.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:07.241 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:33:07.242 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:33:07.311 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:07.344 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:07.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:07.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:07.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:07.491 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:07.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:07.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:07.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[29291]: 6.a scrub starts 2024-09-10T18:33:07.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[29291]: 6.a scrub ok 2024-09-10T18:33:07.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:33:07.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3468215400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:33:07.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:33:07.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[39220]: 6.a scrub starts 2024-09-10T18:33:07.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[39220]: 6.a scrub ok 2024-09-10T18:33:07.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:33:07.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3468215400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:33:07.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:07 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:33:07.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:07 smithi204 ceph-mon[32357]: 6.a scrub starts 2024-09-10T18:33:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:07 smithi204 ceph-mon[32357]: 6.a scrub ok 2024-09-10T18:33:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:07 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:33:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:07 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3468215400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:33:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:07 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:07 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:07 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:07 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:07 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:07 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:07 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:07 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:07.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:07 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:33:07.630 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:07.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:07.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:07.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:07.775 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:07.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:07.876 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:07.898 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:07.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:08.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:08.034 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:08.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:08.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:08.068 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 55 16 94475 94459 107.27 70.5 0.0120065 0.00931716 2024-09-10T18:33:08.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:08.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:08.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:08.287 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:08.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:08.328 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:08.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:08.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:08.475 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:08.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:08.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: 5.f scrub starts 2024-09-10T18:33:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: 5.f scrub ok 2024-09-10T18:33:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: pgmap v520: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-10T18:33:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: Adjusting osd_memory_target on smithi204 to 3248M 2024-09-10T18:33:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: Adjusting osd_memory_target on smithi028 to 3249M 2024-09-10T18:33:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:33:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:33:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:33:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:33:08.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: 5.0 scrub starts 2024-09-10T18:33:08.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[29291]: 5.0 scrub ok 2024-09-10T18:33:08.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: 5.f scrub starts 2024-09-10T18:33:08.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: 5.f scrub ok 2024-09-10T18:33:08.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: pgmap v520: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-10T18:33:08.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: Adjusting osd_memory_target on smithi204 to 3248M 2024-09-10T18:33:08.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: Adjusting osd_memory_target on smithi028 to 3249M 2024-09-10T18:33:08.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:33:08.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:33:08.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:33:08.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:33:08.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: 5.0 scrub starts 2024-09-10T18:33:08.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:08 smithi028 ceph-mon[39220]: 5.0 scrub ok 2024-09-10T18:33:08.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: 5.f scrub starts 2024-09-10T18:33:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: 5.f scrub ok 2024-09-10T18:33:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: pgmap v520: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-10T18:33:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: Adjusting osd_memory_target on smithi204 to 3248M 2024-09-10T18:33:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:33:08.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: Adjusting osd_memory_target on smithi028 to 3249M 2024-09-10T18:33:08.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:33:08.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:33:08.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:33:08.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:33:08.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: 5.0 scrub starts 2024-09-10T18:33:08.606 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:08 smithi204 ceph-mon[32357]: 5.0 scrub ok 2024-09-10T18:33:08.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:08.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:08.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:08.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:08.748 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:08.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:08.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:08.850 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:08.884 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:08.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:08.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:08.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:09.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:09.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:09.068 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 56 16 95778 95762 106.809 81.4375 0.00307915 0.00934922 2024-09-10T18:33:09.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:09.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:09.259 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:09.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:09.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:09.396 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:09.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:09.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:09 smithi028 ceph-mon[29291]: 5.1c scrub starts 2024-09-10T18:33:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:09 smithi028 ceph-mon[29291]: 5.1c scrub ok 2024-09-10T18:33:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:09 smithi028 ceph-mon[29291]: 1.8 scrub starts 2024-09-10T18:33:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:09 smithi028 ceph-mon[29291]: 1.8 scrub ok 2024-09-10T18:33:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:33:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:33:09.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:09 smithi028 ceph-mon[39220]: 5.1c scrub starts 2024-09-10T18:33:09.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:09 smithi028 ceph-mon[39220]: 5.1c scrub ok 2024-09-10T18:33:09.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:09 smithi028 ceph-mon[39220]: 1.8 scrub starts 2024-09-10T18:33:09.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:09 smithi028 ceph-mon[39220]: 1.8 scrub ok 2024-09-10T18:33:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:33:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:33:09.539 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:09.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:09.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:09.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:09 smithi204 ceph-mon[32357]: 5.1c scrub starts 2024-09-10T18:33:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:09 smithi204 ceph-mon[32357]: 5.1c scrub ok 2024-09-10T18:33:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:09 smithi204 ceph-mon[32357]: 1.8 scrub starts 2024-09-10T18:33:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:09 smithi204 ceph-mon[32357]: 1.8 scrub ok 2024-09-10T18:33:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:33:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:33:09.604 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:33:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=sqlstore.transactions t=2024-09-10T18:33:09.148211703Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-10T18:33:09.604 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:33:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=cleanup t=2024-09-10T18:33:09.150723634Z level=info msg="Completed cleanup jobs" duration=12.953568ms 2024-09-10T18:33:09.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:09.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:09.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:09.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:09.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:09.849 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:09.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:09.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:09.951 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:09.984 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:09.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:09.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:10.068 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 57 16 97076 97060 106.359 81.125 0.00609222 0.00939749 2024-09-10T18:33:10.086 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:10.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:10.125 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:10.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:10.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:10.264 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:10.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:10.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:10.365 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:10.405 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:10.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:10.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[29291]: 4.16 scrub starts 2024-09-10T18:33:10.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[29291]: 4.16 scrub ok 2024-09-10T18:33:10.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[29291]: 6.1a scrub starts 2024-09-10T18:33:10.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[29291]: 6.1a scrub ok 2024-09-10T18:33:10.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[29291]: pgmap v521: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-10T18:33:10.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[29291]: 5.2 scrub starts 2024-09-10T18:33:10.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[29291]: 5.2 scrub ok 2024-09-10T18:33:10.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:33:10.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:33:10.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[39220]: 4.16 scrub starts 2024-09-10T18:33:10.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[39220]: 4.16 scrub ok 2024-09-10T18:33:10.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[39220]: 6.1a scrub starts 2024-09-10T18:33:10.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[39220]: 6.1a scrub ok 2024-09-10T18:33:10.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[39220]: pgmap v521: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-10T18:33:10.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[39220]: 5.2 scrub starts 2024-09-10T18:33:10.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[39220]: 5.2 scrub ok 2024-09-10T18:33:10.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:33:10.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:10 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:33:10.507 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:10.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:10.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:10.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:10 smithi204 ceph-mon[32357]: 4.16 scrub starts 2024-09-10T18:33:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:10 smithi204 ceph-mon[32357]: 4.16 scrub ok 2024-09-10T18:33:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:10 smithi204 ceph-mon[32357]: 6.1a scrub starts 2024-09-10T18:33:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:10 smithi204 ceph-mon[32357]: 6.1a scrub ok 2024-09-10T18:33:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:10 smithi204 ceph-mon[32357]: pgmap v521: 158 pgs: 158 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-10T18:33:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:10 smithi204 ceph-mon[32357]: 5.2 scrub starts 2024-09-10T18:33:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:10 smithi204 ceph-mon[32357]: 5.2 scrub ok 2024-09-10T18:33:10.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:10 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:33:10.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:10 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:33:10.639 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:10.674 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:10.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:10.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:10.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:10.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:10.923 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:10.963 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:10.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:11.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:11.068 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 58 16 98415 98399 105.968 83.6875 0.00666249 0.00943204 2024-09-10T18:33:11.099 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:11.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:11.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:11.201 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:11.239 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:11.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:11.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:11 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:33:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:11 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:33:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:11 smithi028 ceph-mon[29291]: 6.1 scrub starts 2024-09-10T18:33:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:11 smithi028 ceph-mon[29291]: 6.1 scrub ok 2024-09-10T18:33:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:11 smithi028 ceph-mon[29291]: osdmap e264: 8 total, 8 up, 8 in 2024-09-10T18:33:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:11 smithi028 ceph-mon[29291]: osdmap e265: 8 total, 8 up, 8 in 2024-09-10T18:33:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:11 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:33:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:11 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:33:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:11 smithi028 ceph-mon[39220]: 6.1 scrub starts 2024-09-10T18:33:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:11 smithi028 ceph-mon[39220]: 6.1 scrub ok 2024-09-10T18:33:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:11 smithi028 ceph-mon[39220]: osdmap e264: 8 total, 8 up, 8 in 2024-09-10T18:33:11.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:11 smithi028 ceph-mon[39220]: osdmap e265: 8 total, 8 up, 8 in 2024-09-10T18:33:11.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:11.376 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:11.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:11.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:11.478 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:11.510 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:11.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:11.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:11.595 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:33:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:33:11] "GET /metrics HTTP/1.1" 200 224462 "" "Prometheus/2.43.0" 2024-09-10T18:33:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:11 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:33:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:11 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:33:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:11 smithi204 ceph-mon[32357]: 6.1 scrub starts 2024-09-10T18:33:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:11 smithi204 ceph-mon[32357]: 6.1 scrub ok 2024-09-10T18:33:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:11 smithi204 ceph-mon[32357]: osdmap e264: 8 total, 8 up, 8 in 2024-09-10T18:33:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:11 smithi204 ceph-mon[32357]: osdmap e265: 8 total, 8 up, 8 in 2024-09-10T18:33:11.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:11.651 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:11.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:11.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:11.753 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:11.783 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:11.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:11.885 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:11.911 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:11.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:11.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:12.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:12.049 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:12.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:12.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:12.068 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 59 16 99710 99694 105.544 80.9375 0.00791298 0.0094697 2024-09-10T18:33:12.151 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:12.185 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:12.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:12.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:12.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:12.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:12.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:12.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:12.455 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:12.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:12.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:12.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:12 smithi028 ceph-mon[39220]: pgmap v524: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-09-10T18:33:12.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:12 smithi028 ceph-mon[39220]: 3.c deep-scrub starts 2024-09-10T18:33:12.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:12 smithi028 ceph-mon[39220]: 3.c deep-scrub ok 2024-09-10T18:33:12.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:12 smithi028 ceph-mon[39220]: 4.3 scrub starts 2024-09-10T18:33:12.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:12 smithi028 ceph-mon[39220]: 4.3 scrub ok 2024-09-10T18:33:12.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:12 smithi028 ceph-mon[39220]: 6.3 scrub starts 2024-09-10T18:33:12.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:12 smithi028 ceph-mon[39220]: 6.3 scrub ok 2024-09-10T18:33:12.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:12 smithi028 ceph-mon[29291]: pgmap v524: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-09-10T18:33:12.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:12 smithi028 ceph-mon[29291]: 3.c deep-scrub starts 2024-09-10T18:33:12.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:12 smithi028 ceph-mon[29291]: 3.c deep-scrub ok 2024-09-10T18:33:12.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:12 smithi028 ceph-mon[29291]: 4.3 scrub starts 2024-09-10T18:33:12.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:12 smithi028 ceph-mon[29291]: 4.3 scrub ok 2024-09-10T18:33:12.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:12 smithi028 ceph-mon[29291]: 6.3 scrub starts 2024-09-10T18:33:12.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:12 smithi028 ceph-mon[29291]: 6.3 scrub ok 2024-09-10T18:33:12.557 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:12.592 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:12.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:12.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:33:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:33:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:33:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:12 smithi204 ceph-mon[32357]: pgmap v524: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-09-10T18:33:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:12 smithi204 ceph-mon[32357]: 3.c deep-scrub starts 2024-09-10T18:33:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:12 smithi204 ceph-mon[32357]: 3.c deep-scrub ok 2024-09-10T18:33:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:12 smithi204 ceph-mon[32357]: 4.3 scrub starts 2024-09-10T18:33:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:12 smithi204 ceph-mon[32357]: 4.3 scrub ok 2024-09-10T18:33:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:12 smithi204 ceph-mon[32357]: 6.3 scrub starts 2024-09-10T18:33:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:12 smithi204 ceph-mon[32357]: 6.3 scrub ok 2024-09-10T18:33:12.695 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:12.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:12.731 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:12.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:12.832 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:12.866 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:12.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:12.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:12.968 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:12.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:12.998 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:12.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:13.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:33:13.068 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:33:13.068056+0000 min lat: 0.00134629 max lat: 0.235622 avg lat: 0.00950062 2024-09-10T18:33:13.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:33:13.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 60 16 101069 101053 105.2 84.9375 0.0122006 0.00950062 2024-09-10T18:33:13.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:13.166 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:13.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:13.267 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:13.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:13.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:13.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:13.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:13.433 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:13.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:13.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:13 smithi028 ceph-mon[39220]: 6.17 scrub starts 2024-09-10T18:33:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:13 smithi028 ceph-mon[39220]: 6.17 scrub ok 2024-09-10T18:33:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:13 smithi028 ceph-mon[39220]: 3.12 scrub starts 2024-09-10T18:33:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:13 smithi028 ceph-mon[39220]: 3.12 scrub ok 2024-09-10T18:33:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:13 smithi028 ceph-mon[29291]: 6.17 scrub starts 2024-09-10T18:33:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:13 smithi028 ceph-mon[29291]: 6.17 scrub ok 2024-09-10T18:33:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:13 smithi028 ceph-mon[29291]: 3.12 scrub starts 2024-09-10T18:33:13.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:13 smithi028 ceph-mon[29291]: 3.12 scrub ok 2024-09-10T18:33:13.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:13.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:13.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:13 smithi204 ceph-mon[32357]: 6.17 scrub starts 2024-09-10T18:33:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:13 smithi204 ceph-mon[32357]: 6.17 scrub ok 2024-09-10T18:33:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:13 smithi204 ceph-mon[32357]: 3.12 scrub starts 2024-09-10T18:33:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:13 smithi204 ceph-mon[32357]: 3.12 scrub ok 2024-09-10T18:33:13.675 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:13.705 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:13.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:13.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:13.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:13.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:13.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:13.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:13.980 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:13.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:13.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:14.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 61 16 102405 102389 104.845 83.5 0.0184614 0.00953311 2024-09-10T18:33:14.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:14.124 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:14.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:14.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:14.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:14.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:14.274 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:14.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:14.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:14.426 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:14.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:14.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:14 smithi028 ceph-mon[39220]: 4.18 scrub starts 2024-09-10T18:33:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:14 smithi028 ceph-mon[39220]: 4.18 scrub ok 2024-09-10T18:33:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:14 smithi028 ceph-mon[39220]: pgmap v525: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-10T18:33:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:14 smithi028 ceph-mon[39220]: 3.b deep-scrub starts 2024-09-10T18:33:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:14 smithi028 ceph-mon[39220]: 3.b deep-scrub ok 2024-09-10T18:33:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:14 smithi028 ceph-mon[39220]: 6.12 scrub starts 2024-09-10T18:33:14.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:14 smithi028 ceph-mon[39220]: 6.12 scrub ok 2024-09-10T18:33:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:14 smithi028 ceph-mon[29291]: 4.18 scrub starts 2024-09-10T18:33:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:14 smithi028 ceph-mon[29291]: 4.18 scrub ok 2024-09-10T18:33:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:14 smithi028 ceph-mon[29291]: pgmap v525: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-10T18:33:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:14 smithi028 ceph-mon[29291]: 3.b deep-scrub starts 2024-09-10T18:33:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:14 smithi028 ceph-mon[29291]: 3.b deep-scrub ok 2024-09-10T18:33:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:14 smithi028 ceph-mon[29291]: 6.12 scrub starts 2024-09-10T18:33:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:14 smithi028 ceph-mon[29291]: 6.12 scrub ok 2024-09-10T18:33:14.522 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:14 smithi204 ceph-mon[32357]: 4.18 scrub starts 2024-09-10T18:33:14.522 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:14 smithi204 ceph-mon[32357]: 4.18 scrub ok 2024-09-10T18:33:14.522 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:14 smithi204 ceph-mon[32357]: pgmap v525: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-10T18:33:14.523 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:14 smithi204 ceph-mon[32357]: 3.b deep-scrub starts 2024-09-10T18:33:14.523 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:14 smithi204 ceph-mon[32357]: 3.b deep-scrub ok 2024-09-10T18:33:14.523 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:14 smithi204 ceph-mon[32357]: 6.12 scrub starts 2024-09-10T18:33:14.523 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:14 smithi204 ceph-mon[32357]: 6.12 scrub ok 2024-09-10T18:33:14.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:14.567 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:14.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:14.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:14.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:14.705 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:14.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:14.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:14.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:14.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:33:14 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:33:14.857 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:14.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:14.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:14.959 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:14.983 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:14.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:14.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:15.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 62 16 103512 103496 104.27 69.1875 0.00310565 0.00957685 2024-09-10T18:33:15.084 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:15.115 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:15.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:15.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:15.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:15.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:15.255 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:15.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:15.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:15.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:15.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:15.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:15.498 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:15.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:15 smithi028 ceph-mon[29291]: 9.9 deep-scrub starts 2024-09-10T18:33:15.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:15 smithi028 ceph-mon[29291]: 3.e scrub starts 2024-09-10T18:33:15.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:15 smithi028 ceph-mon[29291]: 3.e scrub ok 2024-09-10T18:33:15.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:15 smithi028 ceph-mon[39220]: 9.9 deep-scrub starts 2024-09-10T18:33:15.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:15 smithi028 ceph-mon[39220]: 3.e scrub starts 2024-09-10T18:33:15.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:15 smithi028 ceph-mon[39220]: 3.e scrub ok 2024-09-10T18:33:15.535 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:15.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:15.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:15 smithi204 ceph-mon[32357]: 9.9 deep-scrub starts 2024-09-10T18:33:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:15 smithi204 ceph-mon[32357]: 3.e scrub starts 2024-09-10T18:33:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:15 smithi204 ceph-mon[32357]: 3.e scrub ok 2024-09-10T18:33:15.640 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:15.672 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:15.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:15.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:15.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:15.808 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:15.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:15.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:15.910 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:15.947 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:15.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:16.049 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:16.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 63 16 104661 104645 103.755 71.8125 0.0116973 0.0096315 2024-09-10T18:33:16.090 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:16.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:16.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:16.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:16.237 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:16.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:16.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:16.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:16.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:16.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:16.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:16 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:16.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:16 smithi028 ceph-mon[39220]: pgmap v526: 158 pgs: 158 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-10T18:33:16.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:16 smithi028 ceph-mon[39220]: 3.0 deep-scrub starts 2024-09-10T18:33:16.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:16 smithi028 ceph-mon[39220]: 3.0 deep-scrub ok 2024-09-10T18:33:16.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:16 smithi028 ceph-mon[39220]: 4.1e scrub starts 2024-09-10T18:33:16.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:16 smithi028 ceph-mon[39220]: 4.1e scrub ok 2024-09-10T18:33:16.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:16 smithi028 ceph-mon[39220]: 5.9 scrub starts 2024-09-10T18:33:16.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:16 smithi028 ceph-mon[39220]: 5.9 scrub ok 2024-09-10T18:33:16.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:16 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:16.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:16 smithi028 ceph-mon[29291]: pgmap v526: 158 pgs: 158 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-10T18:33:16.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:16 smithi028 ceph-mon[29291]: 3.0 deep-scrub starts 2024-09-10T18:33:16.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:16 smithi028 ceph-mon[29291]: 3.0 deep-scrub ok 2024-09-10T18:33:16.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:16 smithi028 ceph-mon[29291]: 4.1e scrub starts 2024-09-10T18:33:16.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:16 smithi028 ceph-mon[29291]: 4.1e scrub ok 2024-09-10T18:33:16.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:16 smithi028 ceph-mon[29291]: 5.9 scrub starts 2024-09-10T18:33:16.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:16 smithi028 ceph-mon[29291]: 5.9 scrub ok 2024-09-10T18:33:16.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:16 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:16 smithi204 ceph-mon[32357]: pgmap v526: 158 pgs: 158 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-10T18:33:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:16 smithi204 ceph-mon[32357]: 3.0 deep-scrub starts 2024-09-10T18:33:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:16 smithi204 ceph-mon[32357]: 3.0 deep-scrub ok 2024-09-10T18:33:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:16 smithi204 ceph-mon[32357]: 4.1e scrub starts 2024-09-10T18:33:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:16 smithi204 ceph-mon[32357]: 4.1e scrub ok 2024-09-10T18:33:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:16 smithi204 ceph-mon[32357]: 5.9 scrub starts 2024-09-10T18:33:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:16 smithi204 ceph-mon[32357]: 5.9 scrub ok 2024-09-10T18:33:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:16.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:16.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:16.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:16.789 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:16.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:16.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:16.892 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:16.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:16.926 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:16.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:17.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:17.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:17.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:17.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:17.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 64 15 105795 105780 103.242 70.9375 0.101345 0.00968045 2024-09-10T18:33:17.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:17.208 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:17.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:17.263 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:33:17.309 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:17.343 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:17.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:17.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:17.444 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:17.475 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:17.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:17.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:17.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:17 smithi204 ceph-mon[32357]: 4.13 scrub starts 2024-09-10T18:33:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:17 smithi204 ceph-mon[32357]: 4.13 scrub ok 2024-09-10T18:33:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:17 smithi204 ceph-mon[32357]: 4.15 scrub starts 2024-09-10T18:33:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:17 smithi204 ceph-mon[32357]: 4.15 scrub ok 2024-09-10T18:33:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:17 smithi204 ceph-mon[32357]: 4.b scrub starts 2024-09-10T18:33:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:17 smithi204 ceph-mon[32357]: 4.b scrub ok 2024-09-10T18:33:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:17 smithi204 ceph-mon[32357]: 3.11 scrub starts 2024-09-10T18:33:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:17 smithi204 ceph-mon[32357]: 3.11 scrub ok 2024-09-10T18:33:17.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:17 smithi028 ceph-mon[29291]: 4.13 scrub starts 2024-09-10T18:33:17.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:17 smithi028 ceph-mon[29291]: 4.13 scrub ok 2024-09-10T18:33:17.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:17 smithi028 ceph-mon[29291]: 4.15 scrub starts 2024-09-10T18:33:17.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:17 smithi028 ceph-mon[29291]: 4.15 scrub ok 2024-09-10T18:33:17.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:17 smithi028 ceph-mon[29291]: 4.b scrub starts 2024-09-10T18:33:17.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:17 smithi028 ceph-mon[29291]: 4.b scrub ok 2024-09-10T18:33:17.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:17 smithi028 ceph-mon[29291]: 3.11 scrub starts 2024-09-10T18:33:17.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:17 smithi028 ceph-mon[29291]: 3.11 scrub ok 2024-09-10T18:33:17.621 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:17 smithi028 ceph-mon[39220]: 4.13 scrub starts 2024-09-10T18:33:17.621 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:17 smithi028 ceph-mon[39220]: 4.13 scrub ok 2024-09-10T18:33:17.621 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:17 smithi028 ceph-mon[39220]: 4.15 scrub starts 2024-09-10T18:33:17.621 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:17 smithi028 ceph-mon[39220]: 4.15 scrub ok 2024-09-10T18:33:17.621 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:17 smithi028 ceph-mon[39220]: 4.b scrub starts 2024-09-10T18:33:17.621 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:17 smithi028 ceph-mon[39220]: 4.b scrub ok 2024-09-10T18:33:17.621 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:17 smithi028 ceph-mon[39220]: 3.11 scrub starts 2024-09-10T18:33:17.622 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:17 smithi028 ceph-mon[39220]: 3.11 scrub ok 2024-09-10T18:33:17.625 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:17.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:17.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:17.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:17.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:17.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:17.795 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:33:17.795 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:33:17.870 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:17.909 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:17.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:17.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:18.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:18.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:18.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:18.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 65 16 106820 106804 102.639 64 0.0174938 0.0097362 2024-09-10T18:33:18.154 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:18.188 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:18.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:18.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:18.290 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:18.291 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:33:18.321 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:18.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:18.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:18.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:18.466 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:18.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:18.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:18.568 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:18.599 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:18.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:18.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:18 smithi204 ceph-mon[32357]: 3.1d deep-scrub starts 2024-09-10T18:33:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:18 smithi204 ceph-mon[32357]: 3.1d deep-scrub ok 2024-09-10T18:33:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:18 smithi204 ceph-mon[32357]: pgmap v527: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-10T18:33:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:18 smithi204 ceph-mon[32357]: 9.0 scrub starts 2024-09-10T18:33:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:18 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2459777142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:33:18.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:18.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:18.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:18.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:18 smithi028 ceph-mon[39220]: 3.1d deep-scrub starts 2024-09-10T18:33:18.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:18 smithi028 ceph-mon[39220]: 3.1d deep-scrub ok 2024-09-10T18:33:18.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:18 smithi028 ceph-mon[39220]: pgmap v527: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-10T18:33:18.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:18 smithi028 ceph-mon[39220]: 9.0 scrub starts 2024-09-10T18:33:18.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:18 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2459777142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:33:18.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:18 smithi028 ceph-mon[29291]: 3.1d deep-scrub starts 2024-09-10T18:33:18.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:18 smithi028 ceph-mon[29291]: 3.1d deep-scrub ok 2024-09-10T18:33:18.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:18 smithi028 ceph-mon[29291]: pgmap v527: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-10T18:33:18.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:18 smithi028 ceph-mon[29291]: 9.0 scrub starts 2024-09-10T18:33:18.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:18 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2459777142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:33:18.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:18.869 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:18.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:18.970 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:19.003 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:19.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:19.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:19.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 66 16 107889 107873 102.096 66.8125 0.0102378 0.00978682 2024-09-10T18:33:19.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:19.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:19.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:19.245 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:19.277 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:19.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:19.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:19.379 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:19.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:19.420 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:19.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:19.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:19 smithi028 ceph-mon[39220]: 6.1d deep-scrub starts 2024-09-10T18:33:19.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:19 smithi028 ceph-mon[39220]: 6.1d deep-scrub ok 2024-09-10T18:33:19.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:19 smithi028 ceph-mon[39220]: 6.13 scrub starts 2024-09-10T18:33:19.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:19 smithi028 ceph-mon[39220]: 6.13 scrub ok 2024-09-10T18:33:19.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:19 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:33:19.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:19 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2459777142' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:33:19.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:19 smithi028 ceph-mon[39220]: osdmap e266: 8 total, 8 up, 8 in 2024-09-10T18:33:19.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:19 smithi028 ceph-mon[39220]: 3.1b scrub starts 2024-09-10T18:33:19.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:19 smithi028 ceph-mon[29291]: 6.1d deep-scrub starts 2024-09-10T18:33:19.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:19 smithi028 ceph-mon[29291]: 6.1d deep-scrub ok 2024-09-10T18:33:19.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:19 smithi028 ceph-mon[29291]: 6.13 scrub starts 2024-09-10T18:33:19.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:19 smithi028 ceph-mon[29291]: 6.13 scrub ok 2024-09-10T18:33:19.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:19 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:33:19.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:19 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2459777142' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:33:19.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:19 smithi028 ceph-mon[29291]: osdmap e266: 8 total, 8 up, 8 in 2024-09-10T18:33:19.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:19 smithi028 ceph-mon[29291]: 3.1b scrub starts 2024-09-10T18:33:19.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:19.556 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:19.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:19.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:19.597 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:33:19.600 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:33:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:19 smithi204 ceph-mon[32357]: 6.1d deep-scrub starts 2024-09-10T18:33:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:19 smithi204 ceph-mon[32357]: 6.1d deep-scrub ok 2024-09-10T18:33:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:19 smithi204 ceph-mon[32357]: 6.13 scrub starts 2024-09-10T18:33:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:19 smithi204 ceph-mon[32357]: 6.13 scrub ok 2024-09-10T18:33:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:19 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:33:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:19 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2459777142' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:33:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:19 smithi204 ceph-mon[32357]: osdmap e266: 8 total, 8 up, 8 in 2024-09-10T18:33:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:19 smithi204 ceph-mon[32357]: 3.1b scrub starts 2024-09-10T18:33:19.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:19.692 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:19.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:19.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:19.794 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:19.823 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:19.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:19.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:19.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:19.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:19.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:20.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:20.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 67 16 109071 109055 101.675 73.875 0.00666591 0.00982906 2024-09-10T18:33:20.100 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:20.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:20.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:20.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-10T18:33:20.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:20.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:20.280 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:20.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:20.381 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:20.416 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:20.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:20.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:20.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:20 smithi028 ceph-mon[29291]: from='client.35375 v1:172.21.15.28:0/4035439561' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:33:20.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:20 smithi028 ceph-mon[29291]: pgmap v529: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 621 B/s rd, 58 MiB/s wr, 923 op/s 2024-09-10T18:33:20.424 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:20 smithi028 ceph-mon[39220]: from='client.35375 v1:172.21.15.28:0/4035439561' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:33:20.424 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:20 smithi028 ceph-mon[39220]: pgmap v529: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 621 B/s rd, 58 MiB/s wr, 923 op/s 2024-09-10T18:33:20.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:20.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:20.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:20 smithi204 ceph-mon[32357]: from='client.35375 v1:172.21.15.28:0/4035439561' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:33:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:20 smithi204 ceph-mon[32357]: pgmap v529: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 621 B/s rd, 58 MiB/s wr, 923 op/s 2024-09-10T18:33:20.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:20.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:20.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:20.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:20.855 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:20.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:20.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:20.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:21.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:21.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:21.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 68 16 110196 110180 101.214 70.3125 0.0113705 0.0098744 2024-09-10T18:33:21.102 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:21.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:21.138 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:21.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:21.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:21.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:21.276 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:21.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:21.377 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:21.384 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":528,"stamp":"2024-09-10T18:33:18.286012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6805592284,"num_objects":104032,"num_object_clones":0,"num_object_copies":312096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104032,"num_whiteouts":0,"num_read":1844,"num_read_kb":1596,"num_write":104552,"num_write_kb":6645547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79578,"ondisk_log_size":79578,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22555588,"kb_used_data":19833220,"kb_used_omap":19,"kb_used_meta":2722284,"kb_avail":727405628,"statfs":{"total":767960285184,"available":744863363072,"internally_reserved":0,"allocated":20309217280,"data_stored":20305188380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20033,"internal_metadata":2787619263},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":646709248,"num_objects":9868,"num_object_clones":0,"num_object_copies":29604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9868,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":9868,"num_write_kb":631552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.073437"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":265,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.218894+0000","last_change":"2024-09-10T18:33:04.320548+0000","last_active":"2024-09-10T18:33:10.218894+0000","last_peered":"2024-09-10T18:33:10.218894+0000","last_clean":"2024-09-10T18:33:10.218894+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:33:10.218894+0000","last_undegraded":"2024-09-10T18:33:10.218894+0000","last_fullsized":"2024-09-10T18:33:10.218894+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:04.320458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:33:04.320458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.581200+0000","scrub_duration":0.0055421380000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":304,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:12.680293+0000","last_change":"2024-09-10T18:33:12.680256+0000","last_active":"2024-09-10T18:33:12.680293+0000","last_peered":"2024-09-10T18:33:12.680293+0000","last_clean":"2024-09-10T18:33:12.680293+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:33:12.680293+0000","last_undegraded":"2024-09-10T18:33:12.680293+0000","last_fullsized":"2024-09-10T18:33:12.680293+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:33:12.680138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:12.680138+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:19.317278+0000","scrub_duration":0.0033442979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":270,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.222711+0000","last_change":"2024-09-10T18:32:43.156005+0000","last_active":"2024-09-10T18:33:10.222711+0000","last_peered":"2024-09-10T18:33:10.222711+0000","last_clean":"2024-09-10T18:33:10.222711+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:33:10.222711+0000","last_undegraded":"2024-09-10T18:33:10.222711+0000","last_fullsized":"2024-09-10T18:33:10.222711+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:43.155924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:43.155924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:58.337772+0000","scrub_duration":0.007043718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":268,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.228612+0000","last_change":"2024-09-10T18:33:08.643815+0000","last_active":"2024-09-10T18:33:10.228612+0000","last_peered":"2024-09-10T18:33:10.228612+0000","last_clean":"2024-09-10T18:33:10.228612+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:33:10.228612+0000","last_undegraded":"2024-09-10T18:33:10.228612+0000","last_fullsized":"2024-09-10T18:33:10.228612+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:08.643758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.643758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:24.996028+0000","scrub_duration":0.0046354059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":267,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.231328+0000","last_change":"2024-09-10T18:31:54.788489+0000","last_active":"2024-09-10T18:33:10.231328+0000","last_peered":"2024-09-10T18:33:10.231328+0000","last_clean":"2024-09-10T18:33:10.231328+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:33:10.231328+0000","last_undegraded":"2024-09-10T18:33:10.231328+0000","last_fullsized":"2024-09-10T18:33:10.231328+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:54.788436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:54.788436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013008939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":273,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.230056+0000","last_change":"2024-09-10T18:32:00.727061+0000","last_active":"2024-09-10T18:33:10.230056+0000","last_peered":"2024-09-10T18:33:10.230056+0000","last_clean":"2024-09-10T18:33:10.230056+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:33:10.230056+0000","last_undegraded":"2024-09-10T18:33:10.230056+0000","last_fullsized":"2024-09-10T18:33:10.230056+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.727002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.727002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:29.752529+0000","scrub_duration":0.0014900709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'34","reported_seq":314,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.223413+0000","last_change":"2024-09-10T18:32:55.371863+0000","last_active":"2024-09-10T18:33:10.223413+0000","last_peered":"2024-09-10T18:33:10.223413+0000","last_clean":"2024-09-10T18:33:10.223413+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:33:10.223413+0000","last_undegraded":"2024-09-10T18:33:10.223413+0000","last_fullsized":"2024-09-10T18:33:10.223413+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:32:55.371617+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:32:55.371617+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:18.220770+0000","scrub_duration":0.0089694079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":270,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.228278+0000","last_change":"2024-09-10T18:31:48.634526+0000","last_active":"2024-09-10T18:33:10.228278+0000","last_peered":"2024-09-10T18:33:10.228278+0000","last_clean":"2024-09-10T18:33:10.228278+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:33:10.228278+0000","last_undegraded":"2024-09-10T18:33:10.228278+0000","last_fullsized":"2024-09-10T18:33:10.228278+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:48.634465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.634465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:14.831892+0000","scrub_duration":0.0012201320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":275,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.220578+0000","last_change":"2024-09-10T18:31:54.606380+0000","last_active":"2024-09-10T18:33:10.220578+0000","last_peered":"2024-09-10T18:33:10.220578+0000","last_clean":"2024-09-10T18:33:10.220578+0000","last_became_active":"2024-09-10T18:26:27.644961+0000","last_became_peered":"2024-09-10T18:26:27.644961+0000","last_unstale":"2024-09-10T18:33:10.220578+0000","last_undegraded":"2024-09-10T18:33:10.220578+0000","last_fullsized":"2024-09-10T18:33:10.220578+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:54.606321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.606321+0000","last_clean_scrub_stamp":"2024-09-10T18:31:54.606321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016163989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'15","reported_seq":297,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.228837+0000","last_change":"2024-09-10T18:32:45.581485+0000","last_active":"2024-09-10T18:33:10.228837+0000","last_peered":"2024-09-10T18:33:10.228837+0000","last_clean":"2024-09-10T18:33:10.228837+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:33:10.228837+0000","last_undegraded":"2024-09-10T18:33:10.228837+0000","last_fullsized":"2024-09-10T18:33:10.228837+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:32:45.581390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:45.581390+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:01.343758+0000","scrub_duration":0.003880966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":271,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.219572+0000","last_change":"2024-09-10T18:31:58.589684+0000","last_active":"2024-09-10T18:33:10.219572+0000","last_peered":"2024-09-10T18:33:10.219572+0000","last_clean":"2024-09-10T18:33:10.219572+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:33:10.219572+0000","last_undegraded":"2024-09-10T18:33:10.219572+0000","last_fullsized":"2024-09-10T18:33:10.219572+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_clean_scrub_stamp":"2024-09-10T18:31:58.589631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:24.982578+0000","scrub_duration":0.0011763940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":269,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.219240+0000","last_change":"2024-09-10T18:33:00.235163+0000","last_active":"2024-09-10T18:33:10.219240+0000","last_peered":"2024-09-10T18:33:10.219240+0000","last_clean":"2024-09-10T18:33:10.219240+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:33:10.219240+0000","last_undegraded":"2024-09-10T18:33:10.219240+0000","last_fullsized":"2024-09-10T18:33:10.219240+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:00.234931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:00.234931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.372488+0000","scrub_duration":0.0023527999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":275,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.216598+0000","last_change":"2024-09-10T18:32:00.582011+0000","last_active":"2024-09-10T18:33:10.216598+0000","last_peered":"2024-09-10T18:33:10.216598+0000","last_clean":"2024-09-10T18:33:10.216598+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:33:10.216598+0000","last_undegraded":"2024-09-10T18:33:10.216598+0000","last_fullsized":"2024-09-10T18:33:10.216598+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.581959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:17.464205+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.581959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:29.570654+0000","scrub_duration":0.0012934120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":309,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.228584+0000","last_change":"2024-09-10T18:32:48.589267+0000","last_active":"2024-09-10T18:33:10.228584+0000","last_peered":"2024-09-10T18:33:10.228584+0000","last_clean":"2024-09-10T18:33:10.228584+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:33:10.228584+0000","last_undegraded":"2024-09-10T18:33:10.228584+0000","last_fullsized":"2024-09-10T18:33:10.228584+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'9","last_scrub_stamp":"2024-09-10T18:32:48.589167+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:32:48.589167+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:02.172908+0000","scrub_duration":0.0073279920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":270,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.222985+0000","last_change":"2024-09-10T18:32:01.525753+0000","last_active":"2024-09-10T18:33:10.222985+0000","last_peered":"2024-09-10T18:33:10.222985+0000","last_clean":"2024-09-10T18:33:10.222985+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:33:10.222985+0000","last_undegraded":"2024-09-10T18:33:10.222985+0000","last_fullsized":"2024-09-10T18:33:10.222985+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:01.525701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:01.525701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:24.525312+0000","scrub_duration":0.001223902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":262,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.219282+0000","last_change":"2024-09-10T18:31:57.628816+0000","last_active":"2024-09-10T18:33:10.219282+0000","last_peered":"2024-09-10T18:33:10.219282+0000","last_clean":"2024-09-10T18:33:10.219282+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:33:10.219282+0000","last_undegraded":"2024-09-10T18:33:10.219282+0000","last_fullsized":"2024-09-10T18:33:10.219282+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:57.628758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.628758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013180119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":268,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.229206+0000","last_change":"2024-09-10T18:31:57.658865+0000","last_active":"2024-09-10T18:33:10.229206+0000","last_peered":"2024-09-10T18:33:10.229206+0000","last_clean":"2024-09-10T18:33:10.229206+0000","last_became_active":"2024-09-10T18:26:27.650774+0000","last_became_peered":"2024-09-10T18:26:27.650774+0000","last_unstale":"2024-09-10T18:33:10.229206+0000","last_undegraded":"2024-09-10T18:33:10.229206+0000","last_fullsized":"2024-09-10T18:33:10.229206+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:57.658808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.658808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:12.353198+0000","scrub_duration":0.001743421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":255,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.218494+0000","last_change":"2024-09-10T18:31:53.332936+0000","last_active":"2024-09-10T18:33:10.218494+0000","last_peered":"2024-09-10T18:33:10.218494+0000","last_clean":"2024-09-10T18:33:10.218494+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:33:10.218494+0000","last_undegraded":"2024-09-10T18:33:10.218494+0000","last_fullsized":"2024-09-10T18:33:10.218494+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:53.332881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:53.332881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014944929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"240'27","reported_seq":335,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.226395+0000","last_change":"2024-09-10T18:33:04.586182+0000","last_active":"2024-09-10T18:33:10.226395+0000","last_peered":"2024-09-10T18:33:10.226395+0000","last_clean":"2024-09-10T18:33:10.226395+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:33:10.226395+0000","last_undegraded":"2024-09-10T18:33:10.226395+0000","last_fullsized":"2024-09-10T18:33:10.226395+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:33:04.586063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:04.586063+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:25.050823+0000","scrub_duration":0.007748642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":280,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.223432+0000","last_change":"2024-09-10T18:32:58.252469+0000","last_active":"2024-09-10T18:33:10.223432+0000","last_peered":"2024-09-10T18:33:10.223432+0000","last_clean":"2024-09-10T18:33:10.223432+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:33:10.223432+0000","last_undegraded":"2024-09-10T18:33:10.223432+0000","last_fullsized":"2024-09-10T18:33:10.223432+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:58.252419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.252419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.332332+0000","scrub_duration":0.0049362080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":267,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.227296+0000","last_change":"2024-09-10T18:31:53.754090+0000","last_active":"2024-09-10T18:33:10.227296+0000","last_peered":"2024-09-10T18:33:10.227296+0000","last_clean":"2024-09-10T18:33:10.227296+0000","last_became_active":"2024-09-10T18:26:27.640035+0000","last_became_peered":"2024-09-10T18:26:27.640035+0000","last_unstale":"2024-09-10T18:33:10.227296+0000","last_undegraded":"2024-09-10T18:33:10.227296+0000","last_fullsized":"2024-09-10T18:33:10.227296+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:53.754032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:53.754032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:17.934661+0000","scrub_duration":0.0016988699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":274,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.228962+0000","last_change":"2024-09-10T18:32:59.567476+0000","last_active":"2024-09-10T18:33:10.228962+0000","last_peered":"2024-09-10T18:33:10.228962+0000","last_clean":"2024-09-10T18:33:10.228962+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:33:10.228962+0000","last_undegraded":"2024-09-10T18:33:10.228962+0000","last_fullsized":"2024-09-10T18:33:10.228962+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:59.567389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:32:59.567389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:06.727554+0000","scrub_duration":0.0073893559999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'23","reported_seq":301,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.221986+0000","last_change":"2024-09-10T18:32:50.341138+0000","last_active":"2024-09-10T18:33:10.221986+0000","last_peered":"2024-09-10T18:33:10.221986+0000","last_clean":"2024-09-10T18:33:10.221986+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:33:10.221986+0000","last_undegraded":"2024-09-10T18:33:10.221986+0000","last_fullsized":"2024-09-10T18:33:10.221986+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'23","last_scrub_stamp":"2024-09-10T18:32:50.341044+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:32:50.341044+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:02.598697+0000","scrub_duration":0.012050170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":270,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.229994+0000","last_change":"2024-09-10T18:33:07.697770+0000","last_active":"2024-09-10T18:33:10.229994+0000","last_peered":"2024-09-10T18:33:10.229994+0000","last_clean":"2024-09-10T18:33:10.229994+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:33:10.229994+0000","last_undegraded":"2024-09-10T18:33:10.229994+0000","last_fullsized":"2024-09-10T18:33:10.229994+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:07.697667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:33:07.697667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:12.574607+0000","scrub_duration":0.044890945000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":270,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.222189+0000","last_change":"2024-09-10T18:32:57.769495+0000","last_active":"2024-09-10T18:33:10.222189+0000","last_peered":"2024-09-10T18:33:10.222189+0000","last_clean":"2024-09-10T18:33:10.222189+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:33:10.222189+0000","last_undegraded":"2024-09-10T18:33:10.222189+0000","last_fullsized":"2024-09-10T18:33:10.222189+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:32:57.769408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:57.769408+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:20.116093+0000","scrub_duration":0.003757149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":284,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.222823+0000","last_change":"2024-09-10T18:32:51.182762+0000","last_active":"2024-09-10T18:33:10.222823+0000","last_peered":"2024-09-10T18:33:10.222823+0000","last_clean":"2024-09-10T18:33:10.222823+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:33:10.222823+0000","last_undegraded":"2024-09-10T18:33:10.222823+0000","last_fullsized":"2024-09-10T18:33:10.222823+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:51.182684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:32:51.182684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:12.337885+0000","scrub_duration":0.003614637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'25","reported_seq":308,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:15.366365+0000","last_change":"2024-09-10T18:33:15.366317+0000","last_active":"2024-09-10T18:33:15.366365+0000","last_peered":"2024-09-10T18:33:15.366365+0000","last_clean":"2024-09-10T18:33:15.366365+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:33:15.366365+0000","last_undegraded":"2024-09-10T18:33:15.366365+0000","last_fullsized":"2024-09-10T18:33:15.366365+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:33:15.366107+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.366107+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:23.163279+0000","scrub_duration":0.014527711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"217'11","reported_seq":332,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.773332+0000","last_change":"2024-09-10T18:33:03.119057+0000","last_active":"2024-09-10T18:33:10.773332+0000","last_peered":"2024-09-10T18:33:10.773332+0000","last_clean":"2024-09-10T18:33:10.773332+0000","last_became_active":"2024-09-10T18:26:27.652489+0000","last_became_peered":"2024-09-10T18:26:27.652489+0000","last_unstale":"2024-09-10T18:33:10.773332+0000","last_undegraded":"2024-09-10T18:33:10.773332+0000","last_fullsized":"2024-09-10T18:33:10.773332+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11","last_scrub_stamp":"2024-09-10T18:33:03.118959+0000","last_deep_scrub":"217'11","last_deep_scrub_stamp":"2024-09-10T18:30:03.161985+0000","last_clean_scrub_stamp":"2024-09-10T18:33:03.118959+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.565288+0000","scrub_duration":0.002038715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"265'6463","reported_seq":6490,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:15.617455+0000","last_change":"2024-09-10T18:32:03.385708+0000","last_active":"2024-09-10T18:33:15.617455+0000","last_peered":"2024-09-10T18:33:15.617455+0000","last_clean":"2024-09-10T18:33:15.617455+0000","last_became_active":"2024-09-10T18:32:03.385428+0000","last_became_peered":"2024-09-10T18:32:03.385428+0000","last_unstale":"2024-09-10T18:33:15.617455+0000","last_undegraded":"2024-09-10T18:33:15.617455+0000","last_fullsized":"2024-09-10T18:33:15.617455+0000","mapping_epoch":246,"log_start":"251'2000","ondisk_log_start":"251'2000","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4463,"log_dups_size":0,"ondisk_log_size":4463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423559168,"num_objects":6463,"num_object_clones":0,"num_object_copies":19389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6463,"num_write_kb":413632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":181,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.218190+0000","last_change":"2024-09-10T18:31:59.600055+0000","last_active":"2024-09-10T18:33:10.218190+0000","last_peered":"2024-09-10T18:33:10.218190+0000","last_clean":"2024-09-10T18:33:10.218190+0000","last_became_active":"2024-09-10T18:26:28.635249+0000","last_became_peered":"2024-09-10T18:26:28.635249+0000","last_unstale":"2024-09-10T18:33:10.218190+0000","last_undegraded":"2024-09-10T18:33:10.218190+0000","last_fullsized":"2024-09-10T18:33:10.218190+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:59.599999+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:59.599999+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001350027,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"240'34","reported_seq":335,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.227289+0000","last_change":"2024-09-10T18:32:56.285685+0000","last_active":"2024-09-10T18:33:10.227289+0000","last_peered":"2024-09-10T18:33:10.227289+0000","last_clean":"2024-09-10T18:33:10.227289+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:33:10.227289+0000","last_undegraded":"2024-09-10T18:33:10.227289+0000","last_fullsized":"2024-09-10T18:33:10.227289+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:32:56.285589+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:32:56.285589+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:15.128863+0000","scrub_duration":0.0081173400000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":266,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.230219+0000","last_change":"2024-09-10T18:31:59.772883+0000","last_active":"2024-09-10T18:33:10.230219+0000","last_peered":"2024-09-10T18:33:10.230219+0000","last_clean":"2024-09-10T18:33:10.230219+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:33:10.230219+0000","last_undegraded":"2024-09-10T18:33:10.230219+0000","last_fullsized":"2024-09-10T18:33:10.230219+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:59.772816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:59.772816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:16.535228+0000","scrub_duration":0.0015225810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":411,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.229283+0000","last_change":"2024-09-10T18:31:55.634620+0000","last_active":"2024-09-10T18:33:10.229283+0000","last_peered":"2024-09-10T18:33:10.229283+0000","last_clean":"2024-09-10T18:33:10.229283+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:33:10.229283+0000","last_undegraded":"2024-09-10T18:33:10.229283+0000","last_fullsized":"2024-09-10T18:33:10.229283+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:31:55.634564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:55.634564+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:17.377608+0000","scrub_duration":0.001562346,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":269,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.220120+0000","last_change":"2024-09-10T18:32:50.961491+0000","last_active":"2024-09-10T18:33:10.220120+0000","last_peered":"2024-09-10T18:33:10.220120+0000","last_clean":"2024-09-10T18:33:10.220120+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:33:10.220120+0000","last_undegraded":"2024-09-10T18:33:10.220120+0000","last_fullsized":"2024-09-10T18:33:10.220120+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:50.961251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:32:50.961251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:51.179028+0000","scrub_duration":0.0063477669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"265'6260","reported_seq":6287,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:14.867857+0000","last_change":"2024-09-10T18:32:03.379554+0000","last_active":"2024-09-10T18:33:14.867857+0000","last_peered":"2024-09-10T18:33:14.867857+0000","last_clean":"2024-09-10T18:33:14.867857+0000","last_became_active":"2024-09-10T18:32:03.379257+0000","last_became_peered":"2024-09-10T18:32:03.379257+0000","last_unstale":"2024-09-10T18:33:14.867857+0000","last_undegraded":"2024-09-10T18:33:14.867857+0000","last_fullsized":"2024-09-10T18:33:14.867857+0000","mapping_epoch":246,"log_start":"251'1700","ondisk_log_start":"251'1700","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4560,"log_dups_size":0,"ondisk_log_size":4560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410255360,"num_objects":6260,"num_object_clones":0,"num_object_copies":18780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":180,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.229236+0000","last_change":"2024-09-10T18:32:58.561489+0000","last_active":"2024-09-10T18:33:10.229236+0000","last_peered":"2024-09-10T18:33:10.229236+0000","last_clean":"2024-09-10T18:33:10.229236+0000","last_became_active":"2024-09-10T18:26:28.632744+0000","last_became_peered":"2024-09-10T18:26:28.632744+0000","last_unstale":"2024-09-10T18:33:10.229236+0000","last_undegraded":"2024-09-10T18:33:10.229236+0000","last_fullsized":"2024-09-10T18:33:10.229236+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:58.561425+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.561425+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:00.246746+0000","scrub_duration":0.002234073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":314,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.220165+0000","last_change":"2024-09-10T18:32:59.382953+0000","last_active":"2024-09-10T18:33:10.220165+0000","last_peered":"2024-09-10T18:33:10.220165+0000","last_clean":"2024-09-10T18:33:10.220165+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:33:10.220165+0000","last_undegraded":"2024-09-10T18:33:10.220165+0000","last_fullsized":"2024-09-10T18:33:10.220165+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-09-10T18:32:59.382840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:59.382840+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:18.205789+0000","scrub_duration":0.0072140750000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":266,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.229049+0000","last_change":"2024-09-10T18:32:49.346233+0000","last_active":"2024-09-10T18:33:10.229049+0000","last_peered":"2024-09-10T18:33:10.229049+0000","last_clean":"2024-09-10T18:33:10.229049+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:33:10.229049+0000","last_undegraded":"2024-09-10T18:33:10.229049+0000","last_fullsized":"2024-09-10T18:33:10.229049+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:49.342269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.140922+0000","last_clean_scrub_stamp":"2024-09-10T18:32:49.342269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:05.179940+0000","scrub_duration":0.006539385,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":270,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.228369+0000","last_change":"2024-09-10T18:32:52.582545+0000","last_active":"2024-09-10T18:33:10.228369+0000","last_peered":"2024-09-10T18:33:10.228369+0000","last_clean":"2024-09-10T18:33:10.228369+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:33:10.228369+0000","last_undegraded":"2024-09-10T18:33:10.228369+0000","last_fullsized":"2024-09-10T18:33:10.228369+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:52.582464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:32:52.582464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:16.167011+0000","scrub_duration":0.0024277399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":258,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.228770+0000","last_change":"2024-09-10T18:32:44.600295+0000","last_active":"2024-09-10T18:33:10.228770+0000","last_peered":"2024-09-10T18:33:10.228770+0000","last_clean":"2024-09-10T18:33:10.228770+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:33:10.228770+0000","last_undegraded":"2024-09-10T18:33:10.228770+0000","last_fullsized":"2024-09-10T18:33:10.228770+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:44.600200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:44.600200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:53.871071+0000","scrub_duration":0.0073958740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"265'6543","reported_seq":6570,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:15.609555+0000","last_change":"2024-09-10T18:32:03.554760+0000","last_active":"2024-09-10T18:33:15.609555+0000","last_peered":"2024-09-10T18:33:15.609555+0000","last_clean":"2024-09-10T18:33:15.609555+0000","last_became_active":"2024-09-10T18:32:03.554400+0000","last_became_peered":"2024-09-10T18:32:03.554400+0000","last_unstale":"2024-09-10T18:33:15.609555+0000","last_undegraded":"2024-09-10T18:33:15.609555+0000","last_fullsized":"2024-09-10T18:33:15.609555+0000","mapping_epoch":246,"log_start":"251'2100","ondisk_log_start":"251'2100","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4443,"log_dups_size":0,"ondisk_log_size":4443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:30.476745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428802048,"num_objects":6543,"num_object_clones":0,"num_object_copies":19629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6543,"num_write_kb":418752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":171,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.219373+0000","last_change":"2024-09-10T18:33:01.370765+0000","last_active":"2024-09-10T18:33:10.219373+0000","last_peered":"2024-09-10T18:33:10.219373+0000","last_clean":"2024-09-10T18:33:10.219373+0000","last_became_active":"2024-09-10T18:26:18.091896+0000","last_became_peered":"2024-09-10T18:26:18.091896+0000","last_unstale":"2024-09-10T18:33:10.219373+0000","last_undegraded":"2024-09-10T18:33:10.219373+0000","last_fullsized":"2024-09-10T18:33:10.219373+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.370561+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.370561+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:12.358728+0000","scrub_duration":0.0029251390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"240'28","reported_seq":311,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.228560+0000","last_change":"2024-09-10T18:31:46.592515+0000","last_active":"2024-09-10T18:33:10.228560+0000","last_peered":"2024-09-10T18:33:10.228560+0000","last_clean":"2024-09-10T18:33:10.228560+0000","last_became_active":"2024-09-10T18:26:27.651364+0000","last_became_peered":"2024-09-10T18:26:27.651364+0000","last_unstale":"2024-09-10T18:33:10.228560+0000","last_undegraded":"2024-09-10T18:33:10.228560+0000","last_fullsized":"2024-09-10T18:33:10.228560+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'28","last_scrub_stamp":"2024-09-10T18:31:46.592456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:46.592456+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019972359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":265,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.218631+0000","last_change":"2024-09-10T18:32:54.291391+0000","last_active":"2024-09-10T18:33:10.218631+0000","last_peered":"2024-09-10T18:33:10.218631+0000","last_clean":"2024-09-10T18:33:10.218631+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:33:10.218631+0000","last_undegraded":"2024-09-10T18:33:10.218631+0000","last_fullsized":"2024-09-10T18:33:10.218631+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:54.291164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:32:54.291164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:04.937902+0000","scrub_duration":0.002933667,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":412,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.770956+0000","last_change":"2024-09-10T18:33:07.367306+0000","last_active":"2024-09-10T18:33:10.770956+0000","last_peered":"2024-09-10T18:33:10.770956+0000","last_clean":"2024-09-10T18:33:10.770956+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:33:10.770956+0000","last_undegraded":"2024-09-10T18:33:10.770956+0000","last_fullsized":"2024-09-10T18:33:10.770956+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:33:07.367248+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:33:07.367248+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:25.811396+0000","scrub_duration":0.0044945339999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":278,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:11.858913+0000","last_change":"2024-09-10T18:33:11.857892+0000","last_active":"2024-09-10T18:33:11.858913+0000","last_peered":"2024-09-10T18:33:11.858913+0000","last_clean":"2024-09-10T18:33:11.858913+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:33:11.858913+0000","last_undegraded":"2024-09-10T18:33:11.858913+0000","last_fullsized":"2024-09-10T18:33:11.858913+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:11.857685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.857685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:36.504074+0000","scrub_duration":0.0022995870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"265'6427","reported_seq":6454,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:14.912952+0000","last_change":"2024-09-10T18:32:03.384690+0000","last_active":"2024-09-10T18:33:14.912952+0000","last_peered":"2024-09-10T18:33:14.912952+0000","last_clean":"2024-09-10T18:33:14.912952+0000","last_became_active":"2024-09-10T18:32:03.384288+0000","last_became_peered":"2024-09-10T18:32:03.384288+0000","last_unstale":"2024-09-10T18:33:14.912952+0000","last_undegraded":"2024-09-10T18:33:14.912952+0000","last_fullsized":"2024-09-10T18:33:14.912952+0000","mapping_epoch":246,"log_start":"251'1500","ondisk_log_start":"251'1500","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4927,"log_dups_size":0,"ondisk_log_size":4927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421199872,"num_objects":6427,"num_object_clones":0,"num_object_copies":19281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6427,"num_write_kb":411328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":230,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.226859+0000","last_change":"2024-09-10T18:33:01.151484+0000","last_active":"2024-09-10T18:33:10.226859+0000","last_peered":"2024-09-10T18:33:10.226859+0000","last_clean":"2024-09-10T18:33:10.226859+0000","last_became_active":"2024-09-10T18:26:28.632849+0000","last_became_peered":"2024-09-10T18:26:28.632849+0000","last_unstale":"2024-09-10T18:33:10.226859+0000","last_undegraded":"2024-09-10T18:33:10.226859+0000","last_fullsized":"2024-09-10T18:33:10.226859+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.151386+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.151386+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:06.156495+0000","scrub_duration":0.0031748879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":317,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.224328+0000","last_change":"2024-09-10T18:32:55.279937+0000","last_active":"2024-09-10T18:33:10.224328+0000","last_peered":"2024-09-10T18:33:10.224328+0000","last_clean":"2024-09-10T18:33:10.224328+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:33:10.224328+0000","last_undegraded":"2024-09-10T18:33:10.224328+0000","last_fullsized":"2024-09-10T18:33:10.224328+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:32:55.279209+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:32:55.279209+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:56.244187+0000","scrub_duration":0.0091125259999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":265,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.220646+0000","last_change":"2024-09-10T18:32:45.965208+0000","last_active":"2024-09-10T18:33:10.220646+0000","last_peered":"2024-09-10T18:33:10.220646+0000","last_clean":"2024-09-10T18:33:10.220646+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:33:10.220646+0000","last_undegraded":"2024-09-10T18:33:10.220646+0000","last_fullsized":"2024-09-10T18:33:10.220646+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:45.965129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:32:45.965129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:47.511331+0000","scrub_duration":0.0045521829999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"263'11","reported_seq":420,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.766491+0000","last_change":"2024-09-10T18:33:01.283747+0000","last_active":"2024-09-10T18:33:10.766491+0000","last_peered":"2024-09-10T18:33:10.766491+0000","last_clean":"2024-09-10T18:33:10.766491+0000","last_became_active":"2024-09-10T18:26:27.639032+0000","last_became_peered":"2024-09-10T18:26:27.639032+0000","last_unstale":"2024-09-10T18:33:10.766491+0000","last_undegraded":"2024-09-10T18:33:10.766491+0000","last_fullsized":"2024-09-10T18:33:10.766491+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'11","last_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_deep_scrub":"263'11","last_deep_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.283648+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:27.247085+0000","scrub_duration":0.0081388439999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":259,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.219452+0000","last_change":"2024-09-10T18:31:43.338883+0000","last_active":"2024-09-10T18:33:10.219452+0000","last_peered":"2024-09-10T18:33:10.219452+0000","last_clean":"2024-09-10T18:33:10.219452+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:33:10.219452+0000","last_undegraded":"2024-09-10T18:33:10.219452+0000","last_fullsized":"2024-09-10T18:33:10.219452+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:43.338816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:43.338816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011905869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"265'6454","reported_seq":6481,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:14.865175+0000","last_change":"2024-09-10T18:32:03.555680+0000","last_active":"2024-09-10T18:33:14.865175+0000","last_peered":"2024-09-10T18:33:14.865175+0000","last_clean":"2024-09-10T18:33:14.865175+0000","last_became_active":"2024-09-10T18:32:03.554693+0000","last_became_peered":"2024-09-10T18:32:03.554693+0000","last_unstale":"2024-09-10T18:33:14.865175+0000","last_undegraded":"2024-09-10T18:33:14.865175+0000","last_fullsized":"2024-09-10T18:33:14.865175+0000","mapping_epoch":246,"log_start":"251'1900","ondisk_log_start":"251'1900","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4554,"log_dups_size":0,"ondisk_log_size":4554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422969344,"num_objects":6454,"num_object_clones":0,"num_object_copies":19362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6454,"num_write_kb":413056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":230,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.225790+0000","last_change":"2024-09-10T18:32:48.006787+0000","last_active":"2024-09-10T18:33:10.225790+0000","last_peered":"2024-09-10T18:33:10.225790+0000","last_clean":"2024-09-10T18:33:10.225790+0000","last_became_active":"2024-09-10T18:26:28.635542+0000","last_became_peered":"2024-09-10T18:26:28.635542+0000","last_unstale":"2024-09-10T18:33:10.225790+0000","last_undegraded":"2024-09-10T18:33:10.225790+0000","last_fullsized":"2024-09-10T18:33:10.225790+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:48.006695+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:48.006695+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:10.998203+0000","scrub_duration":0.0051030709999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"240'29","reported_seq":324,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:11.310516+0000","last_change":"2024-09-10T18:33:11.310163+0000","last_active":"2024-09-10T18:33:11.310516+0000","last_peered":"2024-09-10T18:33:11.310516+0000","last_clean":"2024-09-10T18:33:11.310516+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:33:11.310516+0000","last_undegraded":"2024-09-10T18:33:11.310516+0000","last_fullsized":"2024-09-10T18:33:11.310516+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'29","last_scrub_stamp":"2024-09-10T18:33:11.310074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.310074+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.669570+0000","scrub_duration":0.0026943760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":264,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.223324+0000","last_change":"2024-09-10T18:31:52.607842+0000","last_active":"2024-09-10T18:33:10.223324+0000","last_peered":"2024-09-10T18:33:10.223324+0000","last_clean":"2024-09-10T18:33:10.223324+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:33:10.223324+0000","last_undegraded":"2024-09-10T18:33:10.223324+0000","last_fullsized":"2024-09-10T18:33:10.223324+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:52.607787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:52.607787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:20.479043+0000","scrub_duration":0.0014830799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":220,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.225968+0000","last_change":"2024-09-10T18:33:09.183070+0000","last_active":"2024-09-10T18:33:10.225968+0000","last_peered":"2024-09-10T18:33:10.225968+0000","last_clean":"2024-09-10T18:33:10.225968+0000","last_became_active":"2024-09-10T18:26:27.642063+0000","last_became_peered":"2024-09-10T18:26:27.642063+0000","last_unstale":"2024-09-10T18:33:10.225968+0000","last_undegraded":"2024-09-10T18:33:10.225968+0000","last_fullsized":"2024-09-10T18:33:10.225968+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:09.182982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:33:09.182982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:14.241099+0000","scrub_duration":0.0023969899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":275,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.222646+0000","last_change":"2024-09-10T18:33:10.137145+0000","last_active":"2024-09-10T18:33:10.222646+0000","last_peered":"2024-09-10T18:33:10.222646+0000","last_clean":"2024-09-10T18:33:10.222646+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:33:10.222646+0000","last_undegraded":"2024-09-10T18:33:10.222646+0000","last_fullsized":"2024-09-10T18:33:10.222646+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:10.137048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:33:10.137048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:31.309662+0000","scrub_duration":0.0040149249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"265'6773","reported_seq":6798,"reported_epoch":265,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-10T18:33:15.870805+0000","last_change":"2024-09-10T18:33:14.280780+0000","last_active":"2024-09-10T18:33:15.870805+0000","last_peered":"2024-09-10T18:33:15.870805+0000","last_clean":"2024-09-10T18:33:15.870805+0000","last_became_active":"2024-09-10T18:32:03.381872+0000","last_became_peered":"2024-09-10T18:32:03.381872+0000","last_unstale":"2024-09-10T18:33:15.870805+0000","last_undegraded":"2024-09-10T18:33:15.870805+0000","last_fullsized":"2024-09-10T18:33:15.870805+0000","mapping_epoch":246,"log_start":"250'1100","ondisk_log_start":"250'1100","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":349,"log_size":5673,"log_dups_size":0,"ondisk_log_size":5673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428933120,"num_objects":6545,"num_object_clones":0,"num_object_copies":19635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6545,"num_write_kb":418880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":166,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.219012+0000","last_change":"2024-09-10T18:31:56.629743+0000","last_active":"2024-09-10T18:33:10.219012+0000","last_peered":"2024-09-10T18:33:10.219012+0000","last_clean":"2024-09-10T18:33:10.219012+0000","last_became_active":"2024-09-10T18:26:18.094747+0000","last_became_peered":"2024-09-10T18:26:18.094747+0000","last_unstale":"2024-09-10T18:33:10.219012+0000","last_undegraded":"2024-09-10T18:33:10.219012+0000","last_fullsized":"2024-09-10T18:33:10.219012+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:56.629683+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:56.629683+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:22.298804+0000","scrub_duration":0.001774133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"240'41","reported_seq":398,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.772582+0000","last_change":"2024-09-10T18:32:55.941738+0000","last_active":"2024-09-10T18:33:10.772582+0000","last_peered":"2024-09-10T18:33:10.772582+0000","last_clean":"2024-09-10T18:33:10.772582+0000","last_became_active":"2024-09-10T18:23:10.067049+0000","last_became_peered":"2024-09-10T18:23:10.067049+0000","last_unstale":"2024-09-10T18:33:10.772582+0000","last_undegraded":"2024-09-10T18:33:10.772582+0000","last_fullsized":"2024-09-10T18:33:10.772582+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'41","last_scrub_stamp":"2024-09-10T18:32:55.941653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:55.941653+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:02.668969+0000","scrub_duration":0.011585564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":271,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.228864+0000","last_change":"2024-09-10T18:31:49.684883+0000","last_active":"2024-09-10T18:33:10.228864+0000","last_peered":"2024-09-10T18:33:10.228864+0000","last_clean":"2024-09-10T18:33:10.228864+0000","last_became_active":"2024-09-10T18:26:27.648690+0000","last_became_peered":"2024-09-10T18:26:27.648690+0000","last_unstale":"2024-09-10T18:33:10.228864+0000","last_undegraded":"2024-09-10T18:33:10.228864+0000","last_fullsized":"2024-09-10T18:33:10.228864+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:49.684824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:49.684824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:19.116090+0000","scrub_duration":0.0012508370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":290,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.218549+0000","last_change":"2024-09-10T18:33:04.936495+0000","last_active":"2024-09-10T18:33:10.218549+0000","last_peered":"2024-09-10T18:33:10.218549+0000","last_clean":"2024-09-10T18:33:10.218549+0000","last_became_active":"2024-09-10T18:26:27.647576+0000","last_became_peered":"2024-09-10T18:26:27.647576+0000","last_unstale":"2024-09-10T18:33:10.218549+0000","last_undegraded":"2024-09-10T18:33:10.218549+0000","last_fullsized":"2024-09-10T18:33:10.218549+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:33:04.936407+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:33:04.936407+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:31.757007+0000","scrub_duration":0.0096149180000000001,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":261,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.218219+0000","last_change":"2024-09-10T18:32:39.119345+0000","last_active":"2024-09-10T18:33:10.218219+0000","last_peered":"2024-09-10T18:33:10.218219+0000","last_clean":"2024-09-10T18:33:10.218219+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:33:10.218219+0000","last_undegraded":"2024-09-10T18:33:10.218219+0000","last_fullsized":"2024-09-10T18:33:10.218219+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:39.119228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.119228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:55.707959+0000","scrub_duration":0.011330534999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":250,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.229712+0000","last_change":"2024-09-10T18:31:04.714213+0000","last_active":"2024-09-10T18:33:10.229712+0000","last_peered":"2024-09-10T18:33:10.229712+0000","last_clean":"2024-09-10T18:33:10.229712+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:33:10.229712+0000","last_undegraded":"2024-09-10T18:33:10.229712+0000","last_fullsized":"2024-09-10T18:33:10.229712+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:04.714112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:04.714112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0085368679999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"265'6516","reported_seq":6543,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:14.934210+0000","last_change":"2024-09-10T18:32:03.380691+0000","last_active":"2024-09-10T18:33:14.934210+0000","last_peered":"2024-09-10T18:33:14.934210+0000","last_clean":"2024-09-10T18:33:14.934210+0000","last_became_active":"2024-09-10T18:32:03.380391+0000","last_became_peered":"2024-09-10T18:32:03.380391+0000","last_unstale":"2024-09-10T18:33:14.934210+0000","last_undegraded":"2024-09-10T18:33:14.934210+0000","last_fullsized":"2024-09-10T18:33:14.934210+0000","mapping_epoch":246,"log_start":"251'1500","ondisk_log_start":"251'1500","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427032576,"num_objects":6516,"num_object_clones":0,"num_object_copies":19548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6516,"num_write_kb":417024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":312,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.222848+0000","last_change":"2024-09-10T18:31:57.474380+0000","last_active":"2024-09-10T18:33:10.222848+0000","last_peered":"2024-09-10T18:33:10.222848+0000","last_clean":"2024-09-10T18:33:10.222848+0000","last_became_active":"2024-09-10T18:26:18.095568+0000","last_became_peered":"2024-09-10T18:26:18.095568+0000","last_unstale":"2024-09-10T18:33:10.222848+0000","last_undegraded":"2024-09-10T18:33:10.222848+0000","last_fullsized":"2024-09-10T18:33:10.222848+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:57.474295+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.474295+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:12.518679+0000","scrub_duration":0.0015998690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"240'17","reported_seq":305,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.227391+0000","last_change":"2024-09-10T18:32:44.186718+0000","last_active":"2024-09-10T18:33:10.227391+0000","last_peered":"2024-09-10T18:33:10.227391+0000","last_clean":"2024-09-10T18:33:10.227391+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:33:10.227391+0000","last_undegraded":"2024-09-10T18:33:10.227391+0000","last_fullsized":"2024-09-10T18:33:10.227391+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'17","last_scrub_stamp":"2024-09-10T18:32:44.186634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:44.186634+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:53.244468+0000","scrub_duration":0.0043121310000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":291,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:15.142297+0000","last_change":"2024-09-10T18:33:15.142062+0000","last_active":"2024-09-10T18:33:15.142297+0000","last_peered":"2024-09-10T18:33:15.142297+0000","last_clean":"2024-09-10T18:33:15.142297+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:33:15.142297+0000","last_undegraded":"2024-09-10T18:33:15.142297+0000","last_fullsized":"2024-09-10T18:33:15.142297+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:15.141982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:15.141982+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.141982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:45.048602+0000","scrub_duration":0.0028956749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":293,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.226759+0000","last_change":"2024-09-10T18:31:54.779251+0000","last_active":"2024-09-10T18:33:10.226759+0000","last_peered":"2024-09-10T18:33:10.226759+0000","last_clean":"2024-09-10T18:33:10.226759+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:33:10.226759+0000","last_undegraded":"2024-09-10T18:33:10.226759+0000","last_fullsized":"2024-09-10T18:33:10.226759+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:31:54.779189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:54.779189+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:22.671479+0000","scrub_duration":0.0018921529999999999,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":280,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.226201+0000","last_change":"2024-09-10T18:32:53.586654+0000","last_active":"2024-09-10T18:33:10.226201+0000","last_peered":"2024-09-10T18:33:10.226201+0000","last_clean":"2024-09-10T18:33:10.226201+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:33:10.226201+0000","last_undegraded":"2024-09-10T18:33:10.226201+0000","last_fullsized":"2024-09-10T18:33:10.226201+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:53.586216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:32:53.586216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:22.249594+0000","scrub_duration":0.0053348709999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":268,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.222270+0000","last_change":"2024-09-10T18:32:00.501396+0000","last_active":"2024-09-10T18:33:10.222270+0000","last_peered":"2024-09-10T18:33:10.222270+0000","last_clean":"2024-09-10T18:33:10.222270+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:33:10.222270+0000","last_undegraded":"2024-09-10T18:33:10.222270+0000","last_fullsized":"2024-09-10T18:33:10.222270+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.501344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.501344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001367442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"265'6592","reported_seq":6619,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:15.904669+0000","last_change":"2024-09-10T18:32:03.555231+0000","last_active":"2024-09-10T18:33:15.904669+0000","last_peered":"2024-09-10T18:33:15.904669+0000","last_clean":"2024-09-10T18:33:15.904669+0000","last_became_active":"2024-09-10T18:32:03.554958+0000","last_became_peered":"2024-09-10T18:32:03.554958+0000","last_unstale":"2024-09-10T18:33:15.904669+0000","last_undegraded":"2024-09-10T18:33:15.904669+0000","last_fullsized":"2024-09-10T18:33:15.904669+0000","mapping_epoch":246,"log_start":"250'900","ondisk_log_start":"250'900","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":5692,"log_dups_size":0,"ondisk_log_size":5692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432013312,"num_objects":6592,"num_object_clones":0,"num_object_copies":19776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6592,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":181,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.224794+0000","last_change":"2024-09-10T18:31:48.466710+0000","last_active":"2024-09-10T18:33:10.224794+0000","last_peered":"2024-09-10T18:33:10.224794+0000","last_clean":"2024-09-10T18:33:10.224794+0000","last_became_active":"2024-09-10T18:26:28.641791+0000","last_became_peered":"2024-09-10T18:26:28.641791+0000","last_unstale":"2024-09-10T18:33:10.224794+0000","last_undegraded":"2024-09-10T18:33:10.224794+0000","last_fullsized":"2024-09-10T18:33:10.224794+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:48.466654+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.466654+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:15.898353+0000","scrub_duration":0.001538871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":299,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.218493+0000","last_change":"2024-09-10T18:32:53.270762+0000","last_active":"2024-09-10T18:33:10.218493+0000","last_peered":"2024-09-10T18:33:10.218493+0000","last_clean":"2024-09-10T18:33:10.218493+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:33:10.218493+0000","last_undegraded":"2024-09-10T18:33:10.218493+0000","last_fullsized":"2024-09-10T18:33:10.218493+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-09-10T18:32:53.270667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:53.270667+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:14.044059+0000","scrub_duration":0.01096307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":275,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.218282+0000","last_change":"2024-09-10T18:32:59.245911+0000","last_active":"2024-09-10T18:33:10.218282+0000","last_peered":"2024-09-10T18:33:10.218282+0000","last_clean":"2024-09-10T18:33:10.218282+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:33:10.218282+0000","last_undegraded":"2024-09-10T18:33:10.218282+0000","last_fullsized":"2024-09-10T18:33:10.218282+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:59.245802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:32:59.245802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:13.023525+0000","scrub_duration":0.028203379000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":978,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:13.710181+0000","last_change":"2024-09-10T18:31:13.536758+0000","last_active":"2024-09-10T18:33:13.710181+0000","last_peered":"2024-09-10T18:33:13.710181+0000","last_clean":"2024-09-10T18:33:13.710181+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:33:13.710181+0000","last_undegraded":"2024-09-10T18:33:13.710181+0000","last_fullsized":"2024-09-10T18:33:13.710181+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:31:13.536419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:13.536419+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043736720000000003,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":696,"num_read_kb":691,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":264,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.219986+0000","last_change":"2024-09-10T18:32:56.888589+0000","last_active":"2024-09-10T18:33:10.219986+0000","last_peered":"2024-09-10T18:33:10.219986+0000","last_clean":"2024-09-10T18:33:10.219986+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:33:10.219986+0000","last_undegraded":"2024-09-10T18:33:10.219986+0000","last_fullsized":"2024-09-10T18:33:10.219986+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:56.888436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.771415+0000","last_clean_scrub_stamp":"2024-09-10T18:32:56.888436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:19.568187+0000","scrub_duration":0.0064579939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":258,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.223035+0000","last_change":"2024-09-10T18:31:57.719333+0000","last_active":"2024-09-10T18:33:10.223035+0000","last_peered":"2024-09-10T18:33:10.223035+0000","last_clean":"2024-09-10T18:33:10.223035+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:33:10.223035+0000","last_undegraded":"2024-09-10T18:33:10.223035+0000","last_fullsized":"2024-09-10T18:33:10.223035+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:57.719278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.719278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:25.197723+0000","scrub_duration":0.0011883760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"265'6358","reported_seq":6385,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:15.620659+0000","last_change":"2024-09-10T18:32:03.382727+0000","last_active":"2024-09-10T18:33:15.620659+0000","last_peered":"2024-09-10T18:33:15.620659+0000","last_clean":"2024-09-10T18:33:15.620659+0000","last_became_active":"2024-09-10T18:32:03.382335+0000","last_became_peered":"2024-09-10T18:32:03.382335+0000","last_unstale":"2024-09-10T18:33:15.620659+0000","last_undegraded":"2024-09-10T18:33:15.620659+0000","last_fullsized":"2024-09-10T18:33:15.620659+0000","mapping_epoch":246,"log_start":"251'1900","ondisk_log_start":"251'1900","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4458,"log_dups_size":0,"ondisk_log_size":4458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416677888,"num_objects":6358,"num_object_clones":0,"num_object_copies":19074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6358,"num_write_kb":406912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'26","reported_seq":257,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.226474+0000","last_change":"2024-09-10T18:32:42.037679+0000","last_active":"2024-09-10T18:33:10.226474+0000","last_peered":"2024-09-10T18:33:10.226474+0000","last_clean":"2024-09-10T18:33:10.226474+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:33:10.226474+0000","last_undegraded":"2024-09-10T18:33:10.226474+0000","last_fullsized":"2024-09-10T18:33:10.226474+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'26","last_scrub_stamp":"2024-09-10T18:32:42.037595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:42.037595+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:07.709417+0000","scrub_duration":0.003897162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":273,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.225672+0000","last_change":"2024-09-10T18:31:52.756369+0000","last_active":"2024-09-10T18:33:10.225672+0000","last_peered":"2024-09-10T18:33:10.225672+0000","last_clean":"2024-09-10T18:33:10.225672+0000","last_became_active":"2024-09-10T18:26:27.639129+0000","last_became_peered":"2024-09-10T18:26:27.639129+0000","last_unstale":"2024-09-10T18:33:10.225672+0000","last_undegraded":"2024-09-10T18:33:10.225672+0000","last_fullsized":"2024-09-10T18:33:10.225672+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:52.756258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:52.756258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:14.824560+0000","scrub_duration":0.002681269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":324,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.221754+0000","last_change":"2024-09-10T18:32:58.780690+0000","last_active":"2024-09-10T18:33:10.221754+0000","last_peered":"2024-09-10T18:33:10.221754+0000","last_clean":"2024-09-10T18:33:10.221754+0000","last_became_active":"2024-09-10T18:26:27.646658+0000","last_became_peered":"2024-09-10T18:26:27.646658+0000","last_unstale":"2024-09-10T18:33:10.221754+0000","last_undegraded":"2024-09-10T18:33:10.221754+0000","last_fullsized":"2024-09-10T18:33:10.221754+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:58.780613+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.780613+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:13.355182+0000","scrub_duration":0.002490309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":270,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.222315+0000","last_change":"2024-09-10T18:31:53.416989+0000","last_active":"2024-09-10T18:33:10.222315+0000","last_peered":"2024-09-10T18:33:10.222315+0000","last_clean":"2024-09-10T18:33:10.222315+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:33:10.222315+0000","last_undegraded":"2024-09-10T18:33:10.222315+0000","last_fullsized":"2024-09-10T18:33:10.222315+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:53.416936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:53.416936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012036219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":262,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.219899+0000","last_change":"2024-09-10T18:32:49.984765+0000","last_active":"2024-09-10T18:33:10.219899+0000","last_peered":"2024-09-10T18:33:10.219899+0000","last_clean":"2024-09-10T18:33:10.219899+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:33:10.219899+0000","last_undegraded":"2024-09-10T18:33:10.219899+0000","last_fullsized":"2024-09-10T18:33:10.219899+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:49.984707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:32:49.984707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:59.221248+0000","scrub_duration":0.0035569920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"265'6529","reported_seq":6556,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:14.879690+0000","last_change":"2024-09-10T18:32:03.384526+0000","last_active":"2024-09-10T18:33:14.879690+0000","last_peered":"2024-09-10T18:33:14.879690+0000","last_clean":"2024-09-10T18:33:14.879690+0000","last_became_active":"2024-09-10T18:32:03.384247+0000","last_became_peered":"2024-09-10T18:32:03.384247+0000","last_unstale":"2024-09-10T18:33:14.879690+0000","last_undegraded":"2024-09-10T18:33:14.879690+0000","last_fullsized":"2024-09-10T18:33:14.879690+0000","mapping_epoch":246,"log_start":"251'2000","ondisk_log_start":"251'2000","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4529,"log_dups_size":0,"ondisk_log_size":4529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427884544,"num_objects":6529,"num_object_clones":0,"num_object_copies":19587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6529,"num_write_kb":417856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"240'19","reported_seq":303,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.229360+0000","last_change":"2024-09-10T18:31:50.655266+0000","last_active":"2024-09-10T18:33:10.229360+0000","last_peered":"2024-09-10T18:33:10.229360+0000","last_clean":"2024-09-10T18:33:10.229360+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:33:10.229360+0000","last_undegraded":"2024-09-10T18:33:10.229360+0000","last_fullsized":"2024-09-10T18:33:10.229360+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:31:50.655208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:50.655208+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:20.517259+0000","scrub_duration":0.0021897840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":264,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.228716+0000","last_change":"2024-09-10T18:32:40.540692+0000","last_active":"2024-09-10T18:33:10.228716+0000","last_peered":"2024-09-10T18:33:10.228716+0000","last_clean":"2024-09-10T18:33:10.228716+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:33:10.228716+0000","last_undegraded":"2024-09-10T18:33:10.228716+0000","last_fullsized":"2024-09-10T18:33:10.228716+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:40.540616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:32:40.540616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:44.048903+0000","scrub_duration":0.002137262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":278,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.219827+0000","last_change":"2024-09-10T18:33:06.960402+0000","last_active":"2024-09-10T18:33:10.219827+0000","last_peered":"2024-09-10T18:33:10.219827+0000","last_clean":"2024-09-10T18:33:10.219827+0000","last_became_active":"2024-09-10T18:26:27.645885+0000","last_became_peered":"2024-09-10T18:26:27.645885+0000","last_unstale":"2024-09-10T18:33:10.219827+0000","last_undegraded":"2024-09-10T18:33:10.219827+0000","last_fullsized":"2024-09-10T18:33:10.219827+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:06.960320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:33:06.960320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:19.113831+0000","scrub_duration":0.0020571560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":267,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.219657+0000","last_change":"2024-09-10T18:31:50.740204+0000","last_active":"2024-09-10T18:33:10.219657+0000","last_peered":"2024-09-10T18:33:10.219657+0000","last_clean":"2024-09-10T18:33:10.219657+0000","last_became_active":"2024-09-10T18:26:27.632071+0000","last_became_peered":"2024-09-10T18:26:27.632071+0000","last_unstale":"2024-09-10T18:33:10.219657+0000","last_undegraded":"2024-09-10T18:33:10.219657+0000","last_fullsized":"2024-09-10T18:33:10.219657+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:50.740104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:50.740104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014989739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"265'6548","reported_seq":6575,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:15.271459+0000","last_change":"2024-09-10T18:32:03.555018+0000","last_active":"2024-09-10T18:33:15.271459+0000","last_peered":"2024-09-10T18:33:15.271459+0000","last_clean":"2024-09-10T18:33:15.271459+0000","last_became_active":"2024-09-10T18:32:03.554412+0000","last_became_peered":"2024-09-10T18:32:03.554412+0000","last_unstale":"2024-09-10T18:33:15.271459+0000","last_undegraded":"2024-09-10T18:33:15.271459+0000","last_fullsized":"2024-09-10T18:33:15.271459+0000","mapping_epoch":246,"log_start":"251'1800","ondisk_log_start":"251'1800","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4748,"log_dups_size":0,"ondisk_log_size":4748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429129728,"num_objects":6548,"num_object_clones":0,"num_object_copies":19644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6548,"num_write_kb":419072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":183,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.219834+0000","last_change":"2024-09-10T18:33:08.009896+0000","last_active":"2024-09-10T18:33:10.219834+0000","last_peered":"2024-09-10T18:33:10.219834+0000","last_clean":"2024-09-10T18:33:10.219834+0000","last_became_active":"2024-09-10T18:26:18.095088+0000","last_became_peered":"2024-09-10T18:26:18.095088+0000","last_unstale":"2024-09-10T18:33:10.219834+0000","last_undegraded":"2024-09-10T18:33:10.219834+0000","last_fullsized":"2024-09-10T18:33:10.219834+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:08.009806+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.009806+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:37.524807+0000","scrub_duration":0.002902614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":311,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.229290+0000","last_change":"2024-09-10T18:31:41.540517+0000","last_active":"2024-09-10T18:33:10.229290+0000","last_peered":"2024-09-10T18:33:10.229290+0000","last_clean":"2024-09-10T18:33:10.229290+0000","last_became_active":"2024-09-10T18:21:33.611203+0000","last_became_peered":"2024-09-10T18:21:33.611203+0000","last_unstale":"2024-09-10T18:33:10.229290+0000","last_undegraded":"2024-09-10T18:33:10.229290+0000","last_fullsized":"2024-09-10T18:33:10.229290+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-09-10T18:31:41.540420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:41.540420+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028551750000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":228,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.226215+0000","last_change":"2024-09-10T18:32:44.035023+0000","last_active":"2024-09-10T18:33:10.226215+0000","last_peered":"2024-09-10T18:33:10.226215+0000","last_clean":"2024-09-10T18:33:10.226215+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:33:10.226215+0000","last_undegraded":"2024-09-10T18:33:10.226215+0000","last_fullsized":"2024-09-10T18:33:10.226215+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:32:44.034968+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:32:44.034968+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:01.163735+0000","scrub_duration":0.001586772,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":270,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.223103+0000","last_change":"2024-09-10T18:32:50.211889+0000","last_active":"2024-09-10T18:33:10.223103+0000","last_peered":"2024-09-10T18:33:10.223103+0000","last_clean":"2024-09-10T18:33:10.223103+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:33:10.223103+0000","last_undegraded":"2024-09-10T18:33:10.223103+0000","last_fullsized":"2024-09-10T18:33:10.223103+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:50.211414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:50.211414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:55.885824+0000","scrub_duration":0.007128512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":288,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.226628+0000","last_change":"2024-09-10T18:32:39.620645+0000","last_active":"2024-09-10T18:33:10.226628+0000","last_peered":"2024-09-10T18:33:10.226628+0000","last_clean":"2024-09-10T18:33:10.226628+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:33:10.226628+0000","last_undegraded":"2024-09-10T18:33:10.226628+0000","last_fullsized":"2024-09-10T18:33:10.226628+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:39.620456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.620456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:50.575419+0000","scrub_duration":0.017312912,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"265'6588","reported_seq":6613,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:15.614967+0000","last_change":"2024-09-10T18:32:03.555761+0000","last_active":"2024-09-10T18:33:15.614967+0000","last_peered":"2024-09-10T18:33:15.614967+0000","last_clean":"2024-09-10T18:33:15.614967+0000","last_became_active":"2024-09-10T18:32:03.554475+0000","last_became_peered":"2024-09-10T18:32:03.554475+0000","last_unstale":"2024-09-10T18:33:15.614967+0000","last_undegraded":"2024-09-10T18:33:15.614967+0000","last_fullsized":"2024-09-10T18:33:15.614967+0000","mapping_epoch":246,"log_start":"251'2100","ondisk_log_start":"251'2100","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431751168,"num_objects":6588,"num_object_clones":0,"num_object_copies":19764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6588,"num_write_kb":421632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":312,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.225926+0000","last_change":"2024-09-10T18:31:10.502369+0000","last_active":"2024-09-10T18:33:10.225926+0000","last_peered":"2024-09-10T18:33:10.225926+0000","last_clean":"2024-09-10T18:33:10.225926+0000","last_became_active":"2024-09-10T18:26:18.092014+0000","last_became_peered":"2024-09-10T18:26:18.092014+0000","last_unstale":"2024-09-10T18:33:10.225926+0000","last_undegraded":"2024-09-10T18:33:10.225926+0000","last_fullsized":"2024-09-10T18:33:10.225926+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:10.502290+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:10.502290+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082798439999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"240'25","reported_seq":32,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.221941+0000","last_change":"2024-09-10T18:32:51.151265+0000","last_active":"2024-09-10T18:33:10.221941+0000","last_peered":"2024-09-10T18:33:10.221941+0000","last_clean":"2024-09-10T18:33:10.221941+0000","last_became_active":"2024-09-10T18:32:51.118251+0000","last_became_peered":"2024-09-10T18:32:51.118251+0000","last_unstale":"2024-09-10T18:33:10.221941+0000","last_undegraded":"2024-09-10T18:33:10.221941+0000","last_fullsized":"2024-09-10T18:33:10.221941+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:32:39.544846+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.544846+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:56.978883+0000","scrub_duration":0.0033922850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":266,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:13.324809+0000","last_change":"2024-09-10T18:33:13.324775+0000","last_active":"2024-09-10T18:33:13.324809+0000","last_peered":"2024-09-10T18:33:13.324809+0000","last_clean":"2024-09-10T18:33:13.324809+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:33:13.324809+0000","last_undegraded":"2024-09-10T18:33:13.324809+0000","last_fullsized":"2024-09-10T18:33:13.324809+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:13.324556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:13.324556+0000","last_clean_scrub_stamp":"2024-09-10T18:33:13.324556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:13.880002+0000","scrub_duration":0.011863357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"189'9","reported_seq":440,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.770959+0000","last_change":"2024-09-10T18:32:42.667266+0000","last_active":"2024-09-10T18:33:10.770959+0000","last_peered":"2024-09-10T18:33:10.770959+0000","last_clean":"2024-09-10T18:33:10.770959+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:33:10.770959+0000","last_undegraded":"2024-09-10T18:33:10.770959+0000","last_fullsized":"2024-09-10T18:33:10.770959+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'9","last_scrub_stamp":"2024-09-10T18:32:42.667117+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:32:42.667117+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:47.771641+0000","scrub_duration":0.0071583269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":261,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.229133+0000","last_change":"2024-09-10T18:32:46.630172+0000","last_active":"2024-09-10T18:33:10.229133+0000","last_peered":"2024-09-10T18:33:10.229133+0000","last_clean":"2024-09-10T18:33:10.229133+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:33:10.229133+0000","last_undegraded":"2024-09-10T18:33:10.229133+0000","last_fullsized":"2024-09-10T18:33:10.229133+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:46.630078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_clean_scrub_stamp":"2024-09-10T18:32:46.630078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:08.759422+0000","scrub_duration":0.0024328319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"265'6538","reported_seq":6564,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:14.937149+0000","last_change":"2024-09-10T18:32:03.384707+0000","last_active":"2024-09-10T18:33:14.937149+0000","last_peered":"2024-09-10T18:33:14.937149+0000","last_clean":"2024-09-10T18:33:14.937149+0000","last_became_active":"2024-09-10T18:32:03.384417+0000","last_became_peered":"2024-09-10T18:32:03.384417+0000","last_unstale":"2024-09-10T18:33:14.937149+0000","last_undegraded":"2024-09-10T18:33:14.937149+0000","last_fullsized":"2024-09-10T18:33:14.937149+0000","mapping_epoch":246,"log_start":"251'1600","ondisk_log_start":"251'1600","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4938,"log_dups_size":0,"ondisk_log_size":4938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428474368,"num_objects":6538,"num_object_clones":0,"num_object_copies":19614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6538,"num_write_kb":418432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"240'15","reported_seq":278,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.218229+0000","last_change":"2024-09-10T18:31:50.320682+0000","last_active":"2024-09-10T18:33:10.218229+0000","last_peered":"2024-09-10T18:33:10.218229+0000","last_clean":"2024-09-10T18:33:10.218229+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:33:10.218229+0000","last_undegraded":"2024-09-10T18:33:10.218229+0000","last_fullsized":"2024-09-10T18:33:10.218229+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:31:50.320624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:50.320624+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002110139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"265'2","reported_seq":292,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:11.047394+0000","last_change":"2024-09-10T18:33:11.047358+0000","last_active":"2024-09-10T18:33:11.047394+0000","last_peered":"2024-09-10T18:33:11.047394+0000","last_clean":"2024-09-10T18:33:11.047394+0000","last_became_active":"2024-09-10T18:26:27.644997+0000","last_became_peered":"2024-09-10T18:26:27.644997+0000","last_unstale":"2024-09-10T18:33:11.047394+0000","last_undegraded":"2024-09-10T18:33:11.047394+0000","last_fullsized":"2024-09-10T18:33:11.047394+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'2","last_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_deep_scrub":"265'2","last_deep_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.047305+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:22.716371+0000","scrub_duration":0.021095425000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":286,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.226373+0000","last_change":"2024-09-10T18:32:52.590088+0000","last_active":"2024-09-10T18:33:10.226373+0000","last_peered":"2024-09-10T18:33:10.226373+0000","last_clean":"2024-09-10T18:33:10.226373+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:33:10.226373+0000","last_undegraded":"2024-09-10T18:33:10.226373+0000","last_fullsized":"2024-09-10T18:33:10.226373+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:52.589997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:52.589997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:18.281059+0000","scrub_duration":0.001802543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":259,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.218194+0000","last_change":"2024-09-10T18:32:44.239399+0000","last_active":"2024-09-10T18:33:10.218194+0000","last_peered":"2024-09-10T18:33:10.218194+0000","last_clean":"2024-09-10T18:33:10.218194+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:33:10.218194+0000","last_undegraded":"2024-09-10T18:33:10.218194+0000","last_fullsized":"2024-09-10T18:33:10.218194+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:44.239347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:44.239347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:11.230674+0000","scrub_duration":0.0011983390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"265'6421","reported_seq":6448,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:14.881992+0000","last_change":"2024-09-10T18:32:03.382716+0000","last_active":"2024-09-10T18:33:14.881992+0000","last_peered":"2024-09-10T18:33:14.881992+0000","last_clean":"2024-09-10T18:33:14.881992+0000","last_became_active":"2024-09-10T18:32:03.382430+0000","last_became_peered":"2024-09-10T18:32:03.382430+0000","last_unstale":"2024-09-10T18:33:14.881992+0000","last_undegraded":"2024-09-10T18:33:14.881992+0000","last_fullsized":"2024-09-10T18:33:14.881992+0000","mapping_epoch":246,"log_start":"251'1500","ondisk_log_start":"251'1500","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4921,"log_dups_size":0,"ondisk_log_size":4921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:27.873094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420806656,"num_objects":6421,"num_object_clones":0,"num_object_copies":19263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6421,"num_write_kb":410944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"240'44","reported_seq":345,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.226048+0000","last_change":"2024-09-10T18:32:39.964570+0000","last_active":"2024-09-10T18:33:10.226048+0000","last_peered":"2024-09-10T18:33:10.226048+0000","last_clean":"2024-09-10T18:33:10.226048+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:33:10.226048+0000","last_undegraded":"2024-09-10T18:33:10.226048+0000","last_fullsized":"2024-09-10T18:33:10.226048+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'44","last_scrub_stamp":"2024-09-10T18:32:39.964484+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.964484+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:43.342879+0000","scrub_duration":0.012248376,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":281,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.222727+0000","last_change":"2024-09-10T18:31:45.496272+0000","last_active":"2024-09-10T18:33:10.222727+0000","last_peered":"2024-09-10T18:33:10.222727+0000","last_clean":"2024-09-10T18:33:10.222727+0000","last_became_active":"2024-09-10T18:26:27.638408+0000","last_became_peered":"2024-09-10T18:26:27.638408+0000","last_unstale":"2024-09-10T18:33:10.222727+0000","last_undegraded":"2024-09-10T18:33:10.222727+0000","last_fullsized":"2024-09-10T18:33:10.222727+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:45.496183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:45.496183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015755630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":290,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.225818+0000","last_change":"2024-09-10T18:32:48.582057+0000","last_active":"2024-09-10T18:33:10.225818+0000","last_peered":"2024-09-10T18:33:10.225818+0000","last_clean":"2024-09-10T18:33:10.225818+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:33:10.225818+0000","last_undegraded":"2024-09-10T18:33:10.225818+0000","last_fullsized":"2024-09-10T18:33:10.225818+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:48.581963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:48.581963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:53.020863+0000","scrub_duration":0.011575692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":271,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.221681+0000","last_change":"2024-09-10T18:31:58.460588+0000","last_active":"2024-09-10T18:33:10.221681+0000","last_peered":"2024-09-10T18:33:10.221681+0000","last_clean":"2024-09-10T18:33:10.221681+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:33:10.221681+0000","last_undegraded":"2024-09-10T18:33:10.221681+0000","last_fullsized":"2024-09-10T18:33:10.221681+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:58.460533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:58.460533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:28.153691+0000","scrub_duration":0.001375914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"265'6503","reported_seq":6530,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:15.272739+0000","last_change":"2024-09-10T18:32:03.554922+0000","last_active":"2024-09-10T18:33:15.272739+0000","last_peered":"2024-09-10T18:33:15.272739+0000","last_clean":"2024-09-10T18:33:15.272739+0000","last_became_active":"2024-09-10T18:32:03.554423+0000","last_became_peered":"2024-09-10T18:32:03.554423+0000","last_unstale":"2024-09-10T18:33:15.272739+0000","last_undegraded":"2024-09-10T18:33:15.272739+0000","last_fullsized":"2024-09-10T18:33:15.272739+0000","mapping_epoch":246,"log_start":"251'1800","ondisk_log_start":"251'1800","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4703,"log_dups_size":0,"ondisk_log_size":4703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426180608,"num_objects":6503,"num_object_clones":0,"num_object_copies":19509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6503,"num_write_kb":416192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"240'18","reported_seq":307,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.228446+0000","last_change":"2024-09-10T18:32:54.528296+0000","last_active":"2024-09-10T18:33:10.228446+0000","last_peered":"2024-09-10T18:33:10.228446+0000","last_clean":"2024-09-10T18:33:10.228446+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:33:10.228446+0000","last_undegraded":"2024-09-10T18:33:10.228446+0000","last_fullsized":"2024-09-10T18:33:10.228446+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-09-10T18:32:54.528209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:54.528209+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.286857+0000","scrub_duration":0.0056889829999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":284,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:14.879927+0000","last_change":"2024-09-10T18:33:14.879881+0000","last_active":"2024-09-10T18:33:14.879927+0000","last_peered":"2024-09-10T18:33:14.879927+0000","last_clean":"2024-09-10T18:33:14.879927+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:33:14.879927+0000","last_undegraded":"2024-09-10T18:33:14.879927+0000","last_fullsized":"2024-09-10T18:33:14.879927+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:14.879787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:33:14.879787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:36.213826+0000","scrub_duration":0.0044267550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":290,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.225536+0000","last_change":"2024-09-10T18:32:50.570319+0000","last_active":"2024-09-10T18:33:10.225536+0000","last_peered":"2024-09-10T18:33:10.225536+0000","last_clean":"2024-09-10T18:33:10.225536+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:33:10.225536+0000","last_undegraded":"2024-09-10T18:33:10.225536+0000","last_fullsized":"2024-09-10T18:33:10.225536+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:50.570113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:50.570113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:05.696833+0000","scrub_duration":0.0059653969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":260,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.231413+0000","last_change":"2024-09-10T18:32:58.358775+0000","last_active":"2024-09-10T18:33:10.231413+0000","last_peered":"2024-09-10T18:33:10.231413+0000","last_clean":"2024-09-10T18:33:10.231413+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:33:10.231413+0000","last_undegraded":"2024-09-10T18:33:10.231413+0000","last_fullsized":"2024-09-10T18:33:10.231413+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:58.358694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.803236+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.358694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:15.808097+0000","scrub_duration":0.011821898000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"265'6551","reported_seq":6577,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:15.931646+0000","last_change":"2024-09-10T18:32:03.383156+0000","last_active":"2024-09-10T18:33:15.931646+0000","last_peered":"2024-09-10T18:33:15.931646+0000","last_clean":"2024-09-10T18:33:15.931646+0000","last_became_active":"2024-09-10T18:32:03.382890+0000","last_became_peered":"2024-09-10T18:32:03.382890+0000","last_unstale":"2024-09-10T18:33:15.931646+0000","last_undegraded":"2024-09-10T18:33:15.931646+0000","last_fullsized":"2024-09-10T18:33:15.931646+0000","mapping_epoch":246,"log_start":"249'800","ondisk_log_start":"249'800","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":5751,"log_dups_size":0,"ondisk_log_size":5751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:17.281396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429326336,"num_objects":6551,"num_object_clones":0,"num_object_copies":19653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6551,"num_write_kb":419264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":313,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.218296+0000","last_change":"2024-09-10T18:31:47.535552+0000","last_active":"2024-09-10T18:33:10.218296+0000","last_peered":"2024-09-10T18:33:10.218296+0000","last_clean":"2024-09-10T18:33:10.218296+0000","last_became_active":"2024-09-10T18:26:27.645932+0000","last_became_peered":"2024-09-10T18:26:27.645932+0000","last_unstale":"2024-09-10T18:33:10.218296+0000","last_undegraded":"2024-09-10T18:33:10.218296+0000","last_fullsized":"2024-09-10T18:33:10.218296+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:31:47.535496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:47.535496+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017425819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":284,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.222763+0000","last_change":"2024-09-10T18:33:05.784757+0000","last_active":"2024-09-10T18:33:10.222763+0000","last_peered":"2024-09-10T18:33:10.222763+0000","last_clean":"2024-09-10T18:33:10.222763+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:33:10.222763+0000","last_undegraded":"2024-09-10T18:33:10.222763+0000","last_fullsized":"2024-09-10T18:33:10.222763+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:33:05.784670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:33:05.784670+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:24.433627+0000","scrub_duration":0.002289031,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":419,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.778102+0000","last_change":"2024-09-10T18:31:03.524139+0000","last_active":"2024-09-10T18:33:10.778102+0000","last_peered":"2024-09-10T18:33:10.778102+0000","last_clean":"2024-09-10T18:33:10.778102+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:33:10.778102+0000","last_undegraded":"2024-09-10T18:33:10.778102+0000","last_fullsized":"2024-09-10T18:33:10.778102+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:31:03.524066+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040323920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":266,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.218310+0000","last_change":"2024-09-10T18:33:05.964339+0000","last_active":"2024-09-10T18:33:10.218310+0000","last_peered":"2024-09-10T18:33:10.218310+0000","last_clean":"2024-09-10T18:33:10.218310+0000","last_became_active":"2024-09-10T18:26:27.647511+0000","last_became_peered":"2024-09-10T18:26:27.647511+0000","last_unstale":"2024-09-10T18:33:10.218310+0000","last_undegraded":"2024-09-10T18:33:10.218310+0000","last_fullsized":"2024-09-10T18:33:10.218310+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:05.964034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:05.964034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:30.675049+0000","scrub_duration":0.0015507839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":219,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.225998+0000","last_change":"2024-09-10T18:32:47.037801+0000","last_active":"2024-09-10T18:33:10.225998+0000","last_peered":"2024-09-10T18:33:10.225998+0000","last_clean":"2024-09-10T18:33:10.225998+0000","last_became_active":"2024-09-10T18:26:27.640922+0000","last_became_peered":"2024-09-10T18:26:27.640922+0000","last_unstale":"2024-09-10T18:33:10.225998+0000","last_undegraded":"2024-09-10T18:33:10.225998+0000","last_fullsized":"2024-09-10T18:33:10.225998+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:47.037651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:32:47.037651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:03.275969+0000","scrub_duration":0.0021178149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"240'12","reported_seq":282,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.227026+0000","last_change":"2024-09-10T18:32:56.384627+0000","last_active":"2024-09-10T18:33:10.227026+0000","last_peered":"2024-09-10T18:33:10.227026+0000","last_clean":"2024-09-10T18:33:10.227026+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:33:10.227026+0000","last_undegraded":"2024-09-10T18:33:10.227026+0000","last_fullsized":"2024-09-10T18:33:10.227026+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-10T18:32:56.384439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:56.384439+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:05.649827+0000","scrub_duration":0.010401713999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":260,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.219233+0000","last_change":"2024-09-10T18:31:53.634789+0000","last_active":"2024-09-10T18:33:10.219233+0000","last_peered":"2024-09-10T18:33:10.219233+0000","last_clean":"2024-09-10T18:33:10.219233+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:33:10.219233+0000","last_undegraded":"2024-09-10T18:33:10.219233+0000","last_fullsized":"2024-09-10T18:33:10.219233+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:53.634704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:53.634704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014977510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":265,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.222195+0000","last_change":"2024-09-10T18:31:01.523924+0000","last_active":"2024-09-10T18:33:10.222195+0000","last_peered":"2024-09-10T18:33:10.222195+0000","last_clean":"2024-09-10T18:33:10.222195+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:33:10.222195+0000","last_undegraded":"2024-09-10T18:33:10.222195+0000","last_fullsized":"2024-09-10T18:33:10.222195+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:01.523839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:01.523839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016753880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"240'19","reported_seq":295,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.218566+0000","last_change":"2024-09-10T18:33:08.284482+0000","last_active":"2024-09-10T18:33:10.218566+0000","last_peered":"2024-09-10T18:33:10.218566+0000","last_clean":"2024-09-10T18:33:10.218566+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:33:10.218566+0000","last_undegraded":"2024-09-10T18:33:10.218566+0000","last_fullsized":"2024-09-10T18:33:10.218566+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:33:08.284394+0000","last_deep_scrub":"189'13","last_deep_scrub_stamp":"2024-09-10T18:28:08.976162+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.284394+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:31.505393+0000","scrub_duration":0.010009881999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":271,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.226099+0000","last_change":"2024-09-10T18:31:00.524076+0000","last_active":"2024-09-10T18:33:10.226099+0000","last_peered":"2024-09-10T18:33:10.226099+0000","last_clean":"2024-09-10T18:33:10.226099+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:33:10.226099+0000","last_undegraded":"2024-09-10T18:33:10.226099+0000","last_fullsized":"2024-09-10T18:33:10.226099+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:00.523985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:31:00.523985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018784559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":262,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.227358+0000","last_change":"2024-09-10T18:32:39.161911+0000","last_active":"2024-09-10T18:33:10.227358+0000","last_peered":"2024-09-10T18:33:10.227358+0000","last_clean":"2024-09-10T18:33:10.227358+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:33:10.227358+0000","last_undegraded":"2024-09-10T18:33:10.227358+0000","last_fullsized":"2024-09-10T18:33:10.227358+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:39.161653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.161653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:58.824821+0000","scrub_duration":0.0093847749999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":278,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.226071+0000","last_change":"2024-09-10T18:32:43.621566+0000","last_active":"2024-09-10T18:33:10.226071+0000","last_peered":"2024-09-10T18:33:10.226071+0000","last_clean":"2024-09-10T18:33:10.226071+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:33:10.226071+0000","last_undegraded":"2024-09-10T18:33:10.226071+0000","last_fullsized":"2024-09-10T18:33:10.226071+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:43.621478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:43.621478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:01.039659+0000","scrub_duration":0.0031857410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"240'19","reported_seq":292,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.220472+0000","last_change":"2024-09-10T18:31:45.544047+0000","last_active":"2024-09-10T18:33:10.220472+0000","last_peered":"2024-09-10T18:33:10.220472+0000","last_clean":"2024-09-10T18:33:10.220472+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:33:10.220472+0000","last_undegraded":"2024-09-10T18:33:10.220472+0000","last_fullsized":"2024-09-10T18:33:10.220472+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:31:45.543996+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:31:45.543996+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016860379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":275,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:12.342577+0000","last_change":"2024-09-10T18:33:12.342537+0000","last_active":"2024-09-10T18:33:12.342577+0000","last_peered":"2024-09-10T18:33:12.342577+0000","last_clean":"2024-09-10T18:33:12.342577+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:33:12.342577+0000","last_undegraded":"2024-09-10T18:33:12.342577+0000","last_fullsized":"2024-09-10T18:33:12.342577+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:12.342432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_clean_scrub_stamp":"2024-09-10T18:33:12.342432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:21.132397+0000","scrub_duration":0.0043002429999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":411,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.774174+0000","last_change":"2024-09-10T18:32:40.183655+0000","last_active":"2024-09-10T18:33:10.774174+0000","last_peered":"2024-09-10T18:33:10.774174+0000","last_clean":"2024-09-10T18:33:10.774174+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:33:10.774174+0000","last_undegraded":"2024-09-10T18:33:10.774174+0000","last_fullsized":"2024-09-10T18:33:10.774174+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:32:40.183571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:40.183571+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:50.634276+0000","scrub_duration":0.014890771000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":268,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:11.727924+0000","last_change":"2024-09-10T18:33:11.727869+0000","last_active":"2024-09-10T18:33:11.727924+0000","last_peered":"2024-09-10T18:33:11.727924+0000","last_clean":"2024-09-10T18:33:11.727924+0000","last_became_active":"2024-09-10T18:21:37.639706+0000","last_became_peered":"2024-09-10T18:21:37.639706+0000","last_unstale":"2024-09-10T18:33:11.727924+0000","last_undegraded":"2024-09-10T18:33:11.727924+0000","last_fullsized":"2024-09-10T18:33:11.727924+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:11.727786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.727786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:37.347798+0000","scrub_duration":0.0092571419999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":294,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.220145+0000","last_change":"2024-09-10T18:31:48.766301+0000","last_active":"2024-09-10T18:33:10.220145+0000","last_peered":"2024-09-10T18:33:10.220145+0000","last_clean":"2024-09-10T18:33:10.220145+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:33:10.220145+0000","last_undegraded":"2024-09-10T18:33:10.220145+0000","last_fullsized":"2024-09-10T18:33:10.220145+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-10T18:31:48.766245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.766245+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001995817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":274,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.224179+0000","last_change":"2024-09-10T18:32:39.535817+0000","last_active":"2024-09-10T18:33:10.224179+0000","last_peered":"2024-09-10T18:33:10.224179+0000","last_clean":"2024-09-10T18:33:10.224179+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:33:10.224179+0000","last_undegraded":"2024-09-10T18:33:10.224179+0000","last_fullsized":"2024-09-10T18:33:10.224179+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:39.535749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.535749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:03.810715+0000","scrub_duration":0.0046018179999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":420,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.773804+0000","last_change":"2024-09-10T18:32:43.942068+0000","last_active":"2024-09-10T18:33:10.773804+0000","last_peered":"2024-09-10T18:33:10.773804+0000","last_clean":"2024-09-10T18:33:10.773804+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:33:10.773804+0000","last_undegraded":"2024-09-10T18:33:10.773804+0000","last_fullsized":"2024-09-10T18:33:10.773804+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:32:43.941975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:43.941975+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:51.834001+0000","scrub_duration":0.0046448039999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":259,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.218219+0000","last_change":"2024-09-10T18:32:42.165903+0000","last_active":"2024-09-10T18:33:10.218219+0000","last_peered":"2024-09-10T18:33:10.218219+0000","last_clean":"2024-09-10T18:33:10.218219+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:33:10.218219+0000","last_undegraded":"2024-09-10T18:33:10.218219+0000","last_fullsized":"2024-09-10T18:33:10.218219+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:42.165812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:42.165812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:03.644151+0000","scrub_duration":0.010729512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":300,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.228667+0000","last_change":"2024-09-10T18:31:47.604036+0000","last_active":"2024-09-10T18:33:10.228667+0000","last_peered":"2024-09-10T18:33:10.228667+0000","last_clean":"2024-09-10T18:33:10.228667+0000","last_became_active":"2024-09-10T18:26:27.649202+0000","last_became_peered":"2024-09-10T18:26:27.649202+0000","last_unstale":"2024-09-10T18:33:10.228667+0000","last_undegraded":"2024-09-10T18:33:10.228667+0000","last_fullsized":"2024-09-10T18:33:10.228667+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:31:47.603984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:47.603984+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018863860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":274,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.230133+0000","last_change":"2024-09-10T18:32:00.649943+0000","last_active":"2024-09-10T18:33:10.230133+0000","last_peered":"2024-09-10T18:33:10.230133+0000","last_clean":"2024-09-10T18:33:10.230133+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:33:10.230133+0000","last_undegraded":"2024-09-10T18:33:10.230133+0000","last_fullsized":"2024-09-10T18:33:10.230133+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.649881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.649881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001205547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":273,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.224255+0000","last_change":"2024-09-10T18:31:56.674517+0000","last_active":"2024-09-10T18:33:10.224255+0000","last_peered":"2024-09-10T18:33:10.224255+0000","last_clean":"2024-09-10T18:33:10.224255+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:33:10.224255+0000","last_undegraded":"2024-09-10T18:33:10.224255+0000","last_fullsized":"2024-09-10T18:33:10.224255+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_clean_scrub_stamp":"2024-09-10T18:31:56.674462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:20.788769+0000","scrub_duration":0.0010991219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":250,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.219577+0000","last_change":"2024-09-10T18:31:45.797979+0000","last_active":"2024-09-10T18:33:10.219577+0000","last_peered":"2024-09-10T18:33:10.219577+0000","last_clean":"2024-09-10T18:33:10.219577+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:33:10.219577+0000","last_undegraded":"2024-09-10T18:33:10.219577+0000","last_fullsized":"2024-09-10T18:33:10.219577+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:45.797924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:31:45.797924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001612093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"240'15","reported_seq":304,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.224400+0000","last_change":"2024-09-10T18:32:41.169276+0000","last_active":"2024-09-10T18:33:10.224400+0000","last_peered":"2024-09-10T18:33:10.224400+0000","last_clean":"2024-09-10T18:33:10.224400+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:33:10.224400+0000","last_undegraded":"2024-09-10T18:33:10.224400+0000","last_fullsized":"2024-09-10T18:33:10.224400+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:32:41.168785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:41.168785+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:53.991999+0000","scrub_duration":0.0037973939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":277,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.222609+0000","last_change":"2024-09-10T18:33:01.781370+0000","last_active":"2024-09-10T18:33:10.222609+0000","last_peered":"2024-09-10T18:33:10.222609+0000","last_clean":"2024-09-10T18:33:10.222609+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:33:10.222609+0000","last_undegraded":"2024-09-10T18:33:10.222609+0000","last_fullsized":"2024-09-10T18:33:10.222609+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:01.781295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.781295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:27.429985+0000","scrub_duration":0.0016784879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":252,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.231477+0000","last_change":"2024-09-10T18:31:44.838204+0000","last_active":"2024-09-10T18:33:10.231477+0000","last_peered":"2024-09-10T18:33:10.231477+0000","last_clean":"2024-09-10T18:33:10.231477+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:33:10.231477+0000","last_undegraded":"2024-09-10T18:33:10.231477+0000","last_fullsized":"2024-09-10T18:33:10.231477+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:44.838141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:31:44.838141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011071239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":259,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.219107+0000","last_change":"2024-09-10T18:32:02.220356+0000","last_active":"2024-09-10T18:33:10.219107+0000","last_peered":"2024-09-10T18:33:10.219107+0000","last_clean":"2024-09-10T18:33:10.219107+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:33:10.219107+0000","last_undegraded":"2024-09-10T18:33:10.219107+0000","last_fullsized":"2024-09-10T18:33:10.219107+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.220305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.220305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:11.568327+0000","scrub_duration":0.0015049099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":302,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.220163+0000","last_change":"2024-09-10T18:32:46.253109+0000","last_active":"2024-09-10T18:33:10.220163+0000","last_peered":"2024-09-10T18:33:10.220163+0000","last_clean":"2024-09-10T18:33:10.220163+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:33:10.220163+0000","last_undegraded":"2024-09-10T18:33:10.220163+0000","last_fullsized":"2024-09-10T18:33:10.220163+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:32:46.252895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:46.252895+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:52.741188+0000","scrub_duration":0.003188967,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":275,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.219298+0000","last_change":"2024-09-10T18:32:01.595970+0000","last_active":"2024-09-10T18:33:10.219298+0000","last_peered":"2024-09-10T18:33:10.219298+0000","last_clean":"2024-09-10T18:33:10.219298+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:33:10.219298+0000","last_undegraded":"2024-09-10T18:33:10.219298+0000","last_fullsized":"2024-09-10T18:33:10.219298+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:01.595917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:32:01.595917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016309899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":270,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.221338+0000","last_change":"2024-09-10T18:31:50.423429+0000","last_active":"2024-09-10T18:33:10.221338+0000","last_peered":"2024-09-10T18:33:10.221338+0000","last_clean":"2024-09-10T18:33:10.221338+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:33:10.221338+0000","last_undegraded":"2024-09-10T18:33:10.221338+0000","last_fullsized":"2024-09-10T18:33:10.221338+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:50.423374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:50.423374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:14.190747+0000","scrub_duration":0.00116716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":267,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.231700+0000","last_change":"2024-09-10T18:31:14.775776+0000","last_active":"2024-09-10T18:33:10.231700+0000","last_peered":"2024-09-10T18:33:10.231700+0000","last_clean":"2024-09-10T18:33:10.231700+0000","last_became_active":"2024-09-10T18:26:27.636996+0000","last_became_peered":"2024-09-10T18:26:27.636996+0000","last_unstale":"2024-09-10T18:33:10.231700+0000","last_undegraded":"2024-09-10T18:33:10.231700+0000","last_fullsized":"2024-09-10T18:33:10.231700+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:14.775693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:14.775693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00163416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":283,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.220077+0000","last_change":"2024-09-10T18:32:47.287083+0000","last_active":"2024-09-10T18:33:10.220077+0000","last_peered":"2024-09-10T18:33:10.220077+0000","last_clean":"2024-09-10T18:33:10.220077+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:33:10.220077+0000","last_undegraded":"2024-09-10T18:33:10.220077+0000","last_fullsized":"2024-09-10T18:33:10.220077+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-09-10T18:32:47.286977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:47.286977+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:11.371082+0000","scrub_duration":0.017807315000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":268,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.218955+0000","last_change":"2024-09-10T18:32:57.214331+0000","last_active":"2024-09-10T18:33:10.218955+0000","last_peered":"2024-09-10T18:33:10.218955+0000","last_clean":"2024-09-10T18:33:10.218955+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:33:10.218955+0000","last_undegraded":"2024-09-10T18:33:10.218955+0000","last_fullsized":"2024-09-10T18:33:10.218955+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:57.214242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:32:57.214242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:12.819926+0000","scrub_duration":0.015466415000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":223,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.225964+0000","last_change":"2024-09-10T18:33:00.161788+0000","last_active":"2024-09-10T18:33:10.225964+0000","last_peered":"2024-09-10T18:33:10.225964+0000","last_clean":"2024-09-10T18:33:10.225964+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:33:10.225964+0000","last_undegraded":"2024-09-10T18:33:10.225964+0000","last_fullsized":"2024-09-10T18:33:10.225964+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:00.161361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:33:00.161361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:05.797066+0000","scrub_duration":0.0035012799999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":268,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-10T18:33:13.845099+0000","last_change":"2024-09-10T18:33:13.844984+0000","last_active":"2024-09-10T18:33:13.845099+0000","last_peered":"2024-09-10T18:33:13.845099+0000","last_clean":"2024-09-10T18:33:13.845099+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:33:13.845099+0000","last_undegraded":"2024-09-10T18:33:13.845099+0000","last_fullsized":"2024-09-10T18:33:13.845099+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:13.844781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:13.844781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:15.129270+0000","scrub_duration":0.0022954300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":268,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.223215+0000","last_change":"2024-09-10T18:31:49.495614+0000","last_active":"2024-09-10T18:33:10.223215+0000","last_peered":"2024-09-10T18:33:10.223215+0000","last_clean":"2024-09-10T18:33:10.223215+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:33:10.223215+0000","last_undegraded":"2024-09-10T18:33:10.223215+0000","last_fullsized":"2024-09-10T18:33:10.223215+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:49.495556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:49.495556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:12.834399+0000","scrub_duration":0.0012999590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":256,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.230148+0000","last_change":"2024-09-10T18:31:43.830584+0000","last_active":"2024-09-10T18:33:10.230148+0000","last_peered":"2024-09-10T18:33:10.230148+0000","last_clean":"2024-09-10T18:33:10.230148+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:33:10.230148+0000","last_undegraded":"2024-09-10T18:33:10.230148+0000","last_fullsized":"2024-09-10T18:33:10.230148+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:43.830528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:31:43.830528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011198810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":274,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.227116+0000","last_change":"2024-09-10T18:31:54.060506+0000","last_active":"2024-09-10T18:33:10.227116+0000","last_peered":"2024-09-10T18:33:10.227116+0000","last_clean":"2024-09-10T18:33:10.227116+0000","last_became_active":"2024-09-10T18:26:27.648112+0000","last_became_peered":"2024-09-10T18:26:27.648112+0000","last_unstale":"2024-09-10T18:33:10.227116+0000","last_undegraded":"2024-09-10T18:33:10.227116+0000","last_fullsized":"2024-09-10T18:33:10.227116+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:31:54.060444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:54.060444+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:19.181773+0000","scrub_duration":0.0019908159999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":261,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-10T18:33:10.218347+0000","last_change":"2024-09-10T18:32:00.271380+0000","last_active":"2024-09-10T18:33:10.218347+0000","last_peered":"2024-09-10T18:33:10.218347+0000","last_clean":"2024-09-10T18:33:10.218347+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:33:10.218347+0000","last_undegraded":"2024-09-10T18:33:10.218347+0000","last_fullsized":"2024-09-10T18:33:10.218347+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.271326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.271326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:12.093852+0000","scrub_duration":0.0014948439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6804996096,"num_objects":103836,"num_object_clones":0,"num_object_copies":311508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103836,"num_write_kb":6645504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20400766976,"data_stored":20400766976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77864,"ondisk_log_size":77864,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":731,"ondisk_log_size":731,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":696,"num_read_kb":691,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":891,"ondisk_log_size":891,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234128,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4033472,"kb_used_data":3723640,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":89711680,"statfs":{"total":95995035648,"available":91864760320,"internally_reserved":0,"allocated":3813007360,"data_stored":3812503990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":2.956,"5min":2.5059999999999998,"15min":1.7310000000000001},"min":{"1min":0.27500000000000002,"5min":0.21099999999999999,"15min":0.20799999999999999},"max":{"1min":50.091999999999999,"5min":75.506,"15min":85.012},"last":2.0209999999999999},{"interface":"front","average":{"1min":2.9100000000000001,"5min":2.3959999999999999,"15min":1.6970000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.222},"max":{"1min":50.104999999999997,"5min":75.710999999999999,"15min":84.665999999999997},"last":2.2909999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.899,"15min":1.1919999999999999},"min":{"1min":0.26300000000000001,"5min":0.214,"15min":0.214},"max":{"1min":5.8940000000000001,"5min":49.317999999999998,"15min":49.317999999999998},"last":1.95},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.7010000000000001,"15min":1.149},"min":{"1min":0.318,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":5.8550000000000004,"5min":49.353000000000002,"15min":49.353000000000002},"last":2.2309999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.84499999999999997,"15min":0.755},"min":{"1min":0.27700000000000002,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":1.1599999999999999,"5min":5.008,"15min":5.3150000000000004},"last":1.575},{"interface":"front","average":{"1min":0.54100000000000004,"5min":1.008,"15min":0.83199999999999996},"min":{"1min":0.32500000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.24,"5min":5.4029999999999996,"15min":9.9450000000000003},"last":1.9750000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":1.0409999999999999,"15min":1.319},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.5190000000000001,"5min":4.4160000000000004,"15min":86.305999999999997},"last":1.6259999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.1519999999999999,"15min":1.3680000000000001},"min":{"1min":0.40200000000000002,"5min":0.29699999999999999,"15min":0.26700000000000002},"max":{"1min":7.1559999999999997,"5min":7.3339999999999996,"15min":88.364000000000004},"last":2.0009999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.60999999999999999,"15min":0.47399999999999998},"min":{"1min":0.26300000000000001,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":1.3140000000000001,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":2.2610000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.65800000000000003,"15min":0.497},"min":{"1min":0.28299999999999997,"5min":0.26300000000000001,"15min":0.223},"max":{"1min":1.147,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":1.607}]},{"osd":5,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.98499999999999999,"15min":0.60399999999999998},"min":{"1min":0.29999999999999999,"5min":0.27600000000000002,"15min":0.23499999999999999},"max":{"1min":1.3280000000000001,"5min":7.3109999999999999,"15min":7.3109999999999999},"last":1.6990000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.79700000000000004,"15min":0.56699999999999995},"min":{"1min":0.33600000000000002,"5min":0.27100000000000002,"15min":0.219},"max":{"1min":1.6240000000000001,"5min":6.3360000000000003,"15min":6.3360000000000003},"last":2.4159999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:32:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.88800000000000001,"15min":0.57299999999999995},"min":{"1min":0.29299999999999998,"5min":0.28899999999999998,"15min":0.27000000000000002},"max":{"1min":1.212,"5min":7.2750000000000004,"15min":7.2750000000000004},"last":1.871},{"interface":"front","average":{"1min":0.5,"5min":0.746,"15min":0.53300000000000003},"min":{"1min":0.36799999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76500000000000001,"5min":4.585,"15min":4.585},"last":2.3199999999999998}]}]},{"osd":6,"up_from":144,"seq":618475290707,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1563056,"kb_used_data":1256296,"kb_used_omap":7,"kb_used_meta":306744,"kb_avail":92182096,"statfs":{"total":95995035648,"available":94394466304,"internally_reserved":0,"allocated":1286447104,"data_stored":1285960290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":314105943},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:32:42 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.575,"15min":1.9330000000000001},"min":{"1min":0.27000000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":7.8860000000000001,"5min":30.629999999999999,"15min":30.629999999999999},"last":0.315},{"interface":"front","average":{"1min":0.51800000000000002,"5min":1.179,"15min":1.4219999999999999},"min":{"1min":0.26500000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.4099999999999999,"5min":30.597000000000001,"15min":30.597000000000001},"last":0.42999999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:32:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":1.282,"15min":0.90800000000000003},"min":{"1min":0.32500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.6779999999999999,"5min":30.984000000000002,"15min":30.984000000000002},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":1.399,"15min":1.137},"min":{"1min":0.25700000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":5.375,"5min":30.571000000000002,"15min":30.571000000000002},"last":1.3240000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:32:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.0209999999999999,"15min":1.079},"min":{"1min":0.28999999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":18.353000000000002,"5min":18.353000000000002,"15min":18.353000000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.997,"15min":0.84699999999999998},"min":{"1min":0.29499999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.7450000000000001,"5min":11.164999999999999,"15min":11.164999999999999},"last":0.34899999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:32:42 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.7190000000000001,"15min":2.028},"min":{"1min":0.33100000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":16.831,"5min":61.703000000000003,"15min":61.703000000000003},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.554,"5min":1.788,"15min":2.0350000000000001},"min":{"1min":0.35299999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":16.719000000000001,"5min":61.814,"15min":61.814},"last":0.46600000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:32:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.64600000000000002,"15min":0.53900000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.23699999999999999},"max":{"1min":0.90200000000000002,"5min":5.4950000000000001,"15min":5.4950000000000001},"last":0.495},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.64500000000000002,"15min":0.64400000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.048,"5min":4.6390000000000002,"15min":4.6390000000000002},"last":0.36399999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:32:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.66300000000000003,"15min":0.53000000000000003},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.24399999999999999},"max":{"1min":1.7929999999999999,"5min":4.6619999999999999,"15min":4.6619999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.67100000000000004,"15min":0.49299999999999999},"min":{"1min":0.31900000000000001,"5min":0.29699999999999999,"15min":0.26800000000000002},"max":{"1min":2.794,"5min":2.794,"15min":2.794},"last":0.64300000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:32:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63200000000000001,"15min":0.53000000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.7040000000000002,"5min":2.7040000000000002,"15min":2.7040000000000002},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.748,"15min":0.56599999999999995},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.95999999999999996,"5min":5.5309999999999997,"15min":5.5309999999999997},"last":0.56299999999999994}]}]},{"osd":5,"up_from":39,"seq":167503724699,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3967304,"kb_used_data":3669952,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":89777848,"statfs":{"total":95995035648,"available":91932516352,"internally_reserved":0,"allocated":3758030848,"data_stored":3757520336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:33:00 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.4099999999999999,"15min":1.4390000000000001},"min":{"1min":0.33600000000000002,"5min":0.23000000000000001,"15min":0.21099999999999999},"max":{"1min":3.2410000000000001,"5min":26.288,"15min":52.033999999999999},"last":1.3999999999999999},{"interface":"front","average":{"1min":0.998,"5min":1.246,"15min":1.2589999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":5.4420000000000002,"5min":24.178000000000001,"15min":52.167999999999999},"last":0.438}]},{"osd":1,"last update":"Tue Sep 10 18:33:00 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.4470000000000001,"15min":0.91500000000000004},"min":{"1min":0.32700000000000001,"5min":0.315,"15min":0.22600000000000001},"max":{"1min":3.2090000000000001,"5min":24.16,"15min":24.16},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.3340000000000001,"15min":0.98299999999999998},"min":{"1min":0.43099999999999999,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":3.234,"5min":24.129999999999999,"15min":24.129999999999999},"last":0.45800000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:33:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4980000000000002,"5min":1.7330000000000001,"15min":1.1779999999999999},"min":{"1min":0.309,"5min":0.245,"15min":0.24299999999999999},"max":{"1min":54.530000000000001,"5min":54.530000000000001,"15min":54.530000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":3.3069999999999999,"5min":1.6990000000000001,"15min":1.1379999999999999},"min":{"1min":0.28999999999999998,"5min":0.247,"15min":0.247},"max":{"1min":54.514000000000003,"5min":54.514000000000003,"15min":54.514000000000003},"last":0.33900000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:33:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.5900000000000001,"15min":1.298},"min":{"1min":0.372,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.8980000000000001,"5min":30.475000000000001,"15min":31.207000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.84399999999999997,"5min":1.5089999999999999,"15min":1.1819999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.29699999999999999},"max":{"1min":1.9850000000000001,"5min":30.172000000000001,"15min":31.199000000000002},"last":2.3300000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:33:00 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.67700000000000005,"15min":0.47499999999999998},"min":{"1min":0.27700000000000002,"5min":0.22900000000000001,"15min":0.20799999999999999},"max":{"1min":0.72599999999999998,"5min":4.5199999999999996,"15min":4.5199999999999996},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.81200000000000006,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.24199999999999999,"15min":0.22700000000000001},"max":{"1min":2.786,"5min":6.798,"15min":6.798},"last":0.48999999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.79900000000000004,"15min":0.63900000000000001},"min":{"1min":0.28599999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.7469999999999999,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.81200000000000006,"15min":0.63},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.82799999999999996,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.42099999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.79600000000000004,"15min":0.55800000000000005},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.22500000000000001},"max":{"1min":0.84399999999999997,"5min":5.569,"15min":5.569},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.67500000000000004,"15min":0.51400000000000001},"min":{"1min":0.309,"5min":0.25900000000000001,"15min":0.24299999999999999},"max":{"1min":2.6259999999999999,"5min":3.1819999999999999,"15min":3.1819999999999999},"last":0.58599999999999997}]}]},{"osd":4,"up_from":33,"seq":141733920927,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1550656,"kb_used_data":1240824,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92194496,"statfs":{"total":95995035648,"available":94407163904,"internally_reserved":0,"allocated":1270603776,"data_stored":1270100217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:32:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":1.2529999999999999,"15min":0.99199999999999999},"min":{"1min":0.22900000000000001,"5min":0.21199999999999999,"15min":0.188},"max":{"1min":1.855,"5min":16.236999999999998,"15min":31.585000000000001},"last":2.4260000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":1.0269999999999999,"15min":0.95499999999999996},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.2730000000000001,"5min":16.280999999999999,"15min":29.802},"last":1.2549999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:32:33 2024","interfaces":[{"interface":"back","average":{"1min":3.3820000000000001,"5min":1.375,"15min":0.93100000000000005},"min":{"1min":0.376,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":50.045000000000002,"5min":50.045000000000002,"15min":50.045000000000002},"last":0.497},{"interface":"front","average":{"1min":3.556,"5min":1.647,"15min":0.996},"min":{"1min":0.315,"5min":0.28699999999999998,"15min":0.22900000000000001},"max":{"1min":50.283000000000001,"5min":50.283000000000001,"15min":50.283000000000001},"last":1.603}]},{"osd":2,"last update":"Tue Sep 10 18:32:33 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.91300000000000003,"15min":0.83199999999999996},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":1.2470000000000001,"5min":5.2380000000000004,"15min":13.898999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.89600000000000002,"15min":0.84999999999999998},"min":{"1min":0.25900000000000001,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":1.7609999999999999,"5min":4.1950000000000003,"15min":14.51},"last":1.8680000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:32:33 2024","interfaces":[{"interface":"back","average":{"1min":3.492,"5min":1.897,"15min":1.2450000000000001},"min":{"1min":0.33100000000000002,"5min":0.26300000000000001,"15min":0.249},"max":{"1min":49.990000000000002,"5min":49.990000000000002,"15min":49.990000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":3.399,"5min":1.9159999999999999,"15min":1.278},"min":{"1min":0.371,"5min":0.28000000000000003,"15min":0.255},"max":{"1min":50.764000000000003,"5min":50.764000000000003,"15min":50.764000000000003},"last":0.66500000000000004}]},{"osd":5,"last update":"Tue Sep 10 18:32:54 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.92800000000000005,"15min":0.55800000000000005},"min":{"1min":0.249,"5min":0.249,"15min":0.20499999999999999},"max":{"1min":1.484,"5min":22.861000000000001,"15min":22.861000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.95299999999999996,"15min":0.59199999999999997},"min":{"1min":0.32300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":2.5950000000000002,"5min":19.611999999999998,"15min":19.611999999999998},"last":0.39800000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.623,"15min":0.46500000000000002},"min":{"1min":0.316,"5min":0.27400000000000002,"15min":0.26400000000000001},"max":{"1min":1.899,"5min":6.7990000000000004,"15min":6.7990000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.66500000000000004,"15min":0.51700000000000002},"min":{"1min":0.307,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.65000000000000002,"5min":7.0300000000000002,"15min":7.0300000000000002},"last":0.38100000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.65500000000000003,"15min":0.52800000000000002},"min":{"1min":0.35099999999999998,"5min":0.28799999999999998,"15min":0.255},"max":{"1min":0.755,"5min":7.0670000000000002,"15min":7.0670000000000002},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.625,"5min":0.69999999999999996,"15min":0.55000000000000004},"min":{"1min":0.35299999999999998,"5min":0.28699999999999998,"15min":0.247},"max":{"1min":1.8200000000000001,"5min":6.5330000000000004,"15min":6.5330000000000004},"last":0.372}]}]},{"osd":3,"up_from":27,"seq":115964117157,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3900432,"kb_used_data":3314696,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":89844720,"statfs":{"total":95995035648,"available":92000993280,"internally_reserved":0,"allocated":3394248704,"data_stored":3393730149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:33:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.272,"15min":1.181},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":4.6079999999999997,"5min":5.4560000000000004,"15min":8.5530000000000008},"last":1.7789999999999999},{"interface":"front","average":{"1min":1.637,"5min":1.393,"15min":1.25},"min":{"1min":0.29899999999999999,"5min":0.193,"15min":0.193},"max":{"1min":8.2110000000000003,"5min":8.2110000000000003,"15min":13.659000000000001},"last":0.54800000000000004}]},{"osd":1,"last update":"Tue Sep 10 18:33:11 2024","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.9670000000000001,"15min":1.327},"min":{"1min":0.34200000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.149,"5min":35.406999999999996,"15min":35.406999999999996},"last":1.407},{"interface":"front","average":{"1min":1.409,"5min":1.974,"15min":1.2649999999999999},"min":{"1min":0.28100000000000003,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":5.0720000000000001,"5min":36.902000000000001,"15min":36.902000000000001},"last":0.62}]},{"osd":2,"last update":"Tue Sep 10 18:33:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.079,"15min":1.0249999999999999},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":5.3810000000000002,"5min":5.3810000000000002,"15min":7.7960000000000003},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.4810000000000001,"15min":1.1890000000000001},"min":{"1min":0.26000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":4.4649999999999999,"5min":14.997999999999999,"15min":14.997999999999999},"last":0.28000000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":1.165,"15min":1.1220000000000001},"min":{"1min":0.34399999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.843,"5min":5.1139999999999999,"15min":23.684999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.623,"5min":1.054,"15min":1.0569999999999999},"min":{"1min":0.41099999999999998,"5min":0.29899999999999999,"15min":0.27200000000000002},"max":{"1min":0.99199999999999999,"5min":5.2160000000000002,"15min":23.649000000000001},"last":0.42899999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:32:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.333,"15min":1.218},"min":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.34499999999999997},"max":{"1min":4.0730000000000004,"5min":5.2850000000000001,"15min":23.719999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.177},"min":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.33100000000000002},"max":{"1min":6.2249999999999996,"5min":6.2249999999999996,"15min":23.707000000000001},"last":0.40000000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.639,"15min":1.133},"min":{"1min":0.379,"5min":0.32200000000000001,"15min":0.317},"max":{"1min":1.0640000000000001,"5min":31.815999999999999,"15min":31.815999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.90800000000000003,"5min":1.7849999999999999,"15min":1.4830000000000001},"min":{"1min":0.38500000000000001,"5min":0.33400000000000002,"15min":0.315},"max":{"1min":3.177,"5min":31.742000000000001,"15min":31.742000000000001},"last":0.69699999999999995}]},{"osd":7,"last update":"Tue Sep 10 18:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":1.2729999999999999,"15min":1.155},"min":{"1min":0.40799999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.875,"5min":5.1449999999999996,"15min":23.748999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.22,"15min":1.1819999999999999},"min":{"1min":0.47399999999999998,"5min":0.35199999999999998,"15min":0.307},"max":{"1min":12.612,"5min":12.612,"15min":25.058},"last":0.60899999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411497,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1546996,"kb_used_data":1241324,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92198156,"statfs":{"total":95995035648,"available":94410911744,"internally_reserved":0,"allocated":1271115776,"data_stored":1270624521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.6479999999999999,"15min":1.278},"min":{"1min":0.217,"5min":0.214,"15min":0.185},"max":{"1min":1.583,"5min":43.890000000000001,"15min":43.890000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":2.0379999999999998,"15min":1.4650000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.16900000000000001},"max":{"1min":2.3439999999999999,"5min":44.331000000000003,"15min":44.331000000000003},"last":0.73199999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":2.0070000000000001,"15min":1.343},"min":{"1min":0.20200000000000001,"5min":0.193,"15min":0.193},"max":{"1min":2.5649999999999999,"5min":44.575000000000003,"15min":44.575000000000003},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":1.8440000000000001,"15min":1.1519999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":1.258,"5min":44.368000000000002,"15min":44.368000000000002},"last":0.25900000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:33:07 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.669,"15min":1.4510000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.251},"max":{"1min":6.5350000000000001,"5min":15.069000000000001,"15min":60.786999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.645,"15min":1.498},"min":{"1min":0.23499999999999999,"5min":0.214,"15min":0.214},"max":{"1min":4.694,"5min":15.055999999999999,"15min":60.116999999999997},"last":0.89000000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:32:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.2050000000000001,"15min":0.98599999999999999},"min":{"1min":0.30499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":12.555999999999999,"5min":12.555999999999999,"15min":12.555999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.129,"15min":0.91400000000000003},"min":{"1min":0.434,"5min":0.32900000000000001,"15min":0.317},"max":{"1min":1.298,"5min":7.0730000000000004,"15min":12.130000000000001},"last":0.52000000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:32:55 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.2969999999999999,"15min":0.96599999999999997},"min":{"1min":0.40699999999999997,"5min":0.30299999999999999,"15min":0.27500000000000002},"max":{"1min":6.3170000000000002,"5min":7.1609999999999996,"15min":12.106},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.4019999999999999,"15min":1.0149999999999999},"min":{"1min":0.371,"5min":0.32200000000000001,"15min":0.31900000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":0.59999999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.1419999999999999,"15min":1.127},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.6309999999999998,"5min":5.9989999999999997,"15min":5.9989999999999997},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":1.194,"15min":0.96199999999999997},"min":{"1min":0.435,"5min":0.38300000000000001,"15min":0.30099999999999999},"max":{"1min":1.2430000000000001,"5min":7.1349999999999998,"15min":7.1349999999999998},"last":0.66900000000000004}]},{"osd":7,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.1830000000000001,"15min":0.98999999999999999},"min":{"1min":0.38900000000000001,"5min":0.35399999999999998,"15min":0.30499999999999999},"max":{"1min":2.6030000000000002,"5min":5.1210000000000004,"15min":7.1639999999999997},"last":0.44400000000000001},{"interface":"front","average":{"1min":1.417,"5min":1.3899999999999999,"15min":1.0680000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.307},"max":{"1min":12.596,"5min":12.596,"15min":12.596},"last":0.57599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575022,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4029964,"kb_used_data":3728452,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":89715188,"statfs":{"total":95995035648,"available":91868352512,"internally_reserved":0,"allocated":3817934848,"data_stored":3817419603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:32:26 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.242,"15min":1.409},"min":{"1min":0.224,"5min":0.20399999999999999,"15min":0.16700000000000001},"max":{"1min":3.6579999999999999,"5min":14.079000000000001,"15min":80.456999999999994},"last":3.206},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.353,"15min":1.5089999999999999},"min":{"1min":0.23799999999999999,"5min":0.187,"15min":0.187},"max":{"1min":3.3959999999999999,"5min":7.3120000000000003,"15min":80.578000000000003},"last":1.1830000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:32:40 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":1.419,"15min":1.129},"min":{"1min":0.27300000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.093,"5min":7.2530000000000001,"15min":13.035},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":1.393,"15min":1.0569999999999999},"min":{"1min":0.24299999999999999,"5min":0.216,"15min":0.19900000000000001},"max":{"1min":2.9809999999999999,"5min":12.222,"15min":12.222},"last":1.97}]},{"osd":3,"last update":"Tue Sep 10 18:32:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.3620000000000001,"15min":1.4379999999999999},"min":{"1min":0.29299999999999998,"5min":0.23899999999999999,"15min":0.22600000000000001},"max":{"1min":3.71,"5min":14.121,"15min":43.076000000000001},"last":1.107},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.53,"15min":1.5269999999999999},"min":{"1min":0.32800000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":4.0990000000000002,"5min":7.3799999999999999,"15min":41.93},"last":1.252}]},{"osd":4,"last update":"Tue Sep 10 18:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.2709999999999999,"15min":0.97399999999999998},"min":{"1min":0.33700000000000002,"5min":0.32100000000000001,"15min":0.29799999999999999},"max":{"1min":3.3570000000000002,"5min":7.2859999999999996,"15min":7.2859999999999996},"last":1.29},{"interface":"front","average":{"1min":1.044,"5min":1.2470000000000001,"15min":1.0760000000000001},"min":{"1min":0.35599999999999998,"5min":0.30499999999999999,"15min":0.28299999999999997},"max":{"1min":3.9910000000000001,"5min":5.7750000000000004,"15min":19.393000000000001},"last":2.2759999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:32:56 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.5600000000000001,"15min":1.1240000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.0139999999999998,"5min":12.206,"15min":15.48},"last":2.1179999999999999},{"interface":"front","average":{"1min":0.999,"5min":1.4319999999999999,"15min":1.03},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.308},"max":{"1min":2.4769999999999999,"5min":9.9359999999999999,"15min":9.9359999999999999},"last":0.56599999999999995}]},{"osd":6,"last update":"Tue Sep 10 18:32:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":1.22,"15min":1.3080000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":2.1829999999999998,"5min":10.039,"15min":10.039},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.099,"5min":1.4019999999999999,"15min":1.3819999999999999},"min":{"1min":0.38100000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":4.6059999999999999,"5min":7.3369999999999997,"15min":7.3369999999999997},"last":6.2670000000000003}]},{"osd":7,"last update":"Tue Sep 10 18:32:40 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":1.284,"15min":1.05},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.34399999999999997},"max":{"1min":2.2050000000000001,"5min":10.005000000000001,"15min":15.545999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.405,"15min":1.0429999999999999},"min":{"1min":0.441,"5min":0.35699999999999998,"15min":0.313},"max":{"1min":5.056,"5min":7.359,"15min":7.359},"last":6.3049999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738547,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1963708,"kb_used_data":1658036,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91781444,"statfs":{"total":95995035648,"available":93984198656,"internally_reserved":0,"allocated":1697828864,"data_stored":1697329274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:32:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":1.8600000000000001,"15min":1.238},"min":{"1min":0.21199999999999999,"5min":0.20499999999999999,"15min":0.193},"max":{"1min":4.1989999999999998,"5min":33.130000000000003,"15min":33.130000000000003},"last":3.0779999999999998},{"interface":"front","average":{"1min":0.505,"5min":1.8939999999999999,"15min":1.335},"min":{"1min":0.23100000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":1.911,"5min":31.329999999999998,"15min":31.329999999999998},"last":3.5329999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:32:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":1.532,"15min":1.242},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.189},"max":{"1min":2.1230000000000002,"5min":30.219999999999999,"15min":30.219999999999999},"last":2.1930000000000001},{"interface":"front","average":{"1min":0.44,"5min":1.5369999999999999,"15min":1.2549999999999999},"min":{"1min":0.23100000000000001,"5min":0.224,"15min":0.22},"max":{"1min":1.3340000000000001,"5min":30.129999999999999,"15min":30.129999999999999},"last":0.31900000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.464,"15min":1.3640000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.157,"5min":8.5709999999999997,"15min":23.143999999999998},"last":0.35199999999999998},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.583,"15min":1.343},"min":{"1min":0.28799999999999998,"5min":0.249,"15min":0.23899999999999999},"max":{"1min":3.5720000000000001,"5min":9.3360000000000003,"15min":23.035},"last":2.1059999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":1.157,"15min":1.0980000000000001},"min":{"1min":0.34399999999999997,"5min":0.29299999999999998,"15min":0.246},"max":{"1min":3.6040000000000001,"5min":8.3699999999999992,"15min":17.151},"last":2.0800000000000001},{"interface":"front","average":{"1min":0.87,"5min":1.075,"15min":1.0920000000000001},"min":{"1min":0.47499999999999998,"5min":0.27500000000000002,"15min":0.25900000000000001},"max":{"1min":3.552,"5min":4.8460000000000001,"15min":18.728999999999999},"last":2.2970000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":1.113,"15min":1.1839999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.4159999999999999,"5min":4.7220000000000004,"15min":18.77},"last":2.3519999999999999},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.264,"15min":1.1259999999999999},"min":{"1min":0.39700000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.609,"5min":6.6079999999999997,"15min":14.991},"last":0.42599999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":1.194,"15min":1.2689999999999999},"min":{"1min":0.378,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":3.6320000000000001,"5min":8.3940000000000001,"15min":8.3940000000000001},"last":2.0470000000000002},{"interface":"front","average":{"1min":0.84699999999999998,"5min":1.236,"15min":1.3560000000000001},"min":{"1min":0.44900000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.585,"5min":9.7859999999999996,"15min":9.7859999999999996},"last":2.3250000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":1.242,"15min":1.097},"min":{"1min":0.48499999999999999,"5min":0.32200000000000001,"15min":0.26700000000000002},"max":{"1min":3.4580000000000002,"5min":5.0430000000000001,"15min":13.711},"last":2.383},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.2,"15min":1.0740000000000001},"min":{"1min":0.437,"5min":0.36199999999999999,"15min":0.33500000000000002},"max":{"1min":1.3879999999999999,"5min":9.6549999999999994,"15min":15.032},"last":0.45600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1708261376,"data_stored":1708261376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3824549888,"data_stored":3824549888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1277427712,"data_stored":1277427712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3407740928,"data_stored":3407740928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1272774656,"data_stored":1272774656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3800236032,"data_stored":3800236032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1296498688,"data_stored":1296498688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3813277696,"data_stored":3813277696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:33:21.392 INFO:tasks.thrashosds.thrasher:Setting 6.17 to [3, 7, 5] 2024-09-10T18:33:21.392 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.17', '3', '7', '5'] 2024-09-10T18:33:21.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pg-upmap 6.17 3 7 5 2024-09-10T18:33:21.407 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:21.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:21.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:21.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:21.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:21.562 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:21.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:21.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:21.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:21.710 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:21.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:21.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:33:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:33:21] "GET /metrics HTTP/1.1" 200 224462 "" "Prometheus/2.43.0" 2024-09-10T18:33:21.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:21.838 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:21.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:21.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:21.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:21.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:21.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:22.069 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 69 16 111328 111312 100.772 70.75 0.0549096 0.0099186 2024-09-10T18:33:22.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:22.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:21 smithi204 ceph-mon[32357]: osdmap e267: 8 total, 8 up, 8 in 2024-09-10T18:33:22.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:21 smithi204 ceph-mon[32357]: pgmap v531: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-09-10T18:33:22.116 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:22.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:22.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:22.218 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:22.248 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:22.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:22.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:22.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:21 smithi028 ceph-mon[39220]: osdmap e267: 8 total, 8 up, 8 in 2024-09-10T18:33:22.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:21 smithi028 ceph-mon[39220]: pgmap v531: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-09-10T18:33:22.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:21 smithi028 ceph-mon[29291]: osdmap e267: 8 total, 8 up, 8 in 2024-09-10T18:33:22.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:21 smithi028 ceph-mon[29291]: pgmap v531: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-09-10T18:33:22.349 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:22.382 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:22.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:22.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:22.432 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:33:22.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:22.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:22.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:22.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:33:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:33:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:33:22.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:22.672 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:22.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:22.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:22.725 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:33:22.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:22.794 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:22.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:22.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:22.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:22.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:22.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:22.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:23.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:23.070 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 70 16 112448 112432 100.332 70 0.0134087 0.00996163 2024-09-10T18:33:23.077 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:23.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:23.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:23.225 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:23.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:23.289 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:33:23.290 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:33:23.321 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:33:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:33:23.327 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:23.350 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:33:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:33:23.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:23.358 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:23.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:23.381 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:33:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:33:23.420 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:33:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:33:23.452 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:33:23.452 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:33:23.459 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:23.484 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:33:23.484 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:33:23.494 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:23.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:23.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:23.514 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:33:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:33:23.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:23.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:23.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:23.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:23.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:23.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:23.877 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:23.909 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:23.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:23.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:24.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:24.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:24.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:24.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:24.070 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 71 16 113599 113583 99.9321 71.9375 0.00532577 0.0100015 2024-09-10T18:33:24.151 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:24.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:24.191 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:24.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:24.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:24 smithi028 ceph-mon[29291]: pgmap v532: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-10T18:33:24.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:24 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4218852676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:33:24.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:24 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:33:24.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:33:24.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:24 smithi028 ceph-mon[39220]: pgmap v532: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-10T18:33:24.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:24 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4218852676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:33:24.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:24 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:33:24.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:33:24.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:24.327 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:24.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:24.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:24 smithi204 ceph-mon[32357]: pgmap v532: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-10T18:33:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:24 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4218852676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:33:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:24 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:33:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:33:24.428 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:24.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:33:24.458 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:24.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:24.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:24.560 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:24.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:24.596 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:24.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:24.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:24.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:24.752 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:24.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:24.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:24.855 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:33:24 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:33:24.900 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:24.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:24.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:25.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:25.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:25.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:25.070 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 72 16 114732 114716 99.5277 70.8125 0.00921501 0.0100408 2024-09-10T18:33:25.142 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:25.175 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:25.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:25.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:25.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:25.313 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:25.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:25.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:25.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:25.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:25.452 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:25.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:25.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:25.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:25.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:25.686 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:25.717 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:25.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:25.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:25.819 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:25.851 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:25.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:25.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:25.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:25.993 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:25.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:26.072 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 73 16 115929 115913 99.1892 74.8125 0.0981852 0.0100753 2024-09-10T18:33:26.094 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:26.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:26.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:26.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:26.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:26.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:26.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:26.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:26 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:26.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:26 smithi028 ceph-mon[29291]: pgmap v533: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-10T18:33:26.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:26 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:26.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:26 smithi028 ceph-mon[39220]: pgmap v533: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-10T18:33:26.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:26 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:26 smithi204 ceph-mon[32357]: pgmap v533: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-10T18:33:26.386 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:26.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:26.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:26.488 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:26.519 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:26.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:26.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:26.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:26.662 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:26.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:26.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:26.764 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:26.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:26.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:26.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:26.908 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:26.938 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:33:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-10T18:33:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-10T18:33:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-10T18:33:26.939 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:33:26.944 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:26.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:26.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:27.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:27.071 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 74 16 116908 116892 98.6758 61.1875 0.0103402 0.0101299 2024-09-10T18:33:27.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:27.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:27.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:27 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:33:27.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:27 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:33:27.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:27 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:33:27.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:27 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:33:27.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:27.235 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:27.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:27.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:27.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:27 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:33:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:27 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:33:27.379 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:27.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:27.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:27.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:33:27.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:27.535 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:27.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:27.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:27.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:27.669 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:27.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:27.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:27.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:27.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:27.918 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:27.952 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:27.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:27.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:28.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:28.071 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 75 16 117958 117942 98.2342 65.625 0.0107682 0.0101748 2024-09-10T18:33:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:set 6.17 pg_upmap mapping to [3,7,5] 2024-09-10T18:33:28.097 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:28.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:28.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:28.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:28.229 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:28.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:28.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:28.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-10T18:33:28.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:28 smithi204 ceph-mon[32357]: pgmap v534: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-10T18:33:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:28 smithi204 ceph-mon[32357]: osdmap e268: 8 total, 8 up, 8 in 2024-09-10T18:33:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:28 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/164466038' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [3, 7, 5]}]: dispatch 2024-09-10T18:33:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:28 smithi204 ceph-mon[32357]: from='client.35384 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [3, 7, 5]}]: dispatch 2024-09-10T18:33:28.377 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:28.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:28.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:28.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:28.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:28 smithi028 ceph-mon[39220]: pgmap v534: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-10T18:33:28.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:28 smithi028 ceph-mon[39220]: osdmap e268: 8 total, 8 up, 8 in 2024-09-10T18:33:28.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:28 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/164466038' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [3, 7, 5]}]: dispatch 2024-09-10T18:33:28.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:28 smithi028 ceph-mon[39220]: from='client.35384 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [3, 7, 5]}]: dispatch 2024-09-10T18:33:28.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:28 smithi028 ceph-mon[29291]: pgmap v534: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-10T18:33:28.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:28 smithi028 ceph-mon[29291]: osdmap e268: 8 total, 8 up, 8 in 2024-09-10T18:33:28.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:28 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/164466038' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [3, 7, 5]}]: dispatch 2024-09-10T18:33:28.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:28 smithi028 ceph-mon[29291]: from='client.35384 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [3, 7, 5]}]: dispatch 2024-09-10T18:33:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:28.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:28.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:28.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:28.668 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:28.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:28.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:28.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:28.799 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:28.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:28.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:28.901 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:28.933 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:28.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:28.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:29.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:29.063 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:29.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:29.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:29.073 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 76 16 119062 119046 97.8479 69 0.0109131 0.0102142 2024-09-10T18:33:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:33:29.091 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:33:29.102 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:33:29.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:29 smithi028 ceph-mon[29291]: from='client.35384 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [3, 7, 5]}]': finished 2024-09-10T18:33:29.122 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:29 smithi028 ceph-mon[29291]: osdmap e269: 8 total, 8 up, 8 in 2024-09-10T18:33:29.165 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:29.195 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:29.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:29.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:29.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:29.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:29.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:29.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:29.444 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:29.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:29 smithi028 ceph-mon[39220]: from='client.35384 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [3, 7, 5]}]': finished 2024-09-10T18:33:29.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:29 smithi028 ceph-mon[39220]: osdmap e269: 8 total, 8 up, 8 in 2024-09-10T18:33:29.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:29.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:29.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:29.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:29 smithi204 ceph-mon[32357]: from='client.35384 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [3, 7, 5]}]': finished 2024-09-10T18:33:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:29 smithi204 ceph-mon[32357]: osdmap e269: 8 total, 8 up, 8 in 2024-09-10T18:33:29.620 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:29.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:29.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:29.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:29.752 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:29.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:29.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:29.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:29.890 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:29.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:29.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:29.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:30.024 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:30.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:30.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:30.073 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 77 16 120229 120213 97.5245 72.9375 0.00452903 0.010249 2024-09-10T18:33:30.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:30.165 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:30.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:30.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:30.267 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:30.300 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:30.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:30.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:30.402 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:30.436 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:30.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:30.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:30.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:30 smithi028 ceph-mon[39220]: pgmap v537: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 712 op/s 2024-09-10T18:33:30.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:30 smithi028 ceph-mon[39220]: osdmap e270: 8 total, 8 up, 8 in 2024-09-10T18:33:30.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:30 smithi028 ceph-mon[29291]: pgmap v537: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 712 op/s 2024-09-10T18:33:30.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:30 smithi028 ceph-mon[29291]: osdmap e270: 8 total, 8 up, 8 in 2024-09-10T18:33:30.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:30.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:30.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:30 smithi204 ceph-mon[32357]: pgmap v537: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 712 op/s 2024-09-10T18:33:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:30 smithi204 ceph-mon[32357]: osdmap e270: 8 total, 8 up, 8 in 2024-09-10T18:33:30.675 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:30.708 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:30.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:30.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:30.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:30.847 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:30.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:30.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:30.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:30.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:30.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:31.073 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 78 16 121392 121376 97.206 72.6875 0.00532245 0.0102803 2024-09-10T18:33:31.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:31.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:31.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:31.240 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:31.274 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:31.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:31.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:31.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:31.417 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:31.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:31.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:31.563 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:31.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:31.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:31.607 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:33:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:33:31] "GET /metrics HTTP/1.1" 200 224417 "" "Prometheus/2.43.0" 2024-09-10T18:33:31.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:31.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:31.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:31.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:31.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:31.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:31.942 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:31.973 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:31.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:31.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:32.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:31 smithi028 ceph-mon[39220]: osdmap e271: 8 total, 8 up, 8 in 2024-09-10T18:33:32.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:31 smithi028 ceph-mon[39220]: pgmap v540: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T18:33:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:31 smithi028 ceph-mon[29291]: osdmap e271: 8 total, 8 up, 8 in 2024-09-10T18:33:32.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:31 smithi028 ceph-mon[29291]: pgmap v540: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T18:33:32.073 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 79 16 122576 122560 96.9123 74 0.0290615 0.0103141 2024-09-10T18:33:32.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:32.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:31 smithi204 ceph-mon[32357]: osdmap e271: 8 total, 8 up, 8 in 2024-09-10T18:33:32.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:31 smithi204 ceph-mon[32357]: pgmap v540: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T18:33:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:32.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:32.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:32.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:32.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:32.258 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:32.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:32.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:32.398 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:32.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:32.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:32.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:32.547 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:32.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:32.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:32.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:33:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:33:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:33:32.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:32.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:32.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:32.827 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:32.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:32.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:32.970 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:32.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:32.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:33.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:32 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:33:33.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:32 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:33:33.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:32 smithi028 ceph-mon[39220]: osdmap e272: 8 total, 8 up, 8 in 2024-09-10T18:33:33.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:32 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:33:33.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:32 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:33:33.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:32 smithi028 ceph-mon[29291]: osdmap e272: 8 total, 8 up, 8 in 2024-09-10T18:33:33.072 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:33.073 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:33:33.072711+0000 min lat: 0.00134629 max lat: 0.235622 avg lat: 0.0103415 2024-09-10T18:33:33.073 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:33:33.073 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 80 16 123759 123743 96.6251 73.9375 0.00379638 0.0103415 2024-09-10T18:33:33.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:32 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:33:33.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:32 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:33:33.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:32 smithi204 ceph-mon[32357]: osdmap e272: 8 total, 8 up, 8 in 2024-09-10T18:33:33.112 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:33.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:33.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:33.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:33.252 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:33.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:33.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:33.354 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:33.394 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:33.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:33.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:33.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:33.534 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:33.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:33.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:33.669 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:33.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:33.771 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:33.800 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:33.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:33.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:33.902 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:33.916 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:33:33.917 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:33:33.917 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-10T18:33:33.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:33:33.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:33.952 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:33.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:34.037 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:33 smithi028 ceph-mon[39220]: pgmap v542: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 70 MiB/s wr, 1.11k op/s 2024-09-10T18:33:34.037 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:33 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:33:34.037 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:33 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:33:34.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:33 smithi028 ceph-mon[29291]: pgmap v542: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 70 MiB/s wr, 1.11k op/s 2024-09-10T18:33:34.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:33 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:33:34.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:33 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:33:34.053 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:34.073 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 81 16 124819 124803 96.2501 66.25 0.00486741 0.0103848 2024-09-10T18:33:34.082 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:34.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:34.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:34.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:33 smithi204 ceph-mon[32357]: pgmap v542: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 70 MiB/s wr, 1.11k op/s 2024-09-10T18:33:34.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:33 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:33:34.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:33 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:33:34.106 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:33:34.106 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-10T18:33:34.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-10T18:33:34.106 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:33:34.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.107 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29819 7.e 7:7d2523b5:::benchmark_data_smithi028_77856_object29818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:33:34.107 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.539630+0000", 2024-09-10T18:33:34.107 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 577.49149218800005, 2024-09-10T18:33:34.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25094886500000002, 2024-09-10T18:33:34.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.107 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:33:34.107 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:33:34.107 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:33:34.107 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:33:34.107 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29819 2024-09-10T18:33:34.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.108 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.539630+0000", 2024-09-10T18:33:34.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.539630+0000", 2024-09-10T18:33:34.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.539627+0000", 2024-09-10T18:33:34.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:33:34.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.539654+0000", 2024-09-10T18:33:34.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.614e-05 2024-09-10T18:33:34.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.539665+0000", 2024-09-10T18:33:34.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1272e-05 2024-09-10T18:33:34.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.539670+0000", 2024-09-10T18:33:34.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5670000000000002e-06 2024-09-10T18:33:34.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.539791+0000", 2024-09-10T18:33:34.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012051 2024-09-10T18:33:34.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.539913+0000", 2024-09-10T18:33:34.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012243599999999999 2024-09-10T18:33:34.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:33:34.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.540029+0000", 2024-09-10T18:33:34.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000115837 2024-09-10T18:33:34.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.542070+0000", 2024-09-10T18:33:34.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020411959999999999 2024-09-10T18:33:34.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:33:34.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.542345+0000", 2024-09-10T18:33:34.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027425000000000003 2024-09-10T18:33:34.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790496+0000", 2024-09-10T18:33:34.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.248150961 2024-09-10T18:33:34.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:33:34.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790543+0000", 2024-09-10T18:33:34.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7553999999999999e-05 2024-09-10T18:33:34.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790578+0000", 2024-09-10T18:33:34.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5324000000000003e-05 2024-09-10T18:33:34.114 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.114 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.114 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.115 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29835 7.8 7:10b6d885:::benchmark_data_smithi028_77856_object29834:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:33:34.115 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.545862+0000", 2024-09-10T18:33:34.115 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 577.48526007199996, 2024-09-10T18:33:34.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.243747676, 2024-09-10T18:33:34.115 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.115 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:33:34.115 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:33:34.115 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:33:34.115 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:33:34.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29835 2024-09-10T18:33:34.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.116 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545862+0000", 2024-09-10T18:33:34.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545862+0000", 2024-09-10T18:33:34.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545860+0000", 2024-09-10T18:33:34.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:33:34.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545975+0000", 2024-09-10T18:33:34.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000115219 2024-09-10T18:33:34.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545986+0000", 2024-09-10T18:33:34.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1394e-05 2024-09-10T18:33:34.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.545993+0000", 2024-09-10T18:33:34.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0430000000000004e-06 2024-09-10T18:33:34.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546086+0000", 2024-09-10T18:33:34.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3540000000000002e-05 2024-09-10T18:33:34.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546185+0000", 2024-09-10T18:33:34.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9204999999999996e-05 2024-09-10T18:33:34.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,7", 2024-09-10T18:33:34.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546298+0000", 2024-09-10T18:33:34.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000112613 2024-09-10T18:33:34.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:33:34.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547603+0000", 2024-09-10T18:33:34.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013053979999999999 2024-09-10T18:33:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789116+0000", 2024-09-10T18:33:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24151288300000001 2024-09-10T18:33:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789526+0000", 2024-09-10T18:33:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00040943499999999999 2024-09-10T18:33:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:33:34.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789569+0000", 2024-09-10T18:33:34.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3273999999999999e-05 2024-09-10T18:33:34.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789609+0000", 2024-09-10T18:33:34.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0463999999999997e-05 2024-09-10T18:33:34.121 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.122 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.122 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.122 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29836 7.e 7:7efd3370:::benchmark_data_smithi028_77856_object29835:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:33:34.122 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.546009+0000", 2024-09-10T18:33:34.122 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 577.48511296300001, 2024-09-10T18:33:34.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24472428600000001, 2024-09-10T18:33:34.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.122 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:33:34.122 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:33:34.122 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:33:34.123 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:33:34.123 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29836 2024-09-10T18:33:34.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.123 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546009+0000", 2024-09-10T18:33:34.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546009+0000", 2024-09-10T18:33:34.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546008+0000", 2024-09-10T18:33:34.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:33:34.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546030+0000", 2024-09-10T18:33:34.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2206000000000001e-05 2024-09-10T18:33:34.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546038+0000", 2024-09-10T18:33:34.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7430000000000002e-06 2024-09-10T18:33:34.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546046+0000", 2024-09-10T18:33:34.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.405e-06 2024-09-10T18:33:34.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546430+0000", 2024-09-10T18:33:34.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00038401400000000001 2024-09-10T18:33:34.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546549+0000", 2024-09-10T18:33:34.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000118958 2024-09-10T18:33:34.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:33:34.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546680+0000", 2024-09-10T18:33:34.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013058199999999999 2024-09-10T18:33:34.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548016+0000", 2024-09-10T18:33:34.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013362160000000001 2024-09-10T18:33:34.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790618+0000", 2024-09-10T18:33:34.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24260189800000001 2024-09-10T18:33:34.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:33:34.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790633+0000", 2024-09-10T18:33:34.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5290000000000001e-05 2024-09-10T18:33:34.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:33:34.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790672+0000", 2024-09-10T18:33:34.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8587999999999997e-05 2024-09-10T18:33:34.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790733+0000", 2024-09-10T18:33:34.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1480000000000001e-05 2024-09-10T18:33:34.129 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.129 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.129 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.129 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:29831 7.a e100/93 7:5aa0b980:::benchmark_data_smithi028_77856_object29830:head v 100'1871, mlcod=100'1869)", 2024-09-10T18:33:34.129 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.546864+0000", 2024-09-10T18:33:34.129 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 577.48425728500001, 2024-09-10T18:33:34.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.242727832, 2024-09-10T18:33:34.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.130 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:33:34.130 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546864+0000", 2024-09-10T18:33:34.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546864+0000", 2024-09-10T18:33:34.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546863+0000", 2024-09-10T18:33:34.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:33:34.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546893+0000", 2024-09-10T18:33:34.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0133000000000001e-05 2024-09-10T18:33:34.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546902+0000", 2024-09-10T18:33:34.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0380000000000005e-06 2024-09-10T18:33:34.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546907+0000", 2024-09-10T18:33:34.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0200000000000002e-06 2024-09-10T18:33:34.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.546986+0000", 2024-09-10T18:33:34.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8712000000000001e-05 2024-09-10T18:33:34.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547004+0000", 2024-09-10T18:33:34.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8357000000000001e-05 2024-09-10T18:33:34.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:33:34.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789563+0000", 2024-09-10T18:33:34.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24255862 2024-09-10T18:33:34.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789592+0000", 2024-09-10T18:33:34.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9212000000000001e-05 2024-09-10T18:33:34.134 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.134 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.134 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.135 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:29834 7.5 e100/93 7:a74714fe:::benchmark_data_smithi028_77856_object29833:head v 100'1873, mlcod=100'1871)", 2024-09-10T18:33:34.135 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.547005+0000", 2024-09-10T18:33:34.135 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 577.48411701999999, 2024-09-10T18:33:34.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24378228900000001, 2024-09-10T18:33:34.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.135 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:33:34.135 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547005+0000", 2024-09-10T18:33:34.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547005+0000", 2024-09-10T18:33:34.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547004+0000", 2024-09-10T18:33:34.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:33:34.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547045+0000", 2024-09-10T18:33:34.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1198000000000001e-05 2024-09-10T18:33:34.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547054+0000", 2024-09-10T18:33:34.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8040000000000007e-06 2024-09-10T18:33:34.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547058+0000", 2024-09-10T18:33:34.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.707e-06 2024-09-10T18:33:34.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547151+0000", 2024-09-10T18:33:34.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2991999999999993e-05 2024-09-10T18:33:34.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.547180+0000", 2024-09-10T18:33:34.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8696000000000001e-05 2024-09-10T18:33:34.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:33:34.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790683+0000", 2024-09-10T18:33:34.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24350297100000001 2024-09-10T18:33:34.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790787+0000", 2024-09-10T18:33:34.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010414 2024-09-10T18:33:34.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.139 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.139 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.14808.0:29835 7.8 e100/93 ondisk, result = 0)", 2024-09-10T18:33:34.140 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.548280+0000", 2024-09-10T18:33:34.140 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 577.48284207100005, 2024-09-10T18:33:34.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24084339199999999, 2024-09-10T18:33:34.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:33:34.140 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548280+0000", 2024-09-10T18:33:34.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548280+0000", 2024-09-10T18:33:34.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548279+0000", 2024-09-10T18:33:34.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:33:34.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548284+0000", 2024-09-10T18:33:34.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5119999999999999e-06 2024-09-10T18:33:34.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548293+0000", 2024-09-10T18:33:34.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2609999999999999e-06 2024-09-10T18:33:34.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548298+0000", 2024-09-10T18:33:34.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.707e-06 2024-09-10T18:33:34.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789037+0000", 2024-09-10T18:33:34.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.240738535 2024-09-10T18:33:34.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789096+0000", 2024-09-10T18:33:34.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9354999999999997e-05 2024-09-10T18:33:34.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789123+0000", 2024-09-10T18:33:34.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7143000000000001e-05 2024-09-10T18:33:34.144 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.144 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.144 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.144 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29837 7.8 7:106d3607:::benchmark_data_smithi028_77856_object29836:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:33:34.144 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.548334+0000", 2024-09-10T18:33:34.144 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 577.48278735999997, 2024-09-10T18:33:34.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24611195, 2024-09-10T18:33:34.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:33:34.145 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:33:34.145 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:33:34.145 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:33:34.145 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29837 2024-09-10T18:33:34.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.145 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548334+0000", 2024-09-10T18:33:34.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548334+0000", 2024-09-10T18:33:34.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548332+0000", 2024-09-10T18:33:34.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:33:34.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548359+0000", 2024-09-10T18:33:34.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6567000000000002e-05 2024-09-10T18:33:34.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548401+0000", 2024-09-10T18:33:34.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1902e-05 2024-09-10T18:33:34.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548407+0000", 2024-09-10T18:33:34.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4500000000000001e-06 2024-09-10T18:33:34.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789639+0000", 2024-09-10T18:33:34.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24123214800000001 2024-09-10T18:33:34.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789814+0000", 2024-09-10T18:33:34.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017425399999999999 2024-09-10T18:33:34.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,7", 2024-09-10T18:33:34.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789956+0000", 2024-09-10T18:33:34.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014267300000000001 2024-09-10T18:33:34.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.794173+0000", 2024-09-10T18:33:34.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042164480000000002 2024-09-10T18:33:34.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:33:34.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.794180+0000", 2024-09-10T18:33:34.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9619999999999996e-06 2024-09-10T18:33:34.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.794367+0000", 2024-09-10T18:33:34.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000186947 2024-09-10T18:33:34.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:33:34.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.794413+0000", 2024-09-10T18:33:34.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6644000000000003e-05 2024-09-10T18:33:34.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.794446+0000", 2024-09-10T18:33:34.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3011e-05 2024-09-10T18:33:34.151 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.151 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.151 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.151 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.14808.0:29835 7.8 e100/93 ondisk, result = 0)", 2024-09-10T18:33:34.152 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.548378+0000", 2024-09-10T18:33:34.152 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 577.48274382700004, 2024-09-10T18:33:34.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.241239025, 2024-09-10T18:33:34.152 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:33:34.152 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548378+0000", 2024-09-10T18:33:34.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548378+0000", 2024-09-10T18:33:34.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548377+0000", 2024-09-10T18:33:34.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:33:34.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548382+0000", 2024-09-10T18:33:34.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7269999999999998e-06 2024-09-10T18:33:34.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548416+0000", 2024-09-10T18:33:34.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3991e-05 2024-09-10T18:33:34.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548419+0000", 2024-09-10T18:33:34.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.298e-06 2024-09-10T18:33:34.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789514+0000", 2024-09-10T18:33:34.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.241094586 2024-09-10T18:33:34.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789520+0000", 2024-09-10T18:33:34.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6680000000000004e-06 2024-09-10T18:33:34.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789617+0000", 2024-09-10T18:33:34.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6757999999999998e-05 2024-09-10T18:33:34.156 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.156 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.156 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.156 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:29838 7.e 7:7d62ddc4:::benchmark_data_smithi028_77856_object29837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-10T18:33:34.157 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.548432+0000", 2024-09-10T18:33:34.157 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 577.48268965399996, 2024-09-10T18:33:34.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25066205800000002, 2024-09-10T18:33:34.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.157 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:33:34.157 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:33:34.157 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:33:34.157 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:33:34.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29838 2024-09-10T18:33:34.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.157 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548432+0000", 2024-09-10T18:33:34.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548432+0000", 2024-09-10T18:33:34.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548431+0000", 2024-09-10T18:33:34.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:33:34.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548453+0000", 2024-09-10T18:33:34.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1889e-05 2024-09-10T18:33:34.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548462+0000", 2024-09-10T18:33:34.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3370000000000006e-06 2024-09-10T18:33:34.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548466+0000", 2024-09-10T18:33:34.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7809999999999999e-06 2024-09-10T18:33:34.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548524+0000", 2024-09-10T18:33:34.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7541999999999999e-05 2024-09-10T18:33:34.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.548653+0000", 2024-09-10T18:33:34.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129154 2024-09-10T18:33:34.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:33:34.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789234+0000", 2024-09-10T18:33:34.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.240580764 2024-09-10T18:33:34.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790805+0000", 2024-09-10T18:33:34.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001571486 2024-09-10T18:33:34.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:33:34.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790966+0000", 2024-09-10T18:33:34.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016126199999999999 2024-09-10T18:33:34.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.799018+0000", 2024-09-10T18:33:34.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080513049999999999 2024-09-10T18:33:34.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:33:34.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.799062+0000", 2024-09-10T18:33:34.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4209000000000003e-05 2024-09-10T18:33:34.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.799094+0000", 2024-09-10T18:33:34.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2149e-05 2024-09-10T18:33:34.163 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.163 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.163 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.164 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.14808.0:29819 7.e e100/93 ondisk, result = 0)", 2024-09-10T18:33:34.164 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.549107+0000", 2024-09-10T18:33:34.164 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 577.48201519199995, 2024-09-10T18:33:34.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.241481533, 2024-09-10T18:33:34.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:33:34.164 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549107+0000", 2024-09-10T18:33:34.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549107+0000", 2024-09-10T18:33:34.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549105+0000", 2024-09-10T18:33:34.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:33:34.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.549111+0000", 2024-09-10T18:33:34.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7409999999999998e-06 2024-09-10T18:33:34.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789574+0000", 2024-09-10T18:33:34.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.240462388 2024-09-10T18:33:34.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789580+0000", 2024-09-10T18:33:34.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4570000000000002e-06 2024-09-10T18:33:34.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790483+0000", 2024-09-10T18:33:34.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00090302299999999998 2024-09-10T18:33:34.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790490+0000", 2024-09-10T18:33:34.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.674e-06 2024-09-10T18:33:34.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.790588+0000", 2024-09-10T18:33:34.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8424000000000005e-05 2024-09-10T18:33:34.168 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.168 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.168 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.168 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:29844 7.5 e100/93 7:a491b2d4:::benchmark_data_smithi028_77856_object29843:head v 100'1874, mlcod=100'1871)", 2024-09-10T18:33:34.168 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:23:56.550888+0000", 2024-09-10T18:33:34.169 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 577.48023367999997, 2024-09-10T18:33:34.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.241289533, 2024-09-10T18:33:34.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.169 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:33:34.169 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.550888+0000", 2024-09-10T18:33:34.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.550888+0000", 2024-09-10T18:33:34.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.550886+0000", 2024-09-10T18:33:34.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:33:34.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.550927+0000", 2024-09-10T18:33:34.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1322999999999997e-05 2024-09-10T18:33:34.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.788973+0000", 2024-09-10T18:33:34.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23804528899999999 2024-09-10T18:33:34.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789001+0000", 2024-09-10T18:33:34.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8816e-05 2024-09-10T18:33:34.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789552+0000", 2024-09-10T18:33:34.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00055091099999999996 2024-09-10T18:33:34.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.789574+0000", 2024-09-10T18:33:34.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1883e-05 2024-09-10T18:33:34.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:33:34.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.792111+0000", 2024-09-10T18:33:34.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025364960000000001 2024-09-10T18:33:34.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:23:56.792178+0000", 2024-09-10T18:33:34.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6939000000000004e-05 2024-09-10T18:33:34.173 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.173 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.174 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.174 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181173 7.e 7:71a1ed00:::benchmark_data_smithi028_77856_object60073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:33:34.174 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.047791+0000", 2024-09-10T18:33:34.174 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 428.98333047599999, 2024-09-10T18:33:34.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14746416900000001, 2024-09-10T18:33:34.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:33:34.174 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:33:34.174 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:33:34.174 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:33:34.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181173 2024-09-10T18:33:34.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.175 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.047791+0000", 2024-09-10T18:33:34.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.047791+0000", 2024-09-10T18:33:34.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.047789+0000", 2024-09-10T18:33:34.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:33:34.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.047796+0000", 2024-09-10T18:33:34.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2169999999999999e-06 2024-09-10T18:33:34.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.047805+0000", 2024-09-10T18:33:34.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3190000000000001e-06 2024-09-10T18:33:34.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.047810+0000", 2024-09-10T18:33:34.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3730000000000001e-06 2024-09-10T18:33:34.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.047856+0000", 2024-09-10T18:33:34.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6e-05 2024-09-10T18:33:34.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.047970+0000", 2024-09-10T18:33:34.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113374 2024-09-10T18:33:34.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:33:34.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048092+0000", 2024-09-10T18:33:34.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012236899999999999 2024-09-10T18:33:34.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:33:34.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.052310+0000", 2024-09-10T18:33:34.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042182859999999999 2024-09-10T18:33:34.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.052984+0000", 2024-09-10T18:33:34.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00067373300000000004 2024-09-10T18:33:34.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.194590+0000", 2024-09-10T18:33:34.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14160546600000001 2024-09-10T18:33:34.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:33:34.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.194834+0000", 2024-09-10T18:33:34.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024398099999999999 2024-09-10T18:33:34.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.195255+0000", 2024-09-10T18:33:34.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00042183200000000002 2024-09-10T18:33:34.181 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.181 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.181 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.181 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:182346 7.e 7:745a108f:::benchmark_data_smithi028_77856_object61246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-10T18:33:34.181 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:26.325845+0000", 2024-09-10T18:33:34.181 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 427.70527659599998, 2024-09-10T18:33:34.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16243428400000001, 2024-09-10T18:33:34.182 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.182 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:33:34.182 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:33:34.182 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:33:34.182 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:33:34.182 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 182346 2024-09-10T18:33:34.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.182 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.325845+0000", 2024-09-10T18:33:34.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.325845+0000", 2024-09-10T18:33:34.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.325843+0000", 2024-09-10T18:33:34.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:33:34.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.325850+0000", 2024-09-10T18:33:34.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6429999999999998e-06 2024-09-10T18:33:34.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.325860+0000", 2024-09-10T18:33:34.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7510000000000003e-06 2024-09-10T18:33:34.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.325865+0000", 2024-09-10T18:33:34.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3850000000000001e-06 2024-09-10T18:33:34.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.326216+0000", 2024-09-10T18:33:34.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00035057099999999999 2024-09-10T18:33:34.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.326311+0000", 2024-09-10T18:33:34.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5045999999999998e-05 2024-09-10T18:33:34.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:33:34.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.327521+0000", 2024-09-10T18:33:34.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012101589999999999 2024-09-10T18:33:34.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:33:34.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.330674+0000", 2024-09-10T18:33:34.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031535589999999998 2024-09-10T18:33:34.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.331189+0000", 2024-09-10T18:33:34.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00051444999999999996 2024-09-10T18:33:34.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.488190+0000", 2024-09-10T18:33:34.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15700107599999999 2024-09-10T18:33:34.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:33:34.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.488236+0000", 2024-09-10T18:33:34.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6341999999999998e-05 2024-09-10T18:33:34.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.488279+0000", 2024-09-10T18:33:34.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3205999999999999e-05 2024-09-10T18:33:34.188 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.188 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.188 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.189 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:182358 7.e 7:7f5527b9:::benchmark_data_smithi028_77856_object61258:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-10T18:33:34.189 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:26.334381+0000", 2024-09-10T18:33:34.189 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 427.696741245, 2024-09-10T18:33:34.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15401279700000001, 2024-09-10T18:33:34.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:33:34.189 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:33:34.189 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:33:34.189 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:33:34.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 182358 2024-09-10T18:33:34.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.190 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.334381+0000", 2024-09-10T18:33:34.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.334381+0000", 2024-09-10T18:33:34.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.334377+0000", 2024-09-10T18:33:34.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-10T18:33:34.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.334388+0000", 2024-09-10T18:33:34.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0210999999999999e-05 2024-09-10T18:33:34.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.334403+0000", 2024-09-10T18:33:34.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5730999999999999e-05 2024-09-10T18:33:34.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.334411+0000", 2024-09-10T18:33:34.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.289e-06 2024-09-10T18:33:34.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.334462+0000", 2024-09-10T18:33:34.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1261999999999998e-05 2024-09-10T18:33:34.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.334576+0000", 2024-09-10T18:33:34.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011425999999999999 2024-09-10T18:33:34.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:33:34.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.334687+0000", 2024-09-10T18:33:34.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111277 2024-09-10T18:33:34.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337933+0000", 2024-09-10T18:33:34.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032456149999999999 2024-09-10T18:33:34.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:33:34.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337941+0000", 2024-09-10T18:33:34.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2379999999999997e-06 2024-09-10T18:33:34.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.488323+0000", 2024-09-10T18:33:34.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15038196200000001 2024-09-10T18:33:34.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:33:34.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.488360+0000", 2024-09-10T18:33:34.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7147999999999997e-05 2024-09-10T18:33:34.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.488393+0000", 2024-09-10T18:33:34.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2839999999999997e-05 2024-09-10T18:33:34.195 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.196 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.196 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.196 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:182362 7.3 e143/93 7:c3249eb8:::benchmark_data_smithi028_77856_object61262:head v 143'11420, mlcod=143'11417)", 2024-09-10T18:33:34.196 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:26.338156+0000", 2024-09-10T18:33:34.196 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 427.69296550799999, 2024-09-10T18:33:34.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15926261, 2024-09-10T18:33:34.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.196 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:33:34.196 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.338156+0000", 2024-09-10T18:33:34.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.338156+0000", 2024-09-10T18:33:34.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.338155+0000", 2024-09-10T18:33:34.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:33:34.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.338162+0000", 2024-09-10T18:33:34.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0759999999999997e-06 2024-09-10T18:33:34.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.338172+0000", 2024-09-10T18:33:34.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8900000000000002e-06 2024-09-10T18:33:34.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.338177+0000", 2024-09-10T18:33:34.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0220000000000004e-06 2024-09-10T18:33:34.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.338231+0000", 2024-09-10T18:33:34.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4307e-05 2024-09-10T18:33:34.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.338248+0000", 2024-09-10T18:33:34.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6680999999999998e-05 2024-09-10T18:33:34.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:33:34.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.496992+0000", 2024-09-10T18:33:34.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15874396199999999 2024-09-10T18:33:34.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.497419+0000", 2024-09-10T18:33:34.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00042691199999999999 2024-09-10T18:33:34.201 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.201 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.201 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.201 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.14808.0:182346 7.e e143/93 ondisk, result = 0)", 2024-09-10T18:33:34.201 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:26.339426+0000", 2024-09-10T18:33:34.201 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 427.69169620100001, 2024-09-10T18:33:34.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14886392900000001, 2024-09-10T18:33:34.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:33:34.202 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.339426+0000", 2024-09-10T18:33:34.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.339426+0000", 2024-09-10T18:33:34.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.339424+0000", 2024-09-10T18:33:34.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:33:34.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.339431+0000", 2024-09-10T18:33:34.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4529999999999999e-06 2024-09-10T18:33:34.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.487928+0000", 2024-09-10T18:33:34.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14849749200000001 2024-09-10T18:33:34.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.487941+0000", 2024-09-10T18:33:34.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2743e-05 2024-09-10T18:33:34.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.488174+0000", 2024-09-10T18:33:34.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023364999999999999 2024-09-10T18:33:34.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.488183+0000", 2024-09-10T18:33:34.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3899999999999993e-06 2024-09-10T18:33:34.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.488289+0000", 2024-09-10T18:33:34.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000106631 2024-09-10T18:33:34.205 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.205 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.205 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.206 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:182380 7.5 e143/93 7:a7182387:::benchmark_data_smithi028_77856_object61280:head v 143'11387, mlcod=143'11385)", 2024-09-10T18:33:34.206 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:26.350071+0000", 2024-09-10T18:33:34.206 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 427.68105085299999, 2024-09-10T18:33:34.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14688066399999999, 2024-09-10T18:33:34.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:33:34.206 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.350071+0000", 2024-09-10T18:33:34.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.350071+0000", 2024-09-10T18:33:34.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.350068+0000", 2024-09-10T18:33:34.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:33:34.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.350080+0000", 2024-09-10T18:33:34.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1165999999999999e-05 2024-09-10T18:33:34.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.487937+0000", 2024-09-10T18:33:34.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13785700000000001 2024-09-10T18:33:34.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.487946+0000", 2024-09-10T18:33:34.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0380000000000005e-06 2024-09-10T18:33:34.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.488060+0000", 2024-09-10T18:33:34.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011403100000000001 2024-09-10T18:33:34.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.488086+0000", 2024-09-10T18:33:34.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6143e-05 2024-09-10T18:33:34.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:33:34.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.496823+0000", 2024-09-10T18:33:34.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087368700000000007 2024-09-10T18:33:34.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.496952+0000", 2024-09-10T18:33:34.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000128942 2024-09-10T18:33:34.210 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.210 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.210 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.211 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:226329 7.e 7:7072a39a:::benchmark_data_smithi028_77856_object105229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-10T18:33:34.211 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:14.384488+0000", 2024-09-10T18:33:34.211 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 379.64663356699998, 2024-09-10T18:33:34.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.143121796, 2024-09-10T18:33:34.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.211 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:33:34.211 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:33:34.211 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:33:34.211 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:33:34.212 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 226329 2024-09-10T18:33:34.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.212 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384488+0000", 2024-09-10T18:33:34.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384488+0000", 2024-09-10T18:33:34.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384486+0000", 2024-09-10T18:33:34.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:33:34.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384494+0000", 2024-09-10T18:33:34.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.43e-06 2024-09-10T18:33:34.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384504+0000", 2024-09-10T18:33:34.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9550000000000004e-06 2024-09-10T18:33:34.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384509+0000", 2024-09-10T18:33:34.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4820000000000002e-06 2024-09-10T18:33:34.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384597+0000", 2024-09-10T18:33:34.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8226999999999994e-05 2024-09-10T18:33:34.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384711+0000", 2024-09-10T18:33:34.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011371599999999999 2024-09-10T18:33:34.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:33:34.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384823+0000", 2024-09-10T18:33:34.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111736 2024-09-10T18:33:34.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.391850+0000", 2024-09-10T18:33:34.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070275429999999998 2024-09-10T18:33:34.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:33:34.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.391857+0000", 2024-09-10T18:33:34.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4080000000000003e-06 2024-09-10T18:33:34.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.527531+0000", 2024-09-10T18:33:34.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.135674605 2024-09-10T18:33:34.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:33:34.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.527574+0000", 2024-09-10T18:33:34.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2877000000000003e-05 2024-09-10T18:33:34.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.527610+0000", 2024-09-10T18:33:34.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5678e-05 2024-09-10T18:33:34.217 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.217 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.218 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.218 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:226330 7.e 7:78b59138:::benchmark_data_smithi028_77856_object105230:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-10T18:33:34.218 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:14.384527+0000", 2024-09-10T18:33:34.218 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 379.64659433399999, 2024-09-10T18:33:34.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.144612923, 2024-09-10T18:33:34.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:33:34.218 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:33:34.218 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:33:34.218 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:33:34.219 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 226330 2024-09-10T18:33:34.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.219 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384527+0000", 2024-09-10T18:33:34.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384527+0000", 2024-09-10T18:33:34.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384527+0000", 2024-09-10T18:33:34.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:33:34.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384531+0000", 2024-09-10T18:33:34.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.33e-06 2024-09-10T18:33:34.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384538+0000", 2024-09-10T18:33:34.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6699999999999997e-06 2024-09-10T18:33:34.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.384542+0000", 2024-09-10T18:33:34.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3109999999999999e-06 2024-09-10T18:33:34.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.388071+0000", 2024-09-10T18:33:34.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035287750000000001 2024-09-10T18:33:34.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.388185+0000", 2024-09-10T18:33:34.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011471700000000001 2024-09-10T18:33:34.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:33:34.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.388306+0000", 2024-09-10T18:33:34.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000120727 2024-09-10T18:33:34.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.391897+0000", 2024-09-10T18:33:34.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035909689999999998 2024-09-10T18:33:34.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:33:34.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.391908+0000", 2024-09-10T18:33:34.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.059e-05 2024-09-10T18:33:34.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.528694+0000", 2024-09-10T18:33:34.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13678641499999999 2024-09-10T18:33:34.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:33:34.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.528951+0000", 2024-09-10T18:33:34.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000256975 2024-09-10T18:33:34.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.529140+0000", 2024-09-10T18:33:34.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018924400000000001 2024-09-10T18:33:34.224 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.225 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.225 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.225 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:226342 7.e 7:7f841004:::benchmark_data_smithi028_77856_object105242:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-10T18:33:34.225 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:27:14.393328+0000", 2024-09-10T18:33:34.225 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 379.637793391, 2024-09-10T18:33:34.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15273182299999999, 2024-09-10T18:33:34.225 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:33:34.225 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:33:34.225 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:33:34.225 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:33:34.226 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:33:34.226 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 226342 2024-09-10T18:33:34.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.226 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:33:34.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:33:34.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.393328+0000", 2024-09-10T18:33:34.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:33:34.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.393328+0000", 2024-09-10T18:33:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:33:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:33:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.393326+0000", 2024-09-10T18:33:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:33:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:33:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.393333+0000", 2024-09-10T18:33:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8569999999999997e-06 2024-09-10T18:33:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:33:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.393345+0000", 2024-09-10T18:33:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1402e-05 2024-09-10T18:33:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:33:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.393350+0000", 2024-09-10T18:33:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8390000000000003e-06 2024-09-10T18:33:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:33:34.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.395360+0000", 2024-09-10T18:33:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002009346 2024-09-10T18:33:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:33:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.395459+0000", 2024-09-10T18:33:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9456000000000005e-05 2024-09-10T18:33:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:33:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.395562+0000", 2024-09-10T18:33:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010313400000000001 2024-09-10T18:33:34.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.523187+0000", 2024-09-10T18:33:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.12762510899999999 2024-09-10T18:33:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:33:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.527675+0000", 2024-09-10T18:33:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044876639999999997 2024-09-10T18:33:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:33:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.545824+0000", 2024-09-10T18:33:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018148487000000001 2024-09-10T18:33:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:33:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.546020+0000", 2024-09-10T18:33:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019589799999999999 2024-09-10T18:33:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:33:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:33:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:33:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:27:14.546060+0000", 2024-09-10T18:33:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0649000000000003e-05 2024-09-10T18:33:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:33:34.232 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:33:34.232 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:33:34.232 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:34.261 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:34.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:34.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:34.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:34.398 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:34.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:34.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:34.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:34.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:34.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:34.639 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:34.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:34.674 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:34.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:34.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:34.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:33:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:33:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:34.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:34.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:34.925 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:34.956 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:34.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:34.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:35.058 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:35.075 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 82 16 125928 125912 95.9205 69.3125 0.00849228 0.0104195 2024-09-10T18:33:35.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:35.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:35.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:35.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:34 smithi204 ceph-mon[32357]: osdmap e273: 8 total, 8 up, 8 in 2024-09-10T18:33:35.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:34 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2452295053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:33:35.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:35.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:35.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:35.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:35.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:34 smithi028 ceph-mon[39220]: osdmap e273: 8 total, 8 up, 8 in 2024-09-10T18:33:35.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:34 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2452295053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:33:35.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:34 smithi028 ceph-mon[29291]: osdmap e273: 8 total, 8 up, 8 in 2024-09-10T18:33:35.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:34 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2452295053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:33:35.336 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:35.370 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:35.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:35.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:35.472 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:35.504 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:35.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:35.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:35.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:35.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:35.644 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:35.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:35.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:35.775 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:35.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:35.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:35.877 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:35.910 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:35.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:35.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:36.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:36.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:36.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:36.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:36.074 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 83 16 127080 127064 95.6323 72 0.02613 0.0104522 2024-09-10T18:33:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:35 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:35 smithi204 ceph-mon[32357]: pgmap v544: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-10T18:33:36.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:36.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:36.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:36.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:35 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:36.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:35 smithi028 ceph-mon[29291]: pgmap v544: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-10T18:33:36.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:35 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:36.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:35 smithi028 ceph-mon[39220]: pgmap v544: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-10T18:33:36.303 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:36.325 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:36.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:36.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:36.426 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:36.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:36.460 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:36.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:36.561 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:36.592 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:36.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:36.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:36.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:36.726 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:36.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:36.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:36.820 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:33:36.820 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:33:36.821 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:33:36.821 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:33:36.821 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:33:36.821 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:33:36.821 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:33:36.821 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:33:36.821 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:33:36.821 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:33:36.821 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:33:36.821 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:33:36.822 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:33:36.822 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:33:36.822 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:33:36.822 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:33:36.828 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:36.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:36.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:36.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:37.007 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:37.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:37.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:37.076 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 84 16 128174 128158 95.3079 68.375 0.0630109 0.0104877 2024-09-10T18:33:37.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:37.148 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:37.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:37.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:37.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-10T18:33:37.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:37.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:37.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:37.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:37.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:37.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:37.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:37.547 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:37.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:37.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:37.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:37.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:37.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:37.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:37.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:37.855 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:37.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:37.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:37.881 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:33:37.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:37.983 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:37.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:37.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:38.076 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 85 16 129263 129247 94.9867 68.0625 0.00950612 0.0105223 2024-09-10T18:33:38.085 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:38.116 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:38.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:38.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:38.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:38.222 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:33:38.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:38.263 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:38.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:38.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:38 smithi028 ceph-mon[29291]: pgmap v545: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 821 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-09-10T18:33:38.351 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:38 smithi028 ceph-mon[39220]: pgmap v545: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 821 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-09-10T18:33:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:38 smithi204 ceph-mon[32357]: pgmap v545: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 821 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-09-10T18:33:38.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:38.406 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:38.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:38.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:38.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:38.546 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:38.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:38.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:38.648 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:38.688 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:38.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:38.790 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:38.817 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:38.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:38.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:38.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:38.956 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:38.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:39.057 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:39.075 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 86 16 130428 130412 94.7289 72.8125 0.00971977 0.0105514 2024-09-10T18:33:39.086 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:33:39.100 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:39.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:39.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:39.201 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:39.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:39.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:39.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:39 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2978246981' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:33:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:39 smithi028 ceph-mon[29291]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:33:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:33:39.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:39 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2978246981' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:33:39.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:39 smithi028 ceph-mon[39220]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:33:39.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:33:39.335 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:39 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2978246981' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:33:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:39 smithi204 ceph-mon[32357]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:33:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:33:39.373 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:39.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:39.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:39.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:39.507 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:39.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:39.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:39.609 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:39.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:33:39.638 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:39.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:39.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:39.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:39.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:39.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:39.898 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:39.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:39.926 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:39.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:40.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:40.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:40.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:40.076 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 87 16 131622 131606 94.497 74.625 0.00602484 0.0105775 2024-09-10T18:33:40.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:40.213 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:40.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:40.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:40.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:40 smithi028 ceph-mon[29291]: pgmap v546: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 719 op/s 2024-09-10T18:33:40.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:40 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:33:40.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:40 smithi028 ceph-mon[29291]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:33:40.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:40 smithi028 ceph-mon[29291]: osdmap e274: 8 total, 8 up, 8 in 2024-09-10T18:33:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:40 smithi028 ceph-mon[39220]: pgmap v546: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 719 op/s 2024-09-10T18:33:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:40 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:33:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:40 smithi028 ceph-mon[39220]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:33:40.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:40 smithi028 ceph-mon[39220]: osdmap e274: 8 total, 8 up, 8 in 2024-09-10T18:33:40.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:40 smithi204 ceph-mon[32357]: pgmap v546: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 719 op/s 2024-09-10T18:33:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:40 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:33:40.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:40 smithi204 ceph-mon[32357]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:33:40.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:40 smithi204 ceph-mon[32357]: osdmap e274: 8 total, 8 up, 8 in 2024-09-10T18:33:40.357 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:40.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:40.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:40.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:40.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:40.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:40.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:40.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:40.630 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:40.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:40.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:40.756 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:40.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:40.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:40.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:40.897 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:40.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:40.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:40.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:41.021 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:41.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:41.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:41.076 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 88 16 132745 132729 94.2208 70.1875 0.00921418 0.0106089 2024-09-10T18:33:41.123 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:41.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:41.157 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:41.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:41.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:41.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:41.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:41.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:41.438 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:41.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:41.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:41.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:41.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:41.574 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:41.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:41.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:41.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:41.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:41.753 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:33:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:33:41] "GET /metrics HTTP/1.1" 200 224454 "" "Prometheus/2.43.0" 2024-09-10T18:33:41.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:41.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:41.829 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:41.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:41.930 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:41.968 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:41.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:41.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:42.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:42.077 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 89 16 133657 133641 93.802 57 0.00799551 0.0106559 2024-09-10T18:33:42.107 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:42.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:42.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:42.209 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:42.241 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:42.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:42.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:42.343 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:42 smithi204 ceph-mon[32357]: pgmap v548: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 8.0 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-10T18:33:42.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:33:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:33:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:33:42.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:42.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:42.471 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:42.503 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:42.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:42 smithi028 ceph-mon[29291]: pgmap v548: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 8.0 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-10T18:33:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:42 smithi028 ceph-mon[39220]: pgmap v548: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 8.0 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-10T18:33:42.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:42.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:42.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:42.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:42.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:42.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:42.788 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:42.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:42.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:42.890 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:42.911 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:42.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:42.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:43.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:43.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:43.049 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:43.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:43.077 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total time run: 90.0066 2024-09-10T18:33:43.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total writes made: 134588 2024-09-10T18:33:43.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Write size: 65536 2024-09-10T18:33:43.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object size: 65536 2024-09-10T18:33:43.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Bandwidth (MB/sec): 93.4571 2024-09-10T18:33:43.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Bandwidth: 35.0044 2024-09-10T18:33:43.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max bandwidth (MB/sec): 218.688 2024-09-10T18:33:43.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min bandwidth (MB/sec): 57 2024-09-10T18:33:43.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average IOPS: 1495 2024-09-10T18:33:43.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev IOPS: 560.07 2024-09-10T18:33:43.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max IOPS: 3499 2024-09-10T18:33:43.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min IOPS: 912 2024-09-10T18:33:43.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average Latency(s): 0.0106962 2024-09-10T18:33:43.078 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Latency(s): 0.010461 2024-09-10T18:33:43.079 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max latency(s): 0.246174 2024-09-10T18:33:43.079 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min latency(s): 0.00134629 2024-09-10T18:33:43.081 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Cleaning up (deleting benchmark objects) 2024-09-10T18:33:43.150 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:43.181 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:43.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:43.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:43.282 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:43.315 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:43.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:43.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:43.417 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:43.448 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:43.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:43.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:43.549 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:43.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:43.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:43.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:43.686 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:43.711 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:43.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:43.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:43.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:43.849 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:43.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:43.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:43.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:43.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:43.988 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:43.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:44.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:44.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:44.234 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:44.255 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:44.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:44 smithi204 ceph-mon[32357]: pgmap v549: 158 pgs: 158 active+clean; 8.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-09-10T18:33:44.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:44.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:44.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:44.496 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:44.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:44 smithi028 ceph-mon[29291]: pgmap v549: 158 pgs: 158 active+clean; 8.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-09-10T18:33:44.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:44 smithi028 ceph-mon[39220]: pgmap v549: 158 pgs: 158 active+clean; 8.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-09-10T18:33:44.527 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:44.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:44.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:44.630 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:44.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:44.664 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:44.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:44.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:44.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:44.802 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:44.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:44.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:33:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:33:44.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:44.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:44.935 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:44.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:45.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:45.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:45.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:45.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:45.198 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:45.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:45.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:45.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:45.340 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:45.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:45.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:45.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:45.479 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:45.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:45.581 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:45.611 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:45.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:45.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:45.713 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:45.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:45.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:45.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:45.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:45.889 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:45.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:45.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:45.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:46.028 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:46.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:46.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:46.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:46.162 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:46.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:46.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:46.264 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:46.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:46.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:46.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:46 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:46 smithi204 ceph-mon[32357]: pgmap v550: 158 pgs: 158 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.12k op/s 2024-09-10T18:33:46.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:46 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:46.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:46 smithi028 ceph-mon[29291]: pgmap v550: 158 pgs: 158 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.12k op/s 2024-09-10T18:33:46.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:46 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:46.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:46 smithi028 ceph-mon[39220]: pgmap v550: 158 pgs: 158 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.12k op/s 2024-09-10T18:33:46.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:46.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:46.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:46.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:46.582 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:46.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:46.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:46.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:46.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:46.727 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:46.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:46.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:46.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:46.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:46.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:46.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:46.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:46.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:46.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:47.099 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:47.127 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:47.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:47.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:47.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:47.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:47.271 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:47.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:47.308 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:33:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:33:47.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:47.407 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:47.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:47.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:47.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:47.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:47.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:47.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:47.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:47.686 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:47.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:47.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:47.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:47.818 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:47.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:47.919 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:47.948 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:47.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:47.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:48.050 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:48.080 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:48.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:48.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:48.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:48.226 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:48.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:48.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:48.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:33:48.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:48.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:48 smithi028 ceph-mon[29291]: pgmap v551: 158 pgs: 158 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 1.10k op/s 2024-09-10T18:33:48.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:48 smithi028 ceph-mon[29291]: from='client.15495 v1:172.21.15.28:0/1801342296' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:33:48.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:48 smithi028 ceph-mon[39220]: pgmap v551: 158 pgs: 158 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 1.10k op/s 2024-09-10T18:33:48.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:48 smithi028 ceph-mon[39220]: from='client.15495 v1:172.21.15.28:0/1801342296' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:33:48.367 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:48.370 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:33:48.370 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:33:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:48.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:48.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:48.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:48.510 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:48.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:48 smithi204 ceph-mon[32357]: pgmap v551: 158 pgs: 158 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 1.10k op/s 2024-09-10T18:33:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:48 smithi204 ceph-mon[32357]: from='client.15495 v1:172.21.15.28:0/1801342296' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:33:48.611 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:48.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:48.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:48.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:48.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:48.767 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:48.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:48.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:48.869 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:48.904 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:48.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:48.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:49.005 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:49.043 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:49.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:49.144 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:49.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:49.183 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:49.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:49.209 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":551,"stamp":"2024-09-10T18:33:47.012066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8641911028,"num_objects":132053,"num_object_clones":0,"num_object_copies":396159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132053,"num_whiteouts":0,"num_read":1875,"num_read_kb":1627,"num_write":138037,"num_write_kb":8613676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79047,"ondisk_log_size":79047,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":28152880,"kb_used_data":25455472,"kb_used_omap":19,"kb_used_meta":2697324,"kb_avail":721808336,"statfs":{"total":767960285184,"available":739131736064,"internally_reserved":0,"allocated":26066403328,"data_stored":26063281588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20033,"internal_metadata":2762060223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":363528216,"num_objects":5548,"num_object_clones":0,"num_object_copies":16644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5548,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":11012,"num_write_kb":529857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004485"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":274,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.805052+0000","last_change":"2024-09-10T18:33:04.320548+0000","last_active":"2024-09-10T18:33:33.805052+0000","last_peered":"2024-09-10T18:33:33.805052+0000","last_clean":"2024-09-10T18:33:33.805052+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:33:33.805052+0000","last_undegraded":"2024-09-10T18:33:33.805052+0000","last_fullsized":"2024-09-10T18:33:33.805052+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:04.320458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:33:04.320458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.581200+0000","scrub_duration":0.0055421380000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":313,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.813976+0000","last_change":"2024-09-10T18:33:12.680256+0000","last_active":"2024-09-10T18:33:33.813976+0000","last_peered":"2024-09-10T18:33:33.813976+0000","last_clean":"2024-09-10T18:33:33.813976+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:33:33.813976+0000","last_undegraded":"2024-09-10T18:33:33.813976+0000","last_fullsized":"2024-09-10T18:33:33.813976+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:33:12.680138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:12.680138+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:19.317278+0000","scrub_duration":0.0033442979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":279,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.810580+0000","last_change":"2024-09-10T18:32:43.156005+0000","last_active":"2024-09-10T18:33:33.810580+0000","last_peered":"2024-09-10T18:33:33.810580+0000","last_clean":"2024-09-10T18:33:33.810580+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:33:33.810580+0000","last_undegraded":"2024-09-10T18:33:33.810580+0000","last_fullsized":"2024-09-10T18:33:33.810580+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:43.155924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:43.155924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:58.337772+0000","scrub_duration":0.007043718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":277,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.814290+0000","last_change":"2024-09-10T18:33:08.643815+0000","last_active":"2024-09-10T18:33:33.814290+0000","last_peered":"2024-09-10T18:33:33.814290+0000","last_clean":"2024-09-10T18:33:33.814290+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:33:33.814290+0000","last_undegraded":"2024-09-10T18:33:33.814290+0000","last_fullsized":"2024-09-10T18:33:33.814290+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:08.643758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.643758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:24.996028+0000","scrub_duration":0.0046354059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":276,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.807642+0000","last_change":"2024-09-10T18:31:54.788489+0000","last_active":"2024-09-10T18:33:33.807642+0000","last_peered":"2024-09-10T18:33:33.807642+0000","last_clean":"2024-09-10T18:33:33.807642+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:33:33.807642+0000","last_undegraded":"2024-09-10T18:33:33.807642+0000","last_fullsized":"2024-09-10T18:33:33.807642+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:54.788436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:54.788436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013008939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":282,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.810021+0000","last_change":"2024-09-10T18:32:00.727061+0000","last_active":"2024-09-10T18:33:33.810021+0000","last_peered":"2024-09-10T18:33:33.810021+0000","last_clean":"2024-09-10T18:33:33.810021+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:33:33.810021+0000","last_undegraded":"2024-09-10T18:33:33.810021+0000","last_fullsized":"2024-09-10T18:33:33.810021+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.727002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.727002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014900709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'34","reported_seq":323,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.808321+0000","last_change":"2024-09-10T18:32:55.371863+0000","last_active":"2024-09-10T18:33:33.808321+0000","last_peered":"2024-09-10T18:33:33.808321+0000","last_clean":"2024-09-10T18:33:33.808321+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:33:33.808321+0000","last_undegraded":"2024-09-10T18:33:33.808321+0000","last_fullsized":"2024-09-10T18:33:33.808321+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:32:55.371617+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:32:55.371617+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:18.220770+0000","scrub_duration":0.0089694079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":279,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.813927+0000","last_change":"2024-09-10T18:31:48.634526+0000","last_active":"2024-09-10T18:33:33.813927+0000","last_peered":"2024-09-10T18:33:33.813927+0000","last_clean":"2024-09-10T18:33:33.813927+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:33:33.813927+0000","last_undegraded":"2024-09-10T18:33:33.813927+0000","last_fullsized":"2024-09-10T18:33:33.813927+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:48.634465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.634465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012201320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":294,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.807826+0000","last_change":"2024-09-10T18:33:16.963182+0000","last_active":"2024-09-10T18:33:33.807826+0000","last_peered":"2024-09-10T18:33:33.807826+0000","last_clean":"2024-09-10T18:33:33.807826+0000","last_became_active":"2024-09-10T18:26:27.644961+0000","last_became_peered":"2024-09-10T18:26:27.644961+0000","last_unstale":"2024-09-10T18:33:33.807826+0000","last_undegraded":"2024-09-10T18:33:33.807826+0000","last_fullsized":"2024-09-10T18:33:33.807826+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_clean_scrub_stamp":"2024-09-10T18:33:16.963102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:44.331853+0000","scrub_duration":0.0019539739999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'15","reported_seq":306,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.814564+0000","last_change":"2024-09-10T18:32:45.581485+0000","last_active":"2024-09-10T18:33:33.814564+0000","last_peered":"2024-09-10T18:33:33.814564+0000","last_clean":"2024-09-10T18:33:33.814564+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:33:33.814564+0000","last_undegraded":"2024-09-10T18:33:33.814564+0000","last_fullsized":"2024-09-10T18:33:33.814564+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:32:45.581390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:45.581390+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:01.343758+0000","scrub_duration":0.003880966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":280,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.806989+0000","last_change":"2024-09-10T18:31:58.589684+0000","last_active":"2024-09-10T18:33:33.806989+0000","last_peered":"2024-09-10T18:33:33.806989+0000","last_clean":"2024-09-10T18:33:33.806989+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:33:33.806989+0000","last_undegraded":"2024-09-10T18:33:33.806989+0000","last_fullsized":"2024-09-10T18:33:33.806989+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_clean_scrub_stamp":"2024-09-10T18:31:58.589631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011763940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":278,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804348+0000","last_change":"2024-09-10T18:33:00.235163+0000","last_active":"2024-09-10T18:33:33.804348+0000","last_peered":"2024-09-10T18:33:33.804348+0000","last_clean":"2024-09-10T18:33:33.804348+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:33:33.804348+0000","last_undegraded":"2024-09-10T18:33:33.804348+0000","last_fullsized":"2024-09-10T18:33:33.804348+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:00.234931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:00.234931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.372488+0000","scrub_duration":0.0023527999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":284,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.807235+0000","last_change":"2024-09-10T18:32:00.582011+0000","last_active":"2024-09-10T18:33:33.807235+0000","last_peered":"2024-09-10T18:33:33.807235+0000","last_clean":"2024-09-10T18:33:33.807235+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:33:33.807235+0000","last_undegraded":"2024-09-10T18:33:33.807235+0000","last_fullsized":"2024-09-10T18:33:33.807235+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.581959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:17.464205+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.581959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012934120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":318,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.814156+0000","last_change":"2024-09-10T18:32:48.589267+0000","last_active":"2024-09-10T18:33:33.814156+0000","last_peered":"2024-09-10T18:33:33.814156+0000","last_clean":"2024-09-10T18:33:33.814156+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:33:33.814156+0000","last_undegraded":"2024-09-10T18:33:33.814156+0000","last_fullsized":"2024-09-10T18:33:33.814156+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'9","last_scrub_stamp":"2024-09-10T18:32:48.589167+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:32:48.589167+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:02.172908+0000","scrub_duration":0.0073279920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":279,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804121+0000","last_change":"2024-09-10T18:32:01.525753+0000","last_active":"2024-09-10T18:33:33.804121+0000","last_peered":"2024-09-10T18:33:33.804121+0000","last_clean":"2024-09-10T18:33:33.804121+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:33:33.804121+0000","last_undegraded":"2024-09-10T18:33:33.804121+0000","last_fullsized":"2024-09-10T18:33:33.804121+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:01.525701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:01.525701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001223902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":271,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.809784+0000","last_change":"2024-09-10T18:31:57.628816+0000","last_active":"2024-09-10T18:33:33.809784+0000","last_peered":"2024-09-10T18:33:33.809784+0000","last_clean":"2024-09-10T18:33:33.809784+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:33:33.809784+0000","last_undegraded":"2024-09-10T18:33:33.809784+0000","last_fullsized":"2024-09-10T18:33:33.809784+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:57.628758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.628758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013180119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":277,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.814935+0000","last_change":"2024-09-10T18:31:57.658865+0000","last_active":"2024-09-10T18:33:33.814935+0000","last_peered":"2024-09-10T18:33:33.814935+0000","last_clean":"2024-09-10T18:33:33.814935+0000","last_became_active":"2024-09-10T18:26:27.650774+0000","last_became_peered":"2024-09-10T18:26:27.650774+0000","last_unstale":"2024-09-10T18:33:33.814935+0000","last_undegraded":"2024-09-10T18:33:33.814935+0000","last_fullsized":"2024-09-10T18:33:33.814935+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:57.658808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.658808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001743421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":270,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804813+0000","last_change":"2024-09-10T18:33:18.319184+0000","last_active":"2024-09-10T18:33:33.804813+0000","last_peered":"2024-09-10T18:33:33.804813+0000","last_clean":"2024-09-10T18:33:33.804813+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:33:33.804813+0000","last_undegraded":"2024-09-10T18:33:33.804813+0000","last_fullsized":"2024-09-10T18:33:33.804813+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:53.332881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:53.332881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014944929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"240'27","reported_seq":344,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.812704+0000","last_change":"2024-09-10T18:33:04.586182+0000","last_active":"2024-09-10T18:33:33.812704+0000","last_peered":"2024-09-10T18:33:33.812704+0000","last_clean":"2024-09-10T18:33:33.812704+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:33:33.812704+0000","last_undegraded":"2024-09-10T18:33:33.812704+0000","last_fullsized":"2024-09-10T18:33:33.812704+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:33:04.586063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:04.586063+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:25.050823+0000","scrub_duration":0.007748642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":289,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.811311+0000","last_change":"2024-09-10T18:32:58.252469+0000","last_active":"2024-09-10T18:33:33.811311+0000","last_peered":"2024-09-10T18:33:33.811311+0000","last_clean":"2024-09-10T18:33:33.811311+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:33:33.811311+0000","last_undegraded":"2024-09-10T18:33:33.811311+0000","last_fullsized":"2024-09-10T18:33:33.811311+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:58.252419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.252419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.332332+0000","scrub_duration":0.0049362080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":276,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.812319+0000","last_change":"2024-09-10T18:31:53.754090+0000","last_active":"2024-09-10T18:33:33.812319+0000","last_peered":"2024-09-10T18:33:33.812319+0000","last_clean":"2024-09-10T18:33:33.812319+0000","last_became_active":"2024-09-10T18:26:27.640035+0000","last_became_peered":"2024-09-10T18:26:27.640035+0000","last_unstale":"2024-09-10T18:33:33.812319+0000","last_undegraded":"2024-09-10T18:33:33.812319+0000","last_fullsized":"2024-09-10T18:33:33.812319+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:53.754032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:53.754032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016988699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":283,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.814698+0000","last_change":"2024-09-10T18:32:59.567476+0000","last_active":"2024-09-10T18:33:33.814698+0000","last_peered":"2024-09-10T18:33:33.814698+0000","last_clean":"2024-09-10T18:33:33.814698+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:33:33.814698+0000","last_undegraded":"2024-09-10T18:33:33.814698+0000","last_fullsized":"2024-09-10T18:33:33.814698+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:59.567389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:32:59.567389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:06.727554+0000","scrub_duration":0.0073893559999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'23","reported_seq":310,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.809281+0000","last_change":"2024-09-10T18:32:50.341138+0000","last_active":"2024-09-10T18:33:33.809281+0000","last_peered":"2024-09-10T18:33:33.809281+0000","last_clean":"2024-09-10T18:33:33.809281+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:33:33.809281+0000","last_undegraded":"2024-09-10T18:33:33.809281+0000","last_fullsized":"2024-09-10T18:33:33.809281+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'23","last_scrub_stamp":"2024-09-10T18:32:50.341044+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:32:50.341044+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:02.598697+0000","scrub_duration":0.012050170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":279,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.815283+0000","last_change":"2024-09-10T18:33:07.697770+0000","last_active":"2024-09-10T18:33:33.815283+0000","last_peered":"2024-09-10T18:33:33.815283+0000","last_clean":"2024-09-10T18:33:33.815283+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:33:33.815283+0000","last_undegraded":"2024-09-10T18:33:33.815283+0000","last_fullsized":"2024-09-10T18:33:33.815283+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:07.697667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:33:07.697667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:12.574607+0000","scrub_duration":0.044890945000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":279,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.805514+0000","last_change":"2024-09-10T18:32:57.769495+0000","last_active":"2024-09-10T18:33:33.805514+0000","last_peered":"2024-09-10T18:33:33.805514+0000","last_clean":"2024-09-10T18:33:33.805514+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:33:33.805514+0000","last_undegraded":"2024-09-10T18:33:33.805514+0000","last_fullsized":"2024-09-10T18:33:33.805514+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:32:57.769408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:57.769408+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:20.116093+0000","scrub_duration":0.003757149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":293,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.810723+0000","last_change":"2024-09-10T18:32:51.182762+0000","last_active":"2024-09-10T18:33:33.810723+0000","last_peered":"2024-09-10T18:33:33.810723+0000","last_clean":"2024-09-10T18:33:33.810723+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:33:33.810723+0000","last_undegraded":"2024-09-10T18:33:33.810723+0000","last_fullsized":"2024-09-10T18:33:33.810723+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:51.182684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:32:51.182684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:12.337885+0000","scrub_duration":0.003614637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'25","reported_seq":317,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804466+0000","last_change":"2024-09-10T18:33:15.366317+0000","last_active":"2024-09-10T18:33:33.804466+0000","last_peered":"2024-09-10T18:33:33.804466+0000","last_clean":"2024-09-10T18:33:33.804466+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:33:33.804466+0000","last_undegraded":"2024-09-10T18:33:33.804466+0000","last_fullsized":"2024-09-10T18:33:33.804466+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:33:15.366107+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.366107+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:23.163279+0000","scrub_duration":0.014527711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"217'11","reported_seq":347,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:39.904835+0000","last_change":"2024-09-10T18:33:03.119057+0000","last_active":"2024-09-10T18:33:39.904835+0000","last_peered":"2024-09-10T18:33:39.904835+0000","last_clean":"2024-09-10T18:33:39.904835+0000","last_became_active":"2024-09-10T18:26:27.652489+0000","last_became_peered":"2024-09-10T18:26:27.652489+0000","last_unstale":"2024-09-10T18:33:39.904835+0000","last_undegraded":"2024-09-10T18:33:39.904835+0000","last_fullsized":"2024-09-10T18:33:39.904835+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11","last_scrub_stamp":"2024-09-10T18:33:03.118959+0000","last_deep_scrub":"217'11","last_deep_scrub_stamp":"2024-09-10T18:30:03.161985+0000","last_clean_scrub_stamp":"2024-09-10T18:33:03.118959+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.565288+0000","scrub_duration":0.002038715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"274'8554","reported_seq":8589,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:45.659980+0000","last_change":"2024-09-10T18:32:03.385708+0000","last_active":"2024-09-10T18:33:45.659980+0000","last_peered":"2024-09-10T18:33:45.659980+0000","last_clean":"2024-09-10T18:33:45.659980+0000","last_became_active":"2024-09-10T18:32:03.385428+0000","last_became_peered":"2024-09-10T18:32:03.385428+0000","last_unstale":"2024-09-10T18:33:45.659980+0000","last_undegraded":"2024-09-10T18:33:45.659980+0000","last_fullsized":"2024-09-10T18:33:45.659980+0000","mapping_epoch":246,"log_start":"258'4200","ondisk_log_start":"258'4200","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4354,"log_dups_size":0,"ondisk_log_size":4354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534773760,"num_objects":8160,"num_object_clones":0,"num_object_copies":24480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8554,"num_write_kb":534848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":190,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.807349+0000","last_change":"2024-09-10T18:31:59.600055+0000","last_active":"2024-09-10T18:33:33.807349+0000","last_peered":"2024-09-10T18:33:33.807349+0000","last_clean":"2024-09-10T18:33:33.807349+0000","last_became_active":"2024-09-10T18:26:28.635249+0000","last_became_peered":"2024-09-10T18:26:28.635249+0000","last_unstale":"2024-09-10T18:33:33.807349+0000","last_undegraded":"2024-09-10T18:33:33.807349+0000","last_fullsized":"2024-09-10T18:33:33.807349+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:59.599999+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:59.599999+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001350027,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"240'34","reported_seq":344,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.814119+0000","last_change":"2024-09-10T18:32:56.285685+0000","last_active":"2024-09-10T18:33:33.814119+0000","last_peered":"2024-09-10T18:33:33.814119+0000","last_clean":"2024-09-10T18:33:33.814119+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:33:33.814119+0000","last_undegraded":"2024-09-10T18:33:33.814119+0000","last_fullsized":"2024-09-10T18:33:33.814119+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:32:56.285589+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:32:56.285589+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:15.128863+0000","scrub_duration":0.0081173400000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":275,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.808433+0000","last_change":"2024-09-10T18:31:59.772883+0000","last_active":"2024-09-10T18:33:33.808433+0000","last_peered":"2024-09-10T18:33:33.808433+0000","last_clean":"2024-09-10T18:33:33.808433+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:33:33.808433+0000","last_undegraded":"2024-09-10T18:33:33.808433+0000","last_fullsized":"2024-09-10T18:33:33.808433+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:59.772816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:59.772816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015225810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":426,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:39.084480+0000","last_change":"2024-09-10T18:31:55.634620+0000","last_active":"2024-09-10T18:33:39.084480+0000","last_peered":"2024-09-10T18:33:39.084480+0000","last_clean":"2024-09-10T18:33:39.084480+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:33:39.084480+0000","last_undegraded":"2024-09-10T18:33:39.084480+0000","last_fullsized":"2024-09-10T18:33:39.084480+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:31:55.634564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:55.634564+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001562346,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":278,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.807469+0000","last_change":"2024-09-10T18:32:50.961491+0000","last_active":"2024-09-10T18:33:33.807469+0000","last_peered":"2024-09-10T18:33:33.807469+0000","last_clean":"2024-09-10T18:33:33.807469+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:33:33.807469+0000","last_undegraded":"2024-09-10T18:33:33.807469+0000","last_fullsized":"2024-09-10T18:33:33.807469+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:50.961251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:32:50.961251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:51.179028+0000","scrub_duration":0.0063477669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"274'8395","reported_seq":8432,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:44.914332+0000","last_change":"2024-09-10T18:32:03.379554+0000","last_active":"2024-09-10T18:33:44.914332+0000","last_peered":"2024-09-10T18:33:44.914332+0000","last_clean":"2024-09-10T18:33:44.914332+0000","last_became_active":"2024-09-10T18:32:03.379257+0000","last_became_peered":"2024-09-10T18:32:03.379257+0000","last_unstale":"2024-09-10T18:33:44.914332+0000","last_undegraded":"2024-09-10T18:33:44.914332+0000","last_fullsized":"2024-09-10T18:33:44.914332+0000","mapping_epoch":246,"log_start":"256'3960","ondisk_log_start":"256'3960","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":532021272,"num_objects":8119,"num_object_clones":0,"num_object_copies":24357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8119,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8395,"num_write_kb":528385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":189,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.814636+0000","last_change":"2024-09-10T18:32:58.561489+0000","last_active":"2024-09-10T18:33:33.814636+0000","last_peered":"2024-09-10T18:33:33.814636+0000","last_clean":"2024-09-10T18:33:33.814636+0000","last_became_active":"2024-09-10T18:26:28.632744+0000","last_became_peered":"2024-09-10T18:26:28.632744+0000","last_unstale":"2024-09-10T18:33:33.814636+0000","last_undegraded":"2024-09-10T18:33:33.814636+0000","last_fullsized":"2024-09-10T18:33:33.814636+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:58.561425+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.561425+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:00.246746+0000","scrub_duration":0.002234073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":323,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.808183+0000","last_change":"2024-09-10T18:32:59.382953+0000","last_active":"2024-09-10T18:33:33.808183+0000","last_peered":"2024-09-10T18:33:33.808183+0000","last_clean":"2024-09-10T18:33:33.808183+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:33:33.808183+0000","last_undegraded":"2024-09-10T18:33:33.808183+0000","last_fullsized":"2024-09-10T18:33:33.808183+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-09-10T18:32:59.382840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:59.382840+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:18.205789+0000","scrub_duration":0.0072140750000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":275,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.812848+0000","last_change":"2024-09-10T18:32:49.346233+0000","last_active":"2024-09-10T18:33:33.812848+0000","last_peered":"2024-09-10T18:33:33.812848+0000","last_clean":"2024-09-10T18:33:33.812848+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:33:33.812848+0000","last_undegraded":"2024-09-10T18:33:33.812848+0000","last_fullsized":"2024-09-10T18:33:33.812848+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:49.342269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.140922+0000","last_clean_scrub_stamp":"2024-09-10T18:32:49.342269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:05.179940+0000","scrub_duration":0.006539385,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":279,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.813996+0000","last_change":"2024-09-10T18:32:52.582545+0000","last_active":"2024-09-10T18:33:33.813996+0000","last_peered":"2024-09-10T18:33:33.813996+0000","last_clean":"2024-09-10T18:33:33.813996+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:33:33.813996+0000","last_undegraded":"2024-09-10T18:33:33.813996+0000","last_fullsized":"2024-09-10T18:33:33.813996+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:52.582464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:32:52.582464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:16.167011+0000","scrub_duration":0.0024277399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":267,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.814484+0000","last_change":"2024-09-10T18:32:44.600295+0000","last_active":"2024-09-10T18:33:33.814484+0000","last_peered":"2024-09-10T18:33:33.814484+0000","last_clean":"2024-09-10T18:33:33.814484+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:33:33.814484+0000","last_undegraded":"2024-09-10T18:33:33.814484+0000","last_fullsized":"2024-09-10T18:33:33.814484+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:44.600200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:44.600200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:53.871071+0000","scrub_duration":0.0073958740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"274'8642","reported_seq":8678,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:45.643243+0000","last_change":"2024-09-10T18:32:03.554760+0000","last_active":"2024-09-10T18:33:45.643243+0000","last_peered":"2024-09-10T18:33:45.643243+0000","last_clean":"2024-09-10T18:33:45.643243+0000","last_became_active":"2024-09-10T18:32:03.554400+0000","last_became_peered":"2024-09-10T18:32:03.554400+0000","last_unstale":"2024-09-10T18:33:45.643243+0000","last_undegraded":"2024-09-10T18:33:45.643243+0000","last_fullsized":"2024-09-10T18:33:45.643243+0000","mapping_epoch":246,"log_start":"258'4252","ondisk_log_start":"258'4252","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4390,"log_dups_size":0,"ondisk_log_size":4390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538443776,"num_objects":8216,"num_object_clones":0,"num_object_copies":24648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8642,"num_write_kb":539456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":180,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.808973+0000","last_change":"2024-09-10T18:33:01.370765+0000","last_active":"2024-09-10T18:33:33.808973+0000","last_peered":"2024-09-10T18:33:33.808973+0000","last_clean":"2024-09-10T18:33:33.808973+0000","last_became_active":"2024-09-10T18:26:18.091896+0000","last_became_peered":"2024-09-10T18:26:18.091896+0000","last_unstale":"2024-09-10T18:33:33.808973+0000","last_undegraded":"2024-09-10T18:33:33.808973+0000","last_fullsized":"2024-09-10T18:33:33.808973+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.370561+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.370561+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:12.358728+0000","scrub_duration":0.0029251390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"240'28","reported_seq":320,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.814257+0000","last_change":"2024-09-10T18:31:46.592515+0000","last_active":"2024-09-10T18:33:33.814257+0000","last_peered":"2024-09-10T18:33:33.814257+0000","last_clean":"2024-09-10T18:33:33.814257+0000","last_became_active":"2024-09-10T18:26:27.651364+0000","last_became_peered":"2024-09-10T18:26:27.651364+0000","last_unstale":"2024-09-10T18:33:33.814257+0000","last_undegraded":"2024-09-10T18:33:33.814257+0000","last_fullsized":"2024-09-10T18:33:33.814257+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'28","last_scrub_stamp":"2024-09-10T18:31:46.592456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:46.592456+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019972359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":274,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804733+0000","last_change":"2024-09-10T18:32:54.291391+0000","last_active":"2024-09-10T18:33:33.804733+0000","last_peered":"2024-09-10T18:33:33.804733+0000","last_clean":"2024-09-10T18:33:33.804733+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:33:33.804733+0000","last_undegraded":"2024-09-10T18:33:33.804733+0000","last_fullsized":"2024-09-10T18:33:33.804733+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:54.291164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:32:54.291164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:04.937902+0000","scrub_duration":0.002933667,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":427,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:39.904918+0000","last_change":"2024-09-10T18:33:07.367306+0000","last_active":"2024-09-10T18:33:39.904918+0000","last_peered":"2024-09-10T18:33:39.904918+0000","last_clean":"2024-09-10T18:33:39.904918+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:33:39.904918+0000","last_undegraded":"2024-09-10T18:33:39.904918+0000","last_fullsized":"2024-09-10T18:33:39.904918+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:33:07.367248+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:33:07.367248+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:25.811396+0000","scrub_duration":0.0044945339999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":287,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804052+0000","last_change":"2024-09-10T18:33:11.857892+0000","last_active":"2024-09-10T18:33:33.804052+0000","last_peered":"2024-09-10T18:33:33.804052+0000","last_clean":"2024-09-10T18:33:33.804052+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:33:33.804052+0000","last_undegraded":"2024-09-10T18:33:33.804052+0000","last_fullsized":"2024-09-10T18:33:33.804052+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:11.857685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.857685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:36.504074+0000","scrub_duration":0.0022995870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"274'8560","reported_seq":8596,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:44.892961+0000","last_change":"2024-09-10T18:32:03.384690+0000","last_active":"2024-09-10T18:33:44.892961+0000","last_peered":"2024-09-10T18:33:44.892961+0000","last_clean":"2024-09-10T18:33:44.892961+0000","last_became_active":"2024-09-10T18:32:03.384288+0000","last_became_peered":"2024-09-10T18:32:03.384288+0000","last_unstale":"2024-09-10T18:33:44.892961+0000","last_undegraded":"2024-09-10T18:33:44.892961+0000","last_fullsized":"2024-09-10T18:33:44.892961+0000","mapping_epoch":246,"log_start":"255'3600","ondisk_log_start":"255'3600","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4960,"log_dups_size":0,"ondisk_log_size":4960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":542113792,"num_objects":8272,"num_object_clones":0,"num_object_copies":24816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8560,"num_write_kb":538624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":239,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.817942+0000","last_change":"2024-09-10T18:33:01.151484+0000","last_active":"2024-09-10T18:33:33.817942+0000","last_peered":"2024-09-10T18:33:33.817942+0000","last_clean":"2024-09-10T18:33:33.817942+0000","last_became_active":"2024-09-10T18:26:28.632849+0000","last_became_peered":"2024-09-10T18:26:28.632849+0000","last_unstale":"2024-09-10T18:33:33.817942+0000","last_undegraded":"2024-09-10T18:33:33.817942+0000","last_fullsized":"2024-09-10T18:33:33.817942+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.151386+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.151386+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:06.156495+0000","scrub_duration":0.0031748879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":326,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.811525+0000","last_change":"2024-09-10T18:32:55.279937+0000","last_active":"2024-09-10T18:33:33.811525+0000","last_peered":"2024-09-10T18:33:33.811525+0000","last_clean":"2024-09-10T18:33:33.811525+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:33:33.811525+0000","last_undegraded":"2024-09-10T18:33:33.811525+0000","last_fullsized":"2024-09-10T18:33:33.811525+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:32:55.279209+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:32:55.279209+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:56.244187+0000","scrub_duration":0.0091125259999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":274,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.807954+0000","last_change":"2024-09-10T18:32:45.965208+0000","last_active":"2024-09-10T18:33:33.807954+0000","last_peered":"2024-09-10T18:33:33.807954+0000","last_clean":"2024-09-10T18:33:33.807954+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:33:33.807954+0000","last_undegraded":"2024-09-10T18:33:33.807954+0000","last_fullsized":"2024-09-10T18:33:33.807954+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:45.965129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:32:45.965129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:47.511331+0000","scrub_duration":0.0045521829999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"263'11","reported_seq":435,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:39.905693+0000","last_change":"2024-09-10T18:33:01.283747+0000","last_active":"2024-09-10T18:33:39.905693+0000","last_peered":"2024-09-10T18:33:39.905693+0000","last_clean":"2024-09-10T18:33:39.905693+0000","last_became_active":"2024-09-10T18:26:27.639032+0000","last_became_peered":"2024-09-10T18:26:27.639032+0000","last_unstale":"2024-09-10T18:33:39.905693+0000","last_undegraded":"2024-09-10T18:33:39.905693+0000","last_fullsized":"2024-09-10T18:33:39.905693+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'11","last_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_deep_scrub":"263'11","last_deep_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.283648+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:27.247085+0000","scrub_duration":0.0081388439999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":268,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804546+0000","last_change":"2024-09-10T18:31:43.338883+0000","last_active":"2024-09-10T18:33:33.804546+0000","last_peered":"2024-09-10T18:33:33.804546+0000","last_clean":"2024-09-10T18:33:33.804546+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:33:33.804546+0000","last_undegraded":"2024-09-10T18:33:33.804546+0000","last_fullsized":"2024-09-10T18:33:33.804546+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:43.338816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:43.338816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011905869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"274'8563","reported_seq":8599,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:44.911897+0000","last_change":"2024-09-10T18:32:03.555680+0000","last_active":"2024-09-10T18:33:44.911897+0000","last_peered":"2024-09-10T18:33:44.911897+0000","last_clean":"2024-09-10T18:33:44.911897+0000","last_became_active":"2024-09-10T18:32:03.554693+0000","last_became_peered":"2024-09-10T18:32:03.554693+0000","last_unstale":"2024-09-10T18:33:44.911897+0000","last_undegraded":"2024-09-10T18:33:44.911897+0000","last_fullsized":"2024-09-10T18:33:44.911897+0000","mapping_epoch":246,"log_start":"258'4145","ondisk_log_start":"258'4145","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4418,"log_dups_size":0,"ondisk_log_size":4418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":542965760,"num_objects":8285,"num_object_clones":0,"num_object_copies":24855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8563,"num_write_kb":539136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":239,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.818299+0000","last_change":"2024-09-10T18:32:48.006787+0000","last_active":"2024-09-10T18:33:33.818299+0000","last_peered":"2024-09-10T18:33:33.818299+0000","last_clean":"2024-09-10T18:33:33.818299+0000","last_became_active":"2024-09-10T18:26:28.635542+0000","last_became_peered":"2024-09-10T18:26:28.635542+0000","last_unstale":"2024-09-10T18:33:33.818299+0000","last_undegraded":"2024-09-10T18:33:33.818299+0000","last_fullsized":"2024-09-10T18:33:33.818299+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:48.006695+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:48.006695+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:10.998203+0000","scrub_duration":0.0051030709999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"240'29","reported_seq":333,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804561+0000","last_change":"2024-09-10T18:33:11.310163+0000","last_active":"2024-09-10T18:33:33.804561+0000","last_peered":"2024-09-10T18:33:33.804561+0000","last_clean":"2024-09-10T18:33:33.804561+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:33:33.804561+0000","last_undegraded":"2024-09-10T18:33:33.804561+0000","last_fullsized":"2024-09-10T18:33:33.804561+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'29","last_scrub_stamp":"2024-09-10T18:33:11.310074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.310074+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.669570+0000","scrub_duration":0.0026943760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":273,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.812070+0000","last_change":"2024-09-10T18:31:52.607842+0000","last_active":"2024-09-10T18:33:33.812070+0000","last_peered":"2024-09-10T18:33:33.812070+0000","last_clean":"2024-09-10T18:33:33.812070+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:33:33.812070+0000","last_undegraded":"2024-09-10T18:33:33.812070+0000","last_fullsized":"2024-09-10T18:33:33.812070+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:52.607787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:52.607787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014830799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":229,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.817847+0000","last_change":"2024-09-10T18:33:09.183070+0000","last_active":"2024-09-10T18:33:33.817847+0000","last_peered":"2024-09-10T18:33:33.817847+0000","last_clean":"2024-09-10T18:33:33.817847+0000","last_became_active":"2024-09-10T18:26:27.642063+0000","last_became_peered":"2024-09-10T18:26:27.642063+0000","last_unstale":"2024-09-10T18:33:33.817847+0000","last_undegraded":"2024-09-10T18:33:33.817847+0000","last_fullsized":"2024-09-10T18:33:33.817847+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:09.182982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:33:09.182982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:14.241099+0000","scrub_duration":0.0023969899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":284,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.810485+0000","last_change":"2024-09-10T18:33:10.137145+0000","last_active":"2024-09-10T18:33:33.810485+0000","last_peered":"2024-09-10T18:33:33.810485+0000","last_clean":"2024-09-10T18:33:33.810485+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:33:33.810485+0000","last_undegraded":"2024-09-10T18:33:33.810485+0000","last_fullsized":"2024-09-10T18:33:33.810485+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:10.137048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:33:10.137048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:31.309662+0000","scrub_duration":0.0040149249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"274'12130","reported_seq":12379,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:46.067814+0000","last_change":"2024-09-10T18:33:39.168175+0000","last_active":"2024-09-10T18:33:46.067814+0000","last_peered":"2024-09-10T18:33:46.067814+0000","last_clean":"2024-09-10T18:33:46.067814+0000","last_became_active":"2024-09-10T18:32:03.381872+0000","last_became_peered":"2024-09-10T18:32:03.381872+0000","last_unstale":"2024-09-10T18:33:46.067814+0000","last_undegraded":"2024-09-10T18:33:46.067814+0000","last_fullsized":"2024-09-10T18:33:46.067814+0000","mapping_epoch":246,"log_start":"265'6401","ondisk_log_start":"265'6401","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":5148,"log_size":5729,"log_dups_size":0,"ondisk_log_size":5729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534052864,"num_objects":8149,"num_object_clones":0,"num_object_copies":24447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8599,"num_write_kb":535936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":175,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.809604+0000","last_change":"2024-09-10T18:31:56.629743+0000","last_active":"2024-09-10T18:33:33.809604+0000","last_peered":"2024-09-10T18:33:33.809604+0000","last_clean":"2024-09-10T18:33:33.809604+0000","last_became_active":"2024-09-10T18:26:18.094747+0000","last_became_peered":"2024-09-10T18:26:18.094747+0000","last_unstale":"2024-09-10T18:33:33.809604+0000","last_undegraded":"2024-09-10T18:33:33.809604+0000","last_fullsized":"2024-09-10T18:33:33.809604+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:56.629683+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:56.629683+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001774133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"240'41","reported_seq":413,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:39.905174+0000","last_change":"2024-09-10T18:32:55.941738+0000","last_active":"2024-09-10T18:33:39.905174+0000","last_peered":"2024-09-10T18:33:39.905174+0000","last_clean":"2024-09-10T18:33:39.905174+0000","last_became_active":"2024-09-10T18:23:10.067049+0000","last_became_peered":"2024-09-10T18:23:10.067049+0000","last_unstale":"2024-09-10T18:33:39.905174+0000","last_undegraded":"2024-09-10T18:33:39.905174+0000","last_fullsized":"2024-09-10T18:33:39.905174+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'41","last_scrub_stamp":"2024-09-10T18:32:55.941653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:55.941653+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:02.668969+0000","scrub_duration":0.011585564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":280,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.814340+0000","last_change":"2024-09-10T18:31:49.684883+0000","last_active":"2024-09-10T18:33:33.814340+0000","last_peered":"2024-09-10T18:33:33.814340+0000","last_clean":"2024-09-10T18:33:33.814340+0000","last_became_active":"2024-09-10T18:26:27.648690+0000","last_became_peered":"2024-09-10T18:26:27.648690+0000","last_unstale":"2024-09-10T18:33:33.814340+0000","last_undegraded":"2024-09-10T18:33:33.814340+0000","last_fullsized":"2024-09-10T18:33:33.814340+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:49.684824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:49.684824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012508370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":299,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.808205+0000","last_change":"2024-09-10T18:33:04.936495+0000","last_active":"2024-09-10T18:33:33.808205+0000","last_peered":"2024-09-10T18:33:33.808205+0000","last_clean":"2024-09-10T18:33:33.808205+0000","last_became_active":"2024-09-10T18:26:27.647576+0000","last_became_peered":"2024-09-10T18:26:27.647576+0000","last_unstale":"2024-09-10T18:33:33.808205+0000","last_undegraded":"2024-09-10T18:33:33.808205+0000","last_fullsized":"2024-09-10T18:33:33.808205+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:33:04.936407+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:33:04.936407+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:31.757007+0000","scrub_duration":0.0096149180000000001,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":270,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804138+0000","last_change":"2024-09-10T18:32:39.119345+0000","last_active":"2024-09-10T18:33:33.804138+0000","last_peered":"2024-09-10T18:33:33.804138+0000","last_clean":"2024-09-10T18:33:33.804138+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:33:33.804138+0000","last_undegraded":"2024-09-10T18:33:33.804138+0000","last_fullsized":"2024-09-10T18:33:33.804138+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:39.119228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.119228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:55.707959+0000","scrub_duration":0.011330534999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":259,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.809628+0000","last_change":"2024-09-10T18:31:04.714213+0000","last_active":"2024-09-10T18:33:33.809628+0000","last_peered":"2024-09-10T18:33:33.809628+0000","last_clean":"2024-09-10T18:33:33.809628+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:33:33.809628+0000","last_undegraded":"2024-09-10T18:33:33.809628+0000","last_fullsized":"2024-09-10T18:33:33.809628+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:04.714112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:04.714112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0085368679999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"274'8649","reported_seq":8685,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:44.914488+0000","last_change":"2024-09-10T18:32:03.380691+0000","last_active":"2024-09-10T18:33:44.914488+0000","last_peered":"2024-09-10T18:33:44.914488+0000","last_clean":"2024-09-10T18:33:44.914488+0000","last_became_active":"2024-09-10T18:32:03.380391+0000","last_became_peered":"2024-09-10T18:32:03.380391+0000","last_unstale":"2024-09-10T18:33:44.914488+0000","last_undegraded":"2024-09-10T18:33:44.914488+0000","last_fullsized":"2024-09-10T18:33:44.914488+0000","mapping_epoch":246,"log_start":"256'3700","ondisk_log_start":"256'3700","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4949,"log_dups_size":0,"ondisk_log_size":4949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":551616512,"num_objects":8417,"num_object_clones":0,"num_object_copies":25251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8649,"num_write_kb":546112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":321,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.803755+0000","last_change":"2024-09-10T18:31:57.474380+0000","last_active":"2024-09-10T18:33:33.803755+0000","last_peered":"2024-09-10T18:33:33.803755+0000","last_clean":"2024-09-10T18:33:33.803755+0000","last_became_active":"2024-09-10T18:26:18.095568+0000","last_became_peered":"2024-09-10T18:26:18.095568+0000","last_unstale":"2024-09-10T18:33:33.803755+0000","last_undegraded":"2024-09-10T18:33:33.803755+0000","last_fullsized":"2024-09-10T18:33:33.803755+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:57.474295+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.474295+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015998690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"240'17","reported_seq":314,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.813753+0000","last_change":"2024-09-10T18:32:44.186718+0000","last_active":"2024-09-10T18:33:33.813753+0000","last_peered":"2024-09-10T18:33:33.813753+0000","last_clean":"2024-09-10T18:33:33.813753+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:33:33.813753+0000","last_undegraded":"2024-09-10T18:33:33.813753+0000","last_fullsized":"2024-09-10T18:33:33.813753+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'17","last_scrub_stamp":"2024-09-10T18:32:44.186634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:44.186634+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:53.244468+0000","scrub_duration":0.0043121310000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":300,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.812297+0000","last_change":"2024-09-10T18:33:15.142062+0000","last_active":"2024-09-10T18:33:33.812297+0000","last_peered":"2024-09-10T18:33:33.812297+0000","last_clean":"2024-09-10T18:33:33.812297+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:33:33.812297+0000","last_undegraded":"2024-09-10T18:33:33.812297+0000","last_fullsized":"2024-09-10T18:33:33.812297+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:15.141982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:15.141982+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.141982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:45.048602+0000","scrub_duration":0.0028956749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":302,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.811630+0000","last_change":"2024-09-10T18:31:54.779251+0000","last_active":"2024-09-10T18:33:33.811630+0000","last_peered":"2024-09-10T18:33:33.811630+0000","last_clean":"2024-09-10T18:33:33.811630+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:33:33.811630+0000","last_undegraded":"2024-09-10T18:33:33.811630+0000","last_fullsized":"2024-09-10T18:33:33.811630+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:31:54.779189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:54.779189+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018921529999999999,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":289,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.812190+0000","last_change":"2024-09-10T18:32:53.586654+0000","last_active":"2024-09-10T18:33:33.812190+0000","last_peered":"2024-09-10T18:33:33.812190+0000","last_clean":"2024-09-10T18:33:33.812190+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:33:33.812190+0000","last_undegraded":"2024-09-10T18:33:33.812190+0000","last_fullsized":"2024-09-10T18:33:33.812190+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:53.586216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:32:53.586216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:22.249594+0000","scrub_duration":0.0053348709999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":277,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.805354+0000","last_change":"2024-09-10T18:32:00.501396+0000","last_active":"2024-09-10T18:33:33.805354+0000","last_peered":"2024-09-10T18:33:33.805354+0000","last_clean":"2024-09-10T18:33:33.805354+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:33:33.805354+0000","last_undegraded":"2024-09-10T18:33:33.805354+0000","last_fullsized":"2024-09-10T18:33:33.805354+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.501344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.501344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001367442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"274'8697","reported_seq":8733,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:46.051164+0000","last_change":"2024-09-10T18:32:03.555231+0000","last_active":"2024-09-10T18:33:46.051164+0000","last_peered":"2024-09-10T18:33:46.051164+0000","last_clean":"2024-09-10T18:33:46.051164+0000","last_became_active":"2024-09-10T18:32:03.554958+0000","last_became_peered":"2024-09-10T18:32:03.554958+0000","last_unstale":"2024-09-10T18:33:46.051164+0000","last_undegraded":"2024-09-10T18:33:46.051164+0000","last_fullsized":"2024-09-10T18:33:46.051164+0000","mapping_epoch":246,"log_start":"252'3000","ondisk_log_start":"252'3000","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":5697,"log_dups_size":0,"ondisk_log_size":5697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540344320,"num_objects":8245,"num_object_clones":0,"num_object_copies":24735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8697,"num_write_kb":542144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":190,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.810499+0000","last_change":"2024-09-10T18:31:48.466710+0000","last_active":"2024-09-10T18:33:33.810499+0000","last_peered":"2024-09-10T18:33:33.810499+0000","last_clean":"2024-09-10T18:33:33.810499+0000","last_became_active":"2024-09-10T18:26:28.641791+0000","last_became_peered":"2024-09-10T18:26:28.641791+0000","last_unstale":"2024-09-10T18:33:33.810499+0000","last_undegraded":"2024-09-10T18:33:33.810499+0000","last_fullsized":"2024-09-10T18:33:33.810499+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:48.466654+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.466654+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":308,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804555+0000","last_change":"2024-09-10T18:32:53.270762+0000","last_active":"2024-09-10T18:33:33.804555+0000","last_peered":"2024-09-10T18:33:33.804555+0000","last_clean":"2024-09-10T18:33:33.804555+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:33:33.804555+0000","last_undegraded":"2024-09-10T18:33:33.804555+0000","last_fullsized":"2024-09-10T18:33:33.804555+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-09-10T18:32:53.270667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:53.270667+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:14.044059+0000","scrub_duration":0.01096307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":284,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.809765+0000","last_change":"2024-09-10T18:32:59.245911+0000","last_active":"2024-09-10T18:33:33.809765+0000","last_peered":"2024-09-10T18:33:33.809765+0000","last_clean":"2024-09-10T18:33:33.809765+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:33:33.809765+0000","last_undegraded":"2024-09-10T18:33:33.809765+0000","last_fullsized":"2024-09-10T18:33:33.809765+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:59.245802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:32:59.245802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:13.023525+0000","scrub_duration":0.028203379000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":1017,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:43.748212+0000","last_change":"2024-09-10T18:31:13.536758+0000","last_active":"2024-09-10T18:33:43.748212+0000","last_peered":"2024-09-10T18:33:43.748212+0000","last_clean":"2024-09-10T18:33:43.748212+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:33:43.748212+0000","last_undegraded":"2024-09-10T18:33:43.748212+0000","last_fullsized":"2024-09-10T18:33:43.748212+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:31:13.536419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:13.536419+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043736720000000003,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":726,"num_read_kb":721,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":273,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.810270+0000","last_change":"2024-09-10T18:32:56.888589+0000","last_active":"2024-09-10T18:33:33.810270+0000","last_peered":"2024-09-10T18:33:33.810270+0000","last_clean":"2024-09-10T18:33:33.810270+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:33:33.810270+0000","last_undegraded":"2024-09-10T18:33:33.810270+0000","last_fullsized":"2024-09-10T18:33:33.810270+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:56.888436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.771415+0000","last_clean_scrub_stamp":"2024-09-10T18:32:56.888436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:19.568187+0000","scrub_duration":0.0064579939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":267,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.811040+0000","last_change":"2024-09-10T18:31:57.719333+0000","last_active":"2024-09-10T18:33:33.811040+0000","last_peered":"2024-09-10T18:33:33.811040+0000","last_clean":"2024-09-10T18:33:33.811040+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:33:33.811040+0000","last_undegraded":"2024-09-10T18:33:33.811040+0000","last_fullsized":"2024-09-10T18:33:33.811040+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:57.719278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.719278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011883760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"274'8486","reported_seq":8522,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:45.648451+0000","last_change":"2024-09-10T18:32:03.382727+0000","last_active":"2024-09-10T18:33:45.648451+0000","last_peered":"2024-09-10T18:33:45.648451+0000","last_clean":"2024-09-10T18:33:45.648451+0000","last_became_active":"2024-09-10T18:32:03.382335+0000","last_became_peered":"2024-09-10T18:32:03.382335+0000","last_unstale":"2024-09-10T18:33:45.648451+0000","last_undegraded":"2024-09-10T18:33:45.648451+0000","last_fullsized":"2024-09-10T18:33:45.648451+0000","mapping_epoch":246,"log_start":"258'4100","ondisk_log_start":"258'4100","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4386,"log_dups_size":0,"ondisk_log_size":4386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":531234816,"num_objects":8106,"num_object_clones":0,"num_object_copies":24318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8486,"num_write_kb":530944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'26","reported_seq":266,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.820481+0000","last_change":"2024-09-10T18:32:42.037679+0000","last_active":"2024-09-10T18:33:33.820481+0000","last_peered":"2024-09-10T18:33:33.820481+0000","last_clean":"2024-09-10T18:33:33.820481+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:33:33.820481+0000","last_undegraded":"2024-09-10T18:33:33.820481+0000","last_fullsized":"2024-09-10T18:33:33.820481+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'26","last_scrub_stamp":"2024-09-10T18:32:42.037595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:42.037595+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:07.709417+0000","scrub_duration":0.003897162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":282,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.808775+0000","last_change":"2024-09-10T18:31:52.756369+0000","last_active":"2024-09-10T18:33:33.808775+0000","last_peered":"2024-09-10T18:33:33.808775+0000","last_clean":"2024-09-10T18:33:33.808775+0000","last_became_active":"2024-09-10T18:26:27.639129+0000","last_became_peered":"2024-09-10T18:26:27.639129+0000","last_unstale":"2024-09-10T18:33:33.808775+0000","last_undegraded":"2024-09-10T18:33:33.808775+0000","last_fullsized":"2024-09-10T18:33:33.808775+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:52.756258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:52.756258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002681269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":333,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.803952+0000","last_change":"2024-09-10T18:32:58.780690+0000","last_active":"2024-09-10T18:33:33.803952+0000","last_peered":"2024-09-10T18:33:33.803952+0000","last_clean":"2024-09-10T18:33:33.803952+0000","last_became_active":"2024-09-10T18:26:27.646658+0000","last_became_peered":"2024-09-10T18:26:27.646658+0000","last_unstale":"2024-09-10T18:33:33.803952+0000","last_undegraded":"2024-09-10T18:33:33.803952+0000","last_fullsized":"2024-09-10T18:33:33.803952+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:58.780613+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.780613+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:13.355182+0000","scrub_duration":0.002490309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":279,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.805652+0000","last_change":"2024-09-10T18:31:53.416989+0000","last_active":"2024-09-10T18:33:33.805652+0000","last_peered":"2024-09-10T18:33:33.805652+0000","last_clean":"2024-09-10T18:33:33.805652+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:33:33.805652+0000","last_undegraded":"2024-09-10T18:33:33.805652+0000","last_fullsized":"2024-09-10T18:33:33.805652+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:53.416936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:53.416936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012036219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":271,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.810156+0000","last_change":"2024-09-10T18:32:49.984765+0000","last_active":"2024-09-10T18:33:33.810156+0000","last_peered":"2024-09-10T18:33:33.810156+0000","last_clean":"2024-09-10T18:33:33.810156+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:33:33.810156+0000","last_undegraded":"2024-09-10T18:33:33.810156+0000","last_fullsized":"2024-09-10T18:33:33.810156+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:49.984707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:32:49.984707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:59.221248+0000","scrub_duration":0.0035569920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"274'8551","reported_seq":8587,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:44.900629+0000","last_change":"2024-09-10T18:32:03.384526+0000","last_active":"2024-09-10T18:33:44.900629+0000","last_peered":"2024-09-10T18:33:44.900629+0000","last_clean":"2024-09-10T18:33:44.900629+0000","last_became_active":"2024-09-10T18:32:03.384247+0000","last_became_peered":"2024-09-10T18:32:03.384247+0000","last_unstale":"2024-09-10T18:33:44.900629+0000","last_undegraded":"2024-09-10T18:33:44.900629+0000","last_fullsized":"2024-09-10T18:33:44.900629+0000","mapping_epoch":246,"log_start":"257'4061","ondisk_log_start":"257'4061","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":543096832,"num_objects":8287,"num_object_clones":0,"num_object_copies":24861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8551,"num_write_kb":538816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"240'19","reported_seq":312,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.814994+0000","last_change":"2024-09-10T18:31:50.655266+0000","last_active":"2024-09-10T18:33:33.814994+0000","last_peered":"2024-09-10T18:33:33.814994+0000","last_clean":"2024-09-10T18:33:33.814994+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:33:33.814994+0000","last_undegraded":"2024-09-10T18:33:33.814994+0000","last_fullsized":"2024-09-10T18:33:33.814994+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:31:50.655208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:50.655208+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021897840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":273,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.814434+0000","last_change":"2024-09-10T18:32:40.540692+0000","last_active":"2024-09-10T18:33:33.814434+0000","last_peered":"2024-09-10T18:33:33.814434+0000","last_clean":"2024-09-10T18:33:33.814434+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:33:33.814434+0000","last_undegraded":"2024-09-10T18:33:33.814434+0000","last_fullsized":"2024-09-10T18:33:33.814434+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:40.540616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:32:40.540616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:44.048903+0000","scrub_duration":0.002137262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":287,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.808355+0000","last_change":"2024-09-10T18:33:06.960402+0000","last_active":"2024-09-10T18:33:33.808355+0000","last_peered":"2024-09-10T18:33:33.808355+0000","last_clean":"2024-09-10T18:33:33.808355+0000","last_became_active":"2024-09-10T18:26:27.645885+0000","last_became_peered":"2024-09-10T18:26:27.645885+0000","last_unstale":"2024-09-10T18:33:33.808355+0000","last_undegraded":"2024-09-10T18:33:33.808355+0000","last_fullsized":"2024-09-10T18:33:33.808355+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:06.960320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:33:06.960320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:19.113831+0000","scrub_duration":0.0020571560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":276,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.809041+0000","last_change":"2024-09-10T18:31:50.740204+0000","last_active":"2024-09-10T18:33:33.809041+0000","last_peered":"2024-09-10T18:33:33.809041+0000","last_clean":"2024-09-10T18:33:33.809041+0000","last_became_active":"2024-09-10T18:26:27.632071+0000","last_became_peered":"2024-09-10T18:26:27.632071+0000","last_unstale":"2024-09-10T18:33:33.809041+0000","last_undegraded":"2024-09-10T18:33:33.809041+0000","last_fullsized":"2024-09-10T18:33:33.809041+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:50.740104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:50.740104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014989739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"274'8653","reported_seq":8812,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:45.347360+0000","last_change":"2024-09-10T18:33:18.299163+0000","last_active":"2024-09-10T18:33:45.347360+0000","last_peered":"2024-09-10T18:33:45.347360+0000","last_clean":"2024-09-10T18:33:45.347360+0000","last_became_active":"2024-09-10T18:32:03.554412+0000","last_became_peered":"2024-09-10T18:32:03.554412+0000","last_unstale":"2024-09-10T18:33:45.347360+0000","last_undegraded":"2024-09-10T18:33:45.347360+0000","last_fullsized":"2024-09-10T18:33:45.347360+0000","mapping_epoch":246,"log_start":"256'3900","ondisk_log_start":"256'3900","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":2999,"log_size":4753,"log_dups_size":0,"ondisk_log_size":4753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":544669696,"num_objects":8311,"num_object_clones":0,"num_object_copies":24933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8653,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":192,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.806312+0000","last_change":"2024-09-10T18:33:08.009896+0000","last_active":"2024-09-10T18:33:33.806312+0000","last_peered":"2024-09-10T18:33:33.806312+0000","last_clean":"2024-09-10T18:33:33.806312+0000","last_became_active":"2024-09-10T18:26:18.095088+0000","last_became_peered":"2024-09-10T18:26:18.095088+0000","last_unstale":"2024-09-10T18:33:33.806312+0000","last_undegraded":"2024-09-10T18:33:33.806312+0000","last_fullsized":"2024-09-10T18:33:33.806312+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:08.009806+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.009806+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:37.524807+0000","scrub_duration":0.002902614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":320,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.814563+0000","last_change":"2024-09-10T18:31:41.540517+0000","last_active":"2024-09-10T18:33:33.814563+0000","last_peered":"2024-09-10T18:33:33.814563+0000","last_clean":"2024-09-10T18:33:33.814563+0000","last_became_active":"2024-09-10T18:21:33.611203+0000","last_became_peered":"2024-09-10T18:21:33.611203+0000","last_unstale":"2024-09-10T18:33:33.814563+0000","last_undegraded":"2024-09-10T18:33:33.814563+0000","last_fullsized":"2024-09-10T18:33:33.814563+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-09-10T18:31:41.540420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:41.540420+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028551750000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":237,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.818003+0000","last_change":"2024-09-10T18:32:44.035023+0000","last_active":"2024-09-10T18:33:33.818003+0000","last_peered":"2024-09-10T18:33:33.818003+0000","last_clean":"2024-09-10T18:33:33.818003+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:33:33.818003+0000","last_undegraded":"2024-09-10T18:33:33.818003+0000","last_fullsized":"2024-09-10T18:33:33.818003+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:32:44.034968+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:32:44.034968+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:01.163735+0000","scrub_duration":0.001586772,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":279,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.811833+0000","last_change":"2024-09-10T18:32:50.211889+0000","last_active":"2024-09-10T18:33:33.811833+0000","last_peered":"2024-09-10T18:33:33.811833+0000","last_clean":"2024-09-10T18:33:33.811833+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:33:33.811833+0000","last_undegraded":"2024-09-10T18:33:33.811833+0000","last_fullsized":"2024-09-10T18:33:33.811833+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:50.211414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:50.211414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:55.885824+0000","scrub_duration":0.007128512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":297,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.812085+0000","last_change":"2024-09-10T18:32:39.620645+0000","last_active":"2024-09-10T18:33:33.812085+0000","last_peered":"2024-09-10T18:33:33.812085+0000","last_clean":"2024-09-10T18:33:33.812085+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:33:33.812085+0000","last_undegraded":"2024-09-10T18:33:33.812085+0000","last_fullsized":"2024-09-10T18:33:33.812085+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:39.620456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.620456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:50.575419+0000","scrub_duration":0.017312912,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"274'8648","reported_seq":8684,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:45.585743+0000","last_change":"2024-09-10T18:32:03.555761+0000","last_active":"2024-09-10T18:33:45.585743+0000","last_peered":"2024-09-10T18:33:45.585743+0000","last_clean":"2024-09-10T18:33:45.585743+0000","last_became_active":"2024-09-10T18:32:03.554475+0000","last_became_peered":"2024-09-10T18:32:03.554475+0000","last_unstale":"2024-09-10T18:33:45.585743+0000","last_undegraded":"2024-09-10T18:33:45.585743+0000","last_fullsized":"2024-09-10T18:33:45.585743+0000","mapping_epoch":246,"log_start":"258'4300","ondisk_log_start":"258'4300","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4348,"log_dups_size":0,"ondisk_log_size":4348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":541720576,"num_objects":8266,"num_object_clones":0,"num_object_copies":24798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8648,"num_write_kb":541248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":321,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804201+0000","last_change":"2024-09-10T18:31:10.502369+0000","last_active":"2024-09-10T18:33:33.804201+0000","last_peered":"2024-09-10T18:33:33.804201+0000","last_clean":"2024-09-10T18:33:33.804201+0000","last_became_active":"2024-09-10T18:26:18.092014+0000","last_became_peered":"2024-09-10T18:26:18.092014+0000","last_unstale":"2024-09-10T18:33:33.804201+0000","last_undegraded":"2024-09-10T18:33:33.804201+0000","last_fullsized":"2024-09-10T18:33:33.804201+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:10.502290+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:10.502290+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082798439999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"240'25","reported_seq":41,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.805225+0000","last_change":"2024-09-10T18:32:51.151265+0000","last_active":"2024-09-10T18:33:33.805225+0000","last_peered":"2024-09-10T18:33:33.805225+0000","last_clean":"2024-09-10T18:33:33.805225+0000","last_became_active":"2024-09-10T18:32:51.118251+0000","last_became_peered":"2024-09-10T18:32:51.118251+0000","last_unstale":"2024-09-10T18:33:33.805225+0000","last_undegraded":"2024-09-10T18:33:33.805225+0000","last_fullsized":"2024-09-10T18:33:33.805225+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:32:39.544846+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.544846+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:56.978883+0000","scrub_duration":0.0033922850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":275,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.808025+0000","last_change":"2024-09-10T18:33:13.324775+0000","last_active":"2024-09-10T18:33:33.808025+0000","last_peered":"2024-09-10T18:33:33.808025+0000","last_clean":"2024-09-10T18:33:33.808025+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:33:33.808025+0000","last_undegraded":"2024-09-10T18:33:33.808025+0000","last_fullsized":"2024-09-10T18:33:33.808025+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:13.324556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:13.324556+0000","last_clean_scrub_stamp":"2024-09-10T18:33:13.324556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:13.880002+0000","scrub_duration":0.011863357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"189'9","reported_seq":455,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:39.904874+0000","last_change":"2024-09-10T18:32:42.667266+0000","last_active":"2024-09-10T18:33:39.904874+0000","last_peered":"2024-09-10T18:33:39.904874+0000","last_clean":"2024-09-10T18:33:39.904874+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:33:39.904874+0000","last_undegraded":"2024-09-10T18:33:39.904874+0000","last_fullsized":"2024-09-10T18:33:39.904874+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'9","last_scrub_stamp":"2024-09-10T18:32:42.667117+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:32:42.667117+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:47.771641+0000","scrub_duration":0.0071583269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":270,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.814891+0000","last_change":"2024-09-10T18:32:46.630172+0000","last_active":"2024-09-10T18:33:33.814891+0000","last_peered":"2024-09-10T18:33:33.814891+0000","last_clean":"2024-09-10T18:33:33.814891+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:33:33.814891+0000","last_undegraded":"2024-09-10T18:33:33.814891+0000","last_fullsized":"2024-09-10T18:33:33.814891+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:46.630078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_clean_scrub_stamp":"2024-09-10T18:32:46.630078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:08.759422+0000","scrub_duration":0.0024328319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"274'8665","reported_seq":8701,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:44.913119+0000","last_change":"2024-09-10T18:32:03.384707+0000","last_active":"2024-09-10T18:33:44.913119+0000","last_peered":"2024-09-10T18:33:44.913119+0000","last_clean":"2024-09-10T18:33:44.913119+0000","last_became_active":"2024-09-10T18:32:03.384417+0000","last_became_peered":"2024-09-10T18:32:03.384417+0000","last_unstale":"2024-09-10T18:33:44.913119+0000","last_undegraded":"2024-09-10T18:33:44.913119+0000","last_fullsized":"2024-09-10T18:33:44.913119+0000","mapping_epoch":246,"log_start":"255'3700","ondisk_log_start":"255'3700","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4965,"log_dups_size":0,"ondisk_log_size":4965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":549388288,"num_objects":8383,"num_object_clones":0,"num_object_copies":25149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8665,"num_write_kb":545536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"240'15","reported_seq":297,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804487+0000","last_change":"2024-09-10T18:33:16.286611+0000","last_active":"2024-09-10T18:33:33.804487+0000","last_peered":"2024-09-10T18:33:33.804487+0000","last_clean":"2024-09-10T18:33:33.804487+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:33:33.804487+0000","last_undegraded":"2024-09-10T18:33:33.804487+0000","last_fullsized":"2024-09-10T18:33:33.804487+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:33:16.286523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:16.286523+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:29.107562+0000","scrub_duration":0.0069638809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"265'2","reported_seq":301,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.807363+0000","last_change":"2024-09-10T18:33:11.047358+0000","last_active":"2024-09-10T18:33:33.807363+0000","last_peered":"2024-09-10T18:33:33.807363+0000","last_clean":"2024-09-10T18:33:33.807363+0000","last_became_active":"2024-09-10T18:26:27.644997+0000","last_became_peered":"2024-09-10T18:26:27.644997+0000","last_unstale":"2024-09-10T18:33:33.807363+0000","last_undegraded":"2024-09-10T18:33:33.807363+0000","last_fullsized":"2024-09-10T18:33:33.807363+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'2","last_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_deep_scrub":"265'2","last_deep_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.047305+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:22.716371+0000","scrub_duration":0.021095425000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":295,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.811738+0000","last_change":"2024-09-10T18:32:52.590088+0000","last_active":"2024-09-10T18:33:33.811738+0000","last_peered":"2024-09-10T18:33:33.811738+0000","last_clean":"2024-09-10T18:33:33.811738+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:33:33.811738+0000","last_undegraded":"2024-09-10T18:33:33.811738+0000","last_fullsized":"2024-09-10T18:33:33.811738+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:52.589997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:52.589997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:18.281059+0000","scrub_duration":0.001802543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":268,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.809695+0000","last_change":"2024-09-10T18:32:44.239399+0000","last_active":"2024-09-10T18:33:33.809695+0000","last_peered":"2024-09-10T18:33:33.809695+0000","last_clean":"2024-09-10T18:33:33.809695+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:33:33.809695+0000","last_undegraded":"2024-09-10T18:33:33.809695+0000","last_fullsized":"2024-09-10T18:33:33.809695+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:44.239347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:44.239347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:11.230674+0000","scrub_duration":0.0011983390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"274'8509","reported_seq":8545,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:44.912784+0000","last_change":"2024-09-10T18:32:03.382716+0000","last_active":"2024-09-10T18:33:44.912784+0000","last_peered":"2024-09-10T18:33:44.912784+0000","last_clean":"2024-09-10T18:33:44.912784+0000","last_became_active":"2024-09-10T18:32:03.382430+0000","last_became_peered":"2024-09-10T18:32:03.382430+0000","last_unstale":"2024-09-10T18:33:44.912784+0000","last_undegraded":"2024-09-10T18:33:44.912784+0000","last_fullsized":"2024-09-10T18:33:44.912784+0000","mapping_epoch":246,"log_start":"255'3500","ondisk_log_start":"255'3500","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540082176,"num_objects":8241,"num_object_clones":0,"num_object_copies":24723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8509,"num_write_kb":536000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"240'44","reported_seq":354,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.817863+0000","last_change":"2024-09-10T18:32:39.964570+0000","last_active":"2024-09-10T18:33:33.817863+0000","last_peered":"2024-09-10T18:33:33.817863+0000","last_clean":"2024-09-10T18:33:33.817863+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:33:33.817863+0000","last_undegraded":"2024-09-10T18:33:33.817863+0000","last_fullsized":"2024-09-10T18:33:33.817863+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'44","last_scrub_stamp":"2024-09-10T18:32:39.964484+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.964484+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:43.342879+0000","scrub_duration":0.012248376,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":290,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.805636+0000","last_change":"2024-09-10T18:31:45.496272+0000","last_active":"2024-09-10T18:33:33.805636+0000","last_peered":"2024-09-10T18:33:33.805636+0000","last_clean":"2024-09-10T18:33:33.805636+0000","last_became_active":"2024-09-10T18:26:27.638408+0000","last_became_peered":"2024-09-10T18:26:27.638408+0000","last_unstale":"2024-09-10T18:33:33.805636+0000","last_undegraded":"2024-09-10T18:33:33.805636+0000","last_fullsized":"2024-09-10T18:33:33.805636+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:45.496183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:45.496183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015755630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":299,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.811799+0000","last_change":"2024-09-10T18:32:48.582057+0000","last_active":"2024-09-10T18:33:33.811799+0000","last_peered":"2024-09-10T18:33:33.811799+0000","last_clean":"2024-09-10T18:33:33.811799+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:33:33.811799+0000","last_undegraded":"2024-09-10T18:33:33.811799+0000","last_fullsized":"2024-09-10T18:33:33.811799+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:48.581963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:48.581963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:53.020863+0000","scrub_duration":0.011575692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":280,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.803916+0000","last_change":"2024-09-10T18:31:58.460588+0000","last_active":"2024-09-10T18:33:33.803916+0000","last_peered":"2024-09-10T18:33:33.803916+0000","last_clean":"2024-09-10T18:33:33.803916+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:33:33.803916+0000","last_undegraded":"2024-09-10T18:33:33.803916+0000","last_fullsized":"2024-09-10T18:33:33.803916+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:58.460533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:58.460533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001375914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"274'8569","reported_seq":8605,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:45.334268+0000","last_change":"2024-09-10T18:32:03.554922+0000","last_active":"2024-09-10T18:33:45.334268+0000","last_peered":"2024-09-10T18:33:45.334268+0000","last_clean":"2024-09-10T18:33:45.334268+0000","last_became_active":"2024-09-10T18:32:03.554423+0000","last_became_peered":"2024-09-10T18:32:03.554423+0000","last_unstale":"2024-09-10T18:33:45.334268+0000","last_undegraded":"2024-09-10T18:33:45.334268+0000","last_fullsized":"2024-09-10T18:33:45.334268+0000","mapping_epoch":246,"log_start":"256'3800","ondisk_log_start":"256'3800","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4769,"log_dups_size":0,"ondisk_log_size":4769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538116096,"num_objects":8211,"num_object_clones":0,"num_object_copies":24633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8569,"num_write_kb":536960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"240'18","reported_seq":316,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.814072+0000","last_change":"2024-09-10T18:32:54.528296+0000","last_active":"2024-09-10T18:33:33.814072+0000","last_peered":"2024-09-10T18:33:33.814072+0000","last_clean":"2024-09-10T18:33:33.814072+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:33:33.814072+0000","last_undegraded":"2024-09-10T18:33:33.814072+0000","last_fullsized":"2024-09-10T18:33:33.814072+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-09-10T18:32:54.528209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:54.528209+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.286857+0000","scrub_duration":0.0056889829999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":293,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.805826+0000","last_change":"2024-09-10T18:33:14.879881+0000","last_active":"2024-09-10T18:33:33.805826+0000","last_peered":"2024-09-10T18:33:33.805826+0000","last_clean":"2024-09-10T18:33:33.805826+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:33:33.805826+0000","last_undegraded":"2024-09-10T18:33:33.805826+0000","last_fullsized":"2024-09-10T18:33:33.805826+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:14.879787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:33:14.879787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:36.213826+0000","scrub_duration":0.0044267550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":299,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.814675+0000","last_change":"2024-09-10T18:32:50.570319+0000","last_active":"2024-09-10T18:33:33.814675+0000","last_peered":"2024-09-10T18:33:33.814675+0000","last_clean":"2024-09-10T18:33:33.814675+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:33:33.814675+0000","last_undegraded":"2024-09-10T18:33:33.814675+0000","last_fullsized":"2024-09-10T18:33:33.814675+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:50.570113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:50.570113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:05.696833+0000","scrub_duration":0.0059653969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":269,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.807725+0000","last_change":"2024-09-10T18:32:58.358775+0000","last_active":"2024-09-10T18:33:33.807725+0000","last_peered":"2024-09-10T18:33:33.807725+0000","last_clean":"2024-09-10T18:33:33.807725+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:33:33.807725+0000","last_undegraded":"2024-09-10T18:33:33.807725+0000","last_fullsized":"2024-09-10T18:33:33.807725+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:58.358694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.803236+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.358694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:15.808097+0000","scrub_duration":0.011821898000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"274'8581","reported_seq":8616,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:46.040584+0000","last_change":"2024-09-10T18:32:03.383156+0000","last_active":"2024-09-10T18:33:46.040584+0000","last_peered":"2024-09-10T18:33:46.040584+0000","last_clean":"2024-09-10T18:33:46.040584+0000","last_became_active":"2024-09-10T18:32:03.382890+0000","last_became_peered":"2024-09-10T18:32:03.382890+0000","last_unstale":"2024-09-10T18:33:46.040584+0000","last_undegraded":"2024-09-10T18:33:46.040584+0000","last_fullsized":"2024-09-10T18:33:46.040584+0000","mapping_epoch":246,"log_start":"252'2900","ondisk_log_start":"252'2900","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536674304,"num_objects":8189,"num_object_clones":0,"num_object_copies":24567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8581,"num_write_kb":536640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":322,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.806638+0000","last_change":"2024-09-10T18:31:47.535552+0000","last_active":"2024-09-10T18:33:33.806638+0000","last_peered":"2024-09-10T18:33:33.806638+0000","last_clean":"2024-09-10T18:33:33.806638+0000","last_became_active":"2024-09-10T18:26:27.645932+0000","last_became_peered":"2024-09-10T18:26:27.645932+0000","last_unstale":"2024-09-10T18:33:33.806638+0000","last_undegraded":"2024-09-10T18:33:33.806638+0000","last_fullsized":"2024-09-10T18:33:33.806638+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:31:47.535496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:47.535496+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017425819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":293,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.806084+0000","last_change":"2024-09-10T18:33:05.784757+0000","last_active":"2024-09-10T18:33:33.806084+0000","last_peered":"2024-09-10T18:33:33.806084+0000","last_clean":"2024-09-10T18:33:33.806084+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:33:33.806084+0000","last_undegraded":"2024-09-10T18:33:33.806084+0000","last_fullsized":"2024-09-10T18:33:33.806084+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:33:05.784670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:33:05.784670+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:24.433627+0000","scrub_duration":0.002289031,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":444,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:39.904791+0000","last_change":"2024-09-10T18:33:15.972246+0000","last_active":"2024-09-10T18:33:39.904791+0000","last_peered":"2024-09-10T18:33:39.904791+0000","last_clean":"2024-09-10T18:33:39.904791+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:33:39.904791+0000","last_undegraded":"2024-09-10T18:33:39.904791+0000","last_fullsized":"2024-09-10T18:33:39.904791+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:33:15.972166+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.972166+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:25.882626+0000","scrub_duration":0.004394907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":275,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.807938+0000","last_change":"2024-09-10T18:33:05.964339+0000","last_active":"2024-09-10T18:33:33.807938+0000","last_peered":"2024-09-10T18:33:33.807938+0000","last_clean":"2024-09-10T18:33:33.807938+0000","last_became_active":"2024-09-10T18:26:27.647511+0000","last_became_peered":"2024-09-10T18:26:27.647511+0000","last_unstale":"2024-09-10T18:33:33.807938+0000","last_undegraded":"2024-09-10T18:33:33.807938+0000","last_fullsized":"2024-09-10T18:33:33.807938+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:05.964034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:05.964034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:30.675049+0000","scrub_duration":0.0015507839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":228,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.820152+0000","last_change":"2024-09-10T18:32:47.037801+0000","last_active":"2024-09-10T18:33:33.820152+0000","last_peered":"2024-09-10T18:33:33.820152+0000","last_clean":"2024-09-10T18:33:33.820152+0000","last_became_active":"2024-09-10T18:26:27.640922+0000","last_became_peered":"2024-09-10T18:26:27.640922+0000","last_unstale":"2024-09-10T18:33:33.820152+0000","last_undegraded":"2024-09-10T18:33:33.820152+0000","last_fullsized":"2024-09-10T18:33:33.820152+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:47.037651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:32:47.037651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:03.275969+0000","scrub_duration":0.0021178149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"240'12","reported_seq":291,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.812647+0000","last_change":"2024-09-10T18:32:56.384627+0000","last_active":"2024-09-10T18:33:33.812647+0000","last_peered":"2024-09-10T18:33:33.812647+0000","last_clean":"2024-09-10T18:33:33.812647+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:33:33.812647+0000","last_undegraded":"2024-09-10T18:33:33.812647+0000","last_fullsized":"2024-09-10T18:33:33.812647+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-10T18:32:56.384439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:56.384439+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:05.649827+0000","scrub_duration":0.010401713999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":269,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.807780+0000","last_change":"2024-09-10T18:31:53.634789+0000","last_active":"2024-09-10T18:33:33.807780+0000","last_peered":"2024-09-10T18:33:33.807780+0000","last_clean":"2024-09-10T18:33:33.807780+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:33:33.807780+0000","last_undegraded":"2024-09-10T18:33:33.807780+0000","last_fullsized":"2024-09-10T18:33:33.807780+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:53.634704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:53.634704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014977510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":274,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.805280+0000","last_change":"2024-09-10T18:31:01.523924+0000","last_active":"2024-09-10T18:33:33.805280+0000","last_peered":"2024-09-10T18:33:33.805280+0000","last_clean":"2024-09-10T18:33:33.805280+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:33:33.805280+0000","last_undegraded":"2024-09-10T18:33:33.805280+0000","last_fullsized":"2024-09-10T18:33:33.805280+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:01.523839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:01.523839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016753880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"240'19","reported_seq":304,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804644+0000","last_change":"2024-09-10T18:33:08.284482+0000","last_active":"2024-09-10T18:33:33.804644+0000","last_peered":"2024-09-10T18:33:33.804644+0000","last_clean":"2024-09-10T18:33:33.804644+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:33:33.804644+0000","last_undegraded":"2024-09-10T18:33:33.804644+0000","last_fullsized":"2024-09-10T18:33:33.804644+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:33:08.284394+0000","last_deep_scrub":"189'13","last_deep_scrub_stamp":"2024-09-10T18:28:08.976162+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.284394+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:31.505393+0000","scrub_duration":0.010009881999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":290,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804432+0000","last_change":"2024-09-10T18:33:16.937511+0000","last_active":"2024-09-10T18:33:33.804432+0000","last_peered":"2024-09-10T18:33:33.804432+0000","last_clean":"2024-09-10T18:33:33.804432+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:33:33.804432+0000","last_undegraded":"2024-09-10T18:33:33.804432+0000","last_fullsized":"2024-09-10T18:33:33.804432+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:16.937390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:33:16.937390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:26.843652+0000","scrub_duration":0.024770533000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":271,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.812392+0000","last_change":"2024-09-10T18:32:39.161911+0000","last_active":"2024-09-10T18:33:33.812392+0000","last_peered":"2024-09-10T18:33:33.812392+0000","last_clean":"2024-09-10T18:33:33.812392+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:33:33.812392+0000","last_undegraded":"2024-09-10T18:33:33.812392+0000","last_fullsized":"2024-09-10T18:33:33.812392+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:39.161653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.161653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:58.824821+0000","scrub_duration":0.0093847749999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":287,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.812368+0000","last_change":"2024-09-10T18:32:43.621566+0000","last_active":"2024-09-10T18:33:33.812368+0000","last_peered":"2024-09-10T18:33:33.812368+0000","last_clean":"2024-09-10T18:33:33.812368+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:33:33.812368+0000","last_undegraded":"2024-09-10T18:33:33.812368+0000","last_fullsized":"2024-09-10T18:33:33.812368+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:43.621478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:43.621478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:01.039659+0000","scrub_duration":0.0031857410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"240'19","reported_seq":311,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.807746+0000","last_change":"2024-09-10T18:33:15.955719+0000","last_active":"2024-09-10T18:33:33.807746+0000","last_peered":"2024-09-10T18:33:33.807746+0000","last_clean":"2024-09-10T18:33:33.807746+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:33:33.807746+0000","last_undegraded":"2024-09-10T18:33:33.807746+0000","last_fullsized":"2024-09-10T18:33:33.807746+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:33:15.955638+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.955638+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:44.846427+0000","scrub_duration":0.021358801,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":284,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.803933+0000","last_change":"2024-09-10T18:33:12.342537+0000","last_active":"2024-09-10T18:33:33.803933+0000","last_peered":"2024-09-10T18:33:33.803933+0000","last_clean":"2024-09-10T18:33:33.803933+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:33:33.803933+0000","last_undegraded":"2024-09-10T18:33:33.803933+0000","last_fullsized":"2024-09-10T18:33:33.803933+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:12.342432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_clean_scrub_stamp":"2024-09-10T18:33:12.342432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:21.132397+0000","scrub_duration":0.0043002429999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":426,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:39.905284+0000","last_change":"2024-09-10T18:32:40.183655+0000","last_active":"2024-09-10T18:33:39.905284+0000","last_peered":"2024-09-10T18:33:39.905284+0000","last_clean":"2024-09-10T18:33:39.905284+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:33:39.905284+0000","last_undegraded":"2024-09-10T18:33:39.905284+0000","last_fullsized":"2024-09-10T18:33:39.905284+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:32:40.183571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:40.183571+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:50.634276+0000","scrub_duration":0.014890771000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.812774+0000","last_change":"2024-09-10T18:33:29.142401+0000","last_active":"2024-09-10T18:33:33.812774+0000","last_peered":"2024-09-10T18:33:33.812774+0000","last_clean":"2024-09-10T18:33:33.812774+0000","last_became_active":"2024-09-10T18:33:29.141936+0000","last_became_peered":"2024-09-10T18:33:29.141936+0000","last_unstale":"2024-09-10T18:33:33.812774+0000","last_undegraded":"2024-09-10T18:33:33.812774+0000","last_fullsized":"2024-09-10T18:33:33.812774+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:11.727786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.727786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:31.539221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":303,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.809293+0000","last_change":"2024-09-10T18:31:48.766301+0000","last_active":"2024-09-10T18:33:33.809293+0000","last_peered":"2024-09-10T18:33:33.809293+0000","last_clean":"2024-09-10T18:33:33.809293+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:33:33.809293+0000","last_undegraded":"2024-09-10T18:33:33.809293+0000","last_fullsized":"2024-09-10T18:33:33.809293+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-10T18:31:48.766245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.766245+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001995817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":283,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.805261+0000","last_change":"2024-09-10T18:32:39.535817+0000","last_active":"2024-09-10T18:33:33.805261+0000","last_peered":"2024-09-10T18:33:33.805261+0000","last_clean":"2024-09-10T18:33:33.805261+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:33:33.805261+0000","last_undegraded":"2024-09-10T18:33:33.805261+0000","last_fullsized":"2024-09-10T18:33:33.805261+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:39.535749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.535749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:03.810715+0000","scrub_duration":0.0046018179999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":435,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-10T18:33:39.905238+0000","last_change":"2024-09-10T18:32:43.942068+0000","last_active":"2024-09-10T18:33:39.905238+0000","last_peered":"2024-09-10T18:33:39.905238+0000","last_clean":"2024-09-10T18:33:39.905238+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:33:39.905238+0000","last_undegraded":"2024-09-10T18:33:39.905238+0000","last_fullsized":"2024-09-10T18:33:39.905238+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:32:43.941975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:43.941975+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:51.834001+0000","scrub_duration":0.0046448039999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":268,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804233+0000","last_change":"2024-09-10T18:32:42.165903+0000","last_active":"2024-09-10T18:33:33.804233+0000","last_peered":"2024-09-10T18:33:33.804233+0000","last_clean":"2024-09-10T18:33:33.804233+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:33:33.804233+0000","last_undegraded":"2024-09-10T18:33:33.804233+0000","last_fullsized":"2024-09-10T18:33:33.804233+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:42.165812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:42.165812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:03.644151+0000","scrub_duration":0.010729512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":319,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.814192+0000","last_change":"2024-09-10T18:33:15.612721+0000","last_active":"2024-09-10T18:33:33.814192+0000","last_peered":"2024-09-10T18:33:33.814192+0000","last_clean":"2024-09-10T18:33:33.814192+0000","last_became_active":"2024-09-10T18:26:27.649202+0000","last_became_peered":"2024-09-10T18:26:27.649202+0000","last_unstale":"2024-09-10T18:33:33.814192+0000","last_undegraded":"2024-09-10T18:33:33.814192+0000","last_fullsized":"2024-09-10T18:33:33.814192+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:33:15.612638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.612638+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:16.544192+0000","scrub_duration":0.002578828,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":283,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.817764+0000","last_change":"2024-09-10T18:32:00.649943+0000","last_active":"2024-09-10T18:33:33.817764+0000","last_peered":"2024-09-10T18:33:33.817764+0000","last_clean":"2024-09-10T18:33:33.817764+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:33:33.817764+0000","last_undegraded":"2024-09-10T18:33:33.817764+0000","last_fullsized":"2024-09-10T18:33:33.817764+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.649881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.649881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001205547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":282,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.811391+0000","last_change":"2024-09-10T18:31:56.674517+0000","last_active":"2024-09-10T18:33:33.811391+0000","last_peered":"2024-09-10T18:33:33.811391+0000","last_clean":"2024-09-10T18:33:33.811391+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:33:33.811391+0000","last_undegraded":"2024-09-10T18:33:33.811391+0000","last_fullsized":"2024-09-10T18:33:33.811391+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_clean_scrub_stamp":"2024-09-10T18:31:56.674462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0010991219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":259,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.807801+0000","last_change":"2024-09-10T18:31:45.797979+0000","last_active":"2024-09-10T18:33:33.807801+0000","last_peered":"2024-09-10T18:33:33.807801+0000","last_clean":"2024-09-10T18:33:33.807801+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:33:33.807801+0000","last_undegraded":"2024-09-10T18:33:33.807801+0000","last_fullsized":"2024-09-10T18:33:33.807801+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:45.797924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:31:45.797924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001612093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"240'15","reported_seq":313,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.811607+0000","last_change":"2024-09-10T18:32:41.169276+0000","last_active":"2024-09-10T18:33:33.811607+0000","last_peered":"2024-09-10T18:33:33.811607+0000","last_clean":"2024-09-10T18:33:33.811607+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:33:33.811607+0000","last_undegraded":"2024-09-10T18:33:33.811607+0000","last_fullsized":"2024-09-10T18:33:33.811607+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:32:41.168785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:41.168785+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:53.991999+0000","scrub_duration":0.0037973939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":286,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.805568+0000","last_change":"2024-09-10T18:33:01.781370+0000","last_active":"2024-09-10T18:33:33.805568+0000","last_peered":"2024-09-10T18:33:33.805568+0000","last_clean":"2024-09-10T18:33:33.805568+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:33:33.805568+0000","last_undegraded":"2024-09-10T18:33:33.805568+0000","last_fullsized":"2024-09-10T18:33:33.805568+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:01.781295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.781295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:27.429985+0000","scrub_duration":0.0016784879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":261,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.807799+0000","last_change":"2024-09-10T18:31:44.838204+0000","last_active":"2024-09-10T18:33:33.807799+0000","last_peered":"2024-09-10T18:33:33.807799+0000","last_clean":"2024-09-10T18:33:33.807799+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:33:33.807799+0000","last_undegraded":"2024-09-10T18:33:33.807799+0000","last_fullsized":"2024-09-10T18:33:33.807799+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:44.838141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:31:44.838141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011071239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":268,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804227+0000","last_change":"2024-09-10T18:32:02.220356+0000","last_active":"2024-09-10T18:33:33.804227+0000","last_peered":"2024-09-10T18:33:33.804227+0000","last_clean":"2024-09-10T18:33:33.804227+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:33:33.804227+0000","last_undegraded":"2024-09-10T18:33:33.804227+0000","last_fullsized":"2024-09-10T18:33:33.804227+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.220305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.220305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015049099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":311,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.808095+0000","last_change":"2024-09-10T18:32:46.253109+0000","last_active":"2024-09-10T18:33:33.808095+0000","last_peered":"2024-09-10T18:33:33.808095+0000","last_clean":"2024-09-10T18:33:33.808095+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:33:33.808095+0000","last_undegraded":"2024-09-10T18:33:33.808095+0000","last_fullsized":"2024-09-10T18:33:33.808095+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:32:46.252895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:46.252895+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:33:52.741188+0000","scrub_duration":0.003188967,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":284,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.807848+0000","last_change":"2024-09-10T18:32:01.595970+0000","last_active":"2024-09-10T18:33:33.807848+0000","last_peered":"2024-09-10T18:33:33.807848+0000","last_clean":"2024-09-10T18:33:33.807848+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:33:33.807848+0000","last_undegraded":"2024-09-10T18:33:33.807848+0000","last_fullsized":"2024-09-10T18:33:33.807848+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:01.595917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:32:01.595917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016309899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":279,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.803590+0000","last_change":"2024-09-10T18:31:50.423429+0000","last_active":"2024-09-10T18:33:33.803590+0000","last_peered":"2024-09-10T18:33:33.803590+0000","last_clean":"2024-09-10T18:33:33.803590+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:33:33.803590+0000","last_undegraded":"2024-09-10T18:33:33.803590+0000","last_fullsized":"2024-09-10T18:33:33.803590+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:50.423374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:50.423374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00116716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":286,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.807947+0000","last_change":"2024-09-10T18:33:18.227516+0000","last_active":"2024-09-10T18:33:33.807947+0000","last_peered":"2024-09-10T18:33:33.807947+0000","last_clean":"2024-09-10T18:33:33.807947+0000","last_became_active":"2024-09-10T18:26:27.636996+0000","last_became_peered":"2024-09-10T18:26:27.636996+0000","last_unstale":"2024-09-10T18:33:33.807947+0000","last_undegraded":"2024-09-10T18:33:33.807947+0000","last_fullsized":"2024-09-10T18:33:33.807947+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:18.227428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:18.227428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:27.758937+0000","scrub_duration":0.0063780959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":292,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.808106+0000","last_change":"2024-09-10T18:32:47.287083+0000","last_active":"2024-09-10T18:33:33.808106+0000","last_peered":"2024-09-10T18:33:33.808106+0000","last_clean":"2024-09-10T18:33:33.808106+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:33:33.808106+0000","last_undegraded":"2024-09-10T18:33:33.808106+0000","last_fullsized":"2024-09-10T18:33:33.808106+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-09-10T18:32:47.286977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:47.286977+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:11.371082+0000","scrub_duration":0.017807315000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":277,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.805125+0000","last_change":"2024-09-10T18:32:57.214331+0000","last_active":"2024-09-10T18:33:33.805125+0000","last_peered":"2024-09-10T18:33:33.805125+0000","last_clean":"2024-09-10T18:33:33.805125+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:33:33.805125+0000","last_undegraded":"2024-09-10T18:33:33.805125+0000","last_fullsized":"2024-09-10T18:33:33.805125+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:57.214242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:32:57.214242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:12.819926+0000","scrub_duration":0.015466415000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":232,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.818191+0000","last_change":"2024-09-10T18:33:00.161788+0000","last_active":"2024-09-10T18:33:33.818191+0000","last_peered":"2024-09-10T18:33:33.818191+0000","last_clean":"2024-09-10T18:33:33.818191+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:33:33.818191+0000","last_undegraded":"2024-09-10T18:33:33.818191+0000","last_fullsized":"2024-09-10T18:33:33.818191+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:00.161361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:33:00.161361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:05.797066+0000","scrub_duration":0.0035012799999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":277,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804038+0000","last_change":"2024-09-10T18:33:13.844984+0000","last_active":"2024-09-10T18:33:33.804038+0000","last_peered":"2024-09-10T18:33:33.804038+0000","last_clean":"2024-09-10T18:33:33.804038+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:33:33.804038+0000","last_undegraded":"2024-09-10T18:33:33.804038+0000","last_fullsized":"2024-09-10T18:33:33.804038+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:13.844781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:13.844781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:15.129270+0000","scrub_duration":0.0022954300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":287,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.810945+0000","last_change":"2024-09-10T18:33:18.177364+0000","last_active":"2024-09-10T18:33:33.810945+0000","last_peered":"2024-09-10T18:33:33.810945+0000","last_clean":"2024-09-10T18:33:33.810945+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:33:33.810945+0000","last_undegraded":"2024-09-10T18:33:33.810945+0000","last_fullsized":"2024-09-10T18:33:33.810945+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_clean_scrub_stamp":"2024-09-10T18:33:18.177264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:41.303362+0000","scrub_duration":0.0029244380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":265,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.808371+0000","last_change":"2024-09-10T18:31:43.830584+0000","last_active":"2024-09-10T18:33:33.808371+0000","last_peered":"2024-09-10T18:33:33.808371+0000","last_clean":"2024-09-10T18:33:33.808371+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:33:33.808371+0000","last_undegraded":"2024-09-10T18:33:33.808371+0000","last_fullsized":"2024-09-10T18:33:33.808371+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:43.830528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:31:43.830528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011198810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":283,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.818036+0000","last_change":"2024-09-10T18:31:54.060506+0000","last_active":"2024-09-10T18:33:33.818036+0000","last_peered":"2024-09-10T18:33:33.818036+0000","last_clean":"2024-09-10T18:33:33.818036+0000","last_became_active":"2024-09-10T18:26:27.648112+0000","last_became_peered":"2024-09-10T18:26:27.648112+0000","last_unstale":"2024-09-10T18:33:33.818036+0000","last_undegraded":"2024-09-10T18:33:33.818036+0000","last_fullsized":"2024-09-10T18:33:33.818036+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:31:54.060444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:54.060444+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019908159999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":270,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-10T18:33:33.804382+0000","last_change":"2024-09-10T18:32:00.271380+0000","last_active":"2024-09-10T18:33:33.804382+0000","last_peered":"2024-09-10T18:33:33.804382+0000","last_clean":"2024-09-10T18:33:33.804382+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:33:33.804382+0000","last_undegraded":"2024-09-10T18:33:33.804382+0000","last_fullsized":"2024-09-10T18:33:33.804382+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.271326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.271326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014948439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":8641314840,"num_objects":131857,"num_object_clones":0,"num_object_copies":395571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131857,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137321,"num_write_kb":8613633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25946304512,"data_stored":25946292296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77333,"ondisk_log_size":77333,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":731,"ondisk_log_size":731,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":726,"num_read_kb":721,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":891,"ondisk_log_size":891,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234134,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5072520,"kb_used_data":4766848,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":88672632,"statfs":{"total":95995035648,"available":90800775168,"internally_reserved":0,"allocated":4881252352,"data_stored":4880859832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":2.956,"5min":2.5059999999999998,"15min":1.7310000000000001},"min":{"1min":0.27500000000000002,"5min":0.21099999999999999,"15min":0.20799999999999999},"max":{"1min":50.091999999999999,"5min":75.506,"15min":85.012},"last":0.33400000000000002},{"interface":"front","average":{"1min":2.9100000000000001,"5min":2.3959999999999999,"15min":1.6970000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.222},"max":{"1min":50.104999999999997,"5min":75.710999999999999,"15min":84.665999999999997},"last":1.546}]},{"osd":1,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.899,"15min":1.1919999999999999},"min":{"1min":0.26300000000000001,"5min":0.214,"15min":0.214},"max":{"1min":5.8940000000000001,"5min":49.317999999999998,"15min":49.317999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.7010000000000001,"15min":1.149},"min":{"1min":0.318,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":5.8550000000000004,"5min":49.353000000000002,"15min":49.353000000000002},"last":1.4710000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.84499999999999997,"15min":0.755},"min":{"1min":0.27700000000000002,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":1.1599999999999999,"5min":5.008,"15min":5.3150000000000004},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.54100000000000004,"5min":1.008,"15min":0.83199999999999996},"min":{"1min":0.32500000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.24,"5min":5.4029999999999996,"15min":9.9450000000000003},"last":1.395}]},{"osd":3,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":1.0409999999999999,"15min":1.319},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.5190000000000001,"5min":4.4160000000000004,"15min":86.305999999999997},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.1519999999999999,"15min":1.3680000000000001},"min":{"1min":0.40200000000000002,"5min":0.29699999999999999,"15min":0.26700000000000002},"max":{"1min":7.1559999999999997,"5min":7.3339999999999996,"15min":88.364000000000004},"last":1.4279999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.60999999999999999,"15min":0.47399999999999998},"min":{"1min":0.26300000000000001,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":1.3140000000000001,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.651},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.65800000000000003,"15min":0.497},"min":{"1min":0.28299999999999997,"5min":0.26300000000000001,"15min":0.223},"max":{"1min":1.147,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":1.3660000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.98499999999999999,"15min":0.60399999999999998},"min":{"1min":0.29999999999999999,"5min":0.27600000000000002,"15min":0.23499999999999999},"max":{"1min":1.3280000000000001,"5min":7.3109999999999999,"15min":7.3109999999999999},"last":1.919},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.79700000000000004,"15min":0.56699999999999995},"min":{"1min":0.33600000000000002,"5min":0.27100000000000002,"15min":0.219},"max":{"1min":1.6240000000000001,"5min":6.3360000000000003,"15min":6.3360000000000003},"last":1.4970000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.89200000000000002,"15min":0.59699999999999998},"min":{"1min":0.33600000000000002,"5min":0.28899999999999998,"15min":0.27000000000000002},"max":{"1min":2.488,"5min":7.2750000000000004,"15min":7.2750000000000004},"last":1.4530000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.79900000000000004,"15min":0.56200000000000006},"min":{"1min":0.378,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.3199999999999998,"5min":4.585,"15min":4.585},"last":1.4430000000000001}]}]},{"osd":6,"up_from":144,"seq":618475290713,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1899380,"kb_used_data":1600940,"kb_used_omap":7,"kb_used_meta":298424,"kb_avail":91845772,"statfs":{"total":95995035648,"available":94050070528,"internally_reserved":0,"allocated":1639362560,"data_stored":1638990668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":305586263},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":1.373,"15min":1.853},"min":{"1min":0.24399999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.831,"5min":8.3949999999999996,"15min":30.629999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.95999999999999996,"15min":1.375},"min":{"1min":0.247,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":4.3310000000000004,"5min":8.4640000000000004,"15min":30.597000000000001},"last":1.1519999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.131,"15min":0.94199999999999995},"min":{"1min":0.29099999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":3.5569999999999999,"5min":8.0329999999999995,"15min":30.984000000000002},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.206,"15min":1.1399999999999999},"min":{"1min":0.30499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.1850000000000001,"5min":6.7859999999999996,"15min":30.571000000000002},"last":0.39500000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":1.0489999999999999,"15min":1.0600000000000001},"min":{"1min":0.27200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.0760000000000001,"5min":18.353000000000002,"15min":18.353000000000002},"last":1.2470000000000001},{"interface":"front","average":{"1min":0.82599999999999996,"5min":1.028,"15min":0.85099999999999998},"min":{"1min":0.28799999999999998,"5min":0.26600000000000001,"15min":0.23400000000000001},"max":{"1min":3.0880000000000001,"5min":11.164999999999999,"15min":11.164999999999999},"last":1.2729999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.2849999999999999,"15min":1.964},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27800000000000002},"max":{"1min":4.8890000000000002,"5min":16.831,"15min":61.703000000000003},"last":1.1779999999999999},{"interface":"front","average":{"1min":2.0230000000000001,"5min":1.494,"15min":2.0259999999999998},"min":{"1min":0.30399999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":17.786000000000001,"5min":17.786000000000001,"15min":61.814},"last":0.31}]},{"osd":4,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.63500000000000001,"15min":0.54500000000000004},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.23699999999999999},"max":{"1min":2.8300000000000001,"5min":5.4950000000000001,"15min":5.4950000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.60099999999999998,"15min":0.63700000000000001},"min":{"1min":0.26900000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.2440000000000002,"5min":4.6390000000000002,"15min":4.6390000000000002},"last":0.35999999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.69099999999999995,"15min":0.54400000000000004},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.24399999999999999},"max":{"1min":2.8969999999999998,"5min":4.6619999999999999,"15min":4.6619999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.68799999999999994,"15min":0.51700000000000002},"min":{"1min":0.309,"5min":0.29699999999999999,"15min":0.26800000000000002},"max":{"1min":1.9830000000000001,"5min":2.794,"15min":2.794},"last":1.022}]},{"osd":7,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.65200000000000002,"15min":0.54500000000000004},"min":{"1min":0.32600000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.929,"5min":2.7040000000000002,"15min":2.7040000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.755,"15min":0.57799999999999996},"min":{"1min":0.34399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.2320000000000002,"5min":5.5309999999999997,"15min":5.5309999999999997},"last":0.51600000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724705,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5063248,"kb_used_data":4765896,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":88681904,"statfs":{"total":95995035648,"available":90810269696,"internally_reserved":0,"allocated":4880277504,"data_stored":4879877842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:33:00 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.4099999999999999,"15min":1.4390000000000001},"min":{"1min":0.33600000000000002,"5min":0.23000000000000001,"15min":0.21099999999999999},"max":{"1min":3.2410000000000001,"5min":26.288,"15min":52.033999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":0.998,"5min":1.246,"15min":1.2589999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":5.4420000000000002,"5min":24.178000000000001,"15min":52.167999999999999},"last":1.2849999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:33:00 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.4470000000000001,"15min":0.91500000000000004},"min":{"1min":0.32700000000000001,"5min":0.315,"15min":0.22600000000000001},"max":{"1min":3.2090000000000001,"5min":24.16,"15min":24.16},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.3340000000000001,"15min":0.98299999999999998},"min":{"1min":0.43099999999999999,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":3.234,"5min":24.129999999999999,"15min":24.129999999999999},"last":0.35399999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:33:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4980000000000002,"5min":1.7330000000000001,"15min":1.1779999999999999},"min":{"1min":0.309,"5min":0.245,"15min":0.24299999999999999},"max":{"1min":54.530000000000001,"5min":54.530000000000001,"15min":54.530000000000001},"last":1.032},{"interface":"front","average":{"1min":3.3069999999999999,"5min":1.6990000000000001,"15min":1.1379999999999999},"min":{"1min":0.28999999999999998,"5min":0.247,"15min":0.247},"max":{"1min":54.514000000000003,"5min":54.514000000000003,"15min":54.514000000000003},"last":0.36699999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:33:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.5900000000000001,"15min":1.298},"min":{"1min":0.372,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.8980000000000001,"5min":30.475000000000001,"15min":31.207000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.84399999999999997,"5min":1.5089999999999999,"15min":1.1819999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.29699999999999999},"max":{"1min":1.9850000000000001,"5min":30.172000000000001,"15min":31.199000000000002},"last":0.44600000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:33:00 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.67700000000000005,"15min":0.47499999999999998},"min":{"1min":0.27700000000000002,"5min":0.22900000000000001,"15min":0.20799999999999999},"max":{"1min":0.72599999999999998,"5min":4.5199999999999996,"15min":4.5199999999999996},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.81200000000000006,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.24199999999999999,"15min":0.22700000000000001},"max":{"1min":2.786,"5min":6.798,"15min":6.798},"last":1.0009999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:33:37 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.76700000000000002,"15min":0.63500000000000001},"min":{"1min":0.29299999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.66100000000000003,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.1859999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.79800000000000004,"15min":0.63300000000000001},"min":{"1min":0.36899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.254,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.42199999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.82399999999999995,"15min":0.58199999999999996},"min":{"1min":0.39700000000000002,"5min":0.25900000000000001,"15min":0.22500000000000001},"max":{"1min":2.5859999999999999,"5min":5.569,"15min":5.569},"last":1.1699999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.73399999999999999,"15min":0.54400000000000004},"min":{"1min":0.36599999999999999,"5min":0.25900000000000001,"15min":0.24299999999999999},"max":{"1min":2.4510000000000001,"5min":3.1819999999999999,"15min":3.1819999999999999},"last":0.38100000000000001}]}]},{"osd":4,"up_from":33,"seq":141733920933,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1894788,"kb_used_data":1593276,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91850364,"statfs":{"total":95995035648,"available":94054772736,"internally_reserved":0,"allocated":1631514624,"data_stored":1631125987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:33:33 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.196,"15min":1.022},"min":{"1min":0.23599999999999999,"5min":0.21199999999999999,"15min":0.188},"max":{"1min":3.7309999999999999,"5min":6.2060000000000004,"15min":31.585000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.95299999999999996,"15min":0.97699999999999998},"min":{"1min":0.27100000000000002,"5min":0.189,"15min":0.189},"max":{"1min":3.4279999999999999,"5min":4.3129999999999997,"15min":29.802},"last":0.29099999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:33:33 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.506,"15min":0.96799999999999997},"min":{"1min":0.26900000000000002,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":3.2490000000000001,"5min":50.045000000000002,"15min":50.045000000000002},"last":0.38},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.766,"15min":1.056},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.22900000000000001},"max":{"1min":4.2789999999999999,"5min":50.283000000000001,"15min":50.283000000000001},"last":0.47699999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96999999999999997,"15min":0.85899999999999999},"min":{"1min":0.30299999999999999,"5min":0.249,"15min":0.249},"max":{"1min":3.0870000000000002,"5min":5.2380000000000004,"15min":13.898999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.999,"5min":0.96799999999999997,"15min":0.877},"min":{"1min":0.307,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":3.2519999999999998,"5min":4.1950000000000003,"15min":14.51},"last":0.42099999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:33:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.665,"15min":1.278},"min":{"1min":0.27400000000000002,"5min":0.26300000000000001,"15min":0.249},"max":{"1min":3.4049999999999998,"5min":49.990000000000002,"15min":49.990000000000002},"last":7.0330000000000004},{"interface":"front","average":{"1min":1.419,"5min":1.75,"15min":1.333},"min":{"1min":0.312,"5min":0.312,"15min":0.255},"max":{"1min":4.9740000000000002,"5min":50.764000000000003,"15min":50.764000000000003},"last":5.0439999999999996}]},{"osd":5,"last update":"Tue Sep 10 18:32:54 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.92800000000000005,"15min":0.55800000000000005},"min":{"1min":0.249,"5min":0.249,"15min":0.20499999999999999},"max":{"1min":1.484,"5min":22.861000000000001,"15min":22.861000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.95299999999999996,"15min":0.59199999999999997},"min":{"1min":0.32300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":2.5950000000000002,"5min":19.611999999999998,"15min":19.611999999999998},"last":0.59399999999999997}]},{"osd":6,"last update":"Tue Sep 10 18:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.70199999999999996,"15min":0.503},"min":{"1min":0.33900000000000002,"5min":0.27400000000000002,"15min":0.26400000000000001},"max":{"1min":3.1139999999999999,"5min":6.7990000000000004,"15min":6.7990000000000004},"last":0.433},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.70899999999999996,"15min":0.54300000000000004},"min":{"1min":0.315,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.21,"5min":7.0300000000000002,"15min":7.0300000000000002},"last":0.55500000000000005}]},{"osd":7,"last update":"Tue Sep 10 18:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.70099999999999996,"15min":0.54900000000000004},"min":{"1min":0.32200000000000001,"5min":0.28799999999999998,"15min":0.255},"max":{"1min":3.145,"5min":7.0670000000000002,"15min":7.0670000000000002},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.72099999999999997,"15min":0.56499999999999995},"min":{"1min":0.33700000000000002,"5min":0.28699999999999998,"15min":0.247},"max":{"1min":3.3450000000000002,"5min":6.5330000000000004,"15min":6.5330000000000004},"last":0.46200000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117163,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4818708,"kb_used_data":4224652,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":88926444,"statfs":{"total":95995035648,"available":91060678656,"internally_reserved":0,"allocated":4326043648,"data_stored":4325640015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:33:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.272,"15min":1.181},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":4.6079999999999997,"5min":5.4560000000000004,"15min":8.5530000000000008},"last":3.9569999999999999},{"interface":"front","average":{"1min":1.637,"5min":1.393,"15min":1.25},"min":{"1min":0.29899999999999999,"5min":0.193,"15min":0.193},"max":{"1min":8.2110000000000003,"5min":8.2110000000000003,"15min":13.659000000000001},"last":7.5979999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:33:11 2024","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.9670000000000001,"15min":1.327},"min":{"1min":0.34200000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.149,"5min":35.406999999999996,"15min":35.406999999999996},"last":4.0220000000000002},{"interface":"front","average":{"1min":1.409,"5min":1.974,"15min":1.2649999999999999},"min":{"1min":0.28100000000000003,"5min":0.20399999999999999,"15min":0.19},"max":{"1min":5.0720000000000001,"5min":36.902000000000001,"15min":36.902000000000001},"last":0.65300000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:33:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.079,"15min":1.0249999999999999},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":5.3810000000000002,"5min":5.3810000000000002,"15min":7.7960000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.4810000000000001,"15min":1.1890000000000001},"min":{"1min":0.26000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":4.4649999999999999,"5min":14.997999999999999,"15min":14.997999999999999},"last":0.45700000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":1.165,"15min":1.1220000000000001},"min":{"1min":0.34399999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.843,"5min":5.1139999999999999,"15min":23.684999999999999},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.623,"5min":1.054,"15min":1.0569999999999999},"min":{"1min":0.41099999999999998,"5min":0.29899999999999999,"15min":0.27200000000000002},"max":{"1min":0.99199999999999999,"5min":5.2160000000000002,"15min":23.649000000000001},"last":3.8570000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:32:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.333,"15min":1.218},"min":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.34499999999999997},"max":{"1min":4.0730000000000004,"5min":5.2850000000000001,"15min":23.719999999999999},"last":0.85499999999999998},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.177},"min":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.33100000000000002},"max":{"1min":6.2249999999999996,"5min":6.2249999999999996,"15min":23.707000000000001},"last":3.8180000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.639,"15min":1.133},"min":{"1min":0.379,"5min":0.32200000000000001,"15min":0.317},"max":{"1min":1.0640000000000001,"5min":31.815999999999999,"15min":31.815999999999999},"last":3.887},{"interface":"front","average":{"1min":0.90800000000000003,"5min":1.7849999999999999,"15min":1.4830000000000001},"min":{"1min":0.38500000000000001,"5min":0.33400000000000002,"15min":0.315},"max":{"1min":3.177,"5min":31.742000000000001,"15min":31.742000000000001},"last":0.91500000000000004}]},{"osd":7,"last update":"Tue Sep 10 18:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":1.2729999999999999,"15min":1.155},"min":{"1min":0.40799999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.875,"5min":5.1449999999999996,"15min":23.748999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.22,"15min":1.1819999999999999},"min":{"1min":0.47399999999999998,"5min":0.35199999999999998,"15min":0.307},"max":{"1min":12.612,"5min":12.612,"15min":25.058},"last":3.919}]}]},{"osd":2,"up_from":18,"seq":77309411503,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1902072,"kb_used_data":1604720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91843080,"statfs":{"total":95995035648,"available":94047313920,"internally_reserved":0,"allocated":1643233280,"data_stored":1642856947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.6479999999999999,"15min":1.278},"min":{"1min":0.217,"5min":0.214,"15min":0.185},"max":{"1min":1.583,"5min":43.890000000000001,"15min":43.890000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":2.0379999999999998,"15min":1.4650000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.16900000000000001},"max":{"1min":2.3439999999999999,"5min":44.331000000000003,"15min":44.331000000000003},"last":1.0820000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":2.0070000000000001,"15min":1.343},"min":{"1min":0.20200000000000001,"5min":0.193,"15min":0.193},"max":{"1min":2.5649999999999999,"5min":44.575000000000003,"15min":44.575000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":1.8440000000000001,"15min":1.1519999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":1.258,"5min":44.368000000000002,"15min":44.368000000000002},"last":0.92600000000000005}]},{"osd":3,"last update":"Tue Sep 10 18:33:07 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.669,"15min":1.4510000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.251},"max":{"1min":6.5350000000000001,"5min":15.069000000000001,"15min":60.786999999999999},"last":1.232},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.645,"15min":1.498},"min":{"1min":0.23499999999999999,"5min":0.214,"15min":0.214},"max":{"1min":4.694,"5min":15.055999999999999,"15min":60.116999999999997},"last":1.266}]},{"osd":4,"last update":"Tue Sep 10 18:33:37 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":1.266,"15min":1.0089999999999999},"min":{"1min":0.36199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":4.5570000000000004,"5min":12.555999999999999,"15min":12.555999999999999},"last":0.441},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.171,"15min":0.92700000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.317},"max":{"1min":6.3639999999999999,"5min":7.0730000000000004,"15min":12.130000000000001},"last":1.0209999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:32:55 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.2969999999999999,"15min":0.96599999999999997},"min":{"1min":0.40699999999999997,"5min":0.30299999999999999,"15min":0.27500000000000002},"max":{"1min":6.3170000000000002,"5min":7.1609999999999996,"15min":12.106},"last":1.048},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.4019999999999999,"15min":1.0149999999999999},"min":{"1min":0.371,"5min":0.32200000000000001,"15min":0.31900000000000001},"max":{"1min":12.529999999999999,"5min":12.529999999999999,"15min":12.529999999999999},"last":2.6949999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:33:37 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.234,"15min":1.125},"min":{"1min":0.34999999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":4.5899999999999999,"5min":5.9989999999999997,"15min":5.9989999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.2,"15min":0.96599999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.30099999999999999},"max":{"1min":6.4089999999999998,"5min":7.1349999999999998,"15min":7.1349999999999998},"last":0.98899999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.1830000000000001,"15min":0.98999999999999999},"min":{"1min":0.38900000000000001,"5min":0.35399999999999998,"15min":0.30499999999999999},"max":{"1min":2.6030000000000002,"5min":5.1210000000000004,"15min":7.1639999999999997},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.417,"5min":1.3899999999999999,"15min":1.0680000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.307},"max":{"1min":12.596,"5min":12.596,"15min":12.596},"last":0.40400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575028,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5098964,"kb_used_data":4797452,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":88646188,"statfs":{"total":95995035648,"available":90773696512,"internally_reserved":0,"allocated":4912590848,"data_stored":4912186453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:33:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.385,"15min":1.4730000000000001},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":4.2000000000000002,"5min":14.079000000000001,"15min":80.456999999999994},"last":2.6970000000000001},{"interface":"front","average":{"1min":1.077,"5min":1.421,"15min":1.556},"min":{"1min":0.22800000000000001,"5min":0.187,"15min":0.187},"max":{"1min":4.5410000000000004,"5min":7.3120000000000003,"15min":80.578000000000003},"last":0.30499999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:33:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":1.4219999999999999,"15min":1.1499999999999999},"min":{"1min":0.25900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.694,"5min":7.2530000000000001,"15min":13.035},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.095,"5min":1.405,"15min":1.1060000000000001},"min":{"1min":0.23799999999999999,"5min":0.216,"15min":0.19900000000000001},"max":{"1min":2.621,"5min":12.222,"15min":12.222},"last":0.28000000000000003}]},{"osd":3,"last update":"Tue Sep 10 18:33:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.4219999999999999,"15min":1.4830000000000001},"min":{"1min":0.41099999999999998,"5min":0.23899999999999999,"15min":0.22600000000000001},"max":{"1min":3.2570000000000001,"5min":14.121,"15min":43.076000000000001},"last":2.7320000000000002},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.5780000000000001,"15min":1.579},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":4.5800000000000001,"5min":7.3799999999999999,"15min":41.93},"last":0.65800000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:33:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.3879999999999999,"15min":1.0169999999999999},"min":{"1min":0.41199999999999998,"5min":0.32100000000000001,"15min":0.29799999999999999},"max":{"1min":4.6289999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"last":0.60799999999999998},{"interface":"front","average":{"1min":2.2069999999999999,"5min":1.4950000000000001,"15min":1.1850000000000001},"min":{"1min":0.308,"5min":0.30499999999999999,"15min":0.28299999999999997},"max":{"1min":7.8769999999999998,"5min":7.8769999999999998,"15min":19.393000000000001},"last":0.439}]},{"osd":5,"last update":"Tue Sep 10 18:32:56 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.5600000000000001,"15min":1.1240000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.0139999999999998,"5min":12.206,"15min":15.48},"last":1.3129999999999999},{"interface":"front","average":{"1min":0.999,"5min":1.4319999999999999,"15min":1.03},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.308},"max":{"1min":2.4769999999999999,"5min":9.9359999999999999,"15min":9.9359999999999999},"last":2.8069999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:33:42 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":1.232,"15min":1.276},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.6479999999999999,"5min":10.039,"15min":10.039},"last":2.831},{"interface":"front","average":{"1min":1.214,"5min":1.4790000000000001,"15min":1.371},"min":{"1min":0.48099999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":6.2670000000000003,"5min":7.3369999999999997,"15min":7.3369999999999997},"last":0.63800000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:33:42 2024","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.407,"15min":1.1080000000000001},"min":{"1min":0.45300000000000001,"5min":0.40400000000000003,"15min":0.34399999999999997},"max":{"1min":8.3650000000000002,"5min":10.005000000000001,"15min":15.545999999999999},"last":2.7559999999999998},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.5109999999999999,"15min":1.095},"min":{"1min":0.57999999999999996,"5min":0.35699999999999998,"15min":0.313},"max":{"1min":6.3049999999999997,"5min":7.359,"15min":7.359},"last":0.67700000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738553,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2403200,"kb_used_data":2101688,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91341952,"statfs":{"total":95995035648,"available":93534158848,"internally_reserved":0,"allocated":2152128512,"data_stored":2151743844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:33:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.9419999999999999,"15min":1.3089999999999999},"min":{"1min":0.221,"5min":0.20499999999999999,"15min":0.193},"max":{"1min":3.5920000000000001,"5min":33.130000000000003,"15min":33.130000000000003},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.9219999999999999,"15min":1.3779999999999999},"min":{"1min":0.35299999999999998,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":3.5329999999999999,"5min":31.329999999999998,"15min":31.329999999999998},"last":2.8650000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:32:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":1.532,"15min":1.242},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.189},"max":{"1min":2.1230000000000002,"5min":30.219999999999999,"15min":30.219999999999999},"last":1.331},{"interface":"front","average":{"1min":0.44,"5min":1.5369999999999999,"15min":1.2549999999999999},"min":{"1min":0.23100000000000001,"5min":0.224,"15min":0.22},"max":{"1min":1.3340000000000001,"5min":30.129999999999999,"15min":30.129999999999999},"last":0.316}]},{"osd":3,"last update":"Tue Sep 10 18:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.464,"15min":1.3640000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.157,"5min":8.5709999999999997,"15min":23.143999999999998},"last":1.8340000000000001},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.583,"15min":1.343},"min":{"1min":0.28799999999999998,"5min":0.249,"15min":0.23899999999999999},"max":{"1min":3.5720000000000001,"5min":9.3360000000000003,"15min":23.035},"last":3.73}]},{"osd":4,"last update":"Tue Sep 10 18:33:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.21,"15min":1.1259999999999999},"min":{"1min":0.36799999999999999,"5min":0.29299999999999998,"15min":0.246},"max":{"1min":4.0170000000000003,"5min":8.3699999999999992,"15min":17.151},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.0660000000000001,"15min":1.1040000000000001},"min":{"1min":0.32600000000000001,"5min":0.27500000000000002,"15min":0.25900000000000001},"max":{"1min":2.5409999999999999,"5min":4.8460000000000001,"15min":18.728999999999999},"last":1.375}]},{"osd":5,"last update":"Tue Sep 10 18:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":1.113,"15min":1.1839999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.4159999999999999,"5min":4.7220000000000004,"15min":18.77},"last":1.5449999999999999},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.264,"15min":1.1259999999999999},"min":{"1min":0.39700000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.609,"5min":6.6079999999999997,"15min":14.991},"last":0.52500000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:33:37 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.294,"15min":1.27},"min":{"1min":0.41499999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.1299999999999999,"5min":8.3940000000000001,"15min":8.3940000000000001},"last":0.872},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.2529999999999999,"15min":1.3300000000000001},"min":{"1min":0.35199999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.6680000000000001,"5min":9.7859999999999996,"15min":9.7859999999999996},"last":1.3979999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:33:37 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.3220000000000001,"15min":1.147},"min":{"1min":0.435,"5min":0.32200000000000001,"15min":0.26700000000000002},"max":{"1min":4.2539999999999996,"5min":5.0430000000000001,"15min":13.711},"last":1.5189999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.218,"15min":1.0920000000000001},"min":{"1min":0.436,"5min":0.36199999999999999,"15min":0.33500000000000002},"max":{"1min":3.1640000000000001,"5min":9.6549999999999994,"15min":15.032},"last":0.38400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2139881472,"data_stored":2139881472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4870180864,"data_stored":4870176792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1632763904,"data_stored":1632763904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4295557120,"data_stored":4295557120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1627062272,"data_stored":1627062272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4878700544,"data_stored":4878696472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1622212608,"data_stored":1622212608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4879945728,"data_stored":4879941656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:33:49.275 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-10T18:33:49.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-09-10T18:33:49.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:49.313 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:49.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:49.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:49.415 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:49.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:49.446 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:49.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:49.548 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:49.575 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:49.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:49.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:49.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:49.718 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:49.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:49.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:49.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:49.828 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:33:49.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-10T18:33:49.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:49.889 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:49.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:33:49.990 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:50.019 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:50.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:50.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:50.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:50.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:50.186 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:50.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:50.287 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:50.320 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:50.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:50.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:50.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:50.458 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:50.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:50.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:50.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:50 smithi028 ceph-mon[29291]: pgmap v552: 158 pgs: 158 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 1.10k op/s 2024-09-10T18:33:50.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:50 smithi028 ceph-mon[39220]: pgmap v552: 158 pgs: 158 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 1.10k op/s 2024-09-10T18:33:50.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:50.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:50.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:50 smithi204 ceph-mon[32357]: pgmap v552: 158 pgs: 158 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 1.10k op/s 2024-09-10T18:33:50.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:50.733 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:50.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:50.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:50.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:50.870 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:50.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:50.951 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:33:50.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:51.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:51.022 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:51.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:51.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:51.160 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:51.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:51.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:51.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:51.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:51.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:51.397 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:51.430 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:51.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:51.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:51.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:51.570 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:51.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:51.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:51.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:51.702 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:51.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:51.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:51.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:33:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:33:51] "GET /metrics HTTP/1.1" 200 224454 "" "Prometheus/2.43.0" 2024-09-10T18:33:51.803 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:51.838 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:51.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:51.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:51.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:51.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:51.975 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:51.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:52.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:52.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:51 smithi204 ceph-mon[32357]: pgmap v553: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 43 MiB/s wr, 1.38k op/s 2024-09-10T18:33:52.119 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:51 smithi028 ceph-mon[39220]: pgmap v553: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 43 MiB/s wr, 1.38k op/s 2024-09-10T18:33:52.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:51 smithi028 ceph-mon[29291]: pgmap v553: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 43 MiB/s wr, 1.38k op/s 2024-09-10T18:33:52.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:52.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:52.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:52.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:52.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:52.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:52.411 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:52.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:52.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:52.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:52.550 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:52.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:52.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:52.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:33:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:33:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:33:52.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:52.701 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:52.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:52.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:52.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:52.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:52.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:52.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:52.979 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:52.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:53.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:52 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:33:53.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:52 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:33:53.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:52 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:33:53.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:52 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:33:53.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:53.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:52 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:33:53.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:52 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:33:53.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:53.111 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:53.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:53.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:53.249 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:53.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:53.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:53.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:53.384 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:53.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:53.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:53.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:53.524 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:53.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:53.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:53.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:53.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:53.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:53.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:53.737 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:33:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:33:53.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:53.783 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:33:53.784 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:33:53.803 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:53.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:53.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:53.822 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:33:53.822 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:33:53.855 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:33:53.855 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:33:53.880 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:33:53.881 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:33:53.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:53.911 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:33:53.911 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:33:53.942 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:33:53.942 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:33:53.946 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:53.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:53.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:53.974 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:33:53.974 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:33:54.049 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:54.083 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:54.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:54.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:54.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:53 smithi204 ceph-mon[32357]: osdmap e275: 8 total, 8 up, 8 in 2024-09-10T18:33:54.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:53 smithi204 ceph-mon[32357]: pgmap v555: 158 pgs: 158 active+clean; 7.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 16 MiB/s wr, 1.21k op/s 2024-09-10T18:33:54.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:53 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2795582762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:33:54.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:53 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:33:54.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:53 smithi028 ceph-mon[39220]: osdmap e275: 8 total, 8 up, 8 in 2024-09-10T18:33:54.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:53 smithi028 ceph-mon[39220]: pgmap v555: 158 pgs: 158 active+clean; 7.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 16 MiB/s wr, 1.21k op/s 2024-09-10T18:33:54.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:53 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2795582762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:33:54.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:53 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:33:54.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:53 smithi028 ceph-mon[29291]: osdmap e275: 8 total, 8 up, 8 in 2024-09-10T18:33:54.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:53 smithi028 ceph-mon[29291]: pgmap v555: 158 pgs: 158 active+clean; 7.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 16 MiB/s wr, 1.21k op/s 2024-09-10T18:33:54.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:53 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2795582762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:33:54.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:53 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:33:54.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:54.214 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:54.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:54.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:54.316 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:54.355 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:54.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:54.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:54.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:54.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:54.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:54.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:54.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:54.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:54.739 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:54.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:54.776 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:54.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:54.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:33:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:33:54.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:54.915 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:54.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:54.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:55.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:55.051 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:55.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:55.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:55.153 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:55.183 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:55.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:55.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:55.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:55.315 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:55.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:55.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:55.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:55.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:55.457 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:55.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:55.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:55.596 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:55.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:55.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:55.759 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:55.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:55.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:55.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:33:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:55.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:55.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:33:55.966 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:33:56.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:56.040 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:56.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:56.077 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:33:56.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:56.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:56.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:56.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:56 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:56 smithi028 ceph-mon[39220]: pgmap v556: 158 pgs: 158 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 7.8 MiB/s wr, 1.29k op/s 2024-09-10T18:33:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:56 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4095233024' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:33:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:56 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:56 smithi028 ceph-mon[29291]: pgmap v556: 158 pgs: 158 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 7.8 MiB/s wr, 1.29k op/s 2024-09-10T18:33:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:56 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4095233024' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:33:56.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:33:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:56.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:56.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:33:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:56 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:33:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:56 smithi204 ceph-mon[32357]: pgmap v556: 158 pgs: 158 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 7.8 MiB/s wr, 1.29k op/s 2024-09-10T18:33:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:56 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4095233024' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:33:56.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:56.483 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:56.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:56.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:56.585 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:56.620 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:56.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:56.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:56.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:56.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:56.754 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:56.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:56.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:56.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:56.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:56.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:56.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:57.027 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:57.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:57.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:57.112 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 1 pg_num to 21 2024-09-10T18:33:57.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:57.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:57.167 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:57.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:57.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:57.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:57.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:57.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:57.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:33:57.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:57 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:33:57.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:57 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4095233024' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:33:57.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:57 smithi028 ceph-mon[29291]: osdmap e276: 8 total, 8 up, 8 in 2024-09-10T18:33:57.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:57 smithi028 ceph-mon[29291]: 1.a scrub starts 2024-09-10T18:33:57.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:57 smithi028 ceph-mon[29291]: 1.a scrub ok 2024-09-10T18:33:57.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:57 smithi028 ceph-mon[29291]: 6.6 scrub starts 2024-09-10T18:33:57.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:57 smithi028 ceph-mon[29291]: 6.6 scrub ok 2024-09-10T18:33:57.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:57 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/346742048' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-09-10T18:33:57.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:57 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:33:57.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:57 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4095233024' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:33:57.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:57 smithi028 ceph-mon[39220]: osdmap e276: 8 total, 8 up, 8 in 2024-09-10T18:33:57.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:57 smithi028 ceph-mon[39220]: 1.a scrub starts 2024-09-10T18:33:57.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:57 smithi028 ceph-mon[39220]: 1.a scrub ok 2024-09-10T18:33:57.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:57 smithi028 ceph-mon[39220]: 6.6 scrub starts 2024-09-10T18:33:57.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:57 smithi028 ceph-mon[39220]: 6.6 scrub ok 2024-09-10T18:33:57.348 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:57 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/346742048' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-09-10T18:33:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:57 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:33:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:57 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4095233024' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:33:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:57 smithi204 ceph-mon[32357]: osdmap e276: 8 total, 8 up, 8 in 2024-09-10T18:33:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:57 smithi204 ceph-mon[32357]: 1.a scrub starts 2024-09-10T18:33:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:57 smithi204 ceph-mon[32357]: 1.a scrub ok 2024-09-10T18:33:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:57 smithi204 ceph-mon[32357]: 6.6 scrub starts 2024-09-10T18:33:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:57 smithi204 ceph-mon[32357]: 6.6 scrub ok 2024-09-10T18:33:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:57 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/346742048' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-09-10T18:33:57.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:57.432 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:57.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:57.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:57.534 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:57.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:57.565 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:57.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:57.666 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:57.697 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:57.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:57.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:57.799 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:57.831 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:57.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:57.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:57.932 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:57.967 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:57.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:57.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:33:58.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:58.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:58.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:58.257 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:58.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:58.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:58.359 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:58.392 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:58.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:58.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:58.493 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:58.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:58 smithi028 ceph-mon[39220]: pgmap v558: 158 pgs: 158 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-09-10T18:33:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:58 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/346742048' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-10T18:33:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:58 smithi028 ceph-mon[39220]: osdmap e277: 8 total, 8 up, 8 in 2024-09-10T18:33:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:58 smithi028 ceph-mon[29291]: pgmap v558: 158 pgs: 158 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-09-10T18:33:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:58 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/346742048' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-10T18:33:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:58 smithi028 ceph-mon[29291]: osdmap e277: 8 total, 8 up, 8 in 2024-09-10T18:33:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:58.531 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:58.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:58 smithi204 ceph-mon[32357]: pgmap v558: 158 pgs: 158 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-09-10T18:33:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:58 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/346742048' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-10T18:33:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:58 smithi204 ceph-mon[32357]: osdmap e277: 8 total, 8 up, 8 in 2024-09-10T18:33:58.633 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:58.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:58.670 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:58.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:58.772 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:58.805 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:58.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:58.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:58.907 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:33:58.938 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:58.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:58.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:33:59.040 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:33:59.069 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:59.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:59.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:33:59.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:33:59.171 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:59.200 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:59.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:59.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:33:59.302 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:33:59.329 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:59.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:59.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:33:59.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:59 smithi028 ceph-mon[39220]: 6.0 scrub starts 2024-09-10T18:33:59.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:59 smithi028 ceph-mon[39220]: 6.0 scrub ok 2024-09-10T18:33:59.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:59 smithi028 ceph-mon[39220]: 6.15 scrub starts 2024-09-10T18:33:59.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:59 smithi028 ceph-mon[39220]: 6.15 scrub ok 2024-09-10T18:33:59.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:59 smithi028 ceph-mon[39220]: 9.4 scrub starts 2024-09-10T18:33:59.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:59 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-10T18:33:59.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:33:59 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-10T18:33:59.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:59 smithi028 ceph-mon[29291]: 6.0 scrub starts 2024-09-10T18:33:59.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:59 smithi028 ceph-mon[29291]: 6.0 scrub ok 2024-09-10T18:33:59.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:59 smithi028 ceph-mon[29291]: 6.15 scrub starts 2024-09-10T18:33:59.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:59 smithi028 ceph-mon[29291]: 6.15 scrub ok 2024-09-10T18:33:59.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:59 smithi028 ceph-mon[29291]: 9.4 scrub starts 2024-09-10T18:33:59.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:59 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-10T18:33:59.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:33:59 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-10T18:33:59.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:59.483 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:59.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:59.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:59.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:33:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:59 smithi204 ceph-mon[32357]: 6.0 scrub starts 2024-09-10T18:33:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:59 smithi204 ceph-mon[32357]: 6.0 scrub ok 2024-09-10T18:33:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:59 smithi204 ceph-mon[32357]: 6.15 scrub starts 2024-09-10T18:33:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:59 smithi204 ceph-mon[32357]: 6.15 scrub ok 2024-09-10T18:33:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:59 smithi204 ceph-mon[32357]: 9.4 scrub starts 2024-09-10T18:33:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:59 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-10T18:33:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:33:59 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-10T18:33:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:59.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:59.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:33:59.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:33:59.768 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:33:59.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:59.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:33:59.870 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:33:59.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:33:59.908 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:33:59.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:00.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:00.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:00.054 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:00.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:00.155 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:00.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:00.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:00.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:00.289 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:00.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:00.319 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:00.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:00.396 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:34:00.396 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:34:00.396 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[39220]: 4.1f scrub starts 2024-09-10T18:34:00.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[39220]: pgmap v560: 158 pgs: 158 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 824 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-10T18:34:00.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[39220]: 4.1f scrub ok 2024-09-10T18:34:00.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-10T18:34:00.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[39220]: osdmap e278: 8 total, 8 up, 8 in 2024-09-10T18:34:00.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[39220]: 4.14 scrub starts 2024-09-10T18:34:00.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[39220]: 4.14 scrub ok 2024-09-10T18:34:00.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:34:00.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:34:00.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[29291]: 4.1f scrub starts 2024-09-10T18:34:00.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[29291]: pgmap v560: 158 pgs: 158 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 824 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-10T18:34:00.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[29291]: 4.1f scrub ok 2024-09-10T18:34:00.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-10T18:34:00.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[29291]: osdmap e278: 8 total, 8 up, 8 in 2024-09-10T18:34:00.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[29291]: 4.14 scrub starts 2024-09-10T18:34:00.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:00 smithi028 ceph-mon[29291]: 4.14 scrub ok 2024-09-10T18:34:00.420 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:00.452 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:34:00.452 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-10T18:34:00.453 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-10T18:34:00.453 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-10T18:34:00.453 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:34:00.455 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:00.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:00.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:00.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:00 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:34:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:00 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:34:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:00 smithi204 ceph-mon[32357]: 4.1f scrub starts 2024-09-10T18:34:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:00 smithi204 ceph-mon[32357]: pgmap v560: 158 pgs: 158 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 824 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-10T18:34:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:00 smithi204 ceph-mon[32357]: 4.1f scrub ok 2024-09-10T18:34:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:00 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-10T18:34:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:00 smithi204 ceph-mon[32357]: osdmap e278: 8 total, 8 up, 8 in 2024-09-10T18:34:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:00 smithi204 ceph-mon[32357]: 4.14 scrub starts 2024-09-10T18:34:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:00 smithi204 ceph-mon[32357]: 4.14 scrub ok 2024-09-10T18:34:00.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:00.606 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:00.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:00.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:00.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:00.851 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:00.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:00.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:00.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:00.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:01.012 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:01.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:01.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:01.114 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:01.148 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:01.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:01.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:01.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:01.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:01.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:01.392 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:01.426 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:01.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:01.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:01.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:34:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:34:01] "GET /metrics HTTP/1.1" 200 224405 "" "Prometheus/2.43.0" 2024-09-10T18:34:01.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:01 smithi028 ceph-mon[39220]: 9.b scrub starts 2024-09-10T18:34:01.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:01 smithi028 ceph-mon[39220]: 4.1 scrub starts 2024-09-10T18:34:01.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:01 smithi028 ceph-mon[39220]: 4.1 scrub ok 2024-09-10T18:34:01.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:01 smithi028 ceph-mon[39220]: osdmap e279: 8 total, 8 up, 8 in 2024-09-10T18:34:01.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:01 smithi028 ceph-mon[39220]: 5.16 scrub starts 2024-09-10T18:34:01.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:01 smithi028 ceph-mon[39220]: 5.16 scrub ok 2024-09-10T18:34:01.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:01 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-09-10T18:34:01.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:01 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-09-10T18:34:01.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:01 smithi028 ceph-mon[29291]: 9.b scrub starts 2024-09-10T18:34:01.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:01 smithi028 ceph-mon[29291]: 4.1 scrub starts 2024-09-10T18:34:01.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:01 smithi028 ceph-mon[29291]: 4.1 scrub ok 2024-09-10T18:34:01.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:01 smithi028 ceph-mon[29291]: osdmap e279: 8 total, 8 up, 8 in 2024-09-10T18:34:01.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:01 smithi028 ceph-mon[29291]: 5.16 scrub starts 2024-09-10T18:34:01.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:01 smithi028 ceph-mon[29291]: 5.16 scrub ok 2024-09-10T18:34:01.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:01 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-09-10T18:34:01.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:01 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-09-10T18:34:01.527 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:01.558 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:01.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:01.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:01 smithi204 ceph-mon[32357]: 9.b scrub starts 2024-09-10T18:34:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:01 smithi204 ceph-mon[32357]: 4.1 scrub starts 2024-09-10T18:34:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:01 smithi204 ceph-mon[32357]: 4.1 scrub ok 2024-09-10T18:34:01.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:01 smithi204 ceph-mon[32357]: osdmap e279: 8 total, 8 up, 8 in 2024-09-10T18:34:01.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:01 smithi204 ceph-mon[32357]: 5.16 scrub starts 2024-09-10T18:34:01.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:01 smithi204 ceph-mon[32357]: 5.16 scrub ok 2024-09-10T18:34:01.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:01 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-09-10T18:34:01.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:01 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-09-10T18:34:01.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:01.705 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:01.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:01.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:01.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:01.845 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:01.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:01.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:01.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:01.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:01.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:02.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:02.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:02.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:02.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:02.266 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:02.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:02.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:02.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-10T18:34:02.368 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:02.387 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:34:02.397 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:02.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:02.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:02.422 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[39220]: 6.10 scrub starts 2024-09-10T18:34:02.422 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[39220]: 6.10 scrub ok 2024-09-10T18:34:02.422 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[39220]: 4.d scrub starts 2024-09-10T18:34:02.423 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[39220]: 4.d scrub ok 2024-09-10T18:34:02.423 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[39220]: pgmap v563: 168 pgs: 1 active+clean+scrubbing, 4 peering, 4 unknown, 159 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-10T18:34:02.423 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[39220]: 6.c scrub starts 2024-09-10T18:34:02.423 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[39220]: 6.c scrub ok 2024-09-10T18:34:02.423 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[39220]: 9.f scrub starts 2024-09-10T18:34:02.423 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-10T18:34:02.423 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[39220]: osdmap e280: 8 total, 8 up, 8 in 2024-09-10T18:34:02.423 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[29291]: 6.10 scrub starts 2024-09-10T18:34:02.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[29291]: 6.10 scrub ok 2024-09-10T18:34:02.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[29291]: 4.d scrub starts 2024-09-10T18:34:02.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[29291]: 4.d scrub ok 2024-09-10T18:34:02.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[29291]: pgmap v563: 168 pgs: 1 active+clean+scrubbing, 4 peering, 4 unknown, 159 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-10T18:34:02.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[29291]: 6.c scrub starts 2024-09-10T18:34:02.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[29291]: 6.c scrub ok 2024-09-10T18:34:02.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[29291]: 9.f scrub starts 2024-09-10T18:34:02.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-10T18:34:02.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:02 smithi028 ceph-mon[29291]: osdmap e280: 8 total, 8 up, 8 in 2024-09-10T18:34:02.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:02.529 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:02.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:02.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:02.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:34:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:34:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:34:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:02 smithi204 ceph-mon[32357]: 6.10 scrub starts 2024-09-10T18:34:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:02 smithi204 ceph-mon[32357]: 6.10 scrub ok 2024-09-10T18:34:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:02 smithi204 ceph-mon[32357]: 4.d scrub starts 2024-09-10T18:34:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:02 smithi204 ceph-mon[32357]: 4.d scrub ok 2024-09-10T18:34:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:02 smithi204 ceph-mon[32357]: pgmap v563: 168 pgs: 1 active+clean+scrubbing, 4 peering, 4 unknown, 159 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-10T18:34:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:02 smithi204 ceph-mon[32357]: 6.c scrub starts 2024-09-10T18:34:02.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:02 smithi204 ceph-mon[32357]: 6.c scrub ok 2024-09-10T18:34:02.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:02 smithi204 ceph-mon[32357]: 9.f scrub starts 2024-09-10T18:34:02.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:02 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-10T18:34:02.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:02 smithi204 ceph-mon[32357]: osdmap e280: 8 total, 8 up, 8 in 2024-09-10T18:34:02.631 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:02.665 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:02.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:02.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:02.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:02.814 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:02.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:02.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:02.916 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:02.947 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:02.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:02.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:03.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:03.091 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:03.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:03.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:03.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:03.226 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:03.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:03.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:03.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:03.387 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:03.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:03.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:03 smithi028 ceph-mon[39220]: 9.b scrub ok 2024-09-10T18:34:03.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:03 smithi028 ceph-mon[39220]: 9.4 scrub ok 2024-09-10T18:34:03.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:03 smithi028 ceph-mon[39220]: 2.0 scrub starts 2024-09-10T18:34:03.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:03 smithi028 ceph-mon[39220]: 2.0 scrub ok 2024-09-10T18:34:03.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:03 smithi028 ceph-mon[39220]: 3.4 scrub starts 2024-09-10T18:34:03.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:03 smithi028 ceph-mon[39220]: 3.4 scrub ok 2024-09-10T18:34:03.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:03 smithi028 ceph-mon[39220]: osdmap e281: 8 total, 8 up, 8 in 2024-09-10T18:34:03.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:03 smithi028 ceph-mon[29291]: 9.b scrub ok 2024-09-10T18:34:03.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:03 smithi028 ceph-mon[29291]: 9.4 scrub ok 2024-09-10T18:34:03.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:03 smithi028 ceph-mon[29291]: 2.0 scrub starts 2024-09-10T18:34:03.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:03 smithi028 ceph-mon[29291]: 2.0 scrub ok 2024-09-10T18:34:03.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:03 smithi028 ceph-mon[29291]: 3.4 scrub starts 2024-09-10T18:34:03.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:03 smithi028 ceph-mon[29291]: 3.4 scrub ok 2024-09-10T18:34:03.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:03 smithi028 ceph-mon[29291]: osdmap e281: 8 total, 8 up, 8 in 2024-09-10T18:34:03.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:03.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:03.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:03 smithi204 ceph-mon[32357]: 9.b scrub ok 2024-09-10T18:34:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:03 smithi204 ceph-mon[32357]: 9.4 scrub ok 2024-09-10T18:34:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:03 smithi204 ceph-mon[32357]: 2.0 scrub starts 2024-09-10T18:34:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:03 smithi204 ceph-mon[32357]: 2.0 scrub ok 2024-09-10T18:34:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:03 smithi204 ceph-mon[32357]: 3.4 scrub starts 2024-09-10T18:34:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:03 smithi204 ceph-mon[32357]: 3.4 scrub ok 2024-09-10T18:34:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:03 smithi204 ceph-mon[32357]: osdmap e281: 8 total, 8 up, 8 in 2024-09-10T18:34:03.629 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:03.661 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:03.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:03.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:03.709 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:34:03.710 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:34:03.710 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-10T18:34:03.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:34:03.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:03.805 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:34:03.805 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:34:03.819 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:03.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:03.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:03.843 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:34:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:34:03.879 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:34:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:34:03.915 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:34:03.915 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:34:03.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:03.947 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:34:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:34:03.969 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:03.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:03.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:03.990 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:34:03.990 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:34:04.026 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:34:04.026 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:34:04.056 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:34:04.056 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:34:04.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:04.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:04.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:04.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:04.200 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:04.230 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:04.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:04.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:04.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[29291]: 5.1e scrub starts 2024-09-10T18:34:04.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[29291]: 5.1e scrub ok 2024-09-10T18:34:04.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[29291]: 4.a scrub starts 2024-09-10T18:34:04.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[29291]: pgmap v566: 168 pgs: 2 active+clean+scrubbing, 4 peering, 4 unknown, 158 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s 2024-09-10T18:34:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[29291]: 4.a scrub ok 2024-09-10T18:34:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[29291]: 3.18 scrub starts 2024-09-10T18:34:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[29291]: 3.18 scrub ok 2024-09-10T18:34:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[29291]: 5.12 scrub starts 2024-09-10T18:34:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[29291]: 5.12 scrub ok 2024-09-10T18:34:04.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3425967499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:34:04.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[39220]: 5.1e scrub starts 2024-09-10T18:34:04.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[39220]: 5.1e scrub ok 2024-09-10T18:34:04.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[39220]: 4.a scrub starts 2024-09-10T18:34:04.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[39220]: pgmap v566: 168 pgs: 2 active+clean+scrubbing, 4 peering, 4 unknown, 158 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s 2024-09-10T18:34:04.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[39220]: 4.a scrub ok 2024-09-10T18:34:04.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[39220]: 3.18 scrub starts 2024-09-10T18:34:04.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[39220]: 3.18 scrub ok 2024-09-10T18:34:04.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[39220]: 5.12 scrub starts 2024-09-10T18:34:04.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[39220]: 5.12 scrub ok 2024-09-10T18:34:04.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:04 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3425967499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:34:04.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:04.370 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:04.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:04.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:04.471 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:04.506 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:04.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:04.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:04.574 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:04 smithi204 ceph-mon[32357]: 5.1e scrub starts 2024-09-10T18:34:04.574 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:04 smithi204 ceph-mon[32357]: 5.1e scrub ok 2024-09-10T18:34:04.574 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:04 smithi204 ceph-mon[32357]: 4.a scrub starts 2024-09-10T18:34:04.574 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:04 smithi204 ceph-mon[32357]: pgmap v566: 168 pgs: 2 active+clean+scrubbing, 4 peering, 4 unknown, 158 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s 2024-09-10T18:34:04.574 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:04 smithi204 ceph-mon[32357]: 4.a scrub ok 2024-09-10T18:34:04.574 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:04 smithi204 ceph-mon[32357]: 3.18 scrub starts 2024-09-10T18:34:04.574 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:04 smithi204 ceph-mon[32357]: 3.18 scrub ok 2024-09-10T18:34:04.574 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:04 smithi204 ceph-mon[32357]: 5.12 scrub starts 2024-09-10T18:34:04.574 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:04 smithi204 ceph-mon[32357]: 5.12 scrub ok 2024-09-10T18:34:04.574 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:04 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3425967499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:34:04.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:04.649 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:04.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:04.750 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:04.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:04.785 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:04.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:04.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:34:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:34:04.886 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:04.913 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:04.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:04.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:05.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:05.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:05.066 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:05.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:05.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:05.207 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:05.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:05.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:05.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:05.355 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:05.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:05.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:05.457 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:05.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:05.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:05.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:05.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:05 smithi028 ceph-mon[39220]: 9.a scrub starts 2024-09-10T18:34:05.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:05 smithi028 ceph-mon[39220]: 9.f scrub ok 2024-09-10T18:34:05.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:05 smithi028 ceph-mon[39220]: 9.2 scrub starts 2024-09-10T18:34:05.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:05 smithi028 ceph-mon[39220]: 9.0 scrub starts 2024-09-10T18:34:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:05 smithi028 ceph-mon[29291]: 9.a scrub starts 2024-09-10T18:34:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:05 smithi028 ceph-mon[29291]: 9.f scrub ok 2024-09-10T18:34:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:05 smithi028 ceph-mon[29291]: 9.2 scrub starts 2024-09-10T18:34:05.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:05 smithi028 ceph-mon[29291]: 9.0 scrub starts 2024-09-10T18:34:05.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:05 smithi204 ceph-mon[32357]: 9.a scrub starts 2024-09-10T18:34:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:05 smithi204 ceph-mon[32357]: 9.f scrub ok 2024-09-10T18:34:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:05 smithi204 ceph-mon[32357]: 9.2 scrub starts 2024-09-10T18:34:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:05 smithi204 ceph-mon[32357]: 9.0 scrub starts 2024-09-10T18:34:05.631 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:05.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:05.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:05.784 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:05.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:05.807 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:34:05.808 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:34:05.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:05.939 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:05.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:05.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:06.041 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:06.077 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:06.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:06.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:06.179 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:06.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:06.207 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:06.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:06.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:06.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:06.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:06.448 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:06.483 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:06.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:06.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:06.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[29291]: 3.14 scrub starts 2024-09-10T18:34:06.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[29291]: 3.14 scrub ok 2024-09-10T18:34:06.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[29291]: 1.6 scrub starts 2024-09-10T18:34:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[29291]: 1.6 scrub ok 2024-09-10T18:34:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[29291]: pgmap v567: 168 pgs: 3 active+clean+scrubbing, 1 unknown, 164 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 8.5 KiB/s rd, 50 KiB/s wr, 1.64k op/s; 36 KiB/s, 0 objects/s recovering 2024-09-10T18:34:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:34:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:34:06.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[29291]: osdmap e282: 8 total, 8 up, 8 in 2024-09-10T18:34:06.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[39220]: 3.14 scrub starts 2024-09-10T18:34:06.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[39220]: 3.14 scrub ok 2024-09-10T18:34:06.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:06.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[39220]: 1.6 scrub starts 2024-09-10T18:34:06.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[39220]: 1.6 scrub ok 2024-09-10T18:34:06.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[39220]: pgmap v567: 168 pgs: 3 active+clean+scrubbing, 1 unknown, 164 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 8.5 KiB/s rd, 50 KiB/s wr, 1.64k op/s; 36 KiB/s, 0 objects/s recovering 2024-09-10T18:34:06.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:34:06.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:34:06.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:06 smithi028 ceph-mon[39220]: osdmap e282: 8 total, 8 up, 8 in 2024-09-10T18:34:06.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:06 smithi204 ceph-mon[32357]: 3.14 scrub starts 2024-09-10T18:34:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:06 smithi204 ceph-mon[32357]: 3.14 scrub ok 2024-09-10T18:34:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:06 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:06 smithi204 ceph-mon[32357]: 1.6 scrub starts 2024-09-10T18:34:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:06 smithi204 ceph-mon[32357]: 1.6 scrub ok 2024-09-10T18:34:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:06 smithi204 ceph-mon[32357]: pgmap v567: 168 pgs: 3 active+clean+scrubbing, 1 unknown, 164 active+clean; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 8.5 KiB/s rd, 50 KiB/s wr, 1.64k op/s; 36 KiB/s, 0 objects/s recovering 2024-09-10T18:34:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:06 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:34:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:06 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:34:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:06 smithi204 ceph-mon[32357]: osdmap e282: 8 total, 8 up, 8 in 2024-09-10T18:34:06.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:06.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:06.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:06.772 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:06.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:06.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:06.874 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:06.910 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:06.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:07.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:07.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:07.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:07.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:07.152 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:07.183 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:07.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:07.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:07.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:07.320 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:07.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:07.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:07.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:07.443 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:07.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:07.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:07 smithi028 ceph-mon[39220]: 1.9 scrub starts 2024-09-10T18:34:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:07 smithi028 ceph-mon[39220]: 1.9 scrub ok 2024-09-10T18:34:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:07 smithi028 ceph-mon[39220]: 4.8 scrub starts 2024-09-10T18:34:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:07 smithi028 ceph-mon[39220]: 4.8 scrub ok 2024-09-10T18:34:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:07 smithi028 ceph-mon[39220]: 2.1 scrub starts 2024-09-10T18:34:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:07 smithi028 ceph-mon[39220]: 2.1 scrub ok 2024-09-10T18:34:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:07 smithi028 ceph-mon[39220]: 9.a scrub ok 2024-09-10T18:34:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:07 smithi028 ceph-mon[29291]: 1.9 scrub starts 2024-09-10T18:34:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:07 smithi028 ceph-mon[29291]: 1.9 scrub ok 2024-09-10T18:34:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:07 smithi028 ceph-mon[29291]: 4.8 scrub starts 2024-09-10T18:34:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:07 smithi028 ceph-mon[29291]: 4.8 scrub ok 2024-09-10T18:34:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:07 smithi028 ceph-mon[29291]: 2.1 scrub starts 2024-09-10T18:34:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:07 smithi028 ceph-mon[29291]: 2.1 scrub ok 2024-09-10T18:34:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:07 smithi028 ceph-mon[29291]: 9.a scrub ok 2024-09-10T18:34:07.546 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:07.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:07.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:07 smithi204 ceph-mon[32357]: 1.9 scrub starts 2024-09-10T18:34:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:07 smithi204 ceph-mon[32357]: 1.9 scrub ok 2024-09-10T18:34:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:07 smithi204 ceph-mon[32357]: 4.8 scrub starts 2024-09-10T18:34:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:07 smithi204 ceph-mon[32357]: 4.8 scrub ok 2024-09-10T18:34:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:07 smithi204 ceph-mon[32357]: 2.1 scrub starts 2024-09-10T18:34:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:07 smithi204 ceph-mon[32357]: 2.1 scrub ok 2024-09-10T18:34:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:07 smithi204 ceph-mon[32357]: 9.a scrub ok 2024-09-10T18:34:07.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:34:07.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:07.760 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:07.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:07.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:07.862 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:07.889 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:07.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:07.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:07.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:08.012 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:08.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:08.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:08.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:08.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:08.155 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:08.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:08.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:08.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:08.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:08.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:08.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:08.433 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:08.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:08 smithi028 ceph-mon[39220]: 5.13 scrub starts 2024-09-10T18:34:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:08 smithi028 ceph-mon[39220]: pgmap v569: 168 pgs: 3 active+clean+scrubbing, 165 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 9.2 KiB/s rd, 62 KiB/s wr, 1.32k op/s; 97 KiB/s, 0 objects/s recovering 2024-09-10T18:34:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:08 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:34:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:08 smithi028 ceph-mon[29291]: 5.13 scrub starts 2024-09-10T18:34:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:08 smithi028 ceph-mon[29291]: pgmap v569: 168 pgs: 3 active+clean+scrubbing, 165 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 9.2 KiB/s rd, 62 KiB/s wr, 1.32k op/s; 97 KiB/s, 0 objects/s recovering 2024-09-10T18:34:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:08 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:34:08.534 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:08.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:08.568 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:08.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:08 smithi204 ceph-mon[32357]: 5.13 scrub starts 2024-09-10T18:34:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:08 smithi204 ceph-mon[32357]: pgmap v569: 168 pgs: 3 active+clean+scrubbing, 165 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 9.2 KiB/s rd, 62 KiB/s wr, 1.32k op/s; 97 KiB/s, 0 objects/s recovering 2024-09-10T18:34:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:08 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:34:08.669 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:08.703 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:08.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:08.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:08.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:08.827 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:08.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:08.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:08.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:08.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:08.975 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:08.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:09.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:09.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:09.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:09.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:09.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:09.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:09.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:09.374 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:09.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:09.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:09.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[29291]: 6.1e scrub starts 2024-09-10T18:34:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[29291]: 6.1e scrub ok 2024-09-10T18:34:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[29291]: 9.2 scrub ok 2024-09-10T18:34:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[29291]: 9.0 scrub ok 2024-09-10T18:34:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[29291]: 1.7 scrub starts 2024-09-10T18:34:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[29291]: 1.7 scrub ok 2024-09-10T18:34:09.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[29291]: 3.16 scrub starts 2024-09-10T18:34:09.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[29291]: 3.16 scrub ok 2024-09-10T18:34:09.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:34:09.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:34:09.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:34:09.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:34:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[39220]: 6.1e scrub starts 2024-09-10T18:34:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[39220]: 6.1e scrub ok 2024-09-10T18:34:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[39220]: 9.2 scrub ok 2024-09-10T18:34:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[39220]: 9.0 scrub ok 2024-09-10T18:34:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[39220]: 1.7 scrub starts 2024-09-10T18:34:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[39220]: 1.7 scrub ok 2024-09-10T18:34:09.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[39220]: 3.16 scrub starts 2024-09-10T18:34:09.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[39220]: 3.16 scrub ok 2024-09-10T18:34:09.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:34:09.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:34:09.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:34:09.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:34:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:09.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:09.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:09 smithi204 ceph-mon[32357]: 6.1e scrub starts 2024-09-10T18:34:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:09 smithi204 ceph-mon[32357]: 6.1e scrub ok 2024-09-10T18:34:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:09 smithi204 ceph-mon[32357]: 9.2 scrub ok 2024-09-10T18:34:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:09 smithi204 ceph-mon[32357]: 9.0 scrub ok 2024-09-10T18:34:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:09 smithi204 ceph-mon[32357]: 1.7 scrub starts 2024-09-10T18:34:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:09 smithi204 ceph-mon[32357]: 1.7 scrub ok 2024-09-10T18:34:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:09 smithi204 ceph-mon[32357]: 3.16 scrub starts 2024-09-10T18:34:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:09 smithi204 ceph-mon[32357]: 3.16 scrub ok 2024-09-10T18:34:09.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:34:09.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:34:09.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:34:09.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:34:09.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:09.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:09.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:09.749 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:09.784 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:09.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:09.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:09.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:09.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:09.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:10.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:10.041 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:10.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:10.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:10.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:10.173 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:10.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:10.275 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:10.296 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:10.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:10.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:10.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:10.427 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:10 smithi028 ceph-mon[39220]: pgmap v570: 168 pgs: 3 active+clean+scrubbing, 165 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 7.6 KiB/s rd, 51 KiB/s wr, 917 op/s; 81 KiB/s, 0 objects/s recovering 2024-09-10T18:34:10.427 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:10 smithi028 ceph-mon[39220]: 6.19 scrub starts 2024-09-10T18:34:10.427 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:10 smithi028 ceph-mon[39220]: 6.19 scrub ok 2024-09-10T18:34:10.427 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:10 smithi028 ceph-mon[39220]: 9.1 scrub starts 2024-09-10T18:34:10.427 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:10 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/408329060' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:34:10.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:10 smithi028 ceph-mon[29291]: pgmap v570: 168 pgs: 3 active+clean+scrubbing, 165 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 7.6 KiB/s rd, 51 KiB/s wr, 917 op/s; 81 KiB/s, 0 objects/s recovering 2024-09-10T18:34:10.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:10 smithi028 ceph-mon[29291]: 6.19 scrub starts 2024-09-10T18:34:10.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:10 smithi028 ceph-mon[29291]: 6.19 scrub ok 2024-09-10T18:34:10.428 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:10 smithi028 ceph-mon[29291]: 9.1 scrub starts 2024-09-10T18:34:10.428 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:10 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/408329060' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:34:10.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:10.434 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:10.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:10.535 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:10.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:10.557 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:10.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:10 smithi204 ceph-mon[32357]: pgmap v570: 168 pgs: 3 active+clean+scrubbing, 165 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 7.6 KiB/s rd, 51 KiB/s wr, 917 op/s; 81 KiB/s, 0 objects/s recovering 2024-09-10T18:34:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:10 smithi204 ceph-mon[32357]: 6.19 scrub starts 2024-09-10T18:34:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:10 smithi204 ceph-mon[32357]: 6.19 scrub ok 2024-09-10T18:34:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:10 smithi204 ceph-mon[32357]: 9.1 scrub starts 2024-09-10T18:34:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:10 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/408329060' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:34:10.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:10.694 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:10.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:10.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:10.796 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:10.829 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:10.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:10.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:10.931 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:10.953 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:10.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:10.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:11.055 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:11.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:11.092 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:11.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:11.186 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:34:11.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:11.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:11.227 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:11.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:11.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:11.358 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:11.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:11.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:11.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:11.492 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:11.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:11.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:11.503 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:34:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:34:11] "GET /metrics HTTP/1.1" 200 224503 "" "Prometheus/2.43.0" 2024-09-10T18:34:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:11 smithi028 ceph-mon[39220]: 5.18 scrub starts 2024-09-10T18:34:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:11 smithi028 ceph-mon[39220]: 5.18 scrub ok 2024-09-10T18:34:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:11 smithi028 ceph-mon[39220]: 3.a scrub starts 2024-09-10T18:34:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:11 smithi028 ceph-mon[39220]: 3.a scrub ok 2024-09-10T18:34:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:11 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:34:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:11 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:34:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:11 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/408329060' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:34:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:11 smithi028 ceph-mon[39220]: osdmap e283: 8 total, 8 up, 8 in 2024-09-10T18:34:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:11 smithi028 ceph-mon[29291]: 5.18 scrub starts 2024-09-10T18:34:11.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:11 smithi028 ceph-mon[29291]: 5.18 scrub ok 2024-09-10T18:34:11.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:11 smithi028 ceph-mon[29291]: 3.a scrub starts 2024-09-10T18:34:11.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:11 smithi028 ceph-mon[29291]: 3.a scrub ok 2024-09-10T18:34:11.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:11 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:34:11.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:11 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:34:11.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:11 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/408329060' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:34:11.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:11 smithi028 ceph-mon[29291]: osdmap e283: 8 total, 8 up, 8 in 2024-09-10T18:34:11.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:11 smithi204 ceph-mon[32357]: 5.18 scrub starts 2024-09-10T18:34:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:11 smithi204 ceph-mon[32357]: 5.18 scrub ok 2024-09-10T18:34:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:11 smithi204 ceph-mon[32357]: 3.a scrub starts 2024-09-10T18:34:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:11 smithi204 ceph-mon[32357]: 3.a scrub ok 2024-09-10T18:34:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:11 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:34:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:11 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:34:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:11 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/408329060' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:34:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:11 smithi204 ceph-mon[32357]: osdmap e283: 8 total, 8 up, 8 in 2024-09-10T18:34:11.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:11.634 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:11.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:11.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:11.773 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:11.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:11.876 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:11.914 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:11.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:11.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:12.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:12.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:12.048 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:12.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:12.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:12.182 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:12.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:12.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:12.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:12.323 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:12.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:12.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:12.425 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:12.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:12.459 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:12.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:12.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:12 smithi028 ceph-mon[29291]: 9.5 scrub starts 2024-09-10T18:34:12.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:12 smithi028 ceph-mon[29291]: pgmap v572: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 7.5 KiB/s rd, 46 KiB/s wr, 1.39k op/s; 73 KiB/s, 0 objects/s recovering 2024-09-10T18:34:12.484 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:12 smithi028 ceph-mon[39220]: 9.5 scrub starts 2024-09-10T18:34:12.484 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:12 smithi028 ceph-mon[39220]: pgmap v572: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 7.5 KiB/s rd, 46 KiB/s wr, 1.39k op/s; 73 KiB/s, 0 objects/s recovering 2024-09-10T18:34:12.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:12.599 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:12.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:12.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:12 smithi204 ceph-mon[32357]: 9.5 scrub starts 2024-09-10T18:34:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:12 smithi204 ceph-mon[32357]: pgmap v572: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 7.5 KiB/s rd, 46 KiB/s wr, 1.39k op/s; 73 KiB/s, 0 objects/s recovering 2024-09-10T18:34:12.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:34:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:34:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:34:12.701 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:12.734 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:12.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:12.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:12.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:12.877 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:12.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:12.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:12.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:13.013 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:13.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:13.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:13.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:13.144 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:13.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:13.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:13.245 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:13.277 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:13.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:13.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:13.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:13.417 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:13.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:13 smithi028 ceph-mon[39220]: 3.d scrub starts 2024-09-10T18:34:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:13 smithi028 ceph-mon[39220]: 3.d scrub ok 2024-09-10T18:34:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:13 smithi028 ceph-mon[39220]: 9.1 scrub ok 2024-09-10T18:34:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:13 smithi028 ceph-mon[39220]: 9.6 deep-scrub starts 2024-09-10T18:34:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:13 smithi028 ceph-mon[39220]: 6.4 scrub starts 2024-09-10T18:34:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:13 smithi028 ceph-mon[39220]: 6.4 scrub ok 2024-09-10T18:34:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:13 smithi028 ceph-mon[39220]: 5.4 scrub starts 2024-09-10T18:34:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:13 smithi028 ceph-mon[39220]: 5.4 scrub ok 2024-09-10T18:34:13.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:13 smithi028 ceph-mon[29291]: 3.d scrub starts 2024-09-10T18:34:13.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:13 smithi028 ceph-mon[29291]: 3.d scrub ok 2024-09-10T18:34:13.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:13 smithi028 ceph-mon[29291]: 9.1 scrub ok 2024-09-10T18:34:13.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:13 smithi028 ceph-mon[29291]: 9.6 deep-scrub starts 2024-09-10T18:34:13.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:13 smithi028 ceph-mon[29291]: 6.4 scrub starts 2024-09-10T18:34:13.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:13 smithi028 ceph-mon[29291]: 6.4 scrub ok 2024-09-10T18:34:13.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:13 smithi028 ceph-mon[29291]: 5.4 scrub starts 2024-09-10T18:34:13.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:13 smithi028 ceph-mon[29291]: 5.4 scrub ok 2024-09-10T18:34:13.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:13.551 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:13.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:13.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:13 smithi204 ceph-mon[32357]: 3.d scrub starts 2024-09-10T18:34:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:13 smithi204 ceph-mon[32357]: 3.d scrub ok 2024-09-10T18:34:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:13 smithi204 ceph-mon[32357]: 9.1 scrub ok 2024-09-10T18:34:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:13 smithi204 ceph-mon[32357]: 9.6 deep-scrub starts 2024-09-10T18:34:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:13 smithi204 ceph-mon[32357]: 6.4 scrub starts 2024-09-10T18:34:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:13 smithi204 ceph-mon[32357]: 6.4 scrub ok 2024-09-10T18:34:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:13 smithi204 ceph-mon[32357]: 5.4 scrub starts 2024-09-10T18:34:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:13 smithi204 ceph-mon[32357]: 5.4 scrub ok 2024-09-10T18:34:13.654 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:13.687 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:13.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:13.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:13.789 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:13.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:13.819 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:13.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:34:13.920 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:13.950 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:13.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:14.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:14.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:14.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:14.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:14.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:14.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:14 smithi028 ceph-mon[29291]: 3.1b scrub starts 2024-09-10T18:34:14.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:14 smithi028 ceph-mon[29291]: 3.1b scrub ok 2024-09-10T18:34:14.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:14 smithi028 ceph-mon[29291]: 9.5 scrub ok 2024-09-10T18:34:14.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:14 smithi028 ceph-mon[29291]: pgmap v573: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.9 KiB/s rd, 9.2 KiB/s wr, 1.14k op/s; 47 KiB/s, 0 objects/s recovering 2024-09-10T18:34:14.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:14 smithi028 ceph-mon[29291]: 4.7 scrub starts 2024-09-10T18:34:14.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:14 smithi028 ceph-mon[29291]: 4.7 scrub ok 2024-09-10T18:34:14.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:14 smithi028 ceph-mon[39220]: 3.1b scrub starts 2024-09-10T18:34:14.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:14 smithi028 ceph-mon[39220]: 3.1b scrub ok 2024-09-10T18:34:14.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:14 smithi028 ceph-mon[39220]: 9.5 scrub ok 2024-09-10T18:34:14.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:14 smithi028 ceph-mon[39220]: pgmap v573: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.9 KiB/s rd, 9.2 KiB/s wr, 1.14k op/s; 47 KiB/s, 0 objects/s recovering 2024-09-10T18:34:14.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:14 smithi028 ceph-mon[39220]: 4.7 scrub starts 2024-09-10T18:34:14.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:14 smithi028 ceph-mon[39220]: 4.7 scrub ok 2024-09-10T18:34:14.238 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:14.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:14.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:14.343 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:14.363 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:14.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:14.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:14.465 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:14.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:14.493 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:14.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:14.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:14 smithi204 ceph-mon[32357]: 3.1b scrub starts 2024-09-10T18:34:14.583 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:14 smithi204 ceph-mon[32357]: 3.1b scrub ok 2024-09-10T18:34:14.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:14 smithi204 ceph-mon[32357]: 9.5 scrub ok 2024-09-10T18:34:14.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:14 smithi204 ceph-mon[32357]: pgmap v573: 168 pgs: 2 active+clean+scrubbing, 166 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.9 KiB/s rd, 9.2 KiB/s wr, 1.14k op/s; 47 KiB/s, 0 objects/s recovering 2024-09-10T18:34:14.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:14 smithi204 ceph-mon[32357]: 4.7 scrub starts 2024-09-10T18:34:14.584 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:14 smithi204 ceph-mon[32357]: 4.7 scrub ok 2024-09-10T18:34:14.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:14.624 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:14.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:14.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:14.725 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:14.755 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:14.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:14.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:14.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:34:14 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:34:14.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:14.887 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:14.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:14.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:14.989 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:15.020 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:15.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:15.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:15.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:15.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:15.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:15.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:15.251 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:15.276 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:15.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:15.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:15.378 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:15.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:15 smithi028 ceph-mon[39220]: 6.11 scrub starts 2024-09-10T18:34:15.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:15 smithi028 ceph-mon[39220]: 6.11 scrub ok 2024-09-10T18:34:15.400 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:15 smithi028 ceph-mon[29291]: 6.11 scrub starts 2024-09-10T18:34:15.400 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:15 smithi028 ceph-mon[29291]: 6.11 scrub ok 2024-09-10T18:34:15.408 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:15.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:15.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:15.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:15.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:15.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:15 smithi204 ceph-mon[32357]: 6.11 scrub starts 2024-09-10T18:34:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:15 smithi204 ceph-mon[32357]: 6.11 scrub ok 2024-09-10T18:34:15.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:15.683 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:15.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:15.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:15.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:15.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:15.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:15.922 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:15.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:15.959 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:15.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:16.060 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:16.093 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:16.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:16.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:16.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:16.233 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:16.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:16.334 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:16.365 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:16.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:16.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:16.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:16.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:16 smithi028 ceph-mon[29291]: 5.e scrub starts 2024-09-10T18:34:16.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:16 smithi028 ceph-mon[29291]: 5.e scrub ok 2024-09-10T18:34:16.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:16 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:16.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:16 smithi028 ceph-mon[29291]: pgmap v574: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 165 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 2.0 KiB/s rd, 7.5 KiB/s wr, 1.19k op/s; 38 KiB/s, 0 objects/s recovering 2024-09-10T18:34:16.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:16 smithi028 ceph-mon[39220]: 5.e scrub starts 2024-09-10T18:34:16.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:16 smithi028 ceph-mon[39220]: 5.e scrub ok 2024-09-10T18:34:16.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:16 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:16.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:16 smithi028 ceph-mon[39220]: pgmap v574: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 165 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 2.0 KiB/s rd, 7.5 KiB/s wr, 1.19k op/s; 38 KiB/s, 0 objects/s recovering 2024-09-10T18:34:16.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:16.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:16.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:16 smithi204 ceph-mon[32357]: 5.e scrub starts 2024-09-10T18:34:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:16 smithi204 ceph-mon[32357]: 5.e scrub ok 2024-09-10T18:34:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:16 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:16 smithi204 ceph-mon[32357]: pgmap v574: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 165 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 2.0 KiB/s rd, 7.5 KiB/s wr, 1.19k op/s; 38 KiB/s, 0 objects/s recovering 2024-09-10T18:34:16.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:16.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:16.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:16.749 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:16.776 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:16.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:16.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:16.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:34:16.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:16.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:16.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:17.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:17.086 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:17.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:17.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:17.188 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:17.214 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:17.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:17.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:17.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:17.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:17.456 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:17.486 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:17.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:17.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:17 smithi028 ceph-mon[39220]: 1.f deep-scrub starts 2024-09-10T18:34:17.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:17 smithi028 ceph-mon[39220]: 1.f deep-scrub ok 2024-09-10T18:34:17.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:17 smithi028 ceph-mon[39220]: 6.1b scrub starts 2024-09-10T18:34:17.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:17 smithi028 ceph-mon[39220]: 6.1b scrub ok 2024-09-10T18:34:17.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:17 smithi028 ceph-mon[29291]: 1.f deep-scrub starts 2024-09-10T18:34:17.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:17 smithi028 ceph-mon[29291]: 1.f deep-scrub ok 2024-09-10T18:34:17.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:17 smithi028 ceph-mon[29291]: 6.1b scrub starts 2024-09-10T18:34:17.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:17 smithi028 ceph-mon[29291]: 6.1b scrub ok 2024-09-10T18:34:17.588 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:17 smithi204 ceph-mon[32357]: 1.f deep-scrub starts 2024-09-10T18:34:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:17 smithi204 ceph-mon[32357]: 1.f deep-scrub ok 2024-09-10T18:34:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:17 smithi204 ceph-mon[32357]: 6.1b scrub starts 2024-09-10T18:34:17.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:17 smithi204 ceph-mon[32357]: 6.1b scrub ok 2024-09-10T18:34:17.616 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:17.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:17.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:17.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:17.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:17.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:17.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:17.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:17.913 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:17.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:18.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:18.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:18.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:18.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:18.185 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:18.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:18.230 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:18 smithi028 ceph-mon[39220]: 3.10 scrub starts 2024-09-10T18:34:18.230 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:18 smithi028 ceph-mon[39220]: 3.10 scrub ok 2024-09-10T18:34:18.230 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:18 smithi028 ceph-mon[39220]: pgmap v575: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T18:34:18.230 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:18 smithi028 ceph-mon[39220]: 5.1b scrub starts 2024-09-10T18:34:18.230 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:18 smithi028 ceph-mon[39220]: 5.1b scrub ok 2024-09-10T18:34:18.230 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:18 smithi028 ceph-mon[39220]: 9.8 scrub starts 2024-09-10T18:34:18.231 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:18 smithi028 ceph-mon[39220]: 5.d scrub starts 2024-09-10T18:34:18.231 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:18 smithi028 ceph-mon[39220]: 5.d scrub ok 2024-09-10T18:34:18.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:18 smithi028 ceph-mon[29291]: 3.10 scrub starts 2024-09-10T18:34:18.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:18 smithi028 ceph-mon[29291]: 3.10 scrub ok 2024-09-10T18:34:18.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:18 smithi028 ceph-mon[29291]: pgmap v575: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T18:34:18.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:18 smithi028 ceph-mon[29291]: 5.1b scrub starts 2024-09-10T18:34:18.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:18 smithi028 ceph-mon[29291]: 5.1b scrub ok 2024-09-10T18:34:18.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:18 smithi028 ceph-mon[29291]: 9.8 scrub starts 2024-09-10T18:34:18.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:18 smithi028 ceph-mon[29291]: 5.d scrub starts 2024-09-10T18:34:18.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:18 smithi028 ceph-mon[29291]: 5.d scrub ok 2024-09-10T18:34:18.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:18.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:18.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:18.474 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:18.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:18.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:18 smithi204 ceph-mon[32357]: 3.10 scrub starts 2024-09-10T18:34:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:18 smithi204 ceph-mon[32357]: 3.10 scrub ok 2024-09-10T18:34:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:18 smithi204 ceph-mon[32357]: pgmap v575: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T18:34:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:18 smithi204 ceph-mon[32357]: 5.1b scrub starts 2024-09-10T18:34:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:18 smithi204 ceph-mon[32357]: 5.1b scrub ok 2024-09-10T18:34:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:18 smithi204 ceph-mon[32357]: 9.8 scrub starts 2024-09-10T18:34:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:18 smithi204 ceph-mon[32357]: 5.d scrub starts 2024-09-10T18:34:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:18 smithi204 ceph-mon[32357]: 5.d scrub ok 2024-09-10T18:34:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:18.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:18.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:18.692 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:34:18.692 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:34:18.692 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:34:18.693 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:34:18.693 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:34:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:34:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:34:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:34:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:34:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:34:18.705 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:34:18.705 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:34:18.717 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:18.741 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:34:18.741 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:34:18.742 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:18.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:34:18.782 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:34:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:34:18.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:18.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:18.873 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:18.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:18.965 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:34:18.968 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:34:18.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:19.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:19.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:19.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:19.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:19.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:19.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:19.259 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:19.291 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:19.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:19.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:19.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:19.423 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:19.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:19.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:19.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:19 smithi028 ceph-mon[29291]: 3.3 scrub starts 2024-09-10T18:34:19.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:19 smithi028 ceph-mon[29291]: 3.3 scrub ok 2024-09-10T18:34:19.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:19 smithi028 ceph-mon[29291]: 9.7 scrub starts 2024-09-10T18:34:19.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:19 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/750032643' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:34:19.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:19 smithi028 ceph-mon[39220]: 3.3 scrub starts 2024-09-10T18:34:19.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:19 smithi028 ceph-mon[39220]: 3.3 scrub ok 2024-09-10T18:34:19.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:19 smithi028 ceph-mon[39220]: 9.7 scrub starts 2024-09-10T18:34:19.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:19 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/750032643' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:34:19.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:19.567 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:19.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:19.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:19 smithi204 ceph-mon[32357]: 3.3 scrub starts 2024-09-10T18:34:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:19 smithi204 ceph-mon[32357]: 3.3 scrub ok 2024-09-10T18:34:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:19 smithi204 ceph-mon[32357]: 9.7 scrub starts 2024-09-10T18:34:19.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:19 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/750032643' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:34:19.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:19.710 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:19.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:19.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:19.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:19.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:19.847 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:19.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:19.948 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:19.975 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:19.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:19.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:20.077 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:20.111 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:20.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:20.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:20.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:20.249 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:20.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:20.277 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:34:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-10T18:34:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-10T18:34:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:34:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:121682 7.a 7:5cc2bbec:::benchmark_data_smithi028_77856_object582:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:34:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:14.512524+0000", 2024-09-10T18:34:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 545.75786073999996, 2024-09-10T18:34:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16768606599999999, 2024-09-10T18:34:20.277 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:34:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 121682 2024-09-10T18:34:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512524+0000", 2024-09-10T18:34:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.278 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512524+0000", 2024-09-10T18:34:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512524+0000", 2024-09-10T18:34:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:34:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512527+0000", 2024-09-10T18:34:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.467e-06 2024-09-10T18:34:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512533+0000", 2024-09-10T18:34:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5679999999999999e-06 2024-09-10T18:34:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.512538+0000", 2024-09-10T18:34:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7119999999999999e-06 2024-09-10T18:34:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521276+0000", 2024-09-10T18:34:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087382520000000002 2024-09-10T18:34:20.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521386+0000", 2024-09-10T18:34:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110188 2024-09-10T18:34:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:34:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.521505+0000", 2024-09-10T18:34:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000119021 2024-09-10T18:34:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.530749+0000", 2024-09-10T18:34:20.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.009244169 2024-09-10T18:34:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.530781+0000", 2024-09-10T18:34:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2119999999999997e-05 2024-09-10T18:34:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680127+0000", 2024-09-10T18:34:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14934560899999999 2024-09-10T18:34:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680182+0000", 2024-09-10T18:34:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5398000000000001e-05 2024-09-10T18:34:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680210+0000", 2024-09-10T18:34:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8191000000000002e-05 2024-09-10T18:34:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:121685 7.a 7:556e904c:::benchmark_data_smithi028_77856_object585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:34:20.283 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:14.516970+0000", 2024-09-10T18:34:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 545.75341501000003, 2024-09-10T18:34:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16365090800000001, 2024-09-10T18:34:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:34:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 121685 2024-09-10T18:34:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.284 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516970+0000", 2024-09-10T18:34:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516970+0000", 2024-09-10T18:34:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516968+0000", 2024-09-10T18:34:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:34:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516977+0000", 2024-09-10T18:34:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1600000000000004e-06 2024-09-10T18:34:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516989+0000", 2024-09-10T18:34:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2094e-05 2024-09-10T18:34:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.516995+0000", 2024-09-10T18:34:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1199999999999999e-06 2024-09-10T18:34:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.524202+0000", 2024-09-10T18:34:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0072076129999999999 2024-09-10T18:34:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.524301+0000", 2024-09-10T18:34:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8858999999999997e-05 2024-09-10T18:34:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:34:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.524413+0000", 2024-09-10T18:34:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011134899999999999 2024-09-10T18:34:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.530754+0000", 2024-09-10T18:34:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006341218 2024-09-10T18:34:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.533574+0000", 2024-09-10T18:34:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002820473 2024-09-10T18:34:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680527+0000", 2024-09-10T18:34:20.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.146952895 2024-09-10T18:34:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680579+0000", 2024-09-10T18:34:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1777999999999998e-05 2024-09-10T18:34:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680621+0000", 2024-09-10T18:34:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1953999999999998e-05 2024-09-10T18:34:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:121686 7.2 7:471809bb:::benchmark_data_smithi028_77856_object586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:34:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:14.518358+0000", 2024-09-10T18:34:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 545.75202738899998, 2024-09-10T18:34:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16205149299999999, 2024-09-10T18:34:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:34:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 121686 2024-09-10T18:34:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.290 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518358+0000", 2024-09-10T18:34:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518358+0000", 2024-09-10T18:34:20.291 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518356+0000", 2024-09-10T18:34:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:34:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518362+0000", 2024-09-10T18:34:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8889999999999999e-06 2024-09-10T18:34:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.292 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518373+0000", 2024-09-10T18:34:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0375e-05 2024-09-10T18:34:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.518380+0000", 2024-09-10T18:34:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.092e-06 2024-09-10T18:34:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.526805+0000", 2024-09-10T18:34:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0084254840000000004 2024-09-10T18:34:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.526929+0000", 2024-09-10T18:34:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012368700000000001 2024-09-10T18:34:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:34:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.527045+0000", 2024-09-10T18:34:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116277 2024-09-10T18:34:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.533582+0000", 2024-09-10T18:34:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0065365359999999999 2024-09-10T18:34:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.542247+0000", 2024-09-10T18:34:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0086657750000000006 2024-09-10T18:34:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680327+0000", 2024-09-10T18:34:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13807940899999999 2024-09-10T18:34:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680372+0000", 2024-09-10T18:34:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5173000000000003e-05 2024-09-10T18:34:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:14.680409+0000", 2024-09-10T18:34:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7194000000000003e-05 2024-09-10T18:34:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:124429 7.2 7:4838b573:::benchmark_data_smithi028_77856_object3329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T18:34:20.296 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:18.054003+0000", 2024-09-10T18:34:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 542.21638216999997, 2024-09-10T18:34:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.166994589, 2024-09-10T18:34:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:34:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 124429 2024-09-10T18:34:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.297 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054003+0000", 2024-09-10T18:34:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054003+0000", 2024-09-10T18:34:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054002+0000", 2024-09-10T18:34:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:34:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054005+0000", 2024-09-10T18:34:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4559999999999999e-06 2024-09-10T18:34:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054009+0000", 2024-09-10T18:34:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3669999999999999e-06 2024-09-10T18:34:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.054013+0000", 2024-09-10T18:34:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.343e-06 2024-09-10T18:34:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.126700+0000", 2024-09-10T18:34:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.072687286000000004 2024-09-10T18:34:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.126814+0000", 2024-09-10T18:34:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114442 2024-09-10T18:34:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:34:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.126963+0000", 2024-09-10T18:34:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014888499999999999 2024-09-10T18:34:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.140882+0000", 2024-09-10T18:34:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013919124 2024-09-10T18:34:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.141158+0000", 2024-09-10T18:34:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000276042 2024-09-10T18:34:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.220924+0000", 2024-09-10T18:34:20.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.079766112 2024-09-10T18:34:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.220962+0000", 2024-09-10T18:34:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7707999999999999e-05 2024-09-10T18:34:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:18.220998+0000", 2024-09-10T18:34:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5349999999999999e-05 2024-09-10T18:34:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:139058 7.2 7:4504e7d4:::benchmark_data_smithi028_77856_object17958:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:34:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:42.798430+0000", 2024-09-10T18:34:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 517.471954665, 2024-09-10T18:34:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.21785926999999999, 2024-09-10T18:34:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:34:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 139058 2024-09-10T18:34:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.798430+0000", 2024-09-10T18:34:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.798430+0000", 2024-09-10T18:34:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.304 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.798429+0000", 2024-09-10T18:34:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:34:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.798435+0000", 2024-09-10T18:34:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4330000000000001e-06 2024-09-10T18:34:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.798449+0000", 2024-09-10T18:34:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3360000000000001e-05 2024-09-10T18:34:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.798455+0000", 2024-09-10T18:34:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1600000000000003e-06 2024-09-10T18:34:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.887408+0000", 2024-09-10T18:34:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.088953303999999997 2024-09-10T18:34:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.887522+0000", 2024-09-10T18:34:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113528 2024-09-10T18:34:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:34:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.887728+0000", 2024-09-10T18:34:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020583900000000001 2024-09-10T18:34:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.896982+0000", 2024-09-10T18:34:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092544399999999992 2024-09-10T18:34:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.896990+0000", 2024-09-10T18:34:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2800000000000003e-06 2024-09-10T18:34:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.016206+0000", 2024-09-10T18:34:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.119215716 2024-09-10T18:34:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.016249+0000", 2024-09-10T18:34:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3247000000000001e-05 2024-09-10T18:34:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.016290+0000", 2024-09-10T18:34:20.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0467999999999999e-05 2024-09-10T18:34:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:139084 7.2 7:4d9fefb8:::benchmark_data_smithi028_77856_object17984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:34:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:42.851594+0000", 2024-09-10T18:34:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 517.41879096000002, 2024-09-10T18:34:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.164888488, 2024-09-10T18:34:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:34:20.309 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 139084 2024-09-10T18:34:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.851594+0000", 2024-09-10T18:34:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.851594+0000", 2024-09-10T18:34:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.851593+0000", 2024-09-10T18:34:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:34:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.851599+0000", 2024-09-10T18:34:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1299999999999998e-06 2024-09-10T18:34:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.851610+0000", 2024-09-10T18:34:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.098e-05 2024-09-10T18:34:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.851616+0000", 2024-09-10T18:34:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0750000000000003e-06 2024-09-10T18:34:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.890507+0000", 2024-09-10T18:34:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.038891336999999998 2024-09-10T18:34:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.890635+0000", 2024-09-10T18:34:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012825400000000001 2024-09-10T18:34:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:34:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.890768+0000", 2024-09-10T18:34:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013264400000000001 2024-09-10T18:34:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.896997+0000", 2024-09-10T18:34:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062283959999999998 2024-09-10T18:34:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:42.897071+0000", 2024-09-10T18:34:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4501999999999999e-05 2024-09-10T18:34:20.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.016410+0000", 2024-09-10T18:34:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.11933859099999999 2024-09-10T18:34:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.016450+0000", 2024-09-10T18:34:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0340999999999999e-05 2024-09-10T18:34:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.016483+0000", 2024-09-10T18:34:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2687000000000001e-05 2024-09-10T18:34:20.314 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:139472 7.2 7:4dfe09d1:::benchmark_data_smithi028_77856_object18372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:34:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:43.522091+0000", 2024-09-10T18:34:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 516.74829388299997, 2024-09-10T18:34:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.22258198400000001, 2024-09-10T18:34:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:34:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:20.315 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 139472 2024-09-10T18:34:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.522091+0000", 2024-09-10T18:34:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.522091+0000", 2024-09-10T18:34:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.522089+0000", 2024-09-10T18:34:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:34:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.522096+0000", 2024-09-10T18:34:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0550000000000002e-06 2024-09-10T18:34:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.522107+0000", 2024-09-10T18:34:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0299000000000001e-05 2024-09-10T18:34:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.522113+0000", 2024-09-10T18:34:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9739999999999999e-06 2024-09-10T18:34:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.600209+0000", 2024-09-10T18:34:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.078096439000000004 2024-09-10T18:34:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.600329+0000", 2024-09-10T18:34:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000119535 2024-09-10T18:34:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:34:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.600449+0000", 2024-09-10T18:34:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000120189 2024-09-10T18:34:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.606584+0000", 2024-09-10T18:34:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061352220000000001 2024-09-10T18:34:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.610809+0000", 2024-09-10T18:34:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042246059999999997 2024-09-10T18:34:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744597+0000", 2024-09-10T18:34:20.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.133788834 2024-09-10T18:34:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744636+0000", 2024-09-10T18:34:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8461999999999999e-05 2024-09-10T18:34:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744673+0000", 2024-09-10T18:34:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7283999999999997e-05 2024-09-10T18:34:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:139485 7.a 7:5828730e:::benchmark_data_smithi028_77856_object18385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:34:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:43.554329+0000", 2024-09-10T18:34:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 516.71605580400001, 2024-09-10T18:34:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.190223159, 2024-09-10T18:34:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:34:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 139485 2024-09-10T18:34:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.554329+0000", 2024-09-10T18:34:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.554329+0000", 2024-09-10T18:34:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.554327+0000", 2024-09-10T18:34:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:34:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.554335+0000", 2024-09-10T18:34:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4039999999999998e-06 2024-09-10T18:34:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.554345+0000", 2024-09-10T18:34:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0973e-05 2024-09-10T18:34:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.554351+0000", 2024-09-10T18:34:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9120000000000003e-06 2024-09-10T18:34:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.597534+0000", 2024-09-10T18:34:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.043182458 2024-09-10T18:34:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.597642+0000", 2024-09-10T18:34:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010785099999999999 2024-09-10T18:34:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:34:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.597766+0000", 2024-09-10T18:34:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012438799999999999 2024-09-10T18:34:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.602961+0000", 2024-09-10T18:34:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.005195224 2024-09-10T18:34:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.606579+0000", 2024-09-10T18:34:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003617529 2024-09-10T18:34:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744464+0000", 2024-09-10T18:34:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.137885326 2024-09-10T18:34:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744513+0000", 2024-09-10T18:34:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8442000000000001e-05 2024-09-10T18:34:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744552+0000", 2024-09-10T18:34:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9870000000000003e-05 2024-09-10T18:34:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:139490 7.2 7:469cd9c4:::benchmark_data_smithi028_77856_object18390:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:34:20.327 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:43.578701+0000", 2024-09-10T18:34:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 516.69168410299994, 2024-09-10T18:34:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.166214733, 2024-09-10T18:34:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:34:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 139490 2024-09-10T18:34:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.578701+0000", 2024-09-10T18:34:20.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.578701+0000", 2024-09-10T18:34:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.578699+0000", 2024-09-10T18:34:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:34:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.578707+0000", 2024-09-10T18:34:20.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5809999999999999e-06 2024-09-10T18:34:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.578720+0000", 2024-09-10T18:34:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3888e-05 2024-09-10T18:34:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.578726+0000", 2024-09-10T18:34:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7740000000000001e-06 2024-09-10T18:34:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.602984+0000", 2024-09-10T18:34:20.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024257595999999999 2024-09-10T18:34:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.603082+0000", 2024-09-10T18:34:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8620000000000001e-05 2024-09-10T18:34:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:34:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.603476+0000", 2024-09-10T18:34:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00039365500000000002 2024-09-10T18:34:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.607531+0000", 2024-09-10T18:34:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004055042 2024-09-10T18:34:20.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.610833+0000", 2024-09-10T18:34:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003302049 2024-09-10T18:34:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744821+0000", 2024-09-10T18:34:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.133988041 2024-09-10T18:34:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744877+0000", 2024-09-10T18:34:20.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6140999999999997e-05 2024-09-10T18:34:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.744916+0000", 2024-09-10T18:34:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8346000000000002e-05 2024-09-10T18:34:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:141043 7.2 7:45059280:::benchmark_data_smithi028_77856_object19943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:34:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:46.309105+0000", 2024-09-10T18:34:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 513.96128057299995, 2024-09-10T18:34:20.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.163191158, 2024-09-10T18:34:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:34:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 141043 2024-09-10T18:34:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.309105+0000", 2024-09-10T18:34:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.309105+0000", 2024-09-10T18:34:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.309103+0000", 2024-09-10T18:34:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:34:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.309110+0000", 2024-09-10T18:34:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6029999999999999e-06 2024-09-10T18:34:20.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.309122+0000", 2024-09-10T18:34:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1989e-05 2024-09-10T18:34:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.309128+0000", 2024-09-10T18:34:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0630000000000002e-06 2024-09-10T18:34:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.379768+0000", 2024-09-10T18:34:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.070639561000000003 2024-09-10T18:34:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.379881+0000", 2024-09-10T18:34:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113529 2024-09-10T18:34:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:34:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.380004+0000", 2024-09-10T18:34:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012301400000000001 2024-09-10T18:34:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.386775+0000", 2024-09-10T18:34:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006770496 2024-09-10T18:34:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.394129+0000", 2024-09-10T18:34:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073538070000000004 2024-09-10T18:34:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.472166+0000", 2024-09-10T18:34:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.078037168000000004 2024-09-10T18:34:20.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.472256+0000", 2024-09-10T18:34:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9802000000000005e-05 2024-09-10T18:34:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.472296+0000", 2024-09-10T18:34:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0132000000000001e-05 2024-09-10T18:34:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.339 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:182362 7.3 7:c3249eb8:::benchmark_data_smithi028_77856_object61262:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-10T18:34:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:26.337733+0000", 2024-09-10T18:34:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 473.93265226900002, 2024-09-10T18:34:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.166142067, 2024-09-10T18:34:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:34:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 182362 2024-09-10T18:34:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337733+0000", 2024-09-10T18:34:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337733+0000", 2024-09-10T18:34:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337731+0000", 2024-09-10T18:34:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:34:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337738+0000", 2024-09-10T18:34:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3520000000000003e-06 2024-09-10T18:34:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337749+0000", 2024-09-10T18:34:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1096e-05 2024-09-10T18:34:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337753+0000", 2024-09-10T18:34:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5480000000000003e-06 2024-09-10T18:34:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337791+0000", 2024-09-10T18:34:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7648999999999998e-05 2024-09-10T18:34:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.337908+0000", 2024-09-10T18:34:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011726 2024-09-10T18:34:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,4", 2024-09-10T18:34:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.338054+0000", 2024-09-10T18:34:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014590299999999999 2024-09-10T18:34:20.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.341103+0000", 2024-09-10T18:34:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030488400000000001 2024-09-10T18:34:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.341345+0000", 2024-09-10T18:34:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024254600000000001 2024-09-10T18:34:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.503780+0000", 2024-09-10T18:34:20.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16243438299999999 2024-09-10T18:34:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.503826+0000", 2024-09-10T18:34:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6640000000000001e-05 2024-09-10T18:34:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.503875+0000", 2024-09-10T18:34:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8497e-05 2024-09-10T18:34:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:182397 7.3 7:c3805c23:::benchmark_data_smithi028_77856_object61297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-10T18:34:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:26.362893+0000", 2024-09-10T18:34:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 473.90749257300001, 2024-09-10T18:34:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16023626699999999, 2024-09-10T18:34:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:34:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 182397 2024-09-10T18:34:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.362893+0000", 2024-09-10T18:34:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.362893+0000", 2024-09-10T18:34:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.362891+0000", 2024-09-10T18:34:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:34:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.362898+0000", 2024-09-10T18:34:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7000000000000002e-06 2024-09-10T18:34:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.362910+0000", 2024-09-10T18:34:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2819999999999999e-05 2024-09-10T18:34:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.362916+0000", 2024-09-10T18:34:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9039999999999997e-06 2024-09-10T18:34:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.362968+0000", 2024-09-10T18:34:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1521999999999999e-05 2024-09-10T18:34:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.363080+0000", 2024-09-10T18:34:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011167699999999999 2024-09-10T18:34:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,4", 2024-09-10T18:34:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.363185+0000", 2024-09-10T18:34:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010522399999999999 2024-09-10T18:34:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.367164+0000", 2024-09-10T18:34:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039788100000000002 2024-09-10T18:34:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.367171+0000", 2024-09-10T18:34:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0990000000000001e-06 2024-09-10T18:34:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.523007+0000", 2024-09-10T18:34:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15583649299999999 2024-09-10T18:34:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.523093+0000", 2024-09-10T18:34:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6192000000000004e-05 2024-09-10T18:34:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:26.523129+0000", 2024-09-10T18:34:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5426000000000003e-05 2024-09-10T18:34:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15117.0:176787 8.b 8:dfe5e725:::benchmark_data_smithi028_109242_object42566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-10T18:34:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:30:18.738527+0000", 2024-09-10T18:34:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 241.53185784199999, 2024-09-10T18:34:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16198654200000001, 2024-09-10T18:34:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15117", 2024-09-10T18:34:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/121353060", 2024-09-10T18:34:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 176787 2024-09-10T18:34:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:18.738527+0000", 2024-09-10T18:34:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:18.738527+0000", 2024-09-10T18:34:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:18.738526+0000", 2024-09-10T18:34:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:34:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:18.738531+0000", 2024-09-10T18:34:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1669999999999998e-06 2024-09-10T18:34:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:18.738545+0000", 2024-09-10T18:34:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3797e-05 2024-09-10T18:34:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:18.738552+0000", 2024-09-10T18:34:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8229999999999997e-06 2024-09-10T18:34:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:18.775633+0000", 2024-09-10T18:34:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.037081752000000003 2024-09-10T18:34:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:18.775738+0000", 2024-09-10T18:34:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000104901 2024-09-10T18:34:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:34:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:18.775850+0000", 2024-09-10T18:34:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111397 2024-09-10T18:34:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:18.778787+0000", 2024-09-10T18:34:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029370199999999998 2024-09-10T18:34:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:18.778865+0000", 2024-09-10T18:34:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8668e-05 2024-09-10T18:34:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:18.900205+0000", 2024-09-10T18:34:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.12133933 2024-09-10T18:34:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:18.900356+0000", 2024-09-10T18:34:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000151256 2024-09-10T18:34:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:18.900514+0000", 2024-09-10T18:34:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015779100000000001 2024-09-10T18:34:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35306.0:153339 9.8 9:1c33a867:::benchmark_data_smithi028_135703_object18748:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-10T18:34:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:33:57.091173+0000", 2024-09-10T18:34:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.179211637000002, 2024-09-10T18:34:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032282308000000003, 2024-09-10T18:34:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35306", 2024-09-10T18:34:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 153339 2024-09-10T18:34:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:57.091173+0000", 2024-09-10T18:34:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:57.091173+0000", 2024-09-10T18:34:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:57.091171+0000", 2024-09-10T18:34:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:34:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:57.091180+0000", 2024-09-10T18:34:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1689999999999998e-06 2024-09-10T18:34:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:57.091193+0000", 2024-09-10T18:34:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3251999999999999e-05 2024-09-10T18:34:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:57.091200+0000", 2024-09-10T18:34:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1500000000000004e-06 2024-09-10T18:34:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:57.091936+0000", 2024-09-10T18:34:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00073672600000000005 2024-09-10T18:34:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:57.092073+0000", 2024-09-10T18:34:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013642099999999999 2024-09-10T18:34:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:34:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:57.092257+0000", 2024-09-10T18:34:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000184112 2024-09-10T18:34:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:57.096534+0000", 2024-09-10T18:34:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042772970000000002 2024-09-10T18:34:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:57.101412+0000", 2024-09-10T18:34:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048774300000000003 2024-09-10T18:34:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:57.123378+0000", 2024-09-10T18:34:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021966669000000001 2024-09-10T18:34:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:57.123418+0000", 2024-09-10T18:34:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9619000000000001e-05 2024-09-10T18:34:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:57.123456+0000", 2024-09-10T18:34:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7910000000000001e-05 2024-09-10T18:34:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35306.0:154748 9.2 9:4f340aed:::benchmark_data_smithi028_135703_object20157:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-09-10T18:34:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:33:58.101931+0000", 2024-09-10T18:34:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.168454261000001, 2024-09-10T18:34:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034716858000000003, 2024-09-10T18:34:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35306", 2024-09-10T18:34:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 154748 2024-09-10T18:34:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:58.101931+0000", 2024-09-10T18:34:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:58.101931+0000", 2024-09-10T18:34:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:58.101930+0000", 2024-09-10T18:34:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:34:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:58.101933+0000", 2024-09-10T18:34:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9289999999999998e-06 2024-09-10T18:34:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:58.101938+0000", 2024-09-10T18:34:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7650000000000001e-06 2024-09-10T18:34:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:58.101942+0000", 2024-09-10T18:34:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5319999999999998e-06 2024-09-10T18:34:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:58.101983+0000", 2024-09-10T18:34:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1486000000000002e-05 2024-09-10T18:34:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:58.102096+0000", 2024-09-10T18:34:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011302300000000001 2024-09-10T18:34:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:34:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:58.102209+0000", 2024-09-10T18:34:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000112537 2024-09-10T18:34:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:58.109410+0000", 2024-09-10T18:34:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0072012639999999998 2024-09-10T18:34:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:58.111370+0000", 2024-09-10T18:34:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019601079999999999 2024-09-10T18:34:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:58.136572+0000", 2024-09-10T18:34:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025202471000000001 2024-09-10T18:34:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:58.136614+0000", 2024-09-10T18:34:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1199999999999999e-05 2024-09-10T18:34:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:58.136648+0000", 2024-09-10T18:34:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4106000000000002e-05 2024-09-10T18:34:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35306.0:156303 9.1 9:83c492c7:::benchmark_data_smithi028_135703_object21712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-10T18:34:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:33:59.252328+0000", 2024-09-10T18:34:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 21.018057322000001, 2024-09-10T18:34:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.033688735999999997, 2024-09-10T18:34:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35306", 2024-09-10T18:34:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 156303 2024-09-10T18:34:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.252328+0000", 2024-09-10T18:34:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.252328+0000", 2024-09-10T18:34:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.252326+0000", 2024-09-10T18:34:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:34:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.252333+0000", 2024-09-10T18:34:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4950000000000002e-06 2024-09-10T18:34:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.252344+0000", 2024-09-10T18:34:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0601000000000001e-05 2024-09-10T18:34:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.252350+0000", 2024-09-10T18:34:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9410000000000004e-06 2024-09-10T18:34:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.260035+0000", 2024-09-10T18:34:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076845960000000001 2024-09-10T18:34:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.260229+0000", 2024-09-10T18:34:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019455399999999999 2024-09-10T18:34:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-09-10T18:34:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.260394+0000", 2024-09-10T18:34:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016476199999999999 2024-09-10T18:34:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.270073+0000", 2024-09-10T18:34:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0096787179999999993 2024-09-10T18:34:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.270107+0000", 2024-09-10T18:34:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4483999999999997e-05 2024-09-10T18:34:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.285948+0000", 2024-09-10T18:34:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015840730000000001 2024-09-10T18:34:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.285984+0000", 2024-09-10T18:34:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6059000000000002e-05 2024-09-10T18:34:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.286017+0000", 2024-09-10T18:34:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2629999999999998e-05 2024-09-10T18:34:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35306.0:156306 9.1 9:827f904f:::benchmark_data_smithi028_135703_object21715:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-10T18:34:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:33:59.252914+0000", 2024-09-10T18:34:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 21.017471514, 2024-09-10T18:34:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035551675999999997, 2024-09-10T18:34:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35306", 2024-09-10T18:34:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 156306 2024-09-10T18:34:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.252914+0000", 2024-09-10T18:34:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.252914+0000", 2024-09-10T18:34:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.252912+0000", 2024-09-10T18:34:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:34:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.252918+0000", 2024-09-10T18:34:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7790000000000001e-06 2024-09-10T18:34:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.252941+0000", 2024-09-10T18:34:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3399000000000001e-05 2024-09-10T18:34:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.252947+0000", 2024-09-10T18:34:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7309999999999998e-06 2024-09-10T18:34:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.263773+0000", 2024-09-10T18:34:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010826234000000001 2024-09-10T18:34:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.264552+0000", 2024-09-10T18:34:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00077863800000000001 2024-09-10T18:34:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-09-10T18:34:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.264663+0000", 2024-09-10T18:34:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011118199999999999 2024-09-10T18:34:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.270113+0000", 2024-09-10T18:34:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054496529999999996 2024-09-10T18:34:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.270144+0000", 2024-09-10T18:34:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1439999999999997e-05 2024-09-10T18:34:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.286068+0000", 2024-09-10T18:34:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015923861000000001 2024-09-10T18:34:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.288430+0000", 2024-09-10T18:34:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023622920000000002 2024-09-10T18:34:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.288465+0000", 2024-09-10T18:34:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4904999999999999e-05 2024-09-10T18:34:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35306.0:156668 9.c 9:30f2f3aa:::benchmark_data_smithi028_135703_object22077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-10T18:34:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:33:59.573921+0000", 2024-09-10T18:34:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 20.696464032000002, 2024-09-10T18:34:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034605549999999999, 2024-09-10T18:34:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35306", 2024-09-10T18:34:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 156668 2024-09-10T18:34:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.573921+0000", 2024-09-10T18:34:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.573921+0000", 2024-09-10T18:34:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.573919+0000", 2024-09-10T18:34:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:34:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.573926+0000", 2024-09-10T18:34:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1249999999999998e-06 2024-09-10T18:34:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.573934+0000", 2024-09-10T18:34:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2819999999999996e-06 2024-09-10T18:34:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.573938+0000", 2024-09-10T18:34:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5770000000000004e-06 2024-09-10T18:34:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.582936+0000", 2024-09-10T18:34:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0089976810000000004 2024-09-10T18:34:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.583036+0000", 2024-09-10T18:34:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010014999999999999 2024-09-10T18:34:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,7", 2024-09-10T18:34:20.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[29291]: 5.11 scrub starts 2024-09-10T18:34:20.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[29291]: 5.11 scrub ok 2024-09-10T18:34:20.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[29291]: from='client.25429 v1:172.21.15.28:0/3994456674' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:34:20.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[29291]: pgmap v576: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T18:34:20.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[29291]: 9.8 scrub ok 2024-09-10T18:34:20.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[29291]: 3.2 scrub starts 2024-09-10T18:34:20.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[29291]: 3.2 scrub ok 2024-09-10T18:34:20.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[29291]: 4.12 scrub starts 2024-09-10T18:34:20.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[29291]: 4.12 scrub ok 2024-09-10T18:34:20.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[39220]: 5.11 scrub starts 2024-09-10T18:34:20.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[39220]: 5.11 scrub ok 2024-09-10T18:34:20.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[39220]: from='client.25429 v1:172.21.15.28:0/3994456674' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:34:20.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[39220]: pgmap v576: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T18:34:20.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[39220]: 9.8 scrub ok 2024-09-10T18:34:20.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[39220]: 3.2 scrub starts 2024-09-10T18:34:20.387 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[39220]: 3.2 scrub ok 2024-09-10T18:34:20.387 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[39220]: 4.12 scrub starts 2024-09-10T18:34:20.387 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:20 smithi028 ceph-mon[39220]: 4.12 scrub ok 2024-09-10T18:34:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.583138+0000", 2024-09-10T18:34:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101686 2024-09-10T18:34:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.594942+0000", 2024-09-10T18:34:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011803602 2024-09-10T18:34:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.596703+0000", 2024-09-10T18:34:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017612820000000001 2024-09-10T18:34:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.608446+0000", 2024-09-10T18:34:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011743181 2024-09-10T18:34:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.608489+0000", 2024-09-10T18:34:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3189000000000003e-05 2024-09-10T18:34:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.608527+0000", 2024-09-10T18:34:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7444999999999998e-05 2024-09-10T18:34:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35306.0:156672 9.c 9:36683f2f:::benchmark_data_smithi028_135703_object22081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-10T18:34:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:33:59.573943+0000", 2024-09-10T18:34:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 20.696442058999999, 2024-09-10T18:34:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034702021, 2024-09-10T18:34:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:34:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35306", 2024-09-10T18:34:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 156672 2024-09-10T18:34:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.573943+0000", 2024-09-10T18:34:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.573943+0000", 2024-09-10T18:34:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.573943+0000", 2024-09-10T18:34:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:34:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.573946+0000", 2024-09-10T18:34:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4989999999999999e-06 2024-09-10T18:34:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.573951+0000", 2024-09-10T18:34:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7790000000000003e-06 2024-09-10T18:34:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.573955+0000", 2024-09-10T18:34:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.907e-06 2024-09-10T18:34:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.587106+0000", 2024-09-10T18:34:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013151302 2024-09-10T18:34:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.587233+0000", 2024-09-10T18:34:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000126902 2024-09-10T18:34:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,7", 2024-09-10T18:34:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.587343+0000", 2024-09-10T18:34:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110339 2024-09-10T18:34:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.596733+0000", 2024-09-10T18:34:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093897670000000003 2024-09-10T18:34:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:34:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.599283+0000", 2024-09-10T18:34:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025497110000000001 2024-09-10T18:34:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.608560+0000", 2024-09-10T18:34:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092770449999999994 2024-09-10T18:34:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.608609+0000", 2024-09-10T18:34:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9385000000000002e-05 2024-09-10T18:34:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:33:59.608645+0000", 2024-09-10T18:34:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5930999999999999e-05 2024-09-10T18:34:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35306.0:160886 9.f e280/246 9:f1cd0df2:::benchmark_data_smithi028_135703_object26295:head v 280'10044, mlcod=280'10042)", 2024-09-10T18:34:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:02.787661+0000", 2024-09-10T18:34:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.482724561000001, 2024-09-10T18:34:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.038373473999999998, 2024-09-10T18:34:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:34:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:34:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:34:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:02.787661+0000", 2024-09-10T18:34:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:34:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:02.787661+0000", 2024-09-10T18:34:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:34:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:34:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:02.787659+0000", 2024-09-10T18:34:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:34:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:34:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:02.787667+0000", 2024-09-10T18:34:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5279999999999998e-06 2024-09-10T18:34:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:34:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:02.787677+0000", 2024-09-10T18:34:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6749999999999997e-06 2024-09-10T18:34:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:34:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:02.787705+0000", 2024-09-10T18:34:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8433999999999999e-05 2024-09-10T18:34:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:34:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:02.799931+0000", 2024-09-10T18:34:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012225840999999999 2024-09-10T18:34:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:34:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:02.799951+0000", 2024-09-10T18:34:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.997e-05 2024-09-10T18:34:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:34:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:02.826015+0000", 2024-09-10T18:34:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026064173 2024-09-10T18:34:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:34:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:34:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:34:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:02.826034+0000", 2024-09-10T18:34:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8960000000000001e-05 2024-09-10T18:34:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:34:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:34:20.399 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:34:20.439 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:20.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:20.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:20.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:20.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:20.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:20.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:20 smithi204 ceph-mon[32357]: 5.11 scrub starts 2024-09-10T18:34:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:20 smithi204 ceph-mon[32357]: 5.11 scrub ok 2024-09-10T18:34:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:20 smithi204 ceph-mon[32357]: from='client.25429 v1:172.21.15.28:0/3994456674' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:34:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:20 smithi204 ceph-mon[32357]: pgmap v576: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T18:34:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:20 smithi204 ceph-mon[32357]: 9.8 scrub ok 2024-09-10T18:34:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:20 smithi204 ceph-mon[32357]: 3.2 scrub starts 2024-09-10T18:34:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:20 smithi204 ceph-mon[32357]: 3.2 scrub ok 2024-09-10T18:34:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:20 smithi204 ceph-mon[32357]: 4.12 scrub starts 2024-09-10T18:34:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:20 smithi204 ceph-mon[32357]: 4.12 scrub ok 2024-09-10T18:34:20.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:20.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:20.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:20.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:20.814 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:20.843 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:20.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:20.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:20.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:20.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:20.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:21.027 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":575,"stamp":"2024-09-10T18:34:17.022457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6118253843,"num_objects":93545,"num_object_clones":0,"num_object_copies":280635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93545,"num_whiteouts":0,"num_read":1932,"num_read_kb":1707,"num_write":176574,"num_write_kb":8614046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":596608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75314,"ondisk_log_size":75314,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21116976,"kb_used_data":18097520,"kb_used_omap":19,"kb_used_meta":3018988,"kb_avail":728844240,"statfs":{"total":767960285184,"available":746336501760,"internally_reserved":0,"allocated":18531860480,"data_stored":18528578436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20033,"internal_metadata":3091444159},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-754515968,"num_objects":-11513,"num_object_clones":0,"num_object_copies":-34539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11513,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11513,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003738"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":283,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.165709+0000","last_change":"2024-09-10T18:33:04.320548+0000","last_active":"2024-09-10T18:34:05.165709+0000","last_peered":"2024-09-10T18:34:05.165709+0000","last_clean":"2024-09-10T18:34:05.165709+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:34:05.165709+0000","last_undegraded":"2024-09-10T18:34:05.165709+0000","last_fullsized":"2024-09-10T18:34:05.165709+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:04.320458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:33:04.320458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.581200+0000","scrub_duration":0.0055421380000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":322,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.173470+0000","last_change":"2024-09-10T18:33:12.680256+0000","last_active":"2024-09-10T18:34:05.173470+0000","last_peered":"2024-09-10T18:34:05.173470+0000","last_clean":"2024-09-10T18:34:05.173470+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:34:05.173470+0000","last_undegraded":"2024-09-10T18:34:05.173470+0000","last_fullsized":"2024-09-10T18:34:05.173470+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:33:12.680138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:12.680138+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:19.317278+0000","scrub_duration":0.0033442979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":288,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164095+0000","last_change":"2024-09-10T18:32:43.156005+0000","last_active":"2024-09-10T18:34:05.164095+0000","last_peered":"2024-09-10T18:34:05.164095+0000","last_clean":"2024-09-10T18:34:05.164095+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:34:05.164095+0000","last_undegraded":"2024-09-10T18:34:05.164095+0000","last_fullsized":"2024-09-10T18:34:05.164095+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:43.155924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:43.155924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007043718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":286,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.173940+0000","last_change":"2024-09-10T18:33:08.643815+0000","last_active":"2024-09-10T18:34:05.173940+0000","last_peered":"2024-09-10T18:34:05.173940+0000","last_clean":"2024-09-10T18:34:05.173940+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:34:05.173940+0000","last_undegraded":"2024-09-10T18:34:05.173940+0000","last_fullsized":"2024-09-10T18:34:05.173940+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:08.643758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.643758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:24.996028+0000","scrub_duration":0.0046354059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":285,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.174182+0000","last_change":"2024-09-10T18:31:54.788489+0000","last_active":"2024-09-10T18:34:05.174182+0000","last_peered":"2024-09-10T18:34:05.174182+0000","last_clean":"2024-09-10T18:34:05.174182+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:34:05.174182+0000","last_undegraded":"2024-09-10T18:34:05.174182+0000","last_fullsized":"2024-09-10T18:34:05.174182+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:54.788436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:54.788436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013008939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":291,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.172875+0000","last_change":"2024-09-10T18:32:00.727061+0000","last_active":"2024-09-10T18:34:05.172875+0000","last_peered":"2024-09-10T18:34:05.172875+0000","last_clean":"2024-09-10T18:34:05.172875+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:34:05.172875+0000","last_undegraded":"2024-09-10T18:34:05.172875+0000","last_fullsized":"2024-09-10T18:34:05.172875+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.727002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.727002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014900709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'34","reported_seq":332,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.169649+0000","last_change":"2024-09-10T18:32:55.371863+0000","last_active":"2024-09-10T18:34:05.169649+0000","last_peered":"2024-09-10T18:34:05.169649+0000","last_clean":"2024-09-10T18:34:05.169649+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:34:05.169649+0000","last_undegraded":"2024-09-10T18:34:05.169649+0000","last_fullsized":"2024-09-10T18:34:05.169649+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:32:55.371617+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:32:55.371617+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:18.220770+0000","scrub_duration":0.0089694079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":298,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:09.440291+0000","last_change":"2024-09-10T18:34:09.440223+0000","last_active":"2024-09-10T18:34:09.440291+0000","last_peered":"2024-09-10T18:34:09.440291+0000","last_clean":"2024-09-10T18:34:09.440291+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:34:09.440291+0000","last_undegraded":"2024-09-10T18:34:09.440291+0000","last_fullsized":"2024-09-10T18:34:09.440291+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:09.440140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:34:09.440140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:16.383699+0000","scrub_duration":0.003425656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":303,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.159325+0000","last_change":"2024-09-10T18:33:16.963182+0000","last_active":"2024-09-10T18:34:05.159325+0000","last_peered":"2024-09-10T18:34:05.159325+0000","last_clean":"2024-09-10T18:34:05.159325+0000","last_became_active":"2024-09-10T18:26:27.644961+0000","last_became_peered":"2024-09-10T18:26:27.644961+0000","last_unstale":"2024-09-10T18:34:05.159325+0000","last_undegraded":"2024-09-10T18:34:05.159325+0000","last_fullsized":"2024-09-10T18:34:05.159325+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_clean_scrub_stamp":"2024-09-10T18:33:16.963102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:44.331853+0000","scrub_duration":0.0019539739999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'15","reported_seq":315,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.174178+0000","last_change":"2024-09-10T18:32:45.581485+0000","last_active":"2024-09-10T18:34:05.174178+0000","last_peered":"2024-09-10T18:34:05.174178+0000","last_clean":"2024-09-10T18:34:05.174178+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:34:05.174178+0000","last_undegraded":"2024-09-10T18:34:05.174178+0000","last_fullsized":"2024-09-10T18:34:05.174178+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:32:45.581390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:45.581390+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003880966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":289,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.157362+0000","last_change":"2024-09-10T18:31:58.589684+0000","last_active":"2024-09-10T18:34:05.157362+0000","last_peered":"2024-09-10T18:34:05.157362+0000","last_clean":"2024-09-10T18:34:05.157362+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:34:05.157362+0000","last_undegraded":"2024-09-10T18:34:05.157362+0000","last_fullsized":"2024-09-10T18:34:05.157362+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_clean_scrub_stamp":"2024-09-10T18:31:58.589631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011763940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":287,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.166955+0000","last_change":"2024-09-10T18:33:00.235163+0000","last_active":"2024-09-10T18:34:05.166955+0000","last_peered":"2024-09-10T18:34:05.166955+0000","last_clean":"2024-09-10T18:34:05.166955+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:34:05.166955+0000","last_undegraded":"2024-09-10T18:34:05.166955+0000","last_fullsized":"2024-09-10T18:34:05.166955+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:00.234931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:00.234931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.372488+0000","scrub_duration":0.0023527999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":293,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.157775+0000","last_change":"2024-09-10T18:32:00.582011+0000","last_active":"2024-09-10T18:34:05.157775+0000","last_peered":"2024-09-10T18:34:05.157775+0000","last_clean":"2024-09-10T18:34:05.157775+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:34:05.157775+0000","last_undegraded":"2024-09-10T18:34:05.157775+0000","last_fullsized":"2024-09-10T18:34:05.157775+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.581959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:17.464205+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.581959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012934120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":327,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.174069+0000","last_change":"2024-09-10T18:32:48.589267+0000","last_active":"2024-09-10T18:34:05.174069+0000","last_peered":"2024-09-10T18:34:05.174069+0000","last_clean":"2024-09-10T18:34:05.174069+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:34:05.174069+0000","last_undegraded":"2024-09-10T18:34:05.174069+0000","last_fullsized":"2024-09-10T18:34:05.174069+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'9","last_scrub_stamp":"2024-09-10T18:32:48.589167+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:32:48.589167+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073279920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":288,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164940+0000","last_change":"2024-09-10T18:32:01.525753+0000","last_active":"2024-09-10T18:34:05.164940+0000","last_peered":"2024-09-10T18:34:05.164940+0000","last_clean":"2024-09-10T18:34:05.164940+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:34:05.164940+0000","last_undegraded":"2024-09-10T18:34:05.164940+0000","last_fullsized":"2024-09-10T18:34:05.164940+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:01.525701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:01.525701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001223902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":290,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:09.152133+0000","last_change":"2024-09-10T18:34:09.151879+0000","last_active":"2024-09-10T18:34:09.152133+0000","last_peered":"2024-09-10T18:34:09.152133+0000","last_clean":"2024-09-10T18:34:09.152133+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:34:09.152133+0000","last_undegraded":"2024-09-10T18:34:09.152133+0000","last_fullsized":"2024-09-10T18:34:09.152133+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:09.151789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:34:09.151789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:10.873083+0000","scrub_duration":0.002651994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":296,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:07.394951+0000","last_change":"2024-09-10T18:34:07.394899+0000","last_active":"2024-09-10T18:34:07.394951+0000","last_peered":"2024-09-10T18:34:07.394951+0000","last_clean":"2024-09-10T18:34:07.394951+0000","last_became_active":"2024-09-10T18:26:27.650774+0000","last_became_peered":"2024-09-10T18:26:27.650774+0000","last_unstale":"2024-09-10T18:34:07.394951+0000","last_undegraded":"2024-09-10T18:34:07.394951+0000","last_fullsized":"2024-09-10T18:34:07.394951+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:07.394835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:07.394835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:13.236135+0000","scrub_duration":0.001116433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":289,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:12.459048+0000","last_change":"2024-09-10T18:34:12.458962+0000","last_active":"2024-09-10T18:34:12.459048+0000","last_peered":"2024-09-10T18:34:12.459048+0000","last_clean":"2024-09-10T18:34:12.459048+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:34:12.459048+0000","last_undegraded":"2024-09-10T18:34:12.459048+0000","last_fullsized":"2024-09-10T18:34:12.459048+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:12.458742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.458742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:40.818274+0000","scrub_duration":0.0077313979999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"240'27","reported_seq":353,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.158246+0000","last_change":"2024-09-10T18:33:04.586182+0000","last_active":"2024-09-10T18:34:05.158246+0000","last_peered":"2024-09-10T18:34:05.158246+0000","last_clean":"2024-09-10T18:34:05.158246+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:34:05.158246+0000","last_undegraded":"2024-09-10T18:34:05.158246+0000","last_fullsized":"2024-09-10T18:34:05.158246+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:33:04.586063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:04.586063+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:25.050823+0000","scrub_duration":0.007748642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":298,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164599+0000","last_change":"2024-09-10T18:32:58.252469+0000","last_active":"2024-09-10T18:34:05.164599+0000","last_peered":"2024-09-10T18:34:05.164599+0000","last_clean":"2024-09-10T18:34:05.164599+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:34:05.164599+0000","last_undegraded":"2024-09-10T18:34:05.164599+0000","last_fullsized":"2024-09-10T18:34:05.164599+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:58.252419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.252419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.332332+0000","scrub_duration":0.0049362080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":285,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.179483+0000","last_change":"2024-09-10T18:31:53.754090+0000","last_active":"2024-09-10T18:34:05.179483+0000","last_peered":"2024-09-10T18:34:05.179483+0000","last_clean":"2024-09-10T18:34:05.179483+0000","last_became_active":"2024-09-10T18:26:27.640035+0000","last_became_peered":"2024-09-10T18:26:27.640035+0000","last_unstale":"2024-09-10T18:34:05.179483+0000","last_undegraded":"2024-09-10T18:34:05.179483+0000","last_fullsized":"2024-09-10T18:34:05.179483+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:53.754032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:53.754032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016988699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":292,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.174291+0000","last_change":"2024-09-10T18:32:59.567476+0000","last_active":"2024-09-10T18:34:05.174291+0000","last_peered":"2024-09-10T18:34:05.174291+0000","last_clean":"2024-09-10T18:34:05.174291+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:34:05.174291+0000","last_undegraded":"2024-09-10T18:34:05.174291+0000","last_fullsized":"2024-09-10T18:34:05.174291+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:59.567389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:32:59.567389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073893559999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'23","reported_seq":319,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.172318+0000","last_change":"2024-09-10T18:32:50.341138+0000","last_active":"2024-09-10T18:34:05.172318+0000","last_peered":"2024-09-10T18:34:05.172318+0000","last_clean":"2024-09-10T18:34:05.172318+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:34:05.172318+0000","last_undegraded":"2024-09-10T18:34:05.172318+0000","last_fullsized":"2024-09-10T18:34:05.172318+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'23","last_scrub_stamp":"2024-09-10T18:32:50.341044+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:32:50.341044+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012050170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":288,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.174608+0000","last_change":"2024-09-10T18:33:07.697770+0000","last_active":"2024-09-10T18:34:05.174608+0000","last_peered":"2024-09-10T18:34:05.174608+0000","last_clean":"2024-09-10T18:34:05.174608+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:34:05.174608+0000","last_undegraded":"2024-09-10T18:34:05.174608+0000","last_fullsized":"2024-09-10T18:34:05.174608+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:07.697667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:33:07.697667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:12.574607+0000","scrub_duration":0.044890945000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":288,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164971+0000","last_change":"2024-09-10T18:32:57.769495+0000","last_active":"2024-09-10T18:34:05.164971+0000","last_peered":"2024-09-10T18:34:05.164971+0000","last_clean":"2024-09-10T18:34:05.164971+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:34:05.164971+0000","last_undegraded":"2024-09-10T18:34:05.164971+0000","last_fullsized":"2024-09-10T18:34:05.164971+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:32:57.769408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:57.769408+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:20.116093+0000","scrub_duration":0.003757149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":302,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164225+0000","last_change":"2024-09-10T18:32:51.182762+0000","last_active":"2024-09-10T18:34:05.164225+0000","last_peered":"2024-09-10T18:34:05.164225+0000","last_clean":"2024-09-10T18:34:05.164225+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:34:05.164225+0000","last_undegraded":"2024-09-10T18:34:05.164225+0000","last_fullsized":"2024-09-10T18:34:05.164225+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:51.182684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:32:51.182684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:12.337885+0000","scrub_duration":0.003614637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'25","reported_seq":326,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.165231+0000","last_change":"2024-09-10T18:33:15.366317+0000","last_active":"2024-09-10T18:34:05.165231+0000","last_peered":"2024-09-10T18:34:05.165231+0000","last_clean":"2024-09-10T18:34:05.165231+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:34:05.165231+0000","last_undegraded":"2024-09-10T18:34:05.165231+0000","last_fullsized":"2024-09-10T18:34:05.165231+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:33:15.366107+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.366107+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:23.163279+0000","scrub_duration":0.014527711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"217'11","reported_seq":362,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:10.434385+0000","last_change":"2024-09-10T18:33:03.119057+0000","last_active":"2024-09-10T18:34:10.434385+0000","last_peered":"2024-09-10T18:34:10.434385+0000","last_clean":"2024-09-10T18:34:10.434385+0000","last_became_active":"2024-09-10T18:26:27.652489+0000","last_became_peered":"2024-09-10T18:26:27.652489+0000","last_unstale":"2024-09-10T18:34:10.434385+0000","last_undegraded":"2024-09-10T18:34:10.434385+0000","last_fullsized":"2024-09-10T18:34:10.434385+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11","last_scrub_stamp":"2024-09-10T18:33:03.118959+0000","last_deep_scrub":"217'11","last_deep_scrub_stamp":"2024-09-10T18:30:03.161985+0000","last_clean_scrub_stamp":"2024-09-10T18:33:03.118959+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.565288+0000","scrub_duration":0.002038715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"283'10986","reported_seq":11299,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:15.693983+0000","last_change":"2024-09-10T18:34:07.618780+0000","last_active":"2024-09-10T18:34:15.693983+0000","last_peered":"2024-09-10T18:34:15.693983+0000","last_clean":"2024-09-10T18:34:15.693983+0000","last_became_active":"2024-09-10T18:32:03.385428+0000","last_became_peered":"2024-09-10T18:32:03.385428+0000","last_unstale":"2024-09-10T18:34:15.693983+0000","last_undegraded":"2024-09-10T18:34:15.693983+0000","last_fullsized":"2024-09-10T18:34:15.693983+0000","mapping_epoch":246,"log_start":"267'6847","ondisk_log_start":"267'6847","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10375","last_scrub_stamp":"2024-09-10T18:34:07.618535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:07.618535+0000","objects_scrubbed":6480,"log_size":4139,"log_dups_size":0,"ondisk_log_size":4139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:25.198149+0000","scrub_duration":3.415268937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375390208,"num_objects":5728,"num_object_clones":0,"num_object_copies":17184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10986,"num_write_kb":534848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":209,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.157670+0000","last_change":"2024-09-10T18:33:56.155435+0000","last_active":"2024-09-10T18:34:05.157670+0000","last_peered":"2024-09-10T18:34:05.157670+0000","last_clean":"2024-09-10T18:34:05.157670+0000","last_became_active":"2024-09-10T18:26:28.635249+0000","last_became_peered":"2024-09-10T18:26:28.635249+0000","last_unstale":"2024-09-10T18:34:05.157670+0000","last_undegraded":"2024-09-10T18:34:05.157670+0000","last_fullsized":"2024-09-10T18:34:05.157670+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:56.155354+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:56.155354+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:03.302112+0000","scrub_duration":0.001679647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"240'34","reported_seq":353,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.165736+0000","last_change":"2024-09-10T18:32:56.285685+0000","last_active":"2024-09-10T18:34:05.165736+0000","last_peered":"2024-09-10T18:34:05.165736+0000","last_clean":"2024-09-10T18:34:05.165736+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:34:05.165736+0000","last_undegraded":"2024-09-10T18:34:05.165736+0000","last_fullsized":"2024-09-10T18:34:05.165736+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:32:56.285589+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:32:56.285589+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:15.128863+0000","scrub_duration":0.0081173400000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":284,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.171037+0000","last_change":"2024-09-10T18:31:59.772883+0000","last_active":"2024-09-10T18:34:05.171037+0000","last_peered":"2024-09-10T18:34:05.171037+0000","last_clean":"2024-09-10T18:34:05.171037+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:34:05.171037+0000","last_undegraded":"2024-09-10T18:34:05.171037+0000","last_fullsized":"2024-09-10T18:34:05.171037+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:59.772816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:59.772816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015225810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":451,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:14.495610+0000","last_change":"2024-09-10T18:34:14.495568+0000","last_active":"2024-09-10T18:34:14.495610+0000","last_peered":"2024-09-10T18:34:14.495610+0000","last_clean":"2024-09-10T18:34:14.495610+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:34:14.495610+0000","last_undegraded":"2024-09-10T18:34:14.495610+0000","last_fullsized":"2024-09-10T18:34:14.495610+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:34:14.495486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:14.495486+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:34.967733+0000","scrub_duration":0.010908085,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":287,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.158582+0000","last_change":"2024-09-10T18:32:50.961491+0000","last_active":"2024-09-10T18:34:05.158582+0000","last_peered":"2024-09-10T18:34:05.158582+0000","last_clean":"2024-09-10T18:34:05.158582+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:34:05.158582+0000","last_undegraded":"2024-09-10T18:34:05.158582+0000","last_fullsized":"2024-09-10T18:34:05.158582+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:50.961251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:32:50.961251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063477669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"283'10763","reported_seq":10809,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:14.949722+0000","last_change":"2024-09-10T18:32:03.379554+0000","last_active":"2024-09-10T18:34:14.949722+0000","last_peered":"2024-09-10T18:34:14.949722+0000","last_clean":"2024-09-10T18:34:14.949722+0000","last_became_active":"2024-09-10T18:32:03.379257+0000","last_became_peered":"2024-09-10T18:32:03.379257+0000","last_unstale":"2024-09-10T18:34:14.949722+0000","last_undegraded":"2024-09-10T18:34:14.949722+0000","last_fullsized":"2024-09-10T18:34:14.949722+0000","mapping_epoch":246,"log_start":"266'6544","ondisk_log_start":"266'6544","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4219,"log_dups_size":0,"ondisk_log_size":4219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376832024,"num_objects":5751,"num_object_clones":0,"num_object_copies":17253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5751,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10763,"num_write_kb":528385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":208,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.174985+0000","last_change":"2024-09-10T18:34:00.150322+0000","last_active":"2024-09-10T18:34:05.174985+0000","last_peered":"2024-09-10T18:34:05.174985+0000","last_clean":"2024-09-10T18:34:05.174985+0000","last_became_active":"2024-09-10T18:34:00.150054+0000","last_became_peered":"2024-09-10T18:34:00.150054+0000","last_unstale":"2024-09-10T18:34:05.174985+0000","last_undegraded":"2024-09-10T18:34:05.174985+0000","last_fullsized":"2024-09-10T18:34:05.174985+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:58.561425+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.561425+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002234073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":332,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.170763+0000","last_change":"2024-09-10T18:32:59.382953+0000","last_active":"2024-09-10T18:34:05.170763+0000","last_peered":"2024-09-10T18:34:05.170763+0000","last_clean":"2024-09-10T18:34:05.170763+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:34:05.170763+0000","last_undegraded":"2024-09-10T18:34:05.170763+0000","last_fullsized":"2024-09-10T18:34:05.170763+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-09-10T18:32:59.382840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:59.382840+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:18.205789+0000","scrub_duration":0.0072140750000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":284,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.179998+0000","last_change":"2024-09-10T18:32:49.346233+0000","last_active":"2024-09-10T18:34:05.179998+0000","last_peered":"2024-09-10T18:34:05.179998+0000","last_clean":"2024-09-10T18:34:05.179998+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:34:05.179998+0000","last_undegraded":"2024-09-10T18:34:05.179998+0000","last_fullsized":"2024-09-10T18:34:05.179998+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:49.342269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.140922+0000","last_clean_scrub_stamp":"2024-09-10T18:32:49.342269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006539385,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":288,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.173587+0000","last_change":"2024-09-10T18:32:52.582545+0000","last_active":"2024-09-10T18:34:05.173587+0000","last_peered":"2024-09-10T18:34:05.173587+0000","last_clean":"2024-09-10T18:34:05.173587+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:34:05.173587+0000","last_undegraded":"2024-09-10T18:34:05.173587+0000","last_fullsized":"2024-09-10T18:34:05.173587+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:52.582464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:32:52.582464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:16.167011+0000","scrub_duration":0.0024277399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":276,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.174112+0000","last_change":"2024-09-10T18:32:44.600295+0000","last_active":"2024-09-10T18:34:05.174112+0000","last_peered":"2024-09-10T18:34:05.174112+0000","last_clean":"2024-09-10T18:34:05.174112+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:34:05.174112+0000","last_undegraded":"2024-09-10T18:34:05.174112+0000","last_fullsized":"2024-09-10T18:34:05.174112+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:44.600200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:44.600200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073958740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"283'10995","reported_seq":11040,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:15.686188+0000","last_change":"2024-09-10T18:32:03.554760+0000","last_active":"2024-09-10T18:34:15.686188+0000","last_peered":"2024-09-10T18:34:15.686188+0000","last_clean":"2024-09-10T18:34:15.686188+0000","last_became_active":"2024-09-10T18:32:03.554400+0000","last_became_peered":"2024-09-10T18:32:03.554400+0000","last_unstale":"2024-09-10T18:34:15.686188+0000","last_undegraded":"2024-09-10T18:34:15.686188+0000","last_fullsized":"2024-09-10T18:34:15.686188+0000","mapping_epoch":246,"log_start":"266'6862","ondisk_log_start":"266'6862","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4133,"log_dups_size":0,"ondisk_log_size":4133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384237568,"num_objects":5863,"num_object_clones":0,"num_object_copies":17589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10995,"num_write_kb":539456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":199,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.173113+0000","last_change":"2024-09-10T18:34:00.166754+0000","last_active":"2024-09-10T18:34:05.173113+0000","last_peered":"2024-09-10T18:34:05.173113+0000","last_clean":"2024-09-10T18:34:05.173113+0000","last_became_active":"2024-09-10T18:34:00.166362+0000","last_became_peered":"2024-09-10T18:34:00.166362+0000","last_unstale":"2024-09-10T18:34:05.173113+0000","last_undegraded":"2024-09-10T18:34:05.173113+0000","last_fullsized":"2024-09-10T18:34:05.173113+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.370561+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.370561+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029251390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"240'28","reported_seq":339,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.173676+0000","last_change":"2024-09-10T18:33:59.587714+0000","last_active":"2024-09-10T18:34:05.173676+0000","last_peered":"2024-09-10T18:34:05.173676+0000","last_clean":"2024-09-10T18:34:05.173676+0000","last_became_active":"2024-09-10T18:26:27.651364+0000","last_became_peered":"2024-09-10T18:26:27.651364+0000","last_unstale":"2024-09-10T18:34:05.173676+0000","last_undegraded":"2024-09-10T18:34:05.173676+0000","last_fullsized":"2024-09-10T18:34:05.173676+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'28","last_scrub_stamp":"2024-09-10T18:33:59.587604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:59.587604+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:21.896292+0000","scrub_duration":0.0024058019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":283,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.165495+0000","last_change":"2024-09-10T18:32:54.291391+0000","last_active":"2024-09-10T18:34:05.165495+0000","last_peered":"2024-09-10T18:34:05.165495+0000","last_clean":"2024-09-10T18:34:05.165495+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:34:05.165495+0000","last_undegraded":"2024-09-10T18:34:05.165495+0000","last_fullsized":"2024-09-10T18:34:05.165495+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:54.291164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:32:54.291164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002933667,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":442,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:10.434604+0000","last_change":"2024-09-10T18:33:07.367306+0000","last_active":"2024-09-10T18:34:10.434604+0000","last_peered":"2024-09-10T18:34:10.434604+0000","last_clean":"2024-09-10T18:34:10.434604+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:34:10.434604+0000","last_undegraded":"2024-09-10T18:34:10.434604+0000","last_fullsized":"2024-09-10T18:34:10.434604+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:33:07.367248+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:33:07.367248+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:25.811396+0000","scrub_duration":0.0044945339999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":296,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164356+0000","last_change":"2024-09-10T18:33:11.857892+0000","last_active":"2024-09-10T18:34:05.164356+0000","last_peered":"2024-09-10T18:34:05.164356+0000","last_clean":"2024-09-10T18:34:05.164356+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:34:05.164356+0000","last_undegraded":"2024-09-10T18:34:05.164356+0000","last_fullsized":"2024-09-10T18:34:05.164356+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:11.857685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.857685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:36.504074+0000","scrub_duration":0.0022995870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"283'10914","reported_seq":11241,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:14.961435+0000","last_change":"2024-09-10T18:34:04.158338+0000","last_active":"2024-09-10T18:34:14.961435+0000","last_peered":"2024-09-10T18:34:14.961435+0000","last_clean":"2024-09-10T18:34:14.961435+0000","last_became_active":"2024-09-10T18:32:03.384288+0000","last_became_peered":"2024-09-10T18:32:03.384288+0000","last_unstale":"2024-09-10T18:34:14.961435+0000","last_undegraded":"2024-09-10T18:34:14.961435+0000","last_fullsized":"2024-09-10T18:34:14.961435+0000","mapping_epoch":246,"log_start":"265'6333","ondisk_log_start":"265'6333","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'10160","last_scrub_stamp":"2024-09-10T18:34:04.158266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:04.158266+0000","objects_scrubbed":6798,"log_size":4581,"log_dups_size":0,"ondisk_log_size":4581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:13.705054+0000","scrub_duration":2.9688170149999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387842048,"num_objects":5918,"num_object_clones":0,"num_object_copies":17754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10914,"num_write_kb":538624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":268,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:08.035581+0000","last_change":"2024-09-10T18:34:08.035514+0000","last_active":"2024-09-10T18:34:08.035581+0000","last_peered":"2024-09-10T18:34:08.035581+0000","last_clean":"2024-09-10T18:34:08.035581+0000","last_became_active":"2024-09-10T18:34:00.155184+0000","last_became_peered":"2024-09-10T18:34:00.155184+0000","last_unstale":"2024-09-10T18:34:08.035581+0000","last_undegraded":"2024-09-10T18:34:08.035581+0000","last_fullsized":"2024-09-10T18:34:08.035581+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:08.035431+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:08.035431+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:29.943428+0000","scrub_duration":0.065049888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":335,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.166481+0000","last_change":"2024-09-10T18:32:55.279937+0000","last_active":"2024-09-10T18:34:05.166481+0000","last_peered":"2024-09-10T18:34:05.166481+0000","last_clean":"2024-09-10T18:34:05.166481+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:34:05.166481+0000","last_undegraded":"2024-09-10T18:34:05.166481+0000","last_fullsized":"2024-09-10T18:34:05.166481+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:32:55.279209+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:32:55.279209+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091125259999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":283,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.159393+0000","last_change":"2024-09-10T18:32:45.965208+0000","last_active":"2024-09-10T18:34:05.159393+0000","last_peered":"2024-09-10T18:34:05.159393+0000","last_clean":"2024-09-10T18:34:05.159393+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:34:05.159393+0000","last_undegraded":"2024-09-10T18:34:05.159393+0000","last_fullsized":"2024-09-10T18:34:05.159393+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:45.965129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:32:45.965129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045521829999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"263'11","reported_seq":450,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:10.426793+0000","last_change":"2024-09-10T18:33:01.283747+0000","last_active":"2024-09-10T18:34:10.426793+0000","last_peered":"2024-09-10T18:34:10.426793+0000","last_clean":"2024-09-10T18:34:10.426793+0000","last_became_active":"2024-09-10T18:26:27.639032+0000","last_became_peered":"2024-09-10T18:26:27.639032+0000","last_unstale":"2024-09-10T18:34:10.426793+0000","last_undegraded":"2024-09-10T18:34:10.426793+0000","last_fullsized":"2024-09-10T18:34:10.426793+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'11","last_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_deep_scrub":"263'11","last_deep_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.283648+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:27.247085+0000","scrub_duration":0.0081388439999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":287,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.167130+0000","last_change":"2024-09-10T18:33:57.483999+0000","last_active":"2024-09-10T18:34:05.167130+0000","last_peered":"2024-09-10T18:34:05.167130+0000","last_clean":"2024-09-10T18:34:05.167130+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:34:05.167130+0000","last_undegraded":"2024-09-10T18:34:05.167130+0000","last_fullsized":"2024-09-10T18:34:05.167130+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:57.483905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:57.483905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:58.495941+0000","scrub_duration":0.023666573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"283'11000","reported_seq":11045,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:14.938896+0000","last_change":"2024-09-10T18:32:03.555680+0000","last_active":"2024-09-10T18:34:14.938896+0000","last_peered":"2024-09-10T18:34:14.938896+0000","last_clean":"2024-09-10T18:34:14.938896+0000","last_became_active":"2024-09-10T18:32:03.554693+0000","last_became_peered":"2024-09-10T18:32:03.554693+0000","last_unstale":"2024-09-10T18:34:14.938896+0000","last_undegraded":"2024-09-10T18:34:14.938896+0000","last_fullsized":"2024-09-10T18:34:14.938896+0000","mapping_epoch":246,"log_start":"266'6772","ondisk_log_start":"266'6772","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4228,"log_dups_size":0,"ondisk_log_size":4228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383254528,"num_objects":5848,"num_object_clones":0,"num_object_copies":17544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11000,"num_write_kb":539136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":268,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.167666+0000","last_change":"2024-09-10T18:34:05.012137+0000","last_active":"2024-09-10T18:34:05.167666+0000","last_peered":"2024-09-10T18:34:05.167666+0000","last_clean":"2024-09-10T18:34:05.167666+0000","last_became_active":"2024-09-10T18:34:00.154611+0000","last_became_peered":"2024-09-10T18:34:00.154611+0000","last_unstale":"2024-09-10T18:34:05.167666+0000","last_undegraded":"2024-09-10T18:34:05.167666+0000","last_fullsized":"2024-09-10T18:34:05.167666+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:05.012053+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:05.012053+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:13.854010+0000","scrub_duration":0.0025346610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"240'29","reported_seq":342,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.166333+0000","last_change":"2024-09-10T18:33:11.310163+0000","last_active":"2024-09-10T18:34:05.166333+0000","last_peered":"2024-09-10T18:34:05.166333+0000","last_clean":"2024-09-10T18:34:05.166333+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:34:05.166333+0000","last_undegraded":"2024-09-10T18:34:05.166333+0000","last_fullsized":"2024-09-10T18:34:05.166333+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'29","last_scrub_stamp":"2024-09-10T18:33:11.310074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.310074+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.669570+0000","scrub_duration":0.0026943760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":292,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.168618+0000","last_change":"2024-09-10T18:34:02.559656+0000","last_active":"2024-09-10T18:34:05.168618+0000","last_peered":"2024-09-10T18:34:05.168618+0000","last_clean":"2024-09-10T18:34:05.168618+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:34:05.168618+0000","last_undegraded":"2024-09-10T18:34:05.168618+0000","last_fullsized":"2024-09-10T18:34:05.168618+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:02.559431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:02.559431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:18.828380+0000","scrub_duration":0.019462604000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":238,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.166332+0000","last_change":"2024-09-10T18:33:09.183070+0000","last_active":"2024-09-10T18:34:05.166332+0000","last_peered":"2024-09-10T18:34:05.166332+0000","last_clean":"2024-09-10T18:34:05.166332+0000","last_became_active":"2024-09-10T18:26:27.642063+0000","last_became_peered":"2024-09-10T18:26:27.642063+0000","last_unstale":"2024-09-10T18:34:05.166332+0000","last_undegraded":"2024-09-10T18:34:05.166332+0000","last_fullsized":"2024-09-10T18:34:05.166332+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:09.182982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:33:09.182982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:14.241099+0000","scrub_duration":0.0023969899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":293,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.163834+0000","last_change":"2024-09-10T18:33:10.137145+0000","last_active":"2024-09-10T18:34:05.163834+0000","last_peered":"2024-09-10T18:34:05.163834+0000","last_clean":"2024-09-10T18:34:05.163834+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:34:05.163834+0000","last_undegraded":"2024-09-10T18:34:05.163834+0000","last_fullsized":"2024-09-10T18:34:05.163834+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:10.137048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:33:10.137048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:31.309662+0000","scrub_duration":0.0040149249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"283'14522","reported_seq":14780,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:16.061985+0000","last_change":"2024-09-10T18:33:39.168175+0000","last_active":"2024-09-10T18:34:16.061985+0000","last_peered":"2024-09-10T18:34:16.061985+0000","last_clean":"2024-09-10T18:34:16.061985+0000","last_became_active":"2024-09-10T18:32:03.381872+0000","last_became_peered":"2024-09-10T18:32:03.381872+0000","last_unstale":"2024-09-10T18:34:16.061985+0000","last_undegraded":"2024-09-10T18:34:16.061985+0000","last_fullsized":"2024-09-10T18:34:16.061985+0000","mapping_epoch":246,"log_start":"267'9097","ondisk_log_start":"267'9097","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":5148,"log_size":5425,"log_dups_size":0,"ondisk_log_size":5425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377290752,"num_objects":5757,"num_object_clones":0,"num_object_copies":17271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10991,"num_write_kb":535936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":194,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.159700+0000","last_change":"2024-09-10T18:34:00.151547+0000","last_active":"2024-09-10T18:34:05.159700+0000","last_peered":"2024-09-10T18:34:05.159700+0000","last_clean":"2024-09-10T18:34:05.159700+0000","last_became_active":"2024-09-10T18:34:00.151017+0000","last_became_peered":"2024-09-10T18:34:00.151017+0000","last_unstale":"2024-09-10T18:34:05.159700+0000","last_undegraded":"2024-09-10T18:34:05.159700+0000","last_fullsized":"2024-09-10T18:34:05.159700+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:56.629683+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:56.629683+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001774133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"240'41","reported_seq":428,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:10.432768+0000","last_change":"2024-09-10T18:32:55.941738+0000","last_active":"2024-09-10T18:34:10.432768+0000","last_peered":"2024-09-10T18:34:10.432768+0000","last_clean":"2024-09-10T18:34:10.432768+0000","last_became_active":"2024-09-10T18:23:10.067049+0000","last_became_peered":"2024-09-10T18:23:10.067049+0000","last_unstale":"2024-09-10T18:34:10.432768+0000","last_undegraded":"2024-09-10T18:34:10.432768+0000","last_fullsized":"2024-09-10T18:34:10.432768+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'41","last_scrub_stamp":"2024-09-10T18:32:55.941653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:55.941653+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011585564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":289,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.174388+0000","last_change":"2024-09-10T18:31:49.684883+0000","last_active":"2024-09-10T18:34:05.174388+0000","last_peered":"2024-09-10T18:34:05.174388+0000","last_clean":"2024-09-10T18:34:05.174388+0000","last_became_active":"2024-09-10T18:26:27.648690+0000","last_became_peered":"2024-09-10T18:26:27.648690+0000","last_unstale":"2024-09-10T18:34:05.174388+0000","last_undegraded":"2024-09-10T18:34:05.174388+0000","last_fullsized":"2024-09-10T18:34:05.174388+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:49.684824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:49.684824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012508370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":308,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.159354+0000","last_change":"2024-09-10T18:33:04.936495+0000","last_active":"2024-09-10T18:34:05.159354+0000","last_peered":"2024-09-10T18:34:05.159354+0000","last_clean":"2024-09-10T18:34:05.159354+0000","last_became_active":"2024-09-10T18:26:27.647576+0000","last_became_peered":"2024-09-10T18:26:27.647576+0000","last_unstale":"2024-09-10T18:34:05.159354+0000","last_undegraded":"2024-09-10T18:34:05.159354+0000","last_fullsized":"2024-09-10T18:34:05.159354+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:33:04.936407+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:33:04.936407+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:31.757007+0000","scrub_duration":0.0096149180000000001,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":279,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.170095+0000","last_change":"2024-09-10T18:32:39.119345+0000","last_active":"2024-09-10T18:34:05.170095+0000","last_peered":"2024-09-10T18:34:05.170095+0000","last_clean":"2024-09-10T18:34:05.170095+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:34:05.170095+0000","last_undegraded":"2024-09-10T18:34:05.170095+0000","last_fullsized":"2024-09-10T18:34:05.170095+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:39.119228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.119228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011330534999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":278,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.172650+0000","last_change":"2024-09-10T18:33:56.168567+0000","last_active":"2024-09-10T18:34:05.172650+0000","last_peered":"2024-09-10T18:34:05.172650+0000","last_clean":"2024-09-10T18:34:05.172650+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:34:05.172650+0000","last_undegraded":"2024-09-10T18:34:05.172650+0000","last_fullsized":"2024-09-10T18:34:05.172650+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:56.168485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:56.168485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:14.414730+0000","scrub_duration":0.002436723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"283'11103","reported_seq":11425,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:14.971687+0000","last_change":"2024-09-10T18:34:06.677663+0000","last_active":"2024-09-10T18:34:14.971687+0000","last_peered":"2024-09-10T18:34:14.971687+0000","last_clean":"2024-09-10T18:34:14.971687+0000","last_became_active":"2024-09-10T18:32:03.380391+0000","last_became_peered":"2024-09-10T18:32:03.380391+0000","last_unstale":"2024-09-10T18:34:14.971687+0000","last_undegraded":"2024-09-10T18:34:14.971687+0000","last_fullsized":"2024-09-10T18:34:14.971687+0000","mapping_epoch":246,"log_start":"265'6472","ondisk_log_start":"265'6472","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10475","last_scrub_stamp":"2024-09-10T18:34:06.677564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.677564+0000","objects_scrubbed":6683,"log_size":4631,"log_dups_size":0,"ondisk_log_size":4631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:12.828051+0000","scrub_duration":2.5274349109999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390791168,"num_objects":5963,"num_object_clones":0,"num_object_copies":17889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11103,"num_write_kb":546112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":340,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164860+0000","last_change":"2024-09-10T18:34:00.154687+0000","last_active":"2024-09-10T18:34:05.164860+0000","last_peered":"2024-09-10T18:34:05.164860+0000","last_clean":"2024-09-10T18:34:05.164860+0000","last_became_active":"2024-09-10T18:34:00.154404+0000","last_became_peered":"2024-09-10T18:34:00.154404+0000","last_unstale":"2024-09-10T18:34:05.164860+0000","last_undegraded":"2024-09-10T18:34:05.164860+0000","last_fullsized":"2024-09-10T18:34:05.164860+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:57.474295+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.474295+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015998690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"240'17","reported_seq":333,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:13.522251+0000","last_change":"2024-09-10T18:34:13.522168+0000","last_active":"2024-09-10T18:34:13.522251+0000","last_peered":"2024-09-10T18:34:13.522251+0000","last_clean":"2024-09-10T18:34:13.522251+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:34:13.522251+0000","last_undegraded":"2024-09-10T18:34:13.522251+0000","last_fullsized":"2024-09-10T18:34:13.522251+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'17","last_scrub_stamp":"2024-09-10T18:34:13.521944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:13.521944+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:35.758470+0000","scrub_duration":0.0029448339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":309,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.165989+0000","last_change":"2024-09-10T18:33:15.142062+0000","last_active":"2024-09-10T18:34:05.165989+0000","last_peered":"2024-09-10T18:34:05.165989+0000","last_clean":"2024-09-10T18:34:05.165989+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:34:05.165989+0000","last_undegraded":"2024-09-10T18:34:05.165989+0000","last_fullsized":"2024-09-10T18:34:05.165989+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:15.141982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:15.141982+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.141982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:45.048602+0000","scrub_duration":0.0028956749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":321,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:06.493594+0000","last_change":"2024-09-10T18:34:06.493534+0000","last_active":"2024-09-10T18:34:06.493594+0000","last_peered":"2024-09-10T18:34:06.493594+0000","last_clean":"2024-09-10T18:34:06.493594+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:34:06.493594+0000","last_undegraded":"2024-09-10T18:34:06.493594+0000","last_fullsized":"2024-09-10T18:34:06.493594+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:34:06.493448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.493448+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:21.109643+0000","scrub_duration":0.011827739,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":298,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.158304+0000","last_change":"2024-09-10T18:32:53.586654+0000","last_active":"2024-09-10T18:34:05.158304+0000","last_peered":"2024-09-10T18:34:05.158304+0000","last_clean":"2024-09-10T18:34:05.158304+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:34:05.158304+0000","last_undegraded":"2024-09-10T18:34:05.158304+0000","last_fullsized":"2024-09-10T18:34:05.158304+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:53.586216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:32:53.586216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:22.249594+0000","scrub_duration":0.0053348709999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":286,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164734+0000","last_change":"2024-09-10T18:32:00.501396+0000","last_active":"2024-09-10T18:34:05.164734+0000","last_peered":"2024-09-10T18:34:05.164734+0000","last_clean":"2024-09-10T18:34:05.164734+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:34:05.164734+0000","last_undegraded":"2024-09-10T18:34:05.164734+0000","last_fullsized":"2024-09-10T18:34:05.164734+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.501344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.501344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001367442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"283'11150","reported_seq":11484,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:16.067374+0000","last_change":"2024-09-10T18:34:01.902715+0000","last_active":"2024-09-10T18:34:16.067374+0000","last_peered":"2024-09-10T18:34:16.067374+0000","last_clean":"2024-09-10T18:34:16.067374+0000","last_became_active":"2024-09-10T18:32:03.554958+0000","last_became_peered":"2024-09-10T18:32:03.554958+0000","last_unstale":"2024-09-10T18:34:16.067374+0000","last_undegraded":"2024-09-10T18:34:16.067374+0000","last_fullsized":"2024-09-10T18:34:16.067374+0000","mapping_epoch":246,"log_start":"263'5770","ondisk_log_start":"263'5770","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'10034","last_scrub_stamp":"2024-09-10T18:34:01.902618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:01.902618+0000","objects_scrubbed":6997,"log_size":5380,"log_dups_size":0,"ondisk_log_size":5380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:18.230790+0000","scrub_duration":2.3941321589999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379584512,"num_objects":5792,"num_object_clones":0,"num_object_copies":17376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11150,"num_write_kb":542144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":209,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.166264+0000","last_change":"2024-09-10T18:34:00.166073+0000","last_active":"2024-09-10T18:34:05.166264+0000","last_peered":"2024-09-10T18:34:05.166264+0000","last_clean":"2024-09-10T18:34:05.166264+0000","last_became_active":"2024-09-10T18:34:00.165384+0000","last_became_peered":"2024-09-10T18:34:00.165384+0000","last_unstale":"2024-09-10T18:34:05.166264+0000","last_undegraded":"2024-09-10T18:34:05.166264+0000","last_fullsized":"2024-09-10T18:34:05.166264+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:48.466654+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.466654+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":317,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.165319+0000","last_change":"2024-09-10T18:32:53.270762+0000","last_active":"2024-09-10T18:34:05.165319+0000","last_peered":"2024-09-10T18:34:05.165319+0000","last_clean":"2024-09-10T18:34:05.165319+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:34:05.165319+0000","last_undegraded":"2024-09-10T18:34:05.165319+0000","last_fullsized":"2024-09-10T18:34:05.165319+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-09-10T18:32:53.270667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:53.270667+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:14.044059+0000","scrub_duration":0.01096307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":293,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.165381+0000","last_change":"2024-09-10T18:32:59.245911+0000","last_active":"2024-09-10T18:34:05.165381+0000","last_peered":"2024-09-10T18:34:05.165381+0000","last_clean":"2024-09-10T18:34:05.165381+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:34:05.165381+0000","last_undegraded":"2024-09-10T18:34:05.165381+0000","last_fullsized":"2024-09-10T18:34:05.165381+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:59.245802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:32:59.245802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:13.023525+0000","scrub_duration":0.028203379000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":1066,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:13.792225+0000","last_change":"2024-09-10T18:34:02.136505+0000","last_active":"2024-09-10T18:34:13.792225+0000","last_peered":"2024-09-10T18:34:13.792225+0000","last_clean":"2024-09-10T18:34:13.792225+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:34:13.792225+0000","last_undegraded":"2024-09-10T18:34:13.792225+0000","last_fullsized":"2024-09-10T18:34:13.792225+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:34:02.136429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:02.136429+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:32.048652+0000","scrub_duration":0.0083841149999999993,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":756,"num_read_kb":751,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":282,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.160162+0000","last_change":"2024-09-10T18:32:56.888589+0000","last_active":"2024-09-10T18:34:05.160162+0000","last_peered":"2024-09-10T18:34:05.160162+0000","last_clean":"2024-09-10T18:34:05.160162+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:34:05.160162+0000","last_undegraded":"2024-09-10T18:34:05.160162+0000","last_fullsized":"2024-09-10T18:34:05.160162+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:56.888436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.771415+0000","last_clean_scrub_stamp":"2024-09-10T18:32:56.888436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:19.568187+0000","scrub_duration":0.0064579939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":286,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:12.539235+0000","last_change":"2024-09-10T18:34:12.539195+0000","last_active":"2024-09-10T18:34:12.539235+0000","last_peered":"2024-09-10T18:34:12.539235+0000","last_clean":"2024-09-10T18:34:12.539235+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:34:12.539235+0000","last_undegraded":"2024-09-10T18:34:12.539235+0000","last_fullsized":"2024-09-10T18:34:12.539235+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:12.539124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.539124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:33.017675+0000","scrub_duration":0.0079965020000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"283'10815","reported_seq":10860,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:15.695513+0000","last_change":"2024-09-10T18:32:03.382727+0000","last_active":"2024-09-10T18:34:15.695513+0000","last_peered":"2024-09-10T18:34:15.695513+0000","last_clean":"2024-09-10T18:34:15.695513+0000","last_became_active":"2024-09-10T18:32:03.382335+0000","last_became_peered":"2024-09-10T18:32:03.382335+0000","last_unstale":"2024-09-10T18:34:15.695513+0000","last_undegraded":"2024-09-10T18:34:15.695513+0000","last_fullsized":"2024-09-10T18:34:15.695513+0000","mapping_epoch":246,"log_start":"266'6698","ondisk_log_start":"266'6698","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4117,"log_dups_size":0,"ondisk_log_size":4117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378601472,"num_objects":5777,"num_object_clones":0,"num_object_copies":17331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10815,"num_write_kb":530944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'26","reported_seq":275,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.167405+0000","last_change":"2024-09-10T18:32:42.037679+0000","last_active":"2024-09-10T18:34:05.167405+0000","last_peered":"2024-09-10T18:34:05.167405+0000","last_clean":"2024-09-10T18:34:05.167405+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:34:05.167405+0000","last_undegraded":"2024-09-10T18:34:05.167405+0000","last_fullsized":"2024-09-10T18:34:05.167405+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'26","last_scrub_stamp":"2024-09-10T18:32:42.037595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:42.037595+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003897162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":291,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.172217+0000","last_change":"2024-09-10T18:31:52.756369+0000","last_active":"2024-09-10T18:34:05.172217+0000","last_peered":"2024-09-10T18:34:05.172217+0000","last_clean":"2024-09-10T18:34:05.172217+0000","last_became_active":"2024-09-10T18:26:27.639129+0000","last_became_peered":"2024-09-10T18:26:27.639129+0000","last_unstale":"2024-09-10T18:34:05.172217+0000","last_undegraded":"2024-09-10T18:34:05.172217+0000","last_fullsized":"2024-09-10T18:34:05.172217+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:52.756258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:31:52.756258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002681269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":352,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164098+0000","last_change":"2024-09-10T18:34:00.160610+0000","last_active":"2024-09-10T18:34:05.164098+0000","last_peered":"2024-09-10T18:34:05.164098+0000","last_clean":"2024-09-10T18:34:05.164098+0000","last_became_active":"2024-09-10T18:34:00.159975+0000","last_became_peered":"2024-09-10T18:34:00.159975+0000","last_unstale":"2024-09-10T18:34:05.164098+0000","last_undegraded":"2024-09-10T18:34:05.164098+0000","last_fullsized":"2024-09-10T18:34:05.164098+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:58.780613+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.780613+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002490309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":298,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:12.920770+0000","last_change":"2024-09-10T18:34:12.920724+0000","last_active":"2024-09-10T18:34:12.920770+0000","last_peered":"2024-09-10T18:34:12.920770+0000","last_clean":"2024-09-10T18:34:12.920770+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:34:12.920770+0000","last_undegraded":"2024-09-10T18:34:12.920770+0000","last_fullsized":"2024-09-10T18:34:12.920770+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:12.920651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.920651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:29.441715+0000","scrub_duration":0.0047636090000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":280,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.160004+0000","last_change":"2024-09-10T18:32:49.984765+0000","last_active":"2024-09-10T18:34:05.160004+0000","last_peered":"2024-09-10T18:34:05.160004+0000","last_clean":"2024-09-10T18:34:05.160004+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:34:05.160004+0000","last_undegraded":"2024-09-10T18:34:05.160004+0000","last_fullsized":"2024-09-10T18:34:05.160004+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:49.984707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:32:49.984707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035569920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"76'81","reported_seq":218,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.166396+0000","last_change":"2024-09-10T18:34:02.816825+0000","last_active":"2024-09-10T18:34:05.166396+0000","last_peered":"2024-09-10T18:34:05.166396+0000","last_clean":"2024-09-10T18:34:05.166396+0000","last_became_active":"2024-09-10T18:34:02.810774+0000","last_became_peered":"2024-09-10T18:34:02.810774+0000","last_unstale":"2024-09-10T18:34:05.166396+0000","last_undegraded":"2024-09-10T18:34:05.166396+0000","last_fullsized":"2024-09-10T18:34:05.166396+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:48.466654+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.466654+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"283'10913","reported_seq":10958,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:14.953416+0000","last_change":"2024-09-10T18:32:03.384526+0000","last_active":"2024-09-10T18:34:14.953416+0000","last_peered":"2024-09-10T18:34:14.953416+0000","last_clean":"2024-09-10T18:34:14.953416+0000","last_became_active":"2024-09-10T18:32:03.384247+0000","last_became_peered":"2024-09-10T18:32:03.384247+0000","last_unstale":"2024-09-10T18:34:14.953416+0000","last_undegraded":"2024-09-10T18:34:14.953416+0000","last_fullsized":"2024-09-10T18:34:14.953416+0000","mapping_epoch":246,"log_start":"265'6684","ondisk_log_start":"265'6684","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4229,"log_dups_size":0,"ondisk_log_size":4229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388300800,"num_objects":5925,"num_object_clones":0,"num_object_copies":17775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10913,"num_write_kb":538816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"240'19","reported_seq":321,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.175312+0000","last_change":"2024-09-10T18:31:50.655266+0000","last_active":"2024-09-10T18:34:05.175312+0000","last_peered":"2024-09-10T18:34:05.175312+0000","last_clean":"2024-09-10T18:34:05.175312+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:34:05.175312+0000","last_undegraded":"2024-09-10T18:34:05.175312+0000","last_fullsized":"2024-09-10T18:34:05.175312+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:31:50.655208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:31:50.655208+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021897840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":282,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.173747+0000","last_change":"2024-09-10T18:32:40.540692+0000","last_active":"2024-09-10T18:34:05.173747+0000","last_peered":"2024-09-10T18:34:05.173747+0000","last_clean":"2024-09-10T18:34:05.173747+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:34:05.173747+0000","last_undegraded":"2024-09-10T18:34:05.173747+0000","last_fullsized":"2024-09-10T18:34:05.173747+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:40.540616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:32:40.540616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002137262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":296,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.159399+0000","last_change":"2024-09-10T18:33:06.960402+0000","last_active":"2024-09-10T18:34:05.159399+0000","last_peered":"2024-09-10T18:34:05.159399+0000","last_clean":"2024-09-10T18:34:05.159399+0000","last_became_active":"2024-09-10T18:26:27.645885+0000","last_became_peered":"2024-09-10T18:26:27.645885+0000","last_unstale":"2024-09-10T18:34:05.159399+0000","last_undegraded":"2024-09-10T18:34:05.159399+0000","last_fullsized":"2024-09-10T18:34:05.159399+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:06.960320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:33:06.960320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:19.113831+0000","scrub_duration":0.0020571560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":295,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.173186+0000","last_change":"2024-09-10T18:34:01.181010+0000","last_active":"2024-09-10T18:34:05.173186+0000","last_peered":"2024-09-10T18:34:05.173186+0000","last_clean":"2024-09-10T18:34:05.173186+0000","last_became_active":"2024-09-10T18:26:27.632071+0000","last_became_peered":"2024-09-10T18:26:27.632071+0000","last_unstale":"2024-09-10T18:34:05.173186+0000","last_undegraded":"2024-09-10T18:34:05.173186+0000","last_fullsized":"2024-09-10T18:34:05.173186+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:01.180920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:01.180920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:10.276096+0000","scrub_duration":0.0082996130000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"283'11109","reported_seq":11553,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:15.404984+0000","last_change":"2024-09-10T18:34:07.700772+0000","last_active":"2024-09-10T18:34:15.404984+0000","last_peered":"2024-09-10T18:34:15.404984+0000","last_clean":"2024-09-10T18:34:15.404984+0000","last_became_active":"2024-09-10T18:32:03.554412+0000","last_became_peered":"2024-09-10T18:32:03.554412+0000","last_unstale":"2024-09-10T18:34:15.404984+0000","last_undegraded":"2024-09-10T18:34:15.404984+0000","last_fullsized":"2024-09-10T18:34:15.404984+0000","mapping_epoch":246,"log_start":"265'6660","ondisk_log_start":"265'6660","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10569","last_scrub_stamp":"2024-09-10T18:34:07.700574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:07.700574+0000","objects_scrubbed":6543,"log_size":4449,"log_dups_size":0,"ondisk_log_size":4449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:32.220981+0000","scrub_duration":3.13969619,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383713280,"num_objects":5855,"num_object_clones":0,"num_object_copies":17565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11109,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":201,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.159549+0000","last_change":"2024-09-10T18:33:08.009896+0000","last_active":"2024-09-10T18:34:05.159549+0000","last_peered":"2024-09-10T18:34:05.159549+0000","last_clean":"2024-09-10T18:34:05.159549+0000","last_became_active":"2024-09-10T18:26:18.095088+0000","last_became_peered":"2024-09-10T18:26:18.095088+0000","last_unstale":"2024-09-10T18:34:05.159549+0000","last_undegraded":"2024-09-10T18:34:05.159549+0000","last_fullsized":"2024-09-10T18:34:05.159549+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:08.009806+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.009806+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:37.524807+0000","scrub_duration":0.002902614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":339,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.174966+0000","last_change":"2024-09-10T18:34:00.542831+0000","last_active":"2024-09-10T18:34:05.174966+0000","last_peered":"2024-09-10T18:34:05.174966+0000","last_clean":"2024-09-10T18:34:05.174966+0000","last_became_active":"2024-09-10T18:21:33.611203+0000","last_became_peered":"2024-09-10T18:21:33.611203+0000","last_unstale":"2024-09-10T18:34:05.174966+0000","last_undegraded":"2024-09-10T18:34:05.174966+0000","last_fullsized":"2024-09-10T18:34:05.174966+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-09-10T18:34:00.542737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:00.542737+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:11.466009+0000","scrub_duration":0.0030351190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":256,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:09.980826+0000","last_change":"2024-09-10T18:34:09.980779+0000","last_active":"2024-09-10T18:34:09.980826+0000","last_peered":"2024-09-10T18:34:09.980826+0000","last_clean":"2024-09-10T18:34:09.980826+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:34:09.980826+0000","last_undegraded":"2024-09-10T18:34:09.980826+0000","last_fullsized":"2024-09-10T18:34:09.980826+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:34:09.980722+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:34:09.980722+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:29.815756+0000","scrub_duration":0.001388053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":288,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.168313+0000","last_change":"2024-09-10T18:32:50.211889+0000","last_active":"2024-09-10T18:34:05.168313+0000","last_peered":"2024-09-10T18:34:05.168313+0000","last_clean":"2024-09-10T18:34:05.168313+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:34:05.168313+0000","last_undegraded":"2024-09-10T18:34:05.168313+0000","last_fullsized":"2024-09-10T18:34:05.168313+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:50.211414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:50.211414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007128512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":306,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.158556+0000","last_change":"2024-09-10T18:32:39.620645+0000","last_active":"2024-09-10T18:34:05.158556+0000","last_peered":"2024-09-10T18:34:05.158556+0000","last_clean":"2024-09-10T18:34:05.158556+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:34:05.158556+0000","last_undegraded":"2024-09-10T18:34:05.158556+0000","last_fullsized":"2024-09-10T18:34:05.158556+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:39.620456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.620456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017312912,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"283'11098","reported_seq":11400,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:15.692014+0000","last_change":"2024-09-10T18:34:12.213950+0000","last_active":"2024-09-10T18:34:15.692014+0000","last_peered":"2024-09-10T18:34:15.692014+0000","last_clean":"2024-09-10T18:34:15.692014+0000","last_became_active":"2024-09-10T18:32:03.554475+0000","last_became_peered":"2024-09-10T18:32:03.554475+0000","last_unstale":"2024-09-10T18:34:15.692014+0000","last_undegraded":"2024-09-10T18:34:15.692014+0000","last_fullsized":"2024-09-10T18:34:15.692014+0000","mapping_epoch":246,"log_start":"267'6942","ondisk_log_start":"267'6942","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'10828","last_scrub_stamp":"2024-09-10T18:34:12.213852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.213852+0000","objects_scrubbed":6169,"log_size":4156,"log_dups_size":0,"ondisk_log_size":4156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:28.363825+0000","scrub_duration":3.0156890789999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381157376,"num_objects":5816,"num_object_clones":0,"num_object_copies":17448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11098,"num_write_kb":541248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":340,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:06.053711+0000","last_change":"2024-09-10T18:34:06.053598+0000","last_active":"2024-09-10T18:34:06.053711+0000","last_peered":"2024-09-10T18:34:06.053711+0000","last_clean":"2024-09-10T18:34:06.053711+0000","last_became_active":"2024-09-10T18:26:18.092014+0000","last_became_peered":"2024-09-10T18:26:18.092014+0000","last_unstale":"2024-09-10T18:34:06.053711+0000","last_undegraded":"2024-09-10T18:34:06.053711+0000","last_fullsized":"2024-09-10T18:34:06.053711+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:06.053455+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.053455+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:30.452954+0000","scrub_duration":0.007740935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"240'25","reported_seq":50,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164268+0000","last_change":"2024-09-10T18:32:51.151265+0000","last_active":"2024-09-10T18:34:05.164268+0000","last_peered":"2024-09-10T18:34:05.164268+0000","last_clean":"2024-09-10T18:34:05.164268+0000","last_became_active":"2024-09-10T18:32:51.118251+0000","last_became_peered":"2024-09-10T18:32:51.118251+0000","last_unstale":"2024-09-10T18:34:05.164268+0000","last_undegraded":"2024-09-10T18:34:05.164268+0000","last_fullsized":"2024-09-10T18:34:05.164268+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:32:39.544846+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.544846+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033922850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":284,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.174739+0000","last_change":"2024-09-10T18:33:13.324775+0000","last_active":"2024-09-10T18:34:05.174739+0000","last_peered":"2024-09-10T18:34:05.174739+0000","last_clean":"2024-09-10T18:34:05.174739+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:34:05.174739+0000","last_undegraded":"2024-09-10T18:34:05.174739+0000","last_fullsized":"2024-09-10T18:34:05.174739+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:13.324556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:13.324556+0000","last_clean_scrub_stamp":"2024-09-10T18:33:13.324556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:13.880002+0000","scrub_duration":0.011863357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"189'9","reported_seq":470,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:10.438280+0000","last_change":"2024-09-10T18:32:42.667266+0000","last_active":"2024-09-10T18:34:10.438280+0000","last_peered":"2024-09-10T18:34:10.438280+0000","last_clean":"2024-09-10T18:34:10.438280+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:34:10.438280+0000","last_undegraded":"2024-09-10T18:34:10.438280+0000","last_fullsized":"2024-09-10T18:34:10.438280+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'9","last_scrub_stamp":"2024-09-10T18:32:42.667117+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:32:42.667117+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071583269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":279,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.175102+0000","last_change":"2024-09-10T18:32:46.630172+0000","last_active":"2024-09-10T18:34:05.175102+0000","last_peered":"2024-09-10T18:34:05.175102+0000","last_clean":"2024-09-10T18:34:05.175102+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:34:05.175102+0000","last_undegraded":"2024-09-10T18:34:05.175102+0000","last_fullsized":"2024-09-10T18:34:05.175102+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:46.630078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_clean_scrub_stamp":"2024-09-10T18:32:46.630078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024328319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"76'81","reported_seq":12,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.174683+0000","last_change":"2024-09-10T18:34:02.801323+0000","last_active":"2024-09-10T18:34:05.174683+0000","last_peered":"2024-09-10T18:34:05.174683+0000","last_clean":"2024-09-10T18:34:05.174683+0000","last_became_active":"2024-09-10T18:34:02.800956+0000","last_became_peered":"2024-09-10T18:34:02.800956+0000","last_unstale":"2024-09-10T18:34:05.174683+0000","last_undegraded":"2024-09-10T18:34:05.174683+0000","last_fullsized":"2024-09-10T18:34:05.174683+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:48.006695+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:48.006695+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"283'11735","reported_seq":11786,"reported_epoch":283,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-10T18:34:14.994433+0000","last_change":"2024-09-10T18:34:12.222670+0000","last_active":"2024-09-10T18:34:14.994433+0000","last_peered":"2024-09-10T18:34:14.994433+0000","last_clean":"2024-09-10T18:34:14.994433+0000","last_became_active":"2024-09-10T18:32:03.384417+0000","last_became_peered":"2024-09-10T18:32:03.384417+0000","last_unstale":"2024-09-10T18:34:14.994433+0000","last_undegraded":"2024-09-10T18:34:14.994433+0000","last_fullsized":"2024-09-10T18:34:14.994433+0000","mapping_epoch":246,"log_start":"267'7146","ondisk_log_start":"267'7146","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":611,"log_size":4589,"log_dups_size":0,"ondisk_log_size":4589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388235264,"num_objects":5924,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11124,"num_write_kb":545536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"240'15","reported_seq":306,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.166260+0000","last_change":"2024-09-10T18:33:16.286611+0000","last_active":"2024-09-10T18:34:05.166260+0000","last_peered":"2024-09-10T18:34:05.166260+0000","last_clean":"2024-09-10T18:34:05.166260+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:34:05.166260+0000","last_undegraded":"2024-09-10T18:34:05.166260+0000","last_fullsized":"2024-09-10T18:34:05.166260+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:33:16.286523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:16.286523+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:29.107562+0000","scrub_duration":0.0069638809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"265'2","reported_seq":310,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.157933+0000","last_change":"2024-09-10T18:33:11.047358+0000","last_active":"2024-09-10T18:34:05.157933+0000","last_peered":"2024-09-10T18:34:05.157933+0000","last_clean":"2024-09-10T18:34:05.157933+0000","last_became_active":"2024-09-10T18:26:27.644997+0000","last_became_peered":"2024-09-10T18:26:27.644997+0000","last_unstale":"2024-09-10T18:34:05.157933+0000","last_undegraded":"2024-09-10T18:34:05.157933+0000","last_fullsized":"2024-09-10T18:34:05.157933+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'2","last_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_deep_scrub":"265'2","last_deep_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.047305+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:22.716371+0000","scrub_duration":0.021095425000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":304,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.157688+0000","last_change":"2024-09-10T18:32:52.590088+0000","last_active":"2024-09-10T18:34:05.157688+0000","last_peered":"2024-09-10T18:34:05.157688+0000","last_clean":"2024-09-10T18:34:05.157688+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:34:05.157688+0000","last_undegraded":"2024-09-10T18:34:05.157688+0000","last_fullsized":"2024-09-10T18:34:05.157688+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:52.589997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:52.589997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:18.281059+0000","scrub_duration":0.001802543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":277,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.165238+0000","last_change":"2024-09-10T18:32:44.239399+0000","last_active":"2024-09-10T18:34:05.165238+0000","last_peered":"2024-09-10T18:34:05.165238+0000","last_clean":"2024-09-10T18:34:05.165238+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:34:05.165238+0000","last_undegraded":"2024-09-10T18:34:05.165238+0000","last_fullsized":"2024-09-10T18:34:05.165238+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:44.239347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:44.239347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:11.230674+0000","scrub_duration":0.0011983390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'81","reported_seq":267,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.167875+0000","last_change":"2024-09-10T18:34:02.804208+0000","last_active":"2024-09-10T18:34:05.167875+0000","last_peered":"2024-09-10T18:34:05.167875+0000","last_clean":"2024-09-10T18:34:05.167875+0000","last_became_active":"2024-09-10T18:34:02.803861+0000","last_became_peered":"2024-09-10T18:34:02.803861+0000","last_unstale":"2024-09-10T18:34:05.167875+0000","last_undegraded":"2024-09-10T18:34:05.167875+0000","last_fullsized":"2024-09-10T18:34:05.167875+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.151386+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.151386+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031748879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"283'10897","reported_seq":10942,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:14.946315+0000","last_change":"2024-09-10T18:32:03.382716+0000","last_active":"2024-09-10T18:34:14.946315+0000","last_peered":"2024-09-10T18:34:14.946315+0000","last_clean":"2024-09-10T18:34:14.946315+0000","last_became_active":"2024-09-10T18:32:03.382430+0000","last_became_peered":"2024-09-10T18:32:03.382430+0000","last_unstale":"2024-09-10T18:34:14.946315+0000","last_undegraded":"2024-09-10T18:34:14.946315+0000","last_fullsized":"2024-09-10T18:34:14.946315+0000","mapping_epoch":246,"log_start":"265'6331","ondisk_log_start":"265'6331","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":4566,"log_dups_size":0,"ondisk_log_size":4566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383582208,"num_objects":5853,"num_object_clones":0,"num_object_copies":17559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10897,"num_write_kb":536000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"240'44","reported_seq":373,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.166417+0000","last_change":"2024-09-10T18:34:03.021478+0000","last_active":"2024-09-10T18:34:05.166417+0000","last_peered":"2024-09-10T18:34:05.166417+0000","last_clean":"2024-09-10T18:34:05.166417+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:34:05.166417+0000","last_undegraded":"2024-09-10T18:34:05.166417+0000","last_fullsized":"2024-09-10T18:34:05.166417+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'44","last_scrub_stamp":"2024-09-10T18:34:03.021369+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:34:03.021369+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:14.897595+0000","scrub_duration":0.0045216129999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":309,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:11.965375+0000","last_change":"2024-09-10T18:34:11.965336+0000","last_active":"2024-09-10T18:34:11.965375+0000","last_peered":"2024-09-10T18:34:11.965375+0000","last_clean":"2024-09-10T18:34:11.965375+0000","last_became_active":"2024-09-10T18:26:27.638408+0000","last_became_peered":"2024-09-10T18:26:27.638408+0000","last_unstale":"2024-09-10T18:34:11.965375+0000","last_undegraded":"2024-09-10T18:34:11.965375+0000","last_fullsized":"2024-09-10T18:34:11.965375+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:11.965263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:11.965263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:21.204938+0000","scrub_duration":0.0034110310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":308,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.157570+0000","last_change":"2024-09-10T18:32:48.582057+0000","last_active":"2024-09-10T18:34:05.157570+0000","last_peered":"2024-09-10T18:34:05.157570+0000","last_clean":"2024-09-10T18:34:05.157570+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:34:05.157570+0000","last_undegraded":"2024-09-10T18:34:05.157570+0000","last_fullsized":"2024-09-10T18:34:05.157570+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:48.581963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:48.581963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011575692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":289,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164020+0000","last_change":"2024-09-10T18:31:58.460588+0000","last_active":"2024-09-10T18:34:05.164020+0000","last_peered":"2024-09-10T18:34:05.164020+0000","last_clean":"2024-09-10T18:34:05.164020+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:34:05.164020+0000","last_undegraded":"2024-09-10T18:34:05.164020+0000","last_fullsized":"2024-09-10T18:34:05.164020+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:58.460533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:58.460533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001375914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"76'81","reported_seq":214,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.173381+0000","last_change":"2024-09-10T18:34:02.827839+0000","last_active":"2024-09-10T18:34:05.173381+0000","last_peered":"2024-09-10T18:34:05.173381+0000","last_clean":"2024-09-10T18:34:05.173381+0000","last_became_active":"2024-09-10T18:34:02.806416+0000","last_became_peered":"2024-09-10T18:34:02.806416+0000","last_unstale":"2024-09-10T18:34:05.173381+0000","last_undegraded":"2024-09-10T18:34:05.173381+0000","last_fullsized":"2024-09-10T18:34:05.173381+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.370561+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.370561+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029251390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"283'10974","reported_seq":11305,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:15.364907+0000","last_change":"2024-09-10T18:34:01.964247+0000","last_active":"2024-09-10T18:34:15.364907+0000","last_peered":"2024-09-10T18:34:15.364907+0000","last_clean":"2024-09-10T18:34:15.364907+0000","last_became_active":"2024-09-10T18:32:03.554423+0000","last_became_peered":"2024-09-10T18:32:03.554423+0000","last_unstale":"2024-09-10T18:34:15.364907+0000","last_undegraded":"2024-09-10T18:34:15.364907+0000","last_fullsized":"2024-09-10T18:34:15.364907+0000","mapping_epoch":246,"log_start":"265'6469","ondisk_log_start":"265'6469","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'10000","last_scrub_stamp":"2024-09-10T18:34:01.964158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:01.964158+0000","objects_scrubbed":6911,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:10.075365+0000","scrub_duration":3.4154075100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380502016,"num_objects":5806,"num_object_clones":0,"num_object_copies":17418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10974,"num_write_kb":536960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"240'18","reported_seq":325,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.173606+0000","last_change":"2024-09-10T18:32:54.528296+0000","last_active":"2024-09-10T18:34:05.173606+0000","last_peered":"2024-09-10T18:34:05.173606+0000","last_clean":"2024-09-10T18:34:05.173606+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:34:05.173606+0000","last_undegraded":"2024-09-10T18:34:05.173606+0000","last_fullsized":"2024-09-10T18:34:05.173606+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-09-10T18:32:54.528209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:54.528209+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:17.286857+0000","scrub_duration":0.0056889829999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":302,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164422+0000","last_change":"2024-09-10T18:33:14.879881+0000","last_active":"2024-09-10T18:34:05.164422+0000","last_peered":"2024-09-10T18:34:05.164422+0000","last_clean":"2024-09-10T18:34:05.164422+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:34:05.164422+0000","last_undegraded":"2024-09-10T18:34:05.164422+0000","last_fullsized":"2024-09-10T18:34:05.164422+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:14.879787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:33:14.879787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:36.213826+0000","scrub_duration":0.0044267550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":308,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.157066+0000","last_change":"2024-09-10T18:32:50.570319+0000","last_active":"2024-09-10T18:34:05.157066+0000","last_peered":"2024-09-10T18:34:05.157066+0000","last_clean":"2024-09-10T18:34:05.157066+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:34:05.157066+0000","last_undegraded":"2024-09-10T18:34:05.157066+0000","last_fullsized":"2024-09-10T18:34:05.157066+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:50.570113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:50.570113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059653969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":278,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.174268+0000","last_change":"2024-09-10T18:32:58.358775+0000","last_active":"2024-09-10T18:34:05.174268+0000","last_peered":"2024-09-10T18:34:05.174268+0000","last_clean":"2024-09-10T18:34:05.174268+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:34:05.174268+0000","last_undegraded":"2024-09-10T18:34:05.174268+0000","last_fullsized":"2024-09-10T18:34:05.174268+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:58.358694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.803236+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.358694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:15.808097+0000","scrub_duration":0.011821898000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"76'81","reported_seq":12,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164559+0000","last_change":"2024-09-10T18:34:02.801016+0000","last_active":"2024-09-10T18:34:05.164559+0000","last_peered":"2024-09-10T18:34:05.164559+0000","last_clean":"2024-09-10T18:34:05.164559+0000","last_became_active":"2024-09-10T18:34:02.800107+0000","last_became_peered":"2024-09-10T18:34:02.800107+0000","last_unstale":"2024-09-10T18:34:05.164559+0000","last_undegraded":"2024-09-10T18:34:05.164559+0000","last_fullsized":"2024-09-10T18:34:05.164559+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:58.561425+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.561425+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"283'10999","reported_seq":11296,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:16.058807+0000","last_change":"2024-09-10T18:34:12.987347+0000","last_active":"2024-09-10T18:34:16.058807+0000","last_peered":"2024-09-10T18:34:16.058807+0000","last_clean":"2024-09-10T18:34:16.058807+0000","last_became_active":"2024-09-10T18:32:03.382890+0000","last_became_peered":"2024-09-10T18:32:03.382890+0000","last_unstale":"2024-09-10T18:34:16.058807+0000","last_undegraded":"2024-09-10T18:34:16.058807+0000","last_fullsized":"2024-09-10T18:34:16.058807+0000","mapping_epoch":246,"log_start":"263'5556","ondisk_log_start":"263'5556","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'10774","last_scrub_stamp":"2024-09-10T18:34:12.987248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.987248+0000","objects_scrubbed":6064,"log_size":5443,"log_dups_size":0,"ondisk_log_size":5443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:26.071275+0000","scrub_duration":2.4958883709999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378208256,"num_objects":5771,"num_object_clones":0,"num_object_copies":17313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10999,"num_write_kb":536640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":341,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:06.163167+0000","last_change":"2024-09-10T18:34:06.163124+0000","last_active":"2024-09-10T18:34:06.163167+0000","last_peered":"2024-09-10T18:34:06.163167+0000","last_clean":"2024-09-10T18:34:06.163167+0000","last_became_active":"2024-09-10T18:26:27.645932+0000","last_became_peered":"2024-09-10T18:26:27.645932+0000","last_unstale":"2024-09-10T18:34:06.163167+0000","last_undegraded":"2024-09-10T18:34:06.163167+0000","last_fullsized":"2024-09-10T18:34:06.163167+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:34:06.163072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.163072+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:32.837570+0000","scrub_duration":0.0016579489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":302,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164495+0000","last_change":"2024-09-10T18:33:05.784757+0000","last_active":"2024-09-10T18:34:05.164495+0000","last_peered":"2024-09-10T18:34:05.164495+0000","last_clean":"2024-09-10T18:34:05.164495+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:34:05.164495+0000","last_undegraded":"2024-09-10T18:34:05.164495+0000","last_fullsized":"2024-09-10T18:34:05.164495+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:33:05.784670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:33:05.784670+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:24.433627+0000","scrub_duration":0.002289031,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":459,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:10.427895+0000","last_change":"2024-09-10T18:33:15.972246+0000","last_active":"2024-09-10T18:34:10.427895+0000","last_peered":"2024-09-10T18:34:10.427895+0000","last_clean":"2024-09-10T18:34:10.427895+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:34:10.427895+0000","last_undegraded":"2024-09-10T18:34:10.427895+0000","last_fullsized":"2024-09-10T18:34:10.427895+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:33:15.972166+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.972166+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:25.882626+0000","scrub_duration":0.004394907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":284,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.159023+0000","last_change":"2024-09-10T18:33:05.964339+0000","last_active":"2024-09-10T18:34:05.159023+0000","last_peered":"2024-09-10T18:34:05.159023+0000","last_clean":"2024-09-10T18:34:05.159023+0000","last_became_active":"2024-09-10T18:26:27.647511+0000","last_became_peered":"2024-09-10T18:26:27.647511+0000","last_unstale":"2024-09-10T18:34:05.159023+0000","last_undegraded":"2024-09-10T18:34:05.159023+0000","last_fullsized":"2024-09-10T18:34:05.159023+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:05.964034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:05.964034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:30.675049+0000","scrub_duration":0.0015507839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'81","reported_seq":12,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.159188+0000","last_change":"2024-09-10T18:34:02.810441+0000","last_active":"2024-09-10T18:34:05.159188+0000","last_peered":"2024-09-10T18:34:05.159188+0000","last_clean":"2024-09-10T18:34:05.159188+0000","last_became_active":"2024-09-10T18:34:02.810126+0000","last_became_peered":"2024-09-10T18:34:02.810126+0000","last_unstale":"2024-09-10T18:34:05.159188+0000","last_undegraded":"2024-09-10T18:34:05.159188+0000","last_fullsized":"2024-09-10T18:34:05.159188+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:57.474295+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.474295+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":10,"num_write":6,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":237,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.166414+0000","last_change":"2024-09-10T18:32:47.037801+0000","last_active":"2024-09-10T18:34:05.166414+0000","last_peered":"2024-09-10T18:34:05.166414+0000","last_clean":"2024-09-10T18:34:05.166414+0000","last_became_active":"2024-09-10T18:26:27.640922+0000","last_became_peered":"2024-09-10T18:26:27.640922+0000","last_unstale":"2024-09-10T18:34:05.166414+0000","last_undegraded":"2024-09-10T18:34:05.166414+0000","last_fullsized":"2024-09-10T18:34:05.166414+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:47.037651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:32:47.037651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021178149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"240'12","reported_seq":300,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.179779+0000","last_change":"2024-09-10T18:32:56.384627+0000","last_active":"2024-09-10T18:34:05.179779+0000","last_peered":"2024-09-10T18:34:05.179779+0000","last_clean":"2024-09-10T18:34:05.179779+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:34:05.179779+0000","last_undegraded":"2024-09-10T18:34:05.179779+0000","last_fullsized":"2024-09-10T18:34:05.179779+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-10T18:32:56.384439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:56.384439+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010401713999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":288,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.160181+0000","last_change":"2024-09-10T18:34:00.199169+0000","last_active":"2024-09-10T18:34:05.160181+0000","last_peered":"2024-09-10T18:34:05.160181+0000","last_clean":"2024-09-10T18:34:05.160181+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:34:05.160181+0000","last_undegraded":"2024-09-10T18:34:05.160181+0000","last_fullsized":"2024-09-10T18:34:05.160181+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:00.199087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:00.199087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:11.766473+0000","scrub_duration":0.016944524999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":293,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164662+0000","last_change":"2024-09-10T18:33:58.161088+0000","last_active":"2024-09-10T18:34:05.164662+0000","last_peered":"2024-09-10T18:34:05.164662+0000","last_clean":"2024-09-10T18:34:05.164662+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:34:05.164662+0000","last_undegraded":"2024-09-10T18:34:05.164662+0000","last_fullsized":"2024-09-10T18:34:05.164662+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:58.161019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:58.161019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:23.203002+0000","scrub_duration":0.0042556759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'81","reported_seq":218,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.166057+0000","last_change":"2024-09-10T18:34:02.816592+0000","last_active":"2024-09-10T18:34:05.166057+0000","last_peered":"2024-09-10T18:34:05.166057+0000","last_clean":"2024-09-10T18:34:05.166057+0000","last_became_active":"2024-09-10T18:34:02.815880+0000","last_became_peered":"2024-09-10T18:34:02.815880+0000","last_unstale":"2024-09-10T18:34:05.166057+0000","last_undegraded":"2024-09-10T18:34:05.166057+0000","last_fullsized":"2024-09-10T18:34:05.166057+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:48.466654+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.466654+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"240'19","reported_seq":313,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.165405+0000","last_change":"2024-09-10T18:33:08.284482+0000","last_active":"2024-09-10T18:34:05.165405+0000","last_peered":"2024-09-10T18:34:05.165405+0000","last_clean":"2024-09-10T18:34:05.165405+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:34:05.165405+0000","last_undegraded":"2024-09-10T18:34:05.165405+0000","last_fullsized":"2024-09-10T18:34:05.165405+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:33:08.284394+0000","last_deep_scrub":"189'13","last_deep_scrub_stamp":"2024-09-10T18:28:08.976162+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.284394+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:31.505393+0000","scrub_duration":0.010009881999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":299,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164678+0000","last_change":"2024-09-10T18:33:16.937511+0000","last_active":"2024-09-10T18:34:05.164678+0000","last_peered":"2024-09-10T18:34:05.164678+0000","last_clean":"2024-09-10T18:34:05.164678+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:34:05.164678+0000","last_undegraded":"2024-09-10T18:34:05.164678+0000","last_fullsized":"2024-09-10T18:34:05.164678+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:16.937390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:33:16.937390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:26.843652+0000","scrub_duration":0.024770533000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":280,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.179548+0000","last_change":"2024-09-10T18:32:39.161911+0000","last_active":"2024-09-10T18:34:05.179548+0000","last_peered":"2024-09-10T18:34:05.179548+0000","last_clean":"2024-09-10T18:34:05.179548+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:34:05.179548+0000","last_undegraded":"2024-09-10T18:34:05.179548+0000","last_fullsized":"2024-09-10T18:34:05.179548+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:39.161653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.161653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0093847749999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":296,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.157886+0000","last_change":"2024-09-10T18:32:43.621566+0000","last_active":"2024-09-10T18:34:05.157886+0000","last_peered":"2024-09-10T18:34:05.157886+0000","last_clean":"2024-09-10T18:34:05.157886+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:34:05.157886+0000","last_undegraded":"2024-09-10T18:34:05.157886+0000","last_fullsized":"2024-09-10T18:34:05.157886+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:43.621478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:43.621478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031857410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'81","reported_seq":12,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.159400+0000","last_change":"2024-09-10T18:34:02.797580+0000","last_active":"2024-09-10T18:34:05.159400+0000","last_peered":"2024-09-10T18:34:05.159400+0000","last_clean":"2024-09-10T18:34:05.159400+0000","last_became_active":"2024-09-10T18:34:02.797304+0000","last_became_peered":"2024-09-10T18:34:02.797304+0000","last_unstale":"2024-09-10T18:34:05.159400+0000","last_undegraded":"2024-09-10T18:34:05.159400+0000","last_fullsized":"2024-09-10T18:34:05.159400+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:58.780613+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.780613+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"240'19","reported_seq":320,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.159248+0000","last_change":"2024-09-10T18:33:15.955719+0000","last_active":"2024-09-10T18:34:05.159248+0000","last_peered":"2024-09-10T18:34:05.159248+0000","last_clean":"2024-09-10T18:34:05.159248+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:34:05.159248+0000","last_undegraded":"2024-09-10T18:34:05.159248+0000","last_fullsized":"2024-09-10T18:34:05.159248+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:33:15.955638+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.955638+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:44.846427+0000","scrub_duration":0.021358801,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":293,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164745+0000","last_change":"2024-09-10T18:33:12.342537+0000","last_active":"2024-09-10T18:34:05.164745+0000","last_peered":"2024-09-10T18:34:05.164745+0000","last_clean":"2024-09-10T18:34:05.164745+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:34:05.164745+0000","last_undegraded":"2024-09-10T18:34:05.164745+0000","last_fullsized":"2024-09-10T18:34:05.164745+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:12.342432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_clean_scrub_stamp":"2024-09-10T18:33:12.342432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:21.132397+0000","scrub_duration":0.0043002429999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":441,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:10.439372+0000","last_change":"2024-09-10T18:32:40.183655+0000","last_active":"2024-09-10T18:34:10.439372+0000","last_peered":"2024-09-10T18:34:10.439372+0000","last_clean":"2024-09-10T18:34:10.439372+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:34:10.439372+0000","last_undegraded":"2024-09-10T18:34:10.439372+0000","last_fullsized":"2024-09-10T18:34:10.439372+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:32:40.183571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:40.183571+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014890771000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":23,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.179923+0000","last_change":"2024-09-10T18:33:29.142401+0000","last_active":"2024-09-10T18:34:05.179923+0000","last_peered":"2024-09-10T18:34:05.179923+0000","last_clean":"2024-09-10T18:34:05.179923+0000","last_became_active":"2024-09-10T18:33:29.141936+0000","last_became_peered":"2024-09-10T18:33:29.141936+0000","last_unstale":"2024-09-10T18:34:05.179923+0000","last_undegraded":"2024-09-10T18:34:05.179923+0000","last_fullsized":"2024-09-10T18:34:05.179923+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:11.727786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.727786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:31.539221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"76'81","reported_seq":73,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.165016+0000","last_change":"2024-09-10T18:34:02.810916+0000","last_active":"2024-09-10T18:34:05.165016+0000","last_peered":"2024-09-10T18:34:05.165016+0000","last_clean":"2024-09-10T18:34:05.165016+0000","last_became_active":"2024-09-10T18:34:02.810381+0000","last_became_peered":"2024-09-10T18:34:02.810381+0000","last_unstale":"2024-09-10T18:34:05.165016+0000","last_undegraded":"2024-09-10T18:34:05.165016+0000","last_fullsized":"2024-09-10T18:34:05.165016+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:56.629683+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:56.629683+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":10,"num_write":6,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":322,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.173616+0000","last_change":"2024-09-10T18:33:59.230861+0000","last_active":"2024-09-10T18:34:05.173616+0000","last_peered":"2024-09-10T18:34:05.173616+0000","last_clean":"2024-09-10T18:34:05.173616+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:34:05.173616+0000","last_undegraded":"2024-09-10T18:34:05.173616+0000","last_fullsized":"2024-09-10T18:34:05.173616+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-10T18:33:59.227198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:59.227198+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:04.026375+0000","scrub_duration":0.012577204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":292,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164679+0000","last_change":"2024-09-10T18:32:39.535817+0000","last_active":"2024-09-10T18:34:05.164679+0000","last_peered":"2024-09-10T18:34:05.164679+0000","last_clean":"2024-09-10T18:34:05.164679+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:34:05.164679+0000","last_undegraded":"2024-09-10T18:34:05.164679+0000","last_fullsized":"2024-09-10T18:34:05.164679+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:39.535749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.535749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046018179999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":450,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:10.434654+0000","last_change":"2024-09-10T18:32:43.942068+0000","last_active":"2024-09-10T18:34:10.434654+0000","last_peered":"2024-09-10T18:34:10.434654+0000","last_clean":"2024-09-10T18:34:10.434654+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:34:10.434654+0000","last_undegraded":"2024-09-10T18:34:10.434654+0000","last_fullsized":"2024-09-10T18:34:10.434654+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:32:43.941975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:43.941975+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046448039999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":277,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.165020+0000","last_change":"2024-09-10T18:32:42.165903+0000","last_active":"2024-09-10T18:34:05.165020+0000","last_peered":"2024-09-10T18:34:05.165020+0000","last_clean":"2024-09-10T18:34:05.165020+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:34:05.165020+0000","last_undegraded":"2024-09-10T18:34:05.165020+0000","last_fullsized":"2024-09-10T18:34:05.165020+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:42.165812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:42.165812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010729512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":328,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.174147+0000","last_change":"2024-09-10T18:33:15.612721+0000","last_active":"2024-09-10T18:34:05.174147+0000","last_peered":"2024-09-10T18:34:05.174147+0000","last_clean":"2024-09-10T18:34:05.174147+0000","last_became_active":"2024-09-10T18:26:27.649202+0000","last_became_peered":"2024-09-10T18:26:27.649202+0000","last_unstale":"2024-09-10T18:34:05.174147+0000","last_undegraded":"2024-09-10T18:34:05.174147+0000","last_fullsized":"2024-09-10T18:34:05.174147+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:33:15.612638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.612638+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:16.544192+0000","scrub_duration":0.002578828,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":302,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.174792+0000","last_change":"2024-09-10T18:34:04.380240+0000","last_active":"2024-09-10T18:34:05.174792+0000","last_peered":"2024-09-10T18:34:05.174792+0000","last_clean":"2024-09-10T18:34:05.174792+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:34:05.174792+0000","last_undegraded":"2024-09-10T18:34:05.174792+0000","last_fullsized":"2024-09-10T18:34:05.174792+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:04.380160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:34:04.380160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:34.095135+0000","scrub_duration":0.0018137730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":301,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.166342+0000","last_change":"2024-09-10T18:34:03.521909+0000","last_active":"2024-09-10T18:34:05.166342+0000","last_peered":"2024-09-10T18:34:05.166342+0000","last_clean":"2024-09-10T18:34:05.166342+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:34:05.166342+0000","last_undegraded":"2024-09-10T18:34:05.166342+0000","last_fullsized":"2024-09-10T18:34:05.166342+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:03.521806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_clean_scrub_stamp":"2024-09-10T18:34:03.521806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:19.398042+0000","scrub_duration":0.005196946,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":278,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-10T18:34:14.263657+0000","last_change":"2024-09-10T18:34:14.263579+0000","last_active":"2024-09-10T18:34:14.263657+0000","last_peered":"2024-09-10T18:34:14.263657+0000","last_clean":"2024-09-10T18:34:14.263657+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:34:14.263657+0000","last_undegraded":"2024-09-10T18:34:14.263657+0000","last_fullsized":"2024-09-10T18:34:14.263657+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:14.263499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:34:14.263499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:30.466656+0000","scrub_duration":0.0019685509999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"240'15","reported_seq":322,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.166565+0000","last_change":"2024-09-10T18:32:41.169276+0000","last_active":"2024-09-10T18:34:05.166565+0000","last_peered":"2024-09-10T18:34:05.166565+0000","last_clean":"2024-09-10T18:34:05.166565+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:34:05.166565+0000","last_undegraded":"2024-09-10T18:34:05.166565+0000","last_fullsized":"2024-09-10T18:34:05.166565+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:32:41.168785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:41.168785+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037973939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":295,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164995+0000","last_change":"2024-09-10T18:33:01.781370+0000","last_active":"2024-09-10T18:34:05.164995+0000","last_peered":"2024-09-10T18:34:05.164995+0000","last_clean":"2024-09-10T18:34:05.164995+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:34:05.164995+0000","last_undegraded":"2024-09-10T18:34:05.164995+0000","last_fullsized":"2024-09-10T18:34:05.164995+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:01.781295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.781295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:27.429985+0000","scrub_duration":0.0016784879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":273,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:06.262732+0000","last_change":"2024-09-10T18:34:06.262708+0000","last_active":"2024-09-10T18:34:06.262732+0000","last_peered":"2024-09-10T18:34:06.262732+0000","last_clean":"2024-09-10T18:34:06.262732+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:34:06.262732+0000","last_undegraded":"2024-09-10T18:34:06.262732+0000","last_fullsized":"2024-09-10T18:34:06.262732+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:44.838141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:31:44.838141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011071239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":287,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.166826+0000","last_change":"2024-09-10T18:34:00.512903+0000","last_active":"2024-09-10T18:34:05.166826+0000","last_peered":"2024-09-10T18:34:05.166826+0000","last_clean":"2024-09-10T18:34:05.166826+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:34:05.166826+0000","last_undegraded":"2024-09-10T18:34:05.166826+0000","last_fullsized":"2024-09-10T18:34:05.166826+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:00.512670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:00.512670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:03.765496+0000","scrub_duration":0.0068783309999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'81","reported_seq":12,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.157738+0000","last_change":"2024-09-10T18:34:02.802928+0000","last_active":"2024-09-10T18:34:05.157738+0000","last_peered":"2024-09-10T18:34:05.157738+0000","last_clean":"2024-09-10T18:34:05.157738+0000","last_became_active":"2024-09-10T18:34:02.802580+0000","last_became_peered":"2024-09-10T18:34:02.802580+0000","last_unstale":"2024-09-10T18:34:05.157738+0000","last_undegraded":"2024-09-10T18:34:05.157738+0000","last_fullsized":"2024-09-10T18:34:05.157738+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.370561+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.370561+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029251390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":320,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.169403+0000","last_change":"2024-09-10T18:32:46.253109+0000","last_active":"2024-09-10T18:34:05.169403+0000","last_peered":"2024-09-10T18:34:05.169403+0000","last_clean":"2024-09-10T18:34:05.169403+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:34:05.169403+0000","last_undegraded":"2024-09-10T18:34:05.169403+0000","last_fullsized":"2024-09-10T18:34:05.169403+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:32:46.252895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:46.252895+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003188967,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":303,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:08.134496+0000","last_change":"2024-09-10T18:34:08.134374+0000","last_active":"2024-09-10T18:34:08.134496+0000","last_peered":"2024-09-10T18:34:08.134496+0000","last_clean":"2024-09-10T18:34:08.134496+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:34:08.134496+0000","last_undegraded":"2024-09-10T18:34:08.134496+0000","last_fullsized":"2024-09-10T18:34:08.134496+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:08.134230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:34:08.134230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:31.369105+0000","scrub_duration":0.0077618749999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":288,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.163673+0000","last_change":"2024-09-10T18:31:50.423429+0000","last_active":"2024-09-10T18:34:05.163673+0000","last_peered":"2024-09-10T18:34:05.163673+0000","last_clean":"2024-09-10T18:34:05.163673+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:34:05.163673+0000","last_undegraded":"2024-09-10T18:34:05.163673+0000","last_fullsized":"2024-09-10T18:34:05.163673+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:50.423374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:50.423374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00116716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":295,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.174674+0000","last_change":"2024-09-10T18:33:18.227516+0000","last_active":"2024-09-10T18:34:05.174674+0000","last_peered":"2024-09-10T18:34:05.174674+0000","last_clean":"2024-09-10T18:34:05.174674+0000","last_became_active":"2024-09-10T18:26:27.636996+0000","last_became_peered":"2024-09-10T18:26:27.636996+0000","last_unstale":"2024-09-10T18:34:05.174674+0000","last_undegraded":"2024-09-10T18:34:05.174674+0000","last_fullsized":"2024-09-10T18:34:05.174674+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:18.227428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:18.227428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:27.758937+0000","scrub_duration":0.0063780959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":301,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.170766+0000","last_change":"2024-09-10T18:32:47.287083+0000","last_active":"2024-09-10T18:34:05.170766+0000","last_peered":"2024-09-10T18:34:05.170766+0000","last_clean":"2024-09-10T18:34:05.170766+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:34:05.170766+0000","last_undegraded":"2024-09-10T18:34:05.170766+0000","last_fullsized":"2024-09-10T18:34:05.170766+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-09-10T18:32:47.286977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:47.286977+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:11.371082+0000","scrub_duration":0.017807315000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":286,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.165786+0000","last_change":"2024-09-10T18:32:57.214331+0000","last_active":"2024-09-10T18:34:05.165786+0000","last_peered":"2024-09-10T18:34:05.165786+0000","last_clean":"2024-09-10T18:34:05.165786+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:34:05.165786+0000","last_undegraded":"2024-09-10T18:34:05.165786+0000","last_fullsized":"2024-09-10T18:34:05.165786+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:57.214242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:32:57.214242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:12.819926+0000","scrub_duration":0.015466415000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":241,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.166170+0000","last_change":"2024-09-10T18:33:00.161788+0000","last_active":"2024-09-10T18:34:05.166170+0000","last_peered":"2024-09-10T18:34:05.166170+0000","last_clean":"2024-09-10T18:34:05.166170+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:34:05.166170+0000","last_undegraded":"2024-09-10T18:34:05.166170+0000","last_fullsized":"2024-09-10T18:34:05.166170+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:00.161361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:33:00.161361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035012799999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":286,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.165077+0000","last_change":"2024-09-10T18:33:13.844984+0000","last_active":"2024-09-10T18:34:05.165077+0000","last_peered":"2024-09-10T18:34:05.165077+0000","last_clean":"2024-09-10T18:34:05.165077+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:34:05.165077+0000","last_undegraded":"2024-09-10T18:34:05.165077+0000","last_fullsized":"2024-09-10T18:34:05.165077+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:13.844781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:13.844781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:15.129270+0000","scrub_duration":0.0022954300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":296,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.164381+0000","last_change":"2024-09-10T18:33:18.177364+0000","last_active":"2024-09-10T18:34:05.164381+0000","last_peered":"2024-09-10T18:34:05.164381+0000","last_clean":"2024-09-10T18:34:05.164381+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:34:05.164381+0000","last_undegraded":"2024-09-10T18:34:05.164381+0000","last_fullsized":"2024-09-10T18:34:05.164381+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_clean_scrub_stamp":"2024-09-10T18:33:18.177264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:41.303362+0000","scrub_duration":0.0029244380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":284,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.170963+0000","last_change":"2024-09-10T18:34:03.175239+0000","last_active":"2024-09-10T18:34:05.170963+0000","last_peered":"2024-09-10T18:34:05.170963+0000","last_clean":"2024-09-10T18:34:05.170963+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:34:05.170963+0000","last_undegraded":"2024-09-10T18:34:05.170963+0000","last_fullsized":"2024-09-10T18:34:05.170963+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:03.175158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:34:03.175158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:17.890582+0000","scrub_duration":0.0046326559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":302,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.167945+0000","last_change":"2024-09-10T18:33:59.017207+0000","last_active":"2024-09-10T18:34:05.167945+0000","last_peered":"2024-09-10T18:34:05.167945+0000","last_clean":"2024-09-10T18:34:05.167945+0000","last_became_active":"2024-09-10T18:26:27.648112+0000","last_became_peered":"2024-09-10T18:26:27.648112+0000","last_unstale":"2024-09-10T18:34:05.167945+0000","last_undegraded":"2024-09-10T18:34:05.167945+0000","last_fullsized":"2024-09-10T18:34:05.167945+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:33:59.017151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:59.017151+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:07.309204+0000","scrub_duration":0.0027087330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":289,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-10T18:34:05.165148+0000","last_change":"2024-09-10T18:34:02.502004+0000","last_active":"2024-09-10T18:34:05.165148+0000","last_peered":"2024-09-10T18:34:05.165148+0000","last_clean":"2024-09-10T18:34:05.165148+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:34:05.165148+0000","last_undegraded":"2024-09-10T18:34:05.165148+0000","last_fullsized":"2024-09-10T18:34:05.165148+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:02.501929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:34:02.501929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:23.332476+0000","scrub_duration":0.0086866479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6117523480,"num_objects":93347,"num_object_clones":0,"num_object_copies":280041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93347,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175831,"num_write_kb":8613633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18375389184,"data_stored":18375376968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72790,"ondisk_log_size":72790,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":731,"ondisk_log_size":731,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":756,"num_read_kb":751,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":724543,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":50,"num_write":27,"num_write_kb":370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":596030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1701,"ondisk_log_size":1701,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234140,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3693496,"kb_used_data":3387760,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":90051656,"statfs":{"total":95995035648,"available":92212895744,"internally_reserved":0,"allocated":3469066240,"data_stored":3468653682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":2.6840000000000002,"15min":1.8140000000000001},"min":{"1min":0.33200000000000002,"5min":0.21099999999999999,"15min":0.20799999999999999},"max":{"1min":6.5209999999999999,"5min":75.506,"15min":85.012},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.98899999999999999,"5min":2.472,"15min":1.728},"min":{"1min":0.35599999999999998,"5min":0.23699999999999999,"15min":0.222},"max":{"1min":2.7130000000000001,"5min":75.710999999999999,"15min":84.665999999999997},"last":2.2480000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.8600000000000001,"15min":1.23},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.214},"max":{"1min":2.4510000000000001,"5min":49.317999999999998,"15min":49.317999999999998},"last":2.6000000000000001},{"interface":"front","average":{"1min":0.83199999999999996,"5min":1.6439999999999999,"15min":1.167},"min":{"1min":0.28599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.2309999999999999,"5min":49.353000000000002,"15min":49.353000000000002},"last":2.3140000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":0.93500000000000005,"15min":0.79900000000000004},"min":{"1min":0.27400000000000002,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":3.254,"5min":5.008,"15min":5.3150000000000004},"last":0.374},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.0660000000000001,"15min":0.871},"min":{"1min":0.32300000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.5150000000000001,"5min":5.4029999999999996,"15min":9.9450000000000003},"last":2.6629999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.163,"15min":1.373},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.7380000000000004,"5min":4.7380000000000004,"15min":86.305999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.2869999999999999,"15min":1.425},"min":{"1min":0.30299999999999999,"5min":0.29699999999999999,"15min":0.26700000000000002},"max":{"1min":2.7200000000000002,"5min":7.3339999999999996,"15min":88.364000000000004},"last":2.6309999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.626,"15min":0.48699999999999999},"min":{"1min":0.26700000000000002,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":2.2610000000000001,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":2.282},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68899999999999995,"15min":0.51000000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.223},"max":{"1min":1.607,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":0.42199999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.0760000000000001,"15min":0.65400000000000003},"min":{"1min":0.30099999999999999,"5min":0.27600000000000002,"15min":0.23499999999999999},"max":{"1min":4.1479999999999997,"5min":7.3109999999999999,"15min":7.3109999999999999},"last":2.7509999999999999},{"interface":"front","average":{"1min":1.244,"5min":0.93000000000000005,"15min":0.61799999999999999},"min":{"1min":0.38900000000000001,"5min":0.27100000000000002,"15min":0.219},"max":{"1min":3.911,"5min":6.3360000000000003,"15min":6.3360000000000003},"last":2.3370000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.89200000000000002,"15min":0.59699999999999998},"min":{"1min":0.33600000000000002,"5min":0.28899999999999998,"15min":0.27000000000000002},"max":{"1min":2.488,"5min":7.2750000000000004,"15min":7.2750000000000004},"last":2.5699999999999998},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.79900000000000004,"15min":0.56200000000000006},"min":{"1min":0.378,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.3199999999999998,"5min":4.585,"15min":4.585},"last":2.2090000000000001}]}]},{"osd":6,"up_from":144,"seq":618475290719,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1442020,"kb_used_data":1131100,"kb_used_omap":7,"kb_used_meta":310904,"kb_avail":92303132,"statfs":{"total":95995035648,"available":94518407168,"internally_reserved":0,"allocated":1158246400,"data_stored":1157854470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":318365783},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":1.373,"15min":1.853},"min":{"1min":0.24399999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.831,"5min":8.3949999999999996,"15min":30.629999999999999},"last":1.3200000000000001},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.95999999999999996,"15min":1.375},"min":{"1min":0.247,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":4.3310000000000004,"5min":8.4640000000000004,"15min":30.597000000000001},"last":3.407}]},{"osd":1,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.131,"15min":0.94199999999999995},"min":{"1min":0.29099999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":3.5569999999999999,"5min":8.0329999999999995,"15min":30.984000000000002},"last":1.546},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.206,"15min":1.1399999999999999},"min":{"1min":0.30499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.1850000000000001,"5min":6.7859999999999996,"15min":30.571000000000002},"last":0.28100000000000003}]},{"osd":2,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":1.0489999999999999,"15min":1.0600000000000001},"min":{"1min":0.27200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.0760000000000001,"5min":18.353000000000002,"15min":18.353000000000002},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.82599999999999996,"5min":1.028,"15min":0.85099999999999998},"min":{"1min":0.28799999999999998,"5min":0.26600000000000001,"15min":0.23400000000000001},"max":{"1min":3.0880000000000001,"5min":11.164999999999999,"15min":11.164999999999999},"last":0.84399999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.2849999999999999,"15min":1.964},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27800000000000002},"max":{"1min":4.8890000000000002,"5min":16.831,"15min":61.703000000000003},"last":2.448},{"interface":"front","average":{"1min":2.0230000000000001,"5min":1.494,"15min":2.0259999999999998},"min":{"1min":0.30399999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":17.786000000000001,"5min":17.786000000000001,"15min":61.814},"last":1.242}]},{"osd":4,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.63500000000000001,"15min":0.54500000000000004},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.23699999999999999},"max":{"1min":2.8300000000000001,"5min":5.4950000000000001,"15min":5.4950000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.60099999999999998,"15min":0.63700000000000001},"min":{"1min":0.26900000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.2440000000000002,"5min":4.6390000000000002,"15min":4.6390000000000002},"last":0.46500000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.69099999999999995,"15min":0.54400000000000004},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.24399999999999999},"max":{"1min":2.8969999999999998,"5min":4.6619999999999999,"15min":4.6619999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.68799999999999994,"15min":0.51700000000000002},"min":{"1min":0.309,"5min":0.29699999999999999,"15min":0.26800000000000002},"max":{"1min":1.9830000000000001,"5min":2.794,"15min":2.794},"last":0.56299999999999994}]},{"osd":7,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.65200000000000002,"15min":0.54500000000000004},"min":{"1min":0.32600000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.929,"5min":2.7040000000000002,"15min":2.7040000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.755,"15min":0.57799999999999996},"min":{"1min":0.34399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.2320000000000002,"5min":5.5309999999999997,"15min":5.5309999999999997},"last":0.54700000000000004}]}]},{"osd":5,"up_from":39,"seq":167503724711,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3716224,"kb_used_data":3418872,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":90028928,"statfs":{"total":95995035648,"available":92189622272,"internally_reserved":0,"allocated":3500924928,"data_stored":3500505228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":1.367,"15min":1.4590000000000001},"min":{"1min":0.249,"5min":0.23499999999999999,"15min":0.21099999999999999},"max":{"1min":2.5259999999999998,"5min":26.288,"15min":52.033999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.218,"15min":1.29},"min":{"1min":0.246,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":3.9820000000000002,"5min":24.178000000000001,"15min":52.167999999999999},"last":0.48799999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.379,"15min":0.93799999999999994},"min":{"1min":0.33900000000000002,"5min":0.315,"15min":0.22600000000000001},"max":{"1min":4.4800000000000004,"5min":24.16,"15min":24.16},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.343,"15min":1.0109999999999999},"min":{"1min":0.30399999999999999,"5min":0.28499999999999998,"15min":0.247},"max":{"1min":2.984,"5min":24.129999999999999,"15min":24.129999999999999},"last":0.45700000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":1.601,"15min":1.1819999999999999},"min":{"1min":0.28899999999999998,"5min":0.245,"15min":0.24299999999999999},"max":{"1min":1.8819999999999999,"5min":54.530000000000001,"15min":54.530000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.79200000000000004,"5min":1.5700000000000001,"15min":1.1539999999999999},"min":{"1min":0.32600000000000001,"5min":0.247,"15min":0.247},"max":{"1min":2.871,"5min":54.514000000000003,"15min":54.514000000000003},"last":0.443}]},{"osd":3,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":2.1320000000000001,"5min":1.782,"15min":1.4039999999999999},"min":{"1min":0.29899999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":17.198,"5min":30.475000000000001,"15min":31.207000000000001},"last":3.5179999999999998},{"interface":"front","average":{"1min":2.472,"5min":1.772,"15min":1.3109999999999999},"min":{"1min":0.35899999999999999,"5min":0.32000000000000001,"15min":0.29699999999999999},"max":{"1min":16.934999999999999,"5min":30.172000000000001,"15min":31.199000000000002},"last":0.60499999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66700000000000004,"15min":0.496},"min":{"1min":0.27900000000000003,"5min":0.22900000000000001,"15min":0.20799999999999999},"max":{"1min":2.3580000000000001,"5min":4.5199999999999996,"15min":4.5199999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.73599999999999999,"15min":0.55200000000000005},"min":{"1min":0.315,"5min":0.24199999999999999,"15min":0.22700000000000001},"max":{"1min":2.4700000000000002,"5min":6.798,"15min":6.798},"last":1.456}]},{"osd":6,"last update":"Tue Sep 10 18:33:37 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.76700000000000002,"15min":0.63500000000000001},"min":{"1min":0.29299999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.66100000000000003,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.79800000000000004,"15min":0.63300000000000001},"min":{"1min":0.36899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.254,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.47199999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.82399999999999995,"15min":0.58199999999999996},"min":{"1min":0.39700000000000002,"5min":0.25900000000000001,"15min":0.22500000000000001},"max":{"1min":2.5859999999999999,"5min":5.569,"15min":5.569},"last":1.5069999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.73399999999999999,"15min":0.54400000000000004},"min":{"1min":0.36599999999999999,"5min":0.25900000000000001,"15min":0.24299999999999999},"max":{"1min":2.4510000000000001,"5min":3.1819999999999999,"15min":3.1819999999999999},"last":1.1299999999999999}]}]},{"osd":4,"up_from":33,"seq":141733920939,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1431796,"kb_used_data":1134444,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":92313356,"statfs":{"total":95995035648,"available":94528876544,"internally_reserved":0,"allocated":1161670656,"data_stored":1161261981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:33:33 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.196,"15min":1.022},"min":{"1min":0.23599999999999999,"5min":0.21199999999999999,"15min":0.188},"max":{"1min":3.7309999999999999,"5min":6.2060000000000004,"15min":31.585000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.95299999999999996,"15min":0.97699999999999998},"min":{"1min":0.27100000000000002,"5min":0.189,"15min":0.189},"max":{"1min":3.4279999999999999,"5min":4.3129999999999997,"15min":29.802},"last":0.372}]},{"osd":1,"last update":"Tue Sep 10 18:33:33 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.506,"15min":0.96799999999999997},"min":{"1min":0.26900000000000002,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":3.2490000000000001,"5min":50.045000000000002,"15min":50.045000000000002},"last":0.502},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.766,"15min":1.056},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.22900000000000001},"max":{"1min":4.2789999999999999,"5min":50.283000000000001,"15min":50.283000000000001},"last":0.48799999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96999999999999997,"15min":0.85899999999999999},"min":{"1min":0.30299999999999999,"5min":0.249,"15min":0.249},"max":{"1min":3.0870000000000002,"5min":5.2380000000000004,"15min":13.898999999999999},"last":0.316},{"interface":"front","average":{"1min":0.999,"5min":0.96799999999999997,"15min":0.877},"min":{"1min":0.307,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":3.2519999999999998,"5min":4.1950000000000003,"15min":14.51},"last":0.42199999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:33:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.665,"15min":1.278},"min":{"1min":0.27400000000000002,"5min":0.26300000000000001,"15min":0.249},"max":{"1min":3.4049999999999998,"5min":49.990000000000002,"15min":49.990000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.419,"5min":1.75,"15min":1.333},"min":{"1min":0.312,"5min":0.312,"15min":0.255},"max":{"1min":4.9740000000000002,"5min":50.764000000000003,"15min":50.764000000000003},"last":0.29799999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:33:58 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.995,"15min":0.59999999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.20499999999999999},"max":{"1min":3.0409999999999999,"5min":22.861000000000001,"15min":22.861000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.98399999999999999,"15min":0.623},"min":{"1min":0.28299999999999997,"5min":0.22,"15min":0.22},"max":{"1min":3.3799999999999999,"5min":19.611999999999998,"15min":19.611999999999998},"last":0.55900000000000005}]},{"osd":6,"last update":"Tue Sep 10 18:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.70199999999999996,"15min":0.503},"min":{"1min":0.33900000000000002,"5min":0.27400000000000002,"15min":0.26400000000000001},"max":{"1min":3.1139999999999999,"5min":6.7990000000000004,"15min":6.7990000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.70899999999999996,"15min":0.54300000000000004},"min":{"1min":0.315,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.21,"5min":7.0300000000000002,"15min":7.0300000000000002},"last":0.35599999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.70099999999999996,"15min":0.54900000000000004},"min":{"1min":0.32200000000000001,"5min":0.28799999999999998,"15min":0.255},"max":{"1min":3.145,"5min":7.0670000000000002,"15min":7.0670000000000002},"last":0.436},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.72099999999999997,"15min":0.56499999999999995},"min":{"1min":0.33700000000000002,"5min":0.28699999999999998,"15min":0.247},"max":{"1min":3.3450000000000002,"5min":6.5330000000000004,"15min":6.5330000000000004},"last":0.80500000000000005}]}]},{"osd":3,"up_from":27,"seq":115964117169,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3589252,"kb_used_data":2995068,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":90155900,"statfs":{"total":95995035648,"available":92319641600,"internally_reserved":0,"allocated":3066949632,"data_stored":3066525961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:12 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2789999999999999,"15min":1.2390000000000001},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.9569999999999999,"5min":5.4560000000000004,"15min":8.5530000000000008},"last":2.089},{"interface":"front","average":{"1min":1.903,"5min":1.464,"15min":1.351},"min":{"1min":0.253,"5min":0.22,"15min":0.193},"max":{"1min":7.5979999999999999,"5min":8.2110000000000003,"15min":13.659000000000001},"last":0.55800000000000005}]},{"osd":1,"last update":"Tue Sep 10 18:34:12 2024","interfaces":[{"interface":"back","average":{"1min":1.9890000000000001,"5min":1.923,"15min":1.431},"min":{"1min":0.28599999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.0419999999999998,"5min":35.406999999999996,"15min":35.406999999999996},"last":1.254},{"interface":"front","average":{"1min":1.556,"5min":1.9590000000000001,"15min":1.343},"min":{"1min":0.375,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":3.843,"5min":36.902000000000001,"15min":36.902000000000001},"last":2.1230000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:34:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.0589999999999999,"15min":1.077},"min":{"1min":0.38700000000000001,"5min":0.222,"15min":0.222},"max":{"1min":3.9060000000000001,"5min":5.3810000000000002,"15min":7.7960000000000003},"last":2.016},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3520000000000001,"15min":1.256},"min":{"1min":0.23000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":6.7290000000000001,"5min":14.997999999999999,"15min":14.997999999999999},"last":2.7519999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":1.3049999999999999,"15min":1.2010000000000001},"min":{"1min":0.33800000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":23.684999999999999},"last":2.5840000000000001},{"interface":"front","average":{"1min":2,"5min":1.2749999999999999,"15min":1.1539999999999999},"min":{"1min":0.315,"5min":0.29899999999999999,"15min":0.27200000000000002},"max":{"1min":6.1929999999999996,"5min":6.1929999999999996,"15min":23.649000000000001},"last":0.61399999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:33:57 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.3540000000000001,"15min":1.288},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":6.601,"5min":6.601,"15min":23.719999999999999},"last":2.726},{"interface":"front","average":{"1min":1.7130000000000001,"5min":1.375,"15min":1.2509999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.3979999999999997,"5min":6.2249999999999996,"15min":23.707000000000001},"last":0.59299999999999997}]},{"osd":6,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":1.966,"5min":1.8440000000000001,"15min":1.2050000000000001},"min":{"1min":0.38500000000000001,"5min":0.32200000000000001,"15min":0.317},"max":{"1min":6.2889999999999997,"5min":31.815999999999999,"15min":31.815999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.978,"15min":1.4970000000000001},"min":{"1min":0.42999999999999999,"5min":0.33400000000000002,"15min":0.315},"max":{"1min":7.0970000000000004,"5min":31.742000000000001,"15min":31.742000000000001},"last":2.3860000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.452,"15min":1.2390000000000001},"min":{"1min":0.56599999999999995,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":6.5730000000000004,"5min":6.5730000000000004,"15min":23.748999999999999},"last":2.6190000000000002},{"interface":"front","average":{"1min":2.2250000000000001,"5min":1.468,"15min":1.286},"min":{"1min":0.46899999999999997,"5min":0.35199999999999998,"15min":0.307},"max":{"1min":6.2530000000000001,"5min":12.612,"15min":25.058},"last":0.65200000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411509,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1444584,"kb_used_data":1134688,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92300568,"statfs":{"total":95995035648,"available":94515781632,"internally_reserved":0,"allocated":1161920512,"data_stored":1161524141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:33:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.7390000000000001,"15min":1.3300000000000001},"min":{"1min":0.22700000000000001,"5min":0.216,"15min":0.185},"max":{"1min":6.5019999999999998,"5min":43.890000000000001,"15min":43.890000000000001},"last":0.435},{"interface":"front","average":{"1min":1.3260000000000001,"5min":2.0920000000000001,"15min":1.5309999999999999},"min":{"1min":0.247,"5min":0.223,"15min":0.16900000000000001},"max":{"1min":10.606999999999999,"5min":44.331000000000003,"15min":44.331000000000003},"last":0.25600000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:33:47 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":2.0230000000000001,"15min":1.3919999999999999},"min":{"1min":0.19400000000000001,"5min":0.193,"15min":0.193},"max":{"1min":7.266,"5min":44.575000000000003,"15min":44.575000000000003},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.9510000000000001,"15min":1.2170000000000001},"min":{"1min":0.25900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":7.0270000000000001,"5min":44.368000000000002,"15min":44.368000000000002},"last":1.069}]},{"osd":3,"last update":"Tue Sep 10 18:34:10 2024","interfaces":[{"interface":"back","average":{"1min":4.4269999999999996,"5min":2.1669999999999998,"15min":1.7150000000000001},"min":{"1min":0.312,"5min":0.26100000000000001,"15min":0.251},"max":{"1min":61.331000000000003,"5min":61.331000000000003,"15min":61.331000000000003},"last":2.0430000000000001},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.1949999999999998,"15min":1.7969999999999999},"min":{"1min":0.33500000000000002,"5min":0.214,"15min":0.214},"max":{"1min":61.250999999999998,"5min":61.250999999999998,"15min":61.250999999999998},"last":1.546}]},{"osd":4,"last update":"Tue Sep 10 18:33:37 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":1.266,"15min":1.0089999999999999},"min":{"1min":0.36199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":4.5570000000000004,"5min":12.555999999999999,"15min":12.555999999999999},"last":0.503},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.171,"15min":0.92700000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.317},"max":{"1min":6.3639999999999999,"5min":7.0730000000000004,"15min":12.130000000000001},"last":0.39800000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:33:56 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":1.262,"15min":0.97299999999999998},"min":{"1min":0.32000000000000001,"5min":0.30299999999999999,"15min":0.27500000000000002},"max":{"1min":2.3980000000000001,"5min":7.1609999999999996,"15min":12.106},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.4430000000000001,"15min":1.0369999999999999},"min":{"1min":0.45000000000000001,"5min":0.32200000000000001,"15min":0.31900000000000001},"max":{"1min":2.6949999999999998,"5min":12.529999999999999,"15min":12.529999999999999},"last":0.49199999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:33:37 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.234,"15min":1.125},"min":{"1min":0.34999999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":4.5899999999999999,"5min":5.9989999999999997,"15min":5.9989999999999997},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.2,"15min":0.96599999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.30099999999999999},"max":{"1min":6.4089999999999998,"5min":7.1349999999999998,"15min":7.1349999999999998},"last":0.55500000000000005}]},{"osd":7,"last update":"Tue Sep 10 18:33:47 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.252,"15min":1.0469999999999999},"min":{"1min":0.42699999999999999,"5min":0.35399999999999998,"15min":0.30499999999999999},"max":{"1min":10.68,"5min":10.68,"15min":10.68},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.202,"5min":1.4790000000000001,"15min":1.107},"min":{"1min":0.40400000000000003,"5min":0.376,"15min":0.307},"max":{"1min":4.6440000000000001,"5min":12.596,"15min":12.596},"last":0.53900000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575034,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4003268,"kb_used_data":3404924,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":89741884,"statfs":{"total":95995035648,"available":91895689216,"internally_reserved":0,"allocated":3486642176,"data_stored":3486217743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:33:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.385,"15min":1.4730000000000001},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":4.2000000000000002,"5min":14.079000000000001,"15min":80.456999999999994},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.077,"5min":1.421,"15min":1.556},"min":{"1min":0.22800000000000001,"5min":0.187,"15min":0.187},"max":{"1min":4.5410000000000004,"5min":7.3120000000000003,"15min":80.578000000000003},"last":2.3780000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:33:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":1.4219999999999999,"15min":1.1499999999999999},"min":{"1min":0.25900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.694,"5min":7.2530000000000001,"15min":13.035},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.095,"5min":1.405,"15min":1.1060000000000001},"min":{"1min":0.23799999999999999,"5min":0.216,"15min":0.19900000000000001},"max":{"1min":2.621,"5min":12.222,"15min":12.222},"last":1.321}]},{"osd":3,"last update":"Tue Sep 10 18:33:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.4219999999999999,"15min":1.4830000000000001},"min":{"1min":0.41099999999999998,"5min":0.23899999999999999,"15min":0.22600000000000001},"max":{"1min":3.2570000000000001,"5min":14.121,"15min":43.076000000000001},"last":1.7889999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.5780000000000001,"15min":1.579},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":4.5800000000000001,"5min":7.3799999999999999,"15min":41.93},"last":1.544}]},{"osd":4,"last update":"Tue Sep 10 18:33:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.3879999999999999,"15min":1.0169999999999999},"min":{"1min":0.41199999999999998,"5min":0.32100000000000001,"15min":0.29799999999999999},"max":{"1min":4.6289999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"last":0.66800000000000004},{"interface":"front","average":{"1min":2.2069999999999999,"5min":1.4950000000000001,"15min":1.1850000000000001},"min":{"1min":0.308,"5min":0.30499999999999999,"15min":0.28299999999999997},"max":{"1min":7.8769999999999998,"5min":7.8769999999999998,"15min":19.393000000000001},"last":1.357}]},{"osd":5,"last update":"Tue Sep 10 18:33:57 2024","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":1.6339999999999999,"15min":1.196},"min":{"1min":0.46999999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":7.2110000000000003,"5min":12.206,"15min":15.48},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.556,"15min":1.103},"min":{"1min":0.443,"5min":0.36599999999999999,"15min":0.308},"max":{"1min":11.414999999999999,"5min":11.414999999999999,"15min":11.414999999999999},"last":0.79200000000000004}]},{"osd":6,"last update":"Tue Sep 10 18:33:42 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":1.232,"15min":1.276},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.6479999999999999,"5min":10.039,"15min":10.039},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.214,"5min":1.4790000000000001,"15min":1.371},"min":{"1min":0.48099999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":6.2670000000000003,"5min":7.3369999999999997,"15min":7.3369999999999997},"last":0.69199999999999995}]},{"osd":7,"last update":"Tue Sep 10 18:33:42 2024","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.407,"15min":1.1080000000000001},"min":{"1min":0.45300000000000001,"5min":0.40400000000000003,"15min":0.34399999999999997},"max":{"1min":8.3650000000000002,"5min":10.005000000000001,"15min":15.545999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.5109999999999999,"15min":1.095},"min":{"1min":0.57999999999999996,"5min":0.35699999999999998,"15min":0.313},"max":{"1min":6.3049999999999997,"5min":7.359,"15min":7.359},"last":0.627}]}]},{"osd":0,"up_from":8,"seq":34359738559,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1796336,"kb_used_data":1490664,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91948816,"statfs":{"total":95995035648,"available":94155587584,"internally_reserved":0,"allocated":1526439936,"data_stored":1526035230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:33:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.9419999999999999,"15min":1.3089999999999999},"min":{"1min":0.221,"5min":0.20499999999999999,"15min":0.193},"max":{"1min":3.5920000000000001,"5min":33.130000000000003,"15min":33.130000000000003},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.9219999999999999,"15min":1.3779999999999999},"min":{"1min":0.35299999999999998,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":3.5329999999999999,"5min":31.329999999999998,"15min":31.329999999999998},"last":0.66000000000000003}]},{"osd":2,"last update":"Tue Sep 10 18:33:48 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.2829999999999999,"15min":1.2849999999999999},"min":{"1min":0.28599999999999998,"5min":0.20699999999999999,"15min":0.189},"max":{"1min":3.137,"5min":7.4329999999999998,"15min":30.219999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":1.2110000000000001,"15min":1.268},"min":{"1min":0.23000000000000001,"5min":0.224,"15min":0.22},"max":{"1min":2.6160000000000001,"5min":7.5119999999999996,"15min":30.129999999999999},"last":0.26200000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:33:13 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.464,"15min":1.3640000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.157,"5min":8.5709999999999997,"15min":23.143999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.583,"15min":1.343},"min":{"1min":0.28799999999999998,"5min":0.249,"15min":0.23899999999999999},"max":{"1min":3.5720000000000001,"5min":9.3360000000000003,"15min":23.035},"last":0.81100000000000005}]},{"osd":4,"last update":"Tue Sep 10 18:33:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.21,"15min":1.1259999999999999},"min":{"1min":0.36799999999999999,"5min":0.29299999999999998,"15min":0.246},"max":{"1min":4.0170000000000003,"5min":8.3699999999999992,"15min":17.151},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.0660000000000001,"15min":1.1040000000000001},"min":{"1min":0.32600000000000001,"5min":0.27500000000000002,"15min":0.25900000000000001},"max":{"1min":2.5409999999999999,"5min":4.8460000000000001,"15min":18.728999999999999},"last":0.34300000000000003}]},{"osd":5,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.167,"15min":1.228},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.3210000000000002,"5min":4.7220000000000004,"15min":18.77},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.2949999999999999,"15min":1.1619999999999999},"min":{"1min":0.42599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.6050000000000004,"5min":6.6079999999999997,"15min":14.991},"last":0.68700000000000006}]},{"osd":6,"last update":"Tue Sep 10 18:33:37 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.294,"15min":1.27},"min":{"1min":0.41499999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.1299999999999999,"5min":8.3940000000000001,"15min":8.3940000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.2529999999999999,"15min":1.3300000000000001},"min":{"1min":0.35199999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.6680000000000001,"5min":9.7859999999999996,"15min":9.7859999999999996},"last":0.51700000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:33:37 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.3220000000000001,"15min":1.147},"min":{"1min":0.435,"5min":0.32200000000000001,"15min":0.26700000000000002},"max":{"1min":4.2539999999999996,"5min":5.0430000000000001,"15min":13.711},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.218,"15min":1.0920000000000001},"min":{"1min":0.436,"5min":0.36199999999999999,"15min":0.33500000000000002},"max":{"1min":3.1640000000000001,"5min":9.6549999999999994,"15min":15.032},"last":0.53500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1510735872,"data_stored":1510735872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3443068928,"data_stored":3443064856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1151533056,"data_stored":1151533056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3047555072,"data_stored":3047555072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1155137536,"data_stored":1155137536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3462008832,"data_stored":3462004760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1142882304,"data_stored":1142882304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3462467584,"data_stored":3462463512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:34:21.036 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-10T18:34:21.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pg_num 31 2024-09-10T18:34:21.082 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:21.110 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:21.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:21.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:21.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:21.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:21.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:21.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:21.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:21.416 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:21.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:21.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[29291]: 4.e deep-scrub starts 2024-09-10T18:34:21.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[29291]: 4.e deep-scrub ok 2024-09-10T18:34:21.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[29291]: 9.9 scrub starts 2024-09-10T18:34:21.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[29291]: 4.5 scrub starts 2024-09-10T18:34:21.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[29291]: 4.5 scrub ok 2024-09-10T18:34:21.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[29291]: 9.7 scrub ok 2024-09-10T18:34:21.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[29291]: 3.8 scrub starts 2024-09-10T18:34:21.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[29291]: 3.8 scrub ok 2024-09-10T18:34:21.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[29291]: 6.f scrub starts 2024-09-10T18:34:21.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[29291]: 6.f scrub ok 2024-09-10T18:34:21.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[29291]: 9.3 scrub starts 2024-09-10T18:34:21.505 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:34:21] "GET /metrics HTTP/1.1" 200 224503 "" "Prometheus/2.43.0" 2024-09-10T18:34:21.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[39220]: 4.e deep-scrub starts 2024-09-10T18:34:21.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[39220]: 4.e deep-scrub ok 2024-09-10T18:34:21.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[39220]: 9.9 scrub starts 2024-09-10T18:34:21.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[39220]: 4.5 scrub starts 2024-09-10T18:34:21.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[39220]: 4.5 scrub ok 2024-09-10T18:34:21.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[39220]: 9.7 scrub ok 2024-09-10T18:34:21.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[39220]: 3.8 scrub starts 2024-09-10T18:34:21.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[39220]: 3.8 scrub ok 2024-09-10T18:34:21.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[39220]: 6.f scrub starts 2024-09-10T18:34:21.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[39220]: 6.f scrub ok 2024-09-10T18:34:21.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:21 smithi028 ceph-mon[39220]: 9.3 scrub starts 2024-09-10T18:34:21.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:21.552 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:21.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:21.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:21.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:21 smithi204 ceph-mon[32357]: 4.e deep-scrub starts 2024-09-10T18:34:21.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:21 smithi204 ceph-mon[32357]: 4.e deep-scrub ok 2024-09-10T18:34:21.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:21 smithi204 ceph-mon[32357]: 9.9 scrub starts 2024-09-10T18:34:21.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:21 smithi204 ceph-mon[32357]: 4.5 scrub starts 2024-09-10T18:34:21.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:21 smithi204 ceph-mon[32357]: 4.5 scrub ok 2024-09-10T18:34:21.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:21 smithi204 ceph-mon[32357]: 9.7 scrub ok 2024-09-10T18:34:21.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:21 smithi204 ceph-mon[32357]: 3.8 scrub starts 2024-09-10T18:34:21.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:21 smithi204 ceph-mon[32357]: 3.8 scrub ok 2024-09-10T18:34:21.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:21 smithi204 ceph-mon[32357]: 6.f scrub starts 2024-09-10T18:34:21.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:21 smithi204 ceph-mon[32357]: 6.f scrub ok 2024-09-10T18:34:21.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:21 smithi204 ceph-mon[32357]: 9.3 scrub starts 2024-09-10T18:34:21.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:21.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:21.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:21.798 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:21.827 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:21.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:21.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:21.929 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:21.959 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:21.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:22.060 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:22.088 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:22.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:22.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:22.190 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:22.223 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:22.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:22.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:22.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:22.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:22.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:22.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:22.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:22 smithi028 ceph-mon[39220]: 3.9 scrub starts 2024-09-10T18:34:22.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:22 smithi028 ceph-mon[39220]: 3.9 scrub ok 2024-09-10T18:34:22.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:22 smithi028 ceph-mon[39220]: pgmap v577: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 965 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-10T18:34:22.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:22 smithi028 ceph-mon[39220]: 9.9 scrub ok 2024-09-10T18:34:22.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:22 smithi028 ceph-mon[39220]: 5.b deep-scrub starts 2024-09-10T18:34:22.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:22 smithi028 ceph-mon[39220]: 5.b deep-scrub ok 2024-09-10T18:34:22.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:22 smithi028 ceph-mon[29291]: 3.9 scrub starts 2024-09-10T18:34:22.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:22 smithi028 ceph-mon[29291]: 3.9 scrub ok 2024-09-10T18:34:22.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:22 smithi028 ceph-mon[29291]: pgmap v577: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 965 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-10T18:34:22.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:22 smithi028 ceph-mon[29291]: 9.9 scrub ok 2024-09-10T18:34:22.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:22 smithi028 ceph-mon[29291]: 5.b deep-scrub starts 2024-09-10T18:34:22.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:22 smithi028 ceph-mon[29291]: 5.b deep-scrub ok 2024-09-10T18:34:22.506 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:34:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:34:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:34:22.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:22.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:22 smithi204 ceph-mon[32357]: 3.9 scrub starts 2024-09-10T18:34:22.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:22 smithi204 ceph-mon[32357]: 3.9 scrub ok 2024-09-10T18:34:22.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:22 smithi204 ceph-mon[32357]: pgmap v577: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 965 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-10T18:34:22.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:22 smithi204 ceph-mon[32357]: 9.9 scrub ok 2024-09-10T18:34:22.507 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:22 smithi204 ceph-mon[32357]: 5.b deep-scrub starts 2024-09-10T18:34:22.507 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:22 smithi204 ceph-mon[32357]: 5.b deep-scrub ok 2024-09-10T18:34:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:22.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:22.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:22.643 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:22.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:22.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:22.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:22.774 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:22.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:22.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:22.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:22.910 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:22.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:22.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:23.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:23.065 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:23.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:23.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:23.167 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:23.195 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:23.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:23.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:23.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:23.328 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:23.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:23.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:23.464 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:23.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:23.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:23.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:23 smithi028 ceph-mon[39220]: 6.2 scrub starts 2024-09-10T18:34:23.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:23 smithi028 ceph-mon[39220]: 6.2 scrub ok 2024-09-10T18:34:23.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:23 smithi028 ceph-mon[39220]: 6.1f scrub starts 2024-09-10T18:34:23.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:23 smithi028 ceph-mon[39220]: 6.1f scrub ok 2024-09-10T18:34:23.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:23 smithi028 ceph-mon[39220]: 5.5 scrub starts 2024-09-10T18:34:23.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:23 smithi028 ceph-mon[39220]: 5.5 scrub ok 2024-09-10T18:34:23.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:23 smithi028 ceph-mon[39220]: 9.3 scrub ok 2024-09-10T18:34:23.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:23 smithi028 ceph-mon[39220]: 9.e scrub starts 2024-09-10T18:34:23.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:23 smithi028 ceph-mon[29291]: 6.2 scrub starts 2024-09-10T18:34:23.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:23 smithi028 ceph-mon[29291]: 6.2 scrub ok 2024-09-10T18:34:23.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:23 smithi028 ceph-mon[29291]: 6.1f scrub starts 2024-09-10T18:34:23.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:23 smithi028 ceph-mon[29291]: 6.1f scrub ok 2024-09-10T18:34:23.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:23 smithi028 ceph-mon[29291]: 5.5 scrub starts 2024-09-10T18:34:23.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:23 smithi028 ceph-mon[29291]: 5.5 scrub ok 2024-09-10T18:34:23.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:23 smithi028 ceph-mon[29291]: 9.3 scrub ok 2024-09-10T18:34:23.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:23 smithi028 ceph-mon[29291]: 9.e scrub starts 2024-09-10T18:34:23.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:23.595 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:23.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:23.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:23.597 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-10T18:34:23.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:23 smithi204 ceph-mon[32357]: 6.2 scrub starts 2024-09-10T18:34:23.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:23 smithi204 ceph-mon[32357]: 6.2 scrub ok 2024-09-10T18:34:23.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:23 smithi204 ceph-mon[32357]: 6.1f scrub starts 2024-09-10T18:34:23.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:23 smithi204 ceph-mon[32357]: 6.1f scrub ok 2024-09-10T18:34:23.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:23 smithi204 ceph-mon[32357]: 5.5 scrub starts 2024-09-10T18:34:23.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:23 smithi204 ceph-mon[32357]: 5.5 scrub ok 2024-09-10T18:34:23.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:23 smithi204 ceph-mon[32357]: 9.3 scrub ok 2024-09-10T18:34:23.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:23 smithi204 ceph-mon[32357]: 9.e scrub starts 2024-09-10T18:34:23.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:34:23.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:23.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:23.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:23.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:23.973 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:34:23.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:23.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:24.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:24.124 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:24.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:24.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:24.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:24.250 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:34:24.260 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:24.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:24.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:24 smithi204 ceph-mon[32357]: 1.5 scrub starts 2024-09-10T18:34:24.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:24 smithi204 ceph-mon[32357]: 1.5 scrub ok 2024-09-10T18:34:24.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:24 smithi204 ceph-mon[32357]: 3.1e scrub starts 2024-09-10T18:34:24.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:24 smithi204 ceph-mon[32357]: 3.1e scrub ok 2024-09-10T18:34:24.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:24 smithi204 ceph-mon[32357]: 1.11 scrub starts 2024-09-10T18:34:24.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:24 smithi204 ceph-mon[32357]: 1.11 scrub ok 2024-09-10T18:34:24.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:24 smithi204 ceph-mon[32357]: 5.2 scrub starts 2024-09-10T18:34:24.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:24 smithi204 ceph-mon[32357]: 5.2 scrub ok 2024-09-10T18:34:24.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:24 smithi204 ceph-mon[32357]: pgmap v578: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 166 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:34:24.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:24 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:34:24.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:34:24.362 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:24.387 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[39220]: 1.5 scrub starts 2024-09-10T18:34:24.387 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[39220]: 1.5 scrub ok 2024-09-10T18:34:24.387 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[39220]: 3.1e scrub starts 2024-09-10T18:34:24.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[39220]: 3.1e scrub ok 2024-09-10T18:34:24.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[39220]: 1.11 scrub starts 2024-09-10T18:34:24.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[39220]: 1.11 scrub ok 2024-09-10T18:34:24.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[39220]: 5.2 scrub starts 2024-09-10T18:34:24.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[39220]: 5.2 scrub ok 2024-09-10T18:34:24.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[39220]: pgmap v578: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 166 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:34:24.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:34:24.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:34:24.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[29291]: 1.5 scrub starts 2024-09-10T18:34:24.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[29291]: 1.5 scrub ok 2024-09-10T18:34:24.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[29291]: 3.1e scrub starts 2024-09-10T18:34:24.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[29291]: 3.1e scrub ok 2024-09-10T18:34:24.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[29291]: 1.11 scrub starts 2024-09-10T18:34:24.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[29291]: 1.11 scrub ok 2024-09-10T18:34:24.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[29291]: 5.2 scrub starts 2024-09-10T18:34:24.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[29291]: 5.2 scrub ok 2024-09-10T18:34:24.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[29291]: pgmap v578: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 166 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:34:24.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:34:24.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:34:24.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:24.401 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:24.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:24.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:24.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:24.546 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:24.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:24.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:24.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:24.684 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:24.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:24.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:24.820 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:24.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:24.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:24.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:34:24 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:34:24.922 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:24.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:24.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:25.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:25.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:25.095 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:25.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:25.196 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:25.227 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:25.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:25.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:34:25.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:25.352 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:25.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:25.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:25.454 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:25.486 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:25.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:25.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:25.494 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:25 smithi204 ceph-mon[32357]: 6.16 scrub starts 2024-09-10T18:34:25.495 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:25 smithi204 ceph-mon[32357]: 6.16 scrub ok 2024-09-10T18:34:25.495 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:25 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/229754235' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:34:25.495 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:25 smithi204 ceph-mon[32357]: from='client.25435 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:34:25.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:25.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:25.622 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:25.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:25.723 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:25.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:25 smithi028 ceph-mon[29291]: 6.16 scrub starts 2024-09-10T18:34:25.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:25 smithi028 ceph-mon[29291]: 6.16 scrub ok 2024-09-10T18:34:25.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:25 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/229754235' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:34:25.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:25 smithi028 ceph-mon[29291]: from='client.25435 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:34:25.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:25 smithi028 ceph-mon[39220]: 6.16 scrub starts 2024-09-10T18:34:25.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:25 smithi028 ceph-mon[39220]: 6.16 scrub ok 2024-09-10T18:34:25.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:25 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/229754235' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:34:25.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:25 smithi028 ceph-mon[39220]: from='client.25435 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:34:25.763 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:25.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:25.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:25.865 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:25.897 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:25.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:25.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:25.949 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:34:25.949 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:34:25.949 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-10T18:34:25.949 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:34:25.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:26.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:26.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:26.173 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:26.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:26.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:26.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:26.321 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:26.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:26.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:26.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:26.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:26.560 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:26.573 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:26 smithi204 ceph-mon[32357]: 9.c deep-scrub starts 2024-09-10T18:34:26.573 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:26 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:26.573 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:26 smithi204 ceph-mon[32357]: pgmap v579: 168 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 165 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:34:26.573 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:26 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:34:26.573 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:26 smithi204 ceph-mon[32357]: from='client.25435 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:34:26.573 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:26 smithi204 ceph-mon[32357]: osdmap e284: 8 total, 8 up, 8 in 2024-09-10T18:34:26.596 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:26.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:26.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:26.680 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-10T18:34:26.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:26.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:26 smithi028 ceph-mon[39220]: 9.c deep-scrub starts 2024-09-10T18:34:26.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:26 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:26.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:26 smithi028 ceph-mon[39220]: pgmap v579: 168 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 165 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:34:26.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:26 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:34:26.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:26 smithi028 ceph-mon[39220]: from='client.25435 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:34:26.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:26 smithi028 ceph-mon[39220]: osdmap e284: 8 total, 8 up, 8 in 2024-09-10T18:34:26.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:26 smithi028 ceph-mon[29291]: 9.c deep-scrub starts 2024-09-10T18:34:26.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:26 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:26.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:26 smithi028 ceph-mon[29291]: pgmap v579: 168 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 165 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:34:26.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:26 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:34:26.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:26 smithi028 ceph-mon[29291]: from='client.25435 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:34:26.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:26 smithi028 ceph-mon[29291]: osdmap e284: 8 total, 8 up, 8 in 2024-09-10T18:34:26.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:26.793 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:26.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:26.894 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:26.938 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:26.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:26.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:27.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:27.073 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:27.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:27.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:27.121 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:34:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:34:27.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:27.211 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:27.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:27.313 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:27.356 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:27.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:27.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:27.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:27.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:27.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:27.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:27.544 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:27 smithi204 ceph-mon[32357]: 4.1a scrub starts 2024-09-10T18:34:27.544 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:27 smithi204 ceph-mon[32357]: 4.1a scrub ok 2024-09-10T18:34:27.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:27.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:27.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:27.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:27.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:27 smithi028 ceph-mon[39220]: 4.1a scrub starts 2024-09-10T18:34:27.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:27 smithi028 ceph-mon[39220]: 4.1a scrub ok 2024-09-10T18:34:27.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:27 smithi028 ceph-mon[29291]: 4.1a scrub starts 2024-09-10T18:34:27.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:27 smithi028 ceph-mon[29291]: 4.1a scrub ok 2024-09-10T18:34:27.737 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:27.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:27.794 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:27.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:27.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:27.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:27.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:28.032 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:28.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:28.063 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:28.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:28.163 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:28.195 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:28.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:28.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:28.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:28.330 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:28.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:28.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:28.432 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:28.461 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:28.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:28.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:28.548 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:28 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:34:28.548 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:28 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:34:28.548 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:28 smithi028 ceph-mon[39220]: pgmap v581: 168 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 165 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-10T18:34:28.548 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:28 smithi028 ceph-mon[39220]: osdmap e285: 8 total, 8 up, 8 in 2024-09-10T18:34:28.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:28 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:34:28.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:28 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:34:28.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:28 smithi028 ceph-mon[29291]: pgmap v581: 168 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 165 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-10T18:34:28.549 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:28 smithi028 ceph-mon[29291]: osdmap e285: 8 total, 8 up, 8 in 2024-09-10T18:34:28.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:28 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:34:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:28 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:34:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:28 smithi204 ceph-mon[32357]: pgmap v581: 168 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 165 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-10T18:34:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:28 smithi204 ceph-mon[32357]: osdmap e285: 8 total, 8 up, 8 in 2024-09-10T18:34:28.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:28.605 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:28.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:28.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:28.742 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:28.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:28.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:28.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:28.879 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:28.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:28.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:28.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:29.020 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:29.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:29.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:29.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:29.152 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:29.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:29.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:29.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:29.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:29.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:29.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:29.295 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 1 pg_num to 31 2024-09-10T18:34:29.386 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:29.426 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:29.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:29.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:29.559 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:29.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:29.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:29.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:29 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2023678482' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-09-10T18:34:29.589 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:29 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2023678482' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-09-10T18:34:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:29 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2023678482' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-09-10T18:34:29.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:29.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:29.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:29.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:29.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:34:29.810 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:29.844 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:29.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:29.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:29.975 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:29.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:29.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:30.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:30.104 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:30.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:30.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:30.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:30.245 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:30.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:30.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:30.347 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:30.355 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:34:30.355 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:34:30.355 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-10T18:34:30.355 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-10T18:34:30.355 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:34:30.379 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:30.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:30.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:30.481 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:30.501 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:30.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:30.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:30.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:30 smithi204 ceph-mon[32357]: pgmap v583: 168 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 165 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 770 op/s 2024-09-10T18:34:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:30 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2023678482' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-10T18:34:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:30 smithi204 ceph-mon[32357]: osdmap e286: 8 total, 8 up, 8 in 2024-09-10T18:34:30.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:30.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:30.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:30.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:30.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:30 smithi028 ceph-mon[29291]: pgmap v583: 168 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 165 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 770 op/s 2024-09-10T18:34:30.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:30 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2023678482' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-10T18:34:30.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:30 smithi028 ceph-mon[29291]: osdmap e286: 8 total, 8 up, 8 in 2024-09-10T18:34:30.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:30 smithi028 ceph-mon[39220]: pgmap v583: 168 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 165 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 770 op/s 2024-09-10T18:34:30.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:30 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2023678482' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-10T18:34:30.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:30 smithi028 ceph-mon[39220]: osdmap e286: 8 total, 8 up, 8 in 2024-09-10T18:34:30.762 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:30.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:30.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:30.863 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:30.890 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:30.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:30.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:30.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:31.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:31.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:31.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:31.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:31.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:31.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:31.251 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:31.279 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:31.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:31.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:31.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:31.420 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:31.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:31.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:31.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:31.553 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:31.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:31.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:31.655 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:31.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-10T18:34:31.685 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:31.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:31.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:34:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:34:31] "GET /metrics HTTP/1.1" 200 224476 "" "Prometheus/2.43.0" 2024-09-10T18:34:31.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:31.822 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:31.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:31.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:31.924 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:31.961 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:31.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:31.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:32.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:32.065 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:34:32.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:31 smithi204 ceph-mon[32357]: pgmap v585: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.27k op/s 2024-09-10T18:34:32.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:31 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-10T18:34:32.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:31 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-10T18:34:32.112 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:32.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:32.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:32.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:32.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:32.253 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:32.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:31 smithi028 ceph-mon[29291]: pgmap v585: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.27k op/s 2024-09-10T18:34:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:31 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-10T18:34:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:31 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-10T18:34:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:31 smithi028 ceph-mon[39220]: pgmap v585: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.27k op/s 2024-09-10T18:34:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:31 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-10T18:34:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:31 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-10T18:34:32.354 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:32.376 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:32.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:32.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:32.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:32.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:32.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:34:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:34:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:34:32.620 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:32.642 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:32.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:32.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:32.744 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:32.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:32.781 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:32.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:32.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:32.903 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:32.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:33.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:33.039 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:33.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:33.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:32 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-10T18:34:33.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:32 smithi204 ceph-mon[32357]: osdmap e287: 8 total, 8 up, 8 in 2024-09-10T18:34:33.140 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:33.168 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:33.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:33.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:33.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:32 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-10T18:34:33.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:32 smithi028 ceph-mon[39220]: osdmap e287: 8 total, 8 up, 8 in 2024-09-10T18:34:33.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:32 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-10T18:34:33.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:32 smithi028 ceph-mon[29291]: osdmap e287: 8 total, 8 up, 8 in 2024-09-10T18:34:33.269 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:33.301 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:33.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:33.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:33.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:33.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:33.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:33.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:33.583 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:33.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:33.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:33.685 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:33.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:33.721 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:33.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:33.822 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:33.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:33.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:33.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:33.958 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:33.981 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:33 smithi204 ceph-mon[32357]: osdmap e288: 8 total, 8 up, 8 in 2024-09-10T18:34:33.981 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:33 smithi204 ceph-mon[32357]: pgmap v588: 178 pgs: 10 unknown, 2 active+clean+scrubbing+deep, 166 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 889 B/s rd, 0 B/s wr, 745 op/s 2024-09-10T18:34:33.981 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:33 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-09-10T18:34:33.981 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:33 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-09-10T18:34:33.990 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:33.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:33.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:34.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:34.114 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:34.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:34.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:34.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:34.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:34.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:33 smithi028 ceph-mon[29291]: osdmap e288: 8 total, 8 up, 8 in 2024-09-10T18:34:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:33 smithi028 ceph-mon[29291]: pgmap v588: 178 pgs: 10 unknown, 2 active+clean+scrubbing+deep, 166 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 889 B/s rd, 0 B/s wr, 745 op/s 2024-09-10T18:34:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:33 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-09-10T18:34:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:33 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-09-10T18:34:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:33 smithi028 ceph-mon[39220]: osdmap e288: 8 total, 8 up, 8 in 2024-09-10T18:34:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:33 smithi028 ceph-mon[39220]: pgmap v588: 178 pgs: 10 unknown, 2 active+clean+scrubbing+deep, 166 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 889 B/s rd, 0 B/s wr, 745 op/s 2024-09-10T18:34:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:33 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-09-10T18:34:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:33 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-09-10T18:34:34.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:34.385 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:34.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:34.425 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:34:34.425 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-10T18:34:34.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-10T18:34:34.425 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:34:34.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.425 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:181520 7.1 7:822e6410:::benchmark_data_smithi028_77856_object60420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:34:34.426 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:25.451480+0000", 2024-09-10T18:34:34.426 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 488.96852372299998, 2024-09-10T18:34:34.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13309078699999999, 2024-09-10T18:34:34.426 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.426 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.426 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.426 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.426 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.426 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 181520 2024-09-10T18:34:34.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.426 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451480+0000", 2024-09-10T18:34:34.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451480+0000", 2024-09-10T18:34:34.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451478+0000", 2024-09-10T18:34:34.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:34:34.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451485+0000", 2024-09-10T18:34:34.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3570000000000003e-06 2024-09-10T18:34:34.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451495+0000", 2024-09-10T18:34:34.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0385e-05 2024-09-10T18:34:34.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451500+0000", 2024-09-10T18:34:34.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8389999999999997e-06 2024-09-10T18:34:34.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451540+0000", 2024-09-10T18:34:34.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0339000000000001e-05 2024-09-10T18:34:34.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451626+0000", 2024-09-10T18:34:34.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5860999999999997e-05 2024-09-10T18:34:34.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-10T18:34:34.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.451694+0000", 2024-09-10T18:34:34.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8169000000000004e-05 2024-09-10T18:34:34.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.457845+0000", 2024-09-10T18:34:34.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006151045 2024-09-10T18:34:34.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.584485+0000", 2024-09-10T18:34:34.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.126639366 2024-09-10T18:34:34.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.584517+0000", 2024-09-10T18:34:34.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2372000000000001e-05 2024-09-10T18:34:34.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.432 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.584571+0000", 2024-09-10T18:34:34.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4137e-05 2024-09-10T18:34:34.432 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.432 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.432 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.432 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:181529 7.1 7:8089af12:::benchmark_data_smithi028_77856_object60429:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:34:34.432 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:25.453132+0000", 2024-09-10T18:34:34.432 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 488.96687228799999, 2024-09-10T18:34:34.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13400395500000001, 2024-09-10T18:34:34.433 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.433 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.433 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.433 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.433 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.433 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 181529 2024-09-10T18:34:34.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.433 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453132+0000", 2024-09-10T18:34:34.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453132+0000", 2024-09-10T18:34:34.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453131+0000", 2024-09-10T18:34:34.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:34:34.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453136+0000", 2024-09-10T18:34:34.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9910000000000002e-06 2024-09-10T18:34:34.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453144+0000", 2024-09-10T18:34:34.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7280000000000001e-06 2024-09-10T18:34:34.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453149+0000", 2024-09-10T18:34:34.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4479999999999996e-06 2024-09-10T18:34:34.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.436 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.436 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.436 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.456233+0000", 2024-09-10T18:34:34.436 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030846509999999999 2024-09-10T18:34:34.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.436 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.436 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.436 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.456312+0000", 2024-09-10T18:34:34.436 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8944999999999997e-05 2024-09-10T18:34:34.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.436 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.436 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-10T18:34:34.436 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.456378+0000", 2024-09-10T18:34:34.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5303999999999997e-05 2024-09-10T18:34:34.437 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.437 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.458107+0000", 2024-09-10T18:34:34.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0017296569999999999 2024-09-10T18:34:34.437 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.437 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.587061+0000", 2024-09-10T18:34:34.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.12895343500000001 2024-09-10T18:34:34.437 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.587089+0000", 2024-09-10T18:34:34.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7996000000000001e-05 2024-09-10T18:34:34.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.587136+0000", 2024-09-10T18:34:34.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6949000000000001e-05 2024-09-10T18:34:34.438 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.438 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.438 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.439 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192360 7.1 7:8a845e96:::benchmark_data_smithi028_77856_object71260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-10T18:34:34.439 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.176590+0000", 2024-09-10T18:34:34.439 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 476.24341371200001, 2024-09-10T18:34:34.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0444927079999999, 2024-09-10T18:34:34.439 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.439 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:34:34.439 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.439 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.439 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.439 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192360 2024-09-10T18:34:34.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.439 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176590+0000", 2024-09-10T18:34:34.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176590+0000", 2024-09-10T18:34:34.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176588+0000", 2024-09-10T18:34:34.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:34:34.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176595+0000", 2024-09-10T18:34:34.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7190000000000004e-06 2024-09-10T18:34:34.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176607+0000", 2024-09-10T18:34:34.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1936000000000001e-05 2024-09-10T18:34:34.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176612+0000", 2024-09-10T18:34:34.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2299999999999999e-06 2024-09-10T18:34:34.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176659+0000", 2024-09-10T18:34:34.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6782e-05 2024-09-10T18:34:34.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176790+0000", 2024-09-10T18:34:34.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013042800000000001 2024-09-10T18:34:34.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:34:34.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.176921+0000", 2024-09-10T18:34:34.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000131552 2024-09-10T18:34:34.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.179313+0000", 2024-09-10T18:34:34.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023916839999999998 2024-09-10T18:34:34.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.180160+0000", 2024-09-10T18:34:34.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00084720399999999995 2024-09-10T18:34:34.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.185626+0000", 2024-09-10T18:34:34.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054663239999999998 2024-09-10T18:34:34.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:34:34.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.185637+0000", 2024-09-10T18:34:34.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.112e-05 2024-09-10T18:34:34.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.220975+0000", 2024-09-10T18:34:34.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0353376249999999 2024-09-10T18:34:34.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.221083+0000", 2024-09-10T18:34:34.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108022 2024-09-10T18:34:34.445 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.445 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.445 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.446 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192414 7.1 7:8900b3fc:::benchmark_data_smithi028_77856_object71314:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:34:34.446 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.234714+0000", 2024-09-10T18:34:34.446 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 476.18528988999998, 2024-09-10T18:34:34.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0099985309999999, 2024-09-10T18:34:34.446 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.446 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.446 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.446 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.446 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.446 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192414 2024-09-10T18:34:34.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.446 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.234714+0000", 2024-09-10T18:34:34.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.234714+0000", 2024-09-10T18:34:34.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.234712+0000", 2024-09-10T18:34:34.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:34:34.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.234720+0000", 2024-09-10T18:34:34.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8809999999999996e-06 2024-09-10T18:34:34.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.234731+0000", 2024-09-10T18:34:34.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1804000000000001e-05 2024-09-10T18:34:34.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.234737+0000", 2024-09-10T18:34:34.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3340000000000001e-06 2024-09-10T18:34:34.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.234776+0000", 2024-09-10T18:34:34.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9490999999999998e-05 2024-09-10T18:34:34.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:34:34.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.234786+0000", 2024-09-10T18:34:34.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5510000000000005e-06 2024-09-10T18:34:34.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.221104+0000", 2024-09-10T18:34:34.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98631840199999998 2024-09-10T18:34:34.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.221216+0000", 2024-09-10T18:34:34.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011218300000000001 2024-09-10T18:34:34.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:34:34.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.221336+0000", 2024-09-10T18:34:34.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011998900000000001 2024-09-10T18:34:34.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.228488+0000", 2024-09-10T18:34:34.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007151704 2024-09-10T18:34:34.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244439+0000", 2024-09-10T18:34:34.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015951402999999999 2024-09-10T18:34:34.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244633+0000", 2024-09-10T18:34:34.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019376699999999999 2024-09-10T18:34:34.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244679+0000", 2024-09-10T18:34:34.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5878999999999999e-05 2024-09-10T18:34:34.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244713+0000", 2024-09-10T18:34:34.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3683000000000003e-05 2024-09-10T18:34:34.452 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.452 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.453 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.453 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192418 7.1 7:82bfeb99:::benchmark_data_smithi028_77856_object71318:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:34:34.453 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.235688+0000", 2024-09-10T18:34:34.453 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 476.18431629999998, 2024-09-10T18:34:34.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.009211074, 2024-09-10T18:34:34.453 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.453 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.453 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.453 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.453 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.453 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192418 2024-09-10T18:34:34.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.454 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.235688+0000", 2024-09-10T18:34:34.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.235688+0000", 2024-09-10T18:34:34.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.235686+0000", 2024-09-10T18:34:34.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:34:34.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.235693+0000", 2024-09-10T18:34:34.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.905e-06 2024-09-10T18:34:34.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.235707+0000", 2024-09-10T18:34:34.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4069e-05 2024-09-10T18:34:34.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.235714+0000", 2024-09-10T18:34:34.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4189999999999999e-06 2024-09-10T18:34:34.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.235760+0000", 2024-09-10T18:34:34.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6245999999999997e-05 2024-09-10T18:34:34.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:34:34.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.235766+0000", 2024-09-10T18:34:34.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4740000000000002e-06 2024-09-10T18:34:34.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.223084+0000", 2024-09-10T18:34:34.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98731729499999998 2024-09-10T18:34:34.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.223188+0000", 2024-09-10T18:34:34.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0001043 2024-09-10T18:34:34.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:34:34.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.223302+0000", 2024-09-10T18:34:34.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000113992 2024-09-10T18:34:34.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.228494+0000", 2024-09-10T18:34:34.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005192543 2024-09-10T18:34:34.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244498+0000", 2024-09-10T18:34:34.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016003567999999999 2024-09-10T18:34:34.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244841+0000", 2024-09-10T18:34:34.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000342963 2024-09-10T18:34:34.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244868+0000", 2024-09-10T18:34:34.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7489999999999999e-05 2024-09-10T18:34:34.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244899+0000", 2024-09-10T18:34:34.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0375999999999999e-05 2024-09-10T18:34:34.460 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.460 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.460 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.460 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192458 7.1 7:825d7094:::benchmark_data_smithi028_77856_object71358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:34:34.460 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.267613+0000", 2024-09-10T18:34:34.461 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 476.15239142399997, 2024-09-10T18:34:34.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97748335900000005, 2024-09-10T18:34:34.461 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.461 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.461 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.461 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.461 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.461 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192458 2024-09-10T18:34:34.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.461 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.461 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.267613+0000", 2024-09-10T18:34:34.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.267613+0000", 2024-09-10T18:34:34.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.267611+0000", 2024-09-10T18:34:34.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:34:34.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.267618+0000", 2024-09-10T18:34:34.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1720000000000001e-06 2024-09-10T18:34:34.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.267628+0000", 2024-09-10T18:34:34.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9000000000000001e-06 2024-09-10T18:34:34.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.267633+0000", 2024-09-10T18:34:34.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4410000000000001e-06 2024-09-10T18:34:34.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.267677+0000", 2024-09-10T18:34:34.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3467999999999998e-05 2024-09-10T18:34:34.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:34:34.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.267683+0000", 2024-09-10T18:34:34.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6250000000000001e-06 2024-09-10T18:34:34.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.224848+0000", 2024-09-10T18:34:34.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95716446499999996 2024-09-10T18:34:34.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.224967+0000", 2024-09-10T18:34:34.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0001192 2024-09-10T18:34:34.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:34:34.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.225087+0000", 2024-09-10T18:34:34.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000120241 2024-09-10T18:34:34.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.233973+0000", 2024-09-10T18:34:34.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088862950000000007 2024-09-10T18:34:34.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244535+0000", 2024-09-10T18:34:34.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010561305 2024-09-10T18:34:34.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245035+0000", 2024-09-10T18:34:34.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00050065799999999996 2024-09-10T18:34:34.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245063+0000", 2024-09-10T18:34:34.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7438999999999999e-05 2024-09-10T18:34:34.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245096+0000", 2024-09-10T18:34:34.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3263999999999999e-05 2024-09-10T18:34:34.467 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.467 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.467 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.468 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192468 7.1 7:8dcd11d8:::benchmark_data_smithi028_77856_object71368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:34:34.468 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.280719+0000", 2024-09-10T18:34:34.468 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 476.139285336, 2024-09-10T18:34:34.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.96446010999999998, 2024-09-10T18:34:34.468 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.468 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.468 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.468 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.468 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.468 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192468 2024-09-10T18:34:34.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.468 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.280719+0000", 2024-09-10T18:34:34.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.280719+0000", 2024-09-10T18:34:34.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.280716+0000", 2024-09-10T18:34:34.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:34:34.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.280724+0000", 2024-09-10T18:34:34.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2220000000000002e-06 2024-09-10T18:34:34.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.280739+0000", 2024-09-10T18:34:34.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4776e-05 2024-09-10T18:34:34.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.280745+0000", 2024-09-10T18:34:34.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0580000000000003e-06 2024-09-10T18:34:34.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.280794+0000", 2024-09-10T18:34:34.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8605e-05 2024-09-10T18:34:34.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:34:34.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.280804+0000", 2024-09-10T18:34:34.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0166000000000001e-05 2024-09-10T18:34:34.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.226133+0000", 2024-09-10T18:34:34.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.94532860699999999 2024-09-10T18:34:34.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.226220+0000", 2024-09-10T18:34:34.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7226999999999997e-05 2024-09-10T18:34:34.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:34:34.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.226289+0000", 2024-09-10T18:34:34.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8881999999999995e-05 2024-09-10T18:34:34.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.233980+0000", 2024-09-10T18:34:34.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0076915480000000003 2024-09-10T18:34:34.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244574+0000", 2024-09-10T18:34:34.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010593981000000001 2024-09-10T18:34:34.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245133+0000", 2024-09-10T18:34:34.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00055907199999999998 2024-09-10T18:34:34.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245159+0000", 2024-09-10T18:34:34.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5573999999999999e-05 2024-09-10T18:34:34.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245179+0000", 2024-09-10T18:34:34.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9879999999999999e-05 2024-09-10T18:34:34.474 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.474 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.474 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.475 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192490 7.1 7:85c9b87e:::benchmark_data_smithi028_77856_object71390:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:34:34.475 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.297655+0000", 2024-09-10T18:34:34.475 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 476.12234895900002, 2024-09-10T18:34:34.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.947659696, 2024-09-10T18:34:34.475 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.475 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.475 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.475 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.475 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.475 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192490 2024-09-10T18:34:34.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.475 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.297655+0000", 2024-09-10T18:34:34.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.297655+0000", 2024-09-10T18:34:34.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.297653+0000", 2024-09-10T18:34:34.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:34:34.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.297661+0000", 2024-09-10T18:34:34.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3289999999999994e-06 2024-09-10T18:34:34.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.297676+0000", 2024-09-10T18:34:34.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4829e-05 2024-09-10T18:34:34.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.297682+0000", 2024-09-10T18:34:34.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5740000000000001e-06 2024-09-10T18:34:34.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.297729+0000", 2024-09-10T18:34:34.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6752999999999998e-05 2024-09-10T18:34:34.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:34:34.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.297738+0000", 2024-09-10T18:34:34.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5720000000000003e-06 2024-09-10T18:34:34.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.227394+0000", 2024-09-10T18:34:34.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92965577099999996 2024-09-10T18:34:34.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.227483+0000", 2024-09-10T18:34:34.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9287000000000001e-05 2024-09-10T18:34:34.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:34:34.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.227548+0000", 2024-09-10T18:34:34.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5151e-05 2024-09-10T18:34:34.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.238564+0000", 2024-09-10T18:34:34.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011016336999999999 2024-09-10T18:34:34.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244614+0000", 2024-09-10T18:34:34.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0060498039999999998 2024-09-10T18:34:34.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245259+0000", 2024-09-10T18:34:34.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00064463900000000002 2024-09-10T18:34:34.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245284+0000", 2024-09-10T18:34:34.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5096000000000002e-05 2024-09-10T18:34:34.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245315+0000", 2024-09-10T18:34:34.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0988999999999998e-05 2024-09-10T18:34:34.481 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.481 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.481 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.482 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192546 7.1 7:8c0c27ca:::benchmark_data_smithi028_77856_object71446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:34:34.482 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.351141+0000", 2024-09-10T18:34:34.482 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 476.06886331999999, 2024-09-10T18:34:34.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.89439569500000005, 2024-09-10T18:34:34.482 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.482 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.482 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.482 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.482 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.482 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192546 2024-09-10T18:34:34.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.482 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.351141+0000", 2024-09-10T18:34:34.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.351141+0000", 2024-09-10T18:34:34.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.351139+0000", 2024-09-10T18:34:34.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:34:34.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.351149+0000", 2024-09-10T18:34:34.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0575e-05 2024-09-10T18:34:34.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.351160+0000", 2024-09-10T18:34:34.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.094e-05 2024-09-10T18:34:34.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.351166+0000", 2024-09-10T18:34:34.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7810000000000002e-06 2024-09-10T18:34:34.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.351206+0000", 2024-09-10T18:34:34.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9694000000000003e-05 2024-09-10T18:34:34.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:34:34.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.351215+0000", 2024-09-10T18:34:34.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2189999999999993e-06 2024-09-10T18:34:34.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.228532+0000", 2024-09-10T18:34:34.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.87731700499999998 2024-09-10T18:34:34.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.228612+0000", 2024-09-10T18:34:34.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0439999999999996e-05 2024-09-10T18:34:34.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:34:34.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.228725+0000", 2024-09-10T18:34:34.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011289 2024-09-10T18:34:34.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.238572+0000", 2024-09-10T18:34:34.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098468070000000008 2024-09-10T18:34:34.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244766+0000", 2024-09-10T18:34:34.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061940500000000004 2024-09-10T18:34:34.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245469+0000", 2024-09-10T18:34:34.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00070266000000000003 2024-09-10T18:34:34.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245502+0000", 2024-09-10T18:34:34.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3154999999999997e-05 2024-09-10T18:34:34.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245536+0000", 2024-09-10T18:34:34.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4672999999999998e-05 2024-09-10T18:34:34.489 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.489 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.489 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.489 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192567 7.1 7:827122b7:::benchmark_data_smithi028_77856_object71467:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:34:34.489 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.367207+0000", 2024-09-10T18:34:34.489 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 476.05279724299999, 2024-09-10T18:34:34.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.87840781999999995, 2024-09-10T18:34:34.489 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.489 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.489 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.490 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.490 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.490 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192567 2024-09-10T18:34:34.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.490 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.367207+0000", 2024-09-10T18:34:34.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.367207+0000", 2024-09-10T18:34:34.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.367204+0000", 2024-09-10T18:34:34.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:34:34.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.367214+0000", 2024-09-10T18:34:34.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0013000000000001e-05 2024-09-10T18:34:34.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.367230+0000", 2024-09-10T18:34:34.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6504e-05 2024-09-10T18:34:34.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.367238+0000", 2024-09-10T18:34:34.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6599999999999995e-06 2024-09-10T18:34:34.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.367294+0000", 2024-09-10T18:34:34.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5603999999999998e-05 2024-09-10T18:34:34.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:34:34.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.367301+0000", 2024-09-10T18:34:34.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9650000000000002e-06 2024-09-10T18:34:34.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.230432+0000", 2024-09-10T18:34:34.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.86313160700000002 2024-09-10T18:34:34.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.230540+0000", 2024-09-10T18:34:34.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107299 2024-09-10T18:34:34.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:34:34.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.230656+0000", 2024-09-10T18:34:34.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116549 2024-09-10T18:34:34.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.238579+0000", 2024-09-10T18:34:34.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0079227970000000005 2024-09-10T18:34:34.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244792+0000", 2024-09-10T18:34:34.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0062134 2024-09-10T18:34:34.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245567+0000", 2024-09-10T18:34:34.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00077483000000000001 2024-09-10T18:34:34.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245593+0000", 2024-09-10T18:34:34.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6061e-05 2024-09-10T18:34:34.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245615+0000", 2024-09-10T18:34:34.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1498e-05 2024-09-10T18:34:34.496 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.496 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.496 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.496 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192573 7.1 7:84aa733b:::benchmark_data_smithi028_77856_object71473:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:34:34.496 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.372364+0000", 2024-09-10T18:34:34.496 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 476.04764018399999, 2024-09-10T18:34:34.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.87491753800000005, 2024-09-10T18:34:34.496 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.496 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.496 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.497 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.497 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.497 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192573 2024-09-10T18:34:34.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.497 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.372364+0000", 2024-09-10T18:34:34.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.372364+0000", 2024-09-10T18:34:34.498 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:34.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.372362+0000", 2024-09-10T18:34:34.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:34:34.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.372370+0000", 2024-09-10T18:34:34.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6400000000000003e-06 2024-09-10T18:34:34.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.372385+0000", 2024-09-10T18:34:34.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4629e-05 2024-09-10T18:34:34.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.372391+0000", 2024-09-10T18:34:34.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.28e-06 2024-09-10T18:34:34.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.372440+0000", 2024-09-10T18:34:34.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9190000000000002e-05 2024-09-10T18:34:34.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:34:34.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.372448+0000", 2024-09-10T18:34:34.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2640000000000002e-06 2024-09-10T18:34:34.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.232611+0000", 2024-09-10T18:34:34.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.86016320099999999 2024-09-10T18:34:34.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.232689+0000", 2024-09-10T18:34:34.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8687000000000001e-05 2024-09-10T18:34:34.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:34:34.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.232757+0000", 2024-09-10T18:34:34.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7353000000000004e-05 2024-09-10T18:34:34.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.503 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.241251+0000", 2024-09-10T18:34:34.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0084944689999999993 2024-09-10T18:34:34.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.503 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244818+0000", 2024-09-10T18:34:34.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035666539999999998 2024-09-10T18:34:34.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.247229+0000", 2024-09-10T18:34:34.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024109600000000002 2024-09-10T18:34:34.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.247261+0000", 2024-09-10T18:34:34.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2400000000000001e-05 2024-09-10T18:34:34.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.247281+0000", 2024-09-10T18:34:34.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0166999999999999e-05 2024-09-10T18:34:34.504 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.504 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.505 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.505 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192579 7.1 7:86f625e9:::benchmark_data_smithi028_77856_object71479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:34:34.505 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.377253+0000", 2024-09-10T18:34:34.505 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 476.04275153100002, 2024-09-10T18:34:34.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.87009756199999999, 2024-09-10T18:34:34.505 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.505 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.505 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.505 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.505 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.505 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192579 2024-09-10T18:34:34.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.506 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.377253+0000", 2024-09-10T18:34:34.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.377253+0000", 2024-09-10T18:34:34.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.377250+0000", 2024-09-10T18:34:34.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:34:34.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.377261+0000", 2024-09-10T18:34:34.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0648999999999999e-05 2024-09-10T18:34:34.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.377277+0000", 2024-09-10T18:34:34.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6158e-05 2024-09-10T18:34:34.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.377283+0000", 2024-09-10T18:34:34.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7370000000000003e-06 2024-09-10T18:34:34.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.377362+0000", 2024-09-10T18:34:34.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9010000000000004e-05 2024-09-10T18:34:34.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:34:34.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.377370+0000", 2024-09-10T18:34:34.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.79e-06 2024-09-10T18:34:34.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.234019+0000", 2024-09-10T18:34:34.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.85664904799999997 2024-09-10T18:34:34.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.234086+0000", 2024-09-10T18:34:34.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7403000000000005e-05 2024-09-10T18:34:34.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:34:34.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.234153+0000", 2024-09-10T18:34:34.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6556000000000004e-05 2024-09-10T18:34:34.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.241258+0000", 2024-09-10T18:34:34.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007105218 2024-09-10T18:34:34.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244949+0000", 2024-09-10T18:34:34.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003690652 2024-09-10T18:34:34.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.247309+0000", 2024-09-10T18:34:34.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023603790000000001 2024-09-10T18:34:34.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.247332+0000", 2024-09-10T18:34:34.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3147999999999999e-05 2024-09-10T18:34:34.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.247350+0000", 2024-09-10T18:34:34.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.808e-05 2024-09-10T18:34:34.511 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.512 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.512 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.512 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192599 7.1 7:83571491:::benchmark_data_smithi028_77856_object71499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:34:34.512 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.395289+0000", 2024-09-10T18:34:34.512 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 476.02471492500001, 2024-09-10T18:34:34.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.85726006200000004, 2024-09-10T18:34:34.512 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.512 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.512 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.512 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.512 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.513 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192599 2024-09-10T18:34:34.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.513 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.395289+0000", 2024-09-10T18:34:34.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.395289+0000", 2024-09-10T18:34:34.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.395287+0000", 2024-09-10T18:34:34.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:34:34.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.395295+0000", 2024-09-10T18:34:34.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3599999999999996e-06 2024-09-10T18:34:34.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.395308+0000", 2024-09-10T18:34:34.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3115999999999999e-05 2024-09-10T18:34:34.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.395315+0000", 2024-09-10T18:34:34.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5849999999999996e-06 2024-09-10T18:34:34.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.395386+0000", 2024-09-10T18:34:34.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1477999999999996e-05 2024-09-10T18:34:34.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:34:34.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.395395+0000", 2024-09-10T18:34:34.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2719999999999997e-06 2024-09-10T18:34:34.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.235362+0000", 2024-09-10T18:34:34.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.83996730600000002 2024-09-10T18:34:34.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.235439+0000", 2024-09-10T18:34:34.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6870000000000001e-05 2024-09-10T18:34:34.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:34:34.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.235501+0000", 2024-09-10T18:34:34.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2680000000000003e-05 2024-09-10T18:34:34.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.241264+0000", 2024-09-10T18:34:34.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0057622319999999999 2024-09-10T18:34:34.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244990+0000", 2024-09-10T18:34:34.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0037264389999999998 2024-09-10T18:34:34.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252467+0000", 2024-09-10T18:34:34.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0074771150000000003 2024-09-10T18:34:34.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252511+0000", 2024-09-10T18:34:34.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3251999999999999e-05 2024-09-10T18:34:34.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252549+0000", 2024-09-10T18:34:34.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8717000000000002e-05 2024-09-10T18:34:34.518 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.519 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.519 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.519 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192616 7.1 7:842d8045:::benchmark_data_smithi028_77856_object71516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:34:34.519 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.412146+0000", 2024-09-10T18:34:34.519 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 476.00785829099999, 2024-09-10T18:34:34.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.84049511600000004, 2024-09-10T18:34:34.519 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.519 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.519 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.519 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.519 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.519 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192616 2024-09-10T18:34:34.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.520 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.412146+0000", 2024-09-10T18:34:34.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.412146+0000", 2024-09-10T18:34:34.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.412143+0000", 2024-09-10T18:34:34.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:34:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.412151+0000", 2024-09-10T18:34:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7749999999999993e-06 2024-09-10T18:34:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.412166+0000", 2024-09-10T18:34:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4347e-05 2024-09-10T18:34:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.412171+0000", 2024-09-10T18:34:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7599999999999999e-06 2024-09-10T18:34:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.412207+0000", 2024-09-10T18:34:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5488000000000003e-05 2024-09-10T18:34:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:34:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.412214+0000", 2024-09-10T18:34:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3010000000000001e-06 2024-09-10T18:34:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.236816+0000", 2024-09-10T18:34:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.82460156100000004 2024-09-10T18:34:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.236904+0000", 2024-09-10T18:34:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8072000000000007e-05 2024-09-10T18:34:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:34:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.236968+0000", 2024-09-10T18:34:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4579999999999995e-05 2024-09-10T18:34:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244125+0000", 2024-09-10T18:34:34.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0071561769999999997 2024-09-10T18:34:34.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245213+0000", 2024-09-10T18:34:34.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001088971 2024-09-10T18:34:34.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252590+0000", 2024-09-10T18:34:34.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0073765030000000004 2024-09-10T18:34:34.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252618+0000", 2024-09-10T18:34:34.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8408e-05 2024-09-10T18:34:34.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252641+0000", 2024-09-10T18:34:34.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2498999999999999e-05 2024-09-10T18:34:34.525 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.525 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.525 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.525 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192633 7.1 7:869757c6:::benchmark_data_smithi028_77856_object71533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:34:34.525 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.436867+0000", 2024-09-10T18:34:34.526 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 475.98313675899999, 2024-09-10T18:34:34.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.815892744, 2024-09-10T18:34:34.526 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.526 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.526 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.526 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.526 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.526 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192633 2024-09-10T18:34:34.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.526 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.526 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.526 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.436867+0000", 2024-09-10T18:34:34.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.436867+0000", 2024-09-10T18:34:34.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.436865+0000", 2024-09-10T18:34:34.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:34:34.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.436878+0000", 2024-09-10T18:34:34.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.237e-05 2024-09-10T18:34:34.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.436890+0000", 2024-09-10T18:34:34.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1688999999999999e-05 2024-09-10T18:34:34.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.436894+0000", 2024-09-10T18:34:34.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5390000000000001e-06 2024-09-10T18:34:34.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.436930+0000", 2024-09-10T18:34:34.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6236999999999999e-05 2024-09-10T18:34:34.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:34:34.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.436937+0000", 2024-09-10T18:34:34.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7689999999999999e-06 2024-09-10T18:34:34.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.238619+0000", 2024-09-10T18:34:34.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80168163400000003 2024-09-10T18:34:34.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.238725+0000", 2024-09-10T18:34:34.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010587 2024-09-10T18:34:34.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:34:34.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.238846+0000", 2024-09-10T18:34:34.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000121402 2024-09-10T18:34:34.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244131+0000", 2024-09-10T18:34:34.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005285334 2024-09-10T18:34:34.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245237+0000", 2024-09-10T18:34:34.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011056600000000001 2024-09-10T18:34:34.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252719+0000", 2024-09-10T18:34:34.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007482201 2024-09-10T18:34:34.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252743+0000", 2024-09-10T18:34:34.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4221999999999999e-05 2024-09-10T18:34:34.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.252760+0000", 2024-09-10T18:34:34.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6670000000000001e-05 2024-09-10T18:34:34.532 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.532 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.532 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.532 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192640 7.1 7:8b08cdd5:::benchmark_data_smithi028_77856_object71540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:34:34.532 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.443619+0000", 2024-09-10T18:34:34.533 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 475.97638516500001, 2024-09-10T18:34:34.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.82085115399999997, 2024-09-10T18:34:34.533 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.533 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.533 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.533 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.533 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.533 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192640 2024-09-10T18:34:34.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.533 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.443619+0000", 2024-09-10T18:34:34.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.443619+0000", 2024-09-10T18:34:34.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.443616+0000", 2024-09-10T18:34:34.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:34:34.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.443625+0000", 2024-09-10T18:34:34.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8899999999999996e-06 2024-09-10T18:34:34.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.443641+0000", 2024-09-10T18:34:34.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6076999999999999e-05 2024-09-10T18:34:34.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.443648+0000", 2024-09-10T18:34:34.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9519999999999997e-06 2024-09-10T18:34:34.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.443687+0000", 2024-09-10T18:34:34.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9224000000000003e-05 2024-09-10T18:34:34.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:34:34.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.443695+0000", 2024-09-10T18:34:34.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4669999999999998e-06 2024-09-10T18:34:34.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.240084+0000", 2024-09-10T18:34:34.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.79638984999999995 2024-09-10T18:34:34.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.240166+0000", 2024-09-10T18:34:34.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1206999999999997e-05 2024-09-10T18:34:34.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:34:34.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.240231+0000", 2024-09-10T18:34:34.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5543000000000006e-05 2024-09-10T18:34:34.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244229+0000", 2024-09-10T18:34:34.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039977229999999999 2024-09-10T18:34:34.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.538 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.538 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245370+0000", 2024-09-10T18:34:34.538 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011409199999999999 2024-09-10T18:34:34.538 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.538 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.538 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264392+0000", 2024-09-10T18:34:34.538 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01902216 2024-09-10T18:34:34.538 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.538 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.538 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264429+0000", 2024-09-10T18:34:34.538 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6761000000000003e-05 2024-09-10T18:34:34.538 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.539 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.539 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264470+0000", 2024-09-10T18:34:34.539 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1295999999999999e-05 2024-09-10T18:34:34.539 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.539 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.539 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.539 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.539 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.539 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192653 7.1 7:8f4ef615:::benchmark_data_smithi028_77856_object71553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:34:34.539 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.468961+0000", 2024-09-10T18:34:34.539 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 475.95104291500002, 2024-09-10T18:34:34.539 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.79562034500000001, 2024-09-10T18:34:34.539 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.539 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.539 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.540 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.540 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.540 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192653 2024-09-10T18:34:34.540 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.540 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.540 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.540 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.540 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.468961+0000", 2024-09-10T18:34:34.540 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.540 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.540 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.540 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.540 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.468961+0000", 2024-09-10T18:34:34.540 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.541 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.541 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.541 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.541 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.468958+0000", 2024-09-10T18:34:34.541 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:34:34.541 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.541 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.541 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.541 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.468968+0000", 2024-09-10T18:34:34.541 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0129000000000001e-05 2024-09-10T18:34:34.541 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.541 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.468984+0000", 2024-09-10T18:34:34.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5835999999999999e-05 2024-09-10T18:34:34.542 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.542 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.468992+0000", 2024-09-10T18:34:34.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7319999999999998e-06 2024-09-10T18:34:34.542 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.542 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.469044+0000", 2024-09-10T18:34:34.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2101999999999997e-05 2024-09-10T18:34:34.543 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.543 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.543 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:34:34.543 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.469055+0000", 2024-09-10T18:34:34.543 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0757999999999999e-05 2024-09-10T18:34:34.543 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.543 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.543 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.543 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.241306+0000", 2024-09-10T18:34:34.543 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.77225105699999996 2024-09-10T18:34:34.543 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.543 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.543 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.543 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.241417+0000", 2024-09-10T18:34:34.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011148499999999999 2024-09-10T18:34:34.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.544 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:34:34.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.241494+0000", 2024-09-10T18:34:34.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6364000000000003e-05 2024-09-10T18:34:34.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.544 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244234+0000", 2024-09-10T18:34:34.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027403029999999999 2024-09-10T18:34:34.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.544 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.545 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245402+0000", 2024-09-10T18:34:34.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001168475 2024-09-10T18:34:34.545 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.545 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.545 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.545 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264519+0000", 2024-09-10T18:34:34.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019116528000000001 2024-09-10T18:34:34.545 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.545 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.545 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.545 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264560+0000", 2024-09-10T18:34:34.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1043e-05 2024-09-10T18:34:34.545 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.545 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.545 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.546 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264582+0000", 2024-09-10T18:34:34.546 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1518999999999998e-05 2024-09-10T18:34:34.546 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.546 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.546 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.546 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.546 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.546 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:192692 7.1 7:844b92c8:::benchmark_data_smithi028_77856_object71592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-10T18:34:34.546 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:38.589431+0000", 2024-09-10T18:34:34.546 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 475.83057266200001, 2024-09-10T18:34:34.546 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.67523672300000004, 2024-09-10T18:34:34.546 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.547 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.547 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:34.547 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:34.547 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:34.547 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 192692 2024-09-10T18:34:34.547 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.547 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.547 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.547 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.547 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.589431+0000", 2024-09-10T18:34:34.547 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.547 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.547 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.547 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.547 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.589431+0000", 2024-09-10T18:34:34.548 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.548 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.548 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.548 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.548 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.589430+0000", 2024-09-10T18:34:34.548 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:34:34.548 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.548 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.548 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.548 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.589436+0000", 2024-09-10T18:34:34.548 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7690000000000002e-06 2024-09-10T18:34:34.548 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.548 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.548 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.548 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.589446+0000", 2024-09-10T18:34:34.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0214000000000001e-05 2024-09-10T18:34:34.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.549 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.549 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.589450+0000", 2024-09-10T18:34:34.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2209999999999999e-06 2024-09-10T18:34:34.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.549 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.549 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.589485+0000", 2024-09-10T18:34:34.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5308999999999997e-05 2024-09-10T18:34:34.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:34:34.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:38.589491+0000", 2024-09-10T18:34:34.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7320000000000003e-06 2024-09-10T18:34:34.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.242569+0000", 2024-09-10T18:34:34.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.65307782199999997 2024-09-10T18:34:34.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.242658+0000", 2024-09-10T18:34:34.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8823e-05 2024-09-10T18:34:34.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,6", 2024-09-10T18:34:34.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.242757+0000", 2024-09-10T18:34:34.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8987000000000006e-05 2024-09-10T18:34:34.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:34.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.244448+0000", 2024-09-10T18:34:34.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016910569999999999 2024-09-10T18:34:34.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.552 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.245435+0000", 2024-09-10T18:34:34.552 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00098748499999999997 2024-09-10T18:34:34.552 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.552 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.552 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:34.552 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264619+0000", 2024-09-10T18:34:34.552 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019183861 2024-09-10T18:34:34.552 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.552 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.552 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.552 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264645+0000", 2024-09-10T18:34:34.552 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6288999999999999e-05 2024-09-10T18:34:34.552 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.553 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.553 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.553 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:39.264668+0000", 2024-09-10T18:34:34.553 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2674e-05 2024-09-10T18:34:34.553 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.553 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.553 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.553 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.553 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.553 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.35306.0:188505 9.c e285/246 9:3de7062d:::benchmark_data_smithi028_135703_object53914:head v 285'12513, mlcod=285'12501)", 2024-09-10T18:34:34.553 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:28.039146+0000", 2024-09-10T18:34:34.553 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 6.3808584350000004, 2024-09-10T18:34:34.553 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.048930365000000003, 2024-09-10T18:34:34.554 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.554 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.554 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.554 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.554 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.554 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.039146+0000", 2024-09-10T18:34:34.555 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.039146+0000", 2024-09-10T18:34:34.555 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.039145+0000", 2024-09-10T18:34:34.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:34:34.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.039151+0000", 2024-09-10T18:34:34.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9000000000000003e-06 2024-09-10T18:34:34.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.039158+0000", 2024-09-10T18:34:34.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9319999999999999e-06 2024-09-10T18:34:34.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.039162+0000", 2024-09-10T18:34:34.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.053e-06 2024-09-10T18:34:34.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.060894+0000", 2024-09-10T18:34:34.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021732656999999999 2024-09-10T18:34:34.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.060940+0000", 2024-09-10T18:34:34.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5451999999999998e-05 2024-09-10T18:34:34.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.088040+0000", 2024-09-10T18:34:34.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.027100099999999998 2024-09-10T18:34:34.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.088076+0000", 2024-09-10T18:34:34.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6155999999999997e-05 2024-09-10T18:34:34.559 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.559 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.559 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.559 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.35306.0:188528 9.c e285/246 9:3f97010b:::benchmark_data_smithi028_135703_object53937:head v 285'12514, mlcod=285'12501)", 2024-09-10T18:34:34.559 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:28.041212+0000", 2024-09-10T18:34:34.559 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 6.3787923830000004, 2024-09-10T18:34:34.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.048885603999999999, 2024-09-10T18:34:34.559 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:34.559 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:34.559 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:34.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:34.560 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.041212+0000", 2024-09-10T18:34:34.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:34.560 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.041212+0000", 2024-09-10T18:34:34.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:34.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:34.560 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.041210+0000", 2024-09-10T18:34:34.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:34:34.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:34.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.041217+0000", 2024-09-10T18:34:34.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0489999999999998e-06 2024-09-10T18:34:34.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:34.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.041228+0000", 2024-09-10T18:34:34.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0599e-05 2024-09-10T18:34:34.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:34.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.041233+0000", 2024-09-10T18:34:34.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2290000000000002e-06 2024-09-10T18:34:34.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:34.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.064920+0000", 2024-09-10T18:34:34.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023687066999999999 2024-09-10T18:34:34.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:34.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.064946+0000", 2024-09-10T18:34:34.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5480999999999999e-05 2024-09-10T18:34:34.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:34.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.090069+0000", 2024-09-10T18:34:34.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025122756 2024-09-10T18:34:34.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:34.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:34.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:34.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:28.090097+0000", 2024-09-10T18:34:34.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8802999999999998e-05 2024-09-10T18:34:34.563 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.563 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.563 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.563 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:34.563 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:34.563 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:34:34.571 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:34:34.583 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:34.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:34.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:34.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:34.723 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:34.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:34.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:34.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:34.841 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:34:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:34:34.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:34.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:34.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:34.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:34.982 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:34.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:35.003 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:34 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-10T18:34:35.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:34 smithi028 ceph-mon[39220]: osdmap e289: 8 total, 8 up, 8 in 2024-09-10T18:34:35.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:34 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-10T18:34:35.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:34 smithi028 ceph-mon[29291]: osdmap e289: 8 total, 8 up, 8 in 2024-09-10T18:34:35.085 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:35.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:34 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-10T18:34:35.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:34 smithi204 ceph-mon[32357]: osdmap e289: 8 total, 8 up, 8 in 2024-09-10T18:34:35.114 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:35.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:35.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:35.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:35.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:35.253 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:35.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:35.354 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:35.384 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:35.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:35.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:35.486 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:35.530 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:35.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:35.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:35.533 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-10T18:34:35.632 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:35.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:35.664 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:35.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:35.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:35.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:35.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:35.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:35.910 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:35.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:35.955 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:35.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:35.962 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:34:35.963 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:34:35.983 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:34:35.992 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:34:35.992 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:34:36.023 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:34:36.023 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:34:36.049 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:34:36.049 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:34:36.049 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:35 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:36.050 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:35 smithi028 ceph-mon[39220]: osdmap e290: 8 total, 8 up, 8 in 2024-09-10T18:34:36.050 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:35 smithi028 ceph-mon[39220]: pgmap v591: 178 pgs: 1 activating, 7 unknown, 2 active+clean+scrubbing+deep, 168 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 2.5 KiB/s rd, 9.2 KiB/s wr, 384 op/s; 6.5 KiB/s, 0 objects/s recovering 2024-09-10T18:34:36.050 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:35 smithi028 ceph-mon[39220]: osdmap e291: 8 total, 8 up, 8 in 2024-09-10T18:34:36.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:35 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:36.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:35 smithi028 ceph-mon[29291]: osdmap e290: 8 total, 8 up, 8 in 2024-09-10T18:34:36.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:35 smithi028 ceph-mon[29291]: pgmap v591: 178 pgs: 1 activating, 7 unknown, 2 active+clean+scrubbing+deep, 168 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 2.5 KiB/s rd, 9.2 KiB/s wr, 384 op/s; 6.5 KiB/s, 0 objects/s recovering 2024-09-10T18:34:36.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:35 smithi028 ceph-mon[29291]: osdmap e291: 8 total, 8 up, 8 in 2024-09-10T18:34:36.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:36.077 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:34:36.077 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:34:36.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:36.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:36.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:35 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:35 smithi204 ceph-mon[32357]: osdmap e290: 8 total, 8 up, 8 in 2024-09-10T18:34:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:35 smithi204 ceph-mon[32357]: pgmap v591: 178 pgs: 1 activating, 7 unknown, 2 active+clean+scrubbing+deep, 168 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 2.5 KiB/s rd, 9.2 KiB/s wr, 384 op/s; 6.5 KiB/s, 0 objects/s recovering 2024-09-10T18:34:36.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:35 smithi204 ceph-mon[32357]: osdmap e291: 8 total, 8 up, 8 in 2024-09-10T18:34:36.112 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:34:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:34:36.141 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:34:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:34:36.170 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:34:36.170 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:34:36.202 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:36.224 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:36.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:36.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:36.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:36.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:36.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:36.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:36.489 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:36.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:36.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:36.591 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:36.620 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:36.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:36.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:36.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:36.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:36.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:36.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:36.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:36.898 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:36.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:36.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:37.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:37.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:37.035 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:37.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:37.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:36 smithi204 ceph-mon[32357]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-10T18:34:37.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:36 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1983393106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:34:37.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:37.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:37.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:37.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:36 smithi028 ceph-mon[39220]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-10T18:34:37.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:36 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1983393106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:34:37.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:36 smithi028 ceph-mon[29291]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-10T18:34:37.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:36 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1983393106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:34:37.277 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:37.307 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:37.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:37.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:37.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:37.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:37.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:37.548 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:37.583 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:37.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:37.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:37.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:37.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:37.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:37.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:37.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:37.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:37.956 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:37.987 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:37.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:37.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:38.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:37 smithi028 ceph-mon[29291]: pgmap v593: 178 pgs: 1 activating, 2 active+clean+scrubbing+deep, 175 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 9.7 KiB/s rd, 61 KiB/s wr, 981 op/s; 66 KiB/s, 0 objects/s recovering 2024-09-10T18:34:38.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:37 smithi028 ceph-mon[29291]: 9.6 deep-scrub ok 2024-09-10T18:34:38.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:37 smithi028 ceph-mon[39220]: pgmap v593: 178 pgs: 1 activating, 2 active+clean+scrubbing+deep, 175 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 9.7 KiB/s rd, 61 KiB/s wr, 981 op/s; 66 KiB/s, 0 objects/s recovering 2024-09-10T18:34:38.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:37 smithi028 ceph-mon[39220]: 9.6 deep-scrub ok 2024-09-10T18:34:38.042 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:37 smithi204 ceph-mon[32357]: pgmap v593: 178 pgs: 1 activating, 2 active+clean+scrubbing+deep, 175 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 9.7 KiB/s rd, 61 KiB/s wr, 981 op/s; 66 KiB/s, 0 objects/s recovering 2024-09-10T18:34:38.042 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:37 smithi204 ceph-mon[32357]: 9.6 deep-scrub ok 2024-09-10T18:34:38.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:38.122 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:38.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:38.190 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:34:38.190 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:34:38.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:38.257 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:38.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:38.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:38.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:38.394 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:38.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:38.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:38.496 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:38.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:38.533 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:38.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:38.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:38.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:38.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:38.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:38.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:38.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:38.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:38.952 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-10T18:34:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:38.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:38.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:39.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:39.090 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:39.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:39.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:39.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:39.223 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:39.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:39.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:39.325 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:39.369 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:39.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:39.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:39.377 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:34:39.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:39.504 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:39.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:39.565 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:34:39.565 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:34:39.565 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_2 2024-09-10T18:34:39.565 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_2 by 10 min 8 2024-09-10T18:34:39.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:34:39.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:39.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:39.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:39.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:39.780 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:39.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:34:39.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:39.920 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:39.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:39.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:39 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:34:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:34:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:39 smithi028 ceph-mon[29291]: pgmap v594: 178 pgs: 1 activating, 2 active+clean+scrubbing+deep, 175 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 6.8 KiB/s rd, 43 KiB/s wr, 690 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-10T18:34:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:39 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/710268037' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:34:40.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:39 smithi028 ceph-mon[29291]: from='client.25444 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:34:40.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:39 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:34:40.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:34:40.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:39 smithi028 ceph-mon[39220]: pgmap v594: 178 pgs: 1 activating, 2 active+clean+scrubbing+deep, 175 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 6.8 KiB/s rd, 43 KiB/s wr, 690 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-10T18:34:40.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:39 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/710268037' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:34:40.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:39 smithi028 ceph-mon[39220]: from='client.25444 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:34:40.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:40.064 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:40.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:40.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:40.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:34:40.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:39 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:34:40.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:34:40.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:39 smithi204 ceph-mon[32357]: pgmap v594: 178 pgs: 1 activating, 2 active+clean+scrubbing+deep, 175 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 6.8 KiB/s rd, 43 KiB/s wr, 690 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-10T18:34:40.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:39 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/710268037' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:34:40.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:39 smithi204 ceph-mon[32357]: from='client.25444 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:34:40.166 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:40.206 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:40.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:40.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:40.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:40.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:40.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:40.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:40.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:40.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:40.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:40.626 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:40.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:40.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:40.728 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:40.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:40.763 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:40.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:40.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:40 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:34:40.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:40 smithi204 ceph-mon[32357]: from='client.25444 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:34:40.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:40 smithi204 ceph-mon[32357]: osdmap e292: 8 total, 8 up, 8 in 2024-09-10T18:34:40.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:40 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:34:40.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:40 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:34:40.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:40.904 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:40.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:41.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:41.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:41.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:41.152 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:41.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:41.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:41.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:40 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:34:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:40 smithi028 ceph-mon[39220]: from='client.25444 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:34:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:40 smithi028 ceph-mon[39220]: osdmap e292: 8 total, 8 up, 8 in 2024-09-10T18:34:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:40 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:34:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:40 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:34:41.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:40 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:34:41.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:40 smithi028 ceph-mon[29291]: from='client.25444 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:34:41.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:40 smithi028 ceph-mon[29291]: osdmap e292: 8 total, 8 up, 8 in 2024-09-10T18:34:41.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:40 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:34:41.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:40 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:34:41.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:41.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:41.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:41.358 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:34:41.359 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:34:41.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:41.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:41.467 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:41.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:41.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:41.636 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:41.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:41.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:34:41.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:41.747 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:34:41.753 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:34:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:34:41] "GET /metrics HTTP/1.1" 200 224592 "" "Prometheus/2.43.0" 2024-09-10T18:34:41.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:41.765 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:41.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:41.866 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:41.887 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:41.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:41.989 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:42.024 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:42.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:42.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:42.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:41 smithi028 ceph-mon[39220]: osdmap e293: 8 total, 8 up, 8 in 2024-09-10T18:34:42.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:41 smithi028 ceph-mon[39220]: pgmap v597: 178 pgs: 178 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 6.0 KiB/s rd, 37 KiB/s wr, 1.33k op/s; 42 KiB/s, 0 objects/s recovering 2024-09-10T18:34:42.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:41 smithi028 ceph-mon[29291]: osdmap e293: 8 total, 8 up, 8 in 2024-09-10T18:34:42.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:41 smithi028 ceph-mon[29291]: pgmap v597: 178 pgs: 178 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 6.0 KiB/s rd, 37 KiB/s wr, 1.33k op/s; 42 KiB/s, 0 objects/s recovering 2024-09-10T18:34:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:41 smithi204 ceph-mon[32357]: osdmap e293: 8 total, 8 up, 8 in 2024-09-10T18:34:42.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:41 smithi204 ceph-mon[32357]: pgmap v597: 178 pgs: 178 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 6.0 KiB/s rd, 37 KiB/s wr, 1.33k op/s; 42 KiB/s, 0 objects/s recovering 2024-09-10T18:34:42.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:42.163 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:42.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:42.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:42.264 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:42.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:42.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:42.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:42.401 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:42.444 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:42.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:42.503 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:34:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:34:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:34:42.543 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-10T18:34:42.545 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:42.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:42.582 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:42.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:42.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:42.718 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:42.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:42.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:42.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:42.825 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:42 smithi204 ceph-mon[32357]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-10T18:34:42.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:42.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:42.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:42.945 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:34:42.965 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:43.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:42 smithi028 ceph-mon[29291]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-10T18:34:43.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:42 smithi028 ceph-mon[39220]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-10T18:34:43.006 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:43.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:43.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:43.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:43.148 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:43.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:43.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:43.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:43.288 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:43.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:43.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:43.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:43.422 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:43.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:43.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:43.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:34:43.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:43.579 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:43.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:43.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:43.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:43.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:43.718 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:43.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:43.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:43.842 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:43.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:43.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:43.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:43.978 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:43.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:43.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:44.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:44.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:44.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:44.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:44.103 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:43 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:34:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:43 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:34:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:43 smithi204 ceph-mon[32357]: osdmap e294: 8 total, 8 up, 8 in 2024-09-10T18:34:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:43 smithi204 ceph-mon[32357]: pgmap v599: 178 pgs: 178 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 961 op/s 2024-09-10T18:34:44.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:44.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:43 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:34:44.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:43 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:34:44.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:43 smithi028 ceph-mon[29291]: osdmap e294: 8 total, 8 up, 8 in 2024-09-10T18:34:44.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:43 smithi028 ceph-mon[29291]: pgmap v599: 178 pgs: 178 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 961 op/s 2024-09-10T18:34:44.268 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:43 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:34:44.268 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:43 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:34:44.269 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:43 smithi028 ceph-mon[39220]: osdmap e294: 8 total, 8 up, 8 in 2024-09-10T18:34:44.269 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:43 smithi028 ceph-mon[39220]: pgmap v599: 178 pgs: 178 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 961 op/s 2024-09-10T18:34:44.291 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:44.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:44.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:44.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:44.435 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:44.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:44.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:44.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:44.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:44.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:44.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:44.736 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:44.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:44.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:44.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:44.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:34:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:34:44.874 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:44.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:44.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:44.976 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:45.008 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:45.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:45.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:45.109 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:45.153 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:45.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:45.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:45.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:45.276 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:45.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:45.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:45.376 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:34:45.376 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:34:45.378 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:45.403 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:45.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:45.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:45.505 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:45.536 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:45.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:45.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:45.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:45.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:45.674 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:45.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:45.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:45.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:45.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:45.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:45.918 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:45.951 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:45.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:45.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:46.053 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:46.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:46.089 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:46.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:46.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-10T18:34:46.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:46.202 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:34:46.256 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:46.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:46.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:46 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:46 smithi204 ceph-mon[32357]: pgmap v600: 178 pgs: 178 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-10T18:34:46.360 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:46.396 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:46.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:46.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:46.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:46 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:46.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:46 smithi028 ceph-mon[39220]: pgmap v600: 178 pgs: 178 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-10T18:34:46.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:46 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:46.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:46 smithi028 ceph-mon[29291]: pgmap v600: 178 pgs: 178 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-10T18:34:46.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:46.569 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:46.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:46.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:46.571 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:34:46.671 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:46.706 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:46.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:46.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:46.808 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:46.833 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:46.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:46.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:46.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:46.979 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:46.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:46.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:47.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:47.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:47.126 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:47.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:47.226 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:47.255 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:47.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:47.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:47.271 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:34:47.272 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:34:47.310 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:34:47.310 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:34:47.342 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:34:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:34:47.356 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:47.378 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:47.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:47.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:47.399 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:34:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:34:47.435 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:34:47.435 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:34:47.463 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:34:47.463 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:34:47.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:47.499 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:34:47.499 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:34:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:47.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:47.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:47.529 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:34:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:34:47.561 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:34:47.561 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:34:47.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:47.659 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:47.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:47.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:47.796 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:47.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:47.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:47.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:47.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:47.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:48.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:48.083 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:48.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:48.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:48.191 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:48 smithi204 ceph-mon[32357]: pgmap v601: 178 pgs: 178 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.68k op/s 2024-09-10T18:34:48.191 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:48 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3666204843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:34:48.191 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:48 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:34:48.191 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:48 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:34:48.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:48.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:48.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:48.334 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:48.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:48.375 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:48.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:48.408 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:48 smithi028 ceph-mon[39220]: pgmap v601: 178 pgs: 178 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.68k op/s 2024-09-10T18:34:48.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:48 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3666204843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:34:48.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:48 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:34:48.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:48 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:34:48.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:48 smithi028 ceph-mon[29291]: pgmap v601: 178 pgs: 178 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.68k op/s 2024-09-10T18:34:48.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:48 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3666204843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:34:48.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:48 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:34:48.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:48 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:34:48.477 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:48.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:48.511 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:48.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:48.612 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:48.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:48.651 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:48.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:48.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:48.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:48.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:48.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:48.887 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:48.928 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:48.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:48.938 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:34:48.938 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:34:48.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.938 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:211698 9.3 9:c2523ddd:::benchmark_data_smithi028_135703_object77107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-09-10T18:34:48.938 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:48.929818+0000", 2024-09-10T18:34:48.938 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0053924489999999997, 2024-09-10T18:34:48.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054227800000000003, 2024-09-10T18:34:48.938 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:48.938 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:34:48.938 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:48.939 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:34:48.939 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:48.939 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 211698 2024-09-10T18:34:48.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.939 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:48.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:48.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.929818+0000", 2024-09-10T18:34:48.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:48.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:48.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.929818+0000", 2024-09-10T18:34:48.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:48.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:48.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.929815+0000", 2024-09-10T18:34:48.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:34:48.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:48.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.929823+0000", 2024-09-10T18:34:48.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9449999999999993e-06 2024-09-10T18:34:48.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:48.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.929837+0000", 2024-09-10T18:34:48.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3293e-05 2024-09-10T18:34:48.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:48.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.929852+0000", 2024-09-10T18:34:48.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5355000000000001e-05 2024-09-10T18:34:48.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:48.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.929932+0000", 2024-09-10T18:34:48.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0069999999999997e-05 2024-09-10T18:34:48.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:48.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.930132+0000", 2024-09-10T18:34:48.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000199901 2024-09-10T18:34:48.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:34:48.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.930271+0000", 2024-09-10T18:34:48.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000139393 2024-09-10T18:34:48.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:48.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.932528+0000", 2024-09-10T18:34:48.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022565279999999998 2024-09-10T18:34:48.943 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:48.943 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:48.943 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:48.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.943 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:211705 9.3 9:c9217449:::benchmark_data_smithi028_135703_object77114:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-09-10T18:34:48.943 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:48.932273+0000", 2024-09-10T18:34:48.943 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0029369940000000001, 2024-09-10T18:34:48.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030572450000000001, 2024-09-10T18:34:48.943 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:48.943 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:34:48.943 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:48.944 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:34:48.944 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:48.944 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 211705 2024-09-10T18:34:48.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.944 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:48.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:48.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.932273+0000", 2024-09-10T18:34:48.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:48.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:48.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.932273+0000", 2024-09-10T18:34:48.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:48.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:48.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.932271+0000", 2024-09-10T18:34:48.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:34:48.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:48.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.932278+0000", 2024-09-10T18:34:48.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7109999999999998e-06 2024-09-10T18:34:48.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:48.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.932293+0000", 2024-09-10T18:34:48.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4626e-05 2024-09-10T18:34:48.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:48.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.932300+0000", 2024-09-10T18:34:48.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8589999999999999e-06 2024-09-10T18:34:48.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:48.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.932570+0000", 2024-09-10T18:34:48.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027037200000000001 2024-09-10T18:34:48.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:48.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.932677+0000", 2024-09-10T18:34:48.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107359 2024-09-10T18:34:48.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:34:48.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.932786+0000", 2024-09-10T18:34:48.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108927 2024-09-10T18:34:48.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:48.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.935308+0000", 2024-09-10T18:34:48.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002521372 2024-09-10T18:34:48.947 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:48.948 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:48.948 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:48.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.948 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.35306.0:211703 9.a e295/246 9:55e25512:::benchmark_data_smithi028_135703_object77112:head v 295'13385, mlcod=295'13383)", 2024-09-10T18:34:48.948 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:48.933454+0000", 2024-09-10T18:34:48.948 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.001755796, 2024-09-10T18:34:48.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019569510000000002, 2024-09-10T18:34:48.948 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:48.948 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:34:48.948 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:48.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:48.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.933454+0000", 2024-09-10T18:34:48.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:48.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:48.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.933454+0000", 2024-09-10T18:34:48.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:48.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:48.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.933453+0000", 2024-09-10T18:34:48.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:34:48.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:48.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.933461+0000", 2024-09-10T18:34:48.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1300000000000001e-06 2024-09-10T18:34:48.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:48.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.933475+0000", 2024-09-10T18:34:48.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4218e-05 2024-09-10T18:34:48.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:48.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.933481+0000", 2024-09-10T18:34:48.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4070000000000001e-06 2024-09-10T18:34:48.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:48.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.933544+0000", 2024-09-10T18:34:48.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3853e-05 2024-09-10T18:34:48.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:48.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.933562+0000", 2024-09-10T18:34:48.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7941e-05 2024-09-10T18:34:48.951 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:48.951 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:48.951 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:48.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.952 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.35306.0:211702 9.c e295/246 9:3866f421:::benchmark_data_smithi028_135703_object77111:head v 295'16675, mlcod=295'16672)", 2024-09-10T18:34:48.952 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:48.934461+0000", 2024-09-10T18:34:48.952 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.00074912700000000004, 2024-09-10T18:34:48.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010099169999999999, 2024-09-10T18:34:48.952 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:48.952 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:34:48.952 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:48.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:48.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.934461+0000", 2024-09-10T18:34:48.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:48.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:48.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.934461+0000", 2024-09-10T18:34:48.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:48.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:48.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.934460+0000", 2024-09-10T18:34:48.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:34:48.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:48.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.934467+0000", 2024-09-10T18:34:48.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9269999999999999e-06 2024-09-10T18:34:48.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:48.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.934476+0000", 2024-09-10T18:34:48.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5959999999999993e-06 2024-09-10T18:34:48.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:48.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.934481+0000", 2024-09-10T18:34:48.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3159999999999996e-06 2024-09-10T18:34:48.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:48.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.934540+0000", 2024-09-10T18:34:48.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8683e-05 2024-09-10T18:34:48.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:48.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.934569+0000", 2024-09-10T18:34:48.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8804000000000001e-05 2024-09-10T18:34:48.955 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:48.955 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:48.955 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:48.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.956 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:211706 9.3 9.ff132e83 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-09-10T18:34:48.956 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:48.935270+0000", 2024-09-10T18:34:48.956 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4294967295.9999404, 2024-09-10T18:34:48.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00026174800000000002, 2024-09-10T18:34:48.956 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:48.956 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:34:48.956 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:48.956 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:34:48.956 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:48.956 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 211706 2024-09-10T18:34:48.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.956 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:48.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:48.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.935270+0000", 2024-09-10T18:34:48.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:48.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:48.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.935270+0000", 2024-09-10T18:34:48.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:48.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:48.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.935268+0000", 2024-09-10T18:34:48.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:34:48.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:48.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.935275+0000", 2024-09-10T18:34:48.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8689999999999998e-06 2024-09-10T18:34:48.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:48.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.935327+0000", 2024-09-10T18:34:48.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2074999999999999e-05 2024-09-10T18:34:48.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:48.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.935336+0000", 2024-09-10T18:34:48.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5869999999999993e-06 2024-09-10T18:34:48.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:48.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.935518+0000", 2024-09-10T18:34:48.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018209300000000001 2024-09-10T18:34:48.959 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:48.959 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:48.959 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:48.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.959 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.35306.0:211698 9.3 e295/246)", 2024-09-10T18:34:48.959 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:48.935540+0000", 2024-09-10T18:34:48.960 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4294967295.99967, 2024-09-10T18:34:48.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9854000000000001e-05, 2024-09-10T18:34:48.960 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:48.960 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "queued for pg", 2024-09-10T18:34:48.960 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:48.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:48.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.935540+0000", 2024-09-10T18:34:48.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:48.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:48.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.935540+0000", 2024-09-10T18:34:48.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:48.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:48.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.935539+0000", 2024-09-10T18:34:48.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:34:48.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:48.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.935544+0000", 2024-09-10T18:34:48.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.429e-06 2024-09-10T18:34:48.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:48.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.935567+0000", 2024-09-10T18:34:48.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2393000000000001e-05 2024-09-10T18:34:48.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:48.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:48.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:48.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:48.935572+0000", 2024-09-10T18:34:48.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2870000000000004e-06 2024-09-10T18:34:48.962 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:48.962 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:48.962 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:48.962 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:48.962 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-09-10T18:34:48.962 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 6 2024-09-10T18:34:48.962 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:34:49.030 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:49.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:49.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:49.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:49.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:49.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:49.204 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:49.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:49.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:49.344 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:49.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:49.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:49.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:49.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:49.483 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:49.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:49.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:49 smithi028 ceph-mon[39220]: from='client.25450 v1:172.21.15.28:0/2055430113' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:34:49.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:49 smithi028 ceph-mon[39220]: osdmap e295: 8 total, 8 up, 8 in 2024-09-10T18:34:49.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:49 smithi028 ceph-mon[29291]: from='client.25450 v1:172.21.15.28:0/2055430113' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:34:49.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:49 smithi028 ceph-mon[29291]: osdmap e295: 8 total, 8 up, 8 in 2024-09-10T18:34:49.585 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:49 smithi204 ceph-mon[32357]: from='client.25450 v1:172.21.15.28:0/2055430113' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:34:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:49 smithi204 ceph-mon[32357]: osdmap e295: 8 total, 8 up, 8 in 2024-09-10T18:34:49.616 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:49.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:49.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:49.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:49.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:49.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:49.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:49.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:49.886 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":601,"stamp":"2024-09-10T18:34:47.032765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4083165988,"num_objects":62490,"num_object_clones":0,"num_object_copies":187470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62490,"num_whiteouts":0,"num_read":1976,"num_read_kb":1763,"num_write":207641,"num_write_kb":8614231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":829631,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71646,"ondisk_log_size":71646,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15396708,"kb_used_data":12101008,"kb_used_omap":19,"kb_used_meta":3294892,"kb_avail":734564508,"statfs":{"total":767960285184,"available":752194056192,"internally_reserved":0,"allocated":12391432192,"data_stored":12387684916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20033,"internal_metadata":3373969855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-796065792,"num_objects":-12147,"num_object_clones":0,"num_object_copies":-36441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12147,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12147,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.244011"},"pg_stats":[{"pgid":"1.1d","version":"76'81","reported_seq":18,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818252+0000","last_change":"2024-09-10T18:34:34.866222+0000","last_active":"2024-09-10T18:34:40.818252+0000","last_peered":"2024-09-10T18:34:40.818252+0000","last_clean":"2024-09-10T18:34:40.818252+0000","last_became_active":"2024-09-10T18:34:34.865894+0000","last_became_peered":"2024-09-10T18:34:34.865894+0000","last_unstale":"2024-09-10T18:34:40.818252+0000","last_undegraded":"2024-09-10T18:34:40.818252+0000","last_fullsized":"2024-09-10T18:34:40.818252+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:58.561425+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.561425+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":294,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.821588+0000","last_change":"2024-09-10T18:33:04.320548+0000","last_active":"2024-09-10T18:34:40.821588+0000","last_peered":"2024-09-10T18:34:40.821588+0000","last_clean":"2024-09-10T18:34:40.821588+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:34:40.821588+0000","last_undegraded":"2024-09-10T18:34:40.821588+0000","last_fullsized":"2024-09-10T18:34:40.821588+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:04.320458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:33:04.320458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055421380000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":333,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.809251+0000","last_change":"2024-09-10T18:33:12.680256+0000","last_active":"2024-09-10T18:34:40.809251+0000","last_peered":"2024-09-10T18:34:40.809251+0000","last_clean":"2024-09-10T18:34:40.809251+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:34:40.809251+0000","last_undegraded":"2024-09-10T18:34:40.809251+0000","last_fullsized":"2024-09-10T18:34:40.809251+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:33:12.680138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:12.680138+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033442979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":299,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.817731+0000","last_change":"2024-09-10T18:32:43.156005+0000","last_active":"2024-09-10T18:34:40.817731+0000","last_peered":"2024-09-10T18:34:40.817731+0000","last_clean":"2024-09-10T18:34:40.817731+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:34:40.817731+0000","last_undegraded":"2024-09-10T18:34:40.817731+0000","last_fullsized":"2024-09-10T18:34:40.817731+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:43.155924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:43.155924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007043718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":297,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.810010+0000","last_change":"2024-09-10T18:33:08.643815+0000","last_active":"2024-09-10T18:34:40.810010+0000","last_peered":"2024-09-10T18:34:40.810010+0000","last_clean":"2024-09-10T18:34:40.810010+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:34:40.810010+0000","last_undegraded":"2024-09-10T18:34:40.810010+0000","last_fullsized":"2024-09-10T18:34:40.810010+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:08.643758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.643758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046354059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'81","reported_seq":14,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820737+0000","last_change":"2024-09-10T18:34:34.868599+0000","last_active":"2024-09-10T18:34:40.820737+0000","last_peered":"2024-09-10T18:34:40.820737+0000","last_clean":"2024-09-10T18:34:40.820737+0000","last_became_active":"2024-09-10T18:34:34.867388+0000","last_became_peered":"2024-09-10T18:34:34.867388+0000","last_unstale":"2024-09-10T18:34:40.820737+0000","last_undegraded":"2024-09-10T18:34:40.820737+0000","last_fullsized":"2024-09-10T18:34:40.820737+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.370561+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.370561+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029251390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54614,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":306,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.815887+0000","last_change":"2024-09-10T18:34:16.293404+0000","last_active":"2024-09-10T18:34:40.815887+0000","last_peered":"2024-09-10T18:34:40.815887+0000","last_clean":"2024-09-10T18:34:40.815887+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:34:40.815887+0000","last_undegraded":"2024-09-10T18:34:40.815887+0000","last_fullsized":"2024-09-10T18:34:40.815887+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:16.293206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:16.293206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:37.489769+0000","scrub_duration":0.0021119960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":312,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.814139+0000","last_change":"2024-09-10T18:34:22.415416+0000","last_active":"2024-09-10T18:34:40.814139+0000","last_peered":"2024-09-10T18:34:40.814139+0000","last_clean":"2024-09-10T18:34:40.814139+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:34:40.814139+0000","last_undegraded":"2024-09-10T18:34:40.814139+0000","last_fullsized":"2024-09-10T18:34:40.814139+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:22.415093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.415093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:23.013807+0000","scrub_duration":0.068845071999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'34","reported_seq":343,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.813979+0000","last_change":"2024-09-10T18:32:55.371863+0000","last_active":"2024-09-10T18:34:40.813979+0000","last_peered":"2024-09-10T18:34:40.813979+0000","last_clean":"2024-09-10T18:34:40.813979+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:34:40.813979+0000","last_undegraded":"2024-09-10T18:34:40.813979+0000","last_fullsized":"2024-09-10T18:34:40.813979+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:32:55.371617+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:32:55.371617+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089694079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":309,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.809175+0000","last_change":"2024-09-10T18:34:09.440223+0000","last_active":"2024-09-10T18:34:40.809175+0000","last_peered":"2024-09-10T18:34:40.809175+0000","last_clean":"2024-09-10T18:34:40.809175+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:34:40.809175+0000","last_undegraded":"2024-09-10T18:34:40.809175+0000","last_fullsized":"2024-09-10T18:34:40.809175+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:09.440140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:34:09.440140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:16.383699+0000","scrub_duration":0.003425656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":314,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818925+0000","last_change":"2024-09-10T18:33:16.963182+0000","last_active":"2024-09-10T18:34:40.818925+0000","last_peered":"2024-09-10T18:34:40.818925+0000","last_clean":"2024-09-10T18:34:40.818925+0000","last_became_active":"2024-09-10T18:26:27.644961+0000","last_became_peered":"2024-09-10T18:26:27.644961+0000","last_unstale":"2024-09-10T18:34:40.818925+0000","last_undegraded":"2024-09-10T18:34:40.818925+0000","last_fullsized":"2024-09-10T18:34:40.818925+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_clean_scrub_stamp":"2024-09-10T18:33:16.963102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:44.331853+0000","scrub_duration":0.0019539739999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'15","reported_seq":336,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.810107+0000","last_change":"2024-09-10T18:34:25.455830+0000","last_active":"2024-09-10T18:34:40.810107+0000","last_peered":"2024-09-10T18:34:40.810107+0000","last_clean":"2024-09-10T18:34:40.810107+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:34:40.810107+0000","last_undegraded":"2024-09-10T18:34:40.810107+0000","last_fullsized":"2024-09-10T18:34:40.810107+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:34:25.455754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:25.455754+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:48.508068+0000","scrub_duration":0.13960720300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":310,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818273+0000","last_change":"2024-09-10T18:34:17.265867+0000","last_active":"2024-09-10T18:34:40.818273+0000","last_peered":"2024-09-10T18:34:40.818273+0000","last_clean":"2024-09-10T18:34:40.818273+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:34:40.818273+0000","last_undegraded":"2024-09-10T18:34:40.818273+0000","last_fullsized":"2024-09-10T18:34:40.818273+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:17.265788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_clean_scrub_stamp":"2024-09-10T18:34:17.265788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:22.473386+0000","scrub_duration":0.0023387740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":298,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820437+0000","last_change":"2024-09-10T18:33:00.235163+0000","last_active":"2024-09-10T18:34:40.820437+0000","last_peered":"2024-09-10T18:34:40.820437+0000","last_clean":"2024-09-10T18:34:40.820437+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:34:40.820437+0000","last_undegraded":"2024-09-10T18:34:40.820437+0000","last_fullsized":"2024-09-10T18:34:40.820437+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:00.234931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:00.234931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023527999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'81","reported_seq":82,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.813973+0000","last_change":"2024-09-10T18:34:34.863425+0000","last_active":"2024-09-10T18:34:40.813973+0000","last_peered":"2024-09-10T18:34:40.813973+0000","last_clean":"2024-09-10T18:34:40.813973+0000","last_became_active":"2024-09-10T18:34:34.863102+0000","last_became_peered":"2024-09-10T18:34:34.863102+0000","last_unstale":"2024-09-10T18:34:40.813973+0000","last_undegraded":"2024-09-10T18:34:40.813973+0000","last_fullsized":"2024-09-10T18:34:40.813973+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:48.006695+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:48.006695+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":304,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818752+0000","last_change":"2024-09-10T18:32:00.582011+0000","last_active":"2024-09-10T18:34:40.818752+0000","last_peered":"2024-09-10T18:34:40.818752+0000","last_clean":"2024-09-10T18:34:40.818752+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:34:40.818752+0000","last_undegraded":"2024-09-10T18:34:40.818752+0000","last_fullsized":"2024-09-10T18:34:40.818752+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.581959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:17.464205+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.581959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012934120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":338,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.810293+0000","last_change":"2024-09-10T18:32:48.589267+0000","last_active":"2024-09-10T18:34:40.810293+0000","last_peered":"2024-09-10T18:34:40.810293+0000","last_clean":"2024-09-10T18:34:40.810293+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:34:40.810293+0000","last_undegraded":"2024-09-10T18:34:40.810293+0000","last_fullsized":"2024-09-10T18:34:40.810293+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'9","last_scrub_stamp":"2024-09-10T18:32:48.589167+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:32:48.589167+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073279920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":299,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.819995+0000","last_change":"2024-09-10T18:32:01.525753+0000","last_active":"2024-09-10T18:34:40.819995+0000","last_peered":"2024-09-10T18:34:40.819995+0000","last_clean":"2024-09-10T18:34:40.819995+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:34:40.819995+0000","last_undegraded":"2024-09-10T18:34:40.819995+0000","last_fullsized":"2024-09-10T18:34:40.819995+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:01.525701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:01.525701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001223902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":301,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.817347+0000","last_change":"2024-09-10T18:34:09.151879+0000","last_active":"2024-09-10T18:34:40.817347+0000","last_peered":"2024-09-10T18:34:40.817347+0000","last_clean":"2024-09-10T18:34:40.817347+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:34:40.817347+0000","last_undegraded":"2024-09-10T18:34:40.817347+0000","last_fullsized":"2024-09-10T18:34:40.817347+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:09.151789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:34:09.151789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:10.873083+0000","scrub_duration":0.002651994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'81","reported_seq":165,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.813294+0000","last_change":"2024-09-10T18:34:34.863449+0000","last_active":"2024-09-10T18:34:40.813294+0000","last_peered":"2024-09-10T18:34:40.813294+0000","last_clean":"2024-09-10T18:34:40.813294+0000","last_became_active":"2024-09-10T18:34:34.862917+0000","last_became_peered":"2024-09-10T18:34:34.862917+0000","last_unstale":"2024-09-10T18:34:40.813294+0000","last_undegraded":"2024-09-10T18:34:40.813294+0000","last_fullsized":"2024-09-10T18:34:40.813294+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:06.053455+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.053455+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003148245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":11,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":307,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.811105+0000","last_change":"2024-09-10T18:34:07.394899+0000","last_active":"2024-09-10T18:34:40.811105+0000","last_peered":"2024-09-10T18:34:40.811105+0000","last_clean":"2024-09-10T18:34:40.811105+0000","last_became_active":"2024-09-10T18:26:27.650774+0000","last_became_peered":"2024-09-10T18:26:27.650774+0000","last_unstale":"2024-09-10T18:34:40.811105+0000","last_undegraded":"2024-09-10T18:34:40.811105+0000","last_fullsized":"2024-09-10T18:34:40.811105+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:07.394835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:07.394835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:13.236135+0000","scrub_duration":0.001116433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":300,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.825943+0000","last_change":"2024-09-10T18:34:12.458962+0000","last_active":"2024-09-10T18:34:40.825943+0000","last_peered":"2024-09-10T18:34:40.825943+0000","last_clean":"2024-09-10T18:34:40.825943+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:34:40.825943+0000","last_undegraded":"2024-09-10T18:34:40.825943+0000","last_fullsized":"2024-09-10T18:34:40.825943+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:12.458742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.458742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:40.818274+0000","scrub_duration":0.0077313979999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"240'27","reported_seq":364,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.828256+0000","last_change":"2024-09-10T18:33:04.586182+0000","last_active":"2024-09-10T18:34:40.828256+0000","last_peered":"2024-09-10T18:34:40.828256+0000","last_clean":"2024-09-10T18:34:40.828256+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:34:40.828256+0000","last_undegraded":"2024-09-10T18:34:40.828256+0000","last_fullsized":"2024-09-10T18:34:40.828256+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:33:04.586063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:04.586063+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007748642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":309,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818786+0000","last_change":"2024-09-10T18:32:58.252469+0000","last_active":"2024-09-10T18:34:40.818786+0000","last_peered":"2024-09-10T18:34:40.818786+0000","last_clean":"2024-09-10T18:34:40.818786+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:34:40.818786+0000","last_undegraded":"2024-09-10T18:34:40.818786+0000","last_fullsized":"2024-09-10T18:34:40.818786+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:58.252419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.252419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049362080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'81","reported_seq":236,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818116+0000","last_change":"2024-09-10T18:34:34.862755+0000","last_active":"2024-09-10T18:34:40.818116+0000","last_peered":"2024-09-10T18:34:40.818116+0000","last_clean":"2024-09-10T18:34:40.818116+0000","last_became_active":"2024-09-10T18:34:34.855693+0000","last_became_peered":"2024-09-10T18:34:34.855693+0000","last_unstale":"2024-09-10T18:34:40.818116+0000","last_undegraded":"2024-09-10T18:34:40.818116+0000","last_fullsized":"2024-09-10T18:34:40.818116+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:08.009806+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.009806+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002902614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":306,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.814466+0000","last_change":"2024-09-10T18:34:21.344309+0000","last_active":"2024-09-10T18:34:40.814466+0000","last_peered":"2024-09-10T18:34:40.814466+0000","last_clean":"2024-09-10T18:34:40.814466+0000","last_became_active":"2024-09-10T18:26:27.640035+0000","last_became_peered":"2024-09-10T18:26:27.640035+0000","last_unstale":"2024-09-10T18:34:40.814466+0000","last_undegraded":"2024-09-10T18:34:40.814466+0000","last_fullsized":"2024-09-10T18:34:40.814466+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:21.344236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.344236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:50.779397+0000","scrub_duration":0.0040101240000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":303,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.810217+0000","last_change":"2024-09-10T18:32:59.567476+0000","last_active":"2024-09-10T18:34:40.810217+0000","last_peered":"2024-09-10T18:34:40.810217+0000","last_clean":"2024-09-10T18:34:40.810217+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:34:40.810217+0000","last_undegraded":"2024-09-10T18:34:40.810217+0000","last_fullsized":"2024-09-10T18:34:40.810217+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:59.567389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:32:59.567389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073893559999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'23","reported_seq":330,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.813405+0000","last_change":"2024-09-10T18:32:50.341138+0000","last_active":"2024-09-10T18:34:40.813405+0000","last_peered":"2024-09-10T18:34:40.813405+0000","last_clean":"2024-09-10T18:34:40.813405+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:34:40.813405+0000","last_undegraded":"2024-09-10T18:34:40.813405+0000","last_fullsized":"2024-09-10T18:34:40.813405+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'23","last_scrub_stamp":"2024-09-10T18:32:50.341044+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:32:50.341044+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012050170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":299,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.810401+0000","last_change":"2024-09-10T18:33:07.697770+0000","last_active":"2024-09-10T18:34:40.810401+0000","last_peered":"2024-09-10T18:34:40.810401+0000","last_clean":"2024-09-10T18:34:40.810401+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:34:40.810401+0000","last_undegraded":"2024-09-10T18:34:40.810401+0000","last_fullsized":"2024-09-10T18:34:40.810401+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:07.697667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:33:07.697667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.044890945000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'81","reported_seq":224,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.817240+0000","last_change":"2024-09-10T18:34:34.855506+0000","last_active":"2024-09-10T18:34:40.817240+0000","last_peered":"2024-09-10T18:34:40.817240+0000","last_clean":"2024-09-10T18:34:40.817240+0000","last_became_active":"2024-09-10T18:34:34.855038+0000","last_became_peered":"2024-09-10T18:34:34.855038+0000","last_unstale":"2024-09-10T18:34:40.817240+0000","last_undegraded":"2024-09-10T18:34:40.817240+0000","last_fullsized":"2024-09-10T18:34:40.817240+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:48.466654+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.466654+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":299,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820651+0000","last_change":"2024-09-10T18:32:57.769495+0000","last_active":"2024-09-10T18:34:40.820651+0000","last_peered":"2024-09-10T18:34:40.820651+0000","last_clean":"2024-09-10T18:34:40.820651+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:34:40.820651+0000","last_undegraded":"2024-09-10T18:34:40.820651+0000","last_fullsized":"2024-09-10T18:34:40.820651+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:32:57.769408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:57.769408+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003757149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":313,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.817863+0000","last_change":"2024-09-10T18:32:51.182762+0000","last_active":"2024-09-10T18:34:40.817863+0000","last_peered":"2024-09-10T18:34:40.817863+0000","last_clean":"2024-09-10T18:34:40.817863+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:34:40.817863+0000","last_undegraded":"2024-09-10T18:34:40.817863+0000","last_fullsized":"2024-09-10T18:34:40.817863+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:51.182684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:32:51.182684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003614637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'25","reported_seq":337,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.821981+0000","last_change":"2024-09-10T18:33:15.366317+0000","last_active":"2024-09-10T18:34:40.821981+0000","last_peered":"2024-09-10T18:34:40.821981+0000","last_clean":"2024-09-10T18:34:40.821981+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:34:40.821981+0000","last_undegraded":"2024-09-10T18:34:40.821981+0000","last_fullsized":"2024-09-10T18:34:40.821981+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:33:15.366107+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.366107+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014527711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"217'11","reported_seq":379,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:42.847666+0000","last_change":"2024-09-10T18:33:03.119057+0000","last_active":"2024-09-10T18:34:42.847666+0000","last_peered":"2024-09-10T18:34:42.847666+0000","last_clean":"2024-09-10T18:34:42.847666+0000","last_became_active":"2024-09-10T18:26:27.652489+0000","last_became_peered":"2024-09-10T18:26:27.652489+0000","last_unstale":"2024-09-10T18:34:42.847666+0000","last_undegraded":"2024-09-10T18:34:42.847666+0000","last_fullsized":"2024-09-10T18:34:42.847666+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11","last_scrub_stamp":"2024-09-10T18:33:03.118959+0000","last_deep_scrub":"217'11","last_deep_scrub_stamp":"2024-09-10T18:30:03.161985+0000","last_clean_scrub_stamp":"2024-09-10T18:33:03.118959+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002038715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"294'12942","reported_seq":13267,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:45.931916+0000","last_change":"2024-09-10T18:34:07.618780+0000","last_active":"2024-09-10T18:34:45.931916+0000","last_peered":"2024-09-10T18:34:45.931916+0000","last_clean":"2024-09-10T18:34:45.931916+0000","last_became_active":"2024-09-10T18:32:03.385428+0000","last_became_peered":"2024-09-10T18:32:03.385428+0000","last_unstale":"2024-09-10T18:34:45.931916+0000","last_undegraded":"2024-09-10T18:34:45.931916+0000","last_fullsized":"2024-09-10T18:34:45.931916+0000","mapping_epoch":246,"log_start":"274'9044","ondisk_log_start":"274'9044","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10375","last_scrub_stamp":"2024-09-10T18:34:07.618535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:07.618535+0000","objects_scrubbed":6480,"log_size":3898,"log_dups_size":0,"ondisk_log_size":3898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:25.198149+0000","scrub_duration":3.415268937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247201792,"num_objects":3772,"num_object_clones":0,"num_object_copies":11316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12942,"num_write_kb":534848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":230,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.817337+0000","last_change":"2024-09-10T18:34:32.823529+0000","last_active":"2024-09-10T18:34:40.817337+0000","last_peered":"2024-09-10T18:34:40.817337+0000","last_clean":"2024-09-10T18:34:40.817337+0000","last_became_active":"2024-09-10T18:34:32.823230+0000","last_became_peered":"2024-09-10T18:34:32.823230+0000","last_unstale":"2024-09-10T18:34:40.817337+0000","last_undegraded":"2024-09-10T18:34:40.817337+0000","last_fullsized":"2024-09-10T18:34:40.817337+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:56.155354+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:56.155354+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001679647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"240'34","reported_seq":364,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.824506+0000","last_change":"2024-09-10T18:32:56.285685+0000","last_active":"2024-09-10T18:34:40.824506+0000","last_peered":"2024-09-10T18:34:40.824506+0000","last_clean":"2024-09-10T18:34:40.824506+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:34:40.824506+0000","last_undegraded":"2024-09-10T18:34:40.824506+0000","last_fullsized":"2024-09-10T18:34:40.824506+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:32:56.285589+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:32:56.285589+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081173400000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":305,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.813200+0000","last_change":"2024-09-10T18:34:20.305626+0000","last_active":"2024-09-10T18:34:40.813200+0000","last_peered":"2024-09-10T18:34:40.813200+0000","last_clean":"2024-09-10T18:34:40.813200+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:34:40.813200+0000","last_undegraded":"2024-09-10T18:34:40.813200+0000","last_fullsized":"2024-09-10T18:34:40.813200+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:20.305547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:20.305547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:42.039144+0000","scrub_duration":0.0046035989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":468,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.811175+0000","last_change":"2024-09-10T18:34:14.495568+0000","last_active":"2024-09-10T18:34:40.811175+0000","last_peered":"2024-09-10T18:34:40.811175+0000","last_clean":"2024-09-10T18:34:40.811175+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:34:40.811175+0000","last_undegraded":"2024-09-10T18:34:40.811175+0000","last_fullsized":"2024-09-10T18:34:40.811175+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:34:14.495486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:14.495486+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:34.967733+0000","scrub_duration":0.010908085,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":298,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818528+0000","last_change":"2024-09-10T18:32:50.961491+0000","last_active":"2024-09-10T18:34:40.818528+0000","last_peered":"2024-09-10T18:34:40.818528+0000","last_clean":"2024-09-10T18:34:40.818528+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:34:40.818528+0000","last_undegraded":"2024-09-10T18:34:40.818528+0000","last_fullsized":"2024-09-10T18:34:40.818528+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:50.961251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:32:50.961251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063477669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"294'12633","reported_seq":12690,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:44.987743+0000","last_change":"2024-09-10T18:32:03.379554+0000","last_active":"2024-09-10T18:34:44.987743+0000","last_peered":"2024-09-10T18:34:44.987743+0000","last_clean":"2024-09-10T18:34:44.987743+0000","last_became_active":"2024-09-10T18:32:03.379257+0000","last_became_peered":"2024-09-10T18:32:03.379257+0000","last_unstale":"2024-09-10T18:34:44.987743+0000","last_undegraded":"2024-09-10T18:34:44.987743+0000","last_fullsized":"2024-09-10T18:34:44.987743+0000","mapping_epoch":246,"log_start":"274'8716","ondisk_log_start":"274'8716","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":0,"log_size":3917,"log_dups_size":0,"ondisk_log_size":3917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254279704,"num_objects":3881,"num_object_clones":0,"num_object_copies":11643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3881,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12633,"num_write_kb":528385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":239,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.810869+0000","last_change":"2024-09-10T18:34:32.816324+0000","last_active":"2024-09-10T18:34:40.810869+0000","last_peered":"2024-09-10T18:34:40.810869+0000","last_clean":"2024-09-10T18:34:40.810869+0000","last_became_active":"2024-09-10T18:34:32.815839+0000","last_became_peered":"2024-09-10T18:34:32.815839+0000","last_unstale":"2024-09-10T18:34:40.810869+0000","last_undegraded":"2024-09-10T18:34:40.810869+0000","last_fullsized":"2024-09-10T18:34:40.810869+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:22.344759+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.344759+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002682837,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":343,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.812922+0000","last_change":"2024-09-10T18:32:59.382953+0000","last_active":"2024-09-10T18:34:40.812922+0000","last_peered":"2024-09-10T18:34:40.812922+0000","last_clean":"2024-09-10T18:34:40.812922+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:34:40.812922+0000","last_undegraded":"2024-09-10T18:34:40.812922+0000","last_fullsized":"2024-09-10T18:34:40.812922+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-09-10T18:32:59.382840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:59.382840+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0072140750000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":295,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.814980+0000","last_change":"2024-09-10T18:32:49.346233+0000","last_active":"2024-09-10T18:34:40.814980+0000","last_peered":"2024-09-10T18:34:40.814980+0000","last_clean":"2024-09-10T18:34:40.814980+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:34:40.814980+0000","last_undegraded":"2024-09-10T18:34:40.814980+0000","last_fullsized":"2024-09-10T18:34:40.814980+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:49.342269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.140922+0000","last_clean_scrub_stamp":"2024-09-10T18:32:49.342269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006539385,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":299,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.809514+0000","last_change":"2024-09-10T18:32:52.582545+0000","last_active":"2024-09-10T18:34:40.809514+0000","last_peered":"2024-09-10T18:34:40.809514+0000","last_clean":"2024-09-10T18:34:40.809514+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:34:40.809514+0000","last_undegraded":"2024-09-10T18:34:40.809514+0000","last_fullsized":"2024-09-10T18:34:40.809514+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:52.582464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:32:52.582464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024277399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":297,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.810668+0000","last_change":"2024-09-10T18:34:21.322560+0000","last_active":"2024-09-10T18:34:40.810668+0000","last_peered":"2024-09-10T18:34:40.810668+0000","last_clean":"2024-09-10T18:34:40.810668+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:34:40.810668+0000","last_undegraded":"2024-09-10T18:34:40.810668+0000","last_fullsized":"2024-09-10T18:34:40.810668+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:21.322454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.322454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:50.552165+0000","scrub_duration":0.0032746329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"294'16399","reported_seq":16599,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:45.939573+0000","last_change":"2024-09-10T18:34:39.851791+0000","last_active":"2024-09-10T18:34:45.939573+0000","last_peered":"2024-09-10T18:34:45.939573+0000","last_clean":"2024-09-10T18:34:45.939573+0000","last_became_active":"2024-09-10T18:32:03.554400+0000","last_became_peered":"2024-09-10T18:32:03.554400+0000","last_unstale":"2024-09-10T18:34:45.939573+0000","last_undegraded":"2024-09-10T18:34:45.939573+0000","last_fullsized":"2024-09-10T18:34:45.939573+0000","mapping_epoch":246,"log_start":"285'12413","ondisk_log_start":"285'12413","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":3435,"log_size":3986,"log_dups_size":0,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255197184,"num_objects":3894,"num_object_clones":0,"num_object_copies":11682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12964,"num_write_kb":539456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":210,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.814722+0000","last_change":"2024-09-10T18:34:00.166754+0000","last_active":"2024-09-10T18:34:40.814722+0000","last_peered":"2024-09-10T18:34:40.814722+0000","last_clean":"2024-09-10T18:34:40.814722+0000","last_became_active":"2024-09-10T18:34:00.166362+0000","last_became_peered":"2024-09-10T18:34:00.166362+0000","last_unstale":"2024-09-10T18:34:40.814722+0000","last_undegraded":"2024-09-10T18:34:40.814722+0000","last_fullsized":"2024-09-10T18:34:40.814722+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.370561+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.370561+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029251390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"240'28","reported_seq":350,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.809694+0000","last_change":"2024-09-10T18:33:59.587714+0000","last_active":"2024-09-10T18:34:40.809694+0000","last_peered":"2024-09-10T18:34:40.809694+0000","last_clean":"2024-09-10T18:34:40.809694+0000","last_became_active":"2024-09-10T18:26:27.651364+0000","last_became_peered":"2024-09-10T18:26:27.651364+0000","last_unstale":"2024-09-10T18:34:40.809694+0000","last_undegraded":"2024-09-10T18:34:40.809694+0000","last_fullsized":"2024-09-10T18:34:40.809694+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'28","last_scrub_stamp":"2024-09-10T18:33:59.587604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:59.587604+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:21.896292+0000","scrub_duration":0.0024058019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":294,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.822208+0000","last_change":"2024-09-10T18:32:54.291391+0000","last_active":"2024-09-10T18:34:40.822208+0000","last_peered":"2024-09-10T18:34:40.822208+0000","last_clean":"2024-09-10T18:34:40.822208+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:34:40.822208+0000","last_undegraded":"2024-09-10T18:34:40.822208+0000","last_fullsized":"2024-09-10T18:34:40.822208+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:54.291164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:32:54.291164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002933667,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":459,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:42.840455+0000","last_change":"2024-09-10T18:33:07.367306+0000","last_active":"2024-09-10T18:34:42.840455+0000","last_peered":"2024-09-10T18:34:42.840455+0000","last_clean":"2024-09-10T18:34:42.840455+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:34:42.840455+0000","last_undegraded":"2024-09-10T18:34:42.840455+0000","last_fullsized":"2024-09-10T18:34:42.840455+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:33:07.367248+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:33:07.367248+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044945339999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":307,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820230+0000","last_change":"2024-09-10T18:33:11.857892+0000","last_active":"2024-09-10T18:34:40.820230+0000","last_peered":"2024-09-10T18:34:40.820230+0000","last_clean":"2024-09-10T18:34:40.820230+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:34:40.820230+0000","last_undegraded":"2024-09-10T18:34:40.820230+0000","last_fullsized":"2024-09-10T18:34:40.820230+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:11.857685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.857685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022995870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"294'12874","reported_seq":13212,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:45.059008+0000","last_change":"2024-09-10T18:34:04.158338+0000","last_active":"2024-09-10T18:34:45.059008+0000","last_peered":"2024-09-10T18:34:45.059008+0000","last_clean":"2024-09-10T18:34:45.059008+0000","last_became_active":"2024-09-10T18:32:03.384288+0000","last_became_peered":"2024-09-10T18:32:03.384288+0000","last_unstale":"2024-09-10T18:34:45.059008+0000","last_undegraded":"2024-09-10T18:34:45.059008+0000","last_fullsized":"2024-09-10T18:34:45.059008+0000","mapping_epoch":246,"log_start":"274'8575","ondisk_log_start":"274'8575","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'10160","last_scrub_stamp":"2024-09-10T18:34:04.158266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:04.158266+0000","objects_scrubbed":6798,"log_size":4299,"log_dups_size":0,"ondisk_log_size":4299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:13.705054+0000","scrub_duration":2.9688170149999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259391488,"num_objects":3958,"num_object_clones":0,"num_object_copies":11874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12874,"num_write_kb":538624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":289,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.824753+0000","last_change":"2024-09-10T18:34:32.818627+0000","last_active":"2024-09-10T18:34:40.824753+0000","last_peered":"2024-09-10T18:34:40.824753+0000","last_clean":"2024-09-10T18:34:40.824753+0000","last_became_active":"2024-09-10T18:34:32.818323+0000","last_became_peered":"2024-09-10T18:34:32.818323+0000","last_unstale":"2024-09-10T18:34:40.824753+0000","last_undegraded":"2024-09-10T18:34:40.824753+0000","last_fullsized":"2024-09-10T18:34:40.824753+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:08.035431+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:08.035431+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.065049888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":346,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.822074+0000","last_change":"2024-09-10T18:32:55.279937+0000","last_active":"2024-09-10T18:34:40.822074+0000","last_peered":"2024-09-10T18:34:40.822074+0000","last_clean":"2024-09-10T18:34:40.822074+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:34:40.822074+0000","last_undegraded":"2024-09-10T18:34:40.822074+0000","last_fullsized":"2024-09-10T18:34:40.822074+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:32:55.279209+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:32:55.279209+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091125259999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":294,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818961+0000","last_change":"2024-09-10T18:32:45.965208+0000","last_active":"2024-09-10T18:34:40.818961+0000","last_peered":"2024-09-10T18:34:40.818961+0000","last_clean":"2024-09-10T18:34:40.818961+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:34:40.818961+0000","last_undegraded":"2024-09-10T18:34:40.818961+0000","last_fullsized":"2024-09-10T18:34:40.818961+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:45.965129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:32:45.965129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045521829999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"263'11","reported_seq":467,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:42.841961+0000","last_change":"2024-09-10T18:33:01.283747+0000","last_active":"2024-09-10T18:34:42.841961+0000","last_peered":"2024-09-10T18:34:42.841961+0000","last_clean":"2024-09-10T18:34:42.841961+0000","last_became_active":"2024-09-10T18:26:27.639032+0000","last_became_peered":"2024-09-10T18:26:27.639032+0000","last_unstale":"2024-09-10T18:34:42.841961+0000","last_undegraded":"2024-09-10T18:34:42.841961+0000","last_fullsized":"2024-09-10T18:34:42.841961+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'11","last_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_deep_scrub":"263'11","last_deep_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.283648+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081388439999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":298,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820642+0000","last_change":"2024-09-10T18:33:57.483999+0000","last_active":"2024-09-10T18:34:40.820642+0000","last_peered":"2024-09-10T18:34:40.820642+0000","last_clean":"2024-09-10T18:34:40.820642+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:34:40.820642+0000","last_undegraded":"2024-09-10T18:34:40.820642+0000","last_fullsized":"2024-09-10T18:34:40.820642+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:57.483905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:57.483905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:58.495941+0000","scrub_duration":0.023666573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"294'12874","reported_seq":13048,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:44.988007+0000","last_change":"2024-09-10T18:34:25.462177+0000","last_active":"2024-09-10T18:34:44.988007+0000","last_peered":"2024-09-10T18:34:44.988007+0000","last_clean":"2024-09-10T18:34:44.988007+0000","last_became_active":"2024-09-10T18:32:03.554693+0000","last_became_peered":"2024-09-10T18:32:03.554693+0000","last_unstale":"2024-09-10T18:34:44.988007+0000","last_undegraded":"2024-09-10T18:34:44.988007+0000","last_fullsized":"2024-09-10T18:34:44.988007+0000","mapping_epoch":246,"log_start":"274'8947","ondisk_log_start":"274'8947","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":2761,"log_size":3927,"log_dups_size":0,"ondisk_log_size":3927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260440064,"num_objects":3974,"num_object_clones":0,"num_object_copies":11922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12874,"num_write_kb":539136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":289,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.825625+0000","last_change":"2024-09-10T18:34:32.919295+0000","last_active":"2024-09-10T18:34:40.825625+0000","last_peered":"2024-09-10T18:34:40.825625+0000","last_clean":"2024-09-10T18:34:40.825625+0000","last_became_active":"2024-09-10T18:34:32.918946+0000","last_became_peered":"2024-09-10T18:34:32.918946+0000","last_unstale":"2024-09-10T18:34:40.825625+0000","last_undegraded":"2024-09-10T18:34:40.825625+0000","last_fullsized":"2024-09-10T18:34:40.825625+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:05.012053+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:05.012053+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025346610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"240'29","reported_seq":353,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.825652+0000","last_change":"2024-09-10T18:33:11.310163+0000","last_active":"2024-09-10T18:34:40.825652+0000","last_peered":"2024-09-10T18:34:40.825652+0000","last_clean":"2024-09-10T18:34:40.825652+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:34:40.825652+0000","last_undegraded":"2024-09-10T18:34:40.825652+0000","last_fullsized":"2024-09-10T18:34:40.825652+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'29","last_scrub_stamp":"2024-09-10T18:33:11.310074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.310074+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026943760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":303,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.821462+0000","last_change":"2024-09-10T18:34:02.559656+0000","last_active":"2024-09-10T18:34:40.821462+0000","last_peered":"2024-09-10T18:34:40.821462+0000","last_clean":"2024-09-10T18:34:40.821462+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:34:40.821462+0000","last_undegraded":"2024-09-10T18:34:40.821462+0000","last_fullsized":"2024-09-10T18:34:40.821462+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:02.559431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:02.559431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:18.828380+0000","scrub_duration":0.019462604000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":259,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.825159+0000","last_change":"2024-09-10T18:34:22.795033+0000","last_active":"2024-09-10T18:34:40.825159+0000","last_peered":"2024-09-10T18:34:40.825159+0000","last_clean":"2024-09-10T18:34:40.825159+0000","last_became_active":"2024-09-10T18:26:27.642063+0000","last_became_peered":"2024-09-10T18:26:27.642063+0000","last_unstale":"2024-09-10T18:34:40.825159+0000","last_undegraded":"2024-09-10T18:34:40.825159+0000","last_fullsized":"2024-09-10T18:34:40.825159+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:22.794935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.794935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:45.015520+0000","scrub_duration":0.0028594010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":304,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.817660+0000","last_change":"2024-09-10T18:33:10.137145+0000","last_active":"2024-09-10T18:34:40.817660+0000","last_peered":"2024-09-10T18:34:40.817660+0000","last_clean":"2024-09-10T18:34:40.817660+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:34:40.817660+0000","last_undegraded":"2024-09-10T18:34:40.817660+0000","last_fullsized":"2024-09-10T18:34:40.817660+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:10.137048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:33:10.137048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040149249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"294'16502","reported_seq":16999,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:46.101952+0000","last_change":"2024-09-10T18:34:21.239886+0000","last_active":"2024-09-10T18:34:46.101952+0000","last_peered":"2024-09-10T18:34:46.101952+0000","last_clean":"2024-09-10T18:34:46.101952+0000","last_became_active":"2024-09-10T18:32:03.381872+0000","last_became_peered":"2024-09-10T18:32:03.381872+0000","last_unstale":"2024-09-10T18:34:46.101952+0000","last_undegraded":"2024-09-10T18:34:46.101952+0000","last_fullsized":"2024-09-10T18:34:46.101952+0000","mapping_epoch":246,"log_start":"273'11452","ondisk_log_start":"273'11452","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'14889","last_scrub_stamp":"2024-09-10T18:34:21.239625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.239625+0000","objects_scrubbed":5453,"log_size":5050,"log_dups_size":0,"ondisk_log_size":5050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:26.853140+0000","scrub_duration":1.7487293830000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247529472,"num_objects":3777,"num_object_clones":0,"num_object_copies":11331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12971,"num_write_kb":535936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":205,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.817247+0000","last_change":"2024-09-10T18:34:00.151547+0000","last_active":"2024-09-10T18:34:40.817247+0000","last_peered":"2024-09-10T18:34:40.817247+0000","last_clean":"2024-09-10T18:34:40.817247+0000","last_became_active":"2024-09-10T18:34:00.151017+0000","last_became_peered":"2024-09-10T18:34:00.151017+0000","last_unstale":"2024-09-10T18:34:40.817247+0000","last_undegraded":"2024-09-10T18:34:40.817247+0000","last_fullsized":"2024-09-10T18:34:40.817247+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:56.629683+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:56.629683+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001774133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"240'41","reported_seq":445,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:42.838827+0000","last_change":"2024-09-10T18:32:55.941738+0000","last_active":"2024-09-10T18:34:42.838827+0000","last_peered":"2024-09-10T18:34:42.838827+0000","last_clean":"2024-09-10T18:34:42.838827+0000","last_became_active":"2024-09-10T18:23:10.067049+0000","last_became_peered":"2024-09-10T18:23:10.067049+0000","last_unstale":"2024-09-10T18:34:42.838827+0000","last_undegraded":"2024-09-10T18:34:42.838827+0000","last_fullsized":"2024-09-10T18:34:42.838827+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'41","last_scrub_stamp":"2024-09-10T18:32:55.941653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:55.941653+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011585564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":310,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.810543+0000","last_change":"2024-09-10T18:34:17.387421+0000","last_active":"2024-09-10T18:34:40.810543+0000","last_peered":"2024-09-10T18:34:40.810543+0000","last_clean":"2024-09-10T18:34:40.810543+0000","last_became_active":"2024-09-10T18:26:27.648690+0000","last_became_peered":"2024-09-10T18:26:27.648690+0000","last_unstale":"2024-09-10T18:34:40.810543+0000","last_undegraded":"2024-09-10T18:34:40.810543+0000","last_fullsized":"2024-09-10T18:34:40.810543+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:17.387344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:17.387344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:33.806832+0000","scrub_duration":0.0051904949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":319,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.817849+0000","last_change":"2024-09-10T18:33:04.936495+0000","last_active":"2024-09-10T18:34:40.817849+0000","last_peered":"2024-09-10T18:34:40.817849+0000","last_clean":"2024-09-10T18:34:40.817849+0000","last_became_active":"2024-09-10T18:26:27.647576+0000","last_became_peered":"2024-09-10T18:26:27.647576+0000","last_unstale":"2024-09-10T18:34:40.817849+0000","last_undegraded":"2024-09-10T18:34:40.817849+0000","last_fullsized":"2024-09-10T18:34:40.817849+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:33:04.936407+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:33:04.936407+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096149180000000001,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":300,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.825092+0000","last_change":"2024-09-10T18:34:21.562999+0000","last_active":"2024-09-10T18:34:40.825092+0000","last_peered":"2024-09-10T18:34:40.825092+0000","last_clean":"2024-09-10T18:34:40.825092+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:34:40.825092+0000","last_undegraded":"2024-09-10T18:34:40.825092+0000","last_fullsized":"2024-09-10T18:34:40.825092+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:21.562920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.562920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:36.317098+0000","scrub_duration":0.0041137129999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":289,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.813831+0000","last_change":"2024-09-10T18:33:56.168567+0000","last_active":"2024-09-10T18:34:40.813831+0000","last_peered":"2024-09-10T18:34:40.813831+0000","last_clean":"2024-09-10T18:34:40.813831+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:34:40.813831+0000","last_undegraded":"2024-09-10T18:34:40.813831+0000","last_fullsized":"2024-09-10T18:34:40.813831+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:56.168485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:56.168485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:14.414730+0000","scrub_duration":0.002436723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"294'13040","reported_seq":13373,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:44.991724+0000","last_change":"2024-09-10T18:34:06.677663+0000","last_active":"2024-09-10T18:34:44.991724+0000","last_peered":"2024-09-10T18:34:44.991724+0000","last_clean":"2024-09-10T18:34:44.991724+0000","last_became_active":"2024-09-10T18:32:03.380391+0000","last_became_peered":"2024-09-10T18:32:03.380391+0000","last_unstale":"2024-09-10T18:34:44.991724+0000","last_undegraded":"2024-09-10T18:34:44.991724+0000","last_fullsized":"2024-09-10T18:34:44.991724+0000","mapping_epoch":246,"log_start":"274'8667","ondisk_log_start":"274'8667","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10475","last_scrub_stamp":"2024-09-10T18:34:06.677564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.677564+0000","objects_scrubbed":6683,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:12.828051+0000","scrub_duration":2.5274349109999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263847936,"num_objects":4026,"num_object_clones":0,"num_object_copies":12078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13040,"num_write_kb":546112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":351,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.819412+0000","last_change":"2024-09-10T18:34:00.154687+0000","last_active":"2024-09-10T18:34:40.819412+0000","last_peered":"2024-09-10T18:34:40.819412+0000","last_clean":"2024-09-10T18:34:40.819412+0000","last_became_active":"2024-09-10T18:34:00.154404+0000","last_became_peered":"2024-09-10T18:34:00.154404+0000","last_unstale":"2024-09-10T18:34:40.819412+0000","last_undegraded":"2024-09-10T18:34:40.819412+0000","last_fullsized":"2024-09-10T18:34:40.819412+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:57.474295+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.474295+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015998690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"240'17","reported_seq":344,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.824576+0000","last_change":"2024-09-10T18:34:13.522168+0000","last_active":"2024-09-10T18:34:40.824576+0000","last_peered":"2024-09-10T18:34:40.824576+0000","last_clean":"2024-09-10T18:34:40.824576+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:34:40.824576+0000","last_undegraded":"2024-09-10T18:34:40.824576+0000","last_fullsized":"2024-09-10T18:34:40.824576+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'17","last_scrub_stamp":"2024-09-10T18:34:13.521944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:13.521944+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:35.758470+0000","scrub_duration":0.0029448339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":320,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.819520+0000","last_change":"2024-09-10T18:33:15.142062+0000","last_active":"2024-09-10T18:34:40.819520+0000","last_peered":"2024-09-10T18:34:40.819520+0000","last_clean":"2024-09-10T18:34:40.819520+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:34:40.819520+0000","last_undegraded":"2024-09-10T18:34:40.819520+0000","last_fullsized":"2024-09-10T18:34:40.819520+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:15.141982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:15.141982+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.141982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:45.048602+0000","scrub_duration":0.0028956749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":332,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.827379+0000","last_change":"2024-09-10T18:34:06.493534+0000","last_active":"2024-09-10T18:34:40.827379+0000","last_peered":"2024-09-10T18:34:40.827379+0000","last_clean":"2024-09-10T18:34:40.827379+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:34:40.827379+0000","last_undegraded":"2024-09-10T18:34:40.827379+0000","last_fullsized":"2024-09-10T18:34:40.827379+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:34:06.493448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.493448+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:21.109643+0000","scrub_duration":0.011827739,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":309,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.830325+0000","last_change":"2024-09-10T18:32:53.586654+0000","last_active":"2024-09-10T18:34:40.830325+0000","last_peered":"2024-09-10T18:34:40.830325+0000","last_clean":"2024-09-10T18:34:40.830325+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:34:40.830325+0000","last_undegraded":"2024-09-10T18:34:40.830325+0000","last_fullsized":"2024-09-10T18:34:40.830325+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:53.586216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:32:53.586216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053348709999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":297,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820339+0000","last_change":"2024-09-10T18:32:00.501396+0000","last_active":"2024-09-10T18:34:40.820339+0000","last_peered":"2024-09-10T18:34:40.820339+0000","last_clean":"2024-09-10T18:34:40.820339+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:34:40.820339+0000","last_undegraded":"2024-09-10T18:34:40.820339+0000","last_fullsized":"2024-09-10T18:34:40.820339+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:00.501344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:00.501344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001367442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"294'13149","reported_seq":13494,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:46.108170+0000","last_change":"2024-09-10T18:34:01.902715+0000","last_active":"2024-09-10T18:34:46.108170+0000","last_peered":"2024-09-10T18:34:46.108170+0000","last_clean":"2024-09-10T18:34:46.108170+0000","last_became_active":"2024-09-10T18:32:03.554958+0000","last_became_peered":"2024-09-10T18:32:03.554958+0000","last_unstale":"2024-09-10T18:34:46.108170+0000","last_undegraded":"2024-09-10T18:34:46.108170+0000","last_fullsized":"2024-09-10T18:34:46.108170+0000","mapping_epoch":246,"log_start":"273'8138","ondisk_log_start":"273'8138","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'10034","last_scrub_stamp":"2024-09-10T18:34:01.902618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:01.902618+0000","objects_scrubbed":6997,"log_size":5011,"log_dups_size":0,"ondisk_log_size":5011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:18.230790+0000","scrub_duration":2.3941321589999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248578048,"num_objects":3793,"num_object_clones":0,"num_object_copies":11379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13149,"num_write_kb":542144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":220,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818247+0000","last_change":"2024-09-10T18:34:00.166073+0000","last_active":"2024-09-10T18:34:40.818247+0000","last_peered":"2024-09-10T18:34:40.818247+0000","last_clean":"2024-09-10T18:34:40.818247+0000","last_became_active":"2024-09-10T18:34:00.165384+0000","last_became_peered":"2024-09-10T18:34:00.165384+0000","last_unstale":"2024-09-10T18:34:40.818247+0000","last_undegraded":"2024-09-10T18:34:40.818247+0000","last_fullsized":"2024-09-10T18:34:40.818247+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:48.466654+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.466654+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":328,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.822059+0000","last_change":"2024-09-10T18:32:53.270762+0000","last_active":"2024-09-10T18:34:40.822059+0000","last_peered":"2024-09-10T18:34:40.822059+0000","last_clean":"2024-09-10T18:34:40.822059+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:34:40.822059+0000","last_undegraded":"2024-09-10T18:34:40.822059+0000","last_fullsized":"2024-09-10T18:34:40.822059+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-09-10T18:32:53.270667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:53.270667+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01096307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":304,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.821361+0000","last_change":"2024-09-10T18:32:59.245911+0000","last_active":"2024-09-10T18:34:40.821361+0000","last_peered":"2024-09-10T18:34:40.821361+0000","last_clean":"2024-09-10T18:34:40.821361+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:34:40.821361+0000","last_undegraded":"2024-09-10T18:34:40.821361+0000","last_fullsized":"2024-09-10T18:34:40.821361+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:59.245802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:32:59.245802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.028203379000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":1107,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:43.831157+0000","last_change":"2024-09-10T18:34:02.136505+0000","last_active":"2024-09-10T18:34:43.831157+0000","last_peered":"2024-09-10T18:34:43.831157+0000","last_clean":"2024-09-10T18:34:43.831157+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:34:43.831157+0000","last_undegraded":"2024-09-10T18:34:43.831157+0000","last_fullsized":"2024-09-10T18:34:43.831157+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:34:02.136429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:02.136429+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:32.048652+0000","scrub_duration":0.0083841149999999993,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":786,"num_read_kb":781,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":293,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.819074+0000","last_change":"2024-09-10T18:32:56.888589+0000","last_active":"2024-09-10T18:34:40.819074+0000","last_peered":"2024-09-10T18:34:40.819074+0000","last_clean":"2024-09-10T18:34:40.819074+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:34:40.819074+0000","last_undegraded":"2024-09-10T18:34:40.819074+0000","last_fullsized":"2024-09-10T18:34:40.819074+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:56.888436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.771415+0000","last_clean_scrub_stamp":"2024-09-10T18:32:56.888436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064579939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":297,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.821128+0000","last_change":"2024-09-10T18:34:12.539195+0000","last_active":"2024-09-10T18:34:40.821128+0000","last_peered":"2024-09-10T18:34:40.821128+0000","last_clean":"2024-09-10T18:34:40.821128+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:34:40.821128+0000","last_undegraded":"2024-09-10T18:34:40.821128+0000","last_fullsized":"2024-09-10T18:34:40.821128+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:12.539124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.539124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:33.017675+0000","scrub_duration":0.0079965020000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"294'12735","reported_seq":13024,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:45.936094+0000","last_change":"2024-09-10T18:34:19.206406+0000","last_active":"2024-09-10T18:34:45.936094+0000","last_peered":"2024-09-10T18:34:45.936094+0000","last_clean":"2024-09-10T18:34:45.936094+0000","last_became_active":"2024-09-10T18:32:03.382335+0000","last_became_peered":"2024-09-10T18:32:03.382335+0000","last_unstale":"2024-09-10T18:34:45.936094+0000","last_undegraded":"2024-09-10T18:34:45.936094+0000","last_fullsized":"2024-09-10T18:34:45.936094+0000","mapping_epoch":246,"log_start":"274'8764","ondisk_log_start":"274'8764","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'11048","last_scrub_stamp":"2024-09-10T18:34:19.206308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.206308+0000","objects_scrubbed":5584,"log_size":3971,"log_dups_size":0,"ondisk_log_size":3971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:45.982181+0000","scrub_duration":1.913810319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252772352,"num_objects":3857,"num_object_clones":0,"num_object_copies":11571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12735,"num_write_kb":530944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'26","reported_seq":296,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.825509+0000","last_change":"2024-09-10T18:34:19.788942+0000","last_active":"2024-09-10T18:34:40.825509+0000","last_peered":"2024-09-10T18:34:40.825509+0000","last_clean":"2024-09-10T18:34:40.825509+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:34:40.825509+0000","last_undegraded":"2024-09-10T18:34:40.825509+0000","last_fullsized":"2024-09-10T18:34:40.825509+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'26","last_scrub_stamp":"2024-09-10T18:34:19.788853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.788853+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:26.351175+0000","scrub_duration":0.0020676890000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":312,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.813320+0000","last_change":"2024-09-10T18:34:19.296515+0000","last_active":"2024-09-10T18:34:40.813320+0000","last_peered":"2024-09-10T18:34:40.813320+0000","last_clean":"2024-09-10T18:34:40.813320+0000","last_became_active":"2024-09-10T18:26:27.639129+0000","last_became_peered":"2024-09-10T18:26:27.639129+0000","last_unstale":"2024-09-10T18:34:40.813320+0000","last_undegraded":"2024-09-10T18:34:40.813320+0000","last_fullsized":"2024-09-10T18:34:40.813320+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:19.296286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.296286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:35.735259+0000","scrub_duration":0.0049170170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":363,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.819692+0000","last_change":"2024-09-10T18:34:00.160610+0000","last_active":"2024-09-10T18:34:40.819692+0000","last_peered":"2024-09-10T18:34:40.819692+0000","last_clean":"2024-09-10T18:34:40.819692+0000","last_became_active":"2024-09-10T18:34:00.159975+0000","last_became_peered":"2024-09-10T18:34:00.159975+0000","last_unstale":"2024-09-10T18:34:40.819692+0000","last_undegraded":"2024-09-10T18:34:40.819692+0000","last_fullsized":"2024-09-10T18:34:40.819692+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:58.780613+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.780613+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002490309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":309,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820723+0000","last_change":"2024-09-10T18:34:12.920724+0000","last_active":"2024-09-10T18:34:40.820723+0000","last_peered":"2024-09-10T18:34:40.820723+0000","last_clean":"2024-09-10T18:34:40.820723+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:34:40.820723+0000","last_undegraded":"2024-09-10T18:34:40.820723+0000","last_fullsized":"2024-09-10T18:34:40.820723+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:12.920651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.920651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:29.441715+0000","scrub_duration":0.0047636090000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":291,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818953+0000","last_change":"2024-09-10T18:32:49.984765+0000","last_active":"2024-09-10T18:34:40.818953+0000","last_peered":"2024-09-10T18:34:40.818953+0000","last_clean":"2024-09-10T18:34:40.818953+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:34:40.818953+0000","last_undegraded":"2024-09-10T18:34:40.818953+0000","last_fullsized":"2024-09-10T18:34:40.818953+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:49.984707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:32:49.984707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035569920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"76'81","reported_seq":239,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818391+0000","last_change":"2024-09-10T18:34:32.927768+0000","last_active":"2024-09-10T18:34:40.818391+0000","last_peered":"2024-09-10T18:34:40.818391+0000","last_clean":"2024-09-10T18:34:40.818391+0000","last_became_active":"2024-09-10T18:34:32.927460+0000","last_became_peered":"2024-09-10T18:34:32.927460+0000","last_unstale":"2024-09-10T18:34:40.818391+0000","last_undegraded":"2024-09-10T18:34:40.818391+0000","last_fullsized":"2024-09-10T18:34:40.818391+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:48.466654+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.466654+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"294'12905","reported_seq":13189,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:44.996533+0000","last_change":"2024-09-10T18:34:22.669084+0000","last_active":"2024-09-10T18:34:44.996533+0000","last_peered":"2024-09-10T18:34:44.996533+0000","last_clean":"2024-09-10T18:34:44.996533+0000","last_became_active":"2024-09-10T18:32:03.384247+0000","last_became_peered":"2024-09-10T18:32:03.384247+0000","last_unstale":"2024-09-10T18:34:44.996533+0000","last_undegraded":"2024-09-10T18:34:44.996533+0000","last_fullsized":"2024-09-10T18:34:44.996533+0000","mapping_epoch":246,"log_start":"274'8988","ondisk_log_start":"274'8988","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'11469","last_scrub_stamp":"2024-09-10T18:34:22.669027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.669027+0000","objects_scrubbed":5424,"log_size":3917,"log_dups_size":0,"ondisk_log_size":3917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:26.515481+0000","scrub_duration":1.750744868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257753088,"num_objects":3933,"num_object_clones":0,"num_object_copies":11799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12905,"num_write_kb":538816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"283'24","reported_seq":347,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.811248+0000","last_change":"2024-09-10T18:34:19.404796+0000","last_active":"2024-09-10T18:34:40.811248+0000","last_peered":"2024-09-10T18:34:40.811248+0000","last_clean":"2024-09-10T18:34:40.811248+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:34:40.811248+0000","last_undegraded":"2024-09-10T18:34:40.811248+0000","last_fullsized":"2024-09-10T18:34:40.811248+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'24","last_scrub_stamp":"2024-09-10T18:34:19.404713+0000","last_deep_scrub":"283'24","last_deep_scrub_stamp":"2024-09-10T18:34:19.404713+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.404713+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:30.068249+0000","scrub_duration":0.037546765000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":303,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.809835+0000","last_change":"2024-09-10T18:34:20.349520+0000","last_active":"2024-09-10T18:34:40.809835+0000","last_peered":"2024-09-10T18:34:40.809835+0000","last_clean":"2024-09-10T18:34:40.809835+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:34:40.809835+0000","last_undegraded":"2024-09-10T18:34:40.809835+0000","last_fullsized":"2024-09-10T18:34:40.809835+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:20.349464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:20.349464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:31.859829+0000","scrub_duration":0.0036650300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":307,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818719+0000","last_change":"2024-09-10T18:33:06.960402+0000","last_active":"2024-09-10T18:34:40.818719+0000","last_peered":"2024-09-10T18:34:40.818719+0000","last_clean":"2024-09-10T18:34:40.818719+0000","last_became_active":"2024-09-10T18:26:27.645885+0000","last_became_peered":"2024-09-10T18:26:27.645885+0000","last_unstale":"2024-09-10T18:34:40.818719+0000","last_undegraded":"2024-09-10T18:34:40.818719+0000","last_fullsized":"2024-09-10T18:34:40.818719+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:06.960320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:33:06.960320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020571560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":306,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.814800+0000","last_change":"2024-09-10T18:34:01.181010+0000","last_active":"2024-09-10T18:34:40.814800+0000","last_peered":"2024-09-10T18:34:40.814800+0000","last_clean":"2024-09-10T18:34:40.814800+0000","last_became_active":"2024-09-10T18:26:27.632071+0000","last_became_peered":"2024-09-10T18:26:27.632071+0000","last_unstale":"2024-09-10T18:34:40.814800+0000","last_undegraded":"2024-09-10T18:34:40.814800+0000","last_fullsized":"2024-09-10T18:34:40.814800+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:01.180920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:01.180920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:10.276096+0000","scrub_duration":0.0082996130000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"294'13077","reported_seq":13532,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:45.448341+0000","last_change":"2024-09-10T18:34:07.700772+0000","last_active":"2024-09-10T18:34:45.448341+0000","last_peered":"2024-09-10T18:34:45.448341+0000","last_clean":"2024-09-10T18:34:45.448341+0000","last_became_active":"2024-09-10T18:32:03.554412+0000","last_became_peered":"2024-09-10T18:32:03.554412+0000","last_unstale":"2024-09-10T18:34:45.448341+0000","last_undegraded":"2024-09-10T18:34:45.448341+0000","last_fullsized":"2024-09-10T18:34:45.448341+0000","mapping_epoch":246,"log_start":"274'8901","ondisk_log_start":"274'8901","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10569","last_scrub_stamp":"2024-09-10T18:34:07.700574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:07.700574+0000","objects_scrubbed":6543,"log_size":4176,"log_dups_size":0,"ondisk_log_size":4176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:32.220981+0000","scrub_duration":3.13969619,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254738432,"num_objects":3887,"num_object_clones":0,"num_object_copies":11661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13077,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":222,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.817992+0000","last_change":"2024-09-10T18:34:32.824505+0000","last_active":"2024-09-10T18:34:40.817992+0000","last_peered":"2024-09-10T18:34:40.817992+0000","last_clean":"2024-09-10T18:34:40.817992+0000","last_became_active":"2024-09-10T18:34:32.823976+0000","last_became_peered":"2024-09-10T18:34:32.823976+0000","last_unstale":"2024-09-10T18:34:40.817992+0000","last_undegraded":"2024-09-10T18:34:40.817992+0000","last_fullsized":"2024-09-10T18:34:40.817992+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:08.009806+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.009806+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002902614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":350,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.810767+0000","last_change":"2024-09-10T18:34:00.542831+0000","last_active":"2024-09-10T18:34:40.810767+0000","last_peered":"2024-09-10T18:34:40.810767+0000","last_clean":"2024-09-10T18:34:40.810767+0000","last_became_active":"2024-09-10T18:21:33.611203+0000","last_became_peered":"2024-09-10T18:21:33.611203+0000","last_unstale":"2024-09-10T18:34:40.810767+0000","last_undegraded":"2024-09-10T18:34:40.810767+0000","last_fullsized":"2024-09-10T18:34:40.810767+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-09-10T18:34:00.542737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:00.542737+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:11.466009+0000","scrub_duration":0.0030351190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":267,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.825399+0000","last_change":"2024-09-10T18:34:09.980779+0000","last_active":"2024-09-10T18:34:40.825399+0000","last_peered":"2024-09-10T18:34:40.825399+0000","last_clean":"2024-09-10T18:34:40.825399+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:34:40.825399+0000","last_undegraded":"2024-09-10T18:34:40.825399+0000","last_fullsized":"2024-09-10T18:34:40.825399+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:34:09.980722+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:34:09.980722+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:29.815756+0000","scrub_duration":0.001388053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":299,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.821198+0000","last_change":"2024-09-10T18:32:50.211889+0000","last_active":"2024-09-10T18:34:40.821198+0000","last_peered":"2024-09-10T18:34:40.821198+0000","last_clean":"2024-09-10T18:34:40.821198+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:34:40.821198+0000","last_undegraded":"2024-09-10T18:34:40.821198+0000","last_fullsized":"2024-09-10T18:34:40.821198+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:50.211414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:50.211414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007128512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":327,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.828587+0000","last_change":"2024-09-10T18:34:20.486108+0000","last_active":"2024-09-10T18:34:40.828587+0000","last_peered":"2024-09-10T18:34:40.828587+0000","last_clean":"2024-09-10T18:34:40.828587+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:34:40.828587+0000","last_undegraded":"2024-09-10T18:34:40.828587+0000","last_fullsized":"2024-09-10T18:34:40.828587+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:20.486034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:20.486034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:45.434346+0000","scrub_duration":0.0026235970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"294'13060","reported_seq":13373,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:45.937847+0000","last_change":"2024-09-10T18:34:12.213950+0000","last_active":"2024-09-10T18:34:45.937847+0000","last_peered":"2024-09-10T18:34:45.937847+0000","last_clean":"2024-09-10T18:34:45.937847+0000","last_became_active":"2024-09-10T18:32:03.554475+0000","last_became_peered":"2024-09-10T18:32:03.554475+0000","last_unstale":"2024-09-10T18:34:45.937847+0000","last_undegraded":"2024-09-10T18:34:45.937847+0000","last_fullsized":"2024-09-10T18:34:45.937847+0000","mapping_epoch":246,"log_start":"274'9163","ondisk_log_start":"274'9163","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'10828","last_scrub_stamp":"2024-09-10T18:34:12.213852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.213852+0000","objects_scrubbed":6169,"log_size":3897,"log_dups_size":0,"ondisk_log_size":3897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:28.363825+0000","scrub_duration":3.0156890789999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252575744,"num_objects":3854,"num_object_clones":0,"num_object_copies":11562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13060,"num_write_kb":541248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":361,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.819835+0000","last_change":"2024-09-10T18:34:32.827812+0000","last_active":"2024-09-10T18:34:40.819835+0000","last_peered":"2024-09-10T18:34:40.819835+0000","last_clean":"2024-09-10T18:34:40.819835+0000","last_became_active":"2024-09-10T18:34:32.827174+0000","last_became_peered":"2024-09-10T18:34:32.827174+0000","last_unstale":"2024-09-10T18:34:40.819835+0000","last_undegraded":"2024-09-10T18:34:40.819835+0000","last_fullsized":"2024-09-10T18:34:40.819835+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:06.053455+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.053455+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007740935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"240'25","reported_seq":61,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820510+0000","last_change":"2024-09-10T18:32:51.151265+0000","last_active":"2024-09-10T18:34:40.820510+0000","last_peered":"2024-09-10T18:34:40.820510+0000","last_clean":"2024-09-10T18:34:40.820510+0000","last_became_active":"2024-09-10T18:32:51.118251+0000","last_became_peered":"2024-09-10T18:32:51.118251+0000","last_unstale":"2024-09-10T18:34:40.820510+0000","last_undegraded":"2024-09-10T18:34:40.820510+0000","last_fullsized":"2024-09-10T18:34:40.820510+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:32:39.544846+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.544846+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033922850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":295,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.816360+0000","last_change":"2024-09-10T18:33:13.324775+0000","last_active":"2024-09-10T18:34:40.816360+0000","last_peered":"2024-09-10T18:34:40.816360+0000","last_clean":"2024-09-10T18:34:40.816360+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:34:40.816360+0000","last_undegraded":"2024-09-10T18:34:40.816360+0000","last_fullsized":"2024-09-10T18:34:40.816360+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:13.324556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:13.324556+0000","last_clean_scrub_stamp":"2024-09-10T18:33:13.324556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011863357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"189'9","reported_seq":497,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:42.846710+0000","last_change":"2024-09-10T18:34:17.470904+0000","last_active":"2024-09-10T18:34:42.846710+0000","last_peered":"2024-09-10T18:34:42.846710+0000","last_clean":"2024-09-10T18:34:42.846710+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:34:42.846710+0000","last_undegraded":"2024-09-10T18:34:42.846710+0000","last_fullsized":"2024-09-10T18:34:42.846710+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'9","last_scrub_stamp":"2024-09-10T18:34:17.470793+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:34:17.470793+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:22.477102+0000","scrub_duration":0.0044525839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":290,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.811032+0000","last_change":"2024-09-10T18:32:46.630172+0000","last_active":"2024-09-10T18:34:40.811032+0000","last_peered":"2024-09-10T18:34:40.811032+0000","last_clean":"2024-09-10T18:34:40.811032+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:34:40.811032+0000","last_undegraded":"2024-09-10T18:34:40.811032+0000","last_fullsized":"2024-09-10T18:34:40.811032+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:46.630078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_clean_scrub_stamp":"2024-09-10T18:32:46.630078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024328319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"76'81","reported_seq":33,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.809429+0000","last_change":"2024-09-10T18:34:32.818578+0000","last_active":"2024-09-10T18:34:40.809429+0000","last_peered":"2024-09-10T18:34:40.809429+0000","last_clean":"2024-09-10T18:34:40.809429+0000","last_became_active":"2024-09-10T18:34:32.818251+0000","last_became_peered":"2024-09-10T18:34:32.818251+0000","last_unstale":"2024-09-10T18:34:40.809429+0000","last_undegraded":"2024-09-10T18:34:40.809429+0000","last_fullsized":"2024-09-10T18:34:40.809429+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:48.006695+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:48.006695+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":5,"num_write":2,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"294'18313","reported_seq":18592,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:45.057477+0000","last_change":"2024-09-10T18:34:37.737676+0000","last_active":"2024-09-10T18:34:45.057477+0000","last_peered":"2024-09-10T18:34:45.057477+0000","last_clean":"2024-09-10T18:34:45.057477+0000","last_became_active":"2024-09-10T18:32:03.384417+0000","last_became_peered":"2024-09-10T18:32:03.384417+0000","last_unstale":"2024-09-10T18:34:45.057477+0000","last_undegraded":"2024-09-10T18:34:45.057477+0000","last_fullsized":"2024-09-10T18:34:45.057477+0000","mapping_epoch":246,"log_start":"283'14004","ondisk_log_start":"283'14004","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'17735","last_scrub_stamp":"2024-09-10T18:34:37.737496+0000","last_deep_scrub":"291'17735","last_deep_scrub_stamp":"2024-09-10T18:34:37.737496+0000","last_clean_scrub_stamp":"2024-09-10T18:34:37.737496+0000","objects_scrubbed":5250,"log_size":4309,"log_dups_size":0,"ondisk_log_size":4309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":26,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:45.542594+0000","scrub_duration":25.514376305999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261160960,"num_objects":3985,"num_object_clones":0,"num_object_copies":11955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13063,"num_write_kb":545536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"240'15","reported_seq":317,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.825583+0000","last_change":"2024-09-10T18:33:16.286611+0000","last_active":"2024-09-10T18:34:40.825583+0000","last_peered":"2024-09-10T18:34:40.825583+0000","last_clean":"2024-09-10T18:34:40.825583+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:34:40.825583+0000","last_undegraded":"2024-09-10T18:34:40.825583+0000","last_fullsized":"2024-09-10T18:34:40.825583+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:33:16.286523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:16.286523+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069638809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"265'2","reported_seq":321,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818692+0000","last_change":"2024-09-10T18:33:11.047358+0000","last_active":"2024-09-10T18:34:40.818692+0000","last_peered":"2024-09-10T18:34:40.818692+0000","last_clean":"2024-09-10T18:34:40.818692+0000","last_became_active":"2024-09-10T18:26:27.644997+0000","last_became_peered":"2024-09-10T18:26:27.644997+0000","last_unstale":"2024-09-10T18:34:40.818692+0000","last_undegraded":"2024-09-10T18:34:40.818692+0000","last_fullsized":"2024-09-10T18:34:40.818692+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'2","last_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_deep_scrub":"265'2","last_deep_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.047305+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021095425000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":315,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.827711+0000","last_change":"2024-09-10T18:32:52.590088+0000","last_active":"2024-09-10T18:34:40.827711+0000","last_peered":"2024-09-10T18:34:40.827711+0000","last_clean":"2024-09-10T18:34:40.827711+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:34:40.827711+0000","last_undegraded":"2024-09-10T18:34:40.827711+0000","last_fullsized":"2024-09-10T18:34:40.827711+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:52.589997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:52.589997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001802543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":288,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.821288+0000","last_change":"2024-09-10T18:32:44.239399+0000","last_active":"2024-09-10T18:34:40.821288+0000","last_peered":"2024-09-10T18:34:40.821288+0000","last_clean":"2024-09-10T18:34:40.821288+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:34:40.821288+0000","last_undegraded":"2024-09-10T18:34:40.821288+0000","last_fullsized":"2024-09-10T18:34:40.821288+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:44.239347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:44.239347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011983390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'81","reported_seq":288,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.825804+0000","last_change":"2024-09-10T18:34:15.938135+0000","last_active":"2024-09-10T18:34:40.825804+0000","last_peered":"2024-09-10T18:34:40.825804+0000","last_clean":"2024-09-10T18:34:40.825804+0000","last_became_active":"2024-09-10T18:34:02.803861+0000","last_became_peered":"2024-09-10T18:34:02.803861+0000","last_unstale":"2024-09-10T18:34:40.825804+0000","last_undegraded":"2024-09-10T18:34:40.825804+0000","last_fullsized":"2024-09-10T18:34:40.825804+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:15.938079+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:34:15.938079+0000","last_clean_scrub_stamp":"2024-09-10T18:34:15.938079+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:40.264047+0000","scrub_duration":0.0015442819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"294'12773","reported_seq":13062,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:45.058797+0000","last_change":"2024-09-10T18:34:19.945087+0000","last_active":"2024-09-10T18:34:45.058797+0000","last_peered":"2024-09-10T18:34:45.058797+0000","last_clean":"2024-09-10T18:34:45.058797+0000","last_became_active":"2024-09-10T18:32:03.382430+0000","last_became_peered":"2024-09-10T18:32:03.382430+0000","last_unstale":"2024-09-10T18:34:45.058797+0000","last_undegraded":"2024-09-10T18:34:45.058797+0000","last_fullsized":"2024-09-10T18:34:45.058797+0000","mapping_epoch":246,"log_start":"274'8413","ondisk_log_start":"274'8413","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'11233","last_scrub_stamp":"2024-09-10T18:34:19.945029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.945029+0000","objects_scrubbed":5568,"log_size":4360,"log_dups_size":0,"ondisk_log_size":4360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:38.722375+0000","scrub_duration":1.7273602640000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260636672,"num_objects":3977,"num_object_clones":0,"num_object_copies":11931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":536000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"240'44","reported_seq":384,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.825234+0000","last_change":"2024-09-10T18:34:03.021478+0000","last_active":"2024-09-10T18:34:40.825234+0000","last_peered":"2024-09-10T18:34:40.825234+0000","last_clean":"2024-09-10T18:34:40.825234+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:34:40.825234+0000","last_undegraded":"2024-09-10T18:34:40.825234+0000","last_fullsized":"2024-09-10T18:34:40.825234+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'44","last_scrub_stamp":"2024-09-10T18:34:03.021369+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:34:03.021369+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:14.897595+0000","scrub_duration":0.0045216129999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":320,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820525+0000","last_change":"2024-09-10T18:34:11.965336+0000","last_active":"2024-09-10T18:34:40.820525+0000","last_peered":"2024-09-10T18:34:40.820525+0000","last_clean":"2024-09-10T18:34:40.820525+0000","last_became_active":"2024-09-10T18:26:27.638408+0000","last_became_peered":"2024-09-10T18:26:27.638408+0000","last_unstale":"2024-09-10T18:34:40.820525+0000","last_undegraded":"2024-09-10T18:34:40.820525+0000","last_fullsized":"2024-09-10T18:34:40.820525+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:11.965263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:11.965263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:21.204938+0000","scrub_duration":0.0034110310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":329,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.827754+0000","last_change":"2024-09-10T18:34:21.447378+0000","last_active":"2024-09-10T18:34:40.827754+0000","last_peered":"2024-09-10T18:34:40.827754+0000","last_clean":"2024-09-10T18:34:40.827754+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:34:40.827754+0000","last_undegraded":"2024-09-10T18:34:40.827754+0000","last_fullsized":"2024-09-10T18:34:40.827754+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:21.447291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:34:21.447291+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.447291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:25.350687+0000","scrub_duration":0.0066377579999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":300,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.819619+0000","last_change":"2024-09-10T18:31:58.460588+0000","last_active":"2024-09-10T18:34:40.819619+0000","last_peered":"2024-09-10T18:34:40.819619+0000","last_clean":"2024-09-10T18:34:40.819619+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:34:40.819619+0000","last_undegraded":"2024-09-10T18:34:40.819619+0000","last_fullsized":"2024-09-10T18:34:40.819619+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:58.460533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:58.460533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001375914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"76'81","reported_seq":236,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.815018+0000","last_change":"2024-09-10T18:34:32.957284+0000","last_active":"2024-09-10T18:34:40.815018+0000","last_peered":"2024-09-10T18:34:40.815018+0000","last_clean":"2024-09-10T18:34:40.815018+0000","last_became_active":"2024-09-10T18:34:32.956761+0000","last_became_peered":"2024-09-10T18:34:32.956761+0000","last_unstale":"2024-09-10T18:34:40.815018+0000","last_undegraded":"2024-09-10T18:34:40.815018+0000","last_fullsized":"2024-09-10T18:34:40.815018+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.370561+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.370561+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029251390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54613,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"294'12848","reported_seq":13190,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:45.464189+0000","last_change":"2024-09-10T18:34:01.964247+0000","last_active":"2024-09-10T18:34:45.464189+0000","last_peered":"2024-09-10T18:34:45.464189+0000","last_clean":"2024-09-10T18:34:45.464189+0000","last_became_active":"2024-09-10T18:32:03.554423+0000","last_became_peered":"2024-09-10T18:32:03.554423+0000","last_unstale":"2024-09-10T18:34:45.464189+0000","last_undegraded":"2024-09-10T18:34:45.464189+0000","last_fullsized":"2024-09-10T18:34:45.464189+0000","mapping_epoch":246,"log_start":"274'8662","ondisk_log_start":"274'8662","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'10000","last_scrub_stamp":"2024-09-10T18:34:01.964158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:01.964158+0000","objects_scrubbed":6911,"log_size":4186,"log_dups_size":0,"ondisk_log_size":4186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:10.075365+0000","scrub_duration":3.4154075100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257687552,"num_objects":3932,"num_object_clones":0,"num_object_copies":11796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12848,"num_write_kb":536960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"240'18","reported_seq":336,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.809583+0000","last_change":"2024-09-10T18:32:54.528296+0000","last_active":"2024-09-10T18:34:40.809583+0000","last_peered":"2024-09-10T18:34:40.809583+0000","last_clean":"2024-09-10T18:34:40.809583+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:34:40.809583+0000","last_undegraded":"2024-09-10T18:34:40.809583+0000","last_fullsized":"2024-09-10T18:34:40.809583+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-09-10T18:32:54.528209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:54.528209+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056889829999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":313,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820204+0000","last_change":"2024-09-10T18:33:14.879881+0000","last_active":"2024-09-10T18:34:40.820204+0000","last_peered":"2024-09-10T18:34:40.820204+0000","last_clean":"2024-09-10T18:34:40.820204+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:34:40.820204+0000","last_undegraded":"2024-09-10T18:34:40.820204+0000","last_fullsized":"2024-09-10T18:34:40.820204+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:14.879787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:33:14.879787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044267550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":319,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.827408+0000","last_change":"2024-09-10T18:32:50.570319+0000","last_active":"2024-09-10T18:34:40.827408+0000","last_peered":"2024-09-10T18:34:40.827408+0000","last_clean":"2024-09-10T18:34:40.827408+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:34:40.827408+0000","last_undegraded":"2024-09-10T18:34:40.827408+0000","last_fullsized":"2024-09-10T18:34:40.827408+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:50.570113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:50.570113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059653969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":289,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.815973+0000","last_change":"2024-09-10T18:32:58.358775+0000","last_active":"2024-09-10T18:34:40.815973+0000","last_peered":"2024-09-10T18:34:40.815973+0000","last_clean":"2024-09-10T18:34:40.815973+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:34:40.815973+0000","last_undegraded":"2024-09-10T18:34:40.815973+0000","last_fullsized":"2024-09-10T18:34:40.815973+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:58.358694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.803236+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.358694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011821898000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"76'81","reported_seq":33,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820320+0000","last_change":"2024-09-10T18:34:32.822681+0000","last_active":"2024-09-10T18:34:40.820320+0000","last_peered":"2024-09-10T18:34:40.820320+0000","last_clean":"2024-09-10T18:34:40.820320+0000","last_became_active":"2024-09-10T18:34:32.822196+0000","last_became_peered":"2024-09-10T18:34:32.822196+0000","last_unstale":"2024-09-10T18:34:40.820320+0000","last_undegraded":"2024-09-10T18:34:40.820320+0000","last_fullsized":"2024-09-10T18:34:40.820320+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:58.561425+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.561425+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":5,"num_write":2,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"294'12977","reported_seq":13284,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:46.113014+0000","last_change":"2024-09-10T18:34:12.987347+0000","last_active":"2024-09-10T18:34:46.113014+0000","last_peered":"2024-09-10T18:34:46.113014+0000","last_clean":"2024-09-10T18:34:46.113014+0000","last_became_active":"2024-09-10T18:32:03.382890+0000","last_became_peered":"2024-09-10T18:32:03.382890+0000","last_unstale":"2024-09-10T18:34:46.113014+0000","last_undegraded":"2024-09-10T18:34:46.113014+0000","last_fullsized":"2024-09-10T18:34:46.113014+0000","mapping_epoch":246,"log_start":"273'7947","ondisk_log_start":"273'7947","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'10774","last_scrub_stamp":"2024-09-10T18:34:12.987248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.987248+0000","objects_scrubbed":6064,"log_size":5030,"log_dups_size":0,"ondisk_log_size":5030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:26.071275+0000","scrub_duration":2.4958883709999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248578048,"num_objects":3793,"num_object_clones":0,"num_object_copies":11379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12977,"num_write_kb":536640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":352,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818195+0000","last_change":"2024-09-10T18:34:06.163124+0000","last_active":"2024-09-10T18:34:40.818195+0000","last_peered":"2024-09-10T18:34:40.818195+0000","last_clean":"2024-09-10T18:34:40.818195+0000","last_became_active":"2024-09-10T18:26:27.645932+0000","last_became_peered":"2024-09-10T18:26:27.645932+0000","last_unstale":"2024-09-10T18:34:40.818195+0000","last_undegraded":"2024-09-10T18:34:40.818195+0000","last_fullsized":"2024-09-10T18:34:40.818195+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:34:06.163072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.163072+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:32.837570+0000","scrub_duration":0.0016579489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":313,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820231+0000","last_change":"2024-09-10T18:33:05.784757+0000","last_active":"2024-09-10T18:34:40.820231+0000","last_peered":"2024-09-10T18:34:40.820231+0000","last_clean":"2024-09-10T18:34:40.820231+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:34:40.820231+0000","last_undegraded":"2024-09-10T18:34:40.820231+0000","last_fullsized":"2024-09-10T18:34:40.820231+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:33:05.784670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:33:05.784670+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002289031,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":476,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:42.841758+0000","last_change":"2024-09-10T18:33:15.972246+0000","last_active":"2024-09-10T18:34:42.841758+0000","last_peered":"2024-09-10T18:34:42.841758+0000","last_clean":"2024-09-10T18:34:42.841758+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:34:42.841758+0000","last_undegraded":"2024-09-10T18:34:42.841758+0000","last_fullsized":"2024-09-10T18:34:42.841758+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:33:15.972166+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.972166+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004394907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":295,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.817476+0000","last_change":"2024-09-10T18:33:05.964339+0000","last_active":"2024-09-10T18:34:40.817476+0000","last_peered":"2024-09-10T18:34:40.817476+0000","last_clean":"2024-09-10T18:34:40.817476+0000","last_became_active":"2024-09-10T18:26:27.647511+0000","last_became_peered":"2024-09-10T18:26:27.647511+0000","last_unstale":"2024-09-10T18:34:40.817476+0000","last_undegraded":"2024-09-10T18:34:40.817476+0000","last_fullsized":"2024-09-10T18:34:40.817476+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:05.964034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:05.964034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015507839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'81","reported_seq":23,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.817684+0000","last_change":"2024-09-10T18:34:02.810441+0000","last_active":"2024-09-10T18:34:40.817684+0000","last_peered":"2024-09-10T18:34:40.817684+0000","last_clean":"2024-09-10T18:34:40.817684+0000","last_became_active":"2024-09-10T18:34:02.810126+0000","last_became_peered":"2024-09-10T18:34:02.810126+0000","last_unstale":"2024-09-10T18:34:40.817684+0000","last_undegraded":"2024-09-10T18:34:40.817684+0000","last_fullsized":"2024-09-10T18:34:40.817684+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:57.474295+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.474295+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":10,"num_write":6,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":258,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.826153+0000","last_change":"2024-09-10T18:34:16.895554+0000","last_active":"2024-09-10T18:34:40.826153+0000","last_peered":"2024-09-10T18:34:40.826153+0000","last_clean":"2024-09-10T18:34:40.826153+0000","last_became_active":"2024-09-10T18:26:27.640922+0000","last_became_peered":"2024-09-10T18:26:27.640922+0000","last_unstale":"2024-09-10T18:34:40.826153+0000","last_undegraded":"2024-09-10T18:34:40.826153+0000","last_fullsized":"2024-09-10T18:34:40.826153+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:16.895472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:34:16.895472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:28.880031+0000","scrub_duration":0.002634066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"240'12","reported_seq":311,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.814767+0000","last_change":"2024-09-10T18:32:56.384627+0000","last_active":"2024-09-10T18:34:40.814767+0000","last_peered":"2024-09-10T18:34:40.814767+0000","last_clean":"2024-09-10T18:34:40.814767+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:34:40.814767+0000","last_undegraded":"2024-09-10T18:34:40.814767+0000","last_fullsized":"2024-09-10T18:34:40.814767+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-10T18:32:56.384439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:56.384439+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010401713999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":299,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818860+0000","last_change":"2024-09-10T18:34:00.199169+0000","last_active":"2024-09-10T18:34:40.818860+0000","last_peered":"2024-09-10T18:34:40.818860+0000","last_clean":"2024-09-10T18:34:40.818860+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:34:40.818860+0000","last_undegraded":"2024-09-10T18:34:40.818860+0000","last_fullsized":"2024-09-10T18:34:40.818860+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:00.199087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:00.199087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:11.766473+0000","scrub_duration":0.016944524999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":304,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820283+0000","last_change":"2024-09-10T18:33:58.161088+0000","last_active":"2024-09-10T18:34:40.820283+0000","last_peered":"2024-09-10T18:34:40.820283+0000","last_clean":"2024-09-10T18:34:40.820283+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:34:40.820283+0000","last_undegraded":"2024-09-10T18:34:40.820283+0000","last_fullsized":"2024-09-10T18:34:40.820283+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:58.161019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:58.161019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:23.203002+0000","scrub_duration":0.0042556759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'81","reported_seq":229,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818058+0000","last_change":"2024-09-10T18:34:02.816592+0000","last_active":"2024-09-10T18:34:40.818058+0000","last_peered":"2024-09-10T18:34:40.818058+0000","last_clean":"2024-09-10T18:34:40.818058+0000","last_became_active":"2024-09-10T18:34:02.815880+0000","last_became_peered":"2024-09-10T18:34:02.815880+0000","last_unstale":"2024-09-10T18:34:40.818058+0000","last_undegraded":"2024-09-10T18:34:40.818058+0000","last_fullsized":"2024-09-10T18:34:40.818058+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:48.466654+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.466654+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"240'19","reported_seq":324,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.822138+0000","last_change":"2024-09-10T18:33:08.284482+0000","last_active":"2024-09-10T18:34:40.822138+0000","last_peered":"2024-09-10T18:34:40.822138+0000","last_clean":"2024-09-10T18:34:40.822138+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:34:40.822138+0000","last_undegraded":"2024-09-10T18:34:40.822138+0000","last_fullsized":"2024-09-10T18:34:40.822138+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:33:08.284394+0000","last_deep_scrub":"189'13","last_deep_scrub_stamp":"2024-09-10T18:28:08.976162+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.284394+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010009881999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":310,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820005+0000","last_change":"2024-09-10T18:33:16.937511+0000","last_active":"2024-09-10T18:34:40.820005+0000","last_peered":"2024-09-10T18:34:40.820005+0000","last_clean":"2024-09-10T18:34:40.820005+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:34:40.820005+0000","last_undegraded":"2024-09-10T18:34:40.820005+0000","last_fullsized":"2024-09-10T18:34:40.820005+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:16.937390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:33:16.937390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.024770533000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":291,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.814528+0000","last_change":"2024-09-10T18:32:39.161911+0000","last_active":"2024-09-10T18:34:40.814528+0000","last_peered":"2024-09-10T18:34:40.814528+0000","last_clean":"2024-09-10T18:34:40.814528+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:34:40.814528+0000","last_undegraded":"2024-09-10T18:34:40.814528+0000","last_fullsized":"2024-09-10T18:34:40.814528+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:39.161653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.161653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0093847749999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":307,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.828014+0000","last_change":"2024-09-10T18:32:43.621566+0000","last_active":"2024-09-10T18:34:40.828014+0000","last_peered":"2024-09-10T18:34:40.828014+0000","last_clean":"2024-09-10T18:34:40.828014+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:34:40.828014+0000","last_undegraded":"2024-09-10T18:34:40.828014+0000","last_fullsized":"2024-09-10T18:34:40.828014+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:43.621478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:43.621478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031857410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'81","reported_seq":23,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.817129+0000","last_change":"2024-09-10T18:34:02.797580+0000","last_active":"2024-09-10T18:34:40.817129+0000","last_peered":"2024-09-10T18:34:40.817129+0000","last_clean":"2024-09-10T18:34:40.817129+0000","last_became_active":"2024-09-10T18:34:02.797304+0000","last_became_peered":"2024-09-10T18:34:02.797304+0000","last_unstale":"2024-09-10T18:34:40.817129+0000","last_undegraded":"2024-09-10T18:34:40.817129+0000","last_fullsized":"2024-09-10T18:34:40.817129+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:58.780613+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.780613+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"240'19","reported_seq":331,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818881+0000","last_change":"2024-09-10T18:33:15.955719+0000","last_active":"2024-09-10T18:34:40.818881+0000","last_peered":"2024-09-10T18:34:40.818881+0000","last_clean":"2024-09-10T18:34:40.818881+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:34:40.818881+0000","last_undegraded":"2024-09-10T18:34:40.818881+0000","last_fullsized":"2024-09-10T18:34:40.818881+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:33:15.955638+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.955638+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:44.846427+0000","scrub_duration":0.021358801,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":304,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820572+0000","last_change":"2024-09-10T18:33:12.342537+0000","last_active":"2024-09-10T18:34:40.820572+0000","last_peered":"2024-09-10T18:34:40.820572+0000","last_clean":"2024-09-10T18:34:40.820572+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:34:40.820572+0000","last_undegraded":"2024-09-10T18:34:40.820572+0000","last_fullsized":"2024-09-10T18:34:40.820572+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:12.342432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_clean_scrub_stamp":"2024-09-10T18:33:12.342432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043002429999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":458,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:42.839034+0000","last_change":"2024-09-10T18:32:40.183655+0000","last_active":"2024-09-10T18:34:42.839034+0000","last_peered":"2024-09-10T18:34:42.839034+0000","last_clean":"2024-09-10T18:34:42.839034+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:34:42.839034+0000","last_undegraded":"2024-09-10T18:34:42.839034+0000","last_fullsized":"2024-09-10T18:34:42.839034+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:32:40.183571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:40.183571+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014890771000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":34,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.814906+0000","last_change":"2024-09-10T18:33:29.142401+0000","last_active":"2024-09-10T18:34:40.814906+0000","last_peered":"2024-09-10T18:34:40.814906+0000","last_clean":"2024-09-10T18:34:40.814906+0000","last_became_active":"2024-09-10T18:33:29.141936+0000","last_became_peered":"2024-09-10T18:33:29.141936+0000","last_unstale":"2024-09-10T18:34:40.814906+0000","last_undegraded":"2024-09-10T18:34:40.814906+0000","last_fullsized":"2024-09-10T18:34:40.814906+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:11.727786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.727786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"76'81","reported_seq":94,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.821089+0000","last_change":"2024-09-10T18:34:22.551031+0000","last_active":"2024-09-10T18:34:40.821089+0000","last_peered":"2024-09-10T18:34:40.821089+0000","last_clean":"2024-09-10T18:34:40.821089+0000","last_became_active":"2024-09-10T18:34:02.810381+0000","last_became_peered":"2024-09-10T18:34:02.810381+0000","last_unstale":"2024-09-10T18:34:40.821089+0000","last_undegraded":"2024-09-10T18:34:40.821089+0000","last_fullsized":"2024-09-10T18:34:40.821089+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:22.550782+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.550782+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:47.692871+0000","scrub_duration":0.0052607840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":333,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.815255+0000","last_change":"2024-09-10T18:33:59.230861+0000","last_active":"2024-09-10T18:34:40.815255+0000","last_peered":"2024-09-10T18:34:40.815255+0000","last_clean":"2024-09-10T18:34:40.815255+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:34:40.815255+0000","last_undegraded":"2024-09-10T18:34:40.815255+0000","last_fullsized":"2024-09-10T18:34:40.815255+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-10T18:33:59.227198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:59.227198+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:04.026375+0000","scrub_duration":0.012577204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":303,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820409+0000","last_change":"2024-09-10T18:32:39.535817+0000","last_active":"2024-09-10T18:34:40.820409+0000","last_peered":"2024-09-10T18:34:40.820409+0000","last_clean":"2024-09-10T18:34:40.820409+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:34:40.820409+0000","last_undegraded":"2024-09-10T18:34:40.820409+0000","last_fullsized":"2024-09-10T18:34:40.820409+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:39.535749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.535749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046018179999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":467,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:34:42.839171+0000","last_change":"2024-09-10T18:32:43.942068+0000","last_active":"2024-09-10T18:34:42.839171+0000","last_peered":"2024-09-10T18:34:42.839171+0000","last_clean":"2024-09-10T18:34:42.839171+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:34:42.839171+0000","last_undegraded":"2024-09-10T18:34:42.839171+0000","last_fullsized":"2024-09-10T18:34:42.839171+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:32:43.941975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:43.941975+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046448039999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":298,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.821765+0000","last_change":"2024-09-10T18:34:23.556824+0000","last_active":"2024-09-10T18:34:40.821765+0000","last_peered":"2024-09-10T18:34:40.821765+0000","last_clean":"2024-09-10T18:34:40.821765+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:34:40.821765+0000","last_undegraded":"2024-09-10T18:34:40.821765+0000","last_fullsized":"2024-09-10T18:34:40.821765+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:23.556773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:23.556773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:47.130693+0000","scrub_duration":0.001126988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'81","reported_seq":14,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818843+0000","last_change":"2024-09-10T18:34:34.881583+0000","last_active":"2024-09-10T18:34:40.818843+0000","last_peered":"2024-09-10T18:34:40.818843+0000","last_clean":"2024-09-10T18:34:40.818843+0000","last_became_active":"2024-09-10T18:34:34.881186+0000","last_became_peered":"2024-09-10T18:34:34.881186+0000","last_unstale":"2024-09-10T18:34:40.818843+0000","last_undegraded":"2024-09-10T18:34:40.818843+0000","last_fullsized":"2024-09-10T18:34:40.818843+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:05.012053+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:05.012053+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":339,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.810292+0000","last_change":"2024-09-10T18:33:15.612721+0000","last_active":"2024-09-10T18:34:40.810292+0000","last_peered":"2024-09-10T18:34:40.810292+0000","last_clean":"2024-09-10T18:34:40.810292+0000","last_became_active":"2024-09-10T18:26:27.649202+0000","last_became_peered":"2024-09-10T18:26:27.649202+0000","last_unstale":"2024-09-10T18:34:40.810292+0000","last_undegraded":"2024-09-10T18:34:40.810292+0000","last_fullsized":"2024-09-10T18:34:40.810292+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:33:15.612638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.612638+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002578828,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":313,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.810578+0000","last_change":"2024-09-10T18:34:04.380240+0000","last_active":"2024-09-10T18:34:40.810578+0000","last_peered":"2024-09-10T18:34:40.810578+0000","last_clean":"2024-09-10T18:34:40.810578+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:34:40.810578+0000","last_undegraded":"2024-09-10T18:34:40.810578+0000","last_fullsized":"2024-09-10T18:34:40.810578+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:04.380160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:34:04.380160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:34.095135+0000","scrub_duration":0.0018137730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":312,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.819882+0000","last_change":"2024-09-10T18:34:03.521909+0000","last_active":"2024-09-10T18:34:40.819882+0000","last_peered":"2024-09-10T18:34:40.819882+0000","last_clean":"2024-09-10T18:34:40.819882+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:34:40.819882+0000","last_undegraded":"2024-09-10T18:34:40.819882+0000","last_fullsized":"2024-09-10T18:34:40.819882+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:03.521806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_clean_scrub_stamp":"2024-09-10T18:34:03.521806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:19.398042+0000","scrub_duration":0.005196946,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":289,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.813413+0000","last_change":"2024-09-10T18:34:14.263579+0000","last_active":"2024-09-10T18:34:40.813413+0000","last_peered":"2024-09-10T18:34:40.813413+0000","last_clean":"2024-09-10T18:34:40.813413+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:34:40.813413+0000","last_undegraded":"2024-09-10T18:34:40.813413+0000","last_fullsized":"2024-09-10T18:34:40.813413+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:14.263499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:34:14.263499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:30.466656+0000","scrub_duration":0.0019685509999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'81","reported_seq":14,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818579+0000","last_change":"2024-09-10T18:34:34.858726+0000","last_active":"2024-09-10T18:34:40.818579+0000","last_peered":"2024-09-10T18:34:40.818579+0000","last_clean":"2024-09-10T18:34:40.818579+0000","last_became_active":"2024-09-10T18:34:34.858291+0000","last_became_peered":"2024-09-10T18:34:34.858291+0000","last_unstale":"2024-09-10T18:34:40.818579+0000","last_undegraded":"2024-09-10T18:34:40.818579+0000","last_fullsized":"2024-09-10T18:34:40.818579+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:08.035431+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:08.035431+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"240'15","reported_seq":343,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.822156+0000","last_change":"2024-09-10T18:34:19.608537+0000","last_active":"2024-09-10T18:34:40.822156+0000","last_peered":"2024-09-10T18:34:40.822156+0000","last_clean":"2024-09-10T18:34:40.822156+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:34:40.822156+0000","last_undegraded":"2024-09-10T18:34:40.822156+0000","last_fullsized":"2024-09-10T18:34:40.822156+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:34:19.608445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.608445+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:24.709727+0000","scrub_duration":0.012630626000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":306,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820471+0000","last_change":"2024-09-10T18:33:01.781370+0000","last_active":"2024-09-10T18:34:40.820471+0000","last_peered":"2024-09-10T18:34:40.820471+0000","last_clean":"2024-09-10T18:34:40.820471+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:34:40.820471+0000","last_undegraded":"2024-09-10T18:34:40.820471+0000","last_fullsized":"2024-09-10T18:34:40.820471+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:01.781295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.781295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016784879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":284,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.816088+0000","last_change":"2024-09-10T18:34:06.262708+0000","last_active":"2024-09-10T18:34:40.816088+0000","last_peered":"2024-09-10T18:34:40.816088+0000","last_clean":"2024-09-10T18:34:40.816088+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:34:40.816088+0000","last_undegraded":"2024-09-10T18:34:40.816088+0000","last_fullsized":"2024-09-10T18:34:40.816088+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:44.838141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:31:44.838141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011071239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":298,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.820151+0000","last_change":"2024-09-10T18:34:00.512903+0000","last_active":"2024-09-10T18:34:40.820151+0000","last_peered":"2024-09-10T18:34:40.820151+0000","last_clean":"2024-09-10T18:34:40.820151+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:34:40.820151+0000","last_undegraded":"2024-09-10T18:34:40.820151+0000","last_fullsized":"2024-09-10T18:34:40.820151+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:00.512670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:00.512670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:03.765496+0000","scrub_duration":0.0068783309999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'81","reported_seq":23,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.827918+0000","last_change":"2024-09-10T18:34:02.802928+0000","last_active":"2024-09-10T18:34:40.827918+0000","last_peered":"2024-09-10T18:34:40.827918+0000","last_clean":"2024-09-10T18:34:40.827918+0000","last_became_active":"2024-09-10T18:34:02.802580+0000","last_became_peered":"2024-09-10T18:34:02.802580+0000","last_unstale":"2024-09-10T18:34:40.827918+0000","last_undegraded":"2024-09-10T18:34:40.827918+0000","last_fullsized":"2024-09-10T18:34:40.827918+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.370561+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.370561+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029251390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":331,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.813686+0000","last_change":"2024-09-10T18:32:46.253109+0000","last_active":"2024-09-10T18:34:40.813686+0000","last_peered":"2024-09-10T18:34:40.813686+0000","last_clean":"2024-09-10T18:34:40.813686+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:34:40.813686+0000","last_undegraded":"2024-09-10T18:34:40.813686+0000","last_fullsized":"2024-09-10T18:34:40.813686+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:32:46.252895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:46.252895+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003188967,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":314,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818893+0000","last_change":"2024-09-10T18:34:08.134374+0000","last_active":"2024-09-10T18:34:40.818893+0000","last_peered":"2024-09-10T18:34:40.818893+0000","last_clean":"2024-09-10T18:34:40.818893+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:34:40.818893+0000","last_undegraded":"2024-09-10T18:34:40.818893+0000","last_fullsized":"2024-09-10T18:34:40.818893+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:08.134230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:34:08.134230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:31.369105+0000","scrub_duration":0.0077618749999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":299,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.819213+0000","last_change":"2024-09-10T18:31:50.423429+0000","last_active":"2024-09-10T18:34:40.819213+0000","last_peered":"2024-09-10T18:34:40.819213+0000","last_clean":"2024-09-10T18:34:40.819213+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:34:40.819213+0000","last_undegraded":"2024-09-10T18:34:40.819213+0000","last_fullsized":"2024-09-10T18:34:40.819213+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:50.423374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:31:50.423374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00116716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":306,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.816292+0000","last_change":"2024-09-10T18:33:18.227516+0000","last_active":"2024-09-10T18:34:40.816292+0000","last_peered":"2024-09-10T18:34:40.816292+0000","last_clean":"2024-09-10T18:34:40.816292+0000","last_became_active":"2024-09-10T18:26:27.636996+0000","last_became_peered":"2024-09-10T18:26:27.636996+0000","last_unstale":"2024-09-10T18:34:40.816292+0000","last_undegraded":"2024-09-10T18:34:40.816292+0000","last_fullsized":"2024-09-10T18:34:40.816292+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:18.227428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:18.227428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063780959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'81","reported_seq":14,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.819821+0000","last_change":"2024-09-10T18:34:34.869108+0000","last_active":"2024-09-10T18:34:40.819821+0000","last_peered":"2024-09-10T18:34:40.819821+0000","last_clean":"2024-09-10T18:34:40.819821+0000","last_became_active":"2024-09-10T18:34:34.868371+0000","last_became_peered":"2024-09-10T18:34:34.868371+0000","last_unstale":"2024-09-10T18:34:40.819821+0000","last_undegraded":"2024-09-10T18:34:40.819821+0000","last_fullsized":"2024-09-10T18:34:40.819821+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:22.344759+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.344759+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":312,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.812840+0000","last_change":"2024-09-10T18:32:47.287083+0000","last_active":"2024-09-10T18:34:40.812840+0000","last_peered":"2024-09-10T18:34:40.812840+0000","last_clean":"2024-09-10T18:34:40.812840+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:34:40.812840+0000","last_undegraded":"2024-09-10T18:34:40.812840+0000","last_fullsized":"2024-09-10T18:34:40.812840+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-09-10T18:32:47.286977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:47.286977+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017807315000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":297,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.821679+0000","last_change":"2024-09-10T18:32:57.214331+0000","last_active":"2024-09-10T18:34:40.821679+0000","last_peered":"2024-09-10T18:34:40.821679+0000","last_clean":"2024-09-10T18:34:40.821679+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:34:40.821679+0000","last_undegraded":"2024-09-10T18:34:40.821679+0000","last_fullsized":"2024-09-10T18:34:40.821679+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:57.214242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:32:57.214242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015466415000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":262,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.825062+0000","last_change":"2024-09-10T18:34:18.819184+0000","last_active":"2024-09-10T18:34:40.825062+0000","last_peered":"2024-09-10T18:34:40.825062+0000","last_clean":"2024-09-10T18:34:40.825062+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:34:40.825062+0000","last_undegraded":"2024-09-10T18:34:40.825062+0000","last_fullsized":"2024-09-10T18:34:40.825062+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:18.819090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:18.819090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:35.004598+0000","scrub_duration":0.0015527620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":297,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.819983+0000","last_change":"2024-09-10T18:33:13.844984+0000","last_active":"2024-09-10T18:34:40.819983+0000","last_peered":"2024-09-10T18:34:40.819983+0000","last_clean":"2024-09-10T18:34:40.819983+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:34:40.819983+0000","last_undegraded":"2024-09-10T18:34:40.819983+0000","last_fullsized":"2024-09-10T18:34:40.819983+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:13.844781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:13.844781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022954300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'81","reported_seq":14,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.809833+0000","last_change":"2024-09-10T18:34:34.867779+0000","last_active":"2024-09-10T18:34:40.809833+0000","last_peered":"2024-09-10T18:34:40.809833+0000","last_clean":"2024-09-10T18:34:40.809833+0000","last_became_active":"2024-09-10T18:34:34.867380+0000","last_became_peered":"2024-09-10T18:34:34.867380+0000","last_unstale":"2024-09-10T18:34:40.809833+0000","last_undegraded":"2024-09-10T18:34:40.809833+0000","last_fullsized":"2024-09-10T18:34:40.809833+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:56.155354+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:56.155354+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":307,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.818562+0000","last_change":"2024-09-10T18:33:18.177364+0000","last_active":"2024-09-10T18:34:40.818562+0000","last_peered":"2024-09-10T18:34:40.818562+0000","last_clean":"2024-09-10T18:34:40.818562+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:34:40.818562+0000","last_undegraded":"2024-09-10T18:34:40.818562+0000","last_fullsized":"2024-09-10T18:34:40.818562+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_clean_scrub_stamp":"2024-09-10T18:33:18.177264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029244380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":295,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.813136+0000","last_change":"2024-09-10T18:34:03.175239+0000","last_active":"2024-09-10T18:34:40.813136+0000","last_peered":"2024-09-10T18:34:40.813136+0000","last_clean":"2024-09-10T18:34:40.813136+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:34:40.813136+0000","last_undegraded":"2024-09-10T18:34:40.813136+0000","last_fullsized":"2024-09-10T18:34:40.813136+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:03.175158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:34:03.175158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:17.890582+0000","scrub_duration":0.0046326559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":313,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.825812+0000","last_change":"2024-09-10T18:33:59.017207+0000","last_active":"2024-09-10T18:34:40.825812+0000","last_peered":"2024-09-10T18:34:40.825812+0000","last_clean":"2024-09-10T18:34:40.825812+0000","last_became_active":"2024-09-10T18:26:27.648112+0000","last_became_peered":"2024-09-10T18:26:27.648112+0000","last_unstale":"2024-09-10T18:34:40.825812+0000","last_undegraded":"2024-09-10T18:34:40.825812+0000","last_fullsized":"2024-09-10T18:34:40.825812+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:33:59.017151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:59.017151+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:07.309204+0000","scrub_duration":0.0027087330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":300,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:34:40.821905+0000","last_change":"2024-09-10T18:34:02.502004+0000","last_active":"2024-09-10T18:34:40.821905+0000","last_peered":"2024-09-10T18:34:40.821905+0000","last_clean":"2024-09-10T18:34:40.821905+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:34:40.821905+0000","last_undegraded":"2024-09-10T18:34:40.821905+0000","last_fullsized":"2024-09-10T18:34:40.821905+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:02.501929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:34:02.501929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:23.332476+0000","scrub_duration":0.0086866479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4082368536,"num_objects":62293,"num_object_clones":0,"num_object_copies":186879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62293,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206885,"num_write_kb":8613633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12271955968,"data_stored":12271943752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68307,"ondisk_log_size":68307,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":736,"ondisk_log_size":736,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":786,"num_read_kb":781,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":791632,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":76,"num_write":40,"num_write_kb":555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":829053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2511,"ondisk_log_size":2511,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234146,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2572288,"kb_used_data":2270392,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91172864,"statfs":{"total":95995035648,"available":93361012736,"internally_reserved":0,"allocated":2324881408,"data_stored":2324407144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":2.6840000000000002,"15min":1.8140000000000001},"min":{"1min":0.33200000000000002,"5min":0.21099999999999999,"15min":0.20799999999999999},"max":{"1min":6.5209999999999999,"5min":75.506,"15min":85.012},"last":2.2029999999999998},{"interface":"front","average":{"1min":0.98899999999999999,"5min":2.472,"15min":1.728},"min":{"1min":0.35599999999999998,"5min":0.23699999999999999,"15min":0.222},"max":{"1min":2.7130000000000001,"5min":75.710999999999999,"15min":84.665999999999997},"last":0.47699999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.8600000000000001,"15min":1.23},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.214},"max":{"1min":2.4510000000000001,"5min":49.317999999999998,"15min":49.317999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.83199999999999996,"5min":1.6439999999999999,"15min":1.167},"min":{"1min":0.28599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.2309999999999999,"5min":49.353000000000002,"15min":49.353000000000002},"last":0.26800000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":0.93500000000000005,"15min":0.79900000000000004},"min":{"1min":0.27400000000000002,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":3.254,"5min":5.008,"15min":5.3150000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.0660000000000001,"15min":0.871},"min":{"1min":0.32300000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.5150000000000001,"5min":5.4029999999999996,"15min":9.9450000000000003},"last":0.53200000000000003}]},{"osd":3,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.163,"15min":1.373},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.7380000000000004,"5min":4.7380000000000004,"15min":86.305999999999997},"last":5.2130000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.2869999999999999,"15min":1.425},"min":{"1min":0.30299999999999999,"5min":0.29699999999999999,"15min":0.26700000000000002},"max":{"1min":2.7200000000000002,"5min":7.3339999999999996,"15min":88.364000000000004},"last":3.3260000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.626,"15min":0.48699999999999999},"min":{"1min":0.26700000000000002,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":2.2610000000000001,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":0.44},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68899999999999995,"15min":0.51000000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.223},"max":{"1min":1.607,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":0.79400000000000004}]},{"osd":5,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.0760000000000001,"15min":0.65400000000000003},"min":{"1min":0.30099999999999999,"5min":0.27600000000000002,"15min":0.23499999999999999},"max":{"1min":4.1479999999999997,"5min":7.3109999999999999,"15min":7.3109999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.244,"5min":0.93000000000000005,"15min":0.61799999999999999},"min":{"1min":0.38900000000000001,"5min":0.27100000000000002,"15min":0.219},"max":{"1min":3.911,"5min":6.3360000000000003,"15min":6.3360000000000003},"last":0.51900000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90100000000000002,"15min":0.629},"min":{"1min":0.32000000000000001,"5min":0.28899999999999998,"15min":0.27000000000000002},"max":{"1min":2.5699999999999998,"5min":7.2750000000000004,"15min":7.2750000000000004},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.81299999999999994,"15min":0.58499999999999996},"min":{"1min":0.373,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.2090000000000001,"5min":4.585,"15min":4.585},"last":0.499}]}]},{"osd":6,"up_from":144,"seq":618475290725,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1055532,"kb_used_data":752928,"kb_used_omap":7,"kb_used_meta":302584,"kb_avail":92689620,"statfs":{"total":95995035648,"available":94914170880,"internally_reserved":0,"allocated":770998272,"data_stored":770548188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":309846103},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":1.373,"15min":1.853},"min":{"1min":0.24399999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.831,"5min":8.3949999999999996,"15min":30.629999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.95999999999999996,"15min":1.375},"min":{"1min":0.247,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":4.3310000000000004,"5min":8.4640000000000004,"15min":30.597000000000001},"last":0.40500000000000003}]},{"osd":1,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.131,"15min":0.94199999999999995},"min":{"1min":0.29099999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":3.5569999999999999,"5min":8.0329999999999995,"15min":30.984000000000002},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.206,"15min":1.1399999999999999},"min":{"1min":0.30499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.1850000000000001,"5min":6.7859999999999996,"15min":30.571000000000002},"last":0.28599999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":1.0489999999999999,"15min":1.0600000000000001},"min":{"1min":0.27200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.0760000000000001,"5min":18.353000000000002,"15min":18.353000000000002},"last":1.052},{"interface":"front","average":{"1min":0.82599999999999996,"5min":1.028,"15min":0.85099999999999998},"min":{"1min":0.28799999999999998,"5min":0.26600000000000001,"15min":0.23400000000000001},"max":{"1min":3.0880000000000001,"5min":11.164999999999999,"15min":11.164999999999999},"last":0.33400000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.2849999999999999,"15min":1.964},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27800000000000002},"max":{"1min":4.8890000000000002,"5min":16.831,"15min":61.703000000000003},"last":0.49099999999999999},{"interface":"front","average":{"1min":2.0230000000000001,"5min":1.494,"15min":2.0259999999999998},"min":{"1min":0.30399999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":17.786000000000001,"5min":17.786000000000001,"15min":61.814},"last":0.70399999999999996}]},{"osd":4,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.63500000000000001,"15min":0.54500000000000004},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.23699999999999999},"max":{"1min":2.8300000000000001,"5min":5.4950000000000001,"15min":5.4950000000000001},"last":0.376},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.60099999999999998,"15min":0.63700000000000001},"min":{"1min":0.26900000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.2440000000000002,"5min":4.6390000000000002,"15min":4.6390000000000002},"last":0.38800000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.69099999999999995,"15min":0.54400000000000004},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.24399999999999999},"max":{"1min":2.8969999999999998,"5min":4.6619999999999999,"15min":4.6619999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.68799999999999994,"15min":0.51700000000000002},"min":{"1min":0.309,"5min":0.29699999999999999,"15min":0.26800000000000002},"max":{"1min":1.9830000000000001,"5min":2.794,"15min":2.794},"last":0.65400000000000003}]},{"osd":7,"last update":"Tue Sep 10 18:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.65200000000000002,"15min":0.54500000000000004},"min":{"1min":0.32600000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.929,"5min":2.7040000000000002,"15min":2.7040000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.755,"15min":0.57799999999999996},"min":{"1min":0.34399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.2320000000000002,"5min":5.5309999999999997,"15min":5.5309999999999997},"last":0.59299999999999997}]}]},{"osd":5,"up_from":39,"seq":167503724717,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2875080,"kb_used_data":2289340,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":90870072,"statfs":{"total":95995035648,"available":93050953728,"internally_reserved":0,"allocated":2344284160,"data_stored":2343806306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":1.367,"15min":1.4590000000000001},"min":{"1min":0.249,"5min":0.23499999999999999,"15min":0.21099999999999999},"max":{"1min":2.5259999999999998,"5min":26.288,"15min":52.033999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.218,"15min":1.29},"min":{"1min":0.246,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":3.9820000000000002,"5min":24.178000000000001,"15min":52.167999999999999},"last":0.33100000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.379,"15min":0.93799999999999994},"min":{"1min":0.33900000000000002,"5min":0.315,"15min":0.22600000000000001},"max":{"1min":4.4800000000000004,"5min":24.16,"15min":24.16},"last":25.190000000000001},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.343,"15min":1.0109999999999999},"min":{"1min":0.30399999999999999,"5min":0.28499999999999998,"15min":0.247},"max":{"1min":2.984,"5min":24.129999999999999,"15min":24.129999999999999},"last":25.210999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":1.601,"15min":1.1819999999999999},"min":{"1min":0.28899999999999998,"5min":0.245,"15min":0.24299999999999999},"max":{"1min":1.8819999999999999,"5min":54.530000000000001,"15min":54.530000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":1.5700000000000001,"15min":1.1539999999999999},"min":{"1min":0.32600000000000001,"5min":0.247,"15min":0.247},"max":{"1min":2.871,"5min":54.514000000000003,"15min":54.514000000000003},"last":0.316}]},{"osd":3,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":2.1320000000000001,"5min":1.782,"15min":1.4039999999999999},"min":{"1min":0.29899999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":17.198,"5min":30.475000000000001,"15min":31.207000000000001},"last":22.323},{"interface":"front","average":{"1min":2.472,"5min":1.772,"15min":1.3109999999999999},"min":{"1min":0.35899999999999999,"5min":0.32000000000000001,"15min":0.29699999999999999},"max":{"1min":16.934999999999999,"5min":30.172000000000001,"15min":31.199000000000002},"last":22.971}]},{"osd":4,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66700000000000004,"15min":0.496},"min":{"1min":0.27900000000000003,"5min":0.22900000000000001,"15min":0.20799999999999999},"max":{"1min":2.3580000000000001,"5min":4.5199999999999996,"15min":4.5199999999999996},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.73599999999999999,"15min":0.55200000000000005},"min":{"1min":0.315,"5min":0.24199999999999999,"15min":0.22700000000000001},"max":{"1min":2.4700000000000002,"5min":6.798,"15min":6.798},"last":0.41999999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.65200000000000002,"15min":0.64500000000000002},"min":{"1min":0.34699999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.1819999999999999,"5min":2.7469999999999999,"15min":4.7859999999999996},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.72899999999999998,"15min":0.65200000000000002},"min":{"1min":0.28100000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.2850000000000001,"5min":5.3680000000000003,"15min":5.3860000000000001},"last":0.36599999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:34:34 2024","interfaces":[{"interface":"back","average":{"1min":2.3029999999999999,"5min":1.0640000000000001,"15min":0.70499999999999996},"min":{"1min":0.35699999999999998,"5min":0.25900000000000001,"15min":0.22500000000000001},"max":{"1min":28.934000000000001,"5min":28.934000000000001,"15min":28.934000000000001},"last":0.432},{"interface":"front","average":{"1min":1.2030000000000001,"5min":0.82299999999999995,"15min":0.59499999999999997},"min":{"1min":0.38100000000000001,"5min":0.25900000000000001,"15min":0.24299999999999999},"max":{"1min":5.2190000000000003,"5min":5.2190000000000003,"15min":5.2190000000000003},"last":0.377}]}]},{"osd":4,"up_from":33,"seq":141733920945,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1072828,"kb_used_data":771312,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92672324,"statfs":{"total":95995035648,"available":94896459776,"internally_reserved":0,"allocated":789823488,"data_stored":789356659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.95499999999999996,"15min":1.024},"min":{"1min":0.248,"5min":0.21199999999999999,"15min":0.188},"max":{"1min":2.5960000000000001,"5min":4.3730000000000002,"15min":31.585000000000001},"last":3.5619999999999998},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.92900000000000005,"15min":1.006},"min":{"1min":0.28799999999999998,"5min":0.189,"15min":0.189},"max":{"1min":4.8929999999999998,"5min":4.8929999999999998,"15min":29.802},"last":1.1839999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:34:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.5069999999999999,"15min":1.0069999999999999},"min":{"1min":0.254,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":4.3390000000000004,"5min":50.045000000000002,"15min":50.045000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.7869999999999999,"15min":1.095},"min":{"1min":0.30299999999999999,"5min":0.27100000000000002,"15min":0.22900000000000001},"max":{"1min":5.5590000000000002,"5min":50.283000000000001,"15min":50.283000000000001},"last":0.48499999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.88300000000000001,"15min":0.86899999999999999},"min":{"1min":0.28499999999999998,"5min":0.249,"15min":0.249},"max":{"1min":1.857,"5min":5.2380000000000004,"15min":13.898999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.876,"15min":0.89100000000000001},"min":{"1min":0.31900000000000001,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":2.9790000000000001,"5min":4.0460000000000003,"15min":14.51},"last":1.151}]},{"osd":3,"last update":"Tue Sep 10 18:34:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.6719999999999999,"15min":1.335},"min":{"1min":0.28899999999999998,"5min":0.26300000000000001,"15min":0.249},"max":{"1min":7.0330000000000004,"5min":49.990000000000002,"15min":49.990000000000002},"last":1.27},{"interface":"front","average":{"1min":1.383,"5min":1.726,"15min":1.385},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.255},"max":{"1min":5.0439999999999996,"5min":50.764000000000003,"15min":50.764000000000003},"last":0.51500000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:33:58 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.995,"15min":0.59999999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.20499999999999999},"max":{"1min":3.0409999999999999,"5min":22.861000000000001,"15min":22.861000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.98399999999999999,"15min":0.623},"min":{"1min":0.28299999999999997,"5min":0.22,"15min":0.22},"max":{"1min":3.3799999999999999,"5min":19.611999999999998,"15min":19.611999999999998},"last":1.2410000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:34:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.63800000000000001,"15min":0.50900000000000001},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.26400000000000001},"max":{"1min":0.77000000000000002,"5min":3.1139999999999999,"15min":6.7990000000000004},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.63100000000000001,"15min":0.54900000000000004},"min":{"1min":0.32100000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.95499999999999996,"5min":3.6419999999999999,"15min":7.0300000000000002},"last":0.42799999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:34:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65200000000000002,"15min":0.55700000000000005},"min":{"1min":0.316,"5min":0.28799999999999998,"15min":0.255},"max":{"1min":1.4330000000000001,"5min":4.8600000000000003,"15min":7.0670000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64400000000000002,"15min":0.57099999999999995},"min":{"1min":0.33800000000000002,"5min":0.28699999999999998,"15min":0.247},"max":{"1min":1.2130000000000001,"5min":3.3450000000000002,"15min":6.5330000000000004},"last":1.2130000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117175,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2585608,"kb_used_data":1987388,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":91159544,"statfs":{"total":95995035648,"available":93347373056,"internally_reserved":0,"allocated":2035085312,"data_stored":2034607039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:12 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2789999999999999,"15min":1.2390000000000001},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.9569999999999999,"5min":5.4560000000000004,"15min":8.5530000000000008},"last":4.2370000000000001},{"interface":"front","average":{"1min":1.903,"5min":1.464,"15min":1.351},"min":{"1min":0.253,"5min":0.22,"15min":0.193},"max":{"1min":7.5979999999999999,"5min":8.2110000000000003,"15min":13.659000000000001},"last":3.9940000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:34:12 2024","interfaces":[{"interface":"back","average":{"1min":1.9890000000000001,"5min":1.923,"15min":1.431},"min":{"1min":0.28599999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.0419999999999998,"5min":35.406999999999996,"15min":35.406999999999996},"last":4.0300000000000002},{"interface":"front","average":{"1min":1.556,"5min":1.9590000000000001,"15min":1.343},"min":{"1min":0.375,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":3.843,"5min":36.902000000000001,"15min":36.902000000000001},"last":4.2130000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:34:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.0589999999999999,"15min":1.077},"min":{"1min":0.38700000000000001,"5min":0.222,"15min":0.222},"max":{"1min":3.9060000000000001,"5min":5.3810000000000002,"15min":7.7960000000000003},"last":4.2569999999999997},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3520000000000001,"15min":1.256},"min":{"1min":0.23000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":6.7290000000000001,"5min":14.997999999999999,"15min":14.997999999999999},"last":0.39000000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":1.3049999999999999,"15min":1.2010000000000001},"min":{"1min":0.33800000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":23.684999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":2,"5min":1.2749999999999999,"15min":1.1539999999999999},"min":{"1min":0.315,"5min":0.29899999999999999,"15min":0.27200000000000002},"max":{"1min":6.1929999999999996,"5min":6.1929999999999996,"15min":23.649000000000001},"last":3.8610000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:33:57 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.3540000000000001,"15min":1.288},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":6.601,"5min":6.601,"15min":23.719999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.7130000000000001,"5min":1.375,"15min":1.2509999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.3979999999999997,"5min":6.2249999999999996,"15min":23.707000000000001},"last":3.8100000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":1.966,"5min":1.8440000000000001,"15min":1.2050000000000001},"min":{"1min":0.38500000000000001,"5min":0.32200000000000001,"15min":0.317},"max":{"1min":6.2889999999999997,"5min":31.815999999999999,"15min":31.815999999999999},"last":3.8999999999999999},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.978,"15min":1.4970000000000001},"min":{"1min":0.42999999999999999,"5min":0.33400000000000002,"15min":0.315},"max":{"1min":7.0970000000000004,"5min":31.742000000000001,"15min":31.742000000000001},"last":4.1870000000000003}]},{"osd":7,"last update":"Tue Sep 10 18:33:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.452,"15min":1.2390000000000001},"min":{"1min":0.56599999999999995,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":6.5730000000000004,"5min":6.5730000000000004,"15min":23.748999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":2.2250000000000001,"5min":1.468,"15min":1.286},"min":{"1min":0.46899999999999997,"5min":0.35199999999999998,"15min":0.307},"max":{"1min":6.2530000000000001,"5min":12.612,"15min":25.058},"last":3.9199999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411515,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1067248,"kb_used_data":761508,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92677904,"statfs":{"total":95995035648,"available":94902173696,"internally_reserved":0,"allocated":779784192,"data_stored":779329667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:33:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.7390000000000001,"15min":1.3300000000000001},"min":{"1min":0.22700000000000001,"5min":0.216,"15min":0.185},"max":{"1min":6.5019999999999998,"5min":43.890000000000001,"15min":43.890000000000001},"last":0.22800000000000001},{"interface":"front","average":{"1min":1.3260000000000001,"5min":2.0920000000000001,"15min":1.5309999999999999},"min":{"1min":0.247,"5min":0.223,"15min":0.16900000000000001},"max":{"1min":10.606999999999999,"5min":44.331000000000003,"15min":44.331000000000003},"last":3.4910000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:33:47 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":2.0230000000000001,"15min":1.3919999999999999},"min":{"1min":0.19400000000000001,"5min":0.193,"15min":0.193},"max":{"1min":7.266,"5min":44.575000000000003,"15min":44.575000000000003},"last":2.3479999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.9510000000000001,"15min":1.2170000000000001},"min":{"1min":0.25900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":7.0270000000000001,"5min":44.368000000000002,"15min":44.368000000000002},"last":0.53600000000000003}]},{"osd":3,"last update":"Tue Sep 10 18:34:10 2024","interfaces":[{"interface":"back","average":{"1min":4.4269999999999996,"5min":2.1669999999999998,"15min":1.7150000000000001},"min":{"1min":0.312,"5min":0.26100000000000001,"15min":0.251},"max":{"1min":61.331000000000003,"5min":61.331000000000003,"15min":61.331000000000003},"last":3.206},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.1949999999999998,"15min":1.7969999999999999},"min":{"1min":0.33500000000000002,"5min":0.214,"15min":0.214},"max":{"1min":61.250999999999998,"5min":61.250999999999998,"15min":61.250999999999998},"last":1.829}]},{"osd":4,"last update":"Tue Sep 10 18:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":1.1879999999999999,"15min":1.016},"min":{"1min":0.30199999999999999,"5min":0.28699999999999998,"15min":0.24199999999999999},"max":{"1min":2.1920000000000002,"5min":12.555999999999999,"15min":12.555999999999999},"last":1.119},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.0589999999999999,"15min":0.92600000000000005},"min":{"1min":0.35099999999999998,"5min":0.33000000000000002,"15min":0.317},"max":{"1min":1.105,"5min":7.0730000000000004,"15min":12.130000000000001},"last":0.56299999999999994}]},{"osd":5,"last update":"Tue Sep 10 18:33:56 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":1.262,"15min":0.97299999999999998},"min":{"1min":0.32000000000000001,"5min":0.30299999999999999,"15min":0.27500000000000002},"max":{"1min":2.3980000000000001,"5min":7.1609999999999996,"15min":12.106},"last":3.1269999999999998},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.4430000000000001,"15min":1.0369999999999999},"min":{"1min":0.45000000000000001,"5min":0.32200000000000001,"15min":0.31900000000000001},"max":{"1min":2.6949999999999998,"5min":12.529999999999999,"15min":12.529999999999999},"last":3.1720000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.177,"15min":1.1160000000000001},"min":{"1min":0.38500000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":4.6440000000000001,"5min":5.9989999999999997,"15min":5.9989999999999997},"last":1.23},{"interface":"front","average":{"1min":0.90600000000000003,"5min":1.105,"15min":0.96999999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.30099999999999999},"max":{"1min":5.4880000000000004,"5min":7.1349999999999998,"15min":7.1349999999999998},"last":0.58999999999999997}]},{"osd":7,"last update":"Tue Sep 10 18:33:47 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.252,"15min":1.0469999999999999},"min":{"1min":0.42699999999999999,"5min":0.35399999999999998,"15min":0.30499999999999999},"max":{"1min":10.68,"5min":10.68,"15min":10.68},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.4790000000000001,"15min":1.107},"min":{"1min":0.40400000000000003,"5min":0.376,"15min":0.307},"max":{"1min":4.6440000000000001,"5min":12.596,"15min":12.596},"last":1.198}]}]},{"osd":1,"up_from":13,"seq":55834575040,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2876936,"kb_used_data":2282688,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":90868216,"statfs":{"total":95995035648,"available":93049053184,"internally_reserved":0,"allocated":2337472512,"data_stored":2336989925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2450000000000001,"15min":1.5329999999999999},"min":{"1min":0.20100000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":8.0030000000000001,"5min":14.079000000000001,"15min":80.456999999999994},"last":0.23400000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3240000000000001,"15min":1.6160000000000001},"min":{"1min":0.247,"5min":0.20399999999999999,"15min":0.187},"max":{"1min":5.6980000000000004,"5min":5.6980000000000004,"15min":80.578000000000003},"last":1.8500000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:34:43 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.1779999999999999,"15min":1.1910000000000001},"min":{"1min":0.24399999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.8380000000000001,"5min":5.8380000000000001,"15min":13.035},"last":0.443},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.2050000000000001,"15min":1.1399999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":2.302,"5min":12.222,"15min":12.222},"last":0.56599999999999995}]},{"osd":3,"last update":"Tue Sep 10 18:34:23 2024","interfaces":[{"interface":"back","average":{"1min":1.8029999999999999,"5min":1.5129999999999999,"15min":1.575},"min":{"1min":0.40000000000000002,"5min":0.23899999999999999,"15min":0.22600000000000001},"max":{"1min":8.0399999999999991,"5min":14.121,"15min":43.076000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.446,"15min":1.6220000000000001},"min":{"1min":0.22500000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.8879999999999999,"5min":7.3799999999999999,"15min":41.93},"last":1.5089999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:34:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.26,"15min":1.079},"min":{"1min":0.312,"5min":0.312,"15min":0.29799999999999999},"max":{"1min":5.9930000000000003,"5min":5.9930000000000003,"15min":7.2859999999999996},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.3979999999999999,"15min":1.2170000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.28299999999999997},"max":{"1min":2.3969999999999998,"5min":7.8769999999999998,"15min":19.393000000000001},"last":0.59799999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:33:57 2024","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":1.6339999999999999,"15min":1.196},"min":{"1min":0.46999999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":7.2110000000000003,"5min":12.206,"15min":15.48},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.556,"15min":1.103},"min":{"1min":0.443,"5min":0.36599999999999999,"15min":0.308},"max":{"1min":11.414999999999999,"5min":11.414999999999999,"15min":11.414999999999999},"last":1.595}]},{"osd":6,"last update":"Tue Sep 10 18:34:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.2170000000000001,"15min":1.2909999999999999},"min":{"1min":0.41499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":8.1340000000000003,"5min":10.039,"15min":10.039},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.373,"15min":1.389},"min":{"1min":0.42699999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":5.9240000000000004,"5min":6.2670000000000003,"15min":7.3369999999999997},"last":0.46600000000000003}]},{"osd":7,"last update":"Tue Sep 10 18:34:43 2024","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.4119999999999999,"15min":1.1879999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":8.1609999999999996,"5min":10.005000000000001,"15min":15.545999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.9870000000000001,"5min":1.5009999999999999,"15min":1.1890000000000001},"min":{"1min":0.50600000000000001,"5min":0.35699999999999998,"15min":0.313},"max":{"1min":6.0300000000000002,"5min":6.3049999999999997,"15min":7.359},"last":1.429}]}]},{"osd":0,"up_from":8,"seq":34359738565,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1291188,"kb_used_data":985452,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92453964,"statfs":{"total":95995035648,"available":94672859136,"internally_reserved":0,"allocated":1009102848,"data_stored":1008639988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:34:21 2024","interfaces":[{"interface":"back","average":{"1min":9.0549999999999997,"5min":3.1669999999999998,"15min":1.8899999999999999},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":146.44499999999999,"5min":146.44499999999999,"15min":146.44499999999999},"last":3.73},{"interface":"front","average":{"1min":9.9619999999999997,"5min":3.2679999999999998,"15min":2.0190000000000001},"min":{"1min":0.38300000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":148.77199999999999,"5min":148.77199999999999,"15min":148.77199999999999},"last":0.29799999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:33:48 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.2829999999999999,"15min":1.2849999999999999},"min":{"1min":0.28599999999999998,"5min":0.20699999999999999,"15min":0.189},"max":{"1min":3.137,"5min":7.4329999999999998,"15min":30.219999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":1.2110000000000001,"15min":1.268},"min":{"1min":0.23000000000000001,"5min":0.224,"15min":0.22},"max":{"1min":2.6160000000000001,"5min":7.5119999999999996,"15min":30.129999999999999},"last":0.67300000000000004}]},{"osd":3,"last update":"Tue Sep 10 18:34:19 2024","interfaces":[{"interface":"back","average":{"1min":8.3469999999999995,"5min":2.8690000000000002,"15min":1.8899999999999999},"min":{"1min":0.28599999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":140.68299999999999,"5min":140.68299999999999,"15min":140.68299999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":8.4179999999999993,"5min":3.004,"15min":1.8740000000000001},"min":{"1min":0.308,"5min":0.26500000000000001,"15min":0.23899999999999999},"max":{"1min":140.572,"5min":140.572,"15min":140.572},"last":0.28000000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":1.1279999999999999,"15min":1.1319999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.246},"max":{"1min":1.9830000000000001,"5min":8.3699999999999992,"15min":17.151},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.004,"5min":1.0580000000000001,"15min":1.127},"min":{"1min":0.31,"5min":0.29899999999999999,"15min":0.25900000000000001},"max":{"1min":3.3849999999999998,"5min":3.7919999999999998,"15min":18.728999999999999},"last":0.48799999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.167,"15min":1.228},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.3210000000000002,"5min":4.7220000000000004,"15min":18.77},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.2949999999999999,"15min":1.1619999999999999},"min":{"1min":0.42599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.6050000000000004,"5min":6.6079999999999997,"15min":14.991},"last":0.72599999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":1.1799999999999999,"15min":1.2350000000000001},"min":{"1min":0.36299999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":2.0190000000000001,"5min":8.3940000000000001,"15min":8.3940000000000001},"last":1.147},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0900000000000001,"15min":1.306},"min":{"1min":0.34399999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.3599999999999999,"5min":3.9079999999999999,"15min":9.7859999999999996},"last":0.52000000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:34:38 2024","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":1.371,"15min":1.228},"min":{"1min":0.377,"5min":0.32200000000000001,"15min":0.26700000000000002},"max":{"1min":10.75,"5min":10.75,"15min":13.711},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.583,"5min":1.2090000000000001,"15min":1.159},"min":{"1min":0.378,"5min":0.378,"15min":0.33500000000000002},"max":{"1min":11.218999999999999,"5min":11.218999999999999,"15min":15.032},"last":0.879}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":996671488,"data_stored":996671488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2307067904,"data_stored":2307063832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":767688704,"data_stored":767688704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2024275968,"data_stored":2024275968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":781647872,"data_stored":781647872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2317946880,"data_stored":2317942808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":754778112,"data_stored":754778112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2321879040,"data_stored":2321874968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:34:49.895 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-10T18:34:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:49.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:49.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:50.005 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:50.038 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:50.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:50.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:50.140 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:50.175 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:50.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:50.202 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:50 smithi204 ceph-mon[32357]: pgmap v603: 178 pgs: 178 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 943 op/s 2024-09-10T18:34:50.241 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-10T18:34:50.277 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:50.317 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:50.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:50.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:50.419 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:50.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:50.462 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:50.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:50.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:50 smithi028 ceph-mon[39220]: pgmap v603: 178 pgs: 178 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 943 op/s 2024-09-10T18:34:50.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:50 smithi028 ceph-mon[29291]: pgmap v603: 178 pgs: 178 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 943 op/s 2024-09-10T18:34:50.564 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:50.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:50.608 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:50.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:50.707 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:34:50.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:50.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:50.745 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:50.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:50.845 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:50.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:50.880 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:50.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:50.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:51.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:51.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:51.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:51.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:51.156 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:51.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:51.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:51.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:51.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:51.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:51.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:51.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:51.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:51.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:51.503 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:34:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:34:51] "GET /metrics HTTP/1.1" 200 224592 "" "Prometheus/2.43.0" 2024-09-10T18:34:51.542 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:51.572 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:51.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:51.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:51.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:51.715 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:51.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:51.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:51.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:34:51.860 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:51.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:51.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:51.962 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:51.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:52.005 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:51 smithi204 ceph-mon[32357]: osdmap e296: 8 total, 8 up, 8 in 2024-09-10T18:34:52.005 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:51 smithi204 ceph-mon[32357]: pgmap v605: 178 pgs: 178 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-09-10T18:34:52.005 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:51 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4116144819' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:34:52.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:34:52.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:52.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:51 smithi028 ceph-mon[29291]: osdmap e296: 8 total, 8 up, 8 in 2024-09-10T18:34:52.112 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:51 smithi028 ceph-mon[29291]: pgmap v605: 178 pgs: 178 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-09-10T18:34:52.112 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:51 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4116144819' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:34:52.112 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:51 smithi028 ceph-mon[39220]: osdmap e296: 8 total, 8 up, 8 in 2024-09-10T18:34:52.112 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:51 smithi028 ceph-mon[39220]: pgmap v605: 178 pgs: 178 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-09-10T18:34:52.112 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:51 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4116144819' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:34:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:52.161 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:52.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:52.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:52.278 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:34:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:34:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:34:52.293 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:52.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:52.395 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:52.428 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:52.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:52.529 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:52.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:52.551 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:52.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:52.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:52.686 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:52.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:52.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:52.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:52.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:52.823 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:52.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:52 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:34:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:52 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4116144819' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:34:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:52 smithi204 ceph-mon[32357]: osdmap e297: 8 total, 8 up, 8 in 2024-09-10T18:34:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:52 smithi204 ceph-mon[32357]: 6.5 scrub starts 2024-09-10T18:34:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:52 smithi204 ceph-mon[32357]: 6.5 scrub ok 2024-09-10T18:34:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:52 smithi204 ceph-mon[32357]: 4.1b scrub starts 2024-09-10T18:34:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:52 smithi204 ceph-mon[32357]: 4.1b scrub ok 2024-09-10T18:34:52.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:52.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:52.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:52.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:53.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:53.085 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:53.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:53.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:53.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:53.226 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:53.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:53.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:52 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:34:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:52 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4116144819' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:34:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:52 smithi028 ceph-mon[29291]: osdmap e297: 8 total, 8 up, 8 in 2024-09-10T18:34:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:52 smithi028 ceph-mon[29291]: 6.5 scrub starts 2024-09-10T18:34:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:52 smithi028 ceph-mon[29291]: 6.5 scrub ok 2024-09-10T18:34:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:52 smithi028 ceph-mon[29291]: 4.1b scrub starts 2024-09-10T18:34:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:52 smithi028 ceph-mon[29291]: 4.1b scrub ok 2024-09-10T18:34:53.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:52 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:34:53.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:52 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4116144819' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:34:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:52 smithi028 ceph-mon[39220]: osdmap e297: 8 total, 8 up, 8 in 2024-09-10T18:34:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:52 smithi028 ceph-mon[39220]: 6.5 scrub starts 2024-09-10T18:34:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:52 smithi028 ceph-mon[39220]: 6.5 scrub ok 2024-09-10T18:34:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:52 smithi028 ceph-mon[39220]: 4.1b scrub starts 2024-09-10T18:34:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:52 smithi028 ceph-mon[39220]: 4.1b scrub ok 2024-09-10T18:34:53.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:53.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:53.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:53.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:53.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:53.506 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:53.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:53.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:53.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:53.644 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:53.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:53.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:53.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:53.786 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:53.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:53.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:53.888 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:53.927 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:53.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:53.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:54.029 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:54.052 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:53 smithi204 ceph-mon[32357]: pgmap v607: 178 pgs: 178 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-10T18:34:54.052 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:53 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:34:54.052 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:53 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:34:54.053 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:53 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:34:54.053 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:53 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:34:54.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:54.064 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:54.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:54.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:54.187 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:54.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:54.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:54.251 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:34:54.251 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:34:54.251 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-10T18:34:54.251 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-10T18:34:54.251 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:34:54.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:53 smithi028 ceph-mon[29291]: pgmap v607: 178 pgs: 178 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-10T18:34:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:53 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:34:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:53 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:34:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:53 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:34:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:53 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:34:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:53 smithi028 ceph-mon[39220]: pgmap v607: 178 pgs: 178 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-10T18:34:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:53 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:34:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:53 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:34:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:53 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:34:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:53 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:34:54.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:54.316 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:54.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:54.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:54.418 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:54.448 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:54.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:54.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:54.550 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:54.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:54.586 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:54.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:54.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:54.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:54.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:54.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:54.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:34:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:34:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:54 smithi204 ceph-mon[32357]: 6.14 scrub starts 2024-09-10T18:34:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:54 smithi204 ceph-mon[32357]: 6.14 scrub ok 2024-09-10T18:34:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:54 smithi204 ceph-mon[32357]: 3.6 scrub starts 2024-09-10T18:34:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:54 smithi204 ceph-mon[32357]: 3.6 scrub ok 2024-09-10T18:34:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:54 smithi204 ceph-mon[32357]: 5.c scrub starts 2024-09-10T18:34:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:54 smithi204 ceph-mon[32357]: 5.c scrub ok 2024-09-10T18:34:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:54 smithi204 ceph-mon[32357]: osdmap e298: 8 total, 8 up, 8 in 2024-09-10T18:34:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:54 smithi204 ceph-mon[32357]: 5.1f scrub starts 2024-09-10T18:34:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:54 smithi204 ceph-mon[32357]: 5.1f scrub ok 2024-09-10T18:34:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:54 smithi204 ceph-mon[32357]: 3.1c scrub starts 2024-09-10T18:34:54.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:54 smithi204 ceph-mon[32357]: 3.1c scrub ok 2024-09-10T18:34:54.862 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:54.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:54.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:54.896 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:34:54.896 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:34:54.896 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-10T18:34:54.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:34:54.964 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:54.982 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-10T18:34:54.999 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:55.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:55.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:55.089 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:34:55.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:55.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:55.137 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:55.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:55.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:55.239 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[29291]: 6.14 scrub starts 2024-09-10T18:34:55.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[29291]: 6.14 scrub ok 2024-09-10T18:34:55.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[29291]: 3.6 scrub starts 2024-09-10T18:34:55.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[29291]: 3.6 scrub ok 2024-09-10T18:34:55.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[29291]: 5.c scrub starts 2024-09-10T18:34:55.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[29291]: 5.c scrub ok 2024-09-10T18:34:55.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[29291]: osdmap e298: 8 total, 8 up, 8 in 2024-09-10T18:34:55.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[29291]: 5.1f scrub starts 2024-09-10T18:34:55.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[29291]: 5.1f scrub ok 2024-09-10T18:34:55.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[29291]: 3.1c scrub starts 2024-09-10T18:34:55.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[29291]: 3.1c scrub ok 2024-09-10T18:34:55.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[39220]: 6.14 scrub starts 2024-09-10T18:34:55.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[39220]: 6.14 scrub ok 2024-09-10T18:34:55.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[39220]: 3.6 scrub starts 2024-09-10T18:34:55.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[39220]: 3.6 scrub ok 2024-09-10T18:34:55.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[39220]: 5.c scrub starts 2024-09-10T18:34:55.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[39220]: 5.c scrub ok 2024-09-10T18:34:55.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[39220]: osdmap e298: 8 total, 8 up, 8 in 2024-09-10T18:34:55.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[39220]: 5.1f scrub starts 2024-09-10T18:34:55.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[39220]: 5.1f scrub ok 2024-09-10T18:34:55.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[39220]: 3.1c scrub starts 2024-09-10T18:34:55.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:54 smithi028 ceph-mon[39220]: 3.1c scrub ok 2024-09-10T18:34:55.282 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:55.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:55.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:55.384 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:55.412 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:34:55.421 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:55.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:55.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:55.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:55.564 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:55.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:55.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:55.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:55.706 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:55.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:55.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:55.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:55.849 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:55.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:55.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:55.951 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:55.986 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:55.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:55.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:56.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:56.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:55 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:56.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:55 smithi204 ceph-mon[32357]: 1.6 scrub starts 2024-09-10T18:34:56.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:55 smithi204 ceph-mon[32357]: 1.6 scrub ok 2024-09-10T18:34:56.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:55 smithi204 ceph-mon[32357]: pgmap v609: 178 pgs: 178 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-10T18:34:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:56.127 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:56.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:56.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:55 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:55 smithi028 ceph-mon[29291]: 1.6 scrub starts 2024-09-10T18:34:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:55 smithi028 ceph-mon[29291]: 1.6 scrub ok 2024-09-10T18:34:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:55 smithi028 ceph-mon[29291]: pgmap v609: 178 pgs: 178 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-10T18:34:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:55 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:34:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:55 smithi028 ceph-mon[39220]: 1.6 scrub starts 2024-09-10T18:34:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:55 smithi028 ceph-mon[39220]: 1.6 scrub ok 2024-09-10T18:34:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:55 smithi028 ceph-mon[39220]: pgmap v609: 178 pgs: 178 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-10T18:34:56.267 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:56.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:56.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:56.369 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:56.404 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:56.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:56.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:56.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:34:56.537 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:56.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:34:56.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:56.684 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:56.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:56.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:56.702 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:34:56.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:56.833 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:56.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:56.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:56 smithi204 ceph-mon[32357]: 9.d scrub starts 2024-09-10T18:34:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:56 smithi204 ceph-mon[32357]: 3.5 scrub starts 2024-09-10T18:34:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:56 smithi204 ceph-mon[32357]: 3.5 scrub ok 2024-09-10T18:34:56.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:56 smithi028 ceph-mon[29291]: 9.d scrub starts 2024-09-10T18:34:56.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:56 smithi028 ceph-mon[29291]: 3.5 scrub starts 2024-09-10T18:34:56.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:56 smithi028 ceph-mon[29291]: 3.5 scrub ok 2024-09-10T18:34:56.857 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:56 smithi028 ceph-mon[39220]: 9.d scrub starts 2024-09-10T18:34:56.857 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:56 smithi028 ceph-mon[39220]: 3.5 scrub starts 2024-09-10T18:34:56.857 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:56 smithi028 ceph-mon[39220]: 3.5 scrub ok 2024-09-10T18:34:56.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:56.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:34:56.971 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:56.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:56.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:57.074 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:57.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:57.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:57.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:57.245 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:57.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:57.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:57.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:57.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:57.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:57.437 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:34:57.438 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-10T18:34:57.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-10T18:34:57.438 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:34:57.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.438 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:181174 7.5 7:a54ae781:::benchmark_data_smithi028_77856_object60074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:34:57.438 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:25.047898+0000", 2024-09-10T18:34:57.438 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 512.38491701500004, 2024-09-10T18:34:57.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15111707499999999, 2024-09-10T18:34:57.438 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.438 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.438 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.438 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:57.439 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:57.439 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 181174 2024-09-10T18:34:57.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.439 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.047898+0000", 2024-09-10T18:34:57.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.047898+0000", 2024-09-10T18:34:57.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.047895+0000", 2024-09-10T18:34:57.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:34:57.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.047904+0000", 2024-09-10T18:34:57.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2579999999999995e-06 2024-09-10T18:34:57.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.047917+0000", 2024-09-10T18:34:57.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3468999999999999e-05 2024-09-10T18:34:57.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.047923+0000", 2024-09-10T18:34:57.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5149999999999997e-06 2024-09-10T18:34:57.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.047960+0000", 2024-09-10T18:34:57.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7038e-05 2024-09-10T18:34:57.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.048048+0000", 2024-09-10T18:34:57.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7979999999999995e-05 2024-09-10T18:34:57.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2", 2024-09-10T18:34:57.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.048126+0000", 2024-09-10T18:34:57.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7867000000000005e-05 2024-09-10T18:34:57.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.049930+0000", 2024-09-10T18:34:57.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001804305 2024-09-10T18:34:57.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.052034+0000", 2024-09-10T18:34:57.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021042790000000001 2024-09-10T18:34:57.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.198945+0000", 2024-09-10T18:34:57.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.146911233 2024-09-10T18:34:57.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.198977+0000", 2024-09-10T18:34:57.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1717999999999997e-05 2024-09-10T18:34:57.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.199015+0000", 2024-09-10T18:34:57.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7988999999999998e-05 2024-09-10T18:34:57.444 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.444 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.444 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.444 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:181175 7.5 7:a07e4061:::benchmark_data_smithi028_77856_object60075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:34:57.444 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:25.050064+0000", 2024-09-10T18:34:57.444 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 512.38275094100004, 2024-09-10T18:34:57.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15533630200000001, 2024-09-10T18:34:57.444 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.444 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.444 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.444 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:57.445 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:57.445 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 181175 2024-09-10T18:34:57.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.445 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.050064+0000", 2024-09-10T18:34:57.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.050064+0000", 2024-09-10T18:34:57.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.050063+0000", 2024-09-10T18:34:57.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:34:57.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.050069+0000", 2024-09-10T18:34:57.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6899999999999997e-06 2024-09-10T18:34:57.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.050079+0000", 2024-09-10T18:34:57.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0253e-05 2024-09-10T18:34:57.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.050083+0000", 2024-09-10T18:34:57.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2540000000000003e-06 2024-09-10T18:34:57.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.050115+0000", 2024-09-10T18:34:57.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2391000000000002e-05 2024-09-10T18:34:57.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.050193+0000", 2024-09-10T18:34:57.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7199000000000004e-05 2024-09-10T18:34:57.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2", 2024-09-10T18:34:57.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.050254+0000", 2024-09-10T18:34:57.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1546999999999999e-05 2024-09-10T18:34:57.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.051346+0000", 2024-09-10T18:34:57.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001092059 2024-09-10T18:34:57.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.054557+0000", 2024-09-10T18:34:57.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032110749999999999 2024-09-10T18:34:57.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.205321+0000", 2024-09-10T18:34:57.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15076334699999999 2024-09-10T18:34:57.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.205365+0000", 2024-09-10T18:34:57.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3899000000000001e-05 2024-09-10T18:34:57.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.205400+0000", 2024-09-10T18:34:57.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5877999999999998e-05 2024-09-10T18:34:57.450 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.450 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.450 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.450 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:181180 7.5 7:ac8f5203:::benchmark_data_smithi028_77856_object60080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:34:57.450 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:25.054624+0000", 2024-09-10T18:34:57.450 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 512.37819120699999, 2024-09-10T18:34:57.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.150917404, 2024-09-10T18:34:57.451 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.451 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.451 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.451 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:57.451 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:57.451 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 181180 2024-09-10T18:34:57.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.451 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.054624+0000", 2024-09-10T18:34:57.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.054624+0000", 2024-09-10T18:34:57.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.054623+0000", 2024-09-10T18:34:57.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:34:57.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.054627+0000", 2024-09-10T18:34:57.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2429999999999999e-06 2024-09-10T18:34:57.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.054634+0000", 2024-09-10T18:34:57.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3880000000000002e-06 2024-09-10T18:34:57.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.054638+0000", 2024-09-10T18:34:57.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3459999999999998e-06 2024-09-10T18:34:57.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.054675+0000", 2024-09-10T18:34:57.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7336000000000003e-05 2024-09-10T18:34:57.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.054752+0000", 2024-09-10T18:34:57.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6587000000000004e-05 2024-09-10T18:34:57.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2", 2024-09-10T18:34:57.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.054818+0000", 2024-09-10T18:34:57.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6147000000000002e-05 2024-09-10T18:34:57.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.056810+0000", 2024-09-10T18:34:57.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019919650000000001 2024-09-10T18:34:57.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.057127+0000", 2024-09-10T18:34:57.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00031722500000000003 2024-09-10T18:34:57.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.205457+0000", 2024-09-10T18:34:57.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.148330189 2024-09-10T18:34:57.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.205506+0000", 2024-09-10T18:34:57.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8294000000000002e-05 2024-09-10T18:34:57.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.205541+0000", 2024-09-10T18:34:57.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5747000000000002e-05 2024-09-10T18:34:57.457 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.457 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.457 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.457 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:181530 7.5 7:a1db56bd:::benchmark_data_smithi028_77856_object60430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:34:57.457 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:25.453307+0000", 2024-09-10T18:34:57.458 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 511.97950854099997, 2024-09-10T18:34:57.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.12854568999999999, 2024-09-10T18:34:57.458 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.458 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.458 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.458 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:57.458 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:57.458 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 181530 2024-09-10T18:34:57.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.458 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453307+0000", 2024-09-10T18:34:57.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453307+0000", 2024-09-10T18:34:57.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453304+0000", 2024-09-10T18:34:57.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:34:57.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453311+0000", 2024-09-10T18:34:57.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8530000000000003e-06 2024-09-10T18:34:57.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453324+0000", 2024-09-10T18:34:57.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2591e-05 2024-09-10T18:34:57.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453328+0000", 2024-09-10T18:34:57.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4800000000000003e-06 2024-09-10T18:34:57.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453369+0000", 2024-09-10T18:34:57.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0967999999999998e-05 2024-09-10T18:34:57.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.461 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453475+0000", 2024-09-10T18:34:57.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105602 2024-09-10T18:34:57.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2", 2024-09-10T18:34:57.461 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.453576+0000", 2024-09-10T18:34:57.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010147199999999999 2024-09-10T18:34:57.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.455709+0000", 2024-09-10T18:34:57.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021325490000000001 2024-09-10T18:34:57.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.456350+0000", 2024-09-10T18:34:57.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00064122699999999996 2024-09-10T18:34:57.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.581768+0000", 2024-09-10T18:34:57.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.125418107 2024-09-10T18:34:57.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.581813+0000", 2024-09-10T18:34:57.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4833000000000003e-05 2024-09-10T18:34:57.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:25.581852+0000", 2024-09-10T18:34:57.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9314999999999999e-05 2024-09-10T18:34:57.463 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.463 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.463 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.463 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:182380 7.5 7:a7182387:::benchmark_data_smithi028_77856_object61280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-10T18:34:57.464 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:26.349688+0000", 2024-09-10T18:34:57.464 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 511.08312726899999, 2024-09-10T18:34:57.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.14745614900000001, 2024-09-10T18:34:57.464 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.464 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.464 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.464 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:57.464 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:57.464 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 182380 2024-09-10T18:34:57.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.464 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.349688+0000", 2024-09-10T18:34:57.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.349688+0000", 2024-09-10T18:34:57.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.349685+0000", 2024-09-10T18:34:57.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:34:57.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.349694+0000", 2024-09-10T18:34:57.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5150000000000006e-06 2024-09-10T18:34:57.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.349708+0000", 2024-09-10T18:34:57.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.403e-05 2024-09-10T18:34:57.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.349713+0000", 2024-09-10T18:34:57.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.412e-06 2024-09-10T18:34:57.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.349760+0000", 2024-09-10T18:34:57.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7077999999999999e-05 2024-09-10T18:34:57.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.349888+0000", 2024-09-10T18:34:57.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012750200000000001 2024-09-10T18:34:57.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2", 2024-09-10T18:34:57.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.350003+0000", 2024-09-10T18:34:57.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000115414 2024-09-10T18:34:57.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.351964+0000", 2024-09-10T18:34:57.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019606889999999998 2024-09-10T18:34:57.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.353985+0000", 2024-09-10T18:34:57.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020216050000000001 2024-09-10T18:34:57.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.497063+0000", 2024-09-10T18:34:57.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.143077185 2024-09-10T18:34:57.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.497108+0000", 2024-09-10T18:34:57.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4912e-05 2024-09-10T18:34:57.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.497144+0000", 2024-09-10T18:34:57.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6368000000000001e-05 2024-09-10T18:34:57.469 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.470 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.470 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.470 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:182387 7.5 7:a4e8b542:::benchmark_data_smithi028_77856_object61287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-10T18:34:57.470 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:26.354767+0000", 2024-09-10T18:34:57.470 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 511.07804788499999, 2024-09-10T18:34:57.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.14252271, 2024-09-10T18:34:57.470 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.470 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.470 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.470 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:57.470 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:57.470 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 182387 2024-09-10T18:34:57.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.471 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.354767+0000", 2024-09-10T18:34:57.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.354767+0000", 2024-09-10T18:34:57.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.354765+0000", 2024-09-10T18:34:57.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:34:57.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.354774+0000", 2024-09-10T18:34:57.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.302e-06 2024-09-10T18:34:57.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.354786+0000", 2024-09-10T18:34:57.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1748e-05 2024-09-10T18:34:57.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.354791+0000", 2024-09-10T18:34:57.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.339e-06 2024-09-10T18:34:57.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.354826+0000", 2024-09-10T18:34:57.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4230000000000003e-05 2024-09-10T18:34:57.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.354911+0000", 2024-09-10T18:34:57.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5437000000000002e-05 2024-09-10T18:34:57.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2", 2024-09-10T18:34:57.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.354981+0000", 2024-09-10T18:34:57.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9653999999999994e-05 2024-09-10T18:34:57.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.358903+0000", 2024-09-10T18:34:57.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039226280000000001 2024-09-10T18:34:57.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.360645+0000", 2024-09-10T18:34:57.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0017417789999999999 2024-09-10T18:34:57.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.497211+0000", 2024-09-10T18:34:57.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13656616099999999 2024-09-10T18:34:57.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.497255+0000", 2024-09-10T18:34:57.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3409999999999999e-05 2024-09-10T18:34:57.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.497290+0000", 2024-09-10T18:34:57.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5135000000000002e-05 2024-09-10T18:34:57.476 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.476 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.476 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.476 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14808.0:182388 7.5 7:a135179f:::benchmark_data_smithi028_77856_object61288:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-10T18:34:57.476 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:26:26.354889+0000", 2024-09-10T18:34:57.476 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 511.07792592999999, 2024-09-10T18:34:57.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.142498191, 2024-09-10T18:34:57.476 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.476 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.477 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.477 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14808", 2024-09-10T18:34:57.477 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:34:57.477 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 182388 2024-09-10T18:34:57.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.477 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.354889+0000", 2024-09-10T18:34:57.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.354889+0000", 2024-09-10T18:34:57.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.354888+0000", 2024-09-10T18:34:57.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:34:57.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.354892+0000", 2024-09-10T18:34:57.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1320000000000004e-06 2024-09-10T18:34:57.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.354901+0000", 2024-09-10T18:34:57.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5399999999999996e-06 2024-09-10T18:34:57.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.354905+0000", 2024-09-10T18:34:57.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2089999999999999e-06 2024-09-10T18:34:57.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.356695+0000", 2024-09-10T18:34:57.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001789817 2024-09-10T18:34:57.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.356809+0000", 2024-09-10T18:34:57.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114454 2024-09-10T18:34:57.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2", 2024-09-10T18:34:57.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.356928+0000", 2024-09-10T18:34:57.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119097 2024-09-10T18:34:57.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.359037+0000", 2024-09-10T18:34:57.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021087990000000002 2024-09-10T18:34:57.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.360686+0000", 2024-09-10T18:34:57.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00164857 2024-09-10T18:34:57.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.497323+0000", 2024-09-10T18:34:57.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13663762400000001 2024-09-10T18:34:57.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.497356+0000", 2024-09-10T18:34:57.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2194999999999999e-05 2024-09-10T18:34:57.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:26:26.497387+0000", 2024-09-10T18:34:57.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1717000000000001e-05 2024-09-10T18:34:57.482 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.482 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.482 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.482 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:36821 9.e 9:77cacad8:::benchmark_data_smithi028_135703_object36820:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-10T18:34:57.483 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:32:27.394296+0000", 2024-09-10T18:34:57.483 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 150.03851859900001, 2024-09-10T18:34:57.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.115622766, 2024-09-10T18:34:57.483 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.483 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.483 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.483 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:34:57.483 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:57.483 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 36821 2024-09-10T18:34:57.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.483 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.394296+0000", 2024-09-10T18:34:57.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.394296+0000", 2024-09-10T18:34:57.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.394295+0000", 2024-09-10T18:34:57.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:34:57.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.394340+0000", 2024-09-10T18:34:57.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4653000000000001e-05 2024-09-10T18:34:57.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.394352+0000", 2024-09-10T18:34:57.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2349e-05 2024-09-10T18:34:57.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.394358+0000", 2024-09-10T18:34:57.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8989999999999998e-06 2024-09-10T18:34:57.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.394405+0000", 2024-09-10T18:34:57.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6526999999999997e-05 2024-09-10T18:34:57.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.394563+0000", 2024-09-10T18:34:57.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015857999999999999 2024-09-10T18:34:57.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:34:57.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.394712+0000", 2024-09-10T18:34:57.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014824099999999999 2024-09-10T18:34:57.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.396370+0000", 2024-09-10T18:34:57.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001658046 2024-09-10T18:34:57.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.396530+0000", 2024-09-10T18:34:57.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016023200000000001 2024-09-10T18:34:57.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.509836+0000", 2024-09-10T18:34:57.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.113306199 2024-09-10T18:34:57.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.509890+0000", 2024-09-10T18:34:57.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4282e-05 2024-09-10T18:34:57.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.509919+0000", 2024-09-10T18:34:57.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8904e-05 2024-09-10T18:34:57.488 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.488 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.489 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.489 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:37590 9.3 9:cc8d2c9e:::benchmark_data_smithi028_135703_object37589:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-10T18:34:57.489 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:32:27.966682+0000", 2024-09-10T18:34:57.489 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 149.46613345099999, 2024-09-10T18:34:57.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.120867366, 2024-09-10T18:34:57.489 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.489 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.489 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.489 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:34:57.489 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:57.489 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 37590 2024-09-10T18:34:57.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.490 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.966682+0000", 2024-09-10T18:34:57.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.966682+0000", 2024-09-10T18:34:57.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.966680+0000", 2024-09-10T18:34:57.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:34:57.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.966771+0000", 2024-09-10T18:34:57.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0958000000000006e-05 2024-09-10T18:34:57.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.966782+0000", 2024-09-10T18:34:57.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1157e-05 2024-09-10T18:34:57.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.966787+0000", 2024-09-10T18:34:57.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9559999999999996e-06 2024-09-10T18:34:57.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.966827+0000", 2024-09-10T18:34:57.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9858999999999999e-05 2024-09-10T18:34:57.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.966956+0000", 2024-09-10T18:34:57.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012976099999999999 2024-09-10T18:34:57.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:34:57.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.967044+0000", 2024-09-10T18:34:57.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7645000000000005e-05 2024-09-10T18:34:57.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.968292+0000", 2024-09-10T18:34:57.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001247876 2024-09-10T18:34:57.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:27.968987+0000", 2024-09-10T18:34:57.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00069539500000000002 2024-09-10T18:34:57.494 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:57.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:28.087431+0000", 2024-09-10T18:34:57.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.118443855 2024-09-10T18:34:57.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:28.087509+0000", 2024-09-10T18:34:57.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7718000000000004e-05 2024-09-10T18:34:57.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:32:28.087549+0000", 2024-09-10T18:34:57.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0068000000000003e-05 2024-09-10T18:34:57.497 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.497 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.497 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.497 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:115963 9.e 9:7cc8a211:::benchmark_data_smithi028_135703_object115962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-10T18:34:57.497 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:33:26.098536+0000", 2024-09-10T18:34:57.497 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 91.334279566999996, 2024-09-10T18:34:57.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.22854117500000001, 2024-09-10T18:34:57.497 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.497 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.497 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.497 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:34:57.498 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:57.498 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 115963 2024-09-10T18:34:57.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.498 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098536+0000", 2024-09-10T18:34:57.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098536+0000", 2024-09-10T18:34:57.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098533+0000", 2024-09-10T18:34:57.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:34:57.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098572+0000", 2024-09-10T18:34:57.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8804999999999999e-05 2024-09-10T18:34:57.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098585+0000", 2024-09-10T18:34:57.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2830000000000001e-05 2024-09-10T18:34:57.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098591+0000", 2024-09-10T18:34:57.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0469999999999999e-06 2024-09-10T18:34:57.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098688+0000", 2024-09-10T18:34:57.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7551000000000002e-05 2024-09-10T18:34:57.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098826+0000", 2024-09-10T18:34:57.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013742999999999999 2024-09-10T18:34:57.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:34:57.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098987+0000", 2024-09-10T18:34:57.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016180599999999999 2024-09-10T18:34:57.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105991+0000", 2024-09-10T18:34:57.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0070032649999999998 2024-09-10T18:34:57.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.106028+0000", 2024-09-10T18:34:57.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7605000000000003e-05 2024-09-10T18:34:57.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.326977+0000", 2024-09-10T18:34:57.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.22094904500000001 2024-09-10T18:34:57.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327031+0000", 2024-09-10T18:34:57.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3706999999999999e-05 2024-09-10T18:34:57.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.503 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327077+0000", 2024-09-10T18:34:57.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5705999999999999e-05 2024-09-10T18:34:57.503 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.503 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.503 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.503 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:115964 9.e 9:7700e295:::benchmark_data_smithi028_135703_object115963:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-10T18:34:57.503 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:33:26.098607+0000", 2024-09-10T18:34:57.503 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 91.334208199000003, 2024-09-10T18:34:57.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.22859473999999999, 2024-09-10T18:34:57.503 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.503 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.503 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.504 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:34:57.504 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:57.504 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 115964 2024-09-10T18:34:57.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.504 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098607+0000", 2024-09-10T18:34:57.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098607+0000", 2024-09-10T18:34:57.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098606+0000", 2024-09-10T18:34:57.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:34:57.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098640+0000", 2024-09-10T18:34:57.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4183000000000001e-05 2024-09-10T18:34:57.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098647+0000", 2024-09-10T18:34:57.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6880000000000002e-06 2024-09-10T18:34:57.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098651+0000", 2024-09-10T18:34:57.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.98e-06 2024-09-10T18:34:57.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.103373+0000", 2024-09-10T18:34:57.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047225339999999996 2024-09-10T18:34:57.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.103468+0000", 2024-09-10T18:34:57.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4591000000000003e-05 2024-09-10T18:34:57.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:34:57.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.103558+0000", 2024-09-10T18:34:57.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0408000000000006e-05 2024-09-10T18:34:57.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.106663+0000", 2024-09-10T18:34:57.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003104557 2024-09-10T18:34:57.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.106735+0000", 2024-09-10T18:34:57.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2008999999999994e-05 2024-09-10T18:34:57.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327116+0000", 2024-09-10T18:34:57.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.220380839 2024-09-10T18:34:57.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327169+0000", 2024-09-10T18:34:57.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3189000000000002e-05 2024-09-10T18:34:57.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327202+0000", 2024-09-10T18:34:57.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2540999999999999e-05 2024-09-10T18:34:57.509 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.509 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.509 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.509 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:115965 9.d 9:be537d29:::benchmark_data_smithi028_135703_object115964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-10T18:34:57.509 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:33:26.099655+0000", 2024-09-10T18:34:57.509 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 91.333160516999996, 2024-09-10T18:34:57.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.22745093699999999, 2024-09-10T18:34:57.509 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.509 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.509 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.509 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:34:57.509 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:57.510 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 115965 2024-09-10T18:34:57.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.510 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.099655+0000", 2024-09-10T18:34:57.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.099655+0000", 2024-09-10T18:34:57.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.099653+0000", 2024-09-10T18:34:57.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:34:57.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.099686+0000", 2024-09-10T18:34:57.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3408000000000003e-05 2024-09-10T18:34:57.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.099700+0000", 2024-09-10T18:34:57.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4396000000000001e-05 2024-09-10T18:34:57.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.099707+0000", 2024-09-10T18:34:57.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5459999999999997e-06 2024-09-10T18:34:57.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.099853+0000", 2024-09-10T18:34:57.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014599900000000001 2024-09-10T18:34:57.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.099979+0000", 2024-09-10T18:34:57.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012636099999999999 2024-09-10T18:34:57.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,5", 2024-09-10T18:34:57.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.100107+0000", 2024-09-10T18:34:57.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012751700000000001 2024-09-10T18:34:57.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.102820+0000", 2024-09-10T18:34:57.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027130349999999999 2024-09-10T18:34:57.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.109028+0000", 2024-09-10T18:34:57.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0062082170000000002 2024-09-10T18:34:57.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327021+0000", 2024-09-10T18:34:57.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.217992985 2024-09-10T18:34:57.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327068+0000", 2024-09-10T18:34:57.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6587e-05 2024-09-10T18:34:57.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327105+0000", 2024-09-10T18:34:57.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7750000000000003e-05 2024-09-10T18:34:57.515 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.515 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.515 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.515 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:115969 9.d 9:ba9f0da3:::benchmark_data_smithi028_135703_object115968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-10T18:34:57.515 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:33:26.105312+0000", 2024-09-10T18:34:57.515 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 91.327503128999993, 2024-09-10T18:34:57.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.22189908699999999, 2024-09-10T18:34:57.515 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.515 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.515 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.516 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:34:57.516 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:57.516 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 115969 2024-09-10T18:34:57.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.516 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105312+0000", 2024-09-10T18:34:57.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105312+0000", 2024-09-10T18:34:57.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105309+0000", 2024-09-10T18:34:57.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:34:57.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105367+0000", 2024-09-10T18:34:57.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7657e-05 2024-09-10T18:34:57.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105404+0000", 2024-09-10T18:34:57.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6470999999999997e-05 2024-09-10T18:34:57.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105415+0000", 2024-09-10T18:34:57.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1557e-05 2024-09-10T18:34:57.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105478+0000", 2024-09-10T18:34:57.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3323000000000003e-05 2024-09-10T18:34:57.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105614+0000", 2024-09-10T18:34:57.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000135978 2024-09-10T18:34:57.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,5", 2024-09-10T18:34:57.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105771+0000", 2024-09-10T18:34:57.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015666599999999999 2024-09-10T18:34:57.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.108661+0000", 2024-09-10T18:34:57.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028897369999999999 2024-09-10T18:34:57.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.109245+0000", 2024-09-10T18:34:57.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00058383000000000003 2024-09-10T18:34:57.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327149+0000", 2024-09-10T18:34:57.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21790462799999999 2024-09-10T18:34:57.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327180+0000", 2024-09-10T18:34:57.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0735999999999998e-05 2024-09-10T18:34:57.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327211+0000", 2024-09-10T18:34:57.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1038999999999999e-05 2024-09-10T18:34:57.521 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.521 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.521 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.521 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:115971 9.d 9:be0cedf3:::benchmark_data_smithi028_135703_object115970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-10T18:34:57.521 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:33:26.105428+0000", 2024-09-10T18:34:57.521 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 91.327387357000006, 2024-09-10T18:34:57.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.22190970199999999, 2024-09-10T18:34:57.521 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.521 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.522 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.522 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:34:57.522 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:57.522 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 115971 2024-09-10T18:34:57.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.522 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105428+0000", 2024-09-10T18:34:57.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105428+0000", 2024-09-10T18:34:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105427+0000", 2024-09-10T18:34:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:34:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.120120+0000", 2024-09-10T18:34:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014693513 2024-09-10T18:34:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.120136+0000", 2024-09-10T18:34:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5260999999999998e-05 2024-09-10T18:34:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.120142+0000", 2024-09-10T18:34:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4609999999999996e-06 2024-09-10T18:34:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.120186+0000", 2024-09-10T18:34:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3498000000000003e-05 2024-09-10T18:34:57.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.120304+0000", 2024-09-10T18:34:57.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000118449 2024-09-10T18:34:57.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,5", 2024-09-10T18:34:57.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.120446+0000", 2024-09-10T18:34:57.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014222200000000001 2024-09-10T18:34:57.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.526 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.123629+0000", 2024-09-10T18:34:57.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0031830719999999999 2024-09-10T18:34:57.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.526 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.526 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.123985+0000", 2024-09-10T18:34:57.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00035541499999999998 2024-09-10T18:34:57.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.526 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.526 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327254+0000", 2024-09-10T18:34:57.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.20326886 2024-09-10T18:34:57.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.526 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327300+0000", 2024-09-10T18:34:57.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6532000000000001e-05 2024-09-10T18:34:57.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327337+0000", 2024-09-10T18:34:57.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7285e-05 2024-09-10T18:34:57.527 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.527 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.527 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.527 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:191077 9.e 9:74bb5a96:::benchmark_data_smithi028_135703_object56486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-10T18:34:57.528 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:31.102004+0000", 2024-09-10T18:34:57.528 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 26.330811285999999, 2024-09-10T18:34:57.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13146555300000001, 2024-09-10T18:34:57.528 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.528 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.528 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.528 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:34:57.528 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:57.528 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 191077 2024-09-10T18:34:57.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.528 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102004+0000", 2024-09-10T18:34:57.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102004+0000", 2024-09-10T18:34:57.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102003+0000", 2024-09-10T18:34:57.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:34:57.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102008+0000", 2024-09-10T18:34:57.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7439999999999998e-06 2024-09-10T18:34:57.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102014+0000", 2024-09-10T18:34:57.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.878e-06 2024-09-10T18:34:57.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102019+0000", 2024-09-10T18:34:57.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3560000000000003e-06 2024-09-10T18:34:57.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.103206+0000", 2024-09-10T18:34:57.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001187378 2024-09-10T18:34:57.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.103309+0000", 2024-09-10T18:34:57.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103371 2024-09-10T18:34:57.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:34:57.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.103406+0000", 2024-09-10T18:34:57.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6100000000000005e-05 2024-09-10T18:34:57.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.107609+0000", 2024-09-10T18:34:57.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042029850000000002 2024-09-10T18:34:57.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.107735+0000", 2024-09-10T18:34:57.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012641700000000001 2024-09-10T18:34:57.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.233388+0000", 2024-09-10T18:34:57.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.12565284400000001 2024-09-10T18:34:57.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.233434+0000", 2024-09-10T18:34:57.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5844999999999999e-05 2024-09-10T18:34:57.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.233469+0000", 2024-09-10T18:34:57.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5663000000000001e-05 2024-09-10T18:34:57.533 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.533 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.533 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.534 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:191078 9.e 9:724867a8:::benchmark_data_smithi028_135703_object56487:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-10T18:34:57.534 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:31.102030+0000", 2024-09-10T18:34:57.534 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 26.330784692999998, 2024-09-10T18:34:57.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.131586658, 2024-09-10T18:34:57.534 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.534 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.534 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.534 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:34:57.534 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:57.534 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 191078 2024-09-10T18:34:57.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.534 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102030+0000", 2024-09-10T18:34:57.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102030+0000", 2024-09-10T18:34:57.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102030+0000", 2024-09-10T18:34:57.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:34:57.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102033+0000", 2024-09-10T18:34:57.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.41e-06 2024-09-10T18:34:57.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102039+0000", 2024-09-10T18:34:57.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9499999999999998e-06 2024-09-10T18:34:57.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102043+0000", 2024-09-10T18:34:57.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8890000000000004e-06 2024-09-10T18:34:57.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.105506+0000", 2024-09-10T18:34:57.538 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034628829999999999 2024-09-10T18:34:57.538 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.538 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.538 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.105611+0000", 2024-09-10T18:34:57.538 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010552099999999999 2024-09-10T18:34:57.538 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.538 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:34:57.538 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.105719+0000", 2024-09-10T18:34:57.538 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010804200000000001 2024-09-10T18:34:57.538 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.538 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.539 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.107833+0000", 2024-09-10T18:34:57.539 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021132019999999998 2024-09-10T18:34:57.539 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.539 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.539 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.539 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.108602+0000", 2024-09-10T18:34:57.539 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00076962999999999999 2024-09-10T18:34:57.539 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.539 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.539 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.539 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.233511+0000", 2024-09-10T18:34:57.539 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.12490902800000001 2024-09-10T18:34:57.539 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.539 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.540 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.540 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.233581+0000", 2024-09-10T18:34:57.540 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0279999999999998e-05 2024-09-10T18:34:57.540 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.540 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.540 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.540 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.233617+0000", 2024-09-10T18:34:57.540 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5568999999999998e-05 2024-09-10T18:34:57.540 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.540 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.540 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.540 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.540 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.540 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:191093 9.e 9:7de99e63:::benchmark_data_smithi028_135703_object56502:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-10T18:34:57.541 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:31.116300+0000", 2024-09-10T18:34:57.541 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 26.316514725000001, 2024-09-10T18:34:57.541 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.125951754, 2024-09-10T18:34:57.541 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.541 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.541 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.541 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:34:57.541 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:57.541 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 191093 2024-09-10T18:34:57.541 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.541 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.541 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.541 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.541 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.116300+0000", 2024-09-10T18:34:57.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.542 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.542 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.116300+0000", 2024-09-10T18:34:57.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.542 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.543 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.543 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.116299+0000", 2024-09-10T18:34:57.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:34:57.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.544 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.116306+0000", 2024-09-10T18:34:57.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1969999999999999e-06 2024-09-10T18:34:57.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.544 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.116316+0000", 2024-09-10T18:34:57.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4450000000000002e-06 2024-09-10T18:34:57.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.545 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.545 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.116321+0000", 2024-09-10T18:34:57.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5319999999999997e-06 2024-09-10T18:34:57.545 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.545 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.545 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.545 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.116365+0000", 2024-09-10T18:34:57.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3587000000000001e-05 2024-09-10T18:34:57.545 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.545 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.545 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.545 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.116441+0000", 2024-09-10T18:34:57.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6031000000000004e-05 2024-09-10T18:34:57.545 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.546 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.546 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:34:57.546 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.116507+0000", 2024-09-10T18:34:57.546 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6163999999999999e-05 2024-09-10T18:34:57.546 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.546 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.546 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.546 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.119086+0000", 2024-09-10T18:34:57.546 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025794839999999999 2024-09-10T18:34:57.546 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.546 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.546 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.546 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.119146+0000", 2024-09-10T18:34:57.546 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9179999999999999e-05 2024-09-10T18:34:57.547 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.547 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.547 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.547 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.242172+0000", 2024-09-10T18:34:57.547 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.123026064 2024-09-10T18:34:57.547 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.547 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.547 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.547 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.242215+0000", 2024-09-10T18:34:57.547 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3560000000000003e-05 2024-09-10T18:34:57.547 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.547 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.547 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.547 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.242252+0000", 2024-09-10T18:34:57.547 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6919999999999999e-05 2024-09-10T18:34:57.548 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.548 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.548 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.548 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.548 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.548 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:193358 9.3 9:c47564e4:::benchmark_data_smithi028_135703_object58767:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-10T18:34:57.548 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:33.992785+0000", 2024-09-10T18:34:57.548 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 23.440030479000001, 2024-09-10T18:34:57.548 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13214885500000001, 2024-09-10T18:34:57.548 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.548 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.548 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.548 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:34:57.548 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:57.549 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 193358 2024-09-10T18:34:57.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.549 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.549 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.549 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.992785+0000", 2024-09-10T18:34:57.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.549 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.549 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.992785+0000", 2024-09-10T18:34:57.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.992783+0000", 2024-09-10T18:34:57.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:34:57.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.992790+0000", 2024-09-10T18:34:57.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9979999999999997e-06 2024-09-10T18:34:57.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.992803+0000", 2024-09-10T18:34:57.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3283e-05 2024-09-10T18:34:57.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.992809+0000", 2024-09-10T18:34:57.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1959999999999996e-06 2024-09-10T18:34:57.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.992864+0000", 2024-09-10T18:34:57.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4421e-05 2024-09-10T18:34:57.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.992998+0000", 2024-09-10T18:34:57.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000134449 2024-09-10T18:34:57.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.552 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.552 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:34:57.552 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.993121+0000", 2024-09-10T18:34:57.552 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012321899999999999 2024-09-10T18:34:57.552 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.552 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.552 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.552 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.995481+0000", 2024-09-10T18:34:57.552 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023597340000000001 2024-09-10T18:34:57.552 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.552 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.552 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.552 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.996352+0000", 2024-09-10T18:34:57.553 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00087137300000000005 2024-09-10T18:34:57.553 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.553 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.553 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.553 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.124849+0000", 2024-09-10T18:34:57.553 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.12849706699999999 2024-09-10T18:34:57.553 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.553 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.553 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.553 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.124899+0000", 2024-09-10T18:34:57.553 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9917999999999999e-05 2024-09-10T18:34:57.553 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.553 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.553 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.554 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.124933+0000", 2024-09-10T18:34:57.554 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4146000000000001e-05 2024-09-10T18:34:57.554 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.554 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.554 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.554 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.554 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.554 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:193362 9.e 9:7a6df2df:::benchmark_data_smithi028_135703_object58771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-10T18:34:57.554 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:34.017062+0000", 2024-09-10T18:34:57.554 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 23.415753347999999, 2024-09-10T18:34:57.554 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.11261545100000001, 2024-09-10T18:34:57.554 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.554 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.554 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.554 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:34:57.554 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:57.555 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 193362 2024-09-10T18:34:57.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.555 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.017062+0000", 2024-09-10T18:34:57.555 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.017062+0000", 2024-09-10T18:34:57.555 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.017059+0000", 2024-09-10T18:34:57.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:34:57.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.017067+0000", 2024-09-10T18:34:57.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4780000000000002e-06 2024-09-10T18:34:57.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.017078+0000", 2024-09-10T18:34:57.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1051e-05 2024-09-10T18:34:57.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.017082+0000", 2024-09-10T18:34:57.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3139999999999997e-06 2024-09-10T18:34:57.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.017131+0000", 2024-09-10T18:34:57.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8600000000000002e-05 2024-09-10T18:34:57.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.017253+0000", 2024-09-10T18:34:57.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000122483 2024-09-10T18:34:57.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:34:57.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.017374+0000", 2024-09-10T18:34:57.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012040999999999999 2024-09-10T18:34:57.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.021525+0000", 2024-09-10T18:34:57.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004150801 2024-09-10T18:34:57.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.021547+0000", 2024-09-10T18:34:57.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2076e-05 2024-09-10T18:34:57.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.129583+0000", 2024-09-10T18:34:57.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.108036538 2024-09-10T18:34:57.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.559 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.559 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.129632+0000", 2024-09-10T18:34:57.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8707e-05 2024-09-10T18:34:57.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.559 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.559 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.129677+0000", 2024-09-10T18:34:57.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5293000000000002e-05 2024-09-10T18:34:57.559 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.559 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.559 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.560 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:193363 9.e 9:7f7dd43f:::benchmark_data_smithi028_135703_object58772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-10T18:34:57.560 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:34.019248+0000", 2024-09-10T18:34:57.560 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 23.413567308000001, 2024-09-10T18:34:57.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.112604315, 2024-09-10T18:34:57.560 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:34:57.560 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:34:57.560 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:34:57.560 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:34:57.560 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:34:57.560 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 193363 2024-09-10T18:34:57.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.560 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:34:57.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:34:57.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.019248+0000", 2024-09-10T18:34:57.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:34:57.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.019248+0000", 2024-09-10T18:34:57.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:34:57.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:34:57.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.019246+0000", 2024-09-10T18:34:57.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:34:57.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:34:57.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.019253+0000", 2024-09-10T18:34:57.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1700000000000002e-06 2024-09-10T18:34:57.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:34:57.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.019264+0000", 2024-09-10T18:34:57.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.163e-05 2024-09-10T18:34:57.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:34:57.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.019270+0000", 2024-09-10T18:34:57.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0700000000000003e-06 2024-09-10T18:34:57.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:34:57.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.019483+0000", 2024-09-10T18:34:57.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021277700000000001 2024-09-10T18:34:57.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:34:57.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.019614+0000", 2024-09-10T18:34:57.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000130576 2024-09-10T18:34:57.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:34:57.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.019749+0000", 2024-09-10T18:34:57.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013536099999999999 2024-09-10T18:34:57.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.021689+0000", 2024-09-10T18:34:57.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001939695 2024-09-10T18:34:57.564 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.564 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.564 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:34:57.564 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.021738+0000", 2024-09-10T18:34:57.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8928999999999999e-05 2024-09-10T18:34:57.564 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.564 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.564 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:34:57.564 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.131753+0000", 2024-09-10T18:34:57.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.110015473 2024-09-10T18:34:57.564 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.564 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.564 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:34:57.564 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.131802+0000", 2024-09-10T18:34:57.565 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8501000000000002e-05 2024-09-10T18:34:57.565 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:34:57.565 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:34:57.565 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:34:57.565 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.131852+0000", 2024-09-10T18:34:57.565 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0541000000000003e-05 2024-09-10T18:34:57.565 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.565 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.565 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.565 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:34:57.565 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:34:57.565 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:34:57.573 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:57.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:57.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:57.675 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:57.678 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:34:57.678 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:34:57.708 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:57.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:57.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:57.709 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:34:57.709 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:34:57.748 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:34:57.748 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:34:57.782 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:34:57.782 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:34:57.809 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:34:57.811 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:34:57.812 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:34:57.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:57.840 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:57.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:34:57.841 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:34:57.841 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:34:57.875 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:34:57.875 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:34:57.908 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:34:57.908 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:34:57.941 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:57.976 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:57.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:57.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:58.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[39220]: 1.14 scrub starts 2024-09-10T18:34:58.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[39220]: 1.14 scrub ok 2024-09-10T18:34:58.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[39220]: 5.13 scrub starts 2024-09-10T18:34:58.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[39220]: 5.13 scrub ok 2024-09-10T18:34:58.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[39220]: 6.9 scrub starts 2024-09-10T18:34:58.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[39220]: 6.9 scrub ok 2024-09-10T18:34:58.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[39220]: 1.7 scrub starts 2024-09-10T18:34:58.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[39220]: 1.7 scrub ok 2024-09-10T18:34:58.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[39220]: 9.d scrub ok 2024-09-10T18:34:58.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[39220]: pgmap v610: 178 pgs: 178 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 820 B/s rd, 0 B/s wr, 1.52k op/s 2024-09-10T18:34:58.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[39220]: 6.d scrub starts 2024-09-10T18:34:58.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[39220]: 6.d scrub ok 2024-09-10T18:34:58.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1371023381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:34:58.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[29291]: 1.14 scrub starts 2024-09-10T18:34:58.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[29291]: 1.14 scrub ok 2024-09-10T18:34:58.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[29291]: 5.13 scrub starts 2024-09-10T18:34:58.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[29291]: 5.13 scrub ok 2024-09-10T18:34:58.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[29291]: 6.9 scrub starts 2024-09-10T18:34:58.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[29291]: 6.9 scrub ok 2024-09-10T18:34:58.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[29291]: 1.7 scrub starts 2024-09-10T18:34:58.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[29291]: 1.7 scrub ok 2024-09-10T18:34:58.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[29291]: 9.d scrub ok 2024-09-10T18:34:58.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[29291]: pgmap v610: 178 pgs: 178 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 820 B/s rd, 0 B/s wr, 1.52k op/s 2024-09-10T18:34:58.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[29291]: 6.d scrub starts 2024-09-10T18:34:58.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[29291]: 6.d scrub ok 2024-09-10T18:34:58.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:57 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1371023381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:34:58.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:58.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:57 smithi204 ceph-mon[32357]: 1.14 scrub starts 2024-09-10T18:34:58.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:57 smithi204 ceph-mon[32357]: 1.14 scrub ok 2024-09-10T18:34:58.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:57 smithi204 ceph-mon[32357]: 5.13 scrub starts 2024-09-10T18:34:58.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:57 smithi204 ceph-mon[32357]: 5.13 scrub ok 2024-09-10T18:34:58.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:57 smithi204 ceph-mon[32357]: 6.9 scrub starts 2024-09-10T18:34:58.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:57 smithi204 ceph-mon[32357]: 6.9 scrub ok 2024-09-10T18:34:58.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:57 smithi204 ceph-mon[32357]: 1.7 scrub starts 2024-09-10T18:34:58.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:57 smithi204 ceph-mon[32357]: 1.7 scrub ok 2024-09-10T18:34:58.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:57 smithi204 ceph-mon[32357]: 9.d scrub ok 2024-09-10T18:34:58.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:57 smithi204 ceph-mon[32357]: pgmap v610: 178 pgs: 178 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 820 B/s rd, 0 B/s wr, 1.52k op/s 2024-09-10T18:34:58.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:57 smithi204 ceph-mon[32357]: 6.d scrub starts 2024-09-10T18:34:58.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:57 smithi204 ceph-mon[32357]: 6.d scrub ok 2024-09-10T18:34:58.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:57 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1371023381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:34:58.114 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:58.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:58.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:58.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:58.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:58.339 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T18:34:58.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:58.405 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:58.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:58.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:58.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:34:58.547 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:58.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:58.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:34:58.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:58.679 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:34:58.691 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:58.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:58.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:58.768 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:34:58.793 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:58.840 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:58.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:58.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:34:58.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:58.981 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:58.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:59.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:58 smithi028 ceph-mon[29291]: 5.8 scrub starts 2024-09-10T18:34:59.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:58 smithi028 ceph-mon[29291]: 5.8 scrub ok 2024-09-10T18:34:59.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:58 smithi028 ceph-mon[29291]: 5.10 scrub starts 2024-09-10T18:34:59.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:58 smithi028 ceph-mon[29291]: 5.10 scrub ok 2024-09-10T18:34:59.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:58 smithi028 ceph-mon[29291]: 5.15 scrub starts 2024-09-10T18:34:59.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:58 smithi028 ceph-mon[29291]: 5.15 scrub ok 2024-09-10T18:34:59.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:58 smithi028 ceph-mon[39220]: 5.8 scrub starts 2024-09-10T18:34:59.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:58 smithi028 ceph-mon[39220]: 5.8 scrub ok 2024-09-10T18:34:59.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:58 smithi028 ceph-mon[39220]: 5.10 scrub starts 2024-09-10T18:34:59.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:58 smithi028 ceph-mon[39220]: 5.10 scrub ok 2024-09-10T18:34:59.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:58 smithi028 ceph-mon[39220]: 5.15 scrub starts 2024-09-10T18:34:59.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:58 smithi028 ceph-mon[39220]: 5.15 scrub ok 2024-09-10T18:34:59.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:59.091 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:58 smithi204 ceph-mon[32357]: 5.8 scrub starts 2024-09-10T18:34:59.091 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:58 smithi204 ceph-mon[32357]: 5.8 scrub ok 2024-09-10T18:34:59.092 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:58 smithi204 ceph-mon[32357]: 5.10 scrub starts 2024-09-10T18:34:59.092 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:58 smithi204 ceph-mon[32357]: 5.10 scrub ok 2024-09-10T18:34:59.092 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:58 smithi204 ceph-mon[32357]: 5.15 scrub starts 2024-09-10T18:34:59.092 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:58 smithi204 ceph-mon[32357]: 5.15 scrub ok 2024-09-10T18:34:59.117 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:59.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:59.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:59.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:59.262 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:59.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:59.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:59.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:34:59.401 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:59.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:34:59.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:34:59.549 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:59.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:59.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:34:59.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:34:59.695 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:34:59.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:59.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:34:59.797 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:34:59.835 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:59.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:34:59.937 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:34:59.970 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:34:59.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:34:59.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:00.072 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:00.104 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:00.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:59 smithi204 ceph-mon[32357]: 5.a scrub starts 2024-09-10T18:35:00.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:59 smithi204 ceph-mon[32357]: 5.a scrub ok 2024-09-10T18:35:00.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:59 smithi204 ceph-mon[32357]: pgmap v611: 178 pgs: 178 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 958 op/s 2024-09-10T18:35:00.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:59 smithi204 ceph-mon[32357]: 1.1 scrub starts 2024-09-10T18:35:00.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:34:59 smithi204 ceph-mon[32357]: 1.1 scrub ok 2024-09-10T18:35:00.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:00.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:00.132 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:59 smithi028 ceph-mon[39220]: 5.a scrub starts 2024-09-10T18:35:00.132 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:59 smithi028 ceph-mon[39220]: 5.a scrub ok 2024-09-10T18:35:00.132 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:59 smithi028 ceph-mon[39220]: pgmap v611: 178 pgs: 178 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 958 op/s 2024-09-10T18:35:00.132 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:59 smithi028 ceph-mon[39220]: 1.1 scrub starts 2024-09-10T18:35:00.132 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:34:59 smithi028 ceph-mon[39220]: 1.1 scrub ok 2024-09-10T18:35:00.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:59 smithi028 ceph-mon[29291]: 5.a scrub starts 2024-09-10T18:35:00.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:59 smithi028 ceph-mon[29291]: 5.a scrub ok 2024-09-10T18:35:00.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:59 smithi028 ceph-mon[29291]: pgmap v611: 178 pgs: 178 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 958 op/s 2024-09-10T18:35:00.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:59 smithi028 ceph-mon[29291]: 1.1 scrub starts 2024-09-10T18:35:00.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:34:59 smithi028 ceph-mon[29291]: 1.1 scrub ok 2024-09-10T18:35:00.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:00.246 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:00.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:00.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:00.347 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:00.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:00.405 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:00.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:00.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:00.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:00.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:00.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:00.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:00.680 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:00.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:00.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:00.781 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:00.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:00.816 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:00.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:00.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:00.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:00.953 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:00.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:01.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:00 smithi028 ceph-mon[29291]: 9.c scrub starts 2024-09-10T18:35:01.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:00 smithi028 ceph-mon[29291]: 5.6 scrub starts 2024-09-10T18:35:01.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:00 smithi028 ceph-mon[29291]: 5.6 scrub ok 2024-09-10T18:35:01.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:00 smithi028 ceph-mon[29291]: 9.e scrub starts 2024-09-10T18:35:01.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:00 smithi028 ceph-mon[29291]: 4.2 scrub starts 2024-09-10T18:35:01.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:00 smithi028 ceph-mon[29291]: 4.2 scrub ok 2024-09-10T18:35:01.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:00 smithi028 ceph-mon[39220]: 9.c scrub starts 2024-09-10T18:35:01.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:00 smithi028 ceph-mon[39220]: 5.6 scrub starts 2024-09-10T18:35:01.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:00 smithi028 ceph-mon[39220]: 5.6 scrub ok 2024-09-10T18:35:01.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:00 smithi028 ceph-mon[39220]: 9.e scrub starts 2024-09-10T18:35:01.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:00 smithi028 ceph-mon[39220]: 4.2 scrub starts 2024-09-10T18:35:01.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:00 smithi028 ceph-mon[39220]: 4.2 scrub ok 2024-09-10T18:35:01.053 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:01.062 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:35:01.062 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:35:01.088 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:01.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:01.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:01.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:00 smithi204 ceph-mon[32357]: 9.c scrub starts 2024-09-10T18:35:01.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:00 smithi204 ceph-mon[32357]: 5.6 scrub starts 2024-09-10T18:35:01.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:00 smithi204 ceph-mon[32357]: 5.6 scrub ok 2024-09-10T18:35:01.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:00 smithi204 ceph-mon[32357]: 9.e scrub starts 2024-09-10T18:35:01.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:00 smithi204 ceph-mon[32357]: 4.2 scrub starts 2024-09-10T18:35:01.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:00 smithi204 ceph-mon[32357]: 4.2 scrub ok 2024-09-10T18:35:01.190 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:01.203 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:35:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:35:01.220 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:01.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:01.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:01.322 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:01.356 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:01.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:01.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:01.458 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:01.493 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:01.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:01.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:01.503 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:35:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:35:01] "GET /metrics HTTP/1.1" 200 224541 "" "Prometheus/2.43.0" 2024-09-10T18:35:01.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:01.632 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:01.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:01.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:01.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:01.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:01.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:01.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-10T18:35:01.876 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:01.896 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:01.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:01.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:02.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:01 smithi028 ceph-mon[29291]: 9.e scrub ok 2024-09-10T18:35:02.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:01 smithi028 ceph-mon[29291]: 4.1c scrub starts 2024-09-10T18:35:02.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:01 smithi028 ceph-mon[29291]: 4.1c scrub ok 2024-09-10T18:35:02.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:01 smithi028 ceph-mon[29291]: 9.c scrub ok 2024-09-10T18:35:02.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:01 smithi028 ceph-mon[29291]: 5.19 scrub starts 2024-09-10T18:35:02.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:01 smithi028 ceph-mon[29291]: 5.19 scrub ok 2024-09-10T18:35:02.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:01 smithi028 ceph-mon[29291]: pgmap v612: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-10T18:35:02.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:01 smithi028 ceph-mon[29291]: from='client.15579 v1:172.21.15.28:0/406749186' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:35:02.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:01 smithi028 ceph-mon[39220]: 9.e scrub ok 2024-09-10T18:35:02.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:01 smithi028 ceph-mon[39220]: 4.1c scrub starts 2024-09-10T18:35:02.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:01 smithi028 ceph-mon[39220]: 4.1c scrub ok 2024-09-10T18:35:02.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:01 smithi028 ceph-mon[39220]: 9.c scrub ok 2024-09-10T18:35:02.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:01 smithi028 ceph-mon[39220]: 5.19 scrub starts 2024-09-10T18:35:02.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:01 smithi028 ceph-mon[39220]: 5.19 scrub ok 2024-09-10T18:35:02.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:01 smithi028 ceph-mon[39220]: pgmap v612: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-10T18:35:02.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:01 smithi028 ceph-mon[39220]: from='client.15579 v1:172.21.15.28:0/406749186' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:35:02.035 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:02.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:02.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:01 smithi204 ceph-mon[32357]: 9.e scrub ok 2024-09-10T18:35:02.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:01 smithi204 ceph-mon[32357]: 4.1c scrub starts 2024-09-10T18:35:02.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:01 smithi204 ceph-mon[32357]: 4.1c scrub ok 2024-09-10T18:35:02.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:01 smithi204 ceph-mon[32357]: 9.c scrub ok 2024-09-10T18:35:02.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:01 smithi204 ceph-mon[32357]: 5.19 scrub starts 2024-09-10T18:35:02.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:01 smithi204 ceph-mon[32357]: 5.19 scrub ok 2024-09-10T18:35:02.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:01 smithi204 ceph-mon[32357]: pgmap v612: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-10T18:35:02.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:01 smithi204 ceph-mon[32357]: from='client.15579 v1:172.21.15.28:0/406749186' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:35:02.136 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:02.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:02.181 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:02.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:02.196 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:35:02.283 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:02.316 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:02.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:02.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:02.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:02.454 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:02.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:02.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:02.473 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:35:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:35:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:35:02.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:02.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:02.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:02.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:35:02.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:02.743 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:02.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:02.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:02.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:02.890 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:02.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:02.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:03.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:02 smithi028 ceph-mon[29291]: 5.14 scrub starts 2024-09-10T18:35:03.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:02 smithi028 ceph-mon[29291]: 5.14 scrub ok 2024-09-10T18:35:03.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:02 smithi028 ceph-mon[29291]: 5.1a scrub starts 2024-09-10T18:35:03.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:02 smithi028 ceph-mon[29291]: 5.1a scrub ok 2024-09-10T18:35:03.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:02 smithi028 ceph-mon[29291]: 3.17 scrub starts 2024-09-10T18:35:03.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:02 smithi028 ceph-mon[29291]: 1.3 scrub starts 2024-09-10T18:35:03.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:02 smithi028 ceph-mon[29291]: 1.3 scrub ok 2024-09-10T18:35:03.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:02 smithi028 ceph-mon[29291]: 3.17 scrub ok 2024-09-10T18:35:03.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:02 smithi028 ceph-mon[39220]: 5.14 scrub starts 2024-09-10T18:35:03.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:02 smithi028 ceph-mon[39220]: 5.14 scrub ok 2024-09-10T18:35:03.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:02 smithi028 ceph-mon[39220]: 5.1a scrub starts 2024-09-10T18:35:03.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:02 smithi028 ceph-mon[39220]: 5.1a scrub ok 2024-09-10T18:35:03.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:02 smithi028 ceph-mon[39220]: 3.17 scrub starts 2024-09-10T18:35:03.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:02 smithi028 ceph-mon[39220]: 1.3 scrub starts 2024-09-10T18:35:03.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:02 smithi028 ceph-mon[39220]: 1.3 scrub ok 2024-09-10T18:35:03.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:02 smithi028 ceph-mon[39220]: 3.17 scrub ok 2024-09-10T18:35:03.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:03.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:03.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:02 smithi204 ceph-mon[32357]: 5.14 scrub starts 2024-09-10T18:35:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:02 smithi204 ceph-mon[32357]: 5.14 scrub ok 2024-09-10T18:35:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:02 smithi204 ceph-mon[32357]: 5.1a scrub starts 2024-09-10T18:35:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:02 smithi204 ceph-mon[32357]: 5.1a scrub ok 2024-09-10T18:35:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:02 smithi204 ceph-mon[32357]: 3.17 scrub starts 2024-09-10T18:35:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:02 smithi204 ceph-mon[32357]: 1.3 scrub starts 2024-09-10T18:35:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:02 smithi204 ceph-mon[32357]: 1.3 scrub ok 2024-09-10T18:35:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:02 smithi204 ceph-mon[32357]: 3.17 scrub ok 2024-09-10T18:35:03.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:03.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:03.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:03.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:03.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:03.282 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:03.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:03.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:03.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:03.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:03.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:03.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:03.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:03.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:03.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:03.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:03.689 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":612,"stamp":"2024-09-10T18:35:01.039891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2819522681,"num_objects":43210,"num_object_clones":0,"num_object_copies":129630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43210,"num_whiteouts":0,"num_read":1991,"num_read_kb":1778,"num_write":226921,"num_write_kb":8614231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":829631,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71826,"ondisk_log_size":71826,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11555600,"kb_used_data":8272784,"kb_used_omap":19,"kb_used_meta":3282412,"kb_avail":738405616,"statfs":{"total":767960285184,"available":756127350784,"internally_reserved":0,"allocated":8471330816,"data_stored":8467411220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20033,"internal_metadata":3361190335},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-791718571,"num_objects":-12079,"num_object_clones":0,"num_object_copies":-36237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12079,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12079,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.221836"},"pg_stats":[{"pgid":"1.1d","version":"76'81","reported_seq":22,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.835903+0000","last_change":"2024-09-10T18:34:34.866222+0000","last_active":"2024-09-10T18:34:51.835903+0000","last_peered":"2024-09-10T18:34:51.835903+0000","last_clean":"2024-09-10T18:34:51.835903+0000","last_became_active":"2024-09-10T18:34:34.865894+0000","last_became_peered":"2024-09-10T18:34:34.865894+0000","last_unstale":"2024-09-10T18:34:51.835903+0000","last_undegraded":"2024-09-10T18:34:51.835903+0000","last_fullsized":"2024-09-10T18:34:51.835903+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:58.561425+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.561425+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":298,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829612+0000","last_change":"2024-09-10T18:33:04.320548+0000","last_active":"2024-09-10T18:34:51.829612+0000","last_peered":"2024-09-10T18:34:51.829612+0000","last_clean":"2024-09-10T18:34:51.829612+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:34:51.829612+0000","last_undegraded":"2024-09-10T18:34:51.829612+0000","last_fullsized":"2024-09-10T18:34:51.829612+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:04.320458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:33:04.320458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055421380000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":337,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.835475+0000","last_change":"2024-09-10T18:33:12.680256+0000","last_active":"2024-09-10T18:34:51.835475+0000","last_peered":"2024-09-10T18:34:51.835475+0000","last_clean":"2024-09-10T18:34:51.835475+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:34:51.835475+0000","last_undegraded":"2024-09-10T18:34:51.835475+0000","last_fullsized":"2024-09-10T18:34:51.835475+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:33:12.680138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:12.680138+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033442979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":303,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.834756+0000","last_change":"2024-09-10T18:32:43.156005+0000","last_active":"2024-09-10T18:34:51.834756+0000","last_peered":"2024-09-10T18:34:51.834756+0000","last_clean":"2024-09-10T18:34:51.834756+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:34:51.834756+0000","last_undegraded":"2024-09-10T18:34:51.834756+0000","last_fullsized":"2024-09-10T18:34:51.834756+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:43.155924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:43.155924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007043718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":301,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.835903+0000","last_change":"2024-09-10T18:33:08.643815+0000","last_active":"2024-09-10T18:34:51.835903+0000","last_peered":"2024-09-10T18:34:51.835903+0000","last_clean":"2024-09-10T18:34:51.835903+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:34:51.835903+0000","last_undegraded":"2024-09-10T18:34:51.835903+0000","last_fullsized":"2024-09-10T18:34:51.835903+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:08.643758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.643758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046354059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'81","reported_seq":18,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830790+0000","last_change":"2024-09-10T18:34:34.868599+0000","last_active":"2024-09-10T18:34:51.830790+0000","last_peered":"2024-09-10T18:34:51.830790+0000","last_clean":"2024-09-10T18:34:51.830790+0000","last_became_active":"2024-09-10T18:34:34.867388+0000","last_became_peered":"2024-09-10T18:34:34.867388+0000","last_unstale":"2024-09-10T18:34:51.830790+0000","last_undegraded":"2024-09-10T18:34:51.830790+0000","last_fullsized":"2024-09-10T18:34:51.830790+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.370561+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.370561+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029251390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54614,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":310,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829430+0000","last_change":"2024-09-10T18:34:16.293404+0000","last_active":"2024-09-10T18:34:51.829430+0000","last_peered":"2024-09-10T18:34:51.829430+0000","last_clean":"2024-09-10T18:34:51.829430+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:34:51.829430+0000","last_undegraded":"2024-09-10T18:34:51.829430+0000","last_fullsized":"2024-09-10T18:34:51.829430+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:16.293206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:16.293206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:37.489769+0000","scrub_duration":0.0021119960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":316,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829607+0000","last_change":"2024-09-10T18:34:22.415416+0000","last_active":"2024-09-10T18:34:51.829607+0000","last_peered":"2024-09-10T18:34:51.829607+0000","last_clean":"2024-09-10T18:34:51.829607+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:34:51.829607+0000","last_undegraded":"2024-09-10T18:34:51.829607+0000","last_fullsized":"2024-09-10T18:34:51.829607+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:22.415093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.415093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:23.013807+0000","scrub_duration":0.068845071999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'34","reported_seq":347,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830196+0000","last_change":"2024-09-10T18:32:55.371863+0000","last_active":"2024-09-10T18:34:51.830196+0000","last_peered":"2024-09-10T18:34:51.830196+0000","last_clean":"2024-09-10T18:34:51.830196+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:34:51.830196+0000","last_undegraded":"2024-09-10T18:34:51.830196+0000","last_fullsized":"2024-09-10T18:34:51.830196+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:32:55.371617+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:32:55.371617+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089694079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":313,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.835403+0000","last_change":"2024-09-10T18:34:09.440223+0000","last_active":"2024-09-10T18:34:51.835403+0000","last_peered":"2024-09-10T18:34:51.835403+0000","last_clean":"2024-09-10T18:34:51.835403+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:34:51.835403+0000","last_undegraded":"2024-09-10T18:34:51.835403+0000","last_fullsized":"2024-09-10T18:34:51.835403+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:09.440140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:34:09.440140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:16.383699+0000","scrub_duration":0.003425656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":318,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.820516+0000","last_change":"2024-09-10T18:33:16.963182+0000","last_active":"2024-09-10T18:34:51.820516+0000","last_peered":"2024-09-10T18:34:51.820516+0000","last_clean":"2024-09-10T18:34:51.820516+0000","last_became_active":"2024-09-10T18:26:27.644961+0000","last_became_peered":"2024-09-10T18:26:27.644961+0000","last_unstale":"2024-09-10T18:34:51.820516+0000","last_undegraded":"2024-09-10T18:34:51.820516+0000","last_fullsized":"2024-09-10T18:34:51.820516+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_clean_scrub_stamp":"2024-09-10T18:33:16.963102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019539739999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'15","reported_seq":340,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836155+0000","last_change":"2024-09-10T18:34:25.455830+0000","last_active":"2024-09-10T18:34:51.836155+0000","last_peered":"2024-09-10T18:34:51.836155+0000","last_clean":"2024-09-10T18:34:51.836155+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:34:51.836155+0000","last_undegraded":"2024-09-10T18:34:51.836155+0000","last_fullsized":"2024-09-10T18:34:51.836155+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:34:25.455754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:25.455754+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:48.508068+0000","scrub_duration":0.13960720300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":314,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.821986+0000","last_change":"2024-09-10T18:34:17.265867+0000","last_active":"2024-09-10T18:34:51.821986+0000","last_peered":"2024-09-10T18:34:51.821986+0000","last_clean":"2024-09-10T18:34:51.821986+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:34:51.821986+0000","last_undegraded":"2024-09-10T18:34:51.821986+0000","last_fullsized":"2024-09-10T18:34:51.821986+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:17.265788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_clean_scrub_stamp":"2024-09-10T18:34:17.265788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:22.473386+0000","scrub_duration":0.0023387740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":302,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.826169+0000","last_change":"2024-09-10T18:33:00.235163+0000","last_active":"2024-09-10T18:34:51.826169+0000","last_peered":"2024-09-10T18:34:51.826169+0000","last_clean":"2024-09-10T18:34:51.826169+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:34:51.826169+0000","last_undegraded":"2024-09-10T18:34:51.826169+0000","last_fullsized":"2024-09-10T18:34:51.826169+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:00.234931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:00.234931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023527999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'81","reported_seq":86,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829433+0000","last_change":"2024-09-10T18:34:34.863425+0000","last_active":"2024-09-10T18:34:51.829433+0000","last_peered":"2024-09-10T18:34:51.829433+0000","last_clean":"2024-09-10T18:34:51.829433+0000","last_became_active":"2024-09-10T18:34:34.863102+0000","last_became_peered":"2024-09-10T18:34:34.863102+0000","last_unstale":"2024-09-10T18:34:51.829433+0000","last_undegraded":"2024-09-10T18:34:51.829433+0000","last_fullsized":"2024-09-10T18:34:51.829433+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:48.006695+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:48.006695+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":318,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:54.513405+0000","last_change":"2024-09-10T18:34:54.513360+0000","last_active":"2024-09-10T18:34:54.513405+0000","last_peered":"2024-09-10T18:34:54.513405+0000","last_clean":"2024-09-10T18:34:54.513405+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:34:54.513405+0000","last_undegraded":"2024-09-10T18:34:54.513405+0000","last_fullsized":"2024-09-10T18:34:54.513405+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:54.513278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:17.464205+0000","last_clean_scrub_stamp":"2024-09-10T18:34:54.513278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:03.541538+0000","scrub_duration":0.0028371450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":352,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:52.235652+0000","last_change":"2024-09-10T18:34:52.235528+0000","last_active":"2024-09-10T18:34:52.235652+0000","last_peered":"2024-09-10T18:34:52.235652+0000","last_clean":"2024-09-10T18:34:52.235652+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:34:52.235652+0000","last_undegraded":"2024-09-10T18:34:52.235652+0000","last_fullsized":"2024-09-10T18:34:52.235652+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'9","last_scrub_stamp":"2024-09-10T18:34:52.235425+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:34:52.235425+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:58.995752+0000","scrub_duration":0.0029466150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":303,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830025+0000","last_change":"2024-09-10T18:32:01.525753+0000","last_active":"2024-09-10T18:34:51.830025+0000","last_peered":"2024-09-10T18:34:51.830025+0000","last_clean":"2024-09-10T18:34:51.830025+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:34:51.830025+0000","last_undegraded":"2024-09-10T18:34:51.830025+0000","last_fullsized":"2024-09-10T18:34:51.830025+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:01.525701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:01.525701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001223902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":305,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.821117+0000","last_change":"2024-09-10T18:34:09.151879+0000","last_active":"2024-09-10T18:34:51.821117+0000","last_peered":"2024-09-10T18:34:51.821117+0000","last_clean":"2024-09-10T18:34:51.821117+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:34:51.821117+0000","last_undegraded":"2024-09-10T18:34:51.821117+0000","last_fullsized":"2024-09-10T18:34:51.821117+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:09.151789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:34:09.151789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:10.873083+0000","scrub_duration":0.002651994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'81","reported_seq":169,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829468+0000","last_change":"2024-09-10T18:34:34.863449+0000","last_active":"2024-09-10T18:34:51.829468+0000","last_peered":"2024-09-10T18:34:51.829468+0000","last_clean":"2024-09-10T18:34:51.829468+0000","last_became_active":"2024-09-10T18:34:34.862917+0000","last_became_peered":"2024-09-10T18:34:34.862917+0000","last_unstale":"2024-09-10T18:34:51.829468+0000","last_undegraded":"2024-09-10T18:34:51.829468+0000","last_fullsized":"2024-09-10T18:34:51.829468+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:06.053455+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.053455+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003148245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":11,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":311,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836894+0000","last_change":"2024-09-10T18:34:07.394899+0000","last_active":"2024-09-10T18:34:51.836894+0000","last_peered":"2024-09-10T18:34:51.836894+0000","last_clean":"2024-09-10T18:34:51.836894+0000","last_became_active":"2024-09-10T18:26:27.650774+0000","last_became_peered":"2024-09-10T18:26:27.650774+0000","last_unstale":"2024-09-10T18:34:51.836894+0000","last_undegraded":"2024-09-10T18:34:51.836894+0000","last_fullsized":"2024-09-10T18:34:51.836894+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:07.394835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:07.394835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:13.236135+0000","scrub_duration":0.001116433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":304,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.833721+0000","last_change":"2024-09-10T18:34:12.458962+0000","last_active":"2024-09-10T18:34:51.833721+0000","last_peered":"2024-09-10T18:34:51.833721+0000","last_clean":"2024-09-10T18:34:51.833721+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:34:51.833721+0000","last_undegraded":"2024-09-10T18:34:51.833721+0000","last_fullsized":"2024-09-10T18:34:51.833721+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:12.458742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.458742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:40.818274+0000","scrub_duration":0.0077313979999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"240'27","reported_seq":368,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830109+0000","last_change":"2024-09-10T18:33:04.586182+0000","last_active":"2024-09-10T18:34:51.830109+0000","last_peered":"2024-09-10T18:34:51.830109+0000","last_clean":"2024-09-10T18:34:51.830109+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:34:51.830109+0000","last_undegraded":"2024-09-10T18:34:51.830109+0000","last_fullsized":"2024-09-10T18:34:51.830109+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:33:04.586063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:04.586063+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007748642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":313,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.835981+0000","last_change":"2024-09-10T18:32:58.252469+0000","last_active":"2024-09-10T18:34:51.835981+0000","last_peered":"2024-09-10T18:34:51.835981+0000","last_clean":"2024-09-10T18:34:51.835981+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:34:51.835981+0000","last_undegraded":"2024-09-10T18:34:51.835981+0000","last_fullsized":"2024-09-10T18:34:51.835981+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:58.252419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.252419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049362080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'81","reported_seq":240,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.819783+0000","last_change":"2024-09-10T18:34:34.862755+0000","last_active":"2024-09-10T18:34:51.819783+0000","last_peered":"2024-09-10T18:34:51.819783+0000","last_clean":"2024-09-10T18:34:51.819783+0000","last_became_active":"2024-09-10T18:34:34.855693+0000","last_became_peered":"2024-09-10T18:34:34.855693+0000","last_unstale":"2024-09-10T18:34:51.819783+0000","last_undegraded":"2024-09-10T18:34:51.819783+0000","last_fullsized":"2024-09-10T18:34:51.819783+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:08.009806+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.009806+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002902614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":310,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830496+0000","last_change":"2024-09-10T18:34:21.344309+0000","last_active":"2024-09-10T18:34:51.830496+0000","last_peered":"2024-09-10T18:34:51.830496+0000","last_clean":"2024-09-10T18:34:51.830496+0000","last_became_active":"2024-09-10T18:26:27.640035+0000","last_became_peered":"2024-09-10T18:26:27.640035+0000","last_unstale":"2024-09-10T18:34:51.830496+0000","last_undegraded":"2024-09-10T18:34:51.830496+0000","last_fullsized":"2024-09-10T18:34:51.830496+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:21.344236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.344236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:50.779397+0000","scrub_duration":0.0040101240000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":307,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836241+0000","last_change":"2024-09-10T18:32:59.567476+0000","last_active":"2024-09-10T18:34:51.836241+0000","last_peered":"2024-09-10T18:34:51.836241+0000","last_clean":"2024-09-10T18:34:51.836241+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:34:51.836241+0000","last_undegraded":"2024-09-10T18:34:51.836241+0000","last_fullsized":"2024-09-10T18:34:51.836241+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:59.567389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:32:59.567389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073893559999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'23","reported_seq":334,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.828838+0000","last_change":"2024-09-10T18:32:50.341138+0000","last_active":"2024-09-10T18:34:51.828838+0000","last_peered":"2024-09-10T18:34:51.828838+0000","last_clean":"2024-09-10T18:34:51.828838+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:34:51.828838+0000","last_undegraded":"2024-09-10T18:34:51.828838+0000","last_fullsized":"2024-09-10T18:34:51.828838+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'23","last_scrub_stamp":"2024-09-10T18:32:50.341044+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:32:50.341044+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012050170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":303,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836189+0000","last_change":"2024-09-10T18:33:07.697770+0000","last_active":"2024-09-10T18:34:51.836189+0000","last_peered":"2024-09-10T18:34:51.836189+0000","last_clean":"2024-09-10T18:34:51.836189+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:34:51.836189+0000","last_undegraded":"2024-09-10T18:34:51.836189+0000","last_fullsized":"2024-09-10T18:34:51.836189+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:07.697667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:33:07.697667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.044890945000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'81","reported_seq":228,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.820476+0000","last_change":"2024-09-10T18:34:34.855506+0000","last_active":"2024-09-10T18:34:51.820476+0000","last_peered":"2024-09-10T18:34:51.820476+0000","last_clean":"2024-09-10T18:34:51.820476+0000","last_became_active":"2024-09-10T18:34:34.855038+0000","last_became_peered":"2024-09-10T18:34:34.855038+0000","last_unstale":"2024-09-10T18:34:51.820476+0000","last_undegraded":"2024-09-10T18:34:51.820476+0000","last_fullsized":"2024-09-10T18:34:51.820476+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:48.466654+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.466654+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":303,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830589+0000","last_change":"2024-09-10T18:32:57.769495+0000","last_active":"2024-09-10T18:34:51.830589+0000","last_peered":"2024-09-10T18:34:51.830589+0000","last_clean":"2024-09-10T18:34:51.830589+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:34:51.830589+0000","last_undegraded":"2024-09-10T18:34:51.830589+0000","last_fullsized":"2024-09-10T18:34:51.830589+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:32:57.769408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:57.769408+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003757149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":317,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.834950+0000","last_change":"2024-09-10T18:32:51.182762+0000","last_active":"2024-09-10T18:34:51.834950+0000","last_peered":"2024-09-10T18:34:51.834950+0000","last_clean":"2024-09-10T18:34:51.834950+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:34:51.834950+0000","last_undegraded":"2024-09-10T18:34:51.834950+0000","last_fullsized":"2024-09-10T18:34:51.834950+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:51.182684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:32:51.182684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003614637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'25","reported_seq":341,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829034+0000","last_change":"2024-09-10T18:33:15.366317+0000","last_active":"2024-09-10T18:34:51.829034+0000","last_peered":"2024-09-10T18:34:51.829034+0000","last_clean":"2024-09-10T18:34:51.829034+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:34:51.829034+0000","last_undegraded":"2024-09-10T18:34:51.829034+0000","last_fullsized":"2024-09-10T18:34:51.829034+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:33:15.366107+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.366107+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014527711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"217'11","reported_seq":396,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:54.920651+0000","last_change":"2024-09-10T18:34:53.933334+0000","last_active":"2024-09-10T18:34:54.920651+0000","last_peered":"2024-09-10T18:34:54.920651+0000","last_clean":"2024-09-10T18:34:54.920651+0000","last_became_active":"2024-09-10T18:26:27.652489+0000","last_became_peered":"2024-09-10T18:26:27.652489+0000","last_unstale":"2024-09-10T18:34:54.920651+0000","last_undegraded":"2024-09-10T18:34:54.920651+0000","last_fullsized":"2024-09-10T18:34:54.920651+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11","last_scrub_stamp":"2024-09-10T18:34:53.933237+0000","last_deep_scrub":"217'11","last_deep_scrub_stamp":"2024-09-10T18:30:03.161985+0000","last_clean_scrub_stamp":"2024-09-10T18:34:53.933237+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:06.297756+0000","scrub_duration":0.0062113710000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"298'14202","reported_seq":14531,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:35:00.936283+0000","last_change":"2024-09-10T18:34:07.618780+0000","last_active":"2024-09-10T18:35:00.936283+0000","last_peered":"2024-09-10T18:35:00.936283+0000","last_clean":"2024-09-10T18:35:00.936283+0000","last_became_active":"2024-09-10T18:32:03.385428+0000","last_became_peered":"2024-09-10T18:32:03.385428+0000","last_unstale":"2024-09-10T18:35:00.936283+0000","last_undegraded":"2024-09-10T18:35:00.936283+0000","last_fullsized":"2024-09-10T18:35:00.936283+0000","mapping_epoch":246,"log_start":"282'10244","ondisk_log_start":"282'10244","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10375","last_scrub_stamp":"2024-09-10T18:34:07.618535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:07.618535+0000","objects_scrubbed":6480,"log_size":3958,"log_dups_size":0,"ondisk_log_size":3958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:25.198149+0000","scrub_duration":3.415268937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164626432,"num_objects":2512,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14202,"num_write_kb":534848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":234,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.821237+0000","last_change":"2024-09-10T18:34:32.823529+0000","last_active":"2024-09-10T18:34:51.821237+0000","last_peered":"2024-09-10T18:34:51.821237+0000","last_clean":"2024-09-10T18:34:51.821237+0000","last_became_active":"2024-09-10T18:34:32.823230+0000","last_became_peered":"2024-09-10T18:34:32.823230+0000","last_unstale":"2024-09-10T18:34:51.821237+0000","last_undegraded":"2024-09-10T18:34:51.821237+0000","last_fullsized":"2024-09-10T18:34:51.821237+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:56.155354+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:56.155354+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001679647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"240'34","reported_seq":368,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.835648+0000","last_change":"2024-09-10T18:32:56.285685+0000","last_active":"2024-09-10T18:34:51.835648+0000","last_peered":"2024-09-10T18:34:51.835648+0000","last_clean":"2024-09-10T18:34:51.835648+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:34:51.835648+0000","last_undegraded":"2024-09-10T18:34:51.835648+0000","last_fullsized":"2024-09-10T18:34:51.835648+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:32:56.285589+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:32:56.285589+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081173400000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":309,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829356+0000","last_change":"2024-09-10T18:34:20.305626+0000","last_active":"2024-09-10T18:34:51.829356+0000","last_peered":"2024-09-10T18:34:51.829356+0000","last_clean":"2024-09-10T18:34:51.829356+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:34:51.829356+0000","last_undegraded":"2024-09-10T18:34:51.829356+0000","last_fullsized":"2024-09-10T18:34:51.829356+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:20.305547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:20.305547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:42.039144+0000","scrub_duration":0.0046035989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":475,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:53.828302+0000","last_change":"2024-09-10T18:34:14.495568+0000","last_active":"2024-09-10T18:34:53.828302+0000","last_peered":"2024-09-10T18:34:53.828302+0000","last_clean":"2024-09-10T18:34:53.828302+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:34:53.828302+0000","last_undegraded":"2024-09-10T18:34:53.828302+0000","last_fullsized":"2024-09-10T18:34:53.828302+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:34:14.495486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:14.495486+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:34.967733+0000","scrub_duration":0.010908085,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":312,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:57.526828+0000","last_change":"2024-09-10T18:34:57.526652+0000","last_active":"2024-09-10T18:34:57.526828+0000","last_peered":"2024-09-10T18:34:57.526828+0000","last_clean":"2024-09-10T18:34:57.526828+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:34:57.526828+0000","last_undegraded":"2024-09-10T18:34:57.526828+0000","last_fullsized":"2024-09-10T18:34:57.526828+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:57.526512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:34:57.526512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:24.046577+0000","scrub_duration":0.0049647529999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"298'13874","reported_seq":14063,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:59.971419+0000","last_change":"2024-09-10T18:34:56.967478+0000","last_active":"2024-09-10T18:34:59.971419+0000","last_peered":"2024-09-10T18:34:59.971419+0000","last_clean":"2024-09-10T18:34:59.971419+0000","last_became_active":"2024-09-10T18:32:03.379257+0000","last_became_peered":"2024-09-10T18:32:03.379257+0000","last_unstale":"2024-09-10T18:34:59.971419+0000","last_undegraded":"2024-09-10T18:34:59.971419+0000","last_fullsized":"2024-09-10T18:34:59.971419+0000","mapping_epoch":246,"log_start":"281'9916","ondisk_log_start":"281'9916","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'13598","last_scrub_stamp":"2024-09-10T18:34:56.967395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:56.967395+0000","objects_scrubbed":2956,"log_size":3958,"log_dups_size":0,"ondisk_log_size":3958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:10.206524+0000","scrub_duration":1.133103159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172949528,"num_objects":2640,"num_object_clones":0,"num_object_copies":7920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2640,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13874,"num_write_kb":528385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":243,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836911+0000","last_change":"2024-09-10T18:34:32.816324+0000","last_active":"2024-09-10T18:34:51.836911+0000","last_peered":"2024-09-10T18:34:51.836911+0000","last_clean":"2024-09-10T18:34:51.836911+0000","last_became_active":"2024-09-10T18:34:32.815839+0000","last_became_peered":"2024-09-10T18:34:32.815839+0000","last_unstale":"2024-09-10T18:34:51.836911+0000","last_undegraded":"2024-09-10T18:34:51.836911+0000","last_fullsized":"2024-09-10T18:34:51.836911+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:22.344759+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.344759+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002682837,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":347,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829068+0000","last_change":"2024-09-10T18:32:59.382953+0000","last_active":"2024-09-10T18:34:51.829068+0000","last_peered":"2024-09-10T18:34:51.829068+0000","last_clean":"2024-09-10T18:34:51.829068+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:34:51.829068+0000","last_undegraded":"2024-09-10T18:34:51.829068+0000","last_fullsized":"2024-09-10T18:34:51.829068+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-09-10T18:32:59.382840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:59.382840+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0072140750000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":299,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.831845+0000","last_change":"2024-09-10T18:32:49.346233+0000","last_active":"2024-09-10T18:34:51.831845+0000","last_peered":"2024-09-10T18:34:51.831845+0000","last_clean":"2024-09-10T18:34:51.831845+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:34:51.831845+0000","last_undegraded":"2024-09-10T18:34:51.831845+0000","last_fullsized":"2024-09-10T18:34:51.831845+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:49.342269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.140922+0000","last_clean_scrub_stamp":"2024-09-10T18:32:49.342269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006539385,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":303,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.835523+0000","last_change":"2024-09-10T18:32:52.582545+0000","last_active":"2024-09-10T18:34:51.835523+0000","last_peered":"2024-09-10T18:34:51.835523+0000","last_clean":"2024-09-10T18:34:51.835523+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:34:51.835523+0000","last_undegraded":"2024-09-10T18:34:51.835523+0000","last_fullsized":"2024-09-10T18:34:51.835523+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:52.582464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:32:52.582464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024277399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":301,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836075+0000","last_change":"2024-09-10T18:34:21.322560+0000","last_active":"2024-09-10T18:34:51.836075+0000","last_peered":"2024-09-10T18:34:51.836075+0000","last_clean":"2024-09-10T18:34:51.836075+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:34:51.836075+0000","last_undegraded":"2024-09-10T18:34:51.836075+0000","last_fullsized":"2024-09-10T18:34:51.836075+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:21.322454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.322454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:50.552165+0000","scrub_duration":0.0032746329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"298'17721","reported_seq":18041,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:35:00.929254+0000","last_change":"2024-09-10T18:35:00.803084+0000","last_active":"2024-09-10T18:35:00.929254+0000","last_peered":"2024-09-10T18:35:00.929254+0000","last_clean":"2024-09-10T18:35:00.929254+0000","last_became_active":"2024-09-10T18:32:03.554400+0000","last_became_peered":"2024-09-10T18:32:03.554400+0000","last_unstale":"2024-09-10T18:35:00.929254+0000","last_undegraded":"2024-09-10T18:35:00.929254+0000","last_fullsized":"2024-09-10T18:35:00.929254+0000","mapping_epoch":246,"log_start":"288'13813","ondisk_log_start":"288'13813","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'17711","last_scrub_stamp":"2024-09-10T18:35:00.803002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:35:00.803002+0000","objects_scrubbed":2635,"log_size":3908,"log_dups_size":0,"ondisk_log_size":3908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:23.133375+0000","scrub_duration":1.1033014139999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168558592,"num_objects":2572,"num_object_clones":0,"num_object_copies":7716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14286,"num_write_kb":539456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":214,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.833135+0000","last_change":"2024-09-10T18:34:00.166754+0000","last_active":"2024-09-10T18:34:51.833135+0000","last_peered":"2024-09-10T18:34:51.833135+0000","last_clean":"2024-09-10T18:34:51.833135+0000","last_became_active":"2024-09-10T18:34:00.166362+0000","last_became_peered":"2024-09-10T18:34:00.166362+0000","last_unstale":"2024-09-10T18:34:51.833135+0000","last_undegraded":"2024-09-10T18:34:51.833135+0000","last_fullsized":"2024-09-10T18:34:51.833135+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.370561+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.370561+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029251390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"240'28","reported_seq":354,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.835723+0000","last_change":"2024-09-10T18:33:59.587714+0000","last_active":"2024-09-10T18:34:51.835723+0000","last_peered":"2024-09-10T18:34:51.835723+0000","last_clean":"2024-09-10T18:34:51.835723+0000","last_became_active":"2024-09-10T18:26:27.651364+0000","last_became_peered":"2024-09-10T18:26:27.651364+0000","last_unstale":"2024-09-10T18:34:51.835723+0000","last_undegraded":"2024-09-10T18:34:51.835723+0000","last_fullsized":"2024-09-10T18:34:51.835723+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'28","last_scrub_stamp":"2024-09-10T18:33:59.587604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:59.587604+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:21.896292+0000","scrub_duration":0.0024058019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":308,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:53.715656+0000","last_change":"2024-09-10T18:34:53.715596+0000","last_active":"2024-09-10T18:34:53.715656+0000","last_peered":"2024-09-10T18:34:53.715656+0000","last_clean":"2024-09-10T18:34:53.715656+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:34:53.715656+0000","last_undegraded":"2024-09-10T18:34:53.715656+0000","last_fullsized":"2024-09-10T18:34:53.715656+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:53.715509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:53.715509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:55.015864+0000","scrub_duration":0.0053715869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":466,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:54.920877+0000","last_change":"2024-09-10T18:33:07.367306+0000","last_active":"2024-09-10T18:34:54.920877+0000","last_peered":"2024-09-10T18:34:54.920877+0000","last_clean":"2024-09-10T18:34:54.920877+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:34:54.920877+0000","last_undegraded":"2024-09-10T18:34:54.920877+0000","last_fullsized":"2024-09-10T18:34:54.920877+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:33:07.367248+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:33:07.367248+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044945339999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":311,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830725+0000","last_change":"2024-09-10T18:33:11.857892+0000","last_active":"2024-09-10T18:34:51.830725+0000","last_peered":"2024-09-10T18:34:51.830725+0000","last_clean":"2024-09-10T18:34:51.830725+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:34:51.830725+0000","last_undegraded":"2024-09-10T18:34:51.830725+0000","last_fullsized":"2024-09-10T18:34:51.830725+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:11.857685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.857685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022995870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"298'14139","reported_seq":14481,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:35:00.074267+0000","last_change":"2024-09-10T18:34:04.158338+0000","last_active":"2024-09-10T18:35:00.074267+0000","last_peered":"2024-09-10T18:35:00.074267+0000","last_clean":"2024-09-10T18:35:00.074267+0000","last_became_active":"2024-09-10T18:32:03.384288+0000","last_became_peered":"2024-09-10T18:32:03.384288+0000","last_unstale":"2024-09-10T18:35:00.074267+0000","last_undegraded":"2024-09-10T18:35:00.074267+0000","last_fullsized":"2024-09-10T18:35:00.074267+0000","mapping_epoch":246,"log_start":"278'9775","ondisk_log_start":"278'9775","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'10160","last_scrub_stamp":"2024-09-10T18:34:04.158266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:04.158266+0000","objects_scrubbed":6798,"log_size":4364,"log_dups_size":0,"ondisk_log_size":4364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:13.705054+0000","scrub_duration":2.9688170149999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176488448,"num_objects":2693,"num_object_clones":0,"num_object_copies":8079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14139,"num_write_kb":538624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":303,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:56.812347+0000","last_change":"2024-09-10T18:34:56.812303+0000","last_active":"2024-09-10T18:34:56.812347+0000","last_peered":"2024-09-10T18:34:56.812347+0000","last_clean":"2024-09-10T18:34:56.812347+0000","last_became_active":"2024-09-10T18:34:32.818323+0000","last_became_peered":"2024-09-10T18:34:32.818323+0000","last_unstale":"2024-09-10T18:34:56.812347+0000","last_undegraded":"2024-09-10T18:34:56.812347+0000","last_fullsized":"2024-09-10T18:34:56.812347+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:56.812219+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:56.812219+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:01.522405+0000","scrub_duration":0.0061304319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":360,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:59.896174+0000","last_change":"2024-09-10T18:34:59.896028+0000","last_active":"2024-09-10T18:34:59.896174+0000","last_peered":"2024-09-10T18:34:59.896174+0000","last_clean":"2024-09-10T18:34:59.896174+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:34:59.896174+0000","last_undegraded":"2024-09-10T18:34:59.896174+0000","last_fullsized":"2024-09-10T18:34:59.896174+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:34:59.895743+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:34:59.895743+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:18.992336+0000","scrub_duration":0.0031796210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":308,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:56.510448+0000","last_change":"2024-09-10T18:34:56.510382+0000","last_active":"2024-09-10T18:34:56.510448+0000","last_peered":"2024-09-10T18:34:56.510448+0000","last_clean":"2024-09-10T18:34:56.510448+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:34:56.510448+0000","last_undegraded":"2024-09-10T18:34:56.510448+0000","last_fullsized":"2024-09-10T18:34:56.510448+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:56.510308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:34:56.510308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:24.356371+0000","scrub_duration":0.0059794410000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"263'11","reported_seq":473,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:53.828648+0000","last_change":"2024-09-10T18:33:01.283747+0000","last_active":"2024-09-10T18:34:53.828648+0000","last_peered":"2024-09-10T18:34:53.828648+0000","last_clean":"2024-09-10T18:34:53.828648+0000","last_became_active":"2024-09-10T18:26:27.639032+0000","last_became_peered":"2024-09-10T18:26:27.639032+0000","last_unstale":"2024-09-10T18:34:53.828648+0000","last_undegraded":"2024-09-10T18:34:53.828648+0000","last_fullsized":"2024-09-10T18:34:53.828648+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'11","last_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_deep_scrub":"263'11","last_deep_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.283648+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081388439999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":302,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.826354+0000","last_change":"2024-09-10T18:33:57.483999+0000","last_active":"2024-09-10T18:34:51.826354+0000","last_peered":"2024-09-10T18:34:51.826354+0000","last_clean":"2024-09-10T18:34:51.826354+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:34:51.826354+0000","last_undegraded":"2024-09-10T18:34:51.826354+0000","last_fullsized":"2024-09-10T18:34:51.826354+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:57.483905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:57.483905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:34:58.495941+0000","scrub_duration":0.023666573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"298'14180","reported_seq":14385,"reported_epoch":298,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T18:34:59.992027+0000","last_change":"2024-09-10T18:34:59.735431+0000","last_active":"2024-09-10T18:34:59.992027+0000","last_peered":"2024-09-10T18:34:59.992027+0000","last_clean":"2024-09-10T18:34:59.992027+0000","last_became_active":"2024-09-10T18:32:03.554693+0000","last_became_peered":"2024-09-10T18:32:03.554693+0000","last_unstale":"2024-09-10T18:34:59.992027+0000","last_undegraded":"2024-09-10T18:34:59.992027+0000","last_fullsized":"2024-09-10T18:34:59.992027+0000","mapping_epoch":246,"log_start":"281'10247","ondisk_log_start":"281'10247","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:32:02.366962+0000","objects_scrubbed":555,"log_size":3933,"log_dups_size":0,"ondisk_log_size":3933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174850048,"num_objects":2668,"num_object_clones":0,"num_object_copies":8004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14180,"num_write_kb":539136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":303,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:54.904317+0000","last_change":"2024-09-10T18:34:54.904272+0000","last_active":"2024-09-10T18:34:54.904317+0000","last_peered":"2024-09-10T18:34:54.904317+0000","last_clean":"2024-09-10T18:34:54.904317+0000","last_became_active":"2024-09-10T18:34:32.918946+0000","last_became_peered":"2024-09-10T18:34:32.918946+0000","last_unstale":"2024-09-10T18:34:54.904317+0000","last_undegraded":"2024-09-10T18:34:54.904317+0000","last_fullsized":"2024-09-10T18:34:54.904317+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:54.904210+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:54.904210+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:55.647203+0000","scrub_duration":0.0075563590000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"240'29","reported_seq":357,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.833436+0000","last_change":"2024-09-10T18:33:11.310163+0000","last_active":"2024-09-10T18:34:51.833436+0000","last_peered":"2024-09-10T18:34:51.833436+0000","last_clean":"2024-09-10T18:34:51.833436+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:34:51.833436+0000","last_undegraded":"2024-09-10T18:34:51.833436+0000","last_fullsized":"2024-09-10T18:34:51.833436+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'29","last_scrub_stamp":"2024-09-10T18:33:11.310074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.310074+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026943760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":307,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836759+0000","last_change":"2024-09-10T18:34:02.559656+0000","last_active":"2024-09-10T18:34:51.836759+0000","last_peered":"2024-09-10T18:34:51.836759+0000","last_clean":"2024-09-10T18:34:51.836759+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:34:51.836759+0000","last_undegraded":"2024-09-10T18:34:51.836759+0000","last_fullsized":"2024-09-10T18:34:51.836759+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:02.559431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:02.559431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:18.828380+0000","scrub_duration":0.019462604000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":263,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829053+0000","last_change":"2024-09-10T18:34:22.795033+0000","last_active":"2024-09-10T18:34:51.829053+0000","last_peered":"2024-09-10T18:34:51.829053+0000","last_clean":"2024-09-10T18:34:51.829053+0000","last_became_active":"2024-09-10T18:26:27.642063+0000","last_became_peered":"2024-09-10T18:26:27.642063+0000","last_unstale":"2024-09-10T18:34:51.829053+0000","last_undegraded":"2024-09-10T18:34:51.829053+0000","last_fullsized":"2024-09-10T18:34:51.829053+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:22.794935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.794935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:45.015520+0000","scrub_duration":0.0028594010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":308,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.834687+0000","last_change":"2024-09-10T18:33:10.137145+0000","last_active":"2024-09-10T18:34:51.834687+0000","last_peered":"2024-09-10T18:34:51.834687+0000","last_clean":"2024-09-10T18:34:51.834687+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:34:51.834687+0000","last_undegraded":"2024-09-10T18:34:51.834687+0000","last_fullsized":"2024-09-10T18:34:51.834687+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:10.137048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:33:10.137048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040149249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"298'17344","reported_seq":17845,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:56.113797+0000","last_change":"2024-09-10T18:34:21.239886+0000","last_active":"2024-09-10T18:34:56.113797+0000","last_peered":"2024-09-10T18:34:56.113797+0000","last_clean":"2024-09-10T18:34:56.113797+0000","last_became_active":"2024-09-10T18:32:03.381872+0000","last_became_peered":"2024-09-10T18:32:03.381872+0000","last_unstale":"2024-09-10T18:34:56.113797+0000","last_undegraded":"2024-09-10T18:34:56.113797+0000","last_fullsized":"2024-09-10T18:34:56.113797+0000","mapping_epoch":246,"log_start":"274'12252","ondisk_log_start":"274'12252","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'14889","last_scrub_stamp":"2024-09-10T18:34:21.239625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.239625+0000","objects_scrubbed":5453,"log_size":5092,"log_dups_size":0,"ondisk_log_size":5092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:26.853140+0000","scrub_duration":1.7487293830000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192348160,"num_objects":2935,"num_object_clones":0,"num_object_copies":8805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13813,"num_write_kb":535936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":219,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:59.440512+0000","last_change":"2024-09-10T18:34:59.440417+0000","last_active":"2024-09-10T18:34:59.440512+0000","last_peered":"2024-09-10T18:34:59.440512+0000","last_clean":"2024-09-10T18:34:59.440512+0000","last_became_active":"2024-09-10T18:34:00.151017+0000","last_became_peered":"2024-09-10T18:34:00.151017+0000","last_unstale":"2024-09-10T18:34:59.440512+0000","last_undegraded":"2024-09-10T18:34:59.440512+0000","last_fullsized":"2024-09-10T18:34:59.440512+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:59.440207+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:59.440207+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:03.111983+0000","scrub_duration":0.0048255449999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"240'41","reported_seq":452,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:54.921079+0000","last_change":"2024-09-10T18:32:55.941738+0000","last_active":"2024-09-10T18:34:54.921079+0000","last_peered":"2024-09-10T18:34:54.921079+0000","last_clean":"2024-09-10T18:34:54.921079+0000","last_became_active":"2024-09-10T18:23:10.067049+0000","last_became_peered":"2024-09-10T18:23:10.067049+0000","last_unstale":"2024-09-10T18:34:54.921079+0000","last_undegraded":"2024-09-10T18:34:54.921079+0000","last_fullsized":"2024-09-10T18:34:54.921079+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'41","last_scrub_stamp":"2024-09-10T18:32:55.941653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:55.941653+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011585564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":314,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836243+0000","last_change":"2024-09-10T18:34:17.387421+0000","last_active":"2024-09-10T18:34:51.836243+0000","last_peered":"2024-09-10T18:34:51.836243+0000","last_clean":"2024-09-10T18:34:51.836243+0000","last_became_active":"2024-09-10T18:26:27.648690+0000","last_became_peered":"2024-09-10T18:26:27.648690+0000","last_unstale":"2024-09-10T18:34:51.836243+0000","last_undegraded":"2024-09-10T18:34:51.836243+0000","last_fullsized":"2024-09-10T18:34:51.836243+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:17.387344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:17.387344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:33.806832+0000","scrub_duration":0.0051904949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":323,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.821648+0000","last_change":"2024-09-10T18:33:04.936495+0000","last_active":"2024-09-10T18:34:51.821648+0000","last_peered":"2024-09-10T18:34:51.821648+0000","last_clean":"2024-09-10T18:34:51.821648+0000","last_became_active":"2024-09-10T18:26:27.647576+0000","last_became_peered":"2024-09-10T18:26:27.647576+0000","last_unstale":"2024-09-10T18:34:51.821648+0000","last_undegraded":"2024-09-10T18:34:51.821648+0000","last_fullsized":"2024-09-10T18:34:51.821648+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:33:04.936407+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:33:04.936407+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096149180000000001,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":304,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830211+0000","last_change":"2024-09-10T18:34:21.562999+0000","last_active":"2024-09-10T18:34:51.830211+0000","last_peered":"2024-09-10T18:34:51.830211+0000","last_clean":"2024-09-10T18:34:51.830211+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:34:51.830211+0000","last_undegraded":"2024-09-10T18:34:51.830211+0000","last_fullsized":"2024-09-10T18:34:51.830211+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:21.562920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.562920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:36.317098+0000","scrub_duration":0.0041137129999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":293,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829281+0000","last_change":"2024-09-10T18:33:56.168567+0000","last_active":"2024-09-10T18:34:51.829281+0000","last_peered":"2024-09-10T18:34:51.829281+0000","last_clean":"2024-09-10T18:34:51.829281+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:34:51.829281+0000","last_undegraded":"2024-09-10T18:34:51.829281+0000","last_fullsized":"2024-09-10T18:34:51.829281+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:56.168485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:56.168485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:14.414730+0000","scrub_duration":0.002436723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"298'14322","reported_seq":14659,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:35:00.061933+0000","last_change":"2024-09-10T18:34:06.677663+0000","last_active":"2024-09-10T18:35:00.061933+0000","last_peered":"2024-09-10T18:35:00.061933+0000","last_clean":"2024-09-10T18:35:00.061933+0000","last_became_active":"2024-09-10T18:32:03.380391+0000","last_became_peered":"2024-09-10T18:32:03.380391+0000","last_unstale":"2024-09-10T18:35:00.061933+0000","last_undegraded":"2024-09-10T18:35:00.061933+0000","last_fullsized":"2024-09-10T18:35:00.061933+0000","mapping_epoch":246,"log_start":"279'9967","ondisk_log_start":"279'9967","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10475","last_scrub_stamp":"2024-09-10T18:34:06.677564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.677564+0000","objects_scrubbed":6683,"log_size":4355,"log_dups_size":0,"ondisk_log_size":4355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:12.828051+0000","scrub_duration":2.5274349109999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179830784,"num_objects":2744,"num_object_clones":0,"num_object_copies":8232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14322,"num_write_kb":546112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":355,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829859+0000","last_change":"2024-09-10T18:34:00.154687+0000","last_active":"2024-09-10T18:34:51.829859+0000","last_peered":"2024-09-10T18:34:51.829859+0000","last_clean":"2024-09-10T18:34:51.829859+0000","last_became_active":"2024-09-10T18:34:00.154404+0000","last_became_peered":"2024-09-10T18:34:00.154404+0000","last_unstale":"2024-09-10T18:34:51.829859+0000","last_undegraded":"2024-09-10T18:34:51.829859+0000","last_fullsized":"2024-09-10T18:34:51.829859+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:57.474295+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.474295+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015998690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"240'17","reported_seq":348,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.835816+0000","last_change":"2024-09-10T18:34:13.522168+0000","last_active":"2024-09-10T18:34:51.835816+0000","last_peered":"2024-09-10T18:34:51.835816+0000","last_clean":"2024-09-10T18:34:51.835816+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:34:51.835816+0000","last_undegraded":"2024-09-10T18:34:51.835816+0000","last_fullsized":"2024-09-10T18:34:51.835816+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'17","last_scrub_stamp":"2024-09-10T18:34:13.521944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:13.521944+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:35.758470+0000","scrub_duration":0.0029448339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":324,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.835321+0000","last_change":"2024-09-10T18:33:15.142062+0000","last_active":"2024-09-10T18:34:51.835321+0000","last_peered":"2024-09-10T18:34:51.835321+0000","last_clean":"2024-09-10T18:34:51.835321+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:34:51.835321+0000","last_undegraded":"2024-09-10T18:34:51.835321+0000","last_fullsized":"2024-09-10T18:34:51.835321+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:15.141982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:15.141982+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.141982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028956749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":336,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.828246+0000","last_change":"2024-09-10T18:34:06.493534+0000","last_active":"2024-09-10T18:34:51.828246+0000","last_peered":"2024-09-10T18:34:51.828246+0000","last_clean":"2024-09-10T18:34:51.828246+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:34:51.828246+0000","last_undegraded":"2024-09-10T18:34:51.828246+0000","last_fullsized":"2024-09-10T18:34:51.828246+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:34:06.493448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.493448+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:21.109643+0000","scrub_duration":0.011827739,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":323,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:59.713802+0000","last_change":"2024-09-10T18:34:59.713703+0000","last_active":"2024-09-10T18:34:59.713802+0000","last_peered":"2024-09-10T18:34:59.713802+0000","last_clean":"2024-09-10T18:34:59.713802+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:34:59.713802+0000","last_undegraded":"2024-09-10T18:34:59.713802+0000","last_fullsized":"2024-09-10T18:34:59.713802+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:59.713536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:34:59.713536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:21.888517+0000","scrub_duration":0.002321769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":311,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.933472+0000","last_change":"2024-09-10T18:34:51.933409+0000","last_active":"2024-09-10T18:34:51.933472+0000","last_peered":"2024-09-10T18:34:51.933472+0000","last_clean":"2024-09-10T18:34:51.933472+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:34:51.933472+0000","last_undegraded":"2024-09-10T18:34:51.933472+0000","last_fullsized":"2024-09-10T18:34:51.933472+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:51.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:51.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:06.988199+0000","scrub_duration":0.0029402349999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"298'13993","reported_seq":14342,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:56.105423+0000","last_change":"2024-09-10T18:34:01.902715+0000","last_active":"2024-09-10T18:34:56.105423+0000","last_peered":"2024-09-10T18:34:56.105423+0000","last_clean":"2024-09-10T18:34:56.105423+0000","last_became_active":"2024-09-10T18:32:03.554958+0000","last_became_peered":"2024-09-10T18:32:03.554958+0000","last_unstale":"2024-09-10T18:34:56.105423+0000","last_undegraded":"2024-09-10T18:34:56.105423+0000","last_fullsized":"2024-09-10T18:34:56.105423+0000","mapping_epoch":246,"log_start":"274'8938","ondisk_log_start":"274'8938","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'10034","last_scrub_stamp":"2024-09-10T18:34:01.902618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:01.902618+0000","objects_scrubbed":6997,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:18.230790+0000","scrub_duration":2.3941321589999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193265664,"num_objects":2949,"num_object_clones":0,"num_object_copies":8847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13993,"num_write_kb":542144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":224,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836147+0000","last_change":"2024-09-10T18:34:00.166073+0000","last_active":"2024-09-10T18:34:51.836147+0000","last_peered":"2024-09-10T18:34:51.836147+0000","last_clean":"2024-09-10T18:34:51.836147+0000","last_became_active":"2024-09-10T18:34:00.165384+0000","last_became_peered":"2024-09-10T18:34:00.165384+0000","last_unstale":"2024-09-10T18:34:51.836147+0000","last_undegraded":"2024-09-10T18:34:51.836147+0000","last_fullsized":"2024-09-10T18:34:51.836147+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:48.466654+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.466654+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":332,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829115+0000","last_change":"2024-09-10T18:32:53.270762+0000","last_active":"2024-09-10T18:34:51.829115+0000","last_peered":"2024-09-10T18:34:51.829115+0000","last_clean":"2024-09-10T18:34:51.829115+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:34:51.829115+0000","last_undegraded":"2024-09-10T18:34:51.829115+0000","last_fullsized":"2024-09-10T18:34:51.829115+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-09-10T18:32:53.270667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:53.270667+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01096307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":308,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836856+0000","last_change":"2024-09-10T18:32:59.245911+0000","last_active":"2024-09-10T18:34:51.836856+0000","last_peered":"2024-09-10T18:34:51.836856+0000","last_clean":"2024-09-10T18:34:51.836856+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:34:51.836856+0000","last_undegraded":"2024-09-10T18:34:51.836856+0000","last_fullsized":"2024-09-10T18:34:51.836856+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:59.245802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:32:59.245802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.028203379000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":1126,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:58.854190+0000","last_change":"2024-09-10T18:34:02.136505+0000","last_active":"2024-09-10T18:34:58.854190+0000","last_peered":"2024-09-10T18:34:58.854190+0000","last_clean":"2024-09-10T18:34:58.854190+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:34:58.854190+0000","last_undegraded":"2024-09-10T18:34:58.854190+0000","last_fullsized":"2024-09-10T18:34:58.854190+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:34:02.136429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:02.136429+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:32.048652+0000","scrub_duration":0.0083841149999999993,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":801,"num_read_kb":796,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":297,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.822233+0000","last_change":"2024-09-10T18:32:56.888589+0000","last_active":"2024-09-10T18:34:51.822233+0000","last_peered":"2024-09-10T18:34:51.822233+0000","last_clean":"2024-09-10T18:34:51.822233+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:34:51.822233+0000","last_undegraded":"2024-09-10T18:34:51.822233+0000","last_fullsized":"2024-09-10T18:34:51.822233+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:56.888436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.771415+0000","last_clean_scrub_stamp":"2024-09-10T18:32:56.888436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064579939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":301,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.835533+0000","last_change":"2024-09-10T18:34:12.539195+0000","last_active":"2024-09-10T18:34:51.835533+0000","last_peered":"2024-09-10T18:34:51.835533+0000","last_clean":"2024-09-10T18:34:51.835533+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:34:51.835533+0000","last_undegraded":"2024-09-10T18:34:51.835533+0000","last_fullsized":"2024-09-10T18:34:51.835533+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:12.539124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.539124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:33.017675+0000","scrub_duration":0.0079965020000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"298'13986","reported_seq":14279,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:35:00.919683+0000","last_change":"2024-09-10T18:34:19.206406+0000","last_active":"2024-09-10T18:35:00.919683+0000","last_peered":"2024-09-10T18:35:00.919683+0000","last_clean":"2024-09-10T18:35:00.919683+0000","last_became_active":"2024-09-10T18:32:03.382335+0000","last_became_peered":"2024-09-10T18:32:03.382335+0000","last_unstale":"2024-09-10T18:35:00.919683+0000","last_undegraded":"2024-09-10T18:35:00.919683+0000","last_fullsized":"2024-09-10T18:35:00.919683+0000","mapping_epoch":246,"log_start":"281'10064","ondisk_log_start":"281'10064","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'11048","last_scrub_stamp":"2024-09-10T18:34:19.206308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.206308+0000","objects_scrubbed":5584,"log_size":3922,"log_dups_size":0,"ondisk_log_size":3922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:45.982181+0000","scrub_duration":1.913810319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170786816,"num_objects":2606,"num_object_clones":0,"num_object_copies":7818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13986,"num_write_kb":530944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'26","reported_seq":300,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829684+0000","last_change":"2024-09-10T18:34:19.788942+0000","last_active":"2024-09-10T18:34:51.829684+0000","last_peered":"2024-09-10T18:34:51.829684+0000","last_clean":"2024-09-10T18:34:51.829684+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:34:51.829684+0000","last_undegraded":"2024-09-10T18:34:51.829684+0000","last_fullsized":"2024-09-10T18:34:51.829684+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'26","last_scrub_stamp":"2024-09-10T18:34:19.788853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.788853+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:26.351175+0000","scrub_duration":0.0020676890000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":316,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.831229+0000","last_change":"2024-09-10T18:34:19.296515+0000","last_active":"2024-09-10T18:34:51.831229+0000","last_peered":"2024-09-10T18:34:51.831229+0000","last_clean":"2024-09-10T18:34:51.831229+0000","last_became_active":"2024-09-10T18:26:27.639129+0000","last_became_peered":"2024-09-10T18:26:27.639129+0000","last_unstale":"2024-09-10T18:34:51.831229+0000","last_undegraded":"2024-09-10T18:34:51.831229+0000","last_fullsized":"2024-09-10T18:34:51.831229+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:19.296286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.296286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:35.735259+0000","scrub_duration":0.0049170170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":367,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830354+0000","last_change":"2024-09-10T18:34:00.160610+0000","last_active":"2024-09-10T18:34:51.830354+0000","last_peered":"2024-09-10T18:34:51.830354+0000","last_clean":"2024-09-10T18:34:51.830354+0000","last_became_active":"2024-09-10T18:34:00.159975+0000","last_became_peered":"2024-09-10T18:34:00.159975+0000","last_unstale":"2024-09-10T18:34:51.830354+0000","last_undegraded":"2024-09-10T18:34:51.830354+0000","last_fullsized":"2024-09-10T18:34:51.830354+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:58.780613+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.780613+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002490309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":313,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830717+0000","last_change":"2024-09-10T18:34:12.920724+0000","last_active":"2024-09-10T18:34:51.830717+0000","last_peered":"2024-09-10T18:34:51.830717+0000","last_clean":"2024-09-10T18:34:51.830717+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:34:51.830717+0000","last_undegraded":"2024-09-10T18:34:51.830717+0000","last_fullsized":"2024-09-10T18:34:51.830717+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:12.920651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.920651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:29.441715+0000","scrub_duration":0.0047636090000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":295,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.822153+0000","last_change":"2024-09-10T18:32:49.984765+0000","last_active":"2024-09-10T18:34:51.822153+0000","last_peered":"2024-09-10T18:34:51.822153+0000","last_clean":"2024-09-10T18:34:51.822153+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:34:51.822153+0000","last_undegraded":"2024-09-10T18:34:51.822153+0000","last_fullsized":"2024-09-10T18:34:51.822153+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:49.984707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:32:49.984707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035569920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"76'81","reported_seq":243,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836285+0000","last_change":"2024-09-10T18:34:32.927768+0000","last_active":"2024-09-10T18:34:51.836285+0000","last_peered":"2024-09-10T18:34:51.836285+0000","last_clean":"2024-09-10T18:34:51.836285+0000","last_became_active":"2024-09-10T18:34:32.927460+0000","last_became_peered":"2024-09-10T18:34:32.927460+0000","last_unstale":"2024-09-10T18:34:51.836285+0000","last_undegraded":"2024-09-10T18:34:51.836285+0000","last_fullsized":"2024-09-10T18:34:51.836285+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:48.466654+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.466654+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"298'14260","reported_seq":14548,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:59.990622+0000","last_change":"2024-09-10T18:34:22.669084+0000","last_active":"2024-09-10T18:34:59.990622+0000","last_peered":"2024-09-10T18:34:59.990622+0000","last_clean":"2024-09-10T18:34:59.990622+0000","last_became_active":"2024-09-10T18:32:03.384247+0000","last_became_peered":"2024-09-10T18:32:03.384247+0000","last_unstale":"2024-09-10T18:34:59.990622+0000","last_undegraded":"2024-09-10T18:34:59.990622+0000","last_fullsized":"2024-09-10T18:34:59.990622+0000","mapping_epoch":246,"log_start":"282'10288","ondisk_log_start":"282'10288","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'11469","last_scrub_stamp":"2024-09-10T18:34:22.669027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.669027+0000","objects_scrubbed":5424,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:26.515481+0000","scrub_duration":1.750744868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168951808,"num_objects":2578,"num_object_clones":0,"num_object_copies":7734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14260,"num_write_kb":538816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"283'24","reported_seq":351,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.837046+0000","last_change":"2024-09-10T18:34:19.404796+0000","last_active":"2024-09-10T18:34:51.837046+0000","last_peered":"2024-09-10T18:34:51.837046+0000","last_clean":"2024-09-10T18:34:51.837046+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:34:51.837046+0000","last_undegraded":"2024-09-10T18:34:51.837046+0000","last_fullsized":"2024-09-10T18:34:51.837046+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'24","last_scrub_stamp":"2024-09-10T18:34:19.404713+0000","last_deep_scrub":"283'24","last_deep_scrub_stamp":"2024-09-10T18:34:19.404713+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.404713+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:30.068249+0000","scrub_duration":0.037546765000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":307,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.835882+0000","last_change":"2024-09-10T18:34:20.349520+0000","last_active":"2024-09-10T18:34:51.835882+0000","last_peered":"2024-09-10T18:34:51.835882+0000","last_clean":"2024-09-10T18:34:51.835882+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:34:51.835882+0000","last_undegraded":"2024-09-10T18:34:51.835882+0000","last_fullsized":"2024-09-10T18:34:51.835882+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:20.349464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:20.349464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:31.859829+0000","scrub_duration":0.0036650300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":311,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.821994+0000","last_change":"2024-09-10T18:33:06.960402+0000","last_active":"2024-09-10T18:34:51.821994+0000","last_peered":"2024-09-10T18:34:51.821994+0000","last_clean":"2024-09-10T18:34:51.821994+0000","last_became_active":"2024-09-10T18:26:27.645885+0000","last_became_peered":"2024-09-10T18:26:27.645885+0000","last_unstale":"2024-09-10T18:34:51.821994+0000","last_undegraded":"2024-09-10T18:34:51.821994+0000","last_fullsized":"2024-09-10T18:34:51.821994+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:06.960320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:33:06.960320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020571560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":310,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.833366+0000","last_change":"2024-09-10T18:34:01.181010+0000","last_active":"2024-09-10T18:34:51.833366+0000","last_peered":"2024-09-10T18:34:51.833366+0000","last_clean":"2024-09-10T18:34:51.833366+0000","last_became_active":"2024-09-10T18:26:27.632071+0000","last_became_peered":"2024-09-10T18:26:27.632071+0000","last_unstale":"2024-09-10T18:34:51.833366+0000","last_undegraded":"2024-09-10T18:34:51.833366+0000","last_fullsized":"2024-09-10T18:34:51.833366+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:01.180920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:01.180920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:10.276096+0000","scrub_duration":0.0082996130000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"298'14351","reported_seq":14810,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:35:00.475830+0000","last_change":"2024-09-10T18:34:07.700772+0000","last_active":"2024-09-10T18:35:00.475830+0000","last_peered":"2024-09-10T18:35:00.475830+0000","last_clean":"2024-09-10T18:35:00.475830+0000","last_became_active":"2024-09-10T18:32:03.554412+0000","last_became_peered":"2024-09-10T18:32:03.554412+0000","last_unstale":"2024-09-10T18:35:00.475830+0000","last_undegraded":"2024-09-10T18:35:00.475830+0000","last_fullsized":"2024-09-10T18:35:00.475830+0000","mapping_epoch":246,"log_start":"281'10201","ondisk_log_start":"281'10201","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10569","last_scrub_stamp":"2024-09-10T18:34:07.700574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:07.700574+0000","objects_scrubbed":6543,"log_size":4150,"log_dups_size":0,"ondisk_log_size":4150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:32.220981+0000","scrub_duration":3.13969619,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171245568,"num_objects":2613,"num_object_clones":0,"num_object_copies":7839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14351,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":226,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.819656+0000","last_change":"2024-09-10T18:34:32.824505+0000","last_active":"2024-09-10T18:34:51.819656+0000","last_peered":"2024-09-10T18:34:51.819656+0000","last_clean":"2024-09-10T18:34:51.819656+0000","last_became_active":"2024-09-10T18:34:32.823976+0000","last_became_peered":"2024-09-10T18:34:32.823976+0000","last_unstale":"2024-09-10T18:34:51.819656+0000","last_undegraded":"2024-09-10T18:34:51.819656+0000","last_fullsized":"2024-09-10T18:34:51.819656+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:08.009806+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.009806+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002902614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":354,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836785+0000","last_change":"2024-09-10T18:34:00.542831+0000","last_active":"2024-09-10T18:34:51.836785+0000","last_peered":"2024-09-10T18:34:51.836785+0000","last_clean":"2024-09-10T18:34:51.836785+0000","last_became_active":"2024-09-10T18:21:33.611203+0000","last_became_peered":"2024-09-10T18:21:33.611203+0000","last_unstale":"2024-09-10T18:34:51.836785+0000","last_undegraded":"2024-09-10T18:34:51.836785+0000","last_fullsized":"2024-09-10T18:34:51.836785+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-09-10T18:34:00.542737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:00.542737+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:11.466009+0000","scrub_duration":0.0030351190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":271,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829246+0000","last_change":"2024-09-10T18:34:09.980779+0000","last_active":"2024-09-10T18:34:51.829246+0000","last_peered":"2024-09-10T18:34:51.829246+0000","last_clean":"2024-09-10T18:34:51.829246+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:34:51.829246+0000","last_undegraded":"2024-09-10T18:34:51.829246+0000","last_fullsized":"2024-09-10T18:34:51.829246+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:34:09.980722+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:34:09.980722+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:29.815756+0000","scrub_duration":0.001388053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":313,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:53.794805+0000","last_change":"2024-09-10T18:34:53.794746+0000","last_active":"2024-09-10T18:34:53.794805+0000","last_peered":"2024-09-10T18:34:53.794805+0000","last_clean":"2024-09-10T18:34:53.794805+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:34:53.794805+0000","last_undegraded":"2024-09-10T18:34:53.794805+0000","last_fullsized":"2024-09-10T18:34:53.794805+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:53.794668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:53.794668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:54.333239+0000","scrub_duration":0.0026034389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":331,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.828603+0000","last_change":"2024-09-10T18:34:20.486108+0000","last_active":"2024-09-10T18:34:51.828603+0000","last_peered":"2024-09-10T18:34:51.828603+0000","last_clean":"2024-09-10T18:34:51.828603+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:34:51.828603+0000","last_undegraded":"2024-09-10T18:34:51.828603+0000","last_fullsized":"2024-09-10T18:34:51.828603+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:20.486034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:20.486034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:45.434346+0000","scrub_duration":0.0026235970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"298'14377","reported_seq":14694,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:35:00.957063+0000","last_change":"2024-09-10T18:34:12.213950+0000","last_active":"2024-09-10T18:35:00.957063+0000","last_peered":"2024-09-10T18:35:00.957063+0000","last_clean":"2024-09-10T18:35:00.957063+0000","last_became_active":"2024-09-10T18:32:03.554475+0000","last_became_peered":"2024-09-10T18:32:03.554475+0000","last_unstale":"2024-09-10T18:35:00.957063+0000","last_undegraded":"2024-09-10T18:35:00.957063+0000","last_fullsized":"2024-09-10T18:35:00.957063+0000","mapping_epoch":246,"log_start":"282'10463","ondisk_log_start":"282'10463","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'10828","last_scrub_stamp":"2024-09-10T18:34:12.213852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.213852+0000","objects_scrubbed":6169,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:28.363825+0000","scrub_duration":3.0156890789999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166264832,"num_objects":2537,"num_object_clones":0,"num_object_copies":7611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14377,"num_write_kb":541248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":365,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.831381+0000","last_change":"2024-09-10T18:34:32.827812+0000","last_active":"2024-09-10T18:34:51.831381+0000","last_peered":"2024-09-10T18:34:51.831381+0000","last_clean":"2024-09-10T18:34:51.831381+0000","last_became_active":"2024-09-10T18:34:32.827174+0000","last_became_peered":"2024-09-10T18:34:32.827174+0000","last_unstale":"2024-09-10T18:34:51.831381+0000","last_undegraded":"2024-09-10T18:34:51.831381+0000","last_fullsized":"2024-09-10T18:34:51.831381+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:06.053455+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.053455+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007740935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"240'25","reported_seq":65,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830299+0000","last_change":"2024-09-10T18:32:51.151265+0000","last_active":"2024-09-10T18:34:51.830299+0000","last_peered":"2024-09-10T18:34:51.830299+0000","last_clean":"2024-09-10T18:34:51.830299+0000","last_became_active":"2024-09-10T18:32:51.118251+0000","last_became_peered":"2024-09-10T18:32:51.118251+0000","last_unstale":"2024-09-10T18:34:51.830299+0000","last_undegraded":"2024-09-10T18:34:51.830299+0000","last_fullsized":"2024-09-10T18:34:51.830299+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:32:39.544846+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.544846+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033922850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":299,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830190+0000","last_change":"2024-09-10T18:33:13.324775+0000","last_active":"2024-09-10T18:34:51.830190+0000","last_peered":"2024-09-10T18:34:51.830190+0000","last_clean":"2024-09-10T18:34:51.830190+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:34:51.830190+0000","last_undegraded":"2024-09-10T18:34:51.830190+0000","last_fullsized":"2024-09-10T18:34:51.830190+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:13.324556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:13.324556+0000","last_clean_scrub_stamp":"2024-09-10T18:33:13.324556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011863357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"189'9","reported_seq":504,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:54.920636+0000","last_change":"2024-09-10T18:34:17.470904+0000","last_active":"2024-09-10T18:34:54.920636+0000","last_peered":"2024-09-10T18:34:54.920636+0000","last_clean":"2024-09-10T18:34:54.920636+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:34:54.920636+0000","last_undegraded":"2024-09-10T18:34:54.920636+0000","last_fullsized":"2024-09-10T18:34:54.920636+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'9","last_scrub_stamp":"2024-09-10T18:34:17.470793+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:34:17.470793+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:22.477102+0000","scrub_duration":0.0044525839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":294,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.837004+0000","last_change":"2024-09-10T18:32:46.630172+0000","last_active":"2024-09-10T18:34:51.837004+0000","last_peered":"2024-09-10T18:34:51.837004+0000","last_clean":"2024-09-10T18:34:51.837004+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:34:51.837004+0000","last_undegraded":"2024-09-10T18:34:51.837004+0000","last_fullsized":"2024-09-10T18:34:51.837004+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:46.630078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_clean_scrub_stamp":"2024-09-10T18:32:46.630078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024328319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"76'81","reported_seq":37,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836670+0000","last_change":"2024-09-10T18:34:32.818578+0000","last_active":"2024-09-10T18:34:51.836670+0000","last_peered":"2024-09-10T18:34:51.836670+0000","last_clean":"2024-09-10T18:34:51.836670+0000","last_became_active":"2024-09-10T18:34:32.818251+0000","last_became_peered":"2024-09-10T18:34:32.818251+0000","last_unstale":"2024-09-10T18:34:51.836670+0000","last_undegraded":"2024-09-10T18:34:51.836670+0000","last_fullsized":"2024-09-10T18:34:51.836670+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:48.006695+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:48.006695+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":5,"num_write":2,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"298'19576","reported_seq":19859,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:35:00.017839+0000","last_change":"2024-09-10T18:34:37.737676+0000","last_active":"2024-09-10T18:35:00.017839+0000","last_peered":"2024-09-10T18:35:00.017839+0000","last_clean":"2024-09-10T18:35:00.017839+0000","last_became_active":"2024-09-10T18:32:03.384417+0000","last_became_peered":"2024-09-10T18:32:03.384417+0000","last_unstale":"2024-09-10T18:35:00.017839+0000","last_undegraded":"2024-09-10T18:35:00.017839+0000","last_fullsized":"2024-09-10T18:35:00.017839+0000","mapping_epoch":246,"log_start":"285'15204","ondisk_log_start":"285'15204","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'17735","last_scrub_stamp":"2024-09-10T18:34:37.737496+0000","last_deep_scrub":"291'17735","last_deep_scrub_stamp":"2024-09-10T18:34:37.737496+0000","last_clean_scrub_stamp":"2024-09-10T18:34:37.737496+0000","objects_scrubbed":5250,"log_size":4372,"log_dups_size":0,"ondisk_log_size":4372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":26,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:45.542594+0000","scrub_duration":25.514376305999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178388992,"num_objects":2722,"num_object_clones":0,"num_object_copies":8166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14326,"num_write_kb":545536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"240'15","reported_seq":321,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.834429+0000","last_change":"2024-09-10T18:33:16.286611+0000","last_active":"2024-09-10T18:34:51.834429+0000","last_peered":"2024-09-10T18:34:51.834429+0000","last_clean":"2024-09-10T18:34:51.834429+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:34:51.834429+0000","last_undegraded":"2024-09-10T18:34:51.834429+0000","last_fullsized":"2024-09-10T18:34:51.834429+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:33:16.286523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:16.286523+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069638809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"265'2","reported_seq":325,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.820373+0000","last_change":"2024-09-10T18:33:11.047358+0000","last_active":"2024-09-10T18:34:51.820373+0000","last_peered":"2024-09-10T18:34:51.820373+0000","last_clean":"2024-09-10T18:34:51.820373+0000","last_became_active":"2024-09-10T18:26:27.644997+0000","last_became_peered":"2024-09-10T18:26:27.644997+0000","last_unstale":"2024-09-10T18:34:51.820373+0000","last_undegraded":"2024-09-10T18:34:51.820373+0000","last_fullsized":"2024-09-10T18:34:51.820373+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'2","last_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_deep_scrub":"265'2","last_deep_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.047305+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021095425000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":329,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:58.759276+0000","last_change":"2024-09-10T18:34:58.759217+0000","last_active":"2024-09-10T18:34:58.759276+0000","last_peered":"2024-09-10T18:34:58.759276+0000","last_clean":"2024-09-10T18:34:58.759276+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:34:58.759276+0000","last_undegraded":"2024-09-10T18:34:58.759276+0000","last_fullsized":"2024-09-10T18:34:58.759276+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:58.759108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:58.759108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:26.795742+0000","scrub_duration":0.010436621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":292,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836740+0000","last_change":"2024-09-10T18:32:44.239399+0000","last_active":"2024-09-10T18:34:51.836740+0000","last_peered":"2024-09-10T18:34:51.836740+0000","last_clean":"2024-09-10T18:34:51.836740+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:34:51.836740+0000","last_undegraded":"2024-09-10T18:34:51.836740+0000","last_fullsized":"2024-09-10T18:34:51.836740+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:44.239347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:32:44.239347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011983390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'81","reported_seq":292,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829883+0000","last_change":"2024-09-10T18:34:15.938135+0000","last_active":"2024-09-10T18:34:51.829883+0000","last_peered":"2024-09-10T18:34:51.829883+0000","last_clean":"2024-09-10T18:34:51.829883+0000","last_became_active":"2024-09-10T18:34:02.803861+0000","last_became_peered":"2024-09-10T18:34:02.803861+0000","last_unstale":"2024-09-10T18:34:51.829883+0000","last_undegraded":"2024-09-10T18:34:51.829883+0000","last_fullsized":"2024-09-10T18:34:51.829883+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:15.938079+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:34:15.938079+0000","last_clean_scrub_stamp":"2024-09-10T18:34:15.938079+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:40.264047+0000","scrub_duration":0.0015442819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"298'14044","reported_seq":14337,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:35:00.049626+0000","last_change":"2024-09-10T18:34:19.945087+0000","last_active":"2024-09-10T18:35:00.049626+0000","last_peered":"2024-09-10T18:35:00.049626+0000","last_clean":"2024-09-10T18:35:00.049626+0000","last_became_active":"2024-09-10T18:32:03.382430+0000","last_became_peered":"2024-09-10T18:32:03.382430+0000","last_unstale":"2024-09-10T18:35:00.049626+0000","last_undegraded":"2024-09-10T18:35:00.049626+0000","last_fullsized":"2024-09-10T18:35:00.049626+0000","mapping_epoch":246,"log_start":"277'9713","ondisk_log_start":"277'9713","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'11233","last_scrub_stamp":"2024-09-10T18:34:19.945029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.945029+0000","objects_scrubbed":5568,"log_size":4331,"log_dups_size":0,"ondisk_log_size":4331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:38.722375+0000","scrub_duration":1.7273602640000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177340416,"num_objects":2706,"num_object_clones":0,"num_object_copies":8118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14044,"num_write_kb":536000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"240'44","reported_seq":388,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829128+0000","last_change":"2024-09-10T18:34:03.021478+0000","last_active":"2024-09-10T18:34:51.829128+0000","last_peered":"2024-09-10T18:34:51.829128+0000","last_clean":"2024-09-10T18:34:51.829128+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:34:51.829128+0000","last_undegraded":"2024-09-10T18:34:51.829128+0000","last_fullsized":"2024-09-10T18:34:51.829128+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'44","last_scrub_stamp":"2024-09-10T18:34:03.021369+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:34:03.021369+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:14.897595+0000","scrub_duration":0.0045216129999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":324,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.831079+0000","last_change":"2024-09-10T18:34:11.965336+0000","last_active":"2024-09-10T18:34:51.831079+0000","last_peered":"2024-09-10T18:34:51.831079+0000","last_clean":"2024-09-10T18:34:51.831079+0000","last_became_active":"2024-09-10T18:26:27.638408+0000","last_became_peered":"2024-09-10T18:26:27.638408+0000","last_unstale":"2024-09-10T18:34:51.831079+0000","last_undegraded":"2024-09-10T18:34:51.831079+0000","last_fullsized":"2024-09-10T18:34:51.831079+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:11.965263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:11.965263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:21.204938+0000","scrub_duration":0.0034110310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":333,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.831286+0000","last_change":"2024-09-10T18:34:21.447378+0000","last_active":"2024-09-10T18:34:51.831286+0000","last_peered":"2024-09-10T18:34:51.831286+0000","last_clean":"2024-09-10T18:34:51.831286+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:34:51.831286+0000","last_undegraded":"2024-09-10T18:34:51.831286+0000","last_fullsized":"2024-09-10T18:34:51.831286+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:21.447291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:34:21.447291+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.447291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:25.350687+0000","scrub_duration":0.0066377579999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":304,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830273+0000","last_change":"2024-09-10T18:31:58.460588+0000","last_active":"2024-09-10T18:34:51.830273+0000","last_peered":"2024-09-10T18:34:51.830273+0000","last_clean":"2024-09-10T18:34:51.830273+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:34:51.830273+0000","last_undegraded":"2024-09-10T18:34:51.830273+0000","last_fullsized":"2024-09-10T18:34:51.830273+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:31:58.460533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:31:58.460533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001375914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"76'81","reported_seq":240,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.833453+0000","last_change":"2024-09-10T18:34:32.957284+0000","last_active":"2024-09-10T18:34:51.833453+0000","last_peered":"2024-09-10T18:34:51.833453+0000","last_clean":"2024-09-10T18:34:51.833453+0000","last_became_active":"2024-09-10T18:34:32.956761+0000","last_became_peered":"2024-09-10T18:34:32.956761+0000","last_unstale":"2024-09-10T18:34:51.833453+0000","last_undegraded":"2024-09-10T18:34:51.833453+0000","last_fullsized":"2024-09-10T18:34:51.833453+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.370561+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.370561+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029251390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54613,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"298'14173","reported_seq":14518,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:35:00.472263+0000","last_change":"2024-09-10T18:34:01.964247+0000","last_active":"2024-09-10T18:35:00.472263+0000","last_peered":"2024-09-10T18:35:00.472263+0000","last_clean":"2024-09-10T18:35:00.472263+0000","last_became_active":"2024-09-10T18:32:03.554423+0000","last_became_peered":"2024-09-10T18:32:03.554423+0000","last_unstale":"2024-09-10T18:35:00.472263+0000","last_undegraded":"2024-09-10T18:35:00.472263+0000","last_fullsized":"2024-09-10T18:35:00.472263+0000","mapping_epoch":246,"log_start":"280'10062","ondisk_log_start":"280'10062","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'10000","last_scrub_stamp":"2024-09-10T18:34:01.964158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:01.964158+0000","objects_scrubbed":6911,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:10.075365+0000","scrub_duration":3.4154075100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170852352,"num_objects":2607,"num_object_clones":0,"num_object_copies":7821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14173,"num_write_kb":536960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"240'18","reported_seq":340,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.835597+0000","last_change":"2024-09-10T18:32:54.528296+0000","last_active":"2024-09-10T18:34:51.835597+0000","last_peered":"2024-09-10T18:34:51.835597+0000","last_clean":"2024-09-10T18:34:51.835597+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:34:51.835597+0000","last_undegraded":"2024-09-10T18:34:51.835597+0000","last_fullsized":"2024-09-10T18:34:51.835597+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-09-10T18:32:54.528209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:54.528209+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056889829999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":317,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.831564+0000","last_change":"2024-09-10T18:33:14.879881+0000","last_active":"2024-09-10T18:34:51.831564+0000","last_peered":"2024-09-10T18:34:51.831564+0000","last_clean":"2024-09-10T18:34:51.831564+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:34:51.831564+0000","last_undegraded":"2024-09-10T18:34:51.831564+0000","last_fullsized":"2024-09-10T18:34:51.831564+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:14.879787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:33:14.879787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044267550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":333,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:57.751460+0000","last_change":"2024-09-10T18:34:57.751011+0000","last_active":"2024-09-10T18:34:57.751460+0000","last_peered":"2024-09-10T18:34:57.751460+0000","last_clean":"2024-09-10T18:34:57.751460+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:34:57.751460+0000","last_undegraded":"2024-09-10T18:34:57.751460+0000","last_fullsized":"2024-09-10T18:34:57.751460+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:57.750908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:57.750908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:03.127730+0000","scrub_duration":0.015397398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":293,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829509+0000","last_change":"2024-09-10T18:32:58.358775+0000","last_active":"2024-09-10T18:34:51.829509+0000","last_peered":"2024-09-10T18:34:51.829509+0000","last_clean":"2024-09-10T18:34:51.829509+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:34:51.829509+0000","last_undegraded":"2024-09-10T18:34:51.829509+0000","last_fullsized":"2024-09-10T18:34:51.829509+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:58.358694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.803236+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.358694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011821898000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"76'81","reported_seq":37,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830589+0000","last_change":"2024-09-10T18:34:32.822681+0000","last_active":"2024-09-10T18:34:51.830589+0000","last_peered":"2024-09-10T18:34:51.830589+0000","last_clean":"2024-09-10T18:34:51.830589+0000","last_became_active":"2024-09-10T18:34:32.822196+0000","last_became_peered":"2024-09-10T18:34:32.822196+0000","last_unstale":"2024-09-10T18:34:51.830589+0000","last_undegraded":"2024-09-10T18:34:51.830589+0000","last_fullsized":"2024-09-10T18:34:51.830589+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:58.561425+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.561425+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":5,"num_write":2,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"298'13839","reported_seq":14151,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:56.120412+0000","last_change":"2024-09-10T18:34:12.987347+0000","last_active":"2024-09-10T18:34:56.120412+0000","last_peered":"2024-09-10T18:34:56.120412+0000","last_clean":"2024-09-10T18:34:56.120412+0000","last_became_active":"2024-09-10T18:32:03.382890+0000","last_became_peered":"2024-09-10T18:32:03.382890+0000","last_unstale":"2024-09-10T18:34:56.120412+0000","last_undegraded":"2024-09-10T18:34:56.120412+0000","last_fullsized":"2024-09-10T18:34:56.120412+0000","mapping_epoch":246,"log_start":"274'8747","ondisk_log_start":"274'8747","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'10774","last_scrub_stamp":"2024-09-10T18:34:12.987248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.987248+0000","objects_scrubbed":6064,"log_size":5092,"log_dups_size":0,"ondisk_log_size":5092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:26.071275+0000","scrub_duration":2.4958883709999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192086016,"num_objects":2931,"num_object_clones":0,"num_object_copies":8793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13839,"num_write_kb":536640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":356,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.819854+0000","last_change":"2024-09-10T18:34:06.163124+0000","last_active":"2024-09-10T18:34:51.819854+0000","last_peered":"2024-09-10T18:34:51.819854+0000","last_clean":"2024-09-10T18:34:51.819854+0000","last_became_active":"2024-09-10T18:26:27.645932+0000","last_became_peered":"2024-09-10T18:26:27.645932+0000","last_unstale":"2024-09-10T18:34:51.819854+0000","last_undegraded":"2024-09-10T18:34:51.819854+0000","last_fullsized":"2024-09-10T18:34:51.819854+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:34:06.163072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.163072+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:32.837570+0000","scrub_duration":0.0016579489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":317,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830562+0000","last_change":"2024-09-10T18:33:05.784757+0000","last_active":"2024-09-10T18:34:51.830562+0000","last_peered":"2024-09-10T18:34:51.830562+0000","last_clean":"2024-09-10T18:34:51.830562+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:34:51.830562+0000","last_undegraded":"2024-09-10T18:34:51.830562+0000","last_fullsized":"2024-09-10T18:34:51.830562+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:33:05.784670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:33:05.784670+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002289031,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":483,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:54.920982+0000","last_change":"2024-09-10T18:33:15.972246+0000","last_active":"2024-09-10T18:34:54.920982+0000","last_peered":"2024-09-10T18:34:54.920982+0000","last_clean":"2024-09-10T18:34:54.920982+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:34:54.920982+0000","last_undegraded":"2024-09-10T18:34:54.920982+0000","last_fullsized":"2024-09-10T18:34:54.920982+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:33:15.972166+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.972166+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004394907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":299,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.821352+0000","last_change":"2024-09-10T18:33:05.964339+0000","last_active":"2024-09-10T18:34:51.821352+0000","last_peered":"2024-09-10T18:34:51.821352+0000","last_clean":"2024-09-10T18:34:51.821352+0000","last_became_active":"2024-09-10T18:26:27.647511+0000","last_became_peered":"2024-09-10T18:26:27.647511+0000","last_unstale":"2024-09-10T18:34:51.821352+0000","last_undegraded":"2024-09-10T18:34:51.821352+0000","last_fullsized":"2024-09-10T18:34:51.821352+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:05.964034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:05.964034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015507839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'81","reported_seq":27,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.821504+0000","last_change":"2024-09-10T18:34:02.810441+0000","last_active":"2024-09-10T18:34:51.821504+0000","last_peered":"2024-09-10T18:34:51.821504+0000","last_clean":"2024-09-10T18:34:51.821504+0000","last_became_active":"2024-09-10T18:34:02.810126+0000","last_became_peered":"2024-09-10T18:34:02.810126+0000","last_unstale":"2024-09-10T18:34:51.821504+0000","last_undegraded":"2024-09-10T18:34:51.821504+0000","last_fullsized":"2024-09-10T18:34:51.821504+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:57.474295+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:57.474295+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":10,"num_write":6,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":262,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829111+0000","last_change":"2024-09-10T18:34:16.895554+0000","last_active":"2024-09-10T18:34:51.829111+0000","last_peered":"2024-09-10T18:34:51.829111+0000","last_clean":"2024-09-10T18:34:51.829111+0000","last_became_active":"2024-09-10T18:26:27.640922+0000","last_became_peered":"2024-09-10T18:26:27.640922+0000","last_unstale":"2024-09-10T18:34:51.829111+0000","last_undegraded":"2024-09-10T18:34:51.829111+0000","last_fullsized":"2024-09-10T18:34:51.829111+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:16.895472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:34:16.895472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:28.880031+0000","scrub_duration":0.002634066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"240'12","reported_seq":315,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830890+0000","last_change":"2024-09-10T18:32:56.384627+0000","last_active":"2024-09-10T18:34:51.830890+0000","last_peered":"2024-09-10T18:34:51.830890+0000","last_clean":"2024-09-10T18:34:51.830890+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:34:51.830890+0000","last_undegraded":"2024-09-10T18:34:51.830890+0000","last_fullsized":"2024-09-10T18:34:51.830890+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-10T18:32:56.384439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:56.384439+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010401713999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":303,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.822343+0000","last_change":"2024-09-10T18:34:00.199169+0000","last_active":"2024-09-10T18:34:51.822343+0000","last_peered":"2024-09-10T18:34:51.822343+0000","last_clean":"2024-09-10T18:34:51.822343+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:34:51.822343+0000","last_undegraded":"2024-09-10T18:34:51.822343+0000","last_fullsized":"2024-09-10T18:34:51.822343+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:00.199087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:00.199087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:11.766473+0000","scrub_duration":0.016944524999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":308,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830719+0000","last_change":"2024-09-10T18:33:58.161088+0000","last_active":"2024-09-10T18:34:51.830719+0000","last_peered":"2024-09-10T18:34:51.830719+0000","last_clean":"2024-09-10T18:34:51.830719+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:34:51.830719+0000","last_undegraded":"2024-09-10T18:34:51.830719+0000","last_fullsized":"2024-09-10T18:34:51.830719+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:58.161019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:58.161019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:23.203002+0000","scrub_duration":0.0042556759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'81","reported_seq":233,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.835943+0000","last_change":"2024-09-10T18:34:02.816592+0000","last_active":"2024-09-10T18:34:51.835943+0000","last_peered":"2024-09-10T18:34:51.835943+0000","last_clean":"2024-09-10T18:34:51.835943+0000","last_became_active":"2024-09-10T18:34:02.815880+0000","last_became_peered":"2024-09-10T18:34:02.815880+0000","last_unstale":"2024-09-10T18:34:51.835943+0000","last_undegraded":"2024-09-10T18:34:51.835943+0000","last_fullsized":"2024-09-10T18:34:51.835943+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:48.466654+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.466654+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"240'19","reported_seq":328,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829193+0000","last_change":"2024-09-10T18:33:08.284482+0000","last_active":"2024-09-10T18:34:51.829193+0000","last_peered":"2024-09-10T18:34:51.829193+0000","last_clean":"2024-09-10T18:34:51.829193+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:34:51.829193+0000","last_undegraded":"2024-09-10T18:34:51.829193+0000","last_fullsized":"2024-09-10T18:34:51.829193+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:33:08.284394+0000","last_deep_scrub":"189'13","last_deep_scrub_stamp":"2024-09-10T18:28:08.976162+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.284394+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010009881999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":314,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.831521+0000","last_change":"2024-09-10T18:33:16.937511+0000","last_active":"2024-09-10T18:34:51.831521+0000","last_peered":"2024-09-10T18:34:51.831521+0000","last_clean":"2024-09-10T18:34:51.831521+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:34:51.831521+0000","last_undegraded":"2024-09-10T18:34:51.831521+0000","last_fullsized":"2024-09-10T18:34:51.831521+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:16.937390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:33:16.937390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.024770533000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":295,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830569+0000","last_change":"2024-09-10T18:32:39.161911+0000","last_active":"2024-09-10T18:34:51.830569+0000","last_peered":"2024-09-10T18:34:51.830569+0000","last_clean":"2024-09-10T18:34:51.830569+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:34:51.830569+0000","last_undegraded":"2024-09-10T18:34:51.830569+0000","last_fullsized":"2024-09-10T18:34:51.830569+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:39.161653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.161653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0093847749999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":321,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:53.654725+0000","last_change":"2024-09-10T18:34:53.654671+0000","last_active":"2024-09-10T18:34:53.654725+0000","last_peered":"2024-09-10T18:34:53.654725+0000","last_clean":"2024-09-10T18:34:53.654725+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:34:53.654725+0000","last_undegraded":"2024-09-10T18:34:53.654725+0000","last_fullsized":"2024-09-10T18:34:53.654725+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:53.654591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:53.654591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:00.061299+0000","scrub_duration":0.0020849879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'81","reported_seq":27,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.819496+0000","last_change":"2024-09-10T18:34:02.797580+0000","last_active":"2024-09-10T18:34:51.819496+0000","last_peered":"2024-09-10T18:34:51.819496+0000","last_clean":"2024-09-10T18:34:51.819496+0000","last_became_active":"2024-09-10T18:34:02.797304+0000","last_became_peered":"2024-09-10T18:34:02.797304+0000","last_unstale":"2024-09-10T18:34:51.819496+0000","last_undegraded":"2024-09-10T18:34:51.819496+0000","last_fullsized":"2024-09-10T18:34:51.819496+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:58.780613+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.780613+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"240'19","reported_seq":335,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.820452+0000","last_change":"2024-09-10T18:33:15.955719+0000","last_active":"2024-09-10T18:34:51.820452+0000","last_peered":"2024-09-10T18:34:51.820452+0000","last_clean":"2024-09-10T18:34:51.820452+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:34:51.820452+0000","last_undegraded":"2024-09-10T18:34:51.820452+0000","last_fullsized":"2024-09-10T18:34:51.820452+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:33:15.955638+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.955638+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021358801,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":308,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829894+0000","last_change":"2024-09-10T18:33:12.342537+0000","last_active":"2024-09-10T18:34:51.829894+0000","last_peered":"2024-09-10T18:34:51.829894+0000","last_clean":"2024-09-10T18:34:51.829894+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:34:51.829894+0000","last_undegraded":"2024-09-10T18:34:51.829894+0000","last_fullsized":"2024-09-10T18:34:51.829894+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:12.342432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_clean_scrub_stamp":"2024-09-10T18:33:12.342432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043002429999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":465,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:54.922066+0000","last_change":"2024-09-10T18:32:40.183655+0000","last_active":"2024-09-10T18:34:54.922066+0000","last_peered":"2024-09-10T18:34:54.922066+0000","last_clean":"2024-09-10T18:34:54.922066+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:34:54.922066+0000","last_undegraded":"2024-09-10T18:34:54.922066+0000","last_fullsized":"2024-09-10T18:34:54.922066+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:32:40.183571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:32:40.183571+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014890771000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":38,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.831056+0000","last_change":"2024-09-10T18:33:29.142401+0000","last_active":"2024-09-10T18:34:51.831056+0000","last_peered":"2024-09-10T18:34:51.831056+0000","last_clean":"2024-09-10T18:34:51.831056+0000","last_became_active":"2024-09-10T18:33:29.141936+0000","last_became_peered":"2024-09-10T18:33:29.141936+0000","last_unstale":"2024-09-10T18:34:51.831056+0000","last_undegraded":"2024-09-10T18:34:51.831056+0000","last_fullsized":"2024-09-10T18:34:51.831056+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:11.727786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.727786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"76'81","reported_seq":98,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836430+0000","last_change":"2024-09-10T18:34:22.551031+0000","last_active":"2024-09-10T18:34:51.836430+0000","last_peered":"2024-09-10T18:34:51.836430+0000","last_clean":"2024-09-10T18:34:51.836430+0000","last_became_active":"2024-09-10T18:34:02.810381+0000","last_became_peered":"2024-09-10T18:34:02.810381+0000","last_unstale":"2024-09-10T18:34:51.836430+0000","last_undegraded":"2024-09-10T18:34:51.836430+0000","last_fullsized":"2024-09-10T18:34:51.836430+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:22.550782+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.550782+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:47.692871+0000","scrub_duration":0.0052607840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":337,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.833598+0000","last_change":"2024-09-10T18:33:59.230861+0000","last_active":"2024-09-10T18:34:51.833598+0000","last_peered":"2024-09-10T18:34:51.833598+0000","last_clean":"2024-09-10T18:34:51.833598+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:34:51.833598+0000","last_undegraded":"2024-09-10T18:34:51.833598+0000","last_fullsized":"2024-09-10T18:34:51.833598+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-10T18:33:59.227198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:59.227198+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:04.026375+0000","scrub_duration":0.012577204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":307,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.831337+0000","last_change":"2024-09-10T18:32:39.535817+0000","last_active":"2024-09-10T18:34:51.831337+0000","last_peered":"2024-09-10T18:34:51.831337+0000","last_clean":"2024-09-10T18:34:51.831337+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:34:51.831337+0000","last_undegraded":"2024-09-10T18:34:51.831337+0000","last_fullsized":"2024-09-10T18:34:51.831337+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:39.535749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:32:39.535749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046018179999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":484,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:58.485795+0000","last_change":"2024-09-10T18:34:58.485675+0000","last_active":"2024-09-10T18:34:58.485795+0000","last_peered":"2024-09-10T18:34:58.485795+0000","last_clean":"2024-09-10T18:34:58.485795+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:34:58.485795+0000","last_undegraded":"2024-09-10T18:34:58.485795+0000","last_fullsized":"2024-09-10T18:34:58.485795+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:34:58.485562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:58.485562+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:19.625439+0000","scrub_duration":0.0059980570000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":302,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.828805+0000","last_change":"2024-09-10T18:34:23.556824+0000","last_active":"2024-09-10T18:34:51.828805+0000","last_peered":"2024-09-10T18:34:51.828805+0000","last_clean":"2024-09-10T18:34:51.828805+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:34:51.828805+0000","last_undegraded":"2024-09-10T18:34:51.828805+0000","last_fullsized":"2024-09-10T18:34:51.828805+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:23.556773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:23.556773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:47.130693+0000","scrub_duration":0.001126988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'81","reported_seq":18,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836941+0000","last_change":"2024-09-10T18:34:34.881583+0000","last_active":"2024-09-10T18:34:51.836941+0000","last_peered":"2024-09-10T18:34:51.836941+0000","last_clean":"2024-09-10T18:34:51.836941+0000","last_became_active":"2024-09-10T18:34:34.881186+0000","last_became_peered":"2024-09-10T18:34:34.881186+0000","last_unstale":"2024-09-10T18:34:51.836941+0000","last_undegraded":"2024-09-10T18:34:51.836941+0000","last_fullsized":"2024-09-10T18:34:51.836941+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:05.012053+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:05.012053+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":343,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836081+0000","last_change":"2024-09-10T18:33:15.612721+0000","last_active":"2024-09-10T18:34:51.836081+0000","last_peered":"2024-09-10T18:34:51.836081+0000","last_clean":"2024-09-10T18:34:51.836081+0000","last_became_active":"2024-09-10T18:26:27.649202+0000","last_became_peered":"2024-09-10T18:26:27.649202+0000","last_unstale":"2024-09-10T18:34:51.836081+0000","last_undegraded":"2024-09-10T18:34:51.836081+0000","last_fullsized":"2024-09-10T18:34:51.836081+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:33:15.612638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.612638+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002578828,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":317,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.836400+0000","last_change":"2024-09-10T18:34:04.380240+0000","last_active":"2024-09-10T18:34:51.836400+0000","last_peered":"2024-09-10T18:34:51.836400+0000","last_clean":"2024-09-10T18:34:51.836400+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:34:51.836400+0000","last_undegraded":"2024-09-10T18:34:51.836400+0000","last_fullsized":"2024-09-10T18:34:51.836400+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:04.380160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:34:04.380160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:34.095135+0000","scrub_duration":0.0018137730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":316,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.837129+0000","last_change":"2024-09-10T18:34:03.521909+0000","last_active":"2024-09-10T18:34:51.837129+0000","last_peered":"2024-09-10T18:34:51.837129+0000","last_clean":"2024-09-10T18:34:51.837129+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:34:51.837129+0000","last_undegraded":"2024-09-10T18:34:51.837129+0000","last_fullsized":"2024-09-10T18:34:51.837129+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:03.521806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_clean_scrub_stamp":"2024-09-10T18:34:03.521806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:19.398042+0000","scrub_duration":0.005196946,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":293,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829604+0000","last_change":"2024-09-10T18:34:14.263579+0000","last_active":"2024-09-10T18:34:51.829604+0000","last_peered":"2024-09-10T18:34:51.829604+0000","last_clean":"2024-09-10T18:34:51.829604+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:34:51.829604+0000","last_undegraded":"2024-09-10T18:34:51.829604+0000","last_fullsized":"2024-09-10T18:34:51.829604+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:14.263499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:34:14.263499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:30.466656+0000","scrub_duration":0.0019685509999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'81","reported_seq":18,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.821999+0000","last_change":"2024-09-10T18:34:34.858726+0000","last_active":"2024-09-10T18:34:51.821999+0000","last_peered":"2024-09-10T18:34:51.821999+0000","last_clean":"2024-09-10T18:34:51.821999+0000","last_became_active":"2024-09-10T18:34:34.858291+0000","last_became_peered":"2024-09-10T18:34:34.858291+0000","last_unstale":"2024-09-10T18:34:51.821999+0000","last_undegraded":"2024-09-10T18:34:51.821999+0000","last_fullsized":"2024-09-10T18:34:51.821999+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:08.035431+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:08.035431+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"240'15","reported_seq":347,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.837288+0000","last_change":"2024-09-10T18:34:19.608537+0000","last_active":"2024-09-10T18:34:51.837288+0000","last_peered":"2024-09-10T18:34:51.837288+0000","last_clean":"2024-09-10T18:34:51.837288+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:34:51.837288+0000","last_undegraded":"2024-09-10T18:34:51.837288+0000","last_fullsized":"2024-09-10T18:34:51.837288+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:34:19.608445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.608445+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:24.709727+0000","scrub_duration":0.012630626000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":310,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.831006+0000","last_change":"2024-09-10T18:33:01.781370+0000","last_active":"2024-09-10T18:34:51.831006+0000","last_peered":"2024-09-10T18:34:51.831006+0000","last_clean":"2024-09-10T18:34:51.831006+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:34:51.831006+0000","last_undegraded":"2024-09-10T18:34:51.831006+0000","last_fullsized":"2024-09-10T18:34:51.831006+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:01.781295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.781295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016784879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":298,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:56.701783+0000","last_change":"2024-09-10T18:34:56.701737+0000","last_active":"2024-09-10T18:34:56.701783+0000","last_peered":"2024-09-10T18:34:56.701783+0000","last_clean":"2024-09-10T18:34:56.701783+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:34:56.701783+0000","last_undegraded":"2024-09-10T18:34:56.701783+0000","last_fullsized":"2024-09-10T18:34:56.701783+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:56.701661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:34:56.701661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:25.109620+0000","scrub_duration":0.0064135620000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":302,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.825251+0000","last_change":"2024-09-10T18:34:00.512903+0000","last_active":"2024-09-10T18:34:51.825251+0000","last_peered":"2024-09-10T18:34:51.825251+0000","last_clean":"2024-09-10T18:34:51.825251+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:34:51.825251+0000","last_undegraded":"2024-09-10T18:34:51.825251+0000","last_fullsized":"2024-09-10T18:34:51.825251+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:00.512670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:00.512670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:03.765496+0000","scrub_duration":0.0068783309999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'81","reported_seq":37,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:56.691180+0000","last_change":"2024-09-10T18:34:56.691141+0000","last_active":"2024-09-10T18:34:56.691180+0000","last_peered":"2024-09-10T18:34:56.691180+0000","last_clean":"2024-09-10T18:34:56.691180+0000","last_became_active":"2024-09-10T18:34:02.802580+0000","last_became_peered":"2024-09-10T18:34:02.802580+0000","last_unstale":"2024-09-10T18:34:56.691180+0000","last_undegraded":"2024-09-10T18:34:56.691180+0000","last_fullsized":"2024-09-10T18:34:56.691180+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:56.691043+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:56.691043+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:06.642829+0000","scrub_duration":0.0021075249999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":335,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829892+0000","last_change":"2024-09-10T18:32:46.253109+0000","last_active":"2024-09-10T18:34:51.829892+0000","last_peered":"2024-09-10T18:34:51.829892+0000","last_clean":"2024-09-10T18:34:51.829892+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:34:51.829892+0000","last_undegraded":"2024-09-10T18:34:51.829892+0000","last_fullsized":"2024-09-10T18:34:51.829892+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:32:46.252895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:46.252895+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003188967,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":318,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.822389+0000","last_change":"2024-09-10T18:34:08.134374+0000","last_active":"2024-09-10T18:34:51.822389+0000","last_peered":"2024-09-10T18:34:51.822389+0000","last_clean":"2024-09-10T18:34:51.822389+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:34:51.822389+0000","last_undegraded":"2024-09-10T18:34:51.822389+0000","last_fullsized":"2024-09-10T18:34:51.822389+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:08.134230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:34:08.134230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:31.369105+0000","scrub_duration":0.0077618749999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":313,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-10T18:34:57.776135+0000","last_change":"2024-09-10T18:34:57.776098+0000","last_active":"2024-09-10T18:34:57.776135+0000","last_peered":"2024-09-10T18:34:57.776135+0000","last_clean":"2024-09-10T18:34:57.776135+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:34:57.776135+0000","last_undegraded":"2024-09-10T18:34:57.776135+0000","last_fullsized":"2024-09-10T18:34:57.776135+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:57.776022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:57.776022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:27.386344+0000","scrub_duration":0.002853441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":310,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829866+0000","last_change":"2024-09-10T18:33:18.227516+0000","last_active":"2024-09-10T18:34:51.829866+0000","last_peered":"2024-09-10T18:34:51.829866+0000","last_clean":"2024-09-10T18:34:51.829866+0000","last_became_active":"2024-09-10T18:26:27.636996+0000","last_became_peered":"2024-09-10T18:26:27.636996+0000","last_unstale":"2024-09-10T18:34:51.829866+0000","last_undegraded":"2024-09-10T18:34:51.829866+0000","last_fullsized":"2024-09-10T18:34:51.829866+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:18.227428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:18.227428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063780959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'81","reported_seq":18,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.830514+0000","last_change":"2024-09-10T18:34:34.869108+0000","last_active":"2024-09-10T18:34:51.830514+0000","last_peered":"2024-09-10T18:34:51.830514+0000","last_clean":"2024-09-10T18:34:51.830514+0000","last_became_active":"2024-09-10T18:34:34.868371+0000","last_became_peered":"2024-09-10T18:34:34.868371+0000","last_unstale":"2024-09-10T18:34:51.830514+0000","last_undegraded":"2024-09-10T18:34:51.830514+0000","last_fullsized":"2024-09-10T18:34:51.830514+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:22.344759+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.344759+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":316,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.828985+0000","last_change":"2024-09-10T18:32:47.287083+0000","last_active":"2024-09-10T18:34:51.828985+0000","last_peered":"2024-09-10T18:34:51.828985+0000","last_clean":"2024-09-10T18:34:51.828985+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:34:51.828985+0000","last_undegraded":"2024-09-10T18:34:51.828985+0000","last_fullsized":"2024-09-10T18:34:51.828985+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-09-10T18:32:47.286977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:47.286977+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017807315000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":301,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829680+0000","last_change":"2024-09-10T18:32:57.214331+0000","last_active":"2024-09-10T18:34:51.829680+0000","last_peered":"2024-09-10T18:34:51.829680+0000","last_clean":"2024-09-10T18:34:51.829680+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:34:51.829680+0000","last_undegraded":"2024-09-10T18:34:51.829680+0000","last_fullsized":"2024-09-10T18:34:51.829680+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:57.214242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:32:57.214242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015466415000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":266,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.828955+0000","last_change":"2024-09-10T18:34:18.819184+0000","last_active":"2024-09-10T18:34:51.828955+0000","last_peered":"2024-09-10T18:34:51.828955+0000","last_clean":"2024-09-10T18:34:51.828955+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:34:51.828955+0000","last_undegraded":"2024-09-10T18:34:51.828955+0000","last_fullsized":"2024-09-10T18:34:51.828955+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:18.819090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:18.819090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:35.004598+0000","scrub_duration":0.0015527620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":301,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829952+0000","last_change":"2024-09-10T18:33:13.844984+0000","last_active":"2024-09-10T18:34:51.829952+0000","last_peered":"2024-09-10T18:34:51.829952+0000","last_clean":"2024-09-10T18:34:51.829952+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:34:51.829952+0000","last_undegraded":"2024-09-10T18:34:51.829952+0000","last_fullsized":"2024-09-10T18:34:51.829952+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:13.844781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:13.844781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022954300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'81","reported_seq":18,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.835678+0000","last_change":"2024-09-10T18:34:34.867779+0000","last_active":"2024-09-10T18:34:51.835678+0000","last_peered":"2024-09-10T18:34:51.835678+0000","last_clean":"2024-09-10T18:34:51.835678+0000","last_became_active":"2024-09-10T18:34:34.867380+0000","last_became_peered":"2024-09-10T18:34:34.867380+0000","last_unstale":"2024-09-10T18:34:51.835678+0000","last_undegraded":"2024-09-10T18:34:51.835678+0000","last_fullsized":"2024-09-10T18:34:51.835678+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:56.155354+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:56.155354+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":311,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.835680+0000","last_change":"2024-09-10T18:33:18.177364+0000","last_active":"2024-09-10T18:34:51.835680+0000","last_peered":"2024-09-10T18:34:51.835680+0000","last_clean":"2024-09-10T18:34:51.835680+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:34:51.835680+0000","last_undegraded":"2024-09-10T18:34:51.835680+0000","last_fullsized":"2024-09-10T18:34:51.835680+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_clean_scrub_stamp":"2024-09-10T18:33:18.177264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029244380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":299,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829282+0000","last_change":"2024-09-10T18:34:03.175239+0000","last_active":"2024-09-10T18:34:51.829282+0000","last_peered":"2024-09-10T18:34:51.829282+0000","last_clean":"2024-09-10T18:34:51.829282+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:34:51.829282+0000","last_undegraded":"2024-09-10T18:34:51.829282+0000","last_fullsized":"2024-09-10T18:34:51.829282+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:03.175158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:34:03.175158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:17.890582+0000","scrub_duration":0.0046326559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":317,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.829958+0000","last_change":"2024-09-10T18:33:59.017207+0000","last_active":"2024-09-10T18:34:51.829958+0000","last_peered":"2024-09-10T18:34:51.829958+0000","last_clean":"2024-09-10T18:34:51.829958+0000","last_became_active":"2024-09-10T18:26:27.648112+0000","last_became_peered":"2024-09-10T18:26:27.648112+0000","last_unstale":"2024-09-10T18:34:51.829958+0000","last_undegraded":"2024-09-10T18:34:51.829958+0000","last_fullsized":"2024-09-10T18:34:51.829958+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:33:59.017151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:59.017151+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:07.309204+0000","scrub_duration":0.0027087330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":304,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-10T18:34:51.828956+0000","last_change":"2024-09-10T18:34:02.502004+0000","last_active":"2024-09-10T18:34:51.828956+0000","last_peered":"2024-09-10T18:34:51.828956+0000","last_clean":"2024-09-10T18:34:51.828956+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:34:51.828956+0000","last_undegraded":"2024-09-10T18:34:51.828956+0000","last_fullsized":"2024-09-10T18:34:51.828956+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:02.501929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:34:02.501929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:23.332476+0000","scrub_duration":0.0086866479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2818834456,"num_objects":43013,"num_object_clones":0,"num_object_copies":129039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43013,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226165,"num_write_kb":8613633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8344580096,"data_stored":8344567880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68487,"ondisk_log_size":68487,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":736,"ondisk_log_size":736,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":801,"num_read_kb":796,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":682405,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":76,"num_write":40,"num_write_kb":555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":829053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2511,"ondisk_log_size":2511,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234149,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1849328,"kb_used_data":1539368,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91895824,"statfs":{"total":95995035648,"available":94101323776,"internally_reserved":0,"allocated":1576312832,"data_stored":1575817028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":2.3799999999999999,"15min":1.8320000000000001},"min":{"1min":0.215,"5min":0.21099999999999999,"15min":0.20799999999999999},"max":{"1min":2.3820000000000001,"5min":75.506,"15min":85.012},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.127,"5min":2.2690000000000001,"15min":1.7689999999999999},"min":{"1min":0.32800000000000001,"5min":0.25600000000000001,"15min":0.222},"max":{"1min":3.4889999999999999,"5min":75.710999999999999,"15min":84.665999999999997},"last":0.48399999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:34:46 2024","interfaces":[{"interface":"back","average":{"1min":3.6459999999999999,"5min":1.661,"15min":1.4379999999999999},"min":{"1min":0.27000000000000002,"5min":0.214,"15min":0.214},"max":{"1min":39.728000000000002,"5min":39.728000000000002,"15min":49.317999999999998},"last":0.26200000000000001},{"interface":"front","average":{"1min":3.6459999999999999,"5min":1.5369999999999999,"15min":1.3720000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":39.738999999999997,"5min":39.738999999999997,"15min":49.353000000000002},"last":0.311}]},{"osd":2,"last update":"Tue Sep 10 18:34:46 2024","interfaces":[{"interface":"back","average":{"1min":8.8529999999999998,"5min":2.5190000000000001,"15min":1.353},"min":{"1min":0.29099999999999998,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":136.648,"5min":136.648,"15min":136.648},"last":0.38400000000000001},{"interface":"front","average":{"1min":9.2149999999999999,"5min":2.625,"15min":1.4490000000000001},"min":{"1min":0.32100000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":136.506,"5min":136.506,"15min":136.506},"last":0.29199999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:34:46 2024","interfaces":[{"interface":"back","average":{"1min":3.7519999999999998,"5min":1.712,"15min":1.587},"min":{"1min":0.34100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":38.225000000000001,"5min":38.225000000000001,"15min":86.305999999999997},"last":0.79300000000000004},{"interface":"front","average":{"1min":3.5720000000000001,"5min":1.7070000000000001,"15min":1.6259999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.26700000000000002},"max":{"1min":38.104999999999997,"5min":38.104999999999997,"15min":88.364000000000004},"last":0.65400000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.63300000000000001,"15min":0.50600000000000001},"min":{"1min":0.27100000000000002,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":2.282,"5min":3.4670000000000001,"15min":3.8839999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.67500000000000004,"15min":0.53200000000000003},"min":{"1min":0.28299999999999997,"5min":0.26200000000000001,"15min":0.223},"max":{"1min":2.4449999999999998,"5min":2.4449999999999998,"15min":2.6589999999999998},"last":0.35099999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:34:46 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":0.98999999999999999,"15min":0.69799999999999995},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.23499999999999999},"max":{"1min":2.7509999999999999,"5min":7.1820000000000004,"15min":7.3109999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.93999999999999995,"15min":0.65100000000000002},"min":{"1min":0.379,"5min":0.27100000000000002,"15min":0.219},"max":{"1min":3.3130000000000002,"5min":6.3360000000000003,"15min":6.3360000000000003},"last":0.61099999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90100000000000002,"15min":0.629},"min":{"1min":0.32000000000000001,"5min":0.28899999999999998,"15min":0.27000000000000002},"max":{"1min":2.5699999999999998,"5min":7.2750000000000004,"15min":7.2750000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.81299999999999994,"15min":0.58499999999999996},"min":{"1min":0.373,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.2090000000000001,"5min":4.585,"15min":4.585},"last":0.36699999999999999}]}]},{"osd":6,"up_from":144,"seq":618475290728,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":796248,"kb_used_data":493648,"kb_used_omap":7,"kb_used_meta":302584,"kb_avail":92948904,"statfs":{"total":95995035648,"available":95179677696,"internally_reserved":0,"allocated":505495552,"data_stored":505023928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":309846103},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.1859999999999999,"15min":1.776},"min":{"1min":0.23300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.9940000000000002,"5min":8.3949999999999996,"15min":30.629999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.93999999999999995,"15min":1.3320000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.22900000000000001},"max":{"1min":3.407,"5min":8.4640000000000004,"15min":30.597000000000001},"last":0.5}]},{"osd":1,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.0660000000000001,"15min":0.95799999999999996},"min":{"1min":0.26000000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":3.806,"5min":8.0329999999999995,"15min":30.984000000000002},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.246,"15min":1.143},"min":{"1min":0.28100000000000003,"5min":0.246,"15min":0.24199999999999999},"max":{"1min":2.766,"5min":6.7859999999999996,"15min":30.571000000000002},"last":0.31900000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.93300000000000005,"15min":1.024},"min":{"1min":0.307,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.661,"5min":18.353000000000002,"15min":18.353000000000002},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.90300000000000002,"15min":0.84299999999999997},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.23400000000000001},"max":{"1min":2.2959999999999998,"5min":5.415,"15min":11.164999999999999},"last":0.55500000000000005}]},{"osd":3,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.2310000000000001,"15min":1.8979999999999999},"min":{"1min":0.28799999999999998,"5min":0.28199999999999997,"15min":0.27800000000000002},"max":{"1min":4.0970000000000004,"5min":16.831,"15min":61.703000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.377,"15min":1.9379999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.5270000000000001,"5min":17.786000000000001,"15min":61.814},"last":0.438}]},{"osd":4,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58399999999999996,"15min":0.55100000000000005},"min":{"1min":0.27700000000000002,"5min":0.24199999999999999,"15min":0.23699999999999999},"max":{"1min":2.4889999999999999,"5min":2.8410000000000002,"15min":5.4950000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60999999999999999,"15min":0.63300000000000001},"min":{"1min":0.314,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.8140000000000001,"5min":4.6390000000000002,"15min":4.6390000000000002},"last":0.27300000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.69099999999999995,"15min":0.55300000000000005},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.24399999999999999},"max":{"1min":1.8109999999999999,"5min":4.6619999999999999,"15min":4.6619999999999999},"last":4.2850000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71199999999999997,"15min":0.53500000000000003},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.26800000000000002},"max":{"1min":2.1259999999999999,"5min":2.794,"15min":2.794},"last":0.79500000000000004}]},{"osd":7,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.69999999999999996,"15min":0.56200000000000006},"min":{"1min":0.28999999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.3650000000000002,"5min":3.3650000000000002,"15min":3.3650000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.68300000000000005,"15min":0.58799999999999997},"min":{"1min":0.33500000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.1480000000000001,"5min":3.9300000000000002,"15min":5.5309999999999997},"last":0.41399999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724720,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2162356,"kb_used_data":1568300,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":91582796,"statfs":{"total":95995035648,"available":93780783104,"internally_reserved":0,"allocated":1605939200,"data_stored":1605439806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":1.367,"15min":1.4590000000000001},"min":{"1min":0.249,"5min":0.23499999999999999,"15min":0.21099999999999999},"max":{"1min":2.5259999999999998,"5min":26.288,"15min":52.033999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.218,"15min":1.29},"min":{"1min":0.246,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":3.9820000000000002,"5min":24.178000000000001,"15min":52.167999999999999},"last":1.4279999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.379,"15min":0.93799999999999994},"min":{"1min":0.33900000000000002,"5min":0.315,"15min":0.22600000000000001},"max":{"1min":4.4800000000000004,"5min":24.16,"15min":24.16},"last":9.8729999999999993},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.343,"15min":1.0109999999999999},"min":{"1min":0.30399999999999999,"5min":0.28499999999999998,"15min":0.247},"max":{"1min":2.984,"5min":24.129999999999999,"15min":24.129999999999999},"last":9.8599999999999994}]},{"osd":2,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":1.601,"15min":1.1819999999999999},"min":{"1min":0.28899999999999998,"5min":0.245,"15min":0.24299999999999999},"max":{"1min":1.8819999999999999,"5min":54.530000000000001,"15min":54.530000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.79200000000000004,"5min":1.5700000000000001,"15min":1.1539999999999999},"min":{"1min":0.32600000000000001,"5min":0.247,"15min":0.247},"max":{"1min":2.871,"5min":54.514000000000003,"15min":54.514000000000003},"last":2.6600000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":2.1320000000000001,"5min":1.782,"15min":1.4039999999999999},"min":{"1min":0.29899999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":17.198,"5min":30.475000000000001,"15min":31.207000000000001},"last":1.2709999999999999},{"interface":"front","average":{"1min":2.472,"5min":1.772,"15min":1.3109999999999999},"min":{"1min":0.35899999999999999,"5min":0.32000000000000001,"15min":0.29699999999999999},"max":{"1min":16.934999999999999,"5min":30.172000000000001,"15min":31.199000000000002},"last":0.88}]},{"osd":4,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66700000000000004,"15min":0.496},"min":{"1min":0.27900000000000003,"5min":0.22900000000000001,"15min":0.20799999999999999},"max":{"1min":2.3580000000000001,"5min":4.5199999999999996,"15min":4.5199999999999996},"last":1.3700000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.73599999999999999,"15min":0.55200000000000005},"min":{"1min":0.315,"5min":0.24199999999999999,"15min":0.22700000000000001},"max":{"1min":2.4700000000000002,"5min":6.798,"15min":6.798},"last":0.43099999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.65200000000000002,"15min":0.64500000000000002},"min":{"1min":0.34699999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.1819999999999999,"5min":2.7469999999999999,"15min":4.7859999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.72899999999999998,"15min":0.65200000000000002},"min":{"1min":0.28100000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.2850000000000001,"5min":5.3680000000000003,"15min":5.3860000000000001},"last":1.5800000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:34:34 2024","interfaces":[{"interface":"back","average":{"1min":2.3029999999999999,"5min":1.0640000000000001,"15min":0.70499999999999996},"min":{"1min":0.35699999999999998,"5min":0.25900000000000001,"15min":0.22500000000000001},"max":{"1min":28.934000000000001,"5min":28.934000000000001,"15min":28.934000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.2030000000000001,"5min":0.82299999999999995,"15min":0.59499999999999997},"min":{"1min":0.38100000000000001,"5min":0.25900000000000001,"15min":0.24299999999999999},"max":{"1min":5.2190000000000003,"5min":5.2190000000000003,"15min":5.2190000000000003},"last":1.403}]}]},{"osd":4,"up_from":33,"seq":141733920948,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":826344,"kb_used_data":524704,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92918808,"statfs":{"total":95995035648,"available":95148859392,"internally_reserved":0,"allocated":537296896,"data_stored":536808527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.95499999999999996,"15min":1.024},"min":{"1min":0.248,"5min":0.21199999999999999,"15min":0.188},"max":{"1min":2.5960000000000001,"5min":4.3730000000000002,"15min":31.585000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.92900000000000005,"15min":1.006},"min":{"1min":0.28799999999999998,"5min":0.189,"15min":0.189},"max":{"1min":4.8929999999999998,"5min":4.8929999999999998,"15min":29.802},"last":1.5329999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:34:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.5069999999999999,"15min":1.0069999999999999},"min":{"1min":0.254,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":4.3390000000000004,"5min":50.045000000000002,"15min":50.045000000000002},"last":1.091},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.7869999999999999,"15min":1.095},"min":{"1min":0.30299999999999999,"5min":0.27100000000000002,"15min":0.22900000000000001},"max":{"1min":5.5590000000000002,"5min":50.283000000000001,"15min":50.283000000000001},"last":0.54400000000000004}]},{"osd":2,"last update":"Tue Sep 10 18:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.88300000000000001,"15min":0.86899999999999999},"min":{"1min":0.28499999999999998,"5min":0.249,"15min":0.249},"max":{"1min":1.857,"5min":5.2380000000000004,"15min":13.898999999999999},"last":0.372},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.876,"15min":0.89100000000000001},"min":{"1min":0.31900000000000001,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":2.9790000000000001,"5min":4.0460000000000003,"15min":14.51},"last":1.4159999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:34:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.6719999999999999,"15min":1.335},"min":{"1min":0.28899999999999998,"5min":0.26300000000000001,"15min":0.249},"max":{"1min":7.0330000000000004,"5min":49.990000000000002,"15min":49.990000000000002},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.383,"5min":1.726,"15min":1.385},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.255},"max":{"1min":5.0439999999999996,"5min":50.764000000000003,"15min":50.764000000000003},"last":0.38700000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:34:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.93799999999999994,"15min":0.61099999999999999},"min":{"1min":0.27600000000000002,"5min":0.223,"15min":0.20499999999999999},"max":{"1min":1.1970000000000001,"5min":22.861000000000001,"15min":22.861000000000001},"last":1.1970000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.93799999999999994,"15min":0.64000000000000001},"min":{"1min":0.27300000000000002,"5min":0.22,"15min":0.22},"max":{"1min":2.387,"5min":19.611999999999998,"15min":19.611999999999998},"last":1.504}]},{"osd":6,"last update":"Tue Sep 10 18:34:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.63800000000000001,"15min":0.50900000000000001},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.26400000000000001},"max":{"1min":0.77000000000000002,"5min":3.1139999999999999,"15min":6.7990000000000004},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.63100000000000001,"15min":0.54900000000000004},"min":{"1min":0.32100000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.95499999999999996,"5min":3.6419999999999999,"15min":7.0300000000000002},"last":1.3799999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:34:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65200000000000002,"15min":0.55700000000000005},"min":{"1min":0.316,"5min":0.28799999999999998,"15min":0.255},"max":{"1min":1.4330000000000001,"5min":4.8600000000000003,"15min":7.0670000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64400000000000002,"15min":0.57099999999999995},"min":{"1min":0.33800000000000002,"5min":0.28699999999999998,"15min":0.247},"max":{"1min":1.2130000000000001,"5min":3.3450000000000002,"15min":6.5330000000000004},"last":1.4510000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117178,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1910324,"kb_used_data":1324588,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":91834828,"statfs":{"total":95995035648,"available":94038863872,"internally_reserved":0,"allocated":1356378112,"data_stored":1355878299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:12 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2789999999999999,"15min":1.2390000000000001},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.9569999999999999,"5min":5.4560000000000004,"15min":8.5530000000000008},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.903,"5min":1.464,"15min":1.351},"min":{"1min":0.253,"5min":0.22,"15min":0.193},"max":{"1min":7.5979999999999999,"5min":8.2110000000000003,"15min":13.659000000000001},"last":0.57599999999999996}]},{"osd":1,"last update":"Tue Sep 10 18:34:12 2024","interfaces":[{"interface":"back","average":{"1min":1.9890000000000001,"5min":1.923,"15min":1.431},"min":{"1min":0.28599999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.0419999999999998,"5min":35.406999999999996,"15min":35.406999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.556,"5min":1.9590000000000001,"15min":1.343},"min":{"1min":0.375,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":3.843,"5min":36.902000000000001,"15min":36.902000000000001},"last":0.625}]},{"osd":2,"last update":"Tue Sep 10 18:34:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.0589999999999999,"15min":1.077},"min":{"1min":0.38700000000000001,"5min":0.222,"15min":0.222},"max":{"1min":3.9060000000000001,"5min":5.3810000000000002,"15min":7.7960000000000003},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3520000000000001,"15min":1.256},"min":{"1min":0.23000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":6.7290000000000001,"5min":14.997999999999999,"15min":14.997999999999999},"last":2.2679999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.208,"15min":1.2390000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":2.7210000000000001,"5min":6.5250000000000004,"15min":23.684999999999999},"last":2.0470000000000002},{"interface":"front","average":{"1min":1.079,"5min":1.2130000000000001,"15min":1.1890000000000001},"min":{"1min":0.33400000000000002,"5min":0.315,"15min":0.27200000000000002},"max":{"1min":3.8610000000000002,"5min":6.1929999999999996,"15min":23.649000000000001},"last":0.55600000000000005}]},{"osd":5,"last update":"Tue Sep 10 18:34:58 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.2789999999999999,"15min":1.343},"min":{"1min":0.34599999999999997,"5min":0.314,"15min":0.314},"max":{"1min":3.4849999999999999,"5min":6.601,"15min":23.719999999999999},"last":2.2029999999999998},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.3360000000000001,"15min":1.3109999999999999},"min":{"1min":0.46200000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.3109999999999999,"5min":6.2249999999999996,"15min":23.707000000000001},"last":1.194}]},{"osd":6,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.587,"15min":1.236},"min":{"1min":0.49399999999999999,"5min":0.32200000000000001,"15min":0.317},"max":{"1min":4.2720000000000002,"5min":31.815999999999999,"15min":31.815999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.8129999999999999,"15min":1.5},"min":{"1min":0.44500000000000001,"5min":0.33400000000000002,"15min":0.315},"max":{"1min":4.1870000000000003,"5min":31.742000000000001,"15min":31.742000000000001},"last":0.45500000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.327,"15min":1.282},"min":{"1min":0.39300000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.3369999999999997,"5min":6.5730000000000004,"15min":23.748999999999999},"last":2.1739999999999999},{"interface":"front","average":{"1min":1.373,"5min":1.454,"15min":1.3340000000000001},"min":{"1min":0.44800000000000001,"5min":0.35199999999999998,"15min":0.307},"max":{"1min":4.2809999999999997,"5min":12.612,"15min":25.058},"last":1.1679999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411518,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":816732,"kb_used_data":515156,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92928420,"statfs":{"total":95995035648,"available":95158702080,"internally_reserved":0,"allocated":527519744,"data_stored":527043679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:50 2024","interfaces":[{"interface":"back","average":{"1min":2.2490000000000001,"5min":1.544,"15min":1.458},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":26.585999999999999,"5min":26.585999999999999,"15min":43.890000000000001},"last":0.373},{"interface":"front","average":{"1min":2.4060000000000001,"5min":1.7310000000000001,"15min":1.669},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.16900000000000001},"max":{"1min":26.48,"5min":26.48,"15min":44.331000000000003},"last":2.1869999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:34:50 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.345,"15min":1.4419999999999999},"min":{"1min":0.251,"5min":0.19400000000000001,"15min":0.193},"max":{"1min":2.3479999999999999,"5min":15.750999999999999,"15min":44.575000000000003},"last":1.647},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.3680000000000001,"15min":1.246},"min":{"1min":0.21199999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.9750000000000001,"5min":14.988,"15min":44.368000000000002},"last":0.24299999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:34:10 2024","interfaces":[{"interface":"back","average":{"1min":4.4269999999999996,"5min":2.1669999999999998,"15min":1.7150000000000001},"min":{"1min":0.312,"5min":0.26100000000000001,"15min":0.251},"max":{"1min":61.331000000000003,"5min":61.331000000000003,"15min":61.331000000000003},"last":2.1120000000000001},{"interface":"front","average":{"1min":4.9180000000000001,"5min":2.1949999999999998,"15min":1.7969999999999999},"min":{"1min":0.33500000000000002,"5min":0.214,"15min":0.214},"max":{"1min":61.250999999999998,"5min":61.250999999999998,"15min":61.250999999999998},"last":2.052}]},{"osd":4,"last update":"Tue Sep 10 18:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":1.1879999999999999,"15min":1.016},"min":{"1min":0.30199999999999999,"5min":0.28699999999999998,"15min":0.24199999999999999},"max":{"1min":2.1920000000000002,"5min":12.555999999999999,"15min":12.555999999999999},"last":0.441},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.0589999999999999,"15min":0.92600000000000005},"min":{"1min":0.35099999999999998,"5min":0.33000000000000002,"15min":0.317},"max":{"1min":1.105,"5min":7.0730000000000004,"15min":12.130000000000001},"last":0.34100000000000003}]},{"osd":5,"last update":"Tue Sep 10 18:33:56 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":1.262,"15min":0.97299999999999998},"min":{"1min":0.32000000000000001,"5min":0.30299999999999999,"15min":0.27500000000000002},"max":{"1min":2.3980000000000001,"5min":7.1609999999999996,"15min":12.106},"last":1.3240000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.4430000000000001,"15min":1.0369999999999999},"min":{"1min":0.45000000000000001,"5min":0.32200000000000001,"15min":0.31900000000000001},"max":{"1min":2.6949999999999998,"5min":12.529999999999999,"15min":12.529999999999999},"last":0.317}]},{"osd":6,"last update":"Tue Sep 10 18:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.177,"15min":1.1160000000000001},"min":{"1min":0.38500000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":4.6440000000000001,"5min":5.9989999999999997,"15min":5.9989999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.90600000000000003,"5min":1.105,"15min":0.96999999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.30099999999999999},"max":{"1min":5.4880000000000004,"5min":7.1349999999999998,"15min":7.1349999999999998},"last":1.2889999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:34:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.198,"15min":1.0780000000000001},"min":{"1min":0.46300000000000002,"5min":0.35399999999999998,"15min":0.30499999999999999},"max":{"1min":4.0430000000000001,"5min":10.68,"15min":10.68},"last":0.5},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.341,"15min":1.129},"min":{"1min":0.40100000000000002,"5min":0.376,"15min":0.307},"max":{"1min":2.2349999999999999,"5min":12.596,"15min":12.596},"last":0.41299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575043,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2124664,"kb_used_data":1538928,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":91620488,"statfs":{"total":95995035648,"available":93819379712,"internally_reserved":0,"allocated":1575862272,"data_stored":1575358145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2450000000000001,"15min":1.5329999999999999},"min":{"1min":0.20100000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":8.0030000000000001,"5min":14.079000000000001,"15min":80.456999999999994},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3240000000000001,"15min":1.6160000000000001},"min":{"1min":0.247,"5min":0.20399999999999999,"15min":0.187},"max":{"1min":5.6980000000000004,"5min":5.6980000000000004,"15min":80.578000000000003},"last":0.26900000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:34:43 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.1779999999999999,"15min":1.1910000000000001},"min":{"1min":0.24399999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.8380000000000001,"5min":5.8380000000000001,"15min":13.035},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.2050000000000001,"15min":1.1399999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":2.302,"5min":12.222,"15min":12.222},"last":0.40200000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:34:23 2024","interfaces":[{"interface":"back","average":{"1min":1.8029999999999999,"5min":1.5129999999999999,"15min":1.575},"min":{"1min":0.40000000000000002,"5min":0.23899999999999999,"15min":0.22600000000000001},"max":{"1min":8.0399999999999991,"5min":14.121,"15min":43.076000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.446,"15min":1.6220000000000001},"min":{"1min":0.22500000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.8879999999999999,"5min":7.3799999999999999,"15min":41.93},"last":0.38200000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:34:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.26,"15min":1.079},"min":{"1min":0.312,"5min":0.312,"15min":0.29799999999999999},"max":{"1min":5.9930000000000003,"5min":5.9930000000000003,"15min":7.2859999999999996},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.3979999999999999,"15min":1.2170000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.28299999999999997},"max":{"1min":2.3969999999999998,"5min":7.8769999999999998,"15min":19.393000000000001},"last":0.55100000000000005}]},{"osd":5,"last update":"Tue Sep 10 18:34:59 2024","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":1.3919999999999999,"15min":1.2789999999999999},"min":{"1min":0.32800000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.5570000000000004,"5min":7.2110000000000003,"15min":15.48},"last":0.52700000000000002},{"interface":"front","average":{"1min":2.1749999999999998,"5min":1.4590000000000001,"15min":1.208},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.308},"max":{"1min":8.1050000000000004,"5min":11.414999999999999,"15min":11.414999999999999},"last":0.89400000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:34:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.2170000000000001,"15min":1.2909999999999999},"min":{"1min":0.41499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":8.1340000000000003,"5min":10.039,"15min":10.039},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.373,"15min":1.389},"min":{"1min":0.42699999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":5.9240000000000004,"5min":6.2670000000000003,"15min":7.3369999999999997},"last":0.59799999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:34:43 2024","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.4119999999999999,"15min":1.1879999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":8.1609999999999996,"5min":10.005000000000001,"15min":15.545999999999999},"last":0.88},{"interface":"front","average":{"1min":1.9870000000000001,"5min":1.5009999999999999,"15min":1.1890000000000001},"min":{"1min":0.50600000000000001,"5min":0.35699999999999998,"15min":0.313},"max":{"1min":6.0300000000000002,"5min":6.3049999999999997,"15min":7.359},"last":0.71199999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738567,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1069604,"kb_used_data":768092,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92675548,"statfs":{"total":95995035648,"available":94899761152,"internally_reserved":0,"allocated":786526208,"data_stored":786041808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:34:21 2024","interfaces":[{"interface":"back","average":{"1min":9.0549999999999997,"5min":3.1669999999999998,"15min":1.8899999999999999},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":146.44499999999999,"5min":146.44499999999999,"15min":146.44499999999999},"last":0.753},{"interface":"front","average":{"1min":9.9619999999999997,"5min":3.2679999999999998,"15min":2.0190000000000001},"min":{"1min":0.38300000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":148.77199999999999,"5min":148.77199999999999,"15min":148.77199999999999},"last":0.80200000000000005}]},{"osd":2,"last update":"Tue Sep 10 18:34:51 2024","interfaces":[{"interface":"back","average":{"1min":9.0990000000000002,"5min":2.7970000000000002,"15min":1.8640000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.189},"max":{"1min":146.56999999999999,"5min":146.56999999999999,"15min":146.56999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":9.1010000000000009,"5min":2.7040000000000002,"15min":1.8480000000000001},"min":{"1min":0.25,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":146.601,"5min":146.601,"15min":146.601},"last":0.42699999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:34:19 2024","interfaces":[{"interface":"back","average":{"1min":8.3469999999999995,"5min":2.8690000000000002,"15min":1.8899999999999999},"min":{"1min":0.28599999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":140.68299999999999,"5min":140.68299999999999,"15min":140.68299999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":8.4179999999999993,"5min":3.004,"15min":1.8740000000000001},"min":{"1min":0.308,"5min":0.26500000000000001,"15min":0.23899999999999999},"max":{"1min":140.572,"5min":140.572,"15min":140.572},"last":0.311}]},{"osd":4,"last update":"Tue Sep 10 18:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":1.1279999999999999,"15min":1.1319999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.246},"max":{"1min":1.9830000000000001,"5min":8.3699999999999992,"15min":17.151},"last":0.34899999999999998},{"interface":"front","average":{"1min":1.004,"5min":1.0580000000000001,"15min":1.127},"min":{"1min":0.31,"5min":0.29899999999999999,"15min":0.25900000000000001},"max":{"1min":3.3849999999999998,"5min":3.7919999999999998,"15min":18.728999999999999},"last":0.52900000000000003}]},{"osd":5,"last update":"Tue Sep 10 18:34:01 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.167,"15min":1.228},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.3210000000000002,"5min":4.7220000000000004,"15min":18.77},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.2949999999999999,"15min":1.1619999999999999},"min":{"1min":0.42599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.6050000000000004,"5min":6.6079999999999997,"15min":14.991},"last":1.9450000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":1.1799999999999999,"15min":1.2350000000000001},"min":{"1min":0.36299999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":2.0190000000000001,"5min":8.3940000000000001,"15min":8.3940000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0900000000000001,"15min":1.306},"min":{"1min":0.34399999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.3599999999999999,"5min":3.9079999999999999,"15min":9.7859999999999996},"last":0.63200000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:34:38 2024","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":1.371,"15min":1.228},"min":{"1min":0.377,"5min":0.32200000000000001,"15min":0.26700000000000002},"max":{"1min":10.75,"5min":10.75,"15min":13.711},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.2090000000000001,"15min":1.159},"min":{"1min":0.378,"5min":0.378,"15min":0.33500000000000002},"max":{"1min":11.218999999999999,"5min":11.218999999999999,"15min":15.032},"last":1.5429999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":776929280,"data_stored":776929280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1545474048,"data_stored":1545469976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":520028160,"data_stored":520028160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1344995328,"data_stored":1344995328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":525074432,"data_stored":525074432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1569787904,"data_stored":1569783832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":502202368,"data_stored":502202368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1560088576,"data_stored":1560084504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:35:03.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pgp_num 31 2024-09-10T18:35:03.707 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:03.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:03.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:03.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:03.839 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:03.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:03.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:03.942 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:03.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:03.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:03.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:04.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:03 smithi028 ceph-mon[29291]: 6.8 scrub starts 2024-09-10T18:35:04.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:03 smithi028 ceph-mon[29291]: 6.8 scrub ok 2024-09-10T18:35:04.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:03 smithi028 ceph-mon[29291]: 3.1 scrub starts 2024-09-10T18:35:04.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:03 smithi028 ceph-mon[29291]: 3.1 scrub ok 2024-09-10T18:35:04.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:03 smithi028 ceph-mon[29291]: pgmap v613: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T18:35:04.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:03 smithi028 ceph-mon[29291]: 6.7 scrub starts 2024-09-10T18:35:04.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:03 smithi028 ceph-mon[29291]: 6.7 scrub ok 2024-09-10T18:35:04.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:03 smithi028 ceph-mon[39220]: 6.8 scrub starts 2024-09-10T18:35:04.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:03 smithi028 ceph-mon[39220]: 6.8 scrub ok 2024-09-10T18:35:04.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:03 smithi028 ceph-mon[39220]: 3.1 scrub starts 2024-09-10T18:35:04.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:03 smithi028 ceph-mon[39220]: 3.1 scrub ok 2024-09-10T18:35:04.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:03 smithi028 ceph-mon[39220]: pgmap v613: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T18:35:04.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:03 smithi028 ceph-mon[39220]: 6.7 scrub starts 2024-09-10T18:35:04.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:03 smithi028 ceph-mon[39220]: 6.7 scrub ok 2024-09-10T18:35:04.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:04.106 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:03 smithi204 ceph-mon[32357]: 6.8 scrub starts 2024-09-10T18:35:04.106 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:03 smithi204 ceph-mon[32357]: 6.8 scrub ok 2024-09-10T18:35:04.106 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:03 smithi204 ceph-mon[32357]: 3.1 scrub starts 2024-09-10T18:35:04.106 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:03 smithi204 ceph-mon[32357]: 3.1 scrub ok 2024-09-10T18:35:04.106 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:03 smithi204 ceph-mon[32357]: pgmap v613: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T18:35:04.106 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:03 smithi204 ceph-mon[32357]: 6.7 scrub starts 2024-09-10T18:35:04.107 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:03 smithi204 ceph-mon[32357]: 6.7 scrub ok 2024-09-10T18:35:04.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:04.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:04.215 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:04.248 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:04.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:04.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:04.350 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:04.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:04.394 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:04.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:04.484 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:35:04.484 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:35:04.495 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:04.528 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:04.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:04.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:04.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:04.648 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:04.650 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:04.706 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:04.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:04.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:04.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:04.850 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:04.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:04.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:35:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:35:04.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:05.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:05.103 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:05.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:04 smithi028 ceph-mon[29291]: 4.11 scrub starts 2024-09-10T18:35:05.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:04 smithi028 ceph-mon[29291]: 4.11 scrub ok 2024-09-10T18:35:05.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:04 smithi028 ceph-mon[29291]: 4.6 scrub starts 2024-09-10T18:35:05.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:04 smithi028 ceph-mon[29291]: 4.6 scrub ok 2024-09-10T18:35:05.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:04 smithi028 ceph-mon[29291]: 3.1a scrub starts 2024-09-10T18:35:05.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:04 smithi028 ceph-mon[29291]: 3.1a scrub ok 2024-09-10T18:35:05.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:04 smithi028 ceph-mon[29291]: 1.10 scrub starts 2024-09-10T18:35:05.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:04 smithi028 ceph-mon[29291]: 1.10 scrub ok 2024-09-10T18:35:05.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:04 smithi028 ceph-mon[39220]: 4.11 scrub starts 2024-09-10T18:35:05.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:04 smithi028 ceph-mon[39220]: 4.11 scrub ok 2024-09-10T18:35:05.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:04 smithi028 ceph-mon[39220]: 4.6 scrub starts 2024-09-10T18:35:05.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:04 smithi028 ceph-mon[39220]: 4.6 scrub ok 2024-09-10T18:35:05.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:04 smithi028 ceph-mon[39220]: 3.1a scrub starts 2024-09-10T18:35:05.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:04 smithi028 ceph-mon[39220]: 3.1a scrub ok 2024-09-10T18:35:05.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:04 smithi028 ceph-mon[39220]: 1.10 scrub starts 2024-09-10T18:35:05.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:04 smithi028 ceph-mon[39220]: 1.10 scrub ok 2024-09-10T18:35:05.135 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:05.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:05.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:05.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:05.280 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:05.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:04 smithi204 ceph-mon[32357]: 4.11 scrub starts 2024-09-10T18:35:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:04 smithi204 ceph-mon[32357]: 4.11 scrub ok 2024-09-10T18:35:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:04 smithi204 ceph-mon[32357]: 4.6 scrub starts 2024-09-10T18:35:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:04 smithi204 ceph-mon[32357]: 4.6 scrub ok 2024-09-10T18:35:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:04 smithi204 ceph-mon[32357]: 3.1a scrub starts 2024-09-10T18:35:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:04 smithi204 ceph-mon[32357]: 3.1a scrub ok 2024-09-10T18:35:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:04 smithi204 ceph-mon[32357]: 1.10 scrub starts 2024-09-10T18:35:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:04 smithi204 ceph-mon[32357]: 1.10 scrub ok 2024-09-10T18:35:05.382 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:05.413 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:05.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:05.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:05.516 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:05.539 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:05.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:05.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:05.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:05.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:05.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:05.715 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-10T18:35:05.779 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:05.824 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:05.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:05.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:05.897 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:35:05.926 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:05.959 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:05.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:05.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:06.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:06.091 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:06.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:06.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:05 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:05 smithi204 ceph-mon[32357]: 4.c scrub starts 2024-09-10T18:35:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:05 smithi204 ceph-mon[32357]: 4.c scrub ok 2024-09-10T18:35:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:05 smithi204 ceph-mon[32357]: 6.18 scrub starts 2024-09-10T18:35:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:05 smithi204 ceph-mon[32357]: 6.18 scrub ok 2024-09-10T18:35:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:05 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3916013434' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:35:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:05 smithi204 ceph-mon[32357]: pgmap v614: 178 pgs: 178 active+clean; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:35:06.129 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:35:06.194 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:06.230 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:06.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:06.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:05 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:05 smithi028 ceph-mon[29291]: 4.c scrub starts 2024-09-10T18:35:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:05 smithi028 ceph-mon[29291]: 4.c scrub ok 2024-09-10T18:35:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:05 smithi028 ceph-mon[29291]: 6.18 scrub starts 2024-09-10T18:35:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:05 smithi028 ceph-mon[29291]: 6.18 scrub ok 2024-09-10T18:35:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:05 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3916013434' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:35:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:05 smithi028 ceph-mon[29291]: pgmap v614: 178 pgs: 178 active+clean; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:35:06.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:05 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:05 smithi028 ceph-mon[39220]: 4.c scrub starts 2024-09-10T18:35:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:05 smithi028 ceph-mon[39220]: 4.c scrub ok 2024-09-10T18:35:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:05 smithi028 ceph-mon[39220]: 6.18 scrub starts 2024-09-10T18:35:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:05 smithi028 ceph-mon[39220]: 6.18 scrub ok 2024-09-10T18:35:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:05 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3916013434' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:35:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:05 smithi028 ceph-mon[39220]: pgmap v614: 178 pgs: 178 active+clean; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:35:06.332 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:06.363 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:06.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:06.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:06.465 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:06.495 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:06.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:06.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:06.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:06.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:06.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:06.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:06.771 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:06.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:06.873 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:06.904 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:06.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:06.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:07.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:07.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:07.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:07.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:07.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:07.187 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:07.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:06 smithi028 ceph-mon[39220]: 1.0 scrub starts 2024-09-10T18:35:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:06 smithi028 ceph-mon[39220]: 1.0 scrub ok 2024-09-10T18:35:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:06 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:35:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:06 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:35:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:06 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3916013434' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:35:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:06 smithi028 ceph-mon[39220]: osdmap e299: 8 total, 8 up, 8 in 2024-09-10T18:35:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:06 smithi028 ceph-mon[39220]: 1.12 scrub starts 2024-09-10T18:35:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:06 smithi028 ceph-mon[39220]: 1.12 scrub ok 2024-09-10T18:35:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:06 smithi028 ceph-mon[29291]: 1.0 scrub starts 2024-09-10T18:35:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:06 smithi028 ceph-mon[29291]: 1.0 scrub ok 2024-09-10T18:35:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:06 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:35:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:06 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:35:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:06 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3916013434' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:35:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:06 smithi028 ceph-mon[29291]: osdmap e299: 8 total, 8 up, 8 in 2024-09-10T18:35:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:06 smithi028 ceph-mon[29291]: 1.12 scrub starts 2024-09-10T18:35:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:06 smithi028 ceph-mon[29291]: 1.12 scrub ok 2024-09-10T18:35:07.289 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:07.298 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:06 smithi204 ceph-mon[32357]: 1.0 scrub starts 2024-09-10T18:35:07.298 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:06 smithi204 ceph-mon[32357]: 1.0 scrub ok 2024-09-10T18:35:07.298 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:06 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:35:07.299 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:06 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:35:07.299 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:06 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3916013434' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:35:07.299 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:06 smithi204 ceph-mon[32357]: osdmap e299: 8 total, 8 up, 8 in 2024-09-10T18:35:07.299 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:06 smithi204 ceph-mon[32357]: 1.12 scrub starts 2024-09-10T18:35:07.299 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:06 smithi204 ceph-mon[32357]: 1.12 scrub ok 2024-09-10T18:35:07.323 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:07.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:07.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:07.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:07.465 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:07.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:07.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:07.566 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:07.606 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:07.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:07.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:07.747 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:07.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:07.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:07.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:07.891 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:07.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:07.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:07.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:07 smithi028 ceph-mon[29291]: pgmap v616: 178 pgs: 178 active+clean; 2.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T18:35:07.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:07 smithi028 ceph-mon[29291]: 6.e scrub starts 2024-09-10T18:35:07.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:07 smithi028 ceph-mon[29291]: 6.e scrub ok 2024-09-10T18:35:07.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:07 smithi028 ceph-mon[29291]: 4.4 scrub starts 2024-09-10T18:35:07.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:07 smithi028 ceph-mon[29291]: 4.4 scrub ok 2024-09-10T18:35:07.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:07 smithi028 ceph-mon[39220]: pgmap v616: 178 pgs: 178 active+clean; 2.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T18:35:07.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:07 smithi028 ceph-mon[39220]: 6.e scrub starts 2024-09-10T18:35:07.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:07 smithi028 ceph-mon[39220]: 6.e scrub ok 2024-09-10T18:35:07.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:07 smithi028 ceph-mon[39220]: 4.4 scrub starts 2024-09-10T18:35:07.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:07 smithi028 ceph-mon[39220]: 4.4 scrub ok 2024-09-10T18:35:07.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:08.030 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:08.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:08.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:08.132 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:08.166 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:08.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:08.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:08.168 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:07 smithi204 ceph-mon[32357]: pgmap v616: 178 pgs: 178 active+clean; 2.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T18:35:08.168 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:07 smithi204 ceph-mon[32357]: 6.e scrub starts 2024-09-10T18:35:08.169 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:07 smithi204 ceph-mon[32357]: 6.e scrub ok 2024-09-10T18:35:08.169 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:07 smithi204 ceph-mon[32357]: 4.4 scrub starts 2024-09-10T18:35:08.169 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:07 smithi204 ceph-mon[32357]: 4.4 scrub ok 2024-09-10T18:35:08.269 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:08.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:08.308 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:08.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:08.316 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:35:08.316 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:35:08.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:08.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:08.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:08.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:08.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:08.594 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:08.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:08.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:08.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:08.736 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:08.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:08.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:08.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:08.889 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:08.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:08.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:09.027 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:09.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:09.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:09.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:08 smithi204 ceph-mon[32357]: 5.1c deep-scrub starts 2024-09-10T18:35:09.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:08 smithi204 ceph-mon[32357]: 5.1c deep-scrub ok 2024-09-10T18:35:09.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:08 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:35:09.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:08 smithi204 ceph-mon[32357]: 5.f scrub starts 2024-09-10T18:35:09.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:08 smithi204 ceph-mon[32357]: 5.f scrub ok 2024-09-10T18:35:09.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:08 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:09.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:08 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:35:09.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:09.163 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:09.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:09.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:09.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:08 smithi028 ceph-mon[39220]: 5.1c deep-scrub starts 2024-09-10T18:35:09.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:08 smithi028 ceph-mon[39220]: 5.1c deep-scrub ok 2024-09-10T18:35:09.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:08 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:35:09.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:08 smithi028 ceph-mon[39220]: 5.f scrub starts 2024-09-10T18:35:09.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:08 smithi028 ceph-mon[39220]: 5.f scrub ok 2024-09-10T18:35:09.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:08 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:09.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:08 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:35:09.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:08 smithi028 ceph-mon[29291]: 5.1c deep-scrub starts 2024-09-10T18:35:09.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:08 smithi028 ceph-mon[29291]: 5.1c deep-scrub ok 2024-09-10T18:35:09.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:08 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:35:09.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:08 smithi028 ceph-mon[29291]: 5.f scrub starts 2024-09-10T18:35:09.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:08 smithi028 ceph-mon[29291]: 5.f scrub ok 2024-09-10T18:35:09.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:08 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:09.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:08 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:35:09.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:09.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:09.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:09.451 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:09.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:09.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:09.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:09.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:09.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:09.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:09.729 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:09.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:09.745 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:35:09.745 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:35:09.779 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:35:09.779 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:35:09.813 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:35:09.813 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:35:09.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:09.850 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:35:09.850 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:35:09.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:09.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:09.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:09.884 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:35:09.884 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:35:09.913 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:35:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:35:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 1 pgp_num to 31 2024-09-10T18:35:09.947 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:35:09.947 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:35:09.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:09.985 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:35:09.985 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:35:09.987 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:09.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:09.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:10.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:35:10.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:10.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:10.133 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:10.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:10.234 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:10.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[29291]: 1.2 scrub starts 2024-09-10T18:35:10.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[29291]: 1.2 scrub ok 2024-09-10T18:35:10.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[29291]: 1.13 scrub starts 2024-09-10T18:35:10.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[29291]: 1.13 scrub ok 2024-09-10T18:35:10.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[29291]: 4.1f scrub starts 2024-09-10T18:35:10.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[29291]: 4.1f scrub ok 2024-09-10T18:35:10.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[29291]: pgmap v617: 178 pgs: 178 active+clean; 2.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T18:35:10.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:35:10.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:35:10.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1434480925' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-09-10T18:35:10.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2163103954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:35:10.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[39220]: 1.2 scrub starts 2024-09-10T18:35:10.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[39220]: 1.2 scrub ok 2024-09-10T18:35:10.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[39220]: 1.13 scrub starts 2024-09-10T18:35:10.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[39220]: 1.13 scrub ok 2024-09-10T18:35:10.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[39220]: 4.1f scrub starts 2024-09-10T18:35:10.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[39220]: 4.1f scrub ok 2024-09-10T18:35:10.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[39220]: pgmap v617: 178 pgs: 178 active+clean; 2.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T18:35:10.258 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:35:10.258 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:35:10.258 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1434480925' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-09-10T18:35:10.258 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:09 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2163103954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:35:10.265 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:10.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:10.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:09 smithi204 ceph-mon[32357]: 1.2 scrub starts 2024-09-10T18:35:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:09 smithi204 ceph-mon[32357]: 1.2 scrub ok 2024-09-10T18:35:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:09 smithi204 ceph-mon[32357]: 1.13 scrub starts 2024-09-10T18:35:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:09 smithi204 ceph-mon[32357]: 1.13 scrub ok 2024-09-10T18:35:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:09 smithi204 ceph-mon[32357]: 4.1f scrub starts 2024-09-10T18:35:10.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:09 smithi204 ceph-mon[32357]: 4.1f scrub ok 2024-09-10T18:35:10.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:09 smithi204 ceph-mon[32357]: pgmap v617: 178 pgs: 178 active+clean; 2.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T18:35:10.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:09 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:35:10.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:09 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:35:10.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:09 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1434480925' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-09-10T18:35:10.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:09 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2163103954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:35:10.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:10.405 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:10.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:10.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:10.507 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:10.545 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:10.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:10.646 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:10.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:10.681 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:10.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:10.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:10.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:10.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T18:35:10.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:11.003 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:11.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:11.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:11.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:11.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:11.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:10 smithi028 ceph-mon[29291]: 3.1f scrub starts 2024-09-10T18:35:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:10 smithi028 ceph-mon[29291]: 3.1f scrub ok 2024-09-10T18:35:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:10 smithi028 ceph-mon[29291]: 3.19 scrub starts 2024-09-10T18:35:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:10 smithi028 ceph-mon[29291]: 3.19 scrub ok 2024-09-10T18:35:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:10 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1434480925' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-09-10T18:35:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:10 smithi028 ceph-mon[29291]: osdmap e300: 8 total, 8 up, 8 in 2024-09-10T18:35:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:10 smithi028 ceph-mon[29291]: 5.1 scrub starts 2024-09-10T18:35:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:10 smithi028 ceph-mon[29291]: 5.1 scrub ok 2024-09-10T18:35:11.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:10 smithi028 ceph-mon[39220]: 3.1f scrub starts 2024-09-10T18:35:11.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:10 smithi028 ceph-mon[39220]: 3.1f scrub ok 2024-09-10T18:35:11.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:10 smithi028 ceph-mon[39220]: 3.19 scrub starts 2024-09-10T18:35:11.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:10 smithi028 ceph-mon[39220]: 3.19 scrub ok 2024-09-10T18:35:11.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:10 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1434480925' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-09-10T18:35:11.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:10 smithi028 ceph-mon[39220]: osdmap e300: 8 total, 8 up, 8 in 2024-09-10T18:35:11.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:10 smithi028 ceph-mon[39220]: 5.1 scrub starts 2024-09-10T18:35:11.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:10 smithi028 ceph-mon[39220]: 5.1 scrub ok 2024-09-10T18:35:11.276 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:11.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:11.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:10 smithi204 ceph-mon[32357]: 3.1f scrub starts 2024-09-10T18:35:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:10 smithi204 ceph-mon[32357]: 3.1f scrub ok 2024-09-10T18:35:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:10 smithi204 ceph-mon[32357]: 3.19 scrub starts 2024-09-10T18:35:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:10 smithi204 ceph-mon[32357]: 3.19 scrub ok 2024-09-10T18:35:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:10 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1434480925' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-09-10T18:35:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:10 smithi204 ceph-mon[32357]: osdmap e300: 8 total, 8 up, 8 in 2024-09-10T18:35:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:10 smithi204 ceph-mon[32357]: 5.1 scrub starts 2024-09-10T18:35:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:10 smithi204 ceph-mon[32357]: 5.1 scrub ok 2024-09-10T18:35:11.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:11.419 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:11.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:11.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:11.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:11.565 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:11.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:11.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:11.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:11.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:11.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:11.753 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:35:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:35:11] "GET /metrics HTTP/1.1" 200 224522 "" "Prometheus/2.43.0" 2024-09-10T18:35:11.809 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:11.851 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:11.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:11.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:11.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:11.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:11.989 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:11.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:12.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:12.126 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:12.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:12.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:12.228 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:11 smithi028 ceph-mon[39220]: 4.3 scrub starts 2024-09-10T18:35:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:11 smithi028 ceph-mon[39220]: 4.3 scrub ok 2024-09-10T18:35:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:11 smithi028 ceph-mon[39220]: pgmap v619: 178 pgs: 178 active+clean; 1.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.56k op/s 2024-09-10T18:35:12.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:11 smithi028 ceph-mon[29291]: 4.3 scrub starts 2024-09-10T18:35:12.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:11 smithi028 ceph-mon[29291]: 4.3 scrub ok 2024-09-10T18:35:12.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:11 smithi028 ceph-mon[29291]: pgmap v619: 178 pgs: 178 active+clean; 1.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.56k op/s 2024-09-10T18:35:12.258 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:12.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:12.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:35:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:35:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:35:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:11 smithi204 ceph-mon[32357]: 4.3 scrub starts 2024-09-10T18:35:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:11 smithi204 ceph-mon[32357]: 4.3 scrub ok 2024-09-10T18:35:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:11 smithi204 ceph-mon[32357]: pgmap v619: 178 pgs: 178 active+clean; 1.8 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.56k op/s 2024-09-10T18:35:12.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:12.395 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:12.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:12.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:12.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:12.542 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:12.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:12.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:12.644 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:12.689 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:12.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:12.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:12.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:12.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:12.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:12.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:12.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:12.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:12.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:12.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:13.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:13.097 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:13.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:13.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:13.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:12 smithi204 ceph-mon[32357]: 4.f scrub starts 2024-09-10T18:35:13.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:12 smithi204 ceph-mon[32357]: 4.f scrub ok 2024-09-10T18:35:13.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:12 smithi204 ceph-mon[32357]: 5.7 scrub starts 2024-09-10T18:35:13.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:12 smithi204 ceph-mon[32357]: 5.7 scrub ok 2024-09-10T18:35:13.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:13.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:13.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:13.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:12 smithi028 ceph-mon[39220]: 4.f scrub starts 2024-09-10T18:35:13.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:12 smithi028 ceph-mon[39220]: 4.f scrub ok 2024-09-10T18:35:13.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:12 smithi028 ceph-mon[39220]: 5.7 scrub starts 2024-09-10T18:35:13.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:12 smithi028 ceph-mon[39220]: 5.7 scrub ok 2024-09-10T18:35:13.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:12 smithi028 ceph-mon[29291]: 4.f scrub starts 2024-09-10T18:35:13.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:12 smithi028 ceph-mon[29291]: 4.f scrub ok 2024-09-10T18:35:13.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:12 smithi028 ceph-mon[29291]: 5.7 scrub starts 2024-09-10T18:35:13.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:12 smithi028 ceph-mon[29291]: 5.7 scrub ok 2024-09-10T18:35:13.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:13.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:13.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:13.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:13.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:13.619 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:13.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:13.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:13.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:13.760 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:13.795 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:13.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:13.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:13.896 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:13.927 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:13.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:13.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:14.029 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:14.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:14.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:14.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:14.195 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:14.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:14.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:14.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:13 smithi028 ceph-mon[39220]: 3.12 scrub starts 2024-09-10T18:35:14.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:13 smithi028 ceph-mon[39220]: 5.17 scrub starts 2024-09-10T18:35:14.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:13 smithi028 ceph-mon[39220]: 3.12 scrub ok 2024-09-10T18:35:14.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:13 smithi028 ceph-mon[39220]: 5.17 scrub ok 2024-09-10T18:35:14.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:13 smithi028 ceph-mon[39220]: pgmap v620: 178 pgs: 178 active+clean; 1.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.56k op/s 2024-09-10T18:35:14.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:13 smithi028 ceph-mon[29291]: 3.12 scrub starts 2024-09-10T18:35:14.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:13 smithi028 ceph-mon[29291]: 5.17 scrub starts 2024-09-10T18:35:14.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:13 smithi028 ceph-mon[29291]: 3.12 scrub ok 2024-09-10T18:35:14.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:13 smithi028 ceph-mon[29291]: 5.17 scrub ok 2024-09-10T18:35:14.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:13 smithi028 ceph-mon[29291]: pgmap v620: 178 pgs: 178 active+clean; 1.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.56k op/s 2024-09-10T18:35:14.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:14.318 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:14.324 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:14.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:14.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:14.330 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:13 smithi204 ceph-mon[32357]: 3.12 scrub starts 2024-09-10T18:35:14.330 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:13 smithi204 ceph-mon[32357]: 5.17 scrub starts 2024-09-10T18:35:14.330 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:13 smithi204 ceph-mon[32357]: 3.12 scrub ok 2024-09-10T18:35:14.330 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:13 smithi204 ceph-mon[32357]: 5.17 scrub ok 2024-09-10T18:35:14.330 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:13 smithi204 ceph-mon[32357]: pgmap v620: 178 pgs: 178 active+clean; 1.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.56k op/s 2024-09-10T18:35:14.335 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:14.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:14.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:14.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:14.570 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:14.600 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:14.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:14.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:14.702 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:14.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:14.734 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:14.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:14.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:14.877 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:14.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:14.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:14.953 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:35:14 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:35:14.979 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:15.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:15.012 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:15.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:15.068 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[39220]: 1.4 scrub starts 2024-09-10T18:35:15.068 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[39220]: 1.4 scrub ok 2024-09-10T18:35:15.069 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[39220]: 5.1d scrub starts 2024-09-10T18:35:15.069 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[39220]: 5.1d scrub ok 2024-09-10T18:35:15.069 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:35:15.069 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:35:15.069 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[39220]: osdmap e301: 8 total, 8 up, 8 in 2024-09-10T18:35:15.069 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[39220]: 4.13 scrub starts 2024-09-10T18:35:15.069 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[39220]: 4.13 scrub ok 2024-09-10T18:35:15.069 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[39220]: 3.c scrub starts 2024-09-10T18:35:15.069 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[39220]: 3.c scrub ok 2024-09-10T18:35:15.069 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[29291]: 1.4 scrub starts 2024-09-10T18:35:15.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[29291]: 1.4 scrub ok 2024-09-10T18:35:15.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[29291]: 5.1d scrub starts 2024-09-10T18:35:15.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[29291]: 5.1d scrub ok 2024-09-10T18:35:15.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:35:15.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:35:15.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[29291]: osdmap e301: 8 total, 8 up, 8 in 2024-09-10T18:35:15.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[29291]: 4.13 scrub starts 2024-09-10T18:35:15.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[29291]: 4.13 scrub ok 2024-09-10T18:35:15.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[29291]: 3.c scrub starts 2024-09-10T18:35:15.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:14 smithi028 ceph-mon[29291]: 3.c scrub ok 2024-09-10T18:35:15.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:15.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:15.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:15.254 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:15.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:15.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:15.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:14 smithi204 ceph-mon[32357]: 1.4 scrub starts 2024-09-10T18:35:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:14 smithi204 ceph-mon[32357]: 1.4 scrub ok 2024-09-10T18:35:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:14 smithi204 ceph-mon[32357]: 5.1d scrub starts 2024-09-10T18:35:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:14 smithi204 ceph-mon[32357]: 5.1d scrub ok 2024-09-10T18:35:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:14 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:35:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:14 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:35:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:14 smithi204 ceph-mon[32357]: osdmap e301: 8 total, 8 up, 8 in 2024-09-10T18:35:15.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:14 smithi204 ceph-mon[32357]: 4.13 scrub starts 2024-09-10T18:35:15.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:14 smithi204 ceph-mon[32357]: 4.13 scrub ok 2024-09-10T18:35:15.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:14 smithi204 ceph-mon[32357]: 3.c scrub starts 2024-09-10T18:35:15.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:14 smithi204 ceph-mon[32357]: 3.c scrub ok 2024-09-10T18:35:15.394 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:15.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:15.425 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:15.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:15.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:15.561 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:15.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:15.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:15.663 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:15.699 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:15.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:15.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:15.780 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:35:15.780 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:35:15.781 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-10T18:35:15.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:35:15.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:15.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:15.870 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:15.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:15.970 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:15.998 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:15.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:15.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[39220]: 4.1d scrub starts 2024-09-10T18:35:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[39220]: 4.1d scrub ok 2024-09-10T18:35:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[39220]: 6.1 scrub starts 2024-09-10T18:35:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[39220]: 6.1 scrub ok 2024-09-10T18:35:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[39220]: pgmap v622: 178 pgs: 178 active+clean; 1.6 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:35:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[39220]: 4.9 deep-scrub starts 2024-09-10T18:35:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[39220]: 4.9 deep-scrub ok 2024-09-10T18:35:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[39220]: osdmap e302: 8 total, 8 up, 8 in 2024-09-10T18:35:16.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:16.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[29291]: 4.1d scrub starts 2024-09-10T18:35:16.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[29291]: 4.1d scrub ok 2024-09-10T18:35:16.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[29291]: 6.1 scrub starts 2024-09-10T18:35:16.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[29291]: 6.1 scrub ok 2024-09-10T18:35:16.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[29291]: pgmap v622: 178 pgs: 178 active+clean; 1.6 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:35:16.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[29291]: 4.9 deep-scrub starts 2024-09-10T18:35:16.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[29291]: 4.9 deep-scrub ok 2024-09-10T18:35:16.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:15 smithi028 ceph-mon[29291]: osdmap e302: 8 total, 8 up, 8 in 2024-09-10T18:35:16.100 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:16.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:16.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:16.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:16.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:35:16.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:16.287 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:16.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:16.302 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:15 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:16.302 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:15 smithi204 ceph-mon[32357]: 4.1d scrub starts 2024-09-10T18:35:16.302 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:15 smithi204 ceph-mon[32357]: 4.1d scrub ok 2024-09-10T18:35:16.302 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:15 smithi204 ceph-mon[32357]: 6.1 scrub starts 2024-09-10T18:35:16.302 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:15 smithi204 ceph-mon[32357]: 6.1 scrub ok 2024-09-10T18:35:16.302 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:15 smithi204 ceph-mon[32357]: pgmap v622: 178 pgs: 178 active+clean; 1.6 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T18:35:16.302 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:15 smithi204 ceph-mon[32357]: 4.9 deep-scrub starts 2024-09-10T18:35:16.302 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:15 smithi204 ceph-mon[32357]: 4.9 deep-scrub ok 2024-09-10T18:35:16.303 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:15 smithi204 ceph-mon[32357]: osdmap e302: 8 total, 8 up, 8 in 2024-09-10T18:35:16.389 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:16.425 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:16.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:16.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:16.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:16.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:16.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:16.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:16.687 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:16.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:16.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:16.789 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:16.822 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:16.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:16.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:16.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:16.962 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:16.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:16.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:17.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:17.092 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:17.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:17.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:17.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:17.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:16 smithi028 ceph-mon[29291]: 3.7 deep-scrub starts 2024-09-10T18:35:17.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:16 smithi028 ceph-mon[29291]: 3.7 deep-scrub ok 2024-09-10T18:35:17.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:16 smithi028 ceph-mon[29291]: 4.1e scrub starts 2024-09-10T18:35:17.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:16 smithi028 ceph-mon[29291]: 4.1e scrub ok 2024-09-10T18:35:17.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:16 smithi028 ceph-mon[29291]: 1.b scrub starts 2024-09-10T18:35:17.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:16 smithi028 ceph-mon[29291]: 1.b scrub ok 2024-09-10T18:35:17.237 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:16 smithi028 ceph-mon[39220]: 3.7 deep-scrub starts 2024-09-10T18:35:17.237 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:16 smithi028 ceph-mon[39220]: 3.7 deep-scrub ok 2024-09-10T18:35:17.237 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:16 smithi028 ceph-mon[39220]: 4.1e scrub starts 2024-09-10T18:35:17.237 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:16 smithi028 ceph-mon[39220]: 4.1e scrub ok 2024-09-10T18:35:17.237 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:16 smithi028 ceph-mon[39220]: 1.b scrub starts 2024-09-10T18:35:17.237 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:16 smithi028 ceph-mon[39220]: 1.b scrub ok 2024-09-10T18:35:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:17.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:17.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:17.246 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:16 smithi204 ceph-mon[32357]: 3.7 deep-scrub starts 2024-09-10T18:35:17.246 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:16 smithi204 ceph-mon[32357]: 3.7 deep-scrub ok 2024-09-10T18:35:17.246 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:16 smithi204 ceph-mon[32357]: 4.1e scrub starts 2024-09-10T18:35:17.246 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:16 smithi204 ceph-mon[32357]: 4.1e scrub ok 2024-09-10T18:35:17.247 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:16 smithi204 ceph-mon[32357]: 1.b scrub starts 2024-09-10T18:35:17.247 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:16 smithi204 ceph-mon[32357]: 1.b scrub ok 2024-09-10T18:35:17.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:17.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:17.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:17.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:17.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:17.533 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:17.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:17.634 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:17.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:17.672 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:17.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:17.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:17.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:17.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:17.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:17.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:17.948 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:17.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:18.050 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:18.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:18.088 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:18.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:18.189 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:18.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:18.227 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:18.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:18.236 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:17 smithi028 ceph-mon[39220]: pgmap v624: 178 pgs: 178 active+clean; 1.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.94k op/s 2024-09-10T18:35:18.236 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:17 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:18.236 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:17 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:18.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:17 smithi028 ceph-mon[29291]: pgmap v624: 178 pgs: 178 active+clean; 1.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.94k op/s 2024-09-10T18:35:18.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:17 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:18.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:17 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:18.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:17 smithi204 ceph-mon[32357]: pgmap v624: 178 pgs: 178 active+clean; 1.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.94k op/s 2024-09-10T18:35:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:17 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:17 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:18.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:18.361 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:18.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:18.462 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:18.492 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:18.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:18.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:18.555 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:18.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:18.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:18.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:18.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:18.790 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:18.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:18.892 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:18.921 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:18.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:18.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:19.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:19.062 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:19.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:19.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:19.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:19.192 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:19.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:19.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:19.237 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:18 smithi028 ceph-mon[39220]: 5.3 scrub starts 2024-09-10T18:35:19.237 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:18 smithi028 ceph-mon[39220]: 5.3 scrub ok 2024-09-10T18:35:19.237 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:18 smithi028 ceph-mon[39220]: 1.1d scrub starts 2024-09-10T18:35:19.237 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:18 smithi028 ceph-mon[39220]: 1.1d scrub ok 2024-09-10T18:35:19.237 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:18 smithi028 ceph-mon[39220]: 3.13 deep-scrub starts 2024-09-10T18:35:19.237 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:18 smithi028 ceph-mon[39220]: 3.13 deep-scrub ok 2024-09-10T18:35:19.238 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:18 smithi028 ceph-mon[29291]: 5.3 scrub starts 2024-09-10T18:35:19.238 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:18 smithi028 ceph-mon[29291]: 5.3 scrub ok 2024-09-10T18:35:19.238 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:18 smithi028 ceph-mon[29291]: 1.1d scrub starts 2024-09-10T18:35:19.238 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:18 smithi028 ceph-mon[29291]: 1.1d scrub ok 2024-09-10T18:35:19.238 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:18 smithi028 ceph-mon[29291]: 3.13 deep-scrub starts 2024-09-10T18:35:19.238 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:18 smithi028 ceph-mon[29291]: 3.13 deep-scrub ok 2024-09-10T18:35:19.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:19.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:19.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:18 smithi204 ceph-mon[32357]: 5.3 scrub starts 2024-09-10T18:35:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:18 smithi204 ceph-mon[32357]: 5.3 scrub ok 2024-09-10T18:35:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:18 smithi204 ceph-mon[32357]: 1.1d scrub starts 2024-09-10T18:35:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:18 smithi204 ceph-mon[32357]: 1.1d scrub ok 2024-09-10T18:35:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:18 smithi204 ceph-mon[32357]: 3.13 deep-scrub starts 2024-09-10T18:35:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:18 smithi204 ceph-mon[32357]: 3.13 deep-scrub ok 2024-09-10T18:35:19.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:19.479 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:19.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:19.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:19.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:19.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:19.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:19.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:19.762 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:19.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:19.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:19.864 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:19.894 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:19.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:19.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:19.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:20.026 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:20.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:20.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:20.128 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:20.160 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:20.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:20.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:20.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:19 smithi028 ceph-mon[39220]: 4.b deep-scrub starts 2024-09-10T18:35:20.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:19 smithi028 ceph-mon[39220]: 4.b deep-scrub ok 2024-09-10T18:35:20.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:19 smithi028 ceph-mon[39220]: 6.12 scrub starts 2024-09-10T18:35:20.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:19 smithi028 ceph-mon[39220]: 6.12 scrub ok 2024-09-10T18:35:20.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:19 smithi028 ceph-mon[39220]: pgmap v625: 178 pgs: 178 active+clean; 1.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-10T18:35:20.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:19 smithi028 ceph-mon[29291]: 4.b deep-scrub starts 2024-09-10T18:35:20.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:19 smithi028 ceph-mon[29291]: 4.b deep-scrub ok 2024-09-10T18:35:20.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:19 smithi028 ceph-mon[29291]: 6.12 scrub starts 2024-09-10T18:35:20.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:19 smithi028 ceph-mon[29291]: 6.12 scrub ok 2024-09-10T18:35:20.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:19 smithi028 ceph-mon[29291]: pgmap v625: 178 pgs: 178 active+clean; 1.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-10T18:35:20.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:20.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:20.306 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:20.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:20.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:19 smithi204 ceph-mon[32357]: 4.b deep-scrub starts 2024-09-10T18:35:20.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:19 smithi204 ceph-mon[32357]: 4.b deep-scrub ok 2024-09-10T18:35:20.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:19 smithi204 ceph-mon[32357]: 6.12 scrub starts 2024-09-10T18:35:20.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:19 smithi204 ceph-mon[32357]: 6.12 scrub ok 2024-09-10T18:35:20.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:19 smithi204 ceph-mon[32357]: pgmap v625: 178 pgs: 178 active+clean; 1.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-10T18:35:20.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:20.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:20.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:20.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:20.590 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:20.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:20.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:20.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:20.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:20.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:20.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:20.878 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:20.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:20.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:20.980 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:21.009 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:35:21.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:21.012 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:21.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:21.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:21.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:21.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:21.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:21.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:21.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:21.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:21.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:21 smithi028 ceph-mon[39220]: 4.16 deep-scrub starts 2024-09-10T18:35:21.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:21 smithi028 ceph-mon[39220]: 4.16 deep-scrub ok 2024-09-10T18:35:21.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:21 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/534997834' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:35:21.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:21 smithi028 ceph-mon[39220]: from='client.35501 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:35:21.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:21 smithi028 ceph-mon[29291]: 4.16 deep-scrub starts 2024-09-10T18:35:21.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:21 smithi028 ceph-mon[29291]: 4.16 deep-scrub ok 2024-09-10T18:35:21.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:21 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/534997834' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:35:21.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:21 smithi028 ceph-mon[29291]: from='client.35501 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:35:21.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:21 smithi204 ceph-mon[32357]: 4.16 deep-scrub starts 2024-09-10T18:35:21.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:21 smithi204 ceph-mon[32357]: 4.16 deep-scrub ok 2024-09-10T18:35:21.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:21 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/534997834' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:35:21.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:21 smithi204 ceph-mon[32357]: from='client.35501 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:35:21.396 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:21.429 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:21.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:21.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:21.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:21.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:21.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:21.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:21.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:21.753 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:35:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:35:21] "GET /metrics HTTP/1.1" 200 224522 "" "Prometheus/2.43.0" 2024-09-10T18:35:21.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:21.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:21.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:21.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:21.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:21.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:22.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:22.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:22.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:22.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:22.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:22 smithi028 ceph-mon[39220]: 4.18 scrub starts 2024-09-10T18:35:22.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:22 smithi028 ceph-mon[39220]: 4.18 scrub ok 2024-09-10T18:35:22.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:22 smithi028 ceph-mon[39220]: 6.1c scrub starts 2024-09-10T18:35:22.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:22 smithi028 ceph-mon[39220]: 6.1c scrub ok 2024-09-10T18:35:22.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:22 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:35:22.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:22 smithi028 ceph-mon[39220]: from='client.35501 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:35:22.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:22 smithi028 ceph-mon[39220]: osdmap e303: 8 total, 8 up, 8 in 2024-09-10T18:35:22.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:22 smithi028 ceph-mon[39220]: pgmap v627: 178 pgs: 178 active+clean; 963 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 721 B/s rd, 0 B/s wr, 1.78k op/s 2024-09-10T18:35:22.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:22 smithi028 ceph-mon[29291]: 4.18 scrub starts 2024-09-10T18:35:22.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:22 smithi028 ceph-mon[29291]: 4.18 scrub ok 2024-09-10T18:35:22.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:22 smithi028 ceph-mon[29291]: 6.1c scrub starts 2024-09-10T18:35:22.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:22 smithi028 ceph-mon[29291]: 6.1c scrub ok 2024-09-10T18:35:22.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:22 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:35:22.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:22 smithi028 ceph-mon[29291]: from='client.35501 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:35:22.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:22 smithi028 ceph-mon[29291]: osdmap e303: 8 total, 8 up, 8 in 2024-09-10T18:35:22.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:22 smithi028 ceph-mon[29291]: pgmap v627: 178 pgs: 178 active+clean; 963 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 721 B/s rd, 0 B/s wr, 1.78k op/s 2024-09-10T18:35:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:22.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:22.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:22.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:35:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:35:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:35:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:22 smithi204 ceph-mon[32357]: 4.18 scrub starts 2024-09-10T18:35:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:22 smithi204 ceph-mon[32357]: 4.18 scrub ok 2024-09-10T18:35:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:22 smithi204 ceph-mon[32357]: 6.1c scrub starts 2024-09-10T18:35:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:22 smithi204 ceph-mon[32357]: 6.1c scrub ok 2024-09-10T18:35:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:22 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:35:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:22 smithi204 ceph-mon[32357]: from='client.35501 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:35:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:22 smithi204 ceph-mon[32357]: osdmap e303: 8 total, 8 up, 8 in 2024-09-10T18:35:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:22 smithi204 ceph-mon[32357]: pgmap v627: 178 pgs: 178 active+clean; 963 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 721 B/s rd, 0 B/s wr, 1.78k op/s 2024-09-10T18:35:22.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:22.421 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:22.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:22.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:22.522 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:22.557 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:22.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:22.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:22.658 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:22.695 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:22.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:22.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:22.796 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:22.834 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:22.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:22.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:22.936 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:22.972 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:22.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:22.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:23.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:23.109 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:35:23.109 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:35:23.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.109 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35306.0:258730 9.9 9:9b3e790b:::benchmark_data_smithi028_135703_object124139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-10T18:35:23.109 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:35:23.082445+0000", 2024-09-10T18:35:23.109 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.012465446, 2024-09-10T18:35:23.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012494633, 2024-09-10T18:35:23.109 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:23.109 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:35:23.109 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:35:23.109 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35306", 2024-09-10T18:35:23.109 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:35:23.109 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 258730 2024-09-10T18:35:23.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.110 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:23.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:23.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.082445+0000", 2024-09-10T18:35:23.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:23.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:23.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.082445+0000", 2024-09-10T18:35:23.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:23.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:23.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.082444+0000", 2024-09-10T18:35:23.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:35:23.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:23.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.082448+0000", 2024-09-10T18:35:23.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2100000000000003e-06 2024-09-10T18:35:23.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:23.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.082455+0000", 2024-09-10T18:35:23.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3329999999999999e-06 2024-09-10T18:35:23.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:23.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.082460+0000", 2024-09-10T18:35:23.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4469999999999999e-06 2024-09-10T18:35:23.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:23.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.087973+0000", 2024-09-10T18:35:23.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055126949999999997 2024-09-10T18:35:23.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:23.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.088081+0000", 2024-09-10T18:35:23.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108087 2024-09-10T18:35:23.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,7", 2024-09-10T18:35:23.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.088189+0000", 2024-09-10T18:35:23.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108268 2024-09-10T18:35:23.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:23.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.091935+0000", 2024-09-10T18:35:23.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037457609999999998 2024-09-10T18:35:23.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:35:23.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.093266+0000", 2024-09-10T18:35:23.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013311530000000001 2024-09-10T18:35:23.113 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:23.113 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:23.113 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:23.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.114 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35306.0:258736 9.5 9:a9888fa1:::benchmark_data_smithi028_135703_object124145:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-10T18:35:23.114 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:35:23.086970+0000", 2024-09-10T18:35:23.114 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0079399009999999992, 2024-09-10T18:35:23.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080602060000000003, 2024-09-10T18:35:23.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:23.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:35:23.114 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:35:23.114 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35306", 2024-09-10T18:35:23.114 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:35:23.114 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 258736 2024-09-10T18:35:23.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.114 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:23.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:23.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.086970+0000", 2024-09-10T18:35:23.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:23.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:23.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.086970+0000", 2024-09-10T18:35:23.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:23.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:23.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.086968+0000", 2024-09-10T18:35:23.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:35:23.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:23.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.086976+0000", 2024-09-10T18:35:23.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4749999999999996e-06 2024-09-10T18:35:23.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:23.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.086996+0000", 2024-09-10T18:35:23.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0095e-05 2024-09-10T18:35:23.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:23.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.087004+0000", 2024-09-10T18:35:23.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1489999999999994e-06 2024-09-10T18:35:23.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:23.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.091129+0000", 2024-09-10T18:35:23.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041247879999999999 2024-09-10T18:35:23.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:23.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.091256+0000", 2024-09-10T18:35:23.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012687700000000001 2024-09-10T18:35:23.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,7", 2024-09-10T18:35:23.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.091413+0000", 2024-09-10T18:35:23.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015752799999999999 2024-09-10T18:35:23.117 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:23.117 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:23.118 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:23.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.118 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35306.0:258744 9.b 9:d7c7a2e8:::benchmark_data_smithi028_135703_object124153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-10T18:35:23.118 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:35:23.092248+0000", 2024-09-10T18:35:23.118 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.002662265, 2024-09-10T18:35:23.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002855256, 2024-09-10T18:35:23.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:23.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:35:23.118 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:35:23.118 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35306", 2024-09-10T18:35:23.118 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:35:23.118 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 258744 2024-09-10T18:35:23.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.118 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:23.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:23.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.092248+0000", 2024-09-10T18:35:23.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:23.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:23.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.092248+0000", 2024-09-10T18:35:23.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:23.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:23.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.092246+0000", 2024-09-10T18:35:23.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:35:23.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:23.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.092253+0000", 2024-09-10T18:35:23.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0369999999999997e-06 2024-09-10T18:35:23.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:23.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.092261+0000", 2024-09-10T18:35:23.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0639999999999994e-06 2024-09-10T18:35:23.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:23.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.092266+0000", 2024-09-10T18:35:23.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2240000000000003e-06 2024-09-10T18:35:23.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:23.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.092315+0000", 2024-09-10T18:35:23.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8958000000000001e-05 2024-09-10T18:35:23.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:23.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.092420+0000", 2024-09-10T18:35:23.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000105107 2024-09-10T18:35:23.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:35:23.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.092521+0000", 2024-09-10T18:35:23.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000100723 2024-09-10T18:35:23.122 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:23.122 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:23.122 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:23.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.122 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.35306.0:258730 9.9 e303/246)", 2024-09-10T18:35:23.122 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:35:23.094712+0000", 2024-09-10T18:35:23.122 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.00019843899999999999, 2024-09-10T18:35:23.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00045697199999999999, 2024-09-10T18:35:23.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:23.122 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-09-10T18:35:23.123 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:23.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:23.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.094712+0000", 2024-09-10T18:35:23.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:23.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:23.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.094712+0000", 2024-09-10T18:35:23.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:23.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:23.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.094711+0000", 2024-09-10T18:35:23.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:35:23.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:23.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.094716+0000", 2024-09-10T18:35:23.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5879999999999997e-06 2024-09-10T18:35:23.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:23.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.094729+0000", 2024-09-10T18:35:23.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3006999999999999e-05 2024-09-10T18:35:23.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:23.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:23.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:23.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:35:23.094735+0000", 2024-09-10T18:35:23.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0939999999999996e-06 2024-09-10T18:35:23.125 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:23.125 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:23.125 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:23.125 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:23.125 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:35:23.125 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 4 2024-09-10T18:35:23.125 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:35:23.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:23.126 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:23.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:23.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:23.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:23.265 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:23.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:23.366 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:23.385 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:35:23.386 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:35:23.400 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:23.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:23.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:23.419 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:35:23.420 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:35:23.449 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:35:23.449 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:35:23.484 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:35:23.484 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:35:23.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:23.521 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:35:23.521 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:35:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:23.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:23.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:23.555 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:35:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:35:23.585 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:35:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:35:23.616 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:35:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:35:23.639 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:23.676 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:23.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:23.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:23.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:23.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:23.928 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:23.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:23.968 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:23.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:24.070 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:24.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:24.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:24.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:24.203 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:24.241 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:24.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:24.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:24.343 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:24.377 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:24.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:24.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:24.470 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:24 smithi028 ceph-mon[39220]: pgmap v628: 178 pgs: 178 active+clean; 884 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.57k op/s 2024-09-10T18:35:24.470 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:24 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1633545382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:35:24.470 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:24 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:24.470 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:35:24.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:24 smithi028 ceph-mon[29291]: pgmap v628: 178 pgs: 178 active+clean; 884 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.57k op/s 2024-09-10T18:35:24.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:24 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1633545382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:35:24.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:24 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:24.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:35:24.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:24.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:35:24.520 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:24.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:24 smithi204 ceph-mon[32357]: pgmap v628: 178 pgs: 178 active+clean; 884 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 1.57k op/s 2024-09-10T18:35:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:24 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1633545382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:35:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:24 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:35:24.622 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:24.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:24.655 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:24.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:24.756 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:24.785 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:24.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:24.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:24.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:24.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:24.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:24.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:25.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:25.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:25.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:25.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:35:24 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:35:25.179 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:25.209 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:25.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:25.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:25.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:25.350 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:25.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:25.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:25.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:25.483 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:25.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:25.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:25.585 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:25.611 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:25.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:25.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:25.713 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:25.748 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:25.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:25.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:25.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:25.887 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:25.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:25.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:25.989 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:26.024 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:26.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:26.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:26.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:26.155 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:26.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:26.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:26.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:26 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:26.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:26 smithi028 ceph-mon[29291]: pgmap v629: 178 pgs: 178 active+clean; 798 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 554 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-10T18:35:26.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:26 smithi028 ceph-mon[29291]: osdmap e304: 8 total, 8 up, 8 in 2024-09-10T18:35:26.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:26 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:26.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:26 smithi028 ceph-mon[39220]: pgmap v629: 178 pgs: 178 active+clean; 798 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 554 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-10T18:35:26.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:26 smithi028 ceph-mon[39220]: osdmap e304: 8 total, 8 up, 8 in 2024-09-10T18:35:26.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:26.309 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:26.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:26.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:26.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:26.451 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:26.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:26.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:26.576 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:26.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:26.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:26.602 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-10T18:35:26.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:26 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:26 smithi204 ceph-mon[32357]: pgmap v629: 178 pgs: 178 active+clean; 798 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 554 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-10T18:35:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:26 smithi204 ceph-mon[32357]: osdmap e304: 8 total, 8 up, 8 in 2024-09-10T18:35:26.678 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:26.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:26.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:26.825 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:26.852 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:26.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:26.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:26.859 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:35:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:35:26.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:26.982 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:26.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:27.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:27.104 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:27.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:27.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:27.205 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:27.226 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:27.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:27.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:35:27.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:27.356 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:27.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:27.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:27.458 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:27.489 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:27.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:27.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:27.591 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:27.619 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:27.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:27.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:27.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:27.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:27.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:27.875 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:27.906 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:27.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:27.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:28.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:28.040 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:28.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:28.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:28.180 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":630,"stamp":"2024-09-10T18:35:25.816566+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657772479,"num_objects":10226,"num_object_clones":0,"num_object_copies":30678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10226,"num_whiteouts":0,"num_read":2002,"num_read_kb":1778,"num_write":259890,"num_write_kb":8614046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":695456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71718,"ondisk_log_size":71718,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5199244,"kb_used_data":1912332,"kb_used_omap":19,"kb_used_meta":3286572,"kb_avail":744761972,"statfs":{"total":767960285184,"available":762636259328,"internally_reserved":0,"allocated":1958227968,"data_stored":1954111962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20033,"internal_metadata":3365450175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,5,3,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-732509290,"num_objects":-11177,"num_object_clones":0,"num_object_copies":-33531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11177,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":11174,"num_write_kb":-37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.769837"},"pg_stats":[{"pgid":"1.1d","version":"76'81","reported_seq":37,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:17.818900+0000","last_change":"2024-09-10T18:35:17.818822+0000","last_active":"2024-09-10T18:35:17.818900+0000","last_peered":"2024-09-10T18:35:17.818900+0000","last_clean":"2024-09-10T18:35:17.818900+0000","last_became_active":"2024-09-10T18:34:34.865894+0000","last_became_peered":"2024-09-10T18:34:34.865894+0000","last_unstale":"2024-09-10T18:35:17.818900+0000","last_undegraded":"2024-09-10T18:35:17.818900+0000","last_fullsized":"2024-09-10T18:35:17.818900+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:17.818727+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:17.818727+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:41.743523+0000","scrub_duration":0.0037738960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":313,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.824897+0000","last_change":"2024-09-10T18:35:09.688177+0000","last_active":"2024-09-10T18:35:15.824897+0000","last_peered":"2024-09-10T18:35:15.824897+0000","last_clean":"2024-09-10T18:35:15.824897+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:35:15.824897+0000","last_undegraded":"2024-09-10T18:35:15.824897+0000","last_fullsized":"2024-09-10T18:35:15.824897+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:09.687954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:35:09.687954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:11.549190+0000","scrub_duration":0.0020960879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":352,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:20.421910+0000","last_change":"2024-09-10T18:35:20.421834+0000","last_active":"2024-09-10T18:35:20.421910+0000","last_peered":"2024-09-10T18:35:20.421910+0000","last_clean":"2024-09-10T18:35:20.421910+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:35:20.421910+0000","last_undegraded":"2024-09-10T18:35:20.421910+0000","last_fullsized":"2024-09-10T18:35:20.421910+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:35:20.421727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:20.421727+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:46.115408+0000","scrub_duration":0.010997518,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":318,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.815749+0000","last_change":"2024-09-10T18:35:00.915434+0000","last_active":"2024-09-10T18:35:15.815749+0000","last_peered":"2024-09-10T18:35:15.815749+0000","last_clean":"2024-09-10T18:35:15.815749+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:35:15.815749+0000","last_undegraded":"2024-09-10T18:35:15.815749+0000","last_fullsized":"2024-09-10T18:35:15.815749+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:00.915210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:00.915210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:27.372155+0000","scrub_duration":0.0052862780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":306,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.810771+0000","last_change":"2024-09-10T18:33:08.643815+0000","last_active":"2024-09-10T18:35:15.810771+0000","last_peered":"2024-09-10T18:35:15.810771+0000","last_clean":"2024-09-10T18:35:15.810771+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:35:15.810771+0000","last_undegraded":"2024-09-10T18:35:15.810771+0000","last_fullsized":"2024-09-10T18:35:15.810771+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:08.643758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.643758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046354059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'81","reported_seq":23,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.817531+0000","last_change":"2024-09-10T18:34:34.868599+0000","last_active":"2024-09-10T18:35:15.817531+0000","last_peered":"2024-09-10T18:35:15.817531+0000","last_clean":"2024-09-10T18:35:15.817531+0000","last_became_active":"2024-09-10T18:34:34.867388+0000","last_became_peered":"2024-09-10T18:34:34.867388+0000","last_unstale":"2024-09-10T18:35:15.817531+0000","last_undegraded":"2024-09-10T18:35:15.817531+0000","last_fullsized":"2024-09-10T18:35:15.817531+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.370561+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.370561+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029251390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54614,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":315,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.826330+0000","last_change":"2024-09-10T18:34:16.293404+0000","last_active":"2024-09-10T18:35:15.826330+0000","last_peered":"2024-09-10T18:35:15.826330+0000","last_clean":"2024-09-10T18:35:15.826330+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:35:15.826330+0000","last_undegraded":"2024-09-10T18:35:15.826330+0000","last_fullsized":"2024-09-10T18:35:15.826330+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:16.293206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:16.293206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:37.489769+0000","scrub_duration":0.0021119960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":321,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.827001+0000","last_change":"2024-09-10T18:34:22.415416+0000","last_active":"2024-09-10T18:35:15.827001+0000","last_peered":"2024-09-10T18:35:15.827001+0000","last_clean":"2024-09-10T18:35:15.827001+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:35:15.827001+0000","last_undegraded":"2024-09-10T18:35:15.827001+0000","last_fullsized":"2024-09-10T18:35:15.827001+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:22.415093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.415093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:23.013807+0000","scrub_duration":0.068845071999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'34","reported_seq":352,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.819622+0000","last_change":"2024-09-10T18:32:55.371863+0000","last_active":"2024-09-10T18:35:15.819622+0000","last_peered":"2024-09-10T18:35:15.819622+0000","last_clean":"2024-09-10T18:35:15.819622+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:35:15.819622+0000","last_undegraded":"2024-09-10T18:35:15.819622+0000","last_fullsized":"2024-09-10T18:35:15.819622+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:32:55.371617+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:32:55.371617+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089694079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":318,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.810611+0000","last_change":"2024-09-10T18:34:09.440223+0000","last_active":"2024-09-10T18:35:15.810611+0000","last_peered":"2024-09-10T18:35:15.810611+0000","last_clean":"2024-09-10T18:35:15.810611+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:35:15.810611+0000","last_undegraded":"2024-09-10T18:35:15.810611+0000","last_fullsized":"2024-09-10T18:35:15.810611+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:09.440140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:34:09.440140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003425656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":323,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.826522+0000","last_change":"2024-09-10T18:33:16.963182+0000","last_active":"2024-09-10T18:35:15.826522+0000","last_peered":"2024-09-10T18:35:15.826522+0000","last_clean":"2024-09-10T18:35:15.826522+0000","last_became_active":"2024-09-10T18:26:27.644961+0000","last_became_peered":"2024-09-10T18:26:27.644961+0000","last_unstale":"2024-09-10T18:35:15.826522+0000","last_undegraded":"2024-09-10T18:35:15.826522+0000","last_fullsized":"2024-09-10T18:35:15.826522+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_clean_scrub_stamp":"2024-09-10T18:33:16.963102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019539739999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'15","reported_seq":345,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.810969+0000","last_change":"2024-09-10T18:34:25.455830+0000","last_active":"2024-09-10T18:35:15.810969+0000","last_peered":"2024-09-10T18:35:15.810969+0000","last_clean":"2024-09-10T18:35:15.810969+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:35:15.810969+0000","last_undegraded":"2024-09-10T18:35:15.810969+0000","last_fullsized":"2024-09-10T18:35:15.810969+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:34:25.455754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:25.455754+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:48.508068+0000","scrub_duration":0.13960720300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":319,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.825407+0000","last_change":"2024-09-10T18:34:17.265867+0000","last_active":"2024-09-10T18:35:15.825407+0000","last_peered":"2024-09-10T18:35:15.825407+0000","last_clean":"2024-09-10T18:35:15.825407+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:35:15.825407+0000","last_undegraded":"2024-09-10T18:35:15.825407+0000","last_fullsized":"2024-09-10T18:35:15.825407+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:17.265788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_clean_scrub_stamp":"2024-09-10T18:34:17.265788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:22.473386+0000","scrub_duration":0.0023387740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":317,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.824992+0000","last_change":"2024-09-10T18:35:04.782030+0000","last_active":"2024-09-10T18:35:15.824992+0000","last_peered":"2024-09-10T18:35:15.824992+0000","last_clean":"2024-09-10T18:35:15.824992+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:35:15.824992+0000","last_undegraded":"2024-09-10T18:35:15.824992+0000","last_fullsized":"2024-09-10T18:35:15.824992+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:04.781946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:04.781946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:25.372151+0000","scrub_duration":0.011382141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'81","reported_seq":91,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.826865+0000","last_change":"2024-09-10T18:34:34.863425+0000","last_active":"2024-09-10T18:35:15.826865+0000","last_peered":"2024-09-10T18:35:15.826865+0000","last_clean":"2024-09-10T18:35:15.826865+0000","last_became_active":"2024-09-10T18:34:34.863102+0000","last_became_peered":"2024-09-10T18:34:34.863102+0000","last_unstale":"2024-09-10T18:35:15.826865+0000","last_undegraded":"2024-09-10T18:35:15.826865+0000","last_fullsized":"2024-09-10T18:35:15.826865+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:48.006695+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:48.006695+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":323,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.825669+0000","last_change":"2024-09-10T18:34:54.513360+0000","last_active":"2024-09-10T18:35:15.825669+0000","last_peered":"2024-09-10T18:35:15.825669+0000","last_clean":"2024-09-10T18:35:15.825669+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:35:15.825669+0000","last_undegraded":"2024-09-10T18:35:15.825669+0000","last_fullsized":"2024-09-10T18:35:15.825669+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:54.513278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:17.464205+0000","last_clean_scrub_stamp":"2024-09-10T18:34:54.513278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:03.541538+0000","scrub_duration":0.0028371450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":357,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.810960+0000","last_change":"2024-09-10T18:34:52.235528+0000","last_active":"2024-09-10T18:35:15.810960+0000","last_peered":"2024-09-10T18:35:15.810960+0000","last_clean":"2024-09-10T18:35:15.810960+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:35:15.810960+0000","last_undegraded":"2024-09-10T18:35:15.810960+0000","last_fullsized":"2024-09-10T18:35:15.810960+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'9","last_scrub_stamp":"2024-09-10T18:34:52.235425+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:34:52.235425+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:58.995752+0000","scrub_duration":0.0029466150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":318,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.817593+0000","last_change":"2024-09-10T18:35:01.711169+0000","last_active":"2024-09-10T18:35:15.817593+0000","last_peered":"2024-09-10T18:35:15.817593+0000","last_clean":"2024-09-10T18:35:15.817593+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:35:15.817593+0000","last_undegraded":"2024-09-10T18:35:15.817593+0000","last_fullsized":"2024-09-10T18:35:15.817593+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:01.710785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:01.710785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:26.695785+0000","scrub_duration":0.0033386729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":310,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.825966+0000","last_change":"2024-09-10T18:34:09.151879+0000","last_active":"2024-09-10T18:35:15.825966+0000","last_peered":"2024-09-10T18:35:15.825966+0000","last_clean":"2024-09-10T18:35:15.825966+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:35:15.825966+0000","last_undegraded":"2024-09-10T18:35:15.825966+0000","last_fullsized":"2024-09-10T18:35:15.825966+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:09.151789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:34:09.151789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002651994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'81","reported_seq":174,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.818654+0000","last_change":"2024-09-10T18:34:34.863449+0000","last_active":"2024-09-10T18:35:15.818654+0000","last_peered":"2024-09-10T18:35:15.818654+0000","last_clean":"2024-09-10T18:35:15.818654+0000","last_became_active":"2024-09-10T18:34:34.862917+0000","last_became_peered":"2024-09-10T18:34:34.862917+0000","last_unstale":"2024-09-10T18:35:15.818654+0000","last_undegraded":"2024-09-10T18:35:15.818654+0000","last_fullsized":"2024-09-10T18:35:15.818654+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:06.053455+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.053455+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003148245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":11,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":316,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.811596+0000","last_change":"2024-09-10T18:34:07.394899+0000","last_active":"2024-09-10T18:35:15.811596+0000","last_peered":"2024-09-10T18:35:15.811596+0000","last_clean":"2024-09-10T18:35:15.811596+0000","last_became_active":"2024-09-10T18:26:27.650774+0000","last_became_peered":"2024-09-10T18:26:27.650774+0000","last_unstale":"2024-09-10T18:35:15.811596+0000","last_undegraded":"2024-09-10T18:35:15.811596+0000","last_fullsized":"2024-09-10T18:35:15.811596+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:07.394835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:07.394835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001116433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":309,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.826189+0000","last_change":"2024-09-10T18:34:12.458962+0000","last_active":"2024-09-10T18:35:15.826189+0000","last_peered":"2024-09-10T18:35:15.826189+0000","last_clean":"2024-09-10T18:35:15.826189+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:35:15.826189+0000","last_undegraded":"2024-09-10T18:35:15.826189+0000","last_fullsized":"2024-09-10T18:35:15.826189+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:12.458742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.458742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:40.818274+0000","scrub_duration":0.0077313979999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"240'27","reported_seq":383,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.823603+0000","last_change":"2024-09-10T18:35:00.736438+0000","last_active":"2024-09-10T18:35:15.823603+0000","last_peered":"2024-09-10T18:35:15.823603+0000","last_clean":"2024-09-10T18:35:15.823603+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:35:15.823603+0000","last_undegraded":"2024-09-10T18:35:15.823603+0000","last_fullsized":"2024-09-10T18:35:15.823603+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:35:00.736347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:00.736347+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:16.200836+0000","scrub_duration":0.0063163400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":328,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.821579+0000","last_change":"2024-09-10T18:35:13.892354+0000","last_active":"2024-09-10T18:35:15.821579+0000","last_peered":"2024-09-10T18:35:15.821579+0000","last_clean":"2024-09-10T18:35:15.821579+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:35:15.821579+0000","last_undegraded":"2024-09-10T18:35:15.821579+0000","last_fullsized":"2024-09-10T18:35:15.821579+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:13.892271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:13.892271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:42.058900+0000","scrub_duration":0.0058623920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'81","reported_seq":245,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.825348+0000","last_change":"2024-09-10T18:34:34.862755+0000","last_active":"2024-09-10T18:35:15.825348+0000","last_peered":"2024-09-10T18:35:15.825348+0000","last_clean":"2024-09-10T18:35:15.825348+0000","last_became_active":"2024-09-10T18:34:34.855693+0000","last_became_peered":"2024-09-10T18:34:34.855693+0000","last_unstale":"2024-09-10T18:35:15.825348+0000","last_undegraded":"2024-09-10T18:35:15.825348+0000","last_fullsized":"2024-09-10T18:35:15.825348+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:08.009806+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.009806+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002902614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":315,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.827220+0000","last_change":"2024-09-10T18:34:21.344309+0000","last_active":"2024-09-10T18:35:15.827220+0000","last_peered":"2024-09-10T18:35:15.827220+0000","last_clean":"2024-09-10T18:35:15.827220+0000","last_became_active":"2024-09-10T18:26:27.640035+0000","last_became_peered":"2024-09-10T18:26:27.640035+0000","last_unstale":"2024-09-10T18:35:15.827220+0000","last_undegraded":"2024-09-10T18:35:15.827220+0000","last_fullsized":"2024-09-10T18:35:15.827220+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:21.344236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.344236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:50.779397+0000","scrub_duration":0.0040101240000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":322,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.811008+0000","last_change":"2024-09-10T18:35:04.319486+0000","last_active":"2024-09-10T18:35:15.811008+0000","last_peered":"2024-09-10T18:35:15.811008+0000","last_clean":"2024-09-10T18:35:15.811008+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:35:15.811008+0000","last_undegraded":"2024-09-10T18:35:15.811008+0000","last_fullsized":"2024-09-10T18:35:15.811008+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:04.319395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:35:04.319395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:24.169782+0000","scrub_duration":0.002865018,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'23","reported_seq":349,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.827270+0000","last_change":"2024-09-10T18:35:14.739218+0000","last_active":"2024-09-10T18:35:15.827270+0000","last_peered":"2024-09-10T18:35:15.827270+0000","last_clean":"2024-09-10T18:35:15.827270+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:35:15.827270+0000","last_undegraded":"2024-09-10T18:35:15.827270+0000","last_fullsized":"2024-09-10T18:35:15.827270+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'23","last_scrub_stamp":"2024-09-10T18:35:14.739123+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:35:14.739123+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:26.680453+0000","scrub_duration":0.0052949870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":318,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.811186+0000","last_change":"2024-09-10T18:35:08.266372+0000","last_active":"2024-09-10T18:35:15.811186+0000","last_peered":"2024-09-10T18:35:15.811186+0000","last_clean":"2024-09-10T18:35:15.811186+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:35:15.811186+0000","last_undegraded":"2024-09-10T18:35:15.811186+0000","last_fullsized":"2024-09-10T18:35:15.811186+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:08.266279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:08.266279+0000","last_clean_scrub_stamp":"2024-09-10T18:35:08.266279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:17.854497+0000","scrub_duration":0.0041938189999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'81","reported_seq":233,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.825603+0000","last_change":"2024-09-10T18:34:34.855506+0000","last_active":"2024-09-10T18:35:15.825603+0000","last_peered":"2024-09-10T18:35:15.825603+0000","last_clean":"2024-09-10T18:35:15.825603+0000","last_became_active":"2024-09-10T18:34:34.855038+0000","last_became_peered":"2024-09-10T18:34:34.855038+0000","last_unstale":"2024-09-10T18:35:15.825603+0000","last_undegraded":"2024-09-10T18:35:15.825603+0000","last_fullsized":"2024-09-10T18:35:15.825603+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:31:48.466654+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:31:48.466654+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":318,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:20.905743+0000","last_change":"2024-09-10T18:35:20.905675+0000","last_active":"2024-09-10T18:35:20.905743+0000","last_peered":"2024-09-10T18:35:20.905743+0000","last_clean":"2024-09-10T18:35:20.905743+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:35:20.905743+0000","last_undegraded":"2024-09-10T18:35:20.905743+0000","last_fullsized":"2024-09-10T18:35:20.905743+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:35:20.905576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:20.905576+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:34.600225+0000","scrub_duration":0.0049065819999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":332,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.815886+0000","last_change":"2024-09-10T18:35:09.892149+0000","last_active":"2024-09-10T18:35:15.815886+0000","last_peered":"2024-09-10T18:35:15.815886+0000","last_clean":"2024-09-10T18:35:15.815886+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:35:15.815886+0000","last_undegraded":"2024-09-10T18:35:15.815886+0000","last_fullsized":"2024-09-10T18:35:15.815886+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:09.892050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:35:09.892050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:21.477379+0000","scrub_duration":0.013937018000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'25","reported_seq":356,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.824953+0000","last_change":"2024-09-10T18:35:15.783820+0000","last_active":"2024-09-10T18:35:15.824953+0000","last_peered":"2024-09-10T18:35:15.824953+0000","last_clean":"2024-09-10T18:35:15.824953+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:35:15.824953+0000","last_undegraded":"2024-09-10T18:35:15.824953+0000","last_fullsized":"2024-09-10T18:35:15.824953+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:35:15.783731+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:35:15.783731+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:33.405736+0000","scrub_duration":0.0047918379999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"217'11","reported_seq":404,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.824418+0000","last_change":"2024-09-10T18:34:53.933334+0000","last_active":"2024-09-10T18:35:15.824418+0000","last_peered":"2024-09-10T18:35:15.824418+0000","last_clean":"2024-09-10T18:35:15.824418+0000","last_became_active":"2024-09-10T18:26:27.652489+0000","last_became_peered":"2024-09-10T18:26:27.652489+0000","last_unstale":"2024-09-10T18:35:15.824418+0000","last_undegraded":"2024-09-10T18:35:15.824418+0000","last_fullsized":"2024-09-10T18:35:15.824418+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11","last_scrub_stamp":"2024-09-10T18:34:53.933237+0000","last_deep_scrub":"217'11","last_deep_scrub_stamp":"2024-09-10T18:30:03.161985+0000","last_clean_scrub_stamp":"2024-09-10T18:34:53.933237+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:06.297756+0000","scrub_duration":0.0062113710000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"303'15880","reported_seq":16214,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:21.023844+0000","last_change":"2024-09-10T18:34:07.618780+0000","last_active":"2024-09-10T18:35:21.023844+0000","last_peered":"2024-09-10T18:35:21.023844+0000","last_clean":"2024-09-10T18:35:21.023844+0000","last_became_active":"2024-09-10T18:32:03.385428+0000","last_became_peered":"2024-09-10T18:32:03.385428+0000","last_unstale":"2024-09-10T18:35:21.023844+0000","last_undegraded":"2024-09-10T18:35:21.023844+0000","last_fullsized":"2024-09-10T18:35:21.023844+0000","mapping_epoch":246,"log_start":"287'11944","ondisk_log_start":"287'11944","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10375","last_scrub_stamp":"2024-09-10T18:34:07.618535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:07.618535+0000","objects_scrubbed":6480,"log_size":3936,"log_dups_size":0,"ondisk_log_size":3936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:25.198149+0000","scrub_duration":3.415268937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54657024,"num_objects":834,"num_object_clones":0,"num_object_copies":2502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15880,"num_write_kb":534848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":239,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.825563+0000","last_change":"2024-09-10T18:34:32.823529+0000","last_active":"2024-09-10T18:35:15.825563+0000","last_peered":"2024-09-10T18:35:15.825563+0000","last_clean":"2024-09-10T18:35:15.825563+0000","last_became_active":"2024-09-10T18:34:32.823230+0000","last_became_peered":"2024-09-10T18:34:32.823230+0000","last_unstale":"2024-09-10T18:35:15.825563+0000","last_undegraded":"2024-09-10T18:35:15.825563+0000","last_fullsized":"2024-09-10T18:35:15.825563+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:56.155354+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:56.155354+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001679647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"240'34","reported_seq":383,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.816499+0000","last_change":"2024-09-10T18:35:11.834455+0000","last_active":"2024-09-10T18:35:15.816499+0000","last_peered":"2024-09-10T18:35:15.816499+0000","last_clean":"2024-09-10T18:35:15.816499+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:35:15.816499+0000","last_undegraded":"2024-09-10T18:35:15.816499+0000","last_fullsized":"2024-09-10T18:35:15.816499+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:35:11.834362+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:35:11.834362+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:13.506612+0000","scrub_duration":0.0040598810000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":314,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.819119+0000","last_change":"2024-09-10T18:34:20.305626+0000","last_active":"2024-09-10T18:35:15.819119+0000","last_peered":"2024-09-10T18:35:15.819119+0000","last_clean":"2024-09-10T18:35:15.819119+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:35:15.819119+0000","last_undegraded":"2024-09-10T18:35:15.819119+0000","last_fullsized":"2024-09-10T18:35:15.819119+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:20.305547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:20.305547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:42.039144+0000","scrub_duration":0.0046035989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":485,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:19.925936+0000","last_change":"2024-09-10T18:34:14.495568+0000","last_active":"2024-09-10T18:35:19.925936+0000","last_peered":"2024-09-10T18:35:19.925936+0000","last_clean":"2024-09-10T18:35:19.925936+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:35:19.925936+0000","last_undegraded":"2024-09-10T18:35:19.925936+0000","last_fullsized":"2024-09-10T18:35:19.925936+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:34:14.495486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:14.495486+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:34.967733+0000","scrub_duration":0.010908085,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":317,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.826199+0000","last_change":"2024-09-10T18:34:57.526652+0000","last_active":"2024-09-10T18:35:15.826199+0000","last_peered":"2024-09-10T18:35:15.826199+0000","last_clean":"2024-09-10T18:35:15.826199+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:35:15.826199+0000","last_undegraded":"2024-09-10T18:35:15.826199+0000","last_fullsized":"2024-09-10T18:35:15.826199+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:57.526512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:34:57.526512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:24.046577+0000","scrub_duration":0.0049647529999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"303'16060","reported_seq":16254,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:25.006477+0000","last_change":"2024-09-10T18:34:56.967478+0000","last_active":"2024-09-10T18:35:25.006477+0000","last_peered":"2024-09-10T18:35:25.006477+0000","last_clean":"2024-09-10T18:35:25.006477+0000","last_became_active":"2024-09-10T18:32:03.379257+0000","last_became_peered":"2024-09-10T18:32:03.379257+0000","last_unstale":"2024-09-10T18:35:25.006477+0000","last_undegraded":"2024-09-10T18:35:25.006477+0000","last_fullsized":"2024-09-10T18:35:25.006477+0000","mapping_epoch":246,"log_start":"291'12116","ondisk_log_start":"291'12116","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'13598","last_scrub_stamp":"2024-09-10T18:34:56.967395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:56.967395+0000","objects_scrubbed":2956,"log_size":3944,"log_dups_size":0,"ondisk_log_size":3944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:10.206524+0000","scrub_duration":1.133103159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29687832,"num_objects":454,"num_object_clones":0,"num_object_copies":1362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16060,"num_write_kb":528385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":248,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.814078+0000","last_change":"2024-09-10T18:34:32.816324+0000","last_active":"2024-09-10T18:35:15.814078+0000","last_peered":"2024-09-10T18:35:15.814078+0000","last_clean":"2024-09-10T18:35:15.814078+0000","last_became_active":"2024-09-10T18:34:32.815839+0000","last_became_peered":"2024-09-10T18:34:32.815839+0000","last_unstale":"2024-09-10T18:35:15.814078+0000","last_undegraded":"2024-09-10T18:35:15.814078+0000","last_fullsized":"2024-09-10T18:35:15.814078+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:22.344759+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.344759+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002682837,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":352,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.818820+0000","last_change":"2024-09-10T18:32:59.382953+0000","last_active":"2024-09-10T18:35:15.818820+0000","last_peered":"2024-09-10T18:35:15.818820+0000","last_clean":"2024-09-10T18:35:15.818820+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:35:15.818820+0000","last_undegraded":"2024-09-10T18:35:15.818820+0000","last_fullsized":"2024-09-10T18:35:15.818820+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-09-10T18:32:59.382840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:59.382840+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0072140750000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":314,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.827817+0000","last_change":"2024-09-10T18:35:15.765590+0000","last_active":"2024-09-10T18:35:15.827817+0000","last_peered":"2024-09-10T18:35:15.827817+0000","last_clean":"2024-09-10T18:35:15.827817+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:35:15.827817+0000","last_undegraded":"2024-09-10T18:35:15.827817+0000","last_fullsized":"2024-09-10T18:35:15.827817+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:15.765392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:15.765392+0000","last_clean_scrub_stamp":"2024-09-10T18:35:15.765392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:37.786212+0000","scrub_duration":0.01380637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":318,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.810616+0000","last_change":"2024-09-10T18:35:10.270036+0000","last_active":"2024-09-10T18:35:15.810616+0000","last_peered":"2024-09-10T18:35:15.810616+0000","last_clean":"2024-09-10T18:35:15.810616+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:35:15.810616+0000","last_undegraded":"2024-09-10T18:35:15.810616+0000","last_fullsized":"2024-09-10T18:35:15.810616+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:10.269951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:35:10.269951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:21.615779+0000","scrub_duration":0.0073211559999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":306,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.810934+0000","last_change":"2024-09-10T18:34:21.322560+0000","last_active":"2024-09-10T18:35:15.810934+0000","last_peered":"2024-09-10T18:35:15.810934+0000","last_clean":"2024-09-10T18:35:15.810934+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:35:15.810934+0000","last_undegraded":"2024-09-10T18:35:15.810934+0000","last_fullsized":"2024-09-10T18:35:15.810934+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:21.322454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.322454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:50.552165+0000","scrub_duration":0.0032746329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"303'19478","reported_seq":19803,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:21.011085+0000","last_change":"2024-09-10T18:35:00.803084+0000","last_active":"2024-09-10T18:35:21.011085+0000","last_peered":"2024-09-10T18:35:21.011085+0000","last_clean":"2024-09-10T18:35:21.011085+0000","last_became_active":"2024-09-10T18:32:03.554400+0000","last_became_peered":"2024-09-10T18:32:03.554400+0000","last_unstale":"2024-09-10T18:35:21.011085+0000","last_undegraded":"2024-09-10T18:35:21.011085+0000","last_fullsized":"2024-09-10T18:35:21.011085+0000","mapping_epoch":246,"log_start":"291'15513","ondisk_log_start":"291'15513","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'17711","last_scrub_stamp":"2024-09-10T18:35:00.803002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:35:00.803002+0000","objects_scrubbed":2635,"log_size":3965,"log_dups_size":0,"ondisk_log_size":3965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:23.133375+0000","scrub_duration":1.1033014139999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53411840,"num_objects":815,"num_object_clones":0,"num_object_copies":2445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16043,"num_write_kb":539456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":229,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.828887+0000","last_change":"2024-09-10T18:35:13.749157+0000","last_active":"2024-09-10T18:35:15.828887+0000","last_peered":"2024-09-10T18:35:15.828887+0000","last_clean":"2024-09-10T18:35:15.828887+0000","last_became_active":"2024-09-10T18:34:00.166362+0000","last_became_peered":"2024-09-10T18:34:00.166362+0000","last_unstale":"2024-09-10T18:35:15.828887+0000","last_undegraded":"2024-09-10T18:35:15.828887+0000","last_fullsized":"2024-09-10T18:35:15.828887+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:13.748963+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:13.748963+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:16.266453+0000","scrub_duration":0.013370432999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"240'28","reported_seq":359,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.810860+0000","last_change":"2024-09-10T18:33:59.587714+0000","last_active":"2024-09-10T18:35:15.810860+0000","last_peered":"2024-09-10T18:35:15.810860+0000","last_clean":"2024-09-10T18:35:15.810860+0000","last_became_active":"2024-09-10T18:26:27.651364+0000","last_became_peered":"2024-09-10T18:26:27.651364+0000","last_unstale":"2024-09-10T18:35:15.810860+0000","last_undegraded":"2024-09-10T18:35:15.810860+0000","last_fullsized":"2024-09-10T18:35:15.810860+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'28","last_scrub_stamp":"2024-09-10T18:33:59.587604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:59.587604+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:21.896292+0000","scrub_duration":0.0024058019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":313,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.825320+0000","last_change":"2024-09-10T18:34:53.715596+0000","last_active":"2024-09-10T18:35:15.825320+0000","last_peered":"2024-09-10T18:35:15.825320+0000","last_clean":"2024-09-10T18:35:15.825320+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:35:15.825320+0000","last_undegraded":"2024-09-10T18:35:15.825320+0000","last_fullsized":"2024-09-10T18:35:15.825320+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:53.715509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:53.715509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:55.015864+0000","scrub_duration":0.0053715869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":475,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:19.925910+0000","last_change":"2024-09-10T18:33:07.367306+0000","last_active":"2024-09-10T18:35:19.925910+0000","last_peered":"2024-09-10T18:35:19.925910+0000","last_clean":"2024-09-10T18:35:19.925910+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:35:19.925910+0000","last_undegraded":"2024-09-10T18:35:19.925910+0000","last_fullsized":"2024-09-10T18:35:19.925910+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:33:07.367248+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:33:07.367248+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044945339999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":316,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.817936+0000","last_change":"2024-09-10T18:33:11.857892+0000","last_active":"2024-09-10T18:35:15.817936+0000","last_peered":"2024-09-10T18:35:15.817936+0000","last_clean":"2024-09-10T18:35:15.817936+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:35:15.817936+0000","last_undegraded":"2024-09-10T18:35:15.817936+0000","last_fullsized":"2024-09-10T18:35:15.817936+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:11.857685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.857685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022995870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"303'16368","reported_seq":16715,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:25.071047+0000","last_change":"2024-09-10T18:34:04.158338+0000","last_active":"2024-09-10T18:35:25.071047+0000","last_peered":"2024-09-10T18:35:25.071047+0000","last_clean":"2024-09-10T18:35:25.071047+0000","last_became_active":"2024-09-10T18:32:03.384288+0000","last_became_peered":"2024-09-10T18:32:03.384288+0000","last_unstale":"2024-09-10T18:35:25.071047+0000","last_undegraded":"2024-09-10T18:35:25.071047+0000","last_fullsized":"2024-09-10T18:35:25.071047+0000","mapping_epoch":246,"log_start":"289'12075","ondisk_log_start":"289'12075","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'10160","last_scrub_stamp":"2024-09-10T18:34:04.158266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:04.158266+0000","objects_scrubbed":6798,"log_size":4293,"log_dups_size":0,"ondisk_log_size":4293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.9688170149999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30408704,"num_objects":464,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16368,"num_write_kb":538624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":307,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-10T18:35:13.965670+0000","last_change":"2024-09-10T18:34:56.812303+0000","last_active":"2024-09-10T18:35:13.965670+0000","last_peered":"2024-09-10T18:35:13.965670+0000","last_clean":"2024-09-10T18:35:13.965670+0000","last_became_active":"2024-09-10T18:34:32.818323+0000","last_became_peered":"2024-09-10T18:34:32.818323+0000","last_unstale":"2024-09-10T18:35:13.965670+0000","last_undegraded":"2024-09-10T18:35:13.965670+0000","last_fullsized":"2024-09-10T18:35:13.965670+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:56.812219+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:56.812219+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:01.522405+0000","scrub_duration":0.0061304319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":365,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.822078+0000","last_change":"2024-09-10T18:34:59.896028+0000","last_active":"2024-09-10T18:35:15.822078+0000","last_peered":"2024-09-10T18:35:15.822078+0000","last_clean":"2024-09-10T18:35:15.822078+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:35:15.822078+0000","last_undegraded":"2024-09-10T18:35:15.822078+0000","last_fullsized":"2024-09-10T18:35:15.822078+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:34:59.895743+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:34:59.895743+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:18.992336+0000","scrub_duration":0.0031796210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":313,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.826549+0000","last_change":"2024-09-10T18:34:56.510382+0000","last_active":"2024-09-10T18:35:15.826549+0000","last_peered":"2024-09-10T18:35:15.826549+0000","last_clean":"2024-09-10T18:35:15.826549+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:35:15.826549+0000","last_undegraded":"2024-09-10T18:35:15.826549+0000","last_fullsized":"2024-09-10T18:35:15.826549+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:56.510308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:34:56.510308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:24.356371+0000","scrub_duration":0.0059794410000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"263'11","reported_seq":493,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:19.925685+0000","last_change":"2024-09-10T18:35:17.801582+0000","last_active":"2024-09-10T18:35:19.925685+0000","last_peered":"2024-09-10T18:35:19.925685+0000","last_clean":"2024-09-10T18:35:19.925685+0000","last_became_active":"2024-09-10T18:26:27.639032+0000","last_became_peered":"2024-09-10T18:26:27.639032+0000","last_unstale":"2024-09-10T18:35:19.925685+0000","last_undegraded":"2024-09-10T18:35:19.925685+0000","last_fullsized":"2024-09-10T18:35:19.925685+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'11","last_scrub_stamp":"2024-09-10T18:35:17.801395+0000","last_deep_scrub":"263'11","last_deep_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_clean_scrub_stamp":"2024-09-10T18:35:17.801395+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:21.357399+0000","scrub_duration":0.0098849520000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":307,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.825319+0000","last_change":"2024-09-10T18:33:57.483999+0000","last_active":"2024-09-10T18:35:15.825319+0000","last_peered":"2024-09-10T18:35:15.825319+0000","last_clean":"2024-09-10T18:35:15.825319+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:35:15.825319+0000","last_undegraded":"2024-09-10T18:35:15.825319+0000","last_fullsized":"2024-09-10T18:35:15.825319+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:57.483905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:57.483905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023666573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"303'16363","reported_seq":16661,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:25.030191+0000","last_change":"2024-09-10T18:35:00.666628+0000","last_active":"2024-09-10T18:35:25.030191+0000","last_peered":"2024-09-10T18:35:25.030191+0000","last_clean":"2024-09-10T18:35:25.030191+0000","last_became_active":"2024-09-10T18:32:03.554693+0000","last_became_peered":"2024-09-10T18:32:03.554693+0000","last_unstale":"2024-09-10T18:35:25.030191+0000","last_undegraded":"2024-09-10T18:35:25.030191+0000","last_fullsized":"2024-09-10T18:35:25.030191+0000","mapping_epoch":246,"log_start":"291'12447","ondisk_log_start":"291'12447","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'14233","last_scrub_stamp":"2024-09-10T18:35:00.666486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:35:00.666486+0000","objects_scrubbed":2644,"log_size":3916,"log_dups_size":0,"ondisk_log_size":3916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:23.352443+0000","scrub_duration":0.930865583,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31784960,"num_objects":485,"num_object_clones":0,"num_object_copies":1455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16363,"num_write_kb":539136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":307,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-10T18:35:13.965533+0000","last_change":"2024-09-10T18:34:54.904272+0000","last_active":"2024-09-10T18:35:13.965533+0000","last_peered":"2024-09-10T18:35:13.965533+0000","last_clean":"2024-09-10T18:35:13.965533+0000","last_became_active":"2024-09-10T18:34:32.918946+0000","last_became_peered":"2024-09-10T18:34:32.918946+0000","last_unstale":"2024-09-10T18:35:13.965533+0000","last_undegraded":"2024-09-10T18:35:13.965533+0000","last_fullsized":"2024-09-10T18:35:13.965533+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:54.904210+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:54.904210+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:55.647203+0000","scrub_duration":0.0075563590000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"240'29","reported_seq":372,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.825924+0000","last_change":"2024-09-10T18:35:10.711922+0000","last_active":"2024-09-10T18:35:15.825924+0000","last_peered":"2024-09-10T18:35:15.825924+0000","last_clean":"2024-09-10T18:35:15.825924+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:35:15.825924+0000","last_undegraded":"2024-09-10T18:35:15.825924+0000","last_fullsized":"2024-09-10T18:35:15.825924+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'29","last_scrub_stamp":"2024-09-10T18:35:10.711649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:10.711649+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:30.444538+0000","scrub_duration":0.013292415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":312,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.824640+0000","last_change":"2024-09-10T18:34:02.559656+0000","last_active":"2024-09-10T18:35:15.824640+0000","last_peered":"2024-09-10T18:35:15.824640+0000","last_clean":"2024-09-10T18:35:15.824640+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:35:15.824640+0000","last_undegraded":"2024-09-10T18:35:15.824640+0000","last_fullsized":"2024-09-10T18:35:15.824640+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:02.559431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:02.559431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019462604000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":267,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-10T18:35:13.965340+0000","last_change":"2024-09-10T18:34:22.795033+0000","last_active":"2024-09-10T18:35:13.965340+0000","last_peered":"2024-09-10T18:35:13.965340+0000","last_clean":"2024-09-10T18:35:13.965340+0000","last_became_active":"2024-09-10T18:26:27.642063+0000","last_became_peered":"2024-09-10T18:26:27.642063+0000","last_unstale":"2024-09-10T18:35:13.965340+0000","last_undegraded":"2024-09-10T18:35:13.965340+0000","last_fullsized":"2024-09-10T18:35:13.965340+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:22.794935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.794935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:45.015520+0000","scrub_duration":0.0028594010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":323,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.815677+0000","last_change":"2024-09-10T18:35:14.848040+0000","last_active":"2024-09-10T18:35:15.815677+0000","last_peered":"2024-09-10T18:35:15.815677+0000","last_clean":"2024-09-10T18:35:15.815677+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:35:15.815677+0000","last_undegraded":"2024-09-10T18:35:15.815677+0000","last_fullsized":"2024-09-10T18:35:15.815677+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:14.847962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:35:14.847962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:35.208051+0000","scrub_duration":0.0053623120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"303'19454","reported_seq":19960,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:21.114197+0000","last_change":"2024-09-10T18:34:21.239886+0000","last_active":"2024-09-10T18:35:21.114197+0000","last_peered":"2024-09-10T18:35:21.114197+0000","last_clean":"2024-09-10T18:35:21.114197+0000","last_became_active":"2024-09-10T18:32:03.381872+0000","last_became_peered":"2024-09-10T18:32:03.381872+0000","last_unstale":"2024-09-10T18:35:21.114197+0000","last_undegraded":"2024-09-10T18:35:21.114197+0000","last_fullsized":"2024-09-10T18:35:21.114197+0000","mapping_epoch":246,"log_start":"283'14452","ondisk_log_start":"283'14452","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'14889","last_scrub_stamp":"2024-09-10T18:34:21.239625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.239625+0000","objects_scrubbed":5453,"log_size":5002,"log_dups_size":0,"ondisk_log_size":5002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:26.853140+0000","scrub_duration":1.7487293830000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54067200,"num_objects":825,"num_object_clones":0,"num_object_copies":2475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15923,"num_write_kb":535936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":224,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.825801+0000","last_change":"2024-09-10T18:34:59.440417+0000","last_active":"2024-09-10T18:35:15.825801+0000","last_peered":"2024-09-10T18:35:15.825801+0000","last_clean":"2024-09-10T18:35:15.825801+0000","last_became_active":"2024-09-10T18:34:00.151017+0000","last_became_peered":"2024-09-10T18:34:00.151017+0000","last_unstale":"2024-09-10T18:35:15.825801+0000","last_undegraded":"2024-09-10T18:35:15.825801+0000","last_fullsized":"2024-09-10T18:35:15.825801+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:59.440207+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:59.440207+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:03.111983+0000","scrub_duration":0.0048255449999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"240'41","reported_seq":472,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:21.011854+0000","last_change":"2024-09-10T18:35:07.535987+0000","last_active":"2024-09-10T18:35:21.011854+0000","last_peered":"2024-09-10T18:35:21.011854+0000","last_clean":"2024-09-10T18:35:21.011854+0000","last_became_active":"2024-09-10T18:23:10.067049+0000","last_became_peered":"2024-09-10T18:23:10.067049+0000","last_unstale":"2024-09-10T18:35:21.011854+0000","last_undegraded":"2024-09-10T18:35:21.011854+0000","last_fullsized":"2024-09-10T18:35:21.011854+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'41","last_scrub_stamp":"2024-09-10T18:35:07.535910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:07.535910+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:14.513038+0000","scrub_duration":0.012729134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":319,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.811203+0000","last_change":"2024-09-10T18:34:17.387421+0000","last_active":"2024-09-10T18:35:15.811203+0000","last_peered":"2024-09-10T18:35:15.811203+0000","last_clean":"2024-09-10T18:35:15.811203+0000","last_became_active":"2024-09-10T18:26:27.648690+0000","last_became_peered":"2024-09-10T18:26:27.648690+0000","last_unstale":"2024-09-10T18:35:15.811203+0000","last_undegraded":"2024-09-10T18:35:15.811203+0000","last_fullsized":"2024-09-10T18:35:15.811203+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:17.387344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:17.387344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:33.806832+0000","scrub_duration":0.0051904949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":328,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.826058+0000","last_change":"2024-09-10T18:33:04.936495+0000","last_active":"2024-09-10T18:35:15.826058+0000","last_peered":"2024-09-10T18:35:15.826058+0000","last_clean":"2024-09-10T18:35:15.826058+0000","last_became_active":"2024-09-10T18:26:27.647576+0000","last_became_peered":"2024-09-10T18:26:27.647576+0000","last_unstale":"2024-09-10T18:35:15.826058+0000","last_undegraded":"2024-09-10T18:35:15.826058+0000","last_fullsized":"2024-09-10T18:35:15.826058+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:33:04.936407+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:33:04.936407+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096149180000000001,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":309,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.826120+0000","last_change":"2024-09-10T18:34:21.562999+0000","last_active":"2024-09-10T18:35:15.826120+0000","last_peered":"2024-09-10T18:35:15.826120+0000","last_clean":"2024-09-10T18:35:15.826120+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:35:15.826120+0000","last_undegraded":"2024-09-10T18:35:15.826120+0000","last_fullsized":"2024-09-10T18:35:15.826120+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:21.562920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.562920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:36.317098+0000","scrub_duration":0.0041137129999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":298,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.826786+0000","last_change":"2024-09-10T18:33:56.168567+0000","last_active":"2024-09-10T18:35:15.826786+0000","last_peered":"2024-09-10T18:35:15.826786+0000","last_clean":"2024-09-10T18:35:15.826786+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:35:15.826786+0000","last_undegraded":"2024-09-10T18:35:15.826786+0000","last_fullsized":"2024-09-10T18:35:15.826786+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:56.168485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:56.168485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002436723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"303'16576","reported_seq":16918,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:25.080993+0000","last_change":"2024-09-10T18:34:06.677663+0000","last_active":"2024-09-10T18:35:25.080993+0000","last_peered":"2024-09-10T18:35:25.080993+0000","last_clean":"2024-09-10T18:35:25.080993+0000","last_became_active":"2024-09-10T18:32:03.380391+0000","last_became_peered":"2024-09-10T18:32:03.380391+0000","last_unstale":"2024-09-10T18:35:25.080993+0000","last_undegraded":"2024-09-10T18:35:25.080993+0000","last_fullsized":"2024-09-10T18:35:25.080993+0000","mapping_epoch":246,"log_start":"290'12267","ondisk_log_start":"290'12267","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10475","last_scrub_stamp":"2024-09-10T18:34:06.677564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.677564+0000","objects_scrubbed":6683,"log_size":4309,"log_dups_size":0,"ondisk_log_size":4309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.5274349109999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32112640,"num_objects":490,"num_object_clones":0,"num_object_copies":1470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16576,"num_write_kb":546112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":370,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.817442+0000","last_change":"2024-09-10T18:35:08.748174+0000","last_active":"2024-09-10T18:35:15.817442+0000","last_peered":"2024-09-10T18:35:15.817442+0000","last_clean":"2024-09-10T18:35:15.817442+0000","last_became_active":"2024-09-10T18:34:00.154404+0000","last_became_peered":"2024-09-10T18:34:00.154404+0000","last_unstale":"2024-09-10T18:35:15.817442+0000","last_undegraded":"2024-09-10T18:35:15.817442+0000","last_fullsized":"2024-09-10T18:35:15.817442+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:08.748082+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:08.748082+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:34.711930+0000","scrub_duration":0.0038002869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"240'17","reported_seq":353,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.816599+0000","last_change":"2024-09-10T18:34:13.522168+0000","last_active":"2024-09-10T18:35:15.816599+0000","last_peered":"2024-09-10T18:35:15.816599+0000","last_clean":"2024-09-10T18:35:15.816599+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:35:15.816599+0000","last_undegraded":"2024-09-10T18:35:15.816599+0000","last_fullsized":"2024-09-10T18:35:15.816599+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'17","last_scrub_stamp":"2024-09-10T18:34:13.521944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:13.521944+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:35.758470+0000","scrub_duration":0.0029448339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":329,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.817040+0000","last_change":"2024-09-10T18:33:15.142062+0000","last_active":"2024-09-10T18:35:15.817040+0000","last_peered":"2024-09-10T18:35:15.817040+0000","last_clean":"2024-09-10T18:35:15.817040+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:35:15.817040+0000","last_undegraded":"2024-09-10T18:35:15.817040+0000","last_fullsized":"2024-09-10T18:35:15.817040+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:15.141982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:15.141982+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.141982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028956749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":341,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.822760+0000","last_change":"2024-09-10T18:34:06.493534+0000","last_active":"2024-09-10T18:35:15.822760+0000","last_peered":"2024-09-10T18:35:15.822760+0000","last_clean":"2024-09-10T18:35:15.822760+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:35:15.822760+0000","last_undegraded":"2024-09-10T18:35:15.822760+0000","last_fullsized":"2024-09-10T18:35:15.822760+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:34:06.493448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.493448+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:21.109643+0000","scrub_duration":0.011827739,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":328,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.822681+0000","last_change":"2024-09-10T18:34:59.713703+0000","last_active":"2024-09-10T18:35:15.822681+0000","last_peered":"2024-09-10T18:35:15.822681+0000","last_clean":"2024-09-10T18:35:15.822681+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:35:15.822681+0000","last_undegraded":"2024-09-10T18:35:15.822681+0000","last_fullsized":"2024-09-10T18:35:15.822681+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:59.713536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:34:59.713536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:21.888517+0000","scrub_duration":0.002321769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":316,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.822926+0000","last_change":"2024-09-10T18:34:51.933409+0000","last_active":"2024-09-10T18:35:15.822926+0000","last_peered":"2024-09-10T18:35:15.822926+0000","last_clean":"2024-09-10T18:35:15.822926+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:35:15.822926+0000","last_undegraded":"2024-09-10T18:35:15.822926+0000","last_fullsized":"2024-09-10T18:35:15.822926+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:51.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:51.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:06.988199+0000","scrub_duration":0.0029402349999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"303'16091","reported_seq":16445,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:21.111105+0000","last_change":"2024-09-10T18:34:01.902715+0000","last_active":"2024-09-10T18:35:21.111105+0000","last_peered":"2024-09-10T18:35:21.111105+0000","last_clean":"2024-09-10T18:35:21.111105+0000","last_became_active":"2024-09-10T18:32:03.554958+0000","last_became_peered":"2024-09-10T18:32:03.554958+0000","last_unstale":"2024-09-10T18:35:21.111105+0000","last_undegraded":"2024-09-10T18:35:21.111105+0000","last_fullsized":"2024-09-10T18:35:21.111105+0000","mapping_epoch":246,"log_start":"283'11038","ondisk_log_start":"283'11038","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'10034","last_scrub_stamp":"2024-09-10T18:34:01.902618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:01.902618+0000","objects_scrubbed":6997,"log_size":5053,"log_dups_size":0,"ondisk_log_size":5053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.3941321589999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55771136,"num_objects":851,"num_object_clones":0,"num_object_copies":2553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16091,"num_write_kb":542144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":239,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.816638+0000","last_change":"2024-09-10T18:35:01.969623+0000","last_active":"2024-09-10T18:35:15.816638+0000","last_peered":"2024-09-10T18:35:15.816638+0000","last_clean":"2024-09-10T18:35:15.816638+0000","last_became_active":"2024-09-10T18:34:00.165384+0000","last_became_peered":"2024-09-10T18:34:00.165384+0000","last_unstale":"2024-09-10T18:35:15.816638+0000","last_undegraded":"2024-09-10T18:35:15.816638+0000","last_fullsized":"2024-09-10T18:35:15.816638+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:01.969528+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:01.969528+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:07.426889+0000","scrub_duration":0.094075444999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":347,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.825050+0000","last_change":"2024-09-10T18:35:03.798928+0000","last_active":"2024-09-10T18:35:15.825050+0000","last_peered":"2024-09-10T18:35:15.825050+0000","last_clean":"2024-09-10T18:35:15.825050+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:35:15.825050+0000","last_undegraded":"2024-09-10T18:35:15.825050+0000","last_fullsized":"2024-09-10T18:35:15.825050+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-09-10T18:35:03.798837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:03.798837+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:09.218914+0000","scrub_duration":0.0065582729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":323,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.824845+0000","last_change":"2024-09-10T18:35:02.838571+0000","last_active":"2024-09-10T18:35:15.824845+0000","last_peered":"2024-09-10T18:35:15.824845+0000","last_clean":"2024-09-10T18:35:15.824845+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:35:15.824845+0000","last_undegraded":"2024-09-10T18:35:15.824845+0000","last_fullsized":"2024-09-10T18:35:15.824845+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:02.838489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:35:02.838489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:25.891598+0000","scrub_duration":0.0090996949999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":1156,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:23.896253+0000","last_change":"2024-09-10T18:34:02.136505+0000","last_active":"2024-09-10T18:35:23.896253+0000","last_peered":"2024-09-10T18:35:23.896253+0000","last_clean":"2024-09-10T18:35:23.896253+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:35:23.896253+0000","last_undegraded":"2024-09-10T18:35:23.896253+0000","last_fullsized":"2024-09-10T18:35:23.896253+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:34:02.136429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:02.136429+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:32.048652+0000","scrub_duration":0.0083841149999999993,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":826,"num_read_kb":821,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":312,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.828891+0000","last_change":"2024-09-10T18:35:12.568766+0000","last_active":"2024-09-10T18:35:15.828891+0000","last_peered":"2024-09-10T18:35:15.828891+0000","last_clean":"2024-09-10T18:35:15.828891+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:35:15.828891+0000","last_undegraded":"2024-09-10T18:35:15.828891+0000","last_fullsized":"2024-09-10T18:35:15.828891+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:12.568627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.771415+0000","last_clean_scrub_stamp":"2024-09-10T18:35:12.568627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:16.053800+0000","scrub_duration":0.0022788050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":306,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.823817+0000","last_change":"2024-09-10T18:34:12.539195+0000","last_active":"2024-09-10T18:35:15.823817+0000","last_peered":"2024-09-10T18:35:15.823817+0000","last_clean":"2024-09-10T18:35:15.823817+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:35:15.823817+0000","last_undegraded":"2024-09-10T18:35:15.823817+0000","last_fullsized":"2024-09-10T18:35:15.823817+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:12.539124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.539124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:33.017675+0000","scrub_duration":0.0079965020000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"303'15726","reported_seq":16024,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:21.018334+0000","last_change":"2024-09-10T18:34:19.206406+0000","last_active":"2024-09-10T18:35:21.018334+0000","last_peered":"2024-09-10T18:35:21.018334+0000","last_clean":"2024-09-10T18:35:21.018334+0000","last_became_active":"2024-09-10T18:32:03.382335+0000","last_became_peered":"2024-09-10T18:32:03.382335+0000","last_unstale":"2024-09-10T18:35:21.018334+0000","last_undegraded":"2024-09-10T18:35:21.018334+0000","last_fullsized":"2024-09-10T18:35:21.018334+0000","mapping_epoch":246,"log_start":"287'11764","ondisk_log_start":"287'11764","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'11048","last_scrub_stamp":"2024-09-10T18:34:19.206308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.206308+0000","objects_scrubbed":5584,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:45.982181+0000","scrub_duration":1.913810319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56754176,"num_objects":866,"num_object_clones":0,"num_object_copies":2598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15726,"num_write_kb":530944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'26","reported_seq":304,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-10T18:35:13.965128+0000","last_change":"2024-09-10T18:34:19.788942+0000","last_active":"2024-09-10T18:35:13.965128+0000","last_peered":"2024-09-10T18:35:13.965128+0000","last_clean":"2024-09-10T18:35:13.965128+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:35:13.965128+0000","last_undegraded":"2024-09-10T18:35:13.965128+0000","last_fullsized":"2024-09-10T18:35:13.965128+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'26","last_scrub_stamp":"2024-09-10T18:34:19.788853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.788853+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:26.351175+0000","scrub_duration":0.0020676890000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":321,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.826671+0000","last_change":"2024-09-10T18:34:19.296515+0000","last_active":"2024-09-10T18:35:15.826671+0000","last_peered":"2024-09-10T18:35:15.826671+0000","last_clean":"2024-09-10T18:35:15.826671+0000","last_became_active":"2024-09-10T18:26:27.639129+0000","last_became_peered":"2024-09-10T18:26:27.639129+0000","last_unstale":"2024-09-10T18:35:15.826671+0000","last_undegraded":"2024-09-10T18:35:15.826671+0000","last_fullsized":"2024-09-10T18:35:15.826671+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:19.296286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.296286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:35.735259+0000","scrub_duration":0.0049170170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":382,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.816931+0000","last_change":"2024-09-10T18:35:05.724010+0000","last_active":"2024-09-10T18:35:15.816931+0000","last_peered":"2024-09-10T18:35:15.816931+0000","last_clean":"2024-09-10T18:35:15.816931+0000","last_became_active":"2024-09-10T18:34:00.159975+0000","last_became_peered":"2024-09-10T18:34:00.159975+0000","last_unstale":"2024-09-10T18:35:15.816931+0000","last_undegraded":"2024-09-10T18:35:15.816931+0000","last_fullsized":"2024-09-10T18:35:15.816931+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:05.723917+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:05.723917+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:21.234670+0000","scrub_duration":0.0022326540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":318,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.817459+0000","last_change":"2024-09-10T18:34:12.920724+0000","last_active":"2024-09-10T18:35:15.817459+0000","last_peered":"2024-09-10T18:35:15.817459+0000","last_clean":"2024-09-10T18:35:15.817459+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:35:15.817459+0000","last_undegraded":"2024-09-10T18:35:15.817459+0000","last_fullsized":"2024-09-10T18:35:15.817459+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:12.920651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.920651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:29.441715+0000","scrub_duration":0.0047636090000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":310,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.828796+0000","last_change":"2024-09-10T18:35:03.488408+0000","last_active":"2024-09-10T18:35:15.828796+0000","last_peered":"2024-09-10T18:35:15.828796+0000","last_clean":"2024-09-10T18:35:15.828796+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:35:15.828796+0000","last_undegraded":"2024-09-10T18:35:15.828796+0000","last_fullsized":"2024-09-10T18:35:15.828796+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:03.488360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:35:03.488360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:15.135119+0000","scrub_duration":0.0061095339999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"76'81","reported_seq":258,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.894803+0000","last_change":"2024-09-10T18:35:15.894737+0000","last_active":"2024-09-10T18:35:15.894803+0000","last_peered":"2024-09-10T18:35:15.894803+0000","last_clean":"2024-09-10T18:35:15.894803+0000","last_became_active":"2024-09-10T18:34:32.927460+0000","last_became_peered":"2024-09-10T18:34:32.927460+0000","last_unstale":"2024-09-10T18:35:15.894803+0000","last_undegraded":"2024-09-10T18:35:15.894803+0000","last_fullsized":"2024-09-10T18:35:15.894803+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:15.894656+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:15.894656+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:40.273824+0000","scrub_duration":0.013574714,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"303'16358","reported_seq":16651,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:25.013937+0000","last_change":"2024-09-10T18:34:22.669084+0000","last_active":"2024-09-10T18:35:25.013937+0000","last_peered":"2024-09-10T18:35:25.013937+0000","last_clean":"2024-09-10T18:35:25.013937+0000","last_became_active":"2024-09-10T18:32:03.384247+0000","last_became_peered":"2024-09-10T18:32:03.384247+0000","last_unstale":"2024-09-10T18:35:25.013937+0000","last_undegraded":"2024-09-10T18:35:25.013937+0000","last_fullsized":"2024-09-10T18:35:25.013937+0000","mapping_epoch":246,"log_start":"291'12388","ondisk_log_start":"291'12388","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'11469","last_scrub_stamp":"2024-09-10T18:34:22.669027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.669027+0000","objects_scrubbed":5424,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:26.515481+0000","scrub_duration":1.750744868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31457280,"num_objects":480,"num_object_clones":0,"num_object_copies":1440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16358,"num_write_kb":538816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"283'24","reported_seq":356,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.811467+0000","last_change":"2024-09-10T18:34:19.404796+0000","last_active":"2024-09-10T18:35:15.811467+0000","last_peered":"2024-09-10T18:35:15.811467+0000","last_clean":"2024-09-10T18:35:15.811467+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:35:15.811467+0000","last_undegraded":"2024-09-10T18:35:15.811467+0000","last_fullsized":"2024-09-10T18:35:15.811467+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'24","last_scrub_stamp":"2024-09-10T18:34:19.404713+0000","last_deep_scrub":"283'24","last_deep_scrub_stamp":"2024-09-10T18:34:19.404713+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.404713+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:30.068249+0000","scrub_duration":0.037546765000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":312,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.811114+0000","last_change":"2024-09-10T18:34:20.349520+0000","last_active":"2024-09-10T18:35:15.811114+0000","last_peered":"2024-09-10T18:35:15.811114+0000","last_clean":"2024-09-10T18:35:15.811114+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:35:15.811114+0000","last_undegraded":"2024-09-10T18:35:15.811114+0000","last_fullsized":"2024-09-10T18:35:15.811114+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:20.349464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:20.349464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:31.859829+0000","scrub_duration":0.0036650300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":326,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.828711+0000","last_change":"2024-09-10T18:35:08.562624+0000","last_active":"2024-09-10T18:35:15.828711+0000","last_peered":"2024-09-10T18:35:15.828711+0000","last_clean":"2024-09-10T18:35:15.828711+0000","last_became_active":"2024-09-10T18:26:27.645885+0000","last_became_peered":"2024-09-10T18:26:27.645885+0000","last_unstale":"2024-09-10T18:35:15.828711+0000","last_undegraded":"2024-09-10T18:35:15.828711+0000","last_fullsized":"2024-09-10T18:35:15.828711+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:08.562551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:35:08.562551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:32.617261+0000","scrub_duration":0.0076930669999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":315,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.828963+0000","last_change":"2024-09-10T18:34:01.181010+0000","last_active":"2024-09-10T18:35:15.828963+0000","last_peered":"2024-09-10T18:35:15.828963+0000","last_clean":"2024-09-10T18:35:15.828963+0000","last_became_active":"2024-09-10T18:26:27.632071+0000","last_became_peered":"2024-09-10T18:26:27.632071+0000","last_unstale":"2024-09-10T18:35:15.828963+0000","last_undegraded":"2024-09-10T18:35:15.828963+0000","last_fullsized":"2024-09-10T18:35:15.828963+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:01.180920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:01.180920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082996130000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"303'16511","reported_seq":16974,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:25.516489+0000","last_change":"2024-09-10T18:34:07.700772+0000","last_active":"2024-09-10T18:35:25.516489+0000","last_peered":"2024-09-10T18:35:25.516489+0000","last_clean":"2024-09-10T18:35:25.516489+0000","last_became_active":"2024-09-10T18:32:03.554412+0000","last_became_peered":"2024-09-10T18:32:03.554412+0000","last_unstale":"2024-09-10T18:35:25.516489+0000","last_undegraded":"2024-09-10T18:35:25.516489+0000","last_fullsized":"2024-09-10T18:35:25.516489+0000","mapping_epoch":246,"log_start":"291'12401","ondisk_log_start":"291'12401","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'10569","last_scrub_stamp":"2024-09-10T18:34:07.700574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:07.700574+0000","objects_scrubbed":6543,"log_size":4110,"log_dups_size":0,"ondisk_log_size":4110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:32.220981+0000","scrub_duration":3.13969619,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29687808,"num_objects":453,"num_object_clones":0,"num_object_copies":1359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16511,"num_write_kb":542848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":231,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.825070+0000","last_change":"2024-09-10T18:34:32.824505+0000","last_active":"2024-09-10T18:35:15.825070+0000","last_peered":"2024-09-10T18:35:15.825070+0000","last_clean":"2024-09-10T18:35:15.825070+0000","last_became_active":"2024-09-10T18:34:32.823976+0000","last_became_peered":"2024-09-10T18:34:32.823976+0000","last_unstale":"2024-09-10T18:35:15.825070+0000","last_undegraded":"2024-09-10T18:35:15.825070+0000","last_fullsized":"2024-09-10T18:35:15.825070+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:08.009806+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.009806+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002902614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":359,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.811677+0000","last_change":"2024-09-10T18:34:00.542831+0000","last_active":"2024-09-10T18:35:15.811677+0000","last_peered":"2024-09-10T18:35:15.811677+0000","last_clean":"2024-09-10T18:35:15.811677+0000","last_became_active":"2024-09-10T18:21:33.611203+0000","last_became_peered":"2024-09-10T18:21:33.611203+0000","last_unstale":"2024-09-10T18:35:15.811677+0000","last_undegraded":"2024-09-10T18:35:15.811677+0000","last_fullsized":"2024-09-10T18:35:15.811677+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-09-10T18:34:00.542737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:00.542737+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030351190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":275,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-10T18:35:13.965600+0000","last_change":"2024-09-10T18:34:09.980779+0000","last_active":"2024-09-10T18:35:13.965600+0000","last_peered":"2024-09-10T18:35:13.965600+0000","last_clean":"2024-09-10T18:35:13.965600+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:35:13.965600+0000","last_undegraded":"2024-09-10T18:35:13.965600+0000","last_fullsized":"2024-09-10T18:35:13.965600+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:34:09.980722+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:34:09.980722+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:29.815756+0000","scrub_duration":0.001388053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":318,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.823901+0000","last_change":"2024-09-10T18:34:53.794746+0000","last_active":"2024-09-10T18:35:15.823901+0000","last_peered":"2024-09-10T18:35:15.823901+0000","last_clean":"2024-09-10T18:35:15.823901+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:35:15.823901+0000","last_undegraded":"2024-09-10T18:35:15.823901+0000","last_fullsized":"2024-09-10T18:35:15.823901+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:53.794668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:53.794668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:54.333239+0000","scrub_duration":0.0026034389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":336,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.824195+0000","last_change":"2024-09-10T18:34:20.486108+0000","last_active":"2024-09-10T18:35:15.824195+0000","last_peered":"2024-09-10T18:35:15.824195+0000","last_clean":"2024-09-10T18:35:15.824195+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:35:15.824195+0000","last_undegraded":"2024-09-10T18:35:15.824195+0000","last_fullsized":"2024-09-10T18:35:15.824195+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:20.486034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:20.486034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:45.434346+0000","scrub_duration":0.0026235970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"303'16083","reported_seq":16404,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:21.020338+0000","last_change":"2024-09-10T18:34:12.213950+0000","last_active":"2024-09-10T18:35:21.020338+0000","last_peered":"2024-09-10T18:35:21.020338+0000","last_clean":"2024-09-10T18:35:21.020338+0000","last_became_active":"2024-09-10T18:32:03.554475+0000","last_became_peered":"2024-09-10T18:32:03.554475+0000","last_unstale":"2024-09-10T18:35:21.020338+0000","last_undegraded":"2024-09-10T18:35:21.020338+0000","last_fullsized":"2024-09-10T18:35:21.020338+0000","mapping_epoch":246,"log_start":"288'12163","ondisk_log_start":"288'12163","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'10828","last_scrub_stamp":"2024-09-10T18:34:12.213852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.213852+0000","objects_scrubbed":6169,"log_size":3920,"log_dups_size":0,"ondisk_log_size":3920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:28.363825+0000","scrub_duration":3.0156890789999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54460416,"num_objects":831,"num_object_clones":0,"num_object_copies":2493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16083,"num_write_kb":541248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":370,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.818316+0000","last_change":"2024-09-10T18:34:32.827812+0000","last_active":"2024-09-10T18:35:15.818316+0000","last_peered":"2024-09-10T18:35:15.818316+0000","last_clean":"2024-09-10T18:35:15.818316+0000","last_became_active":"2024-09-10T18:34:32.827174+0000","last_became_peered":"2024-09-10T18:34:32.827174+0000","last_unstale":"2024-09-10T18:35:15.818316+0000","last_undegraded":"2024-09-10T18:35:15.818316+0000","last_fullsized":"2024-09-10T18:35:15.818316+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:06.053455+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.053455+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007740935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"240'25","reported_seq":80,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.816576+0000","last_change":"2024-09-10T18:35:04.715386+0000","last_active":"2024-09-10T18:35:15.816576+0000","last_peered":"2024-09-10T18:35:15.816576+0000","last_clean":"2024-09-10T18:35:15.816576+0000","last_became_active":"2024-09-10T18:32:51.118251+0000","last_became_peered":"2024-09-10T18:32:51.118251+0000","last_unstale":"2024-09-10T18:35:15.816576+0000","last_undegraded":"2024-09-10T18:35:15.816576+0000","last_fullsized":"2024-09-10T18:35:15.816576+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:35:04.715302+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:35:04.715302+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:22.286903+0000","scrub_duration":0.0065951120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":304,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.828063+0000","last_change":"2024-09-10T18:33:13.324775+0000","last_active":"2024-09-10T18:35:15.828063+0000","last_peered":"2024-09-10T18:35:15.828063+0000","last_clean":"2024-09-10T18:35:15.828063+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:35:15.828063+0000","last_undegraded":"2024-09-10T18:35:15.828063+0000","last_fullsized":"2024-09-10T18:35:15.828063+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:13.324556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:13.324556+0000","last_clean_scrub_stamp":"2024-09-10T18:33:13.324556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011863357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"189'9","reported_seq":514,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:21.018176+0000","last_change":"2024-09-10T18:34:17.470904+0000","last_active":"2024-09-10T18:35:21.018176+0000","last_peered":"2024-09-10T18:35:21.018176+0000","last_clean":"2024-09-10T18:35:21.018176+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:35:21.018176+0000","last_undegraded":"2024-09-10T18:35:21.018176+0000","last_fullsized":"2024-09-10T18:35:21.018176+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'9","last_scrub_stamp":"2024-09-10T18:34:17.470793+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:34:17.470793+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044525839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":309,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.811345+0000","last_change":"2024-09-10T18:35:07.288756+0000","last_active":"2024-09-10T18:35:15.811345+0000","last_peered":"2024-09-10T18:35:15.811345+0000","last_clean":"2024-09-10T18:35:15.811345+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:35:15.811345+0000","last_undegraded":"2024-09-10T18:35:15.811345+0000","last_fullsized":"2024-09-10T18:35:15.811345+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:07.288672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_clean_scrub_stamp":"2024-09-10T18:35:07.288672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:34.906817+0000","scrub_duration":0.0041335299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"76'81","reported_seq":42,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.811360+0000","last_change":"2024-09-10T18:34:32.818578+0000","last_active":"2024-09-10T18:35:15.811360+0000","last_peered":"2024-09-10T18:35:15.811360+0000","last_clean":"2024-09-10T18:35:15.811360+0000","last_became_active":"2024-09-10T18:34:32.818251+0000","last_became_peered":"2024-09-10T18:34:32.818251+0000","last_unstale":"2024-09-10T18:35:15.811360+0000","last_undegraded":"2024-09-10T18:35:15.811360+0000","last_fullsized":"2024-09-10T18:35:15.811360+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:48.006695+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:48.006695+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":5,"num_write":2,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"303'21823","reported_seq":22111,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:25.079817+0000","last_change":"2024-09-10T18:34:37.737676+0000","last_active":"2024-09-10T18:35:25.079817+0000","last_peered":"2024-09-10T18:35:25.079817+0000","last_clean":"2024-09-10T18:35:25.079817+0000","last_became_active":"2024-09-10T18:32:03.384417+0000","last_became_peered":"2024-09-10T18:32:03.384417+0000","last_unstale":"2024-09-10T18:35:25.079817+0000","last_undegraded":"2024-09-10T18:35:25.079817+0000","last_fullsized":"2024-09-10T18:35:25.079817+0000","mapping_epoch":246,"log_start":"291'17504","ondisk_log_start":"291'17504","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'17735","last_scrub_stamp":"2024-09-10T18:34:37.737496+0000","last_deep_scrub":"291'17735","last_deep_scrub_stamp":"2024-09-10T18:34:37.737496+0000","last_clean_scrub_stamp":"2024-09-10T18:34:37.737496+0000","objects_scrubbed":5250,"log_size":4319,"log_dups_size":0,"ondisk_log_size":4319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":26,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:45.542594+0000","scrub_duration":25.514376305999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31129600,"num_objects":475,"num_object_clones":0,"num_object_copies":1425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16573,"num_write_kb":545536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"302'19","reported_seq":340,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:18.836319+0000","last_change":"2024-09-10T18:35:18.836229+0000","last_active":"2024-09-10T18:35:18.836319+0000","last_peered":"2024-09-10T18:35:18.836319+0000","last_clean":"2024-09-10T18:35:18.836319+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:35:18.836319+0000","last_undegraded":"2024-09-10T18:35:18.836319+0000","last_fullsized":"2024-09-10T18:35:18.836319+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'19","last_scrub_stamp":"2024-09-10T18:35:18.836131+0000","last_deep_scrub":"302'19","last_deep_scrub_stamp":"2024-09-10T18:35:18.836131+0000","last_clean_scrub_stamp":"2024-09-10T18:35:18.836131+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:41.789867+0000","scrub_duration":0.019271303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"265'2","reported_seq":340,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.825805+0000","last_change":"2024-09-10T18:35:14.515615+0000","last_active":"2024-09-10T18:35:15.825805+0000","last_peered":"2024-09-10T18:35:15.825805+0000","last_clean":"2024-09-10T18:35:15.825805+0000","last_became_active":"2024-09-10T18:26:27.644997+0000","last_became_peered":"2024-09-10T18:26:27.644997+0000","last_unstale":"2024-09-10T18:35:15.825805+0000","last_undegraded":"2024-09-10T18:35:15.825805+0000","last_fullsized":"2024-09-10T18:35:15.825805+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'2","last_scrub_stamp":"2024-09-10T18:35:14.515533+0000","last_deep_scrub":"265'2","last_deep_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_clean_scrub_stamp":"2024-09-10T18:35:14.515533+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:38.258111+0000","scrub_duration":0.0046328369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":334,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.826728+0000","last_change":"2024-09-10T18:34:58.759217+0000","last_active":"2024-09-10T18:35:15.826728+0000","last_peered":"2024-09-10T18:35:15.826728+0000","last_clean":"2024-09-10T18:35:15.826728+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:35:15.826728+0000","last_undegraded":"2024-09-10T18:35:15.826728+0000","last_fullsized":"2024-09-10T18:35:15.826728+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:58.759108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:58.759108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:26.795742+0000","scrub_duration":0.010436621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":307,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.824773+0000","last_change":"2024-09-10T18:34:56.786523+0000","last_active":"2024-09-10T18:35:15.824773+0000","last_peered":"2024-09-10T18:35:15.824773+0000","last_clean":"2024-09-10T18:35:15.824773+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:35:15.824773+0000","last_undegraded":"2024-09-10T18:35:15.824773+0000","last_fullsized":"2024-09-10T18:35:15.824773+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:56.786436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:56.786436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:02.364716+0000","scrub_duration":0.0050134400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'81","reported_seq":296,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-10T18:35:13.965747+0000","last_change":"2024-09-10T18:34:15.938135+0000","last_active":"2024-09-10T18:35:13.965747+0000","last_peered":"2024-09-10T18:35:13.965747+0000","last_clean":"2024-09-10T18:35:13.965747+0000","last_became_active":"2024-09-10T18:34:02.803861+0000","last_became_peered":"2024-09-10T18:34:02.803861+0000","last_unstale":"2024-09-10T18:35:13.965747+0000","last_undegraded":"2024-09-10T18:35:13.965747+0000","last_fullsized":"2024-09-10T18:35:13.965747+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:15.938079+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:34:15.938079+0000","last_clean_scrub_stamp":"2024-09-10T18:34:15.938079+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:40.264047+0000","scrub_duration":0.0015442819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"303'16292","reported_seq":16590,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:25.077939+0000","last_change":"2024-09-10T18:34:19.945087+0000","last_active":"2024-09-10T18:35:25.077939+0000","last_peered":"2024-09-10T18:35:25.077939+0000","last_clean":"2024-09-10T18:35:25.077939+0000","last_became_active":"2024-09-10T18:32:03.382430+0000","last_became_peered":"2024-09-10T18:32:03.382430+0000","last_unstale":"2024-09-10T18:35:25.077939+0000","last_undegraded":"2024-09-10T18:35:25.077939+0000","last_fullsized":"2024-09-10T18:35:25.077939+0000","mapping_epoch":246,"log_start":"287'11913","ondisk_log_start":"287'11913","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'11233","last_scrub_stamp":"2024-09-10T18:34:19.945029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.945029+0000","objects_scrubbed":5568,"log_size":4379,"log_dups_size":0,"ondisk_log_size":4379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:38.722375+0000","scrub_duration":1.7273602640000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30015488,"num_objects":458,"num_object_clones":0,"num_object_copies":1374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16292,"num_write_kb":536000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"240'44","reported_seq":392,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-10T18:35:13.965418+0000","last_change":"2024-09-10T18:34:03.021478+0000","last_active":"2024-09-10T18:35:13.965418+0000","last_peered":"2024-09-10T18:35:13.965418+0000","last_clean":"2024-09-10T18:35:13.965418+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:35:13.965418+0000","last_undegraded":"2024-09-10T18:35:13.965418+0000","last_fullsized":"2024-09-10T18:35:13.965418+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'44","last_scrub_stamp":"2024-09-10T18:34:03.021369+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:34:03.021369+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045216129999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":329,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.823289+0000","last_change":"2024-09-10T18:34:11.965336+0000","last_active":"2024-09-10T18:35:15.823289+0000","last_peered":"2024-09-10T18:35:15.823289+0000","last_clean":"2024-09-10T18:35:15.823289+0000","last_became_active":"2024-09-10T18:26:27.638408+0000","last_became_peered":"2024-09-10T18:26:27.638408+0000","last_unstale":"2024-09-10T18:35:15.823289+0000","last_undegraded":"2024-09-10T18:35:15.823289+0000","last_fullsized":"2024-09-10T18:35:15.823289+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:11.965263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:11.965263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:21.204938+0000","scrub_duration":0.0034110310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":338,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.823995+0000","last_change":"2024-09-10T18:34:21.447378+0000","last_active":"2024-09-10T18:35:15.823995+0000","last_peered":"2024-09-10T18:35:15.823995+0000","last_clean":"2024-09-10T18:35:15.823995+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:35:15.823995+0000","last_undegraded":"2024-09-10T18:35:15.823995+0000","last_fullsized":"2024-09-10T18:35:15.823995+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:21.447291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:34:21.447291+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.447291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:25.350687+0000","scrub_duration":0.0066377579999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":319,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.816851+0000","last_change":"2024-09-10T18:35:02.706333+0000","last_active":"2024-09-10T18:35:15.816851+0000","last_peered":"2024-09-10T18:35:15.816851+0000","last_clean":"2024-09-10T18:35:15.816851+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:35:15.816851+0000","last_undegraded":"2024-09-10T18:35:15.816851+0000","last_fullsized":"2024-09-10T18:35:15.816851+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:02.706248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:02.706248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:17.386012+0000","scrub_duration":0.0066128610000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"76'81","reported_seq":245,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.829156+0000","last_change":"2024-09-10T18:34:32.957284+0000","last_active":"2024-09-10T18:35:15.829156+0000","last_peered":"2024-09-10T18:35:15.829156+0000","last_clean":"2024-09-10T18:35:15.829156+0000","last_became_active":"2024-09-10T18:34:32.956761+0000","last_became_peered":"2024-09-10T18:34:32.956761+0000","last_unstale":"2024-09-10T18:35:15.829156+0000","last_undegraded":"2024-09-10T18:35:15.829156+0000","last_fullsized":"2024-09-10T18:35:15.829156+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.370561+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.370561+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029251390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54613,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"303'16364","reported_seq":16715,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:25.513741+0000","last_change":"2024-09-10T18:34:01.964247+0000","last_active":"2024-09-10T18:35:25.513741+0000","last_peered":"2024-09-10T18:35:25.513741+0000","last_clean":"2024-09-10T18:35:25.513741+0000","last_became_active":"2024-09-10T18:32:03.554423+0000","last_became_peered":"2024-09-10T18:32:03.554423+0000","last_unstale":"2024-09-10T18:35:25.513741+0000","last_undegraded":"2024-09-10T18:35:25.513741+0000","last_fullsized":"2024-09-10T18:35:25.513741+0000","mapping_epoch":246,"log_start":"291'12162","ondisk_log_start":"291'12162","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'10000","last_scrub_stamp":"2024-09-10T18:34:01.964158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:01.964158+0000","objects_scrubbed":6911,"log_size":4202,"log_dups_size":0,"ondisk_log_size":4202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.4154075100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27262976,"num_objects":416,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16364,"num_write_kb":536960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"301'23","reported_seq":360,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.810698+0000","last_change":"2024-09-10T18:35:15.283425+0000","last_active":"2024-09-10T18:35:15.810698+0000","last_peered":"2024-09-10T18:35:15.810698+0000","last_clean":"2024-09-10T18:35:15.810698+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:35:15.810698+0000","last_undegraded":"2024-09-10T18:35:15.810698+0000","last_fullsized":"2024-09-10T18:35:15.810698+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'23","last_scrub_stamp":"2024-09-10T18:35:15.283373+0000","last_deep_scrub":"301'23","last_deep_scrub_stamp":"2024-09-10T18:35:15.283373+0000","last_clean_scrub_stamp":"2024-09-10T18:35:15.283373+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:25.670747+0000","scrub_duration":0.018617538999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":322,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.820482+0000","last_change":"2024-09-10T18:33:14.879881+0000","last_active":"2024-09-10T18:35:15.820482+0000","last_peered":"2024-09-10T18:35:15.820482+0000","last_clean":"2024-09-10T18:35:15.820482+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:35:15.820482+0000","last_undegraded":"2024-09-10T18:35:15.820482+0000","last_fullsized":"2024-09-10T18:35:15.820482+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:14.879787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:33:14.879787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044267550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":338,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.827313+0000","last_change":"2024-09-10T18:34:57.751011+0000","last_active":"2024-09-10T18:35:15.827313+0000","last_peered":"2024-09-10T18:35:15.827313+0000","last_clean":"2024-09-10T18:35:15.827313+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:35:15.827313+0000","last_undegraded":"2024-09-10T18:35:15.827313+0000","last_fullsized":"2024-09-10T18:35:15.827313+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:57.750908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:57.750908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:03.127730+0000","scrub_duration":0.015397398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":298,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.826535+0000","last_change":"2024-09-10T18:32:58.358775+0000","last_active":"2024-09-10T18:35:15.826535+0000","last_peered":"2024-09-10T18:35:15.826535+0000","last_clean":"2024-09-10T18:35:15.826535+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:35:15.826535+0000","last_undegraded":"2024-09-10T18:35:15.826535+0000","last_fullsized":"2024-09-10T18:35:15.826535+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:32:58.358694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.803236+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.358694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011821898000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"76'81","reported_seq":42,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.819393+0000","last_change":"2024-09-10T18:34:32.822681+0000","last_active":"2024-09-10T18:35:15.819393+0000","last_peered":"2024-09-10T18:35:15.819393+0000","last_clean":"2024-09-10T18:35:15.819393+0000","last_became_active":"2024-09-10T18:34:32.822196+0000","last_became_peered":"2024-09-10T18:34:32.822196+0000","last_unstale":"2024-09-10T18:35:15.819393+0000","last_undegraded":"2024-09-10T18:35:15.819393+0000","last_fullsized":"2024-09-10T18:35:15.819393+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:58.561425+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:58.561425+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":5,"num_write":2,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"303'15937","reported_seq":16254,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:21.095493+0000","last_change":"2024-09-10T18:34:12.987347+0000","last_active":"2024-09-10T18:35:21.095493+0000","last_peered":"2024-09-10T18:35:21.095493+0000","last_clean":"2024-09-10T18:35:21.095493+0000","last_became_active":"2024-09-10T18:32:03.382890+0000","last_became_peered":"2024-09-10T18:32:03.382890+0000","last_unstale":"2024-09-10T18:35:21.095493+0000","last_undegraded":"2024-09-10T18:35:21.095493+0000","last_fullsized":"2024-09-10T18:35:21.095493+0000","mapping_epoch":246,"log_start":"283'10847","ondisk_log_start":"283'10847","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'10774","last_scrub_stamp":"2024-09-10T18:34:12.987248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:32:02.366962+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.987248+0000","objects_scrubbed":6064,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:26.071275+0000","scrub_duration":2.4958883709999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54591488,"num_objects":833,"num_object_clones":0,"num_object_copies":2499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15937,"num_write_kb":536640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":361,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.825424+0000","last_change":"2024-09-10T18:34:06.163124+0000","last_active":"2024-09-10T18:35:15.825424+0000","last_peered":"2024-09-10T18:35:15.825424+0000","last_clean":"2024-09-10T18:35:15.825424+0000","last_became_active":"2024-09-10T18:26:27.645932+0000","last_became_peered":"2024-09-10T18:26:27.645932+0000","last_unstale":"2024-09-10T18:35:15.825424+0000","last_undegraded":"2024-09-10T18:35:15.825424+0000","last_fullsized":"2024-09-10T18:35:15.825424+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:34:06.163072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.163072+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:32.837570+0000","scrub_duration":0.0016579489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":322,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.820860+0000","last_change":"2024-09-10T18:33:05.784757+0000","last_active":"2024-09-10T18:35:15.820860+0000","last_peered":"2024-09-10T18:35:15.820860+0000","last_clean":"2024-09-10T18:35:15.820860+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:35:15.820860+0000","last_undegraded":"2024-09-10T18:35:15.820860+0000","last_fullsized":"2024-09-10T18:35:15.820860+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:33:05.784670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:33:05.784670+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002289031,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":493,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:21.012676+0000","last_change":"2024-09-10T18:33:15.972246+0000","last_active":"2024-09-10T18:35:21.012676+0000","last_peered":"2024-09-10T18:35:21.012676+0000","last_clean":"2024-09-10T18:35:21.012676+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:35:21.012676+0000","last_undegraded":"2024-09-10T18:35:21.012676+0000","last_fullsized":"2024-09-10T18:35:21.012676+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:33:15.972166+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.972166+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004394907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":304,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.825708+0000","last_change":"2024-09-10T18:33:05.964339+0000","last_active":"2024-09-10T18:35:15.825708+0000","last_peered":"2024-09-10T18:35:15.825708+0000","last_clean":"2024-09-10T18:35:15.825708+0000","last_became_active":"2024-09-10T18:26:27.647511+0000","last_became_peered":"2024-09-10T18:26:27.647511+0000","last_unstale":"2024-09-10T18:35:15.825708+0000","last_undegraded":"2024-09-10T18:35:15.825708+0000","last_fullsized":"2024-09-10T18:35:15.825708+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:05.964034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:05.964034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015507839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'81","reported_seq":42,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.825878+0000","last_change":"2024-09-10T18:35:06.518466+0000","last_active":"2024-09-10T18:35:15.825878+0000","last_peered":"2024-09-10T18:35:15.825878+0000","last_clean":"2024-09-10T18:35:15.825878+0000","last_became_active":"2024-09-10T18:34:02.810126+0000","last_became_peered":"2024-09-10T18:34:02.810126+0000","last_unstale":"2024-09-10T18:35:15.825878+0000","last_undegraded":"2024-09-10T18:35:15.825878+0000","last_fullsized":"2024-09-10T18:35:15.825878+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:06.518411+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:06.518411+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:28.281097+0000","scrub_duration":0.008298649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":266,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-10T18:35:13.967946+0000","last_change":"2024-09-10T18:34:16.895554+0000","last_active":"2024-09-10T18:35:13.967946+0000","last_peered":"2024-09-10T18:35:13.967946+0000","last_clean":"2024-09-10T18:35:13.967946+0000","last_became_active":"2024-09-10T18:26:27.640922+0000","last_became_peered":"2024-09-10T18:26:27.640922+0000","last_unstale":"2024-09-10T18:35:13.967946+0000","last_undegraded":"2024-09-10T18:35:13.967946+0000","last_fullsized":"2024-09-10T18:35:13.967946+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:16.895472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:34:16.895472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:28.880031+0000","scrub_duration":0.002634066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"240'12","reported_seq":320,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.827545+0000","last_change":"2024-09-10T18:32:56.384627+0000","last_active":"2024-09-10T18:35:15.827545+0000","last_peered":"2024-09-10T18:35:15.827545+0000","last_clean":"2024-09-10T18:35:15.827545+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:35:15.827545+0000","last_undegraded":"2024-09-10T18:35:15.827545+0000","last_fullsized":"2024-09-10T18:35:15.827545+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-10T18:32:56.384439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:56.384439+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010401713999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":308,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.826510+0000","last_change":"2024-09-10T18:34:00.199169+0000","last_active":"2024-09-10T18:35:15.826510+0000","last_peered":"2024-09-10T18:35:15.826510+0000","last_clean":"2024-09-10T18:35:15.826510+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:35:15.826510+0000","last_undegraded":"2024-09-10T18:35:15.826510+0000","last_fullsized":"2024-09-10T18:35:15.826510+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:00.199087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:00.199087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016944524999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":313,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.822922+0000","last_change":"2024-09-10T18:33:58.161088+0000","last_active":"2024-09-10T18:35:15.822922+0000","last_peered":"2024-09-10T18:35:15.822922+0000","last_clean":"2024-09-10T18:35:15.822922+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:35:15.822922+0000","last_undegraded":"2024-09-10T18:35:15.822922+0000","last_fullsized":"2024-09-10T18:35:15.822922+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:58.161019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:58.161019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:23.203002+0000","scrub_duration":0.0042556759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'81","reported_seq":248,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.815645+0000","last_change":"2024-09-10T18:35:08.832694+0000","last_active":"2024-09-10T18:35:15.815645+0000","last_peered":"2024-09-10T18:35:15.815645+0000","last_clean":"2024-09-10T18:35:15.815645+0000","last_became_active":"2024-09-10T18:34:02.815880+0000","last_became_peered":"2024-09-10T18:34:02.815880+0000","last_unstale":"2024-09-10T18:35:15.815645+0000","last_undegraded":"2024-09-10T18:35:15.815645+0000","last_fullsized":"2024-09-10T18:35:15.815645+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:08.832597+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:08.832597+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:13.648808+0000","scrub_duration":0.00460159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"240'19","reported_seq":343,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:19.802167+0000","last_change":"2024-09-10T18:35:19.802100+0000","last_active":"2024-09-10T18:35:19.802167+0000","last_peered":"2024-09-10T18:35:19.802167+0000","last_clean":"2024-09-10T18:35:19.802167+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:35:19.802167+0000","last_undegraded":"2024-09-10T18:35:19.802167+0000","last_fullsized":"2024-09-10T18:35:19.802167+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:35:19.802016+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-09-10T18:35:19.802016+0000","last_clean_scrub_stamp":"2024-09-10T18:35:19.802016+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:36.919136+0000","scrub_duration":0.010791837,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":319,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.818468+0000","last_change":"2024-09-10T18:33:16.937511+0000","last_active":"2024-09-10T18:35:15.818468+0000","last_peered":"2024-09-10T18:35:15.818468+0000","last_clean":"2024-09-10T18:35:15.818468+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:35:15.818468+0000","last_undegraded":"2024-09-10T18:35:15.818468+0000","last_fullsized":"2024-09-10T18:35:15.818468+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:16.937390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:33:16.937390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.024770533000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":310,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.827296+0000","last_change":"2024-09-10T18:35:12.764509+0000","last_active":"2024-09-10T18:35:15.827296+0000","last_peered":"2024-09-10T18:35:15.827296+0000","last_clean":"2024-09-10T18:35:15.827296+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:35:15.827296+0000","last_undegraded":"2024-09-10T18:35:15.827296+0000","last_fullsized":"2024-09-10T18:35:15.827296+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:12.764433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:12.764433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:22.436741+0000","scrub_duration":0.0069242059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":326,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.823216+0000","last_change":"2024-09-10T18:34:53.654671+0000","last_active":"2024-09-10T18:35:15.823216+0000","last_peered":"2024-09-10T18:35:15.823216+0000","last_clean":"2024-09-10T18:35:15.823216+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:35:15.823216+0000","last_undegraded":"2024-09-10T18:35:15.823216+0000","last_fullsized":"2024-09-10T18:35:15.823216+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:53.654591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:53.654591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:00.061299+0000","scrub_duration":0.0020849879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'81","reported_seq":42,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.824927+0000","last_change":"2024-09-10T18:35:04.516060+0000","last_active":"2024-09-10T18:35:15.824927+0000","last_peered":"2024-09-10T18:35:15.824927+0000","last_clean":"2024-09-10T18:35:15.824927+0000","last_became_active":"2024-09-10T18:34:02.797304+0000","last_became_peered":"2024-09-10T18:34:02.797304+0000","last_unstale":"2024-09-10T18:35:15.824927+0000","last_undegraded":"2024-09-10T18:35:15.824927+0000","last_fullsized":"2024-09-10T18:35:15.824927+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:04.515970+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:04.515970+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:14.514143+0000","scrub_duration":0.0032060309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"240'19","reported_seq":340,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.826479+0000","last_change":"2024-09-10T18:33:15.955719+0000","last_active":"2024-09-10T18:35:15.826479+0000","last_peered":"2024-09-10T18:35:15.826479+0000","last_clean":"2024-09-10T18:35:15.826479+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:35:15.826479+0000","last_undegraded":"2024-09-10T18:35:15.826479+0000","last_fullsized":"2024-09-10T18:35:15.826479+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:33:15.955638+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:33:15.955638+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021358801,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":323,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.827683+0000","last_change":"2024-09-10T18:35:12.758404+0000","last_active":"2024-09-10T18:35:15.827683+0000","last_peered":"2024-09-10T18:35:15.827683+0000","last_clean":"2024-09-10T18:35:15.827683+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:35:15.827683+0000","last_undegraded":"2024-09-10T18:35:15.827683+0000","last_fullsized":"2024-09-10T18:35:15.827683+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:12.758329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_clean_scrub_stamp":"2024-09-10T18:35:12.758329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:36.142548+0000","scrub_duration":0.010517383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":484,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:19.926305+0000","last_change":"2024-09-10T18:35:01.701033+0000","last_active":"2024-09-10T18:35:19.926305+0000","last_peered":"2024-09-10T18:35:19.926305+0000","last_clean":"2024-09-10T18:35:19.926305+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:35:19.926305+0000","last_undegraded":"2024-09-10T18:35:19.926305+0000","last_fullsized":"2024-09-10T18:35:19.926305+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:35:01.700934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:01.700934+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:12.522418+0000","scrub_duration":0.0050961280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":43,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.827720+0000","last_change":"2024-09-10T18:33:29.142401+0000","last_active":"2024-09-10T18:35:15.827720+0000","last_peered":"2024-09-10T18:35:15.827720+0000","last_clean":"2024-09-10T18:35:15.827720+0000","last_became_active":"2024-09-10T18:33:29.141936+0000","last_became_peered":"2024-09-10T18:33:29.141936+0000","last_unstale":"2024-09-10T18:35:15.827720+0000","last_undegraded":"2024-09-10T18:35:15.827720+0000","last_fullsized":"2024-09-10T18:35:15.827720+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:11.727786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.727786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"76'81","reported_seq":103,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.824575+0000","last_change":"2024-09-10T18:34:22.551031+0000","last_active":"2024-09-10T18:35:15.824575+0000","last_peered":"2024-09-10T18:35:15.824575+0000","last_clean":"2024-09-10T18:35:15.824575+0000","last_became_active":"2024-09-10T18:34:02.810381+0000","last_became_peered":"2024-09-10T18:34:02.810381+0000","last_unstale":"2024-09-10T18:35:15.824575+0000","last_undegraded":"2024-09-10T18:35:15.824575+0000","last_fullsized":"2024-09-10T18:35:15.824575+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:22.550782+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.550782+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:47.692871+0000","scrub_duration":0.0052607840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":342,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.829400+0000","last_change":"2024-09-10T18:33:59.230861+0000","last_active":"2024-09-10T18:35:15.829400+0000","last_peered":"2024-09-10T18:35:15.829400+0000","last_clean":"2024-09-10T18:35:15.829400+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:35:15.829400+0000","last_undegraded":"2024-09-10T18:35:15.829400+0000","last_fullsized":"2024-09-10T18:35:15.829400+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-10T18:33:59.227198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:59.227198+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012577204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":322,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:17.826545+0000","last_change":"2024-09-10T18:35:17.826500+0000","last_active":"2024-09-10T18:35:17.826545+0000","last_peered":"2024-09-10T18:35:17.826545+0000","last_clean":"2024-09-10T18:35:17.826545+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:35:17.826545+0000","last_undegraded":"2024-09-10T18:35:17.826545+0000","last_fullsized":"2024-09-10T18:35:17.826545+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:17.826448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:17.826448+0000","last_clean_scrub_stamp":"2024-09-10T18:35:17.826448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:29.812245+0000","scrub_duration":0.002920808,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":494,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-10T18:35:21.014673+0000","last_change":"2024-09-10T18:34:58.485675+0000","last_active":"2024-09-10T18:35:21.014673+0000","last_peered":"2024-09-10T18:35:21.014673+0000","last_clean":"2024-09-10T18:35:21.014673+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:35:21.014673+0000","last_undegraded":"2024-09-10T18:35:21.014673+0000","last_fullsized":"2024-09-10T18:35:21.014673+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:34:58.485562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:58.485562+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:19.625439+0000","scrub_duration":0.0059980570000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":307,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.824715+0000","last_change":"2024-09-10T18:34:23.556824+0000","last_active":"2024-09-10T18:35:15.824715+0000","last_peered":"2024-09-10T18:35:15.824715+0000","last_clean":"2024-09-10T18:35:15.824715+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:35:15.824715+0000","last_undegraded":"2024-09-10T18:35:15.824715+0000","last_fullsized":"2024-09-10T18:35:15.824715+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:23.556773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:23.556773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:47.130693+0000","scrub_duration":0.001126988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'81","reported_seq":23,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.820016+0000","last_change":"2024-09-10T18:34:34.881583+0000","last_active":"2024-09-10T18:35:15.820016+0000","last_peered":"2024-09-10T18:35:15.820016+0000","last_clean":"2024-09-10T18:35:15.820016+0000","last_became_active":"2024-09-10T18:34:34.881186+0000","last_became_peered":"2024-09-10T18:34:34.881186+0000","last_unstale":"2024-09-10T18:35:15.820016+0000","last_undegraded":"2024-09-10T18:35:15.820016+0000","last_fullsized":"2024-09-10T18:35:15.820016+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:05.012053+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:05.012053+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":358,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.811049+0000","last_change":"2024-09-10T18:35:14.262075+0000","last_active":"2024-09-10T18:35:15.811049+0000","last_peered":"2024-09-10T18:35:15.811049+0000","last_clean":"2024-09-10T18:35:15.811049+0000","last_became_active":"2024-09-10T18:26:27.649202+0000","last_became_peered":"2024-09-10T18:26:27.649202+0000","last_unstale":"2024-09-10T18:35:15.811049+0000","last_undegraded":"2024-09-10T18:35:15.811049+0000","last_fullsized":"2024-09-10T18:35:15.811049+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:35:14.261980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:14.261980+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:16.097526+0000","scrub_duration":0.0026192849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":322,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.811511+0000","last_change":"2024-09-10T18:34:04.380240+0000","last_active":"2024-09-10T18:35:15.811511+0000","last_peered":"2024-09-10T18:35:15.811511+0000","last_clean":"2024-09-10T18:35:15.811511+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:35:15.811511+0000","last_undegraded":"2024-09-10T18:35:15.811511+0000","last_fullsized":"2024-09-10T18:35:15.811511+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:04.380160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:34:04.380160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:34.095135+0000","scrub_duration":0.0018137730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":321,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.821940+0000","last_change":"2024-09-10T18:34:03.521909+0000","last_active":"2024-09-10T18:35:15.821940+0000","last_peered":"2024-09-10T18:35:15.821940+0000","last_clean":"2024-09-10T18:35:15.821940+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:35:15.821940+0000","last_undegraded":"2024-09-10T18:35:15.821940+0000","last_fullsized":"2024-09-10T18:35:15.821940+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:03.521806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_clean_scrub_stamp":"2024-09-10T18:34:03.521806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005196946,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":298,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.818835+0000","last_change":"2024-09-10T18:34:14.263579+0000","last_active":"2024-09-10T18:35:15.818835+0000","last_peered":"2024-09-10T18:35:15.818835+0000","last_clean":"2024-09-10T18:35:15.818835+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:35:15.818835+0000","last_undegraded":"2024-09-10T18:35:15.818835+0000","last_fullsized":"2024-09-10T18:35:15.818835+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:14.263499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:34:14.263499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:30.466656+0000","scrub_duration":0.0019685509999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'81","reported_seq":23,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.828535+0000","last_change":"2024-09-10T18:34:34.858726+0000","last_active":"2024-09-10T18:35:15.828535+0000","last_peered":"2024-09-10T18:35:15.828535+0000","last_clean":"2024-09-10T18:35:15.828535+0000","last_became_active":"2024-09-10T18:34:34.858291+0000","last_became_peered":"2024-09-10T18:34:34.858291+0000","last_unstale":"2024-09-10T18:35:15.828535+0000","last_undegraded":"2024-09-10T18:35:15.828535+0000","last_fullsized":"2024-09-10T18:35:15.828535+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:08.035431+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:08.035431+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"240'15","reported_seq":352,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.822161+0000","last_change":"2024-09-10T18:34:19.608537+0000","last_active":"2024-09-10T18:35:15.822161+0000","last_peered":"2024-09-10T18:35:15.822161+0000","last_clean":"2024-09-10T18:35:15.822161+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:35:15.822161+0000","last_undegraded":"2024-09-10T18:35:15.822161+0000","last_fullsized":"2024-09-10T18:35:15.822161+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:34:19.608445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.608445+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:24.709727+0000","scrub_duration":0.012630626000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":315,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.823191+0000","last_change":"2024-09-10T18:33:01.781370+0000","last_active":"2024-09-10T18:35:15.823191+0000","last_peered":"2024-09-10T18:35:15.823191+0000","last_clean":"2024-09-10T18:35:15.823191+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:35:15.823191+0000","last_undegraded":"2024-09-10T18:35:15.823191+0000","last_fullsized":"2024-09-10T18:35:15.823191+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:01.781295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.781295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016784879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":303,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.826662+0000","last_change":"2024-09-10T18:34:56.701737+0000","last_active":"2024-09-10T18:35:15.826662+0000","last_peered":"2024-09-10T18:35:15.826662+0000","last_clean":"2024-09-10T18:35:15.826662+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:35:15.826662+0000","last_undegraded":"2024-09-10T18:35:15.826662+0000","last_fullsized":"2024-09-10T18:35:15.826662+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:56.701661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:34:56.701661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:25.109620+0000","scrub_duration":0.0064135620000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":307,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.824840+0000","last_change":"2024-09-10T18:34:00.512903+0000","last_active":"2024-09-10T18:35:15.824840+0000","last_peered":"2024-09-10T18:35:15.824840+0000","last_clean":"2024-09-10T18:35:15.824840+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:35:15.824840+0000","last_undegraded":"2024-09-10T18:35:15.824840+0000","last_fullsized":"2024-09-10T18:35:15.824840+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:00.512670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:00.512670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068783309999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'81","reported_seq":42,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.823087+0000","last_change":"2024-09-10T18:34:56.691141+0000","last_active":"2024-09-10T18:35:15.823087+0000","last_peered":"2024-09-10T18:35:15.823087+0000","last_clean":"2024-09-10T18:35:15.823087+0000","last_became_active":"2024-09-10T18:34:02.802580+0000","last_became_peered":"2024-09-10T18:34:02.802580+0000","last_unstale":"2024-09-10T18:35:15.823087+0000","last_undegraded":"2024-09-10T18:35:15.823087+0000","last_fullsized":"2024-09-10T18:35:15.823087+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:56.691043+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:56.691043+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:06.642829+0000","scrub_duration":0.0021075249999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":350,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.819341+0000","last_change":"2024-09-10T18:35:03.727242+0000","last_active":"2024-09-10T18:35:15.819341+0000","last_peered":"2024-09-10T18:35:15.819341+0000","last_clean":"2024-09-10T18:35:15.819341+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:35:15.819341+0000","last_undegraded":"2024-09-10T18:35:15.819341+0000","last_fullsized":"2024-09-10T18:35:15.819341+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:35:03.727153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:03.727153+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:03.868588+0000","scrub_duration":0.0069271460000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":323,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.826574+0000","last_change":"2024-09-10T18:34:08.134374+0000","last_active":"2024-09-10T18:35:15.826574+0000","last_peered":"2024-09-10T18:35:15.826574+0000","last_clean":"2024-09-10T18:35:15.826574+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:35:15.826574+0000","last_undegraded":"2024-09-10T18:35:15.826574+0000","last_fullsized":"2024-09-10T18:35:15.826574+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:08.134230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:34:08.134230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:31.369105+0000","scrub_duration":0.0077618749999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":318,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.816395+0000","last_change":"2024-09-10T18:34:57.776098+0000","last_active":"2024-09-10T18:35:15.816395+0000","last_peered":"2024-09-10T18:35:15.816395+0000","last_clean":"2024-09-10T18:35:15.816395+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:35:15.816395+0000","last_undegraded":"2024-09-10T18:35:15.816395+0000","last_fullsized":"2024-09-10T18:35:15.816395+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:57.776022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:57.776022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:27.386344+0000","scrub_duration":0.002853441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":315,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.827790+0000","last_change":"2024-09-10T18:33:18.227516+0000","last_active":"2024-09-10T18:35:15.827790+0000","last_peered":"2024-09-10T18:35:15.827790+0000","last_clean":"2024-09-10T18:35:15.827790+0000","last_became_active":"2024-09-10T18:26:27.636996+0000","last_became_peered":"2024-09-10T18:26:27.636996+0000","last_unstale":"2024-09-10T18:35:15.827790+0000","last_undegraded":"2024-09-10T18:35:15.827790+0000","last_fullsized":"2024-09-10T18:35:15.827790+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:18.227428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:18.227428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063780959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'81","reported_seq":23,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.819320+0000","last_change":"2024-09-10T18:34:34.869108+0000","last_active":"2024-09-10T18:35:15.819320+0000","last_peered":"2024-09-10T18:35:15.819320+0000","last_clean":"2024-09-10T18:35:15.819320+0000","last_became_active":"2024-09-10T18:34:34.868371+0000","last_became_peered":"2024-09-10T18:34:34.868371+0000","last_unstale":"2024-09-10T18:35:15.819320+0000","last_undegraded":"2024-09-10T18:35:15.819320+0000","last_fullsized":"2024-09-10T18:35:15.819320+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:22.344759+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.344759+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":321,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.818735+0000","last_change":"2024-09-10T18:32:47.287083+0000","last_active":"2024-09-10T18:35:15.818735+0000","last_peered":"2024-09-10T18:35:15.818735+0000","last_clean":"2024-09-10T18:35:15.818735+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:35:15.818735+0000","last_undegraded":"2024-09-10T18:35:15.818735+0000","last_fullsized":"2024-09-10T18:35:15.818735+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-09-10T18:32:47.286977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:32:47.286977+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017807315000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":316,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.824973+0000","last_change":"2024-09-10T18:35:01.976534+0000","last_active":"2024-09-10T18:35:15.824973+0000","last_peered":"2024-09-10T18:35:15.824973+0000","last_clean":"2024-09-10T18:35:15.824973+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:35:15.824973+0000","last_undegraded":"2024-09-10T18:35:15.824973+0000","last_fullsized":"2024-09-10T18:35:15.824973+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:01.976429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:35:01.976429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:18.322841+0000","scrub_duration":0.11603027,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":270,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-10T18:35:13.967335+0000","last_change":"2024-09-10T18:34:18.819184+0000","last_active":"2024-09-10T18:35:13.967335+0000","last_peered":"2024-09-10T18:35:13.967335+0000","last_clean":"2024-09-10T18:35:13.967335+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:35:13.967335+0000","last_undegraded":"2024-09-10T18:35:13.967335+0000","last_fullsized":"2024-09-10T18:35:13.967335+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:18.819090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:18.819090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:35.004598+0000","scrub_duration":0.0015527620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":316,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:18.879824+0000","last_change":"2024-09-10T18:35:18.879787+0000","last_active":"2024-09-10T18:35:18.879824+0000","last_peered":"2024-09-10T18:35:18.879824+0000","last_clean":"2024-09-10T18:35:18.879824+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:35:18.879824+0000","last_undegraded":"2024-09-10T18:35:18.879824+0000","last_fullsized":"2024-09-10T18:35:18.879824+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:18.879717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:18.879717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:41.844243+0000","scrub_duration":0.0059146550000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'81","reported_seq":23,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.810862+0000","last_change":"2024-09-10T18:34:34.867779+0000","last_active":"2024-09-10T18:35:15.810862+0000","last_peered":"2024-09-10T18:35:15.810862+0000","last_clean":"2024-09-10T18:35:15.810862+0000","last_became_active":"2024-09-10T18:34:34.867380+0000","last_became_peered":"2024-09-10T18:34:34.867380+0000","last_unstale":"2024-09-10T18:35:15.810862+0000","last_undegraded":"2024-09-10T18:35:15.810862+0000","last_fullsized":"2024-09-10T18:35:15.810862+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:56.155354+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:56.155354+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":316,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.817098+0000","last_change":"2024-09-10T18:33:18.177364+0000","last_active":"2024-09-10T18:35:15.817098+0000","last_peered":"2024-09-10T18:35:15.817098+0000","last_clean":"2024-09-10T18:35:15.817098+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:35:15.817098+0000","last_undegraded":"2024-09-10T18:35:15.817098+0000","last_fullsized":"2024-09-10T18:35:15.817098+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_clean_scrub_stamp":"2024-09-10T18:33:18.177264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029244380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":304,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.819046+0000","last_change":"2024-09-10T18:34:03.175239+0000","last_active":"2024-09-10T18:35:15.819046+0000","last_peered":"2024-09-10T18:35:15.819046+0000","last_clean":"2024-09-10T18:35:15.819046+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:35:15.819046+0000","last_undegraded":"2024-09-10T18:35:15.819046+0000","last_fullsized":"2024-09-10T18:35:15.819046+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:03.175158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:34:03.175158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046326559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":331,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-10T18:35:13.965762+0000","last_change":"2024-09-10T18:35:08.862632+0000","last_active":"2024-09-10T18:35:13.965762+0000","last_peered":"2024-09-10T18:35:13.965762+0000","last_clean":"2024-09-10T18:35:13.965762+0000","last_became_active":"2024-09-10T18:26:27.648112+0000","last_became_peered":"2024-09-10T18:26:27.648112+0000","last_unstale":"2024-09-10T18:35:13.965762+0000","last_undegraded":"2024-09-10T18:35:13.965762+0000","last_fullsized":"2024-09-10T18:35:13.965762+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:35:08.862542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:08.862542+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:37.474846+0000","scrub_duration":0.0029242280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":309,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-10T18:35:15.824859+0000","last_change":"2024-09-10T18:34:02.502004+0000","last_active":"2024-09-10T18:35:15.824859+0000","last_peered":"2024-09-10T18:35:15.824859+0000","last_clean":"2024-09-10T18:35:15.824859+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:35:15.824859+0000","last_undegraded":"2024-09-10T18:35:15.824859+0000","last_fullsized":"2024-09-10T18:35:15.824859+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:02.501929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:34:02.501929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:35:23.332476+0000","scrub_duration":0.0086866479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":657260568,"num_objects":10030,"num_object_clones":0,"num_object_copies":30090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10030,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":259148,"num_write_kb":8613633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1829777408,"data_stored":1829765192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68370,"ondisk_log_size":68370,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":745,"ondisk_log_size":745,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":826,"num_read_kb":821,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":506091,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":51,"num_write":26,"num_write_kb":370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":694878,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2511,"ondisk_log_size":2511,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234154,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":580844,"kb_used_data":283428,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93164308,"statfs":{"total":95995035648,"available":95400251392,"internally_reserved":0,"allocated":290230272,"data_stored":289707888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":2.3799999999999999,"15min":1.8320000000000001},"min":{"1min":0.215,"5min":0.21099999999999999,"15min":0.20799999999999999},"max":{"1min":2.3820000000000001,"5min":75.506,"15min":85.012},"last":1.47},{"interface":"front","average":{"1min":1.127,"5min":2.2690000000000001,"15min":1.7689999999999999},"min":{"1min":0.32800000000000001,"5min":0.25600000000000001,"15min":0.222},"max":{"1min":3.4889999999999999,"5min":75.710999999999999,"15min":84.665999999999997},"last":0.748}]},{"osd":1,"last update":"Tue Sep 10 18:34:46 2024","interfaces":[{"interface":"back","average":{"1min":3.6459999999999999,"5min":1.661,"15min":1.4379999999999999},"min":{"1min":0.27000000000000002,"5min":0.214,"15min":0.214},"max":{"1min":39.728000000000002,"5min":39.728000000000002,"15min":49.317999999999998},"last":1.5960000000000001},{"interface":"front","average":{"1min":3.6459999999999999,"5min":1.5369999999999999,"15min":1.3720000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":39.738999999999997,"5min":39.738999999999997,"15min":49.353000000000002},"last":0.76800000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:34:46 2024","interfaces":[{"interface":"back","average":{"1min":8.8529999999999998,"5min":2.5190000000000001,"15min":1.353},"min":{"1min":0.29099999999999998,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":136.648,"5min":136.648,"15min":136.648},"last":0.84099999999999997},{"interface":"front","average":{"1min":9.2149999999999999,"5min":2.625,"15min":1.4490000000000001},"min":{"1min":0.32100000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":136.506,"5min":136.506,"15min":136.506},"last":1.532}]},{"osd":3,"last update":"Tue Sep 10 18:34:46 2024","interfaces":[{"interface":"back","average":{"1min":3.7519999999999998,"5min":1.712,"15min":1.587},"min":{"1min":0.34100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":38.225000000000001,"5min":38.225000000000001,"15min":86.305999999999997},"last":0.82299999999999995},{"interface":"front","average":{"1min":3.5720000000000001,"5min":1.7070000000000001,"15min":1.6259999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.26700000000000002},"max":{"1min":38.104999999999997,"5min":38.104999999999997,"15min":88.364000000000004},"last":1.617}]},{"osd":4,"last update":"Tue Sep 10 18:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.63300000000000001,"15min":0.50600000000000001},"min":{"1min":0.27100000000000002,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":2.282,"5min":3.4670000000000001,"15min":3.8839999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.67500000000000004,"15min":0.53200000000000003},"min":{"1min":0.28299999999999997,"5min":0.26200000000000001,"15min":0.223},"max":{"1min":2.4449999999999998,"5min":2.4449999999999998,"15min":2.6589999999999998},"last":0.66100000000000003}]},{"osd":5,"last update":"Tue Sep 10 18:34:46 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":0.98999999999999999,"15min":0.69799999999999995},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.23499999999999999},"max":{"1min":2.7509999999999999,"5min":7.1820000000000004,"15min":7.3109999999999999},"last":1.556},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.93999999999999995,"15min":0.65100000000000002},"min":{"1min":0.379,"5min":0.27100000000000002,"15min":0.219},"max":{"1min":3.3130000000000002,"5min":6.3360000000000003,"15min":6.3360000000000003},"last":0.78600000000000003}]},{"osd":6,"last update":"Tue Sep 10 18:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90100000000000002,"15min":0.629},"min":{"1min":0.32000000000000001,"5min":0.28899999999999998,"15min":0.27000000000000002},"max":{"1min":2.5699999999999998,"5min":7.2750000000000004,"15min":7.2750000000000004},"last":1.5009999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.81299999999999994,"15min":0.58499999999999996},"min":{"1min":0.373,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.2090000000000001,"5min":4.585,"15min":4.585},"last":0.80300000000000005}]}]},{"osd":6,"up_from":144,"seq":618475290732,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":477384,"kb_used_data":170624,"kb_used_omap":7,"kb_used_meta":306744,"kb_avail":93267768,"statfs":{"total":95995035648,"available":95506194432,"internally_reserved":0,"allocated":174718976,"data_stored":174226182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":314105943},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.1859999999999999,"15min":1.776},"min":{"1min":0.23300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.9940000000000002,"5min":8.3949999999999996,"15min":30.629999999999999},"last":3.3799999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.93999999999999995,"15min":1.3320000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.22900000000000001},"max":{"1min":3.407,"5min":8.4640000000000004,"15min":30.597000000000001},"last":3.363}]},{"osd":1,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.0660000000000001,"15min":0.95799999999999996},"min":{"1min":0.26000000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":3.806,"5min":8.0329999999999995,"15min":30.984000000000002},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.246,"15min":1.143},"min":{"1min":0.28100000000000003,"5min":0.246,"15min":0.24199999999999999},"max":{"1min":2.766,"5min":6.7859999999999996,"15min":30.571000000000002},"last":7.7939999999999996}]},{"osd":2,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.93300000000000005,"15min":1.024},"min":{"1min":0.307,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.661,"5min":18.353000000000002,"15min":18.353000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.90300000000000002,"15min":0.84299999999999997},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.23400000000000001},"max":{"1min":2.2959999999999998,"5min":5.415,"15min":11.164999999999999},"last":0.36099999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.2310000000000001,"15min":1.8979999999999999},"min":{"1min":0.28799999999999998,"5min":0.28199999999999997,"15min":0.27800000000000002},"max":{"1min":4.0970000000000004,"5min":16.831,"15min":61.703000000000003},"last":1.292},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.377,"15min":1.9379999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.5270000000000001,"5min":17.786000000000001,"15min":61.814},"last":0.30199999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58399999999999996,"15min":0.55100000000000005},"min":{"1min":0.27700000000000002,"5min":0.24199999999999999,"15min":0.23699999999999999},"max":{"1min":2.4889999999999999,"5min":2.8410000000000002,"15min":5.4950000000000001},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60999999999999999,"15min":0.63300000000000001},"min":{"1min":0.314,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.8140000000000001,"5min":4.6390000000000002,"15min":4.6390000000000002},"last":0.32100000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.69099999999999995,"15min":0.55300000000000005},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.24399999999999999},"max":{"1min":1.8109999999999999,"5min":4.6619999999999999,"15min":4.6619999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71199999999999997,"15min":0.53500000000000003},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.26800000000000002},"max":{"1min":2.1259999999999999,"5min":2.794,"15min":2.794},"last":0.40400000000000003}]},{"osd":7,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.69999999999999996,"15min":0.56200000000000006},"min":{"1min":0.28999999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.3650000000000002,"5min":3.3650000000000002,"15min":3.3650000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.68300000000000005,"15min":0.58799999999999997},"min":{"1min":0.33500000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.1480000000000001,"5min":3.9300000000000002,"15min":5.5309999999999997},"last":0.92600000000000005}]}]},{"osd":5,"up_from":39,"seq":167503724725,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895088,"kb_used_data":309352,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":92850064,"statfs":{"total":95995035648,"available":95078465536,"internally_reserved":0,"allocated":316776448,"data_stored":316250474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:35:03 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.4079999999999999,"15min":1.516},"min":{"1min":0.30099999999999999,"5min":0.23499999999999999,"15min":0.21099999999999999},"max":{"1min":15.819000000000001,"5min":26.288,"15min":52.033999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.218,"15min":1.3109999999999999},"min":{"1min":0.26900000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":1.8839999999999999,"5min":24.178000000000001,"15min":52.167999999999999},"last":0.34699999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:35:03 2024","interfaces":[{"interface":"back","average":{"1min":8.7430000000000003,"5min":2.907,"15min":1.484},"min":{"1min":0.40500000000000003,"5min":0.32700000000000001,"15min":0.22600000000000001},"max":{"1min":111.773,"5min":111.773,"15min":111.773},"last":0.67400000000000004},{"interface":"front","average":{"1min":8.9730000000000008,"5min":2.972,"15min":1.5720000000000001},"min":{"1min":0.41699999999999998,"5min":0.28499999999999998,"15min":0.247},"max":{"1min":111.73999999999999,"5min":111.73999999999999,"15min":111.73999999999999},"last":0.317}]},{"osd":2,"last update":"Tue Sep 10 18:35:03 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.5449999999999999,"15min":1.2190000000000001},"min":{"1min":0.29499999999999998,"5min":0.245,"15min":0.24299999999999999},"max":{"1min":5.6299999999999999,"5min":54.530000000000001,"15min":54.530000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.448,"15min":1.1799999999999999},"min":{"1min":0.29999999999999999,"5min":0.247,"15min":0.247},"max":{"1min":3.222,"5min":54.514000000000003,"15min":54.514000000000003},"last":0.69999999999999996}]},{"osd":3,"last update":"Tue Sep 10 18:35:03 2024","interfaces":[{"interface":"back","average":{"1min":7.4969999999999999,"5min":2.9809999999999999,"15min":1.8680000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":106.205,"5min":106.205,"15min":106.205},"last":0.59999999999999998},{"interface":"front","average":{"1min":7.6219999999999999,"5min":3.0659999999999998,"15min":1.7829999999999999},"min":{"1min":0.38100000000000001,"5min":0.32000000000000001,"15min":0.29699999999999999},"max":{"1min":105.878,"5min":105.878,"15min":105.878},"last":0.72699999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.73899999999999999,"15min":0.53500000000000003},"min":{"1min":0.26800000000000002,"5min":0.22900000000000001,"15min":0.20799999999999999},"max":{"1min":5.8040000000000003,"5min":5.8040000000000003,"15min":5.8040000000000003},"last":0.37},{"interface":"front","average":{"1min":0.877,"5min":0.76400000000000001,"15min":0.58499999999999996},"min":{"1min":0.31900000000000001,"5min":0.24199999999999999,"15min":0.22700000000000001},"max":{"1min":5.5499999999999998,"5min":6.798,"15min":6.798},"last":0.47599999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.65200000000000002,"15min":0.64500000000000002},"min":{"1min":0.34699999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.1819999999999999,"5min":2.7469999999999999,"15min":4.7859999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.72899999999999998,"15min":0.65200000000000002},"min":{"1min":0.28100000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.2850000000000001,"5min":5.3680000000000003,"15min":5.3860000000000001},"last":0.65200000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:34:34 2024","interfaces":[{"interface":"back","average":{"1min":2.3029999999999999,"5min":1.0640000000000001,"15min":0.70499999999999996},"min":{"1min":0.35699999999999998,"5min":0.25900000000000001,"15min":0.22500000000000001},"max":{"1min":28.934000000000001,"5min":28.934000000000001,"15min":28.934000000000001},"last":0.628},{"interface":"front","average":{"1min":1.2030000000000001,"5min":0.82299999999999995,"15min":0.59499999999999997},"min":{"1min":0.38100000000000001,"5min":0.25900000000000001,"15min":0.24299999999999999},"max":{"1min":5.2190000000000003,"5min":5.2190000000000003,"15min":5.2190000000000003},"last":0.53500000000000003}]}]},{"osd":4,"up_from":33,"seq":141733920953,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":414244,"kb_used_data":104412,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93330908,"statfs":{"total":95995035648,"available":95570849792,"internally_reserved":0,"allocated":106917888,"data_stored":106402939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.95499999999999996,"15min":1.024},"min":{"1min":0.248,"5min":0.21199999999999999,"15min":0.188},"max":{"1min":2.5960000000000001,"5min":4.3730000000000002,"15min":31.585000000000001},"last":2.4039999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.92900000000000005,"15min":1.006},"min":{"1min":0.28799999999999998,"5min":0.189,"15min":0.189},"max":{"1min":4.8929999999999998,"5min":4.8929999999999998,"15min":29.802},"last":0.69299999999999995}]},{"osd":1,"last update":"Tue Sep 10 18:34:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.5069999999999999,"15min":1.0069999999999999},"min":{"1min":0.254,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":4.3390000000000004,"5min":50.045000000000002,"15min":50.045000000000002},"last":2.4590000000000001},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.7869999999999999,"15min":1.095},"min":{"1min":0.30299999999999999,"5min":0.27100000000000002,"15min":0.22900000000000001},"max":{"1min":5.5590000000000002,"5min":50.283000000000001,"15min":50.283000000000001},"last":0.61099999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.88300000000000001,"15min":0.86899999999999999},"min":{"1min":0.28499999999999998,"5min":0.249,"15min":0.249},"max":{"1min":1.857,"5min":5.2380000000000004,"15min":13.898999999999999},"last":0.314},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.876,"15min":0.89100000000000001},"min":{"1min":0.31900000000000001,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":2.9790000000000001,"5min":4.0460000000000003,"15min":14.51},"last":0.71699999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:34:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.6719999999999999,"15min":1.335},"min":{"1min":0.28899999999999998,"5min":0.26300000000000001,"15min":0.249},"max":{"1min":7.0330000000000004,"5min":49.990000000000002,"15min":49.990000000000002},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.383,"5min":1.726,"15min":1.385},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.255},"max":{"1min":5.0439999999999996,"5min":50.764000000000003,"15min":50.764000000000003},"last":2.4359999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:34:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.93799999999999994,"15min":0.61099999999999999},"min":{"1min":0.27600000000000002,"5min":0.223,"15min":0.20499999999999999},"max":{"1min":1.1970000000000001,"5min":22.861000000000001,"15min":22.861000000000001},"last":1.2190000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.93799999999999994,"15min":0.64000000000000001},"min":{"1min":0.27300000000000002,"5min":0.22,"15min":0.22},"max":{"1min":2.387,"5min":19.611999999999998,"15min":19.611999999999998},"last":1.369}]},{"osd":6,"last update":"Tue Sep 10 18:34:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.63800000000000001,"15min":0.50900000000000001},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.26400000000000001},"max":{"1min":0.77000000000000002,"5min":3.1139999999999999,"15min":6.7990000000000004},"last":2.4820000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.63100000000000001,"15min":0.54900000000000004},"min":{"1min":0.32100000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.95499999999999996,"5min":3.6419999999999999,"15min":7.0300000000000002},"last":0.51200000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:34:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65200000000000002,"15min":0.55700000000000005},"min":{"1min":0.316,"5min":0.28799999999999998,"15min":0.255},"max":{"1min":1.4330000000000001,"5min":4.8600000000000003,"15min":7.0670000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64400000000000002,"15min":0.57099999999999995},"min":{"1min":0.33800000000000002,"5min":0.28699999999999998,"15min":0.247},"max":{"1min":1.2130000000000001,"5min":3.3450000000000002,"15min":6.5330000000000004},"last":0.63400000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117182,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1017700,"kb_used_data":431836,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":92727452,"statfs":{"total":95995035648,"available":94952910848,"internally_reserved":0,"allocated":442200064,"data_stored":441679081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,3,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:35:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.2350000000000001,"15min":1.304},"min":{"1min":0.184,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.2370000000000001,"5min":5.4560000000000004,"15min":8.5530000000000008},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.4670000000000001,"15min":1.3899999999999999},"min":{"1min":0.23799999999999999,"5min":0.23100000000000001,"15min":0.193},"max":{"1min":3.9940000000000002,"5min":8.2110000000000003,"15min":13.659000000000001},"last":0.89100000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:35:13 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.911,"15min":1.484},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":4.0300000000000002,"5min":35.406999999999996,"15min":35.406999999999996},"last":5.641},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.9370000000000001,"15min":1.387},"min":{"1min":0.24199999999999999,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":4.2130000000000001,"5min":36.902000000000001,"15min":36.902000000000001},"last":0.56499999999999995}]},{"osd":2,"last update":"Tue Sep 10 18:35:13 2024","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.1319999999999999,"15min":1.149},"min":{"1min":0.28899999999999998,"5min":0.247,"15min":0.222},"max":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":7.7960000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.506,"5min":1.3440000000000001,"15min":1.3300000000000001},"min":{"1min":0.29399999999999998,"5min":0.23000000000000001,"15min":0.217},"max":{"1min":6.6340000000000003,"5min":14.997999999999999,"15min":14.997999999999999},"last":1.7689999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.208,"15min":1.2390000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":2.7210000000000001,"5min":6.5250000000000004,"15min":23.684999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.079,"5min":1.2130000000000001,"15min":1.1890000000000001},"min":{"1min":0.33400000000000002,"5min":0.315,"15min":0.27200000000000002},"max":{"1min":3.8610000000000002,"5min":6.1929999999999996,"15min":23.649000000000001},"last":0.92100000000000004}]},{"osd":5,"last update":"Tue Sep 10 18:34:58 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.2789999999999999,"15min":1.343},"min":{"1min":0.34599999999999997,"5min":0.314,"15min":0.314},"max":{"1min":3.4849999999999999,"5min":6.601,"15min":23.719999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.3360000000000001,"15min":1.3109999999999999},"min":{"1min":0.46200000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.3109999999999999,"5min":6.2249999999999996,"15min":23.707000000000001},"last":5.5540000000000003}]},{"osd":6,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.587,"15min":1.236},"min":{"1min":0.49399999999999999,"5min":0.32200000000000001,"15min":0.317},"max":{"1min":4.2720000000000002,"5min":31.815999999999999,"15min":31.815999999999999},"last":5.6029999999999998},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.8129999999999999,"15min":1.5},"min":{"1min":0.44500000000000001,"5min":0.33400000000000002,"15min":0.315},"max":{"1min":4.1870000000000003,"5min":31.742000000000001,"15min":31.742000000000001},"last":0.81100000000000005}]},{"osd":7,"last update":"Tue Sep 10 18:34:47 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.327,"15min":1.282},"min":{"1min":0.39300000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.3369999999999997,"5min":6.5730000000000004,"15min":23.748999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.373,"5min":1.454,"15min":1.3340000000000001},"min":{"1min":0.44800000000000001,"5min":0.35199999999999998,"15min":0.307},"max":{"1min":4.2809999999999997,"5min":12.612,"15min":25.058},"last":0.94699999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411523,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":411928,"kb_used_data":102032,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93333224,"statfs":{"total":95995035648,"available":95573221376,"internally_reserved":0,"allocated":104480768,"data_stored":103978123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:50 2024","interfaces":[{"interface":"back","average":{"1min":2.2490000000000001,"5min":1.544,"15min":1.458},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":26.585999999999999,"5min":26.585999999999999,"15min":43.890000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":2.4060000000000001,"5min":1.7310000000000001,"15min":1.669},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.16900000000000001},"max":{"1min":26.48,"5min":26.48,"15min":44.331000000000003},"last":2.355}]},{"osd":1,"last update":"Tue Sep 10 18:34:50 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.345,"15min":1.4419999999999999},"min":{"1min":0.251,"5min":0.19400000000000001,"15min":0.193},"max":{"1min":2.3479999999999999,"5min":15.750999999999999,"15min":44.575000000000003},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.3680000000000001,"15min":1.246},"min":{"1min":0.21199999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.9750000000000001,"5min":14.988,"15min":44.368000000000002},"last":0.40999999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:35:10 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":2.101,"15min":1.792},"min":{"1min":0.441,"5min":0.26100000000000001,"15min":0.251},"max":{"1min":4.8550000000000004,"5min":61.331000000000003,"15min":61.331000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":2.153,"15min":1.861},"min":{"1min":0.28999999999999998,"5min":0.23499999999999999,"15min":0.214},"max":{"1min":6.1429999999999998,"5min":61.250999999999998,"15min":61.250999999999998},"last":0.38500000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":1.1879999999999999,"15min":1.016},"min":{"1min":0.30199999999999999,"5min":0.28699999999999998,"15min":0.24199999999999999},"max":{"1min":2.1920000000000002,"5min":12.555999999999999,"15min":12.555999999999999},"last":0.432},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.0589999999999999,"15min":0.92600000000000005},"min":{"1min":0.35099999999999998,"5min":0.33000000000000002,"15min":0.317},"max":{"1min":1.105,"5min":7.0730000000000004,"15min":12.130000000000001},"last":0.48299999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:35:00 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.2010000000000001,"15min":1.0089999999999999},"min":{"1min":0.32100000000000001,"5min":0.30299999999999999,"15min":0.27500000000000002},"max":{"1min":5.2919999999999998,"5min":7.1609999999999996,"15min":12.106},"last":3.7210000000000001},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.48,"15min":1.1060000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":6.8810000000000002,"5min":12.529999999999999,"15min":12.529999999999999},"last":0.53200000000000003}]},{"osd":6,"last update":"Tue Sep 10 18:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.177,"15min":1.1160000000000001},"min":{"1min":0.38500000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":4.6440000000000001,"5min":5.9989999999999997,"15min":5.9989999999999997},"last":4.4139999999999997},{"interface":"front","average":{"1min":0.90600000000000003,"5min":1.105,"15min":0.96999999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.30099999999999999},"max":{"1min":5.4880000000000004,"5min":7.1349999999999998,"15min":7.1349999999999998},"last":0.57999999999999996}]},{"osd":7,"last update":"Tue Sep 10 18:34:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.198,"15min":1.0780000000000001},"min":{"1min":0.46300000000000002,"5min":0.35399999999999998,"15min":0.30499999999999999},"max":{"1min":4.0430000000000001,"5min":10.68,"15min":10.68},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.341,"15min":1.129},"min":{"1min":0.40100000000000002,"5min":0.376,"15min":0.307},"max":{"1min":2.2349999999999999,"5min":12.596,"15min":12.596},"last":1.032}]}]},{"osd":1,"up_from":13,"seq":55834575048,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":876468,"kb_used_data":286572,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":92868684,"statfs":{"total":95995035648,"available":95097532416,"internally_reserved":0,"allocated":293449728,"data_stored":292919021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:34:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2450000000000001,"15min":1.5329999999999999},"min":{"1min":0.20100000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":8.0030000000000001,"5min":14.079000000000001,"15min":80.456999999999994},"last":3.7559999999999998},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3240000000000001,"15min":1.6160000000000001},"min":{"1min":0.247,"5min":0.20399999999999999,"15min":0.187},"max":{"1min":5.6980000000000004,"5min":5.6980000000000004,"15min":80.578000000000003},"last":3.4279999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:34:43 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.1779999999999999,"15min":1.1910000000000001},"min":{"1min":0.24399999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.8380000000000001,"5min":5.8380000000000001,"15min":13.035},"last":3.5009999999999999},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.2050000000000001,"15min":1.1399999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":2.302,"5min":12.222,"15min":12.222},"last":3.669}]},{"osd":3,"last update":"Tue Sep 10 18:35:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.294,"15min":1.639},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.22600000000000001},"max":{"1min":6.5129999999999999,"5min":8.0399999999999991,"15min":43.076000000000001},"last":6.5129999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.296,"15min":1.6870000000000001},"min":{"1min":0.38200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":4.6509999999999998,"5min":5.8879999999999999,"15min":41.93},"last":3.6019999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:34:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.26,"15min":1.079},"min":{"1min":0.312,"5min":0.312,"15min":0.29799999999999999},"max":{"1min":5.9930000000000003,"5min":5.9930000000000003,"15min":7.2859999999999996},"last":3.5659999999999998},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.3979999999999999,"15min":1.2170000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.28299999999999997},"max":{"1min":2.3969999999999998,"5min":7.8769999999999998,"15min":19.393000000000001},"last":3.714}]},{"osd":5,"last update":"Tue Sep 10 18:34:59 2024","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":1.3919999999999999,"15min":1.2789999999999999},"min":{"1min":0.32800000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.5570000000000004,"5min":7.2110000000000003,"15min":15.48},"last":3.8740000000000001},{"interface":"front","average":{"1min":2.1749999999999998,"5min":1.4590000000000001,"15min":1.208},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.308},"max":{"1min":8.1050000000000004,"5min":11.414999999999999,"15min":11.414999999999999},"last":3.7999999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:34:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.2170000000000001,"15min":1.2909999999999999},"min":{"1min":0.41499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":8.1340000000000003,"5min":10.039,"15min":10.039},"last":3.823},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.373,"15min":1.389},"min":{"1min":0.42699999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":5.9240000000000004,"5min":6.2670000000000003,"15min":7.3369999999999997},"last":3.6560000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:34:43 2024","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.4119999999999999,"15min":1.1879999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":8.1609999999999996,"5min":10.005000000000001,"15min":15.545999999999999},"last":3.8450000000000002},{"interface":"front","average":{"1min":1.9870000000000001,"5min":1.5009999999999999,"15min":1.1890000000000001},"min":{"1min":0.50600000000000001,"5min":0.35699999999999998,"15min":0.313},"max":{"1min":6.0300000000000002,"5min":6.3049999999999997,"15min":7.359},"last":3.6349999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738572,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":525588,"kb_used_data":224076,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93219564,"statfs":{"total":95995035648,"available":95456833536,"internally_reserved":0,"allocated":229453824,"data_stored":228948254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:34:21 2024","interfaces":[{"interface":"back","average":{"1min":9.0549999999999997,"5min":3.1669999999999998,"15min":1.8899999999999999},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":146.44499999999999,"5min":146.44499999999999,"15min":146.44499999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":9.9619999999999997,"5min":3.2679999999999998,"15min":2.0190000000000001},"min":{"1min":0.38300000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":148.77199999999999,"5min":148.77199999999999,"15min":148.77199999999999},"last":0.26300000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:34:51 2024","interfaces":[{"interface":"back","average":{"1min":9.0990000000000002,"5min":2.7970000000000002,"15min":1.8640000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.189},"max":{"1min":146.56999999999999,"5min":146.56999999999999,"15min":146.56999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":9.1010000000000009,"5min":2.7040000000000002,"15min":1.8480000000000001},"min":{"1min":0.25,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":146.601,"5min":146.601,"15min":146.601},"last":0.59999999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:34:19 2024","interfaces":[{"interface":"back","average":{"1min":8.3469999999999995,"5min":2.8690000000000002,"15min":1.8899999999999999},"min":{"1min":0.28599999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":140.68299999999999,"5min":140.68299999999999,"15min":140.68299999999999},"last":2.113},{"interface":"front","average":{"1min":8.4179999999999993,"5min":3.004,"15min":1.8740000000000001},"min":{"1min":0.308,"5min":0.26500000000000001,"15min":0.23899999999999999},"max":{"1min":140.572,"5min":140.572,"15min":140.572},"last":0.71499999999999997}]},{"osd":4,"last update":"Tue Sep 10 18:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":1.1279999999999999,"15min":1.1319999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.246},"max":{"1min":1.9830000000000001,"5min":8.3699999999999992,"15min":17.151},"last":0.51900000000000002},{"interface":"front","average":{"1min":1.004,"5min":1.0580000000000001,"15min":1.127},"min":{"1min":0.31,"5min":0.29899999999999999,"15min":0.25900000000000001},"max":{"1min":3.3849999999999998,"5min":3.7919999999999998,"15min":18.728999999999999},"last":0.438}]},{"osd":5,"last update":"Tue Sep 10 18:35:05 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.151,"15min":1.2709999999999999},"min":{"1min":0.32300000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":8.1259999999999994,"5min":8.1259999999999994,"15min":18.77},"last":0.39400000000000002},{"interface":"front","average":{"1min":2.0230000000000001,"5min":1.3460000000000001,"15min":1.256},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.29099999999999998},"max":{"1min":20.460000000000001,"5min":20.460000000000001,"15min":20.460000000000001},"last":0.53500000000000003}]},{"osd":6,"last update":"Tue Sep 10 18:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":1.1799999999999999,"15min":1.2350000000000001},"min":{"1min":0.36299999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":2.0190000000000001,"5min":8.3940000000000001,"15min":8.3940000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0900000000000001,"15min":1.306},"min":{"1min":0.34399999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.3599999999999999,"5min":3.9079999999999999,"15min":9.7859999999999996},"last":0.77800000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:34:38 2024","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":1.371,"15min":1.228},"min":{"1min":0.377,"5min":0.32200000000000001,"15min":0.26700000000000002},"max":{"1min":10.75,"5min":10.75,"15min":13.711},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.583,"5min":1.2090000000000001,"15min":1.159},"min":{"1min":0.378,"5min":0.378,"15min":0.33500000000000002},"max":{"1min":11.218999999999999,"5min":11.218999999999999,"15min":15.032},"last":0.57399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":220397568,"data_stored":220397568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":254939136,"data_stored":254935064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":93847552,"data_stored":93847552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":436338688,"data_stored":436338688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":97452032,"data_stored":97452032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":278073344,"data_stored":278069272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":168820736,"data_stored":168820736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":279908352,"data_stored":279904280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:35:28.189 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-10T18:35:28.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pg_num 41 2024-09-10T18:35:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:28.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:28.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:28.245 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:28.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:28.315 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:28.351 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:28.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:28.381 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:28.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:28.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:28 smithi028 ceph-mon[39220]: from='client.25501 v1:172.21.15.28:0/1051867109' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:35:28.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:28 smithi028 ceph-mon[39220]: pgmap v631: 178 pgs: 178 active+clean; 430 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.79k op/s 2024-09-10T18:35:28.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:28 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:28.420 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:28 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:28.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:28 smithi028 ceph-mon[29291]: from='client.25501 v1:172.21.15.28:0/1051867109' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:35:28.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:28 smithi028 ceph-mon[29291]: pgmap v631: 178 pgs: 178 active+clean; 430 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.79k op/s 2024-09-10T18:35:28.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:28 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:28.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:28 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:28.483 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:28.509 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:28.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:28.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:28 smithi204 ceph-mon[32357]: from='client.25501 v1:172.21.15.28:0/1051867109' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:35:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:28 smithi204 ceph-mon[32357]: pgmap v631: 178 pgs: 178 active+clean; 430 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.79k op/s 2024-09-10T18:35:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:28 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:28 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:28.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:28.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:28.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:28.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:28.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:28.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:28.911 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:28.944 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:28.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:28.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:29.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:29.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:29.186 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:29.217 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:29.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:29.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:29.319 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:29.351 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:29.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:29.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:29.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:29 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:35:29.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:29 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:35:29.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:29 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:29.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:29 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:35:29.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:29 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:35:29.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:29 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:29.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:29.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:29.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:29.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:29.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:29 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:35:29.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:29 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:35:29.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:29 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:35:29.626 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:29.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:29.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:29.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:29.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:29.873 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:29.907 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:29.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:29.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:30.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:30.049 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:30.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:30.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:30.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:30.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:30.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:30.294 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:30.371 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:30.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:30.472 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:30.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:30 smithi028 ceph-mon[29291]: pgmap v632: 178 pgs: 178 active+clean; 430 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 635 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T18:35:30.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:30 smithi028 ceph-mon[39220]: pgmap v632: 178 pgs: 178 active+clean; 430 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 635 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T18:35:30.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:30.506 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:30.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:30 smithi204 ceph-mon[32357]: pgmap v632: 178 pgs: 178 active+clean; 430 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 635 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T18:35:30.607 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:30.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:30.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:30.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:30.730 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Removed 134588 objects 2024-09-10T18:35:30.730 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Clean up completed and total clean up time :107.648 2024-09-10T18:35:30.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:30.760 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:30.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:30.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:30.862 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:30.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:30.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:30.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:31.009 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:31.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:31.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:31.110 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:31.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:31.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:31.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:31.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:31.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:31.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:31.357 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:31.379 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:31.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:31.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:31.481 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:31.502 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:31.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:31.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:35:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:35:31] "GET /metrics HTTP/1.1" 200 224540 "" "Prometheus/2.43.0" 2024-09-10T18:35:31.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:31.626 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:31.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:31.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:31.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:31.749 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:31.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:31.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:31.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:31.873 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:31.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:31.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:31.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:31.997 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:31.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:31.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:32.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:32.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:31 smithi204 ceph-mon[32357]: pgmap v633: 178 pgs: 178 active+clean; 101 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T18:35:32.119 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:31 smithi028 ceph-mon[29291]: pgmap v633: 178 pgs: 178 active+clean; 101 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T18:35:32.120 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:31 smithi028 ceph-mon[39220]: pgmap v633: 178 pgs: 178 active+clean; 101 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T18:35:32.122 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:32.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:32.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:32.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:32.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:32.247 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:32.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:32.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:32.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:32.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:32.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:32.471 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:32.497 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:32.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:32.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:32.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:35:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:35:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:35:32.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:32.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:32.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:32.747 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:32.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:32.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:32.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:32.872 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:32.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:32.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:32.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:33.000 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:33.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:33.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:33.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:33.127 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:33.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:33.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:33.213 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:35:33.213 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:35:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:35:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:35:33.225 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:35:33.225 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:35:33.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:33.240 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:35:33.240 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:35:33.251 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:33.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:33.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:33.256 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:35:33.257 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:35:33.273 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:35:33.273 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:35:33.289 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:35:33.289 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:35:33.305 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:35:33.305 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:35:33.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:33.375 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:33.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:33.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:33.477 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:33.502 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:33.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:33.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:33.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:33.627 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:33.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:33.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:33.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:33.751 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:33.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:33.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:33.852 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:33.878 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:33.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:33.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:33.934 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:35:33.934 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-10T18:35:33.934 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-10T18:35:33.934 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-10T18:35:33.934 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:35:33.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:34.002 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:34.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:34.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:34.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:34.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:34.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:34.229 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:34.255 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:34.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:34.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:34.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:34.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:34.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:34.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:34 smithi028 ceph-mon[39220]: pgmap v634: 178 pgs: 178 active+clean; 30 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T18:35:34.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:34 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2520900929' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:35:34.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:34 smithi028 ceph-mon[29291]: pgmap v634: 178 pgs: 178 active+clean; 30 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T18:35:34.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:34 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2520900929' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:35:34.480 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:34.505 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:34.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:34.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:34.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:34 smithi204 ceph-mon[32357]: pgmap v634: 178 pgs: 178 active+clean; 30 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T18:35:34.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:34 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2520900929' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:35:34.608 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:34.632 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:34.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:34.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:34.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:34.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:34.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:34.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:34.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:34.985 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:35.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:35.012 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:35.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:35.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:35:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:35:35.112 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:35.116 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:35:35.138 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:35.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:35.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:35.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:35 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:35:35.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:35 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:35:35.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:35 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2797686492' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:35:35.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:35 smithi028 ceph-mon[39220]: from='client.35525 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:35:35.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:35 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:35:35.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:35 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:35:35.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:35 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2797686492' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:35:35.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:35 smithi028 ceph-mon[29291]: from='client.35525 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:35:35.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:35.264 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:35.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:35.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:35.366 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:35.370 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:35.391 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:35.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:35.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:35.493 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:35.518 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:35.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:35.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:35.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:35 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:35:35.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:35 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:35:35.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:35 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2797686492' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:35:35.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:35 smithi204 ceph-mon[32357]: from='client.35525 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:35:35.620 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:35.647 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:35.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:35.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:35.749 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:35.774 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:35.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:35.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:35.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-10T18:35:35.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:35.899 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:35.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:35.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:36.000 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:36.026 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:36.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:36.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:36.128 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:36.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:36.153 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:36.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:36.254 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:36.278 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:36.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:36.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:36.380 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:36.404 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:36.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:36.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:36.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:36 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:36.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:36 smithi028 ceph-mon[39220]: pgmap v635: 178 pgs: 178 active+clean; 18 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-10T18:35:36.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:36 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:35:36.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:36 smithi028 ceph-mon[39220]: from='client.35525 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:35:36.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:36 smithi028 ceph-mon[39220]: osdmap e305: 8 total, 8 up, 8 in 2024-09-10T18:35:36.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:36 smithi028 ceph-mon[39220]: osdmap e306: 8 total, 8 up, 8 in 2024-09-10T18:35:36.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:36 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:36.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:36 smithi028 ceph-mon[29291]: pgmap v635: 178 pgs: 178 active+clean; 18 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-10T18:35:36.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:36 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:35:36.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:36 smithi028 ceph-mon[29291]: from='client.35525 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:35:36.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:36 smithi028 ceph-mon[29291]: osdmap e305: 8 total, 8 up, 8 in 2024-09-10T18:35:36.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:36 smithi028 ceph-mon[29291]: osdmap e306: 8 total, 8 up, 8 in 2024-09-10T18:35:36.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:36.528 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:36.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:36.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:36 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:36 smithi204 ceph-mon[32357]: pgmap v635: 178 pgs: 178 active+clean; 18 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-10T18:35:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:36 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:35:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:36 smithi204 ceph-mon[32357]: from='client.35525 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:35:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:36 smithi204 ceph-mon[32357]: osdmap e305: 8 total, 8 up, 8 in 2024-09-10T18:35:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:36 smithi204 ceph-mon[32357]: osdmap e306: 8 total, 8 up, 8 in 2024-09-10T18:35:36.629 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:36.655 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:36.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:36.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:36.757 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:36.782 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:36.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:36.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:36.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:36.907 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:36.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:36.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:36.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:35:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:37.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:37.031 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:37.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:37.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:37.133 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:37.159 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:37.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:37.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:37.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:37.284 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:37.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:37.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:37.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:37.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:37.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:37.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:37.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:37.533 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:37.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:37.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:37.635 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:37.661 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:37.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:37.763 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:37.787 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:37.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:37.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:37.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:37.911 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:37.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:37.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:38.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:38.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:38.137 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:38.163 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:38.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:38.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:38.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:35:38.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:38.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:38.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:38.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:38.444 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:38.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:38.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:38.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:38 smithi028 ceph-mon[29291]: pgmap v638: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 860 op/s 2024-09-10T18:35:38.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:38 smithi028 ceph-mon[39220]: pgmap v638: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 860 op/s 2024-09-10T18:35:38.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:38.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:38.568 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:38.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:38 smithi204 ceph-mon[32357]: pgmap v638: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 860 op/s 2024-09-10T18:35:38.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:38.698 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:38.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:38.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:38.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:38.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:38.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:38.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:38.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:39.048 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:39.074 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:39.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:39.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:39.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:39.199 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:39.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:39.301 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:39.327 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:39.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:39.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:39.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:39.452 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:39.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:39.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:35:39.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:35:39.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:39.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:39.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:35:39.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:39.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:39.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:39.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:39.830 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:39.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:39.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:39.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:39.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:39.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:39.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:40.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:40.078 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:40.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:40.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:40.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:40 smithi028 ceph-mon[29291]: pgmap v639: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 201 op/s 2024-09-10T18:35:40.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:40 smithi028 ceph-mon[39220]: pgmap v639: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 201 op/s 2024-09-10T18:35:40.180 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:40.206 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:40.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:40.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:40.277 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: specified pg_num 41 < pg_num_max 32 2024-09-10T18:35:40.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:40.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:40.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:40.432 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:40.458 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:40.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:40.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:40.560 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:40.585 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:40.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:40.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:40.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:40 smithi204 ceph-mon[32357]: pgmap v639: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 201 op/s 2024-09-10T18:35:40.687 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:40.713 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:40.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:40.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:40.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:40.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:40.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:40.939 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:40.965 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:40.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:40.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:41.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:41.090 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:41.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:41.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:41.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:41.217 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:41.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:41.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:41.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:41 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3961567769' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-09-10T18:35:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:41 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3961567769' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-09-10T18:35:41.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:41.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:41.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:41.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:41.443 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:41.469 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:41.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:41.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:41.571 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:41.579 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:35:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:35:41] "GET /metrics HTTP/1.1" 200 224357 "" "Prometheus/2.43.0" 2024-09-10T18:35:41.580 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-10T18:35:41.580 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-10T18:35:41.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-10T18:35:41.595 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:41.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:41.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:41 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3961567769' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-09-10T18:35:41.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:41.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:41.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:41.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:41.843 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:41.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:41.944 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:41.970 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:41.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:41.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:42.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:42.097 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:42.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:42.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:42.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:42.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:42.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:42.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:42.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:42.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:42.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:42.470 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:42.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:42.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:42.499 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:42 smithi028 ceph-mon[39220]: pgmap v640: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-10T18:35:42.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:42 smithi028 ceph-mon[29291]: pgmap v640: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-10T18:35:42.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:42.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:42.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:42 smithi204 ceph-mon[32357]: pgmap v640: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-10T18:35:42.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:35:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:35:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:35:42.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:42.719 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:42.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:42.762 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:42.763 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:42.767 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:35:42.767 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-10T18:35:42.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-10T18:35:42.767 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:35:42.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:139702 7.f 7:f2f55ab9:::benchmark_data_smithi028_77856_object18602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:35:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:43.935577+0000", 2024-09-10T18:35:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 598.82730519400002, 2024-09-10T18:35:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15684685600000001, 2024-09-10T18:35:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:35:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:35:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:35:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 139702 2024-09-10T18:35:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935577+0000", 2024-09-10T18:35:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935577+0000", 2024-09-10T18:35:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935576+0000", 2024-09-10T18:35:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:35:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935580+0000", 2024-09-10T18:35:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6879999999999998e-06 2024-09-10T18:35:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935588+0000", 2024-09-10T18:35:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9999999999999996e-06 2024-09-10T18:35:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.935594+0000", 2024-09-10T18:35:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2820000000000004e-06 2024-09-10T18:35:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.982138+0000", 2024-09-10T18:35:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.046544461000000002 2024-09-10T18:35:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.982248+0000", 2024-09-10T18:35:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110157 2024-09-10T18:35:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:35:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.982382+0000", 2024-09-10T18:35:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000133895 2024-09-10T18:35:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:35:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.989604+0000", 2024-09-10T18:35:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007222366 2024-09-10T18:35:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:43.992745+0000", 2024-09-10T18:35:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031404810000000001 2024-09-10T18:35:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:44.092292+0000", 2024-09-10T18:35:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.099547449999999996 2024-09-10T18:35:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:44.092339+0000", 2024-09-10T18:35:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6990000000000002e-05 2024-09-10T18:35:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:44.092423+0000", 2024-09-10T18:35:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4138e-05 2024-09-10T18:35:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:140670 7.f 7:ff7f13ba:::benchmark_data_smithi028_77856_object19570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:35:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:45.605622+0000", 2024-09-10T18:35:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 597.15725951700006, 2024-09-10T18:35:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.17139099099999999, 2024-09-10T18:35:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:35:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:35:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:35:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 140670 2024-09-10T18:35:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605622+0000", 2024-09-10T18:35:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605622+0000", 2024-09-10T18:35:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605620+0000", 2024-09-10T18:35:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:35:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605628+0000", 2024-09-10T18:35:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7510000000000008e-06 2024-09-10T18:35:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605641+0000", 2024-09-10T18:35:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2692e-05 2024-09-10T18:35:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.605647+0000", 2024-09-10T18:35:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4169999999999997e-06 2024-09-10T18:35:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.612161+0000", 2024-09-10T18:35:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0065135159999999996 2024-09-10T18:35:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.612264+0000", 2024-09-10T18:35:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103127 2024-09-10T18:35:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:35:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.612383+0000", 2024-09-10T18:35:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000119157 2024-09-10T18:35:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:35:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.618786+0000", 2024-09-10T18:35:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006402942 2024-09-10T18:35:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.620538+0000", 2024-09-10T18:35:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001752368 2024-09-10T18:35:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.776944+0000", 2024-09-10T18:35:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.156405708 2024-09-10T18:35:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.776979+0000", 2024-09-10T18:35:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4829999999999997e-05 2024-09-10T18:35:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.777013+0000", 2024-09-10T18:35:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4496999999999999e-05 2024-09-10T18:35:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:140692 7.f 7:f303c2e4:::benchmark_data_smithi028_77856_object19592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:35:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:45.628709+0000", 2024-09-10T18:35:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 597.13417263099996, 2024-09-10T18:35:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14840287099999999, 2024-09-10T18:35:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:35:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:35:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:35:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 140692 2024-09-10T18:35:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628709+0000", 2024-09-10T18:35:42.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628709+0000", 2024-09-10T18:35:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628707+0000", 2024-09-10T18:35:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:35:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628714+0000", 2024-09-10T18:35:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4229999999999999e-06 2024-09-10T18:35:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628730+0000", 2024-09-10T18:35:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5991999999999999e-05 2024-09-10T18:35:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628738+0000", 2024-09-10T18:35:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0469999999999996e-06 2024-09-10T18:35:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628829+0000", 2024-09-10T18:35:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1197000000000001e-05 2024-09-10T18:35:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.628955+0000", 2024-09-10T18:35:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012585100000000001 2024-09-10T18:35:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:35:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.629089+0000", 2024-09-10T18:35:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013479200000000001 2024-09-10T18:35:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.634532+0000", 2024-09-10T18:35:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054427750000000004 2024-09-10T18:35:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:35:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.642486+0000", 2024-09-10T18:35:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007953669 2024-09-10T18:35:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.777048+0000", 2024-09-10T18:35:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13456268499999999 2024-09-10T18:35:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.777082+0000", 2024-09-10T18:35:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3025000000000003e-05 2024-09-10T18:35:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:45.777112+0000", 2024-09-10T18:35:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0516999999999999e-05 2024-09-10T18:35:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:141317 7.f 7:f85eab8a:::benchmark_data_smithi028_77856_object20217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-10T18:35:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:25:46.778960+0000", 2024-09-10T18:35:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 595.98392171299997, 2024-09-10T18:35:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15742622000000001, 2024-09-10T18:35:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:35:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:35:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:35:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 141317 2024-09-10T18:35:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778960+0000", 2024-09-10T18:35:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778960+0000", 2024-09-10T18:35:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778958+0000", 2024-09-10T18:35:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:35:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778965+0000", 2024-09-10T18:35:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7599999999999997e-06 2024-09-10T18:35:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778973+0000", 2024-09-10T18:35:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7770000000000002e-06 2024-09-10T18:35:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.778977+0000", 2024-09-10T18:35:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5229999999999997e-06 2024-09-10T18:35:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.789488+0000", 2024-09-10T18:35:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010510472999999999 2024-09-10T18:35:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.789609+0000", 2024-09-10T18:35:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121301 2024-09-10T18:35:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:35:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.789743+0000", 2024-09-10T18:35:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013393199999999999 2024-09-10T18:35:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.793571+0000", 2024-09-10T18:35:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0038282289999999998 2024-09-10T18:35:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:35:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.810048+0000", 2024-09-10T18:35:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0164768 2024-09-10T18:35:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.936295+0000", 2024-09-10T18:35:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.126246568 2024-09-10T18:35:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.936346+0000", 2024-09-10T18:35:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1708e-05 2024-09-10T18:35:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:25:46.936386+0000", 2024-09-10T18:35:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0021000000000002e-05 2024-09-10T18:35:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181160 7.7 7:e2311557:::benchmark_data_smithi028_77856_object60060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:35:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.037797+0000", 2024-09-10T18:35:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 557.72508504999996, 2024-09-10T18:35:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.155207649, 2024-09-10T18:35:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:35:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:35:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:35:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181160 2024-09-10T18:35:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037797+0000", 2024-09-10T18:35:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037797+0000", 2024-09-10T18:35:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037795+0000", 2024-09-10T18:35:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:35:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037810+0000", 2024-09-10T18:35:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4810000000000001e-05 2024-09-10T18:35:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037822+0000", 2024-09-10T18:35:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2568000000000001e-05 2024-09-10T18:35:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.037829+0000", 2024-09-10T18:35:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.562e-06 2024-09-10T18:35:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.038647+0000", 2024-09-10T18:35:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081780299999999998 2024-09-10T18:35:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.038755+0000", 2024-09-10T18:35:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108941 2024-09-10T18:35:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-10T18:35:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.038876+0000", 2024-09-10T18:35:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000120259 2024-09-10T18:35:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:35:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041587+0000", 2024-09-10T18:35:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027112709999999999 2024-09-10T18:35:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191165+0000", 2024-09-10T18:35:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14957848300000001 2024-09-10T18:35:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.192642+0000", 2024-09-10T18:35:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001476148 2024-09-10T18:35:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.193004+0000", 2024-09-10T18:35:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00036274700000000002 2024-09-10T18:35:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181165 7.6 7:69ba312f:::benchmark_data_smithi028_77856_object60065:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:35:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.041719+0000", 2024-09-10T18:35:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 557.72116277299995, 2024-09-10T18:35:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14957322100000001, 2024-09-10T18:35:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:35:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:35:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:35:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181165 2024-09-10T18:35:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041719+0000", 2024-09-10T18:35:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041719+0000", 2024-09-10T18:35:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041718+0000", 2024-09-10T18:35:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:35:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041723+0000", 2024-09-10T18:35:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.48e-06 2024-09-10T18:35:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041733+0000", 2024-09-10T18:35:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6520000000000001e-06 2024-09-10T18:35:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.041738+0000", 2024-09-10T18:35:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.553e-06 2024-09-10T18:35:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044567+0000", 2024-09-10T18:35:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028282770000000001 2024-09-10T18:35:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044684+0000", 2024-09-10T18:35:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117491 2024-09-10T18:35:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:35:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.044768+0000", 2024-09-10T18:35:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4078000000000004e-05 2024-09-10T18:35:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.049844+0000", 2024-09-10T18:35:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050756799999999999 2024-09-10T18:35:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:35:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.050265+0000", 2024-09-10T18:35:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00042087299999999999 2024-09-10T18:35:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191082+0000", 2024-09-10T18:35:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14081718500000001 2024-09-10T18:35:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191174+0000", 2024-09-10T18:35:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1875999999999999e-05 2024-09-10T18:35:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191292+0000", 2024-09-10T18:35:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000118331 2024-09-10T18:35:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:181174 7.5 e142/93 7:a54ae781:::benchmark_data_smithi028_77856_object60074:head v 142'11303, mlcod=142'11301)", 2024-09-10T18:35:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.048963+0000", 2024-09-10T18:35:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 557.71391846500001, 2024-09-10T18:35:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14980299699999999, 2024-09-10T18:35:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048963+0000", 2024-09-10T18:35:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048963+0000", 2024-09-10T18:35:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048962+0000", 2024-09-10T18:35:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:35:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048969+0000", 2024-09-10T18:35:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.08e-06 2024-09-10T18:35:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048979+0000", 2024-09-10T18:35:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.006e-05 2024-09-10T18:35:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.048985+0000", 2024-09-10T18:35:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8780000000000003e-06 2024-09-10T18:35:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.049055+0000", 2024-09-10T18:35:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9560000000000005e-05 2024-09-10T18:35:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.049072+0000", 2024-09-10T18:35:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7476e-05 2024-09-10T18:35:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198732+0000", 2024-09-10T18:35:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14965962499999999 2024-09-10T18:35:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198766+0000", 2024-09-10T18:35:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4514000000000002e-05 2024-09-10T18:35:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:181175 7.5 e142/93 7:a07e4061:::benchmark_data_smithi028_77856_object60075:head v 142'11304, mlcod=142'11301)", 2024-09-10T18:35:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.051631+0000", 2024-09-10T18:35:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 557.71125054300001, 2024-09-10T18:35:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15340213799999999, 2024-09-10T18:35:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051631+0000", 2024-09-10T18:35:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051631+0000", 2024-09-10T18:35:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051630+0000", 2024-09-10T18:35:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:35:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051639+0000", 2024-09-10T18:35:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8189999999999998e-06 2024-09-10T18:35:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051649+0000", 2024-09-10T18:35:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0101999999999999e-05 2024-09-10T18:35:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051655+0000", 2024-09-10T18:35:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9490000000000001e-06 2024-09-10T18:35:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051705+0000", 2024-09-10T18:35:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0328000000000003e-05 2024-09-10T18:35:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.051722+0000", 2024-09-10T18:35:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6617999999999999e-05 2024-09-10T18:35:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205001+0000", 2024-09-10T18:35:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15327945300000001 2024-09-10T18:35:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205033+0000", 2024-09-10T18:35:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2257e-05 2024-09-10T18:35:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14808.0:181177 7.f 7:f9ebf4ca:::benchmark_data_smithi028_77856_object60077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-10T18:35:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.053108+0000", 2024-09-10T18:35:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 557.70977344999994, 2024-09-10T18:35:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15150366200000001, 2024-09-10T18:35:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:35:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14808", 2024-09-10T18:35:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2450227264", 2024-09-10T18:35:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181177 2024-09-10T18:35:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.053108+0000", 2024-09-10T18:35:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.053108+0000", 2024-09-10T18:35:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.053106+0000", 2024-09-10T18:35:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:35:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.053113+0000", 2024-09-10T18:35:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0520000000000004e-06 2024-09-10T18:35:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.190879+0000", 2024-09-10T18:35:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13776606999999999 2024-09-10T18:35:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.190891+0000", 2024-09-10T18:35:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1102e-05 2024-09-10T18:35:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.190985+0000", 2024-09-10T18:35:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4087999999999998e-05 2024-09-10T18:35:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191099+0000", 2024-09-10T18:35:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114473 2024-09-10T18:35:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:35:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191299+0000", 2024-09-10T18:35:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019984899999999999 2024-09-10T18:35:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.197973+0000", 2024-09-10T18:35:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066738869999999999 2024-09-10T18:35:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:35:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198142+0000", 2024-09-10T18:35:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000169006 2024-09-10T18:35:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.204473+0000", 2024-09-10T18:35:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063307729999999996 2024-09-10T18:35:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.204527+0000", 2024-09-10T18:35:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4588e-05 2024-09-10T18:35:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.204612+0000", 2024-09-10T18:35:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.475e-05 2024-09-10T18:35:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14808.0:181183 7.d e142/95 7:b1361f07:::benchmark_data_smithi028_77856_object60083:head v 142'11153, mlcod=142'11151)", 2024-09-10T18:35:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:26:25.055525+0000", 2024-09-10T18:35:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 557.70735708799998, 2024-09-10T18:35:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14994769499999999, 2024-09-10T18:35:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055525+0000", 2024-09-10T18:35:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055525+0000", 2024-09-10T18:35:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055523+0000", 2024-09-10T18:35:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:35:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.055532+0000", 2024-09-10T18:35:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0159999999999997e-06 2024-09-10T18:35:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191429+0000", 2024-09-10T18:35:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13589642099999999 2024-09-10T18:35:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.191435+0000", 2024-09-10T18:35:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9569999999999999e-06 2024-09-10T18:35:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198700+0000", 2024-09-10T18:35:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0072655549999999999 2024-09-10T18:35:42.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.824 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:42.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.198722+0000", 2024-09-10T18:35:42.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1307000000000001e-05 2024-09-10T18:35:42.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205091+0000", 2024-09-10T18:35:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063695779999999999 2024-09-10T18:35:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:26:25.205472+0000", 2024-09-10T18:35:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00038129400000000001 2024-09-10T18:35:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15117.0:170382 8.6 8:688b0bc1:::benchmark_data_smithi028_109242_object36161:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-10T18:35:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:30:10.978519+0000", 2024-09-10T18:35:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 331.78436254600001, 2024-09-10T18:35:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15547377100000001, 2024-09-10T18:35:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:35:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15117", 2024-09-10T18:35:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/121353060", 2024-09-10T18:35:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 170382 2024-09-10T18:35:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.978519+0000", 2024-09-10T18:35:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.978519+0000", 2024-09-10T18:35:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.978517+0000", 2024-09-10T18:35:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:35:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.978525+0000", 2024-09-10T18:35:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6359999999999994e-06 2024-09-10T18:35:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.978540+0000", 2024-09-10T18:35:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4792e-05 2024-09-10T18:35:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.978546+0000", 2024-09-10T18:35:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7960000000000001e-06 2024-09-10T18:35:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.983066+0000", 2024-09-10T18:35:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045201210000000002 2024-09-10T18:35:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.983187+0000", 2024-09-10T18:35:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121585 2024-09-10T18:35:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-09-10T18:35:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.983364+0000", 2024-09-10T18:35:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017670500000000001 2024-09-10T18:35:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:35:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.997893+0000", 2024-09-10T18:35:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014529179999999999 2024-09-10T18:35:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:11.133743+0000", 2024-09-10T18:35:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.135849677 2024-09-10T18:35:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:11.133908+0000", 2024-09-10T18:35:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000164919 2024-09-10T18:35:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:11.133951+0000", 2024-09-10T18:35:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3668000000000003e-05 2024-09-10T18:35:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:11.133993+0000", 2024-09-10T18:35:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1588000000000002e-05 2024-09-10T18:35:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15117.0:170388 8.6 8:67c70fd9:::benchmark_data_smithi028_109242_object36167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-10T18:35:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:30:10.980841+0000", 2024-09-10T18:35:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 331.78204065, 2024-09-10T18:35:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15331135000000001, 2024-09-10T18:35:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:35:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15117", 2024-09-10T18:35:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/121353060", 2024-09-10T18:35:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 170388 2024-09-10T18:35:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.980841+0000", 2024-09-10T18:35:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.980841+0000", 2024-09-10T18:35:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.980839+0000", 2024-09-10T18:35:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:35:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.980846+0000", 2024-09-10T18:35:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1969999999999999e-06 2024-09-10T18:35:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.980860+0000", 2024-09-10T18:35:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.38e-05 2024-09-10T18:35:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.980865+0000", 2024-09-10T18:35:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6210000000000001e-06 2024-09-10T18:35:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.989078+0000", 2024-09-10T18:35:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0082128789999999993 2024-09-10T18:35:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.989184+0000", 2024-09-10T18:35:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000106086 2024-09-10T18:35:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-09-10T18:35:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.989298+0000", 2024-09-10T18:35:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113318 2024-09-10T18:35:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:35:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:10.999946+0000", 2024-09-10T18:35:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010648283 2024-09-10T18:35:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:11.134028+0000", 2024-09-10T18:35:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13408160099999999 2024-09-10T18:35:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:11.134083+0000", 2024-09-10T18:35:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5206e-05 2024-09-10T18:35:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:11.134120+0000", 2024-09-10T18:35:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6808999999999999e-05 2024-09-10T18:35:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:11.134152+0000", 2024-09-10T18:35:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2808000000000002e-05 2024-09-10T18:35:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15117.0:173622 8.6 8:679bf3f0:::benchmark_data_smithi028_109242_object39401:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-09-10T18:35:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:30:14.779803+0000", 2024-09-10T18:35:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 327.983078326, 2024-09-10T18:35:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15561011, 2024-09-10T18:35:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:35:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15117", 2024-09-10T18:35:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/121353060", 2024-09-10T18:35:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 173622 2024-09-10T18:35:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.779803+0000", 2024-09-10T18:35:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.779803+0000", 2024-09-10T18:35:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.779801+0000", 2024-09-10T18:35:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:35:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.779810+0000", 2024-09-10T18:35:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7590000000000003e-06 2024-09-10T18:35:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.779827+0000", 2024-09-10T18:35:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7048999999999999e-05 2024-09-10T18:35:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.779834+0000", 2024-09-10T18:35:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2250000000000003e-06 2024-09-10T18:35:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.813829+0000", 2024-09-10T18:35:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.033995385000000003 2024-09-10T18:35:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.814039+0000", 2024-09-10T18:35:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00021000399999999999 2024-09-10T18:35:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-09-10T18:35:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.814237+0000", 2024-09-10T18:35:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019735 2024-09-10T18:35:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.818552+0000", 2024-09-10T18:35:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004315129 2024-09-10T18:35:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.822712+0000", 2024-09-10T18:35:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041595620000000003 2024-09-10T18:35:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:35:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.935346+0000", 2024-09-10T18:35:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.11263444 2024-09-10T18:35:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.935389+0000", 2024-09-10T18:35:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3339e-05 2024-09-10T18:35:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:14.935414+0000", 2024-09-10T18:35:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4267e-05 2024-09-10T18:35:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15117.0:177491 8.6 8:63c103fa:::benchmark_data_smithi028_109242_object43270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-10T18:35:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:30:19.568763+0000", 2024-09-10T18:35:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 323.19411917999997, 2024-09-10T18:35:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.17489908000000001, 2024-09-10T18:35:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:35:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15117", 2024-09-10T18:35:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/121353060", 2024-09-10T18:35:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 177491 2024-09-10T18:35:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.568763+0000", 2024-09-10T18:35:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.568763+0000", 2024-09-10T18:35:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.568761+0000", 2024-09-10T18:35:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:35:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.568767+0000", 2024-09-10T18:35:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5930000000000002e-06 2024-09-10T18:35:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.568777+0000", 2024-09-10T18:35:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2820000000000002e-06 2024-09-10T18:35:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.568782+0000", 2024-09-10T18:35:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7729999999999996e-06 2024-09-10T18:35:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.637153+0000", 2024-09-10T18:35:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.068370809000000005 2024-09-10T18:35:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.637452+0000", 2024-09-10T18:35:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029882500000000001 2024-09-10T18:35:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-09-10T18:35:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.637873+0000", 2024-09-10T18:35:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00042066599999999999 2024-09-10T18:35:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.644753+0000", 2024-09-10T18:35:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068805439999999997 2024-09-10T18:35:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:35:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.645799+0000", 2024-09-10T18:35:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010459169999999999 2024-09-10T18:35:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.743549+0000", 2024-09-10T18:35:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.097750291000000003 2024-09-10T18:35:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.743620+0000", 2024-09-10T18:35:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0461000000000002e-05 2024-09-10T18:35:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.743662+0000", 2024-09-10T18:35:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1730999999999997e-05 2024-09-10T18:35:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15117.0:177492 8.6 8:66cc57c8:::benchmark_data_smithi028_109242_object43271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-10T18:35:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:30:19.570479+0000", 2024-09-10T18:35:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 323.19240317800001, 2024-09-10T18:35:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.17340629799999999, 2024-09-10T18:35:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:35:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15117", 2024-09-10T18:35:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/121353060", 2024-09-10T18:35:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 177492 2024-09-10T18:35:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.570479+0000", 2024-09-10T18:35:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:42.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.570479+0000", 2024-09-10T18:35:42.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.570477+0000", 2024-09-10T18:35:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:35:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.570483+0000", 2024-09-10T18:35:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.579e-06 2024-09-10T18:35:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.570500+0000", 2024-09-10T18:35:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6141999999999999e-05 2024-09-10T18:35:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.570506+0000", 2024-09-10T18:35:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2210000000000002e-06 2024-09-10T18:35:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.641242+0000", 2024-09-10T18:35:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.07073662 2024-09-10T18:35:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.641352+0000", 2024-09-10T18:35:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109949 2024-09-10T18:35:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-09-10T18:35:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.641492+0000", 2024-09-10T18:35:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013988499999999999 2024-09-10T18:35:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.652310+0000", 2024-09-10T18:35:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010817614 2024-09-10T18:35:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:35:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.652325+0000", 2024-09-10T18:35:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4673e-05 2024-09-10T18:35:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.743825+0000", 2024-09-10T18:35:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.091500026999999998 2024-09-10T18:35:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.743852+0000", 2024-09-10T18:35:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7208000000000001e-05 2024-09-10T18:35:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:30:19.743885+0000", 2024-09-10T18:35:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3067e-05 2024-09-10T18:35:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35306.0:192353 9.9 9:906f3a78:::benchmark_data_smithi028_135703_object57762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-09-10T18:35:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:32.797068+0000", 2024-09-10T18:35:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.965814000999998, 2024-09-10T18:35:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14077943400000001, 2024-09-10T18:35:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:35:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35306", 2024-09-10T18:35:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:35:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192353 2024-09-10T18:35:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:32.797068+0000", 2024-09-10T18:35:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:32.797068+0000", 2024-09-10T18:35:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:32.797065+0000", 2024-09-10T18:35:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:35:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:32.797074+0000", 2024-09-10T18:35:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4950000000000008e-06 2024-09-10T18:35:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:32.836536+0000", 2024-09-10T18:35:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.039461765000000003 2024-09-10T18:35:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:32.836547+0000", 2024-09-10T18:35:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1272e-05 2024-09-10T18:35:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:32.917740+0000", 2024-09-10T18:35:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.081193389000000005 2024-09-10T18:35:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:32.917861+0000", 2024-09-10T18:35:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121153 2024-09-10T18:35:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,7", 2024-09-10T18:35:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:32.917976+0000", 2024-09-10T18:35:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114381 2024-09-10T18:35:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:32.928466+0000", 2024-09-10T18:35:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010490414 2024-09-10T18:35:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:32.928543+0000", 2024-09-10T18:35:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7269999999999997e-05 2024-09-10T18:35:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:35:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:32.937707+0000", 2024-09-10T18:35:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091638359999999999 2024-09-10T18:35:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:32.937804+0000", 2024-09-10T18:35:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7068e-05 2024-09-10T18:35:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:32.937847+0000", 2024-09-10T18:35:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2877999999999998e-05 2024-09-10T18:35:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35306.0:218740 9.8 e298/246 9:180037e8:::benchmark_data_smithi028_135703_object84149:head v 298'13396, mlcod=298'13394)", 2024-09-10T18:35:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:54.118017+0000", 2024-09-10T18:35:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 48.644864984999998, 2024-09-10T18:35:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.17224061900000001, 2024-09-10T18:35:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.118017+0000", 2024-09-10T18:35:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.118017+0000", 2024-09-10T18:35:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.118016+0000", 2024-09-10T18:35:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:35:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.118024+0000", 2024-09-10T18:35:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3820000000000004e-06 2024-09-10T18:35:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.118036+0000", 2024-09-10T18:35:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1666e-05 2024-09-10T18:35:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.118041+0000", 2024-09-10T18:35:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.541e-06 2024-09-10T18:35:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.118184+0000", 2024-09-10T18:35:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014307299999999999 2024-09-10T18:35:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.118203+0000", 2024-09-10T18:35:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8972999999999999e-05 2024-09-10T18:35:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.122883+0000", 2024-09-10T18:35:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004679384 2024-09-10T18:35:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.290257+0000", 2024-09-10T18:35:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16737487200000001 2024-09-10T18:35:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35306.0:218756 9.5 9:a44278ac:::benchmark_data_smithi028_135703_object84165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-10T18:35:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:54.118122+0000", 2024-09-10T18:35:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 48.644760011000002, 2024-09-10T18:35:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.179937702, 2024-09-10T18:35:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:35:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35306", 2024-09-10T18:35:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:35:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 218756 2024-09-10T18:35:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.118122+0000", 2024-09-10T18:35:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.118122+0000", 2024-09-10T18:35:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.118120+0000", 2024-09-10T18:35:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:35:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.118127+0000", 2024-09-10T18:35:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3610000000000003e-06 2024-09-10T18:35:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.118141+0000", 2024-09-10T18:35:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4299e-05 2024-09-10T18:35:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.118146+0000", 2024-09-10T18:35:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5279999999999997e-06 2024-09-10T18:35:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.120501+0000", 2024-09-10T18:35:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023555099999999999 2024-09-10T18:35:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.120605+0000", 2024-09-10T18:35:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103559 2024-09-10T18:35:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,7", 2024-09-10T18:35:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.120710+0000", 2024-09-10T18:35:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000104837 2024-09-10T18:35:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.292464+0000", 2024-09-10T18:35:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.171754034 2024-09-10T18:35:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:35:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.292498+0000", 2024-09-10T18:35:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3986999999999998e-05 2024-09-10T18:35:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:35:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.297968+0000", 2024-09-10T18:35:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054699340000000001 2024-09-10T18:35:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.298023+0000", 2024-09-10T18:35:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5337000000000002e-05 2024-09-10T18:35:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.298059+0000", 2024-09-10T18:35:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6335999999999999e-05 2024-09-10T18:35:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35306.0:218751 9.8 e298/246 9:14a7ae48:::benchmark_data_smithi028_135703_object84160:head v 298'13397, mlcod=298'13394)", 2024-09-10T18:35:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:54.122616+0000", 2024-09-10T18:35:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 48.640265958999997, 2024-09-10T18:35:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25004287200000003, 2024-09-10T18:35:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:35:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.122616+0000", 2024-09-10T18:35:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.122616+0000", 2024-09-10T18:35:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.122614+0000", 2024-09-10T18:35:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:35:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.122623+0000", 2024-09-10T18:35:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8389999999999996e-06 2024-09-10T18:35:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.289997+0000", 2024-09-10T18:35:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.167373734 2024-09-10T18:35:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.290004+0000", 2024-09-10T18:35:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6359999999999994e-06 2024-09-10T18:35:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.290455+0000", 2024-09-10T18:35:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00045102899999999998 2024-09-10T18:35:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.290479+0000", 2024-09-10T18:35:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3087000000000001e-05 2024-09-10T18:35:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:35:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.372623+0000", 2024-09-10T18:35:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.082144537000000004 2024-09-10T18:35:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.372659+0000", 2024-09-10T18:35:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5641e-05 2024-09-10T18:35:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.35306.0:218756 9.5 e298/246 ondisk, result = 0)", 2024-09-10T18:35:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:54.123789+0000", 2024-09-10T18:35:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 48.639093029999998, 2024-09-10T18:35:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16867674299999999, 2024-09-10T18:35:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:35:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:35:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:35:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:35:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.123789+0000", 2024-09-10T18:35:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:35:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.123789+0000", 2024-09-10T18:35:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:35:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:35:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.123787+0000", 2024-09-10T18:35:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:35:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:35:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.123794+0000", 2024-09-10T18:35:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7009999999999999e-06 2024-09-10T18:35:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:35:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.289979+0000", 2024-09-10T18:35:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16618456300000001 2024-09-10T18:35:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:35:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.289992+0000", 2024-09-10T18:35:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2961e-05 2024-09-10T18:35:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:35:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.292452+0000", 2024-09-10T18:35:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024600289999999999 2024-09-10T18:35:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:35:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.292458+0000", 2024-09-10T18:35:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2149999999999998e-06 2024-09-10T18:35:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:35:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:35:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:35:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.292465+0000", 2024-09-10T18:35:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5800000000000003e-06 2024-09-10T18:35:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:35:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:35:42.881 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:35:42.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:42.882 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:42.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:42.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:43.009 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:43.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:43.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:43.110 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:43.137 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:43.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:43.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:43.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:43.262 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:43.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:43.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:43.364 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:43.391 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:43.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:43.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:43.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:43.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:43 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:35:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:43 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:35:43.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:43 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:35:43.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:43 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:35:43.515 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:43.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:43.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:43 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:35:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:43 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:35:43.616 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:43.642 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:43.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:43.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:43.744 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:43.768 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:43.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:43.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:43.895 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:43.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:43.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:43.997 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:44.023 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:44.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:44.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:44.125 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:44.151 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:44.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:44.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:44.253 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:44.278 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:44.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:44.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:44.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:44.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:44 smithi028 ceph-mon[29291]: pgmap v641: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 36 op/s 2024-09-10T18:35:44.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:44 smithi028 ceph-mon[29291]: osdmap e307: 8 total, 8 up, 8 in 2024-09-10T18:35:44.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:44 smithi028 ceph-mon[39220]: pgmap v641: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 36 op/s 2024-09-10T18:35:44.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:44 smithi028 ceph-mon[39220]: osdmap e307: 8 total, 8 up, 8 in 2024-09-10T18:35:44.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:44.529 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:44.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:44.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:44 smithi204 ceph-mon[32357]: pgmap v641: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 36 op/s 2024-09-10T18:35:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:44 smithi204 ceph-mon[32357]: osdmap e307: 8 total, 8 up, 8 in 2024-09-10T18:35:44.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:44.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T18:35:44.655 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:44.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:44.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:44.757 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:44.783 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:44.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:44.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:44.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:44.908 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:44.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:45.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:45.035 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:45.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:45.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:45.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:35:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:35:45.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:45.159 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:45.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:45.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:45.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:45.283 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:45.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:45.385 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:45.413 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:45.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:45.514 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:45.539 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:45.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:45.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:45.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:45.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:45.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:45.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:45.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:45.789 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:45.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:45.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:45.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:45.913 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:45.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:46.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:46.037 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:46.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:46.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:46.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:46.164 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:46.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:46.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:46.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:46.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:46.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:46.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:46.416 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:46.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:46.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:46 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:46 smithi028 ceph-mon[29291]: pgmap v643: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:35:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:46 smithi028 ceph-mon[29291]: osdmap e308: 8 total, 8 up, 8 in 2024-09-10T18:35:46.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:46 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:46.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:46 smithi028 ceph-mon[39220]: pgmap v643: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:35:46.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:46 smithi028 ceph-mon[39220]: osdmap e308: 8 total, 8 up, 8 in 2024-09-10T18:35:46.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:46.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:46.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:46.581 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:35:46.581 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:35:46.582 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-10T18:35:46.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T18:35:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:46 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:46 smithi204 ceph-mon[32357]: pgmap v643: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:35:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:46 smithi204 ceph-mon[32357]: osdmap e308: 8 total, 8 up, 8 in 2024-09-10T18:35:46.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:46.667 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:46.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:46.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:46.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:46.795 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:46.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:46.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:46.897 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:46.921 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:46.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:46.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:47.023 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:47.047 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:47.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:47.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:47.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:47.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:47.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:47.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:47.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:47.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:47.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:47.425 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:47.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:47.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:47.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:47.553 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:47.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:47.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:47.678 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:47.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:47.780 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:47.802 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:47.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:47.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:47.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:47.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:47.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:47.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:48.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:48.055 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:48.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:48.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:48.157 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:48.159 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:48.182 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:48.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:48.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:48.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:48.309 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:48.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:48.387 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:48 smithi028 ceph-mon[39220]: pgmap v645: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:35:48.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:48 smithi028 ceph-mon[29291]: pgmap v645: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:35:48.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:48.437 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:48.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:48.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:48.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:48.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:48 smithi204 ceph-mon[32357]: pgmap v645: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:35:48.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:48.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:48.689 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:48.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:48.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:48.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:48.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:48.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:48.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:48.938 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:48.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:48.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:49.041 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:49.067 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:49.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:49.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:49.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:49.194 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:49.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:49.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:49.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:49.322 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:49.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:49.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:49.342 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:35:49.343 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:35:49.358 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:35:49.358 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:35:49.373 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:35:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:35:49.389 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:35:49.389 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:35:49.404 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:35:49.405 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:35:49.420 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:35:49.421 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:35:49.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:49.437 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:35:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:35:49.450 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:49.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:49.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:49.452 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:35:49.453 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:35:49.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:49.575 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:49.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:49.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:49.677 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:49.702 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:49.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:49.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:49.828 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:49.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:49.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:49.929 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:49.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:49.956 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:49.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:50.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:50.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:50.081 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:50.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:50.182 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:50.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:50.209 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:50.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:50.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:50.335 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:50.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:50.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:50.404 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:50 smithi028 ceph-mon[39220]: pgmap v646: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:35:50.404 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:50 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1966448040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:35:50.404 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:50 smithi028 ceph-mon[29291]: pgmap v646: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:35:50.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:50 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1966448040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:35:50.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:50.461 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:50.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:50.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:50.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:50.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:50.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:50 smithi204 ceph-mon[32357]: pgmap v646: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:35:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:50 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1966448040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:35:50.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:50.709 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:50.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:50.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:50.728 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:35:50.792 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:35:50.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:50.836 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:50.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:50.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:50.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:50.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:50.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:50.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:51.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:51.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:51.085 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:51.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:51.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:51.208 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:51.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:51.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:51.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:51.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:51.336 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:51.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:51.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:51.463 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:51.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:51.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:51.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-10T18:35:51.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:51.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:51.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:51.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:51.645 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:35:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:35:51] "GET /metrics HTTP/1.1" 200 224357 "" "Prometheus/2.43.0" 2024-09-10T18:35:51.689 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:51.714 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:51.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:51.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:51.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:51.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:51.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:51.942 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:51.968 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:51.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:51.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:52.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:51 smithi028 ceph-mon[29291]: pgmap v647: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:35:52.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:51 smithi028 ceph-mon[39220]: pgmap v647: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:35:52.069 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:52.094 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:52.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:52.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:52.103 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:51 smithi204 ceph-mon[32357]: pgmap v647: 178 pgs: 178 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:35:52.196 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:52.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:52.219 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:52.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:52.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:52.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:52.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:52.373 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:35:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:35:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:35:52.444 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:52.466 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:52.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:52.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:52.568 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:52.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:52.590 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:52.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:52.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:52.712 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:52.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:52.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:52.788 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:52.813 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:52.813 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:52.835 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:52.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:52.836 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_2' removed 2024-09-10T18:35:52.836 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:35:52.936 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:52.957 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:52.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:52.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:53.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:52 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2399517849' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:35:53.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:52 smithi028 ceph-mon[39220]: from='client.35543 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:35:53.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:52 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/997929530' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:35:53.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:52 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2399517849' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:35:53.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:52 smithi028 ceph-mon[29291]: from='client.35543 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:35:53.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:52 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/997929530' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:35:53.059 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:53.080 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:53.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:53.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:53.103 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:52 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2399517849' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:35:53.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:52 smithi204 ceph-mon[32357]: from='client.35543 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:35:53.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:52 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/997929530' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:35:53.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:53.205 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:53.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:53.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:53.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:53.329 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:53.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:53.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:53.430 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:53.452 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:53.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:53.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:53.554 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:53.575 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:53.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:53.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:53.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:53.700 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:53.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:53.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:53.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:53.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:53.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:53.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:53.952 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:53.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:53.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:54.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:54.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:54.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:54.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:53 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:35:54.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:53 smithi204 ceph-mon[32357]: from='client.35543 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:35:54.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:53 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/997929530' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:35:54.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:53 smithi204 ceph-mon[32357]: osdmap e309: 8 total, 8 up, 8 in 2024-09-10T18:35:54.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:53 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:35:54.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:53 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:35:54.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:53 smithi204 ceph-mon[32357]: pgmap v649: 162 pgs: 162 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T18:35:54.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:53 smithi204 ceph-mon[32357]: 6.1a scrub starts 2024-09-10T18:35:54.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:53 smithi204 ceph-mon[32357]: 6.a scrub starts 2024-09-10T18:35:54.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:53 smithi204 ceph-mon[32357]: 6.a scrub ok 2024-09-10T18:35:54.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:53 smithi204 ceph-mon[32357]: 1.16 scrub starts 2024-09-10T18:35:54.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:53 smithi204 ceph-mon[32357]: 1.16 scrub ok 2024-09-10T18:35:54.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:53 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:35:54.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:54.201 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:54.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:54.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:35:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[29291]: from='client.35543 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:35:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/997929530' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:35:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[29291]: osdmap e309: 8 total, 8 up, 8 in 2024-09-10T18:35:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:35:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:35:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[29291]: pgmap v649: 162 pgs: 162 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T18:35:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[29291]: 6.1a scrub starts 2024-09-10T18:35:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[29291]: 6.a scrub starts 2024-09-10T18:35:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[29291]: 6.a scrub ok 2024-09-10T18:35:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[29291]: 1.16 scrub starts 2024-09-10T18:35:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[29291]: 1.16 scrub ok 2024-09-10T18:35:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:35:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:35:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[39220]: from='client.35543 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:35:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/997929530' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:35:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[39220]: osdmap e309: 8 total, 8 up, 8 in 2024-09-10T18:35:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:35:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:35:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[39220]: pgmap v649: 162 pgs: 162 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T18:35:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[39220]: 6.1a scrub starts 2024-09-10T18:35:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[39220]: 6.a scrub starts 2024-09-10T18:35:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[39220]: 6.a scrub ok 2024-09-10T18:35:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[39220]: 1.16 scrub starts 2024-09-10T18:35:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[39220]: 1.16 scrub ok 2024-09-10T18:35:54.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:53 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:35:54.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:54.329 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:54.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:54.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:54.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:54.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:54.578 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:54.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:54.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:54.679 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:54.704 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:54.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:54.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:54.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:35:54.806 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:54.830 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:54.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:54.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:54.932 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:54.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:54.956 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:54.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:55.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:55.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:55.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:54 smithi204 ceph-mon[32357]: osdmap e310: 8 total, 8 up, 8 in 2024-09-10T18:35:55.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:35:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:35:55.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:54 smithi028 ceph-mon[29291]: osdmap e310: 8 total, 8 up, 8 in 2024-09-10T18:35:55.117 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:54 smithi028 ceph-mon[39220]: osdmap e310: 8 total, 8 up, 8 in 2024-09-10T18:35:55.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:55.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:55.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:55.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:55.327 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:55.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:55.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:55.429 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:55.455 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:55.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:55.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:55.557 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:55.583 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:55.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:55.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:55.685 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:55.708 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:55.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:55.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:55.810 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:55.834 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:55.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:55.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:55.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:55.958 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:55.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:55.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:56.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:55 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:56.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:55 smithi028 ceph-mon[29291]: 4.a scrub starts 2024-09-10T18:35:56.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:55 smithi028 ceph-mon[29291]: 4.a scrub ok 2024-09-10T18:35:56.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:55 smithi028 ceph-mon[29291]: pgmap v651: 162 pgs: 162 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:35:56.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:55 smithi028 ceph-mon[29291]: 2.2 scrub starts 2024-09-10T18:35:56.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:55 smithi028 ceph-mon[29291]: 2.2 scrub ok 2024-09-10T18:35:56.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:55 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:56.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:55 smithi028 ceph-mon[39220]: 4.a scrub starts 2024-09-10T18:35:56.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:55 smithi028 ceph-mon[39220]: 4.a scrub ok 2024-09-10T18:35:56.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:55 smithi028 ceph-mon[39220]: pgmap v651: 162 pgs: 162 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:35:56.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:55 smithi028 ceph-mon[39220]: 2.2 scrub starts 2024-09-10T18:35:56.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:55 smithi028 ceph-mon[39220]: 2.2 scrub ok 2024-09-10T18:35:56.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:56.083 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:56.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:56.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:56.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:55 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:35:56.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:55 smithi204 ceph-mon[32357]: 4.a scrub starts 2024-09-10T18:35:56.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:55 smithi204 ceph-mon[32357]: 4.a scrub ok 2024-09-10T18:35:56.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:55 smithi204 ceph-mon[32357]: pgmap v651: 162 pgs: 162 active+clean; 500 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:35:56.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:55 smithi204 ceph-mon[32357]: 2.2 scrub starts 2024-09-10T18:35:56.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:55 smithi204 ceph-mon[32357]: 2.2 scrub ok 2024-09-10T18:35:56.121 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:35:56.122 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:35:56.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:56.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:56.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:56.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:56.336 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:56.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:56.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:56.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:56.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:56.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:56.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:56.588 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:56.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:56.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:56.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:56.712 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:56.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:56.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:56.814 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:56.840 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:56.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:56.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:56.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-10T18:35:56.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:56.963 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:56.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:56.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:56.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:35:57.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:57.088 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:57.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:57.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:57.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:56 smithi204 ceph-mon[32357]: 2.1 scrub starts 2024-09-10T18:35:57.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:56 smithi204 ceph-mon[32357]: 2.1 scrub ok 2024-09-10T18:35:57.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:56 smithi204 ceph-mon[32357]: 6.0 scrub starts 2024-09-10T18:35:57.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:56 smithi204 ceph-mon[32357]: 6.0 scrub ok 2024-09-10T18:35:57.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:56 smithi204 ceph-mon[32357]: 4.5 scrub starts 2024-09-10T18:35:57.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:56 smithi204 ceph-mon[32357]: 4.5 scrub ok 2024-09-10T18:35:57.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:56 smithi204 ceph-mon[32357]: 1.5 scrub starts 2024-09-10T18:35:57.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:56 smithi204 ceph-mon[32357]: 1.5 scrub ok 2024-09-10T18:35:57.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:56 smithi204 ceph-mon[32357]: 6.1d scrub starts 2024-09-10T18:35:57.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:56 smithi204 ceph-mon[32357]: 6.1d scrub ok 2024-09-10T18:35:57.190 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:57.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:57.217 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:57.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:57.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[29291]: 2.1 scrub starts 2024-09-10T18:35:57.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[29291]: 2.1 scrub ok 2024-09-10T18:35:57.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[29291]: 6.0 scrub starts 2024-09-10T18:35:57.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[29291]: 6.0 scrub ok 2024-09-10T18:35:57.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[29291]: 4.5 scrub starts 2024-09-10T18:35:57.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[29291]: 4.5 scrub ok 2024-09-10T18:35:57.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[29291]: 1.5 scrub starts 2024-09-10T18:35:57.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[29291]: 1.5 scrub ok 2024-09-10T18:35:57.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[29291]: 6.1d scrub starts 2024-09-10T18:35:57.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[29291]: 6.1d scrub ok 2024-09-10T18:35:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[39220]: 2.1 scrub starts 2024-09-10T18:35:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[39220]: 2.1 scrub ok 2024-09-10T18:35:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[39220]: 6.0 scrub starts 2024-09-10T18:35:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[39220]: 6.0 scrub ok 2024-09-10T18:35:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[39220]: 4.5 scrub starts 2024-09-10T18:35:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[39220]: 4.5 scrub ok 2024-09-10T18:35:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[39220]: 1.5 scrub starts 2024-09-10T18:35:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[39220]: 1.5 scrub ok 2024-09-10T18:35:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[39220]: 6.1d scrub starts 2024-09-10T18:35:57.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:56 smithi028 ceph-mon[39220]: 6.1d scrub ok 2024-09-10T18:35:57.310 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-10T18:35:57.311 INFO:tasks.radosbench:Beginning radosbench... 2024-09-10T18:35:57.311 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-10T18:35:57.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-09-10T18:35:57.318 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:57.344 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:57.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:57.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:57.381 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:57.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:57.467 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:57.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:57.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:57.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:57.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:57.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:57.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:57.719 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:57.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:57.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:57.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:57.847 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:57.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:57.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:57.854 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:57.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:57.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:57.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:58.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:58.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:58.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:58.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:58.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:57 smithi204 ceph-mon[32357]: pgmap v652: 162 pgs: 162 active+clean; 487 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T18:35:58.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:57 smithi204 ceph-mon[32357]: 3.0 scrub starts 2024-09-10T18:35:58.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:57 smithi204 ceph-mon[32357]: 3.0 scrub ok 2024-09-10T18:35:58.105 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:57 smithi028 ceph-mon[39220]: pgmap v652: 162 pgs: 162 active+clean; 487 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T18:35:58.105 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:57 smithi028 ceph-mon[39220]: 3.0 scrub starts 2024-09-10T18:35:58.105 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:57 smithi028 ceph-mon[39220]: 3.0 scrub ok 2024-09-10T18:35:58.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:57 smithi028 ceph-mon[29291]: pgmap v652: 162 pgs: 162 active+clean; 487 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T18:35:58.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:57 smithi028 ceph-mon[29291]: 3.0 scrub starts 2024-09-10T18:35:58.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:57 smithi028 ceph-mon[29291]: 3.0 scrub ok 2024-09-10T18:35:58.200 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:35:58.223 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:58.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:58.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:35:58.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:58.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:58.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:58.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:58.449 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:58.475 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:58.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:58.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:58.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:35:58.602 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:58.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:58.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:35:58.704 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:58.730 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:58.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:58.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:58.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:58.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:58.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:58.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:58.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:58 smithi028 ceph-mon[29291]: 1.e scrub starts 2024-09-10T18:35:58.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:58 smithi028 ceph-mon[29291]: 1.e scrub ok 2024-09-10T18:35:58.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:58 smithi028 ceph-mon[39220]: 1.e scrub starts 2024-09-10T18:35:58.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:58 smithi028 ceph-mon[39220]: 1.e scrub ok 2024-09-10T18:35:58.965 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:35:58.991 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:58.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:35:59.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:59.103 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:58 smithi204 ceph-mon[32357]: 1.e scrub starts 2024-09-10T18:35:59.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:58 smithi204 ceph-mon[32357]: 1.e scrub ok 2024-09-10T18:35:59.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:59.119 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:59.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:59.190 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:35:59.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:35:59.245 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:59.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:59.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:35:59.347 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:35:59.372 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:35:59.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:59.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:35:59.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:35:59.500 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:59.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:59.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:35:59.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:59.625 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:59.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:59.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:35:59.751 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:59.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:35:59.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:35:59.877 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:35:59.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:35:59.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:35:59.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:00.002 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:00.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:00.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:00.103 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:00.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[29291]: 6.10 scrub starts 2024-09-10T18:36:00.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[29291]: 6.10 scrub ok 2024-09-10T18:36:00.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[29291]: 3.f scrub starts 2024-09-10T18:36:00.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[29291]: 3.f scrub ok 2024-09-10T18:36:00.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[29291]: pgmap v653: 162 pgs: 162 active+clean; 487 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:00.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:36:00.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:36:00.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[29291]: 1.a scrub starts 2024-09-10T18:36:00.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[29291]: 1.a scrub ok 2024-09-10T18:36:00.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[39220]: 6.10 scrub starts 2024-09-10T18:36:00.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[39220]: 6.10 scrub ok 2024-09-10T18:36:00.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[39220]: 3.f scrub starts 2024-09-10T18:36:00.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[39220]: 3.f scrub ok 2024-09-10T18:36:00.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[39220]: pgmap v653: 162 pgs: 162 active+clean; 487 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:00.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:36:00.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:36:00.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[39220]: 1.a scrub starts 2024-09-10T18:36:00.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:35:59 smithi028 ceph-mon[39220]: 1.a scrub ok 2024-09-10T18:36:00.130 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:00.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:00.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:00.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:00.254 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:00.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:00.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:59 smithi204 ceph-mon[32357]: 6.10 scrub starts 2024-09-10T18:36:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:59 smithi204 ceph-mon[32357]: 6.10 scrub ok 2024-09-10T18:36:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:59 smithi204 ceph-mon[32357]: 3.f scrub starts 2024-09-10T18:36:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:59 smithi204 ceph-mon[32357]: 3.f scrub ok 2024-09-10T18:36:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:59 smithi204 ceph-mon[32357]: pgmap v653: 162 pgs: 162 active+clean; 487 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:59 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:36:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:59 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:36:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:59 smithi204 ceph-mon[32357]: 1.a scrub starts 2024-09-10T18:36:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:35:59 smithi204 ceph-mon[32357]: 1.a scrub ok 2024-09-10T18:36:00.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:00.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:00.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:00.480 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:00.506 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:00.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:00.575 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:36:00.575 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:36:00.576 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 311, 2024-09-10T18:36:00.576 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "5eababb2-6fa0-11ef-bcea-c7b262605968", 2024-09-10T18:36:00.576 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-10T18:16:20.680373+0000", 2024-09-10T18:36:00.576 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-10T18:35:59.850021+0000", 2024-09-10T18:36:00.576 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-09-10T18:26:26.611338+0000", 2024-09-10T18:36:00.576 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-09-10T18:21:02.114621+0000", 2024-09-10T18:36:00.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T18:36:00.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5804032, 2024-09-10T18:36:00.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-09-10T18:36:00.576 INFO:teuthology.orchestra.run.smithi028.stdout: "nodeep-scrub", 2024-09-10T18:36:00.576 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-09-10T18:36:00.576 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-09-10T18:36:00.576 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-09-10T18:36:00.577 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-09-10T18:36:00.577 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:36:00.577 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 19, 2024-09-10T18:36:00.577 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T18:36:00.577 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T18:36:00.577 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T18:36:00.577 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-09-10T18:36:00.577 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 9, 2024-09-10T18:36:00.577 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-09-10T18:36:00.577 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-09-10T18:36:00.577 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-09-10T18:36:00.577 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "quincy", 2024-09-10T18:36:00.577 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-09-10T18:36:00.578 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.578 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-09-10T18:36:00.578 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-09-10T18:36:00.578 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:19:24.013536+0000", 2024-09-10T18:36:00.578 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:36:00.578 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:36:00.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:36:00.578 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:36:00.578 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:36:00.578 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:36:00.578 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:36:00.578 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:36:00.578 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:36:00.579 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:36:00.579 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:36:00.579 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:36:00.579 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 31, 2024-09-10T18:36:00.579 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 31, 2024-09-10T18:36:00.579 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 31, 2024-09-10T18:36:00.579 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 31, 2024-09-10T18:36:00.579 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 31, 2024-09-10T18:36:00.579 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:36:00.579 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:36:00.579 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:36:00.579 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:36:00.579 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:36:00.579 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:36:00.579 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:36:00.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.580 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "300", 2024-09-10T18:36:00.580 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:36:00.580 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:36:00.580 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "287", 2024-09-10T18:36:00.580 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:36:00.580 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:36:00.580 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:36:00.580 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:36:00.580 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:36:00.580 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:36:00.580 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:36:00.580 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:36:00.581 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:36:00.581 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:36:00.581 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:36:00.581 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:36:00.581 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:36:00.581 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:36:00.581 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:36:00.581 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:36:00.581 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:36:00.581 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:36:00.581 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:36:00.581 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:36:00.581 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:36:00.582 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:36:00.582 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:36:00.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.582 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:36:00.582 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:36:00.582 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:36:00.582 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:36:00.582 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:36:00.582 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:36:00.582 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:36:00.582 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:36:00.582 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:36:00.582 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:36:00.582 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:36:00.582 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-10T18:36:00.583 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-09-10T18:36:00.583 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-09-10T18:36:00.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.583 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:36:00.583 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-09-10T18:36:00.583 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.583 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-09-10T18:36:00.583 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-09-10T18:36:00.583 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:30.065209+0000", 2024-09-10T18:36:00.583 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-09-10T18:36:00.584 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T18:36:00.584 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:36:00.584 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:36:00.584 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:36:00.584 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:36:00.584 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:36:00.584 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:36:00.584 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:36:00.584 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:36:00.584 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:36:00.584 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:36:00.584 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-09-10T18:36:00.584 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-09-10T18:36:00.584 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-09-10T18:36:00.585 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-09-10T18:36:00.585 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-09-10T18:36:00.585 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:36:00.585 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:36:00.585 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:36:00.585 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:36:00.585 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:36:00.585 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:36:00.585 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:36:00.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.585 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "62", 2024-09-10T18:36:00.585 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:36:00.585 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:36:00.585 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:36:00.586 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:36:00.586 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:36:00.586 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 3, 2024-09-10T18:36:00.586 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 62, 2024-09-10T18:36:00.586 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:36:00.586 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:36:00.586 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:36:00.586 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:36:00.586 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:36:00.586 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:36:00.586 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:36:00.586 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:36:00.586 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:36:00.586 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:36:00.586 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:36:00.586 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:36:00.587 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:36:00.587 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:36:00.587 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:36:00.587 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:36:00.587 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:36:00.587 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:36:00.587 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:36:00.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.587 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:36:00.587 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:36:00.587 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:36:00.587 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:36:00.587 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:36:00.587 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:36:00.587 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:36:00.588 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:36:00.588 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:36:00.588 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:36:00.588 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:36:00.588 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:36:00.588 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:36:00.588 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-09-10T18:36:00.588 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.588 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-09-10T18:36:00.588 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-09-10T18:36:00.588 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:30.506441+0000", 2024-09-10T18:36:00.588 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:36:00.588 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:36:00.589 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:36:00.589 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:36:00.589 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:36:00.589 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:36:00.589 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:36:00.589 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:36:00.589 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:36:00.589 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:36:00.589 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:36:00.589 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:36:00.589 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:36:00.589 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:36:00.589 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:36:00.590 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:36:00.590 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:36:00.590 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:36:00.590 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:36:00.590 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:36:00.590 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:36:00.590 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:36:00.590 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:36:00.590 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:36:00.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.590 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "59", 2024-09-10T18:36:00.590 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:36:00.590 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:36:00.590 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:36:00.590 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:36:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:36:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:36:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:36:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:36:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:36:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:36:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:36:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:36:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:36:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:36:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:36:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:36:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:36:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:36:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:36:00.591 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:36:00.592 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:36:00.592 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:36:00.592 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:36:00.592 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:36:00.592 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:36:00.592 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:36:00.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.592 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:36:00.593 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:36:00.593 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:36:00.593 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:36:00.593 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:36:00.593 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:36:00.593 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:36:00.593 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:36:00.593 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:36:00.593 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:36:00.593 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:36:00.594 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:36:00.594 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:36:00.594 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:36:00.594 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.594 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-09-10T18:36:00.594 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-09-10T18:36:00.594 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:32.008130+0000", 2024-09-10T18:36:00.594 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:36:00.594 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:36:00.594 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:36:00.594 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:36:00.595 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:36:00.595 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:36:00.595 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:36:00.595 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:36:00.595 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:36:00.595 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:36:00.595 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:36:00.595 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:36:00.595 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:36:00.595 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:36:00.595 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:36:00.595 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:36:00.595 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:36:00.595 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:36:00.595 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:36:00.596 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:36:00.596 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:36:00.596 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:36:00.596 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:36:00.596 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:36:00.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.596 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "61", 2024-09-10T18:36:00.596 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:36:00.596 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:36:00.596 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:36:00.596 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:36:00.596 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:36:00.596 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:36:00.596 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:36:00.597 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:36:00.597 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:36:00.597 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:36:00.597 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:36:00.597 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:36:00.597 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:36:00.597 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:36:00.597 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:36:00.597 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:36:00.597 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:36:00.597 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:36:00.597 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:36:00.597 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:36:00.597 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:36:00.598 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:36:00.598 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:36:00.598 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:36:00.598 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:36:00.598 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:36:00.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.598 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:36:00.598 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:36:00.598 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:36:00.598 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:36:00.598 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:36:00.598 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:36:00.598 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:36:00.598 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:36:00.598 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:36:00.599 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:36:00.599 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:36:00.599 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:36:00.599 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:36:00.599 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:36:00.599 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.599 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-09-10T18:36:00.599 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-09-10T18:36:00.599 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:34.060571+0000", 2024-09-10T18:36:00.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:36:00.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:36:00.600 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:36:00.600 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:36:00.600 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:36:00.600 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:36:00.600 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:36:00.600 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:36:00.600 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:36:00.600 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:36:00.600 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:36:00.600 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:36:00.600 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:36:00.600 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:36:00.600 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:36:00.601 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:36:00.601 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:36:00.601 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:36:00.601 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:36:00.601 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:36:00.601 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:36:00.601 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:36:00.601 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:36:00.601 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:36:00.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.601 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "63", 2024-09-10T18:36:00.601 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:36:00.601 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:36:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:36:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:36:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:36:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:36:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:36:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:36:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:36:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:36:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:36:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:36:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:36:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:36:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:36:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:36:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:36:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:36:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:36:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:36:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:36:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:36:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:36:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:36:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:36:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:36:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:36:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:36:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:36:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:36:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:36:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:36:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:36:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:36:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:36:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:36:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:36:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:36:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:36:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:36:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-09-10T18:36:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-09-10T18:36:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:36.127330+0000", 2024-09-10T18:36:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:36:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:36:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:36:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:36:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:36:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:36:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:36:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:36:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:36:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:36:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:36:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:36:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:36:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:36:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:36:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:36:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:36:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:36:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:36:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:36:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:36:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:36:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:36:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:36:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "65", 2024-09-10T18:36:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:36:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:36:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:36:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:36:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:36:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:36:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:36:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:36:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:36:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:36:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:36:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:36:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:36:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:36:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:36:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:36:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:36:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:36:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:36:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:36:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:36:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:36:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:36:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:36:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:36:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:36:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:36:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:36:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:36:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:36:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:36:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:36:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:36:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:36:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:36:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:36:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:36:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-10T18:36:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-09-10T18:36:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:36:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:36:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:36:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:36:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-10T18:36:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "f1c834d9-252a-4077-b25e-169181d70251", 2024-09-10T18:36:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:36:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:36:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:36:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:36:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:36:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:36:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-09-10T18:36:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 280, 2024-09-10T18:36:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:36:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:36:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:36:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-09-10T18:36:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:36:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:36:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-09-10T18:36:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:36:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:36:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-09-10T18:36:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:36:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:36:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-09-10T18:36:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:36:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/3188372351", 2024-09-10T18:36:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/3188372351", 2024-09-10T18:36:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/3188372351", 2024-09-10T18:36:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/3188372351", 2024-09-10T18:36:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:36:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:36:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:36:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-10T18:36:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "81f1855d-b946-4ca3-b246-38977b9e20fd", 2024-09-10T18:36:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:36:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:36:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:36:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:36:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:36:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:36:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-10T18:36:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 289, 2024-09-10T18:36:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:36:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:36:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:36:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-09-10T18:36:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:36:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:36:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-09-10T18:36:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:36:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:36:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-09-10T18:36:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:36:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:36:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-09-10T18:36:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:36:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1369912678", 2024-09-10T18:36:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1369912678", 2024-09-10T18:36:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1369912678", 2024-09-10T18:36:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1369912678", 2024-09-10T18:36:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:36:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:36:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:36:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-10T18:36:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "2ad55d2a-36fa-4756-bf22-58d4b049486e", 2024-09-10T18:36:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:36:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:36:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:36:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:36:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:36:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:36:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-09-10T18:36:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 280, 2024-09-10T18:36:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:36:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:36:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:36:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-09-10T18:36:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:36:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:36:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-09-10T18:36:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:36:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:36:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-09-10T18:36:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:36:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:36:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-09-10T18:36:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:36:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/1243406003", 2024-09-10T18:36:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/1243406003", 2024-09-10T18:36:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/1243406003", 2024-09-10T18:36:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1243406003", 2024-09-10T18:36:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:36:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:36:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:36:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-10T18:36:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "242729a5-a5e8-44d5-97c8-14d14e970bff", 2024-09-10T18:36:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:36:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:36:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:36:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:36:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:36:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:36:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-09-10T18:36:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 289, 2024-09-10T18:36:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:36:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:36:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:36:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-09-10T18:36:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:36:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:36:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-09-10T18:36:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:36:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:36:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-09-10T18:36:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:36:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:36:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-09-10T18:36:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:36:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/67703283", 2024-09-10T18:36:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/67703283", 2024-09-10T18:36:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/67703283", 2024-09-10T18:36:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/67703283", 2024-09-10T18:36:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:36:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:36:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:36:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-10T18:36:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "6e38b8ef-e6ed-493f-8eca-755354754787", 2024-09-10T18:36:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:36:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:36:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:36:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:36:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:36:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:36:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 33, 2024-09-10T18:36:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 289, 2024-09-10T18:36:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:36:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:36:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:36:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6800", 2024-09-10T18:36:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:36:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:36:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.629 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6801", 2024-09-10T18:36:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:36:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:36:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6803", 2024-09-10T18:36:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:36:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:36:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6802", 2024-09-10T18:36:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:36:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6800/1619669076", 2024-09-10T18:36:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6801/1619669076", 2024-09-10T18:36:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6803/1619669076", 2024-09-10T18:36:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6802/1619669076", 2024-09-10T18:36:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:36:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:36:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:36:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-10T18:36:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "a92e0441-66c9-498e-a0ac-6465acdbb1f5", 2024-09-10T18:36:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:36:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:36:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:36:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:36:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:36:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:36:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 39, 2024-09-10T18:36:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 289, 2024-09-10T18:36:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:36:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:36:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:36:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6804", 2024-09-10T18:36:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:36:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:36:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6805", 2024-09-10T18:36:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:36:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:36:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6807", 2024-09-10T18:36:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:36:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:36:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6806", 2024-09-10T18:36:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:36:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6804/2717802880", 2024-09-10T18:36:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6805/2717802880", 2024-09-10T18:36:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6807/2717802880", 2024-09-10T18:36:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6806/2717802880", 2024-09-10T18:36:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:36:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:36:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:36:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-09-10T18:36:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "12b02875-6653-45ed-8d2b-b4d33bd88525", 2024-09-10T18:36:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:36:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:36:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:36:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:36:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 46, 2024-09-10T18:36:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 94, 2024-09-10T18:36:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 144, 2024-09-10T18:36:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 287, 2024-09-10T18:36:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 95, 2024-09-10T18:36:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:36:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:36:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6808", 2024-09-10T18:36:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:36:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:36:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6809", 2024-09-10T18:36:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:36:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:36:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6811", 2024-09-10T18:36:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:36:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:36:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6810", 2024-09-10T18:36:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:36:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6808/3479311452", 2024-09-10T18:36:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6809/3479311452", 2024-09-10T18:36:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6811/3479311452", 2024-09-10T18:36:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6810/3479311452", 2024-09-10T18:36:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:36:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:36:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:36:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-09-10T18:36:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "0c4f7aa9-ff95-4192-a146-ca576eee907f", 2024-09-10T18:36:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:36:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:36:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:36:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:36:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:36:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:36:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 54, 2024-09-10T18:36:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 289, 2024-09-10T18:36:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:36:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:36:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:36:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6812", 2024-09-10T18:36:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:36:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:36:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6813", 2024-09-10T18:36:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:36:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:36:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6815", 2024-09-10T18:36:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:36:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:36:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6814", 2024-09-10T18:36:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:36:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6812/3511791879", 2024-09-10T18:36:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6813/3511791879", 2024-09-10T18:36:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6815/3511791879", 2024-09-10T18:36:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6814/3511791879", 2024-09-10T18:36:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:36:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:36:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:36:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:36:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-09-10T18:36:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-10T18:36:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:36:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:36:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:36:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:36:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:36:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:35:59.338144+0000", 2024-09-10T18:36:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:36:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-10T18:36:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:36:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:36:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:36:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:36:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:36:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:34:47.485153+0000", 2024-09-10T18:36:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:36:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-10T18:36:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:36:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:36:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:36:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:36:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:36:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:35:09.495786+0000", 2024-09-10T18:36:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:36:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-10T18:36:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:36:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:36:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:36:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:36:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:36:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:35:52.899686+0000", 2024-09-10T18:36:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:36:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-10T18:36:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:36:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:36:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:36:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:36:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:36:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:35:13.915850+0000", 2024-09-10T18:36:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:36:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-10T18:36:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:36:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:36:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:36:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:36:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:36:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:35:42.401737+0000", 2024-09-10T18:36:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:36:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-09-10T18:36:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:36:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:36:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:36:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:36:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:36:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:35:34.158547+0000", 2024-09-10T18:36:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 94 2024-09-10T18:36:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-09-10T18:36:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:36:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:36:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:36:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:36:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:36:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:34:42.739810+0000", 2024-09-10T18:36:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:36:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:36:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-09-10T18:36:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.c", 2024-09-10T18:36:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:36:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-09-10T18:36:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: 4, 2024-09-10T18:36:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-10T18:36:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "6.17", 2024-09-10T18:36:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:36:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: 3, 2024-09-10T18:36:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-09-10T18:36:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: 5 2024-09-10T18:36:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:36:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-09-10T18:36:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.4", 2024-09-10T18:36:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:36:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:36:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-09-10T18:36:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1b", 2024-09-10T18:36:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:36:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:36:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-09-10T18:36:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1f", 2024-09-10T18:36:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:36:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:36:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-09-10T18:36:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:36:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-09-10T18:36:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-09-10T18:36:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-09-10T18:36:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1745574083": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:36:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3866220552": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:36:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3760775432": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:36:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1746797014": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:36:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1029045895": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:36:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3310949126": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:36:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2874160765": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:36:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2620179974": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:36:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/421194352": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:36:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/1077377902": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:36:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3428757300": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:36:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1759116360": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:36:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2564333506": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:36:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/717120167": "2024-09-11T18:22:08.678985+0000" 2024-09-10T18:36:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-09-10T18:36:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-09-10T18:36:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-09-10T18:36:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-09-10T18:36:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-09-10T18:36:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-09-10T18:36:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-09-10T18:36:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-09-10T18:36:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-09-10T18:36:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-09-10T18:36:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-09-10T18:36:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-09-10T18:36:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-09-10T18:36:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-09-10T18:36:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-09-10T18:36:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-09-10T18:36:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-09-10T18:36:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-09-10T18:36:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-09-10T18:36:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:00.657 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:36:00.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:00.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:00.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:00.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:00.782 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:00.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:00.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:00.884 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:00.909 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:00.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:00.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:01.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:01.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:01.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:01.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:01.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:01.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:01.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[29291]: 5.d scrub starts 2024-09-10T18:36:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[29291]: 5.d scrub ok 2024-09-10T18:36:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[29291]: 5.1e scrub starts 2024-09-10T18:36:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[29291]: 5.1e scrub ok 2024-09-10T18:36:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[29291]: osdmap e311: 8 total, 8 up, 8 in 2024-09-10T18:36:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[29291]: 3.10 scrub starts 2024-09-10T18:36:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[29291]: 3.10 scrub ok 2024-09-10T18:36:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[29291]: 1.1a scrub starts 2024-09-10T18:36:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[29291]: 1.1a scrub ok 2024-09-10T18:36:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[29291]: 5.b scrub starts 2024-09-10T18:36:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[29291]: 5.b scrub ok 2024-09-10T18:36:01.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3053783966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:36:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[39220]: 5.d scrub starts 2024-09-10T18:36:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[39220]: 5.d scrub ok 2024-09-10T18:36:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[39220]: 5.1e scrub starts 2024-09-10T18:36:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[39220]: 5.1e scrub ok 2024-09-10T18:36:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[39220]: osdmap e311: 8 total, 8 up, 8 in 2024-09-10T18:36:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[39220]: 3.10 scrub starts 2024-09-10T18:36:01.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[39220]: 3.10 scrub ok 2024-09-10T18:36:01.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[39220]: 1.1a scrub starts 2024-09-10T18:36:01.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[39220]: 1.1a scrub ok 2024-09-10T18:36:01.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[39220]: 5.b scrub starts 2024-09-10T18:36:01.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[39220]: 5.b scrub ok 2024-09-10T18:36:01.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:00 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3053783966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:36:01.259 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:01.285 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:01.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:01.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:00 smithi204 ceph-mon[32357]: 5.d scrub starts 2024-09-10T18:36:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:00 smithi204 ceph-mon[32357]: 5.d scrub ok 2024-09-10T18:36:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:00 smithi204 ceph-mon[32357]: 5.1e scrub starts 2024-09-10T18:36:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:00 smithi204 ceph-mon[32357]: 5.1e scrub ok 2024-09-10T18:36:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:00 smithi204 ceph-mon[32357]: osdmap e311: 8 total, 8 up, 8 in 2024-09-10T18:36:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:00 smithi204 ceph-mon[32357]: 3.10 scrub starts 2024-09-10T18:36:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:00 smithi204 ceph-mon[32357]: 3.10 scrub ok 2024-09-10T18:36:01.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:00 smithi204 ceph-mon[32357]: 1.1a scrub starts 2024-09-10T18:36:01.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:00 smithi204 ceph-mon[32357]: 1.1a scrub ok 2024-09-10T18:36:01.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:00 smithi204 ceph-mon[32357]: 5.b scrub starts 2024-09-10T18:36:01.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:00 smithi204 ceph-mon[32357]: 5.b scrub ok 2024-09-10T18:36:01.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:00 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3053783966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:36:01.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:01.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:01.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:01.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:01.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:01.545 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:36:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:36:01] "GET /metrics HTTP/1.1" 200 222032 "" "Prometheus/2.43.0" 2024-09-10T18:36:01.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:01.660 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:01.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:01.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:01.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:01.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:01.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:01.908 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:01.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:02.010 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:02.035 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:02.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:02.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:02.061 INFO:tasks.thrashosds.thrasher:j is {'epoch': 311, 'fsid': '5eababb2-6fa0-11ef-bcea-c7b262605968', 'created': '2024-09-10T18:16:20.680373+0000', 'modified': '2024-09-10T18:35:59.850021+0000', 'last_up_change': '2024-09-10T18:26:26.611338+0000', 'last_in_change': '2024-09-10T18:21:02.114621+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T18:19:24.013536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '300', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '287', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-10T18:21:30.065209+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 62, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T18:21:30.506441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T18:21:32.008130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T18:21:34.060571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T18:21:36.127330+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}], 'osds': [{'osd': 0, 'uuid': 'f1c834d9-252a-4077-b25e-169181d70251', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 280, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 3188372351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 3188372351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 3188372351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 3188372351}]}, 'public_addr': '172.21.15.28:6801/3188372351', 'cluster_addr': '172.21.15.28:6802/3188372351', 'heartbeat_back_addr': '172.21.15.28:6804/3188372351', 'heartbeat_front_addr': '172.21.15.28:6803/3188372351', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '81f1855d-b946-4ca3-b246-38977b9e20fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 289, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1369912678}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1369912678}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1369912678}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1369912678}]}, 'public_addr': '172.21.15.28:6805/1369912678', 'cluster_addr': '172.21.15.28:6806/1369912678', 'heartbeat_back_addr': '172.21.15.28:6808/1369912678', 'heartbeat_front_addr': '172.21.15.28:6807/1369912678', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2ad55d2a-36fa-4756-bf22-58d4b049486e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 280, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 1243406003}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 1243406003}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 1243406003}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1243406003}]}, 'public_addr': '172.21.15.28:6809/1243406003', 'cluster_addr': '172.21.15.28:6810/1243406003', 'heartbeat_back_addr': '172.21.15.28:6812/1243406003', 'heartbeat_front_addr': '172.21.15.28:6811/1243406003', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '242729a5-a5e8-44d5-97c8-14d14e970bff', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 289, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 67703283}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 67703283}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 67703283}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 67703283}]}, 'public_addr': '172.21.15.28:6813/67703283', 'cluster_addr': '172.21.15.28:6814/67703283', 'heartbeat_back_addr': '172.21.15.28:6816/67703283', 'heartbeat_front_addr': '172.21.15.28:6815/67703283', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6e38b8ef-e6ed-493f-8eca-755354754787', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 289, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6800', 'nonce': 1619669076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 1619669076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 1619669076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6802', 'nonce': 1619669076}]}, 'public_addr': '172.21.15.204:6800/1619669076', 'cluster_addr': '172.21.15.204:6801/1619669076', 'heartbeat_back_addr': '172.21.15.204:6803/1619669076', 'heartbeat_front_addr': '172.21.15.204:6802/1619669076', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a92e0441-66c9-498e-a0ac-6465acdbb1f5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 289, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6804', 'nonce': 2717802880}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 2717802880}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 2717802880}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6806', 'nonce': 2717802880}]}, 'public_addr': '172.21.15.204:6804/2717802880', 'cluster_addr': '172.21.15.204:6805/2717802880', 'heartbeat_back_addr': '172.21.15.204:6807/2717802880', 'heartbeat_front_addr': '172.21.15.204:6806/2717802880', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '12b02875-6653-45ed-8d2b-b4d33bd88525', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 46, 'last_clean_end': 94, 'up_from': 144, 'up_thru': 287, 'down_at': 95, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6808', 'nonce': 3479311452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 3479311452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 3479311452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6810', 'nonce': 3479311452}]}, 'public_addr': '172.21.15.204:6808/3479311452', 'cluster_addr': '172.21.15.204:6809/3479311452', 'heartbeat_back_addr': '172.21.15.204:6811/3479311452', 'heartbeat_front_addr': '172.21.15.204:6810/3479311452', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0c4f7aa9-ff95-4192-a146-ca576eee907f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 289, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6812', 'nonce': 3511791879}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 3511791879}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 3511791879}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6814', 'nonce': 3511791879}]}, 'public_addr': '172.21.15.204:6812/3511791879', 'cluster_addr': '172.21.15.204:6813/3511791879', 'heartbeat_back_addr': '172.21.15.204:6815/3511791879', 'heartbeat_front_addr': '172.21.15.204:6814/3511791879', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:35:59.338144+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:34:47.485153+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:35:09.495786+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:35:52.899686+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:35:13.915850+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:35:42.401737+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:35:34.158547+0000', 'dead_epoch': 94}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:34:42.739810+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.c', 'osds': [7, 4, 2]}, {'pgid': '6.17', 'osds': [3, 7, 5]}], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/1745574083': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/3866220552': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:6800/3760775432': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:0/1746797014': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:0/1029045895': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:6800/3310949126': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/2874160765': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/2620179974': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/421194352': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:6800/1077377902': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/3428757300': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/1759116360': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/2564333506': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/717120167': '2024-09-11T18:22:08.678985+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T18:36:02.062 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.4 2024-09-10T18:36:02.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd rm-pg-upmap-items 4.4 2024-09-10T18:36:02.120 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:01 smithi204 ceph-mon[32357]: 4.17 scrub starts 2024-09-10T18:36:02.120 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:01 smithi204 ceph-mon[32357]: 4.17 scrub ok 2024-09-10T18:36:02.120 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:01 smithi204 ceph-mon[32357]: 3.a scrub starts 2024-09-10T18:36:02.120 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:01 smithi204 ceph-mon[32357]: 3.a scrub ok 2024-09-10T18:36:02.120 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:01 smithi204 ceph-mon[32357]: pgmap v655: 162 pgs: 162 active+clean; 474 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:02.120 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:01 smithi204 ceph-mon[32357]: 1.8 scrub starts 2024-09-10T18:36:02.120 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:01 smithi204 ceph-mon[32357]: 1.8 scrub ok 2024-09-10T18:36:02.137 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:02.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:02.164 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:02.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:02.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:01 smithi028 ceph-mon[29291]: 4.17 scrub starts 2024-09-10T18:36:02.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:01 smithi028 ceph-mon[29291]: 4.17 scrub ok 2024-09-10T18:36:02.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:01 smithi028 ceph-mon[29291]: 3.a scrub starts 2024-09-10T18:36:02.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:01 smithi028 ceph-mon[29291]: 3.a scrub ok 2024-09-10T18:36:02.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:01 smithi028 ceph-mon[29291]: pgmap v655: 162 pgs: 162 active+clean; 474 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:02.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:01 smithi028 ceph-mon[29291]: 1.8 scrub starts 2024-09-10T18:36:02.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:01 smithi028 ceph-mon[29291]: 1.8 scrub ok 2024-09-10T18:36:02.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:01 smithi028 ceph-mon[39220]: 4.17 scrub starts 2024-09-10T18:36:02.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:01 smithi028 ceph-mon[39220]: 4.17 scrub ok 2024-09-10T18:36:02.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:01 smithi028 ceph-mon[39220]: 3.a scrub starts 2024-09-10T18:36:02.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:01 smithi028 ceph-mon[39220]: 3.a scrub ok 2024-09-10T18:36:02.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:01 smithi028 ceph-mon[39220]: pgmap v655: 162 pgs: 162 active+clean; 474 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:02.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:01 smithi028 ceph-mon[39220]: 1.8 scrub starts 2024-09-10T18:36:02.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:01 smithi028 ceph-mon[39220]: 1.8 scrub ok 2024-09-10T18:36:02.265 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:02.290 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:02.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:02.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:02.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:02.417 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:02.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:02.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:02.541 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:02.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:02.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:02.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:36:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:36:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:36:02.642 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:02.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:02.665 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:02.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:02.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:02.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:02.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:02.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:02.917 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:02.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:02.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:02.924 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:36:02.924 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:36:03.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:03.042 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:03.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:03.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:02 smithi204 ceph-mon[32357]: 4.10 scrub starts 2024-09-10T18:36:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:02 smithi204 ceph-mon[32357]: 4.10 scrub ok 2024-09-10T18:36:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:02 smithi204 ceph-mon[32357]: 5.11 scrub starts 2024-09-10T18:36:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:02 smithi204 ceph-mon[32357]: 5.11 scrub ok 2024-09-10T18:36:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:02 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:36:03.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:02 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:36:03.144 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:03.167 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:03.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:03.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:03.245 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:02 smithi028 ceph-mon[39220]: 4.10 scrub starts 2024-09-10T18:36:03.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:02 smithi028 ceph-mon[39220]: 4.10 scrub ok 2024-09-10T18:36:03.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:02 smithi028 ceph-mon[39220]: 5.11 scrub starts 2024-09-10T18:36:03.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:02 smithi028 ceph-mon[39220]: 5.11 scrub ok 2024-09-10T18:36:03.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:02 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:36:03.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:02 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:36:03.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:02 smithi028 ceph-mon[29291]: 4.10 scrub starts 2024-09-10T18:36:03.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:02 smithi028 ceph-mon[29291]: 4.10 scrub ok 2024-09-10T18:36:03.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:02 smithi028 ceph-mon[29291]: 5.11 scrub starts 2024-09-10T18:36:03.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:02 smithi028 ceph-mon[29291]: 5.11 scrub ok 2024-09-10T18:36:03.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:02 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:36:03.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:02 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:36:03.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:03.292 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:03.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:03.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:03.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:03.419 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:03.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:03.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:03.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:03.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:03.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:03.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:03.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:03.671 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:03.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:03.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:03.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:03.796 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:03.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:03.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:03.898 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:03.920 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:03.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:03.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:04.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:03 smithi028 ceph-mon[29291]: 4.d scrub starts 2024-09-10T18:36:04.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:03 smithi028 ceph-mon[29291]: 4.d scrub ok 2024-09-10T18:36:04.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:03 smithi028 ceph-mon[29291]: 5.5 scrub starts 2024-09-10T18:36:04.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:03 smithi028 ceph-mon[29291]: 5.5 scrub ok 2024-09-10T18:36:04.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:03 smithi028 ceph-mon[29291]: osdmap e312: 8 total, 8 up, 8 in 2024-09-10T18:36:04.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:03 smithi028 ceph-mon[29291]: 5.9 scrub starts 2024-09-10T18:36:04.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:03 smithi028 ceph-mon[29291]: 5.9 scrub ok 2024-09-10T18:36:04.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:03 smithi028 ceph-mon[29291]: pgmap v657: 162 pgs: 162 active+clean; 474 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:36:04.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:03 smithi028 ceph-mon[39220]: 4.d scrub starts 2024-09-10T18:36:04.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:03 smithi028 ceph-mon[39220]: 4.d scrub ok 2024-09-10T18:36:04.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:03 smithi028 ceph-mon[39220]: 5.5 scrub starts 2024-09-10T18:36:04.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:03 smithi028 ceph-mon[39220]: 5.5 scrub ok 2024-09-10T18:36:04.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:03 smithi028 ceph-mon[39220]: osdmap e312: 8 total, 8 up, 8 in 2024-09-10T18:36:04.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:03 smithi028 ceph-mon[39220]: 5.9 scrub starts 2024-09-10T18:36:04.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:03 smithi028 ceph-mon[39220]: 5.9 scrub ok 2024-09-10T18:36:04.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:03 smithi028 ceph-mon[39220]: pgmap v657: 162 pgs: 162 active+clean; 474 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:36:04.021 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:04.043 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:04.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:04.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:04.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:04.167 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:04.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:04.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:04.269 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:04.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:04.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:04.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:03 smithi204 ceph-mon[32357]: 4.d scrub starts 2024-09-10T18:36:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:03 smithi204 ceph-mon[32357]: 4.d scrub ok 2024-09-10T18:36:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:03 smithi204 ceph-mon[32357]: 5.5 scrub starts 2024-09-10T18:36:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:03 smithi204 ceph-mon[32357]: 5.5 scrub ok 2024-09-10T18:36:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:03 smithi204 ceph-mon[32357]: osdmap e312: 8 total, 8 up, 8 in 2024-09-10T18:36:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:03 smithi204 ceph-mon[32357]: 5.9 scrub starts 2024-09-10T18:36:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:03 smithi204 ceph-mon[32357]: 5.9 scrub ok 2024-09-10T18:36:04.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:03 smithi204 ceph-mon[32357]: pgmap v657: 162 pgs: 162 active+clean; 474 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:36:04.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:04.414 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:04.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:04.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:04.516 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:04.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:04.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:04.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:04.640 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:04.661 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:04.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:04.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:04.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-10T18:36:04.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:04.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:04.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:04.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:04.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:04.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:05.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:04 smithi028 ceph-mon[29291]: 3.1b scrub starts 2024-09-10T18:36:05.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:04 smithi028 ceph-mon[29291]: 3.1b scrub ok 2024-09-10T18:36:05.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:04 smithi028 ceph-mon[29291]: 3.b scrub starts 2024-09-10T18:36:05.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:04 smithi028 ceph-mon[29291]: 3.b scrub ok 2024-09-10T18:36:05.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:04 smithi028 ceph-mon[29291]: 6.f scrub starts 2024-09-10T18:36:05.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:04 smithi028 ceph-mon[29291]: 6.f scrub ok 2024-09-10T18:36:05.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:04 smithi028 ceph-mon[39220]: 3.1b scrub starts 2024-09-10T18:36:05.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:04 smithi028 ceph-mon[39220]: 3.1b scrub ok 2024-09-10T18:36:05.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:04 smithi028 ceph-mon[39220]: 3.b scrub starts 2024-09-10T18:36:05.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:04 smithi028 ceph-mon[39220]: 3.b scrub ok 2024-09-10T18:36:05.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:04 smithi028 ceph-mon[39220]: 6.f scrub starts 2024-09-10T18:36:05.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:04 smithi028 ceph-mon[39220]: 6.f scrub ok 2024-09-10T18:36:05.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:05.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:05.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:05.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:36:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:36:05.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:04 smithi204 ceph-mon[32357]: 3.1b scrub starts 2024-09-10T18:36:05.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:04 smithi204 ceph-mon[32357]: 3.1b scrub ok 2024-09-10T18:36:05.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:04 smithi204 ceph-mon[32357]: 3.b scrub starts 2024-09-10T18:36:05.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:04 smithi204 ceph-mon[32357]: 3.b scrub ok 2024-09-10T18:36:05.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:04 smithi204 ceph-mon[32357]: 6.f scrub starts 2024-09-10T18:36:05.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:04 smithi204 ceph-mon[32357]: 6.f scrub ok 2024-09-10T18:36:05.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:05.159 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:05.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:05.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:05.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:05.283 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:05.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:05.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:05.385 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:05.408 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:05.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:05.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:05.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:05.533 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:05.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:05.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:05.635 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:05.662 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:05.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:05.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:05.764 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:05.789 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:05.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:05.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:05.801 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:36:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:36:05.817 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:36:05.817 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:36:05.842 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:36:05.843 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:36:05.858 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:36:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:36:05.873 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:36:05.874 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:36:05.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:05.892 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:36:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:36:05.907 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:36:05.907 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:36:05.914 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:05.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:05.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:05.924 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:36:05.925 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:36:06.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:06.040 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:06.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:06.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:06.142 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:06.165 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:06.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:06.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:06.232 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:36:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:36:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[29291]: 1.f scrub starts 2024-09-10T18:36:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[29291]: 1.f scrub ok 2024-09-10T18:36:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[29291]: 3.11 scrub starts 2024-09-10T18:36:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[29291]: osdmap e313: 8 total, 8 up, 8 in 2024-09-10T18:36:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[29291]: 3.11 scrub ok 2024-09-10T18:36:06.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[29291]: pgmap v659: 162 pgs: 162 active+clean; 448 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:06.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[29291]: 6.14 scrub starts 2024-09-10T18:36:06.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[29291]: 6.14 scrub ok 2024-09-10T18:36:06.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4135011951' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:36:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:36:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:36:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[39220]: 1.f scrub starts 2024-09-10T18:36:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[39220]: 1.f scrub ok 2024-09-10T18:36:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[39220]: 3.11 scrub starts 2024-09-10T18:36:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[39220]: osdmap e313: 8 total, 8 up, 8 in 2024-09-10T18:36:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[39220]: 3.11 scrub ok 2024-09-10T18:36:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[39220]: pgmap v659: 162 pgs: 162 active+clean; 448 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:06.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[39220]: 6.14 scrub starts 2024-09-10T18:36:06.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[39220]: 6.14 scrub ok 2024-09-10T18:36:06.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:05 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4135011951' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:36:06.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:06.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:06.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:06.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:05 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:36:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:05 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:36:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:05 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:05 smithi204 ceph-mon[32357]: 1.f scrub starts 2024-09-10T18:36:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:05 smithi204 ceph-mon[32357]: 1.f scrub ok 2024-09-10T18:36:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:05 smithi204 ceph-mon[32357]: 3.11 scrub starts 2024-09-10T18:36:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:05 smithi204 ceph-mon[32357]: osdmap e313: 8 total, 8 up, 8 in 2024-09-10T18:36:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:05 smithi204 ceph-mon[32357]: 3.11 scrub ok 2024-09-10T18:36:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:05 smithi204 ceph-mon[32357]: pgmap v659: 162 pgs: 162 active+clean; 448 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:05 smithi204 ceph-mon[32357]: 6.14 scrub starts 2024-09-10T18:36:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:05 smithi204 ceph-mon[32357]: 6.14 scrub ok 2024-09-10T18:36:06.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:05 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4135011951' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:36:06.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:06.415 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:06.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:06.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:06.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:06.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:06.541 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:06.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:06.642 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:06.665 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:06.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:06.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:06.767 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:06.790 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:06.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:06.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:06.890 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_3' created 2024-09-10T18:36:06.892 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:06.915 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:06.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:06.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:07.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:07.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:07.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:07.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:07.157 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[29291]: 6.b scrub starts 2024-09-10T18:36:07.157 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[29291]: 6.b scrub ok 2024-09-10T18:36:07.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[29291]: 5.2 scrub starts 2024-09-10T18:36:07.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[29291]: 5.2 scrub ok 2024-09-10T18:36:07.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2621629528' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-10T18:36:07.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[29291]: 5.8 scrub starts 2024-09-10T18:36:07.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[29291]: 5.8 scrub ok 2024-09-10T18:36:07.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[29291]: 3.4 scrub starts 2024-09-10T18:36:07.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[29291]: 3.4 scrub ok 2024-09-10T18:36:07.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[29291]: 1.18 scrub starts 2024-09-10T18:36:07.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[29291]: 1.18 scrub ok 2024-09-10T18:36:07.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[39220]: 6.b scrub starts 2024-09-10T18:36:07.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[39220]: 6.b scrub ok 2024-09-10T18:36:07.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[39220]: 5.2 scrub starts 2024-09-10T18:36:07.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[39220]: 5.2 scrub ok 2024-09-10T18:36:07.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2621629528' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-10T18:36:07.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[39220]: 5.8 scrub starts 2024-09-10T18:36:07.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[39220]: 5.8 scrub ok 2024-09-10T18:36:07.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[39220]: 3.4 scrub starts 2024-09-10T18:36:07.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[39220]: 3.4 scrub ok 2024-09-10T18:36:07.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[39220]: 1.18 scrub starts 2024-09-10T18:36:07.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:06 smithi028 ceph-mon[39220]: 1.18 scrub ok 2024-09-10T18:36:07.163 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:07.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:07.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:07.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:07.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:07.289 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:07.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:06 smithi204 ceph-mon[32357]: 6.b scrub starts 2024-09-10T18:36:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:06 smithi204 ceph-mon[32357]: 6.b scrub ok 2024-09-10T18:36:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:06 smithi204 ceph-mon[32357]: 5.2 scrub starts 2024-09-10T18:36:07.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:06 smithi204 ceph-mon[32357]: 5.2 scrub ok 2024-09-10T18:36:07.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:06 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2621629528' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-10T18:36:07.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:06 smithi204 ceph-mon[32357]: 5.8 scrub starts 2024-09-10T18:36:07.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:06 smithi204 ceph-mon[32357]: 5.8 scrub ok 2024-09-10T18:36:07.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:06 smithi204 ceph-mon[32357]: 3.4 scrub starts 2024-09-10T18:36:07.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:06 smithi204 ceph-mon[32357]: 3.4 scrub ok 2024-09-10T18:36:07.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:06 smithi204 ceph-mon[32357]: 1.18 scrub starts 2024-09-10T18:36:07.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:06 smithi204 ceph-mon[32357]: 1.18 scrub ok 2024-09-10T18:36:07.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:07.416 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:07.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:07.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:07.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:07.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:07.544 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:07.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:07.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:07.668 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:07.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:07.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:07.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:07.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:36:07.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:07.924 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:07.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:07.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:08.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:08.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:08.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:08.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:08.174 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:08.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[39220]: 6.1e scrub starts 2024-09-10T18:36:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[39220]: 6.1e scrub ok 2024-09-10T18:36:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[39220]: 6.16 scrub starts 2024-09-10T18:36:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[39220]: 6.16 scrub ok 2024-09-10T18:36:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2621629528' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-10T18:36:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[39220]: osdmap e314: 8 total, 8 up, 8 in 2024-09-10T18:36:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[39220]: pgmap v661: 178 pgs: 16 unknown, 162 active+clean; 319 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:36:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/46063134' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:36:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[39220]: 5.18 scrub starts 2024-09-10T18:36:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[39220]: 5.18 scrub ok 2024-09-10T18:36:08.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[39220]: 1.14 scrub starts 2024-09-10T18:36:08.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[39220]: 1.14 scrub ok 2024-09-10T18:36:08.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[39220]: 1.1b scrub starts 2024-09-10T18:36:08.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[39220]: 1.1b scrub ok 2024-09-10T18:36:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[29291]: 6.1e scrub starts 2024-09-10T18:36:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[29291]: 6.1e scrub ok 2024-09-10T18:36:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[29291]: 6.16 scrub starts 2024-09-10T18:36:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[29291]: 6.16 scrub ok 2024-09-10T18:36:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2621629528' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-10T18:36:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[29291]: osdmap e314: 8 total, 8 up, 8 in 2024-09-10T18:36:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[29291]: pgmap v661: 178 pgs: 16 unknown, 162 active+clean; 319 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:36:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/46063134' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:36:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[29291]: 5.18 scrub starts 2024-09-10T18:36:08.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[29291]: 5.18 scrub ok 2024-09-10T18:36:08.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[29291]: 1.14 scrub starts 2024-09-10T18:36:08.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[29291]: 1.14 scrub ok 2024-09-10T18:36:08.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[29291]: 1.1b scrub starts 2024-09-10T18:36:08.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:07 smithi028 ceph-mon[29291]: 1.1b scrub ok 2024-09-10T18:36:08.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:08.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:08.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:08.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:07 smithi204 ceph-mon[32357]: 6.1e scrub starts 2024-09-10T18:36:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:07 smithi204 ceph-mon[32357]: 6.1e scrub ok 2024-09-10T18:36:08.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:07 smithi204 ceph-mon[32357]: 6.16 scrub starts 2024-09-10T18:36:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:07 smithi204 ceph-mon[32357]: 6.16 scrub ok 2024-09-10T18:36:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:07 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2621629528' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-10T18:36:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:07 smithi204 ceph-mon[32357]: osdmap e314: 8 total, 8 up, 8 in 2024-09-10T18:36:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:07 smithi204 ceph-mon[32357]: pgmap v661: 178 pgs: 16 unknown, 162 active+clean; 319 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:36:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:07 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/46063134' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:36:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:07 smithi204 ceph-mon[32357]: 5.18 scrub starts 2024-09-10T18:36:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:07 smithi204 ceph-mon[32357]: 5.18 scrub ok 2024-09-10T18:36:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:07 smithi204 ceph-mon[32357]: 1.14 scrub starts 2024-09-10T18:36:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:07 smithi204 ceph-mon[32357]: 1.14 scrub ok 2024-09-10T18:36:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:07 smithi204 ceph-mon[32357]: 1.1b scrub starts 2024-09-10T18:36:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:07 smithi204 ceph-mon[32357]: 1.1b scrub ok 2024-09-10T18:36:08.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:08.425 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:08.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:08.466 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:08.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:08.549 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:08.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:08.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:08.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:08.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:08.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:08.776 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:08.802 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:08.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:08.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:08.927 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:08.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:08.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:09.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:09.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:09.053 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:09.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:09.154 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:09.180 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:09.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:09.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[29291]: 4.0 scrub starts 2024-09-10T18:36:09.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[29291]: 4.0 scrub ok 2024-09-10T18:36:09.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[29291]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:36:09.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:36:09.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:36:09.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/46063134' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:36:09.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[29291]: osdmap e315: 8 total, 8 up, 8 in 2024-09-10T18:36:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[29291]: 3.15 scrub starts 2024-09-10T18:36:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[29291]: 3.15 scrub ok 2024-09-10T18:36:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[29291]: 1.17 scrub starts 2024-09-10T18:36:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[29291]: 1.17 scrub ok 2024-09-10T18:36:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:36:09.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:36:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[39220]: 4.0 scrub starts 2024-09-10T18:36:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[39220]: 4.0 scrub ok 2024-09-10T18:36:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[39220]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:36:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:36:09.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:36:09.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/46063134' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:36:09.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[39220]: osdmap e315: 8 total, 8 up, 8 in 2024-09-10T18:36:09.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[39220]: 3.15 scrub starts 2024-09-10T18:36:09.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[39220]: 3.15 scrub ok 2024-09-10T18:36:09.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[39220]: 1.17 scrub starts 2024-09-10T18:36:09.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[39220]: 1.17 scrub ok 2024-09-10T18:36:09.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:36:09.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:08 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:36:09.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:09.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:09.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:08 smithi204 ceph-mon[32357]: 4.0 scrub starts 2024-09-10T18:36:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:08 smithi204 ceph-mon[32357]: 4.0 scrub ok 2024-09-10T18:36:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:08 smithi204 ceph-mon[32357]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:36:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:08 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:36:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:08 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:36:09.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:08 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/46063134' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:36:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:08 smithi204 ceph-mon[32357]: osdmap e315: 8 total, 8 up, 8 in 2024-09-10T18:36:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:08 smithi204 ceph-mon[32357]: 3.15 scrub starts 2024-09-10T18:36:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:08 smithi204 ceph-mon[32357]: 3.15 scrub ok 2024-09-10T18:36:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:08 smithi204 ceph-mon[32357]: 1.17 scrub starts 2024-09-10T18:36:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:08 smithi204 ceph-mon[32357]: 1.17 scrub ok 2024-09-10T18:36:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:08 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:36:09.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:08 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:36:09.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:09.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:09.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:09.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-10T18:36:09.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:09.553 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:09.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:09.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:09.655 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:09.681 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:09.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:09.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:09.783 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:09.807 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:09.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:09.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:09.909 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:09.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:09.932 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:09.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:09.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:36:10.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:10.056 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:10.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:10.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:10.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:10.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:09 smithi028 ceph-mon[29291]: 1.11 deep-scrub starts 2024-09-10T18:36:10.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:09 smithi028 ceph-mon[29291]: 1.11 deep-scrub ok 2024-09-10T18:36:10.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:09 smithi028 ceph-mon[29291]: 1.6 scrub starts 2024-09-10T18:36:10.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:09 smithi028 ceph-mon[29291]: 1.6 scrub ok 2024-09-10T18:36:10.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:09 smithi028 ceph-mon[29291]: osdmap e316: 8 total, 8 up, 8 in 2024-09-10T18:36:10.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:09 smithi028 ceph-mon[29291]: pgmap v664: 178 pgs: 16 unknown, 162 active+clean; 319 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:36:10.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:09 smithi028 ceph-mon[29291]: 3.1d scrub starts 2024-09-10T18:36:10.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:09 smithi028 ceph-mon[29291]: 3.1d scrub ok 2024-09-10T18:36:10.169 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:09 smithi028 ceph-mon[39220]: 1.11 deep-scrub starts 2024-09-10T18:36:10.169 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:09 smithi028 ceph-mon[39220]: 1.11 deep-scrub ok 2024-09-10T18:36:10.169 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:09 smithi028 ceph-mon[39220]: 1.6 scrub starts 2024-09-10T18:36:10.169 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:09 smithi028 ceph-mon[39220]: 1.6 scrub ok 2024-09-10T18:36:10.169 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:09 smithi028 ceph-mon[39220]: osdmap e316: 8 total, 8 up, 8 in 2024-09-10T18:36:10.169 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:09 smithi028 ceph-mon[39220]: pgmap v664: 178 pgs: 16 unknown, 162 active+clean; 319 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:36:10.169 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:09 smithi028 ceph-mon[39220]: 3.1d scrub starts 2024-09-10T18:36:10.169 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:09 smithi028 ceph-mon[39220]: 3.1d scrub ok 2024-09-10T18:36:10.180 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:10.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:10.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:10.282 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:10.303 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:10.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:10.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:09 smithi204 ceph-mon[32357]: 1.11 deep-scrub starts 2024-09-10T18:36:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:09 smithi204 ceph-mon[32357]: 1.11 deep-scrub ok 2024-09-10T18:36:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:09 smithi204 ceph-mon[32357]: 1.6 scrub starts 2024-09-10T18:36:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:09 smithi204 ceph-mon[32357]: 1.6 scrub ok 2024-09-10T18:36:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:09 smithi204 ceph-mon[32357]: osdmap e316: 8 total, 8 up, 8 in 2024-09-10T18:36:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:09 smithi204 ceph-mon[32357]: pgmap v664: 178 pgs: 16 unknown, 162 active+clean; 319 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:36:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:09 smithi204 ceph-mon[32357]: 3.1d scrub starts 2024-09-10T18:36:10.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:09 smithi204 ceph-mon[32357]: 3.1d scrub ok 2024-09-10T18:36:10.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:10.427 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:10.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:10.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:10.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:10.553 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:10.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:10.654 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:10.677 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:10.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:10.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:10.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:10.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:10.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:10.904 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:10.926 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:10.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:10.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:11.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:11.051 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:11.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:11.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:11.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:11.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:11.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:11.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:11.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:10 smithi028 ceph-mon[29291]: 3.6 scrub starts 2024-09-10T18:36:11.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:10 smithi028 ceph-mon[29291]: 3.6 scrub ok 2024-09-10T18:36:11.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:10 smithi028 ceph-mon[29291]: 1.7 scrub starts 2024-09-10T18:36:11.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:10 smithi028 ceph-mon[29291]: 1.7 scrub ok 2024-09-10T18:36:11.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:10 smithi028 ceph-mon[39220]: 3.6 scrub starts 2024-09-10T18:36:11.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:10 smithi028 ceph-mon[39220]: 3.6 scrub ok 2024-09-10T18:36:11.218 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:10 smithi028 ceph-mon[39220]: 1.7 scrub starts 2024-09-10T18:36:11.218 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:10 smithi028 ceph-mon[39220]: 1.7 scrub ok 2024-09-10T18:36:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:11.278 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:11.301 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:11.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:11.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:10 smithi204 ceph-mon[32357]: 3.6 scrub starts 2024-09-10T18:36:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:10 smithi204 ceph-mon[32357]: 3.6 scrub ok 2024-09-10T18:36:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:10 smithi204 ceph-mon[32357]: 1.7 scrub starts 2024-09-10T18:36:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:10 smithi204 ceph-mon[32357]: 1.7 scrub ok 2024-09-10T18:36:11.403 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:11.425 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:11.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:11.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:11.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:36:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:36:11] "GET /metrics HTTP/1.1" 200 224409 "" "Prometheus/2.43.0" 2024-09-10T18:36:11.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:11.550 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:11.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:11.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:11.652 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:11.674 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:11.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:11.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:11.776 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:11.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:11.801 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:11.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:11.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:36:11.902 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:11.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:11 smithi028 ceph-mon[29291]: 4.19 scrub starts 2024-09-10T18:36:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:11 smithi028 ceph-mon[29291]: 4.19 scrub ok 2024-09-10T18:36:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:11 smithi028 ceph-mon[29291]: pgmap v665: 178 pgs: 2 unknown, 176 active+clean; 434 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:11 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3480592433' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-09-10T18:36:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:11 smithi028 ceph-mon[29291]: from='client.35579 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-09-10T18:36:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:11 smithi028 ceph-mon[29291]: 5.12 scrub starts 2024-09-10T18:36:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:11 smithi028 ceph-mon[29291]: 5.12 scrub ok 2024-09-10T18:36:11.916 INFO:teuthology.orchestra.run.smithi028.stderr:clear 4.4 pg_upmap_items mapping 2024-09-10T18:36:11.930 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:11.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:11.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:12.032 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:12.054 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:12.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:12.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:12.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:12.177 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:12.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:12.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:11 smithi028 ceph-mon[39220]: 4.19 scrub starts 2024-09-10T18:36:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:11 smithi028 ceph-mon[39220]: 4.19 scrub ok 2024-09-10T18:36:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:11 smithi028 ceph-mon[39220]: pgmap v665: 178 pgs: 2 unknown, 176 active+clean; 434 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:11 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3480592433' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-09-10T18:36:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:11 smithi028 ceph-mon[39220]: from='client.35579 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-09-10T18:36:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:11 smithi028 ceph-mon[39220]: 5.12 scrub starts 2024-09-10T18:36:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:11 smithi028 ceph-mon[39220]: 5.12 scrub ok 2024-09-10T18:36:12.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:12.302 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:12.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:12.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:12.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:36:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:36:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:36:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:11 smithi204 ceph-mon[32357]: 4.19 scrub starts 2024-09-10T18:36:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:11 smithi204 ceph-mon[32357]: 4.19 scrub ok 2024-09-10T18:36:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:11 smithi204 ceph-mon[32357]: pgmap v665: 178 pgs: 2 unknown, 176 active+clean; 434 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:11 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3480592433' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-09-10T18:36:12.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:11 smithi204 ceph-mon[32357]: from='client.35579 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]: dispatch 2024-09-10T18:36:12.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:11 smithi204 ceph-mon[32357]: 5.12 scrub starts 2024-09-10T18:36:12.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:11 smithi204 ceph-mon[32357]: 5.12 scrub ok 2024-09-10T18:36:12.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:12.426 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:12.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:12.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:12.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:12.549 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:12.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:12.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:12.674 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:12.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:12.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:12.776 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:12.798 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:12.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:12.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:12.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:12.922 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:12.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:12.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:13.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:13.046 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:13.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:13.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:13.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:13.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:13.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:12 smithi028 ceph-mon[29291]: 5.1f scrub starts 2024-09-10T18:36:13.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:12 smithi028 ceph-mon[29291]: 5.1f scrub ok 2024-09-10T18:36:13.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:12 smithi028 ceph-mon[29291]: from='client.35579 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]': finished 2024-09-10T18:36:13.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:12 smithi028 ceph-mon[29291]: osdmap e317: 8 total, 8 up, 8 in 2024-09-10T18:36:13.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:12 smithi028 ceph-mon[29291]: 1.d scrub starts 2024-09-10T18:36:13.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:12 smithi028 ceph-mon[29291]: 1.d scrub ok 2024-09-10T18:36:13.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:12 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:36:13.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:12 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:36:13.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:12 smithi028 ceph-mon[39220]: 5.1f scrub starts 2024-09-10T18:36:13.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:12 smithi028 ceph-mon[39220]: 5.1f scrub ok 2024-09-10T18:36:13.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:12 smithi028 ceph-mon[39220]: from='client.35579 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]': finished 2024-09-10T18:36:13.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:12 smithi028 ceph-mon[39220]: osdmap e317: 8 total, 8 up, 8 in 2024-09-10T18:36:13.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:12 smithi028 ceph-mon[39220]: 1.d scrub starts 2024-09-10T18:36:13.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:12 smithi028 ceph-mon[39220]: 1.d scrub ok 2024-09-10T18:36:13.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:12 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:36:13.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:12 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:36:13.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:13.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:13.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:12 smithi204 ceph-mon[32357]: 5.1f scrub starts 2024-09-10T18:36:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:12 smithi204 ceph-mon[32357]: 5.1f scrub ok 2024-09-10T18:36:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:12 smithi204 ceph-mon[32357]: from='client.35579 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.4"}]': finished 2024-09-10T18:36:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:12 smithi204 ceph-mon[32357]: osdmap e317: 8 total, 8 up, 8 in 2024-09-10T18:36:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:12 smithi204 ceph-mon[32357]: 1.d scrub starts 2024-09-10T18:36:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:12 smithi204 ceph-mon[32357]: 1.d scrub ok 2024-09-10T18:36:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:12 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:36:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:12 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:36:13.397 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:13.418 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:13.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:13.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:13.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:13.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:13.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:13.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:13.667 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:13.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:13.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:13.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:13.792 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:13.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:13.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:13.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:13.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:13.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:14.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:14.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:14.042 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:14.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:14.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:14.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:14.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:13 smithi028 ceph-mon[29291]: osdmap e318: 8 total, 8 up, 8 in 2024-09-10T18:36:14.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:13 smithi028 ceph-mon[29291]: pgmap v668: 178 pgs: 178 active+clean; 434 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:14.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:13 smithi028 ceph-mon[39220]: osdmap e318: 8 total, 8 up, 8 in 2024-09-10T18:36:14.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:13 smithi028 ceph-mon[39220]: pgmap v668: 178 pgs: 178 active+clean; 434 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:14.267 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:14.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:14.290 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:14.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:13 smithi204 ceph-mon[32357]: osdmap e318: 8 total, 8 up, 8 in 2024-09-10T18:36:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:13 smithi204 ceph-mon[32357]: pgmap v668: 178 pgs: 178 active+clean; 434 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:14.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:14.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:14.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:14.489 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:36:14.489 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-10T18:36:14.489 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-10T18:36:14.489 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:36:14.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:14.539 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:14.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:14.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:14.641 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:14.662 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:14.664 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:14.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:14.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:14.766 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:14.789 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:14.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:14.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:14.891 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:14.913 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:14.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:14.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:15.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:15.039 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:15.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:15.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:15.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:36:14 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:36:15.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:14 smithi204 ceph-mon[32357]: 5.0 scrub starts 2024-09-10T18:36:15.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:14 smithi204 ceph-mon[32357]: 5.0 scrub ok 2024-09-10T18:36:15.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:14 smithi204 ceph-mon[32357]: 1.9 scrub starts 2024-09-10T18:36:15.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:14 smithi204 ceph-mon[32357]: 1.9 scrub ok 2024-09-10T18:36:15.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:14 smithi204 ceph-mon[32357]: 4.15 scrub starts 2024-09-10T18:36:15.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:14 smithi204 ceph-mon[32357]: 4.15 scrub ok 2024-09-10T18:36:15.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:14 smithi028 ceph-mon[29291]: 5.0 scrub starts 2024-09-10T18:36:15.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:14 smithi028 ceph-mon[29291]: 5.0 scrub ok 2024-09-10T18:36:15.122 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:14 smithi028 ceph-mon[29291]: 1.9 scrub starts 2024-09-10T18:36:15.122 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:14 smithi028 ceph-mon[29291]: 1.9 scrub ok 2024-09-10T18:36:15.122 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:14 smithi028 ceph-mon[29291]: 4.15 scrub starts 2024-09-10T18:36:15.122 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:14 smithi028 ceph-mon[29291]: 4.15 scrub ok 2024-09-10T18:36:15.122 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:14 smithi028 ceph-mon[39220]: 5.0 scrub starts 2024-09-10T18:36:15.122 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:14 smithi028 ceph-mon[39220]: 5.0 scrub ok 2024-09-10T18:36:15.122 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:14 smithi028 ceph-mon[39220]: 1.9 scrub starts 2024-09-10T18:36:15.122 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:14 smithi028 ceph-mon[39220]: 1.9 scrub ok 2024-09-10T18:36:15.122 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:14 smithi028 ceph-mon[39220]: 4.15 scrub starts 2024-09-10T18:36:15.122 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:14 smithi028 ceph-mon[39220]: 4.15 scrub ok 2024-09-10T18:36:15.141 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:15.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-10T18:36:15.163 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:15.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:15.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:15.265 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:15.287 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:15.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:15.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:15.389 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:15.411 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:15.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:15.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:15.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:15.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:15.538 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:15.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:15.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:15.662 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:15.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:15.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:15.764 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:15.788 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:15.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:15.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:15.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:15.913 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:15.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:15.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:16.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:16.038 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:16.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:16.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:16.140 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:16.167 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:16.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:16.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:16.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:15 smithi028 ceph-mon[29291]: 6.9 scrub starts 2024-09-10T18:36:16.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:15 smithi028 ceph-mon[29291]: 6.9 scrub ok 2024-09-10T18:36:16.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:15 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:16.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:15 smithi028 ceph-mon[29291]: 1.15 scrub starts 2024-09-10T18:36:16.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:15 smithi028 ceph-mon[29291]: 1.15 scrub ok 2024-09-10T18:36:16.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:15 smithi028 ceph-mon[29291]: pgmap v669: 178 pgs: 178 active+clean; 408 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:16.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:15 smithi028 ceph-mon[29291]: 6.19 scrub starts 2024-09-10T18:36:16.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:15 smithi028 ceph-mon[29291]: 6.19 scrub ok 2024-09-10T18:36:16.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:15 smithi028 ceph-mon[39220]: 6.9 scrub starts 2024-09-10T18:36:16.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:15 smithi028 ceph-mon[39220]: 6.9 scrub ok 2024-09-10T18:36:16.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:15 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:16.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:15 smithi028 ceph-mon[39220]: 1.15 scrub starts 2024-09-10T18:36:16.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:15 smithi028 ceph-mon[39220]: 1.15 scrub ok 2024-09-10T18:36:16.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:15 smithi028 ceph-mon[39220]: pgmap v669: 178 pgs: 178 active+clean; 408 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:16.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:15 smithi028 ceph-mon[39220]: 6.19 scrub starts 2024-09-10T18:36:16.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:15 smithi028 ceph-mon[39220]: 6.19 scrub ok 2024-09-10T18:36:16.269 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:16.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:16.297 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:16.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:15 smithi204 ceph-mon[32357]: 6.9 scrub starts 2024-09-10T18:36:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:15 smithi204 ceph-mon[32357]: 6.9 scrub ok 2024-09-10T18:36:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:15 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:15 smithi204 ceph-mon[32357]: 1.15 scrub starts 2024-09-10T18:36:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:15 smithi204 ceph-mon[32357]: 1.15 scrub ok 2024-09-10T18:36:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:15 smithi204 ceph-mon[32357]: pgmap v669: 178 pgs: 178 active+clean; 408 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T18:36:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:15 smithi204 ceph-mon[32357]: 6.19 scrub starts 2024-09-10T18:36:16.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:15 smithi204 ceph-mon[32357]: 6.19 scrub ok 2024-09-10T18:36:16.398 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:16.424 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:16.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:16.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:16.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:16.551 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:16.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:16.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:16.653 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:16.678 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:16.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:16.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:16.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:16.803 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:16.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:16.905 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:16.928 INFO:teuthology.orchestra.run.smithi028.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-10T18:36:16.931 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:16.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:17.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:17.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:17.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:17.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:17.160 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:17.184 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:17.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:17.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:17.195 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:17.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:16 smithi028 ceph-mon[29291]: 4.6 scrub starts 2024-09-10T18:36:17.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:16 smithi028 ceph-mon[29291]: 4.6 scrub ok 2024-09-10T18:36:17.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:16 smithi028 ceph-mon[29291]: 4.1 scrub starts 2024-09-10T18:36:17.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:16 smithi028 ceph-mon[29291]: 4.1 scrub ok 2024-09-10T18:36:17.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:16 smithi028 ceph-mon[29291]: 4.12 scrub starts 2024-09-10T18:36:17.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:16 smithi028 ceph-mon[29291]: 4.12 scrub ok 2024-09-10T18:36:17.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:16 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1863174378' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:36:17.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:16 smithi028 ceph-mon[29291]: from='client.35585 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:36:17.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:16 smithi028 ceph-mon[39220]: 4.6 scrub starts 2024-09-10T18:36:17.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:16 smithi028 ceph-mon[39220]: 4.6 scrub ok 2024-09-10T18:36:17.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:16 smithi028 ceph-mon[39220]: 4.1 scrub starts 2024-09-10T18:36:17.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:16 smithi028 ceph-mon[39220]: 4.1 scrub ok 2024-09-10T18:36:17.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:16 smithi028 ceph-mon[39220]: 4.12 scrub starts 2024-09-10T18:36:17.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:16 smithi028 ceph-mon[39220]: 4.12 scrub ok 2024-09-10T18:36:17.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:16 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1863174378' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:36:17.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:16 smithi028 ceph-mon[39220]: from='client.35585 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:36:17.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:17.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:17.310 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:17.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:16 smithi204 ceph-mon[32357]: 4.6 scrub starts 2024-09-10T18:36:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:16 smithi204 ceph-mon[32357]: 4.6 scrub ok 2024-09-10T18:36:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:16 smithi204 ceph-mon[32357]: 4.1 scrub starts 2024-09-10T18:36:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:16 smithi204 ceph-mon[32357]: 4.1 scrub ok 2024-09-10T18:36:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:16 smithi204 ceph-mon[32357]: 4.12 scrub starts 2024-09-10T18:36:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:16 smithi204 ceph-mon[32357]: 4.12 scrub ok 2024-09-10T18:36:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:16 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1863174378' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:36:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:16 smithi204 ceph-mon[32357]: from='client.35585 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:36:17.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:17.437 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:17.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:17.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:17.539 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:17.564 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:17.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:17.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:17.666 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:17.689 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:17.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:17.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:17.791 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:17.814 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:17.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:17.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:17.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:36:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:17.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:17.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:18.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:18.065 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:18.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:18.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:18.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[29291]: from='client.35585 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:36:18.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[29291]: osdmap e319: 8 total, 8 up, 8 in 2024-09-10T18:36:18.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[29291]: pgmap v671: 178 pgs: 178 active+clean; 408 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-10T18:36:18.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1328434262' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:36:18.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[29291]: 4.1c scrub starts 2024-09-10T18:36:18.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[29291]: 4.1c scrub ok 2024-09-10T18:36:18.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[29291]: 6.4 scrub starts 2024-09-10T18:36:18.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[29291]: 6.4 scrub ok 2024-09-10T18:36:18.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[29291]: 5.16 scrub starts 2024-09-10T18:36:18.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[29291]: 5.16 scrub ok 2024-09-10T18:36:18.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[39220]: from='client.35585 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:36:18.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[39220]: osdmap e319: 8 total, 8 up, 8 in 2024-09-10T18:36:18.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[39220]: pgmap v671: 178 pgs: 178 active+clean; 408 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-10T18:36:18.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1328434262' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:36:18.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[39220]: 4.1c scrub starts 2024-09-10T18:36:18.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[39220]: 4.1c scrub ok 2024-09-10T18:36:18.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[39220]: 6.4 scrub starts 2024-09-10T18:36:18.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[39220]: 6.4 scrub ok 2024-09-10T18:36:18.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[39220]: 5.16 scrub starts 2024-09-10T18:36:18.096 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:17 smithi028 ceph-mon[39220]: 5.16 scrub ok 2024-09-10T18:36:18.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:18.190 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:18.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:18.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:18.292 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:18.318 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:18.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:18.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:17 smithi204 ceph-mon[32357]: from='client.35585 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:36:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:17 smithi204 ceph-mon[32357]: osdmap e319: 8 total, 8 up, 8 in 2024-09-10T18:36:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:17 smithi204 ceph-mon[32357]: pgmap v671: 178 pgs: 178 active+clean; 408 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-10T18:36:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:17 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1328434262' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:36:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:17 smithi204 ceph-mon[32357]: 4.1c scrub starts 2024-09-10T18:36:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:17 smithi204 ceph-mon[32357]: 4.1c scrub ok 2024-09-10T18:36:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:17 smithi204 ceph-mon[32357]: 6.4 scrub starts 2024-09-10T18:36:18.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:17 smithi204 ceph-mon[32357]: 6.4 scrub ok 2024-09-10T18:36:18.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:17 smithi204 ceph-mon[32357]: 5.16 scrub starts 2024-09-10T18:36:18.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:17 smithi204 ceph-mon[32357]: 5.16 scrub ok 2024-09-10T18:36:18.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:18.442 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:18.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:18.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:18.544 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:18.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:18.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:18.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:18.695 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:18.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:18.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:18.758 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:36:18.759 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:36:18.759 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-10T18:36:18.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T18:36:18.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:18.842 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:18.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:18.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:18.871 DEBUG:teuthology.orchestra.run.smithi028:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-10T18:36:18.886 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-10T18:36:18.886 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-10T18:36:18.944 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:18.969 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:18.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:18.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:19.015 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:hints = 1 2024-09-10T18:36:19.015 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-10T18:36:19.015 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object prefix: benchmark_data_smithi028_161947 2024-09-10T18:36:19.015 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:36:19.015 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 0 0 0 0 0 0 - 0 2024-09-10T18:36:19.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:19.103 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:19.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:19.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:19.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:18 smithi028 ceph-mon[39220]: 3.1f scrub starts 2024-09-10T18:36:19.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:18 smithi028 ceph-mon[39220]: 3.1f scrub ok 2024-09-10T18:36:19.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:18 smithi028 ceph-mon[39220]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:36:19.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:18 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:36:19.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:18 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:36:19.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:18 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1328434262' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:36:19.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:18 smithi028 ceph-mon[39220]: osdmap e320: 8 total, 8 up, 8 in 2024-09-10T18:36:19.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:18 smithi028 ceph-mon[29291]: 3.1f scrub starts 2024-09-10T18:36:19.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:18 smithi028 ceph-mon[29291]: 3.1f scrub ok 2024-09-10T18:36:19.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:18 smithi028 ceph-mon[29291]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:36:19.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:18 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:36:19.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:18 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:36:19.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:18 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1328434262' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:36:19.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:18 smithi028 ceph-mon[29291]: osdmap e320: 8 total, 8 up, 8 in 2024-09-10T18:36:19.205 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:19.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:19.241 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:19.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:19.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:18 smithi204 ceph-mon[32357]: 3.1f scrub starts 2024-09-10T18:36:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:18 smithi204 ceph-mon[32357]: 3.1f scrub ok 2024-09-10T18:36:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:18 smithi204 ceph-mon[32357]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:36:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:18 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:36:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:18 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:36:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:18 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1328434262' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:36:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:18 smithi204 ceph-mon[32357]: osdmap e320: 8 total, 8 up, 8 in 2024-09-10T18:36:19.376 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:19.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:19.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:19.478 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:19.508 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:19.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:19.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:19.537 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:19.610 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:19.644 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:19.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:19.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:19.779 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:19.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:19.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:19.881 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:19.914 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:19.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:19.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:20.015 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1 16 3846 3830 239.369 239.375 0.00364169 0.00416573 2024-09-10T18:36:20.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:20.050 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:20.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:20.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:20.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:20.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:20.193 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:20.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:20.208 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:19 smithi028 ceph-mon[39220]: 6.13 scrub starts 2024-09-10T18:36:20.208 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:19 smithi028 ceph-mon[39220]: pgmap v673: 178 pgs: 178 active+clean; 408 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-10T18:36:20.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:19 smithi028 ceph-mon[29291]: 6.13 scrub starts 2024-09-10T18:36:20.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:19 smithi028 ceph-mon[29291]: pgmap v673: 178 pgs: 178 active+clean; 408 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-10T18:36:20.246 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:20.295 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:20.326 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:20.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:20.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:20.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:19 smithi204 ceph-mon[32357]: 6.13 scrub starts 2024-09-10T18:36:20.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:19 smithi204 ceph-mon[32357]: pgmap v673: 178 pgs: 178 active+clean; 408 KiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-10T18:36:20.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:20.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:20.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:20.575 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:20.607 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:20.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:20.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:20.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:20.748 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:20.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:20.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:20.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:36:20.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:20.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:20.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:20.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:20.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:21.015 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2 16 7409 7393 231.003 222.688 0.00405292 0.00431834 2024-09-10T18:36:21.031 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:21.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:21.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:21.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:21.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:21.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:21.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:21.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:21.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:21.391 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:36:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:36:21] "GET /metrics HTTP/1.1" 200 224409 "" "Prometheus/2.43.0" 2024-09-10T18:36:21.414 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:21.448 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:21.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:21.550 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:21.580 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:21.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:21.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:21.682 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:21.714 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:21.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:21.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:21.728 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:36:21.728 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:36:21.728 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:36:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:36:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:36:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:36:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:36:21.729 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:36:21.729 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:36:21.729 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:36:21.729 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:36:21.729 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:36:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:36:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:36:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:36:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:36:21.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:21.856 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:21.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:21.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:21.958 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:21.991 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:21.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:21.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:22.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3 16 10615 10599 220.788 200.375 0.00339408 0.00452214 2024-09-10T18:36:22.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:22.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:22.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:22.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:22.238 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:22.273 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:22.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:22.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:22.375 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:22.407 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:22.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:22.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:22.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:22 smithi028 ceph-mon[39220]: pgmap v674: 178 pgs: 178 active+clean; 300 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s rd, 37 MiB/s wr, 599 op/s; 0 B/s, 1 objects/s recovering 2024-09-10T18:36:22.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:22 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2112265022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:36:22.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:22 smithi028 ceph-mon[29291]: pgmap v674: 178 pgs: 178 active+clean; 300 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s rd, 37 MiB/s wr, 599 op/s; 0 B/s, 1 objects/s recovering 2024-09-10T18:36:22.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:22 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2112265022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:36:22.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:22.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:22.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:22.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:22 smithi204 ceph-mon[32357]: pgmap v674: 178 pgs: 178 active+clean; 300 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s rd, 37 MiB/s wr, 599 op/s; 0 B/s, 1 objects/s recovering 2024-09-10T18:36:22.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:22 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2112265022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:36:22.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:36:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:36:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:36:22.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:22.683 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:22.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:22.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:22.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:22.820 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:22.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:22.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:22.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:22.956 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:22.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:22.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:23.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4 16 13877 13861 216.556 203.875 0.00302494 0.00460876 2024-09-10T18:36:23.057 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:23.088 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:23.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:23.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:23.190 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:23.224 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:23.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:23.326 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:23.363 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:23.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:23.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:23.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:23.502 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:23.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:23.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:23.604 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:23.634 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:23.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:23.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:23.736 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:23.770 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:23.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:23.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:23.872 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:23.907 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:23.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:23.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:24.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:24.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5 16 16815 16799 209.967 183.625 0.00197176 0.00475559 2024-09-10T18:36:24.042 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:24.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:24.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:24.144 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:24.178 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:24.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:24.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:24.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:24.326 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:24.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:24.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:24 smithi028 ceph-mon[39220]: pgmap v675: 178 pgs: 178 active+clean; 358 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 511 B/s rd, 45 MiB/s wr, 714 op/s; 44 B/s, 1 objects/s recovering 2024-09-10T18:36:24.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:24 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:36:24.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:36:24.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:24 smithi028 ceph-mon[29291]: pgmap v675: 178 pgs: 178 active+clean; 358 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 511 B/s rd, 45 MiB/s wr, 714 op/s; 44 B/s, 1 objects/s recovering 2024-09-10T18:36:24.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:24 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:36:24.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:36:24.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:24.471 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:24.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:24.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:24.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:24 smithi204 ceph-mon[32357]: pgmap v675: 178 pgs: 178 active+clean; 358 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 511 B/s rd, 45 MiB/s wr, 714 op/s; 44 B/s, 1 objects/s recovering 2024-09-10T18:36:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:24 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:36:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:36:24.605 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:24.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:24.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:24.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:24.745 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:24.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:24.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:24.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:24.885 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:24.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:24.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:24.987 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:25.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 6 16 19475 19459 202.678 166.25 0.00362523 0.0049285 2024-09-10T18:36:25.019 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:25.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:25.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:25.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:36:24 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:36:25.121 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:25.155 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:25.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:25.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:25.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:25.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:25.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:25.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:25.397 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:25.427 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:25.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:25.529 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:25.563 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:25.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:25.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:25.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:36:25.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:25.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:25.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:25.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:25.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:25.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:25.852 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:25.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:25.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:25.992 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:25.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:25.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:26.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 7 16 21968 21952 195.981 155.812 0.00779497 0.00509534 2024-09-10T18:36:26.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:26.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:26.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:26.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:26.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:26.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:26.273 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:26.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:26.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:26.399 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:26 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:26.399 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:26 smithi028 ceph-mon[39220]: pgmap v676: 178 pgs: 178 active+clean; 542 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-10T18:36:26.400 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:26 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:26.400 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:26 smithi028 ceph-mon[29291]: pgmap v676: 178 pgs: 178 active+clean; 542 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-10T18:36:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:26.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:26.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:26.516 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:26.552 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:26.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:26.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:26 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:26 smithi204 ceph-mon[32357]: pgmap v676: 178 pgs: 178 active+clean; 542 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-10T18:36:26.654 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:26.665 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:36:26.665 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:36:26.665 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 320, 2024-09-10T18:36:26.665 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "5eababb2-6fa0-11ef-bcea-c7b262605968", 2024-09-10T18:36:26.665 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-10T18:16:20.680373+0000", 2024-09-10T18:36:26.665 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-10T18:36:17.928237+0000", 2024-09-10T18:36:26.665 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-09-10T18:26:26.611338+0000", 2024-09-10T18:36:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-09-10T18:21:02.114621+0000", 2024-09-10T18:36:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T18:36:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5801984, 2024-09-10T18:36:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-09-10T18:36:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "noscrub", 2024-09-10T18:36:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-09-10T18:36:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-09-10T18:36:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-09-10T18:36:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-09-10T18:36:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:36:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 19, 2024-09-10T18:36:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T18:36:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T18:36:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T18:36:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-09-10T18:36:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 10, 2024-09-10T18:36:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-09-10T18:36:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-09-10T18:36:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-09-10T18:36:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "quincy", 2024-09-10T18:36:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-09-10T18:36:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-09-10T18:36:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-09-10T18:36:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:19:24.013536+0000", 2024-09-10T18:36:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:36:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:36:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:36:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:36:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:36:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:36:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:36:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:36:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:36:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:36:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:36:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:36:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 31, 2024-09-10T18:36:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 31, 2024-09-10T18:36:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 31, 2024-09-10T18:36:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 31, 2024-09-10T18:36:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 31, 2024-09-10T18:36:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:36:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:36:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:36:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:36:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:36:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:36:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:36:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "300", 2024-09-10T18:36:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:36:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:36:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "287", 2024-09-10T18:36:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:36:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:36:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:36:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:36:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:36:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:36:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:36:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:36:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:36:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:36:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:36:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:36:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:36:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:36:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:36:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:36:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:36:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:36:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:36:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:36:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:36:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:36:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:36:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:36:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:36:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:36:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:36:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:36:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:36:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:36:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:36:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:36:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:36:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:36:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-10T18:36:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-09-10T18:36:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-09-10T18:36:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:36:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-09-10T18:36:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-09-10T18:36:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-09-10T18:36:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:30.065209+0000", 2024-09-10T18:36:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-09-10T18:36:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T18:36:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:36:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:36:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:36:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:36:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:36:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:36:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:36:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:36:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:36:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:36:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-09-10T18:36:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-09-10T18:36:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-09-10T18:36:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-09-10T18:36:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-09-10T18:36:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:36:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:36:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:36:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:36:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:36:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:36:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:36:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "62", 2024-09-10T18:36:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:36:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:36:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:36:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:36:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:36:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 3, 2024-09-10T18:36:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 62, 2024-09-10T18:36:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:36:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:36:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:36:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:36:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:36:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:36:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:36:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:36:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:36:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:36:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:36:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:36:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:36:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:36:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:36:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:36:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:36:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:36:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:36:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:36:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:36:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:36:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:36:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:36:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:36:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:36:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:36:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:36:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:36:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:36:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:36:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:36:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-09-10T18:36:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-09-10T18:36:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-09-10T18:36:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:30.506441+0000", 2024-09-10T18:36:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:36:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:36:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:36:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:36:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:36:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:36:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:36:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:36:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:36:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:36:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:36:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:36:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:36:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:36:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:36:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:36:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:36:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:36:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:36:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:36:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:36:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:36:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:36:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:36:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "59", 2024-09-10T18:36:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:36:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:36:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:36:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:36:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:36:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:36:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:36:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:36:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:36:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:36:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:36:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:36:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:36:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:36:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:36:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:36:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:36:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:36:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:36:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:36:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:36:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:36:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:36:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:36:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:36:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:36:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:36:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:36:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:36:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:36:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:36:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:36:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:36:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:36:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:36:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:36:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:36:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:36:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:36:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:36:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-09-10T18:36:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-09-10T18:36:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:32.008130+0000", 2024-09-10T18:36:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:36:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:36:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:36:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:36:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:36:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:36:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:36:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:36:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:36:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:36:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:36:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:36:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:36:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:36:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:36:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:36:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:36:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:36:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:36:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:36:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:36:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:36:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:36:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:36:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "61", 2024-09-10T18:36:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:36:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:36:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:36:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:36:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:36:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:36:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:36:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:36:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:36:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:36:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:36:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:36:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:36:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:36:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:36:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:36:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:36:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:36:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:36:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:36:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:36:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:36:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:36:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:36:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:36:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:36:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:36:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:36:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:36:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:36:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:36:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:36:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:36:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:36:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:36:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:36:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:36:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:36:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:36:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:36:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-09-10T18:36:26.693 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-09-10T18:36:26.693 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:34.060571+0000", 2024-09-10T18:36:26.693 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:36:26.693 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:36:26.693 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:36:26.693 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:36:26.693 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:36:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:36:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:36:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:36:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:36:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:36:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:36:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:36:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:36:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:36:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:36:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:36:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:36:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:36:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:36:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:36:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:36:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:36:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:36:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:36:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "63", 2024-09-10T18:36:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:36:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:36:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:36:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:36:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:36:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:36:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:36:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:36:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:36:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:36:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:36:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:36:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:36:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:36:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:36:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:36:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:36:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:36:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:36:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:36:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:36:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:36:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:36:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:36:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:36:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:36:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:36:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:36:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:36:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:36:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:36:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:36:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:36:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:36:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:36:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:36:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:36:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:36:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:36:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:36:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-09-10T18:36:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-09-10T18:36:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:36.127330+0000", 2024-09-10T18:36:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:36:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:36:26.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:36:26.700 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:36:26.700 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:36:26.700 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:36:26.700 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:36:26.700 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:36:26.700 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:36:26.700 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:36:26.700 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:36:26.700 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:36:26.700 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:36:26.700 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:36:26.700 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:36:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:36:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:36:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:36:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:36:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:36:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:36:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:36:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:36:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:36:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "65", 2024-09-10T18:36:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:36:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:36:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:36:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:36:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:36:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:36:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:36:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:36:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:36:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:36:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:36:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:36:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:36:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:36:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:36:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:36:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:36:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:36:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:36:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:36:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:36:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:36:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:36:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:36:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:36:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:36:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:36:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:36:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:36:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:36:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:36:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:36:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:36:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:36:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:36:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:36:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:36:26.705 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-10T18:36:26.705 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-09-10T18:36:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:36:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:36:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 10, 2024-09-10T18:36:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "unique_pool_3", 2024-09-10T18:36:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:36:06.616074+0000", 2024-09-10T18:36:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:36:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:36:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:36:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:36:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:36:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:36:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:36:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:36:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:36:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:36:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:36:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:36:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 16, 2024-09-10T18:36:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 16, 2024-09-10T18:36:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 16, 2024-09-10T18:36:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 16, 2024-09-10T18:36:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 16, 2024-09-10T18:36:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:36:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:36:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:36:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:36:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:36:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:36:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:36:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "319", 2024-09-10T18:36:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:36:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:36:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:36:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:36:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:36:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:36:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:36:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:36:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:36:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:36:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:36:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:36:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:36:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:36:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:36:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:36:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:36:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:36:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:36:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:36:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:36:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:36:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:36:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:36:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:36:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:36:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:36:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:36:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:36:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:36:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:36:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:36:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:36:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:36:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:36:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:36:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:36:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:36:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:36:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-09-10T18:36:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:36:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:36:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-10T18:36:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "f1c834d9-252a-4077-b25e-169181d70251", 2024-09-10T18:36:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:36:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:36:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:36:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:36:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:36:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:36:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-09-10T18:36:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 314, 2024-09-10T18:36:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:36:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:36:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:36:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-09-10T18:36:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:36:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:36:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-09-10T18:36:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:36:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:36:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-09-10T18:36:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:36:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:36:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-09-10T18:36:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:36:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/3188372351", 2024-09-10T18:36:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/3188372351", 2024-09-10T18:36:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/3188372351", 2024-09-10T18:36:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/3188372351", 2024-09-10T18:36:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:36:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:36:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:36:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-10T18:36:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "81f1855d-b946-4ca3-b246-38977b9e20fd", 2024-09-10T18:36:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:36:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:36:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:36:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:36:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:36:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:36:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-10T18:36:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 317, 2024-09-10T18:36:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:36:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:36:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:36:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-09-10T18:36:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:36:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:36:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-09-10T18:36:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:36:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:36:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-09-10T18:36:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:36:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:36:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-09-10T18:36:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:36:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1369912678", 2024-09-10T18:36:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1369912678", 2024-09-10T18:36:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1369912678", 2024-09-10T18:36:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1369912678", 2024-09-10T18:36:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:36:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:36:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:36:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-10T18:36:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "2ad55d2a-36fa-4756-bf22-58d4b049486e", 2024-09-10T18:36:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:36:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:36:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:36:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:36:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:36:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:36:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-09-10T18:36:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 314, 2024-09-10T18:36:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:36:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:36:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:36:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-09-10T18:36:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:36:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:36:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-09-10T18:36:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:36:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:36:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-09-10T18:36:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:36:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:36:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-09-10T18:36:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:36:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/1243406003", 2024-09-10T18:36:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/1243406003", 2024-09-10T18:36:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/1243406003", 2024-09-10T18:36:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1243406003", 2024-09-10T18:36:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:36:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:36:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:36:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-10T18:36:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "242729a5-a5e8-44d5-97c8-14d14e970bff", 2024-09-10T18:36:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:36:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:36:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:36:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:36:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:36:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:36:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-09-10T18:36:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 289, 2024-09-10T18:36:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:36:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:36:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:36:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-09-10T18:36:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:36:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:36:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-09-10T18:36:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:36:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:36:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-09-10T18:36:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:36:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:36:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-09-10T18:36:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:36:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/67703283", 2024-09-10T18:36:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/67703283", 2024-09-10T18:36:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/67703283", 2024-09-10T18:36:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/67703283", 2024-09-10T18:36:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:36:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:36:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:36:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-10T18:36:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "6e38b8ef-e6ed-493f-8eca-755354754787", 2024-09-10T18:36:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:36:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:36:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:36:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:36:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:36:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:36:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 33, 2024-09-10T18:36:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 314, 2024-09-10T18:36:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:36:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:36:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:36:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6800", 2024-09-10T18:36:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:36:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:36:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6801", 2024-09-10T18:36:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:36:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:36:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6803", 2024-09-10T18:36:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:36:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:36:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6802", 2024-09-10T18:36:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:36:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6800/1619669076", 2024-09-10T18:36:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6801/1619669076", 2024-09-10T18:36:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6803/1619669076", 2024-09-10T18:36:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6802/1619669076", 2024-09-10T18:36:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:36:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:36:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:36:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-10T18:36:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "a92e0441-66c9-498e-a0ac-6465acdbb1f5", 2024-09-10T18:36:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:36:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:36:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:36:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:36:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:36:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:36:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 39, 2024-09-10T18:36:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 314, 2024-09-10T18:36:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:36:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:36:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:36:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6804", 2024-09-10T18:36:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:36:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:36:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6805", 2024-09-10T18:36:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:36:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:36:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6807", 2024-09-10T18:36:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:36:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:36:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6806", 2024-09-10T18:36:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:36:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6804/2717802880", 2024-09-10T18:36:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6805/2717802880", 2024-09-10T18:36:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6807/2717802880", 2024-09-10T18:36:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6806/2717802880", 2024-09-10T18:36:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:36:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:36:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:36:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-09-10T18:36:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "12b02875-6653-45ed-8d2b-b4d33bd88525", 2024-09-10T18:36:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:36:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:36:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:36:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:36:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 46, 2024-09-10T18:36:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 94, 2024-09-10T18:36:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 144, 2024-09-10T18:36:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 314, 2024-09-10T18:36:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 95, 2024-09-10T18:36:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:36:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:36:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6808", 2024-09-10T18:36:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:36:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:36:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6809", 2024-09-10T18:36:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:36:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:36:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6811", 2024-09-10T18:36:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:36:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:36:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6810", 2024-09-10T18:36:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:36:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6808/3479311452", 2024-09-10T18:36:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6809/3479311452", 2024-09-10T18:36:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6811/3479311452", 2024-09-10T18:36:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6810/3479311452", 2024-09-10T18:36:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:36:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:36:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:36:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-09-10T18:36:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "0c4f7aa9-ff95-4192-a146-ca576eee907f", 2024-09-10T18:36:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:36:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:36:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:36:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:36:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:36:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:36:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 54, 2024-09-10T18:36:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 314, 2024-09-10T18:36:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:36:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:36:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:36:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6812", 2024-09-10T18:36:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:36:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:36:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6813", 2024-09-10T18:36:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:36:26.749 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.749 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:26.749 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:26.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:36:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6815", 2024-09-10T18:36:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:36:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:36:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:36:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:36:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6814", 2024-09-10T18:36:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:36:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6812/3511791879", 2024-09-10T18:36:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6813/3511791879", 2024-09-10T18:36:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6815/3511791879", 2024-09-10T18:36:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6814/3511791879", 2024-09-10T18:36:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:36:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:36:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:36:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:36:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-09-10T18:36:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-10T18:36:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:36:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:36:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:36:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:36:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:36:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:35:59.338144+0000", 2024-09-10T18:36:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:36:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-10T18:36:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:36:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:36:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:36:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:36:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:36:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:36:02.706822+0000", 2024-09-10T18:36:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:36:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-10T18:36:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:36:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:36:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:36:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:36:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:36:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:36:12.381850+0000", 2024-09-10T18:36:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:36:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-10T18:36:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:36:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:36:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:36:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:36:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:36:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:35:52.899686+0000", 2024-09-10T18:36:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:36:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-10T18:36:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:36:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:36:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:36:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:36:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:36:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:35:13.915850+0000", 2024-09-10T18:36:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:36:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-10T18:36:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:36:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:36:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:36:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:36:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:36:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:35:42.401737+0000", 2024-09-10T18:36:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:36:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-09-10T18:36:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:36:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:36:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:36:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:36:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:36:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:35:34.158547+0000", 2024-09-10T18:36:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 94 2024-09-10T18:36:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-09-10T18:36:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:36:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:36:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:36:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:36:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:36:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:36:04.293204+0000", 2024-09-10T18:36:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:36:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:36:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-09-10T18:36:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.c", 2024-09-10T18:36:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:36:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-09-10T18:36:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: 4, 2024-09-10T18:36:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-10T18:36:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "6.17", 2024-09-10T18:36:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:36:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: 3, 2024-09-10T18:36:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-09-10T18:36:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: 5 2024-09-10T18:36:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:36:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-09-10T18:36:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1b", 2024-09-10T18:36:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:36:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:36:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-09-10T18:36:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1f", 2024-09-10T18:36:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:36:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:36:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-09-10T18:36:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:36:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-09-10T18:36:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-09-10T18:36:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-09-10T18:36:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1745574083": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:36:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3866220552": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:36:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3760775432": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:36:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1746797014": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:36:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1029045895": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:36:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3310949126": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:36:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2874160765": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:36:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2620179974": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:36:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/421194352": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:36:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/1077377902": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:36:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3428757300": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:36:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1759116360": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:36:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2564333506": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:36:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/717120167": "2024-09-11T18:22:08.678985+0000" 2024-09-10T18:36:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-09-10T18:36:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-09-10T18:36:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-09-10T18:36:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-09-10T18:36:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-09-10T18:36:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-09-10T18:36:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-09-10T18:36:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-09-10T18:36:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-09-10T18:36:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-09-10T18:36:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-09-10T18:36:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-09-10T18:36:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-09-10T18:36:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-09-10T18:36:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-09-10T18:36:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-09-10T18:36:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-09-10T18:36:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-09-10T18:36:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-09-10T18:36:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:26.765 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:36:26.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:26.894 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:26.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:26.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:26.997 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:27.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 8 16 24430 24414 190.716 153.875 0.00932879 0.00523695 2024-09-10T18:36:27.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:27.031 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:27.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:27.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:27.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:27.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:27.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:27.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:27 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/514839777' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:36:27.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:27 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/514839777' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:36:27.271 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:27.303 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:27.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:27.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:27.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:27.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:27.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:27.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:27.582 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:27.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:27.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:27.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:27 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/514839777' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:36:27.665 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:36:27.665 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:36:27.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:27.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:27.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:27.824 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:27.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:27.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:27.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:27.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:27.996 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:27.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:27.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:28.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 9 16 27025 27009 187.545 162.188 0.00374122 0.00532695 2024-09-10T18:36:28.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:28.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:28.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:28.237 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:28.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:28.272 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:28.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:28.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:28.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:28.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:28.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:28 smithi028 ceph-mon[29291]: pgmap v677: 178 pgs: 178 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s 2024-09-10T18:36:28.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:28 smithi028 ceph-mon[39220]: pgmap v677: 178 pgs: 178 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s 2024-09-10T18:36:28.515 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:28.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:28.551 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:28.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:28 smithi204 ceph-mon[32357]: pgmap v677: 178 pgs: 178 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s 2024-09-10T18:36:28.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:28.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:28.700 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:28.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:28.716 INFO:tasks.thrashosds.thrasher:j is {'epoch': 320, 'fsid': '5eababb2-6fa0-11ef-bcea-c7b262605968', 'created': '2024-09-10T18:16:20.680373+0000', 'modified': '2024-09-10T18:36:17.928237+0000', 'last_up_change': '2024-09-10T18:26:26.611338+0000', 'last_in_change': '2024-09-10T18:21:02.114621+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T18:19:24.013536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '300', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '287', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-10T18:21:30.065209+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 62, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T18:21:30.506441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T18:21:32.008130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T18:21:34.060571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T18:21:36.127330+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-10T18:36:06.616074+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '319', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'f1c834d9-252a-4077-b25e-169181d70251', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 314, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 3188372351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 3188372351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 3188372351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 3188372351}]}, 'public_addr': '172.21.15.28:6801/3188372351', 'cluster_addr': '172.21.15.28:6802/3188372351', 'heartbeat_back_addr': '172.21.15.28:6804/3188372351', 'heartbeat_front_addr': '172.21.15.28:6803/3188372351', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '81f1855d-b946-4ca3-b246-38977b9e20fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 317, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1369912678}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1369912678}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1369912678}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1369912678}]}, 'public_addr': '172.21.15.28:6805/1369912678', 'cluster_addr': '172.21.15.28:6806/1369912678', 'heartbeat_back_addr': '172.21.15.28:6808/1369912678', 'heartbeat_front_addr': '172.21.15.28:6807/1369912678', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2ad55d2a-36fa-4756-bf22-58d4b049486e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 314, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 1243406003}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 1243406003}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 1243406003}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1243406003}]}, 'public_addr': '172.21.15.28:6809/1243406003', 'cluster_addr': '172.21.15.28:6810/1243406003', 'heartbeat_back_addr': '172.21.15.28:6812/1243406003', 'heartbeat_front_addr': '172.21.15.28:6811/1243406003', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '242729a5-a5e8-44d5-97c8-14d14e970bff', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 289, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 67703283}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 67703283}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 67703283}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 67703283}]}, 'public_addr': '172.21.15.28:6813/67703283', 'cluster_addr': '172.21.15.28:6814/67703283', 'heartbeat_back_addr': '172.21.15.28:6816/67703283', 'heartbeat_front_addr': '172.21.15.28:6815/67703283', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6e38b8ef-e6ed-493f-8eca-755354754787', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 314, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6800', 'nonce': 1619669076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 1619669076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 1619669076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6802', 'nonce': 1619669076}]}, 'public_addr': '172.21.15.204:6800/1619669076', 'cluster_addr': '172.21.15.204:6801/1619669076', 'heartbeat_back_addr': '172.21.15.204:6803/1619669076', 'heartbeat_front_addr': '172.21.15.204:6802/1619669076', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a92e0441-66c9-498e-a0ac-6465acdbb1f5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 314, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6804', 'nonce': 2717802880}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 2717802880}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 2717802880}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6806', 'nonce': 2717802880}]}, 'public_addr': '172.21.15.204:6804/2717802880', 'cluster_addr': '172.21.15.204:6805/2717802880', 'heartbeat_back_addr': '172.21.15.204:6807/2717802880', 'heartbeat_front_addr': '172.21.15.204:6806/2717802880', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '12b02875-6653-45ed-8d2b-b4d33bd88525', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 46, 'last_clean_end': 94, 'up_from': 144, 'up_thru': 314, 'down_at': 95, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6808', 'nonce': 3479311452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 3479311452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 3479311452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6810', 'nonce': 3479311452}]}, 'public_addr': '172.21.15.204:6808/3479311452', 'cluster_addr': '172.21.15.204:6809/3479311452', 'heartbeat_back_addr': '172.21.15.204:6811/3479311452', 'heartbeat_front_addr': '172.21.15.204:6810/3479311452', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0c4f7aa9-ff95-4192-a146-ca576eee907f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 314, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6812', 'nonce': 3511791879}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 3511791879}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 3511791879}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6814', 'nonce': 3511791879}]}, 'public_addr': '172.21.15.204:6812/3511791879', 'cluster_addr': '172.21.15.204:6813/3511791879', 'heartbeat_back_addr': '172.21.15.204:6815/3511791879', 'heartbeat_front_addr': '172.21.15.204:6814/3511791879', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:35:59.338144+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:36:02.706822+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:36:12.381850+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:35:52.899686+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:35:13.915850+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:35:42.401737+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:35:34.158547+0000', 'dead_epoch': 94}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:36:04.293204+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.c', 'osds': [7, 4, 2]}, {'pgid': '6.17', 'osds': [3, 7, 5]}], 'pg_upmap_items': [{'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/1745574083': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/3866220552': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:6800/3760775432': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:0/1746797014': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:0/1029045895': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:6800/3310949126': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/2874160765': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/2620179974': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/421194352': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:6800/1077377902': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/3428757300': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/1759116360': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/2564333506': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/717120167': '2024-09-11T18:22:08.678985+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T18:36:28.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:36:28.801 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:28.835 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:28.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:28.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:28.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:28.990 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:28.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:29.018 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 10 15 29311 29296 183.082 142.938 0.0101225 0.0054567 2024-09-10T18:36:29.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-10T18:36:29.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:29.154 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:29.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:29.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:29 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:36:29.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:29 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:36:29.255 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:29.288 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:29.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:29.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:29.389 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:29.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:29.422 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:29.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:29.523 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:29.552 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:29.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:29.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:29.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:29 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:36:29.654 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:29.659 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:29.665 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:29.680 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:29.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:29.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:29.781 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:29.812 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:29.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:29.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:29.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:29.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:30.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 11 16 31269 31253 177.557 122.312 0.00612887 0.00562723 2024-09-10T18:36:30.056 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:30.088 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:30.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:30.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:30.190 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:30.226 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:30.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:30.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:30.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:30.358 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:30.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:30.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:30.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:30.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:30.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:30.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:30 smithi028 ceph-mon[39220]: pgmap v678: 178 pgs: 178 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 919 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-09-10T18:36:30.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:30 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:36:30.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:30 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:36:30.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:30 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:36:30.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:30 smithi028 ceph-mon[29291]: pgmap v678: 178 pgs: 178 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 919 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-09-10T18:36:30.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:30 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:36:30.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:30 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:36:30.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:30 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:36:30.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:30 smithi204 ceph-mon[32357]: pgmap v678: 178 pgs: 178 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 919 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-09-10T18:36:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:30 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:36:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:30 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:36:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:30 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:36:30.641 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:30.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:30.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:30.742 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:30.780 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:30.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:30.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:30.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:30.919 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:30.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:30.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:31.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 12 16 33539 33523 174.583 141.875 0.00618181 0.00572234 2024-09-10T18:36:31.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:31.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:31.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:31.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:31.201 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:31.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:31.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:31.303 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:31.323 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:31.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:31.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:31.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:31.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:31.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:31.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:31.598 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:31.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:31.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:31.684 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:36:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:36:31] "GET /metrics HTTP/1.1" 200 224531 "" "Prometheus/2.43.0" 2024-09-10T18:36:31.699 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:31.730 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:31.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:31.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:31.831 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:31.852 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:31.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:31.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:31.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:31 smithi028 ceph-mon[39220]: pgmap v679: 178 pgs: 178 active+clean; 1.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-10T18:36:31.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:31 smithi028 ceph-mon[29291]: pgmap v679: 178 pgs: 178 active+clean; 1.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-10T18:36:31.954 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:31.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:31.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:31.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:32.016 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 13 16 35253 35237 169.393 107.125 0.00402691 0.0058987 2024-09-10T18:36:32.100 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:32.103 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:31 smithi204 ceph-mon[32357]: pgmap v679: 178 pgs: 178 active+clean; 1.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-10T18:36:32.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:32.132 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:32.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:32.233 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:32.265 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:32.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:32.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:32.402 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:32.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:32.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:32.504 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:32.536 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:32.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:32.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:32.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:36:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:36:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:36:32.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:32.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:32.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:32.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:32.760 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:32.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:32.798 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:32.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:32.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:32.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:32.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:33.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 14 16 37015 36999 165.159 110.125 0.0086729 0.00604927 2024-09-10T18:36:33.040 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:33.076 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:33.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:33.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:33.177 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:33.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:32 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3337041410' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:36:33.208 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:32 smithi028 ceph-mon[39220]: from='client.25600 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:36:33.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:32 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3337041410' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:36:33.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:32 smithi028 ceph-mon[29291]: from='client.25600 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:36:33.214 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:33.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:33.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:33.316 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:33.352 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:33.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:33.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:32 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3337041410' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:36:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:32 smithi204 ceph-mon[32357]: from='client.25600 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:36:33.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:33.496 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:33.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:33.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:33.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:33.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:33.631 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:33.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:33.732 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:33.769 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:33.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:33.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:33.871 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:33.891 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:33.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:33.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:34.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 15 16 38664 38648 161.019 103.062 0.0101438 0.00620536 2024-09-10T18:36:34.024 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:34.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:34.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:34.127 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:34.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:34.163 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:34.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:33 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:36:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:33 smithi028 ceph-mon[39220]: from='client.25600 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:36:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:33 smithi028 ceph-mon[39220]: osdmap e321: 8 total, 8 up, 8 in 2024-09-10T18:36:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:33 smithi028 ceph-mon[39220]: pgmap v681: 178 pgs: 178 active+clean; 2.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 167 MiB/s wr, 2.67k op/s 2024-09-10T18:36:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:33 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:36:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:33 smithi028 ceph-mon[29291]: from='client.25600 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:36:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:33 smithi028 ceph-mon[29291]: osdmap e321: 8 total, 8 up, 8 in 2024-09-10T18:36:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:33 smithi028 ceph-mon[29291]: pgmap v681: 178 pgs: 178 active+clean; 2.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 167 MiB/s wr, 2.67k op/s 2024-09-10T18:36:34.264 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:34.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:34.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:33 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:36:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:33 smithi204 ceph-mon[32357]: from='client.25600 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:36:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:33 smithi204 ceph-mon[32357]: osdmap e321: 8 total, 8 up, 8 in 2024-09-10T18:36:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:33 smithi204 ceph-mon[32357]: pgmap v681: 178 pgs: 178 active+clean; 2.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 167 MiB/s wr, 2.67k op/s 2024-09-10T18:36:34.400 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:34.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:34.422 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:34.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:34.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:34.559 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:34.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:34.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:34.661 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:34.697 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:34.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:34.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:34.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:34.836 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:34.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:34.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:34.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:34.974 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:34.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:34.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:35.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 16 16 40599 40583 158.513 120.938 0.00713176 0.00630334 2024-09-10T18:36:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:36:35.076 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:35.105 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:36:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:36:35.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:35.113 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:35.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:35.214 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:35.246 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:35.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:35.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:35.348 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:35.381 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:35.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:35.483 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:35.516 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:35.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:35.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:35.578 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:35.617 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:35.651 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:35.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:35.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:35.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:35.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:35.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:35.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:35.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:35.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:36.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 17 16 42647 42631 156.717 128 0.00991065 0.00637552 2024-09-10T18:36:36.032 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:36.062 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:36.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:36.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:36.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:36.203 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:36.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:36.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:36.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:36.345 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:36.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:36.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:36.447 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:36.481 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:36.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:36.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:36.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:36 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:36.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:36 smithi028 ceph-mon[29291]: pgmap v682: 178 pgs: 178 active+clean; 2.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 159 MiB/s wr, 2.55k op/s 2024-09-10T18:36:36.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:36 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:36:36.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:36 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:36:36.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:36 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:36.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:36 smithi028 ceph-mon[39220]: pgmap v682: 178 pgs: 178 active+clean; 2.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 159 MiB/s wr, 2.55k op/s 2024-09-10T18:36:36.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:36 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:36:36.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:36 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:36:36.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:36 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:36 smithi204 ceph-mon[32357]: pgmap v682: 178 pgs: 178 active+clean; 2.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 159 MiB/s wr, 2.55k op/s 2024-09-10T18:36:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:36 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:36:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:36 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:36:36.621 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:36.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:36.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:36.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:36.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:36.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:36.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:36.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:36.912 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:36.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:37.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:37.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 18 16 44562 44546 154.66 119.688 0.00639297 0.00646015 2024-09-10T18:36:37.053 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:37.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:37.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:37.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:37.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:37.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:37.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:37.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:37.336 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:37.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:37.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:37.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:37.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:37.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:37 smithi028 ceph-mon[29291]: osdmap e322: 8 total, 8 up, 8 in 2024-09-10T18:36:37.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:37 smithi028 ceph-mon[39220]: osdmap e322: 8 total, 8 up, 8 in 2024-09-10T18:36:37.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:37.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:37 smithi204 ceph-mon[32357]: osdmap e322: 8 total, 8 up, 8 in 2024-09-10T18:36:37.621 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:37.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:37.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:37.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:37.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:37.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:37.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:37.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:37.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:38.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 19 16 46443 46427 152.705 117.562 0.00712842 0.00654363 2024-09-10T18:36:38.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:38.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:38.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:38.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:36:38.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:38.180 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:38 smithi028 ceph-mon[29291]: pgmap v684: 178 pgs: 178 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 163 MiB/s wr, 2.61k op/s 2024-09-10T18:36:38.181 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:38 smithi028 ceph-mon[39220]: pgmap v684: 178 pgs: 178 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 163 MiB/s wr, 2.61k op/s 2024-09-10T18:36:38.188 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:38.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:38.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:38.289 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:38.321 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:38.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:38.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:38.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:38.456 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:36:38.456 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:36:38.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:38.473 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:38.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:38.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:38.599 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:38.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:38.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:38 smithi204 ceph-mon[32357]: pgmap v684: 178 pgs: 178 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 163 MiB/s wr, 2.61k op/s 2024-09-10T18:36:38.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:38.737 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:38.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:38.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:38.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:38.875 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:38.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:38.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:38.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:39.014 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:39.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:39.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:39.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:36:39.016875+0000 min lat: 0.00134649 max lat: 0.176394 avg lat: 0.00665555 2024-09-10T18:36:39.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:36:39.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 20 16 48063 48047 150.132 101.25 0.0070206 0.00665555 2024-09-10T18:36:39.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:39.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:39.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:39.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:39.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:39.275 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:39.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:39.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:39.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:39.406 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:39.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:39.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:36:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:36:39.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:39.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:39.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:39.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:36:39.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:39.685 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:39.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:39.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:39.816 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:39.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:39.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:39.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:39.956 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:39.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:39.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:40.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 21 16 49440 49424 147.08 86.0625 0.00767118 0.00679379 2024-09-10T18:36:40.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:40.089 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:40.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:40.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:40.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:40.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:40.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:40.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:40.371 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:40.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:40.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:40.473 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:40.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:40 smithi028 ceph-mon[29291]: pgmap v685: 178 pgs: 178 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-10T18:36:40.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:40 smithi028 ceph-mon[39220]: pgmap v685: 178 pgs: 178 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-10T18:36:40.507 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:40.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:40.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:40.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:40 smithi204 ceph-mon[32357]: pgmap v685: 178 pgs: 178 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-10T18:36:40.608 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:40.644 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:40.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:40.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:40.746 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:40.780 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:40.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:40.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:40.881 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:40.917 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:40.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:40.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:41.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 22 16 51213 51197 145.431 110.812 0.0126872 0.00687067 2024-09-10T18:36:41.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:41.049 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:41.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:41.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:41.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:41.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:41.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:41.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:41.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:41.330 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:41.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:41.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:41.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:41.468 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:41.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:41.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:41.570 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:41.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:41.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:41.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:41.746 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:41.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:41.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:41.753 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:36:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:36:41] "GET /metrics HTTP/1.1" 200 224652 "" "Prometheus/2.43.0" 2024-09-10T18:36:41.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:41.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-10T18:36:41.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:41.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:41.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:41.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:42.019 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 23 16 52858 52842 143.578 102.812 0.0105096 0.00695995 2024-09-10T18:36:42.034 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:42.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:42.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:42.061 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:42.136 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:42.165 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:42.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:42.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:42.267 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:42.303 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:42.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:42.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:42.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:36:42.428 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:42 smithi028 ceph-mon[39220]: pgmap v686: 178 pgs: 178 active+clean; 3.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 128 MiB/s wr, 2.05k op/s 2024-09-10T18:36:42.429 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:42 smithi028 ceph-mon[29291]: pgmap v686: 178 pgs: 178 active+clean; 3.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 128 MiB/s wr, 2.05k op/s 2024-09-10T18:36:42.436 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:36:42.439 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:42.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:42.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:42.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:42.576 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:42.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:42.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:42 smithi204 ceph-mon[32357]: pgmap v686: 178 pgs: 178 active+clean; 3.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 128 MiB/s wr, 2.05k op/s 2024-09-10T18:36:42.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:36:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:36:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:36:42.678 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:42.712 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:42.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:42.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:42.814 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:42.833 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:36:42.833 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:36:42.834 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:36:42.834 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:36:42.845 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:36:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:36:42.849 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:42.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:42.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:42.878 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:36:42.878 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:36:42.912 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:36:42.912 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:36:42.942 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:36:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:36:42.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:42.977 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:36:42.978 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:36:42.980 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:42.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:42.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:43.016 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:36:43.016 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:36:43.017 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 24 16 54287 54271 141.317 89.3125 0.00817011 0.00707065 2024-09-10T18:36:43.082 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:43.119 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:43.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:43.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:43.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:43.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:43 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/508569574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:36:43.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:43 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/508569574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:36:43.260 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:43.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:43.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:43.362 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:43.394 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:43.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:43.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:43.496 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:43.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:43.533 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:43.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:43 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/508569574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:36:43.634 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:43.670 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:43.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:43.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:43.772 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:43.804 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:43.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:43.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:43.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:43.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:43.946 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:43.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:44.018 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 25 16 55986 55970 139.911 106.188 0.00682839 0.00714235 2024-09-10T18:36:44.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:44.084 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:44.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:44.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:44.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:44.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:44.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:44.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:44.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:44.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:44.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:44.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:44 smithi028 ceph-mon[29291]: from='client.25606 v1:172.21.15.28:0/2086113865' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:36:44.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:44 smithi028 ceph-mon[29291]: pgmap v687: 178 pgs: 178 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-09-10T18:36:44.498 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:44 smithi028 ceph-mon[39220]: from='client.25606 v1:172.21.15.28:0/2086113865' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:36:44.498 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:44 smithi028 ceph-mon[39220]: pgmap v687: 178 pgs: 178 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-09-10T18:36:44.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:44.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:44.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:44.544 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":686,"stamp":"2024-09-10T18:36:41.071899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3231932612,"num_objects":49505,"num_object_clones":0,"num_object_copies":148515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49505,"num_whiteouts":0,"num_read":2057,"num_read_kb":1816,"num_write":50033,"num_write_kb":3155930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":245281,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52658,"ondisk_log_size":52658,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13332728,"kb_used_data":9150968,"kb_used_omap":19,"kb_used_meta":4181676,"kb_avail":736628488,"statfs":{"total":767960285184,"available":754307571712,"internally_reserved":0,"allocated":9370591232,"data_stored":9367233609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20033,"internal_metadata":4282036671},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1105002496,"num_objects":16861,"num_object_clones":0,"num_object_copies":50583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16861,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":16861,"num_write_kb":1079104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.220144"},"pg_stats":[{"pgid":"1.1d","version":"76'81","reported_seq":56,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.858945+0000","last_change":"2024-09-10T18:35:17.818822+0000","last_active":"2024-09-10T18:36:32.858945+0000","last_peered":"2024-09-10T18:36:32.858945+0000","last_clean":"2024-09-10T18:36:32.858945+0000","last_became_active":"2024-09-10T18:34:34.865894+0000","last_became_peered":"2024-09-10T18:34:34.865894+0000","last_unstale":"2024-09-10T18:36:32.858945+0000","last_undegraded":"2024-09-10T18:36:32.858945+0000","last_fullsized":"2024-09-10T18:36:32.858945+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:17.818727+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:17.818727+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:41.743523+0000","scrub_duration":0.0037738960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":342,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.867042+0000","last_change":"2024-09-10T18:36:17.645197+0000","last_active":"2024-09-10T18:36:32.867042+0000","last_peered":"2024-09-10T18:36:32.867042+0000","last_clean":"2024-09-10T18:36:32.867042+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:36:32.867042+0000","last_undegraded":"2024-09-10T18:36:32.867042+0000","last_fullsized":"2024-09-10T18:36:32.867042+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:17.645146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:36:17.645146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:43.759721+0000","scrub_duration":0.003140618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":370,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.856284+0000","last_change":"2024-09-10T18:35:20.421834+0000","last_active":"2024-09-10T18:36:32.856284+0000","last_peered":"2024-09-10T18:36:32.856284+0000","last_clean":"2024-09-10T18:36:32.856284+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:36:32.856284+0000","last_undegraded":"2024-09-10T18:36:32.856284+0000","last_fullsized":"2024-09-10T18:36:32.856284+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:35:20.421727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:20.421727+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:46.115408+0000","scrub_duration":0.010997518,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":337,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.858390+0000","last_change":"2024-09-10T18:35:00.915434+0000","last_active":"2024-09-10T18:36:32.858390+0000","last_peered":"2024-09-10T18:36:32.858390+0000","last_clean":"2024-09-10T18:36:32.858390+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:36:32.858390+0000","last_undegraded":"2024-09-10T18:36:32.858390+0000","last_fullsized":"2024-09-10T18:36:32.858390+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:00.915210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:00.915210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052862780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":327,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.856421+0000","last_change":"2024-09-10T18:35:53.315831+0000","last_active":"2024-09-10T18:36:32.856421+0000","last_peered":"2024-09-10T18:36:32.856421+0000","last_clean":"2024-09-10T18:36:32.856421+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:36:32.856421+0000","last_undegraded":"2024-09-10T18:36:32.856421+0000","last_fullsized":"2024-09-10T18:36:32.856421+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:08.643758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:33:08.643758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046354059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'81","reported_seq":41,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855779+0000","last_change":"2024-09-10T18:34:34.868599+0000","last_active":"2024-09-10T18:36:32.855779+0000","last_peered":"2024-09-10T18:36:32.855779+0000","last_clean":"2024-09-10T18:36:32.855779+0000","last_became_active":"2024-09-10T18:34:34.867388+0000","last_became_peered":"2024-09-10T18:34:34.867388+0000","last_unstale":"2024-09-10T18:36:32.855779+0000","last_undegraded":"2024-09-10T18:36:32.855779+0000","last_fullsized":"2024-09-10T18:36:32.855779+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.370561+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.370561+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029251390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54614,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":334,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855349+0000","last_change":"2024-09-10T18:34:16.293404+0000","last_active":"2024-09-10T18:36:32.855349+0000","last_peered":"2024-09-10T18:36:32.855349+0000","last_clean":"2024-09-10T18:36:32.855349+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:36:32.855349+0000","last_undegraded":"2024-09-10T18:36:32.855349+0000","last_fullsized":"2024-09-10T18:36:32.855349+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:16.293206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:16.293206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021119960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":340,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855582+0000","last_change":"2024-09-10T18:34:22.415416+0000","last_active":"2024-09-10T18:36:32.855582+0000","last_peered":"2024-09-10T18:36:32.855582+0000","last_clean":"2024-09-10T18:36:32.855582+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:36:32.855582+0000","last_undegraded":"2024-09-10T18:36:32.855582+0000","last_fullsized":"2024-09-10T18:36:32.855582+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:22.415093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.415093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.068845071999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'34","reported_seq":381,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855532+0000","last_change":"2024-09-10T18:36:10.786060+0000","last_active":"2024-09-10T18:36:32.855532+0000","last_peered":"2024-09-10T18:36:32.855532+0000","last_clean":"2024-09-10T18:36:32.855532+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:36:32.855532+0000","last_undegraded":"2024-09-10T18:36:32.855532+0000","last_fullsized":"2024-09-10T18:36:32.855532+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:36:10.786002+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:36:10.786002+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:36.303670+0000","scrub_duration":0.0021577860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":346,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.856208+0000","last_change":"2024-09-10T18:36:07.325812+0000","last_active":"2024-09-10T18:36:32.856208+0000","last_peered":"2024-09-10T18:36:32.856208+0000","last_clean":"2024-09-10T18:36:32.856208+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:36:32.856208+0000","last_undegraded":"2024-09-10T18:36:32.856208+0000","last_fullsized":"2024-09-10T18:36:32.856208+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:07.325744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.325744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:24.078418+0000","scrub_duration":0.001764406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":351,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.853150+0000","last_change":"2024-09-10T18:36:09.700648+0000","last_active":"2024-09-10T18:36:32.853150+0000","last_peered":"2024-09-10T18:36:32.853150+0000","last_clean":"2024-09-10T18:36:32.853150+0000","last_became_active":"2024-09-10T18:26:27.644961+0000","last_became_peered":"2024-09-10T18:26:27.644961+0000","last_unstale":"2024-09-10T18:36:32.853150+0000","last_undegraded":"2024-09-10T18:36:32.853150+0000","last_fullsized":"2024-09-10T18:36:32.853150+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:09.700579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_clean_scrub_stamp":"2024-09-10T18:36:09.700579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:12.857090+0000","scrub_duration":0.00158731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'15","reported_seq":363,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857515+0000","last_change":"2024-09-10T18:34:25.455830+0000","last_active":"2024-09-10T18:36:32.857515+0000","last_peered":"2024-09-10T18:36:32.857515+0000","last_clean":"2024-09-10T18:36:32.857515+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:36:32.857515+0000","last_undegraded":"2024-09-10T18:36:32.857515+0000","last_fullsized":"2024-09-10T18:36:32.857515+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:34:25.455754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:25.455754+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.13960720300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":337,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.853772+0000","last_change":"2024-09-10T18:34:17.265867+0000","last_active":"2024-09-10T18:36:32.853772+0000","last_peered":"2024-09-10T18:36:32.853772+0000","last_clean":"2024-09-10T18:36:32.853772+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:36:32.853772+0000","last_undegraded":"2024-09-10T18:36:32.853772+0000","last_fullsized":"2024-09-10T18:36:32.853772+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:17.265788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_clean_scrub_stamp":"2024-09-10T18:34:17.265788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023387740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":336,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.866109+0000","last_change":"2024-09-10T18:35:04.782030+0000","last_active":"2024-09-10T18:36:32.866109+0000","last_peered":"2024-09-10T18:36:32.866109+0000","last_clean":"2024-09-10T18:36:32.866109+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:36:32.866109+0000","last_undegraded":"2024-09-10T18:36:32.866109+0000","last_fullsized":"2024-09-10T18:36:32.866109+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:04.781946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:04.781946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011382141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'81","reported_seq":110,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855380+0000","last_change":"2024-09-10T18:34:34.863425+0000","last_active":"2024-09-10T18:36:32.855380+0000","last_peered":"2024-09-10T18:36:32.855380+0000","last_clean":"2024-09-10T18:36:32.855380+0000","last_became_active":"2024-09-10T18:34:34.863102+0000","last_became_peered":"2024-09-10T18:34:34.863102+0000","last_unstale":"2024-09-10T18:36:32.855380+0000","last_undegraded":"2024-09-10T18:36:32.855380+0000","last_fullsized":"2024-09-10T18:36:32.855380+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:32:48.006695+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:32:48.006695+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":341,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.852711+0000","last_change":"2024-09-10T18:34:54.513360+0000","last_active":"2024-09-10T18:36:32.852711+0000","last_peered":"2024-09-10T18:36:32.852711+0000","last_clean":"2024-09-10T18:36:32.852711+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:36:32.852711+0000","last_undegraded":"2024-09-10T18:36:32.852711+0000","last_fullsized":"2024-09-10T18:36:32.852711+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:54.513278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:17.464205+0000","last_clean_scrub_stamp":"2024-09-10T18:34:54.513278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028371450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":375,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.856180+0000","last_change":"2024-09-10T18:34:52.235528+0000","last_active":"2024-09-10T18:36:32.856180+0000","last_peered":"2024-09-10T18:36:32.856180+0000","last_clean":"2024-09-10T18:36:32.856180+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:36:32.856180+0000","last_undegraded":"2024-09-10T18:36:32.856180+0000","last_fullsized":"2024-09-10T18:36:32.856180+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'9","last_scrub_stamp":"2024-09-10T18:34:52.235425+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:34:52.235425+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029466150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":336,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.854971+0000","last_change":"2024-09-10T18:35:01.711169+0000","last_active":"2024-09-10T18:36:32.854971+0000","last_peered":"2024-09-10T18:36:32.854971+0000","last_clean":"2024-09-10T18:36:32.854971+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:36:32.854971+0000","last_undegraded":"2024-09-10T18:36:32.854971+0000","last_fullsized":"2024-09-10T18:36:32.854971+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:01.710785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:01.710785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033386729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":338,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.852982+0000","last_change":"2024-09-10T18:36:15.726285+0000","last_active":"2024-09-10T18:36:32.852982+0000","last_peered":"2024-09-10T18:36:32.852982+0000","last_clean":"2024-09-10T18:36:32.852982+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:36:32.852982+0000","last_undegraded":"2024-09-10T18:36:32.852982+0000","last_fullsized":"2024-09-10T18:36:32.852982+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:15.726200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:36:15.726200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:20.772259+0000","scrub_duration":0.0020103579999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'81","reported_seq":193,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.854381+0000","last_change":"2024-09-10T18:34:34.863449+0000","last_active":"2024-09-10T18:36:32.854381+0000","last_peered":"2024-09-10T18:36:32.854381+0000","last_clean":"2024-09-10T18:36:32.854381+0000","last_became_active":"2024-09-10T18:34:34.862917+0000","last_became_peered":"2024-09-10T18:34:34.862917+0000","last_unstale":"2024-09-10T18:36:32.854381+0000","last_undegraded":"2024-09-10T18:36:32.854381+0000","last_fullsized":"2024-09-10T18:36:32.854381+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:06.053455+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.053455+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003148245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":11,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":344,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.858235+0000","last_change":"2024-09-10T18:36:06.360654+0000","last_active":"2024-09-10T18:36:32.858235+0000","last_peered":"2024-09-10T18:36:32.858235+0000","last_clean":"2024-09-10T18:36:32.858235+0000","last_became_active":"2024-09-10T18:26:27.650774+0000","last_became_peered":"2024-09-10T18:26:27.650774+0000","last_unstale":"2024-09-10T18:36:32.858235+0000","last_undegraded":"2024-09-10T18:36:32.858235+0000","last_fullsized":"2024-09-10T18:36:32.858235+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.360595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.360595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:21.064681+0000","scrub_duration":0.0012270849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":338,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.867166+0000","last_change":"2024-09-10T18:36:03.738564+0000","last_active":"2024-09-10T18:36:32.867166+0000","last_peered":"2024-09-10T18:36:32.867166+0000","last_clean":"2024-09-10T18:36:32.867166+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:36:32.867166+0000","last_undegraded":"2024-09-10T18:36:32.867166+0000","last_fullsized":"2024-09-10T18:36:32.867166+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:03.738510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:36:03.738510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:09.910983+0000","scrub_duration":0.0011989469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"240'27","reported_seq":411,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.858216+0000","last_change":"2024-09-10T18:36:17.526735+0000","last_active":"2024-09-10T18:36:32.858216+0000","last_peered":"2024-09-10T18:36:32.858216+0000","last_clean":"2024-09-10T18:36:32.858216+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:36:32.858216+0000","last_undegraded":"2024-09-10T18:36:32.858216+0000","last_fullsized":"2024-09-10T18:36:32.858216+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:36:17.526667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:17.526667+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:45.386424+0000","scrub_duration":0.0019245169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":347,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.859287+0000","last_change":"2024-09-10T18:35:13.892354+0000","last_active":"2024-09-10T18:36:32.859287+0000","last_peered":"2024-09-10T18:36:32.859287+0000","last_clean":"2024-09-10T18:36:32.859287+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:36:32.859287+0000","last_undegraded":"2024-09-10T18:36:32.859287+0000","last_fullsized":"2024-09-10T18:36:32.859287+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:13.892271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:13.892271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:42.058900+0000","scrub_duration":0.0058623920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'81","reported_seq":273,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.853458+0000","last_change":"2024-09-10T18:36:06.623015+0000","last_active":"2024-09-10T18:36:32.853458+0000","last_peered":"2024-09-10T18:36:32.853458+0000","last_clean":"2024-09-10T18:36:32.853458+0000","last_became_active":"2024-09-10T18:34:34.855693+0000","last_became_peered":"2024-09-10T18:34:34.855693+0000","last_unstale":"2024-09-10T18:36:32.853458+0000","last_undegraded":"2024-09-10T18:36:32.853458+0000","last_fullsized":"2024-09-10T18:36:32.853458+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:06.622956+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.622956+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:16.207315+0000","scrub_duration":0.0017570750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":334,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855565+0000","last_change":"2024-09-10T18:34:21.344309+0000","last_active":"2024-09-10T18:36:32.855565+0000","last_peered":"2024-09-10T18:36:32.855565+0000","last_clean":"2024-09-10T18:36:32.855565+0000","last_became_active":"2024-09-10T18:26:27.640035+0000","last_became_peered":"2024-09-10T18:26:27.640035+0000","last_unstale":"2024-09-10T18:36:32.855565+0000","last_undegraded":"2024-09-10T18:36:32.855565+0000","last_fullsized":"2024-09-10T18:36:32.855565+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:21.344236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.344236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040101240000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":340,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857584+0000","last_change":"2024-09-10T18:35:04.319486+0000","last_active":"2024-09-10T18:36:32.857584+0000","last_peered":"2024-09-10T18:36:32.857584+0000","last_clean":"2024-09-10T18:36:32.857584+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:36:32.857584+0000","last_undegraded":"2024-09-10T18:36:32.857584+0000","last_fullsized":"2024-09-10T18:36:32.857584+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:04.319395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:35:04.319395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002865018,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'23","reported_seq":368,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855433+0000","last_change":"2024-09-10T18:35:14.739218+0000","last_active":"2024-09-10T18:36:32.855433+0000","last_peered":"2024-09-10T18:36:32.855433+0000","last_clean":"2024-09-10T18:36:32.855433+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:36:32.855433+0000","last_undegraded":"2024-09-10T18:36:32.855433+0000","last_fullsized":"2024-09-10T18:36:32.855433+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'23","last_scrub_stamp":"2024-09-10T18:35:14.739123+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:35:14.739123+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052949870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":336,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857085+0000","last_change":"2024-09-10T18:35:08.266372+0000","last_active":"2024-09-10T18:36:32.857085+0000","last_peered":"2024-09-10T18:36:32.857085+0000","last_clean":"2024-09-10T18:36:32.857085+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:36:32.857085+0000","last_undegraded":"2024-09-10T18:36:32.857085+0000","last_fullsized":"2024-09-10T18:36:32.857085+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:08.266279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:08.266279+0000","last_clean_scrub_stamp":"2024-09-10T18:35:08.266279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041938189999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'81","reported_seq":261,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.852664+0000","last_change":"2024-09-10T18:36:07.673249+0000","last_active":"2024-09-10T18:36:32.852664+0000","last_peered":"2024-09-10T18:36:32.852664+0000","last_clean":"2024-09-10T18:36:32.852664+0000","last_became_active":"2024-09-10T18:34:34.855038+0000","last_became_peered":"2024-09-10T18:34:34.855038+0000","last_unstale":"2024-09-10T18:36:32.852664+0000","last_undegraded":"2024-09-10T18:36:32.852664+0000","last_fullsized":"2024-09-10T18:36:32.852664+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:07.673176+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.673176+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:33.711087+0000","scrub_duration":0.0018195469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":336,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855561+0000","last_change":"2024-09-10T18:35:20.905675+0000","last_active":"2024-09-10T18:36:32.855561+0000","last_peered":"2024-09-10T18:36:32.855561+0000","last_clean":"2024-09-10T18:36:32.855561+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:36:32.855561+0000","last_undegraded":"2024-09-10T18:36:32.855561+0000","last_fullsized":"2024-09-10T18:36:32.855561+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:35:20.905576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:20.905576+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049065819999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":351,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.858530+0000","last_change":"2024-09-10T18:35:09.892149+0000","last_active":"2024-09-10T18:36:32.858530+0000","last_peered":"2024-09-10T18:36:32.858530+0000","last_clean":"2024-09-10T18:36:32.858530+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:36:32.858530+0000","last_undegraded":"2024-09-10T18:36:32.858530+0000","last_fullsized":"2024-09-10T18:36:32.858530+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:09.892050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:35:09.892050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013937018000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'25","reported_seq":375,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.867518+0000","last_change":"2024-09-10T18:35:15.783820+0000","last_active":"2024-09-10T18:36:32.867518+0000","last_peered":"2024-09-10T18:36:32.867518+0000","last_clean":"2024-09-10T18:36:32.867518+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:36:32.867518+0000","last_undegraded":"2024-09-10T18:36:32.867518+0000","last_fullsized":"2024-09-10T18:36:32.867518+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:35:15.783731+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:35:15.783731+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047918379999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"217'11","reported_seq":450,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:36.147066+0000","last_change":"2024-09-10T18:36:11.854548+0000","last_active":"2024-09-10T18:36:36.147066+0000","last_peered":"2024-09-10T18:36:36.147066+0000","last_clean":"2024-09-10T18:36:36.147066+0000","last_became_active":"2024-09-10T18:26:27.652489+0000","last_became_peered":"2024-09-10T18:26:27.652489+0000","last_unstale":"2024-09-10T18:36:36.147066+0000","last_undegraded":"2024-09-10T18:36:36.147066+0000","last_fullsized":"2024-09-10T18:36:36.147066+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11","last_scrub_stamp":"2024-09-10T18:36:11.854492+0000","last_deep_scrub":"217'11","last_deep_scrub_stamp":"2024-09-10T18:30:03.161985+0000","last_clean_scrub_stamp":"2024-09-10T18:36:11.854492+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:39.199604+0000","scrub_duration":0.0014027550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"322'2996","reported_seq":3012,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:39.869633+0000","last_change":"2024-09-10T18:36:07.904585+0000","last_active":"2024-09-10T18:36:39.869633+0000","last_peered":"2024-09-10T18:36:39.869633+0000","last_clean":"2024-09-10T18:36:39.869633+0000","last_became_active":"2024-09-10T18:36:07.904319+0000","last_became_peered":"2024-09-10T18:36:07.904319+0000","last_unstale":"2024-09-10T18:36:39.869633+0000","last_undegraded":"2024-09-10T18:36:39.869633+0000","last_fullsized":"2024-09-10T18:36:39.869633+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":2996,"log_dups_size":0,"ondisk_log_size":2996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:18.814945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196345856,"num_objects":2996,"num_object_clones":0,"num_object_copies":8988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":191744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":267,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.852534+0000","last_change":"2024-09-10T18:35:59.488083+0000","last_active":"2024-09-10T18:36:32.852534+0000","last_peered":"2024-09-10T18:36:32.852534+0000","last_clean":"2024-09-10T18:36:32.852534+0000","last_became_active":"2024-09-10T18:34:32.823230+0000","last_became_peered":"2024-09-10T18:34:32.823230+0000","last_unstale":"2024-09-10T18:36:32.852534+0000","last_undegraded":"2024-09-10T18:36:32.852534+0000","last_fullsized":"2024-09-10T18:36:32.852534+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:59.488014+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:59.488014+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:22.512352+0000","scrub_duration":0.0017684759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"240'34","reported_seq":402,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.859153+0000","last_change":"2024-09-10T18:35:11.834455+0000","last_active":"2024-09-10T18:36:32.859153+0000","last_peered":"2024-09-10T18:36:32.859153+0000","last_clean":"2024-09-10T18:36:32.859153+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:36:32.859153+0000","last_undegraded":"2024-09-10T18:36:32.859153+0000","last_fullsized":"2024-09-10T18:36:32.859153+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:35:11.834362+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:35:11.834362+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040598810000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":333,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.854659+0000","last_change":"2024-09-10T18:34:20.305626+0000","last_active":"2024-09-10T18:36:32.854659+0000","last_peered":"2024-09-10T18:36:32.854659+0000","last_clean":"2024-09-10T18:36:32.854659+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:36:32.854659+0000","last_undegraded":"2024-09-10T18:36:32.854659+0000","last_fullsized":"2024-09-10T18:36:32.854659+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:20.305547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:20.305547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046035989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":518,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:34.940261+0000","last_change":"2024-09-10T18:34:14.495568+0000","last_active":"2024-09-10T18:36:34.940261+0000","last_peered":"2024-09-10T18:36:34.940261+0000","last_clean":"2024-09-10T18:36:34.940261+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:36:34.940261+0000","last_undegraded":"2024-09-10T18:36:34.940261+0000","last_fullsized":"2024-09-10T18:36:34.940261+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:34:14.495486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:14.495486+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010908085,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":335,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.852673+0000","last_change":"2024-09-10T18:34:57.526652+0000","last_active":"2024-09-10T18:36:32.852673+0000","last_peered":"2024-09-10T18:36:32.852673+0000","last_clean":"2024-09-10T18:36:32.852673+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:36:32.852673+0000","last_undegraded":"2024-09-10T18:36:32.852673+0000","last_fullsized":"2024-09-10T18:36:32.852673+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:57.526512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:34:57.526512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049647529999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"322'3031","reported_seq":3047,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:39.852392+0000","last_change":"2024-09-10T18:36:07.900391+0000","last_active":"2024-09-10T18:36:39.852392+0000","last_peered":"2024-09-10T18:36:39.852392+0000","last_clean":"2024-09-10T18:36:39.852392+0000","last_became_active":"2024-09-10T18:36:07.900059+0000","last_became_peered":"2024-09-10T18:36:07.900059+0000","last_unstale":"2024-09-10T18:36:39.852392+0000","last_undegraded":"2024-09-10T18:36:39.852392+0000","last_fullsized":"2024-09-10T18:36:39.852392+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":3031,"log_dups_size":0,"ondisk_log_size":3031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:24.159285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198639616,"num_objects":3031,"num_object_clones":0,"num_object_copies":9093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3031,"num_write_kb":193984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":276,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857622+0000","last_change":"2024-09-10T18:35:56.307718+0000","last_active":"2024-09-10T18:36:32.857622+0000","last_peered":"2024-09-10T18:36:32.857622+0000","last_clean":"2024-09-10T18:36:32.857622+0000","last_became_active":"2024-09-10T18:34:32.815839+0000","last_became_peered":"2024-09-10T18:34:32.815839+0000","last_unstale":"2024-09-10T18:36:32.857622+0000","last_undegraded":"2024-09-10T18:36:32.857622+0000","last_fullsized":"2024-09-10T18:36:32.857622+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:56.307649+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:56.307649+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:20.644586+0000","scrub_duration":0.0015720840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":381,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.854455+0000","last_change":"2024-09-10T18:36:07.782175+0000","last_active":"2024-09-10T18:36:32.854455+0000","last_peered":"2024-09-10T18:36:32.854455+0000","last_clean":"2024-09-10T18:36:32.854455+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:36:32.854455+0000","last_undegraded":"2024-09-10T18:36:32.854455+0000","last_fullsized":"2024-09-10T18:36:32.854455+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-09-10T18:36:07.782112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.782112+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:16.926976+0000","scrub_duration":0.00244122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":333,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.856263+0000","last_change":"2024-09-10T18:35:15.765590+0000","last_active":"2024-09-10T18:36:32.856263+0000","last_peered":"2024-09-10T18:36:32.856263+0000","last_clean":"2024-09-10T18:36:32.856263+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:36:32.856263+0000","last_undegraded":"2024-09-10T18:36:32.856263+0000","last_fullsized":"2024-09-10T18:36:32.856263+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:15.765392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:15.765392+0000","last_clean_scrub_stamp":"2024-09-10T18:35:15.765392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:37.786212+0000","scrub_duration":0.01380637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":336,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.856161+0000","last_change":"2024-09-10T18:35:10.270036+0000","last_active":"2024-09-10T18:36:32.856161+0000","last_peered":"2024-09-10T18:36:32.856161+0000","last_clean":"2024-09-10T18:36:32.856161+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:36:32.856161+0000","last_undegraded":"2024-09-10T18:36:32.856161+0000","last_fullsized":"2024-09-10T18:36:32.856161+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:10.269951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:35:10.269951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073211559999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":324,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857418+0000","last_change":"2024-09-10T18:34:21.322560+0000","last_active":"2024-09-10T18:36:32.857418+0000","last_peered":"2024-09-10T18:36:32.857418+0000","last_clean":"2024-09-10T18:36:32.857418+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:36:32.857418+0000","last_undegraded":"2024-09-10T18:36:32.857418+0000","last_fullsized":"2024-09-10T18:36:32.857418+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:21.322454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.322454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032746329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"322'3202","reported_seq":3218,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:40.928482+0000","last_change":"2024-09-10T18:36:07.902153+0000","last_active":"2024-09-10T18:36:40.928482+0000","last_peered":"2024-09-10T18:36:40.928482+0000","last_clean":"2024-09-10T18:36:40.928482+0000","last_became_active":"2024-09-10T18:36:07.901822+0000","last_became_peered":"2024-09-10T18:36:07.901822+0000","last_unstale":"2024-09-10T18:36:40.928482+0000","last_undegraded":"2024-09-10T18:36:40.928482+0000","last_fullsized":"2024-09-10T18:36:40.928482+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":3202,"log_dups_size":0,"ondisk_log_size":3202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:27.350431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209846272,"num_objects":3202,"num_object_clones":0,"num_object_copies":9606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3202,"num_write_kb":204928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":248,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.854410+0000","last_change":"2024-09-10T18:35:13.749157+0000","last_active":"2024-09-10T18:36:32.854410+0000","last_peered":"2024-09-10T18:36:32.854410+0000","last_clean":"2024-09-10T18:36:32.854410+0000","last_became_active":"2024-09-10T18:34:00.166362+0000","last_became_peered":"2024-09-10T18:34:00.166362+0000","last_unstale":"2024-09-10T18:36:32.854410+0000","last_undegraded":"2024-09-10T18:36:32.854410+0000","last_fullsized":"2024-09-10T18:36:32.854410+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:13.748963+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:13.748963+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013370432999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"240'28","reported_seq":387,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.856357+0000","last_change":"2024-09-10T18:36:16.155412+0000","last_active":"2024-09-10T18:36:32.856357+0000","last_peered":"2024-09-10T18:36:32.856357+0000","last_clean":"2024-09-10T18:36:32.856357+0000","last_became_active":"2024-09-10T18:26:27.651364+0000","last_became_peered":"2024-09-10T18:26:27.651364+0000","last_unstale":"2024-09-10T18:36:32.856357+0000","last_undegraded":"2024-09-10T18:36:32.856357+0000","last_fullsized":"2024-09-10T18:36:32.856357+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'28","last_scrub_stamp":"2024-09-10T18:36:16.155287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:16.155287+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:31.419623+0000","scrub_duration":0.0027352050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":342,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.868280+0000","last_change":"2024-09-10T18:36:09.716889+0000","last_active":"2024-09-10T18:36:32.868280+0000","last_peered":"2024-09-10T18:36:32.868280+0000","last_clean":"2024-09-10T18:36:32.868280+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:36:32.868280+0000","last_undegraded":"2024-09-10T18:36:32.868280+0000","last_fullsized":"2024-09-10T18:36:32.868280+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:09.716820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:36:09.716820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:34.441003+0000","scrub_duration":0.0037024689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":520,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:36.145225+0000","last_change":"2024-09-10T18:36:13.682715+0000","last_active":"2024-09-10T18:36:36.145225+0000","last_peered":"2024-09-10T18:36:36.145225+0000","last_clean":"2024-09-10T18:36:36.145225+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:36:36.145225+0000","last_undegraded":"2024-09-10T18:36:36.145225+0000","last_fullsized":"2024-09-10T18:36:36.145225+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:36:13.682649+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:36:13.682649+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:30.845820+0000","scrub_duration":0.001699954,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":334,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855156+0000","last_change":"2024-09-10T18:33:11.857892+0000","last_active":"2024-09-10T18:36:32.855156+0000","last_peered":"2024-09-10T18:36:32.855156+0000","last_clean":"2024-09-10T18:36:32.855156+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:36:32.855156+0000","last_undegraded":"2024-09-10T18:36:32.855156+0000","last_fullsized":"2024-09-10T18:36:32.855156+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:11.857685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.857685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022995870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"322'3151","reported_seq":3167,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:40.033780+0000","last_change":"2024-09-10T18:36:07.904923+0000","last_active":"2024-09-10T18:36:40.033780+0000","last_peered":"2024-09-10T18:36:40.033780+0000","last_clean":"2024-09-10T18:36:40.033780+0000","last_became_active":"2024-09-10T18:36:07.904606+0000","last_became_peered":"2024-09-10T18:36:07.904606+0000","last_unstale":"2024-09-10T18:36:40.033780+0000","last_undegraded":"2024-09-10T18:36:40.033780+0000","last_fullsized":"2024-09-10T18:36:40.033780+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":3151,"log_dups_size":0,"ondisk_log_size":3151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:26.465282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206503936,"num_objects":3151,"num_object_clones":0,"num_object_copies":9453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3151,"num_write_kb":201664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":337,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.861200+0000","last_change":"2024-09-10T18:36:09.909074+0000","last_active":"2024-09-10T18:36:32.861200+0000","last_peered":"2024-09-10T18:36:32.861200+0000","last_clean":"2024-09-10T18:36:32.861200+0000","last_became_active":"2024-09-10T18:34:32.818323+0000","last_became_peered":"2024-09-10T18:34:32.818323+0000","last_unstale":"2024-09-10T18:36:32.861200+0000","last_undegraded":"2024-09-10T18:36:32.861200+0000","last_fullsized":"2024-09-10T18:36:32.861200+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:09.908960+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:09.908960+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:25.472837+0000","scrub_duration":0.0019456180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":384,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.860039+0000","last_change":"2024-09-10T18:34:59.896028+0000","last_active":"2024-09-10T18:36:32.860039+0000","last_peered":"2024-09-10T18:36:32.860039+0000","last_clean":"2024-09-10T18:36:32.860039+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:36:32.860039+0000","last_undegraded":"2024-09-10T18:36:32.860039+0000","last_fullsized":"2024-09-10T18:36:32.860039+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:34:59.895743+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:34:59.895743+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031796210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":331,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.853222+0000","last_change":"2024-09-10T18:34:56.510382+0000","last_active":"2024-09-10T18:36:32.853222+0000","last_peered":"2024-09-10T18:36:32.853222+0000","last_clean":"2024-09-10T18:36:32.853222+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:36:32.853222+0000","last_undegraded":"2024-09-10T18:36:32.853222+0000","last_fullsized":"2024-09-10T18:36:32.853222+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:56.510308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:34:56.510308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059794410000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"263'11","reported_seq":527,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:34.940471+0000","last_change":"2024-09-10T18:35:17.801582+0000","last_active":"2024-09-10T18:36:34.940471+0000","last_peered":"2024-09-10T18:36:34.940471+0000","last_clean":"2024-09-10T18:36:34.940471+0000","last_became_active":"2024-09-10T18:26:27.639032+0000","last_became_peered":"2024-09-10T18:26:27.639032+0000","last_unstale":"2024-09-10T18:36:34.940471+0000","last_undegraded":"2024-09-10T18:36:34.940471+0000","last_fullsized":"2024-09-10T18:36:34.940471+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'11","last_scrub_stamp":"2024-09-10T18:35:17.801395+0000","last_deep_scrub":"263'11","last_deep_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_clean_scrub_stamp":"2024-09-10T18:35:17.801395+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0098849520000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":336,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.866222+0000","last_change":"2024-09-10T18:35:55.632357+0000","last_active":"2024-09-10T18:36:32.866222+0000","last_peered":"2024-09-10T18:36:32.866222+0000","last_clean":"2024-09-10T18:36:32.866222+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:36:32.866222+0000","last_undegraded":"2024-09-10T18:36:32.866222+0000","last_fullsized":"2024-09-10T18:36:32.866222+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:55.632306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:55.632306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:24.932351+0000","scrub_duration":0.001072857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"322'3286","reported_seq":3302,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:40.909348+0000","last_change":"2024-09-10T18:36:07.903925+0000","last_active":"2024-09-10T18:36:40.909348+0000","last_peered":"2024-09-10T18:36:40.909348+0000","last_clean":"2024-09-10T18:36:40.909348+0000","last_became_active":"2024-09-10T18:36:07.903637+0000","last_became_peered":"2024-09-10T18:36:07.903637+0000","last_unstale":"2024-09-10T18:36:40.909348+0000","last_undegraded":"2024-09-10T18:36:40.909348+0000","last_fullsized":"2024-09-10T18:36:40.909348+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":3286,"log_dups_size":0,"ondisk_log_size":3286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:09.143847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215351296,"num_objects":3286,"num_object_clones":0,"num_object_copies":9858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3286,"num_write_kb":210304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":337,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.860390+0000","last_change":"2024-09-10T18:36:08.872676+0000","last_active":"2024-09-10T18:36:32.860390+0000","last_peered":"2024-09-10T18:36:32.860390+0000","last_clean":"2024-09-10T18:36:32.860390+0000","last_became_active":"2024-09-10T18:34:32.918946+0000","last_became_peered":"2024-09-10T18:34:32.918946+0000","last_unstale":"2024-09-10T18:36:32.860390+0000","last_undegraded":"2024-09-10T18:36:32.860390+0000","last_fullsized":"2024-09-10T18:36:32.860390+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:08.872575+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:08.872575+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:31.449589+0000","scrub_duration":0.0016711530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"240'29","reported_seq":391,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.866887+0000","last_change":"2024-09-10T18:35:10.711922+0000","last_active":"2024-09-10T18:36:32.866887+0000","last_peered":"2024-09-10T18:36:32.866887+0000","last_clean":"2024-09-10T18:36:32.866887+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:36:32.866887+0000","last_undegraded":"2024-09-10T18:36:32.866887+0000","last_fullsized":"2024-09-10T18:36:32.866887+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'29","last_scrub_stamp":"2024-09-10T18:35:10.711649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:10.711649+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013292415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":341,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.858914+0000","last_change":"2024-09-10T18:36:06.595223+0000","last_active":"2024-09-10T18:36:32.858914+0000","last_peered":"2024-09-10T18:36:32.858914+0000","last_clean":"2024-09-10T18:36:32.858914+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:36:32.858914+0000","last_undegraded":"2024-09-10T18:36:32.858914+0000","last_fullsized":"2024-09-10T18:36:32.858914+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.595159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.595159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:34.639380+0000","scrub_duration":0.0014987100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":297,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.859991+0000","last_change":"2024-09-10T18:36:05.846864+0000","last_active":"2024-09-10T18:36:32.859991+0000","last_peered":"2024-09-10T18:36:32.859991+0000","last_clean":"2024-09-10T18:36:32.859991+0000","last_became_active":"2024-09-10T18:26:27.642063+0000","last_became_peered":"2024-09-10T18:26:27.642063+0000","last_unstale":"2024-09-10T18:36:32.859991+0000","last_undegraded":"2024-09-10T18:36:32.859991+0000","last_fullsized":"2024-09-10T18:36:32.859991+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:05.846805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:05.846805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:10.917223+0000","scrub_duration":0.0013376390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":342,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.858306+0000","last_change":"2024-09-10T18:35:14.848040+0000","last_active":"2024-09-10T18:36:32.858306+0000","last_peered":"2024-09-10T18:36:32.858306+0000","last_clean":"2024-09-10T18:36:32.858306+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:36:32.858306+0000","last_undegraded":"2024-09-10T18:36:32.858306+0000","last_fullsized":"2024-09-10T18:36:32.858306+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:14.847962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:35:14.847962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053623120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"322'3144","reported_seq":3160,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:40.928651+0000","last_change":"2024-09-10T18:36:07.903723+0000","last_active":"2024-09-10T18:36:40.928651+0000","last_peered":"2024-09-10T18:36:40.928651+0000","last_clean":"2024-09-10T18:36:40.928651+0000","last_became_active":"2024-09-10T18:36:07.903339+0000","last_became_peered":"2024-09-10T18:36:07.903339+0000","last_unstale":"2024-09-10T18:36:40.928651+0000","last_undegraded":"2024-09-10T18:36:40.928651+0000","last_fullsized":"2024-09-10T18:36:40.928651+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":3144,"log_dups_size":0,"ondisk_log_size":3144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:25.034412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206045184,"num_objects":3144,"num_object_clones":0,"num_object_copies":9432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3144,"num_write_kb":201216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":242,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.852800+0000","last_change":"2024-09-10T18:34:59.440417+0000","last_active":"2024-09-10T18:36:32.852800+0000","last_peered":"2024-09-10T18:36:32.852800+0000","last_clean":"2024-09-10T18:36:32.852800+0000","last_became_active":"2024-09-10T18:34:00.151017+0000","last_became_peered":"2024-09-10T18:34:00.151017+0000","last_unstale":"2024-09-10T18:36:32.852800+0000","last_undegraded":"2024-09-10T18:36:32.852800+0000","last_fullsized":"2024-09-10T18:36:32.852800+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:34:59.440207+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:34:59.440207+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048255449999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"318'42","reported_seq":42,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:36.138479+0000","last_change":"2024-09-10T18:36:12.932521+0000","last_active":"2024-09-10T18:36:36.138479+0000","last_peered":"2024-09-10T18:36:36.138479+0000","last_clean":"2024-09-10T18:36:36.138479+0000","last_became_active":"2024-09-10T18:36:12.922665+0000","last_became_peered":"2024-09-10T18:36:12.922665+0000","last_unstale":"2024-09-10T18:36:36.138479+0000","last_undegraded":"2024-09-10T18:36:36.138479+0000","last_fullsized":"2024-09-10T18:36:36.138479+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'41","last_scrub_stamp":"2024-09-10T18:35:07.535910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:07.535910+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0098705089999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":337,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857072+0000","last_change":"2024-09-10T18:34:17.387421+0000","last_active":"2024-09-10T18:36:32.857072+0000","last_peered":"2024-09-10T18:36:32.857072+0000","last_clean":"2024-09-10T18:36:32.857072+0000","last_became_active":"2024-09-10T18:26:27.648690+0000","last_became_peered":"2024-09-10T18:26:27.648690+0000","last_unstale":"2024-09-10T18:36:32.857072+0000","last_undegraded":"2024-09-10T18:36:32.857072+0000","last_fullsized":"2024-09-10T18:36:32.857072+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:17.387344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:17.387344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051904949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":356,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.853079+0000","last_change":"2024-09-10T18:35:55.505988+0000","last_active":"2024-09-10T18:36:32.853079+0000","last_peered":"2024-09-10T18:36:32.853079+0000","last_clean":"2024-09-10T18:36:32.853079+0000","last_became_active":"2024-09-10T18:26:27.647576+0000","last_became_peered":"2024-09-10T18:26:27.647576+0000","last_unstale":"2024-09-10T18:36:32.853079+0000","last_undegraded":"2024-09-10T18:36:32.853079+0000","last_fullsized":"2024-09-10T18:36:32.853079+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:35:55.505916+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:35:55.505916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:12.516837+0000","scrub_duration":0.001708209,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":338,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.866898+0000","last_change":"2024-09-10T18:36:02.734328+0000","last_active":"2024-09-10T18:36:32.866898+0000","last_peered":"2024-09-10T18:36:32.866898+0000","last_clean":"2024-09-10T18:36:32.866898+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:36:32.866898+0000","last_undegraded":"2024-09-10T18:36:32.866898+0000","last_fullsized":"2024-09-10T18:36:32.866898+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:02.734269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:02.734269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:23.649134+0000","scrub_duration":0.0016180000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":317,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855265+0000","last_change":"2024-09-10T18:33:56.168567+0000","last_active":"2024-09-10T18:36:32.855265+0000","last_peered":"2024-09-10T18:36:32.855265+0000","last_clean":"2024-09-10T18:36:32.855265+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:36:32.855265+0000","last_undegraded":"2024-09-10T18:36:32.855265+0000","last_fullsized":"2024-09-10T18:36:32.855265+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:56.168485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:56.168485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002436723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"322'3064","reported_seq":3080,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:40.054915+0000","last_change":"2024-09-10T18:36:07.906590+0000","last_active":"2024-09-10T18:36:40.054915+0000","last_peered":"2024-09-10T18:36:40.054915+0000","last_clean":"2024-09-10T18:36:40.054915+0000","last_became_active":"2024-09-10T18:36:07.906264+0000","last_became_peered":"2024-09-10T18:36:07.906264+0000","last_unstale":"2024-09-10T18:36:40.054915+0000","last_undegraded":"2024-09-10T18:36:40.054915+0000","last_fullsized":"2024-09-10T18:36:40.054915+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":3064,"log_dups_size":0,"ondisk_log_size":3064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:13.087655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200802304,"num_objects":3064,"num_object_clones":0,"num_object_copies":9192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3064,"num_write_kb":196096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":388,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857135+0000","last_change":"2024-09-10T18:35:08.748174+0000","last_active":"2024-09-10T18:36:32.857135+0000","last_peered":"2024-09-10T18:36:32.857135+0000","last_clean":"2024-09-10T18:36:32.857135+0000","last_became_active":"2024-09-10T18:34:00.154404+0000","last_became_peered":"2024-09-10T18:34:00.154404+0000","last_unstale":"2024-09-10T18:36:32.857135+0000","last_undegraded":"2024-09-10T18:36:32.857135+0000","last_fullsized":"2024-09-10T18:36:32.857135+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:08.748082+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:08.748082+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038002869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"240'17","reported_seq":372,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.859230+0000","last_change":"2024-09-10T18:34:13.522168+0000","last_active":"2024-09-10T18:36:32.859230+0000","last_peered":"2024-09-10T18:36:32.859230+0000","last_clean":"2024-09-10T18:36:32.859230+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:36:32.859230+0000","last_undegraded":"2024-09-10T18:36:32.859230+0000","last_fullsized":"2024-09-10T18:36:32.859230+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'17","last_scrub_stamp":"2024-09-10T18:34:13.521944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:13.521944+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029448339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":358,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.858936+0000","last_change":"2024-09-10T18:35:57.626741+0000","last_active":"2024-09-10T18:36:32.858936+0000","last_peered":"2024-09-10T18:36:32.858936+0000","last_clean":"2024-09-10T18:36:32.858936+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:36:32.858936+0000","last_undegraded":"2024-09-10T18:36:32.858936+0000","last_fullsized":"2024-09-10T18:36:32.858936+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:57.626681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:15.141982+0000","last_clean_scrub_stamp":"2024-09-10T18:35:57.626681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:13.924653+0000","scrub_duration":0.001556663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":369,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.859014+0000","last_change":"2024-09-10T18:35:55.625472+0000","last_active":"2024-09-10T18:36:32.859014+0000","last_peered":"2024-09-10T18:36:32.859014+0000","last_clean":"2024-09-10T18:36:32.859014+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:36:32.859014+0000","last_undegraded":"2024-09-10T18:36:32.859014+0000","last_fullsized":"2024-09-10T18:36:32.859014+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:35:55.625412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:35:55.625412+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:03.806756+0000","scrub_duration":0.0014018520000000001,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":346,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.860139+0000","last_change":"2024-09-10T18:34:59.713703+0000","last_active":"2024-09-10T18:36:32.860139+0000","last_peered":"2024-09-10T18:36:32.860139+0000","last_clean":"2024-09-10T18:36:32.860139+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:36:32.860139+0000","last_undegraded":"2024-09-10T18:36:32.860139+0000","last_fullsized":"2024-09-10T18:36:32.860139+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:59.713536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:34:59.713536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002321769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":334,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855715+0000","last_change":"2024-09-10T18:34:51.933409+0000","last_active":"2024-09-10T18:36:32.855715+0000","last_peered":"2024-09-10T18:36:32.855715+0000","last_clean":"2024-09-10T18:36:32.855715+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:36:32.855715+0000","last_undegraded":"2024-09-10T18:36:32.855715+0000","last_fullsized":"2024-09-10T18:36:32.855715+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:51.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:51.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029402349999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"321'2668","reported_seq":2684,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:36.150608+0000","last_change":"2024-09-10T18:36:07.902359+0000","last_active":"2024-09-10T18:36:36.150608+0000","last_peered":"2024-09-10T18:36:36.150608+0000","last_clean":"2024-09-10T18:36:36.150608+0000","last_became_active":"2024-09-10T18:36:07.901516+0000","last_became_peered":"2024-09-10T18:36:07.901516+0000","last_unstale":"2024-09-10T18:36:36.150608+0000","last_undegraded":"2024-09-10T18:36:36.150608+0000","last_fullsized":"2024-09-10T18:36:36.150608+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":2668,"log_dups_size":0,"ondisk_log_size":2668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:17.319621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174850048,"num_objects":2668,"num_object_clones":0,"num_object_copies":8004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2668,"num_write_kb":170752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":258,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.858575+0000","last_change":"2024-09-10T18:35:01.969623+0000","last_active":"2024-09-10T18:36:32.858575+0000","last_peered":"2024-09-10T18:36:32.858575+0000","last_clean":"2024-09-10T18:36:32.858575+0000","last_became_active":"2024-09-10T18:34:00.165384+0000","last_became_peered":"2024-09-10T18:34:00.165384+0000","last_unstale":"2024-09-10T18:36:32.858575+0000","last_undegraded":"2024-09-10T18:36:32.858575+0000","last_fullsized":"2024-09-10T18:36:32.858575+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:01.969528+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:01.969528+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.094075444999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":376,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.867613+0000","last_change":"2024-09-10T18:36:15.636499+0000","last_active":"2024-09-10T18:36:32.867613+0000","last_peered":"2024-09-10T18:36:32.867613+0000","last_clean":"2024-09-10T18:36:32.867613+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:36:32.867613+0000","last_undegraded":"2024-09-10T18:36:32.867613+0000","last_fullsized":"2024-09-10T18:36:32.867613+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-09-10T18:36:15.636442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:15.636442+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:35.680984+0000","scrub_duration":0.0017190630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":342,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.868350+0000","last_change":"2024-09-10T18:35:02.838571+0000","last_active":"2024-09-10T18:36:32.868350+0000","last_peered":"2024-09-10T18:36:32.868350+0000","last_clean":"2024-09-10T18:36:32.868350+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:36:32.868350+0000","last_undegraded":"2024-09-10T18:36:32.868350+0000","last_fullsized":"2024-09-10T18:36:32.868350+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:02.838489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:35:02.838489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090996949999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":1249,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:38.998745+0000","last_change":"2024-09-10T18:34:02.136505+0000","last_active":"2024-09-10T18:36:38.998745+0000","last_peered":"2024-09-10T18:36:38.998745+0000","last_clean":"2024-09-10T18:36:38.998745+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:36:38.998745+0000","last_undegraded":"2024-09-10T18:36:38.998745+0000","last_fullsized":"2024-09-10T18:36:38.998745+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:34:02.136429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:02.136429+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083841149999999993,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":901,"num_read_kb":896,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":330,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.854123+0000","last_change":"2024-09-10T18:35:12.568766+0000","last_active":"2024-09-10T18:36:32.854123+0000","last_peered":"2024-09-10T18:36:32.854123+0000","last_clean":"2024-09-10T18:36:32.854123+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:36:32.854123+0000","last_undegraded":"2024-09-10T18:36:32.854123+0000","last_fullsized":"2024-09-10T18:36:32.854123+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:12.568627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:05.771415+0000","last_clean_scrub_stamp":"2024-09-10T18:35:12.568627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022788050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":335,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.858539+0000","last_change":"2024-09-10T18:36:17.593856+0000","last_active":"2024-09-10T18:36:32.858539+0000","last_peered":"2024-09-10T18:36:32.858539+0000","last_clean":"2024-09-10T18:36:32.858539+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:36:32.858539+0000","last_undegraded":"2024-09-10T18:36:32.858539+0000","last_fullsized":"2024-09-10T18:36:32.858539+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:17.593802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:17.593802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:19.360284+0000","scrub_duration":0.0032348469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"322'3139","reported_seq":3155,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:40.919567+0000","last_change":"2024-09-10T18:36:07.903898+0000","last_active":"2024-09-10T18:36:40.919567+0000","last_peered":"2024-09-10T18:36:40.919567+0000","last_clean":"2024-09-10T18:36:40.919567+0000","last_became_active":"2024-09-10T18:36:07.903576+0000","last_became_peered":"2024-09-10T18:36:07.903576+0000","last_unstale":"2024-09-10T18:36:40.919567+0000","last_undegraded":"2024-09-10T18:36:40.919567+0000","last_fullsized":"2024-09-10T18:36:40.919567+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":3139,"log_dups_size":0,"ondisk_log_size":3139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:26.917343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205717504,"num_objects":3139,"num_object_clones":0,"num_object_copies":9417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3139,"num_write_kb":200896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"240'26","reported_seq":334,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.860444+0000","last_change":"2024-09-10T18:35:55.879357+0000","last_active":"2024-09-10T18:36:32.860444+0000","last_peered":"2024-09-10T18:36:32.860444+0000","last_clean":"2024-09-10T18:36:32.860444+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:36:32.860444+0000","last_undegraded":"2024-09-10T18:36:32.860444+0000","last_fullsized":"2024-09-10T18:36:32.860444+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'26","last_scrub_stamp":"2024-09-10T18:35:55.879297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:55.879297+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:16.597074+0000","scrub_duration":0.001846929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":340,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855132+0000","last_change":"2024-09-10T18:34:19.296515+0000","last_active":"2024-09-10T18:36:32.855132+0000","last_peered":"2024-09-10T18:36:32.855132+0000","last_clean":"2024-09-10T18:36:32.855132+0000","last_became_active":"2024-09-10T18:26:27.639129+0000","last_became_peered":"2024-09-10T18:26:27.639129+0000","last_unstale":"2024-09-10T18:36:32.855132+0000","last_undegraded":"2024-09-10T18:36:32.855132+0000","last_fullsized":"2024-09-10T18:36:32.855132+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:19.296286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.296286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049170170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":400,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857587+0000","last_change":"2024-09-10T18:35:05.724010+0000","last_active":"2024-09-10T18:36:32.857587+0000","last_peered":"2024-09-10T18:36:32.857587+0000","last_clean":"2024-09-10T18:36:32.857587+0000","last_became_active":"2024-09-10T18:34:00.159975+0000","last_became_peered":"2024-09-10T18:34:00.159975+0000","last_unstale":"2024-09-10T18:36:32.857587+0000","last_undegraded":"2024-09-10T18:36:32.857587+0000","last_fullsized":"2024-09-10T18:36:32.857587+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:05.723917+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:05.723917+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022326540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":336,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855699+0000","last_change":"2024-09-10T18:34:12.920724+0000","last_active":"2024-09-10T18:36:32.855699+0000","last_peered":"2024-09-10T18:36:32.855699+0000","last_clean":"2024-09-10T18:36:32.855699+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:36:32.855699+0000","last_undegraded":"2024-09-10T18:36:32.855699+0000","last_fullsized":"2024-09-10T18:36:32.855699+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:12.920651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:12.920651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047636090000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":328,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.854057+0000","last_change":"2024-09-10T18:35:03.488408+0000","last_active":"2024-09-10T18:36:32.854057+0000","last_peered":"2024-09-10T18:36:32.854057+0000","last_clean":"2024-09-10T18:36:32.854057+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:36:32.854057+0000","last_undegraded":"2024-09-10T18:36:32.854057+0000","last_fullsized":"2024-09-10T18:36:32.854057+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:03.488360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:35:03.488360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061095339999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"322'3180","reported_seq":3196,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:40.920507+0000","last_change":"2024-09-10T18:36:07.903096+0000","last_active":"2024-09-10T18:36:40.920507+0000","last_peered":"2024-09-10T18:36:40.920507+0000","last_clean":"2024-09-10T18:36:40.920507+0000","last_became_active":"2024-09-10T18:36:07.902779+0000","last_became_peered":"2024-09-10T18:36:07.902779+0000","last_unstale":"2024-09-10T18:36:40.920507+0000","last_undegraded":"2024-09-10T18:36:40.920507+0000","last_fullsized":"2024-09-10T18:36:40.920507+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":3180,"log_dups_size":0,"ondisk_log_size":3180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:30.765220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208404480,"num_objects":3180,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3180,"num_write_kb":203520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"76'81","reported_seq":277,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.858712+0000","last_change":"2024-09-10T18:35:15.894737+0000","last_active":"2024-09-10T18:36:32.858712+0000","last_peered":"2024-09-10T18:36:32.858712+0000","last_clean":"2024-09-10T18:36:32.858712+0000","last_became_active":"2024-09-10T18:34:32.927460+0000","last_became_peered":"2024-09-10T18:34:32.927460+0000","last_unstale":"2024-09-10T18:36:32.858712+0000","last_undegraded":"2024-09-10T18:36:32.858712+0000","last_fullsized":"2024-09-10T18:36:32.858712+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:15.894656+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:15.894656+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:40.273824+0000","scrub_duration":0.013574714,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"283'24","reported_seq":374,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.858423+0000","last_change":"2024-09-10T18:34:19.404796+0000","last_active":"2024-09-10T18:36:32.858423+0000","last_peered":"2024-09-10T18:36:32.858423+0000","last_clean":"2024-09-10T18:36:32.858423+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:36:32.858423+0000","last_undegraded":"2024-09-10T18:36:32.858423+0000","last_fullsized":"2024-09-10T18:36:32.858423+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'24","last_scrub_stamp":"2024-09-10T18:34:19.404713+0000","last_deep_scrub":"283'24","last_deep_scrub_stamp":"2024-09-10T18:34:19.404713+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.404713+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.037546765000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":330,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.856877+0000","last_change":"2024-09-10T18:34:20.349520+0000","last_active":"2024-09-10T18:36:32.856877+0000","last_peered":"2024-09-10T18:36:32.856877+0000","last_clean":"2024-09-10T18:36:32.856877+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:36:32.856877+0000","last_undegraded":"2024-09-10T18:36:32.856877+0000","last_fullsized":"2024-09-10T18:36:32.856877+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:20.349464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:20.349464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036650300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":344,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.853903+0000","last_change":"2024-09-10T18:35:08.562624+0000","last_active":"2024-09-10T18:36:32.853903+0000","last_peered":"2024-09-10T18:36:32.853903+0000","last_clean":"2024-09-10T18:36:32.853903+0000","last_became_active":"2024-09-10T18:26:27.645885+0000","last_became_peered":"2024-09-10T18:26:27.645885+0000","last_unstale":"2024-09-10T18:36:32.853903+0000","last_undegraded":"2024-09-10T18:36:32.853903+0000","last_fullsized":"2024-09-10T18:36:32.853903+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:08.562551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:35:08.562551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076930669999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":334,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.854500+0000","last_change":"2024-09-10T18:34:01.181010+0000","last_active":"2024-09-10T18:36:32.854500+0000","last_peered":"2024-09-10T18:36:32.854500+0000","last_clean":"2024-09-10T18:36:32.854500+0000","last_became_active":"2024-09-10T18:26:27.632071+0000","last_became_peered":"2024-09-10T18:26:27.632071+0000","last_unstale":"2024-09-10T18:36:32.854500+0000","last_undegraded":"2024-09-10T18:36:32.854500+0000","last_fullsized":"2024-09-10T18:36:32.854500+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:01.180920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:01.180920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082996130000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"322'3101","reported_seq":3117,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:40.120276+0000","last_change":"2024-09-10T18:36:07.903712+0000","last_active":"2024-09-10T18:36:40.120276+0000","last_peered":"2024-09-10T18:36:40.120276+0000","last_clean":"2024-09-10T18:36:40.120276+0000","last_became_active":"2024-09-10T18:36:07.903429+0000","last_became_peered":"2024-09-10T18:36:07.903429+0000","last_unstale":"2024-09-10T18:36:40.120276+0000","last_undegraded":"2024-09-10T18:36:40.120276+0000","last_fullsized":"2024-09-10T18:36:40.120276+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":3101,"log_dups_size":0,"ondisk_log_size":3101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:15.985317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203227136,"num_objects":3101,"num_object_clones":0,"num_object_copies":9303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3101,"num_write_kb":198464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":259,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.853318+0000","last_change":"2024-09-10T18:36:01.514457+0000","last_active":"2024-09-10T18:36:32.853318+0000","last_peered":"2024-09-10T18:36:32.853318+0000","last_clean":"2024-09-10T18:36:32.853318+0000","last_became_active":"2024-09-10T18:34:32.823976+0000","last_became_peered":"2024-09-10T18:34:32.823976+0000","last_unstale":"2024-09-10T18:36:32.853318+0000","last_undegraded":"2024-09-10T18:36:32.853318+0000","last_fullsized":"2024-09-10T18:36:32.853318+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:01.514369+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:01.514369+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:10.406688+0000","scrub_duration":0.001652136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":387,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857495+0000","last_change":"2024-09-10T18:36:02.364773+0000","last_active":"2024-09-10T18:36:32.857495+0000","last_peered":"2024-09-10T18:36:32.857495+0000","last_clean":"2024-09-10T18:36:32.857495+0000","last_became_active":"2024-09-10T18:21:33.611203+0000","last_became_peered":"2024-09-10T18:21:33.611203+0000","last_unstale":"2024-09-10T18:36:32.857495+0000","last_undegraded":"2024-09-10T18:36:32.857495+0000","last_fullsized":"2024-09-10T18:36:32.857495+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-09-10T18:36:02.364698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:02.364698+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:19.314989+0000","scrub_duration":0.0025656849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":305,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.860241+0000","last_change":"2024-09-10T18:36:00.897438+0000","last_active":"2024-09-10T18:36:32.860241+0000","last_peered":"2024-09-10T18:36:32.860241+0000","last_clean":"2024-09-10T18:36:32.860241+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:36:32.860241+0000","last_undegraded":"2024-09-10T18:36:32.860241+0000","last_fullsized":"2024-09-10T18:36:32.860241+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:36:00.897355+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:36:00.897355+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:26.619358+0000","scrub_duration":0.0020072419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":337,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.858615+0000","last_change":"2024-09-10T18:34:53.794746+0000","last_active":"2024-09-10T18:36:32.858615+0000","last_peered":"2024-09-10T18:36:32.858615+0000","last_clean":"2024-09-10T18:36:32.858615+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:36:32.858615+0000","last_undegraded":"2024-09-10T18:36:32.858615+0000","last_fullsized":"2024-09-10T18:36:32.858615+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:53.794668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:53.794668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026034389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":364,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.860000+0000","last_change":"2024-09-10T18:36:04.465752+0000","last_active":"2024-09-10T18:36:32.860000+0000","last_peered":"2024-09-10T18:36:32.860000+0000","last_clean":"2024-09-10T18:36:32.860000+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:36:32.860000+0000","last_undegraded":"2024-09-10T18:36:32.860000+0000","last_fullsized":"2024-09-10T18:36:32.860000+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:04.465694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:04.465694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:11.981327+0000","scrub_duration":0.0014277929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"322'3268","reported_seq":3284,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:40.530245+0000","last_change":"2024-09-10T18:36:07.904133+0000","last_active":"2024-09-10T18:36:40.530245+0000","last_peered":"2024-09-10T18:36:40.530245+0000","last_clean":"2024-09-10T18:36:40.530245+0000","last_became_active":"2024-09-10T18:36:07.903825+0000","last_became_peered":"2024-09-10T18:36:07.903825+0000","last_unstale":"2024-09-10T18:36:40.530245+0000","last_undegraded":"2024-09-10T18:36:40.530245+0000","last_fullsized":"2024-09-10T18:36:40.530245+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":3268,"log_dups_size":0,"ondisk_log_size":3268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:10.891232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214171648,"num_objects":3268,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3268,"num_write_kb":209152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":398,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857195+0000","last_change":"2024-09-10T18:36:13.951388+0000","last_active":"2024-09-10T18:36:32.857195+0000","last_peered":"2024-09-10T18:36:32.857195+0000","last_clean":"2024-09-10T18:36:32.857195+0000","last_became_active":"2024-09-10T18:34:32.827174+0000","last_became_peered":"2024-09-10T18:34:32.827174+0000","last_unstale":"2024-09-10T18:36:32.857195+0000","last_undegraded":"2024-09-10T18:36:32.857195+0000","last_fullsized":"2024-09-10T18:36:32.857195+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:13.951321+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:13.951321+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:19.549886+0000","scrub_duration":0.0015868600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"240'25","reported_seq":98,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855368+0000","last_change":"2024-09-10T18:35:04.715386+0000","last_active":"2024-09-10T18:36:32.855368+0000","last_peered":"2024-09-10T18:36:32.855368+0000","last_clean":"2024-09-10T18:36:32.855368+0000","last_became_active":"2024-09-10T18:32:51.118251+0000","last_became_peered":"2024-09-10T18:32:51.118251+0000","last_unstale":"2024-09-10T18:36:32.855368+0000","last_undegraded":"2024-09-10T18:36:32.855368+0000","last_fullsized":"2024-09-10T18:36:32.855368+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:35:04.715302+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:35:04.715302+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065951120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":333,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855895+0000","last_change":"2024-09-10T18:36:03.769108+0000","last_active":"2024-09-10T18:36:32.855895+0000","last_peered":"2024-09-10T18:36:32.855895+0000","last_clean":"2024-09-10T18:36:32.855895+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:36:32.855895+0000","last_undegraded":"2024-09-10T18:36:32.855895+0000","last_fullsized":"2024-09-10T18:36:32.855895+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:03.769053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:13.324556+0000","last_clean_scrub_stamp":"2024-09-10T18:36:03.769053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:29.323478+0000","scrub_duration":0.001666634,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"189'9","reported_seq":557,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:36.140980+0000","last_change":"2024-09-10T18:35:59.584228+0000","last_active":"2024-09-10T18:36:36.140980+0000","last_peered":"2024-09-10T18:36:36.140980+0000","last_clean":"2024-09-10T18:36:36.140980+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:36:36.140980+0000","last_undegraded":"2024-09-10T18:36:36.140980+0000","last_fullsized":"2024-09-10T18:36:36.140980+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'9","last_scrub_stamp":"2024-09-10T18:35:59.584165+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:35:59.584165+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:07.175142+0000","scrub_duration":0.001924006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":327,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.858040+0000","last_change":"2024-09-10T18:35:07.288756+0000","last_active":"2024-09-10T18:36:32.858040+0000","last_peered":"2024-09-10T18:36:32.858040+0000","last_clean":"2024-09-10T18:36:32.858040+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:36:32.858040+0000","last_undegraded":"2024-09-10T18:36:32.858040+0000","last_fullsized":"2024-09-10T18:36:32.858040+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:07.288672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_clean_scrub_stamp":"2024-09-10T18:35:07.288672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041335299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"322'3159","reported_seq":3175,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:40.125386+0000","last_change":"2024-09-10T18:36:07.904381+0000","last_active":"2024-09-10T18:36:40.125386+0000","last_peered":"2024-09-10T18:36:40.125386+0000","last_clean":"2024-09-10T18:36:40.125386+0000","last_became_active":"2024-09-10T18:36:07.903638+0000","last_became_peered":"2024-09-10T18:36:07.903638+0000","last_unstale":"2024-09-10T18:36:40.125386+0000","last_undegraded":"2024-09-10T18:36:40.125386+0000","last_fullsized":"2024-09-10T18:36:40.125386+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":3159,"log_dups_size":0,"ondisk_log_size":3159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:08.035811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207028224,"num_objects":3159,"num_object_clones":0,"num_object_copies":9477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3159,"num_write_kb":202176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"76'81","reported_seq":70,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857813+0000","last_change":"2024-09-10T18:35:58.314505+0000","last_active":"2024-09-10T18:36:32.857813+0000","last_peered":"2024-09-10T18:36:32.857813+0000","last_clean":"2024-09-10T18:36:32.857813+0000","last_became_active":"2024-09-10T18:34:32.818251+0000","last_became_peered":"2024-09-10T18:34:32.818251+0000","last_unstale":"2024-09-10T18:36:32.857813+0000","last_undegraded":"2024-09-10T18:36:32.857813+0000","last_fullsized":"2024-09-10T18:36:32.857813+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:58.314438+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:58.314438+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:59.510307+0000","scrub_duration":0.0015006209999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"302'19","reported_seq":359,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.866775+0000","last_change":"2024-09-10T18:35:18.836229+0000","last_active":"2024-09-10T18:36:32.866775+0000","last_peered":"2024-09-10T18:36:32.866775+0000","last_clean":"2024-09-10T18:36:32.866775+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:36:32.866775+0000","last_undegraded":"2024-09-10T18:36:32.866775+0000","last_fullsized":"2024-09-10T18:36:32.866775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'19","last_scrub_stamp":"2024-09-10T18:35:18.836131+0000","last_deep_scrub":"302'19","last_deep_scrub_stamp":"2024-09-10T18:35:18.836131+0000","last_clean_scrub_stamp":"2024-09-10T18:35:18.836131+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:41.789867+0000","scrub_duration":0.019271303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"265'2","reported_seq":358,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.852862+0000","last_change":"2024-09-10T18:35:14.515615+0000","last_active":"2024-09-10T18:36:32.852862+0000","last_peered":"2024-09-10T18:36:32.852862+0000","last_clean":"2024-09-10T18:36:32.852862+0000","last_became_active":"2024-09-10T18:26:27.644997+0000","last_became_peered":"2024-09-10T18:26:27.644997+0000","last_unstale":"2024-09-10T18:36:32.852862+0000","last_undegraded":"2024-09-10T18:36:32.852862+0000","last_fullsized":"2024-09-10T18:36:32.852862+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'2","last_scrub_stamp":"2024-09-10T18:35:14.515533+0000","last_deep_scrub":"265'2","last_deep_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_clean_scrub_stamp":"2024-09-10T18:35:14.515533+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:38.258111+0000","scrub_duration":0.0046328369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":352,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857617+0000","last_change":"2024-09-10T18:34:58.759217+0000","last_active":"2024-09-10T18:36:32.857617+0000","last_peered":"2024-09-10T18:36:32.857617+0000","last_clean":"2024-09-10T18:36:32.857617+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:36:32.857617+0000","last_undegraded":"2024-09-10T18:36:32.857617+0000","last_fullsized":"2024-09-10T18:36:32.857617+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:58.759108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:58.759108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010436621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":336,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.866547+0000","last_change":"2024-09-10T18:36:14.665247+0000","last_active":"2024-09-10T18:36:32.866547+0000","last_peered":"2024-09-10T18:36:32.866547+0000","last_clean":"2024-09-10T18:36:32.866547+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:36:32.866547+0000","last_undegraded":"2024-09-10T18:36:32.866547+0000","last_fullsized":"2024-09-10T18:36:32.866547+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:14.665183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:14.665183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:23.790484+0000","scrub_duration":0.0014290399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"322'2612","reported_seq":2627,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:36.150005+0000","last_change":"2024-09-10T18:36:07.902622+0000","last_active":"2024-09-10T18:36:36.150005+0000","last_peered":"2024-09-10T18:36:36.150005+0000","last_clean":"2024-09-10T18:36:36.150005+0000","last_became_active":"2024-09-10T18:36:07.902336+0000","last_became_peered":"2024-09-10T18:36:07.902336+0000","last_unstale":"2024-09-10T18:36:36.150005+0000","last_undegraded":"2024-09-10T18:36:36.150005+0000","last_fullsized":"2024-09-10T18:36:36.150005+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":2612,"log_dups_size":0,"ondisk_log_size":2612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:15.028254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171180032,"num_objects":2612,"num_object_clones":0,"num_object_copies":7836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2612,"num_write_kb":167168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'81","reported_seq":326,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.861122+0000","last_change":"2024-09-10T18:36:04.870940+0000","last_active":"2024-09-10T18:36:32.861122+0000","last_peered":"2024-09-10T18:36:32.861122+0000","last_clean":"2024-09-10T18:36:32.861122+0000","last_became_active":"2024-09-10T18:34:02.803861+0000","last_became_peered":"2024-09-10T18:34:02.803861+0000","last_unstale":"2024-09-10T18:36:32.861122+0000","last_undegraded":"2024-09-10T18:36:32.861122+0000","last_fullsized":"2024-09-10T18:36:32.861122+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:04.870861+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:34:15.938079+0000","last_clean_scrub_stamp":"2024-09-10T18:36:04.870861+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:32.989359+0000","scrub_duration":0.001253791,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"240'44","reported_seq":422,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.860068+0000","last_change":"2024-09-10T18:35:54.841099+0000","last_active":"2024-09-10T18:36:32.860068+0000","last_peered":"2024-09-10T18:36:32.860068+0000","last_clean":"2024-09-10T18:36:32.860068+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:36:32.860068+0000","last_undegraded":"2024-09-10T18:36:32.860068+0000","last_fullsized":"2024-09-10T18:36:32.860068+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'44","last_scrub_stamp":"2024-09-10T18:35:54.841032+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:35:54.841032+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:22.032326+0000","scrub_duration":0.0027477270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":347,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855990+0000","last_change":"2024-09-10T18:34:11.965336+0000","last_active":"2024-09-10T18:36:32.855990+0000","last_peered":"2024-09-10T18:36:32.855990+0000","last_clean":"2024-09-10T18:36:32.855990+0000","last_became_active":"2024-09-10T18:26:27.638408+0000","last_became_peered":"2024-09-10T18:26:27.638408+0000","last_unstale":"2024-09-10T18:36:32.855990+0000","last_undegraded":"2024-09-10T18:36:32.855990+0000","last_fullsized":"2024-09-10T18:36:32.855990+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:11.965263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:11.965263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034110310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":366,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.858383+0000","last_change":"2024-09-10T18:36:00.551970+0000","last_active":"2024-09-10T18:36:32.858383+0000","last_peered":"2024-09-10T18:36:32.858383+0000","last_clean":"2024-09-10T18:36:32.858383+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:36:32.858383+0000","last_undegraded":"2024-09-10T18:36:32.858383+0000","last_fullsized":"2024-09-10T18:36:32.858383+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:00.551910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:34:21.447291+0000","last_clean_scrub_stamp":"2024-09-10T18:36:00.551910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:13.069501+0000","scrub_duration":0.0017792039999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":337,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857501+0000","last_change":"2024-09-10T18:35:02.706333+0000","last_active":"2024-09-10T18:36:32.857501+0000","last_peered":"2024-09-10T18:36:32.857501+0000","last_clean":"2024-09-10T18:36:32.857501+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:36:32.857501+0000","last_undegraded":"2024-09-10T18:36:32.857501+0000","last_fullsized":"2024-09-10T18:36:32.857501+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:02.706248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:02.706248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066128610000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"322'3166","reported_seq":3182,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:40.920367+0000","last_change":"2024-09-10T18:36:07.902995+0000","last_active":"2024-09-10T18:36:40.920367+0000","last_peered":"2024-09-10T18:36:40.920367+0000","last_clean":"2024-09-10T18:36:40.920367+0000","last_became_active":"2024-09-10T18:36:07.902661+0000","last_became_peered":"2024-09-10T18:36:07.902661+0000","last_unstale":"2024-09-10T18:36:40.920367+0000","last_undegraded":"2024-09-10T18:36:40.920367+0000","last_fullsized":"2024-09-10T18:36:40.920367+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":3166,"log_dups_size":0,"ondisk_log_size":3166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:36.577552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207486976,"num_objects":3166,"num_object_clones":0,"num_object_copies":9498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3166,"num_write_kb":202624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"76'81","reported_seq":264,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.854727+0000","last_change":"2024-09-10T18:34:32.957284+0000","last_active":"2024-09-10T18:36:32.854727+0000","last_peered":"2024-09-10T18:36:32.854727+0000","last_clean":"2024-09-10T18:36:32.854727+0000","last_became_active":"2024-09-10T18:34:32.956761+0000","last_became_peered":"2024-09-10T18:34:32.956761+0000","last_unstale":"2024-09-10T18:36:32.854727+0000","last_undegraded":"2024-09-10T18:36:32.854727+0000","last_fullsized":"2024-09-10T18:36:32.854727+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:33:01.370561+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:33:01.370561+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029251390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54613,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"301'23","reported_seq":378,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.856233+0000","last_change":"2024-09-10T18:35:15.283425+0000","last_active":"2024-09-10T18:36:32.856233+0000","last_peered":"2024-09-10T18:36:32.856233+0000","last_clean":"2024-09-10T18:36:32.856233+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:36:32.856233+0000","last_undegraded":"2024-09-10T18:36:32.856233+0000","last_fullsized":"2024-09-10T18:36:32.856233+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'23","last_scrub_stamp":"2024-09-10T18:35:15.283373+0000","last_deep_scrub":"301'23","last_deep_scrub_stamp":"2024-09-10T18:35:15.283373+0000","last_clean_scrub_stamp":"2024-09-10T18:35:15.283373+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018617538999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":340,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.856473+0000","last_change":"2024-09-10T18:33:14.879881+0000","last_active":"2024-09-10T18:36:32.856473+0000","last_peered":"2024-09-10T18:36:32.856473+0000","last_clean":"2024-09-10T18:36:32.856473+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:36:32.856473+0000","last_undegraded":"2024-09-10T18:36:32.856473+0000","last_fullsized":"2024-09-10T18:36:32.856473+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:14.879787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:33:14.879787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044267550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":366,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.858008+0000","last_change":"2024-09-10T18:36:06.476756+0000","last_active":"2024-09-10T18:36:32.858008+0000","last_peered":"2024-09-10T18:36:32.858008+0000","last_clean":"2024-09-10T18:36:32.858008+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:36:32.858008+0000","last_undegraded":"2024-09-10T18:36:32.858008+0000","last_fullsized":"2024-09-10T18:36:32.858008+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.476674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.476674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:06.594858+0000","scrub_duration":0.001628991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":327,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855437+0000","last_change":"2024-09-10T18:36:05.766562+0000","last_active":"2024-09-10T18:36:32.855437+0000","last_peered":"2024-09-10T18:36:32.855437+0000","last_clean":"2024-09-10T18:36:32.855437+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:36:32.855437+0000","last_undegraded":"2024-09-10T18:36:32.855437+0000","last_fullsized":"2024-09-10T18:36:32.855437+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:05.766507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.803236+0000","last_clean_scrub_stamp":"2024-09-10T18:36:05.766507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:30.530626+0000","scrub_duration":0.0023023990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"322'3142","reported_seq":3158,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:40.070706+0000","last_change":"2024-09-10T18:36:07.904127+0000","last_active":"2024-09-10T18:36:40.070706+0000","last_peered":"2024-09-10T18:36:40.070706+0000","last_clean":"2024-09-10T18:36:40.070706+0000","last_became_active":"2024-09-10T18:36:07.903791+0000","last_became_peered":"2024-09-10T18:36:07.903791+0000","last_unstale":"2024-09-10T18:36:40.070706+0000","last_undegraded":"2024-09-10T18:36:40.070706+0000","last_fullsized":"2024-09-10T18:36:40.070706+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":3142,"log_dups_size":0,"ondisk_log_size":3142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:11.175247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205914112,"num_objects":3142,"num_object_clones":0,"num_object_copies":9426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3142,"num_write_kb":201088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"76'81","reported_seq":70,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855583+0000","last_change":"2024-09-10T18:36:11.950457+0000","last_active":"2024-09-10T18:36:32.855583+0000","last_peered":"2024-09-10T18:36:32.855583+0000","last_clean":"2024-09-10T18:36:32.855583+0000","last_became_active":"2024-09-10T18:34:32.822196+0000","last_became_peered":"2024-09-10T18:34:32.822196+0000","last_unstale":"2024-09-10T18:36:32.855583+0000","last_undegraded":"2024-09-10T18:36:32.855583+0000","last_fullsized":"2024-09-10T18:36:32.855583+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:11.950311+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:11.950311+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:13.472288+0000","scrub_duration":0.0020040769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":379,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.853543+0000","last_change":"2024-09-10T18:34:06.163124+0000","last_active":"2024-09-10T18:36:32.853543+0000","last_peered":"2024-09-10T18:36:32.853543+0000","last_clean":"2024-09-10T18:36:32.853543+0000","last_became_active":"2024-09-10T18:26:27.645932+0000","last_became_peered":"2024-09-10T18:26:27.645932+0000","last_unstale":"2024-09-10T18:36:32.853543+0000","last_undegraded":"2024-09-10T18:36:32.853543+0000","last_fullsized":"2024-09-10T18:36:32.853543+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:34:06.163072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:34:06.163072+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016579489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":350,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857055+0000","last_change":"2024-09-10T18:35:58.895879+0000","last_active":"2024-09-10T18:36:32.857055+0000","last_peered":"2024-09-10T18:36:32.857055+0000","last_clean":"2024-09-10T18:36:32.857055+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:36:32.857055+0000","last_undegraded":"2024-09-10T18:36:32.857055+0000","last_fullsized":"2024-09-10T18:36:32.857055+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:35:58.895792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:35:58.895792+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:19.782961+0000","scrub_duration":0.0021727780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":536,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:36.145958+0000","last_change":"2024-09-10T18:36:02.894810+0000","last_active":"2024-09-10T18:36:36.145958+0000","last_peered":"2024-09-10T18:36:36.145958+0000","last_clean":"2024-09-10T18:36:36.145958+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:36:36.145958+0000","last_undegraded":"2024-09-10T18:36:36.145958+0000","last_fullsized":"2024-09-10T18:36:36.145958+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:36:02.894094+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:36:02.894094+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:16.074759+0000","scrub_duration":0.0028441780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":332,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.852672+0000","last_change":"2024-09-10T18:35:53.486754+0000","last_active":"2024-09-10T18:36:32.852672+0000","last_peered":"2024-09-10T18:36:32.852672+0000","last_clean":"2024-09-10T18:36:32.852672+0000","last_became_active":"2024-09-10T18:26:27.647511+0000","last_became_peered":"2024-09-10T18:26:27.647511+0000","last_unstale":"2024-09-10T18:36:32.852672+0000","last_undegraded":"2024-09-10T18:36:32.852672+0000","last_fullsized":"2024-09-10T18:36:32.852672+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:53.486678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:53.486678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:22.033025+0000","scrub_duration":0.0035730969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'81","reported_seq":60,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.852896+0000","last_change":"2024-09-10T18:35:06.518466+0000","last_active":"2024-09-10T18:36:32.852896+0000","last_peered":"2024-09-10T18:36:32.852896+0000","last_clean":"2024-09-10T18:36:32.852896+0000","last_became_active":"2024-09-10T18:34:02.810126+0000","last_became_peered":"2024-09-10T18:34:02.810126+0000","last_unstale":"2024-09-10T18:36:32.852896+0000","last_undegraded":"2024-09-10T18:36:32.852896+0000","last_fullsized":"2024-09-10T18:36:32.852896+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:06.518411+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:06.518411+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008298649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":296,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.861485+0000","last_change":"2024-09-10T18:35:59.886309+0000","last_active":"2024-09-10T18:36:32.861485+0000","last_peered":"2024-09-10T18:36:32.861485+0000","last_clean":"2024-09-10T18:36:32.861485+0000","last_became_active":"2024-09-10T18:26:27.640922+0000","last_became_peered":"2024-09-10T18:26:27.640922+0000","last_unstale":"2024-09-10T18:36:32.861485+0000","last_undegraded":"2024-09-10T18:36:32.861485+0000","last_fullsized":"2024-09-10T18:36:32.861485+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:59.886242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:35:59.886242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:27.332998+0000","scrub_duration":0.001788724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"240'12","reported_seq":349,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.856069+0000","last_change":"2024-09-10T18:36:00.854061+0000","last_active":"2024-09-10T18:36:32.856069+0000","last_peered":"2024-09-10T18:36:32.856069+0000","last_clean":"2024-09-10T18:36:32.856069+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:36:32.856069+0000","last_undegraded":"2024-09-10T18:36:32.856069+0000","last_fullsized":"2024-09-10T18:36:32.856069+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-10T18:36:00.854004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:00.854004+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:20.572196+0000","scrub_duration":0.0019737209999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":336,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855751+0000","last_change":"2024-09-10T18:36:17.693584+0000","last_active":"2024-09-10T18:36:32.855751+0000","last_peered":"2024-09-10T18:36:32.855751+0000","last_clean":"2024-09-10T18:36:32.855751+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:36:32.855751+0000","last_undegraded":"2024-09-10T18:36:32.855751+0000","last_fullsized":"2024-09-10T18:36:32.855751+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:17.693527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:17.693527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:18.331648+0000","scrub_duration":0.0014350840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":331,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855652+0000","last_change":"2024-09-10T18:33:58.161088+0000","last_active":"2024-09-10T18:36:32.855652+0000","last_peered":"2024-09-10T18:36:32.855652+0000","last_clean":"2024-09-10T18:36:32.855652+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:36:32.855652+0000","last_undegraded":"2024-09-10T18:36:32.855652+0000","last_fullsized":"2024-09-10T18:36:32.855652+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:58.161019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:58.161019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042556759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'81","reported_seq":267,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.858379+0000","last_change":"2024-09-10T18:35:08.832694+0000","last_active":"2024-09-10T18:36:32.858379+0000","last_peered":"2024-09-10T18:36:32.858379+0000","last_clean":"2024-09-10T18:36:32.858379+0000","last_became_active":"2024-09-10T18:34:02.815880+0000","last_became_peered":"2024-09-10T18:34:02.815880+0000","last_unstale":"2024-09-10T18:36:32.858379+0000","last_undegraded":"2024-09-10T18:36:32.858379+0000","last_fullsized":"2024-09-10T18:36:32.858379+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:08.832597+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:08.832597+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00460159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"240'19","reported_seq":362,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.867701+0000","last_change":"2024-09-10T18:35:19.802100+0000","last_active":"2024-09-10T18:36:32.867701+0000","last_peered":"2024-09-10T18:36:32.867701+0000","last_clean":"2024-09-10T18:36:32.867701+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:36:32.867701+0000","last_undegraded":"2024-09-10T18:36:32.867701+0000","last_fullsized":"2024-09-10T18:36:32.867701+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:35:19.802016+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-09-10T18:35:19.802016+0000","last_clean_scrub_stamp":"2024-09-10T18:35:19.802016+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:36.919136+0000","scrub_duration":0.010791837,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":347,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857362+0000","last_change":"2024-09-10T18:36:04.875938+0000","last_active":"2024-09-10T18:36:32.857362+0000","last_peered":"2024-09-10T18:36:32.857362+0000","last_clean":"2024-09-10T18:36:32.857362+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:36:32.857362+0000","last_undegraded":"2024-09-10T18:36:32.857362+0000","last_fullsized":"2024-09-10T18:36:32.857362+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:04.875692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:36:04.875692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:20.013533+0000","scrub_duration":0.0018984749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":329,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855707+0000","last_change":"2024-09-10T18:35:12.764509+0000","last_active":"2024-09-10T18:36:32.855707+0000","last_peered":"2024-09-10T18:36:32.855707+0000","last_clean":"2024-09-10T18:36:32.855707+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:36:32.855707+0000","last_undegraded":"2024-09-10T18:36:32.855707+0000","last_fullsized":"2024-09-10T18:36:32.855707+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:12.764433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:12.764433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069242059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":354,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857549+0000","last_change":"2024-09-10T18:36:05.455694+0000","last_active":"2024-09-10T18:36:32.857549+0000","last_peered":"2024-09-10T18:36:32.857549+0000","last_clean":"2024-09-10T18:36:32.857549+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:36:32.857549+0000","last_undegraded":"2024-09-10T18:36:32.857549+0000","last_fullsized":"2024-09-10T18:36:32.857549+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:05.455640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:05.455640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:07.467175+0000","scrub_duration":0.019696215999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'81","reported_seq":60,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.853701+0000","last_change":"2024-09-10T18:35:04.516060+0000","last_active":"2024-09-10T18:36:32.853701+0000","last_peered":"2024-09-10T18:36:32.853701+0000","last_clean":"2024-09-10T18:36:32.853701+0000","last_became_active":"2024-09-10T18:34:02.797304+0000","last_became_peered":"2024-09-10T18:34:02.797304+0000","last_unstale":"2024-09-10T18:36:32.853701+0000","last_undegraded":"2024-09-10T18:36:32.853701+0000","last_fullsized":"2024-09-10T18:36:32.853701+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:04.515970+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:04.515970+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032060309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"240'19","reported_seq":368,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.853023+0000","last_change":"2024-09-10T18:36:14.732278+0000","last_active":"2024-09-10T18:36:32.853023+0000","last_peered":"2024-09-10T18:36:32.853023+0000","last_clean":"2024-09-10T18:36:32.853023+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:36:32.853023+0000","last_undegraded":"2024-09-10T18:36:32.853023+0000","last_fullsized":"2024-09-10T18:36:32.853023+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:36:14.732217+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:36:14.732217+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:39.275538+0000","scrub_duration":0.0020637149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":342,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.869756+0000","last_change":"2024-09-10T18:35:12.758404+0000","last_active":"2024-09-10T18:36:32.869756+0000","last_peered":"2024-09-10T18:36:32.869756+0000","last_clean":"2024-09-10T18:36:32.869756+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:36:32.869756+0000","last_undegraded":"2024-09-10T18:36:32.869756+0000","last_fullsized":"2024-09-10T18:36:32.869756+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:12.758329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_clean_scrub_stamp":"2024-09-10T18:35:12.758329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010517383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":519,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:36.147504+0000","last_change":"2024-09-10T18:35:01.701033+0000","last_active":"2024-09-10T18:36:36.147504+0000","last_peered":"2024-09-10T18:36:36.147504+0000","last_clean":"2024-09-10T18:36:36.147504+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:36:36.147504+0000","last_undegraded":"2024-09-10T18:36:36.147504+0000","last_fullsized":"2024-09-10T18:36:36.147504+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:35:01.700934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:01.700934+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050961280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":62,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.856197+0000","last_change":"2024-09-10T18:33:29.142401+0000","last_active":"2024-09-10T18:36:32.856197+0000","last_peered":"2024-09-10T18:36:32.856197+0000","last_clean":"2024-09-10T18:36:32.856197+0000","last_became_active":"2024-09-10T18:33:29.141936+0000","last_became_peered":"2024-09-10T18:33:29.141936+0000","last_unstale":"2024-09-10T18:36:32.856197+0000","last_undegraded":"2024-09-10T18:36:32.856197+0000","last_fullsized":"2024-09-10T18:36:32.856197+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:11.727786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:11.727786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"76'81","reported_seq":132,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.866423+0000","last_change":"2024-09-10T18:36:08.710620+0000","last_active":"2024-09-10T18:36:32.866423+0000","last_peered":"2024-09-10T18:36:32.866423+0000","last_clean":"2024-09-10T18:36:32.866423+0000","last_became_active":"2024-09-10T18:34:02.810381+0000","last_became_peered":"2024-09-10T18:34:02.810381+0000","last_unstale":"2024-09-10T18:36:32.866423+0000","last_undegraded":"2024-09-10T18:36:32.866423+0000","last_fullsized":"2024-09-10T18:36:32.866423+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:08.710566+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:36:08.710566+0000","last_clean_scrub_stamp":"2024-09-10T18:36:08.710566+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:11.559877+0000","scrub_duration":0.0012163009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":361,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.854950+0000","last_change":"2024-09-10T18:33:59.230861+0000","last_active":"2024-09-10T18:36:32.854950+0000","last_peered":"2024-09-10T18:36:32.854950+0000","last_clean":"2024-09-10T18:36:32.854950+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:36:32.854950+0000","last_undegraded":"2024-09-10T18:36:32.854950+0000","last_fullsized":"2024-09-10T18:36:32.854950+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-10T18:33:59.227198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:33:59.227198+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012577204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":340,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855427+0000","last_change":"2024-09-10T18:35:17.826500+0000","last_active":"2024-09-10T18:36:32.855427+0000","last_peered":"2024-09-10T18:36:32.855427+0000","last_clean":"2024-09-10T18:36:32.855427+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:36:32.855427+0000","last_undegraded":"2024-09-10T18:36:32.855427+0000","last_fullsized":"2024-09-10T18:36:32.855427+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:17.826448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:17.826448+0000","last_clean_scrub_stamp":"2024-09-10T18:35:17.826448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002920808,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":527,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-09-10T18:36:36.132558+0000","last_change":"2024-09-10T18:34:58.485675+0000","last_active":"2024-09-10T18:36:36.132558+0000","last_peered":"2024-09-10T18:36:36.132558+0000","last_clean":"2024-09-10T18:36:36.132558+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:36:36.132558+0000","last_undegraded":"2024-09-10T18:36:36.132558+0000","last_fullsized":"2024-09-10T18:36:36.132558+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:34:58.485562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:58.485562+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059980570000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":336,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.867083+0000","last_change":"2024-09-10T18:36:06.720664+0000","last_active":"2024-09-10T18:36:32.867083+0000","last_peered":"2024-09-10T18:36:32.867083+0000","last_clean":"2024-09-10T18:36:32.867083+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:36:32.867083+0000","last_undegraded":"2024-09-10T18:36:32.867083+0000","last_fullsized":"2024-09-10T18:36:32.867083+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.720613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.720613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:13.034436+0000","scrub_duration":0.0015989380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'81","reported_seq":52,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.859634+0000","last_change":"2024-09-10T18:35:53.652838+0000","last_active":"2024-09-10T18:36:32.859634+0000","last_peered":"2024-09-10T18:36:32.859634+0000","last_clean":"2024-09-10T18:36:32.859634+0000","last_became_active":"2024-09-10T18:34:34.881186+0000","last_became_peered":"2024-09-10T18:34:34.881186+0000","last_unstale":"2024-09-10T18:36:32.859634+0000","last_undegraded":"2024-09-10T18:36:32.859634+0000","last_fullsized":"2024-09-10T18:36:32.859634+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:53.652767+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:53.652767+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:17.120912+0000","scrub_duration":0.0013276989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":376,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.856266+0000","last_change":"2024-09-10T18:35:14.262075+0000","last_active":"2024-09-10T18:36:32.856266+0000","last_peered":"2024-09-10T18:36:32.856266+0000","last_clean":"2024-09-10T18:36:32.856266+0000","last_became_active":"2024-09-10T18:26:27.649202+0000","last_became_peered":"2024-09-10T18:26:27.649202+0000","last_unstale":"2024-09-10T18:36:32.856266+0000","last_undegraded":"2024-09-10T18:36:32.856266+0000","last_fullsized":"2024-09-10T18:36:32.856266+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:35:14.261980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:14.261980+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026192849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":340,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857385+0000","last_change":"2024-09-10T18:34:04.380240+0000","last_active":"2024-09-10T18:36:32.857385+0000","last_peered":"2024-09-10T18:36:32.857385+0000","last_clean":"2024-09-10T18:36:32.857385+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:36:32.857385+0000","last_undegraded":"2024-09-10T18:36:32.857385+0000","last_fullsized":"2024-09-10T18:36:32.857385+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:04.380160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:34:04.380160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018137730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":350,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.859905+0000","last_change":"2024-09-10T18:36:11.523432+0000","last_active":"2024-09-10T18:36:32.859905+0000","last_peered":"2024-09-10T18:36:32.859905+0000","last_clean":"2024-09-10T18:36:32.859905+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:36:32.859905+0000","last_undegraded":"2024-09-10T18:36:32.859905+0000","last_fullsized":"2024-09-10T18:36:32.859905+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:11.523380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_clean_scrub_stamp":"2024-09-10T18:36:11.523380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:33.294003+0000","scrub_duration":0.0014936800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":317,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.854520+0000","last_change":"2024-09-10T18:34:14.263579+0000","last_active":"2024-09-10T18:36:32.854520+0000","last_peered":"2024-09-10T18:36:32.854520+0000","last_clean":"2024-09-10T18:36:32.854520+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:36:32.854520+0000","last_undegraded":"2024-09-10T18:36:32.854520+0000","last_fullsized":"2024-09-10T18:36:32.854520+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:14.263499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:34:14.263499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019685509999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'81","reported_seq":51,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.853728+0000","last_change":"2024-09-10T18:36:08.676315+0000","last_active":"2024-09-10T18:36:32.853728+0000","last_peered":"2024-09-10T18:36:32.853728+0000","last_clean":"2024-09-10T18:36:32.853728+0000","last_became_active":"2024-09-10T18:34:34.858291+0000","last_became_peered":"2024-09-10T18:34:34.858291+0000","last_unstale":"2024-09-10T18:36:32.853728+0000","last_undegraded":"2024-09-10T18:36:32.853728+0000","last_fullsized":"2024-09-10T18:36:32.853728+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:08.676247+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:08.676247+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:12.854629+0000","scrub_duration":0.001606512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"240'15","reported_seq":381,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.860026+0000","last_change":"2024-09-10T18:36:16.592923+0000","last_active":"2024-09-10T18:36:32.860026+0000","last_peered":"2024-09-10T18:36:32.860026+0000","last_clean":"2024-09-10T18:36:32.860026+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:36:32.860026+0000","last_undegraded":"2024-09-10T18:36:32.860026+0000","last_fullsized":"2024-09-10T18:36:32.860026+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:36:16.592847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:16.592847+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:39.683840+0000","scrub_duration":0.0020114550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":343,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855879+0000","last_change":"2024-09-10T18:36:07.931194+0000","last_active":"2024-09-10T18:36:32.855879+0000","last_peered":"2024-09-10T18:36:32.855879+0000","last_clean":"2024-09-10T18:36:32.855879+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:36:32.855879+0000","last_undegraded":"2024-09-10T18:36:32.855879+0000","last_fullsized":"2024-09-10T18:36:32.855879+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:07.931108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.931108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:18.035585+0000","scrub_duration":0.0012878189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":322,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855603+0000","last_change":"2024-09-10T18:34:56.701737+0000","last_active":"2024-09-10T18:36:32.855603+0000","last_peered":"2024-09-10T18:36:32.855603+0000","last_clean":"2024-09-10T18:36:32.855603+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:36:32.855603+0000","last_undegraded":"2024-09-10T18:36:32.855603+0000","last_fullsized":"2024-09-10T18:36:32.855603+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:56.701661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:34:56.701661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064135620000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":336,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.865857+0000","last_change":"2024-09-10T18:35:58.713980+0000","last_active":"2024-09-10T18:36:32.865857+0000","last_peered":"2024-09-10T18:36:32.865857+0000","last_clean":"2024-09-10T18:36:32.865857+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:36:32.865857+0000","last_undegraded":"2024-09-10T18:36:32.865857+0000","last_fullsized":"2024-09-10T18:36:32.865857+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:58.713917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:58.713917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:05.732478+0000","scrub_duration":0.0015550169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'81","reported_seq":70,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.857809+0000","last_change":"2024-09-10T18:36:07.443661+0000","last_active":"2024-09-10T18:36:32.857809+0000","last_peered":"2024-09-10T18:36:32.857809+0000","last_clean":"2024-09-10T18:36:32.857809+0000","last_became_active":"2024-09-10T18:34:02.802580+0000","last_became_peered":"2024-09-10T18:34:02.802580+0000","last_unstale":"2024-09-10T18:36:32.857809+0000","last_undegraded":"2024-09-10T18:36:32.857809+0000","last_fullsized":"2024-09-10T18:36:32.857809+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:07.443596+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.443596+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:20.824349+0000","scrub_duration":0.0016421059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":369,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855420+0000","last_change":"2024-09-10T18:35:03.727242+0000","last_active":"2024-09-10T18:36:32.855420+0000","last_peered":"2024-09-10T18:36:32.855420+0000","last_clean":"2024-09-10T18:36:32.855420+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:36:32.855420+0000","last_undegraded":"2024-09-10T18:36:32.855420+0000","last_fullsized":"2024-09-10T18:36:32.855420+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:35:03.727153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:03.727153+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069271460000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":341,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855819+0000","last_change":"2024-09-10T18:34:08.134374+0000","last_active":"2024-09-10T18:36:32.855819+0000","last_peered":"2024-09-10T18:36:32.855819+0000","last_clean":"2024-09-10T18:36:32.855819+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:36:32.855819+0000","last_undegraded":"2024-09-10T18:36:32.855819+0000","last_fullsized":"2024-09-10T18:36:32.855819+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:08.134230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:34:08.134230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077618749999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":336,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855973+0000","last_change":"2024-09-10T18:34:57.776098+0000","last_active":"2024-09-10T18:36:32.855973+0000","last_peered":"2024-09-10T18:36:32.855973+0000","last_clean":"2024-09-10T18:36:32.855973+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:36:32.855973+0000","last_undegraded":"2024-09-10T18:36:32.855973+0000","last_fullsized":"2024-09-10T18:36:32.855973+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:57.776022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:57.776022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002853441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":340,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855813+0000","last_change":"2024-09-10T18:36:18.740400+0000","last_active":"2024-09-10T18:36:32.855813+0000","last_peered":"2024-09-10T18:36:32.855813+0000","last_clean":"2024-09-10T18:36:32.855813+0000","last_became_active":"2024-09-10T18:26:27.636996+0000","last_became_peered":"2024-09-10T18:26:27.636996+0000","last_unstale":"2024-09-10T18:36:32.855813+0000","last_undegraded":"2024-09-10T18:36:32.855813+0000","last_fullsized":"2024-09-10T18:36:32.855813+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:18.227428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:18.227428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063780959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'81","reported_seq":51,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.855511+0000","last_change":"2024-09-10T18:36:14.999809+0000","last_active":"2024-09-10T18:36:32.855511+0000","last_peered":"2024-09-10T18:36:32.855511+0000","last_clean":"2024-09-10T18:36:32.855511+0000","last_became_active":"2024-09-10T18:34:34.868371+0000","last_became_peered":"2024-09-10T18:34:34.868371+0000","last_unstale":"2024-09-10T18:36:32.855511+0000","last_undegraded":"2024-09-10T18:36:32.855511+0000","last_fullsized":"2024-09-10T18:36:32.855511+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:14.999747+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:14.999747+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:41.897947+0000","scrub_duration":0.001461415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":350,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.854326+0000","last_change":"2024-09-10T18:36:01.819314+0000","last_active":"2024-09-10T18:36:32.854326+0000","last_peered":"2024-09-10T18:36:32.854326+0000","last_clean":"2024-09-10T18:36:32.854326+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:36:32.854326+0000","last_undegraded":"2024-09-10T18:36:32.854326+0000","last_fullsized":"2024-09-10T18:36:32.854326+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-09-10T18:36:01.819258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:01.819258+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:17.021892+0000","scrub_duration":0.001608246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":335,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.867124+0000","last_change":"2024-09-10T18:35:01.976534+0000","last_active":"2024-09-10T18:36:32.867124+0000","last_peered":"2024-09-10T18:36:32.867124+0000","last_clean":"2024-09-10T18:36:32.867124+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:36:32.867124+0000","last_undegraded":"2024-09-10T18:36:32.867124+0000","last_fullsized":"2024-09-10T18:36:32.867124+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:01.976429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:35:01.976429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.11603027,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":300,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.859638+0000","last_change":"2024-09-10T18:36:01.890154+0000","last_active":"2024-09-10T18:36:32.859638+0000","last_peered":"2024-09-10T18:36:32.859638+0000","last_clean":"2024-09-10T18:36:32.859638+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:36:32.859638+0000","last_undegraded":"2024-09-10T18:36:32.859638+0000","last_fullsized":"2024-09-10T18:36:32.859638+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:01.890087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:01.890087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:29.323792+0000","scrub_duration":0.001570519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":334,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.854890+0000","last_change":"2024-09-10T18:35:18.879787+0000","last_active":"2024-09-10T18:36:32.854890+0000","last_peered":"2024-09-10T18:36:32.854890+0000","last_clean":"2024-09-10T18:36:32.854890+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:36:32.854890+0000","last_undegraded":"2024-09-10T18:36:32.854890+0000","last_fullsized":"2024-09-10T18:36:32.854890+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:18.879717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:18.879717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:41.844243+0000","scrub_duration":0.0059146550000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'81","reported_seq":51,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.856181+0000","last_change":"2024-09-10T18:36:00.273377+0000","last_active":"2024-09-10T18:36:32.856181+0000","last_peered":"2024-09-10T18:36:32.856181+0000","last_clean":"2024-09-10T18:36:32.856181+0000","last_became_active":"2024-09-10T18:34:34.867380+0000","last_became_peered":"2024-09-10T18:34:34.867380+0000","last_unstale":"2024-09-10T18:36:32.856181+0000","last_undegraded":"2024-09-10T18:36:32.856181+0000","last_fullsized":"2024-09-10T18:36:32.856181+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:00.273308+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:00.273308+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:16.529317+0000","scrub_duration":0.0016513669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":345,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.859040+0000","last_change":"2024-09-10T18:35:56.582580+0000","last_active":"2024-09-10T18:36:32.859040+0000","last_peered":"2024-09-10T18:36:32.859040+0000","last_clean":"2024-09-10T18:36:32.859040+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:36:32.859040+0000","last_undegraded":"2024-09-10T18:36:32.859040+0000","last_fullsized":"2024-09-10T18:36:32.859040+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:56.582525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_clean_scrub_stamp":"2024-09-10T18:35:56.582525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:22.552243+0000","scrub_duration":0.0014918869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":323,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.854588+0000","last_change":"2024-09-10T18:34:03.175239+0000","last_active":"2024-09-10T18:36:32.854588+0000","last_peered":"2024-09-10T18:36:32.854588+0000","last_clean":"2024-09-10T18:36:32.854588+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:36:32.854588+0000","last_undegraded":"2024-09-10T18:36:32.854588+0000","last_fullsized":"2024-09-10T18:36:32.854588+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:03.175158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:34:03.175158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046326559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":351,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.861202+0000","last_change":"2024-09-10T18:35:08.862632+0000","last_active":"2024-09-10T18:36:32.861202+0000","last_peered":"2024-09-10T18:36:32.861202+0000","last_clean":"2024-09-10T18:36:32.861202+0000","last_became_active":"2024-09-10T18:26:27.648112+0000","last_became_peered":"2024-09-10T18:26:27.648112+0000","last_unstale":"2024-09-10T18:36:32.861202+0000","last_undegraded":"2024-09-10T18:36:32.861202+0000","last_fullsized":"2024-09-10T18:36:32.861202+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:35:08.862542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:08.862542+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:36:37.474846+0000","scrub_duration":0.0029242280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":338,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-10T18:36:32.870029+0000","last_change":"2024-09-10T18:35:59.688760+0000","last_active":"2024-09-10T18:36:32.870029+0000","last_peered":"2024-09-10T18:36:32.870029+0000","last_clean":"2024-09-10T18:36:32.870029+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:36:32.870029+0000","last_undegraded":"2024-09-10T18:36:32.870029+0000","last_fullsized":"2024-09-10T18:36:32.870029+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:59.688701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:35:59.688701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:10.509608+0000","scrub_duration":0.0018218869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3231514624,"num_objects":49309,"num_object_clones":0,"num_object_copies":147927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49309,"num_write_kb":3155776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9550233600,"data_stored":9550233600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49309,"ondisk_log_size":49309,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":746,"ondisk_log_size":746,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":901,"num_read_kb":896,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":412168,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":16,"num_write":8,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":244345,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2511,"ondisk_log_size":2511,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234169,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1382100,"kb_used_data":783884,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":92363052,"statfs":{"total":95995035648,"available":94579765248,"internally_reserved":0,"allocated":802697216,"data_stored":802273530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":1.53,"15min":1.857},"min":{"1min":0.29399999999999998,"5min":0.215,"15min":0.20799999999999999},"max":{"1min":2.4889999999999999,"5min":50.091999999999999,"15min":85.012},"last":1.6080000000000001},{"interface":"front","average":{"1min":0.75800000000000001,"5min":1.335,"15min":1.7849999999999999},"min":{"1min":0.35999999999999999,"5min":0.25600000000000001,"15min":0.222},"max":{"1min":1.72,"5min":50.104999999999997,"15min":84.665999999999997},"last":1.228}]},{"osd":1,"last update":"Tue Sep 10 18:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.609,"15min":1.4630000000000001},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.214},"max":{"1min":3.8279999999999998,"5min":39.728000000000002,"15min":49.317999999999998},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":1.4650000000000001,"15min":1.387},"min":{"1min":0.311,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.7280000000000002,"5min":39.738999999999997,"15min":49.353000000000002},"last":0.27600000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:35:49 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":2.5070000000000001,"15min":1.3959999999999999},"min":{"1min":0.309,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":4.1100000000000003,"5min":136.648,"15min":136.648},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.77900000000000003,"5min":2.5470000000000002,"15min":1.464},"min":{"1min":0.29199999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.5110000000000001,"5min":136.506,"15min":136.506},"last":0.432}]},{"osd":3,"last update":"Tue Sep 10 18:35:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.72,"15min":1.625},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.1550000000000002,"5min":38.225000000000001,"15min":86.305999999999997},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.6910000000000001,"15min":1.6619999999999999},"min":{"1min":0.318,"5min":0.29199999999999998,"15min":0.26700000000000002},"max":{"1min":3.6890000000000001,"5min":38.104999999999997,"15min":88.364000000000004},"last":0.48799999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.622,"15min":0.51700000000000002},"min":{"1min":0.24199999999999999,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":1.806,"5min":2.282,"15min":3.8839999999999999},"last":1.5880000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.69999999999999996,"15min":0.56000000000000005},"min":{"1min":0.26800000000000002,"5min":0.26200000000000001,"15min":0.223},"max":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":4.0730000000000004},"last":0.58699999999999997}]},{"osd":5,"last update":"Tue Sep 10 18:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.99099999999999999,"15min":0.72099999999999997},"min":{"1min":0.28799999999999998,"5min":0.27300000000000002,"15min":0.23499999999999999},"max":{"1min":2.8490000000000002,"5min":7.1820000000000004,"15min":7.3109999999999999},"last":1.702},{"interface":"front","average":{"1min":0.879,"5min":0.95699999999999996,"15min":0.67700000000000005},"min":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.219},"max":{"1min":1.829,"5min":6.3360000000000003,"15min":6.3360000000000003},"last":0.56000000000000005}]},{"osd":6,"last update":"Tue Sep 10 18:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.752,"15min":0.67200000000000004},"min":{"1min":0.38100000000000001,"5min":0.29299999999999998,"15min":0.27000000000000002},"max":{"1min":0.83499999999999996,"5min":2.7890000000000001,"15min":7.2750000000000004},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.69599999999999995,"15min":0.61799999999999999},"min":{"1min":0.437,"5min":0.36699999999999999,"15min":0.30199999999999999},"max":{"1min":1.177,"5min":2.3199999999999998,"15min":4.585},"last":0.70899999999999996}]}]},{"osd":6,"up_from":144,"seq":618475290748,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1684800,"kb_used_data":1373880,"kb_used_omap":7,"kb_used_meta":310904,"kb_avail":92060352,"statfs":{"total":95995035648,"available":94269800448,"internally_reserved":0,"allocated":1406853120,"data_stored":1406449542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":318365783},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:35:48 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.111,"15min":1.6990000000000001},"min":{"1min":0.26900000000000002,"5min":0.23300000000000001,"15min":0.23200000000000001},"max":{"1min":3.3799999999999999,"5min":8.3949999999999996,"15min":30.629999999999999},"last":1.3620000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.93200000000000005,"15min":1.2829999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":3.363,"5min":8.4640000000000004,"15min":30.597000000000001},"last":1.726}]},{"osd":1,"last update":"Tue Sep 10 18:35:48 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.99199999999999999,"15min":0.96299999999999997},"min":{"1min":0.31900000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.4460000000000002,"5min":8.0329999999999995,"15min":30.984000000000002},"last":0.435},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.167,"15min":1.165},"min":{"1min":0.31900000000000001,"5min":0.246,"15min":0.24199999999999999},"max":{"1min":7.7939999999999996,"5min":7.7939999999999996,"15min":30.571000000000002},"last":0.46999999999999997}]},{"osd":2,"last update":"Tue Sep 10 18:35:48 2024","interfaces":[{"interface":"back","average":{"1min":2.98,"5min":1.302,"15min":1.149},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":39.329999999999998,"5min":39.329999999999998,"15min":39.329999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":2.706,"5min":1.2370000000000001,"15min":0.97199999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.23400000000000001},"max":{"1min":39.061999999999998,"5min":39.061999999999998,"15min":39.061999999999998},"last":1.3839999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:35:48 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.2370000000000001,"15min":1.837},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.27800000000000002},"max":{"1min":4.6379999999999999,"5min":16.831,"15min":61.703000000000003},"last":1.448},{"interface":"front","average":{"1min":1.012,"5min":1.3280000000000001,"15min":1.8620000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.71,"5min":17.786000000000001,"15min":61.814},"last":0.54000000000000004}]},{"osd":4,"last update":"Tue Sep 10 18:35:48 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.58099999999999996,"15min":0.56000000000000005},"min":{"1min":0.24399999999999999,"5min":0.24199999999999999,"15min":0.23699999999999999},"max":{"1min":2.0169999999999999,"5min":2.8300000000000001,"15min":5.4950000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61499999999999999,"15min":0.63100000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":4.1689999999999996,"5min":4.6390000000000002,"15min":4.6390000000000002},"last":0.49399999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:35:48 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.74199999999999999,"15min":0.58799999999999997},"min":{"1min":0.29699999999999999,"5min":0.27200000000000002,"15min":0.24399999999999999},"max":{"1min":4.476,"5min":4.6619999999999999,"15min":4.6619999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.74099999999999999,"15min":0.56299999999999994},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.2719999999999998,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":1.3320000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:35:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73399999999999999,"15min":0.57999999999999996},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.9889999999999999,"5min":3.3650000000000002,"15min":3.3650000000000002},"last":1.421},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.66200000000000003,"15min":0.60399999999999998},"min":{"1min":0.33000000000000002,"5min":0.28999999999999998,"15min":0.27100000000000002},"max":{"1min":2.6480000000000001,"5min":3.9300000000000002,"15min":5.5309999999999997},"last":0.38700000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724740,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1576264,"kb_used_data":982208,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":92168888,"statfs":{"total":95995035648,"available":94380941312,"internally_reserved":0,"allocated":1005780992,"data_stored":1005369742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.96299999999999997,"15min":1.5229999999999999},"min":{"1min":0.28000000000000003,"5min":0.24199999999999999,"15min":0.21099999999999999},"max":{"1min":2.5129999999999999,"5min":15.819000000000001,"15min":52.033999999999999},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.82299999999999995,"15min":1.3220000000000001},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":3.294,"5min":5.4420000000000002,"15min":52.167999999999999},"last":2.4049999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":2.5760000000000001,"15min":1.5009999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.22600000000000001},"max":{"1min":2.665,"5min":111.773,"15min":111.773},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.84099999999999997,"5min":2.6200000000000001,"15min":1.591},"min":{"1min":0.317,"5min":0.30399999999999999,"15min":0.247},"max":{"1min":2.6930000000000001,"5min":111.73999999999999,"15min":111.73999999999999},"last":0.56699999999999995}]},{"osd":2,"last update":"Tue Sep 10 18:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.3160000000000001,"15min":1.2170000000000001},"min":{"1min":0.32100000000000001,"5min":0.245,"15min":0.24299999999999999},"max":{"1min":1.085,"5min":54.530000000000001,"15min":54.530000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":1.304,"15min":1.1830000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.247},"max":{"1min":1.3959999999999999,"5min":54.514000000000003,"15min":54.514000000000003},"last":0.59299999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":2.4980000000000002,"15min":1.869},"min":{"1min":0.313,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.25,"5min":106.205,"15min":106.205},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.0309999999999999,"5min":2.629,"15min":1.8149999999999999},"min":{"1min":0.40899999999999997,"5min":0.32000000000000001,"15min":0.29699999999999999},"max":{"1min":5.2560000000000002,"5min":105.878,"15min":105.878},"last":0.48399999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.68400000000000005,"15min":0.54600000000000004},"min":{"1min":0.27600000000000002,"5min":0.26800000000000002,"15min":0.20799999999999999},"max":{"1min":1.4399999999999999,"5min":5.8040000000000003,"15min":5.8040000000000003},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.74299999999999999,"15min":0.59299999999999997},"min":{"1min":0.30399999999999999,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":1.0349999999999999,"5min":6.798,"15min":6.798},"last":0.438}]},{"osd":6,"last update":"Tue Sep 10 18:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.621,"15min":0.64500000000000002},"min":{"1min":0.29699999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.0620000000000001,"5min":2.7469999999999999,"15min":4.7859999999999996},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.69899999999999995,"15min":0.66100000000000003},"min":{"1min":0.36599999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.5800000000000001,"5min":5.3680000000000003,"15min":5.3860000000000001},"last":0.39600000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:35:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":1.1140000000000001,"15min":0.73599999999999999},"min":{"1min":0.432,"5min":0.25900000000000001,"15min":0.22500000000000001},"max":{"1min":4.1159999999999997,"5min":28.934000000000001,"15min":28.934000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.167,"5min":0.92100000000000004,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.25900000000000001,"15min":0.24299999999999999},"max":{"1min":5.8259999999999996,"5min":5.8259999999999996,"15min":5.8259999999999996},"last":0.51700000000000002}]}]},{"osd":4,"up_from":33,"seq":141733920968,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1851920,"kb_used_data":1550408,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91893232,"statfs":{"total":95995035648,"available":94098669568,"internally_reserved":0,"allocated":1587617792,"data_stored":1587197469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.81000000000000005,"15min":1.0680000000000001},"min":{"1min":0.23599999999999999,"5min":0.223,"15min":0.188},"max":{"1min":3.1150000000000002,"5min":3.7309999999999999,"15min":31.585000000000001},"last":1.577},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.79100000000000004,"15min":1.014},"min":{"1min":0.22700000000000001,"5min":0.189,"15min":0.189},"max":{"1min":1.153,"5min":4.8929999999999998,"15min":29.802},"last":0.86299999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.46,"15min":1.0349999999999999},"min":{"1min":0.26800000000000002,"5min":0.254,"15min":0.21299999999999999},"max":{"1min":1.3360000000000001,"5min":50.045000000000002,"15min":50.045000000000002},"last":1.615},{"interface":"front","average":{"1min":0.56599999999999995,"5min":1.5189999999999999,"15min":1.111},"min":{"1min":0.40999999999999998,"5min":0.27100000000000002,"15min":0.22900000000000001},"max":{"1min":1.0149999999999999,"5min":50.283000000000001,"15min":50.283000000000001},"last":1.7549999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.66900000000000004,"15min":0.87},"min":{"1min":0.40699999999999997,"5min":0.249,"15min":0.249},"max":{"1min":1.276,"5min":3.0870000000000002,"15min":13.898999999999999},"last":1.792},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.71899999999999997,"15min":0.89400000000000002},"min":{"1min":0.26700000000000002,"5min":0.25900000000000001,"15min":0.22500000000000001},"max":{"1min":1.157,"5min":3.2519999999999998,"15min":14.51},"last":0.49099999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":1.538,"15min":1.3700000000000001},"min":{"1min":0.318,"5min":0.27400000000000002,"15min":0.249},"max":{"1min":1.0640000000000001,"5min":49.990000000000002,"15min":49.990000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.628,"5min":1.6439999999999999,"15min":1.4399999999999999},"min":{"1min":0.32800000000000001,"5min":0.29799999999999999,"15min":0.255},"max":{"1min":1.3520000000000001,"5min":50.764000000000003,"15min":50.764000000000003},"last":1.639}]},{"osd":5,"last update":"Tue Sep 10 18:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.745,"15min":0.64200000000000002},"min":{"1min":0.36599999999999999,"5min":0.223,"15min":0.20499999999999999},"max":{"1min":2.5649999999999999,"5min":3.0979999999999999,"15min":22.861000000000001},"last":1.897},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.77700000000000002,"15min":0.66800000000000004},"min":{"1min":0.34100000000000003,"5min":0.22,"15min":0.22},"max":{"1min":3.3799999999999999,"5min":3.3799999999999999,"15min":19.611999999999998},"last":0.439}]},{"osd":6,"last update":"Tue Sep 10 18:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.67500000000000004,"15min":0.53300000000000003},"min":{"1min":0.28799999999999998,"5min":0.27900000000000003,"15min":0.26400000000000001},"max":{"1min":2.4820000000000002,"5min":3.1139999999999999,"15min":6.7990000000000004},"last":2.0990000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.61899999999999999,"15min":0.56200000000000006},"min":{"1min":0.35199999999999998,"5min":0.307,"15min":0.30599999999999999},"max":{"1min":1.5069999999999999,"5min":3.21,"15min":7.0300000000000002},"last":1.8460000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.69399999999999995,"15min":0.57199999999999995},"min":{"1min":0.379,"5min":0.316,"15min":0.255},"max":{"1min":3.262,"5min":4.8600000000000003,"15min":7.0670000000000002},"last":1.8160000000000001},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.72399999999999998,"15min":0.59599999999999997},"min":{"1min":0.33000000000000002,"5min":0.28699999999999998,"15min":0.247},"max":{"1min":4.0430000000000001,"5min":4.0430000000000001,"15min":6.5330000000000004},"last":0.94299999999999995}]}]},{"osd":3,"up_from":27,"seq":115964117198,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1908572,"kb_used_data":1021140,"kb_used_omap":1,"kb_used_meta":887422,"kb_avail":91836580,"statfs":{"total":95995035648,"available":94040657920,"internally_reserved":0,"allocated":1045647360,"data_stored":1045215593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908720472},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:36:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":1.0369999999999999,"15min":1.3109999999999999},"min":{"1min":0.21199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.4890000000000001,"5min":5.4560000000000004,"15min":8.5530000000000008},"last":1.2969999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":1.179,"15min":1.403},"min":{"1min":0.23300000000000001,"5min":0.23100000000000001,"15min":0.193},"max":{"1min":2.9590000000000001,"5min":8.2110000000000003,"15min":13.659000000000001},"last":2.206}]},{"osd":1,"last update":"Tue Sep 10 18:36:16 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.333,"15min":1.518},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.641,"5min":6.149,"15min":35.406999999999996},"last":0.999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":1.129,"15min":1.3979999999999999},"min":{"1min":0.254,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":2.468,"5min":5.3529999999999998,"15min":36.902000000000001},"last":1.2190000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:36:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":1.0169999999999999,"15min":1.1519999999999999},"min":{"1min":0.29999999999999999,"5min":0.247,"15min":0.222},"max":{"1min":1.401,"5min":6.6479999999999997,"15min":7.7960000000000003},"last":1.252},{"interface":"front","average":{"1min":0.70899999999999996,"5min":1.0920000000000001,"15min":1.351},"min":{"1min":0.26500000000000001,"5min":0.23000000000000001,"15min":0.217},"max":{"1min":3.1640000000000001,"5min":6.7290000000000001,"15min":14.997999999999999},"last":1.5900000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:35:49 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.2350000000000001,"15min":1.2749999999999999},"min":{"1min":0.41199999999999998,"5min":0.313,"15min":0.313},"max":{"1min":2.746,"5min":6.5250000000000004,"15min":23.684999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1859999999999999,"15min":1.2130000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.27200000000000002},"max":{"1min":3.0350000000000001,"5min":6.1929999999999996,"15min":23.649000000000001},"last":1.1519999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:36:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.3129999999999999,"15min":1.377},"min":{"1min":0.38700000000000001,"5min":0.314,"15min":0.314},"max":{"1min":3.177,"5min":6.601,"15min":23.719999999999999},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.329,"5min":1.409,"15min":1.3600000000000001},"min":{"1min":0.35799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":5.5540000000000003,"5min":6.2249999999999996,"15min":23.707000000000001},"last":1.0760000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:35:49 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.296,"15min":1.262},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.317},"max":{"1min":5.6029999999999998,"5min":6.2889999999999997,"15min":31.815999999999999},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.385,"15min":1.4870000000000001},"min":{"1min":0.42899999999999999,"5min":0.38500000000000001,"15min":0.315},"max":{"1min":4.0259999999999998,"5min":7.0970000000000004,"15min":31.742000000000001},"last":1.274}]},{"osd":7,"last update":"Tue Sep 10 18:35:49 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.363,"15min":1.3220000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34100000000000003},"max":{"1min":2.8530000000000002,"5min":6.5730000000000004,"15min":23.748999999999999},"last":1.383},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.5700000000000001,"15min":1.3879999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.307},"max":{"1min":3.581,"5min":12.612,"15min":25.058},"last":1.0489999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411538,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1476100,"kb_used_data":1174588,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92269052,"statfs":{"total":95995035648,"available":94483509248,"internally_reserved":0,"allocated":1202778112,"data_stored":1202370093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":1.3899999999999999,"15min":1.48},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":3.879,"5min":26.585999999999999,"15min":43.890000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.83499999999999996,"5min":1.4670000000000001,"15min":1.702},"min":{"1min":0.24199999999999999,"5min":0.19400000000000001,"15min":0.16900000000000001},"max":{"1min":3.2799999999999998,"5min":26.48,"15min":44.331000000000003},"last":2.048}]},{"osd":1,"last update":"Tue Sep 10 18:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":1.21,"15min":1.4810000000000001},"min":{"1min":0.23699999999999999,"5min":0.19400000000000001,"15min":0.193},"max":{"1min":4.5380000000000003,"5min":15.750999999999999,"15min":44.575000000000003},"last":1.5620000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.1299999999999999,"15min":1.2669999999999999},"min":{"1min":0.24299999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.9950000000000001,"5min":14.988,"15min":44.368000000000002},"last":1.238}]},{"osd":3,"last update":"Tue Sep 10 18:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":1.8540000000000001,"15min":1.804},"min":{"1min":0.36499999999999999,"5min":0.26100000000000001,"15min":0.251},"max":{"1min":1.9590000000000001,"5min":61.331000000000003,"15min":61.331000000000003},"last":1.2050000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":1.9530000000000001,"15min":1.869},"min":{"1min":0.28599999999999998,"5min":0.23499999999999999,"15min":0.214},"max":{"1min":1.7110000000000001,"5min":61.250999999999998,"15min":61.250999999999998},"last":2.0030000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.996,"15min":1.032},"min":{"1min":0.35499999999999998,"5min":0.28699999999999998,"15min":0.24199999999999999},"max":{"1min":2.5790000000000002,"5min":12.555999999999999,"15min":12.555999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.879,"15min":0.93999999999999995},"min":{"1min":0.34100000000000003,"5min":0.33000000000000002,"15min":0.317},"max":{"1min":2.6600000000000001,"5min":7.0730000000000004,"15min":12.130000000000001},"last":0.44500000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:36:03 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.97699999999999998,"15min":1.028},"min":{"1min":0.51400000000000001,"5min":0.32000000000000001,"15min":0.27500000000000002},"max":{"1min":3.7210000000000001,"5min":6.3170000000000002,"15min":12.106},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":1.2030000000000001,"15min":1.109},"min":{"1min":0.47099999999999997,"5min":0.317,"15min":0.317},"max":{"1min":1.1020000000000001,"5min":12.529999999999999,"15min":12.529999999999999},"last":0.51000000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":1.004,"15min":1.105},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":4.4139999999999997,"5min":4.6849999999999996,"15min":5.9989999999999997},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.96299999999999997,"15min":0.96699999999999997},"min":{"1min":0.376,"5min":0.34999999999999998,"15min":0.30099999999999999},"max":{"1min":1.4390000000000001,"5min":7.1349999999999998,"15min":7.1349999999999998},"last":0.53900000000000003}]},{"osd":7,"last update":"Tue Sep 10 18:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":1.1299999999999999,"15min":1.1000000000000001},"min":{"1min":0.38900000000000001,"5min":0.374,"15min":0.30499999999999999},"max":{"1min":4.0609999999999999,"5min":10.68,"15min":10.68},"last":1.49},{"interface":"front","average":{"1min":0.84299999999999997,"5min":1.1799999999999999,"15min":1.1439999999999999},"min":{"1min":0.39300000000000002,"5min":0.376,"15min":0.307},"max":{"1min":2.77,"5min":12.596,"15min":12.596},"last":0.65300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575063,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1180076,"kb_used_data":590180,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":92565076,"statfs":{"total":95995035648,"available":94786637824,"internally_reserved":0,"allocated":604344320,"data_stored":603896285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":1.2849999999999999,"15min":1.681},"min":{"1min":0.20999999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.91100000000000003,"5min":28.893999999999998,"15min":80.456999999999994},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":1.3620000000000001,"15min":1.7789999999999999},"min":{"1min":0.23999999999999999,"5min":0.22800000000000001,"15min":0.187},"max":{"1min":2.8450000000000002,"5min":31.074999999999999,"15min":80.578000000000003},"last":0.46600000000000003}]},{"osd":2,"last update":"Tue Sep 10 18:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":1.002,"15min":1.2150000000000001},"min":{"1min":0.214,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":3.5009999999999999,"5min":5.8380000000000001,"15min":13.035},"last":1.514},{"interface":"front","average":{"1min":0.81999999999999995,"5min":1.012,"15min":1.171},"min":{"1min":0.26600000000000001,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":3.669,"5min":4.0670000000000002,"15min":12.222},"last":1.556}]},{"osd":3,"last update":"Tue Sep 10 18:36:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":1.161,"15min":1.6419999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.22600000000000001},"max":{"1min":1.7470000000000001,"5min":8.0399999999999991,"15min":43.076000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":1.0920000000000001,"15min":1.6839999999999999},"min":{"1min":0.27300000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69799999999999995,"5min":5.8879999999999999,"15min":41.93},"last":0.39900000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":1.085,"15min":1.107},"min":{"1min":0.36799999999999999,"5min":0.312,"15min":0.29799999999999999},"max":{"1min":0.99299999999999999,"5min":5.9930000000000003,"15min":7.2859999999999996},"last":0.435},{"interface":"front","average":{"1min":0.746,"5min":1.224,"15min":1.26},"min":{"1min":0.35199999999999998,"5min":0.30299999999999999,"15min":0.28299999999999997},"max":{"1min":2.8290000000000002,"5min":7.8769999999999998,"15min":19.393000000000001},"last":0.41999999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.2929999999999999,"15min":1.2969999999999999},"min":{"1min":0.34799999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.8740000000000001,"5min":7.2110000000000003,"15min":15.48},"last":0.308},{"interface":"front","average":{"1min":1.032,"5min":1.4350000000000001,"15min":1.236},"min":{"1min":0.39100000000000001,"5min":0.36199999999999999,"15min":0.308},"max":{"1min":3.7999999999999998,"5min":11.414999999999999,"15min":11.414999999999999},"last":0.35099999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:35:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.1359999999999999,"15min":1.2729999999999999},"min":{"1min":0.39400000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.823,"5min":8.1340000000000003,"15min":10.039},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.92800000000000005,"5min":1.246,"15min":1.359},"min":{"1min":0.40400000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.6560000000000001,"5min":6.2670000000000003,"15min":7.3369999999999997},"last":0.54800000000000004}]},{"osd":7,"last update":"Tue Sep 10 18:35:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.3109999999999999,"15min":1.2170000000000001},"min":{"1min":0.43099999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.8450000000000002,"5min":8.3650000000000002,"15min":15.545999999999999},"last":0.378},{"interface":"front","average":{"1min":1.026,"5min":1.4319999999999999,"15min":1.218},"min":{"1min":0.44900000000000001,"5min":0.38600000000000001,"15min":0.313},"max":{"1min":3.6349999999999998,"5min":6.3049999999999997,"15min":7.359},"last":0.51000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738587,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2272896,"kb_used_data":1674680,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":91472256,"statfs":{"total":95995035648,"available":93667590144,"internally_reserved":0,"allocated":1714872320,"data_stored":1714461355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:36:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":2.5800000000000001,"15min":1.956},"min":{"1min":0.221,"5min":0.185,"15min":0.185},"max":{"1min":4.2699999999999996,"5min":146.44499999999999,"15min":146.44499999999999},"last":5.9640000000000004},{"interface":"front","average":{"1min":0.498,"5min":2.5640000000000001,"15min":2.0579999999999998},"min":{"1min":0.23799999999999999,"5min":0.23100000000000001,"15min":0.17999999999999999},"max":{"1min":1.869,"5min":148.77199999999999,"15min":148.77199999999999},"last":0.29799999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:35:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":2.6579999999999999,"15min":1.8759999999999999},"min":{"1min":0.214,"5min":0.19700000000000001,"15min":0.189},"max":{"1min":3.5449999999999999,"5min":146.56999999999999,"15min":146.56999999999999},"last":0.372},{"interface":"front","average":{"1min":0.45400000000000001,"5min":2.5259999999999998,"15min":1.8520000000000001},"min":{"1min":0.24199999999999999,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":0.91300000000000003,"5min":146.601,"15min":146.601},"last":0.621}]},{"osd":3,"last update":"Tue Sep 10 18:36:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":2.3700000000000001,"15min":1.921},"min":{"1min":0.27100000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.2570000000000001,"5min":140.68299999999999,"15min":140.68299999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":2.5259999999999998,"15min":1.9430000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.23899999999999999},"max":{"1min":4.2320000000000002,"5min":140.572,"15min":140.572},"last":6.375}]},{"osd":4,"last update":"Tue Sep 10 18:35:40 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.98699999999999999,"15min":1.1499999999999999},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.246},"max":{"1min":4.2939999999999996,"5min":4.2939999999999996,"15min":17.151},"last":6.1970000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.94099999999999995,"15min":1.125},"min":{"1min":0.33400000000000002,"5min":0.29899999999999999,"15min":0.25900000000000001},"max":{"1min":2.0939999999999999,"5min":3.7919999999999998,"15min":18.728999999999999},"last":0.64000000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:36:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.997,"15min":1.2849999999999999},"min":{"1min":0.34499999999999997,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.54,"5min":8.1259999999999994,"15min":18.77},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.65200000000000002,"5min":1.2030000000000001,"15min":1.2589999999999999},"min":{"1min":0.41199999999999998,"5min":0.32400000000000001,"15min":0.29099999999999998},"max":{"1min":1.99,"5min":20.460000000000001,"15min":20.460000000000001},"last":0.45700000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:35:40 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.0760000000000001,"15min":1.2090000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.30499999999999999},"max":{"1min":4.3179999999999996,"5min":4.3179999999999996,"15min":8.3940000000000001},"last":6.3330000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.99399999999999999,"15min":1.256},"min":{"1min":0.375,"5min":0.34399999999999997,"15min":0.33300000000000002},"max":{"1min":1.7270000000000001,"5min":3.9079999999999999,"15min":9.7859999999999996},"last":0.67000000000000004}]},{"osd":7,"last update":"Tue Sep 10 18:35:40 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":1.2869999999999999,"15min":1.2430000000000001},"min":{"1min":0.42099999999999999,"5min":0.32200000000000001,"15min":0.26700000000000002},"max":{"1min":2.5529999999999999,"5min":10.75,"15min":13.711},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.0760000000000001,"15min":1.179},"min":{"1min":0.437,"5min":0.378,"15min":0.33500000000000002},"max":{"1min":2.0379999999999998,"5min":11.218999999999999,"15min":15.032},"last":0.71499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1764360192,"data_stored":1764360192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":622133248,"data_stored":622133248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1217265664,"data_stored":1217265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1049821184,"data_stored":1049821184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1619001344,"data_stored":1619001344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1011679232,"data_stored":1011679232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1460142080,"data_stored":1460142080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":805830656,"data_stored":805830656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:36:44.552 INFO:tasks.thrashosds.thrasher:Setting 5.1f to [0, 2, 7] 2024-09-10T18:36:44.552 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.1f', '0', '2', '7'] 2024-09-10T18:36:44.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pg-upmap 5.1f 0 2 7 2024-09-10T18:36:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:44 smithi204 ceph-mon[32357]: from='client.25606 v1:172.21.15.28:0/2086113865' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:36:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:44 smithi204 ceph-mon[32357]: pgmap v687: 178 pgs: 178 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-09-10T18:36:44.609 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:44.644 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:44.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:44.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:44.798 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:44.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:44.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:44.901 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:44.934 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:44.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:44.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:45.018 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 26 16 57606 57590 138.424 101.25 0.00960265 0.00721953 2024-09-10T18:36:45.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:45.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:45.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:45.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:36:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:36:45.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:45.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:45.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:45.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:45.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:45.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:45.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:45.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:45.531 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:45.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:45.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:45.633 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:45.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:45.673 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:45.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:45.774 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:45.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:45.807 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:45.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:45.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:45.952 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:45.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:45.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:46.018 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 27 16 59173 59157 136.924 97.9375 0.011327 0.00729846 2024-09-10T18:36:46.053 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:46.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:46.057 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:46.083 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:46.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:46.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:46.217 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:46.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:46.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:46.319 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:46.349 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:46.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:46.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:46.380 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:46 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:46.380 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:46 smithi028 ceph-mon[39220]: pgmap v688: 178 pgs: 178 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-09-10T18:36:46.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:46 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:46.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:46 smithi028 ceph-mon[29291]: pgmap v688: 178 pgs: 178 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-09-10T18:36:46.451 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:46.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:46.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:46.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:46.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:46 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:46 smithi204 ceph-mon[32357]: pgmap v688: 178 pgs: 178 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-09-10T18:36:46.624 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:46.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:46.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:46.769 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:46.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:46.870 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:46.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:46.907 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:46.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:47.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:47.018 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 28 16 60707 60691 135.458 95.875 0.00715092 0.00737726 2024-09-10T18:36:47.049 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:47.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:47.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:47.151 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:47.180 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:47.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:47.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:47.215 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:36:47.272 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:47 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2198727986' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:36:47.272 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:47 smithi028 ceph-mon[39220]: from='client.25618 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:36:47.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:47 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2198727986' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:36:47.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:47 smithi028 ceph-mon[29291]: from='client.25618 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:36:47.282 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:47.315 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:47.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:47.417 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:47.451 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:47.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:47.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:47.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:36:47.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:47.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:47.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:47.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:47 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2198727986' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:36:47.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:47 smithi204 ceph-mon[32357]: from='client.25618 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:36:47.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:47.747 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:47.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:47.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:47.849 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:47.882 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:47.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:47.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:47.984 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:48.015 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:48.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:48.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:48.018 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 29 16 62211 62195 134.028 94 0.00793131 0.00745632 2024-09-10T18:36:48.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:48.154 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:48.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:48 smithi028 ceph-mon[39220]: pgmap v689: 178 pgs: 178 active+clean; 3.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 935 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-09-10T18:36:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:48 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:36:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:48 smithi028 ceph-mon[39220]: from='client.25618 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:36:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:48 smithi028 ceph-mon[39220]: osdmap e323: 8 total, 8 up, 8 in 2024-09-10T18:36:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:48 smithi028 ceph-mon[29291]: pgmap v689: 178 pgs: 178 active+clean; 3.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 935 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-09-10T18:36:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:48 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:36:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:48 smithi028 ceph-mon[29291]: from='client.25618 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:36:48.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:48 smithi028 ceph-mon[29291]: osdmap e323: 8 total, 8 up, 8 in 2024-09-10T18:36:48.256 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:48.294 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:48.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:48.396 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:48.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:48.430 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:48.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:48.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:48.564 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:48.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:48.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:48 smithi204 ceph-mon[32357]: pgmap v689: 178 pgs: 178 active+clean; 3.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 935 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-09-10T18:36:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:48 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:36:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:48 smithi204 ceph-mon[32357]: from='client.25618 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:36:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:48 smithi204 ceph-mon[32357]: osdmap e323: 8 total, 8 up, 8 in 2024-09-10T18:36:48.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:48.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:48.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:48.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:48.848 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:48.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:48.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:48.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:49.011 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:49.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:49.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:49.018 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 30 16 63775 63759 132.818 97.75 0.0216089 0.00752445 2024-09-10T18:36:49.114 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:49.150 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:49.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:49.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:49.252 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:49.283 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:49.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:49.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:49.385 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:49.420 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:49.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:49.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:49.454 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:49.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:49 smithi028 ceph-mon[29291]: 4.1f scrub starts 2024-09-10T18:36:49.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:49 smithi028 ceph-mon[29291]: 4.1f scrub ok 2024-09-10T18:36:49.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:49 smithi028 ceph-mon[39220]: 4.1f scrub starts 2024-09-10T18:36:49.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:49 smithi028 ceph-mon[39220]: 4.1f scrub ok 2024-09-10T18:36:49.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:49.561 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:49.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:49.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:49 smithi204 ceph-mon[32357]: 4.1f scrub starts 2024-09-10T18:36:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:49 smithi204 ceph-mon[32357]: 4.1f scrub ok 2024-09-10T18:36:49.662 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:49.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:49.697 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:49.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:49.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:49.836 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:49.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:49.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:49.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:49.980 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:49.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:49.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:50.018 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 31 16 65231 65215 131.469 91 0.00606604 0.00760177 2024-09-10T18:36:50.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:50.119 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:50.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:50.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:50.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:50.259 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:50.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:50.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:50.360 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:50.392 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:50.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:50.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:50.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:36:50.494 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:50.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:50 smithi028 ceph-mon[39220]: 5.6 scrub starts 2024-09-10T18:36:50.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:50 smithi028 ceph-mon[39220]: 5.6 scrub ok 2024-09-10T18:36:50.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:50 smithi028 ceph-mon[39220]: 1.1c scrub starts 2024-09-10T18:36:50.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:50 smithi028 ceph-mon[39220]: 1.1c scrub ok 2024-09-10T18:36:50.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:50 smithi028 ceph-mon[39220]: pgmap v691: 178 pgs: 178 active+clean; 3.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-10T18:36:50.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:50 smithi028 ceph-mon[29291]: 5.6 scrub starts 2024-09-10T18:36:50.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:50 smithi028 ceph-mon[29291]: 5.6 scrub ok 2024-09-10T18:36:50.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:50 smithi028 ceph-mon[29291]: 1.1c scrub starts 2024-09-10T18:36:50.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:50 smithi028 ceph-mon[29291]: 1.1c scrub ok 2024-09-10T18:36:50.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:50 smithi028 ceph-mon[29291]: pgmap v691: 178 pgs: 178 active+clean; 3.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-10T18:36:50.519 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:50.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:50.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:50 smithi204 ceph-mon[32357]: 5.6 scrub starts 2024-09-10T18:36:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:50 smithi204 ceph-mon[32357]: 5.6 scrub ok 2024-09-10T18:36:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:50 smithi204 ceph-mon[32357]: 1.1c scrub starts 2024-09-10T18:36:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:50 smithi204 ceph-mon[32357]: 1.1c scrub ok 2024-09-10T18:36:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:50 smithi204 ceph-mon[32357]: pgmap v691: 178 pgs: 178 active+clean; 3.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-10T18:36:50.621 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:50.647 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:50.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:50.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:50.749 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:50.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:50.802 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:50.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:50.903 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:50.935 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:50.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:50.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:51.019 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 32 16 66543 66527 129.921 82 0.0149851 0.00769161 2024-09-10T18:36:51.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:51.071 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:51.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:51.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:51.119 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:36:51.119 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:36:51.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:51.119 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:66674 10.4 e323/314 10:25a82a6e:::benchmark_data_smithi028_161947_object66673:head v 323'4102, mlcod=323'4099)", 2024-09-10T18:36:51.119 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:36:51.114038+0000", 2024-09-10T18:36:51.119 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0018671149999999999, 2024-09-10T18:36:51.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018992379999999999, 2024-09-10T18:36:51.119 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:36:51.119 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:36:51.119 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:36:51.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:51.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:36:51.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:36:51.114038+0000", 2024-09-10T18:36:51.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:36:51.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:51.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:51.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:36:51.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:36:51.114038+0000", 2024-09-10T18:36:51.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:36:51.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:51.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:51.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:36:51.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:36:51.114036+0000", 2024-09-10T18:36:51.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:36:51.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:51.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:51.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:36:51.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:36:51.114062+0000", 2024-09-10T18:36:51.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6035000000000001e-05 2024-09-10T18:36:51.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:51.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:51.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:36:51.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:36:51.114077+0000", 2024-09-10T18:36:51.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5316e-05 2024-09-10T18:36:51.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:51.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:51.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:36:51.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:36:51.114083+0000", 2024-09-10T18:36:51.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7450000000000001e-06 2024-09-10T18:36:51.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:51.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:51.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:36:51.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:36:51.114226+0000", 2024-09-10T18:36:51.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000142352 2024-09-10T18:36:51.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:36:51.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:36:51.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:36:51.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:36:51.114275+0000", 2024-09-10T18:36:51.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9214e-05 2024-09-10T18:36:51.122 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:51.122 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:36:51.122 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:51.123 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:36:51.123 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:36:51.123 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 1 2024-09-10T18:36:51.123 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:36:51.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:51.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:51.219 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:51.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:51.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:51.357 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:51.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:51.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:51.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:51.498 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:51.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:51.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:51.503 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:36:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:36:51] "GET /metrics HTTP/1.1" 200 224652 "" "Prometheus/2.43.0" 2024-09-10T18:36:51.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:51 smithi028 ceph-mon[39220]: 3.17 scrub starts 2024-09-10T18:36:51.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:51 smithi028 ceph-mon[39220]: 3.17 scrub ok 2024-09-10T18:36:51.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:51 smithi028 ceph-mon[39220]: 4.7 scrub starts 2024-09-10T18:36:51.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:51 smithi028 ceph-mon[39220]: 4.7 scrub ok 2024-09-10T18:36:51.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:51 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3496804407' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1f", "id": [0, 2, 7]}]: dispatch 2024-09-10T18:36:51.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:51 smithi028 ceph-mon[39220]: from='client.35639 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1f", "id": [0, 2, 7]}]: dispatch 2024-09-10T18:36:51.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:51 smithi028 ceph-mon[29291]: 3.17 scrub starts 2024-09-10T18:36:51.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:51 smithi028 ceph-mon[29291]: 3.17 scrub ok 2024-09-10T18:36:51.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:51 smithi028 ceph-mon[29291]: 4.7 scrub starts 2024-09-10T18:36:51.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:51 smithi028 ceph-mon[29291]: 4.7 scrub ok 2024-09-10T18:36:51.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:51 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3496804407' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1f", "id": [0, 2, 7]}]: dispatch 2024-09-10T18:36:51.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:51 smithi028 ceph-mon[29291]: from='client.35639 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1f", "id": [0, 2, 7]}]: dispatch 2024-09-10T18:36:51.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:51 smithi204 ceph-mon[32357]: 3.17 scrub starts 2024-09-10T18:36:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:51 smithi204 ceph-mon[32357]: 3.17 scrub ok 2024-09-10T18:36:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:51 smithi204 ceph-mon[32357]: 4.7 scrub starts 2024-09-10T18:36:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:51 smithi204 ceph-mon[32357]: 4.7 scrub ok 2024-09-10T18:36:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:51 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3496804407' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1f", "id": [0, 2, 7]}]: dispatch 2024-09-10T18:36:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:51 smithi204 ceph-mon[32357]: from='client.35639 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1f", "id": [0, 2, 7]}]: dispatch 2024-09-10T18:36:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:51.645 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:51.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:51.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:51.778 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:51.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:51.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:51.855 INFO:teuthology.orchestra.run.smithi028.stderr:set 5.1f pg_upmap mapping to [0,2,7] 2024-09-10T18:36:51.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:51.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:51.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:52.019 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 33 16 67996 67980 128.736 90.8125 0.0111202 0.0077622 2024-09-10T18:36:52.023 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:52.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:52.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:52.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:52.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:52.200 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:52.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:52.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:52.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:52.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:52.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:52.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:52.492 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:52.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:52 smithi028 ceph-mon[29291]: 3.e scrub starts 2024-09-10T18:36:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:52 smithi028 ceph-mon[29291]: 3.e scrub ok 2024-09-10T18:36:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:52 smithi028 ceph-mon[29291]: pgmap v692: 178 pgs: 178 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-10T18:36:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:52 smithi028 ceph-mon[29291]: from='client.35639 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1f", "id": [0, 2, 7]}]': finished 2024-09-10T18:36:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:52 smithi028 ceph-mon[29291]: osdmap e324: 8 total, 8 up, 8 in 2024-09-10T18:36:52.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:52 smithi028 ceph-mon[39220]: 3.e scrub starts 2024-09-10T18:36:52.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:52 smithi028 ceph-mon[39220]: 3.e scrub ok 2024-09-10T18:36:52.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:52 smithi028 ceph-mon[39220]: pgmap v692: 178 pgs: 178 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-10T18:36:52.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:52 smithi028 ceph-mon[39220]: from='client.35639 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1f", "id": [0, 2, 7]}]': finished 2024-09-10T18:36:52.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:52 smithi028 ceph-mon[39220]: osdmap e324: 8 total, 8 up, 8 in 2024-09-10T18:36:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:52 smithi204 ceph-mon[32357]: 3.e scrub starts 2024-09-10T18:36:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:52 smithi204 ceph-mon[32357]: 3.e scrub ok 2024-09-10T18:36:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:52 smithi204 ceph-mon[32357]: pgmap v692: 178 pgs: 178 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-10T18:36:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:52 smithi204 ceph-mon[32357]: from='client.35639 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1f", "id": [0, 2, 7]}]': finished 2024-09-10T18:36:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:52 smithi204 ceph-mon[32357]: osdmap e324: 8 total, 8 up, 8 in 2024-09-10T18:36:52.505 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:36:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:36:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:36:52.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:52.629 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:52.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:52.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:52.731 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:52.761 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:52.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:52.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:52.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-10T18:36:52.863 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:52.896 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:52.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:52.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:52.998 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:53.019 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 34 15 69434 69419 127.594 89.9375 0.0101801 0.00783242 2024-09-10T18:36:53.036 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:53.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:53.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:53.138 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:53.161 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:53.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:53.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:53.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:53.305 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:53.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:53.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:53.407 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:53.442 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:53.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:53.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:53.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:53 smithi028 ceph-mon[29291]: 5.3 scrub starts 2024-09-10T18:36:53.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:53 smithi028 ceph-mon[29291]: 5.3 scrub ok 2024-09-10T18:36:53.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:53 smithi028 ceph-mon[29291]: 5.1b scrub starts 2024-09-10T18:36:53.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:53 smithi028 ceph-mon[29291]: 5.1b scrub ok 2024-09-10T18:36:53.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:53 smithi028 ceph-mon[29291]: 6.1a scrub starts 2024-09-10T18:36:53.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:53 smithi028 ceph-mon[29291]: 6.1a scrub ok 2024-09-10T18:36:53.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:53 smithi028 ceph-mon[29291]: osdmap e325: 8 total, 8 up, 8 in 2024-09-10T18:36:53.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:53 smithi028 ceph-mon[39220]: 5.3 scrub starts 2024-09-10T18:36:53.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:53 smithi028 ceph-mon[39220]: 5.3 scrub ok 2024-09-10T18:36:53.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:53 smithi028 ceph-mon[39220]: 5.1b scrub starts 2024-09-10T18:36:53.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:53 smithi028 ceph-mon[39220]: 5.1b scrub ok 2024-09-10T18:36:53.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:53 smithi028 ceph-mon[39220]: 6.1a scrub starts 2024-09-10T18:36:53.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:53 smithi028 ceph-mon[39220]: 6.1a scrub ok 2024-09-10T18:36:53.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:53 smithi028 ceph-mon[39220]: osdmap e325: 8 total, 8 up, 8 in 2024-09-10T18:36:53.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:53.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:53.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:53 smithi204 ceph-mon[32357]: 5.3 scrub starts 2024-09-10T18:36:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:53 smithi204 ceph-mon[32357]: 5.3 scrub ok 2024-09-10T18:36:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:53 smithi204 ceph-mon[32357]: 5.1b scrub starts 2024-09-10T18:36:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:53 smithi204 ceph-mon[32357]: 5.1b scrub ok 2024-09-10T18:36:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:53 smithi204 ceph-mon[32357]: 6.1a scrub starts 2024-09-10T18:36:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:53 smithi204 ceph-mon[32357]: 6.1a scrub ok 2024-09-10T18:36:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:53 smithi204 ceph-mon[32357]: osdmap e325: 8 total, 8 up, 8 in 2024-09-10T18:36:53.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:53.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:53.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:53.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:53.864 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:53.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:53.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:54.009 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:54.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:54.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:54.019 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 35 16 70804 70788 126.393 85.5625 0.00701605 0.00790689 2024-09-10T18:36:54.087 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:36:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:36:54.106 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:54.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:54.123 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:36:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:36:54.152 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:36:54.152 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:36:54.185 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:36:54.185 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:36:54.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:54.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:54.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:54.222 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:36:54.222 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:36:54.255 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:36:54.255 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:36:54.287 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:36:54.288 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:36:54.288 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:54 smithi028 ceph-mon[39220]: 6.13 scrub starts 2024-09-10T18:36:54.288 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:54 smithi028 ceph-mon[39220]: 6.13 scrub ok 2024-09-10T18:36:54.288 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:54 smithi028 ceph-mon[39220]: 3.16 scrub starts 2024-09-10T18:36:54.288 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:54 smithi028 ceph-mon[39220]: 3.16 scrub ok 2024-09-10T18:36:54.289 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:54 smithi028 ceph-mon[39220]: pgmap v695: 178 pgs: 178 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-09-10T18:36:54.289 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:54 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:36:54.289 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:54 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2370694520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:36:54.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:54 smithi028 ceph-mon[29291]: 6.13 scrub starts 2024-09-10T18:36:54.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:54 smithi028 ceph-mon[29291]: 6.13 scrub ok 2024-09-10T18:36:54.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:54 smithi028 ceph-mon[29291]: 3.16 scrub starts 2024-09-10T18:36:54.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:54 smithi028 ceph-mon[29291]: 3.16 scrub ok 2024-09-10T18:36:54.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:54 smithi028 ceph-mon[29291]: pgmap v695: 178 pgs: 178 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-09-10T18:36:54.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:54 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:36:54.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:54 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2370694520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:36:54.294 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:54.321 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:36:54.321 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:36:54.326 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:54.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:54.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:54.428 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:54.456 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:54.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:54.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:54.558 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:54.591 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:54.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:54.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:54 smithi204 ceph-mon[32357]: 6.13 scrub starts 2024-09-10T18:36:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:54 smithi204 ceph-mon[32357]: 6.13 scrub ok 2024-09-10T18:36:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:54 smithi204 ceph-mon[32357]: 3.16 scrub starts 2024-09-10T18:36:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:54 smithi204 ceph-mon[32357]: 3.16 scrub ok 2024-09-10T18:36:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:54 smithi204 ceph-mon[32357]: pgmap v695: 178 pgs: 178 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-09-10T18:36:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:54 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:36:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:54 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2370694520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:36:54.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:54.722 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:54.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:54.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:54.824 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:54.855 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:54.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:54.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:54.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:54.992 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:54.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:55.019 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 36 16 72198 72182 125.302 87.125 0.0211178 0.00797605 2024-09-10T18:36:55.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:55.103 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:36:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:36:55.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:55.130 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:55.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:55.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:55.266 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:55.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:55.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:55.368 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:55.405 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:55.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:55.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:55.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:55 smithi028 ceph-mon[29291]: 6.3 scrub starts 2024-09-10T18:36:55.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:55 smithi028 ceph-mon[29291]: 6.3 scrub ok 2024-09-10T18:36:55.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:55 smithi028 ceph-mon[29291]: 4.8 scrub starts 2024-09-10T18:36:55.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:55 smithi028 ceph-mon[29291]: 4.8 scrub ok 2024-09-10T18:36:55.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:55 smithi028 ceph-mon[39220]: 6.3 scrub starts 2024-09-10T18:36:55.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:55 smithi028 ceph-mon[39220]: 6.3 scrub ok 2024-09-10T18:36:55.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:55 smithi028 ceph-mon[39220]: 4.8 scrub starts 2024-09-10T18:36:55.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:55 smithi028 ceph-mon[39220]: 4.8 scrub ok 2024-09-10T18:36:55.508 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:55.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:55.545 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:55.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:55.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:55 smithi204 ceph-mon[32357]: 6.3 scrub starts 2024-09-10T18:36:55.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:55 smithi204 ceph-mon[32357]: 6.3 scrub ok 2024-09-10T18:36:55.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:55 smithi204 ceph-mon[32357]: 4.8 scrub starts 2024-09-10T18:36:55.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:55 smithi204 ceph-mon[32357]: 4.8 scrub ok 2024-09-10T18:36:55.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:55.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:55.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:55.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:55.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:55.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:55.913 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:55.951 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:55.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:55.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:56.019 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 37 16 73525 73509 124.157 82.9375 0.0124166 0.00804922 2024-09-10T18:36:56.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:56.089 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:56.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:56.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:56.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:56.224 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:56.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:56.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:56.271 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:56.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:36:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:56.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:56.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:36:56.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[39220]: 6.18 scrub starts 2024-09-10T18:36:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[39220]: 6.18 scrub ok 2024-09-10T18:36:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[39220]: pgmap v696: 178 pgs: 178 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-10T18:36:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[39220]: 1.1 scrub starts 2024-09-10T18:36:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[39220]: 1.1 scrub ok 2024-09-10T18:36:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[39220]: osdmap e326: 8 total, 8 up, 8 in 2024-09-10T18:36:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[39220]: 3.d scrub starts 2024-09-10T18:36:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[39220]: 3.d scrub ok 2024-09-10T18:36:56.505 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:56.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[29291]: 6.18 scrub starts 2024-09-10T18:36:56.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[29291]: 6.18 scrub ok 2024-09-10T18:36:56.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:56.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[29291]: pgmap v696: 178 pgs: 178 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-10T18:36:56.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[29291]: 1.1 scrub starts 2024-09-10T18:36:56.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[29291]: 1.1 scrub ok 2024-09-10T18:36:56.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[29291]: osdmap e326: 8 total, 8 up, 8 in 2024-09-10T18:36:56.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[29291]: 3.d scrub starts 2024-09-10T18:36:56.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:56 smithi028 ceph-mon[29291]: 3.d scrub ok 2024-09-10T18:36:56.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:56.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:56 smithi204 ceph-mon[32357]: 6.18 scrub starts 2024-09-10T18:36:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:56 smithi204 ceph-mon[32357]: 6.18 scrub ok 2024-09-10T18:36:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:56 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:36:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:56 smithi204 ceph-mon[32357]: pgmap v696: 178 pgs: 178 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-10T18:36:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:56 smithi204 ceph-mon[32357]: 1.1 scrub starts 2024-09-10T18:36:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:56 smithi204 ceph-mon[32357]: 1.1 scrub ok 2024-09-10T18:36:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:56 smithi204 ceph-mon[32357]: osdmap e326: 8 total, 8 up, 8 in 2024-09-10T18:36:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:56 smithi204 ceph-mon[32357]: 3.d scrub starts 2024-09-10T18:36:56.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:56 smithi204 ceph-mon[32357]: 3.d scrub ok 2024-09-10T18:36:56.607 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:56.642 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:56.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:56.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:56.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:56.777 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:56.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:56.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:56.879 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:56.912 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:56.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:56.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:57.014 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:57.020 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 38 16 74945 74929 123.224 88.75 0.00666837 0.00811007 2024-09-10T18:36:57.046 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:57.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:57.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:57.193 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:57.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:57.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:57.295 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:57.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:57 smithi028 ceph-mon[39220]: 3.1 scrub starts 2024-09-10T18:36:57.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:57 smithi028 ceph-mon[39220]: 3.1 scrub ok 2024-09-10T18:36:57.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:57 smithi028 ceph-mon[39220]: 3.1c scrub starts 2024-09-10T18:36:57.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:57 smithi028 ceph-mon[39220]: 3.1c scrub ok 2024-09-10T18:36:57.377 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:57 smithi028 ceph-mon[39220]: 6.15 scrub starts 2024-09-10T18:36:57.377 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:57 smithi028 ceph-mon[39220]: 6.15 scrub ok 2024-09-10T18:36:57.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:57 smithi028 ceph-mon[29291]: 3.1 scrub starts 2024-09-10T18:36:57.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:57 smithi028 ceph-mon[29291]: 3.1 scrub ok 2024-09-10T18:36:57.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:57 smithi028 ceph-mon[29291]: 3.1c scrub starts 2024-09-10T18:36:57.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:57 smithi028 ceph-mon[29291]: 3.1c scrub ok 2024-09-10T18:36:57.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:57 smithi028 ceph-mon[29291]: 6.15 scrub starts 2024-09-10T18:36:57.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:57 smithi028 ceph-mon[29291]: 6.15 scrub ok 2024-09-10T18:36:57.388 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:57.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:57.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:36:57.490 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:57.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:57.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:57 smithi204 ceph-mon[32357]: 3.1 scrub starts 2024-09-10T18:36:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:57 smithi204 ceph-mon[32357]: 3.1 scrub ok 2024-09-10T18:36:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:57 smithi204 ceph-mon[32357]: 3.1c scrub starts 2024-09-10T18:36:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:57 smithi204 ceph-mon[32357]: 3.1c scrub ok 2024-09-10T18:36:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:57 smithi204 ceph-mon[32357]: 6.15 scrub starts 2024-09-10T18:36:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:57 smithi204 ceph-mon[32357]: 6.15 scrub ok 2024-09-10T18:36:57.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:36:57.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:57.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:57.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:36:57.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:57.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:57.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:57.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:57.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:57.958 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:57.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:57.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:58.020 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 39 16 76261 76245 122.173 82.25 0.00439914 0.00817998 2024-09-10T18:36:58.059 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:58.095 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:58.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:58.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:58.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:36:58.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:58.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:58.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:36:58.333 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:58.372 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:58.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:58.474 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:58 smithi028 ceph-mon[29291]: 5.a scrub starts 2024-09-10T18:36:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:58 smithi028 ceph-mon[29291]: 5.a scrub ok 2024-09-10T18:36:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:58 smithi028 ceph-mon[29291]: pgmap v698: 178 pgs: 178 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.32k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:36:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:58 smithi028 ceph-mon[29291]: 5.4 scrub starts 2024-09-10T18:36:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:58 smithi028 ceph-mon[29291]: 5.4 scrub ok 2024-09-10T18:36:58.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:58 smithi028 ceph-mon[39220]: 5.a scrub starts 2024-09-10T18:36:58.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:58 smithi028 ceph-mon[39220]: 5.a scrub ok 2024-09-10T18:36:58.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:58 smithi028 ceph-mon[39220]: pgmap v698: 178 pgs: 178 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.32k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:36:58.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:58 smithi028 ceph-mon[39220]: 5.4 scrub starts 2024-09-10T18:36:58.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:58 smithi028 ceph-mon[39220]: 5.4 scrub ok 2024-09-10T18:36:58.506 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:58.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:58.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:58 smithi204 ceph-mon[32357]: 5.a scrub starts 2024-09-10T18:36:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:58 smithi204 ceph-mon[32357]: 5.a scrub ok 2024-09-10T18:36:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:58 smithi204 ceph-mon[32357]: pgmap v698: 178 pgs: 178 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.32k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:36:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:58 smithi204 ceph-mon[32357]: 5.4 scrub starts 2024-09-10T18:36:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:58 smithi204 ceph-mon[32357]: 5.4 scrub ok 2024-09-10T18:36:58.609 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:58.645 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:58.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:58.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:58.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:58.776 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:36:58.776 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:36:58.777 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-10T18:36:58.777 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-10T18:36:58.777 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-10T18:36:58.777 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-10T18:36:58.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:58.799 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:58.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:36:58.900 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:58.945 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:58.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:59.020 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:36:59.019638+0000 min lat: 0.00134649 max lat: 0.229265 avg lat: 0.00824973 2024-09-10T18:36:59.020 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:36:59.020 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 40 15 77562 77547 121.153 81.375 0.0124817 0.00824973 2024-09-10T18:36:59.046 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:59.076 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:59.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:59.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:59.169 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:36:59.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:36:59.216 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:59.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:59.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:36:59.261 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:36:59.271 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:36:59.312 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:36:59.312 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:36:59.318 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:36:59.353 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:59.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:59.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:36:59.455 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:36:59.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:59.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:59.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:36:59.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:59 smithi028 ceph-mon[39220]: 5.c scrub starts 2024-09-10T18:36:59.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:59 smithi028 ceph-mon[39220]: 5.c scrub ok 2024-09-10T18:36:59.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:59 smithi028 ceph-mon[39220]: 1.10 scrub starts 2024-09-10T18:36:59.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:59 smithi028 ceph-mon[39220]: 1.10 scrub ok 2024-09-10T18:36:59.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:59 smithi028 ceph-mon[39220]: 4.e scrub starts 2024-09-10T18:36:59.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:59 smithi028 ceph-mon[39220]: 4.e scrub ok 2024-09-10T18:36:59.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:59 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1276953148' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:36:59.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:36:59 smithi028 ceph-mon[39220]: from='client.25633 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:36:59.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:59 smithi028 ceph-mon[29291]: 5.c scrub starts 2024-09-10T18:36:59.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:59 smithi028 ceph-mon[29291]: 5.c scrub ok 2024-09-10T18:36:59.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:59 smithi028 ceph-mon[29291]: 1.10 scrub starts 2024-09-10T18:36:59.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:59 smithi028 ceph-mon[29291]: 1.10 scrub ok 2024-09-10T18:36:59.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:59 smithi028 ceph-mon[29291]: 4.e scrub starts 2024-09-10T18:36:59.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:59 smithi028 ceph-mon[29291]: 4.e scrub ok 2024-09-10T18:36:59.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:59 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1276953148' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:36:59.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:36:59 smithi028 ceph-mon[29291]: from='client.25633 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:36:59.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:59 smithi204 ceph-mon[32357]: 5.c scrub starts 2024-09-10T18:36:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:59 smithi204 ceph-mon[32357]: 5.c scrub ok 2024-09-10T18:36:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:59 smithi204 ceph-mon[32357]: 1.10 scrub starts 2024-09-10T18:36:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:59 smithi204 ceph-mon[32357]: 1.10 scrub ok 2024-09-10T18:36:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:59 smithi204 ceph-mon[32357]: 4.e scrub starts 2024-09-10T18:36:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:59 smithi204 ceph-mon[32357]: 4.e scrub ok 2024-09-10T18:36:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:59 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1276953148' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:36:59.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:36:59 smithi204 ceph-mon[32357]: from='client.25633 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:36:59.615 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:59.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:59.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:59.717 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:36:59.753 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:36:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:59.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:36:59.854 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:36:59.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:36:59.898 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:36:59.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:00.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:00.020 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 41 16 78750 78734 120.007 74.1875 0.0249606 0.00832801 2024-09-10T18:37:00.034 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:00.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:00.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:00.136 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:00.174 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:00.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:00.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:00.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:00.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:00.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:00.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:00.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:00.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:00.456 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:00.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:00.557 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:00.587 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:00.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:00.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:00 smithi204 ceph-mon[32357]: pgmap v699: 178 pgs: 178 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 707 B/s rd, 60 MiB/s wr, 965 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:37:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:00 smithi204 ceph-mon[32357]: 6.7 scrub starts 2024-09-10T18:37:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:00 smithi204 ceph-mon[32357]: 6.7 scrub ok 2024-09-10T18:37:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:00 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:37:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:00 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:37:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:00 smithi204 ceph-mon[32357]: from='client.25633 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:37:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:00 smithi204 ceph-mon[32357]: osdmap e327: 8 total, 8 up, 8 in 2024-09-10T18:37:00.689 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:00.725 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:00.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:00.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:00.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:00 smithi028 ceph-mon[29291]: pgmap v699: 178 pgs: 178 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 707 B/s rd, 60 MiB/s wr, 965 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:37:00.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:00 smithi028 ceph-mon[29291]: 6.7 scrub starts 2024-09-10T18:37:00.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:00 smithi028 ceph-mon[29291]: 6.7 scrub ok 2024-09-10T18:37:00.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:00 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:37:00.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:00 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:37:00.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:00 smithi028 ceph-mon[29291]: from='client.25633 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:37:00.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:00 smithi028 ceph-mon[29291]: osdmap e327: 8 total, 8 up, 8 in 2024-09-10T18:37:00.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:00 smithi028 ceph-mon[39220]: pgmap v699: 178 pgs: 178 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 707 B/s rd, 60 MiB/s wr, 965 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:37:00.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:00 smithi028 ceph-mon[39220]: 6.7 scrub starts 2024-09-10T18:37:00.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:00 smithi028 ceph-mon[39220]: 6.7 scrub ok 2024-09-10T18:37:00.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:00 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:37:00.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:00 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:37:00.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:00 smithi028 ceph-mon[39220]: from='client.25633 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:37:00.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:00 smithi028 ceph-mon[39220]: osdmap e327: 8 total, 8 up, 8 in 2024-09-10T18:37:00.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:00.874 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:00.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:00.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-10T18:37:00.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:01.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:01.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:01.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:01.020 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 42 16 80094 80078 119.149 84 0.0116683 0.00838781 2024-09-10T18:37:01.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:01.137 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:01.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:01.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:01.274 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:01 smithi204 ceph-mon[32357]: 4.3 scrub starts 2024-09-10T18:37:01.274 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:01 smithi204 ceph-mon[32357]: 4.3 scrub ok 2024-09-10T18:37:01.274 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:01 smithi204 ceph-mon[32357]: 1.3 scrub starts 2024-09-10T18:37:01.274 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:01 smithi204 ceph-mon[32357]: 1.3 scrub ok 2024-09-10T18:37:01.275 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:01 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:37:01.275 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:01 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:37:01.275 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:01 smithi204 ceph-mon[32357]: 3.9 scrub starts 2024-09-10T18:37:01.275 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:01 smithi204 ceph-mon[32357]: 3.9 scrub ok 2024-09-10T18:37:01.275 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:01 smithi204 ceph-mon[32357]: osdmap e328: 8 total, 8 up, 8 in 2024-09-10T18:37:01.275 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:01 smithi204 ceph-mon[32357]: osdmap e329: 8 total, 8 up, 8 in 2024-09-10T18:37:01.275 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:01 smithi204 ceph-mon[32357]: 2.0 scrub starts 2024-09-10T18:37:01.275 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:01 smithi204 ceph-mon[32357]: 2.0 scrub ok 2024-09-10T18:37:01.279 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:01.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:01.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:01.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:01.424 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:01.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:01.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:01.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:01.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:01.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:01.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:01.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:01.714 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:01.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:01.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:01.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:37:01] "GET /metrics HTTP/1.1" 200 224684 "" "Prometheus/2.43.0" 2024-09-10T18:37:01.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[39220]: 4.3 scrub starts 2024-09-10T18:37:01.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[39220]: 4.3 scrub ok 2024-09-10T18:37:01.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[39220]: 1.3 scrub starts 2024-09-10T18:37:01.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[39220]: 1.3 scrub ok 2024-09-10T18:37:01.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:37:01.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:37:01.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[39220]: 3.9 scrub starts 2024-09-10T18:37:01.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[39220]: 3.9 scrub ok 2024-09-10T18:37:01.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[39220]: osdmap e328: 8 total, 8 up, 8 in 2024-09-10T18:37:01.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[39220]: osdmap e329: 8 total, 8 up, 8 in 2024-09-10T18:37:01.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[39220]: 2.0 scrub starts 2024-09-10T18:37:01.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[39220]: 2.0 scrub ok 2024-09-10T18:37:01.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[29291]: 4.3 scrub starts 2024-09-10T18:37:01.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[29291]: 4.3 scrub ok 2024-09-10T18:37:01.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[29291]: 1.3 scrub starts 2024-09-10T18:37:01.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[29291]: 1.3 scrub ok 2024-09-10T18:37:01.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:37:01.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:37:01.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[29291]: 3.9 scrub starts 2024-09-10T18:37:01.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[29291]: 3.9 scrub ok 2024-09-10T18:37:01.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[29291]: osdmap e328: 8 total, 8 up, 8 in 2024-09-10T18:37:01.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[29291]: osdmap e329: 8 total, 8 up, 8 in 2024-09-10T18:37:01.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[29291]: 2.0 scrub starts 2024-09-10T18:37:01.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:01 smithi028 ceph-mon[29291]: 2.0 scrub ok 2024-09-10T18:37:01.815 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:01.850 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:01.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:01.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:01.952 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:01.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:01.987 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:01.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:02.020 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 43 16 81375 81359 118.24 80.0625 0.00346529 0.00845202 2024-09-10T18:37:02.088 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:02.116 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:02.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:02.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:02.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:02.252 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:02.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:02.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:02.354 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:02.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:02.391 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:02.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:02.430 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:02 smithi204 ceph-mon[32357]: 4.1e scrub starts 2024-09-10T18:37:02.430 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:02 smithi204 ceph-mon[32357]: 4.1e scrub ok 2024-09-10T18:37:02.430 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:02 smithi204 ceph-mon[32357]: pgmap v703: 178 pgs: 178 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 976 B/s rd, 111 MiB/s wr, 1.77k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:37:02.430 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:02 smithi204 ceph-mon[32357]: 5.7 deep-scrub starts 2024-09-10T18:37:02.431 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:02 smithi204 ceph-mon[32357]: 5.7 deep-scrub ok 2024-09-10T18:37:02.431 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:02 smithi204 ceph-mon[32357]: 3.3 deep-scrub starts 2024-09-10T18:37:02.431 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:02 smithi204 ceph-mon[32357]: 3.3 deep-scrub ok 2024-09-10T18:37:02.431 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:02 smithi204 ceph-mon[32357]: 6.5 scrub starts 2024-09-10T18:37:02.431 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:02 smithi204 ceph-mon[32357]: 6.5 scrub ok 2024-09-10T18:37:02.431 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:37:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:37:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:37:02.492 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[39220]: 4.1e scrub starts 2024-09-10T18:37:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[39220]: 4.1e scrub ok 2024-09-10T18:37:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[39220]: pgmap v703: 178 pgs: 178 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 976 B/s rd, 111 MiB/s wr, 1.77k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:37:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[39220]: 5.7 deep-scrub starts 2024-09-10T18:37:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[39220]: 5.7 deep-scrub ok 2024-09-10T18:37:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[39220]: 3.3 deep-scrub starts 2024-09-10T18:37:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[39220]: 3.3 deep-scrub ok 2024-09-10T18:37:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[39220]: 6.5 scrub starts 2024-09-10T18:37:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[39220]: 6.5 scrub ok 2024-09-10T18:37:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[29291]: 4.1e scrub starts 2024-09-10T18:37:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[29291]: 4.1e scrub ok 2024-09-10T18:37:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[29291]: pgmap v703: 178 pgs: 178 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 976 B/s rd, 111 MiB/s wr, 1.77k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:37:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[29291]: 5.7 deep-scrub starts 2024-09-10T18:37:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[29291]: 5.7 deep-scrub ok 2024-09-10T18:37:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[29291]: 3.3 deep-scrub starts 2024-09-10T18:37:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[29291]: 3.3 deep-scrub ok 2024-09-10T18:37:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[29291]: 6.5 scrub starts 2024-09-10T18:37:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:02 smithi028 ceph-mon[29291]: 6.5 scrub ok 2024-09-10T18:37:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:02.526 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:02.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:02.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:02.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:02.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:02.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:02.766 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:02.794 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:02.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:02.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:02.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:02.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:02.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:02.996 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:37:02.996 INFO:teuthology.orchestra.run.smithi204.stdout: "success": "" 2024-09-10T18:37:02.996 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:37:03.021 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 44 16 82525 82509 117.185 71.875 0.0157371 0.00852848 2024-09-10T18:37:03.048 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:03.083 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:03.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:03.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:03.218 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:03.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:03.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:03.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:03.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:03.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:03.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:03.488 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:03.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:03.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:03.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:03 smithi204 ceph-mon[32357]: 3.14 scrub starts 2024-09-10T18:37:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:03 smithi204 ceph-mon[32357]: 3.14 scrub ok 2024-09-10T18:37:03.627 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:03.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:03.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:03.728 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:03.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:03 smithi028 ceph-mon[39220]: 3.14 scrub starts 2024-09-10T18:37:03.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:03 smithi028 ceph-mon[39220]: 3.14 scrub ok 2024-09-10T18:37:03.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:03 smithi028 ceph-mon[29291]: 3.14 scrub starts 2024-09-10T18:37:03.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:03 smithi028 ceph-mon[29291]: 3.14 scrub ok 2024-09-10T18:37:03.763 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:03.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:03.865 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:03.910 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:03.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:03.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:04.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:04.021 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 45 16 83691 83675 116.2 72.875 0.00614279 0.00859498 2024-09-10T18:37:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:04.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:04.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:04.164 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:04.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:04.265 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:04.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:04.304 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:04.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:04.405 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:04.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:04.442 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:04.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:37:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:37:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[39220]: 6.17 scrub starts 2024-09-10T18:37:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[39220]: 6.17 scrub ok 2024-09-10T18:37:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[39220]: pgmap v704: 178 pgs: 178 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-10T18:37:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[39220]: 5.15 scrub starts 2024-09-10T18:37:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[39220]: 5.15 scrub ok 2024-09-10T18:37:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[39220]: osdmap e330: 8 total, 8 up, 8 in 2024-09-10T18:37:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[39220]: 5.e scrub starts 2024-09-10T18:37:04.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[39220]: 5.e scrub ok 2024-09-10T18:37:04.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[39220]: 6.8 deep-scrub starts 2024-09-10T18:37:04.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[39220]: 6.8 deep-scrub ok 2024-09-10T18:37:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:37:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:37:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[29291]: 6.17 scrub starts 2024-09-10T18:37:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[29291]: 6.17 scrub ok 2024-09-10T18:37:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[29291]: pgmap v704: 178 pgs: 178 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-10T18:37:04.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[29291]: 5.15 scrub starts 2024-09-10T18:37:04.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[29291]: 5.15 scrub ok 2024-09-10T18:37:04.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[29291]: osdmap e330: 8 total, 8 up, 8 in 2024-09-10T18:37:04.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[29291]: 5.e scrub starts 2024-09-10T18:37:04.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[29291]: 5.e scrub ok 2024-09-10T18:37:04.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[29291]: 6.8 deep-scrub starts 2024-09-10T18:37:04.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:04 smithi028 ceph-mon[29291]: 6.8 deep-scrub ok 2024-09-10T18:37:04.543 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:04.582 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:04.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:04.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:04 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:37:04.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:04 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:37:04.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:04 smithi204 ceph-mon[32357]: 6.17 scrub starts 2024-09-10T18:37:04.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:04 smithi204 ceph-mon[32357]: 6.17 scrub ok 2024-09-10T18:37:04.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:04 smithi204 ceph-mon[32357]: pgmap v704: 178 pgs: 178 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-10T18:37:04.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:04 smithi204 ceph-mon[32357]: 5.15 scrub starts 2024-09-10T18:37:04.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:04 smithi204 ceph-mon[32357]: 5.15 scrub ok 2024-09-10T18:37:04.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:04 smithi204 ceph-mon[32357]: osdmap e330: 8 total, 8 up, 8 in 2024-09-10T18:37:04.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:04 smithi204 ceph-mon[32357]: 5.e scrub starts 2024-09-10T18:37:04.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:04 smithi204 ceph-mon[32357]: 5.e scrub ok 2024-09-10T18:37:04.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:04 smithi204 ceph-mon[32357]: 6.8 deep-scrub starts 2024-09-10T18:37:04.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:04 smithi204 ceph-mon[32357]: 6.8 deep-scrub ok 2024-09-10T18:37:04.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:04.706 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:04.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:04.808 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:04.840 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:04.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:04.942 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:04.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:37:04.977 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:04.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:04.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:05.021 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 46 16 84840 84824 115.235 71.8125 0.0106377 0.00867311 2024-09-10T18:37:05.079 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:05.103 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:37:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:37:05.111 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:05.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:05.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:05.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:05.241 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:05.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:05.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:05.346 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:05.378 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:05.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:05.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:05.480 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:05.501 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:05.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:05.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:05.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:05 smithi028 ceph-mon[29291]: 4.f scrub starts 2024-09-10T18:37:05.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:05 smithi028 ceph-mon[29291]: 4.f scrub ok 2024-09-10T18:37:05.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:05 smithi028 ceph-mon[39220]: 4.f scrub starts 2024-09-10T18:37:05.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:05 smithi028 ceph-mon[39220]: 4.f scrub ok 2024-09-10T18:37:05.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:05 smithi204 ceph-mon[32357]: 4.f scrub starts 2024-09-10T18:37:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:05 smithi204 ceph-mon[32357]: 4.f scrub ok 2024-09-10T18:37:05.640 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:05.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:05.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:05.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:05.782 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:05.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:05.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:05.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:05.923 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:05.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:05.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:05.999 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:37:06.021 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 47 16 85799 85783 114.059 59.9375 0.0191573 0.00876292 2024-09-10T18:37:06.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:06.052 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:06.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:06.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:06.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:06.197 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:06.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:06.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:06.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:06.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:06.438 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:06.475 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:06.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:06.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:06.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:06 smithi204 ceph-mon[32357]: 3.12 scrub starts 2024-09-10T18:37:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:06 smithi204 ceph-mon[32357]: 3.12 scrub ok 2024-09-10T18:37:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:06 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:06 smithi204 ceph-mon[32357]: 2.1 scrub starts 2024-09-10T18:37:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:06 smithi204 ceph-mon[32357]: 2.1 scrub ok 2024-09-10T18:37:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:06 smithi204 ceph-mon[32357]: 1.c scrub starts 2024-09-10T18:37:06.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:06 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:37:06.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:06 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:37:06.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:06 smithi204 ceph-mon[32357]: 1.c scrub ok 2024-09-10T18:37:06.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:06 smithi204 ceph-mon[32357]: pgmap v706: 178 pgs: 178 active+clean; 4.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-10T18:37:06.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:06 smithi204 ceph-mon[32357]: osdmap e331: 8 total, 8 up, 8 in 2024-09-10T18:37:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:06.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:06.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[39220]: 3.12 scrub starts 2024-09-10T18:37:06.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[39220]: 3.12 scrub ok 2024-09-10T18:37:06.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:06.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[39220]: 2.1 scrub starts 2024-09-10T18:37:06.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[39220]: 2.1 scrub ok 2024-09-10T18:37:06.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[39220]: 1.c scrub starts 2024-09-10T18:37:06.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:37:06.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:37:06.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[39220]: 1.c scrub ok 2024-09-10T18:37:06.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[39220]: pgmap v706: 178 pgs: 178 active+clean; 4.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-10T18:37:06.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[39220]: osdmap e331: 8 total, 8 up, 8 in 2024-09-10T18:37:06.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[29291]: 3.12 scrub starts 2024-09-10T18:37:06.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[29291]: 3.12 scrub ok 2024-09-10T18:37:06.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:06.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[29291]: 2.1 scrub starts 2024-09-10T18:37:06.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[29291]: 2.1 scrub ok 2024-09-10T18:37:06.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[29291]: 1.c scrub starts 2024-09-10T18:37:06.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:37:06.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:37:06.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[29291]: 1.c scrub ok 2024-09-10T18:37:06.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[29291]: pgmap v706: 178 pgs: 178 active+clean; 4.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-10T18:37:06.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:06 smithi028 ceph-mon[29291]: osdmap e331: 8 total, 8 up, 8 in 2024-09-10T18:37:06.655 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:37:06.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:06.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:06.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:06.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:06.869 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:06.906 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:06.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:06.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:07.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:07.021 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 48 16 87148 87132 113.439 84.3125 0.0170093 0.00881068 2024-09-10T18:37:07.046 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:07.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:07.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:07.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:07.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:07.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:07.287 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:07.318 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:07.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:07.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:07.420 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:07.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:07.443 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:07.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:07.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:07.577 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:07.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:07.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:07 smithi204 ceph-mon[32357]: 4.1a scrub starts 2024-09-10T18:37:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:07 smithi204 ceph-mon[32357]: 4.1a scrub ok 2024-09-10T18:37:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:07 smithi204 ceph-mon[32357]: 4.16 scrub starts 2024-09-10T18:37:07.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:07 smithi204 ceph-mon[32357]: 4.16 scrub ok 2024-09-10T18:37:07.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:07 smithi204 ceph-mon[32357]: 1.13 scrub starts 2024-09-10T18:37:07.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:07 smithi204 ceph-mon[32357]: 1.13 scrub ok 2024-09-10T18:37:07.678 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:07.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:07.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:07.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:07 smithi028 ceph-mon[29291]: 4.1a scrub starts 2024-09-10T18:37:07.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:07 smithi028 ceph-mon[29291]: 4.1a scrub ok 2024-09-10T18:37:07.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:07 smithi028 ceph-mon[29291]: 4.16 scrub starts 2024-09-10T18:37:07.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:07 smithi028 ceph-mon[29291]: 4.16 scrub ok 2024-09-10T18:37:07.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:07 smithi028 ceph-mon[29291]: 1.13 scrub starts 2024-09-10T18:37:07.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:07 smithi028 ceph-mon[29291]: 1.13 scrub ok 2024-09-10T18:37:07.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:07 smithi028 ceph-mon[39220]: 4.1a scrub starts 2024-09-10T18:37:07.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:07 smithi028 ceph-mon[39220]: 4.1a scrub ok 2024-09-10T18:37:07.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:07 smithi028 ceph-mon[39220]: 4.16 scrub starts 2024-09-10T18:37:07.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:07 smithi028 ceph-mon[39220]: 4.16 scrub ok 2024-09-10T18:37:07.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:07 smithi028 ceph-mon[39220]: 1.13 scrub starts 2024-09-10T18:37:07.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:07 smithi028 ceph-mon[39220]: 1.13 scrub ok 2024-09-10T18:37:07.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:07.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:07.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:07.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:07.924 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:07.960 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:07.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:07.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:08.021 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 49 16 88017 88001 112.232 54.3125 0.00520014 0.00890495 2024-09-10T18:37:08.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:08.103 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:08.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:08.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:08.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:08.245 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:08.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:08.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:08.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:08.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:08.488 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:08.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:08 smithi028 ceph-mon[39220]: 6.2 scrub starts 2024-09-10T18:37:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:08 smithi028 ceph-mon[39220]: 6.2 scrub ok 2024-09-10T18:37:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:08 smithi028 ceph-mon[39220]: 5.8 scrub starts 2024-09-10T18:37:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:08 smithi028 ceph-mon[39220]: 5.8 scrub ok 2024-09-10T18:37:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:08 smithi028 ceph-mon[39220]: 1.1e scrub starts 2024-09-10T18:37:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:08 smithi028 ceph-mon[39220]: 1.1e scrub ok 2024-09-10T18:37:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:08 smithi028 ceph-mon[39220]: pgmap v708: 178 pgs: 178 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 820 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-10T18:37:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:08 smithi028 ceph-mon[29291]: 6.2 scrub starts 2024-09-10T18:37:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:08 smithi028 ceph-mon[29291]: 6.2 scrub ok 2024-09-10T18:37:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:08 smithi028 ceph-mon[29291]: 5.8 scrub starts 2024-09-10T18:37:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:08 smithi028 ceph-mon[29291]: 5.8 scrub ok 2024-09-10T18:37:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:08 smithi028 ceph-mon[29291]: 1.1e scrub starts 2024-09-10T18:37:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:08 smithi028 ceph-mon[29291]: 1.1e scrub ok 2024-09-10T18:37:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:08 smithi028 ceph-mon[29291]: pgmap v708: 178 pgs: 178 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 820 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-10T18:37:08.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:08.533 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:08.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:08 smithi204 ceph-mon[32357]: 6.2 scrub starts 2024-09-10T18:37:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:08 smithi204 ceph-mon[32357]: 6.2 scrub ok 2024-09-10T18:37:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:08 smithi204 ceph-mon[32357]: 5.8 scrub starts 2024-09-10T18:37:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:08 smithi204 ceph-mon[32357]: 5.8 scrub ok 2024-09-10T18:37:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:08 smithi204 ceph-mon[32357]: 1.1e scrub starts 2024-09-10T18:37:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:08 smithi204 ceph-mon[32357]: 1.1e scrub ok 2024-09-10T18:37:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:08 smithi204 ceph-mon[32357]: pgmap v708: 178 pgs: 178 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 820 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-10T18:37:08.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:08.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:08.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:08.803 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:08.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:08.856 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:37:08.857 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:37:08.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:37:08.904 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:08.936 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:08.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:09.024 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 50 16 89327 89311 111.625 81.875 0.0148768 0.0089536 2024-09-10T18:37:09.037 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:09.076 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:09.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:09.178 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:09.212 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:09.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:09.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:09.314 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:09.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:09.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:09.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:09.451 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:09.482 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:09.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:09.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:09 smithi028 ceph-mon[29291]: 4.2 scrub starts 2024-09-10T18:37:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:09 smithi028 ceph-mon[29291]: 4.2 scrub ok 2024-09-10T18:37:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:09 smithi028 ceph-mon[29291]: 1.19 scrub starts 2024-09-10T18:37:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:09 smithi028 ceph-mon[29291]: 1.19 scrub ok 2024-09-10T18:37:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:09 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:37:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:09 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:37:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:37:09.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:37:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:09 smithi028 ceph-mon[39220]: 4.2 scrub starts 2024-09-10T18:37:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:09 smithi028 ceph-mon[39220]: 4.2 scrub ok 2024-09-10T18:37:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:09 smithi028 ceph-mon[39220]: 1.19 scrub starts 2024-09-10T18:37:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:09 smithi028 ceph-mon[39220]: 1.19 scrub ok 2024-09-10T18:37:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:09 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:37:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:09 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:37:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:37:09.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:37:09.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:09 smithi204 ceph-mon[32357]: 4.2 scrub starts 2024-09-10T18:37:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:09 smithi204 ceph-mon[32357]: 4.2 scrub ok 2024-09-10T18:37:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:09 smithi204 ceph-mon[32357]: 1.19 scrub starts 2024-09-10T18:37:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:09 smithi204 ceph-mon[32357]: 1.19 scrub ok 2024-09-10T18:37:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:09 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:37:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:09 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:37:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:37:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:37:09.623 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:09.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:09.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:09.725 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:09.764 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:09.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:09.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:09.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:09.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:09.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:10.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:10.027 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 51 16 90644 90628 111.042 82.3125 0.0150662 0.00900055 2024-09-10T18:37:10.058 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:10.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:10.160 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:10.197 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:10.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:10.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:10.309 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:37:10.309 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:37:10.341 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:37:10.342 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:37:10.343 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:10.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:10.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:10.377 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:37:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:37:10.407 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:37:10.407 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:37:10.441 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:37:10.442 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:37:10.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:10.481 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:37:10.481 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:37:10.490 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:10.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:10.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:10.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[39220]: 4.b scrub starts 2024-09-10T18:37:10.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[39220]: 3.19 scrub starts 2024-09-10T18:37:10.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[39220]: 3.19 scrub ok 2024-09-10T18:37:10.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[39220]: 4.b scrub ok 2024-09-10T18:37:10.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[39220]: pgmap v709: 178 pgs: 178 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 829 op/s 2024-09-10T18:37:10.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[39220]: 6.d scrub starts 2024-09-10T18:37:10.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[39220]: 6.d scrub ok 2024-09-10T18:37:10.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[39220]: osdmap e332: 8 total, 8 up, 8 in 2024-09-10T18:37:10.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-10T18:37:10.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-10T18:37:10.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1343810433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:37:10.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[29291]: 4.b scrub starts 2024-09-10T18:37:10.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[29291]: 3.19 scrub starts 2024-09-10T18:37:10.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[29291]: 3.19 scrub ok 2024-09-10T18:37:10.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[29291]: 4.b scrub ok 2024-09-10T18:37:10.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[29291]: pgmap v709: 178 pgs: 178 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 829 op/s 2024-09-10T18:37:10.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[29291]: 6.d scrub starts 2024-09-10T18:37:10.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[29291]: 6.d scrub ok 2024-09-10T18:37:10.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[29291]: osdmap e332: 8 total, 8 up, 8 in 2024-09-10T18:37:10.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-10T18:37:10.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-10T18:37:10.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:10 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1343810433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:37:10.514 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:37:10.514 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:37:10.552 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:37:10.553 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:37:10.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:10 smithi204 ceph-mon[32357]: 4.b scrub starts 2024-09-10T18:37:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:10 smithi204 ceph-mon[32357]: 3.19 scrub starts 2024-09-10T18:37:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:10 smithi204 ceph-mon[32357]: 3.19 scrub ok 2024-09-10T18:37:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:10 smithi204 ceph-mon[32357]: 4.b scrub ok 2024-09-10T18:37:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:10 smithi204 ceph-mon[32357]: pgmap v709: 178 pgs: 178 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 829 op/s 2024-09-10T18:37:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:10 smithi204 ceph-mon[32357]: 6.d scrub starts 2024-09-10T18:37:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:10 smithi204 ceph-mon[32357]: 6.d scrub ok 2024-09-10T18:37:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:10 smithi204 ceph-mon[32357]: osdmap e332: 8 total, 8 up, 8 in 2024-09-10T18:37:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:10 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-10T18:37:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:10 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-10T18:37:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:10 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1343810433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:37:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:10.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:10.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:10.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:10.773 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:10.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:10.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:10.874 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:10.907 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:10.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:10.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:11.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:11.027 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 52 16 91853 91837 110.36 75.5625 0.00481892 0.00905656 2024-09-10T18:37:11.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:11.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:11.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:11.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:11.199 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:11.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:11.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:11.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:11.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:11.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:11.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:11.368 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:37:11.369 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-10T18:37:11.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-10T18:37:11.369 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:37:11.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.369 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35306.0:186254 9.4 e283/246 9:224dddb3:::benchmark_data_smithi028_135703_object51663:head v 283'11617, mlcod=283'11613)", 2024-09-10T18:37:11.369 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.202636+0000", 2024-09-10T18:37:11.369 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.160935464, 2024-09-10T18:37:11.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27734078200000001, 2024-09-10T18:37:11.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.369 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:11.369 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.202636+0000", 2024-09-10T18:37:11.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.202636+0000", 2024-09-10T18:37:11.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.202634+0000", 2024-09-10T18:37:11.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:11.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.202642+0000", 2024-09-10T18:37:11.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0439999999999998e-06 2024-09-10T18:37:11.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.202650+0000", 2024-09-10T18:37:11.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3070000000000003e-06 2024-09-10T18:37:11.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.202655+0000", 2024-09-10T18:37:11.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3109999999999996e-06 2024-09-10T18:37:11.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.202812+0000", 2024-09-10T18:37:11.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015717199999999999 2024-09-10T18:37:11.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.202829+0000", 2024-09-10T18:37:11.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6405999999999998e-05 2024-09-10T18:37:11.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:37:11.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.479939+0000", 2024-09-10T18:37:11.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27711055299999998 2024-09-10T18:37:11.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.479976+0000", 2024-09-10T18:37:11.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7248e-05 2024-09-10T18:37:11.373 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.373 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.373 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.374 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(osd.3.0:167479 9.c e283/246 ondisk, result = 0)", 2024-09-10T18:37:11.374 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.207950+0000", 2024-09-10T18:37:11.374 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.15562112699999, 2024-09-10T18:37:11.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27119121299999999, 2024-09-10T18:37:11.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.374 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:37:11.374 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207950+0000", 2024-09-10T18:37:11.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207950+0000", 2024-09-10T18:37:11.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207949+0000", 2024-09-10T18:37:11.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:11.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207955+0000", 2024-09-10T18:37:11.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7889999999999997e-06 2024-09-10T18:37:11.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207970+0000", 2024-09-10T18:37:11.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4432e-05 2024-09-10T18:37:11.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207976+0000", 2024-09-10T18:37:11.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.818e-06 2024-09-10T18:37:11.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.479125+0000", 2024-09-10T18:37:11.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27114970799999999 2024-09-10T18:37:11.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.479134+0000", 2024-09-10T18:37:11.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1329999999999999e-06 2024-09-10T18:37:11.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.479141+0000", 2024-09-10T18:37:11.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7409999999999992e-06 2024-09-10T18:37:11.377 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.377 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.377 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.378 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(osd.3.0:167479 9.c e283/246 ondisk, result = 0)", 2024-09-10T18:37:11.378 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.207987+0000", 2024-09-10T18:37:11.378 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.15558439500001, 2024-09-10T18:37:11.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.272007416, 2024-09-10T18:37:11.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:37:11.378 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207987+0000", 2024-09-10T18:37:11.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207987+0000", 2024-09-10T18:37:11.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207986+0000", 2024-09-10T18:37:11.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:37:11.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207990+0000", 2024-09-10T18:37:11.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.411e-06 2024-09-10T18:37:11.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207995+0000", 2024-09-10T18:37:11.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0010000000000001e-06 2024-09-10T18:37:11.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207998+0000", 2024-09-10T18:37:11.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4089999999999999e-06 2024-09-10T18:37:11.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.479976+0000", 2024-09-10T18:37:11.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.271978359 2024-09-10T18:37:11.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.479983+0000", 2024-09-10T18:37:11.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3380000000000001e-06 2024-09-10T18:37:11.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.479994+0000", 2024-09-10T18:37:11.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1459e-05 2024-09-10T18:37:11.381 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.382 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.382 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.382 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35306.0:186267 9.c 9:32632338:::benchmark_data_smithi028_135703_object51676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-10T18:37:11.382 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.210599+0000", 2024-09-10T18:37:11.382 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.15297260400001, 2024-09-10T18:37:11.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.30479414999999999, 2024-09-10T18:37:11.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.382 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:11.382 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:37:11.382 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35306", 2024-09-10T18:37:11.382 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:11.382 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 186267 2024-09-10T18:37:11.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.383 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210599+0000", 2024-09-10T18:37:11.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210599+0000", 2024-09-10T18:37:11.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210596+0000", 2024-09-10T18:37:11.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:37:11.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210604+0000", 2024-09-10T18:37:11.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0539999999999995e-06 2024-09-10T18:37:11.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210618+0000", 2024-09-10T18:37:11.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3716e-05 2024-09-10T18:37:11.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210625+0000", 2024-09-10T18:37:11.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6599999999999998e-06 2024-09-10T18:37:11.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.500019+0000", 2024-09-10T18:37:11.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.28939448699999998 2024-09-10T18:37:11.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.500140+0000", 2024-09-10T18:37:11.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012023699999999999 2024-09-10T18:37:11.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,7", 2024-09-10T18:37:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.500290+0000", 2024-09-10T18:37:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015064000000000001 2024-09-10T18:37:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.510862+0000", 2024-09-10T18:37:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010571268 2024-09-10T18:37:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.510925+0000", 2024-09-10T18:37:11.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3137000000000002e-05 2024-09-10T18:37:11.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:37:11.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.515180+0000", 2024-09-10T18:37:11.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042550009999999996 2024-09-10T18:37:11.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:37:11.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.515221+0000", 2024-09-10T18:37:11.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0970000000000002e-05 2024-09-10T18:37:11.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.515393+0000", 2024-09-10T18:37:11.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017211400000000001 2024-09-10T18:37:11.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.389 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.389 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35306.0:186256 9.4 e283/246 9:2dc2a984:::benchmark_data_smithi028_135703_object51665:head v 283'11618, mlcod=283'11613)", 2024-09-10T18:37:11.389 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.210630+0000", 2024-09-10T18:37:11.389 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.15294108399999, 2024-09-10T18:37:11.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.28319999400000001, 2024-09-10T18:37:11.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.390 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:11.390 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210630+0000", 2024-09-10T18:37:11.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210630+0000", 2024-09-10T18:37:11.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210629+0000", 2024-09-10T18:37:11.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:37:11.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210636+0000", 2024-09-10T18:37:11.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2729999999999997e-06 2024-09-10T18:37:11.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210646+0000", 2024-09-10T18:37:11.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5030000000000003e-06 2024-09-10T18:37:11.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210650+0000", 2024-09-10T18:37:11.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4599999999999996e-06 2024-09-10T18:37:11.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.482849+0000", 2024-09-10T18:37:11.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27219835599999997 2024-09-10T18:37:11.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.482865+0000", 2024-09-10T18:37:11.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6714e-05 2024-09-10T18:37:11.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:37:11.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.493803+0000", 2024-09-10T18:37:11.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010937631999999999 2024-09-10T18:37:11.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.493830+0000", 2024-09-10T18:37:11.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7231000000000001e-05 2024-09-10T18:37:11.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.393 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.394 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35306.0:186259 9.4 e283/246 9:2db7fb1c:::benchmark_data_smithi028_135703_object51668:head v 283'11619, mlcod=283'11615)", 2024-09-10T18:37:11.394 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.213559+0000", 2024-09-10T18:37:11.394 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.150012672, 2024-09-10T18:37:11.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.28029324799999999, 2024-09-10T18:37:11.394 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.394 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:11.394 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.213559+0000", 2024-09-10T18:37:11.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.213559+0000", 2024-09-10T18:37:11.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.213557+0000", 2024-09-10T18:37:11.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:11.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.213565+0000", 2024-09-10T18:37:11.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.388e-06 2024-09-10T18:37:11.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.213578+0000", 2024-09-10T18:37:11.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3093e-05 2024-09-10T18:37:11.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.213585+0000", 2024-09-10T18:37:11.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1999999999999999e-06 2024-09-10T18:37:11.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.485084+0000", 2024-09-10T18:37:11.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.271499306 2024-09-10T18:37:11.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.485113+0000", 2024-09-10T18:37:11.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8847e-05 2024-09-10T18:37:11.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:37:11.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.493833+0000", 2024-09-10T18:37:11.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087202779999999997 2024-09-10T18:37:11.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.493852+0000", 2024-09-10T18:37:11.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8729e-05 2024-09-10T18:37:11.398 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.398 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.398 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.398 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(osd.3.0:167482 9.c e283/246 ondisk, result = 0)", 2024-09-10T18:37:11.398 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.220115+0000", 2024-09-10T18:37:11.398 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.14345629100001, 2024-09-10T18:37:11.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.26019199599999998, 2024-09-10T18:37:11.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.398 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:37:11.398 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.220115+0000", 2024-09-10T18:37:11.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.220115+0000", 2024-09-10T18:37:11.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.220114+0000", 2024-09-10T18:37:11.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:11.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.220119+0000", 2024-09-10T18:37:11.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4949999999999999e-06 2024-09-10T18:37:11.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.220132+0000", 2024-09-10T18:37:11.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3123e-05 2024-09-10T18:37:11.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.220138+0000", 2024-09-10T18:37:11.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0270000000000001e-06 2024-09-10T18:37:11.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480053+0000", 2024-09-10T18:37:11.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25991503199999999 2024-09-10T18:37:11.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480060+0000", 2024-09-10T18:37:11.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6719999999999998e-06 2024-09-10T18:37:11.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480307+0000", 2024-09-10T18:37:11.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024703300000000002 2024-09-10T18:37:11.402 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.402 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.402 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.402 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(osd.3.0:167482 9.c e283/246 ondisk, result = 0)", 2024-09-10T18:37:11.402 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.223464+0000", 2024-09-10T18:37:11.402 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.140106933, 2024-09-10T18:37:11.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25655931599999998, 2024-09-10T18:37:11.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.402 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:37:11.402 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.223464+0000", 2024-09-10T18:37:11.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.223464+0000", 2024-09-10T18:37:11.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.223463+0000", 2024-09-10T18:37:11.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:11.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.223469+0000", 2024-09-10T18:37:11.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1700000000000002e-06 2024-09-10T18:37:11.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.223480+0000", 2024-09-10T18:37:11.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0433e-05 2024-09-10T18:37:11.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.223485+0000", 2024-09-10T18:37:11.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4389999999999999e-06 2024-09-10T18:37:11.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480012+0000", 2024-09-10T18:37:11.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25652734999999999 2024-09-10T18:37:11.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480018+0000", 2024-09-10T18:37:11.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3299999999999998e-06 2024-09-10T18:37:11.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480024+0000", 2024-09-10T18:37:11.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8640000000000001e-06 2024-09-10T18:37:11.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.406 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.406 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35306.0:186279 9.4 e283/246 9:28b03174:::benchmark_data_smithi028_135703_object51688:head v 283'11620, mlcod=283'11615)", 2024-09-10T18:37:11.406 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.224752+0000", 2024-09-10T18:37:11.406 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.138819154, 2024-09-10T18:37:11.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.26912506200000003, 2024-09-10T18:37:11.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:11.406 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.224752+0000", 2024-09-10T18:37:11.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.224752+0000", 2024-09-10T18:37:11.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.224751+0000", 2024-09-10T18:37:11.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:11.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.224759+0000", 2024-09-10T18:37:11.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6240000000000008e-06 2024-09-10T18:37:11.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.224773+0000", 2024-09-10T18:37:11.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3796e-05 2024-09-10T18:37:11.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.224779+0000", 2024-09-10T18:37:11.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6450000000000002e-06 2024-09-10T18:37:11.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.487193+0000", 2024-09-10T18:37:11.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.262414598 2024-09-10T18:37:11.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.487211+0000", 2024-09-10T18:37:11.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8011000000000001e-05 2024-09-10T18:37:11.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:37:11.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.493854+0000", 2024-09-10T18:37:11.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066430020000000003 2024-09-10T18:37:11.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.493877+0000", 2024-09-10T18:37:11.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2860000000000001e-05 2024-09-10T18:37:11.410 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.410 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.410 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.410 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35306.0:186281 9.4 e283/246 9:2314a6f4:::benchmark_data_smithi028_135703_object51690:head v 283'11621, mlcod=283'11615)", 2024-09-10T18:37:11.411 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.227701+0000", 2024-09-10T18:37:11.411 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.13587052700001, 2024-09-10T18:37:11.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27006845800000001, 2024-09-10T18:37:11.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.411 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:11.411 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227701+0000", 2024-09-10T18:37:11.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227701+0000", 2024-09-10T18:37:11.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227699+0000", 2024-09-10T18:37:11.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:37:11.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227707+0000", 2024-09-10T18:37:11.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1219999999999995e-06 2024-09-10T18:37:11.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227717+0000", 2024-09-10T18:37:11.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0511e-05 2024-09-10T18:37:11.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227724+0000", 2024-09-10T18:37:11.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1580000000000002e-06 2024-09-10T18:37:11.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.489305+0000", 2024-09-10T18:37:11.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.26158115300000001 2024-09-10T18:37:11.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.489331+0000", 2024-09-10T18:37:11.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5987e-05 2024-09-10T18:37:11.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:37:11.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.497691+0000", 2024-09-10T18:37:11.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0083606610000000001 2024-09-10T18:37:11.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.497769+0000", 2024-09-10T18:37:11.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7673e-05 2024-09-10T18:37:11.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.415 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.416 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.416 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(osd.3.0:167483 9.c e283/246 ondisk, result = 0)", 2024-09-10T18:37:11.416 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.227735+0000", 2024-09-10T18:37:11.416 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.13583574500001, 2024-09-10T18:37:11.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25269138600000002, 2024-09-10T18:37:11.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.416 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:37:11.416 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227735+0000", 2024-09-10T18:37:11.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227735+0000", 2024-09-10T18:37:11.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227734+0000", 2024-09-10T18:37:11.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:11.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227741+0000", 2024-09-10T18:37:11.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8340000000000001e-06 2024-09-10T18:37:11.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227751+0000", 2024-09-10T18:37:11.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6430000000000007e-06 2024-09-10T18:37:11.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227756+0000", 2024-09-10T18:37:11.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6470000000000003e-06 2024-09-10T18:37:11.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480367+0000", 2024-09-10T18:37:11.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25261063700000003 2024-09-10T18:37:11.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480374+0000", 2024-09-10T18:37:11.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4739999999999999e-06 2024-09-10T18:37:11.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480427+0000", 2024-09-10T18:37:11.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2519000000000003e-05 2024-09-10T18:37:11.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.420 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.420 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35306.0:186287 9.4 e283/246 9:2d5b8ef2:::benchmark_data_smithi028_135703_object51696:head v 283'11622, mlcod=283'11615)", 2024-09-10T18:37:11.420 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.232647+0000", 2024-09-10T18:37:11.420 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.13092428100001, 2024-09-10T18:37:11.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.26978486099999999, 2024-09-10T18:37:11.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.420 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:11.420 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232647+0000", 2024-09-10T18:37:11.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232647+0000", 2024-09-10T18:37:11.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232646+0000", 2024-09-10T18:37:11.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:11.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232659+0000", 2024-09-10T18:37:11.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2993000000000001e-05 2024-09-10T18:37:11.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232668+0000", 2024-09-10T18:37:11.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8660000000000001e-06 2024-09-10T18:37:11.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232675+0000", 2024-09-10T18:37:11.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5259999999999999e-06 2024-09-10T18:37:11.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.491585+0000", 2024-09-10T18:37:11.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25890969200000002 2024-09-10T18:37:11.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.491603+0000", 2024-09-10T18:37:11.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8196999999999999e-05 2024-09-10T18:37:11.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:37:11.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.502399+0000", 2024-09-10T18:37:11.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010796155 2024-09-10T18:37:11.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.424 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.424 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.502432+0000", 2024-09-10T18:37:11.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2765000000000002e-05 2024-09-10T18:37:11.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.424 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.424 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(osd.3.0:167485 9.c e283/246 ondisk, result = 0)", 2024-09-10T18:37:11.424 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.232729+0000", 2024-09-10T18:37:11.425 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.13084206100001, 2024-09-10T18:37:11.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24793206400000001, 2024-09-10T18:37:11.425 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:37:11.425 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.425 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.425 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232729+0000", 2024-09-10T18:37:11.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.425 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.425 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232729+0000", 2024-09-10T18:37:11.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232729+0000", 2024-09-10T18:37:11.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:37:11.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232731+0000", 2024-09-10T18:37:11.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5909999999999998e-06 2024-09-10T18:37:11.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232735+0000", 2024-09-10T18:37:11.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0210000000000001e-06 2024-09-10T18:37:11.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232738+0000", 2024-09-10T18:37:11.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7690000000000001e-06 2024-09-10T18:37:11.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480599+0000", 2024-09-10T18:37:11.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.247861209 2024-09-10T18:37:11.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480605+0000", 2024-09-10T18:37:11.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9859999999999999e-06 2024-09-10T18:37:11.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480661+0000", 2024-09-10T18:37:11.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5758999999999999e-05 2024-09-10T18:37:11.428 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.428 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.429 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(osd.3.0:167486 9.c e283/246 ondisk, result = 0)", 2024-09-10T18:37:11.429 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.232741+0000", 2024-09-10T18:37:11.429 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.130830229, 2024-09-10T18:37:11.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24807601900000001, 2024-09-10T18:37:11.429 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.429 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:37:11.429 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232741+0000", 2024-09-10T18:37:11.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232741+0000", 2024-09-10T18:37:11.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232741+0000", 2024-09-10T18:37:11.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967296 2024-09-10T18:37:11.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232743+0000", 2024-09-10T18:37:11.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1280000000000002e-06 2024-09-10T18:37:11.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232747+0000", 2024-09-10T18:37:11.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7799999999999998e-06 2024-09-10T18:37:11.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232749+0000", 2024-09-10T18:37:11.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7819999999999998e-06 2024-09-10T18:37:11.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480756+0000", 2024-09-10T18:37:11.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.248006169 2024-09-10T18:37:11.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480761+0000", 2024-09-10T18:37:11.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6890000000000001e-06 2024-09-10T18:37:11.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480817+0000", 2024-09-10T18:37:11.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5692999999999997e-05 2024-09-10T18:37:11.433 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.433 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.433 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.433 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(osd.3.0:167488 9.c e283/246 ondisk, result = 0)", 2024-09-10T18:37:11.433 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.232772+0000", 2024-09-10T18:37:11.433 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.130798779, 2024-09-10T18:37:11.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.248170683, 2024-09-10T18:37:11.433 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.433 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:37:11.433 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232772+0000", 2024-09-10T18:37:11.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232772+0000", 2024-09-10T18:37:11.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232772+0000", 2024-09-10T18:37:11.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967296 2024-09-10T18:37:11.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232774+0000", 2024-09-10T18:37:11.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1210000000000001e-06 2024-09-10T18:37:11.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232778+0000", 2024-09-10T18:37:11.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1250000000000003e-06 2024-09-10T18:37:11.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232781+0000", 2024-09-10T18:37:11.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.01e-06 2024-09-10T18:37:11.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480888+0000", 2024-09-10T18:37:11.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24810643299999999 2024-09-10T18:37:11.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480894+0000", 2024-09-10T18:37:11.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9819999999999996e-06 2024-09-10T18:37:11.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480943+0000", 2024-09-10T18:37:11.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9246000000000003e-05 2024-09-10T18:37:11.437 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.437 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.437 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.437 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(osd.3.0:167489 9.c e283/246 ondisk, result = 0)", 2024-09-10T18:37:11.437 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.232784+0000", 2024-09-10T18:37:11.437 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.13078707400001, 2024-09-10T18:37:11.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24825338999999999, 2024-09-10T18:37:11.437 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.437 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:37:11.437 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.438 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232784+0000", 2024-09-10T18:37:11.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.438 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232784+0000", 2024-09-10T18:37:11.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.438 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232784+0000", 2024-09-10T18:37:11.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967296 2024-09-10T18:37:11.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232786+0000", 2024-09-10T18:37:11.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1569999999999998e-06 2024-09-10T18:37:11.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232790+0000", 2024-09-10T18:37:11.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8720000000000004e-06 2024-09-10T18:37:11.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232793+0000", 2024-09-10T18:37:11.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7980000000000001e-06 2024-09-10T18:37:11.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480989+0000", 2024-09-10T18:37:11.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24819665599999999 2024-09-10T18:37:11.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.480994+0000", 2024-09-10T18:37:11.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8910000000000003e-06 2024-09-10T18:37:11.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.481037+0000", 2024-09-10T18:37:11.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3231e-05 2024-09-10T18:37:11.441 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.441 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.441 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.441 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35306.0:186291 9.4 e283/246 9:22863b05:::benchmark_data_smithi028_135703_object51700:head v 283'11623, mlcod=283'11615)", 2024-09-10T18:37:11.441 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.239764+0000", 2024-09-10T18:37:11.441 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.12380676800001, 2024-09-10T18:37:11.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27116428799999998, 2024-09-10T18:37:11.441 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.441 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:11.441 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.239764+0000", 2024-09-10T18:37:11.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.239764+0000", 2024-09-10T18:37:11.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.239763+0000", 2024-09-10T18:37:11.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:11.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.443 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.443 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.239770+0000", 2024-09-10T18:37:11.443 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.041e-06 2024-09-10T18:37:11.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:11.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.444 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.444 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.239780+0000", 2024-09-10T18:37:11.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4269999999999997e-06 2024-09-10T18:37:11.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.239785+0000", 2024-09-10T18:37:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2310000000000004e-06 2024-09-10T18:37:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.493902+0000", 2024-09-10T18:37:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25411749500000003 2024-09-10T18:37:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.493921+0000", 2024-09-10T18:37:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8732e-05 2024-09-10T18:37:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:37:11.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.510866+0000", 2024-09-10T18:37:11.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016945248 2024-09-10T18:37:11.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.510929+0000", 2024-09-10T18:37:11.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2322000000000004e-05 2024-09-10T18:37:11.446 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.446 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.446 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.447 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35306.0:186300 9.c 9:37fcaff8:::benchmark_data_smithi028_135703_object51709:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-10T18:37:11.447 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.251425+0000", 2024-09-10T18:37:11.447 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 166.11214599499999, 2024-09-10T18:37:11.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.26482892400000002, 2024-09-10T18:37:11.447 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.447 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:11.447 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:37:11.447 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35306", 2024-09-10T18:37:11.447 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:11.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 186300 2024-09-10T18:37:11.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.447 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.251425+0000", 2024-09-10T18:37:11.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.251425+0000", 2024-09-10T18:37:11.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.251424+0000", 2024-09-10T18:37:11.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:11.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.251430+0000", 2024-09-10T18:37:11.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9880000000000001e-06 2024-09-10T18:37:11.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.452374+0000", 2024-09-10T18:37:11.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.20094495300000001 2024-09-10T18:37:11.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.452387+0000", 2024-09-10T18:37:11.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2755000000000001e-05 2024-09-10T18:37:11.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.502319+0000", 2024-09-10T18:37:11.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.049931907999999997 2024-09-10T18:37:11.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.502432+0000", 2024-09-10T18:37:11.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000112438 2024-09-10T18:37:11.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,7", 2024-09-10T18:37:11.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.502547+0000", 2024-09-10T18:37:11.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000115518 2024-09-10T18:37:11.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:11.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.510946+0000", 2024-09-10T18:37:11.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0083991799999999991 2024-09-10T18:37:11.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:11.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.510969+0000", 2024-09-10T18:37:11.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2801000000000001e-05 2024-09-10T18:37:11.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:37:11.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.515505+0000", 2024-09-10T18:37:11.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004535557 2024-09-10T18:37:11.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:37:11.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.516147+0000", 2024-09-10T18:37:11.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00064279400000000003 2024-09-10T18:37:11.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.516254+0000", 2024-09-10T18:37:11.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000106669 2024-09-10T18:37:11.453 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.453 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.453 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.453 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35306.0:186567 9.2 9:4b26e7d1:::benchmark_data_smithi028_135703_object51976:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-10T18:37:11.453 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.717499+0000", 2024-09-10T18:37:11.453 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 165.64607229800001, 2024-09-10T18:37:11.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.26343703699999999, 2024-09-10T18:37:11.453 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.453 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:11.453 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:37:11.453 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35306", 2024-09-10T18:37:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 186567 2024-09-10T18:37:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.717499+0000", 2024-09-10T18:37:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.717499+0000", 2024-09-10T18:37:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.717497+0000", 2024-09-10T18:37:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.717504+0000", 2024-09-10T18:37:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2439999999999998e-06 2024-09-10T18:37:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.717515+0000", 2024-09-10T18:37:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1256e-05 2024-09-10T18:37:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.717521+0000", 2024-09-10T18:37:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9259999999999997e-06 2024-09-10T18:37:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.717557+0000", 2024-09-10T18:37:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6130000000000001e-05 2024-09-10T18:37:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.717669+0000", 2024-09-10T18:37:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111865 2024-09-10T18:37:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:11.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.809712+0000", 2024-09-10T18:37:11.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.092043258000000003 2024-09-10T18:37:11.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:37:11.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.815271+0000", 2024-09-10T18:37:11.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055588790000000001 2024-09-10T18:37:11.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:11.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.819001+0000", 2024-09-10T18:37:11.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037302450000000001 2024-09-10T18:37:11.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:11.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.980839+0000", 2024-09-10T18:37:11.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.161838174 2024-09-10T18:37:11.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:37:11.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.980899+0000", 2024-09-10T18:37:11.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.927e-05 2024-09-10T18:37:11.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.980936+0000", 2024-09-10T18:37:11.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7307e-05 2024-09-10T18:37:11.459 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.459 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.459 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.459 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35306.0:218751 9.8 9:14a7ae48:::benchmark_data_smithi028_135703_object84160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-10T18:37:11.459 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:54.117579+0000", 2024-09-10T18:37:11.459 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 137.245992538, 2024-09-10T18:37:11.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25553158599999998, 2024-09-10T18:37:11.459 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:37:11.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:11.459 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:37:11.460 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35306", 2024-09-10T18:37:11.460 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:11.460 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 218751 2024-09-10T18:37:11.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.460 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:37:11.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:37:11.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.117579+0000", 2024-09-10T18:37:11.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:37:11.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.117579+0000", 2024-09-10T18:37:11.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:37:11.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:37:11.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.117578+0000", 2024-09-10T18:37:11.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:37:11.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:37:11.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.117580+0000", 2024-09-10T18:37:11.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2170000000000001e-06 2024-09-10T18:37:11.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:37:11.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.117585+0000", 2024-09-10T18:37:11.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2679999999999996e-06 2024-09-10T18:37:11.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:37:11.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.117588+0000", 2024-09-10T18:37:11.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2399999999999999e-06 2024-09-10T18:37:11.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:37:11.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.122330+0000", 2024-09-10T18:37:11.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047416079999999996 2024-09-10T18:37:11.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:37:11.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.122428+0000", 2024-09-10T18:37:11.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8048999999999996e-05 2024-09-10T18:37:11.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-09-10T18:37:11.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.122529+0000", 2024-09-10T18:37:11.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101742 2024-09-10T18:37:11.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:11.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.128718+0000", 2024-09-10T18:37:11.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061888990000000003 2024-09-10T18:37:11.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:37:11.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.128764+0000", 2024-09-10T18:37:11.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5259999999999997e-05 2024-09-10T18:37:11.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:11.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.372858+0000", 2024-09-10T18:37:11.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.244094113 2024-09-10T18:37:11.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:37:11.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.373065+0000", 2024-09-10T18:37:11.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000206975 2024-09-10T18:37:11.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:37:11.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:37:11.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:37:11.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:54.373110+0000", 2024-09-10T18:37:11.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5578999999999998e-05 2024-09-10T18:37:11.465 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.465 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.465 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.465 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:37:11.465 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:37:11.465 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:37:11.466 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:37:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:37:11] "GET /metrics HTTP/1.1" 200 224694 "" "Prometheus/2.43.0" 2024-09-10T18:37:11.466 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:11 smithi028 ceph-mon[39220]: 4.1b scrub starts 2024-09-10T18:37:11.466 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:11 smithi028 ceph-mon[39220]: 4.1b scrub ok 2024-09-10T18:37:11.466 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:11 smithi028 ceph-mon[39220]: 10.d scrub starts 2024-09-10T18:37:11.466 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:11 smithi028 ceph-mon[39220]: 4.14 scrub starts 2024-09-10T18:37:11.466 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:11 smithi028 ceph-mon[39220]: 4.14 scrub ok 2024-09-10T18:37:11.466 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:11 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-10T18:37:11.466 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:11 smithi028 ceph-mon[39220]: osdmap e333: 8 total, 8 up, 8 in 2024-09-10T18:37:11.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:11 smithi028 ceph-mon[29291]: 4.1b scrub starts 2024-09-10T18:37:11.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:11 smithi028 ceph-mon[29291]: 4.1b scrub ok 2024-09-10T18:37:11.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:11 smithi028 ceph-mon[29291]: 10.d scrub starts 2024-09-10T18:37:11.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:11 smithi028 ceph-mon[29291]: 4.14 scrub starts 2024-09-10T18:37:11.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:11 smithi028 ceph-mon[29291]: 4.14 scrub ok 2024-09-10T18:37:11.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:11 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-10T18:37:11.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:11 smithi028 ceph-mon[29291]: osdmap e333: 8 total, 8 up, 8 in 2024-09-10T18:37:11.508 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:11.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:11.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:11 smithi204 ceph-mon[32357]: 4.1b scrub starts 2024-09-10T18:37:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:11 smithi204 ceph-mon[32357]: 4.1b scrub ok 2024-09-10T18:37:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:11 smithi204 ceph-mon[32357]: 10.d scrub starts 2024-09-10T18:37:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:11 smithi204 ceph-mon[32357]: 4.14 scrub starts 2024-09-10T18:37:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:11 smithi204 ceph-mon[32357]: 4.14 scrub ok 2024-09-10T18:37:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:11 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-10T18:37:11.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:11 smithi204 ceph-mon[32357]: osdmap e333: 8 total, 8 up, 8 in 2024-09-10T18:37:11.610 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:11.645 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:11.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:11.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:11.790 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:11.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:11.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:11.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:11.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:11.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:11.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:12.025 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 53 16 93085 93069 109.73 77 0.0105514 0.00910844 2024-09-10T18:37:12.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:12.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:12.075 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:12.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:12.176 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:12.208 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:12.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:12.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:12.344 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:12.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:12.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:12.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:12.482 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:12.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:12.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:12 smithi028 ceph-mon[29291]: 5.d scrub starts 2024-09-10T18:37:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:12 smithi028 ceph-mon[29291]: 5.d scrub ok 2024-09-10T18:37:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:12 smithi028 ceph-mon[29291]: pgmap v712: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-10T18:37:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:12 smithi028 ceph-mon[29291]: 3.5 scrub starts 2024-09-10T18:37:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:12 smithi028 ceph-mon[29291]: 3.5 scrub ok 2024-09-10T18:37:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:12 smithi028 ceph-mon[29291]: osdmap e334: 8 total, 8 up, 8 in 2024-09-10T18:37:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:12 smithi028 ceph-mon[29291]: 1.0 scrub starts 2024-09-10T18:37:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:12 smithi028 ceph-mon[29291]: 1.0 scrub ok 2024-09-10T18:37:12.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:12 smithi028 ceph-mon[39220]: 5.d scrub starts 2024-09-10T18:37:12.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:12 smithi028 ceph-mon[39220]: 5.d scrub ok 2024-09-10T18:37:12.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:12 smithi028 ceph-mon[39220]: pgmap v712: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-10T18:37:12.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:12 smithi028 ceph-mon[39220]: 3.5 scrub starts 2024-09-10T18:37:12.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:12 smithi028 ceph-mon[39220]: 3.5 scrub ok 2024-09-10T18:37:12.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:12 smithi028 ceph-mon[39220]: osdmap e334: 8 total, 8 up, 8 in 2024-09-10T18:37:12.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:12 smithi028 ceph-mon[39220]: 1.0 scrub starts 2024-09-10T18:37:12.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:12 smithi028 ceph-mon[39220]: 1.0 scrub ok 2024-09-10T18:37:12.584 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:12.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:37:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:37:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:37:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:12 smithi204 ceph-mon[32357]: 5.d scrub starts 2024-09-10T18:37:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:12 smithi204 ceph-mon[32357]: 5.d scrub ok 2024-09-10T18:37:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:12 smithi204 ceph-mon[32357]: pgmap v712: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-10T18:37:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:12 smithi204 ceph-mon[32357]: 3.5 scrub starts 2024-09-10T18:37:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:12 smithi204 ceph-mon[32357]: 3.5 scrub ok 2024-09-10T18:37:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:12 smithi204 ceph-mon[32357]: osdmap e334: 8 total, 8 up, 8 in 2024-09-10T18:37:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:12 smithi204 ceph-mon[32357]: 1.0 scrub starts 2024-09-10T18:37:12.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:12 smithi204 ceph-mon[32357]: 1.0 scrub ok 2024-09-10T18:37:12.622 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:12.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:12.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:12.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:12.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:12.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:12.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:12.907 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:12.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:12.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:12.964 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:37:13.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:13.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 54 16 94394 94378 109.213 81.8125 0.00989039 0.00915164 2024-09-10T18:37:13.045 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:13.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:13.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:13.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:13.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:13.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:13.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:13.433 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:13.458 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:13.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:13.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:13 smithi028 ceph-mon[39220]: 6.10 scrub starts 2024-09-10T18:37:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:13 smithi028 ceph-mon[39220]: 6.10 scrub ok 2024-09-10T18:37:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:13 smithi028 ceph-mon[39220]: 10.d scrub ok 2024-09-10T18:37:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:13 smithi028 ceph-mon[29291]: 6.10 scrub starts 2024-09-10T18:37:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:13 smithi028 ceph-mon[29291]: 6.10 scrub ok 2024-09-10T18:37:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:13 smithi028 ceph-mon[29291]: 10.d scrub ok 2024-09-10T18:37:13.561 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:13.592 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:13.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:13.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:13 smithi204 ceph-mon[32357]: 6.10 scrub starts 2024-09-10T18:37:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:13 smithi204 ceph-mon[32357]: 6.10 scrub ok 2024-09-10T18:37:13.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:13 smithi204 ceph-mon[32357]: 10.d scrub ok 2024-09-10T18:37:13.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:13.739 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:13.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:13.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:13.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:13.886 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-10T18:37:13.890 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:13.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:13.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:13.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:14.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 55 15 95675 95660 108.684 80.125 0.0121633 0.00919651 2024-09-10T18:37:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:14.030 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:14.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:14.131 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:14.168 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:14.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:14.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:14.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:14.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:14.320 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:37:14.360 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:37:14.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:14 smithi028 ceph-mon[39220]: 5.19 scrub starts 2024-09-10T18:37:14.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:14 smithi028 ceph-mon[39220]: 5.2 scrub starts 2024-09-10T18:37:14.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:14 smithi028 ceph-mon[39220]: 5.19 scrub ok 2024-09-10T18:37:14.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:14 smithi028 ceph-mon[39220]: 5.2 scrub ok 2024-09-10T18:37:14.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:14 smithi028 ceph-mon[39220]: pgmap v714: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 62 MiB/s wr, 996 op/s 2024-09-10T18:37:14.377 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:14 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/859259533' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:37:14.377 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:14 smithi028 ceph-mon[39220]: 4.c scrub starts 2024-09-10T18:37:14.377 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:14 smithi028 ceph-mon[39220]: 4.c scrub ok 2024-09-10T18:37:14.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:14 smithi028 ceph-mon[29291]: 5.19 scrub starts 2024-09-10T18:37:14.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:14 smithi028 ceph-mon[29291]: 5.2 scrub starts 2024-09-10T18:37:14.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:14 smithi028 ceph-mon[29291]: 5.19 scrub ok 2024-09-10T18:37:14.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:14 smithi028 ceph-mon[29291]: 5.2 scrub ok 2024-09-10T18:37:14.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:14 smithi028 ceph-mon[29291]: pgmap v714: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 62 MiB/s wr, 996 op/s 2024-09-10T18:37:14.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:14 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/859259533' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:37:14.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:14 smithi028 ceph-mon[29291]: 4.c scrub starts 2024-09-10T18:37:14.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:14 smithi028 ceph-mon[29291]: 4.c scrub ok 2024-09-10T18:37:14.412 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:14.456 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:14.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:14.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:14.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:14.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:14.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:14 smithi204 ceph-mon[32357]: 5.19 scrub starts 2024-09-10T18:37:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:14 smithi204 ceph-mon[32357]: 5.2 scrub starts 2024-09-10T18:37:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:14 smithi204 ceph-mon[32357]: 5.19 scrub ok 2024-09-10T18:37:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:14 smithi204 ceph-mon[32357]: 5.2 scrub ok 2024-09-10T18:37:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:14 smithi204 ceph-mon[32357]: pgmap v714: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 62 MiB/s wr, 996 op/s 2024-09-10T18:37:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:14 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/859259533' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:37:14.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:14 smithi204 ceph-mon[32357]: 4.c scrub starts 2024-09-10T18:37:14.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:14 smithi204 ceph-mon[32357]: 4.c scrub ok 2024-09-10T18:37:14.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:14.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:14.732 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:14.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:14.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:37:14.832 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:14.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:14.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:14.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:14.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:14.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:14.995 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:14.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:15.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 56 15 96865 96850 108.071 74.375 0.00885178 0.00924793 2024-09-10T18:37:15.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:15.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:37:14 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:37:15.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:15.129 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:15.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:15.230 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:15.263 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:15.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:15.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:15.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:15.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:15.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:15.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[29291]: 4.13 scrub starts 2024-09-10T18:37:15.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[29291]: 4.13 scrub ok 2024-09-10T18:37:15.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[29291]: 3.1b scrub starts 2024-09-10T18:37:15.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[29291]: 3.1b scrub ok 2024-09-10T18:37:15.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[29291]: 1.12 scrub starts 2024-09-10T18:37:15.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[29291]: 1.12 scrub ok 2024-09-10T18:37:15.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:37:15.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/859259533' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:37:15.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[29291]: osdmap e335: 8 total, 8 up, 8 in 2024-09-10T18:37:15.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[29291]: 5.1c scrub starts 2024-09-10T18:37:15.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[39220]: 4.13 scrub starts 2024-09-10T18:37:15.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[39220]: 4.13 scrub ok 2024-09-10T18:37:15.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[39220]: 3.1b scrub starts 2024-09-10T18:37:15.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[39220]: 3.1b scrub ok 2024-09-10T18:37:15.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[39220]: 1.12 scrub starts 2024-09-10T18:37:15.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[39220]: 1.12 scrub ok 2024-09-10T18:37:15.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:37:15.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/859259533' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:37:15.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[39220]: osdmap e335: 8 total, 8 up, 8 in 2024-09-10T18:37:15.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:15 smithi028 ceph-mon[39220]: 5.1c scrub starts 2024-09-10T18:37:15.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:15.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:15.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:15.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:15.562 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:15 smithi204 ceph-mon[32357]: 4.13 scrub starts 2024-09-10T18:37:15.562 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:15 smithi204 ceph-mon[32357]: 4.13 scrub ok 2024-09-10T18:37:15.562 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:15 smithi204 ceph-mon[32357]: 3.1b scrub starts 2024-09-10T18:37:15.562 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:15 smithi204 ceph-mon[32357]: 3.1b scrub ok 2024-09-10T18:37:15.562 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:15 smithi204 ceph-mon[32357]: 1.12 scrub starts 2024-09-10T18:37:15.562 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:15 smithi204 ceph-mon[32357]: 1.12 scrub ok 2024-09-10T18:37:15.562 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:15 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:37:15.563 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:15 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/859259533' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:37:15.563 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:15 smithi204 ceph-mon[32357]: osdmap e335: 8 total, 8 up, 8 in 2024-09-10T18:37:15.563 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:15 smithi204 ceph-mon[32357]: 5.1c scrub starts 2024-09-10T18:37:15.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:15.691 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:15.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:15.792 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:15.824 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:15.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:15.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:15.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:16.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 57 16 98191 98175 107.628 82.8125 0.00793091 0.00928619 2024-09-10T18:37:16.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:16.080 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:37:16.133 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:16.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:16.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:16.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:16.280 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:16.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:16.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:16 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:16 smithi204 ceph-mon[32357]: pgmap v716: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s rd, 77 MiB/s wr, 1.23k op/s; 0 B/s, 1 objects/s recovering 2024-09-10T18:37:16.381 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:16.418 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:16.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:16.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:16.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:16.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:16 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:16.542 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:16 smithi028 ceph-mon[39220]: pgmap v716: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s rd, 77 MiB/s wr, 1.23k op/s; 0 B/s, 1 objects/s recovering 2024-09-10T18:37:16.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:16 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:16.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:16 smithi028 ceph-mon[29291]: pgmap v716: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s rd, 77 MiB/s wr, 1.23k op/s; 0 B/s, 1 objects/s recovering 2024-09-10T18:37:16.561 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:16.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:16.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:16.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:16.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:16.709 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:16.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:16.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:16.834 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:16.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:16.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:16.977 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:16.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:16.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:17.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 58 16 99298 99282 106.965 69.1875 0.0100458 0.00934442 2024-09-10T18:37:17.079 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:17.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:17.115 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:17.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:17.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:17.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:17.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:17.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:37:17.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:17.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:17.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:17.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:17.501 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:17.536 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:17.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:17.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:17.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:17 smithi028 ceph-mon[29291]: pgmap v717: 178 pgs: 178 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 813 B/s rd, 93 MiB/s wr, 1.49k op/s; 17 B/s, 1 objects/s recovering 2024-09-10T18:37:17.581 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:17 smithi028 ceph-mon[39220]: pgmap v717: 178 pgs: 178 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 813 B/s rd, 93 MiB/s wr, 1.49k op/s; 17 B/s, 1 objects/s recovering 2024-09-10T18:37:17.599 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:37:17.599 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:37:17.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.599 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:99998 10.e 10:7a856110:::benchmark_data_smithi028_161947_object99997:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-10T18:37:17.599 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:37:17.590284+0000", 2024-09-10T18:37:17.599 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0063043650000000001, 2024-09-10T18:37:17.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0063374440000000002, 2024-09-10T18:37:17.599 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:17.600 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:37:17.600 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:17.600 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:37:17.600 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:37:17.600 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 99998 2024-09-10T18:37:17.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.600 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:17.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:17.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.590284+0000", 2024-09-10T18:37:17.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:17.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:17.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.590284+0000", 2024-09-10T18:37:17.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:17.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:17.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.590279+0000", 2024-09-10T18:37:17.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999952 2024-09-10T18:37:17.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:17.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.590330+0000", 2024-09-10T18:37:17.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0407e-05 2024-09-10T18:37:17.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:17.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.590346+0000", 2024-09-10T18:37:17.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6711999999999999e-05 2024-09-10T18:37:17.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:17.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.590356+0000", 2024-09-10T18:37:17.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1870000000000003e-06 2024-09-10T18:37:17.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:17.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.590405+0000", 2024-09-10T18:37:17.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9518999999999998e-05 2024-09-10T18:37:17.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:17.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.590525+0000", 2024-09-10T18:37:17.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011958199999999999 2024-09-10T18:37:17.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,6", 2024-09-10T18:37:17.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.590656+0000", 2024-09-10T18:37:17.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013153199999999999 2024-09-10T18:37:17.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:17.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.593234+0000", 2024-09-10T18:37:17.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025776950000000001 2024-09-10T18:37:17.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:17.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595488+0000", 2024-09-10T18:37:17.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002253873 2024-09-10T18:37:17.604 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:17.604 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:17.604 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:17.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.605 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:100001 10.e 10:736cd153:::benchmark_data_smithi028_161947_object100000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-10T18:37:17.605 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:37:17.593115+0000", 2024-09-10T18:37:17.605 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.003473636, 2024-09-10T18:37:17.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035958460000000002, 2024-09-10T18:37:17.605 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:17.605 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:37:17.605 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:17.605 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:37:17.605 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:37:17.605 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 100001 2024-09-10T18:37:17.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.605 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:17.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:17.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.593115+0000", 2024-09-10T18:37:17.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:17.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:17.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.593115+0000", 2024-09-10T18:37:17.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:17.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:17.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.593112+0000", 2024-09-10T18:37:17.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:37:17.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:17.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.593152+0000", 2024-09-10T18:37:17.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0191999999999997e-05 2024-09-10T18:37:17.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:17.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.593167+0000", 2024-09-10T18:37:17.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4823e-05 2024-09-10T18:37:17.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:17.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.593174+0000", 2024-09-10T18:37:17.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0600000000000002e-06 2024-09-10T18:37:17.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:17.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.593258+0000", 2024-09-10T18:37:17.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3799000000000002e-05 2024-09-10T18:37:17.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:17.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.593395+0000", 2024-09-10T18:37:17.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013706699999999999 2024-09-10T18:37:17.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,6", 2024-09-10T18:37:17.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.593532+0000", 2024-09-10T18:37:17.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013705499999999999 2024-09-10T18:37:17.608 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:17.608 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:17.609 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:17.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.609 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:100007 10.e 10:772f8513:::benchmark_data_smithi028_161947_object100006:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-10T18:37:17.609 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:37:17.595231+0000", 2024-09-10T18:37:17.609 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0013575499999999999, 2024-09-10T18:37:17.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0015480100000000001, 2024-09-10T18:37:17.609 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:17.609 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:37:17.609 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:17.609 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:37:17.609 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:37:17.609 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 100007 2024-09-10T18:37:17.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.610 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:17.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:17.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595231+0000", 2024-09-10T18:37:17.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:17.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:17.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595231+0000", 2024-09-10T18:37:17.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:17.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:17.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595229+0000", 2024-09-10T18:37:17.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:37:17.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:17.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595261+0000", 2024-09-10T18:37:17.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2189e-05 2024-09-10T18:37:17.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:17.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595279+0000", 2024-09-10T18:37:17.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8247e-05 2024-09-10T18:37:17.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:17.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595286+0000", 2024-09-10T18:37:17.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4309999999999999e-06 2024-09-10T18:37:17.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.612 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:17.612 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595513+0000", 2024-09-10T18:37:17.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022701500000000001 2024-09-10T18:37:17.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.612 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:17.612 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595616+0000", 2024-09-10T18:37:17.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103213 2024-09-10T18:37:17.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.613 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,6", 2024-09-10T18:37:17.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595745+0000", 2024-09-10T18:37:17.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000129252 2024-09-10T18:37:17.613 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:17.613 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:17.613 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:17.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.613 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.15693.0:100002 10.d e335/314 10:bf3dd05c:::benchmark_data_smithi028_161947_object100001:head v 335'6264, mlcod=335'6262)", 2024-09-10T18:37:17.613 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:37:17.595673+0000", 2024-09-10T18:37:17.613 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.00091492400000000001, 2024-09-10T18:37:17.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001171334, 2024-09-10T18:37:17.613 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:17.613 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:37:17.614 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:17.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:17.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595673+0000", 2024-09-10T18:37:17.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:17.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:17.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595673+0000", 2024-09-10T18:37:17.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:17.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:17.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595672+0000", 2024-09-10T18:37:17.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:37:17.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:17.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595690+0000", 2024-09-10T18:37:17.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7187e-05 2024-09-10T18:37:17.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:17.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595698+0000", 2024-09-10T18:37:17.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0609999999999996e-06 2024-09-10T18:37:17.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:17.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595701+0000", 2024-09-10T18:37:17.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8199999999999998e-06 2024-09-10T18:37:17.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:17.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595738+0000", 2024-09-10T18:37:17.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6979e-05 2024-09-10T18:37:17.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:17.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:17.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:17.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:17.595757+0000", 2024-09-10T18:37:17.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8437999999999999e-05 2024-09-10T18:37:17.616 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:17.617 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:17.617 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:17.617 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:17.617 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-09-10T18:37:17.617 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 4 2024-09-10T18:37:17.617 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:37:17.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:17.673 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:17.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:17.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:17.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:17.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:17 smithi204 ceph-mon[32357]: pgmap v717: 178 pgs: 178 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 813 B/s rd, 93 MiB/s wr, 1.49k op/s; 17 B/s, 1 objects/s recovering 2024-09-10T18:37:17.873 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:17.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:17.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:17.975 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:18.004 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:18.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:18.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:18.028 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 59 16 100535 100519 106.463 77.3125 0.0235125 0.00938772 2024-09-10T18:37:18.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:37:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:18.184 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:18.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:18.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:18.315 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:18.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:18.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:18.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:18.463 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:37:18.466 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:37:18.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:18.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:18.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:18.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:18.616 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:18.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:18.705 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-10T18:37:18.717 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:18.789 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:18.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:18.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:18.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:18.930 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:18.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:18.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:19.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:37:19.025709+0000 min lat: 0.00134649 max lat: 0.229265 avg lat: 0.00943872 2024-09-10T18:37:19.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:37:19.027 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 60 16 101690 101674 105.891 72.1875 0.0126269 0.00943872 2024-09-10T18:37:19.027 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:37:19.028 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:37:19.032 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:19.053 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:19 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/782489100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:37:19.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:19.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:19.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:19.069 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:37:19.069 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:37:19.098 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:37:19.101 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:37:19.101 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:37:19.133 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:37:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:37:19.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:19.171 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:37:19.172 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:37:19.197 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:19.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:19.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:19.204 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:37:19.205 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:37:19.240 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:37:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:37:19.273 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:37:19.274 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:37:19.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:19 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/782489100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:37:19.291 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:19 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/782489100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:37:19.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:19.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:19.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:19.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:19.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:19.485 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:19.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:19.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:19.587 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:19.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:19.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:19.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:19.769 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:19.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:19.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:19.871 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:19.904 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:19.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:19.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:20.006 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:20.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 61 16 102991 102975 105.488 81.3125 0.0115401 0.00947522 2024-09-10T18:37:20.041 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:20.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:20.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:20.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:20.189 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:20.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:20.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:20.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:20.329 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:20.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:20.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:20.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:20 smithi204 ceph-mon[32357]: from='client.25645 v1:172.21.15.28:0/3537476701' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:37:20.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:20 smithi204 ceph-mon[32357]: pgmap v718: 178 pgs: 178 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 885 op/s; 13 B/s, 0 objects/s recovering 2024-09-10T18:37:20.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:20.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:20.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:20.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:20 smithi028 ceph-mon[29291]: from='client.25645 v1:172.21.15.28:0/3537476701' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:37:20.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:20 smithi028 ceph-mon[29291]: pgmap v718: 178 pgs: 178 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 885 op/s; 13 B/s, 0 objects/s recovering 2024-09-10T18:37:20.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:20 smithi028 ceph-mon[39220]: from='client.25645 v1:172.21.15.28:0/3537476701' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:37:20.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:20 smithi028 ceph-mon[39220]: pgmap v718: 178 pgs: 178 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 885 op/s; 13 B/s, 0 objects/s recovering 2024-09-10T18:37:20.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:20.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:20.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:20.635 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":717,"stamp":"2024-09-10T18:37:17.085952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6398810332,"num_objects":97825,"num_object_clones":0,"num_object_copies":293475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97825,"num_whiteouts":0,"num_read":2083,"num_read_kb":1835,"num_write":98345,"num_write_kb":6248299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79989,"ondisk_log_size":79989,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":535,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22788200,"kb_used_data":18602252,"kb_used_omap":19,"kb_used_meta":4185836,"kb_avail":727173016,"statfs":{"total":767960285184,"available":744625168384,"internally_reserved":0,"allocated":19048706048,"data_stored":19044794222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20033,"internal_metadata":4286296511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,10,6,9,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":614531072,"num_objects":9377,"num_object_clones":0,"num_object_copies":28131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9377,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":9377,"num_write_kb":600128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.293159"},"pg_stats":[{"pgid":"1.1d","version":"76'81","reported_seq":69,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.809714+0000","last_change":"2024-09-10T18:35:17.818822+0000","last_active":"2024-09-10T18:37:11.809714+0000","last_peered":"2024-09-10T18:37:11.809714+0000","last_clean":"2024-09-10T18:37:11.809714+0000","last_became_active":"2024-09-10T18:34:34.865894+0000","last_became_peered":"2024-09-10T18:34:34.865894+0000","last_unstale":"2024-09-10T18:37:11.809714+0000","last_undegraded":"2024-09-10T18:37:11.809714+0000","last_fullsized":"2024-09-10T18:37:11.809714+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:17.818727+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:17.818727+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037738960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":355,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806771+0000","last_change":"2024-09-10T18:36:17.645197+0000","last_active":"2024-09-10T18:37:11.806771+0000","last_peered":"2024-09-10T18:37:11.806771+0000","last_clean":"2024-09-10T18:37:11.806771+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:37:11.806771+0000","last_undegraded":"2024-09-10T18:37:11.806771+0000","last_fullsized":"2024-09-10T18:37:11.806771+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:17.645146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:36:17.645146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:43.759721+0000","scrub_duration":0.003140618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":383,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.801590+0000","last_change":"2024-09-10T18:35:20.421834+0000","last_active":"2024-09-10T18:37:11.801590+0000","last_peered":"2024-09-10T18:37:11.801590+0000","last_clean":"2024-09-10T18:37:11.801590+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:37:11.801590+0000","last_undegraded":"2024-09-10T18:37:11.801590+0000","last_fullsized":"2024-09-10T18:37:11.801590+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:35:20.421727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:20.421727+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010997518,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":360,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:12.855060+0000","last_change":"2024-09-10T18:37:12.854944+0000","last_active":"2024-09-10T18:37:12.855060+0000","last_peered":"2024-09-10T18:37:12.855060+0000","last_clean":"2024-09-10T18:37:12.855060+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:37:12.855060+0000","last_undegraded":"2024-09-10T18:37:12.855060+0000","last_fullsized":"2024-09-10T18:37:12.855060+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:12.854879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:12.854879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:41.911709+0000","scrub_duration":0.019720808999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":350,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.801070+0000","last_change":"2024-09-10T18:36:52.118582+0000","last_active":"2024-09-10T18:37:11.801070+0000","last_peered":"2024-09-10T18:37:11.801070+0000","last_clean":"2024-09-10T18:37:11.801070+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:37:11.801070+0000","last_undegraded":"2024-09-10T18:37:11.801070+0000","last_fullsized":"2024-09-10T18:37:11.801070+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:52.118515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:36:52.118515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:16.344464+0000","scrub_duration":0.0076434099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'81","reported_seq":64,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.808202+0000","last_change":"2024-09-10T18:36:48.858044+0000","last_active":"2024-09-10T18:37:11.808202+0000","last_peered":"2024-09-10T18:37:11.808202+0000","last_clean":"2024-09-10T18:37:11.808202+0000","last_became_active":"2024-09-10T18:34:34.867388+0000","last_became_peered":"2024-09-10T18:34:34.867388+0000","last_unstale":"2024-09-10T18:37:11.808202+0000","last_undegraded":"2024-09-10T18:37:11.808202+0000","last_fullsized":"2024-09-10T18:37:11.808202+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:48.857902+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:48.857902+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:09.493773+0000","scrub_duration":0.0059112879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":347,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.796003+0000","last_change":"2024-09-10T18:34:16.293404+0000","last_active":"2024-09-10T18:37:11.796003+0000","last_peered":"2024-09-10T18:37:11.796003+0000","last_clean":"2024-09-10T18:37:11.796003+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:37:11.796003+0000","last_undegraded":"2024-09-10T18:37:11.796003+0000","last_fullsized":"2024-09-10T18:37:11.796003+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:16.293206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:16.293206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021119960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":353,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.797722+0000","last_change":"2024-09-10T18:34:22.415416+0000","last_active":"2024-09-10T18:37:11.797722+0000","last_peered":"2024-09-10T18:37:11.797722+0000","last_clean":"2024-09-10T18:37:11.797722+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:37:11.797722+0000","last_undegraded":"2024-09-10T18:37:11.797722+0000","last_fullsized":"2024-09-10T18:37:11.797722+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:22.415093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.415093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.068845071999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'34","reported_seq":394,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.796875+0000","last_change":"2024-09-10T18:36:10.786060+0000","last_active":"2024-09-10T18:37:11.796875+0000","last_peered":"2024-09-10T18:37:11.796875+0000","last_clean":"2024-09-10T18:37:11.796875+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:37:11.796875+0000","last_undegraded":"2024-09-10T18:37:11.796875+0000","last_fullsized":"2024-09-10T18:37:11.796875+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:36:10.786002+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:36:10.786002+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:36.303670+0000","scrub_duration":0.0021577860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":359,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.801508+0000","last_change":"2024-09-10T18:36:07.325812+0000","last_active":"2024-09-10T18:37:11.801508+0000","last_peered":"2024-09-10T18:37:11.801508+0000","last_clean":"2024-09-10T18:37:11.801508+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:37:11.801508+0000","last_undegraded":"2024-09-10T18:37:11.801508+0000","last_fullsized":"2024-09-10T18:37:11.801508+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:07.325744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.325744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:24.078418+0000","scrub_duration":0.001764406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":364,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806009+0000","last_change":"2024-09-10T18:36:09.700648+0000","last_active":"2024-09-10T18:37:11.806009+0000","last_peered":"2024-09-10T18:37:11.806009+0000","last_clean":"2024-09-10T18:37:11.806009+0000","last_became_active":"2024-09-10T18:26:27.644961+0000","last_became_peered":"2024-09-10T18:26:27.644961+0000","last_unstale":"2024-09-10T18:37:11.806009+0000","last_undegraded":"2024-09-10T18:37:11.806009+0000","last_fullsized":"2024-09-10T18:37:11.806009+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:09.700579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_clean_scrub_stamp":"2024-09-10T18:36:09.700579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00158731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'15","reported_seq":386,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.801269+0000","last_change":"2024-09-10T18:37:05.365783+0000","last_active":"2024-09-10T18:37:11.801269+0000","last_peered":"2024-09-10T18:37:11.801269+0000","last_clean":"2024-09-10T18:37:11.801269+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:37:11.801269+0000","last_undegraded":"2024-09-10T18:37:11.801269+0000","last_fullsized":"2024-09-10T18:37:11.801269+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:37:05.365702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:05.365702+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:27.722998+0000","scrub_duration":0.0029180819999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":360,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.805888+0000","last_change":"2024-09-10T18:36:52.071972+0000","last_active":"2024-09-10T18:37:11.805888+0000","last_peered":"2024-09-10T18:37:11.805888+0000","last_clean":"2024-09-10T18:37:11.805888+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:37:11.805888+0000","last_undegraded":"2024-09-10T18:37:11.805888+0000","last_fullsized":"2024-09-10T18:37:11.805888+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:52.071894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_clean_scrub_stamp":"2024-09-10T18:36:52.071894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:05.580562+0000","scrub_duration":0.0025636399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":359,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.802717+0000","last_change":"2024-09-10T18:36:54.584423+0000","last_active":"2024-09-10T18:37:11.802717+0000","last_peered":"2024-09-10T18:37:11.802717+0000","last_clean":"2024-09-10T18:37:11.802717+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:37:11.802717+0000","last_undegraded":"2024-09-10T18:37:11.802717+0000","last_fullsized":"2024-09-10T18:37:11.802717+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:54.583785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:54.583785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:23.333666+0000","scrub_duration":0.0082206150000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'81","reported_seq":133,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.797546+0000","last_change":"2024-09-10T18:37:06.945729+0000","last_active":"2024-09-10T18:37:11.797546+0000","last_peered":"2024-09-10T18:37:11.797546+0000","last_clean":"2024-09-10T18:37:11.797546+0000","last_became_active":"2024-09-10T18:34:34.863102+0000","last_became_peered":"2024-09-10T18:34:34.863102+0000","last_unstale":"2024-09-10T18:37:11.797546+0000","last_undegraded":"2024-09-10T18:37:11.797546+0000","last_fullsized":"2024-09-10T18:37:11.797546+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:06.945645+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:06.945645+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:11.913296+0000","scrub_duration":0.0045824999999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":364,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.804899+0000","last_change":"2024-09-10T18:36:56.124404+0000","last_active":"2024-09-10T18:37:11.804899+0000","last_peered":"2024-09-10T18:37:11.804899+0000","last_clean":"2024-09-10T18:37:11.804899+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:37:11.804899+0000","last_undegraded":"2024-09-10T18:37:11.804899+0000","last_fullsized":"2024-09-10T18:37:11.804899+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:56.124318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:17.464205+0000","last_clean_scrub_stamp":"2024-09-10T18:36:56.124318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:10.885488+0000","scrub_duration":0.0046808750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":398,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.800123+0000","last_change":"2024-09-10T18:37:09.383413+0000","last_active":"2024-09-10T18:37:11.800123+0000","last_peered":"2024-09-10T18:37:11.800123+0000","last_clean":"2024-09-10T18:37:11.800123+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:37:11.800123+0000","last_undegraded":"2024-09-10T18:37:11.800123+0000","last_fullsized":"2024-09-10T18:37:11.800123+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'9","last_scrub_stamp":"2024-09-10T18:37:09.383347+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:37:09.383347+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:10.796821+0000","scrub_duration":0.010843228999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":349,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.807096+0000","last_change":"2024-09-10T18:35:01.711169+0000","last_active":"2024-09-10T18:37:11.807096+0000","last_peered":"2024-09-10T18:37:11.807096+0000","last_clean":"2024-09-10T18:37:11.807096+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:37:11.807096+0000","last_undegraded":"2024-09-10T18:37:11.807096+0000","last_fullsized":"2024-09-10T18:37:11.807096+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:01.710785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:01.710785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033386729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":351,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806160+0000","last_change":"2024-09-10T18:36:15.726285+0000","last_active":"2024-09-10T18:37:11.806160+0000","last_peered":"2024-09-10T18:37:11.806160+0000","last_clean":"2024-09-10T18:37:11.806160+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:37:11.806160+0000","last_undegraded":"2024-09-10T18:37:11.806160+0000","last_fullsized":"2024-09-10T18:37:11.806160+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:15.726200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:36:15.726200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:20.772259+0000","scrub_duration":0.0020103579999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'81","reported_seq":216,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.796245+0000","last_change":"2024-09-10T18:37:07.984531+0000","last_active":"2024-09-10T18:37:11.796245+0000","last_peered":"2024-09-10T18:37:11.796245+0000","last_clean":"2024-09-10T18:37:11.796245+0000","last_became_active":"2024-09-10T18:34:34.862917+0000","last_became_peered":"2024-09-10T18:34:34.862917+0000","last_unstale":"2024-09-10T18:37:11.796245+0000","last_undegraded":"2024-09-10T18:37:11.796245+0000","last_fullsized":"2024-09-10T18:37:11.796245+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:07.984446+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:07.984446+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:16.037756+0000","scrub_duration":0.0054176850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":357,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.801492+0000","last_change":"2024-09-10T18:36:06.360654+0000","last_active":"2024-09-10T18:37:11.801492+0000","last_peered":"2024-09-10T18:37:11.801492+0000","last_clean":"2024-09-10T18:37:11.801492+0000","last_became_active":"2024-09-10T18:26:27.650774+0000","last_became_peered":"2024-09-10T18:26:27.650774+0000","last_unstale":"2024-09-10T18:37:11.801492+0000","last_undegraded":"2024-09-10T18:37:11.801492+0000","last_fullsized":"2024-09-10T18:37:11.801492+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.360595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.360595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:21.064681+0000","scrub_duration":0.0012270849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":361,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:13.830464+0000","last_change":"2024-09-10T18:37:13.830411+0000","last_active":"2024-09-10T18:37:13.830464+0000","last_peered":"2024-09-10T18:37:13.830464+0000","last_clean":"2024-09-10T18:37:13.830464+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:37:13.830464+0000","last_undegraded":"2024-09-10T18:37:13.830464+0000","last_fullsized":"2024-09-10T18:37:13.830464+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:13.830325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:13.830325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:15.627331+0000","scrub_duration":0.0070701799999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"240'27","reported_seq":424,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.807092+0000","last_change":"2024-09-10T18:36:17.526735+0000","last_active":"2024-09-10T18:37:11.807092+0000","last_peered":"2024-09-10T18:37:11.807092+0000","last_clean":"2024-09-10T18:37:11.807092+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:37:11.807092+0000","last_undegraded":"2024-09-10T18:37:11.807092+0000","last_fullsized":"2024-09-10T18:37:11.807092+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:36:17.526667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:17.526667+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:45.386424+0000","scrub_duration":0.0019245169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":360,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.809960+0000","last_change":"2024-09-10T18:35:13.892354+0000","last_active":"2024-09-10T18:37:11.809960+0000","last_peered":"2024-09-10T18:37:11.809960+0000","last_clean":"2024-09-10T18:37:11.809960+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:37:11.809960+0000","last_undegraded":"2024-09-10T18:37:11.809960+0000","last_fullsized":"2024-09-10T18:37:11.809960+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:13.892271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:13.892271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058623920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'81","reported_seq":286,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.804659+0000","last_change":"2024-09-10T18:36:06.623015+0000","last_active":"2024-09-10T18:37:11.804659+0000","last_peered":"2024-09-10T18:37:11.804659+0000","last_clean":"2024-09-10T18:37:11.804659+0000","last_became_active":"2024-09-10T18:34:34.855693+0000","last_became_peered":"2024-09-10T18:34:34.855693+0000","last_unstale":"2024-09-10T18:37:11.804659+0000","last_undegraded":"2024-09-10T18:37:11.804659+0000","last_fullsized":"2024-09-10T18:37:11.804659+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:06.622956+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.622956+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:16.207315+0000","scrub_duration":0.0017570750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":347,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.796781+0000","last_change":"2024-09-10T18:34:21.344309+0000","last_active":"2024-09-10T18:37:11.796781+0000","last_peered":"2024-09-10T18:37:11.796781+0000","last_clean":"2024-09-10T18:37:11.796781+0000","last_became_active":"2024-09-10T18:26:27.640035+0000","last_became_peered":"2024-09-10T18:26:27.640035+0000","last_unstale":"2024-09-10T18:37:11.796781+0000","last_undegraded":"2024-09-10T18:37:11.796781+0000","last_fullsized":"2024-09-10T18:37:11.796781+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:21.344236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.344236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040101240000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":353,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.801389+0000","last_change":"2024-09-10T18:35:04.319486+0000","last_active":"2024-09-10T18:37:11.801389+0000","last_peered":"2024-09-10T18:37:11.801389+0000","last_clean":"2024-09-10T18:37:11.801389+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:37:11.801389+0000","last_undegraded":"2024-09-10T18:37:11.801389+0000","last_fullsized":"2024-09-10T18:37:11.801389+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:04.319395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:35:04.319395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002865018,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'23","reported_seq":381,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.796364+0000","last_change":"2024-09-10T18:35:14.739218+0000","last_active":"2024-09-10T18:37:11.796364+0000","last_peered":"2024-09-10T18:37:11.796364+0000","last_clean":"2024-09-10T18:37:11.796364+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:37:11.796364+0000","last_undegraded":"2024-09-10T18:37:11.796364+0000","last_fullsized":"2024-09-10T18:37:11.796364+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'23","last_scrub_stamp":"2024-09-10T18:35:14.739123+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:35:14.739123+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052949870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":355,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:14.371176+0000","last_change":"2024-09-10T18:37:14.371176+0000","last_active":"2024-09-10T18:37:14.371176+0000","last_peered":"2024-09-10T18:37:14.371176+0000","last_clean":"2024-09-10T18:37:14.371176+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:37:14.371176+0000","last_undegraded":"2024-09-10T18:37:14.371176+0000","last_fullsized":"2024-09-10T18:37:14.371176+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:08.266279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:08.266279+0000","last_clean_scrub_stamp":"2024-09-10T18:35:08.266279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041938189999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'81","reported_seq":274,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.805305+0000","last_change":"2024-09-10T18:36:07.673249+0000","last_active":"2024-09-10T18:37:11.805305+0000","last_peered":"2024-09-10T18:37:11.805305+0000","last_clean":"2024-09-10T18:37:11.805305+0000","last_became_active":"2024-09-10T18:34:34.855038+0000","last_became_peered":"2024-09-10T18:34:34.855038+0000","last_unstale":"2024-09-10T18:37:11.805305+0000","last_undegraded":"2024-09-10T18:37:11.805305+0000","last_fullsized":"2024-09-10T18:37:11.805305+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:07.673176+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.673176+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:33.711087+0000","scrub_duration":0.0018195469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":349,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.807994+0000","last_change":"2024-09-10T18:35:20.905675+0000","last_active":"2024-09-10T18:37:11.807994+0000","last_peered":"2024-09-10T18:37:11.807994+0000","last_clean":"2024-09-10T18:37:11.807994+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:37:11.807994+0000","last_undegraded":"2024-09-10T18:37:11.807994+0000","last_fullsized":"2024-09-10T18:37:11.807994+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:35:20.905576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:20.905576+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049065819999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":374,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.805156+0000","last_change":"2024-09-10T18:37:08.821787+0000","last_active":"2024-09-10T18:37:11.805156+0000","last_peered":"2024-09-10T18:37:11.805156+0000","last_clean":"2024-09-10T18:37:11.805156+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:37:11.805156+0000","last_undegraded":"2024-09-10T18:37:11.805156+0000","last_fullsized":"2024-09-10T18:37:11.805156+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:08.821692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:08.821692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:32.765319+0000","scrub_duration":0.0031507520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'25","reported_seq":398,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.803689+0000","last_change":"2024-09-10T18:37:00.670455+0000","last_active":"2024-09-10T18:37:11.803689+0000","last_peered":"2024-09-10T18:37:11.803689+0000","last_clean":"2024-09-10T18:37:11.803689+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:37:11.803689+0000","last_undegraded":"2024-09-10T18:37:11.803689+0000","last_fullsized":"2024-09-10T18:37:11.803689+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:37:00.670369+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:37:00.670369+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:06.286835+0000","scrub_duration":0.0057824440000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"325'12","reported_seq":31,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:14.358266+0000","last_change":"2024-09-10T18:36:52.891388+0000","last_active":"2024-09-10T18:37:14.358266+0000","last_peered":"2024-09-10T18:37:14.358266+0000","last_clean":"2024-09-10T18:37:14.358266+0000","last_became_active":"2024-09-10T18:36:52.882218+0000","last_became_peered":"2024-09-10T18:36:52.882218+0000","last_unstale":"2024-09-10T18:37:14.358266+0000","last_undegraded":"2024-09-10T18:37:14.358266+0000","last_fullsized":"2024-09-10T18:37:14.358266+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11","last_scrub_stamp":"2024-09-10T18:36:11.854492+0000","last_deep_scrub":"217'11","last_deep_scrub_stamp":"2024-09-10T18:30:03.161985+0000","last_clean_scrub_stamp":"2024-09-10T18:36:11.854492+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:29.616202+0000","scrub_duration":0.0073290009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1","version":"335'6007","reported_seq":6036,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:14.888957+0000","last_change":"2024-09-10T18:36:07.904585+0000","last_active":"2024-09-10T18:37:14.888957+0000","last_peered":"2024-09-10T18:37:14.888957+0000","last_clean":"2024-09-10T18:37:14.888957+0000","last_became_active":"2024-09-10T18:36:07.904319+0000","last_became_peered":"2024-09-10T18:36:07.904319+0000","last_unstale":"2024-09-10T18:37:14.888957+0000","last_undegraded":"2024-09-10T18:37:14.888957+0000","last_fullsized":"2024-09-10T18:37:14.888957+0000","mapping_epoch":314,"log_start":"320'2000","ondisk_log_start":"320'2000","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:18.814945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393674752,"num_objects":6007,"num_object_clones":0,"num_object_copies":18021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6007,"num_write_kb":384448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":280,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.804917+0000","last_change":"2024-09-10T18:35:59.488083+0000","last_active":"2024-09-10T18:37:11.804917+0000","last_peered":"2024-09-10T18:37:11.804917+0000","last_clean":"2024-09-10T18:37:11.804917+0000","last_became_active":"2024-09-10T18:34:32.823230+0000","last_became_peered":"2024-09-10T18:34:32.823230+0000","last_unstale":"2024-09-10T18:37:11.804917+0000","last_undegraded":"2024-09-10T18:37:11.804917+0000","last_fullsized":"2024-09-10T18:37:11.804917+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:59.488014+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:59.488014+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:22.512352+0000","scrub_duration":0.0017684759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"240'34","reported_seq":425,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.810701+0000","last_change":"2024-09-10T18:37:03.818405+0000","last_active":"2024-09-10T18:37:11.810701+0000","last_peered":"2024-09-10T18:37:11.810701+0000","last_clean":"2024-09-10T18:37:11.810701+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:37:11.810701+0000","last_undegraded":"2024-09-10T18:37:11.810701+0000","last_fullsized":"2024-09-10T18:37:11.810701+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:37:03.818290+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:37:03.818290+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:17.664855+0000","scrub_duration":0.0051578769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":346,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.798142+0000","last_change":"2024-09-10T18:34:20.305626+0000","last_active":"2024-09-10T18:37:11.798142+0000","last_peered":"2024-09-10T18:37:11.798142+0000","last_clean":"2024-09-10T18:37:11.798142+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:37:11.798142+0000","last_undegraded":"2024-09-10T18:37:11.798142+0000","last_fullsized":"2024-09-10T18:37:11.798142+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:20.305547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:20.305547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046035989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":548,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.801588+0000","last_change":"2024-09-10T18:37:03.315311+0000","last_active":"2024-09-10T18:37:11.801588+0000","last_peered":"2024-09-10T18:37:11.801588+0000","last_clean":"2024-09-10T18:37:11.801588+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:37:11.801588+0000","last_undegraded":"2024-09-10T18:37:11.801588+0000","last_fullsized":"2024-09-10T18:37:11.801588+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:37:03.315226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:03.315226+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:23.900377+0000","scrub_duration":0.017386324000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":358,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.805344+0000","last_change":"2024-09-10T18:37:09.280044+0000","last_active":"2024-09-10T18:37:11.805344+0000","last_peered":"2024-09-10T18:37:11.805344+0000","last_clean":"2024-09-10T18:37:11.805344+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:37:11.805344+0000","last_undegraded":"2024-09-10T18:37:11.805344+0000","last_fullsized":"2024-09-10T18:37:11.805344+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:09.279969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:37:09.279969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:18.441371+0000","scrub_duration":0.0041856610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"335'5950","reported_seq":5979,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:14.884271+0000","last_change":"2024-09-10T18:36:07.900391+0000","last_active":"2024-09-10T18:37:14.884271+0000","last_peered":"2024-09-10T18:37:14.884271+0000","last_clean":"2024-09-10T18:37:14.884271+0000","last_became_active":"2024-09-10T18:36:07.900059+0000","last_became_peered":"2024-09-10T18:36:07.900059+0000","last_unstale":"2024-09-10T18:37:14.884271+0000","last_undegraded":"2024-09-10T18:37:14.884271+0000","last_fullsized":"2024-09-10T18:37:14.884271+0000","mapping_epoch":314,"log_start":"320'1900","ondisk_log_start":"320'1900","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4050,"log_dups_size":0,"ondisk_log_size":4050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:24.159285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389939200,"num_objects":5950,"num_object_clones":0,"num_object_copies":17850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5950,"num_write_kb":380800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":289,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.802090+0000","last_change":"2024-09-10T18:35:56.307718+0000","last_active":"2024-09-10T18:37:11.802090+0000","last_peered":"2024-09-10T18:37:11.802090+0000","last_clean":"2024-09-10T18:37:11.802090+0000","last_became_active":"2024-09-10T18:34:32.815839+0000","last_became_peered":"2024-09-10T18:34:32.815839+0000","last_unstale":"2024-09-10T18:37:11.802090+0000","last_undegraded":"2024-09-10T18:37:11.802090+0000","last_fullsized":"2024-09-10T18:37:11.802090+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:56.307649+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:56.307649+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:20.644586+0000","scrub_duration":0.0015720840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":394,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.797972+0000","last_change":"2024-09-10T18:36:07.782175+0000","last_active":"2024-09-10T18:37:11.797972+0000","last_peered":"2024-09-10T18:37:11.797972+0000","last_clean":"2024-09-10T18:37:11.797972+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:37:11.797972+0000","last_undegraded":"2024-09-10T18:37:11.797972+0000","last_fullsized":"2024-09-10T18:37:11.797972+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-09-10T18:36:07.782112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.782112+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:16.926976+0000","scrub_duration":0.00244122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":346,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.797463+0000","last_change":"2024-09-10T18:35:15.765590+0000","last_active":"2024-09-10T18:37:11.797463+0000","last_peered":"2024-09-10T18:37:11.797463+0000","last_clean":"2024-09-10T18:37:11.797463+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:37:11.797463+0000","last_undegraded":"2024-09-10T18:37:11.797463+0000","last_fullsized":"2024-09-10T18:37:11.797463+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:15.765392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:15.765392+0000","last_clean_scrub_stamp":"2024-09-10T18:35:15.765392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01380637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":349,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.799894+0000","last_change":"2024-09-10T18:35:10.270036+0000","last_active":"2024-09-10T18:37:11.799894+0000","last_peered":"2024-09-10T18:37:11.799894+0000","last_clean":"2024-09-10T18:37:11.799894+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:37:11.799894+0000","last_undegraded":"2024-09-10T18:37:11.799894+0000","last_fullsized":"2024-09-10T18:37:11.799894+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:10.269951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:35:10.269951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073211559999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":347,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.801193+0000","last_change":"2024-09-10T18:37:06.386024+0000","last_active":"2024-09-10T18:37:11.801193+0000","last_peered":"2024-09-10T18:37:11.801193+0000","last_clean":"2024-09-10T18:37:11.801193+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:37:11.801193+0000","last_undegraded":"2024-09-10T18:37:11.801193+0000","last_fullsized":"2024-09-10T18:37:11.801193+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:06.385944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:06.385944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:32.414011+0000","scrub_duration":0.003928651,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"335'6158","reported_seq":6187,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:15.964752+0000","last_change":"2024-09-10T18:36:07.902153+0000","last_active":"2024-09-10T18:37:15.964752+0000","last_peered":"2024-09-10T18:37:15.964752+0000","last_clean":"2024-09-10T18:37:15.964752+0000","last_became_active":"2024-09-10T18:36:07.901822+0000","last_became_peered":"2024-09-10T18:36:07.901822+0000","last_unstale":"2024-09-10T18:37:15.964752+0000","last_undegraded":"2024-09-10T18:37:15.964752+0000","last_fullsized":"2024-09-10T18:37:15.964752+0000","mapping_epoch":314,"log_start":"320'1200","ondisk_log_start":"320'1200","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4958,"log_dups_size":0,"ondisk_log_size":4958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:27.350431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403570688,"num_objects":6158,"num_object_clones":0,"num_object_copies":18474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6158,"num_write_kb":394112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":261,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.800115+0000","last_change":"2024-09-10T18:35:13.749157+0000","last_active":"2024-09-10T18:37:11.800115+0000","last_peered":"2024-09-10T18:37:11.800115+0000","last_clean":"2024-09-10T18:37:11.800115+0000","last_became_active":"2024-09-10T18:34:00.166362+0000","last_became_peered":"2024-09-10T18:34:00.166362+0000","last_unstale":"2024-09-10T18:37:11.800115+0000","last_undegraded":"2024-09-10T18:37:11.800115+0000","last_fullsized":"2024-09-10T18:37:11.800115+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:13.748963+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:13.748963+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013370432999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"240'28","reported_seq":400,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.800100+0000","last_change":"2024-09-10T18:36:16.155412+0000","last_active":"2024-09-10T18:37:11.800100+0000","last_peered":"2024-09-10T18:37:11.800100+0000","last_clean":"2024-09-10T18:37:11.800100+0000","last_became_active":"2024-09-10T18:26:27.651364+0000","last_became_peered":"2024-09-10T18:26:27.651364+0000","last_unstale":"2024-09-10T18:37:11.800100+0000","last_undegraded":"2024-09-10T18:37:11.800100+0000","last_fullsized":"2024-09-10T18:37:11.800100+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'28","last_scrub_stamp":"2024-09-10T18:36:16.155287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:16.155287+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:31.419623+0000","scrub_duration":0.0027352050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":355,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.803854+0000","last_change":"2024-09-10T18:36:09.716889+0000","last_active":"2024-09-10T18:37:11.803854+0000","last_peered":"2024-09-10T18:37:11.803854+0000","last_clean":"2024-09-10T18:37:11.803854+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:37:11.803854+0000","last_undegraded":"2024-09-10T18:37:11.803854+0000","last_fullsized":"2024-09-10T18:37:11.803854+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:09.716820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:36:09.716820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:34.441003+0000","scrub_duration":0.0037024689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":539,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.797821+0000","last_change":"2024-09-10T18:36:13.682715+0000","last_active":"2024-09-10T18:37:11.797821+0000","last_peered":"2024-09-10T18:37:11.797821+0000","last_clean":"2024-09-10T18:37:11.797821+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:37:11.797821+0000","last_undegraded":"2024-09-10T18:37:11.797821+0000","last_fullsized":"2024-09-10T18:37:11.797821+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:36:13.682649+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:36:13.682649+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:30.845820+0000","scrub_duration":0.001699954,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":357,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806662+0000","last_change":"2024-09-10T18:36:53.979146+0000","last_active":"2024-09-10T18:37:11.806662+0000","last_peered":"2024-09-10T18:37:11.806662+0000","last_clean":"2024-09-10T18:37:11.806662+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:37:11.806662+0000","last_undegraded":"2024-09-10T18:37:11.806662+0000","last_fullsized":"2024-09-10T18:37:11.806662+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:53.979093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:36:53.979093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:15.772937+0000","scrub_duration":0.0047648930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"335'6057","reported_seq":6086,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:15.073235+0000","last_change":"2024-09-10T18:36:07.904923+0000","last_active":"2024-09-10T18:37:15.073235+0000","last_peered":"2024-09-10T18:37:15.073235+0000","last_clean":"2024-09-10T18:37:15.073235+0000","last_became_active":"2024-09-10T18:36:07.904606+0000","last_became_peered":"2024-09-10T18:36:07.904606+0000","last_unstale":"2024-09-10T18:37:15.073235+0000","last_undegraded":"2024-09-10T18:37:15.073235+0000","last_fullsized":"2024-09-10T18:37:15.073235+0000","mapping_epoch":314,"log_start":"320'1800","ondisk_log_start":"320'1800","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4257,"log_dups_size":0,"ondisk_log_size":4257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:26.465282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396951552,"num_objects":6057,"num_object_clones":0,"num_object_copies":18171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6057,"num_write_kb":387648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":350,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.797907+0000","last_change":"2024-09-10T18:36:09.909074+0000","last_active":"2024-09-10T18:37:11.797907+0000","last_peered":"2024-09-10T18:37:11.797907+0000","last_clean":"2024-09-10T18:37:11.797907+0000","last_became_active":"2024-09-10T18:34:32.818323+0000","last_became_peered":"2024-09-10T18:34:32.818323+0000","last_unstale":"2024-09-10T18:37:11.797907+0000","last_undegraded":"2024-09-10T18:37:11.797907+0000","last_fullsized":"2024-09-10T18:37:11.797907+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:09.908960+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:09.908960+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:25.472837+0000","scrub_duration":0.0019456180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":407,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.805563+0000","last_change":"2024-09-10T18:37:07.798909+0000","last_active":"2024-09-10T18:37:11.805563+0000","last_peered":"2024-09-10T18:37:11.805563+0000","last_clean":"2024-09-10T18:37:11.805563+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:37:11.805563+0000","last_undegraded":"2024-09-10T18:37:11.805563+0000","last_fullsized":"2024-09-10T18:37:11.805563+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:37:07.798798+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:37:07.798798+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:11.466713+0000","scrub_duration":0.0020202269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":354,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806105+0000","last_change":"2024-09-10T18:37:11.243616+0000","last_active":"2024-09-10T18:37:11.806105+0000","last_peered":"2024-09-10T18:37:11.806105+0000","last_clean":"2024-09-10T18:37:11.806105+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:37:11.806105+0000","last_undegraded":"2024-09-10T18:37:11.806105+0000","last_fullsized":"2024-09-10T18:37:11.806105+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:11.243548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:37:11.243548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:29.339597+0000","scrub_duration":0.015664832,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"263'11","reported_seq":558,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:14.358158+0000","last_change":"2024-09-10T18:36:51.573695+0000","last_active":"2024-09-10T18:37:14.358158+0000","last_peered":"2024-09-10T18:37:14.358158+0000","last_clean":"2024-09-10T18:37:14.358158+0000","last_became_active":"2024-09-10T18:26:27.639032+0000","last_became_peered":"2024-09-10T18:26:27.639032+0000","last_unstale":"2024-09-10T18:37:14.358158+0000","last_undegraded":"2024-09-10T18:37:14.358158+0000","last_fullsized":"2024-09-10T18:37:14.358158+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'11","last_scrub_stamp":"2024-09-10T18:36:51.573610+0000","last_deep_scrub":"263'11","last_deep_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_clean_scrub_stamp":"2024-09-10T18:36:51.573610+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:06.768796+0000","scrub_duration":0.0034993289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":349,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.803070+0000","last_change":"2024-09-10T18:35:55.632357+0000","last_active":"2024-09-10T18:37:11.803070+0000","last_peered":"2024-09-10T18:37:11.803070+0000","last_clean":"2024-09-10T18:37:11.803070+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:37:11.803070+0000","last_undegraded":"2024-09-10T18:37:11.803070+0000","last_fullsized":"2024-09-10T18:37:11.803070+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:55.632306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:55.632306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:24.932351+0000","scrub_duration":0.001072857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"335'6158","reported_seq":6429,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:15.950431+0000","last_change":"2024-09-10T18:37:12.152091+0000","last_active":"2024-09-10T18:37:15.950431+0000","last_peered":"2024-09-10T18:37:15.950431+0000","last_clean":"2024-09-10T18:37:15.950431+0000","last_became_active":"2024-09-10T18:36:07.903637+0000","last_became_peered":"2024-09-10T18:36:07.903637+0000","last_unstale":"2024-09-10T18:37:15.950431+0000","last_undegraded":"2024-09-10T18:37:15.950431+0000","last_fullsized":"2024-09-10T18:37:15.950431+0000","mapping_epoch":314,"log_start":"320'1200","ondisk_log_start":"320'1200","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'5893","last_scrub_stamp":"2024-09-10T18:37:12.151975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:37:12.151975+0000","objects_scrubbed":5802,"log_size":4958,"log_dups_size":0,"ondisk_log_size":4958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:14.669808+0000","scrub_duration":2.3527053489999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403570688,"num_objects":6158,"num_object_clones":0,"num_object_copies":18474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6158,"num_write_kb":394112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":350,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.797658+0000","last_change":"2024-09-10T18:36:08.872676+0000","last_active":"2024-09-10T18:37:11.797658+0000","last_peered":"2024-09-10T18:37:11.797658+0000","last_clean":"2024-09-10T18:37:11.797658+0000","last_became_active":"2024-09-10T18:34:32.918946+0000","last_became_peered":"2024-09-10T18:34:32.918946+0000","last_unstale":"2024-09-10T18:37:11.797658+0000","last_undegraded":"2024-09-10T18:37:11.797658+0000","last_fullsized":"2024-09-10T18:37:11.797658+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:08.872575+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:08.872575+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:31.449589+0000","scrub_duration":0.0016711530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"240'29","reported_seq":414,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.802421+0000","last_change":"2024-09-10T18:36:59.647785+0000","last_active":"2024-09-10T18:37:11.802421+0000","last_peered":"2024-09-10T18:37:11.802421+0000","last_clean":"2024-09-10T18:37:11.802421+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:37:11.802421+0000","last_undegraded":"2024-09-10T18:37:11.802421+0000","last_fullsized":"2024-09-10T18:37:11.802421+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'29","last_scrub_stamp":"2024-09-10T18:36:59.647532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:59.647532+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:24.657907+0000","scrub_duration":0.0067920380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":354,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.809595+0000","last_change":"2024-09-10T18:36:06.595223+0000","last_active":"2024-09-10T18:37:11.809595+0000","last_peered":"2024-09-10T18:37:11.809595+0000","last_clean":"2024-09-10T18:37:11.809595+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:37:11.809595+0000","last_undegraded":"2024-09-10T18:37:11.809595+0000","last_fullsized":"2024-09-10T18:37:11.809595+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.595159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.595159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:34.639380+0000","scrub_duration":0.0014987100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":320,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:12.856219+0000","last_change":"2024-09-10T18:37:12.856158+0000","last_active":"2024-09-10T18:37:12.856219+0000","last_peered":"2024-09-10T18:37:12.856219+0000","last_clean":"2024-09-10T18:37:12.856219+0000","last_became_active":"2024-09-10T18:26:27.642063+0000","last_became_peered":"2024-09-10T18:26:27.642063+0000","last_unstale":"2024-09-10T18:37:12.856219+0000","last_undegraded":"2024-09-10T18:37:12.856219+0000","last_fullsized":"2024-09-10T18:37:12.856219+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:12.856068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:12.856068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:14.246369+0000","scrub_duration":0.020901162000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":355,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.804996+0000","last_change":"2024-09-10T18:35:14.848040+0000","last_active":"2024-09-10T18:37:11.804996+0000","last_peered":"2024-09-10T18:37:11.804996+0000","last_clean":"2024-09-10T18:37:11.804996+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:37:11.804996+0000","last_undegraded":"2024-09-10T18:37:11.804996+0000","last_fullsized":"2024-09-10T18:37:11.804996+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:14.847962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:35:14.847962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053623120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"335'6084","reported_seq":6113,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:15.955233+0000","last_change":"2024-09-10T18:36:07.903723+0000","last_active":"2024-09-10T18:37:15.955233+0000","last_peered":"2024-09-10T18:37:15.955233+0000","last_clean":"2024-09-10T18:37:15.955233+0000","last_became_active":"2024-09-10T18:36:07.903339+0000","last_became_peered":"2024-09-10T18:36:07.903339+0000","last_unstale":"2024-09-10T18:37:15.955233+0000","last_undegraded":"2024-09-10T18:37:15.955233+0000","last_fullsized":"2024-09-10T18:37:15.955233+0000","mapping_epoch":314,"log_start":"320'1100","ondisk_log_start":"320'1100","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4984,"log_dups_size":0,"ondisk_log_size":4984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:25.034412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398721024,"num_objects":6084,"num_object_clones":0,"num_object_copies":18252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6084,"num_write_kb":389376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":265,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.805977+0000","last_change":"2024-09-10T18:36:55.135378+0000","last_active":"2024-09-10T18:37:11.805977+0000","last_peered":"2024-09-10T18:37:11.805977+0000","last_clean":"2024-09-10T18:37:11.805977+0000","last_became_active":"2024-09-10T18:34:00.151017+0000","last_became_peered":"2024-09-10T18:34:00.151017+0000","last_unstale":"2024-09-10T18:37:11.805977+0000","last_undegraded":"2024-09-10T18:37:11.805977+0000","last_fullsized":"2024-09-10T18:37:11.805977+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:55.135191+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:55.135191+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:04.408840+0000","scrub_duration":0.0072107639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"318'42","reported_seq":62,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:14.349043+0000","last_change":"2024-09-10T18:36:12.932521+0000","last_active":"2024-09-10T18:37:14.349043+0000","last_peered":"2024-09-10T18:37:14.349043+0000","last_clean":"2024-09-10T18:37:14.349043+0000","last_became_active":"2024-09-10T18:36:12.922665+0000","last_became_peered":"2024-09-10T18:36:12.922665+0000","last_unstale":"2024-09-10T18:37:14.349043+0000","last_undegraded":"2024-09-10T18:37:14.349043+0000","last_fullsized":"2024-09-10T18:37:14.349043+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'41","last_scrub_stamp":"2024-09-10T18:35:07.535910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:07.535910+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0098705089999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":360,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.800558+0000","last_change":"2024-09-10T18:37:01.274315+0000","last_active":"2024-09-10T18:37:11.800558+0000","last_peered":"2024-09-10T18:37:11.800558+0000","last_clean":"2024-09-10T18:37:11.800558+0000","last_became_active":"2024-09-10T18:26:27.648690+0000","last_became_peered":"2024-09-10T18:26:27.648690+0000","last_unstale":"2024-09-10T18:37:11.800558+0000","last_undegraded":"2024-09-10T18:37:11.800558+0000","last_fullsized":"2024-09-10T18:37:11.800558+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:01.274219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.274219+0000","last_clean_scrub_stamp":"2024-09-10T18:37:01.274219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:17.545348+0000","scrub_duration":0.0059785009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":369,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.805467+0000","last_change":"2024-09-10T18:35:55.505988+0000","last_active":"2024-09-10T18:37:11.805467+0000","last_peered":"2024-09-10T18:37:11.805467+0000","last_clean":"2024-09-10T18:37:11.805467+0000","last_became_active":"2024-09-10T18:26:27.647576+0000","last_became_peered":"2024-09-10T18:26:27.647576+0000","last_unstale":"2024-09-10T18:37:11.805467+0000","last_undegraded":"2024-09-10T18:37:11.805467+0000","last_fullsized":"2024-09-10T18:37:11.805467+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:35:55.505916+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:35:55.505916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001708209,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":351,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.808639+0000","last_change":"2024-09-10T18:36:02.734328+0000","last_active":"2024-09-10T18:37:11.808639+0000","last_peered":"2024-09-10T18:37:11.808639+0000","last_clean":"2024-09-10T18:37:11.808639+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:37:11.808639+0000","last_undegraded":"2024-09-10T18:37:11.808639+0000","last_fullsized":"2024-09-10T18:37:11.808639+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:02.734269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:02.734269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:23.649134+0000","scrub_duration":0.0016180000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":330,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.797416+0000","last_change":"2024-09-10T18:33:56.168567+0000","last_active":"2024-09-10T18:37:11.797416+0000","last_peered":"2024-09-10T18:37:11.797416+0000","last_clean":"2024-09-10T18:37:11.797416+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:37:11.797416+0000","last_undegraded":"2024-09-10T18:37:11.797416+0000","last_fullsized":"2024-09-10T18:37:11.797416+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:33:56.168485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:33:56.168485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002436723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"335'6083","reported_seq":6112,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:15.129197+0000","last_change":"2024-09-10T18:36:07.906590+0000","last_active":"2024-09-10T18:37:15.129197+0000","last_peered":"2024-09-10T18:37:15.129197+0000","last_clean":"2024-09-10T18:37:15.129197+0000","last_became_active":"2024-09-10T18:36:07.906264+0000","last_became_peered":"2024-09-10T18:36:07.906264+0000","last_unstale":"2024-09-10T18:37:15.129197+0000","last_undegraded":"2024-09-10T18:37:15.129197+0000","last_fullsized":"2024-09-10T18:37:15.129197+0000","mapping_epoch":314,"log_start":"320'1451","ondisk_log_start":"320'1451","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398655488,"num_objects":6083,"num_object_clones":0,"num_object_copies":18249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":389312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":401,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806380+0000","last_change":"2024-09-10T18:35:08.748174+0000","last_active":"2024-09-10T18:37:11.806380+0000","last_peered":"2024-09-10T18:37:11.806380+0000","last_clean":"2024-09-10T18:37:11.806380+0000","last_became_active":"2024-09-10T18:34:00.154404+0000","last_became_peered":"2024-09-10T18:34:00.154404+0000","last_unstale":"2024-09-10T18:37:11.806380+0000","last_undegraded":"2024-09-10T18:37:11.806380+0000","last_fullsized":"2024-09-10T18:37:11.806380+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:08.748082+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:08.748082+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038002869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"240'17","reported_seq":395,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.810762+0000","last_change":"2024-09-10T18:36:49.797325+0000","last_active":"2024-09-10T18:37:11.810762+0000","last_peered":"2024-09-10T18:37:11.810762+0000","last_clean":"2024-09-10T18:37:11.810762+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:37:11.810762+0000","last_undegraded":"2024-09-10T18:37:11.810762+0000","last_fullsized":"2024-09-10T18:37:11.810762+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'17","last_scrub_stamp":"2024-09-10T18:36:49.797243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:49.797243+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:53.680622+0000","scrub_duration":0.0030959260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":371,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806355+0000","last_change":"2024-09-10T18:35:57.626741+0000","last_active":"2024-09-10T18:37:11.806355+0000","last_peered":"2024-09-10T18:37:11.806355+0000","last_clean":"2024-09-10T18:37:11.806355+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:37:11.806355+0000","last_undegraded":"2024-09-10T18:37:11.806355+0000","last_fullsized":"2024-09-10T18:37:11.806355+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:57.626681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:15.141982+0000","last_clean_scrub_stamp":"2024-09-10T18:35:57.626681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001556663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":392,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806855+0000","last_change":"2024-09-10T18:37:04.920700+0000","last_active":"2024-09-10T18:37:11.806855+0000","last_peered":"2024-09-10T18:37:11.806855+0000","last_clean":"2024-09-10T18:37:11.806855+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:37:11.806855+0000","last_undegraded":"2024-09-10T18:37:11.806855+0000","last_fullsized":"2024-09-10T18:37:11.806855+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:37:04.920615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:04.920615+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:20.515535+0000","scrub_duration":0.01495838,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":369,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.807570+0000","last_change":"2024-09-10T18:36:48.839724+0000","last_active":"2024-09-10T18:37:11.807570+0000","last_peered":"2024-09-10T18:37:11.807570+0000","last_clean":"2024-09-10T18:37:11.807570+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:37:11.807570+0000","last_undegraded":"2024-09-10T18:37:11.807570+0000","last_fullsized":"2024-09-10T18:37:11.807570+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:48.839647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:36:48.839647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:55.570968+0000","scrub_duration":0.0072244579999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":357,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.807862+0000","last_change":"2024-09-10T18:37:02.114698+0000","last_active":"2024-09-10T18:37:11.807862+0000","last_peered":"2024-09-10T18:37:11.807862+0000","last_clean":"2024-09-10T18:37:11.807862+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:37:11.807862+0000","last_undegraded":"2024-09-10T18:37:11.807862+0000","last_fullsized":"2024-09-10T18:37:11.807862+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:02.114602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:02.114602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:21.588925+0000","scrub_duration":0.0019755279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"335'6133","reported_seq":6162,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:16.219745+0000","last_change":"2024-09-10T18:36:07.902359+0000","last_active":"2024-09-10T18:37:16.219745+0000","last_peered":"2024-09-10T18:37:16.219745+0000","last_clean":"2024-09-10T18:37:16.219745+0000","last_became_active":"2024-09-10T18:36:07.901516+0000","last_became_peered":"2024-09-10T18:36:07.901516+0000","last_unstale":"2024-09-10T18:37:16.219745+0000","last_undegraded":"2024-09-10T18:37:16.219745+0000","last_fullsized":"2024-09-10T18:37:16.219745+0000","mapping_epoch":314,"log_start":"320'1600","ondisk_log_start":"320'1600","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4533,"log_dups_size":0,"ondisk_log_size":4533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:17.319621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401932288,"num_objects":6133,"num_object_clones":0,"num_object_copies":18399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6133,"num_write_kb":392512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":281,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.808970+0000","last_change":"2024-09-10T18:36:59.835814+0000","last_active":"2024-09-10T18:37:11.808970+0000","last_peered":"2024-09-10T18:37:11.808970+0000","last_clean":"2024-09-10T18:37:11.808970+0000","last_became_active":"2024-09-10T18:34:00.165384+0000","last_became_peered":"2024-09-10T18:34:00.165384+0000","last_unstale":"2024-09-10T18:37:11.808970+0000","last_undegraded":"2024-09-10T18:37:11.808970+0000","last_fullsized":"2024-09-10T18:37:11.808970+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:59.835713+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:59.835713+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:16.626287+0000","scrub_duration":0.029401394000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":389,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.803764+0000","last_change":"2024-09-10T18:36:15.636499+0000","last_active":"2024-09-10T18:37:11.803764+0000","last_peered":"2024-09-10T18:37:11.803764+0000","last_clean":"2024-09-10T18:37:11.803764+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:37:11.803764+0000","last_undegraded":"2024-09-10T18:37:11.803764+0000","last_fullsized":"2024-09-10T18:37:11.803764+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-09-10T18:36:15.636442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:15.636442+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:35.680984+0000","scrub_duration":0.0017190630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":365,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.803117+0000","last_change":"2024-09-10T18:36:55.603164+0000","last_active":"2024-09-10T18:37:11.803117+0000","last_peered":"2024-09-10T18:37:11.803117+0000","last_clean":"2024-09-10T18:37:11.803117+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:37:11.803117+0000","last_undegraded":"2024-09-10T18:37:11.803117+0000","last_fullsized":"2024-09-10T18:37:11.803117+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:55.603085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:36:55.603085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:23.221490+0000","scrub_duration":0.012495984999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":1307,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:14.373955+0000","last_change":"2024-09-10T18:37:01.094318+0000","last_active":"2024-09-10T18:37:14.373955+0000","last_peered":"2024-09-10T18:37:14.373955+0000","last_clean":"2024-09-10T18:37:14.373955+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:37:14.373955+0000","last_undegraded":"2024-09-10T18:37:14.373955+0000","last_fullsized":"2024-09-10T18:37:14.373955+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:37:01.094063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:01.094063+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:23.788462+0000","scrub_duration":0.0099775509999999994,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":936,"num_read_kb":931,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":353,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806176+0000","last_change":"2024-09-10T18:37:01.170467+0000","last_active":"2024-09-10T18:37:11.806176+0000","last_peered":"2024-09-10T18:37:11.806176+0000","last_clean":"2024-09-10T18:37:11.806176+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:37:11.806176+0000","last_undegraded":"2024-09-10T18:37:11.806176+0000","last_fullsized":"2024-09-10T18:37:11.806176+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:01.170272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.170272+0000","last_clean_scrub_stamp":"2024-09-10T18:37:01.170272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:05.863430+0000","scrub_duration":0.0065986719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":348,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.808895+0000","last_change":"2024-09-10T18:36:17.593856+0000","last_active":"2024-09-10T18:37:11.808895+0000","last_peered":"2024-09-10T18:37:11.808895+0000","last_clean":"2024-09-10T18:37:11.808895+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:37:11.808895+0000","last_undegraded":"2024-09-10T18:37:11.808895+0000","last_fullsized":"2024-09-10T18:37:11.808895+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:17.593802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:17.593802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:19.360284+0000","scrub_duration":0.0032348469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"335'6080","reported_seq":6109,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:15.944448+0000","last_change":"2024-09-10T18:36:07.903898+0000","last_active":"2024-09-10T18:37:15.944448+0000","last_peered":"2024-09-10T18:37:15.944448+0000","last_clean":"2024-09-10T18:37:15.944448+0000","last_became_active":"2024-09-10T18:36:07.903576+0000","last_became_peered":"2024-09-10T18:36:07.903576+0000","last_unstale":"2024-09-10T18:37:15.944448+0000","last_undegraded":"2024-09-10T18:37:15.944448+0000","last_fullsized":"2024-09-10T18:37:15.944448+0000","mapping_epoch":314,"log_start":"320'1100","ondisk_log_start":"320'1100","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4980,"log_dups_size":0,"ondisk_log_size":4980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:26.917343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398458880,"num_objects":6080,"num_object_clones":0,"num_object_copies":18240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6080,"num_write_kb":389120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"240'26","reported_seq":347,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.797128+0000","last_change":"2024-09-10T18:35:55.879357+0000","last_active":"2024-09-10T18:37:11.797128+0000","last_peered":"2024-09-10T18:37:11.797128+0000","last_clean":"2024-09-10T18:37:11.797128+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:37:11.797128+0000","last_undegraded":"2024-09-10T18:37:11.797128+0000","last_fullsized":"2024-09-10T18:37:11.797128+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'26","last_scrub_stamp":"2024-09-10T18:35:55.879297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:55.879297+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:16.597074+0000","scrub_duration":0.001846929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":353,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.798356+0000","last_change":"2024-09-10T18:34:19.296515+0000","last_active":"2024-09-10T18:37:11.798356+0000","last_peered":"2024-09-10T18:37:11.798356+0000","last_clean":"2024-09-10T18:37:11.798356+0000","last_became_active":"2024-09-10T18:26:27.639129+0000","last_became_peered":"2024-09-10T18:26:27.639129+0000","last_unstale":"2024-09-10T18:37:11.798356+0000","last_undegraded":"2024-09-10T18:37:11.798356+0000","last_fullsized":"2024-09-10T18:37:11.798356+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:19.296286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.296286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049170170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":423,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.982331+0000","last_change":"2024-09-10T18:37:11.982261+0000","last_active":"2024-09-10T18:37:11.982331+0000","last_peered":"2024-09-10T18:37:11.982331+0000","last_clean":"2024-09-10T18:37:11.982331+0000","last_became_active":"2024-09-10T18:34:00.159975+0000","last_became_peered":"2024-09-10T18:34:00.159975+0000","last_unstale":"2024-09-10T18:37:11.982331+0000","last_undegraded":"2024-09-10T18:37:11.982331+0000","last_fullsized":"2024-09-10T18:37:11.982331+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:11.982173+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:11.982173+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:40.227262+0000","scrub_duration":0.001824406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":359,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.808130+0000","last_change":"2024-09-10T18:36:58.015546+0000","last_active":"2024-09-10T18:37:11.808130+0000","last_peered":"2024-09-10T18:37:11.808130+0000","last_clean":"2024-09-10T18:37:11.808130+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:37:11.808130+0000","last_undegraded":"2024-09-10T18:37:11.808130+0000","last_fullsized":"2024-09-10T18:37:11.808130+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:58.015470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:58.015470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:03.398913+0000","scrub_duration":0.0043498190000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":351,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806050+0000","last_change":"2024-09-10T18:36:59.120950+0000","last_active":"2024-09-10T18:37:11.806050+0000","last_peered":"2024-09-10T18:37:11.806050+0000","last_clean":"2024-09-10T18:37:11.806050+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:37:11.806050+0000","last_undegraded":"2024-09-10T18:37:11.806050+0000","last_fullsized":"2024-09-10T18:37:11.806050+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:59.120687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:36:59.120687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:13.759657+0000","scrub_duration":0.0049055239999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"335'6212","reported_seq":6241,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:15.939978+0000","last_change":"2024-09-10T18:36:07.903096+0000","last_active":"2024-09-10T18:37:15.939978+0000","last_peered":"2024-09-10T18:37:15.939978+0000","last_clean":"2024-09-10T18:37:15.939978+0000","last_became_active":"2024-09-10T18:36:07.902779+0000","last_became_peered":"2024-09-10T18:36:07.902779+0000","last_unstale":"2024-09-10T18:37:15.939978+0000","last_undegraded":"2024-09-10T18:37:15.939978+0000","last_fullsized":"2024-09-10T18:37:15.939978+0000","mapping_epoch":314,"log_start":"320'1200","ondisk_log_start":"320'1200","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":5012,"log_dups_size":0,"ondisk_log_size":5012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:30.765220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407109632,"num_objects":6212,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6212,"num_write_kb":397568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"76'81","reported_seq":290,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.809121+0000","last_change":"2024-09-10T18:35:15.894737+0000","last_active":"2024-09-10T18:37:11.809121+0000","last_peered":"2024-09-10T18:37:11.809121+0000","last_clean":"2024-09-10T18:37:11.809121+0000","last_became_active":"2024-09-10T18:34:32.927460+0000","last_became_peered":"2024-09-10T18:34:32.927460+0000","last_unstale":"2024-09-10T18:37:11.809121+0000","last_undegraded":"2024-09-10T18:37:11.809121+0000","last_fullsized":"2024-09-10T18:37:11.809121+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:15.894656+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:15.894656+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013574714,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"283'24","reported_seq":397,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.801672+0000","last_change":"2024-09-10T18:36:58.225291+0000","last_active":"2024-09-10T18:37:11.801672+0000","last_peered":"2024-09-10T18:37:11.801672+0000","last_clean":"2024-09-10T18:37:11.801672+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:37:11.801672+0000","last_undegraded":"2024-09-10T18:37:11.801672+0000","last_fullsized":"2024-09-10T18:37:11.801672+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'24","last_scrub_stamp":"2024-09-10T18:36:58.225206+0000","last_deep_scrub":"283'24","last_deep_scrub_stamp":"2024-09-10T18:34:19.404713+0000","last_clean_scrub_stamp":"2024-09-10T18:36:58.225206+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:16.467424+0000","scrub_duration":0.0055050170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":353,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.800227+0000","last_change":"2024-09-10T18:37:00.246542+0000","last_active":"2024-09-10T18:37:11.800227+0000","last_peered":"2024-09-10T18:37:11.800227+0000","last_clean":"2024-09-10T18:37:11.800227+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:37:11.800227+0000","last_undegraded":"2024-09-10T18:37:11.800227+0000","last_fullsized":"2024-09-10T18:37:11.800227+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:00.246398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:00.246398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:00.862550+0000","scrub_duration":0.003194966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":357,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806320+0000","last_change":"2024-09-10T18:35:08.562624+0000","last_active":"2024-09-10T18:37:11.806320+0000","last_peered":"2024-09-10T18:37:11.806320+0000","last_clean":"2024-09-10T18:37:11.806320+0000","last_became_active":"2024-09-10T18:26:27.645885+0000","last_became_peered":"2024-09-10T18:26:27.645885+0000","last_unstale":"2024-09-10T18:37:11.806320+0000","last_undegraded":"2024-09-10T18:37:11.806320+0000","last_fullsized":"2024-09-10T18:37:11.806320+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:08.562551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:35:08.562551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076930669999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":347,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.800201+0000","last_change":"2024-09-10T18:34:01.181010+0000","last_active":"2024-09-10T18:37:11.800201+0000","last_peered":"2024-09-10T18:37:11.800201+0000","last_clean":"2024-09-10T18:37:11.800201+0000","last_became_active":"2024-09-10T18:26:27.632071+0000","last_became_peered":"2024-09-10T18:26:27.632071+0000","last_unstale":"2024-09-10T18:37:11.800201+0000","last_undegraded":"2024-09-10T18:37:11.800201+0000","last_fullsized":"2024-09-10T18:37:11.800201+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:01.180920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:01.180920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082996130000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"335'5999","reported_seq":6026,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:15.243039+0000","last_change":"2024-09-10T18:36:07.903712+0000","last_active":"2024-09-10T18:37:15.243039+0000","last_peered":"2024-09-10T18:37:15.243039+0000","last_clean":"2024-09-10T18:37:15.243039+0000","last_became_active":"2024-09-10T18:36:07.903429+0000","last_became_peered":"2024-09-10T18:36:07.903429+0000","last_unstale":"2024-09-10T18:37:15.243039+0000","last_undegraded":"2024-09-10T18:37:15.243039+0000","last_fullsized":"2024-09-10T18:37:15.243039+0000","mapping_epoch":314,"log_start":"320'200","ondisk_log_start":"320'200","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":5799,"log_dups_size":0,"ondisk_log_size":5799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:15.985317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393150464,"num_objects":5999,"num_object_clones":0,"num_object_copies":17997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5999,"num_write_kb":383936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":272,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.804572+0000","last_change":"2024-09-10T18:36:01.514457+0000","last_active":"2024-09-10T18:37:11.804572+0000","last_peered":"2024-09-10T18:37:11.804572+0000","last_clean":"2024-09-10T18:37:11.804572+0000","last_became_active":"2024-09-10T18:34:32.823976+0000","last_became_peered":"2024-09-10T18:34:32.823976+0000","last_unstale":"2024-09-10T18:37:11.804572+0000","last_undegraded":"2024-09-10T18:37:11.804572+0000","last_fullsized":"2024-09-10T18:37:11.804572+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:01.514369+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:01.514369+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001652136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":428,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.833663+0000","last_change":"2024-09-10T18:37:11.832716+0000","last_active":"2024-09-10T18:37:11.833663+0000","last_peered":"2024-09-10T18:37:11.833663+0000","last_clean":"2024-09-10T18:37:11.833663+0000","last_became_active":"2024-09-10T18:37:11.817595+0000","last_became_peered":"2024-09-10T18:37:11.817595+0000","last_unstale":"2024-09-10T18:37:11.833663+0000","last_undegraded":"2024-09-10T18:37:11.833663+0000","last_fullsized":"2024-09-10T18:37:11.833663+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-09-10T18:36:02.364698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:02.364698+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025656849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":318,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.797664+0000","last_change":"2024-09-10T18:36:00.897438+0000","last_active":"2024-09-10T18:37:11.797664+0000","last_peered":"2024-09-10T18:37:11.797664+0000","last_clean":"2024-09-10T18:37:11.797664+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:37:11.797664+0000","last_undegraded":"2024-09-10T18:37:11.797664+0000","last_fullsized":"2024-09-10T18:37:11.797664+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:36:00.897355+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:36:00.897355+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:26.619358+0000","scrub_duration":0.0020072419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":360,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.808955+0000","last_change":"2024-09-10T18:36:57.784141+0000","last_active":"2024-09-10T18:37:11.808955+0000","last_peered":"2024-09-10T18:37:11.808955+0000","last_clean":"2024-09-10T18:37:11.808955+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:37:11.808955+0000","last_undegraded":"2024-09-10T18:37:11.808955+0000","last_fullsized":"2024-09-10T18:37:11.808955+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:57.784049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:57.784049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:15.415282+0000","scrub_duration":0.0022794579999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":377,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.807049+0000","last_change":"2024-09-10T18:36:04.465752+0000","last_active":"2024-09-10T18:37:11.807049+0000","last_peered":"2024-09-10T18:37:11.807049+0000","last_clean":"2024-09-10T18:37:11.807049+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:37:11.807049+0000","last_undegraded":"2024-09-10T18:37:11.807049+0000","last_fullsized":"2024-09-10T18:37:11.807049+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:04.465694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:04.465694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014277929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"335'6249","reported_seq":6278,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:15.537589+0000","last_change":"2024-09-10T18:36:07.904133+0000","last_active":"2024-09-10T18:37:15.537589+0000","last_peered":"2024-09-10T18:37:15.537589+0000","last_clean":"2024-09-10T18:37:15.537589+0000","last_became_active":"2024-09-10T18:36:07.903825+0000","last_became_peered":"2024-09-10T18:36:07.903825+0000","last_unstale":"2024-09-10T18:37:15.537589+0000","last_undegraded":"2024-09-10T18:37:15.537589+0000","last_fullsized":"2024-09-10T18:37:15.537589+0000","mapping_epoch":314,"log_start":"320'1700","ondisk_log_start":"320'1700","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4549,"log_dups_size":0,"ondisk_log_size":4549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409534464,"num_objects":6249,"num_object_clones":0,"num_object_copies":18747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6249,"num_write_kb":399936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":411,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.808571+0000","last_change":"2024-09-10T18:36:13.951388+0000","last_active":"2024-09-10T18:37:11.808571+0000","last_peered":"2024-09-10T18:37:11.808571+0000","last_clean":"2024-09-10T18:37:11.808571+0000","last_became_active":"2024-09-10T18:34:32.827174+0000","last_became_peered":"2024-09-10T18:34:32.827174+0000","last_unstale":"2024-09-10T18:37:11.808571+0000","last_undegraded":"2024-09-10T18:37:11.808571+0000","last_fullsized":"2024-09-10T18:37:11.808571+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:13.951321+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:13.951321+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:19.549886+0000","scrub_duration":0.0015868600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"240'25","reported_seq":121,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:14.054730+0000","last_change":"2024-09-10T18:37:14.054480+0000","last_active":"2024-09-10T18:37:14.054730+0000","last_peered":"2024-09-10T18:37:14.054730+0000","last_clean":"2024-09-10T18:37:14.054730+0000","last_became_active":"2024-09-10T18:32:51.118251+0000","last_became_peered":"2024-09-10T18:32:51.118251+0000","last_unstale":"2024-09-10T18:37:14.054730+0000","last_undegraded":"2024-09-10T18:37:14.054730+0000","last_fullsized":"2024-09-10T18:37:14.054730+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:37:14.054170+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:37:14.054170+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:41.275620+0000","scrub_duration":0.019668199000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":346,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.796528+0000","last_change":"2024-09-10T18:36:03.769108+0000","last_active":"2024-09-10T18:37:11.796528+0000","last_peered":"2024-09-10T18:37:11.796528+0000","last_clean":"2024-09-10T18:37:11.796528+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:37:11.796528+0000","last_undegraded":"2024-09-10T18:37:11.796528+0000","last_fullsized":"2024-09-10T18:37:11.796528+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:03.769053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:13.324556+0000","last_clean_scrub_stamp":"2024-09-10T18:36:03.769053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:29.323478+0000","scrub_duration":0.001666634,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"189'9","reported_seq":587,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:14.363034+0000","last_change":"2024-09-10T18:37:10.969567+0000","last_active":"2024-09-10T18:37:14.363034+0000","last_peered":"2024-09-10T18:37:14.363034+0000","last_clean":"2024-09-10T18:37:14.363034+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:37:14.363034+0000","last_undegraded":"2024-09-10T18:37:14.363034+0000","last_fullsized":"2024-09-10T18:37:14.363034+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'9","last_scrub_stamp":"2024-09-10T18:37:10.969454+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:37:10.969454+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:26.277589+0000","scrub_duration":0.0024896530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":340,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.801427+0000","last_change":"2024-09-10T18:35:07.288756+0000","last_active":"2024-09-10T18:37:11.801427+0000","last_peered":"2024-09-10T18:37:11.801427+0000","last_clean":"2024-09-10T18:37:11.801427+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:37:11.801427+0000","last_undegraded":"2024-09-10T18:37:11.801427+0000","last_fullsized":"2024-09-10T18:37:11.801427+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:07.288672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_clean_scrub_stamp":"2024-09-10T18:35:07.288672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041335299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"335'6118","reported_seq":6147,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:15.224393+0000","last_change":"2024-09-10T18:36:07.904381+0000","last_active":"2024-09-10T18:37:15.224393+0000","last_peered":"2024-09-10T18:37:15.224393+0000","last_clean":"2024-09-10T18:37:15.224393+0000","last_became_active":"2024-09-10T18:36:07.903638+0000","last_became_peered":"2024-09-10T18:36:07.903638+0000","last_unstale":"2024-09-10T18:37:15.224393+0000","last_undegraded":"2024-09-10T18:37:15.224393+0000","last_fullsized":"2024-09-10T18:37:15.224393+0000","mapping_epoch":314,"log_start":"320'300","ondisk_log_start":"320'300","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":5818,"log_dups_size":0,"ondisk_log_size":5818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400949248,"num_objects":6118,"num_object_clones":0,"num_object_copies":18354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6118,"num_write_kb":391552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"76'81","reported_seq":83,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.801356+0000","last_change":"2024-09-10T18:35:58.314505+0000","last_active":"2024-09-10T18:37:11.801356+0000","last_peered":"2024-09-10T18:37:11.801356+0000","last_clean":"2024-09-10T18:37:11.801356+0000","last_became_active":"2024-09-10T18:34:32.818251+0000","last_became_peered":"2024-09-10T18:34:32.818251+0000","last_unstale":"2024-09-10T18:37:11.801356+0000","last_undegraded":"2024-09-10T18:37:11.801356+0000","last_fullsized":"2024-09-10T18:37:11.801356+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:58.314438+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:58.314438+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015006209999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"302'19","reported_seq":382,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.802340+0000","last_change":"2024-09-10T18:37:08.823694+0000","last_active":"2024-09-10T18:37:11.802340+0000","last_peered":"2024-09-10T18:37:11.802340+0000","last_clean":"2024-09-10T18:37:11.802340+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:37:11.802340+0000","last_undegraded":"2024-09-10T18:37:11.802340+0000","last_fullsized":"2024-09-10T18:37:11.802340+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'19","last_scrub_stamp":"2024-09-10T18:37:08.823440+0000","last_deep_scrub":"302'19","last_deep_scrub_stamp":"2024-09-10T18:35:18.836131+0000","last_clean_scrub_stamp":"2024-09-10T18:37:08.823440+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:19.183000+0000","scrub_duration":0.012806522000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"265'2","reported_seq":371,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.805023+0000","last_change":"2024-09-10T18:35:14.515615+0000","last_active":"2024-09-10T18:37:11.805023+0000","last_peered":"2024-09-10T18:37:11.805023+0000","last_clean":"2024-09-10T18:37:11.805023+0000","last_became_active":"2024-09-10T18:26:27.644997+0000","last_became_peered":"2024-09-10T18:26:27.644997+0000","last_unstale":"2024-09-10T18:37:11.805023+0000","last_undegraded":"2024-09-10T18:37:11.805023+0000","last_fullsized":"2024-09-10T18:37:11.805023+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'2","last_scrub_stamp":"2024-09-10T18:35:14.515533+0000","last_deep_scrub":"265'2","last_deep_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_clean_scrub_stamp":"2024-09-10T18:35:14.515533+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046328369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":375,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806478+0000","last_change":"2024-09-10T18:36:56.798207+0000","last_active":"2024-09-10T18:37:11.806478+0000","last_peered":"2024-09-10T18:37:11.806478+0000","last_clean":"2024-09-10T18:37:11.806478+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:37:11.806478+0000","last_undegraded":"2024-09-10T18:37:11.806478+0000","last_fullsized":"2024-09-10T18:37:11.806478+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:56.798117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:56.798117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:14.975136+0000","scrub_duration":0.0021655960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":349,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.803055+0000","last_change":"2024-09-10T18:36:14.665247+0000","last_active":"2024-09-10T18:37:11.803055+0000","last_peered":"2024-09-10T18:37:11.803055+0000","last_clean":"2024-09-10T18:37:11.803055+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:37:11.803055+0000","last_undegraded":"2024-09-10T18:37:11.803055+0000","last_fullsized":"2024-09-10T18:37:11.803055+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:14.665183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:14.665183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:23.790484+0000","scrub_duration":0.0014290399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"335'6114","reported_seq":6143,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:16.224026+0000","last_change":"2024-09-10T18:36:07.902622+0000","last_active":"2024-09-10T18:37:16.224026+0000","last_peered":"2024-09-10T18:37:16.224026+0000","last_clean":"2024-09-10T18:37:16.224026+0000","last_became_active":"2024-09-10T18:36:07.902336+0000","last_became_peered":"2024-09-10T18:36:07.902336+0000","last_unstale":"2024-09-10T18:37:16.224026+0000","last_undegraded":"2024-09-10T18:37:16.224026+0000","last_fullsized":"2024-09-10T18:37:16.224026+0000","mapping_epoch":314,"log_start":"320'1600","ondisk_log_start":"320'1600","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400687104,"num_objects":6114,"num_object_clones":0,"num_object_copies":18342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6114,"num_write_kb":391296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'81","reported_seq":339,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.797983+0000","last_change":"2024-09-10T18:36:04.870940+0000","last_active":"2024-09-10T18:37:11.797983+0000","last_peered":"2024-09-10T18:37:11.797983+0000","last_clean":"2024-09-10T18:37:11.797983+0000","last_became_active":"2024-09-10T18:34:02.803861+0000","last_became_peered":"2024-09-10T18:34:02.803861+0000","last_unstale":"2024-09-10T18:37:11.797983+0000","last_undegraded":"2024-09-10T18:37:11.797983+0000","last_fullsized":"2024-09-10T18:37:11.797983+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:04.870861+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:34:15.938079+0000","last_clean_scrub_stamp":"2024-09-10T18:36:04.870861+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:32.989359+0000","scrub_duration":0.001253791,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"240'44","reported_seq":435,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.797486+0000","last_change":"2024-09-10T18:35:54.841099+0000","last_active":"2024-09-10T18:37:11.797486+0000","last_peered":"2024-09-10T18:37:11.797486+0000","last_clean":"2024-09-10T18:37:11.797486+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:37:11.797486+0000","last_undegraded":"2024-09-10T18:37:11.797486+0000","last_fullsized":"2024-09-10T18:37:11.797486+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'44","last_scrub_stamp":"2024-09-10T18:35:54.841032+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:35:54.841032+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:22.032326+0000","scrub_duration":0.0027477270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":370,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.808145+0000","last_change":"2024-09-10T18:36:56.003032+0000","last_active":"2024-09-10T18:37:11.808145+0000","last_peered":"2024-09-10T18:37:11.808145+0000","last_clean":"2024-09-10T18:37:11.808145+0000","last_became_active":"2024-09-10T18:26:27.638408+0000","last_became_peered":"2024-09-10T18:26:27.638408+0000","last_unstale":"2024-09-10T18:37:11.808145+0000","last_undegraded":"2024-09-10T18:37:11.808145+0000","last_fullsized":"2024-09-10T18:37:11.808145+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:56.002946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:36:56.002946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:17.122534+0000","scrub_duration":0.0058026170000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":379,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806794+0000","last_change":"2024-09-10T18:36:00.551970+0000","last_active":"2024-09-10T18:37:11.806794+0000","last_peered":"2024-09-10T18:37:11.806794+0000","last_clean":"2024-09-10T18:37:11.806794+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:37:11.806794+0000","last_undegraded":"2024-09-10T18:37:11.806794+0000","last_fullsized":"2024-09-10T18:37:11.806794+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:00.551910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:34:21.447291+0000","last_clean_scrub_stamp":"2024-09-10T18:36:00.551910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017792039999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":360,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.807083+0000","last_change":"2024-09-10T18:37:04.058735+0000","last_active":"2024-09-10T18:37:11.807083+0000","last_peered":"2024-09-10T18:37:11.807083+0000","last_clean":"2024-09-10T18:37:11.807083+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:37:11.807083+0000","last_undegraded":"2024-09-10T18:37:11.807083+0000","last_fullsized":"2024-09-10T18:37:11.807083+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:04.058646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:04.058646+0000","last_clean_scrub_stamp":"2024-09-10T18:37:04.058646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:21.930918+0000","scrub_duration":0.0034509390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"335'6163","reported_seq":6192,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:15.945332+0000","last_change":"2024-09-10T18:36:07.902995+0000","last_active":"2024-09-10T18:37:15.945332+0000","last_peered":"2024-09-10T18:37:15.945332+0000","last_clean":"2024-09-10T18:37:15.945332+0000","last_became_active":"2024-09-10T18:36:07.902661+0000","last_became_peered":"2024-09-10T18:36:07.902661+0000","last_unstale":"2024-09-10T18:37:15.945332+0000","last_undegraded":"2024-09-10T18:37:15.945332+0000","last_fullsized":"2024-09-10T18:37:15.945332+0000","mapping_epoch":314,"log_start":"320'1200","ondisk_log_start":"320'1200","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4963,"log_dups_size":0,"ondisk_log_size":4963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:36.577552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403898368,"num_objects":6163,"num_object_clones":0,"num_object_copies":18489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6163,"num_write_kb":394432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"76'81","reported_seq":287,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.800435+0000","last_change":"2024-09-10T18:37:04.959623+0000","last_active":"2024-09-10T18:37:11.800435+0000","last_peered":"2024-09-10T18:37:11.800435+0000","last_clean":"2024-09-10T18:37:11.800435+0000","last_became_active":"2024-09-10T18:34:32.956761+0000","last_became_peered":"2024-09-10T18:34:32.956761+0000","last_unstale":"2024-09-10T18:37:11.800435+0000","last_undegraded":"2024-09-10T18:37:11.800435+0000","last_fullsized":"2024-09-10T18:37:11.800435+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:04.959510+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:04.959510+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:10.242143+0000","scrub_duration":0.0070392069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"301'23","reported_seq":391,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.799972+0000","last_change":"2024-09-10T18:35:15.283425+0000","last_active":"2024-09-10T18:37:11.799972+0000","last_peered":"2024-09-10T18:37:11.799972+0000","last_clean":"2024-09-10T18:37:11.799972+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:37:11.799972+0000","last_undegraded":"2024-09-10T18:37:11.799972+0000","last_fullsized":"2024-09-10T18:37:11.799972+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'23","last_scrub_stamp":"2024-09-10T18:35:15.283373+0000","last_deep_scrub":"301'23","last_deep_scrub_stamp":"2024-09-10T18:35:15.283373+0000","last_clean_scrub_stamp":"2024-09-10T18:35:15.283373+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018617538999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":363,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.807367+0000","last_change":"2024-09-10T18:36:50.894694+0000","last_active":"2024-09-10T18:37:11.807367+0000","last_peered":"2024-09-10T18:37:11.807367+0000","last_clean":"2024-09-10T18:37:11.807367+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:37:11.807367+0000","last_undegraded":"2024-09-10T18:37:11.807367+0000","last_fullsized":"2024-09-10T18:37:11.807367+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:50.894615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:36:50.894615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:04.333933+0000","scrub_duration":0.0038639159999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":389,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.807831+0000","last_change":"2024-09-10T18:37:06.917685+0000","last_active":"2024-09-10T18:37:11.807831+0000","last_peered":"2024-09-10T18:37:11.807831+0000","last_clean":"2024-09-10T18:37:11.807831+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:37:11.807831+0000","last_undegraded":"2024-09-10T18:37:11.807831+0000","last_fullsized":"2024-09-10T18:37:11.807831+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:06.917604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:06.917604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:12.452402+0000","scrub_duration":0.013746744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":340,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.796087+0000","last_change":"2024-09-10T18:36:05.766562+0000","last_active":"2024-09-10T18:37:11.796087+0000","last_peered":"2024-09-10T18:37:11.796087+0000","last_clean":"2024-09-10T18:37:11.796087+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:37:11.796087+0000","last_undegraded":"2024-09-10T18:37:11.796087+0000","last_fullsized":"2024-09-10T18:37:11.796087+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:05.766507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.803236+0000","last_clean_scrub_stamp":"2024-09-10T18:36:05.766507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:30.530626+0000","scrub_duration":0.0023023990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"335'6064","reported_seq":6093,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:15.090356+0000","last_change":"2024-09-10T18:36:07.904127+0000","last_active":"2024-09-10T18:37:15.090356+0000","last_peered":"2024-09-10T18:37:15.090356+0000","last_clean":"2024-09-10T18:37:15.090356+0000","last_became_active":"2024-09-10T18:36:07.903791+0000","last_became_peered":"2024-09-10T18:36:07.903791+0000","last_unstale":"2024-09-10T18:37:15.090356+0000","last_undegraded":"2024-09-10T18:37:15.090356+0000","last_fullsized":"2024-09-10T18:37:15.090356+0000","mapping_epoch":314,"log_start":"320'1439","ondisk_log_start":"320'1439","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4625,"log_dups_size":0,"ondisk_log_size":4625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397410304,"num_objects":6064,"num_object_clones":0,"num_object_copies":18192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6064,"num_write_kb":388096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"76'81","reported_seq":83,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806693+0000","last_change":"2024-09-10T18:36:11.950457+0000","last_active":"2024-09-10T18:37:11.806693+0000","last_peered":"2024-09-10T18:37:11.806693+0000","last_clean":"2024-09-10T18:37:11.806693+0000","last_became_active":"2024-09-10T18:34:32.822196+0000","last_became_peered":"2024-09-10T18:34:32.822196+0000","last_unstale":"2024-09-10T18:37:11.806693+0000","last_undegraded":"2024-09-10T18:37:11.806693+0000","last_fullsized":"2024-09-10T18:37:11.806693+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:11.950311+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:11.950311+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020040769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":402,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.804734+0000","last_change":"2024-09-10T18:36:54.095746+0000","last_active":"2024-09-10T18:37:11.804734+0000","last_peered":"2024-09-10T18:37:11.804734+0000","last_clean":"2024-09-10T18:37:11.804734+0000","last_became_active":"2024-09-10T18:26:27.645932+0000","last_became_peered":"2024-09-10T18:26:27.645932+0000","last_unstale":"2024-09-10T18:37:11.804734+0000","last_undegraded":"2024-09-10T18:37:11.804734+0000","last_fullsized":"2024-09-10T18:37:11.804734+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:36:54.095659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:54.095659+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:11.075189+0000","scrub_duration":0.012246685,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":363,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.807712+0000","last_change":"2024-09-10T18:35:58.895879+0000","last_active":"2024-09-10T18:37:11.807712+0000","last_peered":"2024-09-10T18:37:11.807712+0000","last_clean":"2024-09-10T18:37:11.807712+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:37:11.807712+0000","last_undegraded":"2024-09-10T18:37:11.807712+0000","last_fullsized":"2024-09-10T18:37:11.807712+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:35:58.895792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:35:58.895792+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:19.782961+0000","scrub_duration":0.0021727780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":556,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:14.367224+0000","last_change":"2024-09-10T18:36:02.894810+0000","last_active":"2024-09-10T18:37:14.367224+0000","last_peered":"2024-09-10T18:37:14.367224+0000","last_clean":"2024-09-10T18:37:14.367224+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:37:14.367224+0000","last_undegraded":"2024-09-10T18:37:14.367224+0000","last_fullsized":"2024-09-10T18:37:14.367224+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:36:02.894094+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:36:02.894094+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:16.074759+0000","scrub_duration":0.0028441780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":345,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.805037+0000","last_change":"2024-09-10T18:35:53.486754+0000","last_active":"2024-09-10T18:37:11.805037+0000","last_peered":"2024-09-10T18:37:11.805037+0000","last_clean":"2024-09-10T18:37:11.805037+0000","last_became_active":"2024-09-10T18:26:27.647511+0000","last_became_peered":"2024-09-10T18:26:27.647511+0000","last_unstale":"2024-09-10T18:37:11.805037+0000","last_undegraded":"2024-09-10T18:37:11.805037+0000","last_fullsized":"2024-09-10T18:37:11.805037+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:53.486678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:53.486678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:22.033025+0000","scrub_duration":0.0035730969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'81","reported_seq":83,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:14.252327+0000","last_change":"2024-09-10T18:37:14.252272+0000","last_active":"2024-09-10T18:37:14.252327+0000","last_peered":"2024-09-10T18:37:14.252327+0000","last_clean":"2024-09-10T18:37:14.252327+0000","last_became_active":"2024-09-10T18:34:02.810126+0000","last_became_peered":"2024-09-10T18:34:02.810126+0000","last_unstale":"2024-09-10T18:37:14.252327+0000","last_undegraded":"2024-09-10T18:37:14.252327+0000","last_fullsized":"2024-09-10T18:37:14.252327+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:14.252067+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:14.252067+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:18.451121+0000","scrub_duration":0.0039562720000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":309,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.796749+0000","last_change":"2024-09-10T18:35:59.886309+0000","last_active":"2024-09-10T18:37:11.796749+0000","last_peered":"2024-09-10T18:37:11.796749+0000","last_clean":"2024-09-10T18:37:11.796749+0000","last_became_active":"2024-09-10T18:26:27.640922+0000","last_became_peered":"2024-09-10T18:26:27.640922+0000","last_unstale":"2024-09-10T18:37:11.796749+0000","last_undegraded":"2024-09-10T18:37:11.796749+0000","last_fullsized":"2024-09-10T18:37:11.796749+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:59.886242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:35:59.886242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:27.332998+0000","scrub_duration":0.001788724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"240'12","reported_seq":362,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.797265+0000","last_change":"2024-09-10T18:36:00.854061+0000","last_active":"2024-09-10T18:37:11.797265+0000","last_peered":"2024-09-10T18:37:11.797265+0000","last_clean":"2024-09-10T18:37:11.797265+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:37:11.797265+0000","last_undegraded":"2024-09-10T18:37:11.797265+0000","last_fullsized":"2024-09-10T18:37:11.797265+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-10T18:36:00.854004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:00.854004+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:20.572196+0000","scrub_duration":0.0019737209999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":349,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.805424+0000","last_change":"2024-09-10T18:36:17.693584+0000","last_active":"2024-09-10T18:37:11.805424+0000","last_peered":"2024-09-10T18:37:11.805424+0000","last_clean":"2024-09-10T18:37:11.805424+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:37:11.805424+0000","last_undegraded":"2024-09-10T18:37:11.805424+0000","last_fullsized":"2024-09-10T18:37:11.805424+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:17.693527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:17.693527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:18.331648+0000","scrub_duration":0.0014350840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":354,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.807798+0000","last_change":"2024-09-10T18:36:57.022869+0000","last_active":"2024-09-10T18:37:11.807798+0000","last_peered":"2024-09-10T18:37:11.807798+0000","last_clean":"2024-09-10T18:37:11.807798+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:37:11.807798+0000","last_undegraded":"2024-09-10T18:37:11.807798+0000","last_fullsized":"2024-09-10T18:37:11.807798+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:57.022800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:57.022800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:23.678797+0000","scrub_duration":0.0040564720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'81","reported_seq":290,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.808751+0000","last_change":"2024-09-10T18:37:05.807566+0000","last_active":"2024-09-10T18:37:11.808751+0000","last_peered":"2024-09-10T18:37:11.808751+0000","last_clean":"2024-09-10T18:37:11.808751+0000","last_became_active":"2024-09-10T18:34:02.815880+0000","last_became_peered":"2024-09-10T18:34:02.815880+0000","last_unstale":"2024-09-10T18:37:11.808751+0000","last_undegraded":"2024-09-10T18:37:11.808751+0000","last_fullsized":"2024-09-10T18:37:11.808751+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:05.807459+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:05.807459+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:22.170236+0000","scrub_duration":0.0043697320000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"240'19","reported_seq":385,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.803778+0000","last_change":"2024-09-10T18:37:05.753183+0000","last_active":"2024-09-10T18:37:11.803778+0000","last_peered":"2024-09-10T18:37:11.803778+0000","last_clean":"2024-09-10T18:37:11.803778+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:37:11.803778+0000","last_undegraded":"2024-09-10T18:37:11.803778+0000","last_fullsized":"2024-09-10T18:37:11.803778+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:37:05.752891+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-09-10T18:35:19.802016+0000","last_clean_scrub_stamp":"2024-09-10T18:37:05.752891+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:06.037770+0000","scrub_duration":0.0067102660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":360,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.808692+0000","last_change":"2024-09-10T18:36:04.875938+0000","last_active":"2024-09-10T18:37:11.808692+0000","last_peered":"2024-09-10T18:37:11.808692+0000","last_clean":"2024-09-10T18:37:11.808692+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:37:11.808692+0000","last_undegraded":"2024-09-10T18:37:11.808692+0000","last_fullsized":"2024-09-10T18:37:11.808692+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:04.875692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:36:04.875692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:20.013533+0000","scrub_duration":0.0018984749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":342,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.796931+0000","last_change":"2024-09-10T18:35:12.764509+0000","last_active":"2024-09-10T18:37:11.796931+0000","last_peered":"2024-09-10T18:37:11.796931+0000","last_clean":"2024-09-10T18:37:11.796931+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:37:11.796931+0000","last_undegraded":"2024-09-10T18:37:11.796931+0000","last_fullsized":"2024-09-10T18:37:11.796931+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:12.764433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:12.764433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069242059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":367,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806715+0000","last_change":"2024-09-10T18:36:05.455694+0000","last_active":"2024-09-10T18:37:11.806715+0000","last_peered":"2024-09-10T18:37:11.806715+0000","last_clean":"2024-09-10T18:37:11.806715+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:37:11.806715+0000","last_undegraded":"2024-09-10T18:37:11.806715+0000","last_fullsized":"2024-09-10T18:37:11.806715+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:05.455640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:05.455640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019696215999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'81","reported_seq":83,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.804515+0000","last_change":"2024-09-10T18:36:58.147850+0000","last_active":"2024-09-10T18:37:11.804515+0000","last_peered":"2024-09-10T18:37:11.804515+0000","last_clean":"2024-09-10T18:37:11.804515+0000","last_became_active":"2024-09-10T18:34:02.797304+0000","last_became_peered":"2024-09-10T18:34:02.797304+0000","last_unstale":"2024-09-10T18:37:11.804515+0000","last_undegraded":"2024-09-10T18:37:11.804515+0000","last_fullsized":"2024-09-10T18:37:11.804515+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:58.147719+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:58.147719+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:13.451302+0000","scrub_duration":0.009846858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"240'19","reported_seq":381,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806260+0000","last_change":"2024-09-10T18:36:14.732278+0000","last_active":"2024-09-10T18:37:11.806260+0000","last_peered":"2024-09-10T18:37:11.806260+0000","last_clean":"2024-09-10T18:37:11.806260+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:37:11.806260+0000","last_undegraded":"2024-09-10T18:37:11.806260+0000","last_fullsized":"2024-09-10T18:37:11.806260+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:36:14.732217+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:36:14.732217+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:39.275538+0000","scrub_duration":0.0020637149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":365,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.802152+0000","last_change":"2024-09-10T18:37:04.713668+0000","last_active":"2024-09-10T18:37:11.802152+0000","last_peered":"2024-09-10T18:37:11.802152+0000","last_clean":"2024-09-10T18:37:11.802152+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:37:11.802152+0000","last_undegraded":"2024-09-10T18:37:11.802152+0000","last_fullsized":"2024-09-10T18:37:11.802152+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:04.713474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_clean_scrub_stamp":"2024-09-10T18:37:04.713474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:09.956287+0000","scrub_duration":0.0039225839999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":538,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.800341+0000","last_change":"2024-09-10T18:35:01.701033+0000","last_active":"2024-09-10T18:37:11.800341+0000","last_peered":"2024-09-10T18:37:11.800341+0000","last_clean":"2024-09-10T18:37:11.800341+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:37:11.800341+0000","last_undegraded":"2024-09-10T18:37:11.800341+0000","last_fullsized":"2024-09-10T18:37:11.800341+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:35:01.700934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:01.700934+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050961280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":85,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.797391+0000","last_change":"2024-09-10T18:37:02.933234+0000","last_active":"2024-09-10T18:37:11.797391+0000","last_peered":"2024-09-10T18:37:11.797391+0000","last_clean":"2024-09-10T18:37:11.797391+0000","last_became_active":"2024-09-10T18:33:29.141936+0000","last_became_peered":"2024-09-10T18:33:29.141936+0000","last_unstale":"2024-09-10T18:37:11.797391+0000","last_undegraded":"2024-09-10T18:37:11.797391+0000","last_fullsized":"2024-09-10T18:37:11.797391+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:02.933068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:02.933068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:03.633941+0000","scrub_duration":0.0042260099999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"76'81","reported_seq":145,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.802889+0000","last_change":"2024-09-10T18:36:08.710620+0000","last_active":"2024-09-10T18:37:11.802889+0000","last_peered":"2024-09-10T18:37:11.802889+0000","last_clean":"2024-09-10T18:37:11.802889+0000","last_became_active":"2024-09-10T18:34:02.810381+0000","last_became_peered":"2024-09-10T18:34:02.810381+0000","last_unstale":"2024-09-10T18:37:11.802889+0000","last_undegraded":"2024-09-10T18:37:11.802889+0000","last_fullsized":"2024-09-10T18:37:11.802889+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:08.710566+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:36:08.710566+0000","last_clean_scrub_stamp":"2024-09-10T18:36:08.710566+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012163009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":384,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.800635+0000","last_change":"2024-09-10T18:37:09.977307+0000","last_active":"2024-09-10T18:37:11.800635+0000","last_peered":"2024-09-10T18:37:11.800635+0000","last_clean":"2024-09-10T18:37:11.800635+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:37:11.800635+0000","last_undegraded":"2024-09-10T18:37:11.800635+0000","last_fullsized":"2024-09-10T18:37:11.800635+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-10T18:37:09.977172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:09.977172+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:30.689481+0000","scrub_duration":0.01120503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":353,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806940+0000","last_change":"2024-09-10T18:35:17.826500+0000","last_active":"2024-09-10T18:37:11.806940+0000","last_peered":"2024-09-10T18:37:11.806940+0000","last_clean":"2024-09-10T18:37:11.806940+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:37:11.806940+0000","last_undegraded":"2024-09-10T18:37:11.806940+0000","last_fullsized":"2024-09-10T18:37:11.806940+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:17.826448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:17.826448+0000","last_clean_scrub_stamp":"2024-09-10T18:35:17.826448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002920808,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":557,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-10T18:37:14.368558+0000","last_change":"2024-09-10T18:37:03.193346+0000","last_active":"2024-09-10T18:37:14.368558+0000","last_peered":"2024-09-10T18:37:14.368558+0000","last_clean":"2024-09-10T18:37:14.368558+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:37:14.368558+0000","last_undegraded":"2024-09-10T18:37:14.368558+0000","last_fullsized":"2024-09-10T18:37:14.368558+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:37:03.193256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:03.193256+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:24.557501+0000","scrub_duration":0.0034883319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":349,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.803606+0000","last_change":"2024-09-10T18:36:06.720664+0000","last_active":"2024-09-10T18:37:11.803606+0000","last_peered":"2024-09-10T18:37:11.803606+0000","last_clean":"2024-09-10T18:37:11.803606+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:37:11.803606+0000","last_undegraded":"2024-09-10T18:37:11.803606+0000","last_fullsized":"2024-09-10T18:37:11.803606+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.720613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.720613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015989380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'81","reported_seq":65,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.810320+0000","last_change":"2024-09-10T18:35:53.652838+0000","last_active":"2024-09-10T18:37:11.810320+0000","last_peered":"2024-09-10T18:37:11.810320+0000","last_clean":"2024-09-10T18:37:11.810320+0000","last_became_active":"2024-09-10T18:34:34.881186+0000","last_became_peered":"2024-09-10T18:34:34.881186+0000","last_unstale":"2024-09-10T18:37:11.810320+0000","last_undegraded":"2024-09-10T18:37:11.810320+0000","last_fullsized":"2024-09-10T18:37:11.810320+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:53.652767+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:53.652767+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:17.120912+0000","scrub_duration":0.0013276989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":399,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:13.403328+0000","last_change":"2024-09-10T18:37:13.403259+0000","last_active":"2024-09-10T18:37:13.403328+0000","last_peered":"2024-09-10T18:37:13.403328+0000","last_clean":"2024-09-10T18:37:13.403328+0000","last_became_active":"2024-09-10T18:26:27.649202+0000","last_became_peered":"2024-09-10T18:26:27.649202+0000","last_unstale":"2024-09-10T18:37:13.403328+0000","last_undegraded":"2024-09-10T18:37:13.403328+0000","last_fullsized":"2024-09-10T18:37:13.403328+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:37:13.403175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:13.403175+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:34.162155+0000","scrub_duration":0.0070645999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":363,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.801327+0000","last_change":"2024-09-10T18:37:02.270246+0000","last_active":"2024-09-10T18:37:11.801327+0000","last_peered":"2024-09-10T18:37:11.801327+0000","last_clean":"2024-09-10T18:37:11.801327+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:37:11.801327+0000","last_undegraded":"2024-09-10T18:37:11.801327+0000","last_fullsized":"2024-09-10T18:37:11.801327+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:02.270169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:37:02.270169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:15.675634+0000","scrub_duration":0.001637306,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":363,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.805428+0000","last_change":"2024-09-10T18:36:11.523432+0000","last_active":"2024-09-10T18:37:11.805428+0000","last_peered":"2024-09-10T18:37:11.805428+0000","last_clean":"2024-09-10T18:37:11.805428+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:37:11.805428+0000","last_undegraded":"2024-09-10T18:37:11.805428+0000","last_fullsized":"2024-09-10T18:37:11.805428+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:11.523380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_clean_scrub_stamp":"2024-09-10T18:36:11.523380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:33.294003+0000","scrub_duration":0.0014936800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":330,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.796384+0000","last_change":"2024-09-10T18:34:14.263579+0000","last_active":"2024-09-10T18:37:11.796384+0000","last_peered":"2024-09-10T18:37:11.796384+0000","last_clean":"2024-09-10T18:37:11.796384+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:37:11.796384+0000","last_undegraded":"2024-09-10T18:37:11.796384+0000","last_fullsized":"2024-09-10T18:37:11.796384+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:14.263499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:34:14.263499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019685509999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'81","reported_seq":64,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.805838+0000","last_change":"2024-09-10T18:36:08.676315+0000","last_active":"2024-09-10T18:37:11.805838+0000","last_peered":"2024-09-10T18:37:11.805838+0000","last_clean":"2024-09-10T18:37:11.805838+0000","last_became_active":"2024-09-10T18:34:34.858291+0000","last_became_peered":"2024-09-10T18:34:34.858291+0000","last_unstale":"2024-09-10T18:37:11.805838+0000","last_undegraded":"2024-09-10T18:37:11.805838+0000","last_fullsized":"2024-09-10T18:37:11.805838+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:08.676247+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:08.676247+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001606512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"240'15","reported_seq":394,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.805642+0000","last_change":"2024-09-10T18:36:16.592923+0000","last_active":"2024-09-10T18:37:11.805642+0000","last_peered":"2024-09-10T18:37:11.805642+0000","last_clean":"2024-09-10T18:37:11.805642+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:37:11.805642+0000","last_undegraded":"2024-09-10T18:37:11.805642+0000","last_fullsized":"2024-09-10T18:37:11.805642+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:36:16.592847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:16.592847+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:39.683840+0000","scrub_duration":0.0020114550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":356,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.808027+0000","last_change":"2024-09-10T18:36:07.931194+0000","last_active":"2024-09-10T18:37:11.808027+0000","last_peered":"2024-09-10T18:37:11.808027+0000","last_clean":"2024-09-10T18:37:11.808027+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:37:11.808027+0000","last_undegraded":"2024-09-10T18:37:11.808027+0000","last_fullsized":"2024-09-10T18:37:11.808027+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:07.931108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.931108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:18.035585+0000","scrub_duration":0.0012878189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":335,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.796239+0000","last_change":"2024-09-10T18:34:56.701737+0000","last_active":"2024-09-10T18:37:11.796239+0000","last_peered":"2024-09-10T18:37:11.796239+0000","last_clean":"2024-09-10T18:37:11.796239+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:37:11.796239+0000","last_undegraded":"2024-09-10T18:37:11.796239+0000","last_fullsized":"2024-09-10T18:37:11.796239+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:56.701661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:34:56.701661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064135620000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":359,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.863789+0000","last_change":"2024-09-10T18:37:11.863730+0000","last_active":"2024-09-10T18:37:11.863789+0000","last_peered":"2024-09-10T18:37:11.863789+0000","last_clean":"2024-09-10T18:37:11.863789+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:37:11.863789+0000","last_undegraded":"2024-09-10T18:37:11.863789+0000","last_fullsized":"2024-09-10T18:37:11.863789+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:11.863644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:11.863644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:16.121702+0000","scrub_duration":0.0037629410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'81","reported_seq":83,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806575+0000","last_change":"2024-09-10T18:36:07.443661+0000","last_active":"2024-09-10T18:37:11.806575+0000","last_peered":"2024-09-10T18:37:11.806575+0000","last_clean":"2024-09-10T18:37:11.806575+0000","last_became_active":"2024-09-10T18:34:02.802580+0000","last_became_peered":"2024-09-10T18:34:02.802580+0000","last_unstale":"2024-09-10T18:37:11.806575+0000","last_undegraded":"2024-09-10T18:37:11.806575+0000","last_fullsized":"2024-09-10T18:37:11.806575+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:07.443596+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.443596+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:20.824349+0000","scrub_duration":0.0016421059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":382,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.796574+0000","last_change":"2024-09-10T18:35:03.727242+0000","last_active":"2024-09-10T18:37:11.796574+0000","last_peered":"2024-09-10T18:37:11.796574+0000","last_clean":"2024-09-10T18:37:11.796574+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:37:11.796574+0000","last_undegraded":"2024-09-10T18:37:11.796574+0000","last_fullsized":"2024-09-10T18:37:11.796574+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:35:03.727153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:03.727153+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069271460000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":364,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.805239+0000","last_change":"2024-09-10T18:36:53.050798+0000","last_active":"2024-09-10T18:37:11.805239+0000","last_peered":"2024-09-10T18:37:11.805239+0000","last_clean":"2024-09-10T18:37:11.805239+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:37:11.805239+0000","last_undegraded":"2024-09-10T18:37:11.805239+0000","last_fullsized":"2024-09-10T18:37:11.805239+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:53.050726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:36:53.050726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:02.660776+0000","scrub_duration":0.0099588239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":349,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806373+0000","last_change":"2024-09-10T18:34:57.776098+0000","last_active":"2024-09-10T18:37:11.806373+0000","last_peered":"2024-09-10T18:37:11.806373+0000","last_clean":"2024-09-10T18:37:11.806373+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:37:11.806373+0000","last_undegraded":"2024-09-10T18:37:11.806373+0000","last_fullsized":"2024-09-10T18:37:11.806373+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:57.776022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:57.776022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002853441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":363,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.796456+0000","last_change":"2024-09-10T18:36:52.896429+0000","last_active":"2024-09-10T18:37:11.796456+0000","last_peered":"2024-09-10T18:37:11.796456+0000","last_clean":"2024-09-10T18:37:11.796456+0000","last_became_active":"2024-09-10T18:26:27.636996+0000","last_became_peered":"2024-09-10T18:26:27.636996+0000","last_unstale":"2024-09-10T18:37:11.796456+0000","last_undegraded":"2024-09-10T18:37:11.796456+0000","last_fullsized":"2024-09-10T18:37:11.796456+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:52.896322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:52.896322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:58.016566+0000","scrub_duration":0.0029578769999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'81","reported_seq":64,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806605+0000","last_change":"2024-09-10T18:36:14.999809+0000","last_active":"2024-09-10T18:37:11.806605+0000","last_peered":"2024-09-10T18:37:11.806605+0000","last_clean":"2024-09-10T18:37:11.806605+0000","last_became_active":"2024-09-10T18:34:34.868371+0000","last_became_peered":"2024-09-10T18:34:34.868371+0000","last_unstale":"2024-09-10T18:37:11.806605+0000","last_undegraded":"2024-09-10T18:37:11.806605+0000","last_fullsized":"2024-09-10T18:37:11.806605+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:14.999747+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:14.999747+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:41.897947+0000","scrub_duration":0.001461415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":363,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.797902+0000","last_change":"2024-09-10T18:36:01.819314+0000","last_active":"2024-09-10T18:37:11.797902+0000","last_peered":"2024-09-10T18:37:11.797902+0000","last_clean":"2024-09-10T18:37:11.797902+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:37:11.797902+0000","last_undegraded":"2024-09-10T18:37:11.797902+0000","last_fullsized":"2024-09-10T18:37:11.797902+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-09-10T18:36:01.819258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:01.819258+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:17.021892+0000","scrub_duration":0.001608246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":358,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.806841+0000","last_change":"2024-09-10T18:36:49.574521+0000","last_active":"2024-09-10T18:37:11.806841+0000","last_peered":"2024-09-10T18:37:11.806841+0000","last_clean":"2024-09-10T18:37:11.806841+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:37:11.806841+0000","last_undegraded":"2024-09-10T18:37:11.806841+0000","last_fullsized":"2024-09-10T18:37:11.806841+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:49.571125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:36:49.571125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:56.658367+0000","scrub_duration":0.0066675700000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":313,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.796674+0000","last_change":"2024-09-10T18:36:01.890154+0000","last_active":"2024-09-10T18:37:11.796674+0000","last_peered":"2024-09-10T18:37:11.796674+0000","last_clean":"2024-09-10T18:37:11.796674+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:37:11.796674+0000","last_undegraded":"2024-09-10T18:37:11.796674+0000","last_fullsized":"2024-09-10T18:37:11.796674+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:01.890087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:01.890087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:29.323792+0000","scrub_duration":0.001570519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":347,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.807029+0000","last_change":"2024-09-10T18:35:18.879787+0000","last_active":"2024-09-10T18:37:11.807029+0000","last_peered":"2024-09-10T18:37:11.807029+0000","last_clean":"2024-09-10T18:37:11.807029+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:37:11.807029+0000","last_undegraded":"2024-09-10T18:37:11.807029+0000","last_fullsized":"2024-09-10T18:37:11.807029+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:18.879717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:18.879717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059146550000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'81","reported_seq":64,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.800651+0000","last_change":"2024-09-10T18:36:00.273377+0000","last_active":"2024-09-10T18:37:11.800651+0000","last_peered":"2024-09-10T18:37:11.800651+0000","last_clean":"2024-09-10T18:37:11.800651+0000","last_became_active":"2024-09-10T18:34:34.867380+0000","last_became_peered":"2024-09-10T18:34:34.867380+0000","last_unstale":"2024-09-10T18:37:11.800651+0000","last_undegraded":"2024-09-10T18:37:11.800651+0000","last_fullsized":"2024-09-10T18:37:11.800651+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:00.273308+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:00.273308+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:16.529317+0000","scrub_duration":0.0016513669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":358,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.809778+0000","last_change":"2024-09-10T18:35:56.582580+0000","last_active":"2024-09-10T18:37:11.809778+0000","last_peered":"2024-09-10T18:37:11.809778+0000","last_clean":"2024-09-10T18:37:11.809778+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:37:11.809778+0000","last_undegraded":"2024-09-10T18:37:11.809778+0000","last_fullsized":"2024-09-10T18:37:11.809778+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:56.582525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_clean_scrub_stamp":"2024-09-10T18:35:56.582525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:22.552243+0000","scrub_duration":0.0014918869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":336,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.798082+0000","last_change":"2024-09-10T18:34:03.175239+0000","last_active":"2024-09-10T18:37:11.798082+0000","last_peered":"2024-09-10T18:37:11.798082+0000","last_clean":"2024-09-10T18:37:11.798082+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:37:11.798082+0000","last_undegraded":"2024-09-10T18:37:11.798082+0000","last_fullsized":"2024-09-10T18:37:11.798082+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:03.175158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:34:03.175158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046326559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":374,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.798054+0000","last_change":"2024-09-10T18:36:47.856846+0000","last_active":"2024-09-10T18:37:11.798054+0000","last_peered":"2024-09-10T18:37:11.798054+0000","last_clean":"2024-09-10T18:37:11.798054+0000","last_became_active":"2024-09-10T18:26:27.648112+0000","last_became_peered":"2024-09-10T18:26:27.648112+0000","last_unstale":"2024-09-10T18:37:11.798054+0000","last_undegraded":"2024-09-10T18:37:11.798054+0000","last_fullsized":"2024-09-10T18:37:11.798054+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:36:47.856694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:47.856694+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:08.148989+0000","scrub_duration":0.0066546349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":351,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-10T18:37:11.803680+0000","last_change":"2024-09-10T18:35:59.688760+0000","last_active":"2024-09-10T18:37:11.803680+0000","last_peered":"2024-09-10T18:37:11.803680+0000","last_clean":"2024-09-10T18:37:11.803680+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:37:11.803680+0000","last_undegraded":"2024-09-10T18:37:11.803680+0000","last_fullsized":"2024-09-10T18:37:11.803680+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:59.688701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:35:59.688701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018218869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":6398214144,"num_objects":97629,"num_object_clones":0,"num_object_copies":292887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97629,"num_write_kb":6248256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19155582976,"data_stored":19155582976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76639,"ondisk_log_size":76639,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":746,"ondisk_log_size":746,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":936,"num_read_kb":931,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2511,"ondisk_log_size":2511,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234176,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2147124,"kb_used_data":1548904,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":91598028,"statfs":{"total":95995035648,"available":93796380672,"internally_reserved":0,"allocated":1586077696,"data_stored":1585584064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:36:53 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.4810000000000001,"15min":1.8879999999999999},"min":{"1min":0.36799999999999999,"5min":0.215,"15min":0.20799999999999999},"max":{"1min":2.7890000000000001,"5min":50.091999999999999,"15min":85.012},"last":2.8839999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.292,"15min":1.796},"min":{"1min":0.223,"5min":0.223,"15min":0.222},"max":{"1min":1.766,"5min":50.104999999999997,"15min":84.665999999999997},"last":0.46999999999999997}]},{"osd":1,"last update":"Tue Sep 10 18:36:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":1.4099999999999999,"15min":1.468},"min":{"1min":0.24299999999999999,"5min":0.23899999999999999,"15min":0.214},"max":{"1min":2.0649999999999999,"5min":39.728000000000002,"15min":49.317999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":1.369,"15min":1.393},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.4540000000000002,"5min":39.738999999999997,"15min":49.353000000000002},"last":0.308}]},{"osd":2,"last update":"Tue Sep 10 18:36:53 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":2.5089999999999999,"15min":1.4159999999999999},"min":{"1min":0.32900000000000001,"5min":0.27400000000000002,"15min":0.249},"max":{"1min":2.7810000000000001,"5min":136.648,"15min":136.648},"last":3.8090000000000002},{"interface":"front","average":{"1min":0.94099999999999995,"5min":2.5219999999999998,"15min":1.49},"min":{"1min":0.38700000000000001,"5min":0.29199999999999998,"15min":0.24399999999999999},"max":{"1min":4.0979999999999999,"5min":136.506,"15min":136.506},"last":0.38700000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:36:53 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.6160000000000001,"15min":1.653},"min":{"1min":0.36599999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.8039999999999998,"5min":38.225000000000001,"15min":86.305999999999997},"last":1.1859999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":1.5640000000000001,"15min":1.673},"min":{"1min":0.41999999999999998,"5min":0.29199999999999998,"15min":0.26700000000000002},"max":{"1min":2.5609999999999999,"5min":38.104999999999997,"15min":88.364000000000004},"last":0.80400000000000005}]},{"osd":4,"last update":"Tue Sep 10 18:36:53 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.625,"15min":0.53400000000000003},"min":{"1min":0.35599999999999998,"5min":0.24199999999999999,"15min":0.20799999999999999},"max":{"1min":1.7949999999999999,"5min":2.282,"15min":3.8839999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.72799999999999998,"15min":0.58499999999999996},"min":{"1min":0.34699999999999998,"5min":0.26200000000000001,"15min":0.223},"max":{"1min":2.7480000000000002,"5min":4.0730000000000004,"15min":4.0730000000000004},"last":0.29399999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:36:53 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.88300000000000001,"15min":0.745},"min":{"1min":0.32100000000000001,"5min":0.27300000000000002,"15min":0.23499999999999999},"max":{"1min":2.3149999999999999,"5min":4.1479999999999997,"15min":7.3109999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.877,"15min":0.68700000000000006},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.219},"max":{"1min":1.821,"5min":3.911,"15min":6.3360000000000003},"last":0.371}]},{"osd":6,"last update":"Tue Sep 10 18:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.752,"15min":0.67200000000000004},"min":{"1min":0.38100000000000001,"5min":0.29299999999999998,"15min":0.27000000000000002},"max":{"1min":0.83499999999999996,"5min":2.7890000000000001,"15min":7.2750000000000004},"last":0.439},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.69599999999999995,"15min":0.61799999999999999},"min":{"1min":0.437,"5min":0.36699999999999999,"15min":0.30199999999999999},"max":{"1min":1.177,"5min":2.3199999999999998,"15min":4.585},"last":0.34100000000000003}]}]},{"osd":6,"up_from":144,"seq":618475290755,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3040608,"kb_used_data":2738004,"kb_used_omap":7,"kb_used_meta":302584,"kb_avail":90704544,"statfs":{"total":95995035648,"available":92881453056,"internally_reserved":0,"allocated":2803716096,"data_stored":2803242572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":309846103},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:36:48 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.92800000000000005,"15min":1.613},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.343,"5min":7.8860000000000001,"15min":30.629999999999999},"last":0.747},{"interface":"front","average":{"1min":2.129,"5min":1.0389999999999999,"15min":1.3220000000000001},"min":{"1min":0.35299999999999998,"5min":0.217,"15min":0.217},"max":{"1min":18.093,"5min":18.093,"15min":30.597000000000001},"last":0.54200000000000004}]},{"osd":1,"last update":"Tue Sep 10 18:36:48 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.871,"15min":0.96799999999999997},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.0870000000000002,"5min":3.806,"15min":30.984000000000002},"last":1.163},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.97099999999999997,"15min":1.145},"min":{"1min":0.28399999999999997,"5min":0.25700000000000001,"15min":0.24199999999999999},"max":{"1min":1.915,"5min":7.7939999999999996,"15min":30.571000000000002},"last":0.41699999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:36:48 2024","interfaces":[{"interface":"back","average":{"1min":1.9119999999999999,"5min":1.4930000000000001,"15min":1.2030000000000001},"min":{"1min":0.378,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":18.113,"5min":39.329999999999998,"15min":39.329999999999998},"last":0.44},{"interface":"front","average":{"1min":1.083,"5min":1.1639999999999999,"15min":0.99199999999999999},"min":{"1min":0.39100000000000001,"5min":0.26200000000000001,"15min":0.23400000000000001},"max":{"1min":3.1259999999999999,"5min":39.061999999999998,"15min":39.061999999999998},"last":0.29399999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:36:48 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.1839999999999999,"15min":1.758},"min":{"1min":0.36199999999999999,"5min":0.28199999999999997,"15min":0.27800000000000002},"max":{"1min":3.1560000000000001,"5min":16.831,"15min":61.703000000000003},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.85099999999999998,"5min":1.2569999999999999,"15min":1.7749999999999999},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.0459999999999998,"5min":17.786000000000001,"15min":61.814},"last":1.266}]},{"osd":4,"last update":"Tue Sep 10 18:36:48 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.0220000000000002,"5min":4.0220000000000002,"15min":5.4950000000000001},"last":3.6709999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.56799999999999995,"15min":0.63200000000000001},"min":{"1min":0.33400000000000002,"5min":0.222,"15min":0.222},"max":{"1min":1.883,"5min":4.1689999999999996,"15min":4.6390000000000002},"last":0.46200000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:36:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.69199999999999995,"15min":0.59899999999999998},"min":{"1min":0.33100000000000002,"5min":0.27400000000000002,"15min":0.24399999999999999},"max":{"1min":1.857,"5min":4.476,"15min":4.6619999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":1.0269999999999999,"5min":0.78400000000000003,"15min":0.60499999999999998},"min":{"1min":0.30299999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.2290000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.32900000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:36:48 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.747,"15min":0.60899999999999999},"min":{"1min":0.41199999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":3.081,"5min":3.3650000000000002,"15min":3.3650000000000002},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.65100000000000002,"15min":0.622},"min":{"1min":0.36099999999999999,"5min":0.28999999999999998,"15min":0.27100000000000002},"max":{"1min":2.8319999999999999,"5min":3.2320000000000002,"15min":5.5309999999999997},"last":0.51600000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724747,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2513692,"kb_used_data":1915476,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":91231460,"statfs":{"total":95995035648,"available":93421015040,"internally_reserved":0,"allocated":1961447424,"data_stored":1960967619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:37:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.97999999999999998,"15min":1.556},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.21099999999999999},"max":{"1min":2.7200000000000002,"5min":15.819000000000001,"15min":52.033999999999999},"last":1.8959999999999999},{"interface":"front","average":{"1min":1.2490000000000001,"5min":0.95099999999999996,"15min":1.371},"min":{"1min":0.24399999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":2.4300000000000002,"5min":5.4420000000000002,"15min":52.167999999999999},"last":1.2909999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:37:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":2.3929999999999998,"15min":1.5},"min":{"1min":0.33100000000000002,"5min":0.249,"15min":0.22600000000000001},"max":{"1min":1.3440000000000001,"5min":111.773,"15min":111.773},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":2.54,"15min":1.6200000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.247},"max":{"1min":2.6789999999999998,"5min":111.73999999999999,"15min":111.73999999999999},"last":0.35999999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:37:08 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.3360000000000001,"15min":1.242},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.24299999999999999},"max":{"1min":3.3180000000000001,"5min":54.530000000000001,"15min":54.530000000000001},"last":1.014},{"interface":"front","average":{"1min":0.82599999999999996,"5min":1.2889999999999999,"15min":1.202},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":2.4409999999999998,"5min":54.514000000000003,"15min":54.514000000000003},"last":0.58899999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:37:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":2.395,"15min":1.879},"min":{"1min":0.29999999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.4260000000000002,"5min":106.205,"15min":106.205},"last":1.4530000000000001},{"interface":"front","average":{"1min":0.75,"5min":2.5430000000000001,"15min":1.829},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.6760000000000002,"5min":105.878,"15min":105.878},"last":0.46500000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:37:08 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.69699999999999995,"15min":0.58299999999999996},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.20799999999999999},"max":{"1min":2.7120000000000002,"5min":5.8040000000000003,"15min":5.8040000000000003},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.68200000000000005,"15min":0.623},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":2.7509999999999999,"5min":5.5499999999999998,"15min":6.798},"last":0.42699999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58399999999999996,"15min":0.64500000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.27500000000000002},"max":{"1min":0.88,"5min":2.7469999999999999,"15min":4.7859999999999996},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.623,"15min":0.66100000000000003},"min":{"1min":0.29599999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.94899999999999995,"5min":3.2850000000000001,"15min":5.3860000000000001},"last":0.41299999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":1.0109999999999999,"15min":0.73899999999999999},"min":{"1min":0.36299999999999999,"5min":0.25900000000000001,"15min":0.22500000000000001},"max":{"1min":0.89900000000000002,"5min":28.934000000000001,"15min":28.934000000000001},"last":0.437},{"interface":"front","average":{"1min":0.63,"5min":0.91700000000000004,"15min":0.65600000000000003},"min":{"1min":0.33900000000000002,"5min":0.309,"15min":0.24299999999999999},"max":{"1min":1.8779999999999999,"5min":5.8259999999999996,"15min":5.8259999999999996},"last":0.33200000000000002}]}]},{"osd":4,"up_from":33,"seq":141733920975,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3386032,"kb_used_data":3080356,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":90359120,"statfs":{"total":95995035648,"available":92527738880,"internally_reserved":0,"allocated":3154284544,"data_stored":3153794275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.81000000000000005,"15min":1.0680000000000001},"min":{"1min":0.23599999999999999,"5min":0.223,"15min":0.188},"max":{"1min":3.1150000000000002,"5min":3.7309999999999999,"15min":31.585000000000001},"last":1.1559999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.79100000000000004,"15min":1.014},"min":{"1min":0.22700000000000001,"5min":0.189,"15min":0.189},"max":{"1min":1.153,"5min":4.8929999999999998,"15min":29.802},"last":1.296}]},{"osd":1,"last update":"Tue Sep 10 18:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.46,"15min":1.0349999999999999},"min":{"1min":0.26800000000000002,"5min":0.254,"15min":0.21299999999999999},"max":{"1min":1.3360000000000001,"5min":50.045000000000002,"15min":50.045000000000002},"last":1.095},{"interface":"front","average":{"1min":0.56599999999999995,"5min":1.5189999999999999,"15min":1.111},"min":{"1min":0.40999999999999998,"5min":0.27100000000000002,"15min":0.22900000000000001},"max":{"1min":1.0149999999999999,"5min":50.283000000000001,"15min":50.283000000000001},"last":0.39200000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.66900000000000004,"15min":0.87},"min":{"1min":0.40699999999999997,"5min":0.249,"15min":0.249},"max":{"1min":1.276,"5min":3.0870000000000002,"15min":13.898999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.71899999999999997,"15min":0.89400000000000002},"min":{"1min":0.26700000000000002,"5min":0.25900000000000001,"15min":0.22500000000000001},"max":{"1min":1.157,"5min":3.2519999999999998,"15min":14.51},"last":1.24}]},{"osd":3,"last update":"Tue Sep 10 18:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":1.538,"15min":1.3700000000000001},"min":{"1min":0.318,"5min":0.27400000000000002,"15min":0.249},"max":{"1min":1.0640000000000001,"5min":49.990000000000002,"15min":49.990000000000002},"last":1.27},{"interface":"front","average":{"1min":0.628,"5min":1.6439999999999999,"15min":1.4399999999999999},"min":{"1min":0.32800000000000001,"5min":0.29799999999999999,"15min":0.255},"max":{"1min":1.3520000000000001,"5min":50.764000000000003,"15min":50.764000000000003},"last":1.121}]},{"osd":5,"last update":"Tue Sep 10 18:37:03 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.745,"15min":0.66500000000000004},"min":{"1min":0.309,"5min":0.223,"15min":0.20499999999999999},"max":{"1min":1.897,"5min":3.0409999999999999,"15min":22.861000000000001},"last":1.8540000000000001},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.81599999999999995,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.252,"15min":0.22},"max":{"1min":8.2539999999999996,"5min":8.2539999999999996,"15min":19.611999999999998},"last":4.5730000000000004}]},{"osd":6,"last update":"Tue Sep 10 18:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.72499999999999998,"15min":0.56399999999999995},"min":{"1min":0.29999999999999999,"5min":0.28799999999999998,"15min":0.26400000000000001},"max":{"1min":2.1600000000000001,"5min":3.1139999999999999,"15min":6.7990000000000004},"last":1.0569999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.68300000000000005,"15min":0.59599999999999997},"min":{"1min":0.38300000000000001,"5min":0.307,"15min":0.30599999999999999},"max":{"1min":2.827,"5min":3.21,"15min":7.0300000000000002},"last":0.50600000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.69699999999999995,"15min":0.58699999999999997},"min":{"1min":0.34399999999999997,"5min":0.316,"15min":0.255},"max":{"1min":1.8160000000000001,"5min":3.262,"15min":7.0670000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.69899999999999995,"15min":0.60499999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.247},"max":{"1min":1.236,"5min":4.0430000000000001,"15min":6.5330000000000004},"last":1.212}]}]},{"osd":3,"up_from":27,"seq":115964117205,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2802416,"kb_used_data":1910820,"kb_used_omap":1,"kb_used_meta":891582,"kb_avail":90942736,"statfs":{"total":95995035648,"available":93125361664,"internally_reserved":0,"allocated":1956679680,"data_stored":1956183344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912980312},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:36:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":1.0369999999999999,"15min":1.3109999999999999},"min":{"1min":0.21199999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.4890000000000001,"5min":5.4560000000000004,"15min":8.5530000000000008},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.58399999999999996,"5min":1.179,"15min":1.403},"min":{"1min":0.23300000000000001,"5min":0.23100000000000001,"15min":0.193},"max":{"1min":2.9590000000000001,"5min":8.2110000000000003,"15min":13.659000000000001},"last":1.137}]},{"osd":1,"last update":"Tue Sep 10 18:36:16 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.333,"15min":1.518},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.641,"5min":6.149,"15min":35.406999999999996},"last":1.1100000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":1.129,"15min":1.3979999999999999},"min":{"1min":0.254,"5min":0.20799999999999999,"15min":0.19},"max":{"1min":2.468,"5min":5.3529999999999998,"15min":36.902000000000001},"last":1.0249999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:36:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":1.0169999999999999,"15min":1.1519999999999999},"min":{"1min":0.29999999999999999,"5min":0.247,"15min":0.222},"max":{"1min":1.401,"5min":6.6479999999999997,"15min":7.7960000000000003},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.70899999999999996,"5min":1.0920000000000001,"15min":1.351},"min":{"1min":0.26500000000000001,"5min":0.23000000000000001,"15min":0.217},"max":{"1min":3.1640000000000001,"5min":6.7290000000000001,"15min":14.997999999999999},"last":2.1779999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:36:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":1.0620000000000001,"15min":1.2789999999999999},"min":{"1min":0.41299999999999998,"5min":0.313,"15min":0.313},"max":{"1min":1.4259999999999999,"5min":6.5250000000000004,"15min":23.684999999999999},"last":2.2599999999999998},{"interface":"front","average":{"1min":0.69699999999999995,"5min":1.0600000000000001,"15min":1.2230000000000001},"min":{"1min":0.33900000000000002,"5min":0.29799999999999999,"15min":0.27200000000000002},"max":{"1min":2.4870000000000001,"5min":6.1929999999999996,"15min":23.649000000000001},"last":1.2829999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:37:00 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":1.161,"15min":1.3799999999999999},"min":{"1min":0.34499999999999997,"5min":0.314,"15min":0.314},"max":{"1min":1.4570000000000001,"5min":6.601,"15min":23.719999999999999},"last":2.3079999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":1.2989999999999999,"15min":1.3640000000000001},"min":{"1min":0.375,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.492,"5min":6.2249999999999996,"15min":23.707000000000001},"last":1.4630000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:36:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":1.167,"15min":1.2470000000000001},"min":{"1min":0.41399999999999998,"5min":0.33800000000000002,"15min":0.317},"max":{"1min":1.1259999999999999,"5min":6.2889999999999997,"15min":31.815999999999999},"last":2.1099999999999999},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.165,"15min":1.4550000000000001},"min":{"1min":0.39300000000000002,"5min":0.38500000000000001,"15min":0.315},"max":{"1min":1.8999999999999999,"5min":7.0970000000000004,"15min":31.742000000000001},"last":0.998}]},{"osd":7,"last update":"Tue Sep 10 18:36:49 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":1.1859999999999999,"15min":1.3220000000000001},"min":{"1min":0.45900000000000002,"5min":0.34999999999999998,"15min":0.34100000000000003},"max":{"1min":1.383,"5min":6.5730000000000004,"15min":23.748999999999999},"last":2.2130000000000001},{"interface":"front","average":{"1min":0.75700000000000001,"5min":1.452,"15min":1.395},"min":{"1min":0.46200000000000002,"5min":0.40799999999999997,"15min":0.307},"max":{"1min":2.2000000000000002,"5min":12.612,"15min":25.058},"last":1.3069999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411545,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2614948,"kb_used_data":2317592,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91130204,"statfs":{"total":95995035648,"available":93317328896,"internally_reserved":0,"allocated":2373214208,"data_stored":2372736243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.1080000000000001,"15min":1.5149999999999999},"min":{"1min":0.30199999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":3.2570000000000001,"5min":26.585999999999999,"15min":43.890000000000001},"last":1.617},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3009999999999999,"15min":1.76},"min":{"1min":0.26900000000000002,"5min":0.19400000000000001,"15min":0.16900000000000001},"max":{"1min":3.0390000000000001,"5min":26.48,"15min":44.331000000000003},"last":1.7450000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.90900000000000003,"15min":1.5089999999999999},"min":{"1min":0.27100000000000002,"5min":0.19400000000000001,"15min":0.193},"max":{"1min":2.2770000000000001,"5min":7.266,"15min":44.575000000000003},"last":1.256},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.84299999999999997,"15min":1.3009999999999999},"min":{"1min":0.33700000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":3.2909999999999999,"5min":7.0270000000000001,"15min":44.368000000000002},"last":1.0069999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:37:14 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.7290000000000001,"15min":1.8360000000000001},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.339,"5min":61.331000000000003,"15min":61.331000000000003},"last":1.647},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.8879999999999999,"15min":1.9199999999999999},"min":{"1min":0.24199999999999999,"5min":0.23499999999999999,"15min":0.214},"max":{"1min":4.5229999999999997,"5min":61.250999999999998,"15min":61.250999999999998},"last":4.5229999999999997}]},{"osd":4,"last update":"Tue Sep 10 18:36:45 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.91400000000000003,"15min":1.052},"min":{"1min":0.38500000000000001,"5min":0.30199999999999999,"15min":0.24199999999999999},"max":{"1min":2.0779999999999998,"5min":12.555999999999999,"15min":12.555999999999999},"last":2.5699999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.73099999999999998,"15min":0.94899999999999995},"min":{"1min":0.44500000000000001,"5min":0.33000000000000002,"15min":0.317},"max":{"1min":1.6739999999999999,"5min":6.3639999999999999,"15min":12.130000000000001},"last":2.6850000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.93600000000000005,"15min":1.05},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.27500000000000002},"max":{"1min":2.476,"5min":6.3170000000000002,"15min":12.106},"last":1.712},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.1599999999999999,"15min":1.1339999999999999},"min":{"1min":0.38800000000000001,"5min":0.317,"15min":0.317},"max":{"1min":4.5110000000000001,"5min":12.529999999999999,"15min":12.529999999999999},"last":1.6699999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:36:45 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.93799999999999994,"15min":1.095},"min":{"1min":0.53800000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.3100000000000001,"5min":4.6440000000000001,"15min":5.9989999999999997},"last":2.5329999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.82499999999999996,"15min":0.96799999999999997},"min":{"1min":0.53900000000000003,"5min":0.34999999999999998,"15min":0.30099999999999999},"max":{"1min":1.9399999999999999,"5min":6.4089999999999998,"15min":7.1349999999999998},"last":1.6899999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.0589999999999999,"15min":1.121},"min":{"1min":0.497,"5min":0.38900000000000001,"15min":0.30499999999999999},"max":{"1min":2.4889999999999999,"5min":10.68,"15min":10.68},"last":4.5590000000000002},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.075,"15min":1.1679999999999999},"min":{"1min":0.51200000000000001,"5min":0.376,"15min":0.307},"max":{"1min":2.0499999999999998,"5min":12.596,"15min":12.596},"last":1.2889999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575070,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1764616,"kb_used_data":1170556,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":91980536,"statfs":{"total":95995035648,"available":94188068864,"internally_reserved":0,"allocated":1198649344,"data_stored":1198135349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":1.2849999999999999,"15min":1.681},"min":{"1min":0.20999999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.91100000000000003,"5min":28.893999999999998,"15min":80.456999999999994},"last":3.8759999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":1.3620000000000001,"15min":1.7789999999999999},"min":{"1min":0.23999999999999999,"5min":0.22800000000000001,"15min":0.187},"max":{"1min":2.8450000000000002,"5min":31.074999999999999,"15min":80.578000000000003},"last":4.2939999999999996}]},{"osd":2,"last update":"Tue Sep 10 18:36:49 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.81799999999999995,"15min":1.2350000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19500000000000001},"max":{"1min":2.29,"5min":5.8380000000000001,"15min":13.035},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.86799999999999999,"15min":1.1879999999999999},"min":{"1min":0.26600000000000001,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":2.9399999999999999,"5min":3.669,"15min":12.222},"last":5.7770000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:36:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":1.161,"15min":1.6419999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.22600000000000001},"max":{"1min":1.7470000000000001,"5min":8.0399999999999991,"15min":43.076000000000001},"last":3.7280000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":1.0920000000000001,"15min":1.6839999999999999},"min":{"1min":0.27300000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69799999999999995,"5min":5.8879999999999999,"15min":41.93},"last":0.78900000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":1.085,"15min":1.107},"min":{"1min":0.36799999999999999,"5min":0.312,"15min":0.29799999999999999},"max":{"1min":0.99299999999999999,"5min":5.9930000000000003,"15min":7.2859999999999996},"last":0.75},{"interface":"front","average":{"1min":0.746,"5min":1.224,"15min":1.26},"min":{"1min":0.35199999999999998,"5min":0.30299999999999999,"15min":0.28299999999999997},"max":{"1min":2.8290000000000002,"5min":7.8769999999999998,"15min":19.393000000000001},"last":0.71699999999999997}]},{"osd":5,"last update":"Tue Sep 10 18:37:03 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":1.202,"15min":1.3089999999999999},"min":{"1min":0.308,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.7919999999999998,"5min":7.2110000000000003,"15min":15.48},"last":1.7949999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":1.337,"15min":1.248},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.308},"max":{"1min":3.931,"5min":11.414999999999999,"15min":11.414999999999999},"last":3.8170000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:36:49 2024","interfaces":[{"interface":"back","average":{"1min":4.1239999999999997,"5min":1.6830000000000001,"15min":1.458},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":67.287000000000006,"5min":67.287000000000006,"15min":67.287000000000006},"last":3.7810000000000001},{"interface":"front","average":{"1min":4.3550000000000004,"5min":1.8460000000000001,"15min":1.5580000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.33600000000000002},"max":{"1min":67.400000000000006,"5min":67.400000000000006,"15min":67.400000000000006},"last":0.84499999999999997}]},{"osd":7,"last update":"Tue Sep 10 18:36:49 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1.145,"15min":1.2190000000000001},"min":{"1min":0.378,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.5369999999999999,"5min":8.3650000000000002,"15min":15.545999999999999},"last":3.8460000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":1.2729999999999999,"15min":1.2290000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.313},"max":{"1min":2.0950000000000002,"5min":6.3049999999999997,"15min":7.359},"last":0.81899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738595,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4518764,"kb_used_data":3920544,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":89226388,"statfs":{"total":95995035648,"available":91367821312,"internally_reserved":0,"allocated":4014637056,"data_stored":4014150756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:36:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":2.5800000000000001,"15min":1.956},"min":{"1min":0.221,"5min":0.185,"15min":0.185},"max":{"1min":4.2699999999999996,"5min":146.44499999999999,"15min":146.44499999999999},"last":2.835},{"interface":"front","average":{"1min":0.498,"5min":2.5640000000000001,"15min":2.0579999999999998},"min":{"1min":0.23799999999999999,"5min":0.23100000000000001,"15min":0.17999999999999999},"max":{"1min":1.869,"5min":148.77199999999999,"15min":148.77199999999999},"last":4.2389999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:36:56 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":2.484,"15min":1.919},"min":{"1min":0.24199999999999999,"5min":0.19700000000000001,"15min":0.189},"max":{"1min":4.1559999999999997,"5min":146.56999999999999,"15min":146.56999999999999},"last":2.9980000000000002},{"interface":"front","average":{"1min":0.86799999999999999,"5min":2.2919999999999998,"15min":1.8799999999999999},"min":{"1min":0.23799999999999999,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":3.137,"5min":146.601,"15min":146.601},"last":0.55100000000000005}]},{"osd":3,"last update":"Tue Sep 10 18:36:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":2.3700000000000001,"15min":1.921},"min":{"1min":0.27100000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.2570000000000001,"5min":140.68299999999999,"15min":140.68299999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.80400000000000005,"5min":2.5259999999999998,"15min":1.9430000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.23899999999999999},"max":{"1min":4.2320000000000002,"5min":140.572,"15min":140.572},"last":0.81100000000000005}]},{"osd":4,"last update":"Tue Sep 10 18:36:43 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":0.98099999999999998,"15min":1.2090000000000001},"min":{"1min":0.38100000000000001,"5min":0.27900000000000003,"15min":0.246},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":17.151},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.84099999999999997,"15min":1.1359999999999999},"min":{"1min":0.33900000000000002,"5min":0.31,"15min":0.25900000000000001},"max":{"1min":2.919,"5min":3.552,"15min":18.728999999999999},"last":3.0299999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:37:10 2024","interfaces":[{"interface":"back","average":{"1min":1.736,"5min":1.2,"15min":1.361},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.4589999999999996,"5min":8.1259999999999994,"15min":18.77},"last":3.0529999999999999},{"interface":"front","average":{"1min":1.524,"5min":1.262,"15min":1.3200000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":5.5369999999999999,"5min":20.460000000000001,"15min":20.460000000000001},"last":0.73299999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:36:43 2024","interfaces":[{"interface":"back","average":{"1min":1.6699999999999999,"5min":1.0840000000000001,"15min":1.2330000000000001},"min":{"1min":0.40799999999999997,"5min":0.36299999999999999,"15min":0.30499999999999999},"max":{"1min":6.4909999999999997,"5min":6.4909999999999997,"15min":8.3940000000000001},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.88700000000000001,"15min":1.218},"min":{"1min":0.34799999999999998,"5min":0.34399999999999997,"15min":0.33300000000000002},"max":{"1min":3.0190000000000001,"5min":3.585,"15min":9.7859999999999996},"last":3.0950000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:36:43 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.1639999999999999,"15min":1.272},"min":{"1min":0.38600000000000001,"5min":0.377,"15min":0.26700000000000002},"max":{"1min":2.8650000000000002,"5min":10.75,"15min":13.711},"last":3.1179999999999999},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.94599999999999995,"15min":1.1919999999999999},"min":{"1min":0.41299999999999998,"5min":0.378,"15min":0.33500000000000002},"max":{"1min":1.724,"5min":11.218999999999999,"15min":15.032},"last":0.76600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4033347584,"data_stored":4033347584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1209991168,"data_stored":1209991168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2382233600,"data_stored":2382233600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1976369152,"data_stored":1976369152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3174105088,"data_stored":3174105088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1981743104,"data_stored":1981743104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2811953152,"data_stored":2811953152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1585840128,"data_stored":1585840128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:37:20.710 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:20.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:20.749 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:20.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:20.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:20.889 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:20.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:20.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:20.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:21.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 62 16 104311 104295 105.117 82.5 0.0131946 0.00950856 2024-09-10T18:37:21.029 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:21.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:21.130 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:21.169 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:21.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:21.212 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:37:21.212 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:37:21.271 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:21.303 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:21.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:21.405 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:21.442 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:21.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:21.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:21.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:21.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:21.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:21.720 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:21.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:21.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:21.737 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:37:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:37:21] "GET /metrics HTTP/1.1" 200 224694 "" "Prometheus/2.43.0" 2024-09-10T18:37:21.822 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:21.860 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:21.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:21.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:21.962 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:21.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-10T18:37:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:21.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:21.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:22.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 63 16 105612 105596 104.739 81.3125 0.00539433 0.0095427 2024-09-10T18:37:22.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:22.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:22.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:22.233 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:22.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:22.266 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:22.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:22 smithi204 ceph-mon[32357]: pgmap v719: 178 pgs: 178 active+clean; 6.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 11 B/s, 0 objects/s recovering 2024-09-10T18:37:22.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:37:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:37:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:37:22.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:22.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:22 smithi028 ceph-mon[39220]: pgmap v719: 178 pgs: 178 active+clean; 6.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 11 B/s, 0 objects/s recovering 2024-09-10T18:37:22.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:22 smithi028 ceph-mon[29291]: pgmap v719: 178 pgs: 178 active+clean; 6.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 11 B/s, 0 objects/s recovering 2024-09-10T18:37:22.405 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:37:22.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:22.414 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:22.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:22.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:22.564 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:22.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:22.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:22.666 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:22.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:22.702 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:22.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:22.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:22.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:22.843 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:22.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:22.944 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:22.974 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:22.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:22.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:23.026 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 64 16 106770 106754 104.234 72.375 0.0223397 0.00958917 2024-09-10T18:37:23.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:23.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:23.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:23.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:37:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:37:23.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:23.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:23.268 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:23.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:23 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2456551684' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:37:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:23 smithi204 ceph-mon[32357]: from='client.35672 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:37:23.369 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:23.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:23.405 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:23.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:23.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:23 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2456551684' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:37:23.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:23 smithi028 ceph-mon[29291]: from='client.35672 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:37:23.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:23 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2456551684' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:37:23.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:23 smithi028 ceph-mon[39220]: from='client.35672 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:37:23.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:23.536 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:23.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:23.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:23.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:23.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:23.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:23.782 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:23.835 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:23.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:23.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:23.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:23.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:23.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:24.027 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 65 16 107964 107948 103.778 74.625 0.0111426 0.00963157 2024-09-10T18:37:24.078 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:24.123 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:24.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:24.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:24.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:24.262 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:24.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:24.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:24.332 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:37:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:24 smithi204 ceph-mon[32357]: pgmap v720: 178 pgs: 178 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s; 10 B/s, 0 objects/s recovering 2024-09-10T18:37:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:24 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:37:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:24 smithi204 ceph-mon[32357]: from='client.35672 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:37:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:24 smithi204 ceph-mon[32357]: osdmap e336: 8 total, 8 up, 8 in 2024-09-10T18:37:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:24 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:37:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:37:24.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:24.408 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:37:24.408 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:37:24.410 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:24.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:24.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:24.476 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:24 smithi028 ceph-mon[39220]: pgmap v720: 178 pgs: 178 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s; 10 B/s, 0 objects/s recovering 2024-09-10T18:37:24.476 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:24 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:37:24.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:24 smithi028 ceph-mon[39220]: from='client.35672 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:37:24.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:24 smithi028 ceph-mon[39220]: osdmap e336: 8 total, 8 up, 8 in 2024-09-10T18:37:24.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:24 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:37:24.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:37:24.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:24 smithi028 ceph-mon[29291]: pgmap v720: 178 pgs: 178 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s; 10 B/s, 0 objects/s recovering 2024-09-10T18:37:24.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:24 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:37:24.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:24 smithi028 ceph-mon[29291]: from='client.35672 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:37:24.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:24 smithi028 ceph-mon[29291]: osdmap e336: 8 total, 8 up, 8 in 2024-09-10T18:37:24.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:24 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:37:24.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:37:24.512 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:24.550 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:24.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:24.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:24.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:24.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:24.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:24.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:24.846 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:24.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:24.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:24.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:24.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:24.985 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:24.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:25.027 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 66 16 109123 109107 103.303 72.4375 0.0164737 0.00967509 2024-09-10T18:37:25.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:25.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:37:24 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:37:25.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:25.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:25.177 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-10T18:37:25.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:25.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:25.270 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:25.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:25.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:25.408 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:25.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:25.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:25.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:25.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:25.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:25.604 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:37:25.646 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:37:25.646 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:37:25.646 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-10T18:37:25.646 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-10T18:37:25.646 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-10T18:37:25.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-10T18:37:25.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:25.753 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:25.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:25.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:25.854 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:25.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:25.914 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:25.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:26.015 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:26.028 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 67 16 110252 110236 102.812 70.5625 0.012845 0.00972149 2024-09-10T18:37:26.045 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:26.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:26.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:26.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:26.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:26.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:26.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:37:26.250 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:26 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:26.250 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:26 smithi028 ceph-mon[39220]: pgmap v722: 178 pgs: 178 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-09-10T18:37:26.250 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:26 smithi028 ceph-mon[39220]: osdmap e337: 8 total, 8 up, 8 in 2024-09-10T18:37:26.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:26 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:26.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:26 smithi028 ceph-mon[29291]: pgmap v722: 178 pgs: 178 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-09-10T18:37:26.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:26 smithi028 ceph-mon[29291]: osdmap e337: 8 total, 8 up, 8 in 2024-09-10T18:37:26.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:26.350 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:26.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:26 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:26 smithi204 ceph-mon[32357]: pgmap v722: 178 pgs: 178 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-09-10T18:37:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:26 smithi204 ceph-mon[32357]: osdmap e337: 8 total, 8 up, 8 in 2024-09-10T18:37:26.451 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:26.483 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:26.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:26.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:26.487 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:37:26.487 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:37:26.519 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:37:26.520 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:37:26.551 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:37:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:37:26.582 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:37:26.582 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:37:26.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:26.606 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:37:26.623 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:37:26.623 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:37:26.654 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:37:26.654 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:37:26.655 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:26.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:26.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:26.688 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:37:26.688 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:37:26.721 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:37:26.721 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:37:26.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:26.801 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:26.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:26.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:26.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:26.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:26.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:27.028 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 68 16 111386 111370 102.343 70.875 0.0174293 0.00976643 2024-09-10T18:37:27.045 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:27.084 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:27.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:27.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:27.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:27.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:27.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:27.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:27.329 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:27.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:27 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3023595006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:37:27.363 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:27.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:27.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:27.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:27.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:27 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3023595006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:37:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:27 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3023595006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:37:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:27.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:27.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:27.548 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:37:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:37:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:112047 10.6 10:62615bdd:::benchmark_data_smithi028_161947_object112046:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-09-10T18:37:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:37:27.537200+0000", 2024-09-10T18:37:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0090699179999999997, 2024-09-10T18:37:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0091017470000000003, 2024-09-10T18:37:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:37:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:37:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:37:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 112047 2024-09-10T18:37:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.537200+0000", 2024-09-10T18:37:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.537200+0000", 2024-09-10T18:37:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.537197+0000", 2024-09-10T18:37:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:37:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.537237+0000", 2024-09-10T18:37:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9981999999999998e-05 2024-09-10T18:37:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.537250+0000", 2024-09-10T18:37:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2614e-05 2024-09-10T18:37:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.537256+0000", 2024-09-10T18:37:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9599999999999997e-06 2024-09-10T18:37:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.537302+0000", 2024-09-10T18:37:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6310999999999998e-05 2024-09-10T18:37:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.537411+0000", 2024-09-10T18:37:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108627 2024-09-10T18:37:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.553 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:37:27.553 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.537540+0000", 2024-09-10T18:37:27.553 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012869800000000001 2024-09-10T18:37:27.553 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.553 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.553 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:27.553 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.545191+0000", 2024-09-10T18:37:27.553 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0076511929999999997 2024-09-10T18:37:27.553 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.553 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.553 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:27.553 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.545802+0000", 2024-09-10T18:37:27.554 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00061090299999999999 2024-09-10T18:37:27.554 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:27.554 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:27.554 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:27.554 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.554 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.554 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:112059 10.6 10:63bc5028:::benchmark_data_smithi028_161947_object112058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-09-10T18:37:27.554 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:37:27.542356+0000", 2024-09-10T18:37:27.554 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0039141180000000003, 2024-09-10T18:37:27.554 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040328650000000001, 2024-09-10T18:37:27.554 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:27.555 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:37:27.555 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:27.555 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:37:27.555 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:37:27.555 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 112059 2024-09-10T18:37:27.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.555 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:27.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:27.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.542356+0000", 2024-09-10T18:37:27.555 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:27.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:27.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.542356+0000", 2024-09-10T18:37:27.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:27.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:27.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.542353+0000", 2024-09-10T18:37:27.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:37:27.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:27.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.542394+0000", 2024-09-10T18:37:27.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0519000000000003e-05 2024-09-10T18:37:27.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:27.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.542404+0000", 2024-09-10T18:37:27.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0849999999999999e-05 2024-09-10T18:37:27.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:27.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.542410+0000", 2024-09-10T18:37:27.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4099999999999999e-06 2024-09-10T18:37:27.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:27.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.542466+0000", 2024-09-10T18:37:27.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6096e-05 2024-09-10T18:37:27.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:27.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.542574+0000", 2024-09-10T18:37:27.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107638 2024-09-10T18:37:27.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.559 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:37:27.559 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.542701+0000", 2024-09-10T18:37:27.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012747499999999999 2024-09-10T18:37:27.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.559 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:27.559 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.545360+0000", 2024-09-10T18:37:27.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0026590110000000002 2024-09-10T18:37:27.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:27.560 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.545980+0000", 2024-09-10T18:37:27.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00061976100000000003 2024-09-10T18:37:27.560 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:27.560 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:27.560 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:27.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.560 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.15693.0:112058 10.a e338/314 10:51bc3fe1:::benchmark_data_smithi028_161947_object112057:head v 338'6989, mlcod=338'6987)", 2024-09-10T18:37:27.560 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:37:27.545171+0000", 2024-09-10T18:37:27.561 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0010984759999999999, 2024-09-10T18:37:27.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001291941, 2024-09-10T18:37:27.561 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:27.561 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:37:27.561 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:27.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:27.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.545171+0000", 2024-09-10T18:37:27.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:27.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:27.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.545171+0000", 2024-09-10T18:37:27.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:27.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:27.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.545170+0000", 2024-09-10T18:37:27.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:37:27.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:27.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.545196+0000", 2024-09-10T18:37:27.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6441e-05 2024-09-10T18:37:27.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:27.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.545207+0000", 2024-09-10T18:37:27.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1116e-05 2024-09-10T18:37:27.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:27.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.545213+0000", 2024-09-10T18:37:27.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2029999999999997e-06 2024-09-10T18:37:27.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:27.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.545384+0000", 2024-09-10T18:37:27.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000170896 2024-09-10T18:37:27.564 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:27.564 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:27.564 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:27.564 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:27.545402+0000", 2024-09-10T18:37:27.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7812000000000002e-05 2024-09-10T18:37:27.564 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:27.564 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:27.564 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:27.564 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:27.564 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-09-10T18:37:27.564 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 3 2024-09-10T18:37:27.564 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:37:27.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:27.650 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:27.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:27.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:27.752 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:27.797 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:27.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:27.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:27.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:27.934 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:27.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:27.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:28.028 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 69 16 112713 112697 102.061 82.9375 0.00801451 0.00979347 2024-09-10T18:37:28.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:28.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:28.071 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:28.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:28.172 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:28.205 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:28.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:28.307 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:28.343 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:28.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:28.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:28.441 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:28 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:37:28.441 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:28 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:37:28.441 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:28 smithi204 ceph-mon[32357]: pgmap v724: 178 pgs: 178 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-10T18:37:28.441 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:28 smithi204 ceph-mon[32357]: osdmap e338: 8 total, 8 up, 8 in 2024-09-10T18:37:28.441 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:28 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:37:28.441 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:28 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:37:28.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:28.479 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-10T18:37:28.484 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:28.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:28.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:28 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:37:28.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:28 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:37:28.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:28 smithi028 ceph-mon[39220]: pgmap v724: 178 pgs: 178 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-10T18:37:28.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:28 smithi028 ceph-mon[39220]: osdmap e338: 8 total, 8 up, 8 in 2024-09-10T18:37:28.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:28 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:37:28.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:28 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:37:28.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:28 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:37:28.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:28 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:37:28.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:28 smithi028 ceph-mon[29291]: pgmap v724: 178 pgs: 178 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-10T18:37:28.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:28 smithi028 ceph-mon[29291]: osdmap e338: 8 total, 8 up, 8 in 2024-09-10T18:37:28.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:28 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:37:28.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:28 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:37:28.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:28.627 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:28.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:28.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:28.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:28.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:28.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:28.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:28.881 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:37:28.905 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:28.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:28.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:29.007 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:29.028 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 70 16 114015 113999 101.766 81.375 0.00506339 0.00982165 2024-09-10T18:37:29.039 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:29.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:29.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:29.142 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:29.175 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:29.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:29.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:29.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:29.318 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:29.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:29.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:29.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:29 smithi204 ceph-mon[32357]: osdmap e339: 8 total, 8 up, 8 in 2024-09-10T18:37:29.420 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:29.460 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:29.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:29.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:29.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:29 smithi028 ceph-mon[39220]: osdmap e339: 8 total, 8 up, 8 in 2024-09-10T18:37:29.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:29 smithi028 ceph-mon[29291]: osdmap e339: 8 total, 8 up, 8 in 2024-09-10T18:37:29.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:29.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:29.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:29.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:37:29.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:29.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:29.755 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:29.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:29.856 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:29.891 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:29.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:29.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:30.028 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 71 16 115256 115240 101.425 77.5625 0.0157468 0.00985541 2024-09-10T18:37:30.037 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:30.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:30.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:30.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:37:30.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:30.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:30.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:30.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:30.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:30.373 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:30.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:30.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:30.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:30 smithi028 ceph-mon[39220]: pgmap v727: 178 pgs: 178 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 862 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-10T18:37:30.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:30 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:37:30.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:30 smithi028 ceph-mon[29291]: pgmap v727: 178 pgs: 178 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 862 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-10T18:37:30.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:30 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:37:30.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:30.518 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:30.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:30.569 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:30 smithi204 ceph-mon[32357]: pgmap v727: 178 pgs: 178 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 862 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-10T18:37:30.569 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:30 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:37:30.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:30.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:30.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:30.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:30.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:30.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:30.811 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:30.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:30.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:30.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:30.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:31.029 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 72 16 116328 116312 100.947 67 0.017269 0.00990137 2024-09-10T18:37:31.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:31.056 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:37:31.056 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:37:31.056 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-10T18:37:31.056 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-10T18:37:31.056 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:37:31.090 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:31.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:31.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:31.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:31.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:31.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:31.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:31.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:31.379 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:31.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:31.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:31.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:37:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:37:31] "GET /metrics HTTP/1.1" 200 224655 "" "Prometheus/2.43.0" 2024-09-10T18:37:31.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:31 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:37:31.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:31 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:37:31.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:31 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:37:31.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:31 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:37:31.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:31 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:37:31.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:31 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:37:31.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:31.531 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:31.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:31.599 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:37:31.600 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-10T18:37:31.600 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:37:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:31 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:37:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:31 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:37:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:31 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:37:31.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:31.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:31.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:31.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:31.816 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-10T18:37:31.825 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:31.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:31.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:31.960 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:31.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:31.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:32.029 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 73 16 117583 117567 100.638 78.4375 0.00633808 0.00993185 2024-09-10T18:37:32.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:32.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:32.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:32.207 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:32.221 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:37:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:32.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:32.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:32.280 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:37:32.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:32.401 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:32 smithi204 ceph-mon[32357]: pgmap v728: 178 pgs: 178 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-09-10T18:37:32.402 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:37:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:37:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:37:32.405 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:32.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:32.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:32 smithi028 ceph-mon[29291]: pgmap v728: 178 pgs: 178 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-09-10T18:37:32.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:32 smithi028 ceph-mon[39220]: pgmap v728: 178 pgs: 178 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-09-10T18:37:32.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:32.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:32.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:32.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:32.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:32.688 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:32.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:32.789 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:32.826 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:32.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:32.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:32.929 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:32.969 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:32.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:32.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:33.029 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 74 16 118833 118817 100.334 78.125 0.0155881 0.00996237 2024-09-10T18:37:33.070 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:33.106 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:33.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:33.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:33.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:37:33.251 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:33.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:33.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:33.353 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:33.388 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:33.389 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:33 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1809393265' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:37:33.389 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:33 smithi028 ceph-mon[39220]: from='client.35684 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:37:33.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:33 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1809393265' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:37:33.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:33 smithi028 ceph-mon[29291]: from='client.35684 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:37:33.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:33.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:33.491 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:33.528 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:33.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:33.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:33 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1809393265' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:37:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:33 smithi204 ceph-mon[32357]: from='client.35684 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:37:33.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:33.668 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:33.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:33.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:33.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:33.803 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:33.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:33.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:33.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:33.942 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:33.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:33.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:34.029 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 75 16 120144 120128 100.089 81.9375 0.0292184 0.00998464 2024-09-10T18:37:34.043 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:34.084 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:34.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:34.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:34.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:34.209 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:34.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:34.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:34.336 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:34.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:34.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:34.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:34.464 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:34.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:34.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:34 smithi028 ceph-mon[39220]: pgmap v729: 178 pgs: 178 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 706 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-10T18:37:34.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:34 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:37:34.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:34 smithi028 ceph-mon[39220]: from='client.35684 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:37:34.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:34 smithi028 ceph-mon[39220]: osdmap e340: 8 total, 8 up, 8 in 2024-09-10T18:37:34.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:34 smithi028 ceph-mon[29291]: pgmap v729: 178 pgs: 178 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 706 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-10T18:37:34.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:34 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:37:34.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:34 smithi028 ceph-mon[29291]: from='client.35684 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:37:34.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:34 smithi028 ceph-mon[29291]: osdmap e340: 8 total, 8 up, 8 in 2024-09-10T18:37:34.558 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:34 smithi204 ceph-mon[32357]: pgmap v729: 178 pgs: 178 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 706 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-10T18:37:34.558 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:34 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:37:34.558 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:34 smithi204 ceph-mon[32357]: from='client.35684 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:37:34.558 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:34 smithi204 ceph-mon[32357]: osdmap e340: 8 total, 8 up, 8 in 2024-09-10T18:37:34.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:34.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:34.606 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:34.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:34.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:34.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:34.735 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:34.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:34.837 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:34.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:37:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:37:34.873 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:34.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:34.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:34.975 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:35.017 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:35.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:35.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:35.029 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 76 16 121147 121131 99.5964 62.6875 0.0073513 0.0100361 2024-09-10T18:37:35.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:35.152 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:35.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:35.254 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:35.290 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:35.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:35.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:35.392 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:35.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:35.438 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:35.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:35.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:35 smithi204 ceph-mon[32357]: 4.5 scrub starts 2024-09-10T18:37:35.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:35 smithi204 ceph-mon[32357]: 4.5 scrub ok 2024-09-10T18:37:35.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:35 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:37:35.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:35 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:37:35.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:35 smithi204 ceph-mon[32357]: osdmap e341: 8 total, 8 up, 8 in 2024-09-10T18:37:35.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:35 smithi028 ceph-mon[39220]: 4.5 scrub starts 2024-09-10T18:37:35.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:35 smithi028 ceph-mon[39220]: 4.5 scrub ok 2024-09-10T18:37:35.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:35 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:37:35.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:35 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:37:35.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:35 smithi028 ceph-mon[39220]: osdmap e341: 8 total, 8 up, 8 in 2024-09-10T18:37:35.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:35 smithi028 ceph-mon[29291]: 4.5 scrub starts 2024-09-10T18:37:35.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:35 smithi028 ceph-mon[29291]: 4.5 scrub ok 2024-09-10T18:37:35.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:35 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:37:35.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:35 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:37:35.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:35 smithi028 ceph-mon[29291]: osdmap e341: 8 total, 8 up, 8 in 2024-09-10T18:37:35.539 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:35.577 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:35.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:35.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:35.679 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:35.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:35.701 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:35.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:35.802 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:35.837 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:35.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:35.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:35.912 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:37:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-10T18:37:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-10T18:37:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:37:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:185346 9.6 9:64e8f21e:::benchmark_data_smithi028_135703_object50755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-10T18:37:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:24.132408+0000", 2024-09-10T18:37:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 191.35063176200001, 2024-09-10T18:37:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15451896500000001, 2024-09-10T18:37:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 185346 2024-09-10T18:37:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.132408+0000", 2024-09-10T18:37:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.132408+0000", 2024-09-10T18:37:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.132406+0000", 2024-09-10T18:37:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.132412+0000", 2024-09-10T18:37:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7470000000000002e-06 2024-09-10T18:37:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.132421+0000", 2024-09-10T18:37:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5200000000000003e-06 2024-09-10T18:37:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.132427+0000", 2024-09-10T18:37:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.553e-06 2024-09-10T18:37:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.182728+0000", 2024-09-10T18:37:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.050301332999999997 2024-09-10T18:37:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.182849+0000", 2024-09-10T18:37:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012109599999999999 2024-09-10T18:37:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.182998+0000", 2024-09-10T18:37:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014882 2024-09-10T18:37:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.184878+0000", 2024-09-10T18:37:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0018793779999999999 2024-09-10T18:37:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.229602+0000", 2024-09-10T18:37:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.044724201999999998 2024-09-10T18:37:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.286852+0000", 2024-09-10T18:37:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.057250019999999999 2024-09-10T18:37:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.286888+0000", 2024-09-10T18:37:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6477000000000003e-05 2024-09-10T18:37:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.286927+0000", 2024-09-10T18:37:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8139000000000002e-05 2024-09-10T18:37:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:185350 9.6 9:6efd6568:::benchmark_data_smithi028_135703_object50759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-10T18:37:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:24.135751+0000", 2024-09-10T18:37:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 191.34728828999999, 2024-09-10T18:37:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15126341500000001, 2024-09-10T18:37:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 185350 2024-09-10T18:37:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.135751+0000", 2024-09-10T18:37:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.135751+0000", 2024-09-10T18:37:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.135749+0000", 2024-09-10T18:37:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:37:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.135756+0000", 2024-09-10T18:37:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3900000000000004e-06 2024-09-10T18:37:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.135770+0000", 2024-09-10T18:37:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3828999999999999e-05 2024-09-10T18:37:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.135776+0000", 2024-09-10T18:37:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5639999999999996e-06 2024-09-10T18:37:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.185607+0000", 2024-09-10T18:37:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.049830931000000002 2024-09-10T18:37:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.185762+0000", 2024-09-10T18:37:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000155179 2024-09-10T18:37:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.185870+0000", 2024-09-10T18:37:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108096 2024-09-10T18:37:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.193110+0000", 2024-09-10T18:37:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0072403709999999998 2024-09-10T18:37:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.229642+0000", 2024-09-10T18:37:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.036531173 2024-09-10T18:37:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.286963+0000", 2024-09-10T18:37:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.05732172 2024-09-10T18:37:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.286992+0000", 2024-09-10T18:37:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9011999999999999e-05 2024-09-10T18:37:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:24.287014+0000", 2024-09-10T18:37:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2087e-05 2024-09-10T18:37:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:186264 9.6 9:6f7d5358:::benchmark_data_smithi028_135703_object51673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-10T18:37:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:25.208112+0000", 2024-09-10T18:37:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 190.27492754100001, 2024-09-10T18:37:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.253174285, 2024-09-10T18:37:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 186264 2024-09-10T18:37:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.208112+0000", 2024-09-10T18:37:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.208112+0000", 2024-09-10T18:37:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.208109+0000", 2024-09-10T18:37:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:37:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.208116+0000", 2024-09-10T18:37:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0500000000000003e-06 2024-09-10T18:37:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.208125+0000", 2024-09-10T18:37:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8529999999999998e-06 2024-09-10T18:37:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.208130+0000", 2024-09-10T18:37:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6079999999999998e-06 2024-09-10T18:37:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.249795+0000", 2024-09-10T18:37:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.041664999000000001 2024-09-10T18:37:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.249920+0000", 2024-09-10T18:37:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012498000000000001 2024-09-10T18:37:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.250032+0000", 2024-09-10T18:37:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112254 2024-09-10T18:37:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.254003+0000", 2024-09-10T18:37:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003971306 2024-09-10T18:37:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.254041+0000", 2024-09-10T18:37:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7660000000000002e-05 2024-09-10T18:37:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.461208+0000", 2024-09-10T18:37:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.207167453 2024-09-10T18:37:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.461251+0000", 2024-09-10T18:37:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2870999999999997e-05 2024-09-10T18:37:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.461286+0000", 2024-09-10T18:37:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.472e-05 2024-09-10T18:37:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:186290 9.6 9:6009b567:::benchmark_data_smithi028_135703_object51699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-10T18:37:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:25.238775+0000", 2024-09-10T18:37:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 190.24426464999999, 2024-09-10T18:37:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.22295261899999999, 2024-09-10T18:37:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 186290 2024-09-10T18:37:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.238775+0000", 2024-09-10T18:37:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.238775+0000", 2024-09-10T18:37:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.238773+0000", 2024-09-10T18:37:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:37:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.238780+0000", 2024-09-10T18:37:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6359999999999997e-06 2024-09-10T18:37:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.238789+0000", 2024-09-10T18:37:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0739999999999999e-06 2024-09-10T18:37:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.238793+0000", 2024-09-10T18:37:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0439999999999997e-06 2024-09-10T18:37:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.252139+0000", 2024-09-10T18:37:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01334597 2024-09-10T18:37:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.252272+0000", 2024-09-10T18:37:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000133289 2024-09-10T18:37:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.252424+0000", 2024-09-10T18:37:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015193199999999999 2024-09-10T18:37:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.254068+0000", 2024-09-10T18:37:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016438119999999999 2024-09-10T18:37:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.254073+0000", 2024-09-10T18:37:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.203e-06 2024-09-10T18:37:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.461337+0000", 2024-09-10T18:37:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.207264477 2024-09-10T18:37:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.461373+0000", 2024-09-10T18:37:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5967000000000003e-05 2024-09-10T18:37:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.461727+0000", 2024-09-10T18:37:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00035395100000000003 2024-09-10T18:37:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:186582 9.6 9:602ee978:::benchmark_data_smithi028_135703_object51991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-10T18:37:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:25.728385+0000", 2024-09-10T18:37:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 189.754654757, 2024-09-10T18:37:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.27129171000000002, 2024-09-10T18:37:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 186582 2024-09-10T18:37:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.728385+0000", 2024-09-10T18:37:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.728385+0000", 2024-09-10T18:37:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.728382+0000", 2024-09-10T18:37:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:37:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.728390+0000", 2024-09-10T18:37:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9929999999999995e-06 2024-09-10T18:37:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.728404+0000", 2024-09-10T18:37:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3895e-05 2024-09-10T18:37:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.728410+0000", 2024-09-10T18:37:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5230000000000003e-06 2024-09-10T18:37:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.752791+0000", 2024-09-10T18:37:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024381679 2024-09-10T18:37:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.752881+0000", 2024-09-10T18:37:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9269000000000002e-05 2024-09-10T18:37:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.752968+0000", 2024-09-10T18:37:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7183999999999997e-05 2024-09-10T18:37:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.754297+0000", 2024-09-10T18:37:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013290050000000001 2024-09-10T18:37:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.857961+0000", 2024-09-10T18:37:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.103663924 2024-09-10T18:37:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.999587+0000", 2024-09-10T18:37:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.14162615000000001 2024-09-10T18:37:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.999643+0000", 2024-09-10T18:37:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6060999999999997e-05 2024-09-10T18:37:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.999676+0000", 2024-09-10T18:37:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3176999999999998e-05 2024-09-10T18:37:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:186597 9.6 9:65117448:::benchmark_data_smithi028_135703_object52006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-10T18:37:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:25.819854+0000", 2024-09-10T18:37:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 189.66318554899999, 2024-09-10T18:37:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.179976252, 2024-09-10T18:37:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 186597 2024-09-10T18:37:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.819854+0000", 2024-09-10T18:37:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.819854+0000", 2024-09-10T18:37:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.819852+0000", 2024-09-10T18:37:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:37:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.819859+0000", 2024-09-10T18:37:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7090000000000002e-06 2024-09-10T18:37:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.819882+0000", 2024-09-10T18:37:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2294e-05 2024-09-10T18:37:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.819888+0000", 2024-09-10T18:37:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9769999999999997e-06 2024-09-10T18:37:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.820019+0000", 2024-09-10T18:37:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013137000000000001 2024-09-10T18:37:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.820178+0000", 2024-09-10T18:37:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015874300000000001 2024-09-10T18:37:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.820339+0000", 2024-09-10T18:37:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000160848 2024-09-10T18:37:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.825971+0000", 2024-09-10T18:37:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056326140000000002 2024-09-10T18:37:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.859660+0000", 2024-09-10T18:37:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.033688747999999998 2024-09-10T18:37:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.999753+0000", 2024-09-10T18:37:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.14009269199999999 2024-09-10T18:37:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.999796+0000", 2024-09-10T18:37:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3547000000000002e-05 2024-09-10T18:37:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.999830+0000", 2024-09-10T18:37:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3797999999999997e-05 2024-09-10T18:37:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:187302 9.6 9:69518a1d:::benchmark_data_smithi028_135703_object52711:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-10T18:37:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:26.625947+0000", 2024-09-10T18:37:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 188.85709244700001, 2024-09-10T18:37:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.163220799, 2024-09-10T18:37:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 187302 2024-09-10T18:37:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.625947+0000", 2024-09-10T18:37:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.625947+0000", 2024-09-10T18:37:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.625945+0000", 2024-09-10T18:37:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:37:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.625956+0000", 2024-09-10T18:37:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0795000000000001e-05 2024-09-10T18:37:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.625968+0000", 2024-09-10T18:37:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2883e-05 2024-09-10T18:37:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.625975+0000", 2024-09-10T18:37:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1120000000000001e-06 2024-09-10T18:37:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.642501+0000", 2024-09-10T18:37:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016526216999999999 2024-09-10T18:37:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.642632+0000", 2024-09-10T18:37:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013165299999999999 2024-09-10T18:37:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.642763+0000", 2024-09-10T18:37:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013038700000000001 2024-09-10T18:37:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.672423+0000", 2024-09-10T18:37:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029660618 2024-09-10T18:37:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.672665+0000", 2024-09-10T18:37:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00024113800000000001 2024-09-10T18:37:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.789074+0000", 2024-09-10T18:37:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.116409602 2024-09-10T18:37:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.789123+0000", 2024-09-10T18:37:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8550000000000001e-05 2024-09-10T18:37:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.789168+0000", 2024-09-10T18:37:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.498e-05 2024-09-10T18:37:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:187337 9.6 9:6edff631:::benchmark_data_smithi028_135703_object52746:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-10T18:37:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:26.657353+0000", 2024-09-10T18:37:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 188.82568645800001, 2024-09-10T18:37:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15430215899999999, 2024-09-10T18:37:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 187337 2024-09-10T18:37:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.657353+0000", 2024-09-10T18:37:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.657353+0000", 2024-09-10T18:37:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.657351+0000", 2024-09-10T18:37:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:37:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.657358+0000", 2024-09-10T18:37:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0809999999999996e-06 2024-09-10T18:37:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.657367+0000", 2024-09-10T18:37:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1390000000000001e-06 2024-09-10T18:37:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.657372+0000", 2024-09-10T18:37:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4750000000000001e-06 2024-09-10T18:37:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.672895+0000", 2024-09-10T18:37:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015522785000000001 2024-09-10T18:37:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.673019+0000", 2024-09-10T18:37:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012342399999999999 2024-09-10T18:37:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.673137+0000", 2024-09-10T18:37:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011877199999999999 2024-09-10T18:37:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.789209+0000", 2024-09-10T18:37:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.11607144599999999 2024-09-10T18:37:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.789240+0000", 2024-09-10T18:37:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0621999999999999e-05 2024-09-10T18:37:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.811571+0000", 2024-09-10T18:37:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022331806999999999 2024-09-10T18:37:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.811616+0000", 2024-09-10T18:37:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5040000000000002e-05 2024-09-10T18:37:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:26.811655+0000", 2024-09-10T18:37:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8662999999999999e-05 2024-09-10T18:37:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:190734 9.6 9:617d4bed:::benchmark_data_smithi028_135703_object56143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-10T18:37:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:30.579126+0000", 2024-09-10T18:37:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 184.903913405, 2024-09-10T18:37:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.26543867999999998, 2024-09-10T18:37:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 190734 2024-09-10T18:37:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.579126+0000", 2024-09-10T18:37:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.579126+0000", 2024-09-10T18:37:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.579124+0000", 2024-09-10T18:37:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:37:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.579132+0000", 2024-09-10T18:37:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5660000000000001e-06 2024-09-10T18:37:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.579146+0000", 2024-09-10T18:37:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4742e-05 2024-09-10T18:37:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.579152+0000", 2024-09-10T18:37:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.789e-06 2024-09-10T18:37:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.612147+0000", 2024-09-10T18:37:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.032994891999999998 2024-09-10T18:37:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.612270+0000", 2024-09-10T18:37:35.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012282499999999999 2024-09-10T18:37:35.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:35.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.612412+0000", 2024-09-10T18:37:35.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000142192 2024-09-10T18:37:35.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:35.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.617099+0000", 2024-09-10T18:37:35.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0046871730000000002 2024-09-10T18:37:35.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.641697+0000", 2024-09-10T18:37:35.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024598315999999999 2024-09-10T18:37:35.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.844486+0000", 2024-09-10T18:37:35.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.202788783 2024-09-10T18:37:35.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:35.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.844531+0000", 2024-09-10T18:37:35.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4518000000000003e-05 2024-09-10T18:37:35.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:35.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.844565+0000", 2024-09-10T18:37:35.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3793e-05 2024-09-10T18:37:35.973 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.973 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:35.973 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.973 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:190774 9.6 9:65be00cb:::benchmark_data_smithi028_135703_object56183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-10T18:37:35.974 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:30.612834+0000", 2024-09-10T18:37:35.974 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 184.87020572500001, 2024-09-10T18:37:35.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.23186309199999999, 2024-09-10T18:37:35.974 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:35.974 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:35.974 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:35.974 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:35.974 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:35.974 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 190774 2024-09-10T18:37:35.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.974 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:35.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:35.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.612834+0000", 2024-09-10T18:37:35.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:35.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.612834+0000", 2024-09-10T18:37:35.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.976 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:35.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.612832+0000", 2024-09-10T18:37:35.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:37:35.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.977 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:35.977 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.612839+0000", 2024-09-10T18:37:35.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4699999999999996e-06 2024-09-10T18:37:35.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:35.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.612850+0000", 2024-09-10T18:37:35.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0329999999999999e-05 2024-09-10T18:37:35.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:35.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.612855+0000", 2024-09-10T18:37:35.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5840000000000002e-06 2024-09-10T18:37:35.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:35.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.614749+0000", 2024-09-10T18:37:35.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0018937979999999999 2024-09-10T18:37:35.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:35.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.614855+0000", 2024-09-10T18:37:35.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010596500000000001 2024-09-10T18:37:35.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:35.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.614989+0000", 2024-09-10T18:37:35.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000134449 2024-09-10T18:37:35.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:35.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.617246+0000", 2024-09-10T18:37:35.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022568340000000001 2024-09-10T18:37:35.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.641716+0000", 2024-09-10T18:37:35.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024469827999999999 2024-09-10T18:37:35.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.844619+0000", 2024-09-10T18:37:35.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.20290309400000001 2024-09-10T18:37:35.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:35.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.844662+0000", 2024-09-10T18:37:35.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2725999999999997e-05 2024-09-10T18:37:35.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:35.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.844697+0000", 2024-09-10T18:37:35.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4705000000000001e-05 2024-09-10T18:37:35.984 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.984 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:35.984 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.984 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:190777 9.6 9:6832adb2:::benchmark_data_smithi028_135703_object56186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-10T18:37:35.984 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:30.617938+0000", 2024-09-10T18:37:35.985 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 184.86510092500001, 2024-09-10T18:37:35.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.22686962699999999, 2024-09-10T18:37:35.985 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:35.985 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:35.985 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:35.985 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:35.985 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:35.985 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 190777 2024-09-10T18:37:35.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.985 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:35.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:35.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.617938+0000", 2024-09-10T18:37:35.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:35.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.617938+0000", 2024-09-10T18:37:35.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:35.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.617937+0000", 2024-09-10T18:37:35.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:37:35.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:35.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.617943+0000", 2024-09-10T18:37:35.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8290000000000002e-06 2024-09-10T18:37:35.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:35.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.617955+0000", 2024-09-10T18:37:35.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.185e-05 2024-09-10T18:37:35.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:35.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.617961+0000", 2024-09-10T18:37:35.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7250000000000002e-06 2024-09-10T18:37:35.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:35.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.617992+0000", 2024-09-10T18:37:35.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0487000000000001e-05 2024-09-10T18:37:35.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:35.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.618074+0000", 2024-09-10T18:37:35.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2548999999999999e-05 2024-09-10T18:37:35.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:35.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.618203+0000", 2024-09-10T18:37:35.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012857600000000001 2024-09-10T18:37:35.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:35.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.621725+0000", 2024-09-10T18:37:35.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003522533 2024-09-10T18:37:35.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.641741+0000", 2024-09-10T18:37:35.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020015448000000002 2024-09-10T18:37:35.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.844740+0000", 2024-09-10T18:37:35.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.20299948300000001 2024-09-10T18:37:35.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:35.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.844777+0000", 2024-09-10T18:37:35.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6757000000000001e-05 2024-09-10T18:37:35.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:35.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.844808+0000", 2024-09-10T18:37:35.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1143000000000003e-05 2024-09-10T18:37:35.990 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.991 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:35.991 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.992 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:190803 9.6 9:675dc30e:::benchmark_data_smithi028_135703_object56212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-10T18:37:35.992 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:30.662935+0000", 2024-09-10T18:37:35.992 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 184.82010419400001, 2024-09-10T18:37:35.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.18202421299999999, 2024-09-10T18:37:35.992 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:35.992 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:35.992 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:35.992 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:35.992 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:35.993 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 190803 2024-09-10T18:37:35.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.993 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:35.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:35.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.662935+0000", 2024-09-10T18:37:35.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:35.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.662935+0000", 2024-09-10T18:37:35.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:35.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:35.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.662933+0000", 2024-09-10T18:37:35.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:37:35.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:35.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.662941+0000", 2024-09-10T18:37:35.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2409999999999998e-06 2024-09-10T18:37:35.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:35.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.662950+0000", 2024-09-10T18:37:35.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4630000000000007e-06 2024-09-10T18:37:35.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:35.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.662954+0000", 2024-09-10T18:37:35.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.245e-06 2024-09-10T18:37:35.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:35.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.662989+0000", 2024-09-10T18:37:35.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4869000000000001e-05 2024-09-10T18:37:35.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:35.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.663070+0000", 2024-09-10T18:37:35.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0638999999999998e-05 2024-09-10T18:37:35.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:35.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.663160+0000", 2024-09-10T18:37:35.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9788000000000002e-05 2024-09-10T18:37:35.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.665563+0000", 2024-09-10T18:37:35.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002403203 2024-09-10T18:37:35.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:35.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.665666+0000", 2024-09-10T18:37:35.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103559 2024-09-10T18:37:35.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:35.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.844860+0000", 2024-09-10T18:37:35.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.179193252 2024-09-10T18:37:35.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:35.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.844920+0000", 2024-09-10T18:37:35.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9978000000000001e-05 2024-09-10T18:37:35.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:35.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.844959+0000", 2024-09-10T18:37:35.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9746999999999997e-05 2024-09-10T18:37:35.998 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.998 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:35.998 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:35.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.998 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:190808 9.6 9:6b51b553:::benchmark_data_smithi028_135703_object56217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-10T18:37:35.998 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:30.679614+0000", 2024-09-10T18:37:35.999 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 184.803425395, 2024-09-10T18:37:35.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16546308300000001, 2024-09-10T18:37:35.999 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:35.999 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:35.999 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:35.999 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:35.999 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:35.999 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 190808 2024-09-10T18:37:35.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:35.999 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:35.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:35.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:35.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.679614+0000", 2024-09-10T18:37:35.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:36.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:36.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.679614+0000", 2024-09-10T18:37:36.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:36.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:36.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.679612+0000", 2024-09-10T18:37:36.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:37:36.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:36.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.679621+0000", 2024-09-10T18:37:36.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2979999999999997e-06 2024-09-10T18:37:36.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:36.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.679633+0000", 2024-09-10T18:37:36.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.164e-05 2024-09-10T18:37:36.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:36.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.679639+0000", 2024-09-10T18:37:36.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3199999999999996e-06 2024-09-10T18:37:36.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:36.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.679678+0000", 2024-09-10T18:37:36.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8528000000000001e-05 2024-09-10T18:37:36.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:36.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.679763+0000", 2024-09-10T18:37:36.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5057999999999998e-05 2024-09-10T18:37:36.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:36.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.679838+0000", 2024-09-10T18:37:36.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5574000000000005e-05 2024-09-10T18:37:36.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:36.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.681827+0000", 2024-09-10T18:37:36.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019890160000000001 2024-09-10T18:37:36.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:36.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.681885+0000", 2024-09-10T18:37:36.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7834000000000002e-05 2024-09-10T18:37:36.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:36.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.845005+0000", 2024-09-10T18:37:36.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16312001800000001 2024-09-10T18:37:36.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:36.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.845046+0000", 2024-09-10T18:37:36.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1257999999999997e-05 2024-09-10T18:37:36.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:36.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:30.845077+0000", 2024-09-10T18:37:36.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0629000000000001e-05 2024-09-10T18:37:36.005 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:36.005 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:36.005 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:36.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.005 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:191058 9.6 9:6d984771:::benchmark_data_smithi028_135703_object56467:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-10T18:37:36.005 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:31.084634+0000", 2024-09-10T18:37:36.005 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 184.39840560799999, 2024-09-10T18:37:36.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.164485621, 2024-09-10T18:37:36.006 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:36.006 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:36.006 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:36.006 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:36.006 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:36.006 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 191058 2024-09-10T18:37:36.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.006 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:36.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:36.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.084634+0000", 2024-09-10T18:37:36.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:36.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:36.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.084634+0000", 2024-09-10T18:37:36.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:36.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:36.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.084632+0000", 2024-09-10T18:37:36.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:36.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:36.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.084641+0000", 2024-09-10T18:37:36.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1829999999999994e-06 2024-09-10T18:37:36.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:36.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.084662+0000", 2024-09-10T18:37:36.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1084e-05 2024-09-10T18:37:36.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:36.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.084667+0000", 2024-09-10T18:37:36.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8949999999999997e-06 2024-09-10T18:37:36.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:36.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.124005+0000", 2024-09-10T18:37:36.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.039338753999999997 2024-09-10T18:37:36.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:36.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.124130+0000", 2024-09-10T18:37:36.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012451899999999999 2024-09-10T18:37:36.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:36.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.124261+0000", 2024-09-10T18:37:36.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013090199999999999 2024-09-10T18:37:36.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:36.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.127696+0000", 2024-09-10T18:37:36.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034355330000000002 2024-09-10T18:37:36.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:36.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.127922+0000", 2024-09-10T18:37:36.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022599 2024-09-10T18:37:36.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:36.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.248972+0000", 2024-09-10T18:37:36.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.121049912 2024-09-10T18:37:36.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:36.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.249018+0000", 2024-09-10T18:37:36.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6245999999999997e-05 2024-09-10T18:37:36.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:36.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.249119+0000", 2024-09-10T18:37:36.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010095200000000001 2024-09-10T18:37:36.011 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:36.011 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:36.011 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:36.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.011 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:191062 9.6 9:61d32bf3:::benchmark_data_smithi028_135703_object56471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-10T18:37:36.011 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:31.085407+0000", 2024-09-10T18:37:36.012 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 184.397632579, 2024-09-10T18:37:36.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16385010799999999, 2024-09-10T18:37:36.012 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:36.012 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:36.012 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:36.012 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:36.012 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:36.012 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 191062 2024-09-10T18:37:36.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.012 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:36.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:36.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.085407+0000", 2024-09-10T18:37:36.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:36.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:36.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.085407+0000", 2024-09-10T18:37:36.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:36.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:36.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.085406+0000", 2024-09-10T18:37:36.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:37:36.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:36.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.085410+0000", 2024-09-10T18:37:36.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3529999999999997e-06 2024-09-10T18:37:36.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:36.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.085419+0000", 2024-09-10T18:37:36.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3459999999999994e-06 2024-09-10T18:37:36.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:36.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.085422+0000", 2024-09-10T18:37:36.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8e-06 2024-09-10T18:37:36.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:36.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.126156+0000", 2024-09-10T18:37:36.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.040733639000000002 2024-09-10T18:37:36.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:36.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.126265+0000", 2024-09-10T18:37:36.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108849 2024-09-10T18:37:36.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:36.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.126401+0000", 2024-09-10T18:37:36.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013624000000000001 2024-09-10T18:37:36.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:36.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.127822+0000", 2024-09-10T18:37:36.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014206659999999999 2024-09-10T18:37:36.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:36.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.130432+0000", 2024-09-10T18:37:36.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002610556 2024-09-10T18:37:36.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:36.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.249166+0000", 2024-09-10T18:37:36.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.118733848 2024-09-10T18:37:36.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:36.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.249209+0000", 2024-09-10T18:37:36.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2972999999999997e-05 2024-09-10T18:37:36.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:36.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.249257+0000", 2024-09-10T18:37:36.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7766000000000003e-05 2024-09-10T18:37:36.017 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:36.018 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:36.018 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:36.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.018 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:191071 9.6 9:64e4b1f1:::benchmark_data_smithi028_135703_object56480:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-10T18:37:36.018 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:31.095387+0000", 2024-09-10T18:37:36.018 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 184.387652242, 2024-09-10T18:37:36.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.175088514, 2024-09-10T18:37:36.018 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:36.018 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:36.018 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:36.018 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:36.018 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:36.018 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 191071 2024-09-10T18:37:36.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.019 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:36.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:36.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.095387+0000", 2024-09-10T18:37:36.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:36.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:36.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.095387+0000", 2024-09-10T18:37:36.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:36.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:36.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.095385+0000", 2024-09-10T18:37:36.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:37:36.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:36.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.095391+0000", 2024-09-10T18:37:36.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3489999999999999e-06 2024-09-10T18:37:36.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:36.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.095399+0000", 2024-09-10T18:37:36.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5029999999999997e-06 2024-09-10T18:37:36.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:36.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.095404+0000", 2024-09-10T18:37:36.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7480000000000001e-06 2024-09-10T18:37:36.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:36.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.127949+0000", 2024-09-10T18:37:36.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.032545536999999999 2024-09-10T18:37:36.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:36.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.128016+0000", 2024-09-10T18:37:36.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6808000000000001e-05 2024-09-10T18:37:36.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:36.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.128091+0000", 2024-09-10T18:37:36.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5029000000000003e-05 2024-09-10T18:37:36.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:36.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.130487+0000", 2024-09-10T18:37:36.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023956110000000002 2024-09-10T18:37:36.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:36.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.130550+0000", 2024-09-10T18:37:36.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3565000000000006e-05 2024-09-10T18:37:36.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:36.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.270372+0000", 2024-09-10T18:37:36.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13982186599999999 2024-09-10T18:37:36.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:36.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.270426+0000", 2024-09-10T18:37:36.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4035e-05 2024-09-10T18:37:36.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:36.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.270476+0000", 2024-09-10T18:37:36.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9206999999999998e-05 2024-09-10T18:37:36.024 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:36.024 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:36.024 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:36.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.024 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:191097 9.6 9:6be354ed:::benchmark_data_smithi028_135703_object56506:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-10T18:37:36.024 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:31.120259+0000", 2024-09-10T18:37:36.024 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 184.36278069799999, 2024-09-10T18:37:36.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15037183400000001, 2024-09-10T18:37:36.024 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:36.024 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:36.024 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:36.024 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:36.025 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:36.025 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 191097 2024-09-10T18:37:36.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.025 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:36.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:36.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.120259+0000", 2024-09-10T18:37:36.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:36.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:36.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.120259+0000", 2024-09-10T18:37:36.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:36.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.026 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:36.026 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.120257+0000", 2024-09-10T18:37:36.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:37:36.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.026 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:36.026 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.120263+0000", 2024-09-10T18:37:36.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5150000000000003e-06 2024-09-10T18:37:36.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:36.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.120273+0000", 2024-09-10T18:37:36.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.007e-05 2024-09-10T18:37:36.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:36.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.120279+0000", 2024-09-10T18:37:36.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3199999999999999e-06 2024-09-10T18:37:36.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:36.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.129210+0000", 2024-09-10T18:37:36.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0089309929999999999 2024-09-10T18:37:36.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:36.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.129299+0000", 2024-09-10T18:37:36.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9048999999999994e-05 2024-09-10T18:37:36.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:36.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.129364+0000", 2024-09-10T18:37:36.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5083e-05 2024-09-10T18:37:36.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:36.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.130558+0000", 2024-09-10T18:37:36.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001194374 2024-09-10T18:37:36.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:36.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.130583+0000", 2024-09-10T18:37:36.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4499999999999999e-05 2024-09-10T18:37:36.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:36.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.270541+0000", 2024-09-10T18:37:36.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13995775599999999 2024-09-10T18:37:36.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:36.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.270588+0000", 2024-09-10T18:37:36.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7639999999999998e-05 2024-09-10T18:37:36.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:36.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.270630+0000", 2024-09-10T18:37:36.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2268999999999998e-05 2024-09-10T18:37:36.030 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:36.030 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:36.030 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:36.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.030 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:193335 9.6 9:68fa9ae5:::benchmark_data_smithi028_135703_object58744:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-10T18:37:36.030 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:33.971278+0000", 2024-09-10T18:37:36.030 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 181.51176128099999, 2024-09-10T18:37:36.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.20305822500000001, 2024-09-10T18:37:36.031 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:36.031 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:36.031 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:36.031 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:36.031 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:36.031 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 193335 2024-09-10T18:37:36.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.031 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:36.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:36.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.971278+0000", 2024-09-10T18:37:36.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:36.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:36.032 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.971278+0000", 2024-09-10T18:37:36.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:36.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:36.032 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.971276+0000", 2024-09-10T18:37:36.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:36.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:36.032 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.971282+0000", 2024-09-10T18:37:36.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5459999999999999e-06 2024-09-10T18:37:36.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:36.033 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.971295+0000", 2024-09-10T18:37:36.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3431000000000001e-05 2024-09-10T18:37:36.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:36.033 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.971303+0000", 2024-09-10T18:37:36.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4229999999999999e-06 2024-09-10T18:37:36.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:36.033 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.021317+0000", 2024-09-10T18:37:36.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.050014003000000001 2024-09-10T18:37:36.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:36.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.021425+0000", 2024-09-10T18:37:36.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108266 2024-09-10T18:37:36.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:36.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.021566+0000", 2024-09-10T18:37:36.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000140758 2024-09-10T18:37:36.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:36.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.027055+0000", 2024-09-10T18:37:36.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054890299999999998 2024-09-10T18:37:36.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:36.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.028753+0000", 2024-09-10T18:37:36.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016977089999999999 2024-09-10T18:37:36.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:36.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.174249+0000", 2024-09-10T18:37:36.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.1454966 2024-09-10T18:37:36.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:36.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.174299+0000", 2024-09-10T18:37:36.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9917000000000003e-05 2024-09-10T18:37:36.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:36.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.174336+0000", 2024-09-10T18:37:36.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7113000000000002e-05 2024-09-10T18:37:36.036 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:36.036 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:36.036 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:36.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.036 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:193336 9.6 9:6beea92b:::benchmark_data_smithi028_135703_object58745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-10T18:37:36.036 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:33.971453+0000", 2024-09-10T18:37:36.036 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 181.51158662099999, 2024-09-10T18:37:36.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.20300386500000001, 2024-09-10T18:37:36.036 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:36.036 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:36.036 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:36.037 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:36.037 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:36.037 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 193336 2024-09-10T18:37:36.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.037 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:36.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:36.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.971453+0000", 2024-09-10T18:37:36.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:36.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:36.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.971453+0000", 2024-09-10T18:37:36.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:36.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:36.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.971451+0000", 2024-09-10T18:37:36.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:36.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:36.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.971457+0000", 2024-09-10T18:37:36.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9909999999999999e-06 2024-09-10T18:37:36.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:36.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.971469+0000", 2024-09-10T18:37:36.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1545999999999999e-05 2024-09-10T18:37:36.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:36.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.971475+0000", 2024-09-10T18:37:36.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1099999999999999e-06 2024-09-10T18:37:36.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:36.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.023649+0000", 2024-09-10T18:37:36.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.052174231000000001 2024-09-10T18:37:36.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:36.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.023762+0000", 2024-09-10T18:37:36.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000113029 2024-09-10T18:37:36.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:36.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.023908+0000", 2024-09-10T18:37:36.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000145702 2024-09-10T18:37:36.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:36.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.028760+0000", 2024-09-10T18:37:36.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004851892 2024-09-10T18:37:36.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:36.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.028788+0000", 2024-09-10T18:37:36.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8535e-05 2024-09-10T18:37:36.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:36.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.174388+0000", 2024-09-10T18:37:36.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.14559987599999999 2024-09-10T18:37:36.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:36.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.174427+0000", 2024-09-10T18:37:36.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8464000000000003e-05 2024-09-10T18:37:36.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:36.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.174457+0000", 2024-09-10T18:37:36.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9921e-05 2024-09-10T18:37:36.042 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:36.042 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:36.042 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:36.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.042 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:193350 9.6 9:68a49e1f:::benchmark_data_smithi028_135703_object58759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-10T18:37:36.042 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:33.985387+0000", 2024-09-10T18:37:36.042 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 181.49765199500001, 2024-09-10T18:37:36.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.189232181, 2024-09-10T18:37:36.042 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:36.042 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:37:36.043 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:36.043 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:37:36.043 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:37:36.043 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 193350 2024-09-10T18:37:36.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.043 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:36.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:36.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.985387+0000", 2024-09-10T18:37:36.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:36.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:36.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.985387+0000", 2024-09-10T18:37:36.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:36.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:36.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.985385+0000", 2024-09-10T18:37:36.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:37:36.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:36.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.985393+0000", 2024-09-10T18:37:36.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1799999999999996e-06 2024-09-10T18:37:36.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:36.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.985404+0000", 2024-09-10T18:37:36.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0776e-05 2024-09-10T18:37:36.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:36.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.985410+0000", 2024-09-10T18:37:36.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6049999999999997e-06 2024-09-10T18:37:36.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:36.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.025442+0000", 2024-09-10T18:37:36.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.040032127000000001 2024-09-10T18:37:36.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:36.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.025507+0000", 2024-09-10T18:37:36.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4892000000000001e-05 2024-09-10T18:37:36.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:37:36.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.025595+0000", 2024-09-10T18:37:36.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8193000000000001e-05 2024-09-10T18:37:36.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:36.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.028821+0000", 2024-09-10T18:37:36.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032266830000000002 2024-09-10T18:37:36.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:37:36.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.028830+0000", 2024-09-10T18:37:36.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0620000000000001e-06 2024-09-10T18:37:36.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:36.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.174504+0000", 2024-09-10T18:37:36.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.14567424500000001 2024-09-10T18:37:36.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:37:36.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.174582+0000", 2024-09-10T18:37:36.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8610999999999997e-05 2024-09-10T18:37:36.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:36.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:36.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:37:36.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.174619+0000", 2024-09-10T18:37:36.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7083999999999999e-05 2024-09-10T18:37:36.048 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:36.048 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:36.048 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:36.048 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:36.049 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:36.049 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:37:36.049 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:36.050 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 77 16 122149 122133 99.1163 62.625 0.0111903 0.0100847 2024-09-10T18:37:36.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:36.085 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:36.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:36.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:36.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:36.240 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:36.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:36.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:36.381 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:36.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:36.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:36.483 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:36.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:36 smithi028 ceph-mon[39220]: 6.c scrub starts 2024-09-10T18:37:36.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:36 smithi028 ceph-mon[39220]: 6.c scrub ok 2024-09-10T18:37:36.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:36 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:36.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:36 smithi028 ceph-mon[39220]: 4.4 scrub starts 2024-09-10T18:37:36.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:36 smithi028 ceph-mon[39220]: 4.4 scrub ok 2024-09-10T18:37:36.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:36 smithi028 ceph-mon[39220]: pgmap v732: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.3 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-09-10T18:37:36.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:36 smithi028 ceph-mon[39220]: 5.f scrub starts 2024-09-10T18:37:36.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:36 smithi028 ceph-mon[39220]: 5.f scrub ok 2024-09-10T18:37:36.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:36 smithi028 ceph-mon[29291]: 6.c scrub starts 2024-09-10T18:37:36.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:36 smithi028 ceph-mon[29291]: 6.c scrub ok 2024-09-10T18:37:36.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:36 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:36.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:36 smithi028 ceph-mon[29291]: 4.4 scrub starts 2024-09-10T18:37:36.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:36 smithi028 ceph-mon[29291]: 4.4 scrub ok 2024-09-10T18:37:36.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:36 smithi028 ceph-mon[29291]: pgmap v732: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.3 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-09-10T18:37:36.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:36 smithi028 ceph-mon[29291]: 5.f scrub starts 2024-09-10T18:37:36.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:36 smithi028 ceph-mon[29291]: 5.f scrub ok 2024-09-10T18:37:36.521 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:36.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:36.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:36 smithi204 ceph-mon[32357]: 6.c scrub starts 2024-09-10T18:37:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:36 smithi204 ceph-mon[32357]: 6.c scrub ok 2024-09-10T18:37:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:36 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:36.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:36 smithi204 ceph-mon[32357]: 4.4 scrub starts 2024-09-10T18:37:36.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:36 smithi204 ceph-mon[32357]: 4.4 scrub ok 2024-09-10T18:37:36.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:36 smithi204 ceph-mon[32357]: pgmap v732: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.3 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-09-10T18:37:36.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:36 smithi204 ceph-mon[32357]: 5.f scrub starts 2024-09-10T18:37:36.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:36 smithi204 ceph-mon[32357]: 5.f scrub ok 2024-09-10T18:37:36.623 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:36.656 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:36.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:36.762 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:36.803 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:36.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:36.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:36.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:36.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:37.030 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 78 16 123437 123421 98.8772 80.5 0.00967314 0.0101092 2024-09-10T18:37:37.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:37.092 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:37.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:37.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:37.195 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:37.228 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:37.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:37.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:37.326 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-10T18:37:37.330 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:37.407 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:37.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:37.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:37.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:37.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:37 smithi204 ceph-mon[32357]: 5.1 scrub starts 2024-09-10T18:37:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:37 smithi204 ceph-mon[32357]: 5.1 scrub ok 2024-09-10T18:37:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:37 smithi204 ceph-mon[32357]: 4.a scrub starts 2024-09-10T18:37:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:37 smithi204 ceph-mon[32357]: 4.a scrub ok 2024-09-10T18:37:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:37 smithi204 ceph-mon[32357]: 6.14 scrub starts 2024-09-10T18:37:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:37 smithi204 ceph-mon[32357]: 6.14 scrub ok 2024-09-10T18:37:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:37 smithi204 ceph-mon[32357]: 5.1a scrub starts 2024-09-10T18:37:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:37 smithi204 ceph-mon[32357]: 5.1a scrub ok 2024-09-10T18:37:37.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:37.676 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:37.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:37.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:37 smithi028 ceph-mon[29291]: 5.1 scrub starts 2024-09-10T18:37:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:37 smithi028 ceph-mon[29291]: 5.1 scrub ok 2024-09-10T18:37:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:37 smithi028 ceph-mon[29291]: 4.a scrub starts 2024-09-10T18:37:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:37 smithi028 ceph-mon[29291]: 4.a scrub ok 2024-09-10T18:37:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:37 smithi028 ceph-mon[29291]: 6.14 scrub starts 2024-09-10T18:37:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:37 smithi028 ceph-mon[29291]: 6.14 scrub ok 2024-09-10T18:37:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:37 smithi028 ceph-mon[29291]: 5.1a scrub starts 2024-09-10T18:37:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:37 smithi028 ceph-mon[29291]: 5.1a scrub ok 2024-09-10T18:37:37.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:37 smithi028 ceph-mon[39220]: 5.1 scrub starts 2024-09-10T18:37:37.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:37 smithi028 ceph-mon[39220]: 5.1 scrub ok 2024-09-10T18:37:37.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:37 smithi028 ceph-mon[39220]: 4.a scrub starts 2024-09-10T18:37:37.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:37 smithi028 ceph-mon[39220]: 4.a scrub ok 2024-09-10T18:37:37.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:37 smithi028 ceph-mon[39220]: 6.14 scrub starts 2024-09-10T18:37:37.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:37 smithi028 ceph-mon[39220]: 6.14 scrub ok 2024-09-10T18:37:37.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:37 smithi028 ceph-mon[39220]: 5.1a scrub starts 2024-09-10T18:37:37.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:37 smithi028 ceph-mon[39220]: 5.1a scrub ok 2024-09-10T18:37:37.760 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:37:37.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:37.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:37.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:37.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:37.888 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:37:37.888 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:37:37.888 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-10T18:37:37.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:37:37.912 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:37.948 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:37.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:37.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:38.030 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 79 16 124678 124662 98.6064 77.5625 0.014086 0.0101368 2024-09-10T18:37:38.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:38.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:38.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:38.203 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:38.237 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:38.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:38.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:38.339 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:38.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:38.369 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:38.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:38.470 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:38.499 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:38.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:38.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:38.601 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:38 smithi204 ceph-mon[32357]: 5.1c scrub starts 2024-09-10T18:37:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:38 smithi204 ceph-mon[32357]: 5.1c scrub ok 2024-09-10T18:37:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:38 smithi204 ceph-mon[32357]: 6.1 scrub starts 2024-09-10T18:37:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:38 smithi204 ceph-mon[32357]: 6.1 scrub ok 2024-09-10T18:37:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:38 smithi204 ceph-mon[32357]: pgmap v733: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-10T18:37:38.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:38 smithi204 ceph-mon[32357]: 3.c scrub starts 2024-09-10T18:37:38.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:38 smithi204 ceph-mon[32357]: 3.c scrub ok 2024-09-10T18:37:38.639 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:38.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:38.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:38.741 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:38.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:38 smithi028 ceph-mon[39220]: 5.1c scrub starts 2024-09-10T18:37:38.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:38 smithi028 ceph-mon[39220]: 5.1c scrub ok 2024-09-10T18:37:38.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:38 smithi028 ceph-mon[39220]: 6.1 scrub starts 2024-09-10T18:37:38.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:38 smithi028 ceph-mon[39220]: 6.1 scrub ok 2024-09-10T18:37:38.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:38 smithi028 ceph-mon[39220]: pgmap v733: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-10T18:37:38.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:38 smithi028 ceph-mon[39220]: 3.c scrub starts 2024-09-10T18:37:38.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:38 smithi028 ceph-mon[39220]: 3.c scrub ok 2024-09-10T18:37:38.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:38 smithi028 ceph-mon[29291]: 5.1c scrub starts 2024-09-10T18:37:38.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:38 smithi028 ceph-mon[29291]: 5.1c scrub ok 2024-09-10T18:37:38.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:38 smithi028 ceph-mon[29291]: 6.1 scrub starts 2024-09-10T18:37:38.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:38 smithi028 ceph-mon[29291]: 6.1 scrub ok 2024-09-10T18:37:38.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:38 smithi028 ceph-mon[29291]: pgmap v733: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-10T18:37:38.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:38 smithi028 ceph-mon[29291]: 3.c scrub starts 2024-09-10T18:37:38.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:38 smithi028 ceph-mon[29291]: 3.c scrub ok 2024-09-10T18:37:38.762 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:38.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:38.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:38.864 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:38.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:38.900 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:38.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:39.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:39.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:37:39.030173+0000 min lat: 0.00134649 max lat: 0.229265 avg lat: 0.0101818 2024-09-10T18:37:39.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:37:39.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 80 16 125698 125682 98.1703 63.75 0.00736241 0.0101818 2024-09-10T18:37:39.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:39.044 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:39.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:39.146 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:39.183 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:39.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:39.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:39.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:39.336 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:39.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:39.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:39.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:39.473 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:39.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:39.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:39 smithi028 ceph-mon[29291]: 6.6 scrub starts 2024-09-10T18:37:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:39 smithi028 ceph-mon[29291]: 6.6 scrub ok 2024-09-10T18:37:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:39 smithi028 ceph-mon[29291]: 1.b scrub starts 2024-09-10T18:37:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:39 smithi028 ceph-mon[29291]: 1.b scrub ok 2024-09-10T18:37:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:39 smithi028 ceph-mon[29291]: 10.5 scrub starts 2024-09-10T18:37:39.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:37:39.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:39 smithi028 ceph-mon[39220]: 6.6 scrub starts 2024-09-10T18:37:39.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:39 smithi028 ceph-mon[39220]: 6.6 scrub ok 2024-09-10T18:37:39.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:39 smithi028 ceph-mon[39220]: 1.b scrub starts 2024-09-10T18:37:39.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:39 smithi028 ceph-mon[39220]: 1.b scrub ok 2024-09-10T18:37:39.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:39 smithi028 ceph-mon[39220]: 10.5 scrub starts 2024-09-10T18:37:39.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:37:39.575 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:39 smithi204 ceph-mon[32357]: 6.6 scrub starts 2024-09-10T18:37:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:39 smithi204 ceph-mon[32357]: 6.6 scrub ok 2024-09-10T18:37:39.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:39 smithi204 ceph-mon[32357]: 1.b scrub starts 2024-09-10T18:37:39.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:39 smithi204 ceph-mon[32357]: 1.b scrub ok 2024-09-10T18:37:39.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:39 smithi204 ceph-mon[32357]: 10.5 scrub starts 2024-09-10T18:37:39.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:37:39.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:39.607 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:39.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:39.708 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:39.740 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:39.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:39.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:39.885 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:39.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:39.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:39.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:40.024 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:40.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:40.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:40.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 81 16 126818 126802 97.8224 70 0.00749696 0.0102179 2024-09-10T18:37:40.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:40.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:40.181 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:40.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:40.262 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:37:40.262 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:37:40.265 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:37:40.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:40.305 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:37:40.305 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:37:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:40.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:40.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:40.338 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:37:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:37:40.372 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:37:40.373 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:37:40.408 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:37:40.409 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:37:40.435 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:40.444 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:37:40.444 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:37:40.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:40 smithi028 ceph-mon[29291]: pgmap v734: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 511 B/s rd, 52 MiB/s wr, 831 op/s 2024-09-10T18:37:40.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:40 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1709500491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:37:40.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:40 smithi028 ceph-mon[39220]: pgmap v734: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 511 B/s rd, 52 MiB/s wr, 831 op/s 2024-09-10T18:37:40.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:40 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1709500491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:37:40.469 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:40.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:40.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:40.549 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:37:40.550 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:37:40.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:40.581 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:37:40.581 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:37:40.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:40 smithi204 ceph-mon[32357]: pgmap v734: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 511 B/s rd, 52 MiB/s wr, 831 op/s 2024-09-10T18:37:40.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:40 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1709500491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:37:40.610 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:40.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:40.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:40.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:40.739 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:40.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:40.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:40.841 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:40.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:40.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:40.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:40.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:41.022 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:41.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:41.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:41.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 82 16 127857 127841 97.4213 64.9375 0.00596065 0.0102596 2024-09-10T18:37:41.123 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:41.158 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:41.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:41.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:41.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:41.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:41.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:41.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:41.429 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:41.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:41.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:41.565 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:41.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:41.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:41.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:41 smithi204 ceph-mon[32357]: 3.18 scrub starts 2024-09-10T18:37:41.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:41 smithi204 ceph-mon[32357]: 3.18 scrub ok 2024-09-10T18:37:41.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:41 smithi204 ceph-mon[32357]: 1.8 scrub starts 2024-09-10T18:37:41.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:41 smithi204 ceph-mon[32357]: 1.8 scrub ok 2024-09-10T18:37:41.667 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:41.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:37:41.699 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:41.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:41.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:41.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:41 smithi028 ceph-mon[29291]: 3.18 scrub starts 2024-09-10T18:37:41.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:41 smithi028 ceph-mon[29291]: 3.18 scrub ok 2024-09-10T18:37:41.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:41 smithi028 ceph-mon[29291]: 1.8 scrub starts 2024-09-10T18:37:41.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:41 smithi028 ceph-mon[29291]: 1.8 scrub ok 2024-09-10T18:37:41.708 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:37:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:37:41] "GET /metrics HTTP/1.1" 200 224665 "" "Prometheus/2.43.0" 2024-09-10T18:37:41.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:41 smithi028 ceph-mon[39220]: 3.18 scrub starts 2024-09-10T18:37:41.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:41 smithi028 ceph-mon[39220]: 3.18 scrub ok 2024-09-10T18:37:41.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:41 smithi028 ceph-mon[39220]: 1.8 scrub starts 2024-09-10T18:37:41.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:41 smithi028 ceph-mon[39220]: 1.8 scrub ok 2024-09-10T18:37:41.801 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:41.835 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:41.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:41.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:41.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:42.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:42.012 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:42.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:42.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 83 16 128934 128918 97.0579 67.3125 0.0104773 0.0102983 2024-09-10T18:37:42.113 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:42.151 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:42.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:42.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:42.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:42.292 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:42.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:42.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:42.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:42.434 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:42.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:42.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:42.451 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:37:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:37:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:37:42.451 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:42 smithi204 ceph-mon[32357]: 3.1a scrub starts 2024-09-10T18:37:42.451 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:42 smithi204 ceph-mon[32357]: 3.1a scrub ok 2024-09-10T18:37:42.451 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:42 smithi204 ceph-mon[32357]: 10.5 scrub ok 2024-09-10T18:37:42.451 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:42 smithi204 ceph-mon[32357]: 3.1e scrub starts 2024-09-10T18:37:42.451 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:42 smithi204 ceph-mon[32357]: 3.1e scrub ok 2024-09-10T18:37:42.451 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:42 smithi204 ceph-mon[32357]: pgmap v735: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-10T18:37:42.451 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:42 smithi204 ceph-mon[32357]: 5.10 scrub starts 2024-09-10T18:37:42.451 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:42 smithi204 ceph-mon[32357]: 5.10 scrub ok 2024-09-10T18:37:42.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:42.556 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:42 smithi028 ceph-mon[29291]: 3.1a scrub starts 2024-09-10T18:37:42.556 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:42 smithi028 ceph-mon[29291]: 3.1a scrub ok 2024-09-10T18:37:42.556 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:42 smithi028 ceph-mon[29291]: 10.5 scrub ok 2024-09-10T18:37:42.556 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:42 smithi028 ceph-mon[29291]: 3.1e scrub starts 2024-09-10T18:37:42.556 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:42 smithi028 ceph-mon[29291]: 3.1e scrub ok 2024-09-10T18:37:42.556 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:42 smithi028 ceph-mon[29291]: pgmap v735: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-10T18:37:42.556 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:42 smithi028 ceph-mon[29291]: 5.10 scrub starts 2024-09-10T18:37:42.556 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:42 smithi028 ceph-mon[29291]: 5.10 scrub ok 2024-09-10T18:37:42.557 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:42 smithi028 ceph-mon[39220]: 3.1a scrub starts 2024-09-10T18:37:42.557 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:42 smithi028 ceph-mon[39220]: 3.1a scrub ok 2024-09-10T18:37:42.557 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:42 smithi028 ceph-mon[39220]: 10.5 scrub ok 2024-09-10T18:37:42.557 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:42 smithi028 ceph-mon[39220]: 3.1e scrub starts 2024-09-10T18:37:42.557 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:42 smithi028 ceph-mon[39220]: 3.1e scrub ok 2024-09-10T18:37:42.557 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:42 smithi028 ceph-mon[39220]: pgmap v735: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-10T18:37:42.557 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:42 smithi028 ceph-mon[39220]: 5.10 scrub starts 2024-09-10T18:37:42.557 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:42 smithi028 ceph-mon[39220]: 5.10 scrub ok 2024-09-10T18:37:42.579 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:42.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:42.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:42.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:42.723 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:42.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:42.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:42.825 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:42.867 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:42.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:42.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:42.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:43.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:43.007 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:43.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:43.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 84 16 130080 130064 96.7551 71.625 0.0136624 0.0103308 2024-09-10T18:37:43.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:37:43.126 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:37:43.126 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:37:43.149 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:43.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:43.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:43.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:43.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:43.292 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:43.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:43.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:43.423 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:37:43.426 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:37:43.432 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:43.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:43.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:43.536 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:43.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:43.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:43.576 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:43 smithi028 ceph-mon[39220]: 10.a scrub starts 2024-09-10T18:37:43.576 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:43 smithi028 ceph-mon[39220]: 6.11 scrub starts 2024-09-10T18:37:43.576 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:43 smithi028 ceph-mon[39220]: 6.11 scrub ok 2024-09-10T18:37:43.576 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:43 smithi028 ceph-mon[39220]: 1.1d scrub starts 2024-09-10T18:37:43.576 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:43 smithi028 ceph-mon[39220]: 1.1d scrub ok 2024-09-10T18:37:43.576 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:43 smithi028 ceph-mon[39220]: 6.f scrub starts 2024-09-10T18:37:43.576 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:43 smithi028 ceph-mon[39220]: 6.f scrub ok 2024-09-10T18:37:43.576 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:43 smithi028 ceph-mon[39220]: 10.6 scrub starts 2024-09-10T18:37:43.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:43 smithi028 ceph-mon[29291]: 10.a scrub starts 2024-09-10T18:37:43.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:43 smithi028 ceph-mon[29291]: 6.11 scrub starts 2024-09-10T18:37:43.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:43 smithi028 ceph-mon[29291]: 6.11 scrub ok 2024-09-10T18:37:43.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:43 smithi028 ceph-mon[29291]: 1.1d scrub starts 2024-09-10T18:37:43.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:43 smithi028 ceph-mon[29291]: 1.1d scrub ok 2024-09-10T18:37:43.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:43 smithi028 ceph-mon[29291]: 6.f scrub starts 2024-09-10T18:37:43.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:43 smithi028 ceph-mon[29291]: 6.f scrub ok 2024-09-10T18:37:43.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:43 smithi028 ceph-mon[29291]: 10.6 scrub starts 2024-09-10T18:37:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:43 smithi204 ceph-mon[32357]: 10.a scrub starts 2024-09-10T18:37:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:43 smithi204 ceph-mon[32357]: 6.11 scrub starts 2024-09-10T18:37:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:43 smithi204 ceph-mon[32357]: 6.11 scrub ok 2024-09-10T18:37:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:43 smithi204 ceph-mon[32357]: 1.1d scrub starts 2024-09-10T18:37:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:43 smithi204 ceph-mon[32357]: 1.1d scrub ok 2024-09-10T18:37:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:43 smithi204 ceph-mon[32357]: 6.f scrub starts 2024-09-10T18:37:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:43 smithi204 ceph-mon[32357]: 6.f scrub ok 2024-09-10T18:37:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:43 smithi204 ceph-mon[32357]: 10.6 scrub starts 2024-09-10T18:37:43.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:43.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:43.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:43.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:43.841 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:43.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:43.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:43.891 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-10T18:37:43.943 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:44.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:44.027 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:44.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:44.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 85 16 131225 131209 96.4587 71.5625 0.0193045 0.0103625 2024-09-10T18:37:44.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:44.167 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:44.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:44.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:44.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:44.324 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:44.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:44.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:44.362 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:37:44.426 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:44.472 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:44.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:44.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:44.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:37:44.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:44 smithi028 ceph-mon[29291]: 4.9 scrub starts 2024-09-10T18:37:44.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:44 smithi028 ceph-mon[29291]: 4.9 scrub ok 2024-09-10T18:37:44.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:44 smithi028 ceph-mon[29291]: 5.1e scrub starts 2024-09-10T18:37:44.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:44 smithi028 ceph-mon[29291]: 5.1e scrub ok 2024-09-10T18:37:44.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:44 smithi028 ceph-mon[29291]: pgmap v736: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 933 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-09-10T18:37:44.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:44 smithi028 ceph-mon[39220]: 4.9 scrub starts 2024-09-10T18:37:44.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:44 smithi028 ceph-mon[39220]: 4.9 scrub ok 2024-09-10T18:37:44.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:44 smithi028 ceph-mon[39220]: 5.1e scrub starts 2024-09-10T18:37:44.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:44 smithi028 ceph-mon[39220]: 5.1e scrub ok 2024-09-10T18:37:44.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:44 smithi028 ceph-mon[39220]: pgmap v736: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 933 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-09-10T18:37:44.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:44.588 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:44 smithi204 ceph-mon[32357]: 4.9 scrub starts 2024-09-10T18:37:44.588 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:44 smithi204 ceph-mon[32357]: 4.9 scrub ok 2024-09-10T18:37:44.588 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:44 smithi204 ceph-mon[32357]: 5.1e scrub starts 2024-09-10T18:37:44.588 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:44 smithi204 ceph-mon[32357]: 5.1e scrub ok 2024-09-10T18:37:44.588 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:44 smithi204 ceph-mon[32357]: pgmap v736: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 933 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-09-10T18:37:44.622 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:44.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:44.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:44.723 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:44.767 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:44.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:44.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:44.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:37:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:37:44.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:44.911 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:44.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:45.012 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:45.031 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 86 16 132295 132279 96.1147 66.875 0.0232797 0.0103996 2024-09-10T18:37:45.051 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:45.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:45.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:45.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:45.196 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:45.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:45.298 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:45.341 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:45.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:45.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:45.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:45.482 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:45.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:45.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:45.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:45 smithi204 ceph-mon[32357]: from='client.35693 v1:172.21.15.28:0/3583491261' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:37:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:45 smithi204 ceph-mon[32357]: 6.e scrub starts 2024-09-10T18:37:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:45 smithi204 ceph-mon[32357]: 6.e scrub ok 2024-09-10T18:37:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:45 smithi204 ceph-mon[32357]: 3.2 scrub starts 2024-09-10T18:37:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:45 smithi204 ceph-mon[32357]: 3.2 scrub ok 2024-09-10T18:37:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:45 smithi204 ceph-mon[32357]: 1.11 scrub starts 2024-09-10T18:37:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:45 smithi204 ceph-mon[32357]: 1.11 scrub ok 2024-09-10T18:37:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:45 smithi204 ceph-mon[32357]: 5.1d scrub starts 2024-09-10T18:37:45.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:45 smithi204 ceph-mon[32357]: 5.1d scrub ok 2024-09-10T18:37:45.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:45 smithi204 ceph-mon[32357]: 5.b scrub starts 2024-09-10T18:37:45.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:45 smithi204 ceph-mon[32357]: 5.b scrub ok 2024-09-10T18:37:45.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:45 smithi204 ceph-mon[32357]: 6.1b scrub starts 2024-09-10T18:37:45.622 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:45.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:45.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:45.724 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:45.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[39220]: from='client.35693 v1:172.21.15.28:0/3583491261' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:37:45.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[39220]: 6.e scrub starts 2024-09-10T18:37:45.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[39220]: 6.e scrub ok 2024-09-10T18:37:45.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[39220]: 3.2 scrub starts 2024-09-10T18:37:45.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[39220]: 3.2 scrub ok 2024-09-10T18:37:45.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[39220]: 1.11 scrub starts 2024-09-10T18:37:45.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[39220]: 1.11 scrub ok 2024-09-10T18:37:45.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[39220]: 5.1d scrub starts 2024-09-10T18:37:45.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[39220]: 5.1d scrub ok 2024-09-10T18:37:45.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[39220]: 5.b scrub starts 2024-09-10T18:37:45.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[39220]: 5.b scrub ok 2024-09-10T18:37:45.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[39220]: 6.1b scrub starts 2024-09-10T18:37:45.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[29291]: from='client.35693 v1:172.21.15.28:0/3583491261' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:37:45.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[29291]: 6.e scrub starts 2024-09-10T18:37:45.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[29291]: 6.e scrub ok 2024-09-10T18:37:45.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[29291]: 3.2 scrub starts 2024-09-10T18:37:45.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[29291]: 3.2 scrub ok 2024-09-10T18:37:45.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[29291]: 1.11 scrub starts 2024-09-10T18:37:45.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[29291]: 1.11 scrub ok 2024-09-10T18:37:45.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[29291]: 5.1d scrub starts 2024-09-10T18:37:45.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[29291]: 5.1d scrub ok 2024-09-10T18:37:45.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[29291]: 5.b scrub starts 2024-09-10T18:37:45.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[29291]: 5.b scrub ok 2024-09-10T18:37:45.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:45 smithi028 ceph-mon[29291]: 6.1b scrub starts 2024-09-10T18:37:45.764 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:45.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:45.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:45.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:45.911 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:45.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:45.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:46.013 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:46.032 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 87 16 133586 133570 95.9373 80.6875 0.00919852 0.0104188 2024-09-10T18:37:46.051 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:46.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:46.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:46.153 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:46.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:46.202 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:46.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:46.303 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:46.342 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:46.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:46.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:46.439 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:37:46.439 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:37:46.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:46.481 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:46.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:46.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[29291]: 6.1b scrub ok 2024-09-10T18:37:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[29291]: 1.7 scrub starts 2024-09-10T18:37:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[29291]: 1.7 scrub ok 2024-09-10T18:37:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[29291]: 10.a scrub ok 2024-09-10T18:37:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[29291]: pgmap v737: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.8 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.3 KiB/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-10T18:37:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[29291]: 10.6 scrub ok 2024-09-10T18:37:46.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[29291]: 3.8 scrub starts 2024-09-10T18:37:46.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[29291]: 3.8 scrub ok 2024-09-10T18:37:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[39220]: 6.1b scrub ok 2024-09-10T18:37:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[39220]: 1.7 scrub starts 2024-09-10T18:37:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[39220]: 1.7 scrub ok 2024-09-10T18:37:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[39220]: 10.a scrub ok 2024-09-10T18:37:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[39220]: pgmap v737: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.8 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.3 KiB/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-10T18:37:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[39220]: 10.6 scrub ok 2024-09-10T18:37:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[39220]: 3.8 scrub starts 2024-09-10T18:37:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:46 smithi028 ceph-mon[39220]: 3.8 scrub ok 2024-09-10T18:37:46.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:46 smithi204 ceph-mon[32357]: 6.1b scrub ok 2024-09-10T18:37:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:46 smithi204 ceph-mon[32357]: 1.7 scrub starts 2024-09-10T18:37:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:46 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:46 smithi204 ceph-mon[32357]: 1.7 scrub ok 2024-09-10T18:37:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:46 smithi204 ceph-mon[32357]: 10.a scrub ok 2024-09-10T18:37:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:46 smithi204 ceph-mon[32357]: pgmap v737: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.8 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.3 KiB/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-10T18:37:46.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:46 smithi204 ceph-mon[32357]: 10.6 scrub ok 2024-09-10T18:37:46.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:46 smithi204 ceph-mon[32357]: 3.8 scrub starts 2024-09-10T18:37:46.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:46 smithi204 ceph-mon[32357]: 3.8 scrub ok 2024-09-10T18:37:46.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:46.647 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:46.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:46.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:46.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:46.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:46.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:46.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:46.932 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:46.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:47.032 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 88 16 134801 134785 95.71 75.9375 0.0182336 0.0104441 2024-09-10T18:37:47.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:47.065 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:47.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:47.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:47.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:47.207 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:47.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:47.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:47.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:47.345 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:47.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:47.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:47.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:47.471 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":736,"stamp":"2024-09-10T18:37:43.097958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8353945820,"num_objects":127658,"num_object_clones":0,"num_object_copies":382974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127658,"num_whiteouts":0,"num_read":2107,"num_read_kb":1859,"num_write":128178,"num_write_kb":8157611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79822,"ondisk_log_size":79822,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":28516740,"kb_used_data":24339140,"kb_used_omap":19,"kb_used_meta":4177516,"kb_avail":721444476,"statfs":{"total":767960285184,"available":738759143424,"internally_reserved":0,"allocated":24923279360,"data_stored":24920337180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20033,"internal_metadata":4277776831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,4,5,10,3,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":760020992,"num_objects":11597,"num_object_clones":0,"num_object_copies":34791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11597,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":11597,"num_write_kb":742208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.868330"},"pg_stats":[{"pgid":"1.1d","version":"76'81","reported_seq":75,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.253278+0000","last_change":"2024-09-10T18:35:17.818822+0000","last_active":"2024-09-10T18:37:33.253278+0000","last_peered":"2024-09-10T18:37:33.253278+0000","last_clean":"2024-09-10T18:37:33.253278+0000","last_became_active":"2024-09-10T18:34:34.865894+0000","last_became_peered":"2024-09-10T18:34:34.865894+0000","last_unstale":"2024-09-10T18:37:33.253278+0000","last_undegraded":"2024-09-10T18:37:33.253278+0000","last_fullsized":"2024-09-10T18:37:33.253278+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:17.818727+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:17.818727+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037738960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":361,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.247382+0000","last_change":"2024-09-10T18:36:17.645197+0000","last_active":"2024-09-10T18:37:33.247382+0000","last_peered":"2024-09-10T18:37:33.247382+0000","last_clean":"2024-09-10T18:37:33.247382+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:37:33.247382+0000","last_undegraded":"2024-09-10T18:37:33.247382+0000","last_fullsized":"2024-09-10T18:37:33.247382+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:17.645146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:36:17.645146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:43.759721+0000","scrub_duration":0.003140618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":389,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.247300+0000","last_change":"2024-09-10T18:35:20.421834+0000","last_active":"2024-09-10T18:37:33.247300+0000","last_peered":"2024-09-10T18:37:33.247300+0000","last_clean":"2024-09-10T18:37:33.247300+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:37:33.247300+0000","last_undegraded":"2024-09-10T18:37:33.247300+0000","last_fullsized":"2024-09-10T18:37:33.247300+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:35:20.421727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:20.421727+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010997518,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":366,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.251310+0000","last_change":"2024-09-10T18:37:12.854944+0000","last_active":"2024-09-10T18:37:33.251310+0000","last_peered":"2024-09-10T18:37:33.251310+0000","last_clean":"2024-09-10T18:37:33.251310+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:37:33.251310+0000","last_undegraded":"2024-09-10T18:37:33.251310+0000","last_fullsized":"2024-09-10T18:37:33.251310+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:12.854879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:12.854879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:41.911709+0000","scrub_duration":0.019720808999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":356,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.248064+0000","last_change":"2024-09-10T18:36:52.118582+0000","last_active":"2024-09-10T18:37:33.248064+0000","last_peered":"2024-09-10T18:37:33.248064+0000","last_clean":"2024-09-10T18:37:33.248064+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:37:33.248064+0000","last_undegraded":"2024-09-10T18:37:33.248064+0000","last_fullsized":"2024-09-10T18:37:33.248064+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:52.118515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:36:52.118515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:16.344464+0000","scrub_duration":0.0076434099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'81","reported_seq":70,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.238585+0000","last_change":"2024-09-10T18:36:48.858044+0000","last_active":"2024-09-10T18:37:33.238585+0000","last_peered":"2024-09-10T18:37:33.238585+0000","last_clean":"2024-09-10T18:37:33.238585+0000","last_became_active":"2024-09-10T18:34:34.867388+0000","last_became_peered":"2024-09-10T18:34:34.867388+0000","last_unstale":"2024-09-10T18:37:33.238585+0000","last_undegraded":"2024-09-10T18:37:33.238585+0000","last_fullsized":"2024-09-10T18:37:33.238585+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:48.857902+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:48.857902+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:09.493773+0000","scrub_duration":0.0059112879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":353,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.232958+0000","last_change":"2024-09-10T18:34:16.293404+0000","last_active":"2024-09-10T18:37:33.232958+0000","last_peered":"2024-09-10T18:37:33.232958+0000","last_clean":"2024-09-10T18:37:33.232958+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:37:33.232958+0000","last_undegraded":"2024-09-10T18:37:33.232958+0000","last_fullsized":"2024-09-10T18:37:33.232958+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:16.293206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:16.293206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021119960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":359,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.233970+0000","last_change":"2024-09-10T18:34:22.415416+0000","last_active":"2024-09-10T18:37:33.233970+0000","last_peered":"2024-09-10T18:37:33.233970+0000","last_clean":"2024-09-10T18:37:33.233970+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:37:33.233970+0000","last_undegraded":"2024-09-10T18:37:33.233970+0000","last_fullsized":"2024-09-10T18:37:33.233970+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:22.415093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:34:22.415093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.068845071999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'34","reported_seq":400,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.232942+0000","last_change":"2024-09-10T18:36:10.786060+0000","last_active":"2024-09-10T18:37:33.232942+0000","last_peered":"2024-09-10T18:37:33.232942+0000","last_clean":"2024-09-10T18:37:33.232942+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:37:33.232942+0000","last_undegraded":"2024-09-10T18:37:33.232942+0000","last_fullsized":"2024-09-10T18:37:33.232942+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:36:10.786002+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:36:10.786002+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:36.303670+0000","scrub_duration":0.0021577860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":365,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.247190+0000","last_change":"2024-09-10T18:36:07.325812+0000","last_active":"2024-09-10T18:37:33.247190+0000","last_peered":"2024-09-10T18:37:33.247190+0000","last_clean":"2024-09-10T18:37:33.247190+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:37:33.247190+0000","last_undegraded":"2024-09-10T18:37:33.247190+0000","last_fullsized":"2024-09-10T18:37:33.247190+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:07.325744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.325744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001764406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":370,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.246179+0000","last_change":"2024-09-10T18:36:09.700648+0000","last_active":"2024-09-10T18:37:33.246179+0000","last_peered":"2024-09-10T18:37:33.246179+0000","last_clean":"2024-09-10T18:37:33.246179+0000","last_became_active":"2024-09-10T18:26:27.644961+0000","last_became_peered":"2024-09-10T18:26:27.644961+0000","last_unstale":"2024-09-10T18:37:33.246179+0000","last_undegraded":"2024-09-10T18:37:33.246179+0000","last_fullsized":"2024-09-10T18:37:33.246179+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:09.700579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_clean_scrub_stamp":"2024-09-10T18:36:09.700579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00158731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'15","reported_seq":392,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.248333+0000","last_change":"2024-09-10T18:37:05.365783+0000","last_active":"2024-09-10T18:37:33.248333+0000","last_peered":"2024-09-10T18:37:33.248333+0000","last_clean":"2024-09-10T18:37:33.248333+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:37:33.248333+0000","last_undegraded":"2024-09-10T18:37:33.248333+0000","last_fullsized":"2024-09-10T18:37:33.248333+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:37:05.365702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:05.365702+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:27.722998+0000","scrub_duration":0.0029180819999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":366,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.245052+0000","last_change":"2024-09-10T18:36:52.071972+0000","last_active":"2024-09-10T18:37:33.245052+0000","last_peered":"2024-09-10T18:37:33.245052+0000","last_clean":"2024-09-10T18:37:33.245052+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:37:33.245052+0000","last_undegraded":"2024-09-10T18:37:33.245052+0000","last_fullsized":"2024-09-10T18:37:33.245052+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:52.071894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_clean_scrub_stamp":"2024-09-10T18:36:52.071894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:05.580562+0000","scrub_duration":0.0025636399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":365,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.240486+0000","last_change":"2024-09-10T18:36:54.584423+0000","last_active":"2024-09-10T18:37:33.240486+0000","last_peered":"2024-09-10T18:37:33.240486+0000","last_clean":"2024-09-10T18:37:33.240486+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:37:33.240486+0000","last_undegraded":"2024-09-10T18:37:33.240486+0000","last_fullsized":"2024-09-10T18:37:33.240486+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:54.583785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:54.583785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:23.333666+0000","scrub_duration":0.0082206150000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'81","reported_seq":139,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.233723+0000","last_change":"2024-09-10T18:37:06.945729+0000","last_active":"2024-09-10T18:37:33.233723+0000","last_peered":"2024-09-10T18:37:33.233723+0000","last_clean":"2024-09-10T18:37:33.233723+0000","last_became_active":"2024-09-10T18:34:34.863102+0000","last_became_peered":"2024-09-10T18:34:34.863102+0000","last_unstale":"2024-09-10T18:37:33.233723+0000","last_undegraded":"2024-09-10T18:37:33.233723+0000","last_fullsized":"2024-09-10T18:37:33.233723+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:06.945645+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:06.945645+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:11.913296+0000","scrub_duration":0.0045824999999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":370,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.245551+0000","last_change":"2024-09-10T18:36:56.124404+0000","last_active":"2024-09-10T18:37:33.245551+0000","last_peered":"2024-09-10T18:37:33.245551+0000","last_clean":"2024-09-10T18:37:33.245551+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:37:33.245551+0000","last_undegraded":"2024-09-10T18:37:33.245551+0000","last_fullsized":"2024-09-10T18:37:33.245551+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:56.124318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:17.464205+0000","last_clean_scrub_stamp":"2024-09-10T18:36:56.124318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:10.885488+0000","scrub_duration":0.0046808750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":404,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.247924+0000","last_change":"2024-09-10T18:37:09.383413+0000","last_active":"2024-09-10T18:37:33.247924+0000","last_peered":"2024-09-10T18:37:33.247924+0000","last_clean":"2024-09-10T18:37:33.247924+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:37:33.247924+0000","last_undegraded":"2024-09-10T18:37:33.247924+0000","last_fullsized":"2024-09-10T18:37:33.247924+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'9","last_scrub_stamp":"2024-09-10T18:37:09.383347+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:37:09.383347+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:10.796821+0000","scrub_duration":0.010843228999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":365,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:36.885661+0000","last_change":"2024-09-10T18:37:36.885626+0000","last_active":"2024-09-10T18:37:36.885661+0000","last_peered":"2024-09-10T18:37:36.885661+0000","last_clean":"2024-09-10T18:37:36.885661+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:37:36.885661+0000","last_undegraded":"2024-09-10T18:37:36.885661+0000","last_fullsized":"2024-09-10T18:37:36.885661+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:36.885547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:36.885547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:37.837293+0000","scrub_duration":0.0028632850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":357,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.245517+0000","last_change":"2024-09-10T18:36:15.726285+0000","last_active":"2024-09-10T18:37:33.245517+0000","last_peered":"2024-09-10T18:37:33.245517+0000","last_clean":"2024-09-10T18:37:33.245517+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:37:33.245517+0000","last_undegraded":"2024-09-10T18:37:33.245517+0000","last_fullsized":"2024-09-10T18:37:33.245517+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:15.726200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:36:15.726200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020103579999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'81","reported_seq":222,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.232371+0000","last_change":"2024-09-10T18:37:07.984531+0000","last_active":"2024-09-10T18:37:33.232371+0000","last_peered":"2024-09-10T18:37:33.232371+0000","last_clean":"2024-09-10T18:37:33.232371+0000","last_became_active":"2024-09-10T18:34:34.862917+0000","last_became_peered":"2024-09-10T18:34:34.862917+0000","last_unstale":"2024-09-10T18:37:33.232371+0000","last_undegraded":"2024-09-10T18:37:33.232371+0000","last_fullsized":"2024-09-10T18:37:33.232371+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:07.984446+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:07.984446+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:16.037756+0000","scrub_duration":0.0054176850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":363,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.249405+0000","last_change":"2024-09-10T18:36:06.360654+0000","last_active":"2024-09-10T18:37:33.249405+0000","last_peered":"2024-09-10T18:37:33.249405+0000","last_clean":"2024-09-10T18:37:33.249405+0000","last_became_active":"2024-09-10T18:26:27.650774+0000","last_became_peered":"2024-09-10T18:26:27.650774+0000","last_unstale":"2024-09-10T18:37:33.249405+0000","last_undegraded":"2024-09-10T18:37:33.249405+0000","last_fullsized":"2024-09-10T18:37:33.249405+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.360595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.360595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012270849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":367,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.250382+0000","last_change":"2024-09-10T18:37:13.830411+0000","last_active":"2024-09-10T18:37:33.250382+0000","last_peered":"2024-09-10T18:37:33.250382+0000","last_clean":"2024-09-10T18:37:33.250382+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:37:33.250382+0000","last_undegraded":"2024-09-10T18:37:33.250382+0000","last_fullsized":"2024-09-10T18:37:33.250382+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:13.830325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:13.830325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:15.627331+0000","scrub_duration":0.0070701799999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"240'27","reported_seq":430,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.246683+0000","last_change":"2024-09-10T18:36:17.526735+0000","last_active":"2024-09-10T18:37:33.246683+0000","last_peered":"2024-09-10T18:37:33.246683+0000","last_clean":"2024-09-10T18:37:33.246683+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:37:33.246683+0000","last_undegraded":"2024-09-10T18:37:33.246683+0000","last_fullsized":"2024-09-10T18:37:33.246683+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:36:17.526667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:17.526667+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:45.386424+0000","scrub_duration":0.0019245169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":366,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.253573+0000","last_change":"2024-09-10T18:35:13.892354+0000","last_active":"2024-09-10T18:37:33.253573+0000","last_peered":"2024-09-10T18:37:33.253573+0000","last_clean":"2024-09-10T18:37:33.253573+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:37:33.253573+0000","last_undegraded":"2024-09-10T18:37:33.253573+0000","last_fullsized":"2024-09-10T18:37:33.253573+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:13.892271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:13.892271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058623920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'81","reported_seq":292,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.245073+0000","last_change":"2024-09-10T18:36:06.623015+0000","last_active":"2024-09-10T18:37:33.245073+0000","last_peered":"2024-09-10T18:37:33.245073+0000","last_clean":"2024-09-10T18:37:33.245073+0000","last_became_active":"2024-09-10T18:34:34.855693+0000","last_became_peered":"2024-09-10T18:34:34.855693+0000","last_unstale":"2024-09-10T18:37:33.245073+0000","last_undegraded":"2024-09-10T18:37:33.245073+0000","last_fullsized":"2024-09-10T18:37:33.245073+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:06.622956+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.622956+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017570750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":353,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.234184+0000","last_change":"2024-09-10T18:34:21.344309+0000","last_active":"2024-09-10T18:37:33.234184+0000","last_peered":"2024-09-10T18:37:33.234184+0000","last_clean":"2024-09-10T18:37:33.234184+0000","last_became_active":"2024-09-10T18:26:27.640035+0000","last_became_peered":"2024-09-10T18:26:27.640035+0000","last_unstale":"2024-09-10T18:37:33.234184+0000","last_undegraded":"2024-09-10T18:37:33.234184+0000","last_fullsized":"2024-09-10T18:37:33.234184+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:21.344236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:34:21.344236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040101240000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":359,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.248374+0000","last_change":"2024-09-10T18:35:04.319486+0000","last_active":"2024-09-10T18:37:33.248374+0000","last_peered":"2024-09-10T18:37:33.248374+0000","last_clean":"2024-09-10T18:37:33.248374+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:37:33.248374+0000","last_undegraded":"2024-09-10T18:37:33.248374+0000","last_fullsized":"2024-09-10T18:37:33.248374+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:04.319395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:35:04.319395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002865018,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'23","reported_seq":387,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.234322+0000","last_change":"2024-09-10T18:35:14.739218+0000","last_active":"2024-09-10T18:37:33.234322+0000","last_peered":"2024-09-10T18:37:33.234322+0000","last_clean":"2024-09-10T18:37:33.234322+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:37:33.234322+0000","last_undegraded":"2024-09-10T18:37:33.234322+0000","last_fullsized":"2024-09-10T18:37:33.234322+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'23","last_scrub_stamp":"2024-09-10T18:35:14.739123+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:35:14.739123+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052949870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":371,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:36.407934+0000","last_change":"2024-09-10T18:37:36.407892+0000","last_active":"2024-09-10T18:37:36.407934+0000","last_peered":"2024-09-10T18:37:36.407934+0000","last_clean":"2024-09-10T18:37:36.407934+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:37:36.407934+0000","last_undegraded":"2024-09-10T18:37:36.407934+0000","last_fullsized":"2024-09-10T18:37:36.407934+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:36.407806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:08.266279+0000","last_clean_scrub_stamp":"2024-09-10T18:37:36.407806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:40.475305+0000","scrub_duration":0.0037520880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'81","reported_seq":280,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.244823+0000","last_change":"2024-09-10T18:36:07.673249+0000","last_active":"2024-09-10T18:37:33.244823+0000","last_peered":"2024-09-10T18:37:33.244823+0000","last_clean":"2024-09-10T18:37:33.244823+0000","last_became_active":"2024-09-10T18:34:34.855038+0000","last_became_peered":"2024-09-10T18:34:34.855038+0000","last_unstale":"2024-09-10T18:37:33.244823+0000","last_undegraded":"2024-09-10T18:37:33.244823+0000","last_fullsized":"2024-09-10T18:37:33.244823+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:07.673176+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.673176+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018195469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":355,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.238398+0000","last_change":"2024-09-10T18:35:20.905675+0000","last_active":"2024-09-10T18:37:33.238398+0000","last_peered":"2024-09-10T18:37:33.238398+0000","last_clean":"2024-09-10T18:37:33.238398+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:37:33.238398+0000","last_undegraded":"2024-09-10T18:37:33.238398+0000","last_fullsized":"2024-09-10T18:37:33.238398+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:35:20.905576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:20.905576+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049065819999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":380,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.251456+0000","last_change":"2024-09-10T18:37:08.821787+0000","last_active":"2024-09-10T18:37:33.251456+0000","last_peered":"2024-09-10T18:37:33.251456+0000","last_clean":"2024-09-10T18:37:33.251456+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:37:33.251456+0000","last_undegraded":"2024-09-10T18:37:33.251456+0000","last_fullsized":"2024-09-10T18:37:33.251456+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:08.821692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:08.821692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:32.765319+0000","scrub_duration":0.0031507520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'25","reported_seq":404,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.248498+0000","last_change":"2024-09-10T18:37:00.670455+0000","last_active":"2024-09-10T18:37:33.248498+0000","last_peered":"2024-09-10T18:37:33.248498+0000","last_clean":"2024-09-10T18:37:33.248498+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:37:33.248498+0000","last_undegraded":"2024-09-10T18:37:33.248498+0000","last_fullsized":"2024-09-10T18:37:33.248498+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:37:00.670369+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:37:00.670369+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:06.286835+0000","scrub_duration":0.0057824440000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"325'12","reported_seq":42,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:34.953145+0000","last_change":"2024-09-10T18:36:52.891388+0000","last_active":"2024-09-10T18:37:34.953145+0000","last_peered":"2024-09-10T18:37:34.953145+0000","last_clean":"2024-09-10T18:37:34.953145+0000","last_became_active":"2024-09-10T18:36:52.882218+0000","last_became_peered":"2024-09-10T18:36:52.882218+0000","last_unstale":"2024-09-10T18:37:34.953145+0000","last_undegraded":"2024-09-10T18:37:34.953145+0000","last_fullsized":"2024-09-10T18:37:34.953145+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11","last_scrub_stamp":"2024-09-10T18:36:11.854492+0000","last_deep_scrub":"217'11","last_deep_scrub_stamp":"2024-09-10T18:30:03.161985+0000","last_clean_scrub_stamp":"2024-09-10T18:36:11.854492+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073290009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1","version":"341'7838","reported_seq":7873,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:39.920754+0000","last_change":"2024-09-10T18:36:07.904585+0000","last_active":"2024-09-10T18:37:39.920754+0000","last_peered":"2024-09-10T18:37:39.920754+0000","last_clean":"2024-09-10T18:37:39.920754+0000","last_became_active":"2024-09-10T18:36:07.904319+0000","last_became_peered":"2024-09-10T18:36:07.904319+0000","last_unstale":"2024-09-10T18:37:39.920754+0000","last_undegraded":"2024-09-10T18:37:39.920754+0000","last_fullsized":"2024-09-10T18:37:39.920754+0000","mapping_epoch":314,"log_start":"323'3800","ondisk_log_start":"323'3800","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4038,"log_dups_size":0,"ondisk_log_size":4038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513671168,"num_objects":7838,"num_object_clones":0,"num_object_copies":23514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7838,"num_write_kb":501632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":286,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.246106+0000","last_change":"2024-09-10T18:35:59.488083+0000","last_active":"2024-09-10T18:37:33.246106+0000","last_peered":"2024-09-10T18:37:33.246106+0000","last_clean":"2024-09-10T18:37:33.246106+0000","last_became_active":"2024-09-10T18:34:32.823230+0000","last_became_peered":"2024-09-10T18:34:32.823230+0000","last_unstale":"2024-09-10T18:37:33.246106+0000","last_undegraded":"2024-09-10T18:37:33.246106+0000","last_fullsized":"2024-09-10T18:37:33.246106+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:59.488014+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:59.488014+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017684759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"240'34","reported_seq":431,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.253016+0000","last_change":"2024-09-10T18:37:03.818405+0000","last_active":"2024-09-10T18:37:33.253016+0000","last_peered":"2024-09-10T18:37:33.253016+0000","last_clean":"2024-09-10T18:37:33.253016+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:37:33.253016+0000","last_undegraded":"2024-09-10T18:37:33.253016+0000","last_fullsized":"2024-09-10T18:37:33.253016+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:37:03.818290+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:37:03.818290+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:17.664855+0000","scrub_duration":0.0051578769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":352,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.232789+0000","last_change":"2024-09-10T18:34:20.305626+0000","last_active":"2024-09-10T18:37:33.232789+0000","last_peered":"2024-09-10T18:37:33.232789+0000","last_clean":"2024-09-10T18:37:33.232789+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:37:33.232789+0000","last_undegraded":"2024-09-10T18:37:33.232789+0000","last_fullsized":"2024-09-10T18:37:33.232789+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:20.305547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:20.305547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046035989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":559,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:34.291372+0000","last_change":"2024-09-10T18:37:03.315311+0000","last_active":"2024-09-10T18:37:34.291372+0000","last_peered":"2024-09-10T18:37:34.291372+0000","last_clean":"2024-09-10T18:37:34.291372+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:37:34.291372+0000","last_undegraded":"2024-09-10T18:37:34.291372+0000","last_fullsized":"2024-09-10T18:37:34.291372+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:37:03.315226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:03.315226+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:23.900377+0000","scrub_duration":0.017386324000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":364,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.245846+0000","last_change":"2024-09-10T18:37:09.280044+0000","last_active":"2024-09-10T18:37:33.245846+0000","last_peered":"2024-09-10T18:37:33.245846+0000","last_clean":"2024-09-10T18:37:33.245846+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:37:33.245846+0000","last_undegraded":"2024-09-10T18:37:33.245846+0000","last_fullsized":"2024-09-10T18:37:33.245846+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:09.279969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:37:09.279969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:18.441371+0000","scrub_duration":0.0041856610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"341'7828","reported_seq":7863,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:39.848366+0000","last_change":"2024-09-10T18:36:07.900391+0000","last_active":"2024-09-10T18:37:39.848366+0000","last_peered":"2024-09-10T18:37:39.848366+0000","last_clean":"2024-09-10T18:37:39.848366+0000","last_became_active":"2024-09-10T18:36:07.900059+0000","last_became_peered":"2024-09-10T18:36:07.900059+0000","last_unstale":"2024-09-10T18:37:39.848366+0000","last_undegraded":"2024-09-10T18:37:39.848366+0000","last_fullsized":"2024-09-10T18:37:39.848366+0000","mapping_epoch":314,"log_start":"323'3800","ondisk_log_start":"323'3800","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4028,"log_dups_size":0,"ondisk_log_size":4028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513015808,"num_objects":7828,"num_object_clones":0,"num_object_copies":23484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7828,"num_write_kb":500992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":295,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.249515+0000","last_change":"2024-09-10T18:35:56.307718+0000","last_active":"2024-09-10T18:37:33.249515+0000","last_peered":"2024-09-10T18:37:33.249515+0000","last_clean":"2024-09-10T18:37:33.249515+0000","last_became_active":"2024-09-10T18:34:32.815839+0000","last_became_peered":"2024-09-10T18:34:32.815839+0000","last_unstale":"2024-09-10T18:37:33.249515+0000","last_undegraded":"2024-09-10T18:37:33.249515+0000","last_fullsized":"2024-09-10T18:37:33.249515+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:56.307649+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:56.307649+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015720840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":400,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.232573+0000","last_change":"2024-09-10T18:36:07.782175+0000","last_active":"2024-09-10T18:37:33.232573+0000","last_peered":"2024-09-10T18:37:33.232573+0000","last_clean":"2024-09-10T18:37:33.232573+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:37:33.232573+0000","last_undegraded":"2024-09-10T18:37:33.232573+0000","last_fullsized":"2024-09-10T18:37:33.232573+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-09-10T18:36:07.782112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.782112+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00244122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":352,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.234742+0000","last_change":"2024-09-10T18:35:15.765590+0000","last_active":"2024-09-10T18:37:33.234742+0000","last_peered":"2024-09-10T18:37:33.234742+0000","last_clean":"2024-09-10T18:37:33.234742+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:37:33.234742+0000","last_undegraded":"2024-09-10T18:37:33.234742+0000","last_fullsized":"2024-09-10T18:37:33.234742+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:15.765392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:15.765392+0000","last_clean_scrub_stamp":"2024-09-10T18:35:15.765392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01380637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":365,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:35.460663+0000","last_change":"2024-09-10T18:37:35.460385+0000","last_active":"2024-09-10T18:37:35.460663+0000","last_peered":"2024-09-10T18:37:35.460663+0000","last_clean":"2024-09-10T18:37:35.460663+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:37:35.460663+0000","last_undegraded":"2024-09-10T18:37:35.460663+0000","last_fullsized":"2024-09-10T18:37:35.460663+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:35.460269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:37:35.460269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:01.784794+0000","scrub_duration":0.0073608529999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":353,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.248256+0000","last_change":"2024-09-10T18:37:06.386024+0000","last_active":"2024-09-10T18:37:33.248256+0000","last_peered":"2024-09-10T18:37:33.248256+0000","last_clean":"2024-09-10T18:37:33.248256+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:37:33.248256+0000","last_undegraded":"2024-09-10T18:37:33.248256+0000","last_fullsized":"2024-09-10T18:37:33.248256+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:06.385944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:06.385944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:32.414011+0000","scrub_duration":0.003928651,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"341'8088","reported_seq":8123,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:41.019427+0000","last_change":"2024-09-10T18:36:07.902153+0000","last_active":"2024-09-10T18:37:41.019427+0000","last_peered":"2024-09-10T18:37:41.019427+0000","last_clean":"2024-09-10T18:37:41.019427+0000","last_became_active":"2024-09-10T18:36:07.901822+0000","last_became_peered":"2024-09-10T18:36:07.901822+0000","last_unstale":"2024-09-10T18:37:41.019427+0000","last_undegraded":"2024-09-10T18:37:41.019427+0000","last_fullsized":"2024-09-10T18:37:41.019427+0000","mapping_epoch":314,"log_start":"322'3100","ondisk_log_start":"322'3100","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4988,"log_dups_size":0,"ondisk_log_size":4988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":530055168,"num_objects":8088,"num_object_clones":0,"num_object_copies":24264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8088,"num_write_kb":517632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":267,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.233333+0000","last_change":"2024-09-10T18:35:13.749157+0000","last_active":"2024-09-10T18:37:33.233333+0000","last_peered":"2024-09-10T18:37:33.233333+0000","last_clean":"2024-09-10T18:37:33.233333+0000","last_became_active":"2024-09-10T18:34:00.166362+0000","last_became_peered":"2024-09-10T18:34:00.166362+0000","last_unstale":"2024-09-10T18:37:33.233333+0000","last_undegraded":"2024-09-10T18:37:33.233333+0000","last_fullsized":"2024-09-10T18:37:33.233333+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:13.748963+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:13.748963+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013370432999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"240'28","reported_seq":406,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.247886+0000","last_change":"2024-09-10T18:36:16.155412+0000","last_active":"2024-09-10T18:37:33.247886+0000","last_peered":"2024-09-10T18:37:33.247886+0000","last_clean":"2024-09-10T18:37:33.247886+0000","last_became_active":"2024-09-10T18:26:27.651364+0000","last_became_peered":"2024-09-10T18:26:27.651364+0000","last_unstale":"2024-09-10T18:37:33.247886+0000","last_undegraded":"2024-09-10T18:37:33.247886+0000","last_fullsized":"2024-09-10T18:37:33.247886+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'28","last_scrub_stamp":"2024-09-10T18:36:16.155287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:16.155287+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027352050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":361,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.248457+0000","last_change":"2024-09-10T18:36:09.716889+0000","last_active":"2024-09-10T18:37:33.248457+0000","last_peered":"2024-09-10T18:37:33.248457+0000","last_clean":"2024-09-10T18:37:33.248457+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:37:33.248457+0000","last_undegraded":"2024-09-10T18:37:33.248457+0000","last_fullsized":"2024-09-10T18:37:33.248457+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:09.716820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:36:09.716820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:34.441003+0000","scrub_duration":0.0037024689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":550,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:34.295650+0000","last_change":"2024-09-10T18:36:13.682715+0000","last_active":"2024-09-10T18:37:34.295650+0000","last_peered":"2024-09-10T18:37:34.295650+0000","last_clean":"2024-09-10T18:37:34.295650+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:37:34.295650+0000","last_undegraded":"2024-09-10T18:37:34.295650+0000","last_fullsized":"2024-09-10T18:37:34.295650+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:36:13.682649+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:36:13.682649+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001699954,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":363,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.237653+0000","last_change":"2024-09-10T18:36:53.979146+0000","last_active":"2024-09-10T18:37:33.237653+0000","last_peered":"2024-09-10T18:37:33.237653+0000","last_clean":"2024-09-10T18:37:33.237653+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:37:33.237653+0000","last_undegraded":"2024-09-10T18:37:33.237653+0000","last_fullsized":"2024-09-10T18:37:33.237653+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:53.979093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:36:53.979093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:15.772937+0000","scrub_duration":0.0047648930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"341'7894","reported_seq":7929,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:40.086910+0000","last_change":"2024-09-10T18:36:07.904923+0000","last_active":"2024-09-10T18:37:40.086910+0000","last_peered":"2024-09-10T18:37:40.086910+0000","last_clean":"2024-09-10T18:37:40.086910+0000","last_became_active":"2024-09-10T18:36:07.904606+0000","last_became_peered":"2024-09-10T18:36:07.904606+0000","last_unstale":"2024-09-10T18:37:40.086910+0000","last_undegraded":"2024-09-10T18:37:40.086910+0000","last_fullsized":"2024-09-10T18:37:40.086910+0000","mapping_epoch":314,"log_start":"322'3700","ondisk_log_start":"322'3700","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4194,"log_dups_size":0,"ondisk_log_size":4194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":517341184,"num_objects":7894,"num_object_clones":0,"num_object_copies":23682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7894,"num_write_kb":505216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":356,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.243923+0000","last_change":"2024-09-10T18:36:09.909074+0000","last_active":"2024-09-10T18:37:33.243923+0000","last_peered":"2024-09-10T18:37:33.243923+0000","last_clean":"2024-09-10T18:37:33.243923+0000","last_became_active":"2024-09-10T18:34:32.818323+0000","last_became_peered":"2024-09-10T18:34:32.818323+0000","last_unstale":"2024-09-10T18:37:33.243923+0000","last_undegraded":"2024-09-10T18:37:33.243923+0000","last_fullsized":"2024-09-10T18:37:33.243923+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:09.908960+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:09.908960+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019456180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":413,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.254275+0000","last_change":"2024-09-10T18:37:07.798909+0000","last_active":"2024-09-10T18:37:33.254275+0000","last_peered":"2024-09-10T18:37:33.254275+0000","last_clean":"2024-09-10T18:37:33.254275+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:37:33.254275+0000","last_undegraded":"2024-09-10T18:37:33.254275+0000","last_fullsized":"2024-09-10T18:37:33.254275+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:37:07.798798+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:37:07.798798+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:11.466713+0000","scrub_duration":0.0020202269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":360,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.246241+0000","last_change":"2024-09-10T18:37:11.243616+0000","last_active":"2024-09-10T18:37:33.246241+0000","last_peered":"2024-09-10T18:37:33.246241+0000","last_clean":"2024-09-10T18:37:33.246241+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:37:33.246241+0000","last_undegraded":"2024-09-10T18:37:33.246241+0000","last_fullsized":"2024-09-10T18:37:33.246241+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:11.243548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:37:11.243548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:29.339597+0000","scrub_duration":0.015664832,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"263'11","reported_seq":569,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:34.953833+0000","last_change":"2024-09-10T18:36:51.573695+0000","last_active":"2024-09-10T18:37:34.953833+0000","last_peered":"2024-09-10T18:37:34.953833+0000","last_clean":"2024-09-10T18:37:34.953833+0000","last_became_active":"2024-09-10T18:26:27.639032+0000","last_became_peered":"2024-09-10T18:26:27.639032+0000","last_unstale":"2024-09-10T18:37:34.953833+0000","last_undegraded":"2024-09-10T18:37:34.953833+0000","last_fullsized":"2024-09-10T18:37:34.953833+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'11","last_scrub_stamp":"2024-09-10T18:36:51.573610+0000","last_deep_scrub":"263'11","last_deep_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_clean_scrub_stamp":"2024-09-10T18:36:51.573610+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:06.768796+0000","scrub_duration":0.0034993289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":355,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.240699+0000","last_change":"2024-09-10T18:35:55.632357+0000","last_active":"2024-09-10T18:37:33.240699+0000","last_peered":"2024-09-10T18:37:33.240699+0000","last_clean":"2024-09-10T18:37:33.240699+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:37:33.240699+0000","last_undegraded":"2024-09-10T18:37:33.240699+0000","last_fullsized":"2024-09-10T18:37:33.240699+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:55.632306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:55.632306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001072857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"341'7965","reported_seq":8242,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:40.999287+0000","last_change":"2024-09-10T18:37:12.152091+0000","last_active":"2024-09-10T18:37:40.999287+0000","last_peered":"2024-09-10T18:37:40.999287+0000","last_clean":"2024-09-10T18:37:40.999287+0000","last_became_active":"2024-09-10T18:36:07.903637+0000","last_became_peered":"2024-09-10T18:36:07.903637+0000","last_unstale":"2024-09-10T18:37:40.999287+0000","last_undegraded":"2024-09-10T18:37:40.999287+0000","last_fullsized":"2024-09-10T18:37:40.999287+0000","mapping_epoch":314,"log_start":"322'3000","ondisk_log_start":"322'3000","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'5893","last_scrub_stamp":"2024-09-10T18:37:12.151975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:37:12.151975+0000","objects_scrubbed":5802,"log_size":4965,"log_dups_size":0,"ondisk_log_size":4965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:14.669808+0000","scrub_duration":2.3527053489999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521994240,"num_objects":7965,"num_object_clones":0,"num_object_copies":23895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7965,"num_write_kb":509760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":356,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.243478+0000","last_change":"2024-09-10T18:36:08.872676+0000","last_active":"2024-09-10T18:37:33.243478+0000","last_peered":"2024-09-10T18:37:33.243478+0000","last_clean":"2024-09-10T18:37:33.243478+0000","last_became_active":"2024-09-10T18:34:32.918946+0000","last_became_peered":"2024-09-10T18:34:32.918946+0000","last_unstale":"2024-09-10T18:37:33.243478+0000","last_undegraded":"2024-09-10T18:37:33.243478+0000","last_fullsized":"2024-09-10T18:37:33.243478+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:08.872575+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:08.872575+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016711530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"240'29","reported_seq":420,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.247102+0000","last_change":"2024-09-10T18:36:59.647785+0000","last_active":"2024-09-10T18:37:33.247102+0000","last_peered":"2024-09-10T18:37:33.247102+0000","last_clean":"2024-09-10T18:37:33.247102+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:37:33.247102+0000","last_undegraded":"2024-09-10T18:37:33.247102+0000","last_fullsized":"2024-09-10T18:37:33.247102+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'29","last_scrub_stamp":"2024-09-10T18:36:59.647532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:59.647532+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:24.657907+0000","scrub_duration":0.0067920380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":360,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.253125+0000","last_change":"2024-09-10T18:36:06.595223+0000","last_active":"2024-09-10T18:37:33.253125+0000","last_peered":"2024-09-10T18:37:33.253125+0000","last_clean":"2024-09-10T18:37:33.253125+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:37:33.253125+0000","last_undegraded":"2024-09-10T18:37:33.253125+0000","last_fullsized":"2024-09-10T18:37:33.253125+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.595159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.595159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:34.639380+0000","scrub_duration":0.0014987100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":326,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.242669+0000","last_change":"2024-09-10T18:37:12.856158+0000","last_active":"2024-09-10T18:37:33.242669+0000","last_peered":"2024-09-10T18:37:33.242669+0000","last_clean":"2024-09-10T18:37:33.242669+0000","last_became_active":"2024-09-10T18:26:27.642063+0000","last_became_peered":"2024-09-10T18:26:27.642063+0000","last_unstale":"2024-09-10T18:37:33.242669+0000","last_undegraded":"2024-09-10T18:37:33.242669+0000","last_fullsized":"2024-09-10T18:37:33.242669+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:12.856068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:12.856068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:14.246369+0000","scrub_duration":0.020901162000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":371,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:36.899126+0000","last_change":"2024-09-10T18:37:36.899082+0000","last_active":"2024-09-10T18:37:36.899126+0000","last_peered":"2024-09-10T18:37:36.899126+0000","last_clean":"2024-09-10T18:37:36.899126+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:37:36.899126+0000","last_undegraded":"2024-09-10T18:37:36.899126+0000","last_fullsized":"2024-09-10T18:37:36.899126+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:36.898949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:37:36.898949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:02.655713+0000","scrub_duration":0.0023913269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"341'7957","reported_seq":7992,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:41.031102+0000","last_change":"2024-09-10T18:36:07.903723+0000","last_active":"2024-09-10T18:37:41.031102+0000","last_peered":"2024-09-10T18:37:41.031102+0000","last_clean":"2024-09-10T18:37:41.031102+0000","last_became_active":"2024-09-10T18:36:07.903339+0000","last_became_peered":"2024-09-10T18:36:07.903339+0000","last_unstale":"2024-09-10T18:37:41.031102+0000","last_undegraded":"2024-09-10T18:37:41.031102+0000","last_fullsized":"2024-09-10T18:37:41.031102+0000","mapping_epoch":314,"log_start":"322'3000","ondisk_log_start":"322'3000","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521469952,"num_objects":7957,"num_object_clones":0,"num_object_copies":23871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7957,"num_write_kb":509248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":271,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.245350+0000","last_change":"2024-09-10T18:36:55.135378+0000","last_active":"2024-09-10T18:37:33.245350+0000","last_peered":"2024-09-10T18:37:33.245350+0000","last_clean":"2024-09-10T18:37:33.245350+0000","last_became_active":"2024-09-10T18:34:00.151017+0000","last_became_peered":"2024-09-10T18:34:00.151017+0000","last_unstale":"2024-09-10T18:37:33.245350+0000","last_undegraded":"2024-09-10T18:37:33.245350+0000","last_fullsized":"2024-09-10T18:37:33.245350+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:55.135191+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:55.135191+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:04.408840+0000","scrub_duration":0.0072107639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"318'42","reported_seq":83,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:34.952331+0000","last_change":"2024-09-10T18:37:34.820009+0000","last_active":"2024-09-10T18:37:34.952331+0000","last_peered":"2024-09-10T18:37:34.952331+0000","last_clean":"2024-09-10T18:37:34.952331+0000","last_became_active":"2024-09-10T18:36:12.922665+0000","last_became_peered":"2024-09-10T18:36:12.922665+0000","last_unstale":"2024-09-10T18:37:34.952331+0000","last_undegraded":"2024-09-10T18:37:34.952331+0000","last_fullsized":"2024-09-10T18:37:34.952331+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'42","last_scrub_stamp":"2024-09-10T18:37:34.819731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:34.819731+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:47.157368+0000","scrub_duration":0.004494156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":366,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.248303+0000","last_change":"2024-09-10T18:37:01.274315+0000","last_active":"2024-09-10T18:37:33.248303+0000","last_peered":"2024-09-10T18:37:33.248303+0000","last_clean":"2024-09-10T18:37:33.248303+0000","last_became_active":"2024-09-10T18:26:27.648690+0000","last_became_peered":"2024-09-10T18:26:27.648690+0000","last_unstale":"2024-09-10T18:37:33.248303+0000","last_undegraded":"2024-09-10T18:37:33.248303+0000","last_fullsized":"2024-09-10T18:37:33.248303+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:01.274219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.274219+0000","last_clean_scrub_stamp":"2024-09-10T18:37:01.274219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:17.545348+0000","scrub_duration":0.0059785009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":375,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.246529+0000","last_change":"2024-09-10T18:35:55.505988+0000","last_active":"2024-09-10T18:37:33.246529+0000","last_peered":"2024-09-10T18:37:33.246529+0000","last_clean":"2024-09-10T18:37:33.246529+0000","last_became_active":"2024-09-10T18:26:27.647576+0000","last_became_peered":"2024-09-10T18:26:27.647576+0000","last_unstale":"2024-09-10T18:37:33.246529+0000","last_undegraded":"2024-09-10T18:37:33.246529+0000","last_fullsized":"2024-09-10T18:37:33.246529+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:35:55.505916+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:35:55.505916+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001708209,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":357,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.249226+0000","last_change":"2024-09-10T18:36:02.734328+0000","last_active":"2024-09-10T18:37:33.249226+0000","last_peered":"2024-09-10T18:37:33.249226+0000","last_clean":"2024-09-10T18:37:33.249226+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:37:33.249226+0000","last_undegraded":"2024-09-10T18:37:33.249226+0000","last_fullsized":"2024-09-10T18:37:33.249226+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:02.734269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:02.734269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016180000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":346,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:37.700613+0000","last_change":"2024-09-10T18:37:37.700573+0000","last_active":"2024-09-10T18:37:37.700613+0000","last_peered":"2024-09-10T18:37:37.700613+0000","last_clean":"2024-09-10T18:37:37.700613+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:37:37.700613+0000","last_undegraded":"2024-09-10T18:37:37.700613+0000","last_fullsized":"2024-09-10T18:37:37.700613+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:37.700481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:37.700481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:54.313491+0000","scrub_duration":0.0022681839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"341'8024","reported_seq":8059,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:40.125088+0000","last_change":"2024-09-10T18:36:07.906590+0000","last_active":"2024-09-10T18:37:40.125088+0000","last_peered":"2024-09-10T18:37:40.125088+0000","last_clean":"2024-09-10T18:37:40.125088+0000","last_became_active":"2024-09-10T18:36:07.906264+0000","last_became_peered":"2024-09-10T18:36:07.906264+0000","last_unstale":"2024-09-10T18:37:40.125088+0000","last_undegraded":"2024-09-10T18:37:40.125088+0000","last_fullsized":"2024-09-10T18:37:40.125088+0000","mapping_epoch":314,"log_start":"322'3451","ondisk_log_start":"322'3451","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4573,"log_dups_size":0,"ondisk_log_size":4573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":525860864,"num_objects":8024,"num_object_clones":0,"num_object_copies":24072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8024,"num_write_kb":513536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":407,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.237329+0000","last_change":"2024-09-10T18:35:08.748174+0000","last_active":"2024-09-10T18:37:33.237329+0000","last_peered":"2024-09-10T18:37:33.237329+0000","last_clean":"2024-09-10T18:37:33.237329+0000","last_became_active":"2024-09-10T18:34:00.154404+0000","last_became_peered":"2024-09-10T18:34:00.154404+0000","last_unstale":"2024-09-10T18:37:33.237329+0000","last_undegraded":"2024-09-10T18:37:33.237329+0000","last_fullsized":"2024-09-10T18:37:33.237329+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:08.748082+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:08.748082+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038002869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"240'17","reported_seq":401,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.253063+0000","last_change":"2024-09-10T18:36:49.797325+0000","last_active":"2024-09-10T18:37:33.253063+0000","last_peered":"2024-09-10T18:37:33.253063+0000","last_clean":"2024-09-10T18:37:33.253063+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:37:33.253063+0000","last_undegraded":"2024-09-10T18:37:33.253063+0000","last_fullsized":"2024-09-10T18:37:33.253063+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'17","last_scrub_stamp":"2024-09-10T18:36:49.797243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:49.797243+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:53.680622+0000","scrub_duration":0.0030959260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":377,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.252021+0000","last_change":"2024-09-10T18:35:57.626741+0000","last_active":"2024-09-10T18:37:33.252021+0000","last_peered":"2024-09-10T18:37:33.252021+0000","last_clean":"2024-09-10T18:37:33.252021+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:37:33.252021+0000","last_undegraded":"2024-09-10T18:37:33.252021+0000","last_fullsized":"2024-09-10T18:37:33.252021+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:57.626681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:15.141982+0000","last_clean_scrub_stamp":"2024-09-10T18:35:57.626681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001556663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":398,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.249464+0000","last_change":"2024-09-10T18:37:04.920700+0000","last_active":"2024-09-10T18:37:33.249464+0000","last_peered":"2024-09-10T18:37:33.249464+0000","last_clean":"2024-09-10T18:37:33.249464+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:37:33.249464+0000","last_undegraded":"2024-09-10T18:37:33.249464+0000","last_fullsized":"2024-09-10T18:37:33.249464+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:37:04.920615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:04.920615+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:20.515535+0000","scrub_duration":0.01495838,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":375,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.250127+0000","last_change":"2024-09-10T18:36:48.839724+0000","last_active":"2024-09-10T18:37:33.250127+0000","last_peered":"2024-09-10T18:37:33.250127+0000","last_clean":"2024-09-10T18:37:33.250127+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:37:33.250127+0000","last_undegraded":"2024-09-10T18:37:33.250127+0000","last_fullsized":"2024-09-10T18:37:33.250127+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:48.839647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:36:48.839647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:55.570968+0000","scrub_duration":0.0072244579999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":363,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.239648+0000","last_change":"2024-09-10T18:37:02.114698+0000","last_active":"2024-09-10T18:37:33.239648+0000","last_peered":"2024-09-10T18:37:33.239648+0000","last_clean":"2024-09-10T18:37:33.239648+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:37:33.239648+0000","last_undegraded":"2024-09-10T18:37:33.239648+0000","last_fullsized":"2024-09-10T18:37:33.239648+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:02.114602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:02.114602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:21.588925+0000","scrub_duration":0.0019755279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"341'7979","reported_seq":8014,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:41.268065+0000","last_change":"2024-09-10T18:36:07.902359+0000","last_active":"2024-09-10T18:37:41.268065+0000","last_peered":"2024-09-10T18:37:41.268065+0000","last_clean":"2024-09-10T18:37:41.268065+0000","last_became_active":"2024-09-10T18:36:07.901516+0000","last_became_peered":"2024-09-10T18:36:07.901516+0000","last_unstale":"2024-09-10T18:37:41.268065+0000","last_undegraded":"2024-09-10T18:37:41.268065+0000","last_fullsized":"2024-09-10T18:37:41.268065+0000","mapping_epoch":314,"log_start":"322'3500","ondisk_log_start":"322'3500","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4479,"log_dups_size":0,"ondisk_log_size":4479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522911744,"num_objects":7979,"num_object_clones":0,"num_object_copies":23937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7979,"num_write_kb":510656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":287,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.257828+0000","last_change":"2024-09-10T18:36:59.835814+0000","last_active":"2024-09-10T18:37:33.257828+0000","last_peered":"2024-09-10T18:37:33.257828+0000","last_clean":"2024-09-10T18:37:33.257828+0000","last_became_active":"2024-09-10T18:34:00.165384+0000","last_became_peered":"2024-09-10T18:34:00.165384+0000","last_unstale":"2024-09-10T18:37:33.257828+0000","last_undegraded":"2024-09-10T18:37:33.257828+0000","last_fullsized":"2024-09-10T18:37:33.257828+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:59.835713+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:59.835713+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:16.626287+0000","scrub_duration":0.029401394000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":395,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.248309+0000","last_change":"2024-09-10T18:36:15.636499+0000","last_active":"2024-09-10T18:37:33.248309+0000","last_peered":"2024-09-10T18:37:33.248309+0000","last_clean":"2024-09-10T18:37:33.248309+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:37:33.248309+0000","last_undegraded":"2024-09-10T18:37:33.248309+0000","last_fullsized":"2024-09-10T18:37:33.248309+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-09-10T18:36:15.636442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:15.636442+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:35.680984+0000","scrub_duration":0.0017190630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":371,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.246165+0000","last_change":"2024-09-10T18:36:55.603164+0000","last_active":"2024-09-10T18:37:33.246165+0000","last_peered":"2024-09-10T18:37:33.246165+0000","last_clean":"2024-09-10T18:37:33.246165+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:37:33.246165+0000","last_undegraded":"2024-09-10T18:37:33.246165+0000","last_fullsized":"2024-09-10T18:37:33.246165+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:55.603085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:36:55.603085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:23.221490+0000","scrub_duration":0.012495984999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":1337,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:38.097220+0000","last_change":"2024-09-10T18:37:01.094318+0000","last_active":"2024-09-10T18:37:38.097220+0000","last_peered":"2024-09-10T18:37:38.097220+0000","last_clean":"2024-09-10T18:37:38.097220+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:37:38.097220+0000","last_undegraded":"2024-09-10T18:37:38.097220+0000","last_fullsized":"2024-09-10T18:37:38.097220+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:37:01.094063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:01.094063+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:23.788462+0000","scrub_duration":0.0099775509999999994,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":960,"num_read_kb":955,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":359,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.247531+0000","last_change":"2024-09-10T18:37:01.170467+0000","last_active":"2024-09-10T18:37:33.247531+0000","last_peered":"2024-09-10T18:37:33.247531+0000","last_clean":"2024-09-10T18:37:33.247531+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:37:33.247531+0000","last_undegraded":"2024-09-10T18:37:33.247531+0000","last_fullsized":"2024-09-10T18:37:33.247531+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:01.170272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.170272+0000","last_clean_scrub_stamp":"2024-09-10T18:37:01.170272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:05.863430+0000","scrub_duration":0.0065986719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":354,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.252746+0000","last_change":"2024-09-10T18:36:17.593856+0000","last_active":"2024-09-10T18:37:33.252746+0000","last_peered":"2024-09-10T18:37:33.252746+0000","last_clean":"2024-09-10T18:37:33.252746+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:37:33.252746+0000","last_undegraded":"2024-09-10T18:37:33.252746+0000","last_fullsized":"2024-09-10T18:37:33.252746+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:17.593802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:17.593802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032348469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"341'7929","reported_seq":7964,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:41.019354+0000","last_change":"2024-09-10T18:36:07.903898+0000","last_active":"2024-09-10T18:37:41.019354+0000","last_peered":"2024-09-10T18:37:41.019354+0000","last_clean":"2024-09-10T18:37:41.019354+0000","last_became_active":"2024-09-10T18:36:07.903576+0000","last_became_peered":"2024-09-10T18:36:07.903576+0000","last_unstale":"2024-09-10T18:37:41.019354+0000","last_undegraded":"2024-09-10T18:37:41.019354+0000","last_fullsized":"2024-09-10T18:37:41.019354+0000","mapping_epoch":314,"log_start":"322'3000","ondisk_log_start":"322'3000","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4929,"log_dups_size":0,"ondisk_log_size":4929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":519634944,"num_objects":7929,"num_object_clones":0,"num_object_copies":23787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7929,"num_write_kb":507456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"240'26","reported_seq":363,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.681259+0000","last_change":"2024-09-10T18:37:33.681193+0000","last_active":"2024-09-10T18:37:33.681259+0000","last_peered":"2024-09-10T18:37:33.681259+0000","last_clean":"2024-09-10T18:37:33.681259+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:37:33.681259+0000","last_undegraded":"2024-09-10T18:37:33.681259+0000","last_fullsized":"2024-09-10T18:37:33.681259+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'26","last_scrub_stamp":"2024-09-10T18:37:33.681108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:33.681108+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:01.752280+0000","scrub_duration":0.0099252360000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":359,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.233258+0000","last_change":"2024-09-10T18:34:19.296515+0000","last_active":"2024-09-10T18:37:33.233258+0000","last_peered":"2024-09-10T18:37:33.233258+0000","last_clean":"2024-09-10T18:37:33.233258+0000","last_became_active":"2024-09-10T18:26:27.639129+0000","last_became_peered":"2024-09-10T18:26:27.639129+0000","last_unstale":"2024-09-10T18:37:33.233258+0000","last_undegraded":"2024-09-10T18:37:33.233258+0000","last_fullsized":"2024-09-10T18:37:33.233258+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:19.296286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:34:19.296286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049170170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":429,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.238788+0000","last_change":"2024-09-10T18:37:11.982261+0000","last_active":"2024-09-10T18:37:33.238788+0000","last_peered":"2024-09-10T18:37:33.238788+0000","last_clean":"2024-09-10T18:37:33.238788+0000","last_became_active":"2024-09-10T18:34:00.159975+0000","last_became_peered":"2024-09-10T18:34:00.159975+0000","last_unstale":"2024-09-10T18:37:33.238788+0000","last_undegraded":"2024-09-10T18:37:33.238788+0000","last_fullsized":"2024-09-10T18:37:33.238788+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:11.982173+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:11.982173+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:40.227262+0000","scrub_duration":0.001824406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":365,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.238513+0000","last_change":"2024-09-10T18:36:58.015546+0000","last_active":"2024-09-10T18:37:33.238513+0000","last_peered":"2024-09-10T18:37:33.238513+0000","last_clean":"2024-09-10T18:37:33.238513+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:37:33.238513+0000","last_undegraded":"2024-09-10T18:37:33.238513+0000","last_fullsized":"2024-09-10T18:37:33.238513+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:58.015470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:58.015470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:03.398913+0000","scrub_duration":0.0043498190000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":357,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.247417+0000","last_change":"2024-09-10T18:36:59.120950+0000","last_active":"2024-09-10T18:37:33.247417+0000","last_peered":"2024-09-10T18:37:33.247417+0000","last_clean":"2024-09-10T18:37:33.247417+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:37:33.247417+0000","last_undegraded":"2024-09-10T18:37:33.247417+0000","last_fullsized":"2024-09-10T18:37:33.247417+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:59.120687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:36:59.120687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:13.759657+0000","scrub_duration":0.0049055239999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"341'8101","reported_seq":8136,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:41.019514+0000","last_change":"2024-09-10T18:36:07.903096+0000","last_active":"2024-09-10T18:37:41.019514+0000","last_peered":"2024-09-10T18:37:41.019514+0000","last_clean":"2024-09-10T18:37:41.019514+0000","last_became_active":"2024-09-10T18:36:07.902779+0000","last_became_peered":"2024-09-10T18:36:07.902779+0000","last_unstale":"2024-09-10T18:37:41.019514+0000","last_undegraded":"2024-09-10T18:37:41.019514+0000","last_fullsized":"2024-09-10T18:37:41.019514+0000","mapping_epoch":314,"log_start":"322'3100","ondisk_log_start":"322'3100","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":5001,"log_dups_size":0,"ondisk_log_size":5001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":530907136,"num_objects":8101,"num_object_clones":0,"num_object_copies":24303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8101,"num_write_kb":518464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"76'81","reported_seq":306,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:37.918727+0000","last_change":"2024-09-10T18:37:37.918636+0000","last_active":"2024-09-10T18:37:37.918727+0000","last_peered":"2024-09-10T18:37:37.918727+0000","last_clean":"2024-09-10T18:37:37.918727+0000","last_became_active":"2024-09-10T18:34:32.927460+0000","last_became_peered":"2024-09-10T18:34:32.927460+0000","last_unstale":"2024-09-10T18:37:37.918727+0000","last_undegraded":"2024-09-10T18:37:37.918727+0000","last_fullsized":"2024-09-10T18:37:37.918727+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:37.918433+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:37.918433+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:06.609650+0000","scrub_duration":0.0051962880000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"283'24","reported_seq":403,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.249584+0000","last_change":"2024-09-10T18:36:58.225291+0000","last_active":"2024-09-10T18:37:33.249584+0000","last_peered":"2024-09-10T18:37:33.249584+0000","last_clean":"2024-09-10T18:37:33.249584+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:37:33.249584+0000","last_undegraded":"2024-09-10T18:37:33.249584+0000","last_fullsized":"2024-09-10T18:37:33.249584+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'24","last_scrub_stamp":"2024-09-10T18:36:58.225206+0000","last_deep_scrub":"283'24","last_deep_scrub_stamp":"2024-09-10T18:34:19.404713+0000","last_clean_scrub_stamp":"2024-09-10T18:36:58.225206+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:16.467424+0000","scrub_duration":0.0055050170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":359,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.248563+0000","last_change":"2024-09-10T18:37:00.246542+0000","last_active":"2024-09-10T18:37:33.248563+0000","last_peered":"2024-09-10T18:37:33.248563+0000","last_clean":"2024-09-10T18:37:33.248563+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:37:33.248563+0000","last_undegraded":"2024-09-10T18:37:33.248563+0000","last_fullsized":"2024-09-10T18:37:33.248563+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:00.246398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:00.246398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:00.862550+0000","scrub_duration":0.003194966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":373,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:35.388642+0000","last_change":"2024-09-10T18:37:35.385548+0000","last_active":"2024-09-10T18:37:35.388642+0000","last_peered":"2024-09-10T18:37:35.388642+0000","last_clean":"2024-09-10T18:37:35.388642+0000","last_became_active":"2024-09-10T18:26:27.645885+0000","last_became_peered":"2024-09-10T18:26:27.645885+0000","last_unstale":"2024-09-10T18:37:35.388642+0000","last_undegraded":"2024-09-10T18:37:35.388642+0000","last_fullsized":"2024-09-10T18:37:35.388642+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:35.385189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:37:35.385189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:59.661914+0000","scrub_duration":0.0083152009999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":363,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:34.723675+0000","last_change":"2024-09-10T18:37:34.723627+0000","last_active":"2024-09-10T18:37:34.723675+0000","last_peered":"2024-09-10T18:37:34.723675+0000","last_clean":"2024-09-10T18:37:34.723675+0000","last_became_active":"2024-09-10T18:26:27.632071+0000","last_became_peered":"2024-09-10T18:26:27.632071+0000","last_unstale":"2024-09-10T18:37:34.723675+0000","last_undegraded":"2024-09-10T18:37:34.723675+0000","last_fullsized":"2024-09-10T18:37:34.723675+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:34.723544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:34.723544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:04.166049+0000","scrub_duration":0.045211744999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"341'7820","reported_seq":7855,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:40.247188+0000","last_change":"2024-09-10T18:36:07.903712+0000","last_active":"2024-09-10T18:37:40.247188+0000","last_peered":"2024-09-10T18:37:40.247188+0000","last_clean":"2024-09-10T18:37:40.247188+0000","last_became_active":"2024-09-10T18:36:07.903429+0000","last_became_peered":"2024-09-10T18:36:07.903429+0000","last_unstale":"2024-09-10T18:37:40.247188+0000","last_undegraded":"2024-09-10T18:37:40.247188+0000","last_fullsized":"2024-09-10T18:37:40.247188+0000","mapping_epoch":314,"log_start":"320'2000","ondisk_log_start":"320'2000","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":5820,"log_dups_size":0,"ondisk_log_size":5820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512491520,"num_objects":7820,"num_object_clones":0,"num_object_copies":23460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7820,"num_write_kb":500480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":278,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.246283+0000","last_change":"2024-09-10T18:36:01.514457+0000","last_active":"2024-09-10T18:37:33.246283+0000","last_peered":"2024-09-10T18:37:33.246283+0000","last_clean":"2024-09-10T18:37:33.246283+0000","last_became_active":"2024-09-10T18:34:32.823976+0000","last_became_peered":"2024-09-10T18:34:32.823976+0000","last_unstale":"2024-09-10T18:37:33.246283+0000","last_undegraded":"2024-09-10T18:37:33.246283+0000","last_fullsized":"2024-09-10T18:37:33.246283+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:01.514369+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:01.514369+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001652136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":434,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.249430+0000","last_change":"2024-09-10T18:37:11.832716+0000","last_active":"2024-09-10T18:37:33.249430+0000","last_peered":"2024-09-10T18:37:33.249430+0000","last_clean":"2024-09-10T18:37:33.249430+0000","last_became_active":"2024-09-10T18:37:11.817595+0000","last_became_peered":"2024-09-10T18:37:11.817595+0000","last_unstale":"2024-09-10T18:37:33.249430+0000","last_undegraded":"2024-09-10T18:37:33.249430+0000","last_fullsized":"2024-09-10T18:37:33.249430+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-09-10T18:36:02.364698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:02.364698+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025656849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":324,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.243304+0000","last_change":"2024-09-10T18:36:00.897438+0000","last_active":"2024-09-10T18:37:33.243304+0000","last_peered":"2024-09-10T18:37:33.243304+0000","last_clean":"2024-09-10T18:37:33.243304+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:37:33.243304+0000","last_undegraded":"2024-09-10T18:37:33.243304+0000","last_fullsized":"2024-09-10T18:37:33.243304+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:36:00.897355+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:36:00.897355+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020072419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":366,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.252819+0000","last_change":"2024-09-10T18:36:57.784141+0000","last_active":"2024-09-10T18:37:33.252819+0000","last_peered":"2024-09-10T18:37:33.252819+0000","last_clean":"2024-09-10T18:37:33.252819+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:37:33.252819+0000","last_undegraded":"2024-09-10T18:37:33.252819+0000","last_fullsized":"2024-09-10T18:37:33.252819+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:57.784049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:57.784049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:15.415282+0000","scrub_duration":0.0022794579999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":383,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.249664+0000","last_change":"2024-09-10T18:36:04.465752+0000","last_active":"2024-09-10T18:37:33.249664+0000","last_peered":"2024-09-10T18:37:33.249664+0000","last_clean":"2024-09-10T18:37:33.249664+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:37:33.249664+0000","last_undegraded":"2024-09-10T18:37:33.249664+0000","last_fullsized":"2024-09-10T18:37:33.249664+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:04.465694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:04.465694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014277929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"341'8145","reported_seq":8180,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:40.495025+0000","last_change":"2024-09-10T18:36:07.904133+0000","last_active":"2024-09-10T18:37:40.495025+0000","last_peered":"2024-09-10T18:37:40.495025+0000","last_clean":"2024-09-10T18:37:40.495025+0000","last_became_active":"2024-09-10T18:36:07.903825+0000","last_became_peered":"2024-09-10T18:36:07.903825+0000","last_unstale":"2024-09-10T18:37:40.495025+0000","last_undegraded":"2024-09-10T18:37:40.495025+0000","last_fullsized":"2024-09-10T18:37:40.495025+0000","mapping_epoch":314,"log_start":"322'3600","ondisk_log_start":"322'3600","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4545,"log_dups_size":0,"ondisk_log_size":4545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":533790720,"num_objects":8145,"num_object_clones":0,"num_object_copies":24435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8145,"num_write_kb":521280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":417,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.240411+0000","last_change":"2024-09-10T18:36:13.951388+0000","last_active":"2024-09-10T18:37:33.240411+0000","last_peered":"2024-09-10T18:37:33.240411+0000","last_clean":"2024-09-10T18:37:33.240411+0000","last_became_active":"2024-09-10T18:34:32.827174+0000","last_became_peered":"2024-09-10T18:34:32.827174+0000","last_unstale":"2024-09-10T18:37:33.240411+0000","last_undegraded":"2024-09-10T18:37:33.240411+0000","last_fullsized":"2024-09-10T18:37:33.240411+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:13.951321+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:13.951321+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015868600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"240'25","reported_seq":127,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.238208+0000","last_change":"2024-09-10T18:37:14.054480+0000","last_active":"2024-09-10T18:37:33.238208+0000","last_peered":"2024-09-10T18:37:33.238208+0000","last_clean":"2024-09-10T18:37:33.238208+0000","last_became_active":"2024-09-10T18:32:51.118251+0000","last_became_peered":"2024-09-10T18:32:51.118251+0000","last_unstale":"2024-09-10T18:37:33.238208+0000","last_undegraded":"2024-09-10T18:37:33.238208+0000","last_fullsized":"2024-09-10T18:37:33.238208+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:37:14.054170+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:37:14.054170+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:41.275620+0000","scrub_duration":0.019668199000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":352,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.233493+0000","last_change":"2024-09-10T18:36:03.769108+0000","last_active":"2024-09-10T18:37:33.233493+0000","last_peered":"2024-09-10T18:37:33.233493+0000","last_clean":"2024-09-10T18:37:33.233493+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:37:33.233493+0000","last_undegraded":"2024-09-10T18:37:33.233493+0000","last_fullsized":"2024-09-10T18:37:33.233493+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:03.769053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:13.324556+0000","last_clean_scrub_stamp":"2024-09-10T18:36:03.769053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001666634,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"189'9","reported_seq":598,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:34.952816+0000","last_change":"2024-09-10T18:37:10.969567+0000","last_active":"2024-09-10T18:37:34.952816+0000","last_peered":"2024-09-10T18:37:34.952816+0000","last_clean":"2024-09-10T18:37:34.952816+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:37:34.952816+0000","last_undegraded":"2024-09-10T18:37:34.952816+0000","last_fullsized":"2024-09-10T18:37:34.952816+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'9","last_scrub_stamp":"2024-09-10T18:37:10.969454+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:37:10.969454+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:26.277589+0000","scrub_duration":0.0024896530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":346,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.249333+0000","last_change":"2024-09-10T18:35:07.288756+0000","last_active":"2024-09-10T18:37:33.249333+0000","last_peered":"2024-09-10T18:37:33.249333+0000","last_clean":"2024-09-10T18:37:33.249333+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:37:33.249333+0000","last_undegraded":"2024-09-10T18:37:33.249333+0000","last_fullsized":"2024-09-10T18:37:33.249333+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:07.288672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_clean_scrub_stamp":"2024-09-10T18:35:07.288672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041335299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"341'7954","reported_seq":8247,"reported_epoch":341,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T18:37:40.273319+0000","last_change":"2024-09-10T18:37:37.953923+0000","last_active":"2024-09-10T18:37:40.273319+0000","last_peered":"2024-09-10T18:37:40.273319+0000","last_clean":"2024-09-10T18:37:40.273319+0000","last_became_active":"2024-09-10T18:36:07.903638+0000","last_became_peered":"2024-09-10T18:36:07.903638+0000","last_unstale":"2024-09-10T18:37:40.273319+0000","last_undegraded":"2024-09-10T18:37:40.273319+0000","last_fullsized":"2024-09-10T18:37:40.273319+0000","mapping_epoch":314,"log_start":"320'2100","ondisk_log_start":"320'2100","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":6349,"log_size":5854,"log_dups_size":0,"ondisk_log_size":5854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521273344,"num_objects":7954,"num_object_clones":0,"num_object_copies":23862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7954,"num_write_kb":509056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"76'81","reported_seq":89,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.249320+0000","last_change":"2024-09-10T18:35:58.314505+0000","last_active":"2024-09-10T18:37:33.249320+0000","last_peered":"2024-09-10T18:37:33.249320+0000","last_clean":"2024-09-10T18:37:33.249320+0000","last_became_active":"2024-09-10T18:34:32.818251+0000","last_became_peered":"2024-09-10T18:34:32.818251+0000","last_unstale":"2024-09-10T18:37:33.249320+0000","last_undegraded":"2024-09-10T18:37:33.249320+0000","last_fullsized":"2024-09-10T18:37:33.249320+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:58.314438+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:58.314438+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015006209999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"302'19","reported_seq":388,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.247018+0000","last_change":"2024-09-10T18:37:08.823694+0000","last_active":"2024-09-10T18:37:33.247018+0000","last_peered":"2024-09-10T18:37:33.247018+0000","last_clean":"2024-09-10T18:37:33.247018+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:37:33.247018+0000","last_undegraded":"2024-09-10T18:37:33.247018+0000","last_fullsized":"2024-09-10T18:37:33.247018+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'19","last_scrub_stamp":"2024-09-10T18:37:08.823440+0000","last_deep_scrub":"302'19","last_deep_scrub_stamp":"2024-09-10T18:35:18.836131+0000","last_clean_scrub_stamp":"2024-09-10T18:37:08.823440+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:19.183000+0000","scrub_duration":0.012806522000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"265'2","reported_seq":387,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:37.448237+0000","last_change":"2024-09-10T18:37:37.448192+0000","last_active":"2024-09-10T18:37:37.448237+0000","last_peered":"2024-09-10T18:37:37.448237+0000","last_clean":"2024-09-10T18:37:37.448237+0000","last_became_active":"2024-09-10T18:26:27.644997+0000","last_became_peered":"2024-09-10T18:26:27.644997+0000","last_unstale":"2024-09-10T18:37:37.448237+0000","last_undegraded":"2024-09-10T18:37:37.448237+0000","last_fullsized":"2024-09-10T18:37:37.448237+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'2","last_scrub_stamp":"2024-09-10T18:37:37.448103+0000","last_deep_scrub":"265'2","last_deep_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_clean_scrub_stamp":"2024-09-10T18:37:37.448103+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:01.900653+0000","scrub_duration":0.011219089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":381,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.244679+0000","last_change":"2024-09-10T18:36:56.798207+0000","last_active":"2024-09-10T18:37:33.244679+0000","last_peered":"2024-09-10T18:37:33.244679+0000","last_clean":"2024-09-10T18:37:33.244679+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:37:33.244679+0000","last_undegraded":"2024-09-10T18:37:33.244679+0000","last_fullsized":"2024-09-10T18:37:33.244679+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:56.798117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:56.798117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:14.975136+0000","scrub_duration":0.0021655960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":355,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.246089+0000","last_change":"2024-09-10T18:36:14.665247+0000","last_active":"2024-09-10T18:37:33.246089+0000","last_peered":"2024-09-10T18:37:33.246089+0000","last_clean":"2024-09-10T18:37:33.246089+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:37:33.246089+0000","last_undegraded":"2024-09-10T18:37:33.246089+0000","last_fullsized":"2024-09-10T18:37:33.246089+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:14.665183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:14.665183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014290399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"341'7978","reported_seq":8013,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:41.241979+0000","last_change":"2024-09-10T18:36:07.902622+0000","last_active":"2024-09-10T18:37:41.241979+0000","last_peered":"2024-09-10T18:37:41.241979+0000","last_clean":"2024-09-10T18:37:41.241979+0000","last_became_active":"2024-09-10T18:36:07.902336+0000","last_became_peered":"2024-09-10T18:36:07.902336+0000","last_unstale":"2024-09-10T18:37:41.241979+0000","last_undegraded":"2024-09-10T18:37:41.241979+0000","last_fullsized":"2024-09-10T18:37:41.241979+0000","mapping_epoch":314,"log_start":"322'3500","ondisk_log_start":"322'3500","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4478,"log_dups_size":0,"ondisk_log_size":4478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522846208,"num_objects":7978,"num_object_clones":0,"num_object_copies":23934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7978,"num_write_kb":510592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'81","reported_seq":345,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.243311+0000","last_change":"2024-09-10T18:36:04.870940+0000","last_active":"2024-09-10T18:37:33.243311+0000","last_peered":"2024-09-10T18:37:33.243311+0000","last_clean":"2024-09-10T18:37:33.243311+0000","last_became_active":"2024-09-10T18:34:02.803861+0000","last_became_peered":"2024-09-10T18:34:02.803861+0000","last_unstale":"2024-09-10T18:37:33.243311+0000","last_undegraded":"2024-09-10T18:37:33.243311+0000","last_fullsized":"2024-09-10T18:37:33.243311+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:04.870861+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:34:15.938079+0000","last_clean_scrub_stamp":"2024-09-10T18:36:04.870861+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001253791,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"240'44","reported_seq":451,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:35.673853+0000","last_change":"2024-09-10T18:37:35.673812+0000","last_active":"2024-09-10T18:37:35.673853+0000","last_peered":"2024-09-10T18:37:35.673853+0000","last_clean":"2024-09-10T18:37:35.673853+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:37:35.673853+0000","last_undegraded":"2024-09-10T18:37:35.673853+0000","last_fullsized":"2024-09-10T18:37:35.673853+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'44","last_scrub_stamp":"2024-09-10T18:37:35.673719+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:37:35.673719+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:56.474842+0000","scrub_duration":0.0023098609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":376,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.239945+0000","last_change":"2024-09-10T18:36:56.003032+0000","last_active":"2024-09-10T18:37:33.239945+0000","last_peered":"2024-09-10T18:37:33.239945+0000","last_clean":"2024-09-10T18:37:33.239945+0000","last_became_active":"2024-09-10T18:26:27.638408+0000","last_became_peered":"2024-09-10T18:26:27.638408+0000","last_unstale":"2024-09-10T18:37:33.239945+0000","last_undegraded":"2024-09-10T18:37:33.239945+0000","last_fullsized":"2024-09-10T18:37:33.239945+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:56.002946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:36:56.002946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:17.122534+0000","scrub_duration":0.0058026170000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":385,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.245142+0000","last_change":"2024-09-10T18:36:00.551970+0000","last_active":"2024-09-10T18:37:33.245142+0000","last_peered":"2024-09-10T18:37:33.245142+0000","last_clean":"2024-09-10T18:37:33.245142+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:37:33.245142+0000","last_undegraded":"2024-09-10T18:37:33.245142+0000","last_fullsized":"2024-09-10T18:37:33.245142+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:00.551910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:34:21.447291+0000","last_clean_scrub_stamp":"2024-09-10T18:36:00.551910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017792039999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":366,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.238712+0000","last_change":"2024-09-10T18:37:04.058735+0000","last_active":"2024-09-10T18:37:33.238712+0000","last_peered":"2024-09-10T18:37:33.238712+0000","last_clean":"2024-09-10T18:37:33.238712+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:37:33.238712+0000","last_undegraded":"2024-09-10T18:37:33.238712+0000","last_fullsized":"2024-09-10T18:37:33.238712+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:04.058646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:04.058646+0000","last_clean_scrub_stamp":"2024-09-10T18:37:04.058646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:21.930918+0000","scrub_duration":0.0034509390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"341'8002","reported_seq":8037,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:40.990988+0000","last_change":"2024-09-10T18:36:07.902995+0000","last_active":"2024-09-10T18:37:40.990988+0000","last_peered":"2024-09-10T18:37:40.990988+0000","last_clean":"2024-09-10T18:37:40.990988+0000","last_became_active":"2024-09-10T18:36:07.902661+0000","last_became_peered":"2024-09-10T18:36:07.902661+0000","last_unstale":"2024-09-10T18:37:40.990988+0000","last_undegraded":"2024-09-10T18:37:40.990988+0000","last_fullsized":"2024-09-10T18:37:40.990988+0000","mapping_epoch":314,"log_start":"322'3000","ondisk_log_start":"322'3000","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":5002,"log_dups_size":0,"ondisk_log_size":5002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524419072,"num_objects":8002,"num_object_clones":0,"num_object_copies":24006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8002,"num_write_kb":512128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"76'81","reported_seq":293,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.233815+0000","last_change":"2024-09-10T18:37:04.959623+0000","last_active":"2024-09-10T18:37:33.233815+0000","last_peered":"2024-09-10T18:37:33.233815+0000","last_clean":"2024-09-10T18:37:33.233815+0000","last_became_active":"2024-09-10T18:34:32.956761+0000","last_became_peered":"2024-09-10T18:34:32.956761+0000","last_unstale":"2024-09-10T18:37:33.233815+0000","last_undegraded":"2024-09-10T18:37:33.233815+0000","last_fullsized":"2024-09-10T18:37:33.233815+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:04.959510+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:04.959510+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:10.242143+0000","scrub_duration":0.0070392069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"301'23","reported_seq":397,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.247572+0000","last_change":"2024-09-10T18:35:15.283425+0000","last_active":"2024-09-10T18:37:33.247572+0000","last_peered":"2024-09-10T18:37:33.247572+0000","last_clean":"2024-09-10T18:37:33.247572+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:37:33.247572+0000","last_undegraded":"2024-09-10T18:37:33.247572+0000","last_fullsized":"2024-09-10T18:37:33.247572+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'23","last_scrub_stamp":"2024-09-10T18:35:15.283373+0000","last_deep_scrub":"301'23","last_deep_scrub_stamp":"2024-09-10T18:35:15.283373+0000","last_clean_scrub_stamp":"2024-09-10T18:35:15.283373+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018617538999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":369,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.240020+0000","last_change":"2024-09-10T18:36:50.894694+0000","last_active":"2024-09-10T18:37:33.240020+0000","last_peered":"2024-09-10T18:37:33.240020+0000","last_clean":"2024-09-10T18:37:33.240020+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:37:33.240020+0000","last_undegraded":"2024-09-10T18:37:33.240020+0000","last_fullsized":"2024-09-10T18:37:33.240020+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:50.894615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:36:50.894615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:04.333933+0000","scrub_duration":0.0038639159999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":395,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.247232+0000","last_change":"2024-09-10T18:37:06.917685+0000","last_active":"2024-09-10T18:37:33.247232+0000","last_peered":"2024-09-10T18:37:33.247232+0000","last_clean":"2024-09-10T18:37:33.247232+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:37:33.247232+0000","last_undegraded":"2024-09-10T18:37:33.247232+0000","last_fullsized":"2024-09-10T18:37:33.247232+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:06.917604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:06.917604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:12.452402+0000","scrub_duration":0.013746744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":346,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.233032+0000","last_change":"2024-09-10T18:36:05.766562+0000","last_active":"2024-09-10T18:37:33.233032+0000","last_peered":"2024-09-10T18:37:33.233032+0000","last_clean":"2024-09-10T18:37:33.233032+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:37:33.233032+0000","last_undegraded":"2024-09-10T18:37:33.233032+0000","last_fullsized":"2024-09-10T18:37:33.233032+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:05.766507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.803236+0000","last_clean_scrub_stamp":"2024-09-10T18:36:05.766507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023023990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"341'7960","reported_seq":7995,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:40.149087+0000","last_change":"2024-09-10T18:36:07.904127+0000","last_active":"2024-09-10T18:37:40.149087+0000","last_peered":"2024-09-10T18:37:40.149087+0000","last_clean":"2024-09-10T18:37:40.149087+0000","last_became_active":"2024-09-10T18:36:07.903791+0000","last_became_peered":"2024-09-10T18:36:07.903791+0000","last_unstale":"2024-09-10T18:37:40.149087+0000","last_undegraded":"2024-09-10T18:37:40.149087+0000","last_fullsized":"2024-09-10T18:37:40.149087+0000","mapping_epoch":314,"log_start":"322'3339","ondisk_log_start":"322'3339","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521666560,"num_objects":7960,"num_object_clones":0,"num_object_copies":23880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7960,"num_write_kb":509440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"76'81","reported_seq":89,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.239498+0000","last_change":"2024-09-10T18:36:11.950457+0000","last_active":"2024-09-10T18:37:33.239498+0000","last_peered":"2024-09-10T18:37:33.239498+0000","last_clean":"2024-09-10T18:37:33.239498+0000","last_became_active":"2024-09-10T18:34:32.822196+0000","last_became_peered":"2024-09-10T18:34:32.822196+0000","last_unstale":"2024-09-10T18:37:33.239498+0000","last_undegraded":"2024-09-10T18:37:33.239498+0000","last_fullsized":"2024-09-10T18:37:33.239498+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:11.950311+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:11.950311+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020040769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":408,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.245163+0000","last_change":"2024-09-10T18:36:54.095746+0000","last_active":"2024-09-10T18:37:33.245163+0000","last_peered":"2024-09-10T18:37:33.245163+0000","last_clean":"2024-09-10T18:37:33.245163+0000","last_became_active":"2024-09-10T18:26:27.645932+0000","last_became_peered":"2024-09-10T18:26:27.645932+0000","last_unstale":"2024-09-10T18:37:33.245163+0000","last_undegraded":"2024-09-10T18:37:33.245163+0000","last_fullsized":"2024-09-10T18:37:33.245163+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:36:54.095659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:54.095659+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:11.075189+0000","scrub_duration":0.012246685,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":369,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.242132+0000","last_change":"2024-09-10T18:35:58.895879+0000","last_active":"2024-09-10T18:37:33.242132+0000","last_peered":"2024-09-10T18:37:33.242132+0000","last_clean":"2024-09-10T18:37:33.242132+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:37:33.242132+0000","last_undegraded":"2024-09-10T18:37:33.242132+0000","last_fullsized":"2024-09-10T18:37:33.242132+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:35:58.895792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:35:58.895792+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021727780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":567,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:34.956887+0000","last_change":"2024-09-10T18:36:02.894810+0000","last_active":"2024-09-10T18:37:34.956887+0000","last_peered":"2024-09-10T18:37:34.956887+0000","last_clean":"2024-09-10T18:37:34.956887+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:37:34.956887+0000","last_undegraded":"2024-09-10T18:37:34.956887+0000","last_fullsized":"2024-09-10T18:37:34.956887+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:36:02.894094+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:36:02.894094+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028441780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":351,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.246226+0000","last_change":"2024-09-10T18:35:53.486754+0000","last_active":"2024-09-10T18:37:33.246226+0000","last_peered":"2024-09-10T18:37:33.246226+0000","last_clean":"2024-09-10T18:37:33.246226+0000","last_became_active":"2024-09-10T18:26:27.647511+0000","last_became_peered":"2024-09-10T18:26:27.647511+0000","last_unstale":"2024-09-10T18:37:33.246226+0000","last_undegraded":"2024-09-10T18:37:33.246226+0000","last_fullsized":"2024-09-10T18:37:33.246226+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:53.486678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:53.486678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035730969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'81","reported_seq":89,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.246384+0000","last_change":"2024-09-10T18:37:14.252272+0000","last_active":"2024-09-10T18:37:33.246384+0000","last_peered":"2024-09-10T18:37:33.246384+0000","last_clean":"2024-09-10T18:37:33.246384+0000","last_became_active":"2024-09-10T18:34:02.810126+0000","last_became_peered":"2024-09-10T18:34:02.810126+0000","last_unstale":"2024-09-10T18:37:33.246384+0000","last_undegraded":"2024-09-10T18:37:33.246384+0000","last_fullsized":"2024-09-10T18:37:33.246384+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:14.252067+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:14.252067+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:18.451121+0000","scrub_duration":0.0039562720000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":315,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.242213+0000","last_change":"2024-09-10T18:35:59.886309+0000","last_active":"2024-09-10T18:37:33.242213+0000","last_peered":"2024-09-10T18:37:33.242213+0000","last_clean":"2024-09-10T18:37:33.242213+0000","last_became_active":"2024-09-10T18:26:27.640922+0000","last_became_peered":"2024-09-10T18:26:27.640922+0000","last_unstale":"2024-09-10T18:37:33.242213+0000","last_undegraded":"2024-09-10T18:37:33.242213+0000","last_fullsized":"2024-09-10T18:37:33.242213+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:59.886242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:35:59.886242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001788724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"240'12","reported_seq":368,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.234526+0000","last_change":"2024-09-10T18:36:00.854061+0000","last_active":"2024-09-10T18:37:33.234526+0000","last_peered":"2024-09-10T18:37:33.234526+0000","last_clean":"2024-09-10T18:37:33.234526+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:37:33.234526+0000","last_undegraded":"2024-09-10T18:37:33.234526+0000","last_fullsized":"2024-09-10T18:37:33.234526+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-10T18:36:00.854004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:00.854004+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019737209999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":355,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.250477+0000","last_change":"2024-09-10T18:36:17.693584+0000","last_active":"2024-09-10T18:37:33.250477+0000","last_peered":"2024-09-10T18:37:33.250477+0000","last_clean":"2024-09-10T18:37:33.250477+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:37:33.250477+0000","last_undegraded":"2024-09-10T18:37:33.250477+0000","last_fullsized":"2024-09-10T18:37:33.250477+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:17.693527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:17.693527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014350840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":360,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.239578+0000","last_change":"2024-09-10T18:36:57.022869+0000","last_active":"2024-09-10T18:37:33.239578+0000","last_peered":"2024-09-10T18:37:33.239578+0000","last_clean":"2024-09-10T18:37:33.239578+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:37:33.239578+0000","last_undegraded":"2024-09-10T18:37:33.239578+0000","last_fullsized":"2024-09-10T18:37:33.239578+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:57.022800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:57.022800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:23.678797+0000","scrub_duration":0.0040564720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'81","reported_seq":296,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.257630+0000","last_change":"2024-09-10T18:37:05.807566+0000","last_active":"2024-09-10T18:37:33.257630+0000","last_peered":"2024-09-10T18:37:33.257630+0000","last_clean":"2024-09-10T18:37:33.257630+0000","last_became_active":"2024-09-10T18:34:02.815880+0000","last_became_peered":"2024-09-10T18:34:02.815880+0000","last_unstale":"2024-09-10T18:37:33.257630+0000","last_undegraded":"2024-09-10T18:37:33.257630+0000","last_fullsized":"2024-09-10T18:37:33.257630+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:05.807459+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:05.807459+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:22.170236+0000","scrub_duration":0.0043697320000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"240'19","reported_seq":391,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.248385+0000","last_change":"2024-09-10T18:37:05.753183+0000","last_active":"2024-09-10T18:37:33.248385+0000","last_peered":"2024-09-10T18:37:33.248385+0000","last_clean":"2024-09-10T18:37:33.248385+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:37:33.248385+0000","last_undegraded":"2024-09-10T18:37:33.248385+0000","last_fullsized":"2024-09-10T18:37:33.248385+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:37:05.752891+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-09-10T18:35:19.802016+0000","last_clean_scrub_stamp":"2024-09-10T18:37:05.752891+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:06.037770+0000","scrub_duration":0.0067102660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":366,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.240430+0000","last_change":"2024-09-10T18:36:04.875938+0000","last_active":"2024-09-10T18:37:33.240430+0000","last_peered":"2024-09-10T18:37:33.240430+0000","last_clean":"2024-09-10T18:37:33.240430+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:37:33.240430+0000","last_undegraded":"2024-09-10T18:37:33.240430+0000","last_fullsized":"2024-09-10T18:37:33.240430+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:04.875692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:36:04.875692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018984749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":348,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.234322+0000","last_change":"2024-09-10T18:35:12.764509+0000","last_active":"2024-09-10T18:37:33.234322+0000","last_peered":"2024-09-10T18:37:33.234322+0000","last_clean":"2024-09-10T18:37:33.234322+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:37:33.234322+0000","last_undegraded":"2024-09-10T18:37:33.234322+0000","last_fullsized":"2024-09-10T18:37:33.234322+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:12.764433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:12.764433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069242059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":383,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:35.987493+0000","last_change":"2024-09-10T18:37:35.987443+0000","last_active":"2024-09-10T18:37:35.987493+0000","last_peered":"2024-09-10T18:37:35.987493+0000","last_clean":"2024-09-10T18:37:35.987493+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:37:35.987493+0000","last_undegraded":"2024-09-10T18:37:35.987493+0000","last_fullsized":"2024-09-10T18:37:35.987493+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:35.987330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:35.987330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:41.211548+0000","scrub_duration":0.0096016480000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'81","reported_seq":89,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.244767+0000","last_change":"2024-09-10T18:36:58.147850+0000","last_active":"2024-09-10T18:37:33.244767+0000","last_peered":"2024-09-10T18:37:33.244767+0000","last_clean":"2024-09-10T18:37:33.244767+0000","last_became_active":"2024-09-10T18:34:02.797304+0000","last_became_peered":"2024-09-10T18:34:02.797304+0000","last_unstale":"2024-09-10T18:37:33.244767+0000","last_undegraded":"2024-09-10T18:37:33.244767+0000","last_fullsized":"2024-09-10T18:37:33.244767+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:58.147719+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:58.147719+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:13.451302+0000","scrub_duration":0.009846858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"240'19","reported_seq":387,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.246118+0000","last_change":"2024-09-10T18:36:14.732278+0000","last_active":"2024-09-10T18:37:33.246118+0000","last_peered":"2024-09-10T18:37:33.246118+0000","last_clean":"2024-09-10T18:37:33.246118+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:37:33.246118+0000","last_undegraded":"2024-09-10T18:37:33.246118+0000","last_fullsized":"2024-09-10T18:37:33.246118+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:36:14.732217+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:36:14.732217+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:39.275538+0000","scrub_duration":0.0020637149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":371,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.245940+0000","last_change":"2024-09-10T18:37:04.713668+0000","last_active":"2024-09-10T18:37:33.245940+0000","last_peered":"2024-09-10T18:37:33.245940+0000","last_clean":"2024-09-10T18:37:33.245940+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:37:33.245940+0000","last_undegraded":"2024-09-10T18:37:33.245940+0000","last_fullsized":"2024-09-10T18:37:33.245940+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:04.713474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_clean_scrub_stamp":"2024-09-10T18:37:04.713474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:09.956287+0000","scrub_duration":0.0039225839999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":549,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:34.297280+0000","last_change":"2024-09-10T18:35:01.701033+0000","last_active":"2024-09-10T18:37:34.297280+0000","last_peered":"2024-09-10T18:37:34.297280+0000","last_clean":"2024-09-10T18:37:34.297280+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:37:34.297280+0000","last_undegraded":"2024-09-10T18:37:34.297280+0000","last_fullsized":"2024-09-10T18:37:34.297280+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:35:01.700934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:01.700934+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050961280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":91,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.234674+0000","last_change":"2024-09-10T18:37:02.933234+0000","last_active":"2024-09-10T18:37:33.234674+0000","last_peered":"2024-09-10T18:37:33.234674+0000","last_clean":"2024-09-10T18:37:33.234674+0000","last_became_active":"2024-09-10T18:33:29.141936+0000","last_became_peered":"2024-09-10T18:33:29.141936+0000","last_unstale":"2024-09-10T18:37:33.234674+0000","last_undegraded":"2024-09-10T18:37:33.234674+0000","last_fullsized":"2024-09-10T18:37:33.234674+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:02.933068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:02.933068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:03.633941+0000","scrub_duration":0.0042260099999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"76'81","reported_seq":151,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.245874+0000","last_change":"2024-09-10T18:36:08.710620+0000","last_active":"2024-09-10T18:37:33.245874+0000","last_peered":"2024-09-10T18:37:33.245874+0000","last_clean":"2024-09-10T18:37:33.245874+0000","last_became_active":"2024-09-10T18:34:02.810381+0000","last_became_peered":"2024-09-10T18:34:02.810381+0000","last_unstale":"2024-09-10T18:37:33.245874+0000","last_undegraded":"2024-09-10T18:37:33.245874+0000","last_fullsized":"2024-09-10T18:37:33.245874+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:08.710566+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:36:08.710566+0000","last_clean_scrub_stamp":"2024-09-10T18:36:08.710566+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012163009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":390,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.233984+0000","last_change":"2024-09-10T18:37:09.977307+0000","last_active":"2024-09-10T18:37:33.233984+0000","last_peered":"2024-09-10T18:37:33.233984+0000","last_clean":"2024-09-10T18:37:33.233984+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:37:33.233984+0000","last_undegraded":"2024-09-10T18:37:33.233984+0000","last_fullsized":"2024-09-10T18:37:33.233984+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-10T18:37:09.977172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:09.977172+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:30.689481+0000","scrub_duration":0.01120503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":359,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.237894+0000","last_change":"2024-09-10T18:35:17.826500+0000","last_active":"2024-09-10T18:37:33.237894+0000","last_peered":"2024-09-10T18:37:33.237894+0000","last_clean":"2024-09-10T18:37:33.237894+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:37:33.237894+0000","last_undegraded":"2024-09-10T18:37:33.237894+0000","last_fullsized":"2024-09-10T18:37:33.237894+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:17.826448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:17.826448+0000","last_clean_scrub_stamp":"2024-09-10T18:35:17.826448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002920808,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":568,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-10T18:37:34.952338+0000","last_change":"2024-09-10T18:37:03.193346+0000","last_active":"2024-09-10T18:37:34.952338+0000","last_peered":"2024-09-10T18:37:34.952338+0000","last_clean":"2024-09-10T18:37:34.952338+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:37:34.952338+0000","last_undegraded":"2024-09-10T18:37:34.952338+0000","last_fullsized":"2024-09-10T18:37:34.952338+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:37:03.193256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:03.193256+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:24.557501+0000","scrub_duration":0.0034883319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":355,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.248224+0000","last_change":"2024-09-10T18:36:06.720664+0000","last_active":"2024-09-10T18:37:33.248224+0000","last_peered":"2024-09-10T18:37:33.248224+0000","last_clean":"2024-09-10T18:37:33.248224+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:37:33.248224+0000","last_undegraded":"2024-09-10T18:37:33.248224+0000","last_fullsized":"2024-09-10T18:37:33.248224+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.720613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.720613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015989380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'81","reported_seq":71,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.252859+0000","last_change":"2024-09-10T18:35:53.652838+0000","last_active":"2024-09-10T18:37:33.252859+0000","last_peered":"2024-09-10T18:37:33.252859+0000","last_clean":"2024-09-10T18:37:33.252859+0000","last_became_active":"2024-09-10T18:34:34.881186+0000","last_became_peered":"2024-09-10T18:34:34.881186+0000","last_unstale":"2024-09-10T18:37:33.252859+0000","last_undegraded":"2024-09-10T18:37:33.252859+0000","last_fullsized":"2024-09-10T18:37:33.252859+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:53.652767+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:53.652767+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013276989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":405,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.248014+0000","last_change":"2024-09-10T18:37:13.403259+0000","last_active":"2024-09-10T18:37:33.248014+0000","last_peered":"2024-09-10T18:37:33.248014+0000","last_clean":"2024-09-10T18:37:33.248014+0000","last_became_active":"2024-09-10T18:26:27.649202+0000","last_became_peered":"2024-09-10T18:26:27.649202+0000","last_unstale":"2024-09-10T18:37:33.248014+0000","last_undegraded":"2024-09-10T18:37:33.248014+0000","last_fullsized":"2024-09-10T18:37:33.248014+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:37:13.403175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:13.403175+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:34.162155+0000","scrub_duration":0.0070645999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":369,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.248159+0000","last_change":"2024-09-10T18:37:02.270246+0000","last_active":"2024-09-10T18:37:33.248159+0000","last_peered":"2024-09-10T18:37:33.248159+0000","last_clean":"2024-09-10T18:37:33.248159+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:37:33.248159+0000","last_undegraded":"2024-09-10T18:37:33.248159+0000","last_fullsized":"2024-09-10T18:37:33.248159+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:02.270169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:37:02.270169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:15.675634+0000","scrub_duration":0.001637306,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":369,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.252379+0000","last_change":"2024-09-10T18:36:11.523432+0000","last_active":"2024-09-10T18:37:33.252379+0000","last_peered":"2024-09-10T18:37:33.252379+0000","last_clean":"2024-09-10T18:37:33.252379+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:37:33.252379+0000","last_undegraded":"2024-09-10T18:37:33.252379+0000","last_fullsized":"2024-09-10T18:37:33.252379+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:11.523380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_clean_scrub_stamp":"2024-09-10T18:36:11.523380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014936800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":336,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.232501+0000","last_change":"2024-09-10T18:34:14.263579+0000","last_active":"2024-09-10T18:37:33.232501+0000","last_peered":"2024-09-10T18:37:33.232501+0000","last_clean":"2024-09-10T18:37:33.232501+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:37:33.232501+0000","last_undegraded":"2024-09-10T18:37:33.232501+0000","last_fullsized":"2024-09-10T18:37:33.232501+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:14.263499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:34:14.263499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019685509999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'81","reported_seq":70,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.246704+0000","last_change":"2024-09-10T18:36:08.676315+0000","last_active":"2024-09-10T18:37:33.246704+0000","last_peered":"2024-09-10T18:37:33.246704+0000","last_clean":"2024-09-10T18:37:33.246704+0000","last_became_active":"2024-09-10T18:34:34.858291+0000","last_became_peered":"2024-09-10T18:34:34.858291+0000","last_unstale":"2024-09-10T18:37:33.246704+0000","last_undegraded":"2024-09-10T18:37:33.246704+0000","last_fullsized":"2024-09-10T18:37:33.246704+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:08.676247+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:08.676247+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001606512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"240'15","reported_seq":400,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.253718+0000","last_change":"2024-09-10T18:36:16.592923+0000","last_active":"2024-09-10T18:37:33.253718+0000","last_peered":"2024-09-10T18:37:33.253718+0000","last_clean":"2024-09-10T18:37:33.253718+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:37:33.253718+0000","last_undegraded":"2024-09-10T18:37:33.253718+0000","last_fullsized":"2024-09-10T18:37:33.253718+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:36:16.592847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:16.592847+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:39.683840+0000","scrub_duration":0.0020114550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":362,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.239816+0000","last_change":"2024-09-10T18:36:07.931194+0000","last_active":"2024-09-10T18:37:33.239816+0000","last_peered":"2024-09-10T18:37:33.239816+0000","last_clean":"2024-09-10T18:37:33.239816+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:37:33.239816+0000","last_undegraded":"2024-09-10T18:37:33.239816+0000","last_fullsized":"2024-09-10T18:37:33.239816+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:07.931108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.931108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012878189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":341,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.233157+0000","last_change":"2024-09-10T18:34:56.701737+0000","last_active":"2024-09-10T18:37:33.233157+0000","last_peered":"2024-09-10T18:37:33.233157+0000","last_clean":"2024-09-10T18:37:33.233157+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:37:33.233157+0000","last_undegraded":"2024-09-10T18:37:33.233157+0000","last_fullsized":"2024-09-10T18:37:33.233157+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:56.701661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:34:56.701661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064135620000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":365,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.240357+0000","last_change":"2024-09-10T18:37:11.863730+0000","last_active":"2024-09-10T18:37:33.240357+0000","last_peered":"2024-09-10T18:37:33.240357+0000","last_clean":"2024-09-10T18:37:33.240357+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:37:33.240357+0000","last_undegraded":"2024-09-10T18:37:33.240357+0000","last_fullsized":"2024-09-10T18:37:33.240357+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:11.863644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:11.863644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:16.121702+0000","scrub_duration":0.0037629410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'81","reported_seq":89,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.246203+0000","last_change":"2024-09-10T18:36:07.443661+0000","last_active":"2024-09-10T18:37:33.246203+0000","last_peered":"2024-09-10T18:37:33.246203+0000","last_clean":"2024-09-10T18:37:33.246203+0000","last_became_active":"2024-09-10T18:34:02.802580+0000","last_became_peered":"2024-09-10T18:34:02.802580+0000","last_unstale":"2024-09-10T18:37:33.246203+0000","last_undegraded":"2024-09-10T18:37:33.246203+0000","last_fullsized":"2024-09-10T18:37:33.246203+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:07.443596+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.443596+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016421059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":388,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.232666+0000","last_change":"2024-09-10T18:35:03.727242+0000","last_active":"2024-09-10T18:37:33.232666+0000","last_peered":"2024-09-10T18:37:33.232666+0000","last_clean":"2024-09-10T18:37:33.232666+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:37:33.232666+0000","last_undegraded":"2024-09-10T18:37:33.232666+0000","last_fullsized":"2024-09-10T18:37:33.232666+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:35:03.727153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:35:03.727153+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069271460000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":370,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.250502+0000","last_change":"2024-09-10T18:36:53.050798+0000","last_active":"2024-09-10T18:37:33.250502+0000","last_peered":"2024-09-10T18:37:33.250502+0000","last_clean":"2024-09-10T18:37:33.250502+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:37:33.250502+0000","last_undegraded":"2024-09-10T18:37:33.250502+0000","last_fullsized":"2024-09-10T18:37:33.250502+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:53.050726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:36:53.050726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:02.660776+0000","scrub_duration":0.0099588239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":355,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.238850+0000","last_change":"2024-09-10T18:34:57.776098+0000","last_active":"2024-09-10T18:37:33.238850+0000","last_peered":"2024-09-10T18:37:33.238850+0000","last_clean":"2024-09-10T18:37:33.238850+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:37:33.238850+0000","last_undegraded":"2024-09-10T18:37:33.238850+0000","last_fullsized":"2024-09-10T18:37:33.238850+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:57.776022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:34:57.776022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002853441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":369,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.233411+0000","last_change":"2024-09-10T18:36:52.896429+0000","last_active":"2024-09-10T18:37:33.233411+0000","last_peered":"2024-09-10T18:37:33.233411+0000","last_clean":"2024-09-10T18:37:33.233411+0000","last_became_active":"2024-09-10T18:26:27.636996+0000","last_became_peered":"2024-09-10T18:26:27.636996+0000","last_unstale":"2024-09-10T18:37:33.233411+0000","last_undegraded":"2024-09-10T18:37:33.233411+0000","last_fullsized":"2024-09-10T18:37:33.233411+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:52.896322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:52.896322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:58.016566+0000","scrub_duration":0.0029578769999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'81","reported_seq":70,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.239416+0000","last_change":"2024-09-10T18:36:14.999809+0000","last_active":"2024-09-10T18:37:33.239416+0000","last_peered":"2024-09-10T18:37:33.239416+0000","last_clean":"2024-09-10T18:37:33.239416+0000","last_became_active":"2024-09-10T18:34:34.868371+0000","last_became_peered":"2024-09-10T18:34:34.868371+0000","last_unstale":"2024-09-10T18:37:33.239416+0000","last_undegraded":"2024-09-10T18:37:33.239416+0000","last_fullsized":"2024-09-10T18:37:33.239416+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:14.999747+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:14.999747+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:41.897947+0000","scrub_duration":0.001461415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":369,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.232486+0000","last_change":"2024-09-10T18:36:01.819314+0000","last_active":"2024-09-10T18:37:33.232486+0000","last_peered":"2024-09-10T18:37:33.232486+0000","last_clean":"2024-09-10T18:37:33.232486+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:37:33.232486+0000","last_undegraded":"2024-09-10T18:37:33.232486+0000","last_fullsized":"2024-09-10T18:37:33.232486+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-09-10T18:36:01.819258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:01.819258+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001608246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":364,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.248145+0000","last_change":"2024-09-10T18:36:49.574521+0000","last_active":"2024-09-10T18:37:33.248145+0000","last_peered":"2024-09-10T18:37:33.248145+0000","last_clean":"2024-09-10T18:37:33.248145+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:37:33.248145+0000","last_undegraded":"2024-09-10T18:37:33.248145+0000","last_fullsized":"2024-09-10T18:37:33.248145+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:49.571125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:36:49.571125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:37:56.658367+0000","scrub_duration":0.0066675700000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":319,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.241938+0000","last_change":"2024-09-10T18:36:01.890154+0000","last_active":"2024-09-10T18:37:33.241938+0000","last_peered":"2024-09-10T18:37:33.241938+0000","last_clean":"2024-09-10T18:37:33.241938+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:37:33.241938+0000","last_undegraded":"2024-09-10T18:37:33.241938+0000","last_fullsized":"2024-09-10T18:37:33.241938+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:01.890087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:01.890087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001570519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":353,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.237449+0000","last_change":"2024-09-10T18:35:18.879787+0000","last_active":"2024-09-10T18:37:33.237449+0000","last_peered":"2024-09-10T18:37:33.237449+0000","last_clean":"2024-09-10T18:37:33.237449+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:37:33.237449+0000","last_undegraded":"2024-09-10T18:37:33.237449+0000","last_fullsized":"2024-09-10T18:37:33.237449+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:18.879717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:18.879717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059146550000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'81","reported_seq":70,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.247754+0000","last_change":"2024-09-10T18:36:00.273377+0000","last_active":"2024-09-10T18:37:33.247754+0000","last_peered":"2024-09-10T18:37:33.247754+0000","last_clean":"2024-09-10T18:37:33.247754+0000","last_became_active":"2024-09-10T18:34:34.867380+0000","last_became_peered":"2024-09-10T18:34:34.867380+0000","last_unstale":"2024-09-10T18:37:33.247754+0000","last_undegraded":"2024-09-10T18:37:33.247754+0000","last_fullsized":"2024-09-10T18:37:33.247754+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:00.273308+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:00.273308+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016513669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":364,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.253352+0000","last_change":"2024-09-10T18:35:56.582580+0000","last_active":"2024-09-10T18:37:33.253352+0000","last_peered":"2024-09-10T18:37:33.253352+0000","last_clean":"2024-09-10T18:37:33.253352+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:37:33.253352+0000","last_undegraded":"2024-09-10T18:37:33.253352+0000","last_fullsized":"2024-09-10T18:37:33.253352+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:56.582525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_clean_scrub_stamp":"2024-09-10T18:35:56.582525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014918869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":342,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.232710+0000","last_change":"2024-09-10T18:34:03.175239+0000","last_active":"2024-09-10T18:37:33.232710+0000","last_peered":"2024-09-10T18:37:33.232710+0000","last_clean":"2024-09-10T18:37:33.232710+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:37:33.232710+0000","last_undegraded":"2024-09-10T18:37:33.232710+0000","last_fullsized":"2024-09-10T18:37:33.232710+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:03.175158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:34:03.175158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046326559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":380,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.243572+0000","last_change":"2024-09-10T18:36:47.856846+0000","last_active":"2024-09-10T18:37:33.243572+0000","last_peered":"2024-09-10T18:37:33.243572+0000","last_clean":"2024-09-10T18:37:33.243572+0000","last_became_active":"2024-09-10T18:26:27.648112+0000","last_became_peered":"2024-09-10T18:26:27.648112+0000","last_unstale":"2024-09-10T18:37:33.243572+0000","last_undegraded":"2024-09-10T18:37:33.243572+0000","last_fullsized":"2024-09-10T18:37:33.243572+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:36:47.856694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:47.856694+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:08.148989+0000","scrub_duration":0.0066546349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":357,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-10T18:37:33.246669+0000","last_change":"2024-09-10T18:35:59.688760+0000","last_active":"2024-09-10T18:37:33.246669+0000","last_peered":"2024-09-10T18:37:33.246669+0000","last_clean":"2024-09-10T18:37:33.246669+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:37:33.246669+0000","last_undegraded":"2024-09-10T18:37:33.246669+0000","last_fullsized":"2024-09-10T18:37:33.246669+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:59.688701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:35:59.688701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018218869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":8353349632,"num_objects":127462,"num_object_clones":0,"num_object_copies":382386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127462,"num_write_kb":8157568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25026297856,"data_stored":25026297856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76472,"ondisk_log_size":76472,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":746,"ondisk_log_size":746,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":960,"num_read_kb":955,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2511,"ondisk_log_size":2511,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234181,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2617884,"kb_used_data":2023828,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":91127268,"statfs":{"total":95995035648,"available":93314322432,"internally_reserved":0,"allocated":2072399872,"data_stored":2072028790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:36:53 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.4810000000000001,"15min":1.8879999999999999},"min":{"1min":0.36799999999999999,"5min":0.215,"15min":0.20799999999999999},"max":{"1min":2.7890000000000001,"5min":50.091999999999999,"15min":85.012},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.292,"15min":1.796},"min":{"1min":0.223,"5min":0.223,"15min":0.222},"max":{"1min":1.766,"5min":50.104999999999997,"15min":84.665999999999997},"last":0.26900000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:36:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":1.4099999999999999,"15min":1.468},"min":{"1min":0.24299999999999999,"5min":0.23899999999999999,"15min":0.214},"max":{"1min":2.0649999999999999,"5min":39.728000000000002,"15min":49.317999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":1.369,"15min":1.393},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.4540000000000002,"5min":39.738999999999997,"15min":49.353000000000002},"last":0.373}]},{"osd":2,"last update":"Tue Sep 10 18:36:53 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":2.5089999999999999,"15min":1.4159999999999999},"min":{"1min":0.32900000000000001,"5min":0.27400000000000002,"15min":0.249},"max":{"1min":2.7810000000000001,"5min":136.648,"15min":136.648},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.94099999999999995,"5min":2.5219999999999998,"15min":1.49},"min":{"1min":0.38700000000000001,"5min":0.29199999999999998,"15min":0.24399999999999999},"max":{"1min":4.0979999999999999,"5min":136.506,"15min":136.506},"last":0.36099999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:36:53 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.6160000000000001,"15min":1.653},"min":{"1min":0.36599999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.8039999999999998,"5min":38.225000000000001,"15min":86.305999999999997},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":1.5640000000000001,"15min":1.673},"min":{"1min":0.41999999999999998,"5min":0.29199999999999998,"15min":0.26700000000000002},"max":{"1min":2.5609999999999999,"5min":38.104999999999997,"15min":88.364000000000004},"last":0.40200000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:36:53 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.625,"15min":0.53400000000000003},"min":{"1min":0.35599999999999998,"5min":0.24199999999999999,"15min":0.20799999999999999},"max":{"1min":1.7949999999999999,"5min":2.282,"15min":3.8839999999999999},"last":22.059000000000001},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.72799999999999998,"15min":0.58499999999999996},"min":{"1min":0.34699999999999998,"5min":0.26200000000000001,"15min":0.223},"max":{"1min":2.7480000000000002,"5min":4.0730000000000004,"15min":4.0730000000000004},"last":22.047000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:36:53 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.88300000000000001,"15min":0.745},"min":{"1min":0.32100000000000001,"5min":0.27300000000000002,"15min":0.23499999999999999},"max":{"1min":2.3149999999999999,"5min":4.1479999999999997,"15min":7.3109999999999999},"last":0.439},{"interface":"front","average":{"1min":0.621,"5min":0.877,"15min":0.68700000000000006},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.219},"max":{"1min":1.821,"5min":3.911,"15min":6.3360000000000003},"last":0.40899999999999997}]},{"osd":6,"last update":"Tue Sep 10 18:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.752,"15min":0.67200000000000004},"min":{"1min":0.38100000000000001,"5min":0.29299999999999998,"15min":0.27000000000000002},"max":{"1min":0.83499999999999996,"5min":2.7890000000000001,"15min":7.2750000000000004},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.69599999999999995,"15min":0.61799999999999999},"min":{"1min":0.437,"5min":0.36699999999999999,"15min":0.30199999999999999},"max":{"1min":1.177,"5min":2.3199999999999998,"15min":4.585},"last":0.42899999999999999}]}]},{"osd":6,"up_from":144,"seq":618475290760,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3869768,"kb_used_data":3571328,"kb_used_omap":7,"kb_used_meta":298424,"kb_avail":89875384,"statfs":{"total":95995035648,"available":92032393216,"internally_reserved":0,"allocated":3657039872,"data_stored":3656688898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":305586263},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:36:48 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.92800000000000005,"15min":1.613},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.343,"5min":7.8860000000000001,"15min":30.629999999999999},"last":1.288},{"interface":"front","average":{"1min":2.129,"5min":1.0389999999999999,"15min":1.3220000000000001},"min":{"1min":0.35299999999999998,"5min":0.217,"15min":0.217},"max":{"1min":18.093,"5min":18.093,"15min":30.597000000000001},"last":0.58199999999999996}]},{"osd":1,"last update":"Tue Sep 10 18:36:48 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.871,"15min":0.96799999999999997},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.0870000000000002,"5min":3.806,"15min":30.984000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.97099999999999997,"15min":1.145},"min":{"1min":0.28399999999999997,"5min":0.25700000000000001,"15min":0.24199999999999999},"max":{"1min":1.915,"5min":7.7939999999999996,"15min":30.571000000000002},"last":0.38600000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:36:48 2024","interfaces":[{"interface":"back","average":{"1min":1.9119999999999999,"5min":1.4930000000000001,"15min":1.2030000000000001},"min":{"1min":0.378,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":18.113,"5min":39.329999999999998,"15min":39.329999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":1.083,"5min":1.1639999999999999,"15min":0.99199999999999999},"min":{"1min":0.39100000000000001,"5min":0.26200000000000001,"15min":0.23400000000000001},"max":{"1min":3.1259999999999999,"5min":39.061999999999998,"15min":39.061999999999998},"last":1.2649999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:36:48 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.1839999999999999,"15min":1.758},"min":{"1min":0.36199999999999999,"5min":0.28199999999999997,"15min":0.27800000000000002},"max":{"1min":3.1560000000000001,"5min":16.831,"15min":61.703000000000003},"last":1.3360000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":1.2569999999999999,"15min":1.7749999999999999},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.0459999999999998,"5min":17.786000000000001,"15min":61.814},"last":0.60999999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:36:48 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.0220000000000002,"5min":4.0220000000000002,"15min":5.4950000000000001},"last":3.7749999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.56799999999999995,"15min":0.63200000000000001},"min":{"1min":0.33400000000000002,"5min":0.222,"15min":0.222},"max":{"1min":1.883,"5min":4.1689999999999996,"15min":4.6390000000000002},"last":0.42199999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:36:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.69199999999999995,"15min":0.59899999999999998},"min":{"1min":0.33100000000000002,"5min":0.27400000000000002,"15min":0.24399999999999999},"max":{"1min":1.857,"5min":4.476,"15min":4.6619999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.0269999999999999,"5min":0.78400000000000003,"15min":0.60499999999999998},"min":{"1min":0.30299999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.2290000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":1.2370000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:36:48 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.747,"15min":0.60899999999999999},"min":{"1min":0.41199999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":3.081,"5min":3.3650000000000002,"15min":3.3650000000000002},"last":1.3109999999999999},{"interface":"front","average":{"1min":0.751,"5min":0.65100000000000002,"15min":0.622},"min":{"1min":0.36099999999999999,"5min":0.28999999999999998,"15min":0.27100000000000002},"max":{"1min":2.8319999999999999,"5min":3.2320000000000002,"15min":5.5309999999999997},"last":0.439}]}]},{"osd":5,"up_from":39,"seq":167503724752,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3109396,"kb_used_data":2523660,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":90635756,"statfs":{"total":95995035648,"available":92811014144,"internally_reserved":0,"allocated":2584227840,"data_stored":2583865208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:37:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.97999999999999998,"15min":1.556},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.21099999999999999},"max":{"1min":2.7200000000000002,"5min":15.819000000000001,"15min":52.033999999999999},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.2490000000000001,"5min":0.95099999999999996,"15min":1.371},"min":{"1min":0.24399999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":2.4300000000000002,"5min":5.4420000000000002,"15min":52.167999999999999},"last":0.47599999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:37:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":2.3929999999999998,"15min":1.5},"min":{"1min":0.33100000000000002,"5min":0.249,"15min":0.22600000000000001},"max":{"1min":1.3440000000000001,"5min":111.773,"15min":111.773},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.98199999999999998,"5min":2.54,"15min":1.6200000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.247},"max":{"1min":2.6789999999999998,"5min":111.73999999999999,"15min":111.73999999999999},"last":0.44}]},{"osd":2,"last update":"Tue Sep 10 18:37:08 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.3360000000000001,"15min":1.242},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.24299999999999999},"max":{"1min":3.3180000000000001,"5min":54.530000000000001,"15min":54.530000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.82599999999999996,"5min":1.2889999999999999,"15min":1.202},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":2.4409999999999998,"5min":54.514000000000003,"15min":54.514000000000003},"last":1.1319999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:37:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":2.395,"15min":1.879},"min":{"1min":0.29999999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.4260000000000002,"5min":106.205,"15min":106.205},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.75,"5min":2.5430000000000001,"15min":1.829},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.6760000000000002,"5min":105.878,"15min":105.878},"last":0.51900000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:37:08 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.69699999999999995,"15min":0.58299999999999996},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.20799999999999999},"max":{"1min":2.7120000000000002,"5min":5.8040000000000003,"15min":5.8040000000000003},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.68200000000000005,"15min":0.623},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":2.7509999999999999,"5min":5.5499999999999998,"15min":6.798},"last":0.53500000000000003}]},{"osd":6,"last update":"Tue Sep 10 18:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58399999999999996,"15min":0.64500000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.27500000000000002},"max":{"1min":0.88,"5min":2.7469999999999999,"15min":4.7859999999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.623,"15min":0.66100000000000003},"min":{"1min":0.29599999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.94899999999999995,"5min":3.2850000000000001,"15min":5.3860000000000001},"last":0.60399999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":1.0109999999999999,"15min":0.73899999999999999},"min":{"1min":0.36299999999999999,"5min":0.25900000000000001,"15min":0.22500000000000001},"max":{"1min":0.89900000000000002,"5min":28.934000000000001,"15min":28.934000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.63,"5min":0.91700000000000004,"15min":0.65600000000000003},"min":{"1min":0.33900000000000002,"5min":0.309,"15min":0.24299999999999999},"max":{"1min":1.8779999999999999,"5min":5.8259999999999996,"15min":5.8259999999999996},"last":0.81100000000000005}]}]},{"osd":4,"up_from":33,"seq":141733920980,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4352792,"kb_used_data":4042960,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":89392360,"statfs":{"total":95995035648,"available":91537776640,"internally_reserved":0,"allocated":4139991040,"data_stored":4139623321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:37:38 2024","interfaces":[{"interface":"back","average":{"1min":2.0019999999999998,"5min":1.0900000000000001,"15min":1.167},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.188},"max":{"1min":8.3789999999999996,"5min":8.3789999999999996,"15min":31.585000000000001},"last":1.8879999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":0.91400000000000003,"15min":1.0529999999999999},"min":{"1min":0.23300000000000001,"5min":0.22700000000000001,"15min":0.189},"max":{"1min":7.7999999999999998,"5min":7.7999999999999998,"15min":29.802},"last":0.95399999999999996}]},{"osd":1,"last update":"Tue Sep 10 18:37:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.0800000000000001,"15min":1.0860000000000001},"min":{"1min":0.26900000000000002,"5min":0.254,"15min":0.21299999999999999},"max":{"1min":5.2110000000000003,"5min":5.2110000000000003,"15min":50.045000000000002},"last":1.923},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.99299999999999999,"15min":1.1359999999999999},"min":{"1min":0.28699999999999998,"5min":0.27100000000000002,"15min":0.22900000000000001},"max":{"1min":3.5009999999999999,"5min":5.5590000000000002,"15min":50.283000000000001},"last":0.30499999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:37:38 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":0.77400000000000002,"15min":0.88700000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.249},"max":{"1min":5.8470000000000004,"5min":5.8470000000000004,"15min":13.898999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.4910000000000001,"5min":0.90500000000000003,"15min":0.95899999999999996},"min":{"1min":0.318,"5min":0.26700000000000002,"15min":0.22500000000000001},"max":{"1min":7.7279999999999998,"5min":7.7279999999999998,"15min":14.51},"last":0.42699999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:37:38 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.038,"15min":1.4019999999999999},"min":{"1min":0.33000000000000002,"5min":0.27400000000000002,"15min":0.249},"max":{"1min":7.7699999999999996,"5min":7.7699999999999996,"15min":49.990000000000002},"last":2.718},{"interface":"front","average":{"1min":1.446,"5min":1.254,"15min":1.4990000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.255},"max":{"1min":5.1239999999999997,"5min":5.9189999999999996,"15min":50.764000000000003},"last":1.9510000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:37:03 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.745,"15min":0.66500000000000004},"min":{"1min":0.309,"5min":0.223,"15min":0.20499999999999999},"max":{"1min":1.897,"5min":3.0409999999999999,"15min":22.861000000000001},"last":1.7629999999999999},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.81599999999999995,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.252,"15min":0.22},"max":{"1min":8.2539999999999996,"5min":8.2539999999999996,"15min":19.611999999999998},"last":0.40200000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.72499999999999998,"15min":0.56399999999999995},"min":{"1min":0.29999999999999999,"5min":0.28799999999999998,"15min":0.26400000000000001},"max":{"1min":2.1600000000000001,"5min":3.1139999999999999,"15min":6.7990000000000004},"last":2.169},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.68300000000000005,"15min":0.59599999999999997},"min":{"1min":0.38300000000000001,"5min":0.307,"15min":0.30599999999999999},"max":{"1min":2.827,"5min":3.21,"15min":7.0300000000000002},"last":1.73}]},{"osd":7,"last update":"Tue Sep 10 18:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.69699999999999995,"15min":0.58699999999999997},"min":{"1min":0.34399999999999997,"5min":0.316,"15min":0.255},"max":{"1min":1.8160000000000001,"5min":3.262,"15min":7.0670000000000002},"last":1.7969999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.69899999999999995,"15min":0.60499999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.247},"max":{"1min":1.236,"5min":4.0430000000000001,"15min":6.5330000000000004},"last":2.1840000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117210,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3397412,"kb_used_data":2505820,"kb_used_omap":1,"kb_used_meta":891582,"kb_avail":90347740,"statfs":{"total":95995035648,"available":92516085760,"internally_reserved":0,"allocated":2565959680,"data_stored":2565580517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912980312},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:37:18 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.04,"15min":1.347},"min":{"1min":0.39900000000000002,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.8899999999999999,"5min":4.6079999999999997,"15min":8.5530000000000008},"last":1.8109999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3,"15min":1.4690000000000001},"min":{"1min":0.27700000000000002,"5min":0.23300000000000001,"15min":0.193},"max":{"1min":7.1459999999999999,"5min":8.2110000000000003,"15min":13.659000000000001},"last":5.0389999999999997}]},{"osd":1,"last update":"Tue Sep 10 18:37:18 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.3260000000000001,"15min":1.5489999999999999},"min":{"1min":0.224,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.6949999999999998,"5min":6.149,"15min":35.406999999999996},"last":4.7300000000000004},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.0880000000000001,"15min":1.4259999999999999},"min":{"1min":0.33800000000000002,"5min":0.24199999999999999,"15min":0.19},"max":{"1min":2.988,"5min":5.0720000000000001,"15min":36.902000000000001},"last":1.7749999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:37:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.052,"15min":1.1759999999999999},"min":{"1min":0.33900000000000002,"5min":0.25600000000000001,"15min":0.222},"max":{"1min":1.8700000000000001,"5min":6.6479999999999997,"15min":7.7960000000000003},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.2410000000000001,"15min":1.4139999999999999},"min":{"1min":0.23400000000000001,"5min":0.23000000000000001,"15min":0.217},"max":{"1min":6.3730000000000002,"5min":6.7290000000000001,"15min":14.997999999999999},"last":0.46100000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:36:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":1.0620000000000001,"15min":1.2789999999999999},"min":{"1min":0.41299999999999998,"5min":0.313,"15min":0.313},"max":{"1min":1.4259999999999999,"5min":6.5250000000000004,"15min":23.684999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":1.0600000000000001,"15min":1.2230000000000001},"min":{"1min":0.33900000000000002,"5min":0.29799999999999999,"15min":0.27200000000000002},"max":{"1min":2.4870000000000001,"5min":6.1929999999999996,"15min":23.649000000000001},"last":4.819}]},{"osd":5,"last update":"Tue Sep 10 18:37:00 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":1.161,"15min":1.3799999999999999},"min":{"1min":0.34499999999999997,"5min":0.314,"15min":0.314},"max":{"1min":1.4570000000000001,"5min":6.601,"15min":23.719999999999999},"last":0.5},{"interface":"front","average":{"1min":0.67300000000000004,"5min":1.2989999999999999,"15min":1.3640000000000001},"min":{"1min":0.375,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.492,"5min":6.2249999999999996,"15min":23.707000000000001},"last":4.843}]},{"osd":6,"last update":"Tue Sep 10 18:36:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":1.167,"15min":1.2470000000000001},"min":{"1min":0.41399999999999998,"5min":0.33800000000000002,"15min":0.317},"max":{"1min":1.1259999999999999,"5min":6.2889999999999997,"15min":31.815999999999999},"last":4.7670000000000003},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.165,"15min":1.4550000000000001},"min":{"1min":0.39300000000000002,"5min":0.38500000000000001,"15min":0.315},"max":{"1min":1.8999999999999999,"5min":7.0970000000000004,"15min":31.742000000000001},"last":3.718}]},{"osd":7,"last update":"Tue Sep 10 18:36:49 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":1.1859999999999999,"15min":1.3220000000000001},"min":{"1min":0.45900000000000002,"5min":0.34999999999999998,"15min":0.34100000000000003},"max":{"1min":1.383,"5min":6.5730000000000004,"15min":23.748999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":1.452,"15min":1.395},"min":{"1min":0.46200000000000002,"5min":0.40799999999999997,"15min":0.307},"max":{"1min":2.2000000000000002,"5min":12.612,"15min":25.058},"last":4.7939999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411550,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3344716,"kb_used_data":3039044,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":90400436,"statfs":{"total":95995035648,"available":92570046464,"internally_reserved":0,"allocated":3111981056,"data_stored":3111625641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.1080000000000001,"15min":1.5149999999999999},"min":{"1min":0.30199999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":3.2570000000000001,"5min":26.585999999999999,"15min":43.890000000000001},"last":1.462},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3009999999999999,"15min":1.76},"min":{"1min":0.26900000000000002,"5min":0.19400000000000001,"15min":0.16900000000000001},"max":{"1min":3.0390000000000001,"5min":26.48,"15min":44.331000000000003},"last":2.1160000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.90900000000000003,"15min":1.5089999999999999},"min":{"1min":0.27100000000000002,"5min":0.19400000000000001,"15min":0.193},"max":{"1min":2.2770000000000001,"5min":7.266,"15min":44.575000000000003},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.84299999999999997,"15min":1.3009999999999999},"min":{"1min":0.33700000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":3.2909999999999999,"5min":7.0270000000000001,"15min":44.368000000000002},"last":0.376}]},{"osd":3,"last update":"Tue Sep 10 18:37:14 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.7290000000000001,"15min":1.8360000000000001},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.339,"5min":61.331000000000003,"15min":61.331000000000003},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.8879999999999999,"15min":1.9199999999999999},"min":{"1min":0.24199999999999999,"5min":0.23499999999999999,"15min":0.214},"max":{"1min":4.5229999999999997,"5min":61.250999999999998,"15min":61.250999999999998},"last":0.22600000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:36:45 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.91400000000000003,"15min":1.052},"min":{"1min":0.38500000000000001,"5min":0.30199999999999999,"15min":0.24199999999999999},"max":{"1min":2.0779999999999998,"5min":12.555999999999999,"15min":12.555999999999999},"last":1.79},{"interface":"front","average":{"1min":0.75,"5min":0.73099999999999998,"15min":0.94899999999999995},"min":{"1min":0.44500000000000001,"5min":0.33000000000000002,"15min":0.317},"max":{"1min":1.6739999999999999,"5min":6.3639999999999999,"15min":12.130000000000001},"last":0.41899999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.93600000000000005,"15min":1.05},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.27500000000000002},"max":{"1min":2.476,"5min":6.3170000000000002,"15min":12.106},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.1599999999999999,"15min":1.1339999999999999},"min":{"1min":0.38800000000000001,"5min":0.317,"15min":0.317},"max":{"1min":4.5110000000000001,"5min":12.529999999999999,"15min":12.529999999999999},"last":0.80100000000000005}]},{"osd":6,"last update":"Tue Sep 10 18:36:45 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.93799999999999994,"15min":1.095},"min":{"1min":0.53800000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.3100000000000001,"5min":4.6440000000000001,"15min":5.9989999999999997},"last":1.7210000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.82499999999999996,"15min":0.96799999999999997},"min":{"1min":0.53900000000000003,"5min":0.34999999999999998,"15min":0.30099999999999999},"max":{"1min":1.9399999999999999,"5min":6.4089999999999998,"15min":7.1349999999999998},"last":19.335000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.0589999999999999,"15min":1.121},"min":{"1min":0.497,"5min":0.38900000000000001,"15min":0.30499999999999999},"max":{"1min":2.4889999999999999,"5min":10.68,"15min":10.68},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.075,"15min":1.1679999999999999},"min":{"1min":0.51200000000000001,"5min":0.376,"15min":0.307},"max":{"1min":2.0499999999999998,"5min":12.596,"15min":12.596},"last":1.7569999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575075,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2132656,"kb_used_data":1534440,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":91612496,"statfs":{"total":95995035648,"available":93811195904,"internally_reserved":0,"allocated":1571266560,"data_stored":1570875115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:37:32 2024","interfaces":[{"interface":"back","average":{"1min":2.488,"5min":1.5960000000000001,"15min":1.823},"min":{"1min":0.249,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":22.042999999999999,"5min":28.893999999999998,"15min":80.456999999999994},"last":0.439},{"interface":"front","average":{"1min":1.272,"5min":1.3779999999999999,"15min":1.8380000000000001},"min":{"1min":0.312,"5min":0.22800000000000001,"15min":0.187},"max":{"1min":4.2939999999999996,"5min":31.074999999999999,"15min":80.578000000000003},"last":1.115}]},{"osd":2,"last update":"Tue Sep 10 18:36:49 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.81799999999999995,"15min":1.2350000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19500000000000001},"max":{"1min":2.29,"5min":5.8380000000000001,"15min":13.035},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.86799999999999999,"15min":1.1879999999999999},"min":{"1min":0.26600000000000001,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":2.9399999999999999,"5min":3.669,"15min":12.222},"last":0.96199999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:37:30 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.28,"15min":1.726},"min":{"1min":0.25800000000000001,"5min":0.23699999999999999,"15min":0.22600000000000001},"max":{"1min":4.8449999999999998,"5min":8.0399999999999991,"15min":43.076000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.109,"5min":1.0669999999999999,"15min":1.7290000000000001},"min":{"1min":0.33800000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.3929999999999998,"5min":5.8879999999999999,"15min":41.93},"last":1.278}]},{"osd":4,"last update":"Tue Sep 10 18:37:32 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.0760000000000001,"15min":1.1379999999999999},"min":{"1min":0.35999999999999999,"5min":0.312,"15min":0.29799999999999999},"max":{"1min":2.2879999999999998,"5min":5.9930000000000003,"15min":7.2859999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.159,"5min":1.2470000000000001,"15min":1.3},"min":{"1min":0.36599999999999999,"5min":0.30299999999999999,"15min":0.28299999999999997},"max":{"1min":6.2489999999999997,"5min":7.8769999999999998,"15min":19.393000000000001},"last":0.99199999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:37:03 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":1.202,"15min":1.3089999999999999},"min":{"1min":0.308,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.7919999999999998,"5min":7.2110000000000003,"15min":15.48},"last":1.014},{"interface":"front","average":{"1min":0.79000000000000004,"5min":1.337,"15min":1.248},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.308},"max":{"1min":3.931,"5min":11.414999999999999,"15min":11.414999999999999},"last":0.57799999999999996}]},{"osd":6,"last update":"Tue Sep 10 18:36:49 2024","interfaces":[{"interface":"back","average":{"1min":4.1239999999999997,"5min":1.6830000000000001,"15min":1.458},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":67.287000000000006,"5min":67.287000000000006,"15min":67.287000000000006},"last":0.625},{"interface":"front","average":{"1min":4.3550000000000004,"5min":1.8460000000000001,"15min":1.5580000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.33600000000000002},"max":{"1min":67.400000000000006,"5min":67.400000000000006,"15min":67.400000000000006},"last":0.56599999999999995}]},{"osd":7,"last update":"Tue Sep 10 18:36:49 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1.145,"15min":1.2190000000000001},"min":{"1min":0.378,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.5369999999999999,"5min":8.3650000000000002,"15min":15.545999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":1.2729999999999999,"15min":1.2290000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.313},"max":{"1min":2.0950000000000002,"5min":6.3049999999999997,"15min":7.359},"last":0.52200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738600,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5692116,"kb_used_data":5098060,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":88053036,"statfs":{"total":95995035648,"available":90166308864,"internally_reserved":0,"allocated":5220413440,"data_stored":5220049690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:37:26 2024","interfaces":[{"interface":"back","average":{"1min":2.6309999999999998,"5min":2.968,"15min":2.1070000000000002},"min":{"1min":0.29899999999999999,"5min":0.185,"15min":0.185},"max":{"1min":8.2409999999999997,"5min":146.44499999999999,"15min":146.44499999999999},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.841,"5min":2.831,"15min":2.1539999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.17999999999999999},"max":{"1min":5.9009999999999998,"5min":148.77199999999999,"15min":148.77199999999999},"last":0.46800000000000003}]},{"osd":2,"last update":"Tue Sep 10 18:36:56 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":2.484,"15min":1.919},"min":{"1min":0.24199999999999999,"5min":0.19700000000000001,"15min":0.189},"max":{"1min":4.1559999999999997,"5min":146.56999999999999,"15min":146.56999999999999},"last":2.766},{"interface":"front","average":{"1min":0.86799999999999999,"5min":2.2919999999999998,"15min":1.8799999999999999},"min":{"1min":0.23799999999999999,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":3.137,"5min":146.601,"15min":146.601},"last":0.42799999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:37:26 2024","interfaces":[{"interface":"back","average":{"1min":2.4169999999999998,"5min":2.609,"15min":2.0529999999999999},"min":{"1min":0.373,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":7.6870000000000003,"5min":140.68299999999999,"15min":140.68299999999999},"last":3.0779999999999998},{"interface":"front","average":{"1min":2.8140000000000001,"5min":2.9129999999999998,"15min":2.1019999999999999},"min":{"1min":0.54500000000000004,"5min":0.26900000000000002,"15min":0.23899999999999999},"max":{"1min":8.2720000000000002,"5min":140.572,"15min":140.572},"last":4.04}]},{"osd":4,"last update":"Tue Sep 10 18:36:43 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":0.98099999999999998,"15min":1.2090000000000001},"min":{"1min":0.38100000000000001,"5min":0.27900000000000003,"15min":0.246},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":17.151},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.84099999999999997,"15min":1.1359999999999999},"min":{"1min":0.33900000000000002,"5min":0.31,"15min":0.25900000000000001},"max":{"1min":2.919,"5min":3.552,"15min":18.728999999999999},"last":0.62}]},{"osd":5,"last update":"Tue Sep 10 18:37:10 2024","interfaces":[{"interface":"back","average":{"1min":1.736,"5min":1.2,"15min":1.361},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.4589999999999996,"5min":8.1259999999999994,"15min":18.77},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.524,"5min":1.262,"15min":1.3200000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":5.5369999999999999,"5min":20.460000000000001,"15min":20.460000000000001},"last":0.45400000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:36:43 2024","interfaces":[{"interface":"back","average":{"1min":1.6699999999999999,"5min":1.0840000000000001,"15min":1.2330000000000001},"min":{"1min":0.40799999999999997,"5min":0.36299999999999999,"15min":0.30499999999999999},"max":{"1min":6.4909999999999997,"5min":6.4909999999999997,"15min":8.3940000000000001},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.88700000000000001,"15min":1.218},"min":{"1min":0.34799999999999998,"5min":0.34399999999999997,"15min":0.33300000000000002},"max":{"1min":3.0190000000000001,"5min":3.585,"15min":9.7859999999999996},"last":2.8149999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:36:43 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.1639999999999999,"15min":1.272},"min":{"1min":0.38600000000000001,"5min":0.377,"15min":0.26700000000000002},"max":{"1min":2.8650000000000002,"5min":10.75,"15min":13.711},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.94599999999999995,"15min":1.1919999999999999},"min":{"1min":0.41299999999999998,"5min":0.378,"15min":0.33500000000000002},"max":{"1min":1.724,"5min":11.218999999999999,"15min":15.032},"last":0.76800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5244190720,"data_stored":5244190720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1579483136,"data_stored":1579483136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3118792704,"data_stored":3118792704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2587361280,"data_stored":2587361280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4156358656,"data_stored":4156358656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2598371328,"data_stored":2598371328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3666935808,"data_stored":3666935808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2074804224,"data_stored":2074804224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:37:47.479 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-10T18:37:47.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pg_num 51 2024-09-10T18:37:47.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:47.499 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:47.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:47.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:47 smithi028 ceph-mon[29291]: 10.2 scrub starts 2024-09-10T18:37:47.583 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:47 smithi028 ceph-mon[39220]: 10.2 scrub starts 2024-09-10T18:37:47.601 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:47.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:47 smithi204 ceph-mon[32357]: 10.2 scrub starts 2024-09-10T18:37:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:47.627 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:47.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:47.728 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:47.755 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:47.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:47.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:47.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:47.889 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:47.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:47.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:47.990 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:48.017 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:48.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:48.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:48.032 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 89 16 135867 135851 95.3827 66.625 0.0267113 0.0104789 2024-09-10T18:37:48.049 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-10T18:37:48.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:37:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:48.175 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:48.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:48.270 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:37:48.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:48.308 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:48.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:48.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:48.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:48.461 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:37:48.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:48.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:48.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:48.586 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:48 smithi028 ceph-mon[39220]: 3.0 scrub starts 2024-09-10T18:37:48.587 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:48 smithi028 ceph-mon[39220]: 3.0 scrub ok 2024-09-10T18:37:48.587 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:48 smithi028 ceph-mon[39220]: pgmap v738: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 8.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 852 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-10T18:37:48.587 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:48 smithi028 ceph-mon[29291]: 3.0 scrub starts 2024-09-10T18:37:48.587 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:48 smithi028 ceph-mon[29291]: 3.0 scrub ok 2024-09-10T18:37:48.587 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:48 smithi028 ceph-mon[29291]: pgmap v738: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 8.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 852 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-10T18:37:48.591 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:48 smithi204 ceph-mon[32357]: 3.0 scrub starts 2024-09-10T18:37:48.591 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:48 smithi204 ceph-mon[32357]: 3.0 scrub ok 2024-09-10T18:37:48.591 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:48 smithi204 ceph-mon[32357]: pgmap v738: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 8.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 852 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-10T18:37:48.619 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:48.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:48.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:48.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:48.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:48.769 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:48.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:48.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:48.915 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:48.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:48.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:49.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:49.035 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total time run: 90.0107 2024-09-10T18:37:49.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total writes made: 136921 2024-09-10T18:37:49.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Write size: 65536 2024-09-10T18:37:49.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object size: 65536 2024-09-10T18:37:49.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Bandwidth (MB/sec): 95.0727 2024-09-10T18:37:49.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Bandwidth: 36.6333 2024-09-10T18:37:49.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max bandwidth (MB/sec): 239.375 2024-09-10T18:37:49.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min bandwidth (MB/sec): 54.3125 2024-09-10T18:37:49.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average IOPS: 1521 2024-09-10T18:37:49.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev IOPS: 586.132 2024-09-10T18:37:49.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max IOPS: 3830 2024-09-10T18:37:49.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min IOPS: 869 2024-09-10T18:37:49.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average Latency(s): 0.0105138 2024-09-10T18:37:49.036 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Latency(s): 0.00861386 2024-09-10T18:37:49.037 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max latency(s): 0.229265 2024-09-10T18:37:49.037 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min latency(s): 0.00134649 2024-09-10T18:37:49.046 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Cleaning up (deleting benchmark objects) 2024-09-10T18:37:49.055 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:49.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:49.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:49.157 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:49.190 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:49.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:49.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:49.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:49.326 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:49.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:49.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:49.337 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:37:49.429 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:49.464 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:49.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:49.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:49.588 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:49 smithi028 ceph-mon[29291]: 2.2 scrub starts 2024-09-10T18:37:49.589 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:49 smithi028 ceph-mon[29291]: 2.2 scrub ok 2024-09-10T18:37:49.589 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:49 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4130743660' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:37:49.589 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:49 smithi028 ceph-mon[29291]: from='client.25678 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:37:49.589 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:49 smithi028 ceph-mon[29291]: 3.13 scrub starts 2024-09-10T18:37:49.589 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:49 smithi028 ceph-mon[29291]: 3.13 scrub ok 2024-09-10T18:37:49.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:49 smithi028 ceph-mon[39220]: 2.2 scrub starts 2024-09-10T18:37:49.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:49 smithi028 ceph-mon[39220]: 2.2 scrub ok 2024-09-10T18:37:49.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:49 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4130743660' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:37:49.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:49 smithi028 ceph-mon[39220]: from='client.25678 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:37:49.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:49 smithi028 ceph-mon[39220]: 3.13 scrub starts 2024-09-10T18:37:49.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:49 smithi028 ceph-mon[39220]: 3.13 scrub ok 2024-09-10T18:37:49.597 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:49.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:49.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:49 smithi204 ceph-mon[32357]: 2.2 scrub starts 2024-09-10T18:37:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:49 smithi204 ceph-mon[32357]: 2.2 scrub ok 2024-09-10T18:37:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:49 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4130743660' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:37:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:49 smithi204 ceph-mon[32357]: from='client.25678 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:37:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:49 smithi204 ceph-mon[32357]: 3.13 scrub starts 2024-09-10T18:37:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:49 smithi204 ceph-mon[32357]: 3.13 scrub ok 2024-09-10T18:37:49.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:49.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:49.740 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:49.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:49.841 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:49.884 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:49.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:49.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:49.987 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:50.024 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:50.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:50.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:50.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:50.159 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:50.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:50.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:50.261 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:50.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:50.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:50.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:50.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:50.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:50.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:50.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[39220]: 6.16 scrub starts 2024-09-10T18:37:50.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[39220]: 6.16 scrub ok 2024-09-10T18:37:50.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[39220]: 10.3 scrub starts 2024-09-10T18:37:50.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[39220]: 1.16 scrub starts 2024-09-10T18:37:50.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[39220]: 1.16 scrub ok 2024-09-10T18:37:50.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[39220]: pgmap v739: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 8.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 852 B/s rd, 59 MiB/s wr, 947 op/s 2024-09-10T18:37:50.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[39220]: 10.2 scrub ok 2024-09-10T18:37:50.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:37:50.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:37:50.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[39220]: from='client.25678 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:37:50.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[39220]: osdmap e342: 8 total, 8 up, 8 in 2024-09-10T18:37:50.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[39220]: 4.18 deep-scrub starts 2024-09-10T18:37:50.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[39220]: 4.18 deep-scrub ok 2024-09-10T18:37:50.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[39220]: 6.1c scrub starts 2024-09-10T18:37:50.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[39220]: 6.1c scrub ok 2024-09-10T18:37:50.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[29291]: 6.16 scrub starts 2024-09-10T18:37:50.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[29291]: 6.16 scrub ok 2024-09-10T18:37:50.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[29291]: 10.3 scrub starts 2024-09-10T18:37:50.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[29291]: 1.16 scrub starts 2024-09-10T18:37:50.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[29291]: 1.16 scrub ok 2024-09-10T18:37:50.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[29291]: pgmap v739: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 8.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 852 B/s rd, 59 MiB/s wr, 947 op/s 2024-09-10T18:37:50.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[29291]: 10.2 scrub ok 2024-09-10T18:37:50.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:37:50.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:37:50.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[29291]: from='client.25678 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:37:50.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[29291]: osdmap e342: 8 total, 8 up, 8 in 2024-09-10T18:37:50.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[29291]: 4.18 deep-scrub starts 2024-09-10T18:37:50.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[29291]: 4.18 deep-scrub ok 2024-09-10T18:37:50.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[29291]: 6.1c scrub starts 2024-09-10T18:37:50.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:50 smithi028 ceph-mon[29291]: 6.1c scrub ok 2024-09-10T18:37:50.521 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:37:50.522 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:37:50.522 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:50 smithi204 ceph-mon[32357]: 6.16 scrub starts 2024-09-10T18:37:50.522 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:50 smithi204 ceph-mon[32357]: 6.16 scrub ok 2024-09-10T18:37:50.522 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:50 smithi204 ceph-mon[32357]: 10.3 scrub starts 2024-09-10T18:37:50.522 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:50 smithi204 ceph-mon[32357]: 1.16 scrub starts 2024-09-10T18:37:50.522 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:50 smithi204 ceph-mon[32357]: 1.16 scrub ok 2024-09-10T18:37:50.522 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:50 smithi204 ceph-mon[32357]: pgmap v739: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 8.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 852 B/s rd, 59 MiB/s wr, 947 op/s 2024-09-10T18:37:50.522 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:50 smithi204 ceph-mon[32357]: 10.2 scrub ok 2024-09-10T18:37:50.523 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:50 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:37:50.523 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:50 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:37:50.523 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:50 smithi204 ceph-mon[32357]: from='client.25678 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:37:50.523 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:50 smithi204 ceph-mon[32357]: osdmap e342: 8 total, 8 up, 8 in 2024-09-10T18:37:50.523 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:50 smithi204 ceph-mon[32357]: 4.18 deep-scrub starts 2024-09-10T18:37:50.523 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:50 smithi204 ceph-mon[32357]: 4.18 deep-scrub ok 2024-09-10T18:37:50.523 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:50 smithi204 ceph-mon[32357]: 6.1c scrub starts 2024-09-10T18:37:50.523 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:50 smithi204 ceph-mon[32357]: 6.1c scrub ok 2024-09-10T18:37:50.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:50.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:50.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:50.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:50.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:50.721 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:50.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:50.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:50.864 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:50.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:50.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:50.966 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:50.995 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:50.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:50.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:51.097 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:51.123 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:51.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:51.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:51.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:51.272 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:51.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:51.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:51.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:51.415 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:51.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:51.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:51.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:51.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T18:37:51.560 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:51.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:51.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:51.641 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:37:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:37:51] "GET /metrics HTTP/1.1" 200 224665 "" "Prometheus/2.43.0" 2024-09-10T18:37:51.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:51.702 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:51.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:51.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:51.804 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:51.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:51.857 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:51.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:51.928 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:37:51.959 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:51.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:51 smithi028 ceph-mon[39220]: 1.17 scrub starts 2024-09-10T18:37:51.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:51 smithi028 ceph-mon[39220]: 1.17 scrub ok 2024-09-10T18:37:51.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:51 smithi028 ceph-mon[39220]: 6.4 scrub starts 2024-09-10T18:37:51.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:51 smithi028 ceph-mon[39220]: 6.4 scrub ok 2024-09-10T18:37:51.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:51 smithi028 ceph-mon[39220]: pgmap v741: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 8.2 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.1 KiB/s rd, 61 MiB/s wr, 1.14k op/s 2024-09-10T18:37:51.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:51 smithi028 ceph-mon[29291]: 1.17 scrub starts 2024-09-10T18:37:51.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:51 smithi028 ceph-mon[29291]: 1.17 scrub ok 2024-09-10T18:37:51.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:51 smithi028 ceph-mon[29291]: 6.4 scrub starts 2024-09-10T18:37:51.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:51 smithi028 ceph-mon[29291]: 6.4 scrub ok 2024-09-10T18:37:51.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:51 smithi028 ceph-mon[29291]: pgmap v741: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 8.2 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.1 KiB/s rd, 61 MiB/s wr, 1.14k op/s 2024-09-10T18:37:51.993 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:51.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:51.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:52.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:52.122 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:51 smithi204 ceph-mon[32357]: 1.17 scrub starts 2024-09-10T18:37:52.122 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:51 smithi204 ceph-mon[32357]: 1.17 scrub ok 2024-09-10T18:37:52.122 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:51 smithi204 ceph-mon[32357]: 6.4 scrub starts 2024-09-10T18:37:52.122 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:51 smithi204 ceph-mon[32357]: 6.4 scrub ok 2024-09-10T18:37:52.122 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:51 smithi204 ceph-mon[32357]: pgmap v741: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 8.2 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.1 KiB/s rd, 61 MiB/s wr, 1.14k op/s 2024-09-10T18:37:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:52.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:52.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:52.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:52.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:52.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:52.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:52.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:52.427 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:52.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:52.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:52.551 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:52.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:52.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:52.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:37:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:37:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:37:52.653 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:52.690 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:52.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:52.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:52.792 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:52.813 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:52.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:52.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:52.915 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:52.951 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:52.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:52.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:53.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:52 smithi028 ceph-mon[29291]: 1.e scrub starts 2024-09-10T18:37:53.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:52 smithi028 ceph-mon[29291]: 1.e scrub ok 2024-09-10T18:37:53.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:52 smithi028 ceph-mon[29291]: 3.a scrub starts 2024-09-10T18:37:53.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:52 smithi028 ceph-mon[29291]: 3.a scrub ok 2024-09-10T18:37:53.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:52 smithi028 ceph-mon[29291]: 1.14 scrub starts 2024-09-10T18:37:53.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:52 smithi028 ceph-mon[29291]: 1.14 scrub ok 2024-09-10T18:37:53.001 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:52 smithi028 ceph-mon[29291]: 3.1d scrub starts 2024-09-10T18:37:53.001 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:52 smithi028 ceph-mon[29291]: 3.1d scrub ok 2024-09-10T18:37:53.001 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:52 smithi028 ceph-mon[39220]: 1.e scrub starts 2024-09-10T18:37:53.001 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:52 smithi028 ceph-mon[39220]: 1.e scrub ok 2024-09-10T18:37:53.001 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:52 smithi028 ceph-mon[39220]: 3.a scrub starts 2024-09-10T18:37:53.001 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:52 smithi028 ceph-mon[39220]: 3.a scrub ok 2024-09-10T18:37:53.001 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:52 smithi028 ceph-mon[39220]: 1.14 scrub starts 2024-09-10T18:37:53.001 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:52 smithi028 ceph-mon[39220]: 1.14 scrub ok 2024-09-10T18:37:53.001 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:52 smithi028 ceph-mon[39220]: 3.1d scrub starts 2024-09-10T18:37:53.001 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:52 smithi028 ceph-mon[39220]: 3.1d scrub ok 2024-09-10T18:37:53.040 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:52 smithi204 ceph-mon[32357]: 1.e scrub starts 2024-09-10T18:37:53.040 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:52 smithi204 ceph-mon[32357]: 1.e scrub ok 2024-09-10T18:37:53.041 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:52 smithi204 ceph-mon[32357]: 3.a scrub starts 2024-09-10T18:37:53.041 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:52 smithi204 ceph-mon[32357]: 3.a scrub ok 2024-09-10T18:37:53.041 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:52 smithi204 ceph-mon[32357]: 1.14 scrub starts 2024-09-10T18:37:53.041 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:52 smithi204 ceph-mon[32357]: 1.14 scrub ok 2024-09-10T18:37:53.041 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:52 smithi204 ceph-mon[32357]: 3.1d scrub starts 2024-09-10T18:37:53.041 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:52 smithi204 ceph-mon[32357]: 3.1d scrub ok 2024-09-10T18:37:53.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:53.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:53.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:53.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:53.200 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:53.230 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:53.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:53.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:53.332 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:53.371 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:53.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:53.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:53.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:53.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:53.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:53.545 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:37:53.545 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:37:53.545 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:37:53.545 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:37:53.546 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:37:53.546 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:37:53.546 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:37:53.546 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:37:53.546 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:37:53.546 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:37:53.546 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:37:53.546 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:37:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:37:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:37:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:37:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:37:53.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:53.654 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:53.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:53.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:53.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:53.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:53.809 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: specified pg_num 51 < pg_num_max 32 2024-09-10T18:37:53.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:53.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:53.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:54.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:54.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:54.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:54.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:54.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:54.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:54.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[39220]: 10.4 scrub starts 2024-09-10T18:37:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[39220]: 10.b scrub starts 2024-09-10T18:37:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[39220]: 10.3 scrub ok 2024-09-10T18:37:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3478144176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:37:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[39220]: pgmap v742: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 8.2 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 1.14k op/s 2024-09-10T18:37:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[39220]: 4.d deep-scrub starts 2024-09-10T18:37:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[39220]: 10.9 scrub starts 2024-09-10T18:37:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[39220]: 6.1f scrub starts 2024-09-10T18:37:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[39220]: 6.1f scrub ok 2024-09-10T18:37:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:37:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:37:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3181527962' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-09-10T18:37:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[39220]: from='client.35711 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-09-10T18:37:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[29291]: 10.4 scrub starts 2024-09-10T18:37:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[29291]: 10.b scrub starts 2024-09-10T18:37:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[29291]: 10.3 scrub ok 2024-09-10T18:37:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3478144176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:37:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[29291]: pgmap v742: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 8.2 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 1.14k op/s 2024-09-10T18:37:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[29291]: 4.d deep-scrub starts 2024-09-10T18:37:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[29291]: 10.9 scrub starts 2024-09-10T18:37:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[29291]: 6.1f scrub starts 2024-09-10T18:37:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[29291]: 6.1f scrub ok 2024-09-10T18:37:54.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:37:54.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:37:54.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3181527962' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-09-10T18:37:54.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:53 smithi028 ceph-mon[29291]: from='client.35711 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-09-10T18:37:54.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:53 smithi204 ceph-mon[32357]: 10.4 scrub starts 2024-09-10T18:37:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:53 smithi204 ceph-mon[32357]: 10.b scrub starts 2024-09-10T18:37:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:53 smithi204 ceph-mon[32357]: 10.3 scrub ok 2024-09-10T18:37:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:53 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3478144176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:37:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:53 smithi204 ceph-mon[32357]: pgmap v742: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 8.2 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 1.14k op/s 2024-09-10T18:37:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:53 smithi204 ceph-mon[32357]: 4.d deep-scrub starts 2024-09-10T18:37:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:53 smithi204 ceph-mon[32357]: 10.9 scrub starts 2024-09-10T18:37:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:53 smithi204 ceph-mon[32357]: 6.1f scrub starts 2024-09-10T18:37:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:53 smithi204 ceph-mon[32357]: 6.1f scrub ok 2024-09-10T18:37:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:53 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:37:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:53 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:37:54.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:53 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3181527962' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-09-10T18:37:54.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:53 smithi204 ceph-mon[32357]: from='client.35711 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-09-10T18:37:54.396 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:54.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:54.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:54.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:37:54.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:54.552 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:54.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:54.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:54.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:54.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:54.714 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:54.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:54.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:54.872 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:54.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:54.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:54.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:55.010 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:55.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:55.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:55.024 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:54 smithi204 ceph-mon[32357]: 4.d deep-scrub ok 2024-09-10T18:37:55.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:54 smithi204 ceph-mon[32357]: 10.8 scrub starts 2024-09-10T18:37:55.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:54 smithi204 ceph-mon[32357]: 4.1c scrub starts 2024-09-10T18:37:55.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:54 smithi204 ceph-mon[32357]: 4.1c scrub ok 2024-09-10T18:37:55.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:54 smithi204 ceph-mon[32357]: 1.18 scrub starts 2024-09-10T18:37:55.025 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:54 smithi204 ceph-mon[32357]: 1.18 scrub ok 2024-09-10T18:37:55.025 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:37:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:37:55.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:54 smithi028 ceph-mon[39220]: 4.d deep-scrub ok 2024-09-10T18:37:55.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:54 smithi028 ceph-mon[39220]: 10.8 scrub starts 2024-09-10T18:37:55.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:54 smithi028 ceph-mon[39220]: 4.1c scrub starts 2024-09-10T18:37:55.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:54 smithi028 ceph-mon[39220]: 4.1c scrub ok 2024-09-10T18:37:55.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:54 smithi028 ceph-mon[39220]: 1.18 scrub starts 2024-09-10T18:37:55.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:54 smithi028 ceph-mon[39220]: 1.18 scrub ok 2024-09-10T18:37:55.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:54 smithi028 ceph-mon[29291]: 4.d deep-scrub ok 2024-09-10T18:37:55.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:54 smithi028 ceph-mon[29291]: 10.8 scrub starts 2024-09-10T18:37:55.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:54 smithi028 ceph-mon[29291]: 4.1c scrub starts 2024-09-10T18:37:55.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:54 smithi028 ceph-mon[29291]: 4.1c scrub ok 2024-09-10T18:37:55.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:54 smithi028 ceph-mon[29291]: 1.18 scrub starts 2024-09-10T18:37:55.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:54 smithi028 ceph-mon[29291]: 1.18 scrub ok 2024-09-10T18:37:55.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:55.154 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-10T18:37:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:55.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:55.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:55.185 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:37:55.185 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:37:55.185 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.185 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:143954 10.f 10:f5930c9f:::benchmark_data_smithi028_161947_object7030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-09-10T18:37:55.185 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:37:55.167047+0000", 2024-09-10T18:37:55.185 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.011317939000000001, 2024-09-10T18:37:55.185 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011345375, 2024-09-10T18:37:55.185 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:55.185 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:37:55.185 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:55.185 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:37:55.185 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:37:55.185 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 143954 2024-09-10T18:37:55.186 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.186 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:55.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.186 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:55.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.167047+0000", 2024-09-10T18:37:55.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:55.186 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.186 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:55.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.167047+0000", 2024-09-10T18:37:55.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:55.186 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.186 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:55.187 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.167045+0000", 2024-09-10T18:37:55.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:37:55.187 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.187 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.187 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:55.187 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.167052+0000", 2024-09-10T18:37:55.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0790000000000003e-06 2024-09-10T18:37:55.187 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.187 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.187 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:55.187 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.167065+0000", 2024-09-10T18:37:55.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3128e-05 2024-09-10T18:37:55.187 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.187 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.188 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:55.188 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.167071+0000", 2024-09-10T18:37:55.188 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1e-06 2024-09-10T18:37:55.188 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.188 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.188 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:55.188 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.167170+0000", 2024-09-10T18:37:55.188 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8219000000000003e-05 2024-09-10T18:37:55.188 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.188 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.188 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:55.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.167353+0000", 2024-09-10T18:37:55.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018370800000000001 2024-09-10T18:37:55.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3", 2024-09-10T18:37:55.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.167781+0000", 2024-09-10T18:37:55.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00042757500000000003 2024-09-10T18:37:55.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:37:55.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.174778+0000", 2024-09-10T18:37:55.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0069972439999999997 2024-09-10T18:37:55.189 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:55.189 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:55.189 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:55.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.190 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:143955 10.7 10:eaa57474:::benchmark_data_smithi028_161947_object7031:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-09-10T18:37:55.190 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:37:55.167486+0000", 2024-09-10T18:37:55.190 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.010879265000000001, 2024-09-10T18:37:55.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01099822, 2024-09-10T18:37:55.190 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:55.190 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:37:55.190 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:37:55.190 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:37:55.190 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:37:55.190 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 143955 2024-09-10T18:37:55.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.190 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:55.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:55.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.167486+0000", 2024-09-10T18:37:55.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:55.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:55.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.167486+0000", 2024-09-10T18:37:55.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:55.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:55.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.167485+0000", 2024-09-10T18:37:55.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:55.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:55.192 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.167490+0000", 2024-09-10T18:37:55.192 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8100000000000003e-06 2024-09-10T18:37:55.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.192 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.192 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:55.192 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.167501+0000", 2024-09-10T18:37:55.192 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0145e-05 2024-09-10T18:37:55.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.192 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.192 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:55.192 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.167506+0000", 2024-09-10T18:37:55.192 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7939999999999999e-06 2024-09-10T18:37:55.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.192 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.192 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:37:55.193 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.170798+0000", 2024-09-10T18:37:55.193 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00329138 2024-09-10T18:37:55.193 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.193 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.193 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:37:55.193 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.170899+0000", 2024-09-10T18:37:55.193 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101121 2024-09-10T18:37:55.193 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.193 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.193 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,5", 2024-09-10T18:37:55.193 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.170993+0000", 2024-09-10T18:37:55.193 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4246000000000005e-05 2024-09-10T18:37:55.193 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:55.193 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:55.193 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:55.194 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.194 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.194 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.15693.0:143955 10.7 e342/314)", 2024-09-10T18:37:55.194 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:37:55.175477+0000", 2024-09-10T18:37:55.194 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0028883899999999998, 2024-09-10T18:37:55.194 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030773020000000001, 2024-09-10T18:37:55.194 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:55.194 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "queued for pg", 2024-09-10T18:37:55.194 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:55.194 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.194 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:55.194 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.175477+0000", 2024-09-10T18:37:55.194 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:55.194 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.194 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.194 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:55.195 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.175477+0000", 2024-09-10T18:37:55.195 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:55.195 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.195 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.195 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:55.195 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.175475+0000", 2024-09-10T18:37:55.195 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:55.195 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.195 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.195 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:55.195 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.175481+0000", 2024-09-10T18:37:55.195 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5790000000000003e-06 2024-09-10T18:37:55.195 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.196 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.196 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:55.196 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.175490+0000", 2024-09-10T18:37:55.196 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5990000000000002e-06 2024-09-10T18:37:55.196 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.196 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.196 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:55.196 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.175494+0000", 2024-09-10T18:37:55.196 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7940000000000002e-06 2024-09-10T18:37:55.196 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:55.196 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:55.196 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:55.196 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.196 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.197 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.15693.0:143955 10.7 e342/314)", 2024-09-10T18:37:55.197 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:37:55.175476+0000", 2024-09-10T18:37:55.197 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.002888654, 2024-09-10T18:37:55.197 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003125428, 2024-09-10T18:37:55.197 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:37:55.197 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "queued for pg", 2024-09-10T18:37:55.197 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:37:55.197 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.197 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:37:55.197 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.175476+0000", 2024-09-10T18:37:55.197 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:55.197 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.197 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.198 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:37:55.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.175476+0000", 2024-09-10T18:37:55.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:37:55.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.198 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:37:55.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.175475+0000", 2024-09-10T18:37:55.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:37:55.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.198 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:37:55.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.175481+0000", 2024-09-10T18:37:55.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7729999999999996e-06 2024-09-10T18:37:55.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.199 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:37:55.199 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.175491+0000", 2024-09-10T18:37:55.199 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0210000000000001e-05 2024-09-10T18:37:55.199 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:37:55.199 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:37:55.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:37:55.199 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:55.175497+0000", 2024-09-10T18:37:55.199 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5419999999999996e-06 2024-09-10T18:37:55.199 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:55.199 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:37:55.199 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:55.199 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:37:55.200 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-09-10T18:37:55.200 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 4 2024-09-10T18:37:55.200 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:37:55.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:55.264 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:37:55.312 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:55.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:55.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:55.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:55.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:55.456 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:55.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:55.557 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:55.600 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:55.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:55.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:55.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:55.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:55.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:55.886 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:55.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:55.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:55.988 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:56.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:55 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:56.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:55 smithi028 ceph-mon[39220]: 6.1d scrub starts 2024-09-10T18:37:56.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:55 smithi028 ceph-mon[39220]: 6.1d scrub ok 2024-09-10T18:37:56.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:55 smithi028 ceph-mon[39220]: pgmap v743: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 8.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.1 KiB/s rd, 53 MiB/s wr, 1.14k op/s 2024-09-10T18:37:56.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:55 smithi028 ceph-mon[39220]: 5.16 scrub starts 2024-09-10T18:37:56.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:55 smithi028 ceph-mon[39220]: 5.16 scrub ok 2024-09-10T18:37:56.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:55 smithi028 ceph-mon[39220]: 4.11 scrub starts 2024-09-10T18:37:56.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:55 smithi028 ceph-mon[39220]: 4.11 scrub ok 2024-09-10T18:37:56.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:55 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:56.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:55 smithi028 ceph-mon[29291]: 6.1d scrub starts 2024-09-10T18:37:56.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:55 smithi028 ceph-mon[29291]: 6.1d scrub ok 2024-09-10T18:37:56.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:55 smithi028 ceph-mon[29291]: pgmap v743: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 8.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.1 KiB/s rd, 53 MiB/s wr, 1.14k op/s 2024-09-10T18:37:56.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:55 smithi028 ceph-mon[29291]: 5.16 scrub starts 2024-09-10T18:37:56.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:55 smithi028 ceph-mon[29291]: 5.16 scrub ok 2024-09-10T18:37:56.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:55 smithi028 ceph-mon[29291]: 4.11 scrub starts 2024-09-10T18:37:56.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:55 smithi028 ceph-mon[29291]: 4.11 scrub ok 2024-09-10T18:37:56.018 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:55 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:37:56.019 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:55 smithi204 ceph-mon[32357]: 6.1d scrub starts 2024-09-10T18:37:56.019 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:55 smithi204 ceph-mon[32357]: 6.1d scrub ok 2024-09-10T18:37:56.019 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:55 smithi204 ceph-mon[32357]: pgmap v743: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 8.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.1 KiB/s rd, 53 MiB/s wr, 1.14k op/s 2024-09-10T18:37:56.019 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:55 smithi204 ceph-mon[32357]: 5.16 scrub starts 2024-09-10T18:37:56.019 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:55 smithi204 ceph-mon[32357]: 5.16 scrub ok 2024-09-10T18:37:56.019 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:55 smithi204 ceph-mon[32357]: 4.11 scrub starts 2024-09-10T18:37:56.019 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:55 smithi204 ceph-mon[32357]: 4.11 scrub ok 2024-09-10T18:37:56.022 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:56.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:56.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:56.057 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-10T18:37:56.124 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:56.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:56.181 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:56.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:56.282 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:56.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:56.327 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:56.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:56.401 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:37:56.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:56.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:56.502 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:37:56.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:56.626 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:56.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:56.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:56.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:56.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:56.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:56.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:56.913 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:56.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:56.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:57.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[29291]: 1.1a scrub starts 2024-09-10T18:37:57.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[29291]: 1.1a scrub ok 2024-09-10T18:37:57.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[29291]: 1.2 scrub starts 2024-09-10T18:37:57.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[29291]: 10.9 scrub ok 2024-09-10T18:37:57.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[29291]: 5.6 scrub starts 2024-09-10T18:37:57.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[29291]: 5.6 scrub ok 2024-09-10T18:37:57.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[29291]: 5.12 scrub starts 2024-09-10T18:37:57.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[29291]: 5.12 scrub ok 2024-09-10T18:37:57.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[29291]: 10.b scrub ok 2024-09-10T18:37:57.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[29291]: 5.14 deep-scrub starts 2024-09-10T18:37:57.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[29291]: 5.14 deep-scrub ok 2024-09-10T18:37:57.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[39220]: 1.1a scrub starts 2024-09-10T18:37:57.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[39220]: 1.1a scrub ok 2024-09-10T18:37:57.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[39220]: 1.2 scrub starts 2024-09-10T18:37:57.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[39220]: 10.9 scrub ok 2024-09-10T18:37:57.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[39220]: 5.6 scrub starts 2024-09-10T18:37:57.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[39220]: 5.6 scrub ok 2024-09-10T18:37:57.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[39220]: 5.12 scrub starts 2024-09-10T18:37:57.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[39220]: 5.12 scrub ok 2024-09-10T18:37:57.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[39220]: 10.b scrub ok 2024-09-10T18:37:57.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[39220]: 5.14 deep-scrub starts 2024-09-10T18:37:57.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:56 smithi028 ceph-mon[39220]: 5.14 deep-scrub ok 2024-09-10T18:37:57.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:57.042 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:57.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:57.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:57.144 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:57.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:57.181 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:57.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:57.282 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:57.314 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:57.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:57.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:57.345 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:37:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:56 smithi204 ceph-mon[32357]: 1.1a scrub starts 2024-09-10T18:37:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:56 smithi204 ceph-mon[32357]: 1.1a scrub ok 2024-09-10T18:37:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:56 smithi204 ceph-mon[32357]: 1.2 scrub starts 2024-09-10T18:37:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:56 smithi204 ceph-mon[32357]: 10.9 scrub ok 2024-09-10T18:37:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:56 smithi204 ceph-mon[32357]: 5.6 scrub starts 2024-09-10T18:37:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:56 smithi204 ceph-mon[32357]: 5.6 scrub ok 2024-09-10T18:37:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:56 smithi204 ceph-mon[32357]: 5.12 scrub starts 2024-09-10T18:37:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:56 smithi204 ceph-mon[32357]: 5.12 scrub ok 2024-09-10T18:37:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:56 smithi204 ceph-mon[32357]: 10.b scrub ok 2024-09-10T18:37:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:56 smithi204 ceph-mon[32357]: 5.14 deep-scrub starts 2024-09-10T18:37:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:56 smithi204 ceph-mon[32357]: 5.14 deep-scrub ok 2024-09-10T18:37:57.416 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:57.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:57.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:57.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:57.551 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:57.582 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:57.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:57.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:57.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:37:57.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:57.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:57.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:37:57.829 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:57.858 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:57.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:57.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:37:57.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:57.984 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:57 smithi204 ceph-mon[32357]: 5.5 scrub starts 2024-09-10T18:37:57.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:57 smithi204 ceph-mon[32357]: 5.5 scrub ok 2024-09-10T18:37:57.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:57 smithi204 ceph-mon[32357]: 10.4 scrub ok 2024-09-10T18:37:57.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:57 smithi204 ceph-mon[32357]: 3.4 deep-scrub starts 2024-09-10T18:37:57.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:57 smithi204 ceph-mon[32357]: 3.4 deep-scrub ok 2024-09-10T18:37:57.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:57 smithi204 ceph-mon[32357]: pgmap v744: 178 pgs: 4 active+clean+scrubbing, 174 active+clean; 7.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.1 KiB/s rd, 24 MiB/s wr, 1.14k op/s 2024-09-10T18:37:57.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:57 smithi204 ceph-mon[32357]: 10.8 scrub ok 2024-09-10T18:37:57.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:57 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2703090942' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:37:57.985 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:57 smithi204 ceph-mon[32357]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:37:57.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:57.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:57.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:58.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:58.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:58.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:58.237 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:58.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[29291]: 5.5 scrub starts 2024-09-10T18:37:58.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[29291]: 5.5 scrub ok 2024-09-10T18:37:58.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[29291]: 10.4 scrub ok 2024-09-10T18:37:58.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[29291]: 3.4 deep-scrub starts 2024-09-10T18:37:58.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[29291]: 3.4 deep-scrub ok 2024-09-10T18:37:58.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[29291]: pgmap v744: 178 pgs: 4 active+clean+scrubbing, 174 active+clean; 7.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.1 KiB/s rd, 24 MiB/s wr, 1.14k op/s 2024-09-10T18:37:58.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[29291]: 10.8 scrub ok 2024-09-10T18:37:58.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2703090942' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:37:58.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[29291]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:37:58.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[39220]: 5.5 scrub starts 2024-09-10T18:37:58.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[39220]: 5.5 scrub ok 2024-09-10T18:37:58.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[39220]: 10.4 scrub ok 2024-09-10T18:37:58.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[39220]: 3.4 deep-scrub starts 2024-09-10T18:37:58.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[39220]: 3.4 deep-scrub ok 2024-09-10T18:37:58.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[39220]: pgmap v744: 178 pgs: 4 active+clean+scrubbing, 174 active+clean; 7.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.1 KiB/s rd, 24 MiB/s wr, 1.14k op/s 2024-09-10T18:37:58.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[39220]: 10.8 scrub ok 2024-09-10T18:37:58.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2703090942' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:37:58.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:57 smithi028 ceph-mon[39220]: from='client.25684 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:37:58.275 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:58.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:58.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:58.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:58.420 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:58.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:58.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:58.522 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:58.553 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:58.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:58.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:58.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:37:58.693 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:58.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:58.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:37:58.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:37:58.833 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:58.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:58.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:37:58.935 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:58.958 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:58.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:59.003 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:58 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:37:59.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:58 smithi028 ceph-mon[39220]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:37:59.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:58 smithi028 ceph-mon[39220]: osdmap e343: 8 total, 8 up, 8 in 2024-09-10T18:37:59.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:58 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:37:59.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:58 smithi028 ceph-mon[29291]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:37:59.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:58 smithi028 ceph-mon[29291]: osdmap e343: 8 total, 8 up, 8 in 2024-09-10T18:37:59.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:37:59.091 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:59.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:59.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:37:59.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:59.225 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:59.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:59.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:59.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:37:59.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:58 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:37:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:58 smithi204 ceph-mon[32357]: from='client.25684 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:37:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:58 smithi204 ceph-mon[32357]: osdmap e343: 8 total, 8 up, 8 in 2024-09-10T18:37:59.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:59.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:59.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:37:59.464 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:59.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:59.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:59.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:37:59.590 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:59.616 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:59.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:59.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:59.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:37:59.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:37:59.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:59.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:37:59.860 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:37:59.892 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:37:59.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:37:59.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:37:59.994 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:37:59.998 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:37:59 smithi028 ceph-mon[39220]: pgmap v746: 178 pgs: 4 active+clean+scrubbing, 174 active+clean; 7.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.1 KiB/s rd, 24 MiB/s wr, 1.17k op/s 2024-09-10T18:37:59.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:37:59 smithi028 ceph-mon[29291]: pgmap v746: 178 pgs: 4 active+clean+scrubbing, 174 active+clean; 7.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.1 KiB/s rd, 24 MiB/s wr, 1.17k op/s 2024-09-10T18:38:00.038 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:00.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:00.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:00.049 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:38:00.049 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:38:00.082 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:38:00.083 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:38:00.118 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:38:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:38:00.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:00.155 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:38:00.155 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:38:00.155 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-10T18:38:00.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:38:00.156 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:38:00.157 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:38:00.189 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:00.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:00.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:00.195 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:38:00.196 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:38:00.209 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:37:59 smithi204 ceph-mon[32357]: pgmap v746: 178 pgs: 4 active+clean+scrubbing, 174 active+clean; 7.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.1 KiB/s rd, 24 MiB/s wr, 1.17k op/s 2024-09-10T18:38:00.228 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:38:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:38:00.257 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:38:00.258 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:38:00.288 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:38:00.288 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:38:00.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:00.358 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:00.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:00.459 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:00.495 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:00.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:00.596 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:00.632 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:00.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:00.638 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:38:00.638 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:38:00.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:00.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:00.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:00.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:00.911 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:00.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:01.013 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:01.059 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:01.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:01.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:01.160 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:01.193 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:01.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:01.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:00 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:38:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:00 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:38:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:00 smithi028 ceph-mon[29291]: osdmap e344: 8 total, 8 up, 8 in 2024-09-10T18:38:01.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:00 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1165031156' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:38:01.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:00 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:38:01.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:00 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:38:01.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:00 smithi028 ceph-mon[39220]: osdmap e344: 8 total, 8 up, 8 in 2024-09-10T18:38:01.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:00 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1165031156' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:38:01.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:01.336 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:01.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:01.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:01.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:00 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:38:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:00 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:38:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:00 smithi204 ceph-mon[32357]: osdmap e344: 8 total, 8 up, 8 in 2024-09-10T18:38:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:00 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1165031156' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:38:01.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:01.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:01.483 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:01.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:01.497 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:38:01.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-10T18:38:01.574 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:38:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:38:01] "GET /metrics HTTP/1.1" 200 224681 "" "Prometheus/2.43.0" 2024-09-10T18:38:01.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:38:01.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:01.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:01.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:01.786 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:01.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:01.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:01.888 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:01.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:01.926 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:01.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:01.941 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:38:02.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:02.068 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:02.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:02.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:02.170 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:02.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:02.206 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:02.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:02.241 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:01 smithi204 ceph-mon[32357]: pgmap v748: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-10T18:38:02.241 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:38:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:38:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:38:02.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:01 smithi028 ceph-mon[29291]: pgmap v748: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-10T18:38:02.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:01 smithi028 ceph-mon[39220]: pgmap v748: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-10T18:38:02.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:02.345 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:02.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:02.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:02.447 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:02.485 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:02.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:02.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:02.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:02.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:02.729 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:02.764 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:02.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:02.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:02.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:02.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:02.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:38:03.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:03.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:03.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:03.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:03.152 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:03.184 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:03.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:03.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:03.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:03.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:03.328 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:03.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:03.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:03.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:38:03.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:03.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:03.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:03.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:03.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:03.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:03.762 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:03.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:03.864 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:03.895 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:03.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:03.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:03.930 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:38:03.931 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:38:03.997 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:04.019 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:04.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:04.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:04.156 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:04.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:04.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:04.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:04.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:04.304 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:04.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:04.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:04.440 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:04.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:04.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:04.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:04 smithi028 ceph-mon[29291]: pgmap v749: 178 pgs: 178 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T18:38:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:04 smithi028 ceph-mon[39220]: pgmap v749: 178 pgs: 178 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T18:38:04.543 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:04.575 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:04.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:04.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:04.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:04 smithi204 ceph-mon[32357]: pgmap v749: 178 pgs: 178 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T18:38:04.677 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:04.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:04.714 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:04.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:04.815 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:04.843 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:04.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:04.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:04.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:04.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:04.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:05.084 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:05.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:38:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:38:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:05.117 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:05.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:05.218 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:05.244 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:05.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:05.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:05.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:05.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:05.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:05.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:05.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:05.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:05.672 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:05.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:05.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:05.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:05.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:05.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:05.925 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:05.957 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:05.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:05.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:06.059 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:06.094 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:06.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:06.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:06.251 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:06.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:06.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:06.321 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:38:06.353 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:06.377 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:06.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:06.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:38:06.406 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:06 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:06.406 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:06 smithi028 ceph-mon[39220]: pgmap v750: 178 pgs: 178 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 948 op/s 2024-09-10T18:38:06.407 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:06 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:06.407 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:06 smithi028 ceph-mon[29291]: pgmap v750: 178 pgs: 178 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 948 op/s 2024-09-10T18:38:06.414 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:38:06.479 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:06.506 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:06.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:06.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:06 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:06 smithi204 ceph-mon[32357]: pgmap v750: 178 pgs: 178 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 948 op/s 2024-09-10T18:38:06.607 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:06.650 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:06.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:06.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:06.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T18:38:06.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:06.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:06.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:06.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:06.924 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:06.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:06.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:07.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:07.067 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:07.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:07.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:07.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:07.183 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:38:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:07.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:07.324 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:07.360 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:07.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:07.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:07.461 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:07.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:07.492 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:07.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:07 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/253509035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:38:07.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:07 smithi028 ceph-mon[29291]: from='client.35726 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:38:07.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:07 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/253509035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:38:07.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:07 smithi028 ceph-mon[39220]: from='client.35726 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:38:07.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:07.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:07 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/253509035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:38:07.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:07 smithi204 ceph-mon[32357]: from='client.35726 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:38:07.630 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:07.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:07.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:07.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:07.768 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:07.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:07.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:07.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:07.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:07.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:08.018 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:08.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:08.052 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:08.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:08.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:08.190 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:08.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:08.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:08.293 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:08.327 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:08.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:08.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:08.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:08.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:08.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:08.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:08 smithi028 ceph-mon[39220]: from='client.15825 v1:172.21.15.28:0/1858921973' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:38:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:08 smithi028 ceph-mon[39220]: pgmap v751: 178 pgs: 178 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-10T18:38:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:08 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:38:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:08 smithi028 ceph-mon[39220]: from='client.35726 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:38:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:08 smithi028 ceph-mon[39220]: osdmap e345: 8 total, 8 up, 8 in 2024-09-10T18:38:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:08 smithi028 ceph-mon[29291]: from='client.15825 v1:172.21.15.28:0/1858921973' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:38:08.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:08 smithi028 ceph-mon[29291]: pgmap v751: 178 pgs: 178 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-10T18:38:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:08 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:38:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:08 smithi028 ceph-mon[29291]: from='client.35726 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:38:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:08 smithi028 ceph-mon[29291]: osdmap e345: 8 total, 8 up, 8 in 2024-09-10T18:38:08.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:08.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:08.597 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:08.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:08.604 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:38:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=sqlstore.transactions t=2024-09-10T18:38:08.322887739Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-10T18:38:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:08 smithi204 ceph-mon[32357]: from='client.15825 v1:172.21.15.28:0/1858921973' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:38:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:08 smithi204 ceph-mon[32357]: pgmap v751: 178 pgs: 178 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-10T18:38:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:08 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:38:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:08 smithi204 ceph-mon[32357]: from='client.35726 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:38:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:08 smithi204 ceph-mon[32357]: osdmap e345: 8 total, 8 up, 8 in 2024-09-10T18:38:08.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:08.728 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:08.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:08.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:08.751 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":750,"stamp":"2024-09-10T18:38:05.105200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7979669748,"num_objects":121948,"num_object_clones":0,"num_object_copies":365844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121948,"num_whiteouts":0,"num_read":2133,"num_read_kb":1885,"num_write":152808,"num_write_kb":8762988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80065,"ondisk_log_size":80065,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":27808628,"kb_used_data":23327908,"kb_used_omap":19,"kb_used_meta":4480364,"kb_avail":722152588,"statfs":{"total":767960285184,"available":739484250112,"internally_reserved":0,"allocated":23887777792,"data_stored":23884700075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20033,"internal_metadata":4587893183},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,7,4,5,5,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-496697344,"num_objects":-7579,"num_object_clones":0,"num_object_copies":-22737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7579,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7579,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002624"},"pg_stats":[{"pgid":"1.1d","version":"76'81","reported_seq":88,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.932180+0000","last_change":"2024-09-10T18:37:41.875430+0000","last_active":"2024-09-10T18:37:57.932180+0000","last_peered":"2024-09-10T18:37:57.932180+0000","last_clean":"2024-09-10T18:37:57.932180+0000","last_became_active":"2024-09-10T18:34:34.865894+0000","last_became_peered":"2024-09-10T18:34:34.865894+0000","last_unstale":"2024-09-10T18:37:57.932180+0000","last_undegraded":"2024-09-10T18:37:57.932180+0000","last_fullsized":"2024-09-10T18:37:57.932180+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:41.875334+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:41.875334+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:06.380545+0000","scrub_duration":0.015259258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":364,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.929819+0000","last_change":"2024-09-10T18:36:17.645197+0000","last_active":"2024-09-10T18:37:57.929819+0000","last_peered":"2024-09-10T18:37:57.929819+0000","last_clean":"2024-09-10T18:37:57.929819+0000","last_became_active":"2024-09-10T18:21:31.525236+0000","last_became_peered":"2024-09-10T18:21:31.525236+0000","last_unstale":"2024-09-10T18:37:57.929819+0000","last_undegraded":"2024-09-10T18:37:57.929819+0000","last_fullsized":"2024-09-10T18:37:57.929819+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:17.645146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:36:17.645146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003140618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"342'19","reported_seq":406,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.922106+0000","last_change":"2024-09-10T18:37:49.354831+0000","last_active":"2024-09-10T18:37:57.922106+0000","last_peered":"2024-09-10T18:37:57.922106+0000","last_clean":"2024-09-10T18:37:57.922106+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:37:57.922106+0000","last_undegraded":"2024-09-10T18:37:57.922106+0000","last_fullsized":"2024-09-10T18:37:57.922106+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'19","last_scrub_stamp":"2024-09-10T18:37:49.354781+0000","last_deep_scrub":"342'19","last_deep_scrub_stamp":"2024-09-10T18:37:49.354781+0000","last_clean_scrub_stamp":"2024-09-10T18:37:49.354781+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:15.025756+0000","scrub_duration":0.0096597390000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":369,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.930907+0000","last_change":"2024-09-10T18:37:12.854944+0000","last_active":"2024-09-10T18:37:57.930907+0000","last_peered":"2024-09-10T18:37:57.930907+0000","last_clean":"2024-09-10T18:37:57.930907+0000","last_became_active":"2024-09-10T18:21:35.619619+0000","last_became_peered":"2024-09-10T18:21:35.619619+0000","last_unstale":"2024-09-10T18:37:57.930907+0000","last_undegraded":"2024-09-10T18:37:57.930907+0000","last_fullsized":"2024-09-10T18:37:57.930907+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:12.854879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:12.854879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:41.911709+0000","scrub_duration":0.019720808999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":359,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.922912+0000","last_change":"2024-09-10T18:36:52.118582+0000","last_active":"2024-09-10T18:37:57.922912+0000","last_peered":"2024-09-10T18:37:57.922912+0000","last_clean":"2024-09-10T18:37:57.922912+0000","last_became_active":"2024-09-10T18:21:37.641520+0000","last_became_peered":"2024-09-10T18:21:37.641520+0000","last_unstale":"2024-09-10T18:37:57.922912+0000","last_undegraded":"2024-09-10T18:37:57.922912+0000","last_fullsized":"2024-09-10T18:37:57.922912+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:52.118515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:36:52.118515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:16.344464+0000","scrub_duration":0.0076434099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'81","reported_seq":73,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.928910+0000","last_change":"2024-09-10T18:36:48.858044+0000","last_active":"2024-09-10T18:37:57.928910+0000","last_peered":"2024-09-10T18:37:57.928910+0000","last_clean":"2024-09-10T18:37:57.928910+0000","last_became_active":"2024-09-10T18:34:34.867388+0000","last_became_peered":"2024-09-10T18:34:34.867388+0000","last_unstale":"2024-09-10T18:37:57.928910+0000","last_undegraded":"2024-09-10T18:37:57.928910+0000","last_fullsized":"2024-09-10T18:37:57.928910+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:48.857902+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:48.857902+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:09.493773+0000","scrub_duration":0.0059112879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":366,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.918217+0000","last_change":"2024-09-10T18:37:44.696143+0000","last_active":"2024-09-10T18:37:57.918217+0000","last_peered":"2024-09-10T18:37:57.918217+0000","last_clean":"2024-09-10T18:37:57.918217+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:37:57.918217+0000","last_undegraded":"2024-09-10T18:37:57.918217+0000","last_fullsized":"2024-09-10T18:37:57.918217+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:44.696052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:44.696052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:08.231801+0000","scrub_duration":0.044397580999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":372,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.923359+0000","last_change":"2024-09-10T18:37:40.756679+0000","last_active":"2024-09-10T18:37:57.923359+0000","last_peered":"2024-09-10T18:37:57.923359+0000","last_clean":"2024-09-10T18:37:57.923359+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:37:57.923359+0000","last_undegraded":"2024-09-10T18:37:57.923359+0000","last_fullsized":"2024-09-10T18:37:57.923359+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:40.756589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:37:40.756589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:51.667288+0000","scrub_duration":0.0073347990000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'34","reported_seq":403,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.919590+0000","last_change":"2024-09-10T18:36:10.786060+0000","last_active":"2024-09-10T18:37:57.919590+0000","last_peered":"2024-09-10T18:37:57.919590+0000","last_clean":"2024-09-10T18:37:57.919590+0000","last_became_active":"2024-09-10T18:21:33.613523+0000","last_became_peered":"2024-09-10T18:21:33.613523+0000","last_unstale":"2024-09-10T18:37:57.919590+0000","last_undegraded":"2024-09-10T18:37:57.919590+0000","last_fullsized":"2024-09-10T18:37:57.919590+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:36:10.786002+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:36:10.786002+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021577860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":368,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.922099+0000","last_change":"2024-09-10T18:36:07.325812+0000","last_active":"2024-09-10T18:37:57.922099+0000","last_peered":"2024-09-10T18:37:57.922099+0000","last_clean":"2024-09-10T18:37:57.922099+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:37:57.922099+0000","last_undegraded":"2024-09-10T18:37:57.922099+0000","last_fullsized":"2024-09-10T18:37:57.922099+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:07.325744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.325744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001764406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":383,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926981+0000","last_change":"2024-09-10T18:37:52.436944+0000","last_active":"2024-09-10T18:37:57.926981+0000","last_peered":"2024-09-10T18:37:57.926981+0000","last_clean":"2024-09-10T18:37:57.926981+0000","last_became_active":"2024-09-10T18:26:27.644961+0000","last_became_peered":"2024-09-10T18:26:27.644961+0000","last_unstale":"2024-09-10T18:37:57.926981+0000","last_undegraded":"2024-09-10T18:37:57.926981+0000","last_fullsized":"2024-09-10T18:37:57.926981+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:52.436889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_clean_scrub_stamp":"2024-09-10T18:37:52.436889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:18.456163+0000","scrub_duration":0.0013785469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'15","reported_seq":395,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.923341+0000","last_change":"2024-09-10T18:37:05.365783+0000","last_active":"2024-09-10T18:37:57.923341+0000","last_peered":"2024-09-10T18:37:57.923341+0000","last_clean":"2024-09-10T18:37:57.923341+0000","last_became_active":"2024-09-10T18:21:33.610493+0000","last_became_peered":"2024-09-10T18:21:33.610493+0000","last_unstale":"2024-09-10T18:37:57.923341+0000","last_undegraded":"2024-09-10T18:37:57.923341+0000","last_fullsized":"2024-09-10T18:37:57.923341+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:37:05.365702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:05.365702+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:27.722998+0000","scrub_duration":0.0029180819999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":369,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926579+0000","last_change":"2024-09-10T18:36:52.071972+0000","last_active":"2024-09-10T18:37:57.926579+0000","last_peered":"2024-09-10T18:37:57.926579+0000","last_clean":"2024-09-10T18:37:57.926579+0000","last_became_active":"2024-09-10T18:21:35.623796+0000","last_became_peered":"2024-09-10T18:21:35.623796+0000","last_unstale":"2024-09-10T18:37:57.926579+0000","last_undegraded":"2024-09-10T18:37:57.926579+0000","last_fullsized":"2024-09-10T18:37:57.926579+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:52.071894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_clean_scrub_stamp":"2024-09-10T18:36:52.071894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:05.580562+0000","scrub_duration":0.0025636399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":368,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.930524+0000","last_change":"2024-09-10T18:36:54.584423+0000","last_active":"2024-09-10T18:37:57.930524+0000","last_peered":"2024-09-10T18:37:57.930524+0000","last_clean":"2024-09-10T18:37:57.930524+0000","last_became_active":"2024-09-10T18:21:37.641705+0000","last_became_peered":"2024-09-10T18:21:37.641705+0000","last_unstale":"2024-09-10T18:37:57.930524+0000","last_undegraded":"2024-09-10T18:37:57.930524+0000","last_fullsized":"2024-09-10T18:37:57.930524+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:54.583785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:54.583785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:23.333666+0000","scrub_duration":0.0082206150000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'81","reported_seq":142,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.923172+0000","last_change":"2024-09-10T18:37:06.945729+0000","last_active":"2024-09-10T18:37:57.923172+0000","last_peered":"2024-09-10T18:37:57.923172+0000","last_clean":"2024-09-10T18:37:57.923172+0000","last_became_active":"2024-09-10T18:34:34.863102+0000","last_became_peered":"2024-09-10T18:34:34.863102+0000","last_unstale":"2024-09-10T18:37:57.923172+0000","last_undegraded":"2024-09-10T18:37:57.923172+0000","last_fullsized":"2024-09-10T18:37:57.923172+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:06.945645+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:06.945645+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:11.913296+0000","scrub_duration":0.0045824999999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":373,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926738+0000","last_change":"2024-09-10T18:36:56.124404+0000","last_active":"2024-09-10T18:37:57.926738+0000","last_peered":"2024-09-10T18:37:57.926738+0000","last_clean":"2024-09-10T18:37:57.926738+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:37:57.926738+0000","last_undegraded":"2024-09-10T18:37:57.926738+0000","last_fullsized":"2024-09-10T18:37:57.926738+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:56.124318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:17.464205+0000","last_clean_scrub_stamp":"2024-09-10T18:36:56.124318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:10.885488+0000","scrub_duration":0.0046808750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":407,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.920962+0000","last_change":"2024-09-10T18:37:09.383413+0000","last_active":"2024-09-10T18:37:57.920962+0000","last_peered":"2024-09-10T18:37:57.920962+0000","last_clean":"2024-09-10T18:37:57.920962+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:37:57.920962+0000","last_undegraded":"2024-09-10T18:37:57.920962+0000","last_fullsized":"2024-09-10T18:37:57.920962+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'9","last_scrub_stamp":"2024-09-10T18:37:09.383347+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:37:09.383347+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:10.796821+0000","scrub_duration":0.010843228999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":368,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.928320+0000","last_change":"2024-09-10T18:37:36.885626+0000","last_active":"2024-09-10T18:37:57.928320+0000","last_peered":"2024-09-10T18:37:57.928320+0000","last_clean":"2024-09-10T18:37:57.928320+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:37:57.928320+0000","last_undegraded":"2024-09-10T18:37:57.928320+0000","last_fullsized":"2024-09-10T18:37:57.928320+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:36.885547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:36.885547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:37.837293+0000","scrub_duration":0.0028632850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":360,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926292+0000","last_change":"2024-09-10T18:36:15.726285+0000","last_active":"2024-09-10T18:37:57.926292+0000","last_peered":"2024-09-10T18:37:57.926292+0000","last_clean":"2024-09-10T18:37:57.926292+0000","last_became_active":"2024-09-10T18:21:37.638973+0000","last_became_peered":"2024-09-10T18:21:37.638973+0000","last_unstale":"2024-09-10T18:37:57.926292+0000","last_undegraded":"2024-09-10T18:37:57.926292+0000","last_fullsized":"2024-09-10T18:37:57.926292+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:15.726200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:36:15.726200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020103579999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'81","reported_seq":225,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.918805+0000","last_change":"2024-09-10T18:37:07.984531+0000","last_active":"2024-09-10T18:37:57.918805+0000","last_peered":"2024-09-10T18:37:57.918805+0000","last_clean":"2024-09-10T18:37:57.918805+0000","last_became_active":"2024-09-10T18:34:34.862917+0000","last_became_peered":"2024-09-10T18:34:34.862917+0000","last_unstale":"2024-09-10T18:37:57.918805+0000","last_undegraded":"2024-09-10T18:37:57.918805+0000","last_fullsized":"2024-09-10T18:37:57.918805+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:07.984446+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:07.984446+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:16.037756+0000","scrub_duration":0.0054176850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":366,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.921229+0000","last_change":"2024-09-10T18:36:06.360654+0000","last_active":"2024-09-10T18:37:57.921229+0000","last_peered":"2024-09-10T18:37:57.921229+0000","last_clean":"2024-09-10T18:37:57.921229+0000","last_became_active":"2024-09-10T18:26:27.650774+0000","last_became_peered":"2024-09-10T18:26:27.650774+0000","last_unstale":"2024-09-10T18:37:57.921229+0000","last_undegraded":"2024-09-10T18:37:57.921229+0000","last_fullsized":"2024-09-10T18:37:57.921229+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.360595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.360595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012270849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":370,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.932186+0000","last_change":"2024-09-10T18:37:13.830411+0000","last_active":"2024-09-10T18:37:57.932186+0000","last_peered":"2024-09-10T18:37:57.932186+0000","last_clean":"2024-09-10T18:37:57.932186+0000","last_became_active":"2024-09-10T18:21:31.529587+0000","last_became_peered":"2024-09-10T18:21:31.529587+0000","last_unstale":"2024-09-10T18:37:57.932186+0000","last_undegraded":"2024-09-10T18:37:57.932186+0000","last_fullsized":"2024-09-10T18:37:57.932186+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:13.830325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:13.830325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:15.627331+0000","scrub_duration":0.0070701799999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"240'27","reported_seq":443,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.937858+0000","last_change":"2024-09-10T18:37:54.000296+0000","last_active":"2024-09-10T18:37:57.937858+0000","last_peered":"2024-09-10T18:37:57.937858+0000","last_clean":"2024-09-10T18:37:57.937858+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:37:57.937858+0000","last_undegraded":"2024-09-10T18:37:57.937858+0000","last_fullsized":"2024-09-10T18:37:57.937858+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:37:54.000202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:54.000202+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:07.885459+0000","scrub_duration":0.010729143999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":379,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.932475+0000","last_change":"2024-09-10T18:37:43.881718+0000","last_active":"2024-09-10T18:37:57.932475+0000","last_peered":"2024-09-10T18:37:57.932475+0000","last_clean":"2024-09-10T18:37:57.932475+0000","last_became_active":"2024-09-10T18:21:35.618247+0000","last_became_peered":"2024-09-10T18:21:35.618247+0000","last_unstale":"2024-09-10T18:37:57.932475+0000","last_undegraded":"2024-09-10T18:37:57.932475+0000","last_fullsized":"2024-09-10T18:37:57.932475+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:43.881588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:43.881588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:58.268455+0000","scrub_duration":0.001545126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'81","reported_seq":305,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.925041+0000","last_change":"2024-09-10T18:37:54.457037+0000","last_active":"2024-09-10T18:37:57.925041+0000","last_peered":"2024-09-10T18:37:57.925041+0000","last_clean":"2024-09-10T18:37:57.925041+0000","last_became_active":"2024-09-10T18:34:34.855693+0000","last_became_peered":"2024-09-10T18:34:34.855693+0000","last_unstale":"2024-09-10T18:37:57.925041+0000","last_undegraded":"2024-09-10T18:37:57.925041+0000","last_fullsized":"2024-09-10T18:37:57.925041+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:54.456947+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:54.456947+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:21.324625+0000","scrub_duration":0.0066326629999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":366,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.924347+0000","last_change":"2024-09-10T18:37:53.656341+0000","last_active":"2024-09-10T18:37:57.924347+0000","last_peered":"2024-09-10T18:37:57.924347+0000","last_clean":"2024-09-10T18:37:57.924347+0000","last_became_active":"2024-09-10T18:26:27.640035+0000","last_became_peered":"2024-09-10T18:26:27.640035+0000","last_unstale":"2024-09-10T18:37:57.924347+0000","last_undegraded":"2024-09-10T18:37:57.924347+0000","last_fullsized":"2024-09-10T18:37:57.924347+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:53.656232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:53.656232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:20.243836+0000","scrub_duration":0.014811193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":372,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.923536+0000","last_change":"2024-09-10T18:37:40.447566+0000","last_active":"2024-09-10T18:37:57.923536+0000","last_peered":"2024-09-10T18:37:57.923536+0000","last_clean":"2024-09-10T18:37:57.923536+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:37:57.923536+0000","last_undegraded":"2024-09-10T18:37:57.923536+0000","last_fullsized":"2024-09-10T18:37:57.923536+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:40.447489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:37:40.447489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:52.412017+0000","scrub_duration":0.0030489810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'23","reported_seq":390,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.923516+0000","last_change":"2024-09-10T18:35:14.739218+0000","last_active":"2024-09-10T18:37:57.923516+0000","last_peered":"2024-09-10T18:37:57.923516+0000","last_clean":"2024-09-10T18:37:57.923516+0000","last_became_active":"2024-09-10T18:21:33.617307+0000","last_became_peered":"2024-09-10T18:21:33.617307+0000","last_unstale":"2024-09-10T18:37:57.923516+0000","last_undegraded":"2024-09-10T18:37:57.923516+0000","last_fullsized":"2024-09-10T18:37:57.923516+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'23","last_scrub_stamp":"2024-09-10T18:35:14.739123+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:35:14.739123+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052949870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":374,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.922319+0000","last_change":"2024-09-10T18:37:36.407892+0000","last_active":"2024-09-10T18:37:57.922319+0000","last_peered":"2024-09-10T18:37:57.922319+0000","last_clean":"2024-09-10T18:37:57.922319+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:37:57.922319+0000","last_undegraded":"2024-09-10T18:37:57.922319+0000","last_fullsized":"2024-09-10T18:37:57.922319+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:36.407806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:08.266279+0000","last_clean_scrub_stamp":"2024-09-10T18:37:36.407806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:40.475305+0000","scrub_duration":0.0037520880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'81","reported_seq":283,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926386+0000","last_change":"2024-09-10T18:36:07.673249+0000","last_active":"2024-09-10T18:37:57.926386+0000","last_peered":"2024-09-10T18:37:57.926386+0000","last_clean":"2024-09-10T18:37:57.926386+0000","last_became_active":"2024-09-10T18:34:34.855038+0000","last_became_peered":"2024-09-10T18:34:34.855038+0000","last_unstale":"2024-09-10T18:37:57.926386+0000","last_undegraded":"2024-09-10T18:37:57.926386+0000","last_fullsized":"2024-09-10T18:37:57.926386+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:07.673176+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.673176+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018195469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":368,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.928785+0000","last_change":"2024-09-10T18:37:49.941956+0000","last_active":"2024-09-10T18:37:57.928785+0000","last_peered":"2024-09-10T18:37:57.928785+0000","last_clean":"2024-09-10T18:37:57.928785+0000","last_became_active":"2024-09-10T18:21:37.646349+0000","last_became_peered":"2024-09-10T18:21:37.646349+0000","last_unstale":"2024-09-10T18:37:57.928785+0000","last_undegraded":"2024-09-10T18:37:57.928785+0000","last_fullsized":"2024-09-10T18:37:57.928785+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:37:49.941880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:49.941880+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:58.420486+0000","scrub_duration":0.0041872790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":383,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.931204+0000","last_change":"2024-09-10T18:37:08.821787+0000","last_active":"2024-09-10T18:37:57.931204+0000","last_peered":"2024-09-10T18:37:57.931204+0000","last_clean":"2024-09-10T18:37:57.931204+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:37:57.931204+0000","last_undegraded":"2024-09-10T18:37:57.931204+0000","last_fullsized":"2024-09-10T18:37:57.931204+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:08.821692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:08.821692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:32.765319+0000","scrub_duration":0.0031507520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'25","reported_seq":407,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.929904+0000","last_change":"2024-09-10T18:37:00.670455+0000","last_active":"2024-09-10T18:37:57.929904+0000","last_peered":"2024-09-10T18:37:57.929904+0000","last_clean":"2024-09-10T18:37:57.929904+0000","last_became_active":"2024-09-10T18:21:34.046877+0000","last_became_peered":"2024-09-10T18:21:34.046877+0000","last_unstale":"2024-09-10T18:37:57.929904+0000","last_undegraded":"2024-09-10T18:37:57.929904+0000","last_fullsized":"2024-09-10T18:37:57.929904+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:37:00.670369+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:37:00.670369+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:06.286835+0000","scrub_duration":0.0057824440000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"325'12","reported_seq":49,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:37:59.951753+0000","last_change":"2024-09-10T18:36:52.891388+0000","last_active":"2024-09-10T18:37:59.951753+0000","last_peered":"2024-09-10T18:37:59.951753+0000","last_clean":"2024-09-10T18:37:59.951753+0000","last_became_active":"2024-09-10T18:36:52.882218+0000","last_became_peered":"2024-09-10T18:36:52.882218+0000","last_unstale":"2024-09-10T18:37:59.951753+0000","last_undegraded":"2024-09-10T18:37:59.951753+0000","last_fullsized":"2024-09-10T18:37:59.951753+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11","last_scrub_stamp":"2024-09-10T18:36:11.854492+0000","last_deep_scrub":"217'11","last_deep_scrub_stamp":"2024-09-10T18:30:03.161985+0000","last_clean_scrub_stamp":"2024-09-10T18:36:11.854492+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073290009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1","version":"344'9656","reported_seq":9694,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:38:04.916165+0000","last_change":"2024-09-10T18:36:07.904585+0000","last_active":"2024-09-10T18:38:04.916165+0000","last_peered":"2024-09-10T18:38:04.916165+0000","last_clean":"2024-09-10T18:38:04.916165+0000","last_became_active":"2024-09-10T18:36:07.904319+0000","last_became_peered":"2024-09-10T18:36:07.904319+0000","last_unstale":"2024-09-10T18:38:04.916165+0000","last_undegraded":"2024-09-10T18:38:04.916165+0000","last_fullsized":"2024-09-10T18:38:04.916165+0000","mapping_epoch":314,"log_start":"332'5600","ondisk_log_start":"332'5600","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4056,"log_dups_size":0,"ondisk_log_size":4056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478412800,"num_objects":7300,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9656,"num_write_kb":542592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":289,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926226+0000","last_change":"2024-09-10T18:35:59.488083+0000","last_active":"2024-09-10T18:37:57.926226+0000","last_peered":"2024-09-10T18:37:57.926226+0000","last_clean":"2024-09-10T18:37:57.926226+0000","last_became_active":"2024-09-10T18:34:32.823230+0000","last_became_peered":"2024-09-10T18:34:32.823230+0000","last_unstale":"2024-09-10T18:37:57.926226+0000","last_undegraded":"2024-09-10T18:37:57.926226+0000","last_fullsized":"2024-09-10T18:37:57.926226+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:59.488014+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:59.488014+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017684759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"240'34","reported_seq":434,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.933742+0000","last_change":"2024-09-10T18:37:03.818405+0000","last_active":"2024-09-10T18:37:57.933742+0000","last_peered":"2024-09-10T18:37:57.933742+0000","last_clean":"2024-09-10T18:37:57.933742+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:37:57.933742+0000","last_undegraded":"2024-09-10T18:37:57.933742+0000","last_fullsized":"2024-09-10T18:37:57.933742+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:37:03.818290+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:37:03.818290+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:17.664855+0000","scrub_duration":0.0051578769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":365,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.918699+0000","last_change":"2024-09-10T18:37:45.669336+0000","last_active":"2024-09-10T18:37:57.918699+0000","last_peered":"2024-09-10T18:37:57.918699+0000","last_clean":"2024-09-10T18:37:57.918699+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:37:57.918699+0000","last_undegraded":"2024-09-10T18:37:57.918699+0000","last_fullsized":"2024-09-10T18:37:57.918699+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:45.669248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:45.669248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:49.609493+0000","scrub_duration":0.0028065640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":567,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:37:59.935371+0000","last_change":"2024-09-10T18:37:03.315311+0000","last_active":"2024-09-10T18:37:59.935371+0000","last_peered":"2024-09-10T18:37:59.935371+0000","last_clean":"2024-09-10T18:37:59.935371+0000","last_became_active":"2024-09-10T18:21:35.624041+0000","last_became_peered":"2024-09-10T18:21:35.624041+0000","last_unstale":"2024-09-10T18:37:59.935371+0000","last_undegraded":"2024-09-10T18:37:59.935371+0000","last_fullsized":"2024-09-10T18:37:59.935371+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:37:03.315226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:03.315226+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:23.900377+0000","scrub_duration":0.017386324000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":367,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926638+0000","last_change":"2024-09-10T18:37:09.280044+0000","last_active":"2024-09-10T18:37:57.926638+0000","last_peered":"2024-09-10T18:37:57.926638+0000","last_clean":"2024-09-10T18:37:57.926638+0000","last_became_active":"2024-09-10T18:21:37.643752+0000","last_became_peered":"2024-09-10T18:21:37.643752+0000","last_unstale":"2024-09-10T18:37:57.926638+0000","last_undegraded":"2024-09-10T18:37:57.926638+0000","last_fullsized":"2024-09-10T18:37:57.926638+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:09.279969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:37:09.279969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:18.441371+0000","scrub_duration":0.0041856610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"344'9672","reported_seq":9710,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:38:04.906677+0000","last_change":"2024-09-10T18:36:07.900391+0000","last_active":"2024-09-10T18:38:04.906677+0000","last_peered":"2024-09-10T18:38:04.906677+0000","last_clean":"2024-09-10T18:38:04.906677+0000","last_became_active":"2024-09-10T18:36:07.900059+0000","last_became_peered":"2024-09-10T18:36:07.900059+0000","last_unstale":"2024-09-10T18:38:04.906677+0000","last_undegraded":"2024-09-10T18:38:04.906677+0000","last_fullsized":"2024-09-10T18:38:04.906677+0000","mapping_epoch":314,"log_start":"332'5600","ondisk_log_start":"332'5600","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4072,"log_dups_size":0,"ondisk_log_size":4072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476839936,"num_objects":7276,"num_object_clones":0,"num_object_copies":21828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9672,"num_write_kb":542336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":298,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.921662+0000","last_change":"2024-09-10T18:35:56.307718+0000","last_active":"2024-09-10T18:37:57.921662+0000","last_peered":"2024-09-10T18:37:57.921662+0000","last_clean":"2024-09-10T18:37:57.921662+0000","last_became_active":"2024-09-10T18:34:32.815839+0000","last_became_peered":"2024-09-10T18:34:32.815839+0000","last_unstale":"2024-09-10T18:37:57.921662+0000","last_undegraded":"2024-09-10T18:37:57.921662+0000","last_fullsized":"2024-09-10T18:37:57.921662+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:56.307649+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:56.307649+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015720840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":403,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.918489+0000","last_change":"2024-09-10T18:36:07.782175+0000","last_active":"2024-09-10T18:37:57.918489+0000","last_peered":"2024-09-10T18:37:57.918489+0000","last_clean":"2024-09-10T18:37:57.918489+0000","last_became_active":"2024-09-10T18:21:34.047027+0000","last_became_peered":"2024-09-10T18:21:34.047027+0000","last_unstale":"2024-09-10T18:37:57.918489+0000","last_undegraded":"2024-09-10T18:37:57.918489+0000","last_fullsized":"2024-09-10T18:37:57.918489+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-09-10T18:36:07.782112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.782112+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00244122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":355,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.924901+0000","last_change":"2024-09-10T18:35:15.765590+0000","last_active":"2024-09-10T18:37:57.924901+0000","last_peered":"2024-09-10T18:37:57.924901+0000","last_clean":"2024-09-10T18:37:57.924901+0000","last_became_active":"2024-09-10T18:21:31.536093+0000","last_became_peered":"2024-09-10T18:21:31.536093+0000","last_unstale":"2024-09-10T18:37:57.924901+0000","last_undegraded":"2024-09-10T18:37:57.924901+0000","last_fullsized":"2024-09-10T18:37:57.924901+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:15.765392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:15.765392+0000","last_clean_scrub_stamp":"2024-09-10T18:35:15.765392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01380637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":368,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.920887+0000","last_change":"2024-09-10T18:37:35.460385+0000","last_active":"2024-09-10T18:37:57.920887+0000","last_peered":"2024-09-10T18:37:57.920887+0000","last_clean":"2024-09-10T18:37:57.920887+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:37:57.920887+0000","last_undegraded":"2024-09-10T18:37:57.920887+0000","last_fullsized":"2024-09-10T18:37:57.920887+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:35.460269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:37:35.460269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:01.784794+0000","scrub_duration":0.0073608529999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":356,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.923163+0000","last_change":"2024-09-10T18:37:06.386024+0000","last_active":"2024-09-10T18:37:57.923163+0000","last_peered":"2024-09-10T18:37:57.923163+0000","last_clean":"2024-09-10T18:37:57.923163+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:37:57.923163+0000","last_undegraded":"2024-09-10T18:37:57.923163+0000","last_fullsized":"2024-09-10T18:37:57.923163+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:06.385944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:06.385944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:32.414011+0000","scrub_duration":0.003928651,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"344'9616","reported_seq":9654,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:38:00.978804+0000","last_change":"2024-09-10T18:36:07.902153+0000","last_active":"2024-09-10T18:38:00.978804+0000","last_peered":"2024-09-10T18:38:00.978804+0000","last_clean":"2024-09-10T18:38:00.978804+0000","last_became_active":"2024-09-10T18:36:07.901822+0000","last_became_peered":"2024-09-10T18:36:07.901822+0000","last_unstale":"2024-09-10T18:38:00.978804+0000","last_undegraded":"2024-09-10T18:38:00.978804+0000","last_fullsized":"2024-09-10T18:38:00.978804+0000","mapping_epoch":314,"log_start":"326'4600","ondisk_log_start":"326'4600","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503185408,"num_objects":7678,"num_object_clones":0,"num_object_copies":23034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9616,"num_write_kb":553408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":270,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.920453+0000","last_change":"2024-09-10T18:35:13.749157+0000","last_active":"2024-09-10T18:37:57.920453+0000","last_peered":"2024-09-10T18:37:57.920453+0000","last_clean":"2024-09-10T18:37:57.920453+0000","last_became_active":"2024-09-10T18:34:00.166362+0000","last_became_peered":"2024-09-10T18:34:00.166362+0000","last_unstale":"2024-09-10T18:37:57.920453+0000","last_undegraded":"2024-09-10T18:37:57.920453+0000","last_fullsized":"2024-09-10T18:37:57.920453+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:13.748963+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:13.748963+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013370432999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"240'28","reported_seq":409,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.921842+0000","last_change":"2024-09-10T18:36:16.155412+0000","last_active":"2024-09-10T18:37:57.921842+0000","last_peered":"2024-09-10T18:37:57.921842+0000","last_clean":"2024-09-10T18:37:57.921842+0000","last_became_active":"2024-09-10T18:26:27.651364+0000","last_became_peered":"2024-09-10T18:26:27.651364+0000","last_unstale":"2024-09-10T18:37:57.921842+0000","last_undegraded":"2024-09-10T18:37:57.921842+0000","last_fullsized":"2024-09-10T18:37:57.921842+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'28","last_scrub_stamp":"2024-09-10T18:36:16.155287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:16.155287+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027352050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":364,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.930337+0000","last_change":"2024-09-10T18:36:09.716889+0000","last_active":"2024-09-10T18:37:57.930337+0000","last_peered":"2024-09-10T18:37:57.930337+0000","last_clean":"2024-09-10T18:37:57.930337+0000","last_became_active":"2024-09-10T18:21:31.530988+0000","last_became_peered":"2024-09-10T18:21:31.530988+0000","last_unstale":"2024-09-10T18:37:57.930337+0000","last_undegraded":"2024-09-10T18:37:57.930337+0000","last_fullsized":"2024-09-10T18:37:57.930337+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:09.716820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:36:09.716820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037024689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":558,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:37:59.940659+0000","last_change":"2024-09-10T18:36:13.682715+0000","last_active":"2024-09-10T18:37:59.940659+0000","last_peered":"2024-09-10T18:37:59.940659+0000","last_clean":"2024-09-10T18:37:59.940659+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:37:59.940659+0000","last_undegraded":"2024-09-10T18:37:59.940659+0000","last_fullsized":"2024-09-10T18:37:59.940659+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:36:13.682649+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:36:13.682649+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001699954,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":366,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.927778+0000","last_change":"2024-09-10T18:36:53.979146+0000","last_active":"2024-09-10T18:37:57.927778+0000","last_peered":"2024-09-10T18:37:57.927778+0000","last_clean":"2024-09-10T18:37:57.927778+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:37:57.927778+0000","last_undegraded":"2024-09-10T18:37:57.927778+0000","last_fullsized":"2024-09-10T18:37:57.927778+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:53.979093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:36:53.979093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:15.772937+0000","scrub_duration":0.0047648930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"344'9823","reported_seq":9861,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:38:05.079439+0000","last_change":"2024-09-10T18:36:07.904923+0000","last_active":"2024-09-10T18:38:05.079439+0000","last_peered":"2024-09-10T18:38:05.079439+0000","last_clean":"2024-09-10T18:38:05.079439+0000","last_became_active":"2024-09-10T18:36:07.904606+0000","last_became_peered":"2024-09-10T18:36:07.904606+0000","last_unstale":"2024-09-10T18:38:05.079439+0000","last_undegraded":"2024-09-10T18:38:05.079439+0000","last_fullsized":"2024-09-10T18:38:05.079439+0000","mapping_epoch":314,"log_start":"332'5600","ondisk_log_start":"332'5600","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4223,"log_dups_size":0,"ondisk_log_size":4223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474677248,"num_objects":7243,"num_object_clones":0,"num_object_copies":21729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9823,"num_write_kb":546112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":369,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.925956+0000","last_change":"2024-09-10T18:37:44.836073+0000","last_active":"2024-09-10T18:37:57.925956+0000","last_peered":"2024-09-10T18:37:57.925956+0000","last_clean":"2024-09-10T18:37:57.925956+0000","last_became_active":"2024-09-10T18:34:32.818323+0000","last_became_peered":"2024-09-10T18:34:32.818323+0000","last_unstale":"2024-09-10T18:37:57.925956+0000","last_undegraded":"2024-09-10T18:37:57.925956+0000","last_fullsized":"2024-09-10T18:37:57.925956+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:44.835979+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:44.835979+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:54.965482+0000","scrub_duration":0.075098165999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":416,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.931786+0000","last_change":"2024-09-10T18:37:07.798909+0000","last_active":"2024-09-10T18:37:57.931786+0000","last_peered":"2024-09-10T18:37:57.931786+0000","last_clean":"2024-09-10T18:37:57.931786+0000","last_became_active":"2024-09-10T18:21:33.611409+0000","last_became_peered":"2024-09-10T18:21:33.611409+0000","last_unstale":"2024-09-10T18:37:57.931786+0000","last_undegraded":"2024-09-10T18:37:57.931786+0000","last_fullsized":"2024-09-10T18:37:57.931786+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:37:07.798798+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:37:07.798798+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:11.466713+0000","scrub_duration":0.0020202269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":363,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.927100+0000","last_change":"2024-09-10T18:37:11.243616+0000","last_active":"2024-09-10T18:37:57.927100+0000","last_peered":"2024-09-10T18:37:57.927100+0000","last_clean":"2024-09-10T18:37:57.927100+0000","last_became_active":"2024-09-10T18:21:31.533919+0000","last_became_peered":"2024-09-10T18:21:31.533919+0000","last_unstale":"2024-09-10T18:37:57.927100+0000","last_undegraded":"2024-09-10T18:37:57.927100+0000","last_fullsized":"2024-09-10T18:37:57.927100+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:11.243548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:37:11.243548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:29.339597+0000","scrub_duration":0.015664832,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"263'11","reported_seq":576,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:37:59.946825+0000","last_change":"2024-09-10T18:36:51.573695+0000","last_active":"2024-09-10T18:37:59.946825+0000","last_peered":"2024-09-10T18:37:59.946825+0000","last_clean":"2024-09-10T18:37:59.946825+0000","last_became_active":"2024-09-10T18:26:27.639032+0000","last_became_peered":"2024-09-10T18:26:27.639032+0000","last_unstale":"2024-09-10T18:37:59.946825+0000","last_undegraded":"2024-09-10T18:37:59.946825+0000","last_fullsized":"2024-09-10T18:37:59.946825+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'11","last_scrub_stamp":"2024-09-10T18:36:51.573610+0000","last_deep_scrub":"263'11","last_deep_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_clean_scrub_stamp":"2024-09-10T18:36:51.573610+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:06.768796+0000","scrub_duration":0.0034993289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":358,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.932969+0000","last_change":"2024-09-10T18:35:55.632357+0000","last_active":"2024-09-10T18:37:57.932969+0000","last_peered":"2024-09-10T18:37:57.932969+0000","last_clean":"2024-09-10T18:37:57.932969+0000","last_became_active":"2024-09-10T18:21:37.640117+0000","last_became_peered":"2024-09-10T18:21:37.640117+0000","last_unstale":"2024-09-10T18:37:57.932969+0000","last_undegraded":"2024-09-10T18:37:57.932969+0000","last_fullsized":"2024-09-10T18:37:57.932969+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:55.632306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:55.632306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001072857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"344'9456","reported_seq":9737,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:38:01.030217+0000","last_change":"2024-09-10T18:37:12.152091+0000","last_active":"2024-09-10T18:38:01.030217+0000","last_peered":"2024-09-10T18:38:01.030217+0000","last_clean":"2024-09-10T18:38:01.030217+0000","last_became_active":"2024-09-10T18:36:07.903637+0000","last_became_peered":"2024-09-10T18:36:07.903637+0000","last_unstale":"2024-09-10T18:38:01.030217+0000","last_undegraded":"2024-09-10T18:38:01.030217+0000","last_fullsized":"2024-09-10T18:38:01.030217+0000","mapping_epoch":314,"log_start":"325'4500","ondisk_log_start":"325'4500","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'5893","last_scrub_stamp":"2024-09-10T18:37:12.151975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:37:12.151975+0000","objects_scrubbed":5802,"log_size":4956,"log_dups_size":0,"ondisk_log_size":4956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:14.669808+0000","scrub_duration":2.3527053489999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":496959512,"num_objects":7584,"num_object_clones":0,"num_object_copies":22752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7584,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9456,"num_write_kb":545217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":359,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.925735+0000","last_change":"2024-09-10T18:36:08.872676+0000","last_active":"2024-09-10T18:37:57.925735+0000","last_peered":"2024-09-10T18:37:57.925735+0000","last_clean":"2024-09-10T18:37:57.925735+0000","last_became_active":"2024-09-10T18:34:32.918946+0000","last_became_peered":"2024-09-10T18:34:32.918946+0000","last_unstale":"2024-09-10T18:37:57.925735+0000","last_undegraded":"2024-09-10T18:37:57.925735+0000","last_fullsized":"2024-09-10T18:37:57.925735+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:08.872575+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:08.872575+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016711530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"240'29","reported_seq":423,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.931910+0000","last_change":"2024-09-10T18:36:59.647785+0000","last_active":"2024-09-10T18:37:57.931910+0000","last_peered":"2024-09-10T18:37:57.931910+0000","last_clean":"2024-09-10T18:37:57.931910+0000","last_became_active":"2024-09-10T18:21:34.046188+0000","last_became_peered":"2024-09-10T18:21:34.046188+0000","last_unstale":"2024-09-10T18:37:57.931910+0000","last_undegraded":"2024-09-10T18:37:57.931910+0000","last_fullsized":"2024-09-10T18:37:57.931910+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'29","last_scrub_stamp":"2024-09-10T18:36:59.647532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:59.647532+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:24.657907+0000","scrub_duration":0.0067920380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":373,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.932029+0000","last_change":"2024-09-10T18:37:57.001102+0000","last_active":"2024-09-10T18:37:57.932029+0000","last_peered":"2024-09-10T18:37:57.932029+0000","last_clean":"2024-09-10T18:37:57.932029+0000","last_became_active":"2024-09-10T18:21:31.529827+0000","last_became_peered":"2024-09-10T18:21:31.529827+0000","last_unstale":"2024-09-10T18:37:57.932029+0000","last_undegraded":"2024-09-10T18:37:57.932029+0000","last_fullsized":"2024-09-10T18:37:57.932029+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:57.001003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:57.001003+0000","last_clean_scrub_stamp":"2024-09-10T18:37:57.001003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:23.820070+0000","scrub_duration":0.0076916049999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":329,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926613+0000","last_change":"2024-09-10T18:37:12.856158+0000","last_active":"2024-09-10T18:37:57.926613+0000","last_peered":"2024-09-10T18:37:57.926613+0000","last_clean":"2024-09-10T18:37:57.926613+0000","last_became_active":"2024-09-10T18:26:27.642063+0000","last_became_peered":"2024-09-10T18:26:27.642063+0000","last_unstale":"2024-09-10T18:37:57.926613+0000","last_undegraded":"2024-09-10T18:37:57.926613+0000","last_fullsized":"2024-09-10T18:37:57.926613+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:12.856068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:12.856068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:14.246369+0000","scrub_duration":0.020901162000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":374,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.930774+0000","last_change":"2024-09-10T18:37:36.899082+0000","last_active":"2024-09-10T18:37:57.930774+0000","last_peered":"2024-09-10T18:37:57.930774+0000","last_clean":"2024-09-10T18:37:57.930774+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:37:57.930774+0000","last_undegraded":"2024-09-10T18:37:57.930774+0000","last_fullsized":"2024-09-10T18:37:57.930774+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:36.898949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:37:36.898949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:02.655713+0000","scrub_duration":0.0023913269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"344'9441","reported_seq":9814,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:38:01.038780+0000","last_change":"2024-09-10T18:37:45.092212+0000","last_active":"2024-09-10T18:38:01.038780+0000","last_peered":"2024-09-10T18:38:01.038780+0000","last_clean":"2024-09-10T18:38:01.038780+0000","last_became_active":"2024-09-10T18:36:07.903339+0000","last_became_peered":"2024-09-10T18:36:07.903339+0000","last_unstale":"2024-09-10T18:38:01.038780+0000","last_undegraded":"2024-09-10T18:38:01.038780+0000","last_fullsized":"2024-09-10T18:38:01.038780+0000","mapping_epoch":314,"log_start":"325'4500","ondisk_log_start":"325'4500","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'8250","last_scrub_stamp":"2024-09-10T18:37:45.092127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:37:45.092127+0000","objects_scrubbed":8128,"log_size":4941,"log_dups_size":0,"ondisk_log_size":4941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:53.542795+0000","scrub_duration":3.3537295650000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499843072,"num_objects":7627,"num_object_clones":0,"num_object_copies":22881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9441,"num_write_kb":546176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":274,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.925268+0000","last_change":"2024-09-10T18:36:55.135378+0000","last_active":"2024-09-10T18:37:57.925268+0000","last_peered":"2024-09-10T18:37:57.925268+0000","last_clean":"2024-09-10T18:37:57.925268+0000","last_became_active":"2024-09-10T18:34:00.151017+0000","last_became_peered":"2024-09-10T18:34:00.151017+0000","last_unstale":"2024-09-10T18:37:57.925268+0000","last_undegraded":"2024-09-10T18:37:57.925268+0000","last_fullsized":"2024-09-10T18:37:57.925268+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:55.135191+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:55.135191+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:04.408840+0000","scrub_duration":0.0072107639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"318'42","reported_seq":90,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:37:59.944888+0000","last_change":"2024-09-10T18:37:34.820009+0000","last_active":"2024-09-10T18:37:59.944888+0000","last_peered":"2024-09-10T18:37:59.944888+0000","last_clean":"2024-09-10T18:37:59.944888+0000","last_became_active":"2024-09-10T18:36:12.922665+0000","last_became_peered":"2024-09-10T18:36:12.922665+0000","last_unstale":"2024-09-10T18:37:59.944888+0000","last_undegraded":"2024-09-10T18:37:59.944888+0000","last_fullsized":"2024-09-10T18:37:59.944888+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'42","last_scrub_stamp":"2024-09-10T18:37:34.819731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:34.819731+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:47.157368+0000","scrub_duration":0.004494156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":369,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.921252+0000","last_change":"2024-09-10T18:37:01.274315+0000","last_active":"2024-09-10T18:37:57.921252+0000","last_peered":"2024-09-10T18:37:57.921252+0000","last_clean":"2024-09-10T18:37:57.921252+0000","last_became_active":"2024-09-10T18:26:27.648690+0000","last_became_peered":"2024-09-10T18:26:27.648690+0000","last_unstale":"2024-09-10T18:37:57.921252+0000","last_undegraded":"2024-09-10T18:37:57.921252+0000","last_fullsized":"2024-09-10T18:37:57.921252+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:01.274219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.274219+0000","last_clean_scrub_stamp":"2024-09-10T18:37:01.274219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:17.545348+0000","scrub_duration":0.0059785009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":388,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926818+0000","last_change":"2024-09-10T18:37:48.497659+0000","last_active":"2024-09-10T18:37:57.926818+0000","last_peered":"2024-09-10T18:37:57.926818+0000","last_clean":"2024-09-10T18:37:57.926818+0000","last_became_active":"2024-09-10T18:26:27.647576+0000","last_became_peered":"2024-09-10T18:26:27.647576+0000","last_unstale":"2024-09-10T18:37:57.926818+0000","last_undegraded":"2024-09-10T18:37:57.926818+0000","last_fullsized":"2024-09-10T18:37:57.926818+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:37:48.497410+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:37:48.497410+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:52.992263+0000","scrub_duration":0.0087104109999999995,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":370,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.932242+0000","last_change":"2024-09-10T18:37:56.658238+0000","last_active":"2024-09-10T18:37:57.932242+0000","last_peered":"2024-09-10T18:37:57.932242+0000","last_clean":"2024-09-10T18:37:57.932242+0000","last_became_active":"2024-09-10T18:21:35.622411+0000","last_became_peered":"2024-09-10T18:21:35.622411+0000","last_unstale":"2024-09-10T18:37:57.932242+0000","last_undegraded":"2024-09-10T18:37:57.932242+0000","last_fullsized":"2024-09-10T18:37:57.932242+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:56.658164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:56.658164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:18.698426+0000","scrub_duration":0.0062309560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":349,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.923040+0000","last_change":"2024-09-10T18:37:37.700573+0000","last_active":"2024-09-10T18:37:57.923040+0000","last_peered":"2024-09-10T18:37:57.923040+0000","last_clean":"2024-09-10T18:37:57.923040+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:37:57.923040+0000","last_undegraded":"2024-09-10T18:37:57.923040+0000","last_fullsized":"2024-09-10T18:37:57.923040+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:37.700481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:37.700481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:54.313491+0000","scrub_duration":0.0022681839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"344'9445","reported_seq":9822,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:38:00.147681+0000","last_change":"2024-09-10T18:37:55.970768+0000","last_active":"2024-09-10T18:38:00.147681+0000","last_peered":"2024-09-10T18:38:00.147681+0000","last_clean":"2024-09-10T18:38:00.147681+0000","last_became_active":"2024-09-10T18:36:07.906264+0000","last_became_peered":"2024-09-10T18:36:07.906264+0000","last_unstale":"2024-09-10T18:38:00.147681+0000","last_undegraded":"2024-09-10T18:38:00.147681+0000","last_fullsized":"2024-09-10T18:38:00.147681+0000","mapping_epoch":314,"log_start":"326'4851","ondisk_log_start":"326'4851","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'9120","last_scrub_stamp":"2024-09-10T18:37:55.970682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:37:55.970682+0000","objects_scrubbed":8249,"log_size":4594,"log_dups_size":0,"ondisk_log_size":4594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:09.820371+0000","scrub_duration":2.5588804870000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513736704,"num_objects":7839,"num_object_clones":0,"num_object_copies":23517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9445,"num_write_kb":553088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":413,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.928136+0000","last_change":"2024-09-10T18:37:55.910086+0000","last_active":"2024-09-10T18:37:57.928136+0000","last_peered":"2024-09-10T18:37:57.928136+0000","last_clean":"2024-09-10T18:37:57.928136+0000","last_became_active":"2024-09-10T18:34:00.154404+0000","last_became_peered":"2024-09-10T18:34:00.154404+0000","last_unstale":"2024-09-10T18:37:57.928136+0000","last_undegraded":"2024-09-10T18:37:57.928136+0000","last_fullsized":"2024-09-10T18:37:57.928136+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:35:08.748082+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:35:08.748082+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038002869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"240'17","reported_seq":404,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.933845+0000","last_change":"2024-09-10T18:36:49.797325+0000","last_active":"2024-09-10T18:37:57.933845+0000","last_peered":"2024-09-10T18:37:57.933845+0000","last_clean":"2024-09-10T18:37:57.933845+0000","last_became_active":"2024-09-10T18:21:33.617444+0000","last_became_peered":"2024-09-10T18:21:33.617444+0000","last_unstale":"2024-09-10T18:37:57.933845+0000","last_undegraded":"2024-09-10T18:37:57.933845+0000","last_fullsized":"2024-09-10T18:37:57.933845+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'17","last_scrub_stamp":"2024-09-10T18:36:49.797243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:49.797243+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030959260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":390,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.933663+0000","last_change":"2024-09-10T18:37:46.930362+0000","last_active":"2024-09-10T18:37:57.933663+0000","last_peered":"2024-09-10T18:37:57.933663+0000","last_clean":"2024-09-10T18:37:57.933663+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:37:57.933663+0000","last_undegraded":"2024-09-10T18:37:57.933663+0000","last_fullsized":"2024-09-10T18:37:57.933663+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:46.930282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:15.141982+0000","last_clean_scrub_stamp":"2024-09-10T18:37:46.930282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:54.341486+0000","scrub_duration":0.0067994529999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":401,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.938806+0000","last_change":"2024-09-10T18:37:04.920700+0000","last_active":"2024-09-10T18:37:57.938806+0000","last_peered":"2024-09-10T18:37:57.938806+0000","last_clean":"2024-09-10T18:37:57.938806+0000","last_became_active":"2024-09-10T18:21:31.525519+0000","last_became_peered":"2024-09-10T18:21:31.525519+0000","last_unstale":"2024-09-10T18:37:57.938806+0000","last_undegraded":"2024-09-10T18:37:57.938806+0000","last_fullsized":"2024-09-10T18:37:57.938806+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:37:04.920615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:04.920615+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:20.515535+0000","scrub_duration":0.01495838,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":388,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.939309+0000","last_change":"2024-09-10T18:37:55.985494+0000","last_active":"2024-09-10T18:37:57.939309+0000","last_peered":"2024-09-10T18:37:57.939309+0000","last_clean":"2024-09-10T18:37:57.939309+0000","last_became_active":"2024-09-10T18:21:35.620330+0000","last_became_peered":"2024-09-10T18:21:35.620330+0000","last_unstale":"2024-09-10T18:37:57.939309+0000","last_undegraded":"2024-09-10T18:37:57.939309+0000","last_fullsized":"2024-09-10T18:37:57.939309+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:55.985235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:37:55.985235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:02.408912+0000","scrub_duration":0.003013997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":366,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.928966+0000","last_change":"2024-09-10T18:37:02.114698+0000","last_active":"2024-09-10T18:37:57.928966+0000","last_peered":"2024-09-10T18:37:57.928966+0000","last_clean":"2024-09-10T18:37:57.928966+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:37:57.928966+0000","last_undegraded":"2024-09-10T18:37:57.928966+0000","last_fullsized":"2024-09-10T18:37:57.928966+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:02.114602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:02.114602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:21.588925+0000","scrub_duration":0.0019755279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"344'9466","reported_seq":9836,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:38:01.316286+0000","last_change":"2024-09-10T18:37:57.112987+0000","last_active":"2024-09-10T18:38:01.316286+0000","last_peered":"2024-09-10T18:38:01.316286+0000","last_clean":"2024-09-10T18:38:01.316286+0000","last_became_active":"2024-09-10T18:36:07.901516+0000","last_became_peered":"2024-09-10T18:36:07.901516+0000","last_unstale":"2024-09-10T18:38:01.316286+0000","last_undegraded":"2024-09-10T18:38:01.316286+0000","last_fullsized":"2024-09-10T18:38:01.316286+0000","mapping_epoch":314,"log_start":"328'4900","ondisk_log_start":"328'4900","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'9133","last_scrub_stamp":"2024-09-10T18:37:57.112895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:37:57.112895+0000","objects_scrubbed":8065,"log_size":4566,"log_dups_size":0,"ondisk_log_size":4566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:08.848837+0000","scrub_duration":3.4771762449999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499908608,"num_objects":7628,"num_object_clones":0,"num_object_copies":22884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9466,"num_write_kb":547008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":290,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.931443+0000","last_change":"2024-09-10T18:36:59.835814+0000","last_active":"2024-09-10T18:37:57.931443+0000","last_peered":"2024-09-10T18:37:57.931443+0000","last_clean":"2024-09-10T18:37:57.931443+0000","last_became_active":"2024-09-10T18:34:00.165384+0000","last_became_peered":"2024-09-10T18:34:00.165384+0000","last_unstale":"2024-09-10T18:37:57.931443+0000","last_undegraded":"2024-09-10T18:37:57.931443+0000","last_fullsized":"2024-09-10T18:37:57.931443+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:59.835713+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:59.835713+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:16.626287+0000","scrub_duration":0.029401394000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":398,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.929990+0000","last_change":"2024-09-10T18:36:15.636499+0000","last_active":"2024-09-10T18:37:57.929990+0000","last_peered":"2024-09-10T18:37:57.929990+0000","last_clean":"2024-09-10T18:37:57.929990+0000","last_became_active":"2024-09-10T18:21:33.613499+0000","last_became_peered":"2024-09-10T18:21:33.613499+0000","last_unstale":"2024-09-10T18:37:57.929990+0000","last_undegraded":"2024-09-10T18:37:57.929990+0000","last_fullsized":"2024-09-10T18:37:57.929990+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-09-10T18:36:15.636442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:15.636442+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017190630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":374,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.930019+0000","last_change":"2024-09-10T18:36:55.603164+0000","last_active":"2024-09-10T18:37:57.930019+0000","last_peered":"2024-09-10T18:37:57.930019+0000","last_clean":"2024-09-10T18:37:57.930019+0000","last_became_active":"2024-09-10T18:21:31.534235+0000","last_became_peered":"2024-09-10T18:21:31.534235+0000","last_unstale":"2024-09-10T18:37:57.930019+0000","last_undegraded":"2024-09-10T18:37:57.930019+0000","last_fullsized":"2024-09-10T18:37:57.930019+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:55.603085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:36:55.603085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:23.221490+0000","scrub_duration":0.012495984999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":1365,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:38:03.145285+0000","last_change":"2024-09-10T18:37:01.094318+0000","last_active":"2024-09-10T18:38:03.145285+0000","last_peered":"2024-09-10T18:38:03.145285+0000","last_clean":"2024-09-10T18:38:03.145285+0000","last_became_active":"2024-09-10T18:21:31.528887+0000","last_became_peered":"2024-09-10T18:21:31.528887+0000","last_unstale":"2024-09-10T18:38:03.145285+0000","last_undegraded":"2024-09-10T18:38:03.145285+0000","last_fullsized":"2024-09-10T18:38:03.145285+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:37:01.094063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:01.094063+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:23.788462+0000","scrub_duration":0.0099775509999999994,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":985,"num_read_kb":980,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":362,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.927319+0000","last_change":"2024-09-10T18:37:01.170467+0000","last_active":"2024-09-10T18:37:57.927319+0000","last_peered":"2024-09-10T18:37:57.927319+0000","last_clean":"2024-09-10T18:37:57.927319+0000","last_became_active":"2024-09-10T18:21:35.619484+0000","last_became_peered":"2024-09-10T18:21:35.619484+0000","last_unstale":"2024-09-10T18:37:57.927319+0000","last_undegraded":"2024-09-10T18:37:57.927319+0000","last_fullsized":"2024-09-10T18:37:57.927319+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:01.170272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.170272+0000","last_clean_scrub_stamp":"2024-09-10T18:37:01.170272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:05.863430+0000","scrub_duration":0.0065986719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":367,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.931631+0000","last_change":"2024-09-10T18:37:51.000441+0000","last_active":"2024-09-10T18:37:57.931631+0000","last_peered":"2024-09-10T18:37:57.931631+0000","last_clean":"2024-09-10T18:37:57.931631+0000","last_became_active":"2024-09-10T18:21:37.639599+0000","last_became_peered":"2024-09-10T18:21:37.639599+0000","last_unstale":"2024-09-10T18:37:57.931631+0000","last_undegraded":"2024-09-10T18:37:57.931631+0000","last_fullsized":"2024-09-10T18:37:57.931631+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:51.000352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:51.000352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:11.836888+0000","scrub_duration":0.005761296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"344'9385","reported_seq":9758,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:38:01.012380+0000","last_change":"2024-09-10T18:37:56.338904+0000","last_active":"2024-09-10T18:38:01.012380+0000","last_peered":"2024-09-10T18:38:01.012380+0000","last_clean":"2024-09-10T18:38:01.012380+0000","last_became_active":"2024-09-10T18:36:07.903576+0000","last_became_peered":"2024-09-10T18:36:07.903576+0000","last_unstale":"2024-09-10T18:38:01.012380+0000","last_undegraded":"2024-09-10T18:38:01.012380+0000","last_fullsized":"2024-09-10T18:38:01.012380+0000","mapping_epoch":314,"log_start":"325'4400","ondisk_log_start":"325'4400","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'9007","last_scrub_stamp":"2024-09-10T18:37:56.338431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:37:56.338431+0000","objects_scrubbed":8091,"log_size":4985,"log_dups_size":0,"ondisk_log_size":4985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:20.060186+0000","scrub_duration":3.5911181970000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":497876992,"num_objects":7597,"num_object_clones":0,"num_object_copies":22791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9385,"num_write_kb":543424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"240'26","reported_seq":366,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.927247+0000","last_change":"2024-09-10T18:37:33.681193+0000","last_active":"2024-09-10T18:37:57.927247+0000","last_peered":"2024-09-10T18:37:57.927247+0000","last_clean":"2024-09-10T18:37:57.927247+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:37:57.927247+0000","last_undegraded":"2024-09-10T18:37:57.927247+0000","last_fullsized":"2024-09-10T18:37:57.927247+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'26","last_scrub_stamp":"2024-09-10T18:37:33.681108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:33.681108+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:01.752280+0000","scrub_duration":0.0099252360000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":372,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926123+0000","last_change":"2024-09-10T18:37:43.711029+0000","last_active":"2024-09-10T18:37:57.926123+0000","last_peered":"2024-09-10T18:37:57.926123+0000","last_clean":"2024-09-10T18:37:57.926123+0000","last_became_active":"2024-09-10T18:26:27.639129+0000","last_became_peered":"2024-09-10T18:26:27.639129+0000","last_unstale":"2024-09-10T18:37:57.926123+0000","last_undegraded":"2024-09-10T18:37:57.926123+0000","last_fullsized":"2024-09-10T18:37:57.926123+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:43.710936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:43.710936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:58.627087+0000","scrub_duration":0.010235932999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":432,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.928630+0000","last_change":"2024-09-10T18:37:11.982261+0000","last_active":"2024-09-10T18:37:57.928630+0000","last_peered":"2024-09-10T18:37:57.928630+0000","last_clean":"2024-09-10T18:37:57.928630+0000","last_became_active":"2024-09-10T18:34:00.159975+0000","last_became_peered":"2024-09-10T18:34:00.159975+0000","last_unstale":"2024-09-10T18:37:57.928630+0000","last_undegraded":"2024-09-10T18:37:57.928630+0000","last_fullsized":"2024-09-10T18:37:57.928630+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:11.982173+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:11.982173+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:40.227262+0000","scrub_duration":0.001824406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":368,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.928870+0000","last_change":"2024-09-10T18:36:58.015546+0000","last_active":"2024-09-10T18:37:57.928870+0000","last_peered":"2024-09-10T18:37:57.928870+0000","last_clean":"2024-09-10T18:37:57.928870+0000","last_became_active":"2024-09-10T18:21:35.621062+0000","last_became_peered":"2024-09-10T18:21:35.621062+0000","last_unstale":"2024-09-10T18:37:57.928870+0000","last_undegraded":"2024-09-10T18:37:57.928870+0000","last_fullsized":"2024-09-10T18:37:57.928870+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:58.015470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:58.015470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:03.398913+0000","scrub_duration":0.0043498190000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":360,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.927188+0000","last_change":"2024-09-10T18:36:59.120950+0000","last_active":"2024-09-10T18:37:57.927188+0000","last_peered":"2024-09-10T18:37:57.927188+0000","last_clean":"2024-09-10T18:37:57.927188+0000","last_became_active":"2024-09-10T18:21:37.638558+0000","last_became_peered":"2024-09-10T18:21:37.638558+0000","last_unstale":"2024-09-10T18:37:57.927188+0000","last_undegraded":"2024-09-10T18:37:57.927188+0000","last_fullsized":"2024-09-10T18:37:57.927188+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:59.120687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:36:59.120687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:13.759657+0000","scrub_duration":0.0049055239999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"344'9616","reported_seq":9653,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:38:01.041567+0000","last_change":"2024-09-10T18:36:07.903096+0000","last_active":"2024-09-10T18:38:01.041567+0000","last_peered":"2024-09-10T18:38:01.041567+0000","last_clean":"2024-09-10T18:38:01.041567+0000","last_became_active":"2024-09-10T18:36:07.902779+0000","last_became_peered":"2024-09-10T18:36:07.902779+0000","last_unstale":"2024-09-10T18:38:01.041567+0000","last_undegraded":"2024-09-10T18:38:01.041567+0000","last_fullsized":"2024-09-10T18:38:01.041567+0000","mapping_epoch":314,"log_start":"325'4600","ondisk_log_start":"325'4600","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508297216,"num_objects":7756,"num_object_clones":0,"num_object_copies":23268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9616,"num_write_kb":555904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"76'81","reported_seq":309,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.931590+0000","last_change":"2024-09-10T18:37:37.918636+0000","last_active":"2024-09-10T18:37:57.931590+0000","last_peered":"2024-09-10T18:37:57.931590+0000","last_clean":"2024-09-10T18:37:57.931590+0000","last_became_active":"2024-09-10T18:34:32.927460+0000","last_became_peered":"2024-09-10T18:34:32.927460+0000","last_unstale":"2024-09-10T18:37:57.931590+0000","last_undegraded":"2024-09-10T18:37:57.931590+0000","last_fullsized":"2024-09-10T18:37:57.931590+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:37.918433+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:37.918433+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:06.609650+0000","scrub_duration":0.0051962880000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"283'24","reported_seq":406,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.921376+0000","last_change":"2024-09-10T18:36:58.225291+0000","last_active":"2024-09-10T18:37:57.921376+0000","last_peered":"2024-09-10T18:37:57.921376+0000","last_clean":"2024-09-10T18:37:57.921376+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:37:57.921376+0000","last_undegraded":"2024-09-10T18:37:57.921376+0000","last_fullsized":"2024-09-10T18:37:57.921376+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'24","last_scrub_stamp":"2024-09-10T18:36:58.225206+0000","last_deep_scrub":"283'24","last_deep_scrub_stamp":"2024-09-10T18:34:19.404713+0000","last_clean_scrub_stamp":"2024-09-10T18:36:58.225206+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:16.467424+0000","scrub_duration":0.0055050170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":362,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.921997+0000","last_change":"2024-09-10T18:37:00.246542+0000","last_active":"2024-09-10T18:37:57.921997+0000","last_peered":"2024-09-10T18:37:57.921997+0000","last_clean":"2024-09-10T18:37:57.921997+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:37:57.921997+0000","last_undegraded":"2024-09-10T18:37:57.921997+0000","last_fullsized":"2024-09-10T18:37:57.921997+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:00.246398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:00.246398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:00.862550+0000","scrub_duration":0.003194966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":376,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926953+0000","last_change":"2024-09-10T18:37:35.385548+0000","last_active":"2024-09-10T18:37:57.926953+0000","last_peered":"2024-09-10T18:37:57.926953+0000","last_clean":"2024-09-10T18:37:57.926953+0000","last_became_active":"2024-09-10T18:26:27.645885+0000","last_became_peered":"2024-09-10T18:26:27.645885+0000","last_unstale":"2024-09-10T18:37:57.926953+0000","last_undegraded":"2024-09-10T18:37:57.926953+0000","last_fullsized":"2024-09-10T18:37:57.926953+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:35.385189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:37:35.385189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:59.661914+0000","scrub_duration":0.0083152009999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":366,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.920852+0000","last_change":"2024-09-10T18:37:34.723627+0000","last_active":"2024-09-10T18:37:57.920852+0000","last_peered":"2024-09-10T18:37:57.920852+0000","last_clean":"2024-09-10T18:37:57.920852+0000","last_became_active":"2024-09-10T18:26:27.632071+0000","last_became_peered":"2024-09-10T18:26:27.632071+0000","last_unstale":"2024-09-10T18:37:57.920852+0000","last_undegraded":"2024-09-10T18:37:57.920852+0000","last_fullsized":"2024-09-10T18:37:57.920852+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:34.723544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:34.723544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:04.166049+0000","scrub_duration":0.045211744999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"344'9255","reported_seq":9634,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:38:00.250714+0000","last_change":"2024-09-10T18:37:52.837433+0000","last_active":"2024-09-10T18:38:00.250714+0000","last_peered":"2024-09-10T18:38:00.250714+0000","last_clean":"2024-09-10T18:38:00.250714+0000","last_became_active":"2024-09-10T18:36:07.903429+0000","last_became_peered":"2024-09-10T18:36:07.903429+0000","last_unstale":"2024-09-10T18:38:00.250714+0000","last_undegraded":"2024-09-10T18:38:00.250714+0000","last_fullsized":"2024-09-10T18:38:00.250714+0000","mapping_epoch":314,"log_start":"322'3400","ondisk_log_start":"322'3400","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'8736","last_scrub_stamp":"2024-09-10T18:37:52.837373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:37:52.837373+0000","objects_scrubbed":8281,"log_size":5855,"log_dups_size":0,"ondisk_log_size":5855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:56.830472+0000","scrub_duration":3.8888567959999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499843072,"num_objects":7627,"num_object_clones":0,"num_object_copies":22881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9255,"num_write_kb":540224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":291,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.924916+0000","last_change":"2024-09-10T18:37:40.540315+0000","last_active":"2024-09-10T18:37:57.924916+0000","last_peered":"2024-09-10T18:37:57.924916+0000","last_clean":"2024-09-10T18:37:57.924916+0000","last_became_active":"2024-09-10T18:34:32.823976+0000","last_became_peered":"2024-09-10T18:34:32.823976+0000","last_unstale":"2024-09-10T18:37:57.924916+0000","last_undegraded":"2024-09-10T18:37:57.924916+0000","last_fullsized":"2024-09-10T18:37:57.924916+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:40.540266+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:40.540266+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:51.190401+0000","scrub_duration":0.001149256,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"342'39","reported_seq":455,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.921530+0000","last_change":"2024-09-10T18:37:53.386639+0000","last_active":"2024-09-10T18:37:57.921530+0000","last_peered":"2024-09-10T18:37:57.921530+0000","last_clean":"2024-09-10T18:37:57.921530+0000","last_became_active":"2024-09-10T18:37:11.817595+0000","last_became_peered":"2024-09-10T18:37:11.817595+0000","last_unstale":"2024-09-10T18:37:57.921530+0000","last_undegraded":"2024-09-10T18:37:57.921530+0000","last_fullsized":"2024-09-10T18:37:57.921530+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'39","last_scrub_stamp":"2024-09-10T18:37:53.386563+0000","last_deep_scrub":"342'39","last_deep_scrub_stamp":"2024-09-10T18:37:53.386563+0000","last_clean_scrub_stamp":"2024-09-10T18:37:53.386563+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:21.503072+0000","scrub_duration":0.031998117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":337,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926868+0000","last_change":"2024-09-10T18:37:51.713223+0000","last_active":"2024-09-10T18:37:57.926868+0000","last_peered":"2024-09-10T18:37:57.926868+0000","last_clean":"2024-09-10T18:37:57.926868+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:37:57.926868+0000","last_undegraded":"2024-09-10T18:37:57.926868+0000","last_fullsized":"2024-09-10T18:37:57.926868+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:37:51.712547+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:37:51.712547+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:52.493485+0000","scrub_duration":0.014827761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":369,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.931743+0000","last_change":"2024-09-10T18:36:57.784141+0000","last_active":"2024-09-10T18:37:57.931743+0000","last_peered":"2024-09-10T18:37:57.931743+0000","last_clean":"2024-09-10T18:37:57.931743+0000","last_became_active":"2024-09-10T18:21:35.624880+0000","last_became_peered":"2024-09-10T18:21:35.624880+0000","last_unstale":"2024-09-10T18:37:57.931743+0000","last_undegraded":"2024-09-10T18:37:57.931743+0000","last_fullsized":"2024-09-10T18:37:57.931743+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:57.784049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:57.784049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:15.415282+0000","scrub_duration":0.0022794579999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":396,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.939205+0000","last_change":"2024-09-10T18:37:41.969779+0000","last_active":"2024-09-10T18:37:57.939205+0000","last_peered":"2024-09-10T18:37:57.939205+0000","last_clean":"2024-09-10T18:37:57.939205+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:37:57.939205+0000","last_undegraded":"2024-09-10T18:37:57.939205+0000","last_fullsized":"2024-09-10T18:37:57.939205+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:41.969690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:41.969690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:54.387221+0000","scrub_duration":0.013948959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"344'9643","reported_seq":10036,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:38:00.568291+0000","last_change":"2024-09-10T18:37:49.295534+0000","last_active":"2024-09-10T18:38:00.568291+0000","last_peered":"2024-09-10T18:38:00.568291+0000","last_clean":"2024-09-10T18:38:00.568291+0000","last_became_active":"2024-09-10T18:36:07.903825+0000","last_became_peered":"2024-09-10T18:36:07.903825+0000","last_unstale":"2024-09-10T18:38:00.568291+0000","last_undegraded":"2024-09-10T18:38:00.568291+0000","last_fullsized":"2024-09-10T18:38:00.568291+0000","mapping_epoch":314,"log_start":"329'5100","ondisk_log_start":"329'5100","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'8746","last_scrub_stamp":"2024-09-10T18:37:49.295432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:37:49.295432+0000","objects_scrubbed":8632,"log_size":4543,"log_dups_size":0,"ondisk_log_size":4543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:56.953942+0000","scrub_duration":3.4094893449999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511901696,"num_objects":7811,"num_object_clones":0,"num_object_copies":23433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9643,"num_write_kb":558528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":420,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.927818+0000","last_change":"2024-09-10T18:36:13.951388+0000","last_active":"2024-09-10T18:37:57.927818+0000","last_peered":"2024-09-10T18:37:57.927818+0000","last_clean":"2024-09-10T18:37:57.927818+0000","last_became_active":"2024-09-10T18:34:32.827174+0000","last_became_peered":"2024-09-10T18:34:32.827174+0000","last_unstale":"2024-09-10T18:37:57.927818+0000","last_undegraded":"2024-09-10T18:37:57.927818+0000","last_fullsized":"2024-09-10T18:37:57.927818+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:13.951321+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:13.951321+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015868600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"240'25","reported_seq":130,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.928688+0000","last_change":"2024-09-10T18:37:14.054480+0000","last_active":"2024-09-10T18:37:57.928688+0000","last_peered":"2024-09-10T18:37:57.928688+0000","last_clean":"2024-09-10T18:37:57.928688+0000","last_became_active":"2024-09-10T18:32:51.118251+0000","last_became_peered":"2024-09-10T18:32:51.118251+0000","last_unstale":"2024-09-10T18:37:57.928688+0000","last_undegraded":"2024-09-10T18:37:57.928688+0000","last_fullsized":"2024-09-10T18:37:57.928688+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:37:14.054170+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:37:14.054170+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:41.275620+0000","scrub_duration":0.019668199000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":355,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.919378+0000","last_change":"2024-09-10T18:36:03.769108+0000","last_active":"2024-09-10T18:37:57.919378+0000","last_peered":"2024-09-10T18:37:57.919378+0000","last_clean":"2024-09-10T18:37:57.919378+0000","last_became_active":"2024-09-10T18:21:31.535158+0000","last_became_peered":"2024-09-10T18:21:31.535158+0000","last_unstale":"2024-09-10T18:37:57.919378+0000","last_undegraded":"2024-09-10T18:37:57.919378+0000","last_fullsized":"2024-09-10T18:37:57.919378+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:03.769053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:13.324556+0000","last_clean_scrub_stamp":"2024-09-10T18:36:03.769053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001666634,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"189'9","reported_seq":605,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:37:59.936257+0000","last_change":"2024-09-10T18:37:10.969567+0000","last_active":"2024-09-10T18:37:59.936257+0000","last_peered":"2024-09-10T18:37:59.936257+0000","last_clean":"2024-09-10T18:37:59.936257+0000","last_became_active":"2024-09-10T18:21:35.622211+0000","last_became_peered":"2024-09-10T18:21:35.622211+0000","last_unstale":"2024-09-10T18:37:59.936257+0000","last_undegraded":"2024-09-10T18:37:59.936257+0000","last_fullsized":"2024-09-10T18:37:59.936257+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'9","last_scrub_stamp":"2024-09-10T18:37:10.969454+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:37:10.969454+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:26.277589+0000","scrub_duration":0.0024896530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":359,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.921156+0000","last_change":"2024-09-10T18:37:43.480404+0000","last_active":"2024-09-10T18:37:57.921156+0000","last_peered":"2024-09-10T18:37:57.921156+0000","last_clean":"2024-09-10T18:37:57.921156+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:37:57.921156+0000","last_undegraded":"2024-09-10T18:37:57.921156+0000","last_fullsized":"2024-09-10T18:37:57.921156+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:43.480325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_clean_scrub_stamp":"2024-09-10T18:37:43.480325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:44.950473+0000","scrub_duration":0.011972396999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"344'9418","reported_seq":9782,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:38:00.274220+0000","last_change":"2024-09-10T18:37:40.743516+0000","last_active":"2024-09-10T18:38:00.274220+0000","last_peered":"2024-09-10T18:38:00.274220+0000","last_clean":"2024-09-10T18:38:00.274220+0000","last_became_active":"2024-09-10T18:36:07.903638+0000","last_became_peered":"2024-09-10T18:36:07.903638+0000","last_unstale":"2024-09-10T18:38:00.274220+0000","last_undegraded":"2024-09-10T18:38:00.274220+0000","last_fullsized":"2024-09-10T18:38:00.274220+0000","mapping_epoch":314,"log_start":"322'3600","ondisk_log_start":"322'3600","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'7977","last_scrub_stamp":"2024-09-10T18:37:40.743449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:37:40.743449+0000","objects_scrubbed":7887,"log_size":5818,"log_dups_size":0,"ondisk_log_size":5818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:51.036099+0000","scrub_duration":2.789328641,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505806848,"num_objects":7718,"num_object_clones":0,"num_object_copies":23154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9418,"num_write_kb":548352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"76'81","reported_seq":102,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.921075+0000","last_change":"2024-09-10T18:37:51.400051+0000","last_active":"2024-09-10T18:37:57.921075+0000","last_peered":"2024-09-10T18:37:57.921075+0000","last_clean":"2024-09-10T18:37:57.921075+0000","last_became_active":"2024-09-10T18:34:32.818251+0000","last_became_peered":"2024-09-10T18:34:32.818251+0000","last_unstale":"2024-09-10T18:37:57.921075+0000","last_undegraded":"2024-09-10T18:37:57.921075+0000","last_fullsized":"2024-09-10T18:37:57.921075+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:51.400000+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:51.400000+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:53.107345+0000","scrub_duration":0.0013808830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"302'19","reported_seq":391,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.931804+0000","last_change":"2024-09-10T18:37:08.823694+0000","last_active":"2024-09-10T18:37:57.931804+0000","last_peered":"2024-09-10T18:37:57.931804+0000","last_clean":"2024-09-10T18:37:57.931804+0000","last_became_active":"2024-09-10T18:21:33.614234+0000","last_became_peered":"2024-09-10T18:21:33.614234+0000","last_unstale":"2024-09-10T18:37:57.931804+0000","last_undegraded":"2024-09-10T18:37:57.931804+0000","last_fullsized":"2024-09-10T18:37:57.931804+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'19","last_scrub_stamp":"2024-09-10T18:37:08.823440+0000","last_deep_scrub":"302'19","last_deep_scrub_stamp":"2024-09-10T18:35:18.836131+0000","last_clean_scrub_stamp":"2024-09-10T18:37:08.823440+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:19.183000+0000","scrub_duration":0.012806522000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"265'2","reported_seq":390,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926881+0000","last_change":"2024-09-10T18:37:37.448192+0000","last_active":"2024-09-10T18:37:57.926881+0000","last_peered":"2024-09-10T18:37:57.926881+0000","last_clean":"2024-09-10T18:37:57.926881+0000","last_became_active":"2024-09-10T18:26:27.644997+0000","last_became_peered":"2024-09-10T18:26:27.644997+0000","last_unstale":"2024-09-10T18:37:57.926881+0000","last_undegraded":"2024-09-10T18:37:57.926881+0000","last_fullsized":"2024-09-10T18:37:57.926881+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'2","last_scrub_stamp":"2024-09-10T18:37:37.448103+0000","last_deep_scrub":"265'2","last_deep_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_clean_scrub_stamp":"2024-09-10T18:37:37.448103+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:01.900653+0000","scrub_duration":0.011219089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":384,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.937234+0000","last_change":"2024-09-10T18:36:56.798207+0000","last_active":"2024-09-10T18:37:57.937234+0000","last_peered":"2024-09-10T18:37:57.937234+0000","last_clean":"2024-09-10T18:37:57.937234+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:37:57.937234+0000","last_undegraded":"2024-09-10T18:37:57.937234+0000","last_fullsized":"2024-09-10T18:37:57.937234+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:56.798117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:56.798117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:14.975136+0000","scrub_duration":0.0021655960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":358,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.929934+0000","last_change":"2024-09-10T18:36:14.665247+0000","last_active":"2024-09-10T18:37:57.929934+0000","last_peered":"2024-09-10T18:37:57.929934+0000","last_clean":"2024-09-10T18:37:57.929934+0000","last_became_active":"2024-09-10T18:21:37.640941+0000","last_became_peered":"2024-09-10T18:21:37.640941+0000","last_unstale":"2024-09-10T18:37:57.929934+0000","last_undegraded":"2024-09-10T18:37:57.929934+0000","last_fullsized":"2024-09-10T18:37:57.929934+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:14.665183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:14.665183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014290399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"344'9366","reported_seq":9740,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:38:01.307542+0000","last_change":"2024-09-10T18:37:56.702387+0000","last_active":"2024-09-10T18:38:01.307542+0000","last_peered":"2024-09-10T18:38:01.307542+0000","last_clean":"2024-09-10T18:38:01.307542+0000","last_became_active":"2024-09-10T18:36:07.902336+0000","last_became_peered":"2024-09-10T18:36:07.902336+0000","last_unstale":"2024-09-10T18:38:01.307542+0000","last_undegraded":"2024-09-10T18:38:01.307542+0000","last_fullsized":"2024-09-10T18:38:01.307542+0000","mapping_epoch":314,"log_start":"326'4800","ondisk_log_start":"326'4800","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'8996","last_scrub_stamp":"2024-09-10T18:37:56.702297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:37:56.702297+0000","objects_scrubbed":8154,"log_size":4566,"log_dups_size":0,"ondisk_log_size":4566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:18.958724+0000","scrub_duration":4.0256546780000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":501743616,"num_objects":7656,"num_object_clones":0,"num_object_copies":22968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9366,"num_write_kb":544704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'81","reported_seq":348,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926000+0000","last_change":"2024-09-10T18:36:04.870940+0000","last_active":"2024-09-10T18:37:57.926000+0000","last_peered":"2024-09-10T18:37:57.926000+0000","last_clean":"2024-09-10T18:37:57.926000+0000","last_became_active":"2024-09-10T18:34:02.803861+0000","last_became_peered":"2024-09-10T18:34:02.803861+0000","last_unstale":"2024-09-10T18:37:57.926000+0000","last_undegraded":"2024-09-10T18:37:57.926000+0000","last_fullsized":"2024-09-10T18:37:57.926000+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:04.870861+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:34:15.938079+0000","last_clean_scrub_stamp":"2024-09-10T18:36:04.870861+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001253791,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"240'44","reported_seq":454,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926693+0000","last_change":"2024-09-10T18:37:35.673812+0000","last_active":"2024-09-10T18:37:57.926693+0000","last_peered":"2024-09-10T18:37:57.926693+0000","last_clean":"2024-09-10T18:37:57.926693+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:37:57.926693+0000","last_undegraded":"2024-09-10T18:37:57.926693+0000","last_fullsized":"2024-09-10T18:37:57.926693+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'44","last_scrub_stamp":"2024-09-10T18:37:35.673719+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:37:35.673719+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:56.474842+0000","scrub_duration":0.0023098609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":379,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.929155+0000","last_change":"2024-09-10T18:36:56.003032+0000","last_active":"2024-09-10T18:37:57.929155+0000","last_peered":"2024-09-10T18:37:57.929155+0000","last_clean":"2024-09-10T18:37:57.929155+0000","last_became_active":"2024-09-10T18:26:27.638408+0000","last_became_peered":"2024-09-10T18:26:27.638408+0000","last_unstale":"2024-09-10T18:37:57.929155+0000","last_undegraded":"2024-09-10T18:37:57.929155+0000","last_fullsized":"2024-09-10T18:37:57.929155+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:56.002946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:36:56.002946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:17.122534+0000","scrub_duration":0.0058026170000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":398,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.937578+0000","last_change":"2024-09-10T18:37:43.969249+0000","last_active":"2024-09-10T18:37:57.937578+0000","last_peered":"2024-09-10T18:37:57.937578+0000","last_clean":"2024-09-10T18:37:57.937578+0000","last_became_active":"2024-09-10T18:21:35.618921+0000","last_became_peered":"2024-09-10T18:21:35.618921+0000","last_unstale":"2024-09-10T18:37:57.937578+0000","last_undegraded":"2024-09-10T18:37:57.937578+0000","last_fullsized":"2024-09-10T18:37:57.937578+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:43.969035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:34:21.447291+0000","last_clean_scrub_stamp":"2024-09-10T18:37:43.969035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:09.326309+0000","scrub_duration":0.0041965149999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":369,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.928548+0000","last_change":"2024-09-10T18:37:04.058735+0000","last_active":"2024-09-10T18:37:57.928548+0000","last_peered":"2024-09-10T18:37:57.928548+0000","last_clean":"2024-09-10T18:37:57.928548+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:37:57.928548+0000","last_undegraded":"2024-09-10T18:37:57.928548+0000","last_fullsized":"2024-09-10T18:37:57.928548+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:04.058646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:04.058646+0000","last_clean_scrub_stamp":"2024-09-10T18:37:04.058646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:21.930918+0000","scrub_duration":0.0034509390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"344'9439","reported_seq":9475,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:38:01.017415+0000","last_change":"2024-09-10T18:36:07.902995+0000","last_active":"2024-09-10T18:38:01.017415+0000","last_peered":"2024-09-10T18:38:01.017415+0000","last_clean":"2024-09-10T18:38:01.017415+0000","last_became_active":"2024-09-10T18:36:07.902661+0000","last_became_peered":"2024-09-10T18:36:07.902661+0000","last_unstale":"2024-09-10T18:38:01.017415+0000","last_undegraded":"2024-09-10T18:38:01.017415+0000","last_fullsized":"2024-09-10T18:38:01.017415+0000","mapping_epoch":314,"log_start":"325'4500","ondisk_log_start":"325'4500","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:36:06.880082+0000","objects_scrubbed":0,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":501547008,"num_objects":7653,"num_object_clones":0,"num_object_copies":22959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9439,"num_write_kb":546944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"76'81","reported_seq":296,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.921669+0000","last_change":"2024-09-10T18:37:04.959623+0000","last_active":"2024-09-10T18:37:57.921669+0000","last_peered":"2024-09-10T18:37:57.921669+0000","last_clean":"2024-09-10T18:37:57.921669+0000","last_became_active":"2024-09-10T18:34:32.956761+0000","last_became_peered":"2024-09-10T18:34:32.956761+0000","last_unstale":"2024-09-10T18:37:57.921669+0000","last_undegraded":"2024-09-10T18:37:57.921669+0000","last_fullsized":"2024-09-10T18:37:57.921669+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:04.959510+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:04.959510+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:10.242143+0000","scrub_duration":0.0070392069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"301'23","reported_seq":410,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.920961+0000","last_change":"2024-09-10T18:37:42.463354+0000","last_active":"2024-09-10T18:37:57.920961+0000","last_peered":"2024-09-10T18:37:57.920961+0000","last_clean":"2024-09-10T18:37:57.920961+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:37:57.920961+0000","last_undegraded":"2024-09-10T18:37:57.920961+0000","last_fullsized":"2024-09-10T18:37:57.920961+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'23","last_scrub_stamp":"2024-09-10T18:37:42.463250+0000","last_deep_scrub":"301'23","last_deep_scrub_stamp":"2024-09-10T18:35:15.283373+0000","last_clean_scrub_stamp":"2024-09-10T18:37:42.463250+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:46.370234+0000","scrub_duration":0.0078345689999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":372,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.929168+0000","last_change":"2024-09-10T18:36:50.894694+0000","last_active":"2024-09-10T18:37:57.929168+0000","last_peered":"2024-09-10T18:37:57.929168+0000","last_clean":"2024-09-10T18:37:57.929168+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:37:57.929168+0000","last_undegraded":"2024-09-10T18:37:57.929168+0000","last_fullsized":"2024-09-10T18:37:57.929168+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:50.894615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:36:50.894615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:04.333933+0000","scrub_duration":0.0038639159999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":398,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.937154+0000","last_change":"2024-09-10T18:37:06.917685+0000","last_active":"2024-09-10T18:37:57.937154+0000","last_peered":"2024-09-10T18:37:57.937154+0000","last_clean":"2024-09-10T18:37:57.937154+0000","last_became_active":"2024-09-10T18:21:35.621174+0000","last_became_peered":"2024-09-10T18:21:35.621174+0000","last_unstale":"2024-09-10T18:37:57.937154+0000","last_undegraded":"2024-09-10T18:37:57.937154+0000","last_fullsized":"2024-09-10T18:37:57.937154+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:06.917604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:06.917604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:12.452402+0000","scrub_duration":0.013746744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":349,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.918306+0000","last_change":"2024-09-10T18:36:05.766562+0000","last_active":"2024-09-10T18:37:57.918306+0000","last_peered":"2024-09-10T18:37:57.918306+0000","last_clean":"2024-09-10T18:37:57.918306+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:37:57.918306+0000","last_undegraded":"2024-09-10T18:37:57.918306+0000","last_fullsized":"2024-09-10T18:37:57.918306+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:05.766507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.803236+0000","last_clean_scrub_stamp":"2024-09-10T18:36:05.766507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023023990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"344'9395","reported_seq":9773,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:38:00.151043+0000","last_change":"2024-09-10T18:37:45.627395+0000","last_active":"2024-09-10T18:38:00.151043+0000","last_peered":"2024-09-10T18:38:00.151043+0000","last_clean":"2024-09-10T18:38:00.151043+0000","last_became_active":"2024-09-10T18:36:07.903791+0000","last_became_peered":"2024-09-10T18:36:07.903791+0000","last_unstale":"2024-09-10T18:38:00.151043+0000","last_undegraded":"2024-09-10T18:38:00.151043+0000","last_fullsized":"2024-09-10T18:38:00.151043+0000","mapping_epoch":314,"log_start":"326'4839","ondisk_log_start":"326'4839","created":314,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'8353","last_scrub_stamp":"2024-09-10T18:37:45.627312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:36:06.880082+0000","last_clean_scrub_stamp":"2024-09-10T18:37:45.627312+0000","objects_scrubbed":8260,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:50.954313+0000","scrub_duration":3.0873044790000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508493824,"num_objects":7759,"num_object_clones":0,"num_object_copies":23277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9395,"num_write_kb":548928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"76'81","reported_seq":92,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.928879+0000","last_change":"2024-09-10T18:36:11.950457+0000","last_active":"2024-09-10T18:37:57.928879+0000","last_peered":"2024-09-10T18:37:57.928879+0000","last_clean":"2024-09-10T18:37:57.928879+0000","last_became_active":"2024-09-10T18:34:32.822196+0000","last_became_peered":"2024-09-10T18:34:32.822196+0000","last_unstale":"2024-09-10T18:37:57.928879+0000","last_undegraded":"2024-09-10T18:37:57.928879+0000","last_fullsized":"2024-09-10T18:37:57.928879+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:11.950311+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:11.950311+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020040769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":411,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.925112+0000","last_change":"2024-09-10T18:36:54.095746+0000","last_active":"2024-09-10T18:37:57.925112+0000","last_peered":"2024-09-10T18:37:57.925112+0000","last_clean":"2024-09-10T18:37:57.925112+0000","last_became_active":"2024-09-10T18:26:27.645932+0000","last_became_peered":"2024-09-10T18:26:27.645932+0000","last_unstale":"2024-09-10T18:37:57.925112+0000","last_undegraded":"2024-09-10T18:37:57.925112+0000","last_fullsized":"2024-09-10T18:37:57.925112+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:36:54.095659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:54.095659+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:11.075189+0000","scrub_duration":0.012246685,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":372,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.929228+0000","last_change":"2024-09-10T18:35:58.895879+0000","last_active":"2024-09-10T18:37:57.929228+0000","last_peered":"2024-09-10T18:37:57.929228+0000","last_clean":"2024-09-10T18:37:57.929228+0000","last_became_active":"2024-09-10T18:21:31.530336+0000","last_became_peered":"2024-09-10T18:21:31.530336+0000","last_unstale":"2024-09-10T18:37:57.929228+0000","last_undegraded":"2024-09-10T18:37:57.929228+0000","last_fullsized":"2024-09-10T18:37:57.929228+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:35:58.895792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:35:58.895792+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021727780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":575,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:37:59.957452+0000","last_change":"2024-09-10T18:36:02.894810+0000","last_active":"2024-09-10T18:37:59.957452+0000","last_peered":"2024-09-10T18:37:59.957452+0000","last_clean":"2024-09-10T18:37:59.957452+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:37:59.957452+0000","last_undegraded":"2024-09-10T18:37:59.957452+0000","last_fullsized":"2024-09-10T18:37:59.957452+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:36:02.894094+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:36:02.894094+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028441780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":354,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926360+0000","last_change":"2024-09-10T18:35:53.486754+0000","last_active":"2024-09-10T18:37:57.926360+0000","last_peered":"2024-09-10T18:37:57.926360+0000","last_clean":"2024-09-10T18:37:57.926360+0000","last_became_active":"2024-09-10T18:26:27.647511+0000","last_became_peered":"2024-09-10T18:26:27.647511+0000","last_unstale":"2024-09-10T18:37:57.926360+0000","last_undegraded":"2024-09-10T18:37:57.926360+0000","last_fullsized":"2024-09-10T18:37:57.926360+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:53.486678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:53.486678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035730969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'81","reported_seq":92,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926685+0000","last_change":"2024-09-10T18:37:14.252272+0000","last_active":"2024-09-10T18:37:57.926685+0000","last_peered":"2024-09-10T18:37:57.926685+0000","last_clean":"2024-09-10T18:37:57.926685+0000","last_became_active":"2024-09-10T18:34:02.810126+0000","last_became_peered":"2024-09-10T18:34:02.810126+0000","last_unstale":"2024-09-10T18:37:57.926685+0000","last_undegraded":"2024-09-10T18:37:57.926685+0000","last_fullsized":"2024-09-10T18:37:57.926685+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:14.252067+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:14.252067+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:18.451121+0000","scrub_duration":0.0039562720000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":318,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926305+0000","last_change":"2024-09-10T18:35:59.886309+0000","last_active":"2024-09-10T18:37:57.926305+0000","last_peered":"2024-09-10T18:37:57.926305+0000","last_clean":"2024-09-10T18:37:57.926305+0000","last_became_active":"2024-09-10T18:26:27.640922+0000","last_became_peered":"2024-09-10T18:26:27.640922+0000","last_unstale":"2024-09-10T18:37:57.926305+0000","last_undegraded":"2024-09-10T18:37:57.926305+0000","last_fullsized":"2024-09-10T18:37:57.926305+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:59.886242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:35:59.886242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001788724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"240'12","reported_seq":371,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.924676+0000","last_change":"2024-09-10T18:36:00.854061+0000","last_active":"2024-09-10T18:37:57.924676+0000","last_peered":"2024-09-10T18:37:57.924676+0000","last_clean":"2024-09-10T18:37:57.924676+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:37:57.924676+0000","last_undegraded":"2024-09-10T18:37:57.924676+0000","last_fullsized":"2024-09-10T18:37:57.924676+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-10T18:36:00.854004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:00.854004+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019737209999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":368,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926687+0000","last_change":"2024-09-10T18:37:55.492560+0000","last_active":"2024-09-10T18:37:57.926687+0000","last_peered":"2024-09-10T18:37:57.926687+0000","last_clean":"2024-09-10T18:37:57.926687+0000","last_became_active":"2024-09-10T18:21:35.621963+0000","last_became_peered":"2024-09-10T18:21:35.621963+0000","last_unstale":"2024-09-10T18:37:57.926687+0000","last_undegraded":"2024-09-10T18:37:57.926687+0000","last_fullsized":"2024-09-10T18:37:57.926687+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:55.492428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:55.492428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:22.327464+0000","scrub_duration":0.0032492630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":363,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.928884+0000","last_change":"2024-09-10T18:36:57.022869+0000","last_active":"2024-09-10T18:37:57.928884+0000","last_peered":"2024-09-10T18:37:57.928884+0000","last_clean":"2024-09-10T18:37:57.928884+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:37:57.928884+0000","last_undegraded":"2024-09-10T18:37:57.928884+0000","last_fullsized":"2024-09-10T18:37:57.928884+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:57.022800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:57.022800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:23.678797+0000","scrub_duration":0.0040564720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'81","reported_seq":299,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.931154+0000","last_change":"2024-09-10T18:37:05.807566+0000","last_active":"2024-09-10T18:37:57.931154+0000","last_peered":"2024-09-10T18:37:57.931154+0000","last_clean":"2024-09-10T18:37:57.931154+0000","last_became_active":"2024-09-10T18:34:02.815880+0000","last_became_peered":"2024-09-10T18:34:02.815880+0000","last_unstale":"2024-09-10T18:37:57.931154+0000","last_undegraded":"2024-09-10T18:37:57.931154+0000","last_fullsized":"2024-09-10T18:37:57.931154+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:05.807459+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:05.807459+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:22.170236+0000","scrub_duration":0.0043697320000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"240'19","reported_seq":394,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.930075+0000","last_change":"2024-09-10T18:37:05.753183+0000","last_active":"2024-09-10T18:37:57.930075+0000","last_peered":"2024-09-10T18:37:57.930075+0000","last_clean":"2024-09-10T18:37:57.930075+0000","last_became_active":"2024-09-10T18:21:34.047416+0000","last_became_peered":"2024-09-10T18:21:34.047416+0000","last_unstale":"2024-09-10T18:37:57.930075+0000","last_undegraded":"2024-09-10T18:37:57.930075+0000","last_fullsized":"2024-09-10T18:37:57.930075+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:37:05.752891+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-09-10T18:35:19.802016+0000","last_clean_scrub_stamp":"2024-09-10T18:37:05.752891+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:06.037770+0000","scrub_duration":0.0067102660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":369,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.927951+0000","last_change":"2024-09-10T18:36:04.875938+0000","last_active":"2024-09-10T18:37:57.927951+0000","last_peered":"2024-09-10T18:37:57.927951+0000","last_clean":"2024-09-10T18:37:57.927951+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:37:57.927951+0000","last_undegraded":"2024-09-10T18:37:57.927951+0000","last_fullsized":"2024-09-10T18:37:57.927951+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:04.875692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:36:04.875692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018984749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":351,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.924487+0000","last_change":"2024-09-10T18:35:12.764509+0000","last_active":"2024-09-10T18:37:57.924487+0000","last_peered":"2024-09-10T18:37:57.924487+0000","last_clean":"2024-09-10T18:37:57.924487+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:37:57.924487+0000","last_undegraded":"2024-09-10T18:37:57.924487+0000","last_fullsized":"2024-09-10T18:37:57.924487+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:12.764433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:35:12.764433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069242059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":386,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.937564+0000","last_change":"2024-09-10T18:37:35.987443+0000","last_active":"2024-09-10T18:37:57.937564+0000","last_peered":"2024-09-10T18:37:57.937564+0000","last_clean":"2024-09-10T18:37:57.937564+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:37:57.937564+0000","last_undegraded":"2024-09-10T18:37:57.937564+0000","last_fullsized":"2024-09-10T18:37:57.937564+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:35.987330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:35.987330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:41.211548+0000","scrub_duration":0.0096016480000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'81","reported_seq":92,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.924829+0000","last_change":"2024-09-10T18:36:58.147850+0000","last_active":"2024-09-10T18:37:57.924829+0000","last_peered":"2024-09-10T18:37:57.924829+0000","last_clean":"2024-09-10T18:37:57.924829+0000","last_became_active":"2024-09-10T18:34:02.797304+0000","last_became_peered":"2024-09-10T18:34:02.797304+0000","last_unstale":"2024-09-10T18:37:57.924829+0000","last_undegraded":"2024-09-10T18:37:57.924829+0000","last_fullsized":"2024-09-10T18:37:57.924829+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:58.147719+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:58.147719+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:13.451302+0000","scrub_duration":0.009846858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"240'19","reported_seq":390,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926917+0000","last_change":"2024-09-10T18:36:14.732278+0000","last_active":"2024-09-10T18:37:57.926917+0000","last_peered":"2024-09-10T18:37:57.926917+0000","last_clean":"2024-09-10T18:37:57.926917+0000","last_became_active":"2024-09-10T18:21:34.046687+0000","last_became_peered":"2024-09-10T18:21:34.046687+0000","last_unstale":"2024-09-10T18:37:57.926917+0000","last_undegraded":"2024-09-10T18:37:57.926917+0000","last_fullsized":"2024-09-10T18:37:57.926917+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:36:14.732217+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:36:14.732217+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020637149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":374,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.933085+0000","last_change":"2024-09-10T18:37:04.713668+0000","last_active":"2024-09-10T18:37:57.933085+0000","last_peered":"2024-09-10T18:37:57.933085+0000","last_clean":"2024-09-10T18:37:57.933085+0000","last_became_active":"2024-09-10T18:21:31.530128+0000","last_became_peered":"2024-09-10T18:21:31.530128+0000","last_unstale":"2024-09-10T18:37:57.933085+0000","last_undegraded":"2024-09-10T18:37:57.933085+0000","last_fullsized":"2024-09-10T18:37:57.933085+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:04.713474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_clean_scrub_stamp":"2024-09-10T18:37:04.713474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:09.956287+0000","scrub_duration":0.0039225839999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"342'9","reported_seq":568,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:37:59.942553+0000","last_change":"2024-09-10T18:37:56.641912+0000","last_active":"2024-09-10T18:37:59.942553+0000","last_peered":"2024-09-10T18:37:59.942553+0000","last_clean":"2024-09-10T18:37:59.942553+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:37:59.942553+0000","last_undegraded":"2024-09-10T18:37:59.942553+0000","last_fullsized":"2024-09-10T18:37:59.942553+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'9","last_scrub_stamp":"2024-09-10T18:37:56.641708+0000","last_deep_scrub":"342'9","last_deep_scrub_stamp":"2024-09-10T18:37:56.641708+0000","last_clean_scrub_stamp":"2024-09-10T18:37:56.641708+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:05.418046+0000","scrub_duration":0.023870353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":94,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.924807+0000","last_change":"2024-09-10T18:37:02.933234+0000","last_active":"2024-09-10T18:37:57.924807+0000","last_peered":"2024-09-10T18:37:57.924807+0000","last_clean":"2024-09-10T18:37:57.924807+0000","last_became_active":"2024-09-10T18:33:29.141936+0000","last_became_peered":"2024-09-10T18:33:29.141936+0000","last_unstale":"2024-09-10T18:37:57.924807+0000","last_undegraded":"2024-09-10T18:37:57.924807+0000","last_fullsized":"2024-09-10T18:37:57.924807+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:02.933068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:02.933068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:03.633941+0000","scrub_duration":0.0042260099999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"76'81","reported_seq":164,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.929717+0000","last_change":"2024-09-10T18:37:43.810850+0000","last_active":"2024-09-10T18:37:57.929717+0000","last_peered":"2024-09-10T18:37:57.929717+0000","last_clean":"2024-09-10T18:37:57.929717+0000","last_became_active":"2024-09-10T18:34:02.810381+0000","last_became_peered":"2024-09-10T18:34:02.810381+0000","last_unstale":"2024-09-10T18:37:57.929717+0000","last_undegraded":"2024-09-10T18:37:57.929717+0000","last_fullsized":"2024-09-10T18:37:57.929717+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:43.810696+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:36:08.710566+0000","last_clean_scrub_stamp":"2024-09-10T18:37:43.810696+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:03.468981+0000","scrub_duration":0.0083017330000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":393,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.922040+0000","last_change":"2024-09-10T18:37:09.977307+0000","last_active":"2024-09-10T18:37:57.922040+0000","last_peered":"2024-09-10T18:37:57.922040+0000","last_clean":"2024-09-10T18:37:57.922040+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:37:57.922040+0000","last_undegraded":"2024-09-10T18:37:57.922040+0000","last_fullsized":"2024-09-10T18:37:57.922040+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-10T18:37:09.977172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:09.977172+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:30.689481+0000","scrub_duration":0.01120503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":372,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.928051+0000","last_change":"2024-09-10T18:37:48.936256+0000","last_active":"2024-09-10T18:37:57.928051+0000","last_peered":"2024-09-10T18:37:57.928051+0000","last_clean":"2024-09-10T18:37:57.928051+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:37:57.928051+0000","last_undegraded":"2024-09-10T18:37:57.928051+0000","last_fullsized":"2024-09-10T18:37:57.928051+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:48.936179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:17.826448+0000","last_clean_scrub_stamp":"2024-09-10T18:37:48.936179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:17.605483+0000","scrub_duration":0.011322904999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":575,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-10T18:37:59.940041+0000","last_change":"2024-09-10T18:37:03.193346+0000","last_active":"2024-09-10T18:37:59.940041+0000","last_peered":"2024-09-10T18:37:59.940041+0000","last_clean":"2024-09-10T18:37:59.940041+0000","last_became_active":"2024-09-10T18:21:35.618666+0000","last_became_peered":"2024-09-10T18:21:35.618666+0000","last_unstale":"2024-09-10T18:37:59.940041+0000","last_undegraded":"2024-09-10T18:37:59.940041+0000","last_fullsized":"2024-09-10T18:37:59.940041+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-09-10T18:37:03.193256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:03.193256+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:24.557501+0000","scrub_duration":0.0034883319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":368,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.929636+0000","last_change":"2024-09-10T18:37:48.826749+0000","last_active":"2024-09-10T18:37:57.929636+0000","last_peered":"2024-09-10T18:37:57.929636+0000","last_clean":"2024-09-10T18:37:57.929636+0000","last_became_active":"2024-09-10T18:21:37.643770+0000","last_became_peered":"2024-09-10T18:21:37.643770+0000","last_unstale":"2024-09-10T18:37:57.929636+0000","last_undegraded":"2024-09-10T18:37:57.929636+0000","last_fullsized":"2024-09-10T18:37:57.929636+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:48.826669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:48.826669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:02.896648+0000","scrub_duration":0.0073207899999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'81","reported_seq":84,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.932794+0000","last_change":"2024-09-10T18:37:48.962531+0000","last_active":"2024-09-10T18:37:57.932794+0000","last_peered":"2024-09-10T18:37:57.932794+0000","last_clean":"2024-09-10T18:37:57.932794+0000","last_became_active":"2024-09-10T18:34:34.881186+0000","last_became_peered":"2024-09-10T18:34:34.881186+0000","last_unstale":"2024-09-10T18:37:57.932794+0000","last_undegraded":"2024-09-10T18:37:57.932794+0000","last_fullsized":"2024-09-10T18:37:57.932794+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:48.962443+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:48.962443+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:12.129961+0000","scrub_duration":0.0052595840000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":408,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.921009+0000","last_change":"2024-09-10T18:37:13.403259+0000","last_active":"2024-09-10T18:37:57.921009+0000","last_peered":"2024-09-10T18:37:57.921009+0000","last_clean":"2024-09-10T18:37:57.921009+0000","last_became_active":"2024-09-10T18:26:27.649202+0000","last_became_peered":"2024-09-10T18:26:27.649202+0000","last_unstale":"2024-09-10T18:37:57.921009+0000","last_undegraded":"2024-09-10T18:37:57.921009+0000","last_fullsized":"2024-09-10T18:37:57.921009+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:37:13.403175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:13.403175+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:34.162155+0000","scrub_duration":0.0070645999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":372,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.922478+0000","last_change":"2024-09-10T18:37:02.270246+0000","last_active":"2024-09-10T18:37:57.922478+0000","last_peered":"2024-09-10T18:37:57.922478+0000","last_clean":"2024-09-10T18:37:57.922478+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:37:57.922478+0000","last_undegraded":"2024-09-10T18:37:57.922478+0000","last_fullsized":"2024-09-10T18:37:57.922478+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:02.270169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:37:02.270169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:15.675634+0000","scrub_duration":0.001637306,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":382,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.931653+0000","last_change":"2024-09-10T18:37:56.027633+0000","last_active":"2024-09-10T18:37:57.931653+0000","last_peered":"2024-09-10T18:37:57.931653+0000","last_clean":"2024-09-10T18:37:57.931653+0000","last_became_active":"2024-09-10T18:21:35.620124+0000","last_became_peered":"2024-09-10T18:21:35.620124+0000","last_unstale":"2024-09-10T18:37:57.931653+0000","last_undegraded":"2024-09-10T18:37:57.931653+0000","last_fullsized":"2024-09-10T18:37:57.931653+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:56.027540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_clean_scrub_stamp":"2024-09-10T18:37:56.027540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:20.395091+0000","scrub_duration":0.0065405710000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":349,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.919152+0000","last_change":"2024-09-10T18:37:41.752156+0000","last_active":"2024-09-10T18:37:57.919152+0000","last_peered":"2024-09-10T18:37:57.919152+0000","last_clean":"2024-09-10T18:37:57.919152+0000","last_became_active":"2024-09-10T18:21:37.641570+0000","last_became_peered":"2024-09-10T18:21:37.641570+0000","last_unstale":"2024-09-10T18:37:57.919152+0000","last_undegraded":"2024-09-10T18:37:57.919152+0000","last_fullsized":"2024-09-10T18:37:57.919152+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:41.752076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:37:41.752076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:08.814473+0000","scrub_duration":0.010075051999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'81","reported_seq":83,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926657+0000","last_change":"2024-09-10T18:37:50.487282+0000","last_active":"2024-09-10T18:37:57.926657+0000","last_peered":"2024-09-10T18:37:57.926657+0000","last_clean":"2024-09-10T18:37:57.926657+0000","last_became_active":"2024-09-10T18:34:34.858291+0000","last_became_peered":"2024-09-10T18:34:34.858291+0000","last_unstale":"2024-09-10T18:37:57.926657+0000","last_undegraded":"2024-09-10T18:37:57.926657+0000","last_fullsized":"2024-09-10T18:37:57.926657+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:50.487145+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:50.487145+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:06.010311+0000","scrub_duration":0.0028683850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"240'15","reported_seq":403,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.931863+0000","last_change":"2024-09-10T18:36:16.592923+0000","last_active":"2024-09-10T18:37:57.931863+0000","last_peered":"2024-09-10T18:37:57.931863+0000","last_clean":"2024-09-10T18:37:57.931863+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:37:57.931863+0000","last_undegraded":"2024-09-10T18:37:57.931863+0000","last_fullsized":"2024-09-10T18:37:57.931863+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:36:16.592847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:16.592847+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020114550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":365,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.929040+0000","last_change":"2024-09-10T18:36:07.931194+0000","last_active":"2024-09-10T18:37:57.929040+0000","last_peered":"2024-09-10T18:37:57.929040+0000","last_clean":"2024-09-10T18:37:57.929040+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:37:57.929040+0000","last_undegraded":"2024-09-10T18:37:57.929040+0000","last_fullsized":"2024-09-10T18:37:57.929040+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:07.931108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:36:07.931108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012878189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":344,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.919125+0000","last_change":"2024-09-10T18:34:56.701737+0000","last_active":"2024-09-10T18:37:57.919125+0000","last_peered":"2024-09-10T18:37:57.919125+0000","last_clean":"2024-09-10T18:37:57.919125+0000","last_became_active":"2024-09-10T18:21:35.622773+0000","last_became_peered":"2024-09-10T18:21:35.622773+0000","last_unstale":"2024-09-10T18:37:57.919125+0000","last_undegraded":"2024-09-10T18:37:57.919125+0000","last_fullsized":"2024-09-10T18:37:57.919125+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:34:56.701661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:34:56.701661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064135620000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":368,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.930391+0000","last_change":"2024-09-10T18:37:11.863730+0000","last_active":"2024-09-10T18:37:57.930391+0000","last_peered":"2024-09-10T18:37:57.930391+0000","last_clean":"2024-09-10T18:37:57.930391+0000","last_became_active":"2024-09-10T18:21:37.641494+0000","last_became_peered":"2024-09-10T18:21:37.641494+0000","last_unstale":"2024-09-10T18:37:57.930391+0000","last_undegraded":"2024-09-10T18:37:57.930391+0000","last_fullsized":"2024-09-10T18:37:57.930391+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:11.863644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:11.863644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:16.121702+0000","scrub_duration":0.0037629410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'81","reported_seq":102,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.937434+0000","last_change":"2024-09-10T18:37:52.057056+0000","last_active":"2024-09-10T18:37:57.937434+0000","last_peered":"2024-09-10T18:37:57.937434+0000","last_clean":"2024-09-10T18:37:57.937434+0000","last_became_active":"2024-09-10T18:34:02.802580+0000","last_became_peered":"2024-09-10T18:34:02.802580+0000","last_unstale":"2024-09-10T18:37:57.937434+0000","last_undegraded":"2024-09-10T18:37:57.937434+0000","last_fullsized":"2024-09-10T18:37:57.937434+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:52.056953+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:52.056953+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:05.089494+0000","scrub_duration":0.0049009659999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":401,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.919331+0000","last_change":"2024-09-10T18:37:55.624202+0000","last_active":"2024-09-10T18:37:57.919331+0000","last_peered":"2024-09-10T18:37:57.919331+0000","last_clean":"2024-09-10T18:37:57.919331+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:37:57.919331+0000","last_undegraded":"2024-09-10T18:37:57.919331+0000","last_fullsized":"2024-09-10T18:37:57.919331+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:37:55.624117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:55.624117+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:16.417554+0000","scrub_duration":0.0051350730000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":373,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926762+0000","last_change":"2024-09-10T18:36:53.050798+0000","last_active":"2024-09-10T18:37:57.926762+0000","last_peered":"2024-09-10T18:37:57.926762+0000","last_clean":"2024-09-10T18:37:57.926762+0000","last_became_active":"2024-09-10T18:21:31.529639+0000","last_became_peered":"2024-09-10T18:21:31.529639+0000","last_unstale":"2024-09-10T18:37:57.926762+0000","last_undegraded":"2024-09-10T18:37:57.926762+0000","last_fullsized":"2024-09-10T18:37:57.926762+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:53.050726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:36:53.050726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:02.660776+0000","scrub_duration":0.0099588239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":368,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.927517+0000","last_change":"2024-09-10T18:37:41.891601+0000","last_active":"2024-09-10T18:37:57.927517+0000","last_peered":"2024-09-10T18:37:57.927517+0000","last_clean":"2024-09-10T18:37:57.927517+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:37:57.927517+0000","last_undegraded":"2024-09-10T18:37:57.927517+0000","last_fullsized":"2024-09-10T18:37:57.927517+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:41.891433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:41.891433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:54.252040+0000","scrub_duration":0.002300341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":372,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.919314+0000","last_change":"2024-09-10T18:36:52.896429+0000","last_active":"2024-09-10T18:37:57.919314+0000","last_peered":"2024-09-10T18:37:57.919314+0000","last_clean":"2024-09-10T18:37:57.919314+0000","last_became_active":"2024-09-10T18:26:27.636996+0000","last_became_peered":"2024-09-10T18:26:27.636996+0000","last_unstale":"2024-09-10T18:37:57.919314+0000","last_undegraded":"2024-09-10T18:37:57.919314+0000","last_fullsized":"2024-09-10T18:37:57.919314+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:52.896322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:36:52.896322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029578769999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'81","reported_seq":73,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.928771+0000","last_change":"2024-09-10T18:36:14.999809+0000","last_active":"2024-09-10T18:37:57.928771+0000","last_peered":"2024-09-10T18:37:57.928771+0000","last_clean":"2024-09-10T18:37:57.928771+0000","last_became_active":"2024-09-10T18:34:34.868371+0000","last_became_peered":"2024-09-10T18:34:34.868371+0000","last_unstale":"2024-09-10T18:37:57.928771+0000","last_undegraded":"2024-09-10T18:37:57.928771+0000","last_fullsized":"2024-09-10T18:37:57.928771+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:14.999747+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:14.999747+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001461415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":372,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.918402+0000","last_change":"2024-09-10T18:36:01.819314+0000","last_active":"2024-09-10T18:37:57.918402+0000","last_peered":"2024-09-10T18:37:57.918402+0000","last_clean":"2024-09-10T18:37:57.918402+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:37:57.918402+0000","last_undegraded":"2024-09-10T18:37:57.918402+0000","last_fullsized":"2024-09-10T18:37:57.918402+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-09-10T18:36:01.819258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:01.819258+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001608246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":367,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.929898+0000","last_change":"2024-09-10T18:36:49.574521+0000","last_active":"2024-09-10T18:37:57.929898+0000","last_peered":"2024-09-10T18:37:57.929898+0000","last_clean":"2024-09-10T18:37:57.929898+0000","last_became_active":"2024-09-10T18:21:31.529451+0000","last_became_peered":"2024-09-10T18:21:31.529451+0000","last_unstale":"2024-09-10T18:37:57.929898+0000","last_undegraded":"2024-09-10T18:37:57.929898+0000","last_fullsized":"2024-09-10T18:37:57.929898+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:49.571125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:36:49.571125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066675700000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":322,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.925476+0000","last_change":"2024-09-10T18:36:01.890154+0000","last_active":"2024-09-10T18:37:57.925476+0000","last_peered":"2024-09-10T18:37:57.925476+0000","last_clean":"2024-09-10T18:37:57.925476+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:37:57.925476+0000","last_undegraded":"2024-09-10T18:37:57.925476+0000","last_fullsized":"2024-09-10T18:37:57.925476+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:01.890087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:01.890087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001570519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":356,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.928209+0000","last_change":"2024-09-10T18:35:18.879787+0000","last_active":"2024-09-10T18:37:57.928209+0000","last_peered":"2024-09-10T18:37:57.928209+0000","last_clean":"2024-09-10T18:37:57.928209+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:37:57.928209+0000","last_undegraded":"2024-09-10T18:37:57.928209+0000","last_fullsized":"2024-09-10T18:37:57.928209+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:35:18.879717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:35:18.879717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059146550000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'81","reported_seq":83,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.922555+0000","last_change":"2024-09-10T18:37:55.419411+0000","last_active":"2024-09-10T18:37:57.922555+0000","last_peered":"2024-09-10T18:37:57.922555+0000","last_clean":"2024-09-10T18:37:57.922555+0000","last_became_active":"2024-09-10T18:34:34.867380+0000","last_became_peered":"2024-09-10T18:34:34.867380+0000","last_unstale":"2024-09-10T18:37:57.922555+0000","last_undegraded":"2024-09-10T18:37:57.922555+0000","last_fullsized":"2024-09-10T18:37:57.922555+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:55.419180+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:55.419180+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:00.001250+0000","scrub_duration":0.0067995520000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":377,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.932254+0000","last_change":"2024-09-10T18:37:55.062853+0000","last_active":"2024-09-10T18:37:57.932254+0000","last_peered":"2024-09-10T18:37:57.932254+0000","last_clean":"2024-09-10T18:37:57.932254+0000","last_became_active":"2024-09-10T18:21:37.640943+0000","last_became_peered":"2024-09-10T18:21:37.640943+0000","last_unstale":"2024-09-10T18:37:57.932254+0000","last_undegraded":"2024-09-10T18:37:57.932254+0000","last_fullsized":"2024-09-10T18:37:57.932254+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:55.062765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_clean_scrub_stamp":"2024-09-10T18:37:55.062765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:13.220829+0000","scrub_duration":0.0046630359999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":355,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.918621+0000","last_change":"2024-09-10T18:37:39.736362+0000","last_active":"2024-09-10T18:37:57.918621+0000","last_peered":"2024-09-10T18:37:57.918621+0000","last_clean":"2024-09-10T18:37:57.918621+0000","last_became_active":"2024-09-10T18:21:31.534970+0000","last_became_peered":"2024-09-10T18:21:31.534970+0000","last_unstale":"2024-09-10T18:37:57.918621+0000","last_undegraded":"2024-09-10T18:37:57.918621+0000","last_fullsized":"2024-09-10T18:37:57.918621+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:39.736272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:37:39.736272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:41.193887+0000","scrub_duration":0.001864079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":383,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.926041+0000","last_change":"2024-09-10T18:36:47.856846+0000","last_active":"2024-09-10T18:37:57.926041+0000","last_peered":"2024-09-10T18:37:57.926041+0000","last_clean":"2024-09-10T18:37:57.926041+0000","last_became_active":"2024-09-10T18:26:27.648112+0000","last_became_peered":"2024-09-10T18:26:27.648112+0000","last_unstale":"2024-09-10T18:37:57.926041+0000","last_undegraded":"2024-09-10T18:37:57.926041+0000","last_fullsized":"2024-09-10T18:37:57.926041+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:36:47.856694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:36:47.856694+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:38:08.148989+0000","scrub_duration":0.0066546349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":370,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-10T18:37:57.929789+0000","last_change":"2024-09-10T18:37:42.795852+0000","last_active":"2024-09-10T18:37:57.929789+0000","last_peered":"2024-09-10T18:37:57.929789+0000","last_clean":"2024-09-10T18:37:57.929789+0000","last_became_active":"2024-09-10T18:21:35.620452+0000","last_became_peered":"2024-09-10T18:21:35.620452+0000","last_unstale":"2024-09-10T18:37:57.929789+0000","last_undegraded":"2024-09-10T18:37:57.929789+0000","last_fullsized":"2024-09-10T18:37:57.929789+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:42.795003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:37:42.795003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:05.307442+0000","scrub_duration":0.040172088000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":7979073560,"num_objects":121752,"num_object_clones":0,"num_object_copies":365256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121752,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152092,"num_write_kb":8762945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23763103744,"data_stored":23763091528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76702,"ondisk_log_size":76702,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":758,"ondisk_log_size":758,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":985,"num_read_kb":980,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2511,"ondisk_log_size":2511,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234186,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2468992,"kb_used_data":1879096,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":91276160,"statfs":{"total":95995035648,"available":93466787840,"internally_reserved":0,"allocated":1924194304,"data_stored":1923807141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:37:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.103,"15min":1.9239999999999999},"min":{"1min":0.254,"5min":0.215,"15min":0.20799999999999999},"max":{"1min":3.2469999999999999,"5min":6.5209999999999999,"15min":85.012},"last":3.0129999999999999},{"interface":"front","average":{"1min":1.2849999999999999,"5min":0.96699999999999997,"15min":1.847},"min":{"1min":0.25800000000000001,"5min":0.223,"15min":0.222},"max":{"1min":5.4909999999999997,"5min":5.4909999999999997,"15min":84.665999999999997},"last":2.702}]},{"osd":1,"last update":"Tue Sep 10 18:37:54 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":1.4139999999999999,"15min":1.492},"min":{"1min":0.29899999999999999,"5min":0.23899999999999999,"15min":0.214},"max":{"1min":3.4609999999999999,"5min":39.728000000000002,"15min":49.317999999999998},"last":3.7690000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.343,"15min":1.4079999999999999},"min":{"1min":0.28000000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.9510000000000001,"5min":39.738999999999997,"15min":49.353000000000002},"last":3.1760000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:37:54 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":2.6600000000000001,"15min":1.4650000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.249},"max":{"1min":3.8180000000000001,"5min":136.648,"15min":136.648},"last":0.33100000000000002},{"interface":"front","average":{"1min":1.022,"5min":2.6179999999999999,"15min":1.5209999999999999},"min":{"1min":0.32800000000000001,"5min":0.29199999999999998,"15min":0.24399999999999999},"max":{"1min":3.9700000000000002,"5min":136.506,"15min":136.506},"last":2.2829999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:37:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.6679999999999999,"15min":1.696},"min":{"1min":0.311,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":3.8700000000000001,"5min":38.225000000000001,"15min":86.305999999999997},"last":2.0550000000000002},{"interface":"front","average":{"1min":1.147,"5min":1.585,"15min":1.712},"min":{"1min":0.377,"5min":0.29199999999999998,"15min":0.26700000000000002},"max":{"1min":3.4550000000000001,"5min":38.104999999999997,"15min":88.364000000000004},"last":3.806}]},{"osd":4,"last update":"Tue Sep 10 18:37:54 2024","interfaces":[{"interface":"back","average":{"1min":3.2370000000000001,"5min":1.175,"15min":0.71999999999999997},"min":{"1min":0.248,"5min":0.24199999999999999,"15min":0.20799999999999999},"max":{"1min":23.346,"5min":23.346,"15min":23.346},"last":0.25700000000000001},{"interface":"front","average":{"1min":3.2280000000000002,"5min":1.264,"15min":0.77000000000000002},"min":{"1min":0.27200000000000002,"5min":0.26200000000000001,"15min":0.223},"max":{"1min":23.323,"5min":23.323,"15min":23.323},"last":0.28699999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:37:54 2024","interfaces":[{"interface":"back","average":{"1min":1.903,"5min":1.1339999999999999,"15min":0.84399999999999997},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.23499999999999999},"max":{"1min":16.756,"5min":16.756,"15min":16.756},"last":2.226},{"interface":"front","average":{"1min":2.3199999999999998,"5min":1.2070000000000001,"15min":0.80900000000000005},"min":{"1min":0.224,"5min":0.224,"15min":0.219},"max":{"1min":20.495999999999999,"5min":20.495999999999999,"15min":20.495999999999999},"last":2.6659999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:37:41 2024","interfaces":[{"interface":"back","average":{"1min":2.4550000000000001,"5min":1.121,"15min":0.80800000000000005},"min":{"1min":0.374,"5min":0.316,"15min":0.27000000000000002},"max":{"1min":25.902999999999999,"5min":25.902999999999999,"15min":25.902999999999999},"last":2.339},{"interface":"front","average":{"1min":2.4470000000000001,"5min":1.085,"15min":0.753},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.30199999999999999},"max":{"1min":24.074000000000002,"5min":24.074000000000002,"15min":24.074000000000002},"last":0.63100000000000001}]}]},{"osd":6,"up_from":144,"seq":618475290764,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3734388,"kb_used_data":3431784,"kb_used_omap":7,"kb_used_meta":302584,"kb_avail":90010764,"statfs":{"total":95995035648,"available":92171022336,"internally_reserved":0,"allocated":3514146816,"data_stored":3513775689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":309846103},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:37:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.93700000000000006,"15min":1.5409999999999999},"min":{"1min":0.308,"5min":0.192,"15min":0.192},"max":{"1min":3.218,"5min":3.9940000000000002,"15min":30.629999999999999},"last":3.4660000000000002},{"interface":"front","average":{"1min":1.145,"5min":1.165,"15min":1.296},"min":{"1min":0.253,"5min":0.217,"15min":0.217},"max":{"1min":3.8900000000000001,"5min":18.093,"15min":30.597000000000001},"last":0.93600000000000005}]},{"osd":1,"last update":"Tue Sep 10 18:37:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.91700000000000004,"15min":0.97299999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.8109999999999999,"5min":3.806,"15min":30.984000000000002},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.96099999999999997,"15min":1.1220000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.24199999999999999},"max":{"1min":2.101,"5min":7.7939999999999996,"15min":30.571000000000002},"last":3.8559999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:37:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.5049999999999999,"15min":1.212},"min":{"1min":0.31,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.0990000000000002,"5min":39.329999999999998,"15min":39.329999999999998},"last":3.9009999999999998},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.2889999999999999,"15min":1.02},"min":{"1min":0.29399999999999998,"5min":0.26200000000000001,"15min":0.23400000000000001},"max":{"1min":6.4059999999999997,"5min":39.061999999999998,"15min":39.061999999999998},"last":3.0800000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:37:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.1479999999999999,"15min":1.6830000000000001},"min":{"1min":0.30199999999999999,"5min":0.28199999999999997,"15min":0.27800000000000002},"max":{"1min":4.4210000000000003,"5min":4.8890000000000002,"15min":61.703000000000003},"last":3.5009999999999999},{"interface":"front","average":{"1min":0.85799999999999998,"5min":1.1180000000000001,"15min":1.6879999999999999},"min":{"1min":0.29599999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.3740000000000001,"5min":17.786000000000001,"15min":61.814},"last":1.0029999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:37:51 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.70599999999999996,"15min":0.61199999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.7749999999999999,"5min":4.0220000000000002,"15min":5.4950000000000001},"last":0.873},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.63100000000000001,"15min":0.64000000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":4.1829999999999998,"5min":4.1829999999999998,"15min":4.6390000000000002},"last":11.531000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:37:51 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.73599999999999999,"15min":0.621},"min":{"1min":0.28199999999999997,"5min":0.28000000000000003,"15min":0.24399999999999999},"max":{"1min":4.2190000000000003,"5min":4.476,"15min":4.6619999999999999},"last":4.4139999999999997},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.78900000000000003,"15min":0.621},"min":{"1min":0.248,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.2370000000000001,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":4.4509999999999996}]},{"osd":7,"last update":"Tue Sep 10 18:37:51 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.76700000000000002,"15min":0.626},"min":{"1min":0.27100000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.873,"5min":3.3650000000000002,"15min":3.3650000000000002},"last":11.449},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.70599999999999996,"15min":0.64300000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.27100000000000002},"max":{"1min":3.7559999999999998,"5min":3.7559999999999998,"15min":5.5309999999999997},"last":0.91000000000000003}]}]},{"osd":5,"up_from":39,"seq":167503724756,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3093164,"kb_used_data":2494884,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":90651988,"statfs":{"total":95995035648,"available":92827635712,"internally_reserved":0,"allocated":2554761216,"data_stored":2554387830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:37:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.97999999999999998,"15min":1.556},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.21099999999999999},"max":{"1min":2.7200000000000002,"5min":15.819000000000001,"15min":52.033999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.2490000000000001,"5min":0.95099999999999996,"15min":1.371},"min":{"1min":0.24399999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":2.4300000000000002,"5min":5.4420000000000002,"15min":52.167999999999999},"last":0.85599999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:37:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":2.3929999999999998,"15min":1.5},"min":{"1min":0.33100000000000002,"5min":0.249,"15min":0.22600000000000001},"max":{"1min":1.3440000000000001,"5min":111.773,"15min":111.773},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.98199999999999998,"5min":2.54,"15min":1.6200000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.247},"max":{"1min":2.6789999999999998,"5min":111.73999999999999,"15min":111.73999999999999},"last":0.83299999999999996}]},{"osd":2,"last update":"Tue Sep 10 18:37:08 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.3360000000000001,"15min":1.242},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.24299999999999999},"max":{"1min":3.3180000000000001,"5min":54.530000000000001,"15min":54.530000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.82599999999999996,"5min":1.2889999999999999,"15min":1.202},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":2.4409999999999998,"5min":54.514000000000003,"15min":54.514000000000003},"last":0.318}]},{"osd":3,"last update":"Tue Sep 10 18:37:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":2.395,"15min":1.879},"min":{"1min":0.29999999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.4260000000000002,"5min":106.205,"15min":106.205},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.75,"5min":2.5430000000000001,"15min":1.829},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.6760000000000002,"5min":105.878,"15min":105.878},"last":0.37}]},{"osd":4,"last update":"Tue Sep 10 18:37:08 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.69699999999999995,"15min":0.58299999999999996},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.20799999999999999},"max":{"1min":2.7120000000000002,"5min":5.8040000000000003,"15min":5.8040000000000003},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.68200000000000005,"15min":0.623},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":2.7509999999999999,"5min":5.5499999999999998,"15min":6.798},"last":0.98899999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:37:46 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.64800000000000002,"15min":0.66900000000000004},"min":{"1min":0.34499999999999997,"5min":0.29299999999999998,"15min":0.27500000000000002},"max":{"1min":2.4630000000000001,"5min":2.4630000000000001,"15min":4.7859999999999996},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.5069999999999999,"5min":0.82699999999999996,"15min":0.72599999999999998},"min":{"1min":0.38300000000000001,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":5.6420000000000003,"5min":5.6420000000000003,"15min":5.6420000000000003},"last":3.556}]},{"osd":7,"last update":"Tue Sep 10 18:37:46 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":1.0860000000000001,"15min":0.76800000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.22500000000000001},"max":{"1min":2.8530000000000002,"5min":28.934000000000001,"15min":28.934000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.98899999999999999,"15min":0.69799999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.25900000000000001},"max":{"1min":2.6379999999999999,"5min":5.8259999999999996,"15min":5.8259999999999996},"last":0.878}]}]},{"osd":4,"up_from":33,"seq":141733920985,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4077568,"kb_used_data":3780216,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":89667584,"statfs":{"total":95995035648,"available":91819606016,"internally_reserved":0,"allocated":3870941184,"data_stored":3870553312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:37:38 2024","interfaces":[{"interface":"back","average":{"1min":2.0019999999999998,"5min":1.0900000000000001,"15min":1.167},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.188},"max":{"1min":8.3789999999999996,"5min":8.3789999999999996,"15min":31.585000000000001},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":0.91400000000000003,"15min":1.0529999999999999},"min":{"1min":0.23300000000000001,"5min":0.22700000000000001,"15min":0.189},"max":{"1min":7.7999999999999998,"5min":7.7999999999999998,"15min":29.802},"last":0.21299999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:37:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.0800000000000001,"15min":1.0860000000000001},"min":{"1min":0.26900000000000002,"5min":0.254,"15min":0.21299999999999999},"max":{"1min":5.2110000000000003,"5min":5.2110000000000003,"15min":50.045000000000002},"last":1.3129999999999999},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.99299999999999999,"15min":1.1359999999999999},"min":{"1min":0.28699999999999998,"5min":0.27100000000000002,"15min":0.22900000000000001},"max":{"1min":3.5009999999999999,"5min":5.5590000000000002,"15min":50.283000000000001},"last":0.69299999999999995}]},{"osd":2,"last update":"Tue Sep 10 18:37:38 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":0.77400000000000002,"15min":0.88700000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.249},"max":{"1min":5.8470000000000004,"5min":5.8470000000000004,"15min":13.898999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":1.4910000000000001,"5min":0.90500000000000003,"15min":0.95899999999999996},"min":{"1min":0.318,"5min":0.26700000000000002,"15min":0.22500000000000001},"max":{"1min":7.7279999999999998,"5min":7.7279999999999998,"15min":14.51},"last":0.24099999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:37:38 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.038,"15min":1.4019999999999999},"min":{"1min":0.33000000000000002,"5min":0.27400000000000002,"15min":0.249},"max":{"1min":7.7699999999999996,"5min":7.7699999999999996,"15min":49.990000000000002},"last":0.26200000000000001},{"interface":"front","average":{"1min":1.446,"5min":1.254,"15min":1.4990000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.255},"max":{"1min":5.1239999999999997,"5min":5.9189999999999996,"15min":50.764000000000003},"last":1.3420000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:37:03 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.745,"15min":0.66500000000000004},"min":{"1min":0.309,"5min":0.223,"15min":0.20499999999999999},"max":{"1min":1.897,"5min":3.0409999999999999,"15min":22.861000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.81599999999999995,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.252,"15min":0.22},"max":{"1min":8.2539999999999996,"5min":8.2539999999999996,"15min":19.611999999999998},"last":0.747}]},{"osd":6,"last update":"Tue Sep 10 18:37:49 2024","interfaces":[{"interface":"back","average":{"1min":3.7669999999999999,"5min":1.359,"15min":0.78900000000000003},"min":{"1min":0.30199999999999999,"5min":0.28799999999999998,"15min":0.26400000000000001},"max":{"1min":51.722000000000001,"5min":51.722000000000001,"15min":51.722000000000001},"last":1.49},{"interface":"front","average":{"1min":3.3220000000000001,"5min":1.256,"15min":0.78800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.30599999999999999},"max":{"1min":51.862000000000002,"5min":51.862000000000002,"15min":51.862000000000002},"last":1.3}]},{"osd":7,"last update":"Tue Sep 10 18:37:49 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.79900000000000004,"15min":0.61799999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.255},"max":{"1min":3.4830000000000001,"5min":3.4830000000000001,"15min":7.0670000000000002},"last":0.71499999999999997},{"interface":"front","average":{"1min":1.3859999999999999,"5min":0.85099999999999998,"15min":0.66300000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.247},"max":{"1min":7.8250000000000002,"5min":7.8250000000000002,"15min":7.8250000000000002},"last":0.78800000000000003}]}]},{"osd":3,"up_from":27,"seq":115964117215,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3252936,"kb_used_data":2357120,"kb_used_omap":1,"kb_used_meta":895742,"kb_avail":90492216,"statfs":{"total":95995035648,"available":92664029184,"internally_reserved":0,"allocated":2413690880,"data_stored":2413295636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917240152},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:37:18 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.04,"15min":1.347},"min":{"1min":0.39900000000000002,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.8899999999999999,"5min":4.6079999999999997,"15min":8.5530000000000008},"last":1.8280000000000001},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3,"15min":1.4690000000000001},"min":{"1min":0.27700000000000002,"5min":0.23300000000000001,"15min":0.193},"max":{"1min":7.1459999999999999,"5min":8.2110000000000003,"15min":13.659000000000001},"last":0.52000000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:37:18 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.3260000000000001,"15min":1.5489999999999999},"min":{"1min":0.224,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.6949999999999998,"5min":6.149,"15min":35.406999999999996},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.0880000000000001,"15min":1.4259999999999999},"min":{"1min":0.33800000000000002,"5min":0.24199999999999999,"15min":0.19},"max":{"1min":2.988,"5min":5.0720000000000001,"15min":36.902000000000001},"last":1.792}]},{"osd":2,"last update":"Tue Sep 10 18:37:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.052,"15min":1.1759999999999999},"min":{"1min":0.33900000000000002,"5min":0.25600000000000001,"15min":0.222},"max":{"1min":1.8700000000000001,"5min":6.6479999999999997,"15min":7.7960000000000003},"last":1.8899999999999999},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.2410000000000001,"15min":1.4139999999999999},"min":{"1min":0.23400000000000001,"5min":0.23000000000000001,"15min":0.217},"max":{"1min":6.3730000000000002,"5min":6.7290000000000001,"15min":14.997999999999999},"last":0.65000000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:37:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.1659999999999999,"15min":1.3260000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.0259999999999998,"5min":6.5250000000000004,"15min":23.684999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.238,"15min":1.282},"min":{"1min":0.33700000000000002,"5min":0.29799999999999999,"15min":0.27200000000000002},"max":{"1min":4.819,"5min":6.1929999999999996,"15min":23.649000000000001},"last":0.55100000000000005}]},{"osd":5,"last update":"Tue Sep 10 18:38:04 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.2310000000000001,"15min":1.3819999999999999},"min":{"1min":0.32300000000000001,"5min":0.314,"15min":0.314},"max":{"1min":3.2599999999999998,"5min":6.601,"15min":23.719999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.643,"5min":1.371,"15min":1.4279999999999999},"min":{"1min":0.34000000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.843,"5min":5.5540000000000003,"15min":23.707000000000001},"last":0.71699999999999997}]},{"osd":6,"last update":"Tue Sep 10 18:37:51 2024","interfaces":[{"interface":"back","average":{"1min":1.6910000000000001,"5min":1.3859999999999999,"15min":1.3009999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":4.7670000000000003,"5min":6.2889999999999997,"15min":31.815999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.361,"5min":1.2549999999999999,"15min":1.4570000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.315},"max":{"1min":3.718,"5min":7.0970000000000004,"15min":31.742000000000001},"last":1.859}]},{"osd":7,"last update":"Tue Sep 10 18:37:51 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.2250000000000001,"15min":1.353},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.859,"5min":6.5730000000000004,"15min":23.748999999999999},"last":0.501},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.307},"max":{"1min":4.7939999999999996,"5min":6.2530000000000001,"15min":25.058},"last":0.78400000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411554,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3275956,"kb_used_data":2970152,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":90469196,"statfs":{"total":95995035648,"available":92640456704,"internally_reserved":0,"allocated":3041435648,"data_stored":3041064152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:37:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.2649999999999999,"15min":1.579},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":3.6000000000000001,"5min":26.585999999999999,"15min":43.890000000000001},"last":3.1070000000000002},{"interface":"front","average":{"1min":1.657,"5min":1.5049999999999999,"15min":1.843},"min":{"1min":0.19700000000000001,"5min":0.19400000000000001,"15min":0.16900000000000001},"max":{"1min":4.7160000000000002,"5min":26.48,"15min":44.331000000000003},"last":3.5840000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:37:55 2024","interfaces":[{"interface":"back","average":{"1min":1.9179999999999999,"5min":1.161,"15min":1.611},"min":{"1min":0.222,"5min":0.19400000000000001,"15min":0.193},"max":{"1min":9.7759999999999998,"5min":9.7759999999999998,"15min":44.575000000000003},"last":5.6909999999999998},{"interface":"front","average":{"1min":1.012,"5min":0.93799999999999994,"15min":1.3340000000000001},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.19500000000000001},"max":{"1min":2.6459999999999999,"5min":7.0270000000000001,"15min":44.368000000000002},"last":2.9809999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:37:14 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.7290000000000001,"15min":1.8360000000000001},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.339,"5min":61.331000000000003,"15min":61.331000000000003},"last":2.8460000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.8879999999999999,"15min":1.9199999999999999},"min":{"1min":0.24199999999999999,"5min":0.23499999999999999,"15min":0.214},"max":{"1min":4.5229999999999997,"5min":61.250999999999998,"15min":61.250999999999998},"last":3.4940000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:37:49 2024","interfaces":[{"interface":"back","average":{"1min":2.1219999999999999,"5min":1.0760000000000001,"15min":1.155},"min":{"1min":0.32300000000000001,"5min":0.30199999999999999,"15min":0.24199999999999999},"max":{"1min":9.8810000000000002,"5min":9.8810000000000002,"15min":12.555999999999999},"last":5.5990000000000002},{"interface":"front","average":{"1min":1.821,"5min":0.96599999999999997,"15min":1.03},"min":{"1min":0.38800000000000001,"5min":0.33000000000000002,"15min":0.317},"max":{"1min":7.4770000000000003,"5min":7.4770000000000003,"15min":12.130000000000001},"last":2.8889999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.93600000000000005,"15min":1.05},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.27500000000000002},"max":{"1min":2.476,"5min":6.3170000000000002,"15min":12.106},"last":2.9119999999999999},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.1599999999999999,"15min":1.1339999999999999},"min":{"1min":0.38800000000000001,"5min":0.317,"15min":0.317},"max":{"1min":4.5110000000000001,"5min":12.529999999999999,"15min":12.529999999999999},"last":2.952}]},{"osd":6,"last update":"Tue Sep 10 18:37:49 2024","interfaces":[{"interface":"back","average":{"1min":4.4859999999999998,"5min":1.696,"15min":1.319},"min":{"1min":0.375,"5min":0.34300000000000003,"15min":0.32100000000000001},"max":{"1min":53.359999999999999,"5min":53.359999999999999,"15min":53.359999999999999},"last":5.6349999999999998},{"interface":"front","average":{"1min":5.1580000000000004,"5min":1.728,"15min":1.256},"min":{"1min":0.441,"5min":0.34999999999999998,"15min":0.30099999999999999},"max":{"1min":53.438000000000002,"5min":53.438000000000002,"15min":53.438000000000002},"last":2.9329999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:37:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1359999999999999,"15min":1.1579999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.30499999999999999},"max":{"1min":4.5960000000000001,"5min":10.68,"15min":10.68},"last":0.83299999999999996},{"interface":"front","average":{"1min":2.0579999999999998,"5min":1.2030000000000001,"15min":1.2649999999999999},"min":{"1min":0.42599999999999999,"5min":0.39300000000000002,"15min":0.307},"max":{"1min":9.8499999999999996,"5min":9.8499999999999996,"15min":12.596},"last":5.6619999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575079,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2394136,"kb_used_data":1497228,"kb_used_omap":1,"kb_used_meta":896894,"kb_avail":91351016,"statfs":{"total":95995035648,"available":93543440384,"internally_reserved":0,"allocated":1533161472,"data_stored":1532753946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918419800},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:37:32 2024","interfaces":[{"interface":"back","average":{"1min":2.488,"5min":1.5960000000000001,"15min":1.823},"min":{"1min":0.249,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":22.042999999999999,"5min":28.893999999999998,"15min":80.456999999999994},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.272,"5min":1.3779999999999999,"15min":1.8380000000000001},"min":{"1min":0.312,"5min":0.22800000000000001,"15min":0.187},"max":{"1min":4.2939999999999996,"5min":31.074999999999999,"15min":80.578000000000003},"last":1.639}]},{"osd":2,"last update":"Tue Sep 10 18:37:52 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.83099999999999996,"15min":1.2729999999999999},"min":{"1min":0.314,"5min":0.20999999999999999,"15min":0.19500000000000001},"max":{"1min":3.2759999999999998,"5min":5.8380000000000001,"15min":13.035},"last":1.6140000000000001},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.99199999999999999,"15min":1.26},"min":{"1min":0.26000000000000001,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":6.2069999999999999,"5min":6.2069999999999999,"15min":12.222},"last":2.3410000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:37:30 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.28,"15min":1.726},"min":{"1min":0.25800000000000001,"5min":0.23699999999999999,"15min":0.22600000000000001},"max":{"1min":4.8449999999999998,"5min":8.0399999999999991,"15min":43.076000000000001},"last":0.443},{"interface":"front","average":{"1min":1.109,"5min":1.0669999999999999,"15min":1.7290000000000001},"min":{"1min":0.33800000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.3929999999999998,"5min":5.8879999999999999,"15min":41.93},"last":1.5840000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:37:32 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.0760000000000001,"15min":1.1379999999999999},"min":{"1min":0.35999999999999999,"5min":0.312,"15min":0.29799999999999999},"max":{"1min":2.2879999999999998,"5min":5.9930000000000003,"15min":7.2859999999999996},"last":4.9420000000000002},{"interface":"front","average":{"1min":1.159,"5min":1.2470000000000001,"15min":1.3},"min":{"1min":0.36599999999999999,"5min":0.30299999999999999,"15min":0.28299999999999997},"max":{"1min":6.2489999999999997,"5min":7.8769999999999998,"15min":19.393000000000001},"last":3.0259999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:37:03 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":1.202,"15min":1.3089999999999999},"min":{"1min":0.308,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.7919999999999998,"5min":7.2110000000000003,"15min":15.48},"last":2.4220000000000002},{"interface":"front","average":{"1min":0.79000000000000004,"5min":1.337,"15min":1.248},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.308},"max":{"1min":3.931,"5min":11.414999999999999,"15min":11.414999999999999},"last":0.80100000000000005}]},{"osd":6,"last update":"Tue Sep 10 18:37:52 2024","interfaces":[{"interface":"back","average":{"1min":5.1980000000000004,"5min":2.569,"15min":1.714},"min":{"1min":0.36099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":38.347000000000001,"5min":67.287000000000006,"15min":67.287000000000006},"last":1.2410000000000001},{"interface":"front","average":{"1min":7.3659999999999997,"5min":3.1000000000000001,"15min":1.958},"min":{"1min":0.52100000000000002,"5min":0.374,"15min":0.33600000000000002},"max":{"1min":49.127000000000002,"5min":67.400000000000006,"15min":67.400000000000006},"last":0.61199999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:37:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.2809999999999999,"15min":1.278},"min":{"1min":0.37,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":4.7089999999999996,"5min":8.3650000000000002,"15min":15.545999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.2470000000000001,"15min":1.2649999999999999},"min":{"1min":0.40400000000000003,"5min":0.40300000000000002,"15min":0.313},"max":{"1min":4.3170000000000002,"5min":6.3049999999999997,"15min":7.359},"last":0.47699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738604,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5511488,"kb_used_data":4917428,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":88233664,"statfs":{"total":95995035648,"available":90351271936,"internally_reserved":0,"allocated":5035446272,"data_stored":5035062369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,0,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:37:26 2024","interfaces":[{"interface":"back","average":{"1min":2.6309999999999998,"5min":2.968,"15min":2.1070000000000002},"min":{"1min":0.29899999999999999,"5min":0.185,"15min":0.185},"max":{"1min":8.2409999999999997,"5min":146.44499999999999,"15min":146.44499999999999},"last":2.2189999999999999},{"interface":"front","average":{"1min":1.841,"5min":2.831,"15min":2.1539999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.17999999999999999},"max":{"1min":5.9009999999999998,"5min":148.77199999999999,"15min":148.77199999999999},"last":0.42099999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:37:59 2024","interfaces":[{"interface":"back","average":{"1min":2.3679999999999999,"5min":2.8359999999999999,"15min":2.0510000000000002},"min":{"1min":0.39600000000000002,"5min":0.19700000000000001,"15min":0.189},"max":{"1min":9.2370000000000001,"5min":146.56999999999999,"15min":146.56999999999999},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.72,"5min":2.548,"15min":1.9690000000000001},"min":{"1min":0.36599999999999999,"5min":0.23000000000000001,"15min":0.22},"max":{"1min":6.5209999999999999,"5min":146.601,"15min":146.601},"last":0.81999999999999995}]},{"osd":3,"last update":"Tue Sep 10 18:37:26 2024","interfaces":[{"interface":"back","average":{"1min":2.4169999999999998,"5min":2.609,"15min":2.0529999999999999},"min":{"1min":0.373,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":7.6870000000000003,"5min":140.68299999999999,"15min":140.68299999999999},"last":0.872},{"interface":"front","average":{"1min":2.8140000000000001,"5min":2.9129999999999998,"15min":2.1019999999999999},"min":{"1min":0.54500000000000004,"5min":0.26900000000000002,"15min":0.23899999999999999},"max":{"1min":8.2720000000000002,"5min":140.572,"15min":140.572},"last":2.6680000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:37:48 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.1599999999999999,"15min":1.2769999999999999},"min":{"1min":0.504,"5min":0.27900000000000003,"15min":0.246},"max":{"1min":5.3170000000000002,"5min":6.1970000000000001,"15min":17.151},"last":2.3439999999999999},{"interface":"front","average":{"1min":1.954,"5min":1.0569999999999999,"15min":1.2270000000000001},"min":{"1min":0.62,"5min":0.31,"15min":0.25900000000000001},"max":{"1min":4.5549999999999997,"5min":4.5549999999999997,"15min":18.728999999999999},"last":0.59699999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:37:10 2024","interfaces":[{"interface":"back","average":{"1min":1.736,"5min":1.2,"15min":1.361},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.4589999999999996,"5min":8.1259999999999994,"15min":18.77},"last":0.627},{"interface":"front","average":{"1min":1.524,"5min":1.262,"15min":1.3200000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":5.5369999999999999,"5min":20.460000000000001,"15min":20.460000000000001},"last":1.018}]},{"osd":6,"last update":"Tue Sep 10 18:37:48 2024","interfaces":[{"interface":"back","average":{"1min":1.9039999999999999,"5min":1.319,"15min":1.2729999999999999},"min":{"1min":0.47699999999999998,"5min":0.36299999999999999,"15min":0.30499999999999999},"max":{"1min":5.3529999999999998,"5min":6.4909999999999997,"15min":8.3940000000000001},"last":2.379},{"interface":"front","average":{"1min":2.2349999999999999,"5min":1.165,"15min":1.272},"min":{"1min":0.60299999999999998,"5min":0.34399999999999997,"15min":0.33300000000000002},"max":{"1min":7.5629999999999997,"5min":7.5629999999999997,"15min":9.7859999999999996},"last":0.998}]},{"osd":7,"last update":"Tue Sep 10 18:37:48 2024","interfaces":[{"interface":"back","average":{"1min":1.776,"5min":1.3400000000000001,"15min":1.3520000000000001},"min":{"1min":0.38700000000000001,"5min":0.377,"15min":0.26700000000000002},"max":{"1min":4.4980000000000002,"5min":10.75,"15min":13.711},"last":0.65100000000000002},{"interface":"front","average":{"1min":1.9810000000000001,"5min":1.2070000000000001,"15min":1.2849999999999999},"min":{"1min":0.38100000000000001,"5min":0.378,"15min":0.33500000000000002},"max":{"1min":6.4630000000000001,"5min":11.218999999999999,"15min":15.032},"last":0.89200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4996272128,"data_stored":4996268056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1523908608,"data_stored":1523908608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3031302144,"data_stored":3031302144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2407202816,"data_stored":2407202816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3848081408,"data_stored":3848077336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2533163008,"data_stored":2533163008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3504803840,"data_stored":3504799768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1918369792,"data_stored":1918369792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:38:08.760 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-10T18:38:08.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pg_num 61 2024-09-10T18:38:08.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:08.880 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:08.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:08.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:08.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:09.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:09.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:09.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:09.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:09.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:09.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:09.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:09.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:38:09.443 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:09.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:09.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:09.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:09.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:09.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:09.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:09.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:09.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:09.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:09.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:09.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:09.890 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:09.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:09.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:09.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:10.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:10.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:38:10.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:09 smithi028 ceph-mon[29291]: pgmap v753: 178 pgs: 178 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-10T18:38:10.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:10.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:38:10.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:09 smithi028 ceph-mon[39220]: pgmap v753: 178 pgs: 178 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-10T18:38:10.030 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:10.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:10.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:10.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:10.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:38:10.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:09 smithi204 ceph-mon[32357]: pgmap v753: 178 pgs: 178 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-10T18:38:10.131 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:10.163 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:10.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:10.265 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:10.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:10.298 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:10.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:10.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:38:10.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:10.431 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:10.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:10.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:10.532 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:10.558 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:10.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:10.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:10.660 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:10.686 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:10.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:10.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:10.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:10.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:10.839 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:10.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:10.940 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:10.970 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:10.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:10.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:11.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:11.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:11.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:11.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:11.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:11.249 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:11.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:11.350 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:11.385 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:11.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:11.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:11.486 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:11.514 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:11.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:11.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:11.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:11.656 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:11.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:11.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:11.753 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:38:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:38:11] "GET /metrics HTTP/1.1" 200 224698 "" "Prometheus/2.43.0" 2024-09-10T18:38:11.758 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:11.793 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:11.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:11.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:11.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:11.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:11.933 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:11.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:12.034 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:12.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:12.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:12.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:12.122 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:11 smithi204 ceph-mon[32357]: pgmap v754: 178 pgs: 178 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T18:38:12.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:11 smithi028 ceph-mon[39220]: pgmap v754: 178 pgs: 178 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T18:38:12.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:11 smithi028 ceph-mon[29291]: pgmap v754: 178 pgs: 178 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T18:38:12.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:12.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:12.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:12.317 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:12.346 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:12.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:12.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:12.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:12.484 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:12.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:12.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:12.586 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:12.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:38:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:38:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:38:12.618 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:12.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:12.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:12.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:12.755 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:12.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:12.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:12.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:12.894 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:12.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:12.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:13.016 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:13.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:13.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:13.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:13.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:13.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:13.245 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:13.273 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:13.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:13.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:13.375 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:13.407 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:13.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:13.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:13.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:13.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:13.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:13.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:13.677 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:13.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:13.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:13.779 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:13.814 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:13.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:13.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:13.916 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:13.952 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:13.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:13.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:14.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:14.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:14.091 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:14.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:14.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:14.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:14.229 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:14.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:14.330 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:14.358 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:14.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:14.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:14.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:14.492 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:14.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:14.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:14.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:14 smithi028 ceph-mon[39220]: pgmap v755: 178 pgs: 178 active+clean; 6.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:38:14.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:14 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/666033252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:38:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:14 smithi028 ceph-mon[29291]: pgmap v755: 178 pgs: 178 active+clean; 6.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:38:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:14 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/666033252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:38:14.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:14.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:14 smithi204 ceph-mon[32357]: pgmap v755: 178 pgs: 178 active+clean; 6.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:38:14.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:14 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/666033252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:38:14.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:14.616 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:14.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:14.717 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:14.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:14.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:14.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:14.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:14.890 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:14.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:14.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:14.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:15.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:15.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:15.104 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:38:14 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:38:15.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:15.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:15.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:15.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:15.300 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:15.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:15.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:15.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:15.443 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:15.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:15.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:15.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:15.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:15.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:15.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:15.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:15.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:15.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:15.864 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:15.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:15.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:15.966 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:15.998 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:15.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:15.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:16.100 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:16.130 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:16.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:16.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:16.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:16.266 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:16.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:16.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:16.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:16.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:16.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:16.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:16 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:16.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:16 smithi028 ceph-mon[29291]: pgmap v756: 178 pgs: 178 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T18:38:16.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:16 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:16.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:16 smithi028 ceph-mon[39220]: pgmap v756: 178 pgs: 178 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T18:38:16.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:16.536 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:16.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:16.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:16 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:16 smithi204 ceph-mon[32357]: pgmap v756: 178 pgs: 178 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T18:38:16.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:16.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:16.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:16.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:16.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:16.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:16.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:16.926 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:16.958 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:16.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:16.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:17.060 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:17.096 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:17.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:17.197 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:38:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:38:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:38:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:38:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:38:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:38:17.200 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:38:17.200 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:38:17.200 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:38:17.200 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:38:17.200 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:38:17.200 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:38:17.200 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:38:17.200 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:38:17.200 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:38:17.200 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:38:17.231 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:17.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:17.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:17.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:17.370 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:17.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:17.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:38:17.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:17.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:17.525 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:17.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:17.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:17.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:17.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:17.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:17.765 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:17.797 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:17.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:17.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:17.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:17.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:18.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:18.085 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:18.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:18.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:18.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:18 smithi028 ceph-mon[29291]: pgmap v757: 178 pgs: 178 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:38:18.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:18 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:38:18.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:18 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:38:18.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:18.225 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:18.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:18.327 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:18.349 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:38:18.353 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:18.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:18.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:18.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:18.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:18 smithi028 ceph-mon[39220]: pgmap v757: 178 pgs: 178 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:38:18.493 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:18 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:38:18.493 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:18 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:38:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:18.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:18.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:18 smithi204 ceph-mon[32357]: pgmap v757: 178 pgs: 178 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:38:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:18 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:38:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:18 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:38:18.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:18.654 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:18.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:18.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:18.756 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:18.787 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:18.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:18.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:18.855 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:38:18.856 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:38:18.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.856 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:174579 10.6 e346/314 10:66cb8a9b:::benchmark_data_smithi028_161947_object37655:head v 346'10959, mlcod=346'10957)", 2024-09-10T18:38:18.856 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:18.775602+0000", 2024-09-10T18:38:18.856 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.020919219999999999, 2024-09-10T18:38:18.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020945657999999999, 2024-09-10T18:38:18.856 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:18.856 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:38:18.856 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:18.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:18.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.775602+0000", 2024-09-10T18:38:18.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:18.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.775602+0000", 2024-09-10T18:38:18.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:18.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.775600+0000", 2024-09-10T18:38:18.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:38:18.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:18.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.775608+0000", 2024-09-10T18:38:18.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.79e-06 2024-09-10T18:38:18.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:18.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.775619+0000", 2024-09-10T18:38:18.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0625000000000001e-05 2024-09-10T18:38:18.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:18.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.775625+0000", 2024-09-10T18:38:18.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1009999999999997e-06 2024-09-10T18:38:18.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:38:18.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.775684+0000", 2024-09-10T18:38:18.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8696000000000002e-05 2024-09-10T18:38:18.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:38:18.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.775704+0000", 2024-09-10T18:38:18.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0292000000000002e-05 2024-09-10T18:38:18.859 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.859 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:18.860 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.860 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.15693.0:174606 10.4 e346/314)", 2024-09-10T18:38:18.860 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:18.796036+0000", 2024-09-10T18:38:18.860 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.00048539200000000002, 2024-09-10T18:38:18.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057167800000000005, 2024-09-10T18:38:18.860 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:18.860 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-09-10T18:38:18.860 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:18.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:18.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.796036+0000", 2024-09-10T18:38:18.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:18.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.796036+0000", 2024-09-10T18:38:18.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:18.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.796035+0000", 2024-09-10T18:38:18.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:38:18.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:18.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.796040+0000", 2024-09-10T18:38:18.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8409999999999999e-06 2024-09-10T18:38:18.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:18.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.796047+0000", 2024-09-10T18:38:18.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1210000000000001e-06 2024-09-10T18:38:18.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:18.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.796051+0000", 2024-09-10T18:38:18.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1119999999999998e-06 2024-09-10T18:38:18.862 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.862 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:18.863 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.863 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:174586 10.5 e346/314 10:a65821cd:::benchmark_data_smithi028_161947_object37662:head v 346'10929, mlcod=346'10927)", 2024-09-10T18:38:18.863 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:18.778866+0000", 2024-09-10T18:38:18.863 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.017655173999999999, 2024-09-10T18:38:18.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017788622, 2024-09-10T18:38:18.863 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:18.863 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:38:18.863 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:18.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:18.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.778866+0000", 2024-09-10T18:38:18.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:18.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.778866+0000", 2024-09-10T18:38:18.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:18.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.778865+0000", 2024-09-10T18:38:18.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:38:18.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:18.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.778885+0000", 2024-09-10T18:38:18.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0392000000000001e-05 2024-09-10T18:38:18.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:18.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.778896+0000", 2024-09-10T18:38:18.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1259e-05 2024-09-10T18:38:18.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:18.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.778902+0000", 2024-09-10T18:38:18.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5829999999999997e-06 2024-09-10T18:38:18.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:38:18.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.779068+0000", 2024-09-10T18:38:18.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016628300000000001 2024-09-10T18:38:18.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:38:18.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.779106+0000", 2024-09-10T18:38:18.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7991000000000003e-05 2024-09-10T18:38:18.866 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.866 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:18.866 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.866 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:174581 10.6 e346/314 10:688b9bc3:::benchmark_data_smithi028_161947_object37657:head v 346'10960, mlcod=346'10957)", 2024-09-10T18:38:18.866 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:18.778960+0000", 2024-09-10T18:38:18.867 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.017561568, 2024-09-10T18:38:18.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017757274, 2024-09-10T18:38:18.867 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:18.867 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:38:18.867 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:18.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:18.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.778960+0000", 2024-09-10T18:38:18.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:18.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.778960+0000", 2024-09-10T18:38:18.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:18.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.778959+0000", 2024-09-10T18:38:18.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:38:18.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:18.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.778964+0000", 2024-09-10T18:38:18.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.848e-06 2024-09-10T18:38:18.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:18.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.778969+0000", 2024-09-10T18:38:18.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5509999999999999e-06 2024-09-10T18:38:18.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:18.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.778973+0000", 2024-09-10T18:38:18.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6090000000000001e-06 2024-09-10T18:38:18.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:38:18.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.786170+0000", 2024-09-10T18:38:18.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071973200000000001 2024-09-10T18:38:18.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:38:18.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.786191+0000", 2024-09-10T18:38:18.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1154000000000001e-05 2024-09-10T18:38:18.870 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.870 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:18.870 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.870 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:174589 10.5 e346/314 10:a2bb0b0b:::benchmark_data_smithi028_161947_object37665:head v 346'10930, mlcod=346'10927)", 2024-09-10T18:38:18.870 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:18.782622+0000", 2024-09-10T18:38:18.870 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0138993, 2024-09-10T18:38:18.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014160122000000001, 2024-09-10T18:38:18.870 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:18.870 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:38:18.870 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:18.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:18.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.782622+0000", 2024-09-10T18:38:18.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:18.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.782622+0000", 2024-09-10T18:38:18.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:18.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.782621+0000", 2024-09-10T18:38:18.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:38:18.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:18.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.782628+0000", 2024-09-10T18:38:18.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2849999999999997e-06 2024-09-10T18:38:18.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:18.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.782642+0000", 2024-09-10T18:38:18.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3803e-05 2024-09-10T18:38:18.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:18.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.782647+0000", 2024-09-10T18:38:18.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6239999999999999e-06 2024-09-10T18:38:18.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:38:18.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.786265+0000", 2024-09-10T18:38:18.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003617814 2024-09-10T18:38:18.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:38:18.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.786284+0000", 2024-09-10T18:38:18.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8487000000000001e-05 2024-09-10T18:38:18.874 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.874 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:18.874 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.874 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:174596 10.5 e346/314 10:a8be3465:::benchmark_data_smithi028_161947_object37672:head v 346'10931, mlcod=346'10927)", 2024-09-10T18:38:18.874 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:18.787188+0000", 2024-09-10T18:38:18.874 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0093328179999999997, 2024-09-10T18:38:18.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0097614720000000002, 2024-09-10T18:38:18.874 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:18.874 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:38:18.874 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:18.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:18.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787188+0000", 2024-09-10T18:38:18.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:18.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787188+0000", 2024-09-10T18:38:18.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:18.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787188+0000", 2024-09-10T18:38:18.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:38:18.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:18.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787193+0000", 2024-09-10T18:38:18.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6579999999999999e-06 2024-09-10T18:38:18.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:18.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787199+0000", 2024-09-10T18:38:18.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4530000000000001e-06 2024-09-10T18:38:18.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:18.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787202+0000", 2024-09-10T18:38:18.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5999999999999998e-06 2024-09-10T18:38:18.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:38:18.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.789907+0000", 2024-09-10T18:38:18.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027046459999999998 2024-09-10T18:38:18.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:38:18.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.789925+0000", 2024-09-10T18:38:18.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7892000000000001e-05 2024-09-10T18:38:18.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.877 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:18.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.878 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:174595 10.d e346/314 10:bfa1387a:::benchmark_data_smithi028_161947_object37671:head v 346'10953, mlcod=346'10951)", 2024-09-10T18:38:18.878 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:18.787241+0000", 2024-09-10T18:38:18.878 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0092801029999999996, 2024-09-10T18:38:18.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0097696190000000002, 2024-09-10T18:38:18.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:18.878 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:38:18.878 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:18.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:18.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787241+0000", 2024-09-10T18:38:18.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:18.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787241+0000", 2024-09-10T18:38:18.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:18.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787240+0000", 2024-09-10T18:38:18.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:38:18.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:18.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787246+0000", 2024-09-10T18:38:18.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4389999999999999e-06 2024-09-10T18:38:18.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:18.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787252+0000", 2024-09-10T18:38:18.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7349999999999999e-06 2024-09-10T18:38:18.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:18.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787256+0000", 2024-09-10T18:38:18.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.799e-06 2024-09-10T18:38:18.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:38:18.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.793298+0000", 2024-09-10T18:38:18.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060423200000000003 2024-09-10T18:38:18.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:38:18.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.793315+0000", 2024-09-10T18:38:18.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6880000000000001e-05 2024-09-10T18:38:18.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.881 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:18.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.881 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15693.0:174598 10.4 10:2bc8a13e:::benchmark_data_smithi028_161947_object37674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-10T18:38:18.881 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:18.787512+0000", 2024-09-10T18:38:18.882 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0090091489999999993, 2024-09-10T18:38:18.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095658549999999998, 2024-09-10T18:38:18.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:18.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:38:18.882 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:38:18.882 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15693", 2024-09-10T18:38:18.882 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:38:18.882 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 174598 2024-09-10T18:38:18.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.882 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:18.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:18.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787512+0000", 2024-09-10T18:38:18.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:18.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787512+0000", 2024-09-10T18:38:18.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:18.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787510+0000", 2024-09-10T18:38:18.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:38:18.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:18.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787517+0000", 2024-09-10T18:38:18.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.438e-06 2024-09-10T18:38:18.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:18.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787530+0000", 2024-09-10T18:38:18.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3210000000000001e-05 2024-09-10T18:38:18.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:18.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.787535+0000", 2024-09-10T18:38:18.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4009999999999996e-06 2024-09-10T18:38:18.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:38:18.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.789065+0000", 2024-09-10T18:38:18.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015295020000000001 2024-09-10T18:38:18.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:38:18.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.789171+0000", 2024-09-10T18:38:18.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000106383 2024-09-10T18:38:18.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-09-10T18:38:18.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.789274+0000", 2024-09-10T18:38:18.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103347 2024-09-10T18:38:18.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:38:18.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.793696+0000", 2024-09-10T18:38:18.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044219790000000004 2024-09-10T18:38:18.886 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.886 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:18.886 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.886 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:174601 10.d e346/314)", 2024-09-10T18:38:18.886 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:18.790752+0000", 2024-09-10T18:38:18.886 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0057696529999999996, 2024-09-10T18:38:18.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006405076, 2024-09-10T18:38:18.887 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:18.887 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-09-10T18:38:18.887 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:18.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:18.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.790752+0000", 2024-09-10T18:38:18.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:18.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.790752+0000", 2024-09-10T18:38:18.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:18.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.790750+0000", 2024-09-10T18:38:18.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:38:18.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:18.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.790757+0000", 2024-09-10T18:38:18.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1570000000000003e-06 2024-09-10T18:38:18.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:18.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.790767+0000", 2024-09-10T18:38:18.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0010000000000001e-05 2024-09-10T18:38:18.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:18.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.790773+0000", 2024-09-10T18:38:18.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6509999999999998e-06 2024-09-10T18:38:18.889 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.889 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:18.889 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.889 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:174605 10.c e346/314 10:3345506e:::benchmark_data_smithi028_161947_object37681:head v 346'10959, mlcod=346'10957)", 2024-09-10T18:38:18.889 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:18.792713+0000", 2024-09-10T18:38:18.889 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0038082250000000002, 2024-09-10T18:38:18.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044897219999999998, 2024-09-10T18:38:18.890 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:18.890 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:38:18.890 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:18.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:18.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.792713+0000", 2024-09-10T18:38:18.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:18.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.792713+0000", 2024-09-10T18:38:18.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:18.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.792713+0000", 2024-09-10T18:38:18.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:38:18.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:18.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.792718+0000", 2024-09-10T18:38:18.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9490000000000004e-06 2024-09-10T18:38:18.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:18.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.792722+0000", 2024-09-10T18:38:18.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6949999999999999e-06 2024-09-10T18:38:18.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:18.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.792725+0000", 2024-09-10T18:38:18.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1389999999999999e-06 2024-09-10T18:38:18.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:38:18.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.792777+0000", 2024-09-10T18:38:18.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1279000000000002e-05 2024-09-10T18:38:18.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:38:18.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.792797+0000", 2024-09-10T18:38:18.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0197e-05 2024-09-10T18:38:18.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:38:18.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.797174+0000", 2024-09-10T18:38:18.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0043776630000000004 2024-09-10T18:38:18.893 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.893 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:18.893 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.894 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15693.0:174606 10.4 10:21a55d26:::benchmark_data_smithi028_161947_object37682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-10T18:38:18.894 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:18.793641+0000", 2024-09-10T18:38:18.894 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.002880194, 2024-09-10T18:38:18.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0036308579999999998, 2024-09-10T18:38:18.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:18.894 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:38:18.894 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:38:18.894 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15693", 2024-09-10T18:38:18.894 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:38:18.894 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 174606 2024-09-10T18:38:18.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.895 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:18.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:18.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.793641+0000", 2024-09-10T18:38:18.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:18.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.793641+0000", 2024-09-10T18:38:18.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:18.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.793639+0000", 2024-09-10T18:38:18.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:38:18.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:18.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.793646+0000", 2024-09-10T18:38:18.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7279999999999995e-06 2024-09-10T18:38:18.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:18.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.793661+0000", 2024-09-10T18:38:18.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4724e-05 2024-09-10T18:38:18.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:18.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.793667+0000", 2024-09-10T18:38:18.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5180000000000004e-06 2024-09-10T18:38:18.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:38:18.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.793723+0000", 2024-09-10T18:38:18.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6696000000000001e-05 2024-09-10T18:38:18.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:38:18.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.793826+0000", 2024-09-10T18:38:18.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000102784 2024-09-10T18:38:18.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-09-10T18:38:18.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.793941+0000", 2024-09-10T18:38:18.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000115108 2024-09-10T18:38:18.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:38:18.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.797248+0000", 2024-09-10T18:38:18.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033065360000000001 2024-09-10T18:38:18.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:38:18.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.797273+0000", 2024-09-10T18:38:18.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5641000000000001e-05 2024-09-10T18:38:18.900 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.900 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:18.900 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.900 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:174604 10.2 e346/314 10:47180d50:::benchmark_data_smithi028_161947_object37680:head v 346'11201, mlcod=346'11198)", 2024-09-10T18:38:18.900 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:18.794530+0000", 2024-09-10T18:38:18.901 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0019908259999999998, 2024-09-10T18:38:18.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028225379999999999, 2024-09-10T18:38:18.901 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:18.901 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:38:18.901 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:18.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:18.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.794530+0000", 2024-09-10T18:38:18.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:18.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.794530+0000", 2024-09-10T18:38:18.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:18.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.794529+0000", 2024-09-10T18:38:18.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:38:18.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:18.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.794536+0000", 2024-09-10T18:38:18.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1550000000000001e-06 2024-09-10T18:38:18.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:18.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.794549+0000", 2024-09-10T18:38:18.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2126999999999999e-05 2024-09-10T18:38:18.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:18.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.794553+0000", 2024-09-10T18:38:18.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5619999999999997e-06 2024-09-10T18:38:18.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:38:18.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.796307+0000", 2024-09-10T18:38:18.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001753562 2024-09-10T18:38:18.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:38:18.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.796324+0000", 2024-09-10T18:38:18.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7580000000000001e-05 2024-09-10T18:38:18.904 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.904 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:18.904 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.904 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:174603 10.5 e346/314)", 2024-09-10T18:38:18.904 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:18.795891+0000", 2024-09-10T18:38:18.904 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.00062993399999999996, 2024-09-10T18:38:18.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015140220000000001, 2024-09-10T18:38:18.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:18.904 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-09-10T18:38:18.904 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:18.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:18.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.795891+0000", 2024-09-10T18:38:18.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:18.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.795891+0000", 2024-09-10T18:38:18.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:18.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:18.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.795891+0000", 2024-09-10T18:38:18.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:38:18.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:18.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.795896+0000", 2024-09-10T18:38:18.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4840000000000003e-06 2024-09-10T18:38:18.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:18.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.795902+0000", 2024-09-10T18:38:18.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6450000000000002e-06 2024-09-10T18:38:18.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:18.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:18.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:18.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:18.795905+0000", 2024-09-10T18:38:18.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.591e-06 2024-09-10T18:38:18.907 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.907 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:18.907 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.907 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:18.907 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:38:18.907 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 13 2024-09-10T18:38:18.907 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:38:18.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:18.944 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:18.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:18.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:19.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:19.092 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:19.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:19.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:19.194 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:19.224 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:19.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:19.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:19.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:19.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:19.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:19.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:19.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:19 smithi028 ceph-mon[29291]: osdmap e346: 8 total, 8 up, 8 in 2024-09-10T18:38:19.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:19 smithi028 ceph-mon[39220]: osdmap e346: 8 total, 8 up, 8 in 2024-09-10T18:38:19.505 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:19.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:19.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:19.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:19 smithi204 ceph-mon[32357]: osdmap e346: 8 total, 8 up, 8 in 2024-09-10T18:38:19.607 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:19.642 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:19.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:19.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:19.744 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:19.773 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:19.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:19.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:19.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:19.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:19.910 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:19.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:19.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:38:20.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:20.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:20.072 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:20.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:20.173 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:20.200 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:20.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:20.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:20.302 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:20.336 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:20.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:20.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:20.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:20.472 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:20.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:20.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:20.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:20 smithi028 ceph-mon[29291]: pgmap v759: 178 pgs: 178 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:38:20.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:20 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:38:20.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:20 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:38:20.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:20 smithi028 ceph-mon[29291]: osdmap e347: 8 total, 8 up, 8 in 2024-09-10T18:38:20.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:20 smithi028 ceph-mon[39220]: pgmap v759: 178 pgs: 178 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:38:20.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:20 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:38:20.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:20 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:38:20.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:20 smithi028 ceph-mon[39220]: osdmap e347: 8 total, 8 up, 8 in 2024-09-10T18:38:20.574 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:20 smithi204 ceph-mon[32357]: pgmap v759: 178 pgs: 178 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:38:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:20 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:38:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:20 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:38:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:20 smithi204 ceph-mon[32357]: osdmap e347: 8 total, 8 up, 8 in 2024-09-10T18:38:20.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:20.605 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:20.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:20.706 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:20.736 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:20.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:20.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:20.838 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:20.866 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:20.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:20.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:20.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:21.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:21.014 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:21.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:21.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:21.151 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:21.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:21.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:21.253 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:21.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:21.291 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:21.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:21.392 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:21.423 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:21.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:21.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:21.525 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:21.557 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:21.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:21.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:21.590 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:38:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:38:21] "GET /metrics HTTP/1.1" 200 224698 "" "Prometheus/2.43.0" 2024-09-10T18:38:21.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:21.694 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:21.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:21.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:21.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:21.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:21.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:21.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:21.982 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:21.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:21.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:22.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:22.121 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:22.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:22.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:22.223 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:22.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:21 smithi028 ceph-mon[29291]: pgmap v761: 178 pgs: 178 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-09-10T18:38:22.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:21 smithi028 ceph-mon[39220]: pgmap v761: 178 pgs: 178 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-09-10T18:38:22.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:22.256 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:22.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:22.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-10T18:38:22.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:21 smithi204 ceph-mon[32357]: pgmap v761: 178 pgs: 178 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-09-10T18:38:22.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:38:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:38:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:38:22.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:22.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:22.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:22.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:22.551 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:22.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:22.582 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:22.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:22.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:22.743 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:22.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:22.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:22.845 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:22.883 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:22.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:22.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:22.985 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:23.019 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:23.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:23.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:23.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:23.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:23.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:23.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:23.309 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:23.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:23.410 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:23.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:23.440 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:23.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:23.497 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:38:23.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:23.576 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:23.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:23.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:23.678 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:23.715 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:23.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:23.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:23.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:23.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:23.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:23.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:23.996 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:23.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:23.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:24.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:24.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:24.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:24.178 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:38:24.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:24 smithi028 ceph-mon[39220]: pgmap v762: 178 pgs: 178 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.50k op/s 2024-09-10T18:38:24.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:24 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:38:24.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:24 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:38:24.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:38:24.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:24 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3853304443' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:38:24.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:24 smithi028 ceph-mon[29291]: pgmap v762: 178 pgs: 178 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.50k op/s 2024-09-10T18:38:24.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:24 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:38:24.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:24 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:38:24.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:38:24.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:24 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3853304443' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:38:24.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:24.271 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:24.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:24.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:24.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:24.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:24.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:24.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:24.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:24.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:24.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:24.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:24 smithi204 ceph-mon[32357]: pgmap v762: 178 pgs: 178 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.50k op/s 2024-09-10T18:38:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:24 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:38:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:24 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:38:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:38:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:24 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3853304443' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:38:24.640 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:24.672 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:24.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:24.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:24.774 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:24.808 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:24.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:24.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:24.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:24.945 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:24.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:24.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:25.048 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:25.086 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:25.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:25.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:25.183 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:38:24 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:38:25.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:25.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:25.240 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:25.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:25.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:25.373 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:25 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:38:25.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:25 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3853304443' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:38:25.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:25 smithi028 ceph-mon[39220]: osdmap e348: 8 total, 8 up, 8 in 2024-09-10T18:38:25.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:25 smithi028 ceph-mon[39220]: 6.19 scrub starts 2024-09-10T18:38:25.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:25 smithi028 ceph-mon[39220]: 6.19 scrub ok 2024-09-10T18:38:25.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:25 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:38:25.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:25 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3853304443' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:38:25.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:25 smithi028 ceph-mon[29291]: osdmap e348: 8 total, 8 up, 8 in 2024-09-10T18:38:25.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:25 smithi028 ceph-mon[29291]: 6.19 scrub starts 2024-09-10T18:38:25.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:25 smithi028 ceph-mon[29291]: 6.19 scrub ok 2024-09-10T18:38:25.386 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:25.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:25.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:25.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:25.522 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:38:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:25.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:25.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:25.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:25 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:38:25.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:25 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3853304443' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:38:25.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:25 smithi204 ceph-mon[32357]: osdmap e348: 8 total, 8 up, 8 in 2024-09-10T18:38:25.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:25 smithi204 ceph-mon[32357]: 6.19 scrub starts 2024-09-10T18:38:25.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:25 smithi204 ceph-mon[32357]: 6.19 scrub ok 2024-09-10T18:38:25.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:25.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:25.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:25.822 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:25.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:25.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:25.888 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: specified pg_num 61 < pg_num_max 32 2024-09-10T18:38:25.924 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:25.956 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:25.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:25.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:26.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:26.101 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:26.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:26.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:26.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:26.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:26.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:26.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:26.378 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:26.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:26.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:26.480 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:26.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:26.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[29291]: 1.2 scrub starts 2024-09-10T18:38:26.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[29291]: 1.2 scrub ok 2024-09-10T18:38:26.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[29291]: 4.12 scrub starts 2024-09-10T18:38:26.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[29291]: 4.12 scrub ok 2024-09-10T18:38:26.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[29291]: pgmap v764: 178 pgs: 178 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-10T18:38:26.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[29291]: 6.a scrub starts 2024-09-10T18:38:26.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[29291]: 6.a scrub ok 2024-09-10T18:38:26.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4286580857' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"}]: dispatch 2024-09-10T18:38:26.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[29291]: from='client.25708 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"}]: dispatch 2024-09-10T18:38:26.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:26.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[39220]: 1.2 scrub starts 2024-09-10T18:38:26.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[39220]: 1.2 scrub ok 2024-09-10T18:38:26.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[39220]: 4.12 scrub starts 2024-09-10T18:38:26.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[39220]: 4.12 scrub ok 2024-09-10T18:38:26.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[39220]: pgmap v764: 178 pgs: 178 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-10T18:38:26.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[39220]: 6.a scrub starts 2024-09-10T18:38:26.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[39220]: 6.a scrub ok 2024-09-10T18:38:26.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4286580857' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"}]: dispatch 2024-09-10T18:38:26.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:26 smithi028 ceph-mon[39220]: from='client.25708 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"}]: dispatch 2024-09-10T18:38:26.511 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:26.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:26.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:26 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:26 smithi204 ceph-mon[32357]: 1.2 scrub starts 2024-09-10T18:38:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:26 smithi204 ceph-mon[32357]: 1.2 scrub ok 2024-09-10T18:38:26.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:26 smithi204 ceph-mon[32357]: 4.12 scrub starts 2024-09-10T18:38:26.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:26 smithi204 ceph-mon[32357]: 4.12 scrub ok 2024-09-10T18:38:26.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:26 smithi204 ceph-mon[32357]: pgmap v764: 178 pgs: 178 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-10T18:38:26.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:26 smithi204 ceph-mon[32357]: 6.a scrub starts 2024-09-10T18:38:26.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:26 smithi204 ceph-mon[32357]: 6.a scrub ok 2024-09-10T18:38:26.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:26 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4286580857' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"}]: dispatch 2024-09-10T18:38:26.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:26 smithi204 ceph-mon[32357]: from='client.25708 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"}]: dispatch 2024-09-10T18:38:26.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:26.654 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:26.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:26.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:26.756 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:26.792 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:26.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:26.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:26.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:26.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:26.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:27.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:27.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:27.071 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:27.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:27.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:38:27.172 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:27.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:27.195 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:27.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:27.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:27.330 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:27.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:27.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:27.432 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:27.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[29291]: 10.1 scrub starts 2024-09-10T18:38:27.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[29291]: 3.10 scrub starts 2024-09-10T18:38:27.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[29291]: 3.10 scrub ok 2024-09-10T18:38:27.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[29291]: 6.12 scrub starts 2024-09-10T18:38:27.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[29291]: 6.12 scrub ok 2024-09-10T18:38:27.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[29291]: 5.8 scrub starts 2024-09-10T18:38:27.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[29291]: 5.8 scrub ok 2024-09-10T18:38:27.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[29291]: 1.4 scrub starts 2024-09-10T18:38:27.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[29291]: 1.4 scrub ok 2024-09-10T18:38:27.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[39220]: 10.1 scrub starts 2024-09-10T18:38:27.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[39220]: 3.10 scrub starts 2024-09-10T18:38:27.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[39220]: 3.10 scrub ok 2024-09-10T18:38:27.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[39220]: 6.12 scrub starts 2024-09-10T18:38:27.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[39220]: 6.12 scrub ok 2024-09-10T18:38:27.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[39220]: 5.8 scrub starts 2024-09-10T18:38:27.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[39220]: 5.8 scrub ok 2024-09-10T18:38:27.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[39220]: 1.4 scrub starts 2024-09-10T18:38:27.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:27 smithi028 ceph-mon[39220]: 1.4 scrub ok 2024-09-10T18:38:27.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:27.468 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:27.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:27.569 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:27 smithi204 ceph-mon[32357]: 10.1 scrub starts 2024-09-10T18:38:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:27 smithi204 ceph-mon[32357]: 3.10 scrub starts 2024-09-10T18:38:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:27 smithi204 ceph-mon[32357]: 3.10 scrub ok 2024-09-10T18:38:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:27 smithi204 ceph-mon[32357]: 6.12 scrub starts 2024-09-10T18:38:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:27 smithi204 ceph-mon[32357]: 6.12 scrub ok 2024-09-10T18:38:27.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:27 smithi204 ceph-mon[32357]: 5.8 scrub starts 2024-09-10T18:38:27.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:27 smithi204 ceph-mon[32357]: 5.8 scrub ok 2024-09-10T18:38:27.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:27 smithi204 ceph-mon[32357]: 1.4 scrub starts 2024-09-10T18:38:27.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:27 smithi204 ceph-mon[32357]: 1.4 scrub ok 2024-09-10T18:38:27.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:27.606 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:27.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:27.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:27.746 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:27.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:27.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:27.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:27.885 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:27.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:27.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:27.973 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-10T18:38:27.986 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:28.024 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:28.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:28.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:28.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:28.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:28.159 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:28.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:28.260 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:28.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:28.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:28.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:28.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:28.436 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:28.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:28.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:28.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:28 smithi028 ceph-mon[29291]: 1.d scrub starts 2024-09-10T18:38:28.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:28 smithi028 ceph-mon[29291]: 1.d scrub ok 2024-09-10T18:38:28.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:28 smithi028 ceph-mon[29291]: pgmap v765: 178 pgs: 178 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-10T18:38:28.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:28 smithi028 ceph-mon[29291]: 5.17 scrub starts 2024-09-10T18:38:28.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:28 smithi028 ceph-mon[29291]: 5.17 scrub ok 2024-09-10T18:38:28.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:28 smithi028 ceph-mon[39220]: 1.d scrub starts 2024-09-10T18:38:28.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:28 smithi028 ceph-mon[39220]: 1.d scrub ok 2024-09-10T18:38:28.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:28 smithi028 ceph-mon[39220]: pgmap v765: 178 pgs: 178 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-10T18:38:28.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:28 smithi028 ceph-mon[39220]: 5.17 scrub starts 2024-09-10T18:38:28.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:28 smithi028 ceph-mon[39220]: 5.17 scrub ok 2024-09-10T18:38:28.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:28.595 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:28.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:28 smithi204 ceph-mon[32357]: 1.d scrub starts 2024-09-10T18:38:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:28 smithi204 ceph-mon[32357]: 1.d scrub ok 2024-09-10T18:38:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:28 smithi204 ceph-mon[32357]: pgmap v765: 178 pgs: 178 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-10T18:38:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:28 smithi204 ceph-mon[32357]: 5.17 scrub starts 2024-09-10T18:38:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:28 smithi204 ceph-mon[32357]: 5.17 scrub ok 2024-09-10T18:38:28.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:28.740 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:28.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:28.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:28.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:28.880 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:28.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:28.982 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:29.019 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:29.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:29.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:29.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:29.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:29.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:29.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:29.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:29.316 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:29.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:29.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[29291]: 10.1 scrub ok 2024-09-10T18:38:29.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[29291]: 6.9 scrub starts 2024-09-10T18:38:29.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[29291]: 6.9 scrub ok 2024-09-10T18:38:29.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[29291]: 10.f scrub starts 2024-09-10T18:38:29.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[29291]: 5.9 scrub starts 2024-09-10T18:38:29.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[29291]: 5.9 scrub ok 2024-09-10T18:38:29.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[29291]: 1.a scrub starts 2024-09-10T18:38:29.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[29291]: 1.a scrub ok 2024-09-10T18:38:29.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[29291]: 5.13 scrub starts 2024-09-10T18:38:29.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[29291]: 5.13 scrub ok 2024-09-10T18:38:29.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[39220]: 10.1 scrub ok 2024-09-10T18:38:29.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[39220]: 6.9 scrub starts 2024-09-10T18:38:29.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[39220]: 6.9 scrub ok 2024-09-10T18:38:29.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[39220]: 10.f scrub starts 2024-09-10T18:38:29.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[39220]: 5.9 scrub starts 2024-09-10T18:38:29.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[39220]: 5.9 scrub ok 2024-09-10T18:38:29.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[39220]: 1.a scrub starts 2024-09-10T18:38:29.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[39220]: 1.a scrub ok 2024-09-10T18:38:29.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[39220]: 5.13 scrub starts 2024-09-10T18:38:29.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:29 smithi028 ceph-mon[39220]: 5.13 scrub ok 2024-09-10T18:38:29.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:29.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:29.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:29.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:29.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:29.599 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:29.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:29.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:29 smithi204 ceph-mon[32357]: 10.1 scrub ok 2024-09-10T18:38:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:29 smithi204 ceph-mon[32357]: 6.9 scrub starts 2024-09-10T18:38:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:29 smithi204 ceph-mon[32357]: 6.9 scrub ok 2024-09-10T18:38:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:29 smithi204 ceph-mon[32357]: 10.f scrub starts 2024-09-10T18:38:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:29 smithi204 ceph-mon[32357]: 5.9 scrub starts 2024-09-10T18:38:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:29 smithi204 ceph-mon[32357]: 5.9 scrub ok 2024-09-10T18:38:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:29 smithi204 ceph-mon[32357]: 1.a scrub starts 2024-09-10T18:38:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:29 smithi204 ceph-mon[32357]: 1.a scrub ok 2024-09-10T18:38:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:29 smithi204 ceph-mon[32357]: 5.13 scrub starts 2024-09-10T18:38:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:29 smithi204 ceph-mon[32357]: 5.13 scrub ok 2024-09-10T18:38:29.700 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:29.737 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:29.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:29.839 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:29.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:29.873 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:29.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:29.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:30.017 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:30.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:30.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:30.155 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:30.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:30.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:30.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:30.279 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:30.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:30.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:30.381 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:30.405 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:30.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:30.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:30.490 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:30 smithi028 ceph-mon[39220]: 6.0 scrub starts 2024-09-10T18:38:30.491 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:30 smithi028 ceph-mon[39220]: 6.0 scrub ok 2024-09-10T18:38:30.491 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:30 smithi028 ceph-mon[39220]: 5.a scrub starts 2024-09-10T18:38:30.491 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:30 smithi028 ceph-mon[39220]: 5.a scrub ok 2024-09-10T18:38:30.491 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:30 smithi028 ceph-mon[39220]: pgmap v766: 178 pgs: 178 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T18:38:30.491 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:30 smithi028 ceph-mon[39220]: 4.7 scrub starts 2024-09-10T18:38:30.491 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:30 smithi028 ceph-mon[39220]: 4.7 scrub ok 2024-09-10T18:38:30.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:30 smithi028 ceph-mon[29291]: 6.0 scrub starts 2024-09-10T18:38:30.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:30 smithi028 ceph-mon[29291]: 6.0 scrub ok 2024-09-10T18:38:30.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:30 smithi028 ceph-mon[29291]: 5.a scrub starts 2024-09-10T18:38:30.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:30 smithi028 ceph-mon[29291]: 5.a scrub ok 2024-09-10T18:38:30.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:30 smithi028 ceph-mon[29291]: pgmap v766: 178 pgs: 178 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T18:38:30.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:30 smithi028 ceph-mon[29291]: 4.7 scrub starts 2024-09-10T18:38:30.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:30 smithi028 ceph-mon[29291]: 4.7 scrub ok 2024-09-10T18:38:30.508 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:30.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:30.537 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:30.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:30 smithi204 ceph-mon[32357]: 6.0 scrub starts 2024-09-10T18:38:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:30 smithi204 ceph-mon[32357]: 6.0 scrub ok 2024-09-10T18:38:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:30 smithi204 ceph-mon[32357]: 5.a scrub starts 2024-09-10T18:38:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:30 smithi204 ceph-mon[32357]: 5.a scrub ok 2024-09-10T18:38:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:30 smithi204 ceph-mon[32357]: pgmap v766: 178 pgs: 178 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T18:38:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:30 smithi204 ceph-mon[32357]: 4.7 scrub starts 2024-09-10T18:38:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:30 smithi204 ceph-mon[32357]: 4.7 scrub ok 2024-09-10T18:38:30.639 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:30.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:30.678 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:30.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:30.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:30.818 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:30.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:30.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:30.920 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:30.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:30.958 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:30.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:31.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:31.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:31.103 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:31.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:31.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:31.238 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:31.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:31.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:31.340 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:31.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:31.397 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:31.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:31.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:31.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:38:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:38:31] "GET /metrics HTTP/1.1" 200 224761 "" "Prometheus/2.43.0" 2024-09-10T18:38:31.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:31 smithi028 ceph-mon[29291]: 3.15 scrub starts 2024-09-10T18:38:31.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:31 smithi028 ceph-mon[29291]: 3.15 scrub ok 2024-09-10T18:38:31.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:31 smithi028 ceph-mon[29291]: 5.11 scrub starts 2024-09-10T18:38:31.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:31 smithi028 ceph-mon[29291]: 5.11 scrub ok 2024-09-10T18:38:31.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:31 smithi028 ceph-mon[29291]: 2.1 scrub starts 2024-09-10T18:38:31.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:31 smithi028 ceph-mon[29291]: 2.1 scrub ok 2024-09-10T18:38:31.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:31 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:38:31.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:31 smithi028 ceph-mon[39220]: 3.15 scrub starts 2024-09-10T18:38:31.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:31 smithi028 ceph-mon[39220]: 3.15 scrub ok 2024-09-10T18:38:31.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:31 smithi028 ceph-mon[39220]: 5.11 scrub starts 2024-09-10T18:38:31.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:31 smithi028 ceph-mon[39220]: 5.11 scrub ok 2024-09-10T18:38:31.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:31 smithi028 ceph-mon[39220]: 2.1 scrub starts 2024-09-10T18:38:31.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:31 smithi028 ceph-mon[39220]: 2.1 scrub ok 2024-09-10T18:38:31.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:31 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:38:31.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:31.540 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:31.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:31 smithi204 ceph-mon[32357]: 3.15 scrub starts 2024-09-10T18:38:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:31 smithi204 ceph-mon[32357]: 3.15 scrub ok 2024-09-10T18:38:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:31 smithi204 ceph-mon[32357]: 5.11 scrub starts 2024-09-10T18:38:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:31 smithi204 ceph-mon[32357]: 5.11 scrub ok 2024-09-10T18:38:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:31 smithi204 ceph-mon[32357]: 2.1 scrub starts 2024-09-10T18:38:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:31 smithi204 ceph-mon[32357]: 2.1 scrub ok 2024-09-10T18:38:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:31 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:38:31.641 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:31.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:31.678 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:31.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:31.779 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:31.821 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:31.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:31.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:31.923 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:31.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:31.950 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:31.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:32.051 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:32.080 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:38:32.080 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-10T18:38:32.080 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-10T18:38:32.080 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-10T18:38:32.080 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:38:32.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:32.089 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:32.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:32.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:32.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:32.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:32.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:32.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:32.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:32.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:32.391 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[39220]: 10.f scrub ok 2024-09-10T18:38:32.391 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[39220]: 10.0 scrub starts 2024-09-10T18:38:32.391 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[39220]: pgmap v767: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 5.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T18:38:32.391 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[39220]: 1.1b scrub starts 2024-09-10T18:38:32.391 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[39220]: 1.1b scrub ok 2024-09-10T18:38:32.391 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:32.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:32.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[39220]: 4.1d scrub starts 2024-09-10T18:38:32.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[39220]: 4.1d scrub ok 2024-09-10T18:38:32.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[29291]: 10.f scrub ok 2024-09-10T18:38:32.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[29291]: 10.0 scrub starts 2024-09-10T18:38:32.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[29291]: pgmap v767: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 5.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T18:38:32.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[29291]: 1.1b scrub starts 2024-09-10T18:38:32.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[29291]: 1.1b scrub ok 2024-09-10T18:38:32.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:32.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:32.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[29291]: 4.1d scrub starts 2024-09-10T18:38:32.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:32 smithi028 ceph-mon[29291]: 4.1d scrub ok 2024-09-10T18:38:32.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:32.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:32.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:32.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:32.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:38:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:38:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:38:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:32 smithi204 ceph-mon[32357]: 10.f scrub ok 2024-09-10T18:38:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:32 smithi204 ceph-mon[32357]: 10.0 scrub starts 2024-09-10T18:38:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:32 smithi204 ceph-mon[32357]: pgmap v767: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 5.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T18:38:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:32 smithi204 ceph-mon[32357]: 1.1b scrub starts 2024-09-10T18:38:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:32 smithi204 ceph-mon[32357]: 1.1b scrub ok 2024-09-10T18:38:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:32 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:32.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:32 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:32.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:32 smithi204 ceph-mon[32357]: 4.1d scrub starts 2024-09-10T18:38:32.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:32 smithi204 ceph-mon[32357]: 4.1d scrub ok 2024-09-10T18:38:32.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:32.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:32.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:32.780 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:32.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:32.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:32.873 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:38:32.873 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:38:32.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:32.913 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:38:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:38:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:32.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:32.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:32.962 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:38:32.962 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:38:32.974 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:38:32.974 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:38:32.974 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T18:38:32.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd out 5 2024-09-10T18:38:33.003 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:38:33.004 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:38:33.035 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:38:33.035 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:38:33.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:33.067 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:38:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:38:33.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:33.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:33.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:33.107 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:38:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:38:33.141 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:38:33.142 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:38:33.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:33.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:33.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:33.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:33.384 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:33.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:33.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:33.486 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:33.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[39220]: 5.1f scrub starts 2024-09-10T18:38:33.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[39220]: 5.1f scrub ok 2024-09-10T18:38:33.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[39220]: 1.6 scrub starts 2024-09-10T18:38:33.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[39220]: 1.6 scrub ok 2024-09-10T18:38:33.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[39220]: 5.d scrub starts 2024-09-10T18:38:33.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[39220]: 5.d scrub ok 2024-09-10T18:38:33.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:33.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:33.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/92400953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:38:33.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[29291]: 5.1f scrub starts 2024-09-10T18:38:33.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[29291]: 5.1f scrub ok 2024-09-10T18:38:33.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[29291]: 1.6 scrub starts 2024-09-10T18:38:33.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[29291]: 1.6 scrub ok 2024-09-10T18:38:33.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[29291]: 5.d scrub starts 2024-09-10T18:38:33.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[29291]: 5.d scrub ok 2024-09-10T18:38:33.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:33.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:33.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:33 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/92400953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:38:33.516 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:33.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:33.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:33 smithi204 ceph-mon[32357]: 5.1f scrub starts 2024-09-10T18:38:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:33 smithi204 ceph-mon[32357]: 5.1f scrub ok 2024-09-10T18:38:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:33 smithi204 ceph-mon[32357]: 1.6 scrub starts 2024-09-10T18:38:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:33 smithi204 ceph-mon[32357]: 1.6 scrub ok 2024-09-10T18:38:33.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:33 smithi204 ceph-mon[32357]: 5.d scrub starts 2024-09-10T18:38:33.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:33 smithi204 ceph-mon[32357]: 5.d scrub ok 2024-09-10T18:38:33.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:33 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:33.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:33 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:33.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:33 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/92400953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:38:33.618 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:33.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:33.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:33.761 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:33.795 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:33.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:33.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:33.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:33.935 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:33.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:33.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-10T18:38:34.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:34.067 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:34.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:34.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:34.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:34.199 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:34.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:34.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:34.231 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:38:34.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:34.355 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:34.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:34.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:34.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:34.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:34.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[29291]: 1.5 scrub starts 2024-09-10T18:38:34.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[29291]: 1.5 scrub ok 2024-09-10T18:38:34.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[29291]: 1.9 scrub starts 2024-09-10T18:38:34.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[29291]: 1.9 scrub ok 2024-09-10T18:38:34.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:38:34.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:38:34.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[29291]: 10.0 scrub ok 2024-09-10T18:38:34.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[29291]: pgmap v768: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T18:38:34.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[29291]: 4.2 scrub starts 2024-09-10T18:38:34.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[29291]: 4.2 scrub ok 2024-09-10T18:38:34.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[29291]: osdmap e349: 8 total, 8 up, 8 in 2024-09-10T18:38:34.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:38:34.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:38:34.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:34.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[39220]: 1.5 scrub starts 2024-09-10T18:38:34.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[39220]: 1.5 scrub ok 2024-09-10T18:38:34.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[39220]: 1.9 scrub starts 2024-09-10T18:38:34.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[39220]: 1.9 scrub ok 2024-09-10T18:38:34.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:38:34.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:38:34.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[39220]: 10.0 scrub ok 2024-09-10T18:38:34.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[39220]: pgmap v768: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T18:38:34.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[39220]: 4.2 scrub starts 2024-09-10T18:38:34.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[39220]: 4.2 scrub ok 2024-09-10T18:38:34.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[39220]: osdmap e349: 8 total, 8 up, 8 in 2024-09-10T18:38:34.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:38:34.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:38:34.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:34 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:34.507 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:34 smithi204 ceph-mon[32357]: 1.5 scrub starts 2024-09-10T18:38:34.507 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:34 smithi204 ceph-mon[32357]: 1.5 scrub ok 2024-09-10T18:38:34.507 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:34 smithi204 ceph-mon[32357]: 1.9 scrub starts 2024-09-10T18:38:34.507 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:34 smithi204 ceph-mon[32357]: 1.9 scrub ok 2024-09-10T18:38:34.507 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:34 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:38:34.507 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:34 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:38:34.507 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:34 smithi204 ceph-mon[32357]: 10.0 scrub ok 2024-09-10T18:38:34.507 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:34 smithi204 ceph-mon[32357]: pgmap v768: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T18:38:34.507 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:34 smithi204 ceph-mon[32357]: 4.2 scrub starts 2024-09-10T18:38:34.507 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:34 smithi204 ceph-mon[32357]: 4.2 scrub ok 2024-09-10T18:38:34.507 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:34 smithi204 ceph-mon[32357]: osdmap e349: 8 total, 8 up, 8 in 2024-09-10T18:38:34.507 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:34 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:38:34.507 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:34 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:38:34.507 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:34 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:34.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:34.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:34.630 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:34.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:34.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:34.773 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:34.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:34.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:34.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:34.935 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:38:34.948 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:38:34.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:34.986 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:34.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:35.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:35.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:35.127 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:35.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:35.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:35.245 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:38:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:38:35.279 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:35.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:35.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:35.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:35.415 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:35.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:35.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:35.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:35 smithi028 ceph-mon[39220]: 6.1e scrub starts 2024-09-10T18:38:35.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:35 smithi028 ceph-mon[39220]: 6.1e scrub ok 2024-09-10T18:38:35.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:35 smithi028 ceph-mon[39220]: 3.7 scrub starts 2024-09-10T18:38:35.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:35 smithi028 ceph-mon[39220]: 3.7 scrub ok 2024-09-10T18:38:35.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:35 smithi028 ceph-mon[29291]: 6.1e scrub starts 2024-09-10T18:38:35.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:35 smithi028 ceph-mon[29291]: 6.1e scrub ok 2024-09-10T18:38:35.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:35 smithi028 ceph-mon[29291]: 3.7 scrub starts 2024-09-10T18:38:35.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:35 smithi028 ceph-mon[29291]: 3.7 scrub ok 2024-09-10T18:38:35.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:35.551 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:35.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:35.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:35.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:35 smithi204 ceph-mon[32357]: 6.1e scrub starts 2024-09-10T18:38:35.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:35 smithi204 ceph-mon[32357]: 6.1e scrub ok 2024-09-10T18:38:35.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:35 smithi204 ceph-mon[32357]: 3.7 scrub starts 2024-09-10T18:38:35.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:35 smithi204 ceph-mon[32357]: 3.7 scrub ok 2024-09-10T18:38:35.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:35.688 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:35.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:35.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:35.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:35.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:35.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:35.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:35.976 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:35.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:35.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:36.078 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:36.115 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:36.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:36.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:36.217 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:36.257 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:36.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:36.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:36.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:36.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:36.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:36.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:36.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:36 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:36.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:36 smithi028 ceph-mon[29291]: 1.f scrub starts 2024-09-10T18:38:36.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:36 smithi028 ceph-mon[29291]: 1.f scrub ok 2024-09-10T18:38:36.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:36 smithi028 ceph-mon[29291]: pgmap v770: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-09-10T18:38:36.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:36 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:36.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:36 smithi028 ceph-mon[39220]: 1.f scrub starts 2024-09-10T18:38:36.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:36 smithi028 ceph-mon[39220]: 1.f scrub ok 2024-09-10T18:38:36.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:36 smithi028 ceph-mon[39220]: pgmap v770: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-09-10T18:38:36.529 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:36.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:36 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:36 smithi204 ceph-mon[32357]: 1.f scrub starts 2024-09-10T18:38:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:36 smithi204 ceph-mon[32357]: 1.f scrub ok 2024-09-10T18:38:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:36 smithi204 ceph-mon[32357]: pgmap v770: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-09-10T18:38:36.632 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:36.663 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:36.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:36.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:36.767 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:36.797 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:36.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:36.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:36.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:38:36.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:36.971 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:36.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:37.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:37.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:37.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:37.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:37.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:37.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:37.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:37 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:38:37.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:37 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:38:37.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:37 smithi028 ceph-mon[29291]: osdmap e350: 8 total, 8 up, 8 in 2024-09-10T18:38:37.322 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:37 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:38:37.322 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:37 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:38:37.322 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:37 smithi028 ceph-mon[39220]: osdmap e350: 8 total, 8 up, 8 in 2024-09-10T18:38:37.362 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:37.398 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:37.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:37.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:37.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:37.547 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:37.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:37.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:37 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:38:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:37 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:38:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:37 smithi204 ceph-mon[32357]: osdmap e350: 8 total, 8 up, 8 in 2024-09-10T18:38:37.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:37.684 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:37.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:37.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:37.785 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:37.816 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:37.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:37.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:37.951 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:37.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:37.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:38.053 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:38.092 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:38.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:38.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:38.194 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:38.229 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:38.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:38.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:38.331 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:38.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:38.369 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:38.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:38.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:38.492 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:38.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:38.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:38 smithi204 ceph-mon[32357]: 10.7 scrub starts 2024-09-10T18:38:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:38 smithi204 ceph-mon[32357]: pgmap v772: 178 pgs: 178 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-10T18:38:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:38 smithi204 ceph-mon[32357]: 4.15 scrub starts 2024-09-10T18:38:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:38 smithi204 ceph-mon[32357]: 4.15 scrub ok 2024-09-10T18:38:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:38 smithi204 ceph-mon[32357]: 3.f scrub starts 2024-09-10T18:38:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:38 smithi204 ceph-mon[32357]: 3.f scrub ok 2024-09-10T18:38:38.629 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:38.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:38.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:38.731 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:38.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:38 smithi028 ceph-mon[39220]: 10.7 scrub starts 2024-09-10T18:38:38.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:38 smithi028 ceph-mon[39220]: pgmap v772: 178 pgs: 178 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-10T18:38:38.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:38 smithi028 ceph-mon[39220]: 4.15 scrub starts 2024-09-10T18:38:38.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:38 smithi028 ceph-mon[39220]: 4.15 scrub ok 2024-09-10T18:38:38.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:38 smithi028 ceph-mon[39220]: 3.f scrub starts 2024-09-10T18:38:38.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:38 smithi028 ceph-mon[39220]: 3.f scrub ok 2024-09-10T18:38:38.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:38 smithi028 ceph-mon[29291]: 10.7 scrub starts 2024-09-10T18:38:38.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:38 smithi028 ceph-mon[29291]: pgmap v772: 178 pgs: 178 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-10T18:38:38.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:38 smithi028 ceph-mon[29291]: 4.15 scrub starts 2024-09-10T18:38:38.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:38 smithi028 ceph-mon[29291]: 4.15 scrub ok 2024-09-10T18:38:38.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:38 smithi028 ceph-mon[29291]: 3.f scrub starts 2024-09-10T18:38:38.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:38 smithi028 ceph-mon[29291]: 3.f scrub ok 2024-09-10T18:38:38.765 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:38.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:38.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:38.867 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:38.887 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:38.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:38.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:38.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:39.025 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:39.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:39.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:39.127 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:39.162 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:39.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:39.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:39.264 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:39.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:39.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:39.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:39.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:39.437 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:39.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:39.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:39.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:39.585 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:39.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:39.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:39 smithi204 ceph-mon[32357]: 4.1f scrub starts 2024-09-10T18:38:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:39 smithi204 ceph-mon[32357]: 4.1f scrub ok 2024-09-10T18:38:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:39 smithi204 ceph-mon[32357]: 5.c scrub starts 2024-09-10T18:38:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:39 smithi204 ceph-mon[32357]: 5.c scrub ok 2024-09-10T18:38:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:39 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:38:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:39 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:38:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:38:39.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:39 smithi028 ceph-mon[39220]: 4.1f scrub starts 2024-09-10T18:38:39.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:39 smithi028 ceph-mon[39220]: 4.1f scrub ok 2024-09-10T18:38:39.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:39 smithi028 ceph-mon[39220]: 5.c scrub starts 2024-09-10T18:38:39.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:39 smithi028 ceph-mon[39220]: 5.c scrub ok 2024-09-10T18:38:39.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:39 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:38:39.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:39 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:38:39.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:38:39.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:39 smithi028 ceph-mon[29291]: 4.1f scrub starts 2024-09-10T18:38:39.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:39 smithi028 ceph-mon[29291]: 4.1f scrub ok 2024-09-10T18:38:39.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:39 smithi028 ceph-mon[29291]: 5.c scrub starts 2024-09-10T18:38:39.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:39 smithi028 ceph-mon[29291]: 5.c scrub ok 2024-09-10T18:38:39.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:39 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:38:39.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:39 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:38:39.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:38:39.687 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:39.752 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:39.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:39.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:39.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:39.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:39.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:39.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:39.987 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:40.023 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:40.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:40.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:40.125 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:40.157 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:40.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:40.259 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:40.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:40.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:40.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:40.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:40.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:40 smithi028 ceph-mon[39220]: 3.6 scrub starts 2024-09-10T18:38:40.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:40 smithi028 ceph-mon[39220]: 3.6 scrub ok 2024-09-10T18:38:40.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:40 smithi028 ceph-mon[39220]: 10.7 scrub ok 2024-09-10T18:38:40.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:40 smithi028 ceph-mon[39220]: pgmap v773: 178 pgs: 178 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 861 op/s 2024-09-10T18:38:40.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:40 smithi028 ceph-mon[39220]: osdmap e351: 8 total, 8 up, 8 in 2024-09-10T18:38:40.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:40 smithi028 ceph-mon[39220]: 3.11 scrub starts 2024-09-10T18:38:40.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:40 smithi028 ceph-mon[39220]: 3.11 scrub ok 2024-09-10T18:38:40.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:40 smithi028 ceph-mon[29291]: 3.6 scrub starts 2024-09-10T18:38:40.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:40 smithi028 ceph-mon[29291]: 3.6 scrub ok 2024-09-10T18:38:40.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:40 smithi028 ceph-mon[29291]: 10.7 scrub ok 2024-09-10T18:38:40.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:40 smithi028 ceph-mon[29291]: pgmap v773: 178 pgs: 178 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 861 op/s 2024-09-10T18:38:40.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:40 smithi028 ceph-mon[29291]: osdmap e351: 8 total, 8 up, 8 in 2024-09-10T18:38:40.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:40 smithi028 ceph-mon[29291]: 3.11 scrub starts 2024-09-10T18:38:40.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:40 smithi028 ceph-mon[29291]: 3.11 scrub ok 2024-09-10T18:38:40.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:40.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:40.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:40.536 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:40.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:40.571 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:40.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:40.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:40 smithi204 ceph-mon[32357]: 3.6 scrub starts 2024-09-10T18:38:40.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:40 smithi204 ceph-mon[32357]: 3.6 scrub ok 2024-09-10T18:38:40.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:40 smithi204 ceph-mon[32357]: 10.7 scrub ok 2024-09-10T18:38:40.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:40 smithi204 ceph-mon[32357]: pgmap v773: 178 pgs: 178 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 861 op/s 2024-09-10T18:38:40.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:40 smithi204 ceph-mon[32357]: osdmap e351: 8 total, 8 up, 8 in 2024-09-10T18:38:40.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:40 smithi204 ceph-mon[32357]: 3.11 scrub starts 2024-09-10T18:38:40.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:40 smithi204 ceph-mon[32357]: 3.11 scrub ok 2024-09-10T18:38:40.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:40.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:40.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:40.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:40.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:40.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:40.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:40.978 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:38:40.978 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:38:40.981 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:41.010 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:41.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:41.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:41.112 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:41.143 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:41.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:41.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:41.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:41.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:41.293 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:41.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:41.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:41.435 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:41.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:41.536 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:41.568 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:41.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:41.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:41.571 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:38:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:38:41] "GET /metrics HTTP/1.1" 200 224776 "" "Prometheus/2.43.0" 2024-09-10T18:38:41.572 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:41 smithi028 ceph-mon[39220]: 5.2 scrub starts 2024-09-10T18:38:41.572 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:41 smithi028 ceph-mon[39220]: 5.2 scrub ok 2024-09-10T18:38:41.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:41 smithi028 ceph-mon[29291]: 5.2 scrub starts 2024-09-10T18:38:41.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:41 smithi028 ceph-mon[29291]: 5.2 scrub ok 2024-09-10T18:38:41.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:41 smithi204 ceph-mon[32357]: 5.2 scrub starts 2024-09-10T18:38:41.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:41 smithi204 ceph-mon[32357]: 5.2 scrub ok 2024-09-10T18:38:41.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:41.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:41.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:41.822 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:41.858 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:41.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:41.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:41.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:41.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:41.996 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:41.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:42.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a[29287]: 2024-09-10T18:38:41.569+0000 7fed21057640 -1 mon.a@0(leader).osd e351 definitely_dead 0 2024-09-10T18:38:42.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:42.127 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:42.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:42.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:42.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:38:42.271 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:42.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:42.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:42.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T18:38:42.335 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:38:42.338 INFO:teuthology.orchestra.run.smithi028.stderr:marked out osd.5. 2024-09-10T18:38:42.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:42.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[39220]: 5.18 scrub starts 2024-09-10T18:38:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[39220]: 5.18 scrub ok 2024-09-10T18:38:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[39220]: 4.6 scrub starts 2024-09-10T18:38:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[39220]: 4.6 scrub ok 2024-09-10T18:38:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[39220]: 10.d scrub starts 2024-09-10T18:38:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[39220]: pgmap v775: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-09-10T18:38:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2603740433' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-09-10T18:38:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[39220]: from='client.35744 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-09-10T18:38:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[39220]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-10T18:38:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1333118751' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:38:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[39220]: from='client.35750 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:38:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[29291]: 5.18 scrub starts 2024-09-10T18:38:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[29291]: 5.18 scrub ok 2024-09-10T18:38:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[29291]: 4.6 scrub starts 2024-09-10T18:38:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[29291]: 4.6 scrub ok 2024-09-10T18:38:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[29291]: 10.d scrub starts 2024-09-10T18:38:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[29291]: pgmap v775: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-09-10T18:38:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2603740433' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-09-10T18:38:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[29291]: from='client.35744 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-09-10T18:38:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[29291]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-10T18:38:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1333118751' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:38:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:42 smithi028 ceph-mon[29291]: from='client.35750 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:38:42.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:42.540 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:42.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:42.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:42.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:38:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:38:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:38:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:42 smithi204 ceph-mon[32357]: 5.18 scrub starts 2024-09-10T18:38:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:42 smithi204 ceph-mon[32357]: 5.18 scrub ok 2024-09-10T18:38:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:42 smithi204 ceph-mon[32357]: 4.6 scrub starts 2024-09-10T18:38:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:42 smithi204 ceph-mon[32357]: 4.6 scrub ok 2024-09-10T18:38:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:42 smithi204 ceph-mon[32357]: 10.d scrub starts 2024-09-10T18:38:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:42 smithi204 ceph-mon[32357]: pgmap v775: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-09-10T18:38:42.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:42 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2603740433' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-09-10T18:38:42.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:42 smithi204 ceph-mon[32357]: from='client.35744 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-09-10T18:38:42.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:42 smithi204 ceph-mon[32357]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-10T18:38:42.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:42 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1333118751' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:38:42.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:42 smithi204 ceph-mon[32357]: from='client.35750 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:38:42.641 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:42.662 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:42.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:42.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:42.764 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:42.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:42.789 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:42.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:42.890 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:42.911 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:42.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:42.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:43.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:43.036 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:43.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:43.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:43.138 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:43.160 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:43.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:43.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:43.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:43.284 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:43.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:43.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:43.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:43.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:43.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[39220]: 10.e scrub starts 2024-09-10T18:38:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[39220]: 3.1f scrub starts 2024-09-10T18:38:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[39220]: 3.1f scrub ok 2024-09-10T18:38:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[39220]: 6.14 scrub starts 2024-09-10T18:38:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[39220]: 6.14 scrub ok 2024-09-10T18:38:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:38:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:38:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[39220]: from='client.35744 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-10T18:38:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[39220]: from='client.35750 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:38:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[39220]: osdmap e352: 8 total, 8 up, 7 in 2024-09-10T18:38:43.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[39220]: 10.d scrub ok 2024-09-10T18:38:43.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[39220]: 4.0 scrub starts 2024-09-10T18:38:43.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[39220]: 4.0 scrub ok 2024-09-10T18:38:43.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[39220]: osdmap e353: 8 total, 8 up, 7 in 2024-09-10T18:38:43.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[29291]: 10.e scrub starts 2024-09-10T18:38:43.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[29291]: 3.1f scrub starts 2024-09-10T18:38:43.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[29291]: 3.1f scrub ok 2024-09-10T18:38:43.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[29291]: 6.14 scrub starts 2024-09-10T18:38:43.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[29291]: 6.14 scrub ok 2024-09-10T18:38:43.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:38:43.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:38:43.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[29291]: from='client.35744 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-10T18:38:43.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[29291]: from='client.35750 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:38:43.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[29291]: osdmap e352: 8 total, 8 up, 7 in 2024-09-10T18:38:43.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[29291]: 10.d scrub ok 2024-09-10T18:38:43.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[29291]: 4.0 scrub starts 2024-09-10T18:38:43.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[29291]: 4.0 scrub ok 2024-09-10T18:38:43.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:43 smithi028 ceph-mon[29291]: osdmap e353: 8 total, 8 up, 7 in 2024-09-10T18:38:43.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:43.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:43.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:43 smithi204 ceph-mon[32357]: 10.e scrub starts 2024-09-10T18:38:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:43 smithi204 ceph-mon[32357]: 3.1f scrub starts 2024-09-10T18:38:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:43 smithi204 ceph-mon[32357]: 3.1f scrub ok 2024-09-10T18:38:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:43 smithi204 ceph-mon[32357]: 6.14 scrub starts 2024-09-10T18:38:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:43 smithi204 ceph-mon[32357]: 6.14 scrub ok 2024-09-10T18:38:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:43 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:38:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:43 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:38:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:43 smithi204 ceph-mon[32357]: from='client.35744 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-10T18:38:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:43 smithi204 ceph-mon[32357]: from='client.35750 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:38:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:43 smithi204 ceph-mon[32357]: osdmap e352: 8 total, 8 up, 7 in 2024-09-10T18:38:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:43 smithi204 ceph-mon[32357]: 10.d scrub ok 2024-09-10T18:38:43.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:43 smithi204 ceph-mon[32357]: 4.0 scrub starts 2024-09-10T18:38:43.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:43 smithi204 ceph-mon[32357]: 4.0 scrub ok 2024-09-10T18:38:43.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:43 smithi204 ceph-mon[32357]: osdmap e353: 8 total, 8 up, 7 in 2024-09-10T18:38:43.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:43.657 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:43.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:43.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:43.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:43.781 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:43.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:43.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:43.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:43.905 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:43.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:43.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:44.007 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:44.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:44.030 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:44.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:44.131 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:44.153 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:44.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:44.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:44.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:44.281 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:44.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:44.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:44.383 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:44.406 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:44.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:44.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:44.508 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:44.537 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:44.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:44.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:44 smithi204 ceph-mon[32357]: 10.e scrub ok 2024-09-10T18:38:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:44 smithi204 ceph-mon[32357]: pgmap v777: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 745 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-10T18:38:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:44 smithi204 ceph-mon[32357]: 4.10 scrub starts 2024-09-10T18:38:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:44 smithi204 ceph-mon[32357]: 4.10 scrub ok 2024-09-10T18:38:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:44 smithi204 ceph-mon[32357]: osdmap e354: 8 total, 8 up, 7 in 2024-09-10T18:38:44.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:44.682 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:44.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:44.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:44 smithi028 ceph-mon[29291]: 10.e scrub ok 2024-09-10T18:38:44.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:44 smithi028 ceph-mon[29291]: pgmap v777: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 745 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-10T18:38:44.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:44 smithi028 ceph-mon[29291]: 4.10 scrub starts 2024-09-10T18:38:44.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:44 smithi028 ceph-mon[29291]: 4.10 scrub ok 2024-09-10T18:38:44.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:44 smithi028 ceph-mon[29291]: osdmap e354: 8 total, 8 up, 7 in 2024-09-10T18:38:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:44 smithi028 ceph-mon[39220]: 10.e scrub ok 2024-09-10T18:38:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:44 smithi028 ceph-mon[39220]: pgmap v777: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 745 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-10T18:38:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:44 smithi028 ceph-mon[39220]: 4.10 scrub starts 2024-09-10T18:38:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:44 smithi028 ceph-mon[39220]: 4.10 scrub ok 2024-09-10T18:38:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:44 smithi028 ceph-mon[39220]: osdmap e354: 8 total, 8 up, 7 in 2024-09-10T18:38:44.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:44.852 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:44.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:44.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:44.954 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:44.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:44.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:45.100 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:45.138 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:45.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:45.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:45.240 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:45.272 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:45.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:45.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:45.323 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:38:45.353 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:38:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:38:45.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:45.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:45.428 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:45.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:45.529 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:45.565 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:45.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:45.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:45.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:45 smithi028 ceph-mon[39220]: 4.1 scrub starts 2024-09-10T18:38:45.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:45 smithi028 ceph-mon[39220]: 4.1 scrub ok 2024-09-10T18:38:45.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:45 smithi028 ceph-mon[39220]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 351'12839 2024-09-10T18:38:45.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:45 smithi028 ceph-mon[39220]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 351'12742 2024-09-10T18:38:45.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:45 smithi028 ceph-mon[39220]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 351'12869 2024-09-10T18:38:45.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:45 smithi028 ceph-mon[39220]: 10.3 starting backfill to osd.4 from (0'0,0'0] MAX to 351'12632 2024-09-10T18:38:45.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:45 smithi028 ceph-mon[39220]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 351'12779 2024-09-10T18:38:45.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:45 smithi028 ceph-mon[29291]: 4.1 scrub starts 2024-09-10T18:38:45.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:45 smithi028 ceph-mon[29291]: 4.1 scrub ok 2024-09-10T18:38:45.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:45 smithi028 ceph-mon[29291]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 351'12839 2024-09-10T18:38:45.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:45 smithi028 ceph-mon[29291]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 351'12742 2024-09-10T18:38:45.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:45 smithi028 ceph-mon[29291]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 351'12869 2024-09-10T18:38:45.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:45 smithi028 ceph-mon[29291]: 10.3 starting backfill to osd.4 from (0'0,0'0] MAX to 351'12632 2024-09-10T18:38:45.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:45 smithi028 ceph-mon[29291]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 351'12779 2024-09-10T18:38:45.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:45.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:45.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:45.844 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:45.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:45.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:45.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:45 smithi204 ceph-mon[32357]: 4.1 scrub starts 2024-09-10T18:38:45.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:45 smithi204 ceph-mon[32357]: 4.1 scrub ok 2024-09-10T18:38:45.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:45 smithi204 ceph-mon[32357]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 351'12839 2024-09-10T18:38:45.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:45 smithi204 ceph-mon[32357]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 351'12742 2024-09-10T18:38:45.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:45 smithi204 ceph-mon[32357]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 351'12869 2024-09-10T18:38:45.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:45 smithi204 ceph-mon[32357]: 10.3 starting backfill to osd.4 from (0'0,0'0] MAX to 351'12632 2024-09-10T18:38:45.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:45 smithi204 ceph-mon[32357]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 351'12779 2024-09-10T18:38:45.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:45.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:45.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:45.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:46.086 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:46.123 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:46.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:46.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:46.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:46.256 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:46.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:46.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:46.358 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:46.394 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:46.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:46.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:46.496 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:46.528 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:46.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:46.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:46.617 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:46 smithi028 ceph-mon[39220]: 6.1a scrub starts 2024-09-10T18:38:46.617 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:46 smithi028 ceph-mon[39220]: 6.1a scrub ok 2024-09-10T18:38:46.617 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:46 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:46.617 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:46 smithi028 ceph-mon[39220]: pgmap v780: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.27k op/s; 271 B/s, 5 objects/s recovering 2024-09-10T18:38:46.617 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:46 smithi028 ceph-mon[39220]: 5.7 scrub starts 2024-09-10T18:38:46.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:46 smithi028 ceph-mon[39220]: 5.7 scrub ok 2024-09-10T18:38:46.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:46 smithi028 ceph-mon[29291]: 6.1a scrub starts 2024-09-10T18:38:46.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:46 smithi028 ceph-mon[29291]: 6.1a scrub ok 2024-09-10T18:38:46.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:46 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:46.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:46 smithi028 ceph-mon[29291]: pgmap v780: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.27k op/s; 271 B/s, 5 objects/s recovering 2024-09-10T18:38:46.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:46 smithi028 ceph-mon[29291]: 5.7 scrub starts 2024-09-10T18:38:46.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:46 smithi028 ceph-mon[29291]: 5.7 scrub ok 2024-09-10T18:38:46.630 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:46.669 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:46.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:46.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:46.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:38:46.771 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:46.799 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:46.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:46.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:46.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:46 smithi204 ceph-mon[32357]: 6.1a scrub starts 2024-09-10T18:38:46.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:46 smithi204 ceph-mon[32357]: 6.1a scrub ok 2024-09-10T18:38:46.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:46 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:46.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:46 smithi204 ceph-mon[32357]: pgmap v780: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.27k op/s; 271 B/s, 5 objects/s recovering 2024-09-10T18:38:46.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:46 smithi204 ceph-mon[32357]: 5.7 scrub starts 2024-09-10T18:38:46.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:46 smithi204 ceph-mon[32357]: 5.7 scrub ok 2024-09-10T18:38:46.902 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:46.928 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:46.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:47.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:47.079 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:47.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:47.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:47.181 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:47.219 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:47.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:47.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:47.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:47.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:47.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:47.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:47.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:47.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:47 smithi028 ceph-mon[29291]: 5.15 scrub starts 2024-09-10T18:38:47.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:47 smithi028 ceph-mon[29291]: 5.15 scrub ok 2024-09-10T18:38:47.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:47 smithi028 ceph-mon[29291]: 4.17 scrub starts 2024-09-10T18:38:47.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:47 smithi028 ceph-mon[29291]: 4.17 scrub ok 2024-09-10T18:38:47.456 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:47 smithi028 ceph-mon[39220]: 5.15 scrub starts 2024-09-10T18:38:47.456 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:47 smithi028 ceph-mon[39220]: 5.15 scrub ok 2024-09-10T18:38:47.457 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:47 smithi028 ceph-mon[39220]: 4.17 scrub starts 2024-09-10T18:38:47.457 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:47 smithi028 ceph-mon[39220]: 4.17 scrub ok 2024-09-10T18:38:47.502 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:47.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:47.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:47.590 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:38:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:38:47.598 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:38:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:38:47.604 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:47.629 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:47.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:47.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:47.637 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:38:47.637 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:38:47.681 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:38:47.681 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:38:47.715 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:38:47.715 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:38:47.731 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:47.746 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:38:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:38:47.755 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:38:47.755 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:38:47.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.755 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15693.0:208834 10.2 10:464147d2:::benchmark_data_smithi028_161947_object71910:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:38:47.755 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:47.603082+0000", 2024-09-10T18:38:47.755 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.011928296999999999, 2024-09-10T18:38:47.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011957986, 2024-09-10T18:38:47.756 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:47.756 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:38:47.756 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:38:47.756 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15693", 2024-09-10T18:38:47.756 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:38:47.756 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 208834 2024-09-10T18:38:47.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.756 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:47.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:47.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.603082+0000", 2024-09-10T18:38:47.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:47.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:47.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.603082+0000", 2024-09-10T18:38:47.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:47.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:47.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.603081+0000", 2024-09-10T18:38:47.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:38:47.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:47.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.603087+0000", 2024-09-10T18:38:47.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3169999999999998e-06 2024-09-10T18:38:47.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:47.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.603098+0000", 2024-09-10T18:38:47.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1215e-05 2024-09-10T18:38:47.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:47.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.603105+0000", 2024-09-10T18:38:47.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3439999999999997e-06 2024-09-10T18:38:47.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:38:47.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.609544+0000", 2024-09-10T18:38:47.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064396619999999996 2024-09-10T18:38:47.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:38:47.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.609657+0000", 2024-09-10T18:38:47.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113013 2024-09-10T18:38:47.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:38:47.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.609768+0000", 2024-09-10T18:38:47.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111146 2024-09-10T18:38:47.760 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:47.760 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:47.760 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:47.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.760 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15693.0:208840 10.2 10:4640a6a4:::benchmark_data_smithi028_161947_object71916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:38:47.760 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:47.607721+0000", 2024-09-10T18:38:47.760 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0072896910000000001, 2024-09-10T18:38:47.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073996239999999996, 2024-09-10T18:38:47.760 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:47.760 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:38:47.760 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:38:47.761 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15693", 2024-09-10T18:38:47.761 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:38:47.761 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 208840 2024-09-10T18:38:47.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.761 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:47.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.761 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:47.761 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.607721+0000", 2024-09-10T18:38:47.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:47.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:47.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.607721+0000", 2024-09-10T18:38:47.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:47.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:47.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.607718+0000", 2024-09-10T18:38:47.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-10T18:38:47.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:47.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.607727+0000", 2024-09-10T18:38:47.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7350000000000002e-06 2024-09-10T18:38:47.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:47.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.607747+0000", 2024-09-10T18:38:47.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0078999999999999e-05 2024-09-10T18:38:47.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:47.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.607754+0000", 2024-09-10T18:38:47.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4590000000000001e-06 2024-09-10T18:38:47.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:38:47.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.612133+0000", 2024-09-10T18:38:47.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0043785960000000002 2024-09-10T18:38:47.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:38:47.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.612376+0000", 2024-09-10T18:38:47.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024282700000000001 2024-09-10T18:38:47.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:38:47.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.612817+0000", 2024-09-10T18:38:47.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00044087899999999998 2024-09-10T18:38:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15693.0:208842 10.2 10.f9afd462 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:38:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:47.612052+0000", 2024-09-10T18:38:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0029585480000000001, 2024-09-10T18:38:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031359579999999999, 2024-09-10T18:38:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-09-10T18:38:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:38:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15693", 2024-09-10T18:38:47.765 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:38:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 208842 2024-09-10T18:38:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.612052+0000", 2024-09-10T18:38:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.612052+0000", 2024-09-10T18:38:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.612049+0000", 2024-09-10T18:38:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-10T18:38:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.612058+0000", 2024-09-10T18:38:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4490000000000005e-06 2024-09-10T18:38:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.612117+0000", 2024-09-10T18:38:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8496999999999999e-05 2024-09-10T18:38:47.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.612130+0000", 2024-09-10T18:38:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3171e-05 2024-09-10T18:38:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15693.0:208844 10.2 10.fd664aa2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:38:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:47.612143+0000", 2024-09-10T18:38:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0028675290000000002, 2024-09-10T18:38:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030947330000000001, 2024-09-10T18:38:47.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-09-10T18:38:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:38:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15693", 2024-09-10T18:38:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:38:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 208844 2024-09-10T18:38:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.612143+0000", 2024-09-10T18:38:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.612143+0000", 2024-09-10T18:38:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.612142+0000", 2024-09-10T18:38:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:38:47.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.612147+0000", 2024-09-10T18:38:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7360000000000001e-06 2024-09-10T18:38:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.612153+0000", 2024-09-10T18:38:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4749999999999998e-06 2024-09-10T18:38:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.612157+0000", 2024-09-10T18:38:47.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.98e-06 2024-09-10T18:38:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.15693.0:208834 10.2 e354/314)", 2024-09-10T18:38:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:47.613122+0000", 2024-09-10T18:38:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0018887019999999999, 2024-09-10T18:38:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021654719999999999, 2024-09-10T18:38:47.772 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-09-10T18:38:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.613122+0000", 2024-09-10T18:38:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.613122+0000", 2024-09-10T18:38:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.613120+0000", 2024-09-10T18:38:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:38:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.613126+0000", 2024-09-10T18:38:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5350000000000004e-06 2024-09-10T18:38:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.613136+0000", 2024-09-10T18:38:47.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0426e-05 2024-09-10T18:38:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.613142+0000", 2024-09-10T18:38:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3809999999999998e-06 2024-09-10T18:38:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15693.0:208833 10.2 10:4837661a:::benchmark_data_smithi028_161947_object71909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:38:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:47.602689+0000", 2024-09-10T18:38:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.012321367, 2024-09-10T18:38:47.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012645584, 2024-09-10T18:38:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:38:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:38:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:38:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15693", 2024-09-10T18:38:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:38:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 208833 2024-09-10T18:38:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:38:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:38:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.602689+0000", 2024-09-10T18:38:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.777 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:38:47.777 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.602689+0000", 2024-09-10T18:38:47.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:38:47.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.777 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:38:47.777 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.602686+0000", 2024-09-10T18:38:47.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:38:47.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.777 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:38:47.777 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.602696+0000", 2024-09-10T18:38:47.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7680000000000003e-06 2024-09-10T18:38:47.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:38:47.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.602718+0000", 2024-09-10T18:38:47.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1443000000000001e-05 2024-09-10T18:38:47.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:38:47.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.602726+0000", 2024-09-10T18:38:47.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4470000000000006e-06 2024-09-10T18:38:47.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:38:47.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.602779+0000", 2024-09-10T18:38:47.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2604e-05 2024-09-10T18:38:47.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:38:47.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.602901+0000", 2024-09-10T18:38:47.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121863 2024-09-10T18:38:47.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:38:47.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.603009+0000", 2024-09-10T18:38:47.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010878300000000001 2024-09-10T18:38:47.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:38:47.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:38:47.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:38:47.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:47.609522+0000", 2024-09-10T18:38:47.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006512945 2024-09-10T18:38:47.780 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:47.780 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:38:47.780 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:47.780 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:38:47.780 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:38:47.780 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 6 2024-09-10T18:38:47.780 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:38:47.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:47.781 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:47.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:47.782 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:38:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:38:47.813 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:38:47.813 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:38:47.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:47 smithi204 ceph-mon[32357]: 5.15 scrub starts 2024-09-10T18:38:47.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:47 smithi204 ceph-mon[32357]: 5.15 scrub ok 2024-09-10T18:38:47.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:47 smithi204 ceph-mon[32357]: 4.17 scrub starts 2024-09-10T18:38:47.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:47 smithi204 ceph-mon[32357]: 4.17 scrub ok 2024-09-10T18:38:47.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:47.919 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:47.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:48.021 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:48.059 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:48.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:48.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:48.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:48.181 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:38:48.182 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:38:48.182 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-10T18:38:48.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd in 5 2024-09-10T18:38:48.195 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:48.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:48.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:48.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:48.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:48.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:48.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:48.472 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:48.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:48.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:48.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:48 smithi028 ceph-mon[29291]: pgmap v781: 178 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 173 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 9.7 KiB/s rd, 68 KiB/s wr, 791 op/s; 20601/202584 objects misplaced (10.169%); 4.7 MiB/s, 83 objects/s recovering 2024-09-10T18:38:48.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:48 smithi028 ceph-mon[29291]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T18:38:48.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:48 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2373574081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:38:48.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:48 smithi028 ceph-mon[39220]: pgmap v781: 178 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 173 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 9.7 KiB/s rd, 68 KiB/s wr, 791 op/s; 20601/202584 objects misplaced (10.169%); 4.7 MiB/s, 83 objects/s recovering 2024-09-10T18:38:48.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:48 smithi028 ceph-mon[39220]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T18:38:48.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:48 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2373574081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:38:48.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:48.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:48.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:48.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:48.765 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:48.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:48.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:48.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:48 smithi204 ceph-mon[32357]: pgmap v781: 178 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 173 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 9.7 KiB/s rd, 68 KiB/s wr, 791 op/s; 20601/202584 objects misplaced (10.169%); 4.7 MiB/s, 83 objects/s recovering 2024-09-10T18:38:48.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:48 smithi204 ceph-mon[32357]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T18:38:48.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:48 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2373574081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:38:48.866 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:48.887 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:48.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:48.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:48.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:49.029 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:49.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:49.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:49.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:49.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:49.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:49.274 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:49.308 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:49.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:49.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:49.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:49.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:49.452 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:49.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:49.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:49.588 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:49.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:49.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:49.731 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:49.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:49.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:49.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:49 smithi028 ceph-mon[39220]: 1.3 scrub starts 2024-09-10T18:38:49.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:49 smithi028 ceph-mon[39220]: 3.b scrub starts 2024-09-10T18:38:49.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:49 smithi028 ceph-mon[39220]: 3.b scrub ok 2024-09-10T18:38:49.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:49 smithi028 ceph-mon[39220]: pgmap v782: 178 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 173 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 8.5 KiB/s rd, 60 KiB/s wr, 575 op/s; 20601/202584 objects misplaced (10.169%); 4.1 MiB/s, 73 objects/s recovering 2024-09-10T18:38:49.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:49 smithi028 ceph-mon[29291]: 1.3 scrub starts 2024-09-10T18:38:49.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:49 smithi028 ceph-mon[29291]: 3.b scrub starts 2024-09-10T18:38:49.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:49 smithi028 ceph-mon[29291]: 3.b scrub ok 2024-09-10T18:38:49.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:49 smithi028 ceph-mon[29291]: pgmap v782: 178 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 173 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 8.5 KiB/s rd, 60 KiB/s wr, 575 op/s; 20601/202584 objects misplaced (10.169%); 4.1 MiB/s, 73 objects/s recovering 2024-09-10T18:38:49.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:49.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:49 smithi204 ceph-mon[32357]: 1.3 scrub starts 2024-09-10T18:38:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:49 smithi204 ceph-mon[32357]: 3.b scrub starts 2024-09-10T18:38:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:49 smithi204 ceph-mon[32357]: 3.b scrub ok 2024-09-10T18:38:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:49 smithi204 ceph-mon[32357]: pgmap v782: 178 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 173 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 8.5 KiB/s rd, 60 KiB/s wr, 575 op/s; 20601/202584 objects misplaced (10.169%); 4.1 MiB/s, 73 objects/s recovering 2024-09-10T18:38:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:49.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:49.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:49.975 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:50.013 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:50.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:50.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:50.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:50.151 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:50.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:50.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:50.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:50.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:50.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:50.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:50.428 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:50.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:50.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:50.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:50.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:50.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:50.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:50.693 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:50.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:50.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:50.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:50 smithi028 ceph-mon[39220]: 5.1b scrub starts 2024-09-10T18:38:50.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:50 smithi028 ceph-mon[39220]: 5.1b scrub ok 2024-09-10T18:38:50.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:50 smithi028 ceph-mon[39220]: 4.f scrub starts 2024-09-10T18:38:50.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:50 smithi028 ceph-mon[39220]: 4.f scrub ok 2024-09-10T18:38:50.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:50 smithi028 ceph-mon[39220]: 10.c scrub starts 2024-09-10T18:38:50.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:50 smithi028 ceph-mon[29291]: 5.1b scrub starts 2024-09-10T18:38:50.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:50 smithi028 ceph-mon[29291]: 5.1b scrub ok 2024-09-10T18:38:50.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:50 smithi028 ceph-mon[29291]: 4.f scrub starts 2024-09-10T18:38:50.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:50 smithi028 ceph-mon[29291]: 4.f scrub ok 2024-09-10T18:38:50.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:50 smithi028 ceph-mon[29291]: 10.c scrub starts 2024-09-10T18:38:50.795 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:50.829 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:50.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:50.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:50.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:50 smithi204 ceph-mon[32357]: 5.1b scrub starts 2024-09-10T18:38:50.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:50 smithi204 ceph-mon[32357]: 5.1b scrub ok 2024-09-10T18:38:50.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:50 smithi204 ceph-mon[32357]: 4.f scrub starts 2024-09-10T18:38:50.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:50 smithi204 ceph-mon[32357]: 4.f scrub ok 2024-09-10T18:38:50.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:50 smithi204 ceph-mon[32357]: 10.c scrub starts 2024-09-10T18:38:50.931 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:50.977 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:50.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:50.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:51.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:51.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:51.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:51.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:51.276 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:51.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:51.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:51.378 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:51.411 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:51.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:51.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:51.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:51 smithi028 ceph-mon[39220]: 3.17 scrub starts 2024-09-10T18:38:51.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:51 smithi028 ceph-mon[39220]: pgmap v783: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 172 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 7.9 KiB/s rd, 51 KiB/s wr, 1.13k op/s; 18007/187269 objects misplaced (9.616%); 12 MiB/s, 199 objects/s recovering 2024-09-10T18:38:51.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:38:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:38:51] "GET /metrics HTTP/1.1" 200 224776 "" "Prometheus/2.43.0" 2024-09-10T18:38:51.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:51 smithi028 ceph-mon[29291]: 3.17 scrub starts 2024-09-10T18:38:51.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:51 smithi028 ceph-mon[29291]: pgmap v783: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 172 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 7.9 KiB/s rd, 51 KiB/s wr, 1.13k op/s; 18007/187269 objects misplaced (9.616%); 12 MiB/s, 199 objects/s recovering 2024-09-10T18:38:51.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:51.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:51.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:51.654 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:51.683 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:51.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:51.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:51.785 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:51.818 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:51.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:51.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:51.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:51 smithi204 ceph-mon[32357]: 3.17 scrub starts 2024-09-10T18:38:51.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:51 smithi204 ceph-mon[32357]: pgmap v783: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 172 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 7.9 KiB/s rd, 51 KiB/s wr, 1.13k op/s; 18007/187269 objects misplaced (9.616%); 12 MiB/s, 199 objects/s recovering 2024-09-10T18:38:51.920 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:51.954 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:51.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:51.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:52.056 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:52.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-10T18:38:52.071 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:38:52.081 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:52.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:52.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:38:52.183 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:52.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:52.205 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:52.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:52.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:52.332 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:52.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:52.400 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:38:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:38:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:38:52.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:52.457 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:52.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:52.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:52.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:52.607 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:52.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:52.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:52.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:52 smithi028 ceph-mon[39220]: 6.b deep-scrub starts 2024-09-10T18:38:52.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:52 smithi028 ceph-mon[39220]: 6.b deep-scrub ok 2024-09-10T18:38:52.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:52 smithi028 ceph-mon[39220]: 10.c scrub ok 2024-09-10T18:38:52.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:52 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:38:52.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:52 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:38:52.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:52 smithi028 ceph-mon[39220]: 1.15 scrub starts 2024-09-10T18:38:52.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:52 smithi028 ceph-mon[39220]: 1.15 scrub ok 2024-09-10T18:38:52.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:52 smithi028 ceph-mon[29291]: 6.b deep-scrub starts 2024-09-10T18:38:52.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:52 smithi028 ceph-mon[29291]: 6.b deep-scrub ok 2024-09-10T18:38:52.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:52 smithi028 ceph-mon[29291]: 10.c scrub ok 2024-09-10T18:38:52.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:52 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:38:52.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:52 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:38:52.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:52 smithi028 ceph-mon[29291]: 1.15 scrub starts 2024-09-10T18:38:52.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:52 smithi028 ceph-mon[29291]: 1.15 scrub ok 2024-09-10T18:38:52.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:52.745 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:52.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:52.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:52.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:52 smithi204 ceph-mon[32357]: 6.b deep-scrub starts 2024-09-10T18:38:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:52 smithi204 ceph-mon[32357]: 6.b deep-scrub ok 2024-09-10T18:38:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:52 smithi204 ceph-mon[32357]: 10.c scrub ok 2024-09-10T18:38:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:52 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:38:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:52 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:38:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:52 smithi204 ceph-mon[32357]: 1.15 scrub starts 2024-09-10T18:38:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:52 smithi204 ceph-mon[32357]: 1.15 scrub ok 2024-09-10T18:38:52.886 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:52.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:52.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:52.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:53.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:53.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:53.128 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:53.165 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:53.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:53.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:53.267 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:53.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:38:53.300 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:53.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:53.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:53.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:53.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:53.435 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:53.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:53.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:53.586 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:53.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:53.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:53.688 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:53.720 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:53.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:53.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:53.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:53 smithi028 ceph-mon[39220]: osdmap e355: 8 total, 8 up, 7 in 2024-09-10T18:38:53.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:53 smithi028 ceph-mon[39220]: pgmap v785: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 172 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 7.2 KiB/s rd, 46 KiB/s wr, 1.16k op/s; 17612/183729 objects misplaced (9.586%); 11 MiB/s, 181 objects/s recovering 2024-09-10T18:38:53.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:53 smithi028 ceph-mon[29291]: osdmap e355: 8 total, 8 up, 7 in 2024-09-10T18:38:53.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:53 smithi028 ceph-mon[29291]: pgmap v785: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 172 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 7.2 KiB/s rd, 46 KiB/s wr, 1.16k op/s; 17612/183729 objects misplaced (9.586%); 11 MiB/s, 181 objects/s recovering 2024-09-10T18:38:53.821 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:53.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:53 smithi204 ceph-mon[32357]: osdmap e355: 8 total, 8 up, 7 in 2024-09-10T18:38:53.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:53 smithi204 ceph-mon[32357]: pgmap v785: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 172 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 7.2 KiB/s rd, 46 KiB/s wr, 1.16k op/s; 17612/183729 objects misplaced (9.586%); 11 MiB/s, 181 objects/s recovering 2024-09-10T18:38:53.858 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:53.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:53.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:53.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:54.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:54.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:54.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:54.102 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:54.142 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:54.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:54.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:54.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:54.267 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:54.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:54.369 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:54.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:54.407 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:54.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:54.508 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:54.532 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:54.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:54.633 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:54.673 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:54.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:54 smithi028 ceph-mon[39220]: 3.9 scrub starts 2024-09-10T18:38:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:54 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:54 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:38:54.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:54 smithi028 ceph-mon[29291]: 3.9 scrub starts 2024-09-10T18:38:54.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:54 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:54.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:54 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:38:54.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:54.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:54.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:54.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:54 smithi204 ceph-mon[32357]: 3.9 scrub starts 2024-09-10T18:38:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:54 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:38:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:54 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:38:54.919 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:54.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:54.960 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:54.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:55.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:55.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:55.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:55.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:55.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:55.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:55.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:55.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:55.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:38:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:38:55.369 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:55.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:55.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:55.471 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:55.504 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:55.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:55.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:55.606 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:55.643 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:55.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:55.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:55.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:55 smithi028 ceph-mon[29291]: 6.17 scrub starts 2024-09-10T18:38:55.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:55 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:55.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:55 smithi028 ceph-mon[29291]: pgmap v786: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 172 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 6.0 KiB/s rd, 41 KiB/s wr, 1.02k op/s; 17612/181554 objects misplaced (9.701%); 9.7 MiB/s, 156 objects/s recovering 2024-09-10T18:38:55.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:55 smithi028 ceph-mon[39220]: 6.17 scrub starts 2024-09-10T18:38:55.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:55 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:55.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:55 smithi028 ceph-mon[39220]: pgmap v786: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 172 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 6.0 KiB/s rd, 41 KiB/s wr, 1.02k op/s; 17612/181554 objects misplaced (9.701%); 9.7 MiB/s, 156 objects/s recovering 2024-09-10T18:38:55.773 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:55.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:55.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:55.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:55 smithi204 ceph-mon[32357]: 6.17 scrub starts 2024-09-10T18:38:55.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:55 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:38:55.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:55 smithi204 ceph-mon[32357]: pgmap v786: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 172 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 6.0 KiB/s rd, 41 KiB/s wr, 1.02k op/s; 17612/181554 objects misplaced (9.701%); 9.7 MiB/s, 156 objects/s recovering 2024-09-10T18:38:55.876 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:55.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:55.902 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:55.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:56.003 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:56.038 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:56.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:56.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:56.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:56.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:56.179 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:56.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:56.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:56.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:56.312 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:56.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:56.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:56.434 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:56.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:56.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:56.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:56.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:56.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:56.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:56.697 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:56.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:56.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:56.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:56 smithi028 ceph-mon[29291]: 5.0 scrub starts 2024-09-10T18:38:56.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:56 smithi028 ceph-mon[29291]: 5.0 scrub ok 2024-09-10T18:38:56.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:56 smithi028 ceph-mon[29291]: 1.13 scrub starts 2024-09-10T18:38:56.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:56 smithi028 ceph-mon[29291]: 1.13 scrub ok 2024-09-10T18:38:56.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:56 smithi028 ceph-mon[39220]: 5.0 scrub starts 2024-09-10T18:38:56.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:56 smithi028 ceph-mon[39220]: 5.0 scrub ok 2024-09-10T18:38:56.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:56 smithi028 ceph-mon[39220]: 1.13 scrub starts 2024-09-10T18:38:56.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:56 smithi028 ceph-mon[39220]: 1.13 scrub ok 2024-09-10T18:38:56.799 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:56.829 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:56.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:56.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:56 smithi204 ceph-mon[32357]: 5.0 scrub starts 2024-09-10T18:38:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:56 smithi204 ceph-mon[32357]: 5.0 scrub ok 2024-09-10T18:38:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:56 smithi204 ceph-mon[32357]: 1.13 scrub starts 2024-09-10T18:38:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:56 smithi204 ceph-mon[32357]: 1.13 scrub ok 2024-09-10T18:38:56.932 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:56.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:56.965 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:56.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:57.066 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:38:57.097 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:57.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:57.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:38:57.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:57.241 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:57.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:57.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:57.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:38:57.377 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:57.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:57.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:38:57.457 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:38:57.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:57.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:57.531 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:57.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:57.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:57.676 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:57.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:57.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:57.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:57 smithi028 ceph-mon[29291]: 4.19 scrub starts 2024-09-10T18:38:57.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:57 smithi028 ceph-mon[29291]: pgmap v787: 178 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 173 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 15012/167433 objects misplaced (8.966%); 14 MiB/s, 228 objects/s recovering 2024-09-10T18:38:57.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:57 smithi028 ceph-mon[39220]: 4.19 scrub starts 2024-09-10T18:38:57.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:57 smithi028 ceph-mon[39220]: pgmap v787: 178 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 173 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 15012/167433 objects misplaced (8.966%); 14 MiB/s, 228 objects/s recovering 2024-09-10T18:38:57.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:57.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:57.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:57.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:57 smithi204 ceph-mon[32357]: 4.19 scrub starts 2024-09-10T18:38:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:57 smithi204 ceph-mon[32357]: pgmap v787: 178 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 173 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 15012/167433 objects misplaced (8.966%); 14 MiB/s, 228 objects/s recovering 2024-09-10T18:38:57.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:57.957 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:57.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:57.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:58.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:38:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:58.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:58.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:38:58.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:58.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:58.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:58.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:58.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:38:58.397 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:58.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:58.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:38:58.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:58.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:58.542 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:58.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:58.642 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:58.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:58 smithi028 ceph-mon[29291]: 5.3 scrub starts 2024-09-10T18:38:58.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:58 smithi028 ceph-mon[29291]: 3.e scrub starts 2024-09-10T18:38:58.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:58 smithi028 ceph-mon[29291]: 3.e scrub ok 2024-09-10T18:38:58.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:58 smithi028 ceph-mon[39220]: 5.3 scrub starts 2024-09-10T18:38:58.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:58 smithi028 ceph-mon[39220]: 3.e scrub starts 2024-09-10T18:38:58.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:58 smithi028 ceph-mon[39220]: 3.e scrub ok 2024-09-10T18:38:58.682 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:58.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:58.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:58.812 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:38:58.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:58.843 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:58.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:58.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:58 smithi204 ceph-mon[32357]: 5.3 scrub starts 2024-09-10T18:38:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:58 smithi204 ceph-mon[32357]: 3.e scrub starts 2024-09-10T18:38:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:58 smithi204 ceph-mon[32357]: 3.e scrub ok 2024-09-10T18:38:58.944 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:58.977 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:58.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:58.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:59.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:38:59.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:59.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:59.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:38:59.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:59.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:59.270 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:59.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:38:59.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:59.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:59.411 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:59.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:59.512 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:38:59.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:59.548 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:38:59.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:38:59.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:59.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:59 smithi028 ceph-mon[29291]: 5.e scrub starts 2024-09-10T18:38:59.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:59 smithi028 ceph-mon[29291]: pgmap v788: 178 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 173 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 15012/167433 objects misplaced (8.966%); 14 MiB/s, 228 objects/s recovering 2024-09-10T18:38:59.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:59 smithi028 ceph-mon[29291]: 1.1c scrub starts 2024-09-10T18:38:59.657 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:59 smithi028 ceph-mon[39220]: 5.e scrub starts 2024-09-10T18:38:59.657 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:59 smithi028 ceph-mon[39220]: pgmap v788: 178 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 173 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 15012/167433 objects misplaced (8.966%); 14 MiB/s, 228 objects/s recovering 2024-09-10T18:38:59.657 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:38:59 smithi028 ceph-mon[39220]: 1.1c scrub starts 2024-09-10T18:38:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:59.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:59.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:59.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:38:59.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:59.841 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:59.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:38:59.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:59 smithi204 ceph-mon[32357]: 5.e scrub starts 2024-09-10T18:38:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:59 smithi204 ceph-mon[32357]: pgmap v788: 178 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 173 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 15012/167433 objects misplaced (8.966%); 14 MiB/s, 228 objects/s recovering 2024-09-10T18:38:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:38:59 smithi204 ceph-mon[32357]: 1.1c scrub starts 2024-09-10T18:38:59.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:38:59.981 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:38:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:38:59.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:00.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:00.138 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:00.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:00.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:00.180 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:38:59 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a[29287]: 2024-09-10T18:38:59.910+0000 7fed21057640 -1 mon.a@0(leader).osd e355 definitely_dead 0 2024-09-10T18:39:00.240 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:00.274 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:00.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:00.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:00.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:00.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:00.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:00.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:00.467 INFO:teuthology.orchestra.run.smithi028.stderr:marked in osd.5. 2024-09-10T18:39:00.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:00 smithi028 ceph-mon[29291]: 6.13 scrub starts 2024-09-10T18:39:00.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:00 smithi028 ceph-mon[29291]: 4.16 scrub starts 2024-09-10T18:39:00.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:00 smithi028 ceph-mon[29291]: 4.16 scrub ok 2024-09-10T18:39:00.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:00 smithi028 ceph-mon[29291]: 3.19 scrub starts 2024-09-10T18:39:00.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:00 smithi028 ceph-mon[29291]: 3.19 scrub ok 2024-09-10T18:39:00.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:00 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2467869835' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-09-10T18:39:00.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:00 smithi028 ceph-mon[29291]: from='client.35762 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-09-10T18:39:00.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:00 smithi028 ceph-mon[39220]: 6.13 scrub starts 2024-09-10T18:39:00.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:00 smithi028 ceph-mon[39220]: 4.16 scrub starts 2024-09-10T18:39:00.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:00 smithi028 ceph-mon[39220]: 4.16 scrub ok 2024-09-10T18:39:00.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:00 smithi028 ceph-mon[39220]: 3.19 scrub starts 2024-09-10T18:39:00.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:00 smithi028 ceph-mon[39220]: 3.19 scrub ok 2024-09-10T18:39:00.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:00 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2467869835' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-09-10T18:39:00.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:00 smithi028 ceph-mon[39220]: from='client.35762 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-09-10T18:39:00.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:00.547 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:00.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:00.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:00.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:00.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:00.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:00.796 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:00.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:00.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:00 smithi204 ceph-mon[32357]: 6.13 scrub starts 2024-09-10T18:39:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:00 smithi204 ceph-mon[32357]: 4.16 scrub starts 2024-09-10T18:39:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:00 smithi204 ceph-mon[32357]: 4.16 scrub ok 2024-09-10T18:39:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:00 smithi204 ceph-mon[32357]: 3.19 scrub starts 2024-09-10T18:39:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:00 smithi204 ceph-mon[32357]: 3.19 scrub ok 2024-09-10T18:39:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:00 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2467869835' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-09-10T18:39:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:00 smithi204 ceph-mon[32357]: from='client.35762 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-09-10T18:39:00.879 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:39:00.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:00.932 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:00.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:00.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:01.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:01.054 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:01.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:01.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:01.118 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:39:01.118 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-10T18:39:01.118 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-10T18:39:01.118 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-10T18:39:01.118 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:39:01.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:01.178 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:01.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:01.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:01.280 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:01.301 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:01.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:01.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:01.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:01.426 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:01.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:01.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:01.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:01.549 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:01.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:01.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:01.651 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:01.672 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:01.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:01.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:01.682 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:39:01] "GET /metrics HTTP/1.1" 200 224864 "" "Prometheus/2.43.0" 2024-09-10T18:39:01.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[39220]: 6.7 scrub starts 2024-09-10T18:39:01.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[39220]: 6.7 scrub ok 2024-09-10T18:39:01.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[39220]: from='client.35762 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-10T18:39:01.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[39220]: osdmap e356: 8 total, 8 up, 8 in 2024-09-10T18:39:01.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:39:01.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:39:01.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2854226028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:39:01.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[39220]: from='client.35768 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:39:01.683 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:39:01.683 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[39220]: from='client.35768 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:39:01.683 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[39220]: osdmap e357: 8 total, 8 up, 8 in 2024-09-10T18:39:01.683 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[39220]: pgmap v791: 178 pgs: 8 peering, 3 active+remapped+backfill_toofull, 167 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 9781/153165 objects misplaced (6.386%); 17 MiB/s, 265 objects/s recovering 2024-09-10T18:39:01.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[29291]: 6.7 scrub starts 2024-09-10T18:39:01.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[29291]: 6.7 scrub ok 2024-09-10T18:39:01.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[29291]: from='client.35762 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-10T18:39:01.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[29291]: osdmap e356: 8 total, 8 up, 8 in 2024-09-10T18:39:01.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:39:01.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:39:01.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2854226028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:39:01.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[29291]: from='client.35768 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:39:01.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:39:01.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[29291]: from='client.35768 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:39:01.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[29291]: osdmap e357: 8 total, 8 up, 8 in 2024-09-10T18:39:01.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:01 smithi028 ceph-mon[29291]: pgmap v791: 178 pgs: 8 peering, 3 active+remapped+backfill_toofull, 167 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 9781/153165 objects misplaced (6.386%); 17 MiB/s, 265 objects/s recovering 2024-09-10T18:39:01.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:01.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:01.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:01 smithi204 ceph-mon[32357]: 6.7 scrub starts 2024-09-10T18:39:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:01 smithi204 ceph-mon[32357]: 6.7 scrub ok 2024-09-10T18:39:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:01 smithi204 ceph-mon[32357]: from='client.35762 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-10T18:39:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:01 smithi204 ceph-mon[32357]: osdmap e356: 8 total, 8 up, 8 in 2024-09-10T18:39:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:01 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:39:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:01 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:39:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:01 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2854226028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:39:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:01 smithi204 ceph-mon[32357]: from='client.35768 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:39:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:01 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:39:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:01 smithi204 ceph-mon[32357]: from='client.35768 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:39:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:01 smithi204 ceph-mon[32357]: osdmap e357: 8 total, 8 up, 8 in 2024-09-10T18:39:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:01 smithi204 ceph-mon[32357]: pgmap v791: 178 pgs: 8 peering, 3 active+remapped+backfill_toofull, 167 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 9781/153165 objects misplaced (6.386%); 17 MiB/s, 265 objects/s recovering 2024-09-10T18:39:01.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:01.934 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:01.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:01.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:02.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:02.072 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:02.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:02.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:02.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:02.212 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:02.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:02.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:02.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:02.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:02.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:02.455 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:02.463 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:39:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:39:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:39:02.484 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:02.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:02.586 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:02.619 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:02.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:02.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:02.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:02 smithi028 ceph-mon[29291]: 1.c scrub starts 2024-09-10T18:39:02.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:02 smithi028 ceph-mon[29291]: 1.c scrub ok 2024-09-10T18:39:02.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:02 smithi028 ceph-mon[29291]: osdmap e358: 8 total, 8 up, 8 in 2024-09-10T18:39:02.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:02 smithi028 ceph-mon[39220]: 1.c scrub starts 2024-09-10T18:39:02.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:02 smithi028 ceph-mon[39220]: 1.c scrub ok 2024-09-10T18:39:02.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:02 smithi028 ceph-mon[39220]: osdmap e358: 8 total, 8 up, 8 in 2024-09-10T18:39:02.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:02.760 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:02.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:02.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:02.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:02 smithi204 ceph-mon[32357]: 1.c scrub starts 2024-09-10T18:39:02.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:02 smithi204 ceph-mon[32357]: 1.c scrub ok 2024-09-10T18:39:02.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:02 smithi204 ceph-mon[32357]: osdmap e358: 8 total, 8 up, 8 in 2024-09-10T18:39:02.862 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:02.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:02.895 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:02.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:02.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:03.034 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:03.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:03.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:03.136 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:03.167 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:03.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:03.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:03.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:03.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:03.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:03.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:03.442 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:03.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:03.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:03.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:03.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:03.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:03.691 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:03.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:03.738 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:03.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:03.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:03 smithi028 ceph-mon[39220]: pgmap v793: 178 pgs: 8 peering, 3 active+remapped+backfill_toofull, 167 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 900 op/s; 9781/151239 objects misplaced (6.467%); 9.7 MiB/s, 158 objects/s recovering 2024-09-10T18:39:03.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:03 smithi028 ceph-mon[29291]: pgmap v793: 178 pgs: 8 peering, 3 active+remapped+backfill_toofull, 167 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 900 op/s; 9781/151239 objects misplaced (6.467%); 9.7 MiB/s, 158 objects/s recovering 2024-09-10T18:39:03.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:03.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:03 smithi204 ceph-mon[32357]: pgmap v793: 178 pgs: 8 peering, 3 active+remapped+backfill_toofull, 167 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 900 op/s; 9781/151239 objects misplaced (6.467%); 9.7 MiB/s, 158 objects/s recovering 2024-09-10T18:39:03.876 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:03.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:03.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:03.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:04.023 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:04.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:04.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:04.125 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:04.161 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:04.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:04.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:04.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:04.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:04.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:04.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:04.446 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:04.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:04.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:04.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:04.585 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:04.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:04.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:04.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:04.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:04.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:04.799 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:04 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1984631819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:39:04.800 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:04 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1984631819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:39:04.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:04.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-10T18:39:04.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:04 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1984631819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:39:04.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:04.863 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:04.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:04.950 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:39:04.951 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:39:04.951 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:39:04.951 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:39:04.951 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:39:04.952 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:39:04.952 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:39:04.952 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:39:04.952 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:39:04.964 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:04.992 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:04.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:04.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:05.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:05.134 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:05.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:05.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:05.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:05.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:05.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:39:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:39:05.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:05.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:05.459 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:05.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:05.560 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:05.597 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:05.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:05.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:05.638 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-10T18:39:05.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:05.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:05.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:05.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:05.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:05 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:05.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:05 smithi204 ceph-mon[32357]: pgmap v794: 178 pgs: 8 peering, 3 active+remapped+backfill_toofull, 167 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s; 9781/149370 objects misplaced (6.548%); 9.7 MiB/s, 159 objects/s recovering 2024-09-10T18:39:05.877 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:05.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:05.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:05.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:39:05.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:05 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:05.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:05 smithi028 ceph-mon[29291]: pgmap v794: 178 pgs: 8 peering, 3 active+remapped+backfill_toofull, 167 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s; 9781/149370 objects misplaced (6.548%); 9.7 MiB/s, 159 objects/s recovering 2024-09-10T18:39:05.962 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:05 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:05.963 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:05 smithi028 ceph-mon[39220]: pgmap v794: 178 pgs: 8 peering, 3 active+remapped+backfill_toofull, 167 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s; 9781/149370 objects misplaced (6.548%); 9.7 MiB/s, 159 objects/s recovering 2024-09-10T18:39:05.979 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:06.013 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:06.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:06.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:06.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:06.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:06.165 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:06.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:06.258 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:06.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:06.294 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:06.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:06.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:06.396 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:06.432 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:06.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:06.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:06.535 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:06.571 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:06.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:06.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:06.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:06.712 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:06.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:06.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:06.814 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:06.847 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:06.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:06.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:06.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:06.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:07.090 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:07.125 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:07.126 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:07.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:07.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:07.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:07.274 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:07.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:07.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:07.376 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:07.409 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:07.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:07.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:07.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:07 smithi028 ceph-mon[29291]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-09-10T18:39:07.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:07 smithi028 ceph-mon[39220]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-09-10T18:39:07.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:07.550 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:07.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:07.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:07.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:07 smithi204 ceph-mon[32357]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-09-10T18:39:07.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:07.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:07.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:07.815 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:07.854 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:07.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:07.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:07.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:07.994 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:07.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:07.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:08.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:08.131 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:08.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:08.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:08.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:39:08.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:08.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:08.272 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:08.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:08.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:08 smithi028 ceph-mon[29291]: pgmap v795: 178 pgs: 178 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.37k op/s; 8.7 MiB/s, 145 objects/s recovering 2024-09-10T18:39:08.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:08 smithi028 ceph-mon[39220]: pgmap v795: 178 pgs: 178 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.37k op/s; 8.7 MiB/s, 145 objects/s recovering 2024-09-10T18:39:08.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:08.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:08.432 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:08.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:08.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:08.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:08.582 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:08.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:08 smithi204 ceph-mon[32357]: pgmap v795: 178 pgs: 178 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 1.37k op/s; 8.7 MiB/s, 145 objects/s recovering 2024-09-10T18:39:08.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:08.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:08.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:08.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:08.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:08.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:08.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:08.971 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:09.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:09.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:09.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:09.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:09.145 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:09.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:09.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:09.197 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:39:09.197 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:39:09.247 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:09.279 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:09.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:09.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:09.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:09.423 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:09.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:09.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:09.525 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:09.558 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:09.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:09.660 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:09.698 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:09.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:09.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:09.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:09.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:09.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:09.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:09.984 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:09.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:09.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:10.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:10.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:39:10.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:39:10.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:09 smithi204 ceph-mon[32357]: pgmap v796: 178 pgs: 178 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 619 B/s rd, 0 B/s wr, 1.05k op/s; 7.1 MiB/s, 117 objects/s recovering 2024-09-10T18:39:10.126 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:10.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:10.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:10.229 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:39:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:39:10.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:09 smithi028 ceph-mon[29291]: pgmap v796: 178 pgs: 178 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 619 B/s rd, 0 B/s wr, 1.05k op/s; 7.1 MiB/s, 117 objects/s recovering 2024-09-10T18:39:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:39:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:39:10.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:09 smithi028 ceph-mon[39220]: pgmap v796: 178 pgs: 178 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 619 B/s rd, 0 B/s wr, 1.05k op/s; 7.1 MiB/s, 117 objects/s recovering 2024-09-10T18:39:10.261 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:10.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:10.364 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:10.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-10T18:39:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:10.395 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:10.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:10.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:10.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:10.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:10.639 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:39:10.639 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:39:10.639 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4, 6, 7, 5] 2024-09-10T18:39:10.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd out 0 2024-09-10T18:39:10.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:10.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:10.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:10.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:10.870 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:10.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:10.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:10.929 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:10 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-09-10T18:39:10.929 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:10 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-09-10T18:39:10.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:10 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-09-10T18:39:10.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:10 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-09-10T18:39:10.945 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:10.973 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:10.997 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:10.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:10.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:11.099 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:11.129 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:11.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:11.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:11.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:11.269 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:11.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:11.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:11.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:10 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-09-10T18:39:11.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:10 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-09-10T18:39:11.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:11.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:11.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:11.514 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:11.544 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:11.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:11.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:11.634 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:39:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:39:11] "GET /metrics HTTP/1.1" 200 224836 "" "Prometheus/2.43.0" 2024-09-10T18:39:11.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:11.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:11.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:11.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:11.826 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:11.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:11.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:11.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:11 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-09-10T18:39:11.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:11 smithi028 ceph-mon[39220]: osdmap e359: 8 total, 8 up, 8 in 2024-09-10T18:39:11.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:11 smithi028 ceph-mon[39220]: pgmap v798: 178 pgs: 178 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.29k op/s; 76 B/s, 2 objects/s recovering 2024-09-10T18:39:11.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:11 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2134867276' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:39:11.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:11 smithi028 ceph-mon[39220]: from='client.35777 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:39:11.889 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:11 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-09-10T18:39:11.889 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:11 smithi028 ceph-mon[29291]: osdmap e359: 8 total, 8 up, 8 in 2024-09-10T18:39:11.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:11 smithi028 ceph-mon[29291]: pgmap v798: 178 pgs: 178 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.29k op/s; 76 B/s, 2 objects/s recovering 2024-09-10T18:39:11.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:11 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2134867276' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:39:11.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:11 smithi028 ceph-mon[29291]: from='client.35777 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:39:11.890 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:39:11.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:11.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:11.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:12.065 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:12.099 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:12.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:12.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:12.119 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:11 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-09-10T18:39:12.120 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:11 smithi204 ceph-mon[32357]: osdmap e359: 8 total, 8 up, 8 in 2024-09-10T18:39:12.120 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:11 smithi204 ceph-mon[32357]: pgmap v798: 178 pgs: 178 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.29k op/s; 76 B/s, 2 objects/s recovering 2024-09-10T18:39:12.120 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:11 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2134867276' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:39:12.120 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:11 smithi204 ceph-mon[32357]: from='client.35777 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:39:12.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:12.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:12.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:12.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:12.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:12.374 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:12.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:12.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:12.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:12.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:12.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:39:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:39:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:39:12.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:12.665 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:12.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:12.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:12.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:12.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:12.802 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:12.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:12.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:12.943 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:12.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:12.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:13.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:12 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:39:13.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:12 smithi028 ceph-mon[29291]: from='client.35777 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:39:13.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:12 smithi028 ceph-mon[29291]: osdmap e360: 8 total, 8 up, 8 in 2024-09-10T18:39:13.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:12 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:39:13.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:12 smithi028 ceph-mon[39220]: from='client.35777 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:39:13.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:12 smithi028 ceph-mon[39220]: osdmap e360: 8 total, 8 up, 8 in 2024-09-10T18:39:13.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:13.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:13.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:13.183 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:13.214 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:13.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:13.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:13.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:13.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:13.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:13.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:12 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:39:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:12 smithi204 ceph-mon[32357]: from='client.35777 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:39:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:12 smithi204 ceph-mon[32357]: osdmap e360: 8 total, 8 up, 8 in 2024-09-10T18:39:13.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:13.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:13.480 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:13.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:13.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:13.620 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:13.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:13.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:13.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:13.753 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:13.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:13.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:13.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:13.897 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:13.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:13.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:13.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:14.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:13 smithi028 ceph-mon[29291]: pgmap v800: 178 pgs: 178 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 88 B/s, 1 objects/s recovering 2024-09-10T18:39:14.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:13 smithi028 ceph-mon[39220]: pgmap v800: 178 pgs: 178 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 88 B/s, 1 objects/s recovering 2024-09-10T18:39:14.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:14.033 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:14.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:14.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:14.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:14.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:14.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:14.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:14.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:14.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:14.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:13 smithi204 ceph-mon[32357]: pgmap v800: 178 pgs: 178 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 88 B/s, 1 objects/s recovering 2024-09-10T18:39:14.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:14.461 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:14.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:14.562 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:14.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:14.565 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:14.597 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:14.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:14.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:14.700 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:14.735 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:14.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:14.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:14.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:14.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:14.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:14.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:15.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:15.012 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:15.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:15.113 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:15.145 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:15.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:15.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:15.247 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:15.280 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:15.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:15.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:15.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:39:14 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:39:15.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:15.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:15.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:15.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:15.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:15.551 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:15.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:15.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:15.689 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:15.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:15.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:15.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:15.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:15.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:15.931 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:15.968 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:15.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:15.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:16.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:16.115 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:16.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:16.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:16.217 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:16.248 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:16.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:16.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:16.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:16.384 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:16.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:16.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:16.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:16.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:16 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:16.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:16 smithi028 ceph-mon[39220]: pgmap v801: 178 pgs: 178 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 957 op/s; 27 B/s, 0 objects/s recovering 2024-09-10T18:39:16.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:16 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:16.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:16 smithi028 ceph-mon[29291]: pgmap v801: 178 pgs: 178 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 957 op/s; 27 B/s, 0 objects/s recovering 2024-09-10T18:39:16.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:39:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:16.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:16.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:16.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:16 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:16 smithi204 ceph-mon[32357]: pgmap v801: 178 pgs: 178 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 957 op/s; 27 B/s, 0 objects/s recovering 2024-09-10T18:39:16.623 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:16.651 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:16.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:16.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:16.753 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:16.782 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:16.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:16.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:16.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:16.941 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:16.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:16.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:17.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:17.080 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:17.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:17.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:17.182 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:17.216 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:17.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:17.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:17.318 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:17.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:17.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:17.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:17.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:17.491 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:17.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:17.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:17.593 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:17.619 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:17.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:17.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:17.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:17.757 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:17.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:17.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:17.859 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:17.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:17.890 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:17.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:17.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:18.025 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:18.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:18.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:18.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:18.151 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:18.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:18.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:18.252 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:18.288 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:18.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:18.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:18.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:18.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:18.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:18.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:18.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:18 smithi028 ceph-mon[29291]: pgmap v802: 178 pgs: 178 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s; 27 B/s, 0 objects/s recovering 2024-09-10T18:39:18.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:18 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3052604530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:39:18.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:18 smithi028 ceph-mon[39220]: pgmap v802: 178 pgs: 178 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s; 27 B/s, 0 objects/s recovering 2024-09-10T18:39:18.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:18 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3052604530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:39:18.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:18.565 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:18.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:18.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:18.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:18 smithi204 ceph-mon[32357]: pgmap v802: 178 pgs: 178 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s; 27 B/s, 0 objects/s recovering 2024-09-10T18:39:18.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:18 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3052604530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:39:18.667 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:39:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:39:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:39:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:39:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:39:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:39:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:39:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:39:18.680 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:39:18.688 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:18.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:18.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:18.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:18.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:18.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:18.916 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:18.937 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:18.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:18.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:19.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:19.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:19.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:19.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:19.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:19.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:19.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:19.359 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:19.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:19.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:19.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:19.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:19.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:19.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:19.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:19.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:19.778 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:19.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:19.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:19.880 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:19.920 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:19.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:19.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:20.022 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:20.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:20.065 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:20.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:20.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:20.217 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:20.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:20.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:20.238 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:20.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:20.366 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:20.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:20.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:20.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:20.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:20 smithi028 ceph-mon[39220]: pgmap v803: 178 pgs: 178 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 618 B/s rd, 0 B/s wr, 1.21k op/s; 26 B/s, 0 objects/s recovering 2024-09-10T18:39:20.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:20 smithi028 ceph-mon[29291]: pgmap v803: 178 pgs: 178 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 618 B/s rd, 0 B/s wr, 1.21k op/s; 26 B/s, 0 objects/s recovering 2024-09-10T18:39:20.508 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:20.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:20.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:20 smithi204 ceph-mon[32357]: pgmap v803: 178 pgs: 178 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 618 B/s rd, 0 B/s wr, 1.21k op/s; 26 B/s, 0 objects/s recovering 2024-09-10T18:39:20.609 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:20.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:20.649 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:20.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:20.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:20.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:20.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:20.897 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:20.924 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:20.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:20.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:21.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:21.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:21.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:21.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:21.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:21.274 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:21.306 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:21.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:21.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:21.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:21.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:21.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:21.549 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:21.580 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:21.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:21.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:21.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:21.685 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:39:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:39:21] "GET /metrics HTTP/1.1" 200 224836 "" "Prometheus/2.43.0" 2024-09-10T18:39:21.707 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:21.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:21.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:21.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:21.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:21.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:21.954 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:21.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:21.992 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:21.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:21.995 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:21 smithi028 ceph-mon[29291]: pgmap v804: 178 pgs: 178 active+clean; 1.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 21 B/s, 0 objects/s recovering 2024-09-10T18:39:21.995 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:21 smithi028 ceph-mon[39220]: pgmap v804: 178 pgs: 178 active+clean; 1.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 21 B/s, 0 objects/s recovering 2024-09-10T18:39:22.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:22.128 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:22.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:22.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:22.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:22.267 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:22.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:21 smithi204 ceph-mon[32357]: pgmap v804: 178 pgs: 178 active+clean; 1.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 21 B/s, 0 objects/s recovering 2024-09-10T18:39:22.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:39:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:39:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:39:22.368 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:22.403 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:22.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:22.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:22.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:22.561 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:22.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:22.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:22.607 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:39:22.608 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:39:22.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.608 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15693.0:251379 10.3 10:cae36e08:::benchmark_data_smithi028_161947_object114455:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-10T18:39:22.608 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:39:22.524419+0000", 2024-09-10T18:39:22.608 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.015005193999999999, 2024-09-10T18:39:22.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015037132, 2024-09-10T18:39:22.608 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:39:22.608 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:39:22.608 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:39:22.608 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15693", 2024-09-10T18:39:22.608 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:39:22.608 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 251379 2024-09-10T18:39:22.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.609 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:39:22.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:39:22.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.524419+0000", 2024-09-10T18:39:22.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:39:22.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:39:22.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.524419+0000", 2024-09-10T18:39:22.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:39:22.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:39:22.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.524417+0000", 2024-09-10T18:39:22.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:39:22.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:39:22.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.524425+0000", 2024-09-10T18:39:22.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4109999999999996e-06 2024-09-10T18:39:22.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:39:22.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.524440+0000", 2024-09-10T18:39:22.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4508999999999999e-05 2024-09-10T18:39:22.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:39:22.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.524445+0000", 2024-09-10T18:39:22.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.772e-06 2024-09-10T18:39:22.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:39:22.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.524514+0000", 2024-09-10T18:39:22.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8503999999999993e-05 2024-09-10T18:39:22.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:39:22.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.524661+0000", 2024-09-10T18:39:22.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014693400000000001 2024-09-10T18:39:22.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5", 2024-09-10T18:39:22.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.524861+0000", 2024-09-10T18:39:22.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019997 2024-09-10T18:39:22.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:39:22.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.532444+0000", 2024-09-10T18:39:22.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007583256 2024-09-10T18:39:22.612 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:22.612 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:22.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:22.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.613 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15693.0:251380 10.5 10:a7d1a44d:::benchmark_data_smithi028_161947_object114456:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-10T18:39:22.613 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:39:22.528314+0000", 2024-09-10T18:39:22.613 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.011110273, 2024-09-10T18:39:22.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01240837, 2024-09-10T18:39:22.613 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:39:22.613 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:39:22.613 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:39:22.613 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15693", 2024-09-10T18:39:22.613 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:39:22.613 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 251380 2024-09-10T18:39:22.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.614 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:39:22.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:39:22.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.528314+0000", 2024-09-10T18:39:22.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:39:22.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:39:22.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.528314+0000", 2024-09-10T18:39:22.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:39:22.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:39:22.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.528311+0000", 2024-09-10T18:39:22.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-10T18:39:22.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:39:22.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.528321+0000", 2024-09-10T18:39:22.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9350000000000006e-06 2024-09-10T18:39:22.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:39:22.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.528339+0000", 2024-09-10T18:39:22.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8136999999999999e-05 2024-09-10T18:39:22.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:39:22.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.528347+0000", 2024-09-10T18:39:22.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0499999999999992e-06 2024-09-10T18:39:22.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:39:22.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.528435+0000", 2024-09-10T18:39:22.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8367000000000003e-05 2024-09-10T18:39:22.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:39:22.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.528563+0000", 2024-09-10T18:39:22.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012744399999999999 2024-09-10T18:39:22.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,3", 2024-09-10T18:39:22.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.528683+0000", 2024-09-10T18:39:22.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000119697 2024-09-10T18:39:22.617 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:22.617 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:22.617 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:22.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.617 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15693.0:251387 10.5 10:afb0f585:::benchmark_data_smithi028_161947_object114463:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-10T18:39:22.617 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:39:22.529531+0000", 2024-09-10T18:39:22.617 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0098937399999999998, 2024-09-10T18:39:22.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011270835, 2024-09-10T18:39:22.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:39:22.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:39:22.618 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:39:22.618 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15693", 2024-09-10T18:39:22.618 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:39:22.618 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 251387 2024-09-10T18:39:22.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.618 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:39:22.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:39:22.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529531+0000", 2024-09-10T18:39:22.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:39:22.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:39:22.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529531+0000", 2024-09-10T18:39:22.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:39:22.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:39:22.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529528+0000", 2024-09-10T18:39:22.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:39:22.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:39:22.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529536+0000", 2024-09-10T18:39:22.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2300000000000002e-06 2024-09-10T18:39:22.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:39:22.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529547+0000", 2024-09-10T18:39:22.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1209999999999999e-05 2024-09-10T18:39:22.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:39:22.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529553+0000", 2024-09-10T18:39:22.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2060000000000004e-06 2024-09-10T18:39:22.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:39:22.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.532326+0000", 2024-09-10T18:39:22.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027726840000000001 2024-09-10T18:39:22.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:39:22.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.532447+0000", 2024-09-10T18:39:22.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012174499999999999 2024-09-10T18:39:22.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,3", 2024-09-10T18:39:22.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.532563+0000", 2024-09-10T18:39:22.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011596500000000001 2024-09-10T18:39:22.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:22.621 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:22.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:22.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.621 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15693.0:251388 10.3 10:cfb3aff2:::benchmark_data_smithi028_161947_object114464:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-10T18:39:22.621 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:39:22.529575+0000", 2024-09-10T18:39:22.622 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0098493720000000003, 2024-09-10T18:39:22.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011313963, 2024-09-10T18:39:22.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:39:22.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:39:22.622 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:39:22.622 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15693", 2024-09-10T18:39:22.622 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:39:22.622 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 251388 2024-09-10T18:39:22.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.622 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:39:22.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:39:22.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529575+0000", 2024-09-10T18:39:22.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:39:22.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:39:22.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529575+0000", 2024-09-10T18:39:22.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:39:22.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:39:22.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529574+0000", 2024-09-10T18:39:22.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:39:22.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:39:22.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529579+0000", 2024-09-10T18:39:22.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2920000000000003e-06 2024-09-10T18:39:22.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:39:22.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529588+0000", 2024-09-10T18:39:22.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6070000000000008e-06 2024-09-10T18:39:22.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:39:22.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529593+0000", 2024-09-10T18:39:22.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1100000000000002e-06 2024-09-10T18:39:22.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:39:22.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.532470+0000", 2024-09-10T18:39:22.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028771299999999999 2024-09-10T18:39:22.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:39:22.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.532586+0000", 2024-09-10T18:39:22.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116311 2024-09-10T18:39:22.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5", 2024-09-10T18:39:22.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.532715+0000", 2024-09-10T18:39:22.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012814900000000001 2024-09-10T18:39:22.626 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:22.626 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:22.626 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:22.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.626 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15693.0:251390 10.3 10:c62f4c7d:::benchmark_data_smithi028_161947_object114466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-10T18:39:22.627 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:39:22.529602+0000", 2024-09-10T18:39:22.627 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.009822664, 2024-09-10T18:39:22.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011367098000000001, 2024-09-10T18:39:22.627 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:39:22.627 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:39:22.627 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:39:22.627 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15693", 2024-09-10T18:39:22.627 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:39:22.627 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 251390 2024-09-10T18:39:22.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.627 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:39:22.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:39:22.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529602+0000", 2024-09-10T18:39:22.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:39:22.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:39:22.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529602+0000", 2024-09-10T18:39:22.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:39:22.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:39:22.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529601+0000", 2024-09-10T18:39:22.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:39:22.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:39:22.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529605+0000", 2024-09-10T18:39:22.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1670000000000001e-06 2024-09-10T18:39:22.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:39:22.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529612+0000", 2024-09-10T18:39:22.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1940000000000001e-06 2024-09-10T18:39:22.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:39:22.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.529617+0000", 2024-09-10T18:39:22.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.566e-06 2024-09-10T18:39:22.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:39:22.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.536142+0000", 2024-09-10T18:39:22.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0065254650000000003 2024-09-10T18:39:22.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:39:22.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.536265+0000", 2024-09-10T18:39:22.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122588 2024-09-10T18:39:22.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5", 2024-09-10T18:39:22.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.536385+0000", 2024-09-10T18:39:22.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011993999999999999 2024-09-10T18:39:22.631 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:22.631 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:22.632 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:22.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.632 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15693.0:251392 10.5 10:aa71db6f:::benchmark_data_smithi028_161947_object114468:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-10T18:39:22.632 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:39:22.532917+0000", 2024-09-10T18:39:22.632 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0065077720000000002, 2024-09-10T18:39:22.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0081258460000000008, 2024-09-10T18:39:22.632 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:39:22.632 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-09-10T18:39:22.632 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:39:22.632 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15693", 2024-09-10T18:39:22.632 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:39:22.632 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 251392 2024-09-10T18:39:22.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.633 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:39:22.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:39:22.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.532917+0000", 2024-09-10T18:39:22.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:39:22.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:39:22.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.532917+0000", 2024-09-10T18:39:22.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:39:22.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:39:22.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.532914+0000", 2024-09-10T18:39:22.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:39:22.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:39:22.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.532922+0000", 2024-09-10T18:39:22.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5190000000000003e-06 2024-09-10T18:39:22.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:39:22.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.532933+0000", 2024-09-10T18:39:22.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1134e-05 2024-09-10T18:39:22.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:39:22.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.532938+0000", 2024-09-10T18:39:22.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7450000000000001e-06 2024-09-10T18:39:22.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:39:22.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.541014+0000", 2024-09-10T18:39:22.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080755170000000008 2024-09-10T18:39:22.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:22.635 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:22.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:22.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.636 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:251397 10.9 e360/357 10:927647d7:::benchmark_data_smithi028_161947_object114473:head v 360'15825, mlcod=360'15823)", 2024-09-10T18:39:22.636 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:39:22.540736+0000", 2024-09-10T18:39:22.636 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4294967295.9986887, 2024-09-10T18:39:22.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00036627600000000001, 2024-09-10T18:39:22.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:39:22.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:39:22.636 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:39:22.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:39:22.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.540736+0000", 2024-09-10T18:39:22.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:39:22.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:39:22.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.540736+0000", 2024-09-10T18:39:22.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:39:22.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:39:22.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.540734+0000", 2024-09-10T18:39:22.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:39:22.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:39:22.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.540745+0000", 2024-09-10T18:39:22.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1103e-05 2024-09-10T18:39:22.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:39:22.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.540756+0000", 2024-09-10T18:39:22.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0495e-05 2024-09-10T18:39:22.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:39:22.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.540761+0000", 2024-09-10T18:39:22.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6949999999999997e-06 2024-09-10T18:39:22.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:39:22.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.540817+0000", 2024-09-10T18:39:22.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5674999999999998e-05 2024-09-10T18:39:22.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:22.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:22.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:39:22.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:22.540838+0000", 2024-09-10T18:39:22.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0599e-05 2024-09-10T18:39:22.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:22.639 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:22.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:22.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:22.639 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:39:22.639 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 7 2024-09-10T18:39:22.639 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:39:22.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:22.707 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:22.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:22.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:22.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:22.853 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:22.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:22.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:39:22.954 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:22.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:23.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:23.132 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:23.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:23.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:23.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:23.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:23.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:23.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:23.386 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:23.419 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:23.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:23.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:23.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:23.559 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:23.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:23.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:23.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:23.707 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:23.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:23.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:23.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:23.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:23.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:23.999 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:24.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:24.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:24.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:24.132 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:24.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:marked out osd.0. 2024-09-10T18:39:24.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:24.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-10T18:39:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:24.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:24.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:24.413 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:24.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:24.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:24.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:24 smithi028 ceph-mon[39220]: pgmap v805: 178 pgs: 178 active+clean; 1.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 908 B/s rd, 0 B/s wr, 1.14k op/s; 19 B/s, 0 objects/s recovering 2024-09-10T18:39:24.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:39:24.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:24 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2808005685' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-09-10T18:39:24.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:24 smithi028 ceph-mon[39220]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-10T18:39:24.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:24 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a[29287]: 2024-09-10T18:39:24.090+0000 7fed21057640 -1 mon.a@0(leader).osd e360 definitely_dead 0 2024-09-10T18:39:24.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:24 smithi028 ceph-mon[29291]: pgmap v805: 178 pgs: 178 active+clean; 1.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 908 B/s rd, 0 B/s wr, 1.14k op/s; 19 B/s, 0 objects/s recovering 2024-09-10T18:39:24.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:39:24.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:24 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2808005685' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-09-10T18:39:24.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:24 smithi028 ceph-mon[29291]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-10T18:39:24.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:24.539 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:24.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:24.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:24 smithi204 ceph-mon[32357]: pgmap v805: 178 pgs: 178 active+clean; 1.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 908 B/s rd, 0 B/s wr, 1.14k op/s; 19 B/s, 0 objects/s recovering 2024-09-10T18:39:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:39:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:24 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2808005685' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-09-10T18:39:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:24 smithi204 ceph-mon[32357]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-10T18:39:24.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:24.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:24.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:24.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:24.766 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:24.789 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:24.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:24.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:24.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:24.922 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:24.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:24.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:25.024 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:25.050 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:25.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:25.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:25.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:25.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:25.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:25.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:39:24 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:39:25.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:25.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:25.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:25.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:25 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2808005685' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-10T18:39:25.359 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:25 smithi028 ceph-mon[29291]: osdmap e361: 8 total, 8 up, 7 in 2024-09-10T18:39:25.359 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:25 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2808005685' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-10T18:39:25.359 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:25 smithi028 ceph-mon[39220]: osdmap e361: 8 total, 8 up, 7 in 2024-09-10T18:39:25.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:25.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:25.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:25.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:25.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:25.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:25.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:25 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2808005685' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-10T18:39:25.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:25 smithi204 ceph-mon[32357]: osdmap e361: 8 total, 8 up, 7 in 2024-09-10T18:39:25.652 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:25.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:25.678 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:25.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:25.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:25.803 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:25.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:25.904 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:25.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:25.929 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:25.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:26.030 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:26.055 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:26.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:26.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:26.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:26.180 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:26.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:26.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:26.225 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:39:26.282 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:26.311 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:26.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:26.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:26.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:26.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:26.443 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:26.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:26.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:26 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:26.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:26 smithi028 ceph-mon[39220]: pgmap v807: 178 pgs: 19 peering, 159 active+clean; 1.4 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T18:39:26.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:26 smithi028 ceph-mon[39220]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-10T18:39:26.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:26 smithi028 ceph-mon[39220]: osdmap e362: 8 total, 8 up, 7 in 2024-09-10T18:39:26.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:26 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1008337467' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:39:26.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:26 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:26.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:26 smithi028 ceph-mon[29291]: pgmap v807: 178 pgs: 19 peering, 159 active+clean; 1.4 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T18:39:26.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:26 smithi028 ceph-mon[29291]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-10T18:39:26.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:26 smithi028 ceph-mon[29291]: osdmap e362: 8 total, 8 up, 7 in 2024-09-10T18:39:26.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:26 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1008337467' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:39:26.544 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:26.582 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:26.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:26.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:26 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:26 smithi204 ceph-mon[32357]: pgmap v807: 178 pgs: 19 peering, 159 active+clean; 1.4 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T18:39:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:26 smithi204 ceph-mon[32357]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-10T18:39:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:26 smithi204 ceph-mon[32357]: osdmap e362: 8 total, 8 up, 7 in 2024-09-10T18:39:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:26 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1008337467' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:39:26.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:26.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:26.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:26.825 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:26.857 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:26.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:26.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:26.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:26.988 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:26.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:26.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:27.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:27.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:27.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:27.214 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:27.250 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:27.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:27.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:27.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:27.387 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:27.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:27.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:27.489 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:39:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[29291]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 360'15685 2024-09-10T18:39:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1008337467' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:39:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[29291]: osdmap e363: 8 total, 8 up, 7 in 2024-09-10T18:39:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[29291]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 360'15832 2024-09-10T18:39:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[29291]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 360'15750 2024-09-10T18:39:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[29291]: 10.5 starting backfill to osd.7 from (0'0,0'0] MAX to 360'15898 2024-09-10T18:39:27.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[29291]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 360'15790 2024-09-10T18:39:27.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[29291]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 360'15886 2024-09-10T18:39:27.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[29291]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 360'16203 2024-09-10T18:39:27.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[29291]: 10.a starting backfill to osd.2 from (344'9617,355'13942] MIN to 360'15787 2024-09-10T18:39:27.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[29291]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 360'15690 2024-09-10T18:39:27.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[29291]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 360'15825 2024-09-10T18:39:27.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:39:27.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[39220]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 360'15685 2024-09-10T18:39:27.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1008337467' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:39:27.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[39220]: osdmap e363: 8 total, 8 up, 7 in 2024-09-10T18:39:27.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[39220]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 360'15832 2024-09-10T18:39:27.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[39220]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 360'15750 2024-09-10T18:39:27.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[39220]: 10.5 starting backfill to osd.7 from (0'0,0'0] MAX to 360'15898 2024-09-10T18:39:27.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[39220]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 360'15790 2024-09-10T18:39:27.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[39220]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 360'15886 2024-09-10T18:39:27.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[39220]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 360'16203 2024-09-10T18:39:27.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[39220]: 10.a starting backfill to osd.2 from (344'9617,355'13942] MIN to 360'15787 2024-09-10T18:39:27.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[39220]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 360'15690 2024-09-10T18:39:27.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:27 smithi028 ceph-mon[39220]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 360'15825 2024-09-10T18:39:27.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:27.527 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:27.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:27 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:39:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:27 smithi204 ceph-mon[32357]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 360'15685 2024-09-10T18:39:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:27 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1008337467' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:39:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:27 smithi204 ceph-mon[32357]: osdmap e363: 8 total, 8 up, 7 in 2024-09-10T18:39:27.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:27 smithi204 ceph-mon[32357]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 360'15832 2024-09-10T18:39:27.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:27 smithi204 ceph-mon[32357]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 360'15750 2024-09-10T18:39:27.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:27 smithi204 ceph-mon[32357]: 10.5 starting backfill to osd.7 from (0'0,0'0] MAX to 360'15898 2024-09-10T18:39:27.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:27 smithi204 ceph-mon[32357]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 360'15790 2024-09-10T18:39:27.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:27 smithi204 ceph-mon[32357]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 360'15886 2024-09-10T18:39:27.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:27 smithi204 ceph-mon[32357]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 360'16203 2024-09-10T18:39:27.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:27 smithi204 ceph-mon[32357]: 10.a starting backfill to osd.2 from (344'9617,355'13942] MIN to 360'15787 2024-09-10T18:39:27.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:27 smithi204 ceph-mon[32357]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 360'15690 2024-09-10T18:39:27.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:27 smithi204 ceph-mon[32357]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 360'15825 2024-09-10T18:39:27.629 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:27.664 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:27.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:27.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:27.766 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:27.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:27.811 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:27.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:27.912 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:27.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:27.954 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:27.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:28.055 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:28.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:28.092 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:28.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:28.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:28.231 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:28.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:28.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:28.333 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:28.375 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:28.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:28.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:28.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:28.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:28 smithi028 ceph-mon[39220]: pgmap v810: 178 pgs: 6 remapped+peering, 43 peering, 129 active+clean; 1.2 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 170 B/s rd, 0 B/s wr, 828 op/s; 18 B/s, 1 objects/s recovering 2024-09-10T18:39:28.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:28 smithi028 ceph-mon[39220]: 3.d scrub starts 2024-09-10T18:39:28.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:28 smithi028 ceph-mon[29291]: pgmap v810: 178 pgs: 6 remapped+peering, 43 peering, 129 active+clean; 1.2 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 170 B/s rd, 0 B/s wr, 828 op/s; 18 B/s, 1 objects/s recovering 2024-09-10T18:39:28.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:28 smithi028 ceph-mon[29291]: 3.d scrub starts 2024-09-10T18:39:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:28.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:28.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:28 smithi204 ceph-mon[32357]: pgmap v810: 178 pgs: 6 remapped+peering, 43 peering, 129 active+clean; 1.2 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 170 B/s rd, 0 B/s wr, 828 op/s; 18 B/s, 1 objects/s recovering 2024-09-10T18:39:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:28 smithi204 ceph-mon[32357]: 3.d scrub starts 2024-09-10T18:39:28.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:28.667 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:28.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:28.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:28.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:28.806 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:28.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:28.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:28.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:28.957 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:28.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:29.058 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:29.092 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:29.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:29.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:29.194 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:29.225 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:29.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:29.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:29.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:29.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:29.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:29.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:29.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:29.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:29.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:29.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:29.656 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:29.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:29.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:29.802 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:29.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:29.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:29.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:29.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:29.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:30.047 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:30.087 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:30.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:30.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:30.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:30.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:30.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:30.335 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:30.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:30.377 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:30.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:30.383 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:39:30.383 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:39:30.383 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-10T18:39:30.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T18:39:30.478 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:30.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:30 smithi028 ceph-mon[39220]: pgmap v811: 178 pgs: 6 remapped+peering, 43 peering, 129 active+clean; 1.2 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 170 B/s rd, 0 B/s wr, 705 op/s; 18 B/s, 1 objects/s recovering 2024-09-10T18:39:30.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:30 smithi028 ceph-mon[29291]: pgmap v811: 178 pgs: 6 remapped+peering, 43 peering, 129 active+clean; 1.2 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 170 B/s rd, 0 B/s wr, 705 op/s; 18 B/s, 1 objects/s recovering 2024-09-10T18:39:30.511 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:30.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:30.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:30.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:30 smithi204 ceph-mon[32357]: pgmap v811: 178 pgs: 6 remapped+peering, 43 peering, 129 active+clean; 1.2 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 170 B/s rd, 0 B/s wr, 705 op/s; 18 B/s, 1 objects/s recovering 2024-09-10T18:39:30.613 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:30.648 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:30.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:30.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:30.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:30.804 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:30.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:30.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:30.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:30.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:30.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:31.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:31.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:31.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:31.190 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:31.231 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:31.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:31.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:31.333 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:31.371 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:31.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:31.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:31.473 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:31.494 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:31.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:31.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:31.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:31 smithi028 ceph-mon[29291]: Health check update: Reduced data availability: 7 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-09-10T18:39:31.537 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:39:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:39:31] "GET /metrics HTTP/1.1" 200 224780 "" "Prometheus/2.43.0" 2024-09-10T18:39:31.537 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:31 smithi028 ceph-mon[39220]: Health check update: Reduced data availability: 7 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-09-10T18:39:31.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:31.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:31 smithi204 ceph-mon[32357]: Health check update: Reduced data availability: 7 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-09-10T18:39:31.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:31.632 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:31.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:31.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:31.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:31.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:31.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:39:31.872 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:31.903 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:31.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:31.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:32.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:32.052 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:32.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:32.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:32.154 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:32.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:32.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:32.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:32.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:32.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:32.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:32.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:32.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:32.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:32.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:32.524 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:32 smithi028 ceph-mon[39220]: pgmap v812: 178 pgs: 1 active+clean+remapped, 7 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 168 active+clean; 1.0 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 886 B/s rd, 0 B/s wr, 1.16k op/s; 5112/50049 objects misplaced (10.214%); 49 MiB/s, 794 objects/s recovering 2024-09-10T18:39:32.524 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:32 smithi028 ceph-mon[39220]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T18:39:32.524 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:32 smithi028 ceph-mon[39220]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 9 pgs peering) 2024-09-10T18:39:32.524 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:32 smithi028 ceph-mon[39220]: osdmap e364: 8 total, 8 up, 7 in 2024-09-10T18:39:32.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:32 smithi028 ceph-mon[29291]: pgmap v812: 178 pgs: 1 active+clean+remapped, 7 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 168 active+clean; 1.0 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 886 B/s rd, 0 B/s wr, 1.16k op/s; 5112/50049 objects misplaced (10.214%); 49 MiB/s, 794 objects/s recovering 2024-09-10T18:39:32.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:32 smithi028 ceph-mon[29291]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T18:39:32.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:32 smithi028 ceph-mon[29291]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 9 pgs peering) 2024-09-10T18:39:32.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:32 smithi028 ceph-mon[29291]: osdmap e364: 8 total, 8 up, 7 in 2024-09-10T18:39:32.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:32.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:32.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:32.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:39:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:39:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:39:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:32 smithi204 ceph-mon[32357]: pgmap v812: 178 pgs: 1 active+clean+remapped, 7 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 168 active+clean; 1.0 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 886 B/s rd, 0 B/s wr, 1.16k op/s; 5112/50049 objects misplaced (10.214%); 49 MiB/s, 794 objects/s recovering 2024-09-10T18:39:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:32 smithi204 ceph-mon[32357]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T18:39:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:32 smithi204 ceph-mon[32357]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 9 pgs peering) 2024-09-10T18:39:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:32 smithi204 ceph-mon[32357]: osdmap e364: 8 total, 8 up, 7 in 2024-09-10T18:39:32.678 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:32.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:32.699 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:32.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:32.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:32.823 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:32.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:32.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:32.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:32.952 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:32.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:33.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:33.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:33.141 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:39:33.141 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:39:33.141 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:39:33.141 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:39:33.142 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:39:33.142 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:39:33.142 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:39:33.142 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:39:33.142 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:39:33.142 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:39:33.142 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:39:33.143 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:39:33.143 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:39:33.143 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:39:33.143 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:39:33.143 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:39:33.178 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:33.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:33.202 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:33.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:33.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:33.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:33.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:33.428 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:33.450 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:33.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:33.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:33.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:33 smithi028 ceph-mon[39220]: 5.4 scrub starts 2024-09-10T18:39:33.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:33 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3159095888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:39:33.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:33 smithi028 ceph-mon[39220]: osdmap e365: 8 total, 8 up, 7 in 2024-09-10T18:39:33.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:33 smithi028 ceph-mon[29291]: 5.4 scrub starts 2024-09-10T18:39:33.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:33 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3159095888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:39:33.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:33 smithi028 ceph-mon[29291]: osdmap e365: 8 total, 8 up, 7 in 2024-09-10T18:39:33.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:33.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:33.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:33 smithi204 ceph-mon[32357]: 5.4 scrub starts 2024-09-10T18:39:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:33 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3159095888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:39:33.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:33 smithi204 ceph-mon[32357]: osdmap e365: 8 total, 8 up, 7 in 2024-09-10T18:39:33.678 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:33.703 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:33.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:33.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:33.792 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:33.805 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:33.830 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:33.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:33.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:33.932 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:33.973 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:33.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:33.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:34.018 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:39:34.018 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:39:34.075 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:34.112 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:34.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:34.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:34.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:34.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:34.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:34.356 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:34.395 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:34.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:34.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:34.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:34.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:34 smithi028 ceph-mon[39220]: 1.3 scrub starts 2024-09-10T18:39:34.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:34 smithi028 ceph-mon[39220]: 1.3 scrub ok 2024-09-10T18:39:34.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:34 smithi028 ceph-mon[39220]: pgmap v815: 178 pgs: 1 active+clean+remapped, 7 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 168 active+clean; 1.0 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 738 B/s rd, 0 B/s wr, 551 op/s; 5112/50049 objects misplaced (10.214%); 49 MiB/s, 793 objects/s recovering 2024-09-10T18:39:34.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:34 smithi028 ceph-mon[39220]: osdmap e366: 8 total, 8 up, 7 in 2024-09-10T18:39:34.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:34 smithi028 ceph-mon[29291]: 1.3 scrub starts 2024-09-10T18:39:34.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:34 smithi028 ceph-mon[29291]: 1.3 scrub ok 2024-09-10T18:39:34.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:34 smithi028 ceph-mon[29291]: pgmap v815: 178 pgs: 1 active+clean+remapped, 7 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 168 active+clean; 1.0 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 738 B/s rd, 0 B/s wr, 551 op/s; 5112/50049 objects misplaced (10.214%); 49 MiB/s, 793 objects/s recovering 2024-09-10T18:39:34.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:34 smithi028 ceph-mon[29291]: osdmap e366: 8 total, 8 up, 7 in 2024-09-10T18:39:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:34.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:34.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:34.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:34 smithi204 ceph-mon[32357]: 1.3 scrub starts 2024-09-10T18:39:34.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:34 smithi204 ceph-mon[32357]: 1.3 scrub ok 2024-09-10T18:39:34.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:34 smithi204 ceph-mon[32357]: pgmap v815: 178 pgs: 1 active+clean+remapped, 7 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 168 active+clean; 1.0 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 738 B/s rd, 0 B/s wr, 551 op/s; 5112/50049 objects misplaced (10.214%); 49 MiB/s, 793 objects/s recovering 2024-09-10T18:39:34.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:34 smithi204 ceph-mon[32357]: osdmap e366: 8 total, 8 up, 7 in 2024-09-10T18:39:34.651 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:34.687 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:34.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:34.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:34.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:34.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:34.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:34.925 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:34.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:34.960 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:34.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:35.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-10T18:39:35.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:35.119 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:35.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:35.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:35.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:35.264 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:39:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:39:35.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:35 smithi204 ceph-mon[32357]: 4.19 scrub starts 2024-09-10T18:39:35.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:35 smithi204 ceph-mon[32357]: 4.19 scrub ok 2024-09-10T18:39:35.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:35 smithi204 ceph-mon[32357]: 4.3 scrub starts 2024-09-10T18:39:35.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:35 smithi204 ceph-mon[32357]: 4.3 scrub ok 2024-09-10T18:39:35.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:35 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:39:35.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:35 smithi204 ceph-mon[32357]: 5.3 scrub starts 2024-09-10T18:39:35.266 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:35 smithi204 ceph-mon[32357]: 5.3 scrub ok 2024-09-10T18:39:35.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:35.271 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:35.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:35.372 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:35.409 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:35.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:35.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:35.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:35 smithi028 ceph-mon[29291]: 4.19 scrub starts 2024-09-10T18:39:35.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:35 smithi028 ceph-mon[29291]: 4.19 scrub ok 2024-09-10T18:39:35.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:35 smithi028 ceph-mon[29291]: 4.3 scrub starts 2024-09-10T18:39:35.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:35 smithi028 ceph-mon[29291]: 4.3 scrub ok 2024-09-10T18:39:35.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:35 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:39:35.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:35 smithi028 ceph-mon[29291]: 5.3 scrub starts 2024-09-10T18:39:35.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:35 smithi028 ceph-mon[29291]: 5.3 scrub ok 2024-09-10T18:39:35.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:35 smithi028 ceph-mon[39220]: 4.19 scrub starts 2024-09-10T18:39:35.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:35 smithi028 ceph-mon[39220]: 4.19 scrub ok 2024-09-10T18:39:35.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:35 smithi028 ceph-mon[39220]: 4.3 scrub starts 2024-09-10T18:39:35.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:35 smithi028 ceph-mon[39220]: 4.3 scrub ok 2024-09-10T18:39:35.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:35 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:39:35.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:35 smithi028 ceph-mon[39220]: 5.3 scrub starts 2024-09-10T18:39:35.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:35 smithi028 ceph-mon[39220]: 5.3 scrub ok 2024-09-10T18:39:35.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:35.546 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:35.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:35.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:35.648 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:35.682 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:35.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:35.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:35.784 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:35.820 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:35.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:35.921 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:35.952 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:35.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:36.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:36.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:36.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:36.194 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:36.224 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:36.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:36.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:36.326 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:36.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:36.366 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:36.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:36.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:36.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:36.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:36.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:36 smithi204 ceph-mon[32357]: 6.17 scrub starts 2024-09-10T18:39:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:36 smithi204 ceph-mon[32357]: 6.17 scrub ok 2024-09-10T18:39:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:36 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:36 smithi204 ceph-mon[32357]: 6.f scrub starts 2024-09-10T18:39:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:36 smithi204 ceph-mon[32357]: 6.f scrub ok 2024-09-10T18:39:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:36 smithi204 ceph-mon[32357]: pgmap v817: 178 pgs: 1 active+clean+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 992 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 739 op/s; 3995/48189 objects misplaced (8.290%); 67 MiB/s, 1.09k objects/s recovering 2024-09-10T18:39:36.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:36 smithi204 ceph-mon[32357]: 1.1c scrub starts 2024-09-10T18:39:36.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:36 smithi204 ceph-mon[32357]: 1.1c scrub ok 2024-09-10T18:39:36.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:36 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:39:36.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:36 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:39:36.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:36 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:39:36.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:36 smithi204 ceph-mon[32357]: 6.3 scrub starts 2024-09-10T18:39:36.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:36 smithi204 ceph-mon[32357]: 6.3 scrub ok 2024-09-10T18:39:36.605 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:36.638 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:36.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:36.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:36.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[29291]: 6.17 scrub starts 2024-09-10T18:39:36.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[29291]: 6.17 scrub ok 2024-09-10T18:39:36.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:36.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[29291]: 6.f scrub starts 2024-09-10T18:39:36.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[29291]: 6.f scrub ok 2024-09-10T18:39:36.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[29291]: pgmap v817: 178 pgs: 1 active+clean+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 992 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 739 op/s; 3995/48189 objects misplaced (8.290%); 67 MiB/s, 1.09k objects/s recovering 2024-09-10T18:39:36.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[29291]: 1.1c scrub starts 2024-09-10T18:39:36.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[29291]: 1.1c scrub ok 2024-09-10T18:39:36.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:39:36.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:39:36.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:39:36.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[29291]: 6.3 scrub starts 2024-09-10T18:39:36.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[29291]: 6.3 scrub ok 2024-09-10T18:39:36.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[39220]: 6.17 scrub starts 2024-09-10T18:39:36.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[39220]: 6.17 scrub ok 2024-09-10T18:39:36.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:36.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[39220]: 6.f scrub starts 2024-09-10T18:39:36.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[39220]: 6.f scrub ok 2024-09-10T18:39:36.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[39220]: pgmap v817: 178 pgs: 1 active+clean+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 992 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 739 op/s; 3995/48189 objects misplaced (8.290%); 67 MiB/s, 1.09k objects/s recovering 2024-09-10T18:39:36.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[39220]: 1.1c scrub starts 2024-09-10T18:39:36.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[39220]: 1.1c scrub ok 2024-09-10T18:39:36.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:39:36.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:39:36.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:39:36.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[39220]: 6.3 scrub starts 2024-09-10T18:39:36.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:36 smithi028 ceph-mon[39220]: 6.3 scrub ok 2024-09-10T18:39:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:36.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:36.781 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:36.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:36.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:36.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:36.921 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:36.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:36.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:37.023 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:37.062 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:37.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:37.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:37.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:37.198 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:37.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:37.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:37.300 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:37.335 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:37.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:37.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:37.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:37.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:37.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:37.576 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:37 smithi204 ceph-mon[32357]: 1.1e scrub starts 2024-09-10T18:39:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:37 smithi204 ceph-mon[32357]: 1.1e scrub ok 2024-09-10T18:39:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:37 smithi204 ceph-mon[32357]: 4.8 scrub starts 2024-09-10T18:39:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:37 smithi204 ceph-mon[32357]: 4.8 scrub ok 2024-09-10T18:39:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:37 smithi204 ceph-mon[32357]: 1.14 scrub starts 2024-09-10T18:39:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:37 smithi204 ceph-mon[32357]: 1.14 scrub ok 2024-09-10T18:39:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:37 smithi204 ceph-mon[32357]: 3.a scrub starts 2024-09-10T18:39:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:37 smithi204 ceph-mon[32357]: 3.a scrub ok 2024-09-10T18:39:37.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:37.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:37.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:37.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:37.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:37.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:37 smithi028 ceph-mon[29291]: 1.1e scrub starts 2024-09-10T18:39:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:37 smithi028 ceph-mon[29291]: 1.1e scrub ok 2024-09-10T18:39:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:37 smithi028 ceph-mon[29291]: 4.8 scrub starts 2024-09-10T18:39:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:37 smithi028 ceph-mon[29291]: 4.8 scrub ok 2024-09-10T18:39:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:37 smithi028 ceph-mon[29291]: 1.14 scrub starts 2024-09-10T18:39:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:37 smithi028 ceph-mon[29291]: 1.14 scrub ok 2024-09-10T18:39:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:37 smithi028 ceph-mon[29291]: 3.a scrub starts 2024-09-10T18:39:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:37 smithi028 ceph-mon[29291]: 3.a scrub ok 2024-09-10T18:39:37.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:37 smithi028 ceph-mon[39220]: 1.1e scrub starts 2024-09-10T18:39:37.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:37 smithi028 ceph-mon[39220]: 1.1e scrub ok 2024-09-10T18:39:37.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:37 smithi028 ceph-mon[39220]: 4.8 scrub starts 2024-09-10T18:39:37.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:37 smithi028 ceph-mon[39220]: 4.8 scrub ok 2024-09-10T18:39:37.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:37 smithi028 ceph-mon[39220]: 1.14 scrub starts 2024-09-10T18:39:37.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:37 smithi028 ceph-mon[39220]: 1.14 scrub ok 2024-09-10T18:39:37.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:37 smithi028 ceph-mon[39220]: 3.a scrub starts 2024-09-10T18:39:37.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:37 smithi028 ceph-mon[39220]: 3.a scrub ok 2024-09-10T18:39:37.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:37.896 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:37.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:37.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:37.998 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:38.033 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:38.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:38.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:38.135 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:38.170 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:38.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:38.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:38.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:38.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:38.314 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:38.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:38.415 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:38.456 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:38.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:38.557 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:38.599 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:38.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:38.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:38 smithi204 ceph-mon[32357]: 3.9 scrub starts 2024-09-10T18:39:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:38 smithi204 ceph-mon[32357]: 3.9 scrub ok 2024-09-10T18:39:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:38 smithi204 ceph-mon[32357]: pgmap v818: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 798 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 620 op/s; 1646/38892 objects misplaced (4.232%); 29 MiB/s, 470 objects/s recovering 2024-09-10T18:39:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:38 smithi204 ceph-mon[32357]: 1.7 scrub starts 2024-09-10T18:39:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:38 smithi204 ceph-mon[32357]: 1.7 scrub ok 2024-09-10T18:39:38.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:39:38.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:38.737 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:38.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:38.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:38.739 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:38 smithi028 ceph-mon[39220]: 3.9 scrub starts 2024-09-10T18:39:38.739 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:38 smithi028 ceph-mon[39220]: 3.9 scrub ok 2024-09-10T18:39:38.739 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:38 smithi028 ceph-mon[39220]: pgmap v818: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 798 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 620 op/s; 1646/38892 objects misplaced (4.232%); 29 MiB/s, 470 objects/s recovering 2024-09-10T18:39:38.739 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:38 smithi028 ceph-mon[39220]: 1.7 scrub starts 2024-09-10T18:39:38.739 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:38 smithi028 ceph-mon[39220]: 1.7 scrub ok 2024-09-10T18:39:38.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:38 smithi028 ceph-mon[29291]: 3.9 scrub starts 2024-09-10T18:39:38.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:38 smithi028 ceph-mon[29291]: 3.9 scrub ok 2024-09-10T18:39:38.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:38 smithi028 ceph-mon[29291]: pgmap v818: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 798 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 620 op/s; 1646/38892 objects misplaced (4.232%); 29 MiB/s, 470 objects/s recovering 2024-09-10T18:39:38.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:38 smithi028 ceph-mon[29291]: 1.7 scrub starts 2024-09-10T18:39:38.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:38 smithi028 ceph-mon[29291]: 1.7 scrub ok 2024-09-10T18:39:38.839 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:38.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:38.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:38.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:38.975 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:39.011 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:39.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:39.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:39.112 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:39.148 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:39.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:39.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:39.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:39.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:39.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:39.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:39.440 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:39.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:39.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:39.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:39.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:39.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:39 smithi204 ceph-mon[32357]: 1.19 scrub starts 2024-09-10T18:39:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:39 smithi204 ceph-mon[32357]: 1.19 scrub ok 2024-09-10T18:39:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:39 smithi204 ceph-mon[32357]: 6.18 scrub starts 2024-09-10T18:39:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:39 smithi204 ceph-mon[32357]: 6.18 scrub ok 2024-09-10T18:39:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:39 smithi204 ceph-mon[32357]: 3.16 scrub starts 2024-09-10T18:39:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:39 smithi204 ceph-mon[32357]: 3.16 scrub ok 2024-09-10T18:39:39.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:39 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:39:39.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:39 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:39:39.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:39 smithi204 ceph-mon[32357]: osdmap e367: 8 total, 8 up, 7 in 2024-09-10T18:39:39.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:39 smithi204 ceph-mon[32357]: 4.a scrub starts 2024-09-10T18:39:39.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:39 smithi204 ceph-mon[32357]: 4.a scrub ok 2024-09-10T18:39:39.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:39 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:39:39.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:39:39.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:39 smithi204 ceph-mon[32357]: 4.1e scrub starts 2024-09-10T18:39:39.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:39 smithi204 ceph-mon[32357]: 4.1e scrub ok 2024-09-10T18:39:39.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[39220]: 1.19 scrub starts 2024-09-10T18:39:39.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[39220]: 1.19 scrub ok 2024-09-10T18:39:39.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[39220]: 6.18 scrub starts 2024-09-10T18:39:39.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[39220]: 6.18 scrub ok 2024-09-10T18:39:39.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[39220]: 3.16 scrub starts 2024-09-10T18:39:39.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[39220]: 3.16 scrub ok 2024-09-10T18:39:39.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:39:39.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:39:39.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[39220]: osdmap e367: 8 total, 8 up, 7 in 2024-09-10T18:39:39.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[39220]: 4.a scrub starts 2024-09-10T18:39:39.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[39220]: 4.a scrub ok 2024-09-10T18:39:39.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:39:39.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:39:39.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[39220]: 4.1e scrub starts 2024-09-10T18:39:39.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[39220]: 4.1e scrub ok 2024-09-10T18:39:39.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[29291]: 1.19 scrub starts 2024-09-10T18:39:39.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[29291]: 1.19 scrub ok 2024-09-10T18:39:39.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[29291]: 6.18 scrub starts 2024-09-10T18:39:39.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[29291]: 6.18 scrub ok 2024-09-10T18:39:39.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[29291]: 3.16 scrub starts 2024-09-10T18:39:39.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[29291]: 3.16 scrub ok 2024-09-10T18:39:39.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:39:39.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:39:39.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[29291]: osdmap e367: 8 total, 8 up, 7 in 2024-09-10T18:39:39.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[29291]: 4.a scrub starts 2024-09-10T18:39:39.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[29291]: 4.a scrub ok 2024-09-10T18:39:39.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:39:39.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:39:39.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[29291]: 4.1e scrub starts 2024-09-10T18:39:39.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:39 smithi028 ceph-mon[29291]: 4.1e scrub ok 2024-09-10T18:39:39.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:39.720 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:39.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:39.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:39.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:39.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:39.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:39.973 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:40.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:40.014 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:40.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:40.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:40.148 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:40.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:40.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:40.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:40.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:40.320 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:39:40.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:40.435 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:40.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:40.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:40.540 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:40.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:40.580 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:40.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:40.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:40 smithi028 ceph-mon[39220]: 6.13 scrub starts 2024-09-10T18:39:40.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:40 smithi028 ceph-mon[39220]: 6.13 scrub ok 2024-09-10T18:39:40.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:40 smithi028 ceph-mon[39220]: pgmap v820: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 798 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 819 B/s rd, 0 B/s wr, 595 op/s; 1646/38892 objects misplaced (4.232%); 28 MiB/s, 452 objects/s recovering 2024-09-10T18:39:40.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:40 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:39:40.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:40 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:39:40.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:40 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/952043086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:39:40.601 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:40 smithi028 ceph-mon[29291]: 6.13 scrub starts 2024-09-10T18:39:40.601 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:40 smithi028 ceph-mon[29291]: 6.13 scrub ok 2024-09-10T18:39:40.601 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:40 smithi028 ceph-mon[29291]: pgmap v820: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 798 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 819 B/s rd, 0 B/s wr, 595 op/s; 1646/38892 objects misplaced (4.232%); 28 MiB/s, 452 objects/s recovering 2024-09-10T18:39:40.601 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:40 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:39:40.601 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:40 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:39:40.601 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:40 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/952043086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:39:40.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:40 smithi204 ceph-mon[32357]: 6.13 scrub starts 2024-09-10T18:39:40.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:40 smithi204 ceph-mon[32357]: 6.13 scrub ok 2024-09-10T18:39:40.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:40 smithi204 ceph-mon[32357]: pgmap v820: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 798 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 819 B/s rd, 0 B/s wr, 595 op/s; 1646/38892 objects misplaced (4.232%); 28 MiB/s, 452 objects/s recovering 2024-09-10T18:39:40.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:40 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:39:40.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:40 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:39:40.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:40 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/952043086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:39:40.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:40.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:40.716 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:40.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:40.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:40.853 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:40.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:40.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:40.937 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:39:40.937 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:39:40.937 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 368, 2024-09-10T18:39:40.937 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "5eababb2-6fa0-11ef-bcea-c7b262605968", 2024-09-10T18:39:40.937 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-10T18:16:20.680373+0000", 2024-09-10T18:39:40.937 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-10T18:39:40.300739+0000", 2024-09-10T18:39:40.937 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-09-10T18:26:26.611338+0000", 2024-09-10T18:39:40.938 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-09-10T18:39:24.201588+0000", 2024-09-10T18:39:40.938 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T18:39:40.938 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5799936, 2024-09-10T18:39:40.938 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-09-10T18:39:40.938 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-09-10T18:39:40.938 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-09-10T18:39:40.938 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-09-10T18:39:40.938 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-09-10T18:39:40.938 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:39:40.938 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 19, 2024-09-10T18:39:40.938 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T18:39:40.938 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T18:39:40.938 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T18:39:40.939 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-09-10T18:39:40.939 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 10, 2024-09-10T18:39:40.939 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-09-10T18:39:40.939 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-09-10T18:39:40.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-09-10T18:39:40.939 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "quincy", 2024-09-10T18:39:40.939 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-09-10T18:39:40.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.939 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-09-10T18:39:40.939 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-09-10T18:39:40.939 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:19:24.013536+0000", 2024-09-10T18:39:40.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:39:40.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:39:40.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:39:40.940 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:39:40.940 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:39:40.940 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:39:40.940 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:39:40.940 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:39:40.940 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:39:40.940 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:39:40.940 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:39:40.940 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:39:40.940 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 31, 2024-09-10T18:39:40.940 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 31, 2024-09-10T18:39:40.940 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 31, 2024-09-10T18:39:40.941 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 31, 2024-09-10T18:39:40.941 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 31, 2024-09-10T18:39:40.941 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:39:40.941 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:39:40.941 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:39:40.941 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:39:40.941 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:39:40.941 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:39:40.942 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:39:40.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.942 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "300", 2024-09-10T18:39:40.942 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:39:40.942 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:39:40.942 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "287", 2024-09-10T18:39:40.943 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:39:40.943 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:39:40.943 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:39:40.943 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:39:40.943 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:39:40.943 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:39:40.944 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:39:40.945 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:39:40.945 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:39:40.945 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:39:40.945 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:39:40.945 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:39:40.948 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:39:40.948 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:39:40.948 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:39:40.948 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:39:40.948 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:39:40.948 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:39:40.948 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:39:40.949 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:39:40.949 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:39:40.949 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:39:40.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:39:40.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.949 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:39:40.949 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:39:40.949 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:39:40.949 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:39:40.950 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:39:40.950 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:39:40.950 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:39:40.950 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:39:40.950 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:39:40.950 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:39:40.950 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:39:40.950 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-10T18:39:40.950 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-09-10T18:39:40.950 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-09-10T18:39:40.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.950 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:39:40.950 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-09-10T18:39:40.951 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.951 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-09-10T18:39:40.951 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-09-10T18:39:40.951 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:30.065209+0000", 2024-09-10T18:39:40.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-09-10T18:39:40.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T18:39:40.951 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:39:40.951 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:39:40.951 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:39:40.951 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:39:40.951 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:39:40.952 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:39:40.952 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:39:40.952 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:39:40.952 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:39:40.952 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:39:40.952 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-09-10T18:39:40.952 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-09-10T18:39:40.952 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-09-10T18:39:40.952 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-09-10T18:39:40.953 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-09-10T18:39:40.953 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:39:40.953 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:39:40.953 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:39:40.953 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:39:40.953 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:39:40.953 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:39:40.953 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:39:40.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.953 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "62", 2024-09-10T18:39:40.954 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:39:40.954 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:39:40.954 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:39:40.954 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:39:40.954 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:39:40.954 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 3, 2024-09-10T18:39:40.954 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 62, 2024-09-10T18:39:40.954 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:39:40.954 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:39:40.954 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:39:40.954 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:39:40.954 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:39:40.955 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:39:40.955 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:39:40.955 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:39:40.955 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:39:40.955 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:39:40.955 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:39:40.955 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:39:40.955 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:39:40.955 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:39:40.955 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:39:40.955 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:39:40.955 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:39:40.955 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:39:40.956 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:39:40.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.956 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:39:40.956 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:39:40.956 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:39:40.956 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:39:40.956 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:39:40.956 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:39:40.956 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:39:40.956 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:39:40.956 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:39:40.956 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:39:40.956 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:39:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:39:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:39:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-09-10T18:39:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-09-10T18:39:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-09-10T18:39:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:30.506441+0000", 2024-09-10T18:39:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:39:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:39:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:39:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:39:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:39:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:39:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:39:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:39:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:39:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:39:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:39:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:39:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:39:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:39:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:39:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:39:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:39:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:39:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:39:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:39:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:39:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:39:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:39:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:39:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "59", 2024-09-10T18:39:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:39:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:39:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:39:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:39:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:39:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:39:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:39:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:39:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:39:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:39:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:39:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:39:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:39:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:39:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:39:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:39:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:39:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:39:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:39:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:39:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:39:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:39:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:39:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:39:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:39:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:39:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:39:40.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:39:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:39:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:39:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:39:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:39:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:39:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:39:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:39:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:39:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:39:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:39:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:39:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:39:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-09-10T18:39:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-09-10T18:39:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:32.008130+0000", 2024-09-10T18:39:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:39:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:39:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:39:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:39:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:39:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:39:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:39:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:39:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:39:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:39:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:39:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:39:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:39:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:39:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:39:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:39:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:39:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:39:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:39:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:39:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:39:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:39:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:39:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:39:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "61", 2024-09-10T18:39:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:39:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:39:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:39:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:39:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:39:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:39:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:39:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:39:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:39:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:39:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:39:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:39:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:39:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:39:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:39:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:39:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:39:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:39:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:39:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:39:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:39:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:39:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:39:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:39:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:39:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:39:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:39:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:39:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:39:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:39:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:39:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:39:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:39:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:39:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:39:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:39:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:39:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:39:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:39:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:39:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-09-10T18:39:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-09-10T18:39:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:34.060571+0000", 2024-09-10T18:39:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:39:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:39:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:39:40.971 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:39:40.971 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:39:40.971 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:39:40.971 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:39:40.971 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:39:40.971 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:39:40.971 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:39:40.971 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:39:40.971 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:39:40.971 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:39:40.971 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:39:40.971 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:39:40.971 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:39:40.971 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:39:40.972 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:39:40.972 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:39:40.972 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:39:40.972 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:39:40.972 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:39:40.972 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:39:40.972 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:39:40.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.972 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "63", 2024-09-10T18:39:40.972 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:39:40.972 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:39:40.972 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:39:40.972 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:39:40.973 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:39:40.973 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:39:40.973 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:39:40.973 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:39:40.973 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:39:40.973 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:39:40.973 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:39:40.973 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:39:40.973 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:39:40.973 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:39:40.973 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:39:40.973 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:39:40.973 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:39:40.973 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:39:40.974 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:39:40.974 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:39:40.974 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:39:40.974 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:39:40.974 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:39:40.974 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:39:40.974 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:39:40.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:39:40.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.974 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:39:40.974 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:39:40.974 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:39:40.974 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:39:40.975 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:39:40.975 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:39:40.975 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:39:40.975 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:39:40.975 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:39:40.975 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:39:40.975 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:39:40.975 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:39:40.975 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:39:40.975 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:39:40.975 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.975 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-09-10T18:39:40.976 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-09-10T18:39:40.976 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:36.127330+0000", 2024-09-10T18:39:40.976 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:39:40.976 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:39:40.976 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:39:40.976 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:39:40.976 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:39:40.976 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:39:40.976 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:39:40.976 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:39:40.976 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:39:40.976 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:39:40.977 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:39:40.977 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:39:40.977 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:39:40.977 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:39:40.977 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:39:40.977 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:39:40.977 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:39:40.977 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:39:40.977 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:39:40.977 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:39:40.977 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:39:40.977 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:39:40.977 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:39:40.977 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:39:40.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.978 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "65", 2024-09-10T18:39:40.978 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:39:40.978 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:39:40.978 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:39:40.978 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:39:40.978 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:39:40.978 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:39:40.978 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:39:40.978 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:39:40.978 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:39:40.978 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:39:40.978 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:39:40.978 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:39:40.979 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:39:40.979 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:39:40.979 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:39:40.979 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:39:40.979 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:39:40.979 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:39:40.979 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:39:40.979 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:39:40.979 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:39:40.979 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:39:40.979 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:39:40.979 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:39:40.979 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:39:40.980 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:39:40.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.980 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:39:40.980 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:39:40.980 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:39:40.980 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:39:40.980 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:39:40.980 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:39:40.980 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:39:40.980 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:39:40.980 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:39:40.980 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:39:40.980 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:39:40.980 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-10T18:39:40.981 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-09-10T18:39:40.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.981 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:39:40.981 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:39:40.981 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.981 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 10, 2024-09-10T18:39:40.981 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "unique_pool_3", 2024-09-10T18:39:40.981 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:36:06.616074+0000", 2024-09-10T18:39:40.981 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:39:40.981 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:39:40.982 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:39:40.982 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:39:40.982 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:39:40.982 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:39:40.982 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:39:40.982 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:39:40.982 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:39:40.982 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:39:40.982 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:39:40.982 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:39:40.982 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 16, 2024-09-10T18:39:40.982 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 16, 2024-09-10T18:39:40.982 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 16, 2024-09-10T18:39:40.982 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 16, 2024-09-10T18:39:40.983 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 16, 2024-09-10T18:39:40.983 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:39:40.983 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:39:40.983 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:39:40.983 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:39:40.983 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:39:40.983 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:39:40.983 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:39:40.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.983 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "319", 2024-09-10T18:39:40.983 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:39:40.983 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:39:40.984 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:39:40.984 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:39:40.984 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:39:40.984 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:39:40.984 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:39:40.984 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:39:40.984 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:39:40.984 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:39:40.984 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:39:40.984 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:39:40.984 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:39:40.984 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:39:40.985 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:39:40.985 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:39:40.985 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:39:40.985 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:39:40.985 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:39:40.985 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:39:40.985 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:39:40.985 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:39:40.985 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:39:40.985 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:39:40.985 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:39:40.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:39:40.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.986 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:39:40.986 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:39:40.986 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:39:40.986 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:39:40.986 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:39:40.986 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:39:40.986 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:39:40.986 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:39:40.986 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:39:40.986 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:39:40.986 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:39:40.986 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:39:40.986 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:39:40.987 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-09-10T18:39:40.987 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.987 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.987 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:39:40.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:39:40.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-10T18:39:40.987 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "f1c834d9-252a-4077-b25e-169181d70251", 2024-09-10T18:39:40.987 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:39:40.987 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 0, 2024-09-10T18:39:40.987 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-10T18:39:40.987 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:39:40.987 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:39:40.988 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:39:40.988 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-09-10T18:39:40.988 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 356, 2024-09-10T18:39:40.988 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:39:40.988 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:39:40.988 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:39:40.988 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:40.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:40.988 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-09-10T18:39:40.988 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:39:40.988 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.989 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:40.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.989 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:39:40.989 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:40.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:40.989 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-09-10T18:39:40.989 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:39:40.989 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.989 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:40.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.990 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:39:40.990 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:40.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.990 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:40.990 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-09-10T18:39:40.990 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:39:40.990 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.990 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:40.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.990 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:39:40.990 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:40.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.990 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:40.991 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-09-10T18:39:40.991 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:39:40.991 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.991 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:40.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.991 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/3188372351", 2024-09-10T18:39:40.991 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/3188372351", 2024-09-10T18:39:40.991 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/3188372351", 2024-09-10T18:39:40.991 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/3188372351", 2024-09-10T18:39:40.991 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:39:40.991 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:39:40.991 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:39:40.992 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:40.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.992 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-10T18:39:40.992 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "81f1855d-b946-4ca3-b246-38977b9e20fd", 2024-09-10T18:39:40.992 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:39:40.992 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:39:40.992 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:39:40.992 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:39:40.992 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:39:40.992 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:39:40.992 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-10T18:39:40.993 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 364, 2024-09-10T18:39:40.993 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:39:40.993 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:39:40.993 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:39:40.993 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:40.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.993 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:40.993 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-09-10T18:39:40.993 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:39:40.993 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.993 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:40.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.994 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:39:40.994 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:40.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:40.994 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-09-10T18:39:40.994 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:39:40.994 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.994 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:40.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.994 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:39:40.994 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:40.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:40.995 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-09-10T18:39:40.995 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:39:40.995 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.995 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:40.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.995 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:39:40.995 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:40.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.995 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:40.995 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-09-10T18:39:40.995 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:39:40.995 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.995 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:40.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.996 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1369912678", 2024-09-10T18:39:40.996 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1369912678", 2024-09-10T18:39:40.996 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1369912678", 2024-09-10T18:39:40.996 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1369912678", 2024-09-10T18:39:40.996 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:39:40.996 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:39:40.996 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:39:40.996 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:40.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.996 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-10T18:39:40.996 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "2ad55d2a-36fa-4756-bf22-58d4b049486e", 2024-09-10T18:39:40.996 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:39:40.997 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:39:40.997 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:39:40.997 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:39:40.997 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:39:40.997 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:39:40.997 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-09-10T18:39:40.997 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 365, 2024-09-10T18:39:40.997 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:39:40.997 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:39:40.997 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:39:40.997 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:40.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:40.998 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-09-10T18:39:40.998 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:39:40.998 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.998 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:40.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.998 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:39:40.998 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:40.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:40.998 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-09-10T18:39:40.998 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:39:40.998 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.998 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:40.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.998 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:39:40.999 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:40.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.999 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:40.999 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-09-10T18:39:40.999 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:39:40.999 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:40.999 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:40.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:40.999 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:39:40.999 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:40.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:40.999 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:40.999 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-09-10T18:39:40.999 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:39:41.000 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.000 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.000 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/1243406003", 2024-09-10T18:39:41.000 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/1243406003", 2024-09-10T18:39:41.000 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/1243406003", 2024-09-10T18:39:41.000 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1243406003", 2024-09-10T18:39:41.000 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:39:41.000 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:39:41.000 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:39:41.000 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.001 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-10T18:39:41.001 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "242729a5-a5e8-44d5-97c8-14d14e970bff", 2024-09-10T18:39:41.001 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:39:41.001 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:39:41.001 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:39:41.001 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:39:41.001 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:39:41.001 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:39:41.001 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-09-10T18:39:41.001 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 361, 2024-09-10T18:39:41.001 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:39:41.001 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:39:41.002 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:39:41.002 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.002 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-09-10T18:39:41.002 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:39:41.002 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.002 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.003 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:39:41.003 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.003 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-09-10T18:39:41.003 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:39:41.003 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.003 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.003 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:39:41.004 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.004 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-09-10T18:39:41.004 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:39:41.004 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.004 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.004 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:39:41.004 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.004 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-09-10T18:39:41.004 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:39:41.005 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.005 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.005 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/67703283", 2024-09-10T18:39:41.005 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/67703283", 2024-09-10T18:39:41.005 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/67703283", 2024-09-10T18:39:41.005 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/67703283", 2024-09-10T18:39:41.005 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:39:41.005 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:39:41.005 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:39:41.005 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-10T18:39:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "6e38b8ef-e6ed-493f-8eca-755354754787", 2024-09-10T18:39:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:39:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:39:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:39:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:39:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:39:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:39:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 33, 2024-09-10T18:39:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 365, 2024-09-10T18:39:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:39:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:39:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:39:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.007 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.007 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6800", 2024-09-10T18:39:41.007 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:39:41.007 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.007 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.007 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:39:41.007 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.007 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.007 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6801", 2024-09-10T18:39:41.007 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:39:41.008 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.008 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.008 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:39:41.008 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.008 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6803", 2024-09-10T18:39:41.008 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:39:41.008 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.008 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.008 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:39:41.009 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.009 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6802", 2024-09-10T18:39:41.009 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:39:41.009 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.009 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.009 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6800/1619669076", 2024-09-10T18:39:41.009 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6801/1619669076", 2024-09-10T18:39:41.009 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6803/1619669076", 2024-09-10T18:39:41.009 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6802/1619669076", 2024-09-10T18:39:41.009 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:39:41.009 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:39:41.010 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:39:41.010 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.010 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-10T18:39:41.010 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "a92e0441-66c9-498e-a0ac-6465acdbb1f5", 2024-09-10T18:39:41.010 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:39:41.010 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:39:41.010 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:39:41.010 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:39:41.010 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:39:41.010 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:39:41.010 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 39, 2024-09-10T18:39:41.010 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 365, 2024-09-10T18:39:41.011 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:39:41.011 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:39:41.011 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:39:41.011 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.011 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.011 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6804", 2024-09-10T18:39:41.011 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:39:41.011 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.011 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.011 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:39:41.011 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.012 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6805", 2024-09-10T18:39:41.012 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:39:41.012 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.012 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.012 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:39:41.012 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.012 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6807", 2024-09-10T18:39:41.012 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:39:41.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.013 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.013 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:39:41.013 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.013 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.013 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6806", 2024-09-10T18:39:41.013 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:39:41.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.013 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.013 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6804/2717802880", 2024-09-10T18:39:41.013 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6805/2717802880", 2024-09-10T18:39:41.014 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6807/2717802880", 2024-09-10T18:39:41.014 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6806/2717802880", 2024-09-10T18:39:41.014 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:39:41.014 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:39:41.014 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:39:41.014 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-09-10T18:39:41.014 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "12b02875-6653-45ed-8d2b-b4d33bd88525", 2024-09-10T18:39:41.014 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:39:41.014 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:39:41.014 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:39:41.014 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:39:41.015 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 46, 2024-09-10T18:39:41.015 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 94, 2024-09-10T18:39:41.015 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 144, 2024-09-10T18:39:41.015 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 365, 2024-09-10T18:39:41.015 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 95, 2024-09-10T18:39:41.015 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:39:41.015 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:39:41.015 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.015 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6808", 2024-09-10T18:39:41.015 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:39:41.015 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.016 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.016 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:39:41.016 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.016 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6809", 2024-09-10T18:39:41.016 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:39:41.016 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.016 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.016 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:39:41.016 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.017 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.017 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6811", 2024-09-10T18:39:41.017 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:39:41.017 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.017 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.017 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:39:41.017 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.017 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.017 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6810", 2024-09-10T18:39:41.017 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:39:41.017 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.018 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.018 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6808/3479311452", 2024-09-10T18:39:41.018 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6809/3479311452", 2024-09-10T18:39:41.018 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6811/3479311452", 2024-09-10T18:39:41.018 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6810/3479311452", 2024-09-10T18:39:41.018 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:39:41.018 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:39:41.018 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:39:41.018 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.018 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-09-10T18:39:41.018 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "0c4f7aa9-ff95-4192-a146-ca576eee907f", 2024-09-10T18:39:41.019 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:39:41.019 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:39:41.019 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:39:41.019 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:39:41.019 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:39:41.019 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:39:41.019 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 54, 2024-09-10T18:39:41.019 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 362, 2024-09-10T18:39:41.019 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:39:41.019 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:39:41.019 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:39:41.019 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.019 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.020 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6812", 2024-09-10T18:39:41.020 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:39:41.020 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.020 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.020 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:39:41.020 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.020 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6813", 2024-09-10T18:39:41.020 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:39:41.020 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.020 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.021 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:39:41.021 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.021 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6815", 2024-09-10T18:39:41.021 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:39:41.021 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.021 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.021 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:39:41.021 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:39:41.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.022 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:39:41.022 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6814", 2024-09-10T18:39:41.022 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:39:41.022 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.022 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.022 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6812/3511791879", 2024-09-10T18:39:41.022 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6813/3511791879", 2024-09-10T18:39:41.022 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6815/3511791879", 2024-09-10T18:39:41.022 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6814/3511791879", 2024-09-10T18:39:41.022 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:39:41.022 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:39:41.022 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:39:41.023 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.023 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.023 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:39:41.023 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-09-10T18:39:41.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.023 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-10T18:39:41.023 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:39:41.023 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:39:41.023 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:39:41.023 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:39:41.023 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 65536, 2024-09-10T18:39:41.023 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:38:35.746172+0000", 2024-09-10T18:39:41.023 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:39:41.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.024 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-10T18:39:41.024 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:39:41.024 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:39:41.024 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:39:41.024 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:39:41.024 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:39:41.024 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:38:32.969141+0000", 2024-09-10T18:39:41.024 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:39:41.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.024 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-10T18:39:41.024 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:39:41.025 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:39:41.025 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:39:41.025 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:39:41.025 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:39:41.025 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:38:51.561656+0000", 2024-09-10T18:39:41.025 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:39:41.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.025 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-10T18:39:41.025 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:39:41.025 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:39:41.025 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:39:41.025 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:39:41.025 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:39:41.026 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:38:19.137945+0000", 2024-09-10T18:39:41.026 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:39:41.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.026 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-10T18:39:41.026 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:39:41.026 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:39:41.026 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:39:41.026 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:39:41.026 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:39:41.026 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:39:00.681732+0000", 2024-09-10T18:39:41.026 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:39:41.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.027 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-10T18:39:41.027 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:39:41.027 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:39:41.027 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:39:41.027 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:39:41.027 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:39:41.027 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:39:38.239071+0000", 2024-09-10T18:39:41.027 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:39:41.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.027 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-09-10T18:39:41.027 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:39:41.027 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:39:41.028 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:39:41.028 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:39:41.028 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:39:41.028 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:39:39.624768+0000", 2024-09-10T18:39:41.028 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 94 2024-09-10T18:39:41.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.028 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-09-10T18:39:41.028 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:39:41.028 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:39:41.028 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:39:41.028 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:39:41.028 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:39:41.028 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:38:38.477565+0000", 2024-09-10T18:39:41.029 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:39:41.029 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.029 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:39:41.029 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-09-10T18:39:41.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.029 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.c", 2024-09-10T18:39:41.029 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:39:41.029 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-09-10T18:39:41.029 INFO:teuthology.orchestra.run.smithi028.stdout: 4, 2024-09-10T18:39:41.029 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-10T18:39:41.029 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.030 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.1f", 2024-09-10T18:39:41.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:39:41.030 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-10T18:39:41.030 INFO:teuthology.orchestra.run.smithi028.stdout: 2, 2024-09-10T18:39:41.030 INFO:teuthology.orchestra.run.smithi028.stdout: 7 2024-09-10T18:39:41.030 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.030 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "6.17", 2024-09-10T18:39:41.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:39:41.030 INFO:teuthology.orchestra.run.smithi028.stdout: 3, 2024-09-10T18:39:41.030 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-09-10T18:39:41.030 INFO:teuthology.orchestra.run.smithi028.stdout: 5 2024-09-10T18:39:41.031 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.031 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.031 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:39:41.031 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-09-10T18:39:41.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.031 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1b", 2024-09-10T18:39:41.031 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:39:41.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.031 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:39:41.031 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-09-10T18:39:41.031 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.031 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.032 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1d", 2024-09-10T18:39:41.032 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:39:41.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.032 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:39:41.032 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-09-10T18:39:41.032 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.032 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.033 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1f", 2024-09-10T18:39:41.033 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:39:41.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.033 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:39:41.033 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-09-10T18:39:41.033 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.033 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.033 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.033 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:39:41.033 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [ 2024-09-10T18:39:41.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.034 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "10.5", 2024-09-10T18:39:41.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:39:41.034 INFO:teuthology.orchestra.run.smithi028.stdout: 2, 2024-09-10T18:39:41.034 INFO:teuthology.orchestra.run.smithi028.stdout: 3, 2024-09-10T18:39:41.034 INFO:teuthology.orchestra.run.smithi028.stdout: 0 2024-09-10T18:39:41.034 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:39:41.034 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "10.c", 2024-09-10T18:39:41.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:39:41.034 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-09-10T18:39:41.034 INFO:teuthology.orchestra.run.smithi028.stdout: 4, 2024-09-10T18:39:41.034 INFO:teuthology.orchestra.run.smithi028.stdout: 0 2024-09-10T18:39:41.035 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:39:41.035 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.035 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:39:41.035 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-09-10T18:39:41.035 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-09-10T18:39:41.035 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1745574083": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:39:41.035 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3866220552": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:39:41.035 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3760775432": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:39:41.035 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1746797014": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:39:41.035 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1029045895": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:39:41.035 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3310949126": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:39:41.035 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2874160765": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:39:41.035 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2620179974": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:39:41.035 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/421194352": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:39:41.036 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/1077377902": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:39:41.036 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3428757300": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:39:41.036 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1759116360": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:39:41.036 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2564333506": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:39:41.036 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/717120167": "2024-09-11T18:22:08.678985+0000" 2024-09-10T18:39:41.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.036 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-09-10T18:39:41.036 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-09-10T18:39:41.036 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-09-10T18:39:41.036 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-09-10T18:39:41.036 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-09-10T18:39:41.036 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-09-10T18:39:41.037 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-09-10T18:39:41.037 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-09-10T18:39:41.037 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:39:41.037 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-09-10T18:39:41.037 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-09-10T18:39:41.037 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-09-10T18:39:41.037 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-09-10T18:39:41.037 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-09-10T18:39:41.037 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-09-10T18:39:41.037 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-09-10T18:39:41.037 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-09-10T18:39:41.037 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-09-10T18:39:41.038 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-09-10T18:39:41.038 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-09-10T18:39:41.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:39:41.038 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:39:41.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:41.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:41.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:41.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:41.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:41.193 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:41.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:41.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:41.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:41.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:41.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:41.468 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:41.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:41.570 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:41.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:41 smithi204 ceph-mon[32357]: 4.14 scrub starts 2024-09-10T18:39:41.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:41 smithi204 ceph-mon[32357]: 4.14 scrub ok 2024-09-10T18:39:41.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:41 smithi204 ceph-mon[32357]: 4.b scrub starts 2024-09-10T18:39:41.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:41 smithi204 ceph-mon[32357]: 4.b scrub ok 2024-09-10T18:39:41.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:41 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:39:41.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:41 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/952043086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:39:41.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:41 smithi204 ceph-mon[32357]: osdmap e368: 8 total, 8 up, 7 in 2024-09-10T18:39:41.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:41 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3910029240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:39:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:41.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:41.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:41.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:41 smithi028 ceph-mon[29291]: 4.14 scrub starts 2024-09-10T18:39:41.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:41 smithi028 ceph-mon[29291]: 4.14 scrub ok 2024-09-10T18:39:41.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:41 smithi028 ceph-mon[29291]: 4.b scrub starts 2024-09-10T18:39:41.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:41 smithi028 ceph-mon[29291]: 4.b scrub ok 2024-09-10T18:39:41.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:41 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:39:41.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:41 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/952043086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:39:41.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:41 smithi028 ceph-mon[29291]: osdmap e368: 8 total, 8 up, 7 in 2024-09-10T18:39:41.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:41 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3910029240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:39:41.643 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:39:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:39:41] "GET /metrics HTTP/1.1" 200 224890 "" "Prometheus/2.43.0" 2024-09-10T18:39:41.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:41 smithi028 ceph-mon[39220]: 4.14 scrub starts 2024-09-10T18:39:41.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:41 smithi028 ceph-mon[39220]: 4.14 scrub ok 2024-09-10T18:39:41.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:41 smithi028 ceph-mon[39220]: 4.b scrub starts 2024-09-10T18:39:41.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:41 smithi028 ceph-mon[39220]: 4.b scrub ok 2024-09-10T18:39:41.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:41 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:39:41.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:41 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/952043086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:39:41.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:41 smithi028 ceph-mon[39220]: osdmap e368: 8 total, 8 up, 7 in 2024-09-10T18:39:41.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:41 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3910029240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:39:41.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:41.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:41.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:41.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:41.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:41.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:41.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:42.024 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:42.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:42.053 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:39:42.053 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:39:42.125 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:42.161 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:42.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:42.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:42.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:42.299 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:42.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:42.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:42.401 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:42.437 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:42.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:42.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:42.539 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:42.578 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:42.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:42.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:42 smithi204 ceph-mon[32357]: 3.17 scrub starts 2024-09-10T18:39:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:42 smithi204 ceph-mon[32357]: 3.17 scrub ok 2024-09-10T18:39:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:42 smithi204 ceph-mon[32357]: 10.8 scrub starts 2024-09-10T18:39:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:42 smithi204 ceph-mon[32357]: 10.8 scrub ok 2024-09-10T18:39:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:42 smithi204 ceph-mon[32357]: pgmap v822: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 344 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.52k op/s; 730/17064 objects misplaced (4.278%); 24 MiB/s, 390 objects/s recovering 2024-09-10T18:39:42.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:39:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:39:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:39:42.648 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:42 smithi028 ceph-mon[39220]: 3.17 scrub starts 2024-09-10T18:39:42.648 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:42 smithi028 ceph-mon[39220]: 3.17 scrub ok 2024-09-10T18:39:42.648 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:42 smithi028 ceph-mon[39220]: 10.8 scrub starts 2024-09-10T18:39:42.648 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:42 smithi028 ceph-mon[39220]: 10.8 scrub ok 2024-09-10T18:39:42.648 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:42 smithi028 ceph-mon[39220]: pgmap v822: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 344 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.52k op/s; 730/17064 objects misplaced (4.278%); 24 MiB/s, 390 objects/s recovering 2024-09-10T18:39:42.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:42 smithi028 ceph-mon[29291]: 3.17 scrub starts 2024-09-10T18:39:42.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:42 smithi028 ceph-mon[29291]: 3.17 scrub ok 2024-09-10T18:39:42.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:42 smithi028 ceph-mon[29291]: 10.8 scrub starts 2024-09-10T18:39:42.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:42 smithi028 ceph-mon[29291]: 10.8 scrub ok 2024-09-10T18:39:42.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:42 smithi028 ceph-mon[29291]: pgmap v822: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 344 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.52k op/s; 730/17064 objects misplaced (4.278%); 24 MiB/s, 390 objects/s recovering 2024-09-10T18:39:42.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:42.719 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:42.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:42.821 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:42.858 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:42.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:42.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:42.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-10T18:39:42.959 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:42.994 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:42.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:42.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:43.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:43.146 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:43.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:43.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:43.249 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:43.280 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:43.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:43.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:43.382 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:43.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:43.424 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:43.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:43.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:43.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:43.567 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:43.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:43 smithi204 ceph-mon[32357]: 6.c scrub starts 2024-09-10T18:39:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:43 smithi204 ceph-mon[32357]: 6.c scrub ok 2024-09-10T18:39:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:43 smithi204 ceph-mon[32357]: 4.5 scrub starts 2024-09-10T18:39:43.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:43.702 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:43.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:43.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:43.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:43 smithi028 ceph-mon[39220]: 6.c scrub starts 2024-09-10T18:39:43.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:43 smithi028 ceph-mon[39220]: 6.c scrub ok 2024-09-10T18:39:43.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:43 smithi028 ceph-mon[39220]: 4.5 scrub starts 2024-09-10T18:39:43.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:43 smithi028 ceph-mon[29291]: 6.c scrub starts 2024-09-10T18:39:43.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:43 smithi028 ceph-mon[29291]: 6.c scrub ok 2024-09-10T18:39:43.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:43 smithi028 ceph-mon[29291]: 4.5 scrub starts 2024-09-10T18:39:43.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:43.839 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:43.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:43.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:43.941 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:43.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:43.984 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:43.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:44.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:44.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:44.121 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:44.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:44.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:44.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:44.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:44.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:44.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:44.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:44.424 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Removed 136921 objects 2024-09-10T18:39:44.424 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Clean up completed and total clean up time :115.378 2024-09-10T18:39:44.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:44.519 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:44.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:44.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:44.543 INFO:tasks.thrashosds.thrasher:j is {'epoch': 368, 'fsid': '5eababb2-6fa0-11ef-bcea-c7b262605968', 'created': '2024-09-10T18:16:20.680373+0000', 'modified': '2024-09-10T18:39:40.300739+0000', 'last_up_change': '2024-09-10T18:26:26.611338+0000', 'last_in_change': '2024-09-10T18:39:24.201588+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T18:19:24.013536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '300', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '287', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-10T18:21:30.065209+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 62, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T18:21:30.506441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T18:21:32.008130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T18:21:34.060571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T18:21:36.127330+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-10T18:36:06.616074+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '319', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'f1c834d9-252a-4077-b25e-169181d70251', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 356, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 3188372351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 3188372351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 3188372351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 3188372351}]}, 'public_addr': '172.21.15.28:6801/3188372351', 'cluster_addr': '172.21.15.28:6802/3188372351', 'heartbeat_back_addr': '172.21.15.28:6804/3188372351', 'heartbeat_front_addr': '172.21.15.28:6803/3188372351', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '81f1855d-b946-4ca3-b246-38977b9e20fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1369912678}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1369912678}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1369912678}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1369912678}]}, 'public_addr': '172.21.15.28:6805/1369912678', 'cluster_addr': '172.21.15.28:6806/1369912678', 'heartbeat_back_addr': '172.21.15.28:6808/1369912678', 'heartbeat_front_addr': '172.21.15.28:6807/1369912678', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2ad55d2a-36fa-4756-bf22-58d4b049486e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 365, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 1243406003}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 1243406003}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 1243406003}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1243406003}]}, 'public_addr': '172.21.15.28:6809/1243406003', 'cluster_addr': '172.21.15.28:6810/1243406003', 'heartbeat_back_addr': '172.21.15.28:6812/1243406003', 'heartbeat_front_addr': '172.21.15.28:6811/1243406003', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '242729a5-a5e8-44d5-97c8-14d14e970bff', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 361, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 67703283}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 67703283}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 67703283}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 67703283}]}, 'public_addr': '172.21.15.28:6813/67703283', 'cluster_addr': '172.21.15.28:6814/67703283', 'heartbeat_back_addr': '172.21.15.28:6816/67703283', 'heartbeat_front_addr': '172.21.15.28:6815/67703283', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6e38b8ef-e6ed-493f-8eca-755354754787', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 365, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6800', 'nonce': 1619669076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 1619669076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 1619669076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6802', 'nonce': 1619669076}]}, 'public_addr': '172.21.15.204:6800/1619669076', 'cluster_addr': '172.21.15.204:6801/1619669076', 'heartbeat_back_addr': '172.21.15.204:6803/1619669076', 'heartbeat_front_addr': '172.21.15.204:6802/1619669076', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a92e0441-66c9-498e-a0ac-6465acdbb1f5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 365, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6804', 'nonce': 2717802880}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 2717802880}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 2717802880}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6806', 'nonce': 2717802880}]}, 'public_addr': '172.21.15.204:6804/2717802880', 'cluster_addr': '172.21.15.204:6805/2717802880', 'heartbeat_back_addr': '172.21.15.204:6807/2717802880', 'heartbeat_front_addr': '172.21.15.204:6806/2717802880', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '12b02875-6653-45ed-8d2b-b4d33bd88525', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 46, 'last_clean_end': 94, 'up_from': 144, 'up_thru': 365, 'down_at': 95, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6808', 'nonce': 3479311452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 3479311452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 3479311452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6810', 'nonce': 3479311452}]}, 'public_addr': '172.21.15.204:6808/3479311452', 'cluster_addr': '172.21.15.204:6809/3479311452', 'heartbeat_back_addr': '172.21.15.204:6811/3479311452', 'heartbeat_front_addr': '172.21.15.204:6810/3479311452', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0c4f7aa9-ff95-4192-a146-ca576eee907f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 362, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6812', 'nonce': 3511791879}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 3511791879}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 3511791879}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6814', 'nonce': 3511791879}]}, 'public_addr': '172.21.15.204:6812/3511791879', 'cluster_addr': '172.21.15.204:6813/3511791879', 'heartbeat_back_addr': '172.21.15.204:6815/3511791879', 'heartbeat_front_addr': '172.21.15.204:6814/3511791879', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-10T18:38:35.746172+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:38:32.969141+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:38:51.561656+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:38:19.137945+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:39:00.681732+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:39:38.239071+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:39:39.624768+0000', 'dead_epoch': 94}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:38:38.477565+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.c', 'osds': [7, 4, 2]}, {'pgid': '5.1f', 'osds': [0, 2, 7]}, {'pgid': '6.17', 'osds': [3, 7, 5]}], 'pg_upmap_items': [{'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [{'pgid': '10.5', 'osds': [2, 3, 0]}, {'pgid': '10.c', 'osds': [7, 4, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/1745574083': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/3866220552': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:6800/3760775432': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:0/1746797014': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:0/1029045895': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:6800/3310949126': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/2874160765': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/2620179974': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/421194352': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:6800/1077377902': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/3428757300': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/1759116360': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/2564333506': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/717120167': '2024-09-11T18:22:08.678985+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T18:39:44.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:39:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:44.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:44 smithi204 ceph-mon[32357]: 4.5 scrub ok 2024-09-10T18:39:44.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:44 smithi204 ceph-mon[32357]: 4.1c scrub starts 2024-09-10T18:39:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:44 smithi204 ceph-mon[32357]: 4.1c scrub ok 2024-09-10T18:39:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:44 smithi204 ceph-mon[32357]: pgmap v823: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 344 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 730/17064 objects misplaced (4.278%); 14 MiB/s, 225 objects/s recovering 2024-09-10T18:39:44.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:44.637 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:44.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:44.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:44.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:44.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:44 smithi028 ceph-mon[29291]: 4.5 scrub ok 2024-09-10T18:39:44.718 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:44 smithi028 ceph-mon[29291]: 4.1c scrub starts 2024-09-10T18:39:44.718 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:44 smithi028 ceph-mon[29291]: 4.1c scrub ok 2024-09-10T18:39:44.718 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:44 smithi028 ceph-mon[29291]: pgmap v823: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 344 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 730/17064 objects misplaced (4.278%); 14 MiB/s, 225 objects/s recovering 2024-09-10T18:39:44.718 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:44 smithi028 ceph-mon[39220]: 4.5 scrub ok 2024-09-10T18:39:44.718 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:44 smithi028 ceph-mon[39220]: 4.1c scrub starts 2024-09-10T18:39:44.718 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:44 smithi028 ceph-mon[39220]: 4.1c scrub ok 2024-09-10T18:39:44.718 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:44 smithi028 ceph-mon[39220]: pgmap v823: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 344 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 730/17064 objects misplaced (4.278%); 14 MiB/s, 225 objects/s recovering 2024-09-10T18:39:44.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:44.770 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:44.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:44.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:44.872 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:44.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:44.899 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:44.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:45.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:45.024 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:45.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:45.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:45.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:45.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:45.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:45.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:45.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:45.274 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:45.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:45.334 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:39:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:39:45.376 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:45.402 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:45.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:45.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:45.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:45.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:45 smithi028 ceph-mon[29291]: 3.18 scrub starts 2024-09-10T18:39:45.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:45 smithi028 ceph-mon[29291]: 3.18 scrub ok 2024-09-10T18:39:45.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:45 smithi028 ceph-mon[39220]: 3.18 scrub starts 2024-09-10T18:39:45.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:45 smithi028 ceph-mon[39220]: 3.18 scrub ok 2024-09-10T18:39:45.528 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:45.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:45.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:45 smithi204 ceph-mon[32357]: 3.18 scrub starts 2024-09-10T18:39:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:45 smithi204 ceph-mon[32357]: 3.18 scrub ok 2024-09-10T18:39:45.631 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:45.656 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:45.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:45.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:45.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:45.782 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:45.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:45.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:45.908 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:45.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:45.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:46.010 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:46.035 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:46.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:46.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:46.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:39:46.137 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:46.163 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:46.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:46.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:46.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:46.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:46.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:46.334 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-10T18:39:46.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-10T18:39:46.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:46.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:46.417 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:46.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:46.518 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:46.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:46.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:46 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:46 smithi204 ceph-mon[32357]: pgmap v824: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 247 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 363/12435 objects misplaced (2.919%) 2024-09-10T18:39:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:46 smithi204 ceph-mon[32357]: 10.b scrub starts 2024-09-10T18:39:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:46 smithi204 ceph-mon[32357]: 10.b scrub ok 2024-09-10T18:39:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:46 smithi204 ceph-mon[32357]: 6.5 scrub starts 2024-09-10T18:39:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:46 smithi204 ceph-mon[32357]: 6.5 scrub ok 2024-09-10T18:39:46.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:46.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:46.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:46.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:46.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:46 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:46.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:46 smithi028 ceph-mon[39220]: pgmap v824: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 247 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 363/12435 objects misplaced (2.919%) 2024-09-10T18:39:46.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:46 smithi028 ceph-mon[39220]: 10.b scrub starts 2024-09-10T18:39:46.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:46 smithi028 ceph-mon[39220]: 10.b scrub ok 2024-09-10T18:39:46.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:46 smithi028 ceph-mon[39220]: 6.5 scrub starts 2024-09-10T18:39:46.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:46 smithi028 ceph-mon[39220]: 6.5 scrub ok 2024-09-10T18:39:46.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:46 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:46.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:46 smithi028 ceph-mon[29291]: pgmap v824: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 247 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 363/12435 objects misplaced (2.919%) 2024-09-10T18:39:46.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:46 smithi028 ceph-mon[29291]: 10.b scrub starts 2024-09-10T18:39:46.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:46 smithi028 ceph-mon[29291]: 10.b scrub ok 2024-09-10T18:39:46.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:46 smithi028 ceph-mon[29291]: 6.5 scrub starts 2024-09-10T18:39:46.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:46 smithi028 ceph-mon[29291]: 6.5 scrub ok 2024-09-10T18:39:46.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:46.796 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:46.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:46.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:46.901 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:46.927 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:46.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:46.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:47.029 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:47.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:47.056 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:47.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:47.157 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:47.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:47.184 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:47.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:47.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:47.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:47.314 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:47.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:47.415 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:47.442 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:47.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:47.544 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:47.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:47.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:47.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:47.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:47 smithi204 ceph-mon[32357]: 4.1b scrub starts 2024-09-10T18:39:47.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:47 smithi204 ceph-mon[32357]: 4.1b scrub ok 2024-09-10T18:39:47.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:47 smithi204 ceph-mon[32357]: 6.10 scrub starts 2024-09-10T18:39:47.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:47 smithi204 ceph-mon[32357]: 6.10 scrub ok 2024-09-10T18:39:47.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:47 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:39:47.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:47 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:39:47.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:47 smithi204 ceph-mon[32357]: osdmap e369: 8 total, 8 up, 7 in 2024-09-10T18:39:47.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:47.696 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:47.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:47.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:47.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:47 smithi028 ceph-mon[29291]: 4.1b scrub starts 2024-09-10T18:39:47.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:47 smithi028 ceph-mon[29291]: 4.1b scrub ok 2024-09-10T18:39:47.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:47 smithi028 ceph-mon[29291]: 6.10 scrub starts 2024-09-10T18:39:47.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:47 smithi028 ceph-mon[29291]: 6.10 scrub ok 2024-09-10T18:39:47.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:47 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:39:47.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:47 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:39:47.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:47 smithi028 ceph-mon[29291]: osdmap e369: 8 total, 8 up, 7 in 2024-09-10T18:39:47.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:47 smithi028 ceph-mon[39220]: 4.1b scrub starts 2024-09-10T18:39:47.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:47 smithi028 ceph-mon[39220]: 4.1b scrub ok 2024-09-10T18:39:47.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:47 smithi028 ceph-mon[39220]: 6.10 scrub starts 2024-09-10T18:39:47.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:47 smithi028 ceph-mon[39220]: 6.10 scrub ok 2024-09-10T18:39:47.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:47 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:39:47.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:47 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:39:47.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:47 smithi028 ceph-mon[39220]: osdmap e369: 8 total, 8 up, 7 in 2024-09-10T18:39:47.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:47.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:47.928 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:47.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:47.952 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:47.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:48.053 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:48.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:48.076 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:48.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:48.177 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:48.200 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:48.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:48.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:48.302 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:48.325 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:48.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:48.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:48.427 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:48.451 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:48.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:48.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:48.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:48.575 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:48.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:48.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:48.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:48 smithi204 ceph-mon[32357]: pgmap v826: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-09-10T18:39:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:48 smithi204 ceph-mon[32357]: 5.19 scrub starts 2024-09-10T18:39:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:48 smithi204 ceph-mon[32357]: 5.19 scrub ok 2024-09-10T18:39:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:48 smithi204 ceph-mon[32357]: 6.8 scrub starts 2024-09-10T18:39:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:48 smithi204 ceph-mon[32357]: 6.8 scrub ok 2024-09-10T18:39:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:48 smithi204 ceph-mon[32357]: 10.a scrub starts 2024-09-10T18:39:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:48 smithi204 ceph-mon[32357]: 10.a scrub ok 2024-09-10T18:39:48.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:48 smithi028 ceph-mon[39220]: pgmap v826: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-09-10T18:39:48.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:48 smithi028 ceph-mon[39220]: 5.19 scrub starts 2024-09-10T18:39:48.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:48 smithi028 ceph-mon[39220]: 5.19 scrub ok 2024-09-10T18:39:48.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:48 smithi028 ceph-mon[39220]: 6.8 scrub starts 2024-09-10T18:39:48.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:48 smithi028 ceph-mon[39220]: 6.8 scrub ok 2024-09-10T18:39:48.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:48 smithi028 ceph-mon[39220]: 10.a scrub starts 2024-09-10T18:39:48.631 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:48 smithi028 ceph-mon[39220]: 10.a scrub ok 2024-09-10T18:39:48.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:48 smithi028 ceph-mon[29291]: pgmap v826: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-09-10T18:39:48.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:48 smithi028 ceph-mon[29291]: 5.19 scrub starts 2024-09-10T18:39:48.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:48 smithi028 ceph-mon[29291]: 5.19 scrub ok 2024-09-10T18:39:48.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:48 smithi028 ceph-mon[29291]: 6.8 scrub starts 2024-09-10T18:39:48.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:48 smithi028 ceph-mon[29291]: 6.8 scrub ok 2024-09-10T18:39:48.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:48 smithi028 ceph-mon[29291]: 10.a scrub starts 2024-09-10T18:39:48.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:48 smithi028 ceph-mon[29291]: 10.a scrub ok 2024-09-10T18:39:48.677 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:48.699 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:48.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:48.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:48.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:48.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:48.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:48.931 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:48.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:48.953 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:48.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:49.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:49.076 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:49.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:49.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:49.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:49.204 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:49.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:49.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:49.306 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:49.328 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:49.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:49.407 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:49.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:49.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:49.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:49.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:49.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:49.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:49.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:49 smithi028 ceph-mon[29291]: 3.14 scrub starts 2024-09-10T18:39:49.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:49 smithi028 ceph-mon[29291]: 3.14 scrub ok 2024-09-10T18:39:49.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:49 smithi028 ceph-mon[29291]: 5.b scrub starts 2024-09-10T18:39:49.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:49 smithi028 ceph-mon[29291]: 5.b scrub ok 2024-09-10T18:39:49.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:49 smithi028 ceph-mon[39220]: 3.14 scrub starts 2024-09-10T18:39:49.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:49 smithi028 ceph-mon[39220]: 3.14 scrub ok 2024-09-10T18:39:49.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:49 smithi028 ceph-mon[39220]: 5.b scrub starts 2024-09-10T18:39:49.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:49 smithi028 ceph-mon[39220]: 5.b scrub ok 2024-09-10T18:39:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:49 smithi204 ceph-mon[32357]: 3.14 scrub starts 2024-09-10T18:39:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:49 smithi204 ceph-mon[32357]: 3.14 scrub ok 2024-09-10T18:39:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:49 smithi204 ceph-mon[32357]: 5.b scrub starts 2024-09-10T18:39:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:49 smithi204 ceph-mon[32357]: 5.b scrub ok 2024-09-10T18:39:49.677 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:49.701 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:49.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:49.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:49.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:49.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:49.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:49.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:49.952 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:49.975 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:49.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:49.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:50.077 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:50.100 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:50.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:50.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:50.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:50.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:50.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:50.253 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:39:50.254 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:39:50.256 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:39:50.256 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:39:50.264 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:50.279 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:39:50.279 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:39:50.295 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:39:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:39:50.311 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:39:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:39:50.327 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:39:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:39:50.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:50.343 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:39:50.344 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:39:50.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:50.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:50.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:50.372 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:39:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:39:50.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:50.478 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:50.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:50.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:50 smithi028 ceph-mon[29291]: 4.e scrub starts 2024-09-10T18:39:50.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:50 smithi028 ceph-mon[29291]: 4.e scrub ok 2024-09-10T18:39:50.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:50 smithi028 ceph-mon[29291]: 3.2 scrub starts 2024-09-10T18:39:50.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:50 smithi028 ceph-mon[29291]: 3.2 scrub ok 2024-09-10T18:39:50.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:50 smithi028 ceph-mon[29291]: pgmap v827: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 579 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:39:50.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:50 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2704386736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:39:50.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:50 smithi028 ceph-mon[39220]: 4.e scrub starts 2024-09-10T18:39:50.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:50 smithi028 ceph-mon[39220]: 4.e scrub ok 2024-09-10T18:39:50.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:50 smithi028 ceph-mon[39220]: 3.2 scrub starts 2024-09-10T18:39:50.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:50 smithi028 ceph-mon[39220]: 3.2 scrub ok 2024-09-10T18:39:50.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:50 smithi028 ceph-mon[39220]: pgmap v827: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 579 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:39:50.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:50 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2704386736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:39:50.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:50.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:50.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:50 smithi204 ceph-mon[32357]: 4.e scrub starts 2024-09-10T18:39:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:50 smithi204 ceph-mon[32357]: 4.e scrub ok 2024-09-10T18:39:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:50 smithi204 ceph-mon[32357]: 3.2 scrub starts 2024-09-10T18:39:50.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:50 smithi204 ceph-mon[32357]: 3.2 scrub ok 2024-09-10T18:39:50.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:50 smithi204 ceph-mon[32357]: pgmap v827: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 579 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:39:50.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:50 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2704386736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:39:50.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:50.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:50.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:50.832 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:50.858 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:50.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:50.960 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:50.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:50.985 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:50.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:51.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:51.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:51.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:51.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:51.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:51.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:51.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:51.335 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:51.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:51.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:51.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:51.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:51.487 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:51.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:51.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:51 smithi204 ceph-mon[32357]: 3.1 scrub starts 2024-09-10T18:39:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:51 smithi204 ceph-mon[32357]: 3.1 scrub ok 2024-09-10T18:39:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:51 smithi204 ceph-mon[32357]: 3.8 scrub starts 2024-09-10T18:39:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:51 smithi204 ceph-mon[32357]: 3.8 scrub ok 2024-09-10T18:39:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:51 smithi204 ceph-mon[32357]: 3.1c deep-scrub starts 2024-09-10T18:39:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:51 smithi204 ceph-mon[32357]: 3.1c deep-scrub ok 2024-09-10T18:39:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:51 smithi204 ceph-mon[32357]: 4.4 scrub starts 2024-09-10T18:39:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:51 smithi204 ceph-mon[32357]: 4.4 scrub ok 2024-09-10T18:39:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:51 smithi204 ceph-mon[32357]: 4.1f deep-scrub starts 2024-09-10T18:39:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:51 smithi204 ceph-mon[32357]: 4.1f deep-scrub ok 2024-09-10T18:39:51.611 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:51.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:51.635 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:39:51] "GET /metrics HTTP/1.1" 200 224890 "" "Prometheus/2.43.0" 2024-09-10T18:39:51.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[39220]: 3.1 scrub starts 2024-09-10T18:39:51.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[39220]: 3.1 scrub ok 2024-09-10T18:39:51.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[39220]: 3.8 scrub starts 2024-09-10T18:39:51.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[39220]: 3.8 scrub ok 2024-09-10T18:39:51.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[39220]: 3.1c deep-scrub starts 2024-09-10T18:39:51.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[39220]: 3.1c deep-scrub ok 2024-09-10T18:39:51.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[39220]: 4.4 scrub starts 2024-09-10T18:39:51.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[39220]: 4.4 scrub ok 2024-09-10T18:39:51.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[39220]: 4.1f deep-scrub starts 2024-09-10T18:39:51.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[39220]: 4.1f deep-scrub ok 2024-09-10T18:39:51.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[29291]: 3.1 scrub starts 2024-09-10T18:39:51.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[29291]: 3.1 scrub ok 2024-09-10T18:39:51.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[29291]: 3.8 scrub starts 2024-09-10T18:39:51.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[29291]: 3.8 scrub ok 2024-09-10T18:39:51.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[29291]: 3.1c deep-scrub starts 2024-09-10T18:39:51.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[29291]: 3.1c deep-scrub ok 2024-09-10T18:39:51.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[29291]: 4.4 scrub starts 2024-09-10T18:39:51.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[29291]: 4.4 scrub ok 2024-09-10T18:39:51.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[29291]: 4.1f deep-scrub starts 2024-09-10T18:39:51.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:51 smithi028 ceph-mon[29291]: 4.1f deep-scrub ok 2024-09-10T18:39:51.713 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:51.739 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:51.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:51.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:51.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:51.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:51.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:51.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:51.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:51.990 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:51.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:52.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:52.114 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:52.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:52.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:52.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:52.239 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:52.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:52.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:52.341 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:52.367 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:52.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:52.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:52.386 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:39:52.386 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-10T18:39:52.386 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-10T18:39:52.386 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:39:52.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:52.495 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:52.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:52.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:52.505 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:39:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:39:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:39:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:52 smithi204 ceph-mon[32357]: 5.e scrub starts 2024-09-10T18:39:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:52 smithi204 ceph-mon[32357]: 5.e scrub ok 2024-09-10T18:39:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:52 smithi204 ceph-mon[32357]: 3.5 scrub starts 2024-09-10T18:39:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:52 smithi204 ceph-mon[32357]: 3.5 scrub ok 2024-09-10T18:39:52.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:52 smithi204 ceph-mon[32357]: 5.6 scrub starts 2024-09-10T18:39:52.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:52 smithi204 ceph-mon[32357]: 5.6 scrub ok 2024-09-10T18:39:52.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:52 smithi204 ceph-mon[32357]: pgmap v828: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 550 op/s 2024-09-10T18:39:52.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:52 smithi204 ceph-mon[32357]: 10.7 scrub starts 2024-09-10T18:39:52.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:52 smithi204 ceph-mon[32357]: 10.7 scrub ok 2024-09-10T18:39:52.597 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:52.623 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:52.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:52.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[39220]: 5.e scrub starts 2024-09-10T18:39:52.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[39220]: 5.e scrub ok 2024-09-10T18:39:52.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[39220]: 3.5 scrub starts 2024-09-10T18:39:52.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[39220]: 3.5 scrub ok 2024-09-10T18:39:52.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[39220]: 5.6 scrub starts 2024-09-10T18:39:52.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[39220]: 5.6 scrub ok 2024-09-10T18:39:52.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[39220]: pgmap v828: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 550 op/s 2024-09-10T18:39:52.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[39220]: 10.7 scrub starts 2024-09-10T18:39:52.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[39220]: 10.7 scrub ok 2024-09-10T18:39:52.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[29291]: 5.e scrub starts 2024-09-10T18:39:52.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[29291]: 5.e scrub ok 2024-09-10T18:39:52.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[29291]: 3.5 scrub starts 2024-09-10T18:39:52.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[29291]: 3.5 scrub ok 2024-09-10T18:39:52.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[29291]: 5.6 scrub starts 2024-09-10T18:39:52.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[29291]: 5.6 scrub ok 2024-09-10T18:39:52.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[29291]: pgmap v828: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 550 op/s 2024-09-10T18:39:52.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[29291]: 10.7 scrub starts 2024-09-10T18:39:52.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:52 smithi028 ceph-mon[29291]: 10.7 scrub ok 2024-09-10T18:39:52.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:52.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:52.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:52.851 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:52.876 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:52.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:52.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:52.911 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:52.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:53.003 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:53.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:53.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:53.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:53.131 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:53.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:53.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:53.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:53.258 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:53.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:53.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:53.360 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:53.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:53.386 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:53.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:53.488 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:53.513 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:53.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:53.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:53 smithi204 ceph-mon[32357]: 6.d scrub starts 2024-09-10T18:39:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:53 smithi204 ceph-mon[32357]: 6.d scrub ok 2024-09-10T18:39:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:53 smithi204 ceph-mon[32357]: 6.15 scrub starts 2024-09-10T18:39:53.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:53.643 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:53.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:53.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:53.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:53.745 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:53 smithi028 ceph-mon[29291]: 6.d scrub starts 2024-09-10T18:39:53.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:53 smithi028 ceph-mon[29291]: 6.d scrub ok 2024-09-10T18:39:53.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:53 smithi028 ceph-mon[29291]: 6.15 scrub starts 2024-09-10T18:39:53.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:53 smithi028 ceph-mon[39220]: 6.d scrub starts 2024-09-10T18:39:53.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:53 smithi028 ceph-mon[39220]: 6.d scrub ok 2024-09-10T18:39:53.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:53 smithi028 ceph-mon[39220]: 6.15 scrub starts 2024-09-10T18:39:53.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:53.772 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:53.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:53.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-10T18:39:53.874 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:53.899 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:53.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:53.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:54.001 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:54.025 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:54.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:54.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:54.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:54.150 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:54.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:54.252 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:54.278 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:54.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:54.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:54.381 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:54.406 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:54.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:54.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:54.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:54.531 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:54.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:54.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:54.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:54.657 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:54.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[39220]: pgmap v829: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 550 op/s 2024-09-10T18:39:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[39220]: 6.15 scrub ok 2024-09-10T18:39:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[39220]: 5.1d scrub starts 2024-09-10T18:39:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[39220]: 5.1d scrub ok 2024-09-10T18:39:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:39:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:39:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[39220]: 1.6 scrub starts 2024-09-10T18:39:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[39220]: 1.6 scrub ok 2024-09-10T18:39:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:39:54.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:39:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[29291]: pgmap v829: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 550 op/s 2024-09-10T18:39:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[29291]: 6.15 scrub ok 2024-09-10T18:39:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[29291]: 5.1d scrub starts 2024-09-10T18:39:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[29291]: 5.1d scrub ok 2024-09-10T18:39:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:39:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:39:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[29291]: 1.6 scrub starts 2024-09-10T18:39:54.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[29291]: 1.6 scrub ok 2024-09-10T18:39:54.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:39:54.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:54 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:39:54.758 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:54.782 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:54.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:54.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:54 smithi204 ceph-mon[32357]: pgmap v829: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 550 op/s 2024-09-10T18:39:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:54 smithi204 ceph-mon[32357]: 6.15 scrub ok 2024-09-10T18:39:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:54 smithi204 ceph-mon[32357]: 5.1d scrub starts 2024-09-10T18:39:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:54 smithi204 ceph-mon[32357]: 5.1d scrub ok 2024-09-10T18:39:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:54 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:39:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:54 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:39:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:54 smithi204 ceph-mon[32357]: 1.6 scrub starts 2024-09-10T18:39:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:54 smithi204 ceph-mon[32357]: 1.6 scrub ok 2024-09-10T18:39:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:54 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:39:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:54 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:39:54.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:54.908 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:54.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:54.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:55.010 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:55.035 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:55.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:55.137 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:55.160 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:55.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:55.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:55.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:55.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:55.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:55.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:39:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:39:55.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:39:55.367 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_3' removed 2024-09-10T18:39:55.387 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:55.414 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:55.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:55.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:55.516 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:55.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:55.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:55.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:55.640 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:55.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:55.664 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:55.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:55.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[29291]: 6.11 scrub starts 2024-09-10T18:39:55.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[29291]: 6.11 scrub ok 2024-09-10T18:39:55.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[29291]: 3.c scrub starts 2024-09-10T18:39:55.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[29291]: 3.c scrub ok 2024-09-10T18:39:55.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[29291]: osdmap e370: 8 total, 8 up, 7 in 2024-09-10T18:39:55.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[29291]: 1.b scrub starts 2024-09-10T18:39:55.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[29291]: 1.b scrub ok 2024-09-10T18:39:55.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[29291]: 3.10 scrub starts 2024-09-10T18:39:55.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[29291]: 3.10 scrub ok 2024-09-10T18:39:55.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/425066098' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:39:55.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[29291]: from='client.35822 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:39:55.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[39220]: 6.11 scrub starts 2024-09-10T18:39:55.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[39220]: 6.11 scrub ok 2024-09-10T18:39:55.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[39220]: 3.c scrub starts 2024-09-10T18:39:55.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[39220]: 3.c scrub ok 2024-09-10T18:39:55.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[39220]: osdmap e370: 8 total, 8 up, 7 in 2024-09-10T18:39:55.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[39220]: 1.b scrub starts 2024-09-10T18:39:55.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[39220]: 1.b scrub ok 2024-09-10T18:39:55.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[39220]: 3.10 scrub starts 2024-09-10T18:39:55.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[39220]: 3.10 scrub ok 2024-09-10T18:39:55.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/425066098' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:39:55.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:55 smithi028 ceph-mon[39220]: from='client.35822 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:39:55.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:55.789 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:55.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:55.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:55.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:55 smithi204 ceph-mon[32357]: 6.11 scrub starts 2024-09-10T18:39:55.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:55 smithi204 ceph-mon[32357]: 6.11 scrub ok 2024-09-10T18:39:55.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:55 smithi204 ceph-mon[32357]: 3.c scrub starts 2024-09-10T18:39:55.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:55 smithi204 ceph-mon[32357]: 3.c scrub ok 2024-09-10T18:39:55.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:55 smithi204 ceph-mon[32357]: osdmap e370: 8 total, 8 up, 7 in 2024-09-10T18:39:55.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:55 smithi204 ceph-mon[32357]: 1.b scrub starts 2024-09-10T18:39:55.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:55 smithi204 ceph-mon[32357]: 1.b scrub ok 2024-09-10T18:39:55.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:55 smithi204 ceph-mon[32357]: 3.10 scrub starts 2024-09-10T18:39:55.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:55 smithi204 ceph-mon[32357]: 3.10 scrub ok 2024-09-10T18:39:55.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:55 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/425066098' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:39:55.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:55 smithi204 ceph-mon[32357]: from='client.35822 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:39:55.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:55.923 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:55.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:56.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:56.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:56.048 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:56.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:56.069 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-10T18:39:56.070 INFO:tasks.radosbench:Beginning radosbench... 2024-09-10T18:39:56.071 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-10T18:39:56.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-09-10T18:39:56.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:56.170 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:56.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:56.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:56.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:56.293 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:56.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:56.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:56.395 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:56.417 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:56.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:56.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:56 smithi028 ceph-mon[29291]: 3.1e scrub starts 2024-09-10T18:39:56.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:56 smithi028 ceph-mon[29291]: 3.1e scrub ok 2024-09-10T18:39:56.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:56 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:56.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:56 smithi028 ceph-mon[29291]: 5.f scrub starts 2024-09-10T18:39:56.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:56 smithi028 ceph-mon[29291]: 5.f scrub ok 2024-09-10T18:39:56.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:56 smithi028 ceph-mon[29291]: pgmap v831: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:39:56.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:56 smithi028 ceph-mon[29291]: from='client.35822 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:39:56.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:56 smithi028 ceph-mon[29291]: osdmap e371: 8 total, 8 up, 7 in 2024-09-10T18:39:56.499 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:56 smithi028 ceph-mon[39220]: 3.1e scrub starts 2024-09-10T18:39:56.499 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:56 smithi028 ceph-mon[39220]: 3.1e scrub ok 2024-09-10T18:39:56.499 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:56 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:56.499 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:56 smithi028 ceph-mon[39220]: 5.f scrub starts 2024-09-10T18:39:56.499 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:56 smithi028 ceph-mon[39220]: 5.f scrub ok 2024-09-10T18:39:56.499 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:56 smithi028 ceph-mon[39220]: pgmap v831: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:39:56.499 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:56 smithi028 ceph-mon[39220]: from='client.35822 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:39:56.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:56 smithi028 ceph-mon[39220]: osdmap e371: 8 total, 8 up, 7 in 2024-09-10T18:39:56.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:56.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:56.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:56.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:56.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:39:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:56.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:56.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:39:56.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:56.795 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:56.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:56.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:56 smithi204 ceph-mon[32357]: 3.1e scrub starts 2024-09-10T18:39:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:56 smithi204 ceph-mon[32357]: 3.1e scrub ok 2024-09-10T18:39:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:56 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:39:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:56 smithi204 ceph-mon[32357]: 5.f scrub starts 2024-09-10T18:39:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:56 smithi204 ceph-mon[32357]: 5.f scrub ok 2024-09-10T18:39:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:56 smithi204 ceph-mon[32357]: pgmap v831: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:39:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:56 smithi204 ceph-mon[32357]: from='client.35822 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:39:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:56 smithi204 ceph-mon[32357]: osdmap e371: 8 total, 8 up, 7 in 2024-09-10T18:39:56.897 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:56.920 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:56.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:56.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:57.021 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:57.043 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:57.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:57.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:57.145 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:39:57.167 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:57.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:57.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:39:57.269 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:57.290 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:57.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:57.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:57.392 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:57.413 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:57.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:57.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:57.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:57.542 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:57.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:57.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:57.644 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:57.652 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:39:57.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[29291]: 6.6 scrub starts 2024-09-10T18:39:57.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[29291]: 1.10 deep-scrub starts 2024-09-10T18:39:57.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[29291]: 1.10 deep-scrub ok 2024-09-10T18:39:57.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:39:57.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:39:57.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[29291]: 5.2 scrub starts 2024-09-10T18:39:57.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[29291]: 5.2 scrub ok 2024-09-10T18:39:57.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[29291]: 2.0 scrub starts 2024-09-10T18:39:57.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[29291]: 2.0 scrub ok 2024-09-10T18:39:57.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[39220]: 6.6 scrub starts 2024-09-10T18:39:57.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[39220]: 1.10 deep-scrub starts 2024-09-10T18:39:57.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[39220]: 1.10 deep-scrub ok 2024-09-10T18:39:57.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:39:57.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:39:57.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[39220]: 5.2 scrub starts 2024-09-10T18:39:57.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[39220]: 5.2 scrub ok 2024-09-10T18:39:57.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[39220]: 2.0 scrub starts 2024-09-10T18:39:57.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:57 smithi028 ceph-mon[39220]: 2.0 scrub ok 2024-09-10T18:39:57.662 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:39:57.665 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:57.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:57.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:57.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:57.790 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:57.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:57.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:57 smithi204 ceph-mon[32357]: 6.6 scrub starts 2024-09-10T18:39:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:57 smithi204 ceph-mon[32357]: 1.10 deep-scrub starts 2024-09-10T18:39:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:57 smithi204 ceph-mon[32357]: 1.10 deep-scrub ok 2024-09-10T18:39:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:57 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:39:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:57 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:39:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:57 smithi204 ceph-mon[32357]: 5.2 scrub starts 2024-09-10T18:39:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:57 smithi204 ceph-mon[32357]: 5.2 scrub ok 2024-09-10T18:39:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:57 smithi204 ceph-mon[32357]: 2.0 scrub starts 2024-09-10T18:39:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:57 smithi204 ceph-mon[32357]: 2.0 scrub ok 2024-09-10T18:39:57.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:57.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:57.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:58.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:39:58.039 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:58.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:58.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:39:58.140 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:58.164 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:58.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:58.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:58.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:58.289 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:58.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:58.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:58.372 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:39:58.391 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:58.414 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:58.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:58.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:58.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:58.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:58.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:58.590 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":834,"stamp":"2024-09-10T18:39:57.367748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":481092,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":2265,"num_read_kb":2043,"num_write":745,"num_write_kb":450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":891815,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3380,"ondisk_log_size":3380,"up":486,"acting":486,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":4776156,"kb_used_data":6968,"kb_used_omap":17,"kb_used_meta":4768494,"kb_avail":651439908,"statfs":{"total":671965249536,"available":667074465792,"internally_reserved":0,"allocated":7135232,"data_stored":4494618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18329,"internal_metadata":4882937959},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":-273844,"num_write_kb":-8762945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11944,"num_bytes_recovered":-782303256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.213722"},"pg_stats":[{"pgid":"1.1d","version":"76'81","reported_seq":115,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.376832+0000","last_change":"2024-09-10T18:37:41.875430+0000","last_active":"2024-09-10T18:39:54.376832+0000","last_peered":"2024-09-10T18:39:54.376832+0000","last_clean":"2024-09-10T18:39:54.376832+0000","last_became_active":"2024-09-10T18:34:34.865894+0000","last_became_peered":"2024-09-10T18:34:34.865894+0000","last_unstale":"2024-09-10T18:39:54.376832+0000","last_undegraded":"2024-09-10T18:39:54.376832+0000","last_fullsized":"2024-09-10T18:39:54.376832+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:41.875334+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:41.875334+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015259258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":419,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.972375+0000","last_change":"2024-09-10T18:39:25.252966+0000","last_active":"2024-09-10T18:39:46.972375+0000","last_peered":"2024-09-10T18:39:46.972375+0000","last_clean":"2024-09-10T18:39:46.972375+0000","last_became_active":"2024-09-10T18:39:25.247259+0000","last_became_peered":"2024-09-10T18:39:25.247259+0000","last_unstale":"2024-09-10T18:39:46.972375+0000","last_undegraded":"2024-09-10T18:39:46.972375+0000","last_fullsized":"2024-09-10T18:39:46.972375+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:41.925519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:38:41.925519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002035645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"342'19","reported_seq":432,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.888080+0000","last_change":"2024-09-10T18:37:49.354831+0000","last_active":"2024-09-10T18:39:46.888080+0000","last_peered":"2024-09-10T18:39:46.888080+0000","last_clean":"2024-09-10T18:39:46.888080+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:39:46.888080+0000","last_undegraded":"2024-09-10T18:39:46.888080+0000","last_fullsized":"2024-09-10T18:39:46.888080+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'19","last_scrub_stamp":"2024-09-10T18:37:49.354781+0000","last_deep_scrub":"342'19","last_deep_scrub_stamp":"2024-09-10T18:37:49.354781+0000","last_clean_scrub_stamp":"2024-09-10T18:37:49.354781+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096597390000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":426,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.376309+0000","last_change":"2024-09-10T18:39:47.237682+0000","last_active":"2024-09-10T18:39:54.376309+0000","last_peered":"2024-09-10T18:39:54.376309+0000","last_clean":"2024-09-10T18:39:54.376309+0000","last_became_active":"2024-09-10T18:39:00.910095+0000","last_became_peered":"2024-09-10T18:39:00.910095+0000","last_unstale":"2024-09-10T18:39:54.376309+0000","last_undegraded":"2024-09-10T18:39:54.376309+0000","last_fullsized":"2024-09-10T18:39:54.376309+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:47.237622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:39:47.237622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:52.041545+0000","scrub_duration":0.0017128670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":415,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.888298+0000","last_change":"2024-09-10T18:39:00.890197+0000","last_active":"2024-09-10T18:39:46.888298+0000","last_peered":"2024-09-10T18:39:46.888298+0000","last_clean":"2024-09-10T18:39:46.888298+0000","last_became_active":"2024-09-10T18:39:00.889898+0000","last_became_peered":"2024-09-10T18:39:00.889898+0000","last_unstale":"2024-09-10T18:39:46.888298+0000","last_undegraded":"2024-09-10T18:39:46.888298+0000","last_fullsized":"2024-09-10T18:39:46.888298+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:44.663294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:38:44.663294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:56.194009+0000","scrub_duration":0.0051412879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'81","reported_seq":112,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.973450+0000","last_change":"2024-09-10T18:39:35.192518+0000","last_active":"2024-09-10T18:39:46.973450+0000","last_peered":"2024-09-10T18:39:46.973450+0000","last_clean":"2024-09-10T18:39:46.973450+0000","last_became_active":"2024-09-10T18:34:34.867388+0000","last_became_peered":"2024-09-10T18:34:34.867388+0000","last_unstale":"2024-09-10T18:39:46.973450+0000","last_undegraded":"2024-09-10T18:39:46.973450+0000","last_fullsized":"2024-09-10T18:39:46.973450+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:39:35.192426+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:39:35.192426+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:53.840593+0000","scrub_duration":0.013381291999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":392,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970766+0000","last_change":"2024-09-10T18:37:44.696143+0000","last_active":"2024-09-10T18:39:46.970766+0000","last_peered":"2024-09-10T18:39:46.970766+0000","last_clean":"2024-09-10T18:39:46.970766+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:39:46.970766+0000","last_undegraded":"2024-09-10T18:39:46.970766+0000","last_fullsized":"2024-09-10T18:39:46.970766+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:44.696052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:44.696052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.044397580999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":398,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.994768+0000","last_change":"2024-09-10T18:37:40.756679+0000","last_active":"2024-09-10T18:39:46.994768+0000","last_peered":"2024-09-10T18:39:46.994768+0000","last_clean":"2024-09-10T18:39:46.994768+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:39:46.994768+0000","last_undegraded":"2024-09-10T18:39:46.994768+0000","last_fullsized":"2024-09-10T18:39:46.994768+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:40.756589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:37:40.756589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073347990000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'34","reported_seq":468,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.976145+0000","last_change":"2024-09-10T18:39:33.886353+0000","last_active":"2024-09-10T18:39:46.976145+0000","last_peered":"2024-09-10T18:39:46.976145+0000","last_clean":"2024-09-10T18:39:46.976145+0000","last_became_active":"2024-09-10T18:39:25.237997+0000","last_became_peered":"2024-09-10T18:39:25.237997+0000","last_unstale":"2024-09-10T18:39:46.976145+0000","last_undegraded":"2024-09-10T18:39:46.976145+0000","last_fullsized":"2024-09-10T18:39:46.976145+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:39:33.886231+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:39:33.886231+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:52.460691+0000","scrub_duration":0.0027348149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":404,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.888025+0000","last_change":"2024-09-10T18:38:40.684653+0000","last_active":"2024-09-10T18:39:46.888025+0000","last_peered":"2024-09-10T18:39:46.888025+0000","last_clean":"2024-09-10T18:39:46.888025+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:39:46.888025+0000","last_undegraded":"2024-09-10T18:39:46.888025+0000","last_fullsized":"2024-09-10T18:39:46.888025+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:40.684556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:38:40.684556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:08.981709+0000","scrub_duration":0.003115544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.972084+0000","last_change":"2024-09-10T18:39:00.905360+0000","last_active":"2024-09-10T18:39:46.972084+0000","last_peered":"2024-09-10T18:39:46.972084+0000","last_clean":"2024-09-10T18:39:46.972084+0000","last_became_active":"2024-09-10T18:39:00.904380+0000","last_became_peered":"2024-09-10T18:39:00.904380+0000","last_unstale":"2024-09-10T18:39:46.972084+0000","last_undegraded":"2024-09-10T18:39:46.972084+0000","last_fullsized":"2024-09-10T18:39:46.972084+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:52.436889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_clean_scrub_stamp":"2024-09-10T18:37:52.436889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014302690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'15","reported_seq":441,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.889048+0000","last_change":"2024-09-10T18:39:25.257538+0000","last_active":"2024-09-10T18:39:46.889048+0000","last_peered":"2024-09-10T18:39:46.889048+0000","last_clean":"2024-09-10T18:39:46.889048+0000","last_became_active":"2024-09-10T18:39:25.237946+0000","last_became_peered":"2024-09-10T18:39:25.237946+0000","last_unstale":"2024-09-10T18:39:46.889048+0000","last_undegraded":"2024-09-10T18:39:46.889048+0000","last_fullsized":"2024-09-10T18:39:46.889048+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:37:05.365702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:05.365702+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029180819999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":33,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.972382+0000","last_change":"2024-09-10T18:39:25.240051+0000","last_active":"2024-09-10T18:39:46.972382+0000","last_peered":"2024-09-10T18:39:46.972382+0000","last_clean":"2024-09-10T18:39:46.972382+0000","last_became_active":"2024-09-10T18:39:25.239738+0000","last_became_peered":"2024-09-10T18:39:25.239738+0000","last_unstale":"2024-09-10T18:39:46.972382+0000","last_undegraded":"2024-09-10T18:39:46.972382+0000","last_fullsized":"2024-09-10T18:39:46.972382+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:48.906390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_clean_scrub_stamp":"2024-09-10T18:38:48.906390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:12.930584+0000","scrub_duration":0.005914379,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":32,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.376652+0000","last_change":"2024-09-10T18:39:38.022074+0000","last_active":"2024-09-10T18:39:54.376652+0000","last_peered":"2024-09-10T18:39:54.376652+0000","last_clean":"2024-09-10T18:39:54.376652+0000","last_became_active":"2024-09-10T18:39:25.255362+0000","last_became_peered":"2024-09-10T18:39:25.255362+0000","last_unstale":"2024-09-10T18:39:54.376652+0000","last_undegraded":"2024-09-10T18:39:54.376652+0000","last_fullsized":"2024-09-10T18:39:54.376652+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:38.021856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:39:38.021856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:01.112662+0000","scrub_duration":0.0083569290000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"76'81","reported_seq":178,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.977232+0000","last_change":"2024-09-10T18:39:35.857769+0000","last_active":"2024-09-10T18:39:46.977232+0000","last_peered":"2024-09-10T18:39:46.977232+0000","last_clean":"2024-09-10T18:39:46.977232+0000","last_became_active":"2024-09-10T18:34:34.863102+0000","last_became_peered":"2024-09-10T18:34:34.863102+0000","last_unstale":"2024-09-10T18:39:46.977232+0000","last_undegraded":"2024-09-10T18:39:46.977232+0000","last_fullsized":"2024-09-10T18:39:46.977232+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:39:35.857664+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:39:35.857664+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:40.007808+0000","scrub_duration":0.0078980279999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:49.974676+0000","last_change":"2024-09-10T18:39:49.974639+0000","last_active":"2024-09-10T18:39:49.974676+0000","last_peered":"2024-09-10T18:39:49.974676+0000","last_clean":"2024-09-10T18:39:49.974676+0000","last_became_active":"2024-09-10T18:39:00.915294+0000","last_became_peered":"2024-09-10T18:39:00.915294+0000","last_unstale":"2024-09-10T18:39:49.974676+0000","last_undegraded":"2024-09-10T18:39:49.974676+0000","last_fullsized":"2024-09-10T18:39:49.974676+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:49.974584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:39:49.974584+0000","last_clean_scrub_stamp":"2024-09-10T18:39:49.974584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:12.803914+0000","scrub_duration":0.001326858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":443,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.888651+0000","last_change":"2024-09-10T18:39:45.533685+0000","last_active":"2024-09-10T18:39:46.888651+0000","last_peered":"2024-09-10T18:39:46.888651+0000","last_clean":"2024-09-10T18:39:46.888651+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:39:46.888651+0000","last_undegraded":"2024-09-10T18:39:46.888651+0000","last_fullsized":"2024-09-10T18:39:46.888651+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'9","last_scrub_stamp":"2024-09-10T18:39:45.533620+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:39:45.533620+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:03.038647+0000","scrub_duration":0.0019598749999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":394,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971637+0000","last_change":"2024-09-10T18:37:36.885626+0000","last_active":"2024-09-10T18:39:46.971637+0000","last_peered":"2024-09-10T18:39:46.971637+0000","last_clean":"2024-09-10T18:39:46.971637+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:39:46.971637+0000","last_undegraded":"2024-09-10T18:39:46.971637+0000","last_fullsized":"2024-09-10T18:39:46.971637+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:36.885547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:36.885547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028632850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":23,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970750+0000","last_change":"2024-09-10T18:39:00.917283+0000","last_active":"2024-09-10T18:39:46.970750+0000","last_peered":"2024-09-10T18:39:46.970750+0000","last_clean":"2024-09-10T18:39:46.970750+0000","last_became_active":"2024-09-10T18:39:00.917007+0000","last_became_peered":"2024-09-10T18:39:00.917007+0000","last_unstale":"2024-09-10T18:39:46.970750+0000","last_undegraded":"2024-09-10T18:39:46.970750+0000","last_fullsized":"2024-09-10T18:39:46.970750+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:24.353951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:38:24.353951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'81","reported_seq":271,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970413+0000","last_change":"2024-09-10T18:39:37.865324+0000","last_active":"2024-09-10T18:39:46.970413+0000","last_peered":"2024-09-10T18:39:46.970413+0000","last_clean":"2024-09-10T18:39:46.970413+0000","last_became_active":"2024-09-10T18:39:25.241941+0000","last_became_peered":"2024-09-10T18:39:25.241941+0000","last_unstale":"2024-09-10T18:39:46.970413+0000","last_undegraded":"2024-09-10T18:39:46.970413+0000","last_fullsized":"2024-09-10T18:39:46.970413+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:39:37.865240+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:39:37.865240+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:56.185363+0000","scrub_duration":0.0038263540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":422,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.889539+0000","last_change":"2024-09-10T18:39:00.896583+0000","last_active":"2024-09-10T18:39:46.889539+0000","last_peered":"2024-09-10T18:39:46.889539+0000","last_clean":"2024-09-10T18:39:46.889539+0000","last_became_active":"2024-09-10T18:39:00.895860+0000","last_became_peered":"2024-09-10T18:39:00.895860+0000","last_unstale":"2024-09-10T18:39:46.889539+0000","last_undegraded":"2024-09-10T18:39:46.889539+0000","last_fullsized":"2024-09-10T18:39:46.889539+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:33.604672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:38:33.604672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003188331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.889015+0000","last_change":"2024-09-10T18:39:25.232266+0000","last_active":"2024-09-10T18:39:46.889015+0000","last_peered":"2024-09-10T18:39:46.889015+0000","last_clean":"2024-09-10T18:39:46.889015+0000","last_became_active":"2024-09-10T18:39:25.231931+0000","last_became_peered":"2024-09-10T18:39:25.231931+0000","last_unstale":"2024-09-10T18:39:46.889015+0000","last_undegraded":"2024-09-10T18:39:46.889015+0000","last_fullsized":"2024-09-10T18:39:46.889015+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:13.830325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:13.830325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"240'27","reported_seq":479,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.976003+0000","last_change":"2024-09-10T18:39:43.018460+0000","last_active":"2024-09-10T18:39:46.976003+0000","last_peered":"2024-09-10T18:39:46.976003+0000","last_clean":"2024-09-10T18:39:46.976003+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:39:46.976003+0000","last_undegraded":"2024-09-10T18:39:46.976003+0000","last_fullsized":"2024-09-10T18:39:46.976003+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:39:43.018375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:39:43.018375+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:57.990021+0000","scrub_duration":0.0061920930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":426,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.377076+0000","last_change":"2024-09-10T18:39:53.330296+0000","last_active":"2024-09-10T18:39:54.377076+0000","last_peered":"2024-09-10T18:39:54.377076+0000","last_clean":"2024-09-10T18:39:54.377076+0000","last_became_active":"2024-09-10T18:39:25.246541+0000","last_became_peered":"2024-09-10T18:39:25.246541+0000","last_unstale":"2024-09-10T18:39:54.377076+0000","last_undegraded":"2024-09-10T18:39:54.377076+0000","last_fullsized":"2024-09-10T18:39:54.377076+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:53.330231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:39:53.330231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:01.549113+0000","scrub_duration":0.0020390809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'81","reported_seq":27,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970660+0000","last_change":"2024-09-10T18:39:00.933989+0000","last_active":"2024-09-10T18:39:46.970660+0000","last_peered":"2024-09-10T18:39:46.970660+0000","last_clean":"2024-09-10T18:39:46.970660+0000","last_became_active":"2024-09-10T18:39:00.907303+0000","last_became_peered":"2024-09-10T18:39:00.907303+0000","last_unstale":"2024-09-10T18:39:46.970660+0000","last_undegraded":"2024-09-10T18:39:46.970660+0000","last_fullsized":"2024-09-10T18:39:46.970660+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:54.456947+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:54.456947+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":316358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":412,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.977075+0000","last_change":"2024-09-10T18:39:00.905552+0000","last_active":"2024-09-10T18:39:46.977075+0000","last_peered":"2024-09-10T18:39:46.977075+0000","last_clean":"2024-09-10T18:39:46.977075+0000","last_became_active":"2024-09-10T18:39:00.904769+0000","last_became_peered":"2024-09-10T18:39:00.904769+0000","last_unstale":"2024-09-10T18:39:46.977075+0000","last_undegraded":"2024-09-10T18:39:46.977075+0000","last_fullsized":"2024-09-10T18:39:46.977075+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:53.656232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:53.656232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014811193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":398,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.888025+0000","last_change":"2024-09-10T18:37:40.447566+0000","last_active":"2024-09-10T18:39:46.888025+0000","last_peered":"2024-09-10T18:39:46.888025+0000","last_clean":"2024-09-10T18:39:46.888025+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:39:46.888025+0000","last_undegraded":"2024-09-10T18:39:46.888025+0000","last_fullsized":"2024-09-10T18:39:46.888025+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:40.447489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:37:40.447489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030489810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'23","reported_seq":448,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.977027+0000","last_change":"2024-09-10T18:39:11.919606+0000","last_active":"2024-09-10T18:39:46.977027+0000","last_peered":"2024-09-10T18:39:46.977027+0000","last_clean":"2024-09-10T18:39:46.977027+0000","last_became_active":"2024-09-10T18:39:11.896502+0000","last_became_peered":"2024-09-10T18:39:11.896502+0000","last_unstale":"2024-09-10T18:39:46.977027+0000","last_undegraded":"2024-09-10T18:39:46.977027+0000","last_fullsized":"2024-09-10T18:39:46.977027+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'23","last_scrub_stamp":"2024-09-10T18:38:31.528491+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:38:31.528491+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018604077,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":400,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.888933+0000","last_change":"2024-09-10T18:37:36.407892+0000","last_active":"2024-09-10T18:39:46.888933+0000","last_peered":"2024-09-10T18:39:46.888933+0000","last_clean":"2024-09-10T18:39:46.888933+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:39:46.888933+0000","last_undegraded":"2024-09-10T18:39:46.888933+0000","last_fullsized":"2024-09-10T18:39:46.888933+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:36.407806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:08.266279+0000","last_clean_scrub_stamp":"2024-09-10T18:37:36.407806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037520880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'81","reported_seq":33,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.972298+0000","last_change":"2024-09-10T18:39:25.247750+0000","last_active":"2024-09-10T18:39:46.972298+0000","last_peered":"2024-09-10T18:39:46.972298+0000","last_clean":"2024-09-10T18:39:46.972298+0000","last_became_active":"2024-09-10T18:39:25.240472+0000","last_became_peered":"2024-09-10T18:39:25.240472+0000","last_unstale":"2024-09-10T18:39:46.972298+0000","last_undegraded":"2024-09-10T18:39:46.972298+0000","last_fullsized":"2024-09-10T18:39:46.972298+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:38:31.377926+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:38:31.377926+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":426,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.973329+0000","last_change":"2024-09-10T18:39:00.916263+0000","last_active":"2024-09-10T18:39:46.973329+0000","last_peered":"2024-09-10T18:39:46.973329+0000","last_clean":"2024-09-10T18:39:46.973329+0000","last_became_active":"2024-09-10T18:39:00.891321+0000","last_became_peered":"2024-09-10T18:39:00.891321+0000","last_unstale":"2024-09-10T18:39:46.973329+0000","last_undegraded":"2024-09-10T18:39:46.973329+0000","last_fullsized":"2024-09-10T18:39:46.973329+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:37:49.941880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:49.941880+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041872790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":420,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.376483+0000","last_change":"2024-09-10T18:38:58.997405+0000","last_active":"2024-09-10T18:39:54.376483+0000","last_peered":"2024-09-10T18:39:54.376483+0000","last_clean":"2024-09-10T18:39:54.376483+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:39:54.376483+0000","last_undegraded":"2024-09-10T18:39:54.376483+0000","last_fullsized":"2024-09-10T18:39:54.376483+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:58.997325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:38:58.997325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:59.883952+0000","scrub_duration":0.0054166630000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'25","reported_seq":304,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971781+0000","last_change":"2024-09-10T18:39:39.111380+0000","last_active":"2024-09-10T18:39:46.971781+0000","last_peered":"2024-09-10T18:39:46.971781+0000","last_clean":"2024-09-10T18:39:46.971781+0000","last_became_active":"2024-09-10T18:39:25.244435+0000","last_became_peered":"2024-09-10T18:39:25.244435+0000","last_unstale":"2024-09-10T18:39:46.971781+0000","last_undegraded":"2024-09-10T18:39:46.971781+0000","last_fullsized":"2024-09-10T18:39:46.971781+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:39:39.111295+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:39:39.111295+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:55.938277+0000","scrub_duration":0.003091229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"362'13","reported_seq":32,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.982710+0000","last_change":"2024-09-10T18:39:25.255493+0000","last_active":"2024-09-10T18:39:54.982710+0000","last_peered":"2024-09-10T18:39:54.982710+0000","last_clean":"2024-09-10T18:39:54.982710+0000","last_became_active":"2024-09-10T18:39:25.246551+0000","last_became_peered":"2024-09-10T18:39:25.246551+0000","last_unstale":"2024-09-10T18:39:54.982710+0000","last_undegraded":"2024-09-10T18:39:54.982710+0000","last_fullsized":"2024-09-10T18:39:54.982710+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-09-10T18:38:31.814076+0000","last_deep_scrub":"217'11","last_deep_scrub_stamp":"2024-09-10T18:30:03.161985+0000","last_clean_scrub_stamp":"2024-09-10T18:38:31.814076+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:01.159677+0000","scrub_duration":0.0020820050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":23,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971681+0000","last_change":"2024-09-10T18:39:00.909682+0000","last_active":"2024-09-10T18:39:46.971681+0000","last_peered":"2024-09-10T18:39:46.971681+0000","last_clean":"2024-09-10T18:39:46.971681+0000","last_became_active":"2024-09-10T18:39:00.909261+0000","last_became_peered":"2024-09-10T18:39:00.909261+0000","last_unstale":"2024-09-10T18:39:46.971681+0000","last_undegraded":"2024-09-10T18:39:46.971681+0000","last_fullsized":"2024-09-10T18:39:46.971681+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:38:28.333813+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:38:28.333813+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"240'34","reported_seq":471,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.377019+0000","last_change":"2024-09-10T18:38:49.087975+0000","last_active":"2024-09-10T18:39:54.377019+0000","last_peered":"2024-09-10T18:39:54.377019+0000","last_clean":"2024-09-10T18:39:54.377019+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:39:54.377019+0000","last_undegraded":"2024-09-10T18:39:54.377019+0000","last_fullsized":"2024-09-10T18:39:54.377019+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:38:49.087882+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:38:49.087882+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:06.890419+0000","scrub_duration":0.0078540599999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":401,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:49.902479+0000","last_change":"2024-09-10T18:39:49.902444+0000","last_active":"2024-09-10T18:39:49.902479+0000","last_peered":"2024-09-10T18:39:49.902479+0000","last_clean":"2024-09-10T18:39:49.902479+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:39:49.902479+0000","last_undegraded":"2024-09-10T18:39:49.902479+0000","last_fullsized":"2024-09-10T18:39:49.902479+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:49.902390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:39:49.902390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:52.698043+0000","scrub_duration":0.001638205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"362'11","reported_seq":673,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.356539+0000","last_change":"2024-09-10T18:39:50.576923+0000","last_active":"2024-09-10T18:39:54.356539+0000","last_peered":"2024-09-10T18:39:54.356539+0000","last_clean":"2024-09-10T18:39:54.356539+0000","last_became_active":"2024-09-10T18:39:25.239165+0000","last_became_peered":"2024-09-10T18:39:25.239165+0000","last_unstale":"2024-09-10T18:39:54.356539+0000","last_undegraded":"2024-09-10T18:39:54.356539+0000","last_fullsized":"2024-09-10T18:39:54.356539+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'11","last_scrub_stamp":"2024-09-10T18:39:50.576841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:39:50.576841+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:10.876250+0000","scrub_duration":0.0017290719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":43,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:52.004436+0000","last_change":"2024-09-10T18:39:52.004349+0000","last_active":"2024-09-10T18:39:52.004436+0000","last_peered":"2024-09-10T18:39:52.004436+0000","last_clean":"2024-09-10T18:39:52.004436+0000","last_became_active":"2024-09-10T18:39:25.256173+0000","last_became_peered":"2024-09-10T18:39:25.256173+0000","last_unstale":"2024-09-10T18:39:52.004436+0000","last_undegraded":"2024-09-10T18:39:52.004436+0000","last_fullsized":"2024-09-10T18:39:52.004436+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:52.004281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:39:52.004281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:12.590496+0000","scrub_duration":0.001700701,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":334,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.889351+0000","last_change":"2024-09-10T18:38:32.571227+0000","last_active":"2024-09-10T18:39:46.889351+0000","last_peered":"2024-09-10T18:39:46.889351+0000","last_clean":"2024-09-10T18:39:46.889351+0000","last_became_active":"2024-09-10T18:34:32.815839+0000","last_became_peered":"2024-09-10T18:34:32.815839+0000","last_unstale":"2024-09-10T18:39:46.889351+0000","last_undegraded":"2024-09-10T18:39:46.889351+0000","last_fullsized":"2024-09-10T18:39:46.889351+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:38:32.571179+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:38:32.571179+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:02.203193+0000","scrub_duration":0.0023252170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":467,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970133+0000","last_change":"2024-09-10T18:39:25.287698+0000","last_active":"2024-09-10T18:39:46.970133+0000","last_peered":"2024-09-10T18:39:46.970133+0000","last_clean":"2024-09-10T18:39:46.970133+0000","last_became_active":"2024-09-10T18:39:25.249915+0000","last_became_peered":"2024-09-10T18:39:25.249915+0000","last_unstale":"2024-09-10T18:39:46.970133+0000","last_undegraded":"2024-09-10T18:39:46.970133+0000","last_fullsized":"2024-09-10T18:39:46.970133+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-09-10T18:38:42.640928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:38:42.640928+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021375999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":401,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.994790+0000","last_change":"2024-09-10T18:39:25.237151+0000","last_active":"2024-09-10T18:39:46.994790+0000","last_peered":"2024-09-10T18:39:46.994790+0000","last_clean":"2024-09-10T18:39:46.994790+0000","last_became_active":"2024-09-10T18:39:25.236559+0000","last_became_peered":"2024-09-10T18:39:25.236559+0000","last_unstale":"2024-09-10T18:39:46.994790+0000","last_undegraded":"2024-09-10T18:39:46.994790+0000","last_fullsized":"2024-09-10T18:39:46.994790+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:34.546546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:15.765392+0000","last_clean_scrub_stamp":"2024-09-10T18:38:34.546546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:55.362631+0000","scrub_duration":0.0069621259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":394,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.888073+0000","last_change":"2024-09-10T18:37:35.460385+0000","last_active":"2024-09-10T18:39:46.888073+0000","last_peered":"2024-09-10T18:39:46.888073+0000","last_clean":"2024-09-10T18:39:46.888073+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:39:46.888073+0000","last_undegraded":"2024-09-10T18:39:46.888073+0000","last_fullsized":"2024-09-10T18:39:46.888073+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:35.460269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:37:35.460269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073608529999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":382,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.888200+0000","last_change":"2024-09-10T18:37:06.386024+0000","last_active":"2024-09-10T18:39:46.888200+0000","last_peered":"2024-09-10T18:39:46.888200+0000","last_clean":"2024-09-10T18:39:46.888200+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:39:46.888200+0000","last_undegraded":"2024-09-10T18:39:46.888200+0000","last_fullsized":"2024-09-10T18:39:46.888200+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:06.385944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:06.385944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003928651,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":326,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.976145+0000","last_change":"2024-09-10T18:39:00.895492+0000","last_active":"2024-09-10T18:39:46.976145+0000","last_peered":"2024-09-10T18:39:46.976145+0000","last_clean":"2024-09-10T18:39:46.976145+0000","last_became_active":"2024-09-10T18:39:00.894864+0000","last_became_peered":"2024-09-10T18:39:00.894864+0000","last_unstale":"2024-09-10T18:39:46.976145+0000","last_undegraded":"2024-09-10T18:39:46.976145+0000","last_fullsized":"2024-09-10T18:39:46.976145+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:38:26.438911+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:38:26.438911+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082642849999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"240'28","reported_seq":497,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.888402+0000","last_change":"2024-09-10T18:39:00.931907+0000","last_active":"2024-09-10T18:39:46.888402+0000","last_peered":"2024-09-10T18:39:46.888402+0000","last_clean":"2024-09-10T18:39:46.888402+0000","last_became_active":"2024-09-10T18:39:00.903341+0000","last_became_peered":"2024-09-10T18:39:00.903341+0000","last_unstale":"2024-09-10T18:39:46.888402+0000","last_undegraded":"2024-09-10T18:39:46.888402+0000","last_fullsized":"2024-09-10T18:39:46.888402+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'28","last_scrub_stamp":"2024-09-10T18:38:43.695366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:38:43.695366+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:08.902177+0000","scrub_duration":0.001723888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.377333+0000","last_change":"2024-09-10T18:39:25.238641+0000","last_active":"2024-09-10T18:39:54.377333+0000","last_peered":"2024-09-10T18:39:54.377333+0000","last_clean":"2024-09-10T18:39:54.377333+0000","last_became_active":"2024-09-10T18:39:25.238382+0000","last_became_peered":"2024-09-10T18:39:25.238382+0000","last_unstale":"2024-09-10T18:39:54.377333+0000","last_undegraded":"2024-09-10T18:39:54.377333+0000","last_fullsized":"2024-09-10T18:39:54.377333+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:38.833410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:38:38.833410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:06.517119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":616,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:49.981435+0000","last_change":"2024-09-10T18:38:54.835487+0000","last_active":"2024-09-10T18:39:49.981435+0000","last_peered":"2024-09-10T18:39:49.981435+0000","last_clean":"2024-09-10T18:39:49.981435+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:39:49.981435+0000","last_undegraded":"2024-09-10T18:39:49.981435+0000","last_fullsized":"2024-09-10T18:39:49.981435+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:38:54.835389+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:38:54.835389+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:20.864906+0000","scrub_duration":0.013144446000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":402,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970649+0000","last_change":"2024-09-10T18:39:36.159006+0000","last_active":"2024-09-10T18:39:46.970649+0000","last_peered":"2024-09-10T18:39:46.970649+0000","last_clean":"2024-09-10T18:39:46.970649+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:39:46.970649+0000","last_undegraded":"2024-09-10T18:39:46.970649+0000","last_fullsized":"2024-09-10T18:39:46.970649+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:36.158926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:39:36.158926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:37.733999+0000","scrub_duration":0.0082744229999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":406,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.646673+0000","last_change":"2024-09-10T18:39:37.653447+0000","last_active":"2024-09-10T18:39:54.646673+0000","last_peered":"2024-09-10T18:39:54.646673+0000","last_clean":"2024-09-10T18:39:54.646673+0000","last_became_active":"2024-09-10T18:34:32.818323+0000","last_became_peered":"2024-09-10T18:34:32.818323+0000","last_unstale":"2024-09-10T18:39:54.646673+0000","last_undegraded":"2024-09-10T18:39:54.646673+0000","last_fullsized":"2024-09-10T18:39:54.646673+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:39:37.653365+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:39:37.653365+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:04.821608+0000","scrub_duration":0.0070539720000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":497,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.376541+0000","last_change":"2024-09-10T18:39:00.931701+0000","last_active":"2024-09-10T18:39:54.376541+0000","last_peered":"2024-09-10T18:39:54.376541+0000","last_clean":"2024-09-10T18:39:54.376541+0000","last_became_active":"2024-09-10T18:39:00.897648+0000","last_became_peered":"2024-09-10T18:39:00.897648+0000","last_unstale":"2024-09-10T18:39:54.376541+0000","last_undegraded":"2024-09-10T18:39:54.376541+0000","last_fullsized":"2024-09-10T18:39:54.376541+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:38:33.190375+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:38:33.190375+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011596955000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:50.954377+0000","last_change":"2024-09-10T18:39:50.954314+0000","last_active":"2024-09-10T18:39:50.954377+0000","last_peered":"2024-09-10T18:39:50.954377+0000","last_clean":"2024-09-10T18:39:50.954377+0000","last_became_active":"2024-09-10T18:39:00.909066+0000","last_became_peered":"2024-09-10T18:39:00.909066+0000","last_unstale":"2024-09-10T18:39:50.954377+0000","last_undegraded":"2024-09-10T18:39:50.954377+0000","last_fullsized":"2024-09-10T18:39:50.954377+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:50.954250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:39:50.954250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:58.063541+0000","scrub_duration":0.001791281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"362'14","reported_seq":452,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.982641+0000","last_change":"2024-09-10T18:39:34.649121+0000","last_active":"2024-09-10T18:39:54.982641+0000","last_peered":"2024-09-10T18:39:54.982641+0000","last_clean":"2024-09-10T18:39:54.982641+0000","last_became_active":"2024-09-10T18:39:25.251198+0000","last_became_peered":"2024-09-10T18:39:25.251198+0000","last_unstale":"2024-09-10T18:39:54.982641+0000","last_undegraded":"2024-09-10T18:39:54.982641+0000","last_fullsized":"2024-09-10T18:39:54.982641+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'14","last_scrub_stamp":"2024-09-10T18:39:34.649041+0000","last_deep_scrub":"263'11","last_deep_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_clean_scrub_stamp":"2024-09-10T18:39:34.649041+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:53.641445+0000","scrub_duration":0.0080304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.969973+0000","last_change":"2024-09-10T18:39:25.235786+0000","last_active":"2024-09-10T18:39:46.969973+0000","last_peered":"2024-09-10T18:39:46.969973+0000","last_clean":"2024-09-10T18:39:46.969973+0000","last_became_active":"2024-09-10T18:39:25.235363+0000","last_became_peered":"2024-09-10T18:39:25.235363+0000","last_unstale":"2024-09-10T18:39:46.969973+0000","last_undegraded":"2024-09-10T18:39:46.969973+0000","last_fullsized":"2024-09-10T18:39:46.969973+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:28.790184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:38:28.790184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":416,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.646325+0000","last_change":"2024-09-10T18:39:53.620517+0000","last_active":"2024-09-10T18:39:54.646325+0000","last_peered":"2024-09-10T18:39:54.646325+0000","last_clean":"2024-09-10T18:39:54.646325+0000","last_became_active":"2024-09-10T18:39:25.249543+0000","last_became_peered":"2024-09-10T18:39:25.249543+0000","last_unstale":"2024-09-10T18:39:54.646325+0000","last_undegraded":"2024-09-10T18:39:54.646325+0000","last_fullsized":"2024-09-10T18:39:54.646325+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:39:53.620447+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:39:53.620447+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:19.982559+0000","scrub_duration":0.00162853,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"240'29","reported_seq":514,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.972125+0000","last_change":"2024-09-10T18:39:34.074832+0000","last_active":"2024-09-10T18:39:46.972125+0000","last_peered":"2024-09-10T18:39:46.972125+0000","last_clean":"2024-09-10T18:39:46.972125+0000","last_became_active":"2024-09-10T18:39:25.256454+0000","last_became_peered":"2024-09-10T18:39:25.256454+0000","last_unstale":"2024-09-10T18:39:46.972125+0000","last_undegraded":"2024-09-10T18:39:46.972125+0000","last_fullsized":"2024-09-10T18:39:46.972125+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'29","last_scrub_stamp":"2024-09-10T18:39:34.074748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:39:34.074748+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:48.806028+0000","scrub_duration":0.003408286,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":420,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.377164+0000","last_change":"2024-09-10T18:39:00.897691+0000","last_active":"2024-09-10T18:39:54.377164+0000","last_peered":"2024-09-10T18:39:54.377164+0000","last_clean":"2024-09-10T18:39:54.377164+0000","last_became_active":"2024-09-10T18:39:00.897360+0000","last_became_peered":"2024-09-10T18:39:00.897360+0000","last_unstale":"2024-09-10T18:39:54.377164+0000","last_undegraded":"2024-09-10T18:39:54.377164+0000","last_fullsized":"2024-09-10T18:39:54.377164+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:57.001003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:57.001003+0000","last_clean_scrub_stamp":"2024-09-10T18:37:57.001003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076916049999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":396,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.646239+0000","last_change":"2024-09-10T18:39:25.246391+0000","last_active":"2024-09-10T18:39:54.646239+0000","last_peered":"2024-09-10T18:39:54.646239+0000","last_clean":"2024-09-10T18:39:54.646239+0000","last_became_active":"2024-09-10T18:39:25.243348+0000","last_became_peered":"2024-09-10T18:39:25.243348+0000","last_unstale":"2024-09-10T18:39:54.646239+0000","last_undegraded":"2024-09-10T18:39:54.646239+0000","last_fullsized":"2024-09-10T18:39:54.646239+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:40.013491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:38:40.013491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012957038000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":401,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.376230+0000","last_change":"2024-09-10T18:37:36.899082+0000","last_active":"2024-09-10T18:39:54.376230+0000","last_peered":"2024-09-10T18:39:54.376230+0000","last_clean":"2024-09-10T18:39:54.376230+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:39:54.376230+0000","last_undegraded":"2024-09-10T18:39:54.376230+0000","last_fullsized":"2024-09-10T18:39:54.376230+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:36.898949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:37:36.898949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023913269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":33,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970834+0000","last_change":"2024-09-10T18:39:25.247311+0000","last_active":"2024-09-10T18:39:46.970834+0000","last_peered":"2024-09-10T18:39:46.970834+0000","last_clean":"2024-09-10T18:39:46.970834+0000","last_became_active":"2024-09-10T18:39:25.246989+0000","last_became_peered":"2024-09-10T18:39:25.246989+0000","last_unstale":"2024-09-10T18:39:46.970834+0000","last_undegraded":"2024-09-10T18:39:46.970834+0000","last_fullsized":"2024-09-10T18:39:46.970834+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:55.135191+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:55.135191+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"318'42","reported_seq":150,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.982522+0000","last_change":"2024-09-10T18:39:50.224997+0000","last_active":"2024-09-10T18:39:54.982522+0000","last_peered":"2024-09-10T18:39:54.982522+0000","last_clean":"2024-09-10T18:39:54.982522+0000","last_became_active":"2024-09-10T18:36:12.922665+0000","last_became_peered":"2024-09-10T18:36:12.922665+0000","last_unstale":"2024-09-10T18:39:54.982522+0000","last_undegraded":"2024-09-10T18:39:54.982522+0000","last_fullsized":"2024-09-10T18:39:54.982522+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'42","last_scrub_stamp":"2024-09-10T18:39:50.224877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:39:50.224877+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:05.427957+0000","scrub_duration":0.002927163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":405,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.889197+0000","last_change":"2024-09-10T18:39:25.243068+0000","last_active":"2024-09-10T18:39:46.889197+0000","last_peered":"2024-09-10T18:39:46.889197+0000","last_clean":"2024-09-10T18:39:46.889197+0000","last_became_active":"2024-09-10T18:39:25.242740+0000","last_became_peered":"2024-09-10T18:39:25.242740+0000","last_unstale":"2024-09-10T18:39:46.889197+0000","last_undegraded":"2024-09-10T18:39:46.889197+0000","last_fullsized":"2024-09-10T18:39:46.889197+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:01.274219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.274219+0000","last_clean_scrub_stamp":"2024-09-10T18:37:01.274219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059785009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":27,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971776+0000","last_change":"2024-09-10T18:39:00.950469+0000","last_active":"2024-09-10T18:39:46.971776+0000","last_peered":"2024-09-10T18:39:46.971776+0000","last_clean":"2024-09-10T18:39:46.971776+0000","last_became_active":"2024-09-10T18:39:00.909579+0000","last_became_peered":"2024-09-10T18:39:00.909579+0000","last_unstale":"2024-09-10T18:39:46.971776+0000","last_undegraded":"2024-09-10T18:39:46.971776+0000","last_fullsized":"2024-09-10T18:39:46.971776+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:37:48.497410+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:37:48.497410+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023324999999999999,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":22,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.376993+0000","last_change":"2024-09-10T18:39:25.263334+0000","last_active":"2024-09-10T18:39:54.376993+0000","last_peered":"2024-09-10T18:39:54.376993+0000","last_clean":"2024-09-10T18:39:54.376993+0000","last_became_active":"2024-09-10T18:39:25.255658+0000","last_became_peered":"2024-09-10T18:39:25.255658+0000","last_unstale":"2024-09-10T18:39:54.376993+0000","last_undegraded":"2024-09-10T18:39:54.376993+0000","last_fullsized":"2024-09-10T18:39:54.376993+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:56.658164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:56.658164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":375,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.977100+0000","last_change":"2024-09-10T18:37:37.700573+0000","last_active":"2024-09-10T18:39:46.977100+0000","last_peered":"2024-09-10T18:39:46.977100+0000","last_clean":"2024-09-10T18:39:46.977100+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:39:46.977100+0000","last_undegraded":"2024-09-10T18:39:46.977100+0000","last_fullsized":"2024-09-10T18:39:46.977100+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:37.700481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:37.700481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022681839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":459,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971492+0000","last_change":"2024-09-10T18:39:25.240421+0000","last_active":"2024-09-10T18:39:46.971492+0000","last_peered":"2024-09-10T18:39:46.971492+0000","last_clean":"2024-09-10T18:39:46.971492+0000","last_became_active":"2024-09-10T18:39:25.240048+0000","last_became_peered":"2024-09-10T18:39:25.240048+0000","last_unstale":"2024-09-10T18:39:46.971492+0000","last_undegraded":"2024-09-10T18:39:46.971492+0000","last_fullsized":"2024-09-10T18:39:46.971492+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:38:24.997586+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:38:24.997586+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080711930000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"240'17","reported_seq":507,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.377122+0000","last_change":"2024-09-10T18:39:25.278372+0000","last_active":"2024-09-10T18:39:54.377122+0000","last_peered":"2024-09-10T18:39:54.377122+0000","last_clean":"2024-09-10T18:39:54.377122+0000","last_became_active":"2024-09-10T18:39:25.263098+0000","last_became_peered":"2024-09-10T18:39:25.263098+0000","last_unstale":"2024-09-10T18:39:54.377122+0000","last_undegraded":"2024-09-10T18:39:54.377122+0000","last_fullsized":"2024-09-10T18:39:54.377122+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'17","last_scrub_stamp":"2024-09-10T18:38:29.169751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:38:29.169751+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063612620000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":417,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.376026+0000","last_change":"2024-09-10T18:37:46.930362+0000","last_active":"2024-09-10T18:39:54.376026+0000","last_peered":"2024-09-10T18:39:54.376026+0000","last_clean":"2024-09-10T18:39:54.376026+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:39:54.376026+0000","last_undegraded":"2024-09-10T18:39:54.376026+0000","last_fullsized":"2024-09-10T18:39:54.376026+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:46.930282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:15.141982+0000","last_clean_scrub_stamp":"2024-09-10T18:37:46.930282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067994529999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":451,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970245+0000","last_change":"2024-09-10T18:39:25.267255+0000","last_active":"2024-09-10T18:39:46.970245+0000","last_peered":"2024-09-10T18:39:46.970245+0000","last_clean":"2024-09-10T18:39:46.970245+0000","last_became_active":"2024-09-10T18:39:25.249566+0000","last_became_peered":"2024-09-10T18:39:25.249566+0000","last_unstale":"2024-09-10T18:39:46.970245+0000","last_undegraded":"2024-09-10T18:39:46.970245+0000","last_fullsized":"2024-09-10T18:39:46.970245+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:38:30.394331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:38:30.394331+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27967430799999998,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":444,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:50.988162+0000","last_change":"2024-09-10T18:39:50.988127+0000","last_active":"2024-09-10T18:39:50.988162+0000","last_peered":"2024-09-10T18:39:50.988162+0000","last_clean":"2024-09-10T18:39:50.988162+0000","last_became_active":"2024-09-10T18:39:00.910574+0000","last_became_peered":"2024-09-10T18:39:00.910574+0000","last_unstale":"2024-09-10T18:39:50.988162+0000","last_undegraded":"2024-09-10T18:39:50.988162+0000","last_fullsized":"2024-09-10T18:39:50.988162+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:50.988074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:39:50.988074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:00.120924+0000","scrub_duration":0.0012388569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":402,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971081+0000","last_change":"2024-09-10T18:39:46.257970+0000","last_active":"2024-09-10T18:39:46.971081+0000","last_peered":"2024-09-10T18:39:46.971081+0000","last_clean":"2024-09-10T18:39:46.971081+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:39:46.971081+0000","last_undegraded":"2024-09-10T18:39:46.971081+0000","last_fullsized":"2024-09-10T18:39:46.971081+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:46.257902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:39:46.257902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:02.919086+0000","scrub_duration":0.0015239800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":330,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.376804+0000","last_change":"2024-09-10T18:39:32.968445+0000","last_active":"2024-09-10T18:39:54.376804+0000","last_peered":"2024-09-10T18:39:54.376804+0000","last_clean":"2024-09-10T18:39:54.376804+0000","last_became_active":"2024-09-10T18:34:00.165384+0000","last_became_peered":"2024-09-10T18:34:00.165384+0000","last_unstale":"2024-09-10T18:39:54.376804+0000","last_undegraded":"2024-09-10T18:39:54.376804+0000","last_fullsized":"2024-09-10T18:39:54.376804+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:39:32.968354+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:39:32.968354+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:50.456757+0000","scrub_duration":0.017467021999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":298,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.377022+0000","last_change":"2024-09-10T18:39:25.278979+0000","last_active":"2024-09-10T18:39:54.377022+0000","last_peered":"2024-09-10T18:39:54.377022+0000","last_clean":"2024-09-10T18:39:54.377022+0000","last_became_active":"2024-09-10T18:39:25.263465+0000","last_became_peered":"2024-09-10T18:39:25.263465+0000","last_unstale":"2024-09-10T18:39:54.377022+0000","last_undegraded":"2024-09-10T18:39:54.377022+0000","last_fullsized":"2024-09-10T18:39:54.377022+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-09-10T18:38:40.888821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:38:40.888821+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.138665703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":31,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:49.543192+0000","last_change":"2024-09-10T18:39:49.543154+0000","last_active":"2024-09-10T18:39:49.543192+0000","last_peered":"2024-09-10T18:39:49.543192+0000","last_clean":"2024-09-10T18:39:49.543192+0000","last_became_active":"2024-09-10T18:39:25.239068+0000","last_became_peered":"2024-09-10T18:39:25.239068+0000","last_unstale":"2024-09-10T18:39:49.543192+0000","last_undegraded":"2024-09-10T18:39:49.543192+0000","last_fullsized":"2024-09-10T18:39:49.543192+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:49.543095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:39:49.543095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:56.463009+0000","scrub_duration":0.0012870239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":1514,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.296623+0000","last_change":"2024-09-10T18:39:25.270069+0000","last_active":"2024-09-10T18:39:54.296623+0000","last_peered":"2024-09-10T18:39:54.296623+0000","last_clean":"2024-09-10T18:39:54.296623+0000","last_became_active":"2024-09-10T18:39:25.245563+0000","last_became_peered":"2024-09-10T18:39:25.245563+0000","last_unstale":"2024-09-10T18:39:54.296623+0000","last_undegraded":"2024-09-10T18:39:54.296623+0000","last_fullsized":"2024-09-10T18:39:54.296623+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:37:01.094063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:37:01.094063+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099775509999999994,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1094,"num_read_kb":1089,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":33,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.972453+0000","last_change":"2024-09-10T18:39:25.256405+0000","last_active":"2024-09-10T18:39:46.972453+0000","last_peered":"2024-09-10T18:39:46.972453+0000","last_clean":"2024-09-10T18:39:46.972453+0000","last_became_active":"2024-09-10T18:39:25.255881+0000","last_became_peered":"2024-09-10T18:39:25.255881+0000","last_unstale":"2024-09-10T18:39:46.972453+0000","last_undegraded":"2024-09-10T18:39:46.972453+0000","last_fullsized":"2024-09-10T18:39:46.972453+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:45.155542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.170272+0000","last_clean_scrub_stamp":"2024-09-10T18:38:45.155542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":414,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.376535+0000","last_change":"2024-09-10T18:39:00.897823+0000","last_active":"2024-09-10T18:39:54.376535+0000","last_peered":"2024-09-10T18:39:54.376535+0000","last_clean":"2024-09-10T18:39:54.376535+0000","last_became_active":"2024-09-10T18:39:00.897525+0000","last_became_peered":"2024-09-10T18:39:00.897525+0000","last_unstale":"2024-09-10T18:39:54.376535+0000","last_undegraded":"2024-09-10T18:39:54.376535+0000","last_fullsized":"2024-09-10T18:39:54.376535+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:51.000352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:51.000352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005761296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"240'26","reported_seq":403,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.646594+0000","last_change":"2024-09-10T18:39:42.722994+0000","last_active":"2024-09-10T18:39:54.646594+0000","last_peered":"2024-09-10T18:39:54.646594+0000","last_clean":"2024-09-10T18:39:54.646594+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:39:54.646594+0000","last_undegraded":"2024-09-10T18:39:54.646594+0000","last_fullsized":"2024-09-10T18:39:54.646594+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'26","last_scrub_stamp":"2024-09-10T18:39:42.722913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:39:42.722913+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:11.670047+0000","scrub_duration":0.016138739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":428,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:48.899712+0000","last_change":"2024-09-10T18:39:48.899653+0000","last_active":"2024-09-10T18:39:48.899712+0000","last_peered":"2024-09-10T18:39:48.899712+0000","last_clean":"2024-09-10T18:39:48.899712+0000","last_became_active":"2024-09-10T18:39:00.894778+0000","last_became_peered":"2024-09-10T18:39:00.894778+0000","last_unstale":"2024-09-10T18:39:48.899712+0000","last_undegraded":"2024-09-10T18:39:48.899712+0000","last_fullsized":"2024-09-10T18:39:48.899712+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:48.899587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:39:48.899587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:14.603883+0000","scrub_duration":0.001397414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":468,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.973063+0000","last_change":"2024-09-10T18:39:25.252266+0000","last_active":"2024-09-10T18:39:46.973063+0000","last_peered":"2024-09-10T18:39:46.973063+0000","last_clean":"2024-09-10T18:39:46.973063+0000","last_became_active":"2024-09-10T18:39:25.251895+0000","last_became_peered":"2024-09-10T18:39:25.251895+0000","last_unstale":"2024-09-10T18:39:46.973063+0000","last_undegraded":"2024-09-10T18:39:46.973063+0000","last_fullsized":"2024-09-10T18:39:46.973063+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:11.982173+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:11.982173+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001824406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":417,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.973395+0000","last_change":"2024-09-10T18:39:32.185264+0000","last_active":"2024-09-10T18:39:46.973395+0000","last_peered":"2024-09-10T18:39:46.973395+0000","last_clean":"2024-09-10T18:39:46.973395+0000","last_became_active":"2024-09-10T18:39:00.890829+0000","last_became_peered":"2024-09-10T18:39:00.890829+0000","last_unstale":"2024-09-10T18:39:46.973395+0000","last_undegraded":"2024-09-10T18:39:46.973395+0000","last_fullsized":"2024-09-10T18:39:46.973395+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:58.015470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:36:58.015470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043498190000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":23,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.972576+0000","last_change":"2024-09-10T18:39:00.901002+0000","last_active":"2024-09-10T18:39:46.972576+0000","last_peered":"2024-09-10T18:39:46.972576+0000","last_clean":"2024-09-10T18:39:46.972576+0000","last_became_active":"2024-09-10T18:39:00.900703+0000","last_became_peered":"2024-09-10T18:39:00.900703+0000","last_unstale":"2024-09-10T18:39:46.972576+0000","last_undegraded":"2024-09-10T18:39:46.972576+0000","last_fullsized":"2024-09-10T18:39:46.972576+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:59.825361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:38:59.825361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:14.349923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"76'81","reported_seq":376,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.377417+0000","last_change":"2024-09-10T18:39:54.377371+0000","last_active":"2024-09-10T18:39:54.377417+0000","last_peered":"2024-09-10T18:39:54.377417+0000","last_clean":"2024-09-10T18:39:54.377417+0000","last_became_active":"2024-09-10T18:39:25.263369+0000","last_became_peered":"2024-09-10T18:39:25.263369+0000","last_unstale":"2024-09-10T18:39:54.377417+0000","last_undegraded":"2024-09-10T18:39:54.377417+0000","last_fullsized":"2024-09-10T18:39:54.377417+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:39:54.377316+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:39:54.377316+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:01.556500+0000","scrub_duration":0.0027816300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"283'24","reported_seq":442,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:48.523169+0000","last_change":"2024-09-10T18:39:48.523096+0000","last_active":"2024-09-10T18:39:48.523169+0000","last_peered":"2024-09-10T18:39:48.523169+0000","last_clean":"2024-09-10T18:39:48.523169+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:39:48.523169+0000","last_undegraded":"2024-09-10T18:39:48.523169+0000","last_fullsized":"2024-09-10T18:39:48.523169+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'24","last_scrub_stamp":"2024-09-10T18:39:48.523027+0000","last_deep_scrub":"283'24","last_deep_scrub_stamp":"2024-09-10T18:34:19.404713+0000","last_clean_scrub_stamp":"2024-09-10T18:39:48.523027+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:53.574195+0000","scrub_duration":0.00247957,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":401,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.888559+0000","last_change":"2024-09-10T18:39:36.476434+0000","last_active":"2024-09-10T18:39:46.888559+0000","last_peered":"2024-09-10T18:39:46.888559+0000","last_clean":"2024-09-10T18:39:46.888559+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:39:46.888559+0000","last_undegraded":"2024-09-10T18:39:46.888559+0000","last_fullsized":"2024-09-10T18:39:46.888559+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:36.476221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:39:36.476221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:36.807591+0000","scrub_duration":0.0075989880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":33,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.976033+0000","last_change":"2024-09-10T18:39:54.975996+0000","last_active":"2024-09-10T18:39:54.976033+0000","last_peered":"2024-09-10T18:39:54.976033+0000","last_clean":"2024-09-10T18:39:54.976033+0000","last_became_active":"2024-09-10T18:39:00.907231+0000","last_became_peered":"2024-09-10T18:39:00.907231+0000","last_unstale":"2024-09-10T18:39:54.976033+0000","last_undegraded":"2024-09-10T18:39:54.976033+0000","last_fullsized":"2024-09-10T18:39:54.976033+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:54.975936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:39:54.975936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:19.803658+0000","scrub_duration":0.0013369269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":422,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.976234+0000","last_change":"2024-09-10T18:39:41.925032+0000","last_active":"2024-09-10T18:39:46.976234+0000","last_peered":"2024-09-10T18:39:46.976234+0000","last_clean":"2024-09-10T18:39:46.976234+0000","last_became_active":"2024-09-10T18:39:00.894963+0000","last_became_peered":"2024-09-10T18:39:00.894963+0000","last_unstale":"2024-09-10T18:39:46.976234+0000","last_undegraded":"2024-09-10T18:39:46.976234+0000","last_fullsized":"2024-09-10T18:39:46.976234+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:41.924977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:39:41.924977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:00.164410+0000","scrub_duration":0.011688601999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":23,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971009+0000","last_change":"2024-09-10T18:39:00.909522+0000","last_active":"2024-09-10T18:39:46.971009+0000","last_peered":"2024-09-10T18:39:46.971009+0000","last_clean":"2024-09-10T18:39:46.971009+0000","last_became_active":"2024-09-10T18:39:00.909151+0000","last_became_peered":"2024-09-10T18:39:00.909151+0000","last_unstale":"2024-09-10T18:39:46.971009+0000","last_undegraded":"2024-09-10T18:39:46.971009+0000","last_fullsized":"2024-09-10T18:39:46.971009+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:40.540266+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:40.540266+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"342'39","reported_seq":509,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.889286+0000","last_change":"2024-09-10T18:38:43.363005+0000","last_active":"2024-09-10T18:39:46.889286+0000","last_peered":"2024-09-10T18:39:46.889286+0000","last_clean":"2024-09-10T18:39:46.889286+0000","last_became_active":"2024-09-10T18:38:43.333186+0000","last_became_peered":"2024-09-10T18:38:43.333186+0000","last_unstale":"2024-09-10T18:39:46.889286+0000","last_undegraded":"2024-09-10T18:39:46.889286+0000","last_fullsized":"2024-09-10T18:39:46.889286+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'39","last_scrub_stamp":"2024-09-10T18:37:53.386563+0000","last_deep_scrub":"342'39","last_deep_scrub_stamp":"2024-09-10T18:37:53.386563+0000","last_clean_scrub_stamp":"2024-09-10T18:37:53.386563+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.031998117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":374,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.646532+0000","last_change":"2024-09-10T18:39:36.690146+0000","last_active":"2024-09-10T18:39:54.646532+0000","last_peered":"2024-09-10T18:39:54.646532+0000","last_clean":"2024-09-10T18:39:54.646532+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:39:54.646532+0000","last_undegraded":"2024-09-10T18:39:54.646532+0000","last_fullsized":"2024-09-10T18:39:54.646532+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:39:36.690065+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:39:36.690065+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:41.839230+0000","scrub_duration":0.0044532160000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":416,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.376910+0000","last_change":"2024-09-10T18:39:25.251245+0000","last_active":"2024-09-10T18:39:54.376910+0000","last_peered":"2024-09-10T18:39:54.376910+0000","last_clean":"2024-09-10T18:39:54.376910+0000","last_became_active":"2024-09-10T18:39:25.249337+0000","last_became_peered":"2024-09-10T18:39:25.249337+0000","last_unstale":"2024-09-10T18:39:54.376910+0000","last_undegraded":"2024-09-10T18:39:54.376910+0000","last_fullsized":"2024-09-10T18:39:54.376910+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:38.132080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:38:38.132080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:55.714458+0000","scrub_duration":0.010439053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":432,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.975466+0000","last_change":"2024-09-10T18:39:35.143470+0000","last_active":"2024-09-10T18:39:46.975466+0000","last_peered":"2024-09-10T18:39:46.975466+0000","last_clean":"2024-09-10T18:39:46.975466+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:39:46.975466+0000","last_undegraded":"2024-09-10T18:39:46.975466+0000","last_fullsized":"2024-09-10T18:39:46.975466+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:35.143047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:39:35.143047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:37.843944+0000","scrub_duration":0.0070909090000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":456,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.973060+0000","last_change":"2024-09-10T18:38:32.940690+0000","last_active":"2024-09-10T18:39:46.973060+0000","last_peered":"2024-09-10T18:39:46.973060+0000","last_clean":"2024-09-10T18:39:46.973060+0000","last_became_active":"2024-09-10T18:34:32.827174+0000","last_became_peered":"2024-09-10T18:34:32.827174+0000","last_unstale":"2024-09-10T18:39:46.973060+0000","last_undegraded":"2024-09-10T18:39:46.973060+0000","last_fullsized":"2024-09-10T18:39:46.973060+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:38:32.940591+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:38:32.940591+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:55.958331+0000","scrub_duration":0.0099489020000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"240'25","reported_seq":156,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.973193+0000","last_change":"2024-09-10T18:37:14.054480+0000","last_active":"2024-09-10T18:39:46.973193+0000","last_peered":"2024-09-10T18:39:46.973193+0000","last_clean":"2024-09-10T18:39:46.973193+0000","last_became_active":"2024-09-10T18:32:51.118251+0000","last_became_peered":"2024-09-10T18:32:51.118251+0000","last_unstale":"2024-09-10T18:39:46.973193+0000","last_undegraded":"2024-09-10T18:39:46.973193+0000","last_fullsized":"2024-09-10T18:39:46.973193+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:37:14.054170+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:37:14.054170+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019668199000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":401,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.975721+0000","last_change":"2024-09-10T18:39:25.252292+0000","last_active":"2024-09-10T18:39:46.975721+0000","last_peered":"2024-09-10T18:39:46.975721+0000","last_clean":"2024-09-10T18:39:46.975721+0000","last_became_active":"2024-09-10T18:39:25.251419+0000","last_became_peered":"2024-09-10T18:39:25.251419+0000","last_unstale":"2024-09-10T18:39:46.975721+0000","last_undegraded":"2024-09-10T18:39:46.975721+0000","last_fullsized":"2024-09-10T18:39:46.975721+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:48.662777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:13.324556+0000","last_clean_scrub_stamp":"2024-09-10T18:38:48.662777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056001039999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"357'11","reported_seq":693,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.354811+0000","last_change":"2024-09-10T18:39:00.933886+0000","last_active":"2024-09-10T18:39:54.354811+0000","last_peered":"2024-09-10T18:39:54.354811+0000","last_clean":"2024-09-10T18:39:54.354811+0000","last_became_active":"2024-09-10T18:39:00.909747+0000","last_became_peered":"2024-09-10T18:39:00.909747+0000","last_unstale":"2024-09-10T18:39:54.354811+0000","last_undegraded":"2024-09-10T18:39:54.354811+0000","last_fullsized":"2024-09-10T18:39:54.354811+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'9","last_scrub_stamp":"2024-09-10T18:38:32.143157+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:38:32.143157+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055150269999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":385,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.889539+0000","last_change":"2024-09-10T18:37:43.480404+0000","last_active":"2024-09-10T18:39:46.889539+0000","last_peered":"2024-09-10T18:39:46.889539+0000","last_clean":"2024-09-10T18:39:46.889539+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:39:46.889539+0000","last_undegraded":"2024-09-10T18:39:46.889539+0000","last_fullsized":"2024-09-10T18:39:46.889539+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:43.480325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:16.088718+0000","last_clean_scrub_stamp":"2024-09-10T18:37:43.480325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011972396999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"76'81","reported_seq":128,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.889429+0000","last_change":"2024-09-10T18:37:51.400051+0000","last_active":"2024-09-10T18:39:46.889429+0000","last_peered":"2024-09-10T18:39:46.889429+0000","last_clean":"2024-09-10T18:39:46.889429+0000","last_became_active":"2024-09-10T18:34:32.818251+0000","last_became_peered":"2024-09-10T18:34:32.818251+0000","last_unstale":"2024-09-10T18:39:46.889429+0000","last_undegraded":"2024-09-10T18:39:46.889429+0000","last_fullsized":"2024-09-10T18:39:46.889429+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:51.400000+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:51.400000+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013808830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"302'19","reported_seq":371,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.376832+0000","last_change":"2024-09-10T18:39:40.061732+0000","last_active":"2024-09-10T18:39:54.376832+0000","last_peered":"2024-09-10T18:39:54.376832+0000","last_clean":"2024-09-10T18:39:54.376832+0000","last_became_active":"2024-09-10T18:39:25.240389+0000","last_became_peered":"2024-09-10T18:39:25.240389+0000","last_unstale":"2024-09-10T18:39:54.376832+0000","last_undegraded":"2024-09-10T18:39:54.376832+0000","last_fullsized":"2024-09-10T18:39:54.376832+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'19","last_scrub_stamp":"2024-09-10T18:39:40.061643+0000","last_deep_scrub":"302'19","last_deep_scrub_stamp":"2024-09-10T18:35:18.836131+0000","last_clean_scrub_stamp":"2024-09-10T18:39:40.061643+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:53.085543+0000","scrub_duration":0.0059734920000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"265'2","reported_seq":37,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:53.939103+0000","last_change":"2024-09-10T18:39:53.939060+0000","last_active":"2024-09-10T18:39:53.939103+0000","last_peered":"2024-09-10T18:39:53.939103+0000","last_clean":"2024-09-10T18:39:53.939103+0000","last_became_active":"2024-09-10T18:39:00.904967+0000","last_became_peered":"2024-09-10T18:39:00.904967+0000","last_unstale":"2024-09-10T18:39:53.939103+0000","last_undegraded":"2024-09-10T18:39:53.939103+0000","last_fullsized":"2024-09-10T18:39:53.939103+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'2","last_scrub_stamp":"2024-09-10T18:39:53.938998+0000","last_deep_scrub":"265'2","last_deep_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_clean_scrub_stamp":"2024-09-10T18:39:53.938998+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:13.505976+0000","scrub_duration":0.001572473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":420,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.975728+0000","last_change":"2024-09-10T18:38:29.098961+0000","last_active":"2024-09-10T18:39:46.975728+0000","last_peered":"2024-09-10T18:39:46.975728+0000","last_clean":"2024-09-10T18:39:46.975728+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:39:46.975728+0000","last_undegraded":"2024-09-10T18:39:46.975728+0000","last_fullsized":"2024-09-10T18:39:46.975728+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:29.098879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:38:29.098879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022192790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.972938+0000","last_change":"2024-09-10T18:39:25.237468+0000","last_active":"2024-09-10T18:39:46.972938+0000","last_peered":"2024-09-10T18:39:46.972938+0000","last_clean":"2024-09-10T18:39:46.972938+0000","last_became_active":"2024-09-10T18:39:25.237150+0000","last_became_peered":"2024-09-10T18:39:25.237150+0000","last_unstale":"2024-09-10T18:39:46.972938+0000","last_undegraded":"2024-09-10T18:39:46.972938+0000","last_fullsized":"2024-09-10T18:39:46.972938+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:27.812952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:38:27.812952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"76'81","reported_seq":405,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.646696+0000","last_change":"2024-09-10T18:39:00.922386+0000","last_active":"2024-09-10T18:39:54.646696+0000","last_peered":"2024-09-10T18:39:54.646696+0000","last_clean":"2024-09-10T18:39:54.646696+0000","last_became_active":"2024-09-10T18:39:00.921868+0000","last_became_peered":"2024-09-10T18:39:00.921868+0000","last_unstale":"2024-09-10T18:39:54.646696+0000","last_undegraded":"2024-09-10T18:39:54.646696+0000","last_fullsized":"2024-09-10T18:39:54.646696+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:38:35.028678+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:34:15.938079+0000","last_clean_scrub_stamp":"2024-09-10T18:38:35.028678+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022136859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"240'44","reported_seq":491,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.646371+0000","last_change":"2024-09-10T18:39:38.641783+0000","last_active":"2024-09-10T18:39:54.646371+0000","last_peered":"2024-09-10T18:39:54.646371+0000","last_clean":"2024-09-10T18:39:54.646371+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:39:54.646371+0000","last_undegraded":"2024-09-10T18:39:54.646371+0000","last_fullsized":"2024-09-10T18:39:54.646371+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'44","last_scrub_stamp":"2024-09-10T18:39:38.641706+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:39:38.641706+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:44.925193+0000","scrub_duration":0.0070197519999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":428,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.973535+0000","last_change":"2024-09-10T18:39:27.158252+0000","last_active":"2024-09-10T18:39:46.973535+0000","last_peered":"2024-09-10T18:39:46.973535+0000","last_clean":"2024-09-10T18:39:46.973535+0000","last_became_active":"2024-09-10T18:39:00.895121+0000","last_became_peered":"2024-09-10T18:39:00.895121+0000","last_unstale":"2024-09-10T18:39:46.973535+0000","last_undegraded":"2024-09-10T18:39:46.973535+0000","last_fullsized":"2024-09-10T18:39:46.973535+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:36:56.002946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:36:56.002946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058026170000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":464,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:48.035793+0000","last_change":"2024-09-10T18:39:48.035749+0000","last_active":"2024-09-10T18:39:48.035793+0000","last_peered":"2024-09-10T18:39:48.035793+0000","last_clean":"2024-09-10T18:39:48.035793+0000","last_became_active":"2024-09-10T18:39:25.240291+0000","last_became_peered":"2024-09-10T18:39:25.240291+0000","last_unstale":"2024-09-10T18:39:48.035793+0000","last_undegraded":"2024-09-10T18:39:48.035793+0000","last_fullsized":"2024-09-10T18:39:48.035793+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:48.035691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:34:21.447291+0000","last_clean_scrub_stamp":"2024-09-10T18:39:48.035691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:05.227840+0000","scrub_duration":0.0019947799999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":405,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:47.292934+0000","last_change":"2024-09-10T18:39:47.292863+0000","last_active":"2024-09-10T18:39:47.292934+0000","last_peered":"2024-09-10T18:39:47.292934+0000","last_clean":"2024-09-10T18:39:47.292934+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:39:47.292934+0000","last_undegraded":"2024-09-10T18:39:47.292934+0000","last_fullsized":"2024-09-10T18:39:47.292934+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:47.292801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:04.058646+0000","last_clean_scrub_stamp":"2024-09-10T18:39:47.292801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:49.446186+0000","scrub_duration":0.001761461,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"76'81","reported_seq":346,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.976451+0000","last_change":"2024-09-10T18:39:25.255653+0000","last_active":"2024-09-10T18:39:46.976451+0000","last_peered":"2024-09-10T18:39:46.976451+0000","last_clean":"2024-09-10T18:39:46.976451+0000","last_became_active":"2024-09-10T18:39:25.242996+0000","last_became_peered":"2024-09-10T18:39:25.242996+0000","last_unstale":"2024-09-10T18:39:46.976451+0000","last_undegraded":"2024-09-10T18:39:46.976451+0000","last_fullsized":"2024-09-10T18:39:46.976451+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:39:00.859607+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:39:00.859607+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:07.896692+0000","scrub_duration":0.0018973250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"301'23","reported_seq":436,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.888296+0000","last_change":"2024-09-10T18:37:42.463354+0000","last_active":"2024-09-10T18:39:46.888296+0000","last_peered":"2024-09-10T18:39:46.888296+0000","last_clean":"2024-09-10T18:39:46.888296+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:39:46.888296+0000","last_undegraded":"2024-09-10T18:39:46.888296+0000","last_fullsized":"2024-09-10T18:39:46.888296+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'23","last_scrub_stamp":"2024-09-10T18:37:42.463250+0000","last_deep_scrub":"301'23","last_deep_scrub_stamp":"2024-09-10T18:35:15.283373+0000","last_clean_scrub_stamp":"2024-09-10T18:37:42.463250+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0078345689999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":408,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971854+0000","last_change":"2024-09-10T18:38:58.108942+0000","last_active":"2024-09-10T18:39:46.971854+0000","last_peered":"2024-09-10T18:39:46.971854+0000","last_clean":"2024-09-10T18:39:46.971854+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:39:46.971854+0000","last_undegraded":"2024-09-10T18:39:46.971854+0000","last_fullsized":"2024-09-10T18:39:46.971854+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:58.108852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:38:58.108852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:13.147762+0000","scrub_duration":0.0054565940000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":444,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.969916+0000","last_change":"2024-09-10T18:39:25.265084+0000","last_active":"2024-09-10T18:39:46.969916+0000","last_peered":"2024-09-10T18:39:46.969916+0000","last_clean":"2024-09-10T18:39:46.969916+0000","last_became_active":"2024-09-10T18:39:25.264561+0000","last_became_peered":"2024-09-10T18:39:25.264561+0000","last_unstale":"2024-09-10T18:39:46.969916+0000","last_undegraded":"2024-09-10T18:39:46.969916+0000","last_fullsized":"2024-09-10T18:39:46.969916+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:26.177423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:38:26.177423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003821648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":385,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.976077+0000","last_change":"2024-09-10T18:38:50.727336+0000","last_active":"2024-09-10T18:39:46.976077+0000","last_peered":"2024-09-10T18:39:46.976077+0000","last_clean":"2024-09-10T18:39:46.976077+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:39:46.976077+0000","last_undegraded":"2024-09-10T18:39:46.976077+0000","last_fullsized":"2024-09-10T18:39:46.976077+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:50.727231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:38:50.727231+0000","last_clean_scrub_stamp":"2024-09-10T18:38:50.727231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067197899999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"76'81","reported_seq":128,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970895+0000","last_change":"2024-09-10T18:38:27.050301+0000","last_active":"2024-09-10T18:39:46.970895+0000","last_peered":"2024-09-10T18:39:46.970895+0000","last_clean":"2024-09-10T18:39:46.970895+0000","last_became_active":"2024-09-10T18:34:32.822196+0000","last_became_peered":"2024-09-10T18:34:32.822196+0000","last_unstale":"2024-09-10T18:39:46.970895+0000","last_undegraded":"2024-09-10T18:39:46.970895+0000","last_fullsized":"2024-09-10T18:39:46.970895+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:38:27.050229+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:38:27.050229+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:56.113241+0000","scrub_duration":0.002388977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":49,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970746+0000","last_change":"2024-09-10T18:39:36.061676+0000","last_active":"2024-09-10T18:39:46.970746+0000","last_peered":"2024-09-10T18:39:46.970746+0000","last_clean":"2024-09-10T18:39:46.970746+0000","last_became_active":"2024-09-10T18:39:00.902783+0000","last_became_peered":"2024-09-10T18:39:00.902783+0000","last_unstale":"2024-09-10T18:39:46.970746+0000","last_undegraded":"2024-09-10T18:39:46.970746+0000","last_fullsized":"2024-09-10T18:39:46.970746+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:39:36.061500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:39:36.061500+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:53.952436+0000","scrub_duration":0.0081230469999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":424,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971411+0000","last_change":"2024-09-10T18:39:25.268275+0000","last_active":"2024-09-10T18:39:46.971411+0000","last_peered":"2024-09-10T18:39:46.971411+0000","last_clean":"2024-09-10T18:39:46.971411+0000","last_became_active":"2024-09-10T18:39:25.238037+0000","last_became_peered":"2024-09-10T18:39:25.238037+0000","last_unstale":"2024-09-10T18:39:46.971411+0000","last_undegraded":"2024-09-10T18:39:46.971411+0000","last_fullsized":"2024-09-10T18:39:46.971411+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:38:37.999176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:38:37.999176+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:55.768601+0000","scrub_duration":0.0049257620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":633,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.882079+0000","last_change":"2024-09-10T18:38:28.026891+0000","last_active":"2024-09-10T18:39:54.882079+0000","last_peered":"2024-09-10T18:39:54.882079+0000","last_clean":"2024-09-10T18:39:54.882079+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:39:54.882079+0000","last_undegraded":"2024-09-10T18:39:54.882079+0000","last_fullsized":"2024-09-10T18:39:54.882079+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:38:28.026796+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:38:28.026796+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010171827,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":33,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971425+0000","last_change":"2024-09-10T18:39:25.242805+0000","last_active":"2024-09-10T18:39:46.971425+0000","last_peered":"2024-09-10T18:39:46.971425+0000","last_clean":"2024-09-10T18:39:46.971425+0000","last_became_active":"2024-09-10T18:39:25.242197+0000","last_became_peered":"2024-09-10T18:39:25.242197+0000","last_unstale":"2024-09-10T18:39:46.971425+0000","last_undegraded":"2024-09-10T18:39:46.971425+0000","last_fullsized":"2024-09-10T18:39:46.971425+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:25.349323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:38:25.349323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'81","reported_seq":23,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971587+0000","last_change":"2024-09-10T18:39:00.911067+0000","last_active":"2024-09-10T18:39:46.971587+0000","last_peered":"2024-09-10T18:39:46.971587+0000","last_clean":"2024-09-10T18:39:46.971587+0000","last_became_active":"2024-09-10T18:39:00.910728+0000","last_became_peered":"2024-09-10T18:39:00.910728+0000","last_unstale":"2024-09-10T18:39:46.971587+0000","last_undegraded":"2024-09-10T18:39:46.971587+0000","last_fullsized":"2024-09-10T18:39:46.971587+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:14.252067+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:14.252067+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":10,"num_write":6,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":395,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.645975+0000","last_change":"2024-09-10T18:39:54.645454+0000","last_active":"2024-09-10T18:39:54.645975+0000","last_peered":"2024-09-10T18:39:54.645975+0000","last_clean":"2024-09-10T18:39:54.645975+0000","last_became_active":"2024-09-10T18:39:25.243214+0000","last_became_peered":"2024-09-10T18:39:25.243214+0000","last_unstale":"2024-09-10T18:39:54.645975+0000","last_undegraded":"2024-09-10T18:39:54.645975+0000","last_fullsized":"2024-09-10T18:39:54.645975+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:54.645384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:39:54.645384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:16.949017+0000","scrub_duration":0.0016590590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"240'12","reported_seq":407,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.977262+0000","last_change":"2024-09-10T18:38:46.685534+0000","last_active":"2024-09-10T18:39:46.977262+0000","last_peered":"2024-09-10T18:39:46.977262+0000","last_clean":"2024-09-10T18:39:46.977262+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:39:46.977262+0000","last_undegraded":"2024-09-10T18:39:46.977262+0000","last_fullsized":"2024-09-10T18:39:46.977262+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-10T18:38:46.685110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:38:46.685110+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:14.478434+0000","scrub_duration":0.014007402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":23,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.973442+0000","last_change":"2024-09-10T18:39:00.917262+0000","last_active":"2024-09-10T18:39:46.973442+0000","last_peered":"2024-09-10T18:39:46.973442+0000","last_clean":"2024-09-10T18:39:46.973442+0000","last_became_active":"2024-09-10T18:39:00.916952+0000","last_became_peered":"2024-09-10T18:39:00.916952+0000","last_unstale":"2024-09-10T18:39:46.973442+0000","last_undegraded":"2024-09-10T18:39:46.973442+0000","last_fullsized":"2024-09-10T18:39:46.973442+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:55.492428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:55.492428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":399,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:53.274301+0000","last_change":"2024-09-10T18:39:53.274245+0000","last_active":"2024-09-10T18:39:53.274301+0000","last_peered":"2024-09-10T18:39:53.274301+0000","last_clean":"2024-09-10T18:39:53.274301+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:39:53.274301+0000","last_undegraded":"2024-09-10T18:39:53.274301+0000","last_fullsized":"2024-09-10T18:39:53.274301+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:53.274176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:39:53.274176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:10.722646+0000","scrub_duration":0.0016102950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'81","reported_seq":336,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.376492+0000","last_change":"2024-09-10T18:38:55.091585+0000","last_active":"2024-09-10T18:39:54.376492+0000","last_peered":"2024-09-10T18:39:54.376492+0000","last_clean":"2024-09-10T18:39:54.376492+0000","last_became_active":"2024-09-10T18:34:02.815880+0000","last_became_peered":"2024-09-10T18:34:02.815880+0000","last_unstale":"2024-09-10T18:39:54.376492+0000","last_undegraded":"2024-09-10T18:39:54.376492+0000","last_fullsized":"2024-09-10T18:39:54.376492+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:38:55.091406+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:38:55.091406+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032583999999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"240'19","reported_seq":289,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.994676+0000","last_change":"2024-09-10T18:39:25.252021+0000","last_active":"2024-09-10T18:39:46.994676+0000","last_peered":"2024-09-10T18:39:46.994676+0000","last_clean":"2024-09-10T18:39:46.994676+0000","last_became_active":"2024-09-10T18:39:25.236628+0000","last_became_peered":"2024-09-10T18:39:25.236628+0000","last_unstale":"2024-09-10T18:39:46.994676+0000","last_undegraded":"2024-09-10T18:39:46.994676+0000","last_fullsized":"2024-09-10T18:39:46.994676+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:38:58.913293+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-09-10T18:35:19.802016+0000","last_clean_scrub_stamp":"2024-09-10T18:38:58.913293+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:09.513371+0000","scrub_duration":0.0093093749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":405,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.973246+0000","last_change":"2024-09-10T18:38:39.973754+0000","last_active":"2024-09-10T18:39:46.973246+0000","last_peered":"2024-09-10T18:39:46.973246+0000","last_clean":"2024-09-10T18:39:46.973246+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:39:46.973246+0000","last_undegraded":"2024-09-10T18:39:46.973246+0000","last_fullsized":"2024-09-10T18:39:46.973246+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:39.973681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:38:39.973681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:56.866203+0000","scrub_duration":0.0046437229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":387,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.977164+0000","last_change":"2024-09-10T18:38:27.407066+0000","last_active":"2024-09-10T18:39:46.977164+0000","last_peered":"2024-09-10T18:39:46.977164+0000","last_clean":"2024-09-10T18:39:46.977164+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:39:46.977164+0000","last_undegraded":"2024-09-10T18:39:46.977164+0000","last_fullsized":"2024-09-10T18:39:46.977164+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:27.406539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:38:27.406539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011598275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":422,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970250+0000","last_change":"2024-09-10T18:38:42.139761+0000","last_active":"2024-09-10T18:39:46.970250+0000","last_peered":"2024-09-10T18:39:46.970250+0000","last_clean":"2024-09-10T18:39:46.970250+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:39:46.970250+0000","last_undegraded":"2024-09-10T18:39:46.970250+0000","last_fullsized":"2024-09-10T18:39:46.970250+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:42.139672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:38:42.139672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023629609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'81","reported_seq":23,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970832+0000","last_change":"2024-09-10T18:39:00.905340+0000","last_active":"2024-09-10T18:39:46.970832+0000","last_peered":"2024-09-10T18:39:46.970832+0000","last_clean":"2024-09-10T18:39:46.970832+0000","last_became_active":"2024-09-10T18:39:00.902585+0000","last_became_peered":"2024-09-10T18:39:00.902585+0000","last_unstale":"2024-09-10T18:39:46.970832+0000","last_undegraded":"2024-09-10T18:39:46.970832+0000","last_fullsized":"2024-09-10T18:39:46.970832+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:36:58.147719+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:36:58.147719+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002490309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"240'19","reported_seq":33,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.972177+0000","last_change":"2024-09-10T18:39:00.930793+0000","last_active":"2024-09-10T18:39:46.972177+0000","last_peered":"2024-09-10T18:39:46.972177+0000","last_clean":"2024-09-10T18:39:46.972177+0000","last_became_active":"2024-09-10T18:39:00.899968+0000","last_became_peered":"2024-09-10T18:39:00.899968+0000","last_unstale":"2024-09-10T18:39:46.972177+0000","last_undegraded":"2024-09-10T18:39:46.972177+0000","last_fullsized":"2024-09-10T18:39:46.972177+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:38:37.381389+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:38:37.381389+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053157309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971776+0000","last_change":"2024-09-10T18:39:25.245711+0000","last_active":"2024-09-10T18:39:46.971776+0000","last_peered":"2024-09-10T18:39:46.971776+0000","last_clean":"2024-09-10T18:39:46.971776+0000","last_became_active":"2024-09-10T18:39:25.245366+0000","last_became_peered":"2024-09-10T18:39:25.245366+0000","last_unstale":"2024-09-10T18:39:46.971776+0000","last_undegraded":"2024-09-10T18:39:46.971776+0000","last_fullsized":"2024-09-10T18:39:46.971776+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:04.713474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_clean_scrub_stamp":"2024-09-10T18:37:04.713474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"342'9","reported_seq":616,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:49.981473+0000","last_change":"2024-09-10T18:37:56.641912+0000","last_active":"2024-09-10T18:39:49.981473+0000","last_peered":"2024-09-10T18:39:49.981473+0000","last_clean":"2024-09-10T18:39:49.981473+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:39:49.981473+0000","last_undegraded":"2024-09-10T18:39:49.981473+0000","last_fullsized":"2024-09-10T18:39:49.981473+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'9","last_scrub_stamp":"2024-09-10T18:37:56.641708+0000","last_deep_scrub":"342'9","last_deep_scrub_stamp":"2024-09-10T18:37:56.641708+0000","last_clean_scrub_stamp":"2024-09-10T18:37:56.641708+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023870353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":33,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.994756+0000","last_change":"2024-09-10T18:39:34.901254+0000","last_active":"2024-09-10T18:39:46.994756+0000","last_peered":"2024-09-10T18:39:46.994756+0000","last_clean":"2024-09-10T18:39:46.994756+0000","last_became_active":"2024-09-10T18:39:00.903167+0000","last_became_peered":"2024-09-10T18:39:00.903167+0000","last_unstale":"2024-09-10T18:39:46.994756+0000","last_undegraded":"2024-09-10T18:39:46.994756+0000","last_fullsized":"2024-09-10T18:39:46.994756+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:34.901182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:39:34.901182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:58.531078+0000","scrub_duration":0.00237777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"76'81","reported_seq":209,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970990+0000","last_change":"2024-09-10T18:39:25.251550+0000","last_active":"2024-09-10T18:39:46.970990+0000","last_peered":"2024-09-10T18:39:46.970990+0000","last_clean":"2024-09-10T18:39:46.970990+0000","last_became_active":"2024-09-10T18:39:25.249841+0000","last_became_peered":"2024-09-10T18:39:25.249841+0000","last_unstale":"2024-09-10T18:39:46.970990+0000","last_undegraded":"2024-09-10T18:39:46.970990+0000","last_fullsized":"2024-09-10T18:39:46.970990+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:43.810696+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:36:08.710566+0000","last_clean_scrub_stamp":"2024-09-10T18:37:43.810696+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083017330000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":429,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.976646+0000","last_change":"2024-09-10T18:39:39.855932+0000","last_active":"2024-09-10T18:39:46.976646+0000","last_peered":"2024-09-10T18:39:46.976646+0000","last_clean":"2024-09-10T18:39:46.976646+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:39:46.976646+0000","last_undegraded":"2024-09-10T18:39:46.976646+0000","last_fullsized":"2024-09-10T18:39:46.976646+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-10T18:39:39.855841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:39:39.855841+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:42.026378+0000","scrub_duration":0.008828766,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":398,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971950+0000","last_change":"2024-09-10T18:37:48.936256+0000","last_active":"2024-09-10T18:39:46.971950+0000","last_peered":"2024-09-10T18:39:46.971950+0000","last_clean":"2024-09-10T18:39:46.971950+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:39:46.971950+0000","last_undegraded":"2024-09-10T18:39:46.971950+0000","last_fullsized":"2024-09-10T18:39:46.971950+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:48.936179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:17.826448+0000","last_clean_scrub_stamp":"2024-09-10T18:37:48.936179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011322904999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"362'11","reported_seq":54,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.380196+0000","last_change":"2024-09-10T18:39:25.278276+0000","last_active":"2024-09-10T18:39:54.380196+0000","last_peered":"2024-09-10T18:39:54.380196+0000","last_clean":"2024-09-10T18:39:54.380196+0000","last_became_active":"2024-09-10T18:39:25.256047+0000","last_became_peered":"2024-09-10T18:39:25.256047+0000","last_unstale":"2024-09-10T18:39:54.380196+0000","last_undegraded":"2024-09-10T18:39:54.380196+0000","last_fullsized":"2024-09-10T18:39:54.380196+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'9","last_scrub_stamp":"2024-09-10T18:38:46.139966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:38:46.139966+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971303+0000","last_change":"2024-09-10T18:39:25.247548+0000","last_active":"2024-09-10T18:39:46.971303+0000","last_peered":"2024-09-10T18:39:46.971303+0000","last_clean":"2024-09-10T18:39:46.971303+0000","last_became_active":"2024-09-10T18:39:25.247171+0000","last_became_peered":"2024-09-10T18:39:25.247171+0000","last_unstale":"2024-09-10T18:39:46.971303+0000","last_undegraded":"2024-09-10T18:39:46.971303+0000","last_fullsized":"2024-09-10T18:39:46.971303+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:48.826669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:37:48.826669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.16","version":"76'81","reported_seq":121,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.377135+0000","last_change":"2024-09-10T18:39:25.253479+0000","last_active":"2024-09-10T18:39:54.377135+0000","last_peered":"2024-09-10T18:39:54.377135+0000","last_clean":"2024-09-10T18:39:54.377135+0000","last_became_active":"2024-09-10T18:39:25.253130+0000","last_became_peered":"2024-09-10T18:39:25.253130+0000","last_unstale":"2024-09-10T18:39:54.377135+0000","last_undegraded":"2024-09-10T18:39:54.377135+0000","last_fullsized":"2024-09-10T18:39:54.377135+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:48.962443+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:48.962443+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052595840000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":434,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.888702+0000","last_change":"2024-09-10T18:37:13.403259+0000","last_active":"2024-09-10T18:39:46.888702+0000","last_peered":"2024-09-10T18:39:46.888702+0000","last_clean":"2024-09-10T18:39:46.888702+0000","last_became_active":"2024-09-10T18:26:27.649202+0000","last_became_peered":"2024-09-10T18:26:27.649202+0000","last_unstale":"2024-09-10T18:39:46.888702+0000","last_undegraded":"2024-09-10T18:39:46.888702+0000","last_fullsized":"2024-09-10T18:39:46.888702+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:37:13.403175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:13.403175+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070645999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":408,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:47.552578+0000","last_change":"2024-09-10T18:39:47.552533+0000","last_active":"2024-09-10T18:39:47.552578+0000","last_peered":"2024-09-10T18:39:47.552578+0000","last_clean":"2024-09-10T18:39:47.552578+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:39:47.552578+0000","last_undegraded":"2024-09-10T18:39:47.552578+0000","last_fullsized":"2024-09-10T18:39:47.552578+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:47.552466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:39:47.552466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:11.007804+0000","scrub_duration":0.001793968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":429,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.376436+0000","last_change":"2024-09-10T18:39:00.898175+0000","last_active":"2024-09-10T18:39:54.376436+0000","last_peered":"2024-09-10T18:39:54.376436+0000","last_clean":"2024-09-10T18:39:54.376436+0000","last_became_active":"2024-09-10T18:39:00.897018+0000","last_became_peered":"2024-09-10T18:39:00.897018+0000","last_unstale":"2024-09-10T18:39:54.376436+0000","last_undegraded":"2024-09-10T18:39:54.376436+0000","last_fullsized":"2024-09-10T18:39:54.376436+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:56.027540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_clean_scrub_stamp":"2024-09-10T18:37:56.027540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065405710000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":415,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:53.911317+0000","last_change":"2024-09-10T18:39:53.911283+0000","last_active":"2024-09-10T18:39:53.911317+0000","last_peered":"2024-09-10T18:39:53.911317+0000","last_clean":"2024-09-10T18:39:53.911317+0000","last_became_active":"2024-09-10T18:39:25.237500+0000","last_became_peered":"2024-09-10T18:39:25.237500+0000","last_unstale":"2024-09-10T18:39:53.911317+0000","last_undegraded":"2024-09-10T18:39:53.911317+0000","last_fullsized":"2024-09-10T18:39:53.911317+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:53.911229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:39:53.911229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:20.504157+0000","scrub_duration":0.001589023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'81","reported_seq":33,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971930+0000","last_change":"2024-09-10T18:39:25.252278+0000","last_active":"2024-09-10T18:39:46.971930+0000","last_peered":"2024-09-10T18:39:46.971930+0000","last_clean":"2024-09-10T18:39:46.971930+0000","last_became_active":"2024-09-10T18:39:25.244612+0000","last_became_peered":"2024-09-10T18:39:25.244612+0000","last_unstale":"2024-09-10T18:39:46.971930+0000","last_undegraded":"2024-09-10T18:39:46.971930+0000","last_fullsized":"2024-09-10T18:39:46.971930+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:50.487145+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:50.487145+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"240'15","reported_seq":440,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.376579+0000","last_change":"2024-09-10T18:38:25.084413+0000","last_active":"2024-09-10T18:39:54.376579+0000","last_peered":"2024-09-10T18:39:54.376579+0000","last_clean":"2024-09-10T18:39:54.376579+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:39:54.376579+0000","last_undegraded":"2024-09-10T18:39:54.376579+0000","last_fullsized":"2024-09-10T18:39:54.376579+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:38:25.084319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:38:25.084319+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010195724999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":401,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971297+0000","last_change":"2024-09-10T18:38:29.953557+0000","last_active":"2024-09-10T18:39:46.971297+0000","last_peered":"2024-09-10T18:39:46.971297+0000","last_clean":"2024-09-10T18:39:46.971297+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:39:46.971297+0000","last_undegraded":"2024-09-10T18:39:46.971297+0000","last_fullsized":"2024-09-10T18:39:46.971297+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:29.953473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:38:29.953473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063514089999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":390,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.975512+0000","last_change":"2024-09-10T18:39:25.252063+0000","last_active":"2024-09-10T18:39:46.975512+0000","last_peered":"2024-09-10T18:39:46.975512+0000","last_clean":"2024-09-10T18:39:46.975512+0000","last_became_active":"2024-09-10T18:39:25.251469+0000","last_became_peered":"2024-09-10T18:39:25.251469+0000","last_unstale":"2024-09-10T18:39:46.975512+0000","last_undegraded":"2024-09-10T18:39:46.975512+0000","last_fullsized":"2024-09-10T18:39:46.975512+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:28.424692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:38:28.424692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019717892000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":423,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970927+0000","last_change":"2024-09-10T18:39:46.020028+0000","last_active":"2024-09-10T18:39:46.970927+0000","last_peered":"2024-09-10T18:39:46.970927+0000","last_clean":"2024-09-10T18:39:46.970927+0000","last_became_active":"2024-09-10T18:39:25.264554+0000","last_became_peered":"2024-09-10T18:39:25.264554+0000","last_unstale":"2024-09-10T18:39:46.970927+0000","last_undegraded":"2024-09-10T18:39:46.970927+0000","last_fullsized":"2024-09-10T18:39:46.970927+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:46.019956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:39:46.019956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:13.137974+0000","scrub_duration":0.0018874110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"76'81","reported_seq":138,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970081+0000","last_change":"2024-09-10T18:39:36.098104+0000","last_active":"2024-09-10T18:39:46.970081+0000","last_peered":"2024-09-10T18:39:46.970081+0000","last_clean":"2024-09-10T18:39:46.970081+0000","last_became_active":"2024-09-10T18:34:02.802580+0000","last_became_peered":"2024-09-10T18:34:02.802580+0000","last_unstale":"2024-09-10T18:39:46.970081+0000","last_undegraded":"2024-09-10T18:39:46.970081+0000","last_fullsized":"2024-09-10T18:39:46.970081+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:39:36.098019+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:39:36.098019+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:58.475815+0000","scrub_duration":0.0025792079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":427,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.976152+0000","last_change":"2024-09-10T18:37:55.624202+0000","last_active":"2024-09-10T18:39:46.976152+0000","last_peered":"2024-09-10T18:39:46.976152+0000","last_clean":"2024-09-10T18:39:46.976152+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:39:46.976152+0000","last_undegraded":"2024-09-10T18:39:46.976152+0000","last_fullsized":"2024-09-10T18:39:46.976152+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:37:55.624117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:37:55.624117+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051350730000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.973414+0000","last_change":"2024-09-10T18:39:38.044203+0000","last_active":"2024-09-10T18:39:46.973414+0000","last_peered":"2024-09-10T18:39:46.973414+0000","last_clean":"2024-09-10T18:39:46.973414+0000","last_became_active":"2024-09-10T18:39:00.916983+0000","last_became_peered":"2024-09-10T18:39:00.916983+0000","last_unstale":"2024-09-10T18:39:46.973414+0000","last_undegraded":"2024-09-10T18:39:46.973414+0000","last_fullsized":"2024-09-10T18:39:46.973414+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:38.044074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:39:38.044074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:42.512067+0000","scrub_duration":0.00273858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":394,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.972441+0000","last_change":"2024-09-10T18:37:41.891601+0000","last_active":"2024-09-10T18:39:46.972441+0000","last_peered":"2024-09-10T18:39:46.972441+0000","last_clean":"2024-09-10T18:39:46.972441+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:39:46.972441+0000","last_undegraded":"2024-09-10T18:39:46.972441+0000","last_fullsized":"2024-09-10T18:39:46.972441+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:41.891433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:37:41.891433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002300341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":421,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.975664+0000","last_change":"2024-09-10T18:39:38.888765+0000","last_active":"2024-09-10T18:39:46.975664+0000","last_peered":"2024-09-10T18:39:46.975664+0000","last_clean":"2024-09-10T18:39:46.975664+0000","last_became_active":"2024-09-10T18:39:25.242900+0000","last_became_peered":"2024-09-10T18:39:25.242900+0000","last_unstale":"2024-09-10T18:39:46.975664+0000","last_undegraded":"2024-09-10T18:39:46.975664+0000","last_fullsized":"2024-09-10T18:39:46.975664+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:38.888625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:39:38.888625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:56.849207+0000","scrub_duration":0.0020721170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'81","reported_seq":129,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970784+0000","last_change":"2024-09-10T18:39:00.890324+0000","last_active":"2024-09-10T18:39:46.970784+0000","last_peered":"2024-09-10T18:39:46.970784+0000","last_clean":"2024-09-10T18:39:46.970784+0000","last_became_active":"2024-09-10T18:39:00.889998+0000","last_became_peered":"2024-09-10T18:39:00.889998+0000","last_unstale":"2024-09-10T18:39:46.970784+0000","last_undegraded":"2024-09-10T18:39:46.970784+0000","last_fullsized":"2024-09-10T18:39:46.970784+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:38:52.175598+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:38:52.175598+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:17.275581+0000","scrub_duration":0.00138405,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":408,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970051+0000","last_change":"2024-09-10T18:38:43.659772+0000","last_active":"2024-09-10T18:39:46.970051+0000","last_peered":"2024-09-10T18:39:46.970051+0000","last_clean":"2024-09-10T18:39:46.970051+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:39:46.970051+0000","last_undegraded":"2024-09-10T18:39:46.970051+0000","last_fullsized":"2024-09-10T18:39:46.970051+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-09-10T18:38:43.659715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:38:43.659715+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:04.140989+0000","scrub_duration":0.0019870550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":425,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971823+0000","last_change":"2024-09-10T18:39:41.057048+0000","last_active":"2024-09-10T18:39:46.971823+0000","last_peered":"2024-09-10T18:39:46.971823+0000","last_clean":"2024-09-10T18:39:46.971823+0000","last_became_active":"2024-09-10T18:39:25.242295+0000","last_became_peered":"2024-09-10T18:39:25.242295+0000","last_unstale":"2024-09-10T18:39:46.971823+0000","last_undegraded":"2024-09-10T18:39:46.971823+0000","last_fullsized":"2024-09-10T18:39:46.971823+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:41.056996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:39:41.056996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:44.326721+0000","scrub_duration":0.00123715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":359,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.645785+0000","last_change":"2024-09-10T18:38:30.051120+0000","last_active":"2024-09-10T18:39:54.645785+0000","last_peered":"2024-09-10T18:39:54.645785+0000","last_clean":"2024-09-10T18:39:54.645785+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:39:54.645785+0000","last_undegraded":"2024-09-10T18:39:54.645785+0000","last_fullsized":"2024-09-10T18:39:54.645785+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:30.051035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:38:30.051035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:39:57.086453+0000","scrub_duration":0.001892555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":392,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971568+0000","last_change":"2024-09-10T18:38:26.018093+0000","last_active":"2024-09-10T18:39:46.971568+0000","last_peered":"2024-09-10T18:39:46.971568+0000","last_clean":"2024-09-10T18:39:46.971568+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:39:46.971568+0000","last_undegraded":"2024-09-10T18:39:46.971568+0000","last_fullsized":"2024-09-10T18:39:46.971568+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:38:26.018016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:38:26.018016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015546189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'81","reported_seq":109,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.888584+0000","last_change":"2024-09-10T18:37:55.419411+0000","last_active":"2024-09-10T18:39:46.888584+0000","last_peered":"2024-09-10T18:39:46.888584+0000","last_clean":"2024-09-10T18:39:46.888584+0000","last_became_active":"2024-09-10T18:34:34.867380+0000","last_became_peered":"2024-09-10T18:34:34.867380+0000","last_unstale":"2024-09-10T18:39:46.888584+0000","last_undegraded":"2024-09-10T18:39:46.888584+0000","last_fullsized":"2024-09-10T18:39:46.888584+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:37:55.419180+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:37:55.419180+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067995520000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":424,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.376772+0000","last_change":"2024-09-10T18:39:00.910777+0000","last_active":"2024-09-10T18:39:54.376772+0000","last_peered":"2024-09-10T18:39:54.376772+0000","last_clean":"2024-09-10T18:39:54.376772+0000","last_became_active":"2024-09-10T18:39:00.909906+0000","last_became_peered":"2024-09-10T18:39:00.909906+0000","last_unstale":"2024-09-10T18:39:54.376772+0000","last_undegraded":"2024-09-10T18:39:54.376772+0000","last_fullsized":"2024-09-10T18:39:54.376772+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:55.062765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_clean_scrub_stamp":"2024-09-10T18:37:55.062765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046630359999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":401,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.970644+0000","last_change":"2024-09-10T18:39:43.927945+0000","last_active":"2024-09-10T18:39:46.970644+0000","last_peered":"2024-09-10T18:39:46.970644+0000","last_clean":"2024-09-10T18:39:46.970644+0000","last_became_active":"2024-09-10T18:39:25.250206+0000","last_became_peered":"2024-09-10T18:39:25.250206+0000","last_unstale":"2024-09-10T18:39:46.970644+0000","last_undegraded":"2024-09-10T18:39:46.970644+0000","last_fullsized":"2024-09-10T18:39:46.970644+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:43.927882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:39:43.927882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:56.323128+0000","scrub_duration":0.0017896539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"369'24","reported_seq":479,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-10T18:39:54.646759+0000","last_change":"2024-09-10T18:39:50.690906+0000","last_active":"2024-09-10T18:39:54.646759+0000","last_peered":"2024-09-10T18:39:54.646759+0000","last_clean":"2024-09-10T18:39:54.646759+0000","last_became_active":"2024-09-10T18:39:00.921476+0000","last_became_peered":"2024-09-10T18:39:00.921476+0000","last_unstale":"2024-09-10T18:39:54.646759+0000","last_undegraded":"2024-09-10T18:39:54.646759+0000","last_fullsized":"2024-09-10T18:39:54.646759+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'24","last_scrub_stamp":"2024-09-10T18:39:50.690827+0000","last_deep_scrub":"369'24","last_deep_scrub_stamp":"2024-09-10T18:39:50.690827+0000","last_clean_scrub_stamp":"2024-09-10T18:39:50.690827+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:40:53.577831+0000","scrub_duration":0.0041180970000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":21,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:39:46.971434+0000","last_change":"2024-09-10T18:39:25.244864+0000","last_active":"2024-09-10T18:39:46.971434+0000","last_peered":"2024-09-10T18:39:46.971434+0000","last_clean":"2024-09-10T18:39:46.971434+0000","last_became_active":"2024-09-10T18:39:25.243964+0000","last_became_peered":"2024-09-10T18:39:25.243964+0000","last_unstale":"2024-09-10T18:39:46.971434+0000","last_undegraded":"2024-09-10T18:39:46.971434+0000","last_fullsized":"2024-09-10T18:39:46.971434+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:37:42.795003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:37:42.795003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":4126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466944,"data_stored":12426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":763,"ondisk_log_size":763,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1094,"num_read_kb":1089,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":475272,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":29,"num_read_kb":55,"num_write":29,"num_write_kb":407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":885551,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2511,"ondisk_log_size":2511,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234208,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595144,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93150008,"statfs":{"total":95995035648,"available":95385608192,"internally_reserved":0,"allocated":1114112,"data_stored":695182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:38:55 2024","interfaces":[{"interface":"back","average":{"1min":3.4119999999999999,"5min":1.4299999999999999,"15min":2.1179999999999999},"min":{"1min":0.219,"5min":0.215,"15min":0.20799999999999999},"max":{"1min":29.443999999999999,"5min":29.443999999999999,"15min":85.012},"last":0.503},{"interface":"front","average":{"1min":3.1040000000000001,"5min":1.3899999999999999,"15min":2.0209999999999999},"min":{"1min":0.25700000000000001,"5min":0.223,"15min":0.222},"max":{"1min":29.530999999999999,"5min":29.530999999999999,"15min":84.665999999999997},"last":0.46600000000000003}]},{"osd":1,"last update":"Tue Sep 10 18:38:55 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.407,"15min":1.5269999999999999},"min":{"1min":0.245,"5min":0.23899999999999999,"15min":0.214},"max":{"1min":5.9240000000000004,"5min":39.728000000000002,"15min":49.317999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.403,"15min":1.448},"min":{"1min":0.29499999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":5.9370000000000003,"5min":39.738999999999997,"15min":49.353000000000002},"last":0.52900000000000003}]},{"osd":2,"last update":"Tue Sep 10 18:38:55 2024","interfaces":[{"interface":"back","average":{"1min":5.4809999999999999,"5min":3.52,"15min":1.788},"min":{"1min":0.31,"5min":0.27100000000000002,"15min":0.249},"max":{"1min":44.728000000000002,"5min":136.648,"15min":136.648},"last":0.58999999999999997},{"interface":"front","average":{"1min":6.0369999999999999,"5min":3.5979999999999999,"15min":1.8799999999999999},"min":{"1min":0.29399999999999998,"5min":0.29199999999999998,"15min":0.24399999999999999},"max":{"1min":44.664999999999999,"5min":136.506,"15min":136.506},"last":0.56100000000000005}]},{"osd":3,"last update":"Tue Sep 10 18:38:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.6220000000000001,"15min":1.728},"min":{"1min":0.30199999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":5.5490000000000004,"5min":38.225000000000001,"15min":86.305999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.181,"5min":1.5389999999999999,"15min":1.754},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.26700000000000002},"max":{"1min":3.806,"5min":38.104999999999997,"15min":88.364000000000004},"last":0.64400000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:38:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.2649999999999999,"15min":0.76400000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20799999999999999},"max":{"1min":3.6640000000000001,"5min":23.346,"15min":23.346},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":1.2909999999999999,"15min":0.79100000000000004},"min":{"1min":0.27500000000000002,"5min":0.26800000000000002,"15min":0.223},"max":{"1min":3.1819999999999999,"5min":23.323,"15min":23.323},"last":0.54800000000000004}]},{"osd":5,"last update":"Tue Sep 10 18:38:55 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":1.083,"15min":0.873},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.23499999999999999},"max":{"1min":2.5630000000000002,"5min":16.756,"15min":16.756},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":1.1499999999999999,"15min":0.83899999999999997},"min":{"1min":0.30399999999999999,"5min":0.224,"15min":0.219},"max":{"1min":3.4329999999999998,"5min":20.495999999999999,"15min":20.495999999999999},"last":0.57099999999999995}]},{"osd":6,"last update":"Tue Sep 10 18:39:47 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":1.1779999999999999,"15min":0.88400000000000001},"min":{"1min":0.317,"5min":0.316,"15min":0.27000000000000002},"max":{"1min":5.976,"5min":25.902999999999999,"15min":25.902999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.873,"5min":1.1399999999999999,"15min":0.82299999999999995},"min":{"1min":0.33400000000000002,"5min":0.32400000000000001,"15min":0.30199999999999999},"max":{"1min":3.2410000000000001,"5min":24.074000000000002,"15min":24.074000000000002},"last":0.626}]}]},{"osd":6,"up_from":144,"seq":618475290787,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303736,"kb_used_data":1012,"kb_used_omap":7,"kb_used_meta":302584,"kb_avail":93441416,"statfs":{"total":95995035648,"available":95684009984,"internally_reserved":0,"allocated":1036288,"data_stored":699161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":309846103},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:39:53 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":1.079,"15min":1.4319999999999999},"min":{"1min":0.27700000000000002,"5min":0.192,"15min":0.192},"max":{"1min":2.9870000000000001,"5min":5.5469999999999997,"15min":30.629999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.264,"15min":1.2390000000000001},"min":{"1min":0.35399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":4.9489999999999998,"5min":18.093,"15min":30.597000000000001},"last":0.62}]},{"osd":1,"last update":"Tue Sep 10 18:39:53 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":0.86199999999999999,"15min":1.0049999999999999},"min":{"1min":0.41399999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.9800000000000004,"5min":4.9800000000000004,"15min":30.984000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.115,"5min":1.0920000000000001,"15min":1.171},"min":{"1min":0.26000000000000001,"5min":0.253,"15min":0.24199999999999999},"max":{"1min":4.2110000000000003,"5min":9.7140000000000004,"15min":30.571000000000002},"last":0.49399999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:39:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.772,"15min":1.246},"min":{"1min":0.29099999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.3380000000000001,"5min":39.329999999999998,"15min":39.329999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.4570000000000001,"15min":1.071},"min":{"1min":0.35099999999999998,"5min":0.26200000000000001,"15min":0.23400000000000001},"max":{"1min":3.6989999999999998,"5min":39.061999999999998,"15min":39.061999999999998},"last":0.48099999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:39:53 2024","interfaces":[{"interface":"back","average":{"1min":17.928999999999998,"5min":4.6029999999999998,"15min":2.7050000000000001},"min":{"1min":0.39200000000000002,"5min":0.30199999999999999,"15min":0.27800000000000002},"max":{"1min":162.61600000000001,"5min":162.61600000000001,"15min":162.61600000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":18.940999999999999,"5min":4.5140000000000002,"15min":2.7229999999999999},"min":{"1min":0.41099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":163.39599999999999,"5min":163.39599999999999,"15min":163.39599999999999},"last":0.58799999999999997}]},{"osd":4,"last update":"Tue Sep 10 18:39:53 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.81399999999999995,"15min":0.66000000000000003},"min":{"1min":0.27900000000000003,"5min":0.189,"15min":0.189},"max":{"1min":4.9050000000000002,"5min":4.9050000000000002,"15min":5.4950000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.80600000000000005,"15min":0.68700000000000006},"min":{"1min":0.27800000000000002,"5min":0.219,"15min":0.219},"max":{"1min":1.7190000000000001,"5min":11.531000000000001,"15min":11.531000000000001},"last":0.52400000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:39:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":0.878,"15min":0.69099999999999995},"min":{"1min":0.254,"5min":0.254,"15min":0.24399999999999999},"max":{"1min":3.54,"5min":4.476,"15min":4.6619999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.91300000000000003,"15min":0.70499999999999996},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.0089999999999999,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":0.38100000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:39:53 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.94799999999999995,"15min":0.71899999999999997},"min":{"1min":0.26700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.9369999999999998,"5min":11.449,"15min":11.449},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.79000000000000004,"15min":0.69299999999999995},"min":{"1min":0.38500000000000001,"5min":0.32000000000000001,"15min":0.27100000000000002},"max":{"1min":5.0149999999999997,"5min":5.0149999999999997,"15min":5.5309999999999997},"last":0.46400000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724779,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900556,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":899454,"kb_avail":92844596,"statfs":{"total":95995035648,"available":95072866304,"internally_reserved":0,"allocated":991232,"data_stored":634475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":921041240},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:39:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.2290000000000001,"15min":1.6899999999999999},"min":{"1min":0.23200000000000001,"5min":0.224,"15min":0.21099999999999999},"max":{"1min":5.1989999999999998,"5min":15.819000000000001,"15min":52.033999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.002,"15min":1.45},"min":{"1min":0.26000000000000001,"5min":0.24399999999999999,"15min":0.216},"max":{"1min":3.7120000000000002,"5min":3.7120000000000002,"15min":52.167999999999999},"last":0.41899999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:39:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":2.5259999999999998,"15min":1.5940000000000001},"min":{"1min":0.32700000000000001,"5min":0.249,"15min":0.22600000000000001},"max":{"1min":8.7490000000000006,"5min":111.773,"15min":111.773},"last":0.437},{"interface":"front","average":{"1min":1.03,"5min":2.524,"15min":1.667},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.247},"max":{"1min":3.5139999999999998,"5min":111.73999999999999,"15min":111.73999999999999},"last":0.55300000000000005}]},{"osd":2,"last update":"Tue Sep 10 18:39:09 2024","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":1.1890000000000001,"15min":1.395},"min":{"1min":0.30299999999999999,"5min":0.26900000000000002,"15min":0.24299999999999999},"max":{"1min":9.8759999999999994,"5min":9.8759999999999994,"15min":54.530000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.94099999999999995,"15min":1.2869999999999999},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":2.7069999999999999,"5min":3.9780000000000002,"15min":54.514000000000003},"last":0.59699999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:39:09 2024","interfaces":[{"interface":"back","average":{"1min":1.8149999999999999,"5min":2.3740000000000001,"15min":2.0129999999999999},"min":{"1min":0.29999999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":8.7769999999999992,"5min":106.205,"15min":106.205},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.0329999999999999,"5min":2.3460000000000001,"15min":1.9159999999999999},"min":{"1min":0.376,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.5510000000000002,"5min":105.878,"15min":105.878},"last":0.50700000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:39:09 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.83499999999999996,"15min":0.65000000000000002},"min":{"1min":0.26900000000000002,"5min":0.23799999999999999,"15min":0.20799999999999999},"max":{"1min":2.8900000000000001,"5min":5.8040000000000003,"15min":5.8040000000000003},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.998,"5min":0.91600000000000004,"15min":0.72499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.22700000000000001},"max":{"1min":5.7309999999999999,"5min":8.7270000000000003,"15min":8.7270000000000003},"last":0.52400000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:39:48 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.77500000000000002,"15min":0.71699999999999997},"min":{"1min":0.31900000000000001,"5min":0.29699999999999999,"15min":0.27500000000000002},"max":{"1min":2.266,"5min":7.9969999999999999,"15min":7.9969999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.752,"5min":0.995,"15min":0.80400000000000005},"min":{"1min":0.32400000000000001,"5min":0.29599999999999999,"15min":0.27000000000000002},"max":{"1min":2.3929999999999998,"5min":8.4220000000000006,"15min":8.4220000000000006},"last":0.66100000000000003}]},{"osd":7,"last update":"Tue Sep 10 18:39:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.85899999999999999,"15min":0.84399999999999997},"min":{"1min":0.39900000000000002,"5min":0.32200000000000001,"15min":0.22500000000000001},"max":{"1min":1.6339999999999999,"5min":8.3870000000000005,"15min":28.934000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.89100000000000001,"15min":0.745},"min":{"1min":0.314,"5min":0.314,"15min":0.25900000000000001},"max":{"1min":1.861,"5min":5.8259999999999996,"15min":5.8259999999999996},"last":0.64600000000000002}]}]},{"osd":4,"up_from":33,"seq":141733921007,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93149944,"statfs":{"total":95995035648,"available":95385542656,"internally_reserved":0,"allocated":1052672,"data_stored":698800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:39:43 2024","interfaces":[{"interface":"back","average":{"1min":2.718,"5min":1.625,"15min":1.387},"min":{"1min":0.23599999999999999,"5min":0.20899999999999999,"15min":0.188},"max":{"1min":21.468,"5min":21.468,"15min":31.585000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":2.6000000000000001,"5min":1.3240000000000001,"15min":1.2210000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.189},"max":{"1min":24.277999999999999,"5min":24.277999999999999,"15min":29.802},"last":0.68400000000000005}]},{"osd":1,"last update":"Tue Sep 10 18:39:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.2390000000000001,"15min":1.216},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":3.0720000000000001,"5min":12.696,"15min":50.045000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.04,"5min":1.137,"15min":1.2769999999999999},"min":{"1min":0.28199999999999997,"5min":0.28000000000000003,"15min":0.22900000000000001},"max":{"1min":2.956,"5min":13.632,"15min":50.283000000000001},"last":0.61899999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:39:43 2024","interfaces":[{"interface":"back","average":{"1min":7.024,"5min":2.0590000000000002,"15min":1.355},"min":{"1min":0.28199999999999997,"5min":0.251,"15min":0.249},"max":{"1min":76.843000000000004,"5min":76.843000000000004,"15min":76.843000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":6.7960000000000003,"5min":2.1400000000000001,"15min":1.4159999999999999},"min":{"1min":0.26100000000000001,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":76.591999999999999,"5min":76.591999999999999,"15min":76.591999999999999},"last":0.73099999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:39:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":0.96899999999999997,"15min":1.47},"min":{"1min":0.27400000000000002,"5min":0.26200000000000001,"15min":0.249},"max":{"1min":3.9860000000000002,"5min":7.7699999999999996,"15min":49.990000000000002},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.5589999999999999,"5min":1.4390000000000001,"15min":1.647},"min":{"1min":0.32400000000000001,"5min":0.28999999999999998,"15min":0.28000000000000003},"max":{"1min":5.9790000000000001,"5min":10.119999999999999,"15min":50.764000000000003},"last":0.51800000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:39:06 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.79400000000000004,"15min":0.73299999999999998},"min":{"1min":0.29199999999999998,"5min":0.25,"15min":0.20499999999999999},"max":{"1min":2.1640000000000001,"5min":3.492,"15min":22.861000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.95899999999999996,"15min":0.80200000000000005},"min":{"1min":0.27500000000000002,"5min":0.252,"15min":0.22},"max":{"1min":3.3849999999999998,"5min":8.2539999999999996,"15min":19.611999999999998},"last":0.80300000000000005}]},{"osd":6,"last update":"Tue Sep 10 18:39:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.5549999999999999,"15min":0.89800000000000002},"min":{"1min":0.34200000000000003,"5min":0.28000000000000003,"15min":0.26400000000000001},"max":{"1min":4.1600000000000001,"5min":51.722000000000001,"15min":51.722000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.4730000000000001,"15min":0.89200000000000002},"min":{"1min":0.33000000000000002,"5min":0.31,"15min":0.30599999999999999},"max":{"1min":3.121,"5min":51.862000000000002,"15min":51.862000000000002},"last":0.66600000000000004}]},{"osd":7,"last update":"Tue Sep 10 18:39:51 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":0.97299999999999998,"15min":0.72199999999999998},"min":{"1min":0.35599999999999998,"5min":0.30299999999999999,"15min":0.255},"max":{"1min":7.6260000000000003,"5min":7.6260000000000003,"15min":7.6260000000000003},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.143,"5min":1.0129999999999999,"15min":0.751},"min":{"1min":0.30199999999999999,"5min":0.28000000000000003,"15min":0.247},"max":{"1min":4.718,"5min":7.8250000000000002,"15min":7.8250000000000002},"last":0.78100000000000003}]}]},{"osd":3,"up_from":27,"seq":115964117237,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896908,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":895742,"kb_avail":92848244,"statfs":{"total":95995035648,"available":95076601856,"internally_reserved":0,"allocated":1183744,"data_stored":760998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917240152},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:39:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.369,"15min":1.5529999999999999},"min":{"1min":0.255,"5min":0.184,"15min":0.18099999999999999},"max":{"1min":3.0720000000000001,"5min":29.981999999999999,"15min":29.981999999999999},"last":0.442},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.595,"15min":1.744},"min":{"1min":0.23499999999999999,"5min":0.223,"15min":0.193},"max":{"1min":4.1180000000000003,"5min":28.754000000000001,"15min":28.754000000000001},"last":0.38300000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:39:21 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.3340000000000001,"15min":1.7330000000000001},"min":{"1min":0.24099999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":3.9790000000000001,"5min":28.539000000000001,"15min":35.406999999999996},"last":0.317},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.2110000000000001,"15min":1.609},"min":{"1min":0.247,"5min":0.215,"15min":0.19},"max":{"1min":2.7429999999999999,"5min":29.032,"15min":36.902000000000001},"last":0.35499999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:39:21 2024","interfaces":[{"interface":"back","average":{"1min":5.5430000000000001,"5min":1.931,"15min":1.5629999999999999},"min":{"1min":0.38400000000000001,"5min":0.191,"15min":0.191},"max":{"1min":79.975999999999999,"5min":79.975999999999999,"15min":79.975999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":6.3570000000000002,"5min":2.444,"15min":1.9099999999999999},"min":{"1min":0.34100000000000003,"5min":0.22,"15min":0.217},"max":{"1min":78.278000000000006,"5min":78.278000000000006,"15min":78.278000000000006},"last":0.33200000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:38:55 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.131,"15min":1.391},"min":{"1min":0.34000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.1699999999999999,"5min":6.1699999999999999,"15min":23.684999999999999},"last":0.622},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.1140000000000001,"15min":1.3340000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27200000000000002},"max":{"1min":4.8499999999999996,"5min":4.8499999999999996,"15min":23.649000000000001},"last":0.48099999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:39:04 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.1890000000000001,"15min":1.4399999999999999},"min":{"1min":0.34699999999999998,"5min":0.32300000000000001,"15min":0.314},"max":{"1min":6.2370000000000001,"5min":6.2370000000000001,"15min":23.719999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.2470000000000001,"15min":1.4630000000000001},"min":{"1min":0.39300000000000002,"5min":0.34000000000000002,"15min":0.28799999999999998},"max":{"1min":4.8970000000000002,"5min":5.5540000000000003,"15min":23.707000000000001},"last":0.60999999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:38:55 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.248,"15min":1.3280000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":4.9669999999999996,"5min":5.6029999999999998,"15min":31.815999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.319,"5min":1.21,"15min":1.456},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.315},"max":{"1min":3.2679999999999998,"5min":4.1870000000000003,"15min":31.742000000000001},"last":0.68100000000000005}]},{"osd":7,"last update":"Tue Sep 10 18:38:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":1.153,"15min":1.415},"min":{"1min":0.376,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":6.2119999999999997,"5min":6.2119999999999997,"15min":23.748999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.2330000000000001,"15min":1.4790000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.307},"max":{"1min":4.9370000000000003,"5min":4.9370000000000003,"15min":25.058},"last":0.65600000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411577,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":590980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93154172,"statfs":{"total":95995035648,"available":95389872128,"internally_reserved":0,"allocated":983040,"data_stored":633500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:38:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.353,"15min":1.657},"min":{"1min":0.30399999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":3.6360000000000001,"5min":26.585999999999999,"15min":43.890000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":2.7519999999999998,"5min":1.79,"15min":2.0030000000000001},"min":{"1min":0.32500000000000001,"5min":0.19400000000000001,"15min":0.16900000000000001},"max":{"1min":10.307,"5min":26.48,"15min":44.331000000000003},"last":0.27500000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:38:57 2024","interfaces":[{"interface":"back","average":{"1min":1.909,"5min":1.327,"15min":1.716},"min":{"1min":0.26600000000000001,"5min":0.222,"15min":0.193},"max":{"1min":6.3369999999999997,"5min":9.7759999999999998,"15min":44.575000000000003},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.3340000000000001,"5min":0.94199999999999995,"15min":1.3979999999999999},"min":{"1min":0.28499999999999998,"5min":0.21199999999999999,"15min":0.19500000000000001},"max":{"1min":3.5739999999999998,"5min":3.5739999999999998,"15min":44.368000000000002},"last":0.34300000000000003}]},{"osd":3,"last update":"Tue Sep 10 18:39:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6679999999999999,"5min":1.3260000000000001,"15min":1.9850000000000001},"min":{"1min":0.23999999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.2409999999999997,"5min":7.2409999999999997,"15min":61.331000000000003},"last":0.40400000000000003},{"interface":"front","average":{"1min":2.2549999999999999,"5min":1.464,"15min":2.129},"min":{"1min":0.247,"5min":0.22600000000000001,"15min":0.214},"max":{"1min":7.0670000000000002,"5min":7.0670000000000002,"15min":61.250999999999998},"last":0.36799999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:39:53 2024","interfaces":[{"interface":"back","average":{"1min":1.6859999999999999,"5min":1.526,"15min":1.319},"min":{"1min":0.33200000000000002,"5min":0.32300000000000001,"15min":0.24199999999999999},"max":{"1min":6.2300000000000004,"5min":9.8810000000000002,"15min":12.555999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.613,"5min":1.4179999999999999,"15min":1.181},"min":{"1min":0.39500000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.0300000000000002,"5min":19.696000000000002,"15min":19.696000000000002},"last":0.59899999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:39:10 2024","interfaces":[{"interface":"back","average":{"1min":2.2719999999999998,"5min":1.5609999999999999,"15min":1.292},"min":{"1min":0.33700000000000002,"5min":0.30599999999999999,"15min":0.27500000000000002},"max":{"1min":19.632999999999999,"5min":25.177,"15min":25.177},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.486,"15min":1.3280000000000001},"min":{"1min":0.34999999999999998,"5min":0.317,"15min":0.317},"max":{"1min":4.0640000000000001,"5min":24.986999999999998,"15min":24.986999999999998},"last":0.504}]},{"osd":6,"last update":"Tue Sep 10 18:39:53 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":2.2000000000000002,"15min":1.476},"min":{"1min":0.371,"5min":0.34300000000000003,"15min":0.32100000000000001},"max":{"1min":6.3070000000000004,"5min":53.359999999999999,"15min":53.359999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.9690000000000001,"15min":1.345},"min":{"1min":0.42699999999999999,"5min":0.32900000000000001,"15min":0.30099999999999999},"max":{"1min":4.0899999999999999,"5min":53.438000000000002,"15min":53.438000000000002},"last":0.64100000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:38:57 2024","interfaces":[{"interface":"back","average":{"1min":1.988,"5min":1.236,"15min":1.2509999999999999},"min":{"1min":0.38600000000000001,"5min":0.36499999999999999,"15min":0.30499999999999999},"max":{"1min":5.4450000000000003,"5min":5.4450000000000003,"15min":10.68},"last":0.54100000000000004},{"interface":"front","average":{"1min":2.0539999999999998,"5min":1.3740000000000001,"15min":1.361},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.307},"max":{"1min":6.2750000000000004,"5min":9.8499999999999996,"15min":12.596},"last":0.77600000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575102,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":893624,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":892734,"kb_avail":92851528,"statfs":{"total":95995035648,"available":95079964672,"internally_reserved":0,"allocated":774144,"data_stored":372502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":914159960},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.6160000000000001,"15min":1.9239999999999999},"min":{"1min":0.19900000000000001,"5min":0.183,"15min":0.16600000000000001},"max":{"1min":6.6760000000000002,"5min":28.893999999999998,"15min":80.456999999999994},"last":0.39100000000000001},{"interface":"front","average":{"1min":1.8540000000000001,"5min":1.623,"15min":2.0030000000000001},"min":{"1min":0.29399999999999998,"5min":0.19800000000000001,"15min":0.187},"max":{"1min":5.5140000000000002,"5min":31.074999999999999,"15min":80.578000000000003},"last":0.34100000000000003}]},{"osd":2,"last update":"Tue Sep 10 18:38:53 2024","interfaces":[{"interface":"back","average":{"1min":1.859,"5min":1.0660000000000001,"15min":1.371},"min":{"1min":0.29599999999999999,"5min":0.20999999999999999,"15min":0.19500000000000001},"max":{"1min":6.0430000000000001,"5min":6.0430000000000001,"15min":13.035},"last":0.375},{"interface":"front","average":{"1min":2.1110000000000002,"5min":1.196,"15min":1.3779999999999999},"min":{"1min":0.252,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":6.3470000000000004,"5min":6.3470000000000004,"15min":12.222},"last":0.35699999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:39:31 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.135,"15min":1.8100000000000001},"min":{"1min":0.26700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.649,"5min":6.5129999999999999,"15min":43.076000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":1.373,"5min":1.1870000000000001,"15min":1.859},"min":{"1min":0.29999999999999999,"5min":0.27300000000000002,"15min":0.20499999999999999},"max":{"1min":4.8380000000000001,"5min":4.8380000000000001,"15min":41.93},"last":0.41999999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":8.8889999999999993,"5min":2.6110000000000002,"15min":1.742},"min":{"1min":0.33800000000000002,"5min":0.32900000000000001,"15min":0.29799999999999999},"max":{"1min":127.86199999999999,"5min":127.86199999999999,"15min":127.86199999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":8.6310000000000002,"5min":2.5720000000000001,"15min":1.8009999999999999},"min":{"1min":0.34100000000000003,"5min":0.33900000000000002,"15min":0.28299999999999997},"max":{"1min":127.84399999999999,"5min":127.84399999999999,"15min":127.84399999999999},"last":0.55300000000000005}]},{"osd":5,"last update":"Tue Sep 10 18:39:05 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1739999999999999,"15min":1.4039999999999999},"min":{"1min":0.36199999999999999,"5min":0.29799999999999999,"15min":0.27900000000000003},"max":{"1min":4.3179999999999996,"5min":6.2729999999999997,"15min":15.48},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.95799999999999996,"5min":1.3029999999999999,"15min":1.339},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.308},"max":{"1min":2.569,"5min":8.1050000000000004,"15min":11.414999999999999},"last":0.57999999999999996}]},{"osd":6,"last update":"Tue Sep 10 18:38:53 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":2.5870000000000002,"15min":1.6879999999999999},"min":{"1min":0.51200000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":2.7690000000000001,"5min":67.287000000000006,"15min":67.287000000000006},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.7909999999999999,"5min":3.2149999999999999,"15min":1.986},"min":{"1min":0.46700000000000003,"5min":0.374,"15min":0.33600000000000002},"max":{"1min":4.657,"5min":67.400000000000006,"15min":67.400000000000006},"last":0.504}]},{"osd":7,"last update":"Tue Sep 10 18:38:53 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":1.1519999999999999,"15min":1.294},"min":{"1min":0.36399999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.609,"5min":8.1609999999999996,"15min":15.545999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":1.204,"5min":1.2150000000000001,"15min":1.3049999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.313},"max":{"1min":3.3759999999999999,"5min":6.0300000000000002,"15min":7.359},"last":0.64700000000000002}]}]},{"osd":0,"up_from":0,"seq":34359738627,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:39:58.599 INFO:tasks.thrashosds.thrasher:Setting 3.1d to [0, 2, 5, 7, 1, 6] 2024-09-10T18:39:58.599 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.1d', '0', '2', '5', '7', '1', '6'] 2024-09-10T18:39:58.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pg-upmap-items 3.1d 0 2 5 7 1 6 2024-09-10T18:39:58.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:58 smithi028 ceph-mon[29291]: pgmap v833: 162 pgs: 162 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:39:58.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:58 smithi028 ceph-mon[29291]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-10T18:39:58.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:58 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:39:58.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:58 smithi028 ceph-mon[29291]: osdmap e372: 8 total, 8 up, 7 in 2024-09-10T18:39:58.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:58 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1920730892' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:39:58.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:58 smithi028 ceph-mon[29291]: from='client.35828 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:39:58.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:58 smithi028 ceph-mon[39220]: pgmap v833: 162 pgs: 162 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:39:58.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:58 smithi028 ceph-mon[39220]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-10T18:39:58.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:58 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:39:58.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:58 smithi028 ceph-mon[39220]: osdmap e372: 8 total, 8 up, 7 in 2024-09-10T18:39:58.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:58 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1920730892' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:39:58.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:58 smithi028 ceph-mon[39220]: from='client.35828 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:39:58.641 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:58.664 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:58.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:58.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:58.766 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:39:58.790 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:58.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:58.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:39:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:58 smithi204 ceph-mon[32357]: pgmap v833: 162 pgs: 162 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:39:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:58 smithi204 ceph-mon[32357]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-10T18:39:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:58 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:39:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:58 smithi204 ceph-mon[32357]: osdmap e372: 8 total, 8 up, 7 in 2024-09-10T18:39:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:58 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1920730892' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:39:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:58 smithi204 ceph-mon[32357]: from='client.35828 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:39:58.863 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:58.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:39:58.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:58.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:39:59.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:59.039 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:59.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:59.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:59.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:59.164 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:59.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:59.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:59.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:59.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:59.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:59.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:59.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:59.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:59.418 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:59.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:59.519 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:39:59.544 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:59.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:59.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:39:59.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:59.670 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:59.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:59.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:39:59.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:59 smithi028 ceph-mon[39220]: from='client.35834 v1:172.21.15.28:0/2682159238' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:39:59.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:59 smithi028 ceph-mon[39220]: 3.0 deep-scrub starts 2024-09-10T18:39:59.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:59 smithi028 ceph-mon[39220]: 3.0 deep-scrub ok 2024-09-10T18:39:59.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:59 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:39:59.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:59 smithi028 ceph-mon[39220]: from='client.35828 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:39:59.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:39:59 smithi028 ceph-mon[39220]: osdmap e373: 8 total, 8 up, 7 in 2024-09-10T18:39:59.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:59 smithi028 ceph-mon[29291]: from='client.35834 v1:172.21.15.28:0/2682159238' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:39:59.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:59 smithi028 ceph-mon[29291]: 3.0 deep-scrub starts 2024-09-10T18:39:59.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:59 smithi028 ceph-mon[29291]: 3.0 deep-scrub ok 2024-09-10T18:39:59.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:59 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:39:59.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:59 smithi028 ceph-mon[29291]: from='client.35828 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:39:59.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:39:59 smithi028 ceph-mon[29291]: osdmap e373: 8 total, 8 up, 7 in 2024-09-10T18:39:59.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:39:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:39:59.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:59.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:39:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:59 smithi204 ceph-mon[32357]: from='client.35834 v1:172.21.15.28:0/2682159238' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:39:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:59 smithi204 ceph-mon[32357]: 3.0 deep-scrub starts 2024-09-10T18:39:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:59 smithi204 ceph-mon[32357]: 3.0 deep-scrub ok 2024-09-10T18:39:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:59 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:39:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:59 smithi204 ceph-mon[32357]: from='client.35828 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:39:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:39:59 smithi204 ceph-mon[32357]: osdmap e373: 8 total, 8 up, 7 in 2024-09-10T18:39:59.897 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:39:59.923 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:39:59.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:39:59.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:00.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:00.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:00.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:00.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:00.152 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:00.178 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:00.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:00.280 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:00.306 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:00.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:00.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:00.408 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:00.431 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:00.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:00.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:00.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:00 smithi028 ceph-mon[29291]: pgmap v836: 162 pgs: 162 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:40:00.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:00 smithi028 ceph-mon[29291]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-10T18:40:00.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:00 smithi028 ceph-mon[29291]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-10T18:40:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:00 smithi028 ceph-mon[39220]: pgmap v836: 162 pgs: 162 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:40:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:00 smithi028 ceph-mon[39220]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-10T18:40:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:00 smithi028 ceph-mon[39220]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-10T18:40:00.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:00.557 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:00.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:00.616 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:00.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:00.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:00.683 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:00.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:00.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:00.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:00.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:00.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:00 smithi204 ceph-mon[32357]: pgmap v836: 162 pgs: 162 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:40:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:00 smithi204 ceph-mon[32357]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-10T18:40:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:00 smithi204 ceph-mon[32357]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-10T18:40:00.909 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:00.933 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:00.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:00.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:01.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:01.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:01.059 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:01.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:01.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:01.183 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:01.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:01.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:01.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:01.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:01.310 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:01.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:01.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:40:01.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:01.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:01.436 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:01.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:01.537 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:01.560 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:01.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:01.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:01.662 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:01.690 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:01.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:01.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:01.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:40:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:40:01] "GET /metrics HTTP/1.1" 200 224744 "" "Prometheus/2.43.0" 2024-09-10T18:40:01.792 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:01.814 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:01.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:01.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:01.916 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:01.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:01.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:01.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:02.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:02.055 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:02.071 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:02.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:02.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:02.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:02.201 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:02.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:02.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:02.303 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:02.329 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:02.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:02.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:02.382 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:40:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:40:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:40:02.431 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:02.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:02.457 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:02.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:02.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:02.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:02.583 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:02.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:02.633 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:02 smithi028 ceph-mon[39220]: pgmap v837: 162 pgs: 162 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T18:40:02.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:02 smithi028 ceph-mon[29291]: pgmap v837: 162 pgs: 162 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T18:40:02.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:02.709 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:02.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:02.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:02.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:02.816 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:40:02.816 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:40:02.816 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:40:02.816 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:40:02.823 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:40:02.824 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:40:02.835 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:02.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:02.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:02.842 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:40:02.842 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:40:02.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:02 smithi204 ceph-mon[32357]: pgmap v837: 162 pgs: 162 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T18:40:02.861 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:40:02.861 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:40:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:40:02.878 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:40:02.894 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:40:02.894 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:40:02.912 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:40:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:40:02.937 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:02.962 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:02.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:02.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:03.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:03.088 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:03.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:03.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:03.210 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:03.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:03.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:03.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:03.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:03.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:03.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:03.463 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:03.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:03.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:03.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:03.590 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:03.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:03.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:03.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:03 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2762346540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:40:03.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:03 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2762346540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:40:03.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:03.718 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:03.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:03.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:03.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:03.844 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:03.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:03.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:03.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:03 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2762346540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:40:03.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:03.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:03.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:03.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:04.074 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:04.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:04.100 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:04.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:04.201 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:04.224 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:04.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:04.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:04.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:04.349 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:04.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:04.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:04.451 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:04.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:04.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:04.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:04.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:04.602 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:04.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:04.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:04.703 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:04.727 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:04.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:04.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:04.794 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_4' created 2024-09-10T18:40:04.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:04.852 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:04.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:04.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:04.954 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:04.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:04.977 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:04.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:05.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:04 smithi028 ceph-mon[29291]: pgmap v838: 162 pgs: 162 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 656 B/s rd, 0 op/s 2024-09-10T18:40:05.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:04 smithi028 ceph-mon[39220]: pgmap v838: 162 pgs: 162 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 656 B/s rd, 0 op/s 2024-09-10T18:40:05.079 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:05.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:05.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:05.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:05.103 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:04 smithi204 ceph-mon[32357]: pgmap v838: 162 pgs: 162 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 656 B/s rd, 0 op/s 2024-09-10T18:40:05.103 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:40:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:40:05.203 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:05.225 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:05.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:05.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:05.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:05.350 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:05.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:05.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:05.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:05.476 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:05.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:05.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:05.599 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:05.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:05.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:05.700 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:05.722 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:05.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:05.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:05.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:05.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:05.847 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:05.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:05.948 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:05.970 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:05.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:06.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:06.095 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:06.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:06.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:05 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:40:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:05 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:40:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:05 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3481049358' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-10T18:40:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:05 smithi204 ceph-mon[32357]: from='client.35846 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-10T18:40:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:05 smithi204 ceph-mon[32357]: from='client.35846 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-10T18:40:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:05 smithi204 ceph-mon[32357]: osdmap e374: 8 total, 8 up, 7 in 2024-09-10T18:40:06.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:05 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:06.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:05 smithi204 ceph-mon[32357]: pgmap v840: 178 pgs: 8 creating+peering, 8 unknown, 162 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:40:06.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:05 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:40:06.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:05 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:40:06.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:05 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3481049358' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-10T18:40:06.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:05 smithi028 ceph-mon[39220]: from='client.35846 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-10T18:40:06.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:05 smithi028 ceph-mon[39220]: from='client.35846 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-10T18:40:06.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:05 smithi028 ceph-mon[39220]: osdmap e374: 8 total, 8 up, 7 in 2024-09-10T18:40:06.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:05 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:06.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:05 smithi028 ceph-mon[39220]: pgmap v840: 178 pgs: 8 creating+peering, 8 unknown, 162 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:40:06.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:05 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:40:06.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:05 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:40:06.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:05 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3481049358' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-10T18:40:06.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:05 smithi028 ceph-mon[29291]: from='client.35846 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-10T18:40:06.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:05 smithi028 ceph-mon[29291]: from='client.35846 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-10T18:40:06.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:05 smithi028 ceph-mon[29291]: osdmap e374: 8 total, 8 up, 7 in 2024-09-10T18:40:06.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:05 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:06.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:05 smithi028 ceph-mon[29291]: pgmap v840: 178 pgs: 8 creating+peering, 8 unknown, 162 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:40:06.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:06.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-10T18:40:06.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:06.221 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:06.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:06.227 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:06.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:06.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:06.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:06.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:40:06.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:06.452 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:40:06.453 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:40:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:06.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:06.575 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:06.602 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:06.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:06.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:06.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:06.726 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:06.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:06.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:06.828 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:06.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:06.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:06.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:06.952 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:06.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:06.981 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:06.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:07.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:07.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:06 smithi204 ceph-mon[32357]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:40:07.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:06 smithi204 ceph-mon[32357]: osdmap e375: 8 total, 8 up, 7 in 2024-09-10T18:40:07.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:07.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:07.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:07.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:07.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:07.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:07.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:07.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:06 smithi028 ceph-mon[39220]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:40:07.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:06 smithi028 ceph-mon[39220]: osdmap e375: 8 total, 8 up, 7 in 2024-09-10T18:40:07.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:06 smithi028 ceph-mon[29291]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:40:07.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:06 smithi028 ceph-mon[29291]: osdmap e375: 8 total, 8 up, 7 in 2024-09-10T18:40:07.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:07.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:07.357 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:07.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:07.458 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:07.485 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:07.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:07.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:07.587 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:07.609 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:07.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:07.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:07.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:07.733 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:07.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:07.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:07.835 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:07.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:07.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:07.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:07.986 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:07.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:07.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:08.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-10T18:40:08.081 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:07 smithi028 ceph-mon[29291]: osdmap e376: 8 total, 8 up, 7 in 2024-09-10T18:40:08.081 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:07 smithi028 ceph-mon[29291]: pgmap v843: 178 pgs: 14 creating+peering, 164 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:40:08.082 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:07 smithi028 ceph-mon[39220]: osdmap e376: 8 total, 8 up, 7 in 2024-09-10T18:40:08.082 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:07 smithi028 ceph-mon[39220]: pgmap v843: 178 pgs: 14 creating+peering, 164 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:40:08.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:08.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:07 smithi204 ceph-mon[32357]: osdmap e376: 8 total, 8 up, 7 in 2024-09-10T18:40:08.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:07 smithi204 ceph-mon[32357]: pgmap v843: 178 pgs: 14 creating+peering, 164 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:40:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:08.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:08.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:08.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:08.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:08.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:08.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:08.336 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:08.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:08.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:08.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:08.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:08.486 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:08.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:08.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:08.588 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:08.614 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:08.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:08.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:08.716 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:08.742 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:08.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:08.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:08.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:08.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:08.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:08.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:08.991 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:08.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:08.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:09.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:09.103 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:08 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:40:09.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:08 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:40:09.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:08 smithi204 ceph-mon[32357]: osdmap e377: 8 total, 8 up, 7 in 2024-09-10T18:40:09.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:08 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:40:09.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:08 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:40:09.119 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:09.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:09.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:09.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:08 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:40:09.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:08 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:40:09.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:08 smithi028 ceph-mon[29291]: osdmap e377: 8 total, 8 up, 7 in 2024-09-10T18:40:09.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:08 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:40:09.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:08 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:40:09.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:08 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:40:09.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:08 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:40:09.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:08 smithi028 ceph-mon[39220]: osdmap e377: 8 total, 8 up, 7 in 2024-09-10T18:40:09.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:08 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:40:09.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:08 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:40:09.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:09.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:09.245 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:09.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:09.346 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:09.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:09.373 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:09.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:09.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:09.500 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:09.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:09.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:09.602 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:09.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:09.625 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:09.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:09.726 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:09.753 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:09.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:09.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:09.808 INFO:teuthology.orchestra.run.smithi028.stderr:set 3.1d pg_upmap_items mapping to [0->2,5->7,1->6] 2024-09-10T18:40:09.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:09.879 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:09.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:09.981 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:10.004 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:10.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:10.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:10.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:09 smithi204 ceph-mon[32357]: pgmap v845: 178 pgs: 14 creating+peering, 164 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:40:10.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:09 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3503104965' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.1d", "id": [0, 2, 5, 7, 1, 6]}]: dispatch 2024-09-10T18:40:10.106 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:10.130 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:10.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:10.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:10.207 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:09 smithi028 ceph-mon[29291]: pgmap v845: 178 pgs: 14 creating+peering, 164 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:40:10.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:09 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3503104965' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.1d", "id": [0, 2, 5, 7, 1, 6]}]: dispatch 2024-09-10T18:40:10.208 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:09 smithi028 ceph-mon[39220]: pgmap v845: 178 pgs: 14 creating+peering, 164 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:40:10.208 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:09 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3503104965' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.1d", "id": [0, 2, 5, 7, 1, 6]}]: dispatch 2024-09-10T18:40:10.232 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:10.255 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:10.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:10.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:10.357 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:10.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:10.383 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:10.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:10.484 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:10.509 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:10.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:10.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:10.636 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:10.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:10.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:10.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:10.764 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:10.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:10.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:10.866 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:10.891 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:10.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:10.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:10.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:11.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:11.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:11.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:10 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3503104965' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.1d", "id": [0, 2, 5, 7, 1, 6]}]': finished 2024-09-10T18:40:11.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:10 smithi204 ceph-mon[32357]: osdmap e378: 8 total, 8 up, 7 in 2024-09-10T18:40:11.112 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:10 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3503104965' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.1d", "id": [0, 2, 5, 7, 1, 6]}]': finished 2024-09-10T18:40:11.112 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:10 smithi028 ceph-mon[39220]: osdmap e378: 8 total, 8 up, 7 in 2024-09-10T18:40:11.112 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:10 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3503104965' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.1d", "id": [0, 2, 5, 7, 1, 6]}]': finished 2024-09-10T18:40:11.112 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:10 smithi028 ceph-mon[29291]: osdmap e378: 8 total, 8 up, 7 in 2024-09-10T18:40:11.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:11.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:11.245 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:11.269 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:11.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:11.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:11.344 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:11.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:11.395 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:11.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:11.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:11.460 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:40:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:40:11] "GET /metrics HTTP/1.1" 200 224645 "" "Prometheus/2.43.0" 2024-09-10T18:40:11.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:11.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:11.524 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:11.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:11.625 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:11.650 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:11.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:11.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:11.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:11.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:11.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:11.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:11.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:11.906 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:11.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:12.007 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:12.033 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:12.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:12.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:12.038 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:12.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:11 smithi028 ceph-mon[39220]: osdmap e379: 8 total, 8 up, 7 in 2024-09-10T18:40:12.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:11 smithi028 ceph-mon[39220]: pgmap v848: 178 pgs: 1 peering, 177 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:40:12.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:11 smithi028 ceph-mon[29291]: osdmap e379: 8 total, 8 up, 7 in 2024-09-10T18:40:12.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:11 smithi028 ceph-mon[29291]: pgmap v848: 178 pgs: 1 peering, 177 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:40:12.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:11 smithi204 ceph-mon[32357]: osdmap e379: 8 total, 8 up, 7 in 2024-09-10T18:40:12.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:11 smithi204 ceph-mon[32357]: pgmap v848: 178 pgs: 1 peering, 177 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:40:12.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:12.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:12.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:12.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:12.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:12.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:12.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:12.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:12.387 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:12.413 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:12.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:12.514 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:12.539 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:12.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:12.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:12.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:40:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:40:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:40:12.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:12.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:12.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:12.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:12.766 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:12.792 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:12.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:12.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:12.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:12.907 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:40:12.920 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:12.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:12.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:13.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:13.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:13.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:13.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:13.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:13.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:13.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:13.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:12 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2774224997' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:40:13.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:12 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2774224997' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:40:13.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:13.297 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:13.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:13.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:13.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:12 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2774224997' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:40:13.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:13.422 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:13.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:13.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:13.547 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:13.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:13.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:13.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:13.675 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:13.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:13.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:13.777 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:13.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:13.803 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:13.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:13.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:13.927 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:13.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:13.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:14.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:13 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:40:14.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:13 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2774224997' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:40:14.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:13 smithi028 ceph-mon[39220]: osdmap e380: 8 total, 8 up, 7 in 2024-09-10T18:40:14.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:13 smithi028 ceph-mon[39220]: pgmap v850: 178 pgs: 1 peering, 177 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 955 B/s rd, 0 op/s 2024-09-10T18:40:14.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:13 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:40:14.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:13 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2774224997' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:40:14.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:13 smithi028 ceph-mon[29291]: osdmap e380: 8 total, 8 up, 7 in 2024-09-10T18:40:14.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:13 smithi028 ceph-mon[29291]: pgmap v850: 178 pgs: 1 peering, 177 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 955 B/s rd, 0 op/s 2024-09-10T18:40:14.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:14.051 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:14.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:14.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:14.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:14.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:14.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:14.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:14.300 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:14.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:13 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:40:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:13 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2774224997' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:40:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:13 smithi204 ceph-mon[32357]: osdmap e380: 8 total, 8 up, 7 in 2024-09-10T18:40:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:13 smithi204 ceph-mon[32357]: pgmap v850: 178 pgs: 1 peering, 177 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 955 B/s rd, 0 op/s 2024-09-10T18:40:14.403 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:14.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:14.430 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:14.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:14.531 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:14.556 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:14.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:14.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:14.681 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:14.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:14.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:14.783 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:14.809 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:14.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:14.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:14.911 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:14.912 INFO:teuthology.orchestra.run.smithi028.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-10T18:40:14.936 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:14.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:14.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:15.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:15.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:15.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:15.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:15.185 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:15.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:15.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:15.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:14 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2663649448' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:40:15.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:14 smithi028 ceph-mon[29291]: from='client.35864 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:40:15.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:14 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2663649448' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:40:15.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:14 smithi028 ceph-mon[39220]: from='client.35864 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:40:15.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:15.309 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:15.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:15.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:15.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:14 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2663649448' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:40:15.357 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:14 smithi204 ceph-mon[32357]: from='client.35864 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:40:15.357 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:40:14 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:40:15.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:15.434 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:15.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:15.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:15.536 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:15.562 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:15.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:15.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:15.664 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:15.689 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:15.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:15.791 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:15.816 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:15.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:15.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:15.918 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:15.924 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:40:15.924 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:40:15.924 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-10T18:40:15.924 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-10T18:40:15.925 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-10T18:40:15.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-10T18:40:15.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:15.944 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:15.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:16.045 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:16.070 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:16.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:16.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:16.172 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:16.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:15 smithi028 ceph-mon[29291]: from='client.35864 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:40:16.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:15 smithi028 ceph-mon[29291]: osdmap e381: 8 total, 8 up, 7 in 2024-09-10T18:40:16.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:15 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:16.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:15 smithi028 ceph-mon[29291]: pgmap v852: 178 pgs: 178 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-09-10T18:40:16.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:15 smithi028 ceph-mon[39220]: from='client.35864 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:40:16.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:15 smithi028 ceph-mon[39220]: osdmap e381: 8 total, 8 up, 7 in 2024-09-10T18:40:16.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:15 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:16.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:15 smithi028 ceph-mon[39220]: pgmap v852: 178 pgs: 178 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-09-10T18:40:16.197 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:16.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:16.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:16.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:16.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:16.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:16.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:15 smithi204 ceph-mon[32357]: from='client.35864 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:40:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:15 smithi204 ceph-mon[32357]: osdmap e381: 8 total, 8 up, 7 in 2024-09-10T18:40:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:15 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:15 smithi204 ceph-mon[32357]: pgmap v852: 178 pgs: 178 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-09-10T18:40:16.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:16.425 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:40:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:40:16.446 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:40:16.446 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:40:16.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:16.450 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:16.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:16.461 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:40:16.462 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:40:16.476 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:40:16.476 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:40:16.492 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:40:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:40:16.508 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:40:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:40:16.525 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:40:16.525 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:40:16.541 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:40:16.541 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:40:16.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:16.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:16.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:16.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:16.676 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:16.701 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:16.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:16.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:16.803 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:16.829 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:16.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:16.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:16.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:40:16.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:16.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:16.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:16.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:17.055 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:17.082 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:17.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:17.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:17.184 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:17.209 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:17.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:17.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:17.225 DEBUG:teuthology.orchestra.run.smithi028:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-10T18:40:17.228 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-10T18:40:17.228 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-10T18:40:17.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:16 smithi028 ceph-mon[29291]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:40:17.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:16 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1390622225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:40:17.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:16 smithi028 ceph-mon[39220]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:40:17.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:16 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1390622225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:40:17.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:17.314 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:hints = 1 2024-09-10T18:40:17.314 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-10T18:40:17.314 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object prefix: benchmark_data_smithi028_186962 2024-09-10T18:40:17.314 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:40:17.314 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 0 0 0 0 0 0 - 0 2024-09-10T18:40:17.317 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:17.343 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:17.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:17.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:17.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:16 smithi204 ceph-mon[32357]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:40:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:16 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1390622225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:40:17.445 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:17.480 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:17.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:17.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:17.582 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:17.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:17.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:17.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:17.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:17.755 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:17.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:17.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:17.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:17.888 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:17.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:17.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:17.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:18.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:18.035 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:18.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:18.137 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:18.171 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:18.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:18.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:18.177 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:40:18.177 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-10T18:40:18.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-10T18:40:18.177 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:40:18.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.177 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35306.0:186254 9.4 e283/246 9:224dddb3:::benchmark_data_smithi028_135703_object51663:head v 283'11617, mlcod=283'11613)", 2024-09-10T18:40:18.177 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.202636+0000", 2024-09-10T18:40:18.177 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 352.96903536299999, 2024-09-10T18:40:18.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27734078200000001, 2024-09-10T18:40:18.177 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.177 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:18.178 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.202636+0000", 2024-09-10T18:40:18.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.202636+0000", 2024-09-10T18:40:18.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.202634+0000", 2024-09-10T18:40:18.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:18.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.202642+0000", 2024-09-10T18:40:18.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0439999999999998e-06 2024-09-10T18:40:18.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.202650+0000", 2024-09-10T18:40:18.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3070000000000003e-06 2024-09-10T18:40:18.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.202655+0000", 2024-09-10T18:40:18.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3109999999999996e-06 2024-09-10T18:40:18.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.202812+0000", 2024-09-10T18:40:18.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015717199999999999 2024-09-10T18:40:18.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.202829+0000", 2024-09-10T18:40:18.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6405999999999998e-05 2024-09-10T18:40:18.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:40:18.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.479939+0000", 2024-09-10T18:40:18.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27711055299999998 2024-09-10T18:40:18.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.479976+0000", 2024-09-10T18:40:18.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7248e-05 2024-09-10T18:40:18.182 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.182 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.182 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.182 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(osd.3.0:167479 9.c e283/246 ondisk, result = 0)", 2024-09-10T18:40:18.182 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.207950+0000", 2024-09-10T18:40:18.182 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 352.96372102599997, 2024-09-10T18:40:18.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27119121299999999, 2024-09-10T18:40:18.182 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:40:18.183 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207950+0000", 2024-09-10T18:40:18.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207950+0000", 2024-09-10T18:40:18.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207949+0000", 2024-09-10T18:40:18.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:18.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207955+0000", 2024-09-10T18:40:18.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7889999999999997e-06 2024-09-10T18:40:18.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207970+0000", 2024-09-10T18:40:18.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4432e-05 2024-09-10T18:40:18.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207976+0000", 2024-09-10T18:40:18.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.818e-06 2024-09-10T18:40:18.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.479125+0000", 2024-09-10T18:40:18.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27114970799999999 2024-09-10T18:40:18.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.479134+0000", 2024-09-10T18:40:18.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1329999999999999e-06 2024-09-10T18:40:18.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.479141+0000", 2024-09-10T18:40:18.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7409999999999992e-06 2024-09-10T18:40:18.186 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.186 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.186 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.186 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(osd.3.0:167479 9.c e283/246 ondisk, result = 0)", 2024-09-10T18:40:18.187 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.207987+0000", 2024-09-10T18:40:18.187 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 352.96368429400002, 2024-09-10T18:40:18.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.272007416, 2024-09-10T18:40:18.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:40:18.187 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207987+0000", 2024-09-10T18:40:18.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207987+0000", 2024-09-10T18:40:18.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207986+0000", 2024-09-10T18:40:18.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:40:18.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207990+0000", 2024-09-10T18:40:18.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.411e-06 2024-09-10T18:40:18.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207995+0000", 2024-09-10T18:40:18.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0010000000000001e-06 2024-09-10T18:40:18.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.207998+0000", 2024-09-10T18:40:18.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4089999999999999e-06 2024-09-10T18:40:18.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.479976+0000", 2024-09-10T18:40:18.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.271978359 2024-09-10T18:40:18.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.479983+0000", 2024-09-10T18:40:18.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3380000000000001e-06 2024-09-10T18:40:18.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.479994+0000", 2024-09-10T18:40:18.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1459e-05 2024-09-10T18:40:18.190 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.190 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.190 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.191 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35306.0:186267 9.c 9:32632338:::benchmark_data_smithi028_135703_object51676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-10T18:40:18.191 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.210599+0000", 2024-09-10T18:40:18.191 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 352.96107250300003, 2024-09-10T18:40:18.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.30479414999999999, 2024-09-10T18:40:18.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:18.191 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:40:18.191 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35306", 2024-09-10T18:40:18.191 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:40:18.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 186267 2024-09-10T18:40:18.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.191 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210599+0000", 2024-09-10T18:40:18.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210599+0000", 2024-09-10T18:40:18.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210596+0000", 2024-09-10T18:40:18.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:40:18.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210604+0000", 2024-09-10T18:40:18.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0539999999999995e-06 2024-09-10T18:40:18.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210618+0000", 2024-09-10T18:40:18.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3716e-05 2024-09-10T18:40:18.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210625+0000", 2024-09-10T18:40:18.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6599999999999998e-06 2024-09-10T18:40:18.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.500019+0000", 2024-09-10T18:40:18.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.28939448699999998 2024-09-10T18:40:18.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.500140+0000", 2024-09-10T18:40:18.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012023699999999999 2024-09-10T18:40:18.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,7", 2024-09-10T18:40:18.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.500290+0000", 2024-09-10T18:40:18.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015064000000000001 2024-09-10T18:40:18.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:18.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.510862+0000", 2024-09-10T18:40:18.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010571268 2024-09-10T18:40:18.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:18.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.510925+0000", 2024-09-10T18:40:18.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3137000000000002e-05 2024-09-10T18:40:18.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:40:18.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.515180+0000", 2024-09-10T18:40:18.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042550009999999996 2024-09-10T18:40:18.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:40:18.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.515221+0000", 2024-09-10T18:40:18.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0970000000000002e-05 2024-09-10T18:40:18.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.515393+0000", 2024-09-10T18:40:18.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017211400000000001 2024-09-10T18:40:18.197 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.197 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.197 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.197 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35306.0:186256 9.4 e283/246 9:2dc2a984:::benchmark_data_smithi028_135703_object51665:head v 283'11618, mlcod=283'11613)", 2024-09-10T18:40:18.197 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.210630+0000", 2024-09-10T18:40:18.197 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 352.96104098299998, 2024-09-10T18:40:18.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.28319999400000001, 2024-09-10T18:40:18.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.197 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:18.197 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210630+0000", 2024-09-10T18:40:18.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210630+0000", 2024-09-10T18:40:18.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210629+0000", 2024-09-10T18:40:18.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:40:18.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210636+0000", 2024-09-10T18:40:18.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2729999999999997e-06 2024-09-10T18:40:18.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210646+0000", 2024-09-10T18:40:18.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5030000000000003e-06 2024-09-10T18:40:18.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.210650+0000", 2024-09-10T18:40:18.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4599999999999996e-06 2024-09-10T18:40:18.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.482849+0000", 2024-09-10T18:40:18.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27219835599999997 2024-09-10T18:40:18.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.482865+0000", 2024-09-10T18:40:18.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6714e-05 2024-09-10T18:40:18.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:40:18.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.493803+0000", 2024-09-10T18:40:18.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010937631999999999 2024-09-10T18:40:18.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.493830+0000", 2024-09-10T18:40:18.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7231000000000001e-05 2024-09-10T18:40:18.201 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.201 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.201 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.202 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35306.0:186259 9.4 e283/246 9:2db7fb1c:::benchmark_data_smithi028_135703_object51668:head v 283'11619, mlcod=283'11615)", 2024-09-10T18:40:18.202 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.213559+0000", 2024-09-10T18:40:18.202 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 352.95811257100002, 2024-09-10T18:40:18.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.28029324799999999, 2024-09-10T18:40:18.202 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.202 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:18.202 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.213559+0000", 2024-09-10T18:40:18.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.213559+0000", 2024-09-10T18:40:18.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.213557+0000", 2024-09-10T18:40:18.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:18.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.213565+0000", 2024-09-10T18:40:18.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.388e-06 2024-09-10T18:40:18.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.213578+0000", 2024-09-10T18:40:18.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3093e-05 2024-09-10T18:40:18.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.213585+0000", 2024-09-10T18:40:18.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1999999999999999e-06 2024-09-10T18:40:18.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.485084+0000", 2024-09-10T18:40:18.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.271499306 2024-09-10T18:40:18.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.485113+0000", 2024-09-10T18:40:18.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8847e-05 2024-09-10T18:40:18.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:40:18.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.493833+0000", 2024-09-10T18:40:18.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087202779999999997 2024-09-10T18:40:18.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.493852+0000", 2024-09-10T18:40:18.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8729e-05 2024-09-10T18:40:18.206 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.206 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.206 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.206 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35306.0:186279 9.4 e283/246 9:28b03174:::benchmark_data_smithi028_135703_object51688:head v 283'11620, mlcod=283'11615)", 2024-09-10T18:40:18.206 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.224752+0000", 2024-09-10T18:40:18.206 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 352.94691905299999, 2024-09-10T18:40:18.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.26912506200000003, 2024-09-10T18:40:18.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:17 smithi028 ceph-mon[39220]: pgmap v853: 178 pgs: 178 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 806 B/s rd, 0 op/s 2024-09-10T18:40:18.207 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:17 smithi028 ceph-mon[29291]: pgmap v853: 178 pgs: 178 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 806 B/s rd, 0 op/s 2024-09-10T18:40:18.208 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:18.208 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.224752+0000", 2024-09-10T18:40:18.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.224752+0000", 2024-09-10T18:40:18.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.224751+0000", 2024-09-10T18:40:18.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:18.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.224759+0000", 2024-09-10T18:40:18.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6240000000000008e-06 2024-09-10T18:40:18.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.224773+0000", 2024-09-10T18:40:18.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3796e-05 2024-09-10T18:40:18.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.224779+0000", 2024-09-10T18:40:18.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6450000000000002e-06 2024-09-10T18:40:18.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.487193+0000", 2024-09-10T18:40:18.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.262414598 2024-09-10T18:40:18.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.487211+0000", 2024-09-10T18:40:18.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8011000000000001e-05 2024-09-10T18:40:18.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:40:18.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.493854+0000", 2024-09-10T18:40:18.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066430020000000003 2024-09-10T18:40:18.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.493877+0000", 2024-09-10T18:40:18.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2860000000000001e-05 2024-09-10T18:40:18.211 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.212 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.212 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.212 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35306.0:186281 9.4 e283/246 9:2314a6f4:::benchmark_data_smithi028_135703_object51690:head v 283'11621, mlcod=283'11615)", 2024-09-10T18:40:18.212 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.227701+0000", 2024-09-10T18:40:18.212 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 352.94397042600002, 2024-09-10T18:40:18.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27006845800000001, 2024-09-10T18:40:18.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:18.212 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227701+0000", 2024-09-10T18:40:18.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227701+0000", 2024-09-10T18:40:18.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227699+0000", 2024-09-10T18:40:18.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:40:18.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227707+0000", 2024-09-10T18:40:18.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1219999999999995e-06 2024-09-10T18:40:18.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227717+0000", 2024-09-10T18:40:18.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0511e-05 2024-09-10T18:40:18.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.227724+0000", 2024-09-10T18:40:18.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1580000000000002e-06 2024-09-10T18:40:18.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.489305+0000", 2024-09-10T18:40:18.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.26158115300000001 2024-09-10T18:40:18.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.489331+0000", 2024-09-10T18:40:18.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5987e-05 2024-09-10T18:40:18.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:40:18.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.497691+0000", 2024-09-10T18:40:18.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0083606610000000001 2024-09-10T18:40:18.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.497769+0000", 2024-09-10T18:40:18.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7673e-05 2024-09-10T18:40:18.216 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.216 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.216 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.216 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35306.0:186287 9.4 e283/246 9:2d5b8ef2:::benchmark_data_smithi028_135703_object51696:head v 283'11622, mlcod=283'11615)", 2024-09-10T18:40:18.216 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.232647+0000", 2024-09-10T18:40:18.217 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 352.93902417999999, 2024-09-10T18:40:18.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.26978486099999999, 2024-09-10T18:40:18.217 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.217 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:18.217 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232647+0000", 2024-09-10T18:40:18.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232647+0000", 2024-09-10T18:40:18.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232646+0000", 2024-09-10T18:40:18.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:18.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232659+0000", 2024-09-10T18:40:18.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2993000000000001e-05 2024-09-10T18:40:18.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232668+0000", 2024-09-10T18:40:18.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8660000000000001e-06 2024-09-10T18:40:18.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.232675+0000", 2024-09-10T18:40:18.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5259999999999999e-06 2024-09-10T18:40:18.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.491585+0000", 2024-09-10T18:40:18.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25890969200000002 2024-09-10T18:40:18.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.491603+0000", 2024-09-10T18:40:18.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8196999999999999e-05 2024-09-10T18:40:18.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:40:18.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.502399+0000", 2024-09-10T18:40:18.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010796155 2024-09-10T18:40:18.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.502432+0000", 2024-09-10T18:40:18.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2765000000000002e-05 2024-09-10T18:40:18.220 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.221 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.221 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.221 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35306.0:186291 9.4 e283/246 9:22863b05:::benchmark_data_smithi028_135703_object51700:head v 283'11623, mlcod=283'11615)", 2024-09-10T18:40:18.221 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:34:25.239764+0000", 2024-09-10T18:40:18.221 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 352.93190666700002, 2024-09-10T18:40:18.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27116428799999998, 2024-09-10T18:40:18.221 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:18.221 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.239764+0000", 2024-09-10T18:40:18.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.239764+0000", 2024-09-10T18:40:18.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.239763+0000", 2024-09-10T18:40:18.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:18.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.239770+0000", 2024-09-10T18:40:18.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.041e-06 2024-09-10T18:40:18.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.239780+0000", 2024-09-10T18:40:18.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4269999999999997e-06 2024-09-10T18:40:18.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.239785+0000", 2024-09-10T18:40:18.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2310000000000004e-06 2024-09-10T18:40:18.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.493902+0000", 2024-09-10T18:40:18.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25411749500000003 2024-09-10T18:40:18.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.493921+0000", 2024-09-10T18:40:18.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8732e-05 2024-09-10T18:40:18.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:40:18.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.510866+0000", 2024-09-10T18:40:18.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016945248 2024-09-10T18:40:18.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:34:25.510929+0000", 2024-09-10T18:40:18.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2322000000000004e-05 2024-09-10T18:40:18.226 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.226 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.227 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.227 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:189285 10.f e348/314 10:fd4804a8:::benchmark_data_smithi028_161947_object52361:head v 348'11938, mlcod=348'11936)", 2024-09-10T18:40:18.227 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:30.121829+0000", 2024-09-10T18:40:18.227 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 108.049842266, 2024-09-10T18:40:18.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27094062400000002, 2024-09-10T18:40:18.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:18.227 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.121829+0000", 2024-09-10T18:40:18.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.121829+0000", 2024-09-10T18:40:18.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.121827+0000", 2024-09-10T18:40:18.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:18.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.121837+0000", 2024-09-10T18:40:18.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6989999999999998e-06 2024-09-10T18:40:18.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.121848+0000", 2024-09-10T18:40:18.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1029999999999999e-05 2024-09-10T18:40:18.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.121854+0000", 2024-09-10T18:40:18.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4019999999999999e-06 2024-09-10T18:40:18.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.125200+0000", 2024-09-10T18:40:18.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033453390000000001 2024-09-10T18:40:18.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.125220+0000", 2024-09-10T18:40:18.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0452e-05 2024-09-10T18:40:18.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:40:18.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.392748+0000", 2024-09-10T18:40:18.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.267528027 2024-09-10T18:40:18.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.392769+0000", 2024-09-10T18:40:18.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1285999999999999e-05 2024-09-10T18:40:18.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.231 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.231 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:189283 10.5 e348/314 10:a2bd8b5d:::benchmark_data_smithi028_161947_object52359:head v 348'11919, mlcod=348'11915)", 2024-09-10T18:40:18.231 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:30.121920+0000", 2024-09-10T18:40:18.231 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 108.049751297, 2024-09-10T18:40:18.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.270387761, 2024-09-10T18:40:18.232 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.232 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:18.232 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.121920+0000", 2024-09-10T18:40:18.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.121920+0000", 2024-09-10T18:40:18.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.121919+0000", 2024-09-10T18:40:18.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:40:18.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.121924+0000", 2024-09-10T18:40:18.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2520000000000001e-06 2024-09-10T18:40:18.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.121928+0000", 2024-09-10T18:40:18.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4950000000000002e-06 2024-09-10T18:40:18.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.121931+0000", 2024-09-10T18:40:18.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0520000000000002e-06 2024-09-10T18:40:18.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.121972+0000", 2024-09-10T18:40:18.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0596000000000002e-05 2024-09-10T18:40:18.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.121991+0000", 2024-09-10T18:40:18.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9363000000000001e-05 2024-09-10T18:40:18.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:40:18.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.128047+0000", 2024-09-10T18:40:18.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060556500000000001 2024-09-10T18:40:18.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.392308+0000", 2024-09-10T18:40:18.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.26426073300000003 2024-09-10T18:40:18.236 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.236 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.236 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.236 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:189294 10.7 e348/314 10:e11028da:::benchmark_data_smithi028_161947_object52370:head v 348'11793, mlcod=348'11791)", 2024-09-10T18:40:18.236 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:30.122420+0000", 2024-09-10T18:40:18.236 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 108.049250601, 2024-09-10T18:40:18.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27346151299999999, 2024-09-10T18:40:18.237 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.237 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:18.237 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.122420+0000", 2024-09-10T18:40:18.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.122420+0000", 2024-09-10T18:40:18.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.122419+0000", 2024-09-10T18:40:18.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:40:18.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.122426+0000", 2024-09-10T18:40:18.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2609999999999999e-06 2024-09-10T18:40:18.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.122434+0000", 2024-09-10T18:40:18.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1780000000000003e-06 2024-09-10T18:40:18.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.122438+0000", 2024-09-10T18:40:18.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5580000000000002e-06 2024-09-10T18:40:18.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.126823+0000", 2024-09-10T18:40:18.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0043844899999999996 2024-09-10T18:40:18.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.126842+0000", 2024-09-10T18:40:18.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8944999999999999e-05 2024-09-10T18:40:18.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:40:18.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.395839+0000", 2024-09-10T18:40:18.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.26899716600000001 2024-09-10T18:40:18.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.395882+0000", 2024-09-10T18:40:18.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2964999999999999e-05 2024-09-10T18:40:18.241 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.241 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.241 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.241 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:189303 10.f e348/314 10:f047d9db:::benchmark_data_smithi028_161947_object52379:head v 348'11939, mlcod=348'11936)", 2024-09-10T18:40:18.241 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:38:30.128935+0000", 2024-09-10T18:40:18.242 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 108.042736252, 2024-09-10T18:40:18.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.275062739, 2024-09-10T18:40:18.242 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:18.242 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.128935+0000", 2024-09-10T18:40:18.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.128935+0000", 2024-09-10T18:40:18.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.128933+0000", 2024-09-10T18:40:18.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:18.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.128942+0000", 2024-09-10T18:40:18.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2640000000000008e-06 2024-09-10T18:40:18.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.392255+0000", 2024-09-10T18:40:18.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.263312975 2024-09-10T18:40:18.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.392272+0000", 2024-09-10T18:40:18.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7167999999999999e-05 2024-09-10T18:40:18.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.392825+0000", 2024-09-10T18:40:18.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00055316700000000003 2024-09-10T18:40:18.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.392845+0000", 2024-09-10T18:40:18.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0265e-05 2024-09-10T18:40:18.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:40:18.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.403957+0000", 2024-09-10T18:40:18.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011111607000000001 2024-09-10T18:40:18.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:38:30.403998+0000", 2024-09-10T18:40:18.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0837000000000002e-05 2024-09-10T18:40:18.246 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.246 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.246 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.246 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:253957 10.7 e363/356 10:ec157e1b:::benchmark_data_smithi028_161947_object117033:head v 363'15867, mlcod=363'15864)", 2024-09-10T18:40:18.246 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:39:26.815361+0000", 2024-09-10T18:40:18.246 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 51.356309905000003, 2024-09-10T18:40:18.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.35679616800000002, 2024-09-10T18:40:18.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.246 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:18.246 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.815361+0000", 2024-09-10T18:40:18.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.815361+0000", 2024-09-10T18:40:18.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.815359+0000", 2024-09-10T18:40:18.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:40:18.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.815368+0000", 2024-09-10T18:40:18.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3410000000000003e-06 2024-09-10T18:40:18.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.815379+0000", 2024-09-10T18:40:18.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1243000000000001e-05 2024-09-10T18:40:18.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.815385+0000", 2024-09-10T18:40:18.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2169999999999999e-06 2024-09-10T18:40:18.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.815467+0000", 2024-09-10T18:40:18.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1343999999999999e-05 2024-09-10T18:40:18.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.815488+0000", 2024-09-10T18:40:18.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1073999999999998e-05 2024-09-10T18:40:18.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:40:18.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.172121+0000", 2024-09-10T18:40:18.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.35663373900000001 2024-09-10T18:40:18.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.172157+0000", 2024-09-10T18:40:18.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5917999999999997e-05 2024-09-10T18:40:18.250 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.250 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.250 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.250 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:253955 10.5 e363/362 10:a63fc3e3:::benchmark_data_smithi028_161947_object117031:head v 363'15921, mlcod=363'15919)", 2024-09-10T18:40:18.250 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:39:26.818102+0000", 2024-09-10T18:40:18.250 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 51.353568611, 2024-09-10T18:40:18.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.35426194999999999, 2024-09-10T18:40:18.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.250 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:18.250 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.818102+0000", 2024-09-10T18:40:18.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.818102+0000", 2024-09-10T18:40:18.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.818102+0000", 2024-09-10T18:40:18.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:40:18.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.818108+0000", 2024-09-10T18:40:18.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7689999999999999e-06 2024-09-10T18:40:18.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.818116+0000", 2024-09-10T18:40:18.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7530000000000001e-06 2024-09-10T18:40:18.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.818121+0000", 2024-09-10T18:40:18.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9570000000000001e-06 2024-09-10T18:40:18.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.818198+0000", 2024-09-10T18:40:18.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7280000000000005e-05 2024-09-10T18:40:18.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.818219+0000", 2024-09-10T18:40:18.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1126000000000001e-05 2024-09-10T18:40:18.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:40:18.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.172337+0000", 2024-09-10T18:40:18.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.35411800300000001 2024-09-10T18:40:18.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.172364+0000", 2024-09-10T18:40:18.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6954e-05 2024-09-10T18:40:18.254 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.254 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.254 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.255 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:253960 10.4 e363/362 10:2096eec1:::benchmark_data_smithi028_161947_object117036:head v 363'15783, mlcod=363'15780)", 2024-09-10T18:40:18.255 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:39:26.818268+0000", 2024-09-10T18:40:18.255 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 51.353403231999998, 2024-09-10T18:40:18.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.35936031899999998, 2024-09-10T18:40:18.255 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.255 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:18.255 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.818268+0000", 2024-09-10T18:40:18.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.818268+0000", 2024-09-10T18:40:18.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.818267+0000", 2024-09-10T18:40:18.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:18.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.818274+0000", 2024-09-10T18:40:18.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5780000000000001e-06 2024-09-10T18:40:18.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.818285+0000", 2024-09-10T18:40:18.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0349999999999999e-05 2024-09-10T18:40:18.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.818290+0000", 2024-09-10T18:40:18.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7019999999999998e-06 2024-09-10T18:40:18.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.172346+0000", 2024-09-10T18:40:18.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.35405577900000001 2024-09-10T18:40:18.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.172370+0000", 2024-09-10T18:40:18.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4372999999999999e-05 2024-09-10T18:40:18.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:40:18.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.177587+0000", 2024-09-10T18:40:18.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052161569999999999 2024-09-10T18:40:18.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.177628+0000", 2024-09-10T18:40:18.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1587e-05 2024-09-10T18:40:18.259 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.259 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.259 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.259 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(10.d 363/362 [PushOp(10:b07e223d:::benchmark_data_smithi028_161947_object119516:head, version: 340'7480, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(10:b07e223d:::benchmark_data_smithi028_161947_object119516:head@340'7480, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-10T18:40:18.259 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:39:26.821205+0000", 2024-09-10T18:40:18.259 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 51.350466349000001, 2024-09-10T18:40:18.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.35120765300000001, 2024-09-10T18:40:18.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:40:18.259 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.821205+0000", 2024-09-10T18:40:18.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.821205+0000", 2024-09-10T18:40:18.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.821203+0000", 2024-09-10T18:40:18.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:18.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.821246+0000", 2024-09-10T18:40:18.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2372e-05 2024-09-10T18:40:18.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.171477+0000", 2024-09-10T18:40:18.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.35023110400000002 2024-09-10T18:40:18.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.171493+0000", 2024-09-10T18:40:18.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6492000000000001e-05 2024-09-10T18:40:18.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.172225+0000", 2024-09-10T18:40:18.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00073233200000000004 2024-09-10T18:40:18.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.172234+0000", 2024-09-10T18:40:18.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7190000000000007e-06 2024-09-10T18:40:18.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.172412+0000", 2024-09-10T18:40:18.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000178164 2024-09-10T18:40:18.263 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.263 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.263 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.263 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:253970 10.2 e363/362 10:4aad18a8:::benchmark_data_smithi028_161947_object117046:head v 363'16226, mlcod=363'16224)", 2024-09-10T18:40:18.263 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:39:26.826352+0000", 2024-09-10T18:40:18.264 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 51.345319471000003, 2024-09-10T18:40:18.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.34899627700000002, 2024-09-10T18:40:18.264 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.264 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:18.264 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.826352+0000", 2024-09-10T18:40:18.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.826352+0000", 2024-09-10T18:40:18.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.826350+0000", 2024-09-10T18:40:18.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:40:18.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.826361+0000", 2024-09-10T18:40:18.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1189000000000001e-05 2024-09-10T18:40:18.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.171462+0000", 2024-09-10T18:40:18.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.345101618 2024-09-10T18:40:18.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.171468+0000", 2024-09-10T18:40:18.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4569999999999996e-06 2024-09-10T18:40:18.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.171743+0000", 2024-09-10T18:40:18.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027548299999999998 2024-09-10T18:40:18.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.171766+0000", 2024-09-10T18:40:18.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2266e-05 2024-09-10T18:40:18.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:40:18.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.175315+0000", 2024-09-10T18:40:18.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003549642 2024-09-10T18:40:18.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.175348+0000", 2024-09-10T18:40:18.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2718000000000001e-05 2024-09-10T18:40:18.267 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.268 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.268 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.268 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.15693.0:253971 10.4 e363/362 10:2bc86aca:::benchmark_data_smithi028_161947_object117047:head v 363'15784, mlcod=363'15781)", 2024-09-10T18:40:18.268 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:39:26.829359+0000", 2024-09-10T18:40:18.268 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 51.342311635999998, 2024-09-10T18:40:18.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.34833098099999998, 2024-09-10T18:40:18.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:40:18.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:18.268 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:40:18.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:40:18.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.829359+0000", 2024-09-10T18:40:18.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:40:18.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.829359+0000", 2024-09-10T18:40:18.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:40:18.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:40:18.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.829358+0000", 2024-09-10T18:40:18.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:18.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:40:18.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:26.829367+0000", 2024-09-10T18:40:18.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4839999999999993e-06 2024-09-10T18:40:18.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:40:18.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.171436+0000", 2024-09-10T18:40:18.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.342068751 2024-09-10T18:40:18.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:40:18.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.171452+0000", 2024-09-10T18:40:18.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.605e-05 2024-09-10T18:40:18.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:40:18.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.174260+0000", 2024-09-10T18:40:18.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028082409999999999 2024-09-10T18:40:18.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:40:18.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.174280+0000", 2024-09-10T18:40:18.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0097000000000001e-05 2024-09-10T18:40:18.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-10T18:40:18.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.177633+0000", 2024-09-10T18:40:18.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033528310000000001 2024-09-10T18:40:18.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:40:18.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:40:18.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-10T18:40:18.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:39:27.177690+0000", 2024-09-10T18:40:18.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7122e-05 2024-09-10T18:40:18.272 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.272 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.272 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.272 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:40:18.272 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:40:18.272 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:40:18.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:18.306 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:18.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:18.314 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1 16 3469 3453 215.8 215.812 0.00322539 0.00461558 2024-09-10T18:40:18.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:17 smithi204 ceph-mon[32357]: pgmap v853: 178 pgs: 178 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 806 B/s rd, 0 op/s 2024-09-10T18:40:18.408 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:18.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:18.442 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:18.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:18.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:18.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:18.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:18.687 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:18.719 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:18.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:18.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:18.821 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:18.853 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:18.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:18.955 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:18.988 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:18.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:18.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:19.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:19.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:19.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:19.264 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:19.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:19.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:19.315 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2 16 6795 6779 211.828 207.875 0.00619649 0.00470977 2024-09-10T18:40:19.365 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:19.397 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:19.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:19.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:19.531 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:19.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:19.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:19.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:19.672 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:19.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:19.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:19.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:19.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:19.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:19.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:19.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:19.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:20.059 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:20.093 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:20.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:20.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:20.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:20.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:20.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:20.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:20.315 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3 16 9954 9938 207.026 197.438 0.00483921 0.00481979 2024-09-10T18:40:20.335 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:20.369 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:20.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:20.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:20.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:20.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:20 smithi028 ceph-mon[39220]: pgmap v854: 178 pgs: 178 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:40:20.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:20 smithi028 ceph-mon[29291]: pgmap v854: 178 pgs: 178 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:40:20.511 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:20.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:20.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:20 smithi204 ceph-mon[32357]: pgmap v854: 178 pgs: 178 active+clean; 470 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:40:20.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:20.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:20.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:20.754 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:20.786 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:20.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:20.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:20.882 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:20.888 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:20.919 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:20.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:20.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:21.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:21.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:40:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:21.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:21.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:21.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:21.195 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:21.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:21.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:21.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:21.314 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4 16 13174 13158 205.577 201.25 0.0020959 0.00485292 2024-09-10T18:40:21.322 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:21.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:21.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:21.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:21.457 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:21.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:21.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:21.559 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:21.594 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:21.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:21.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:21.696 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:21.729 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:21.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:21.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:21.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:40:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:40:21] "GET /metrics HTTP/1.1" 200 224645 "" "Prometheus/2.43.0" 2024-09-10T18:40:21.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:21.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:21.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:21.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:22.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:22.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:22.089 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-10T18:40:22.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:22.172 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:22.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:22.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:22.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:22.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:22.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:22.315 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5 16 16205 16189 202.329 189.438 0.00627839 0.00493609 2024-09-10T18:40:22.406 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:22 smithi204 ceph-mon[32357]: pgmap v855: 178 pgs: 178 active+clean; 619 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-10T18:40:22.406 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:40:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:40:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:40:22.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:22.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:22.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:22.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:22 smithi028 ceph-mon[39220]: pgmap v855: 178 pgs: 178 active+clean; 619 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-10T18:40:22.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:22 smithi028 ceph-mon[29291]: pgmap v855: 178 pgs: 178 active+clean; 619 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-10T18:40:22.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:22.569 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:40:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:22.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:22.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:22.685 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:22.724 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:22.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:22.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:22.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:22.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:22.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:22.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:22.889 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:40:22.889 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "" 2024-09-10T18:40:22.889 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:40:22.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:23.002 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:23.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:23.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:23.104 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:23.141 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:23.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:23.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:23.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:23.277 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:23.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:23.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:23.315 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 6 16 18674 18658 194.325 154.312 0.00726583 0.00513918 2024-09-10T18:40:23.379 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:23.413 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:23.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:23.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:23.515 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:23.550 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:23.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:23.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:23.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:23.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:23.691 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:23.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:23.792 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:23.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:23.826 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:23.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:23.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:23.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:23.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:24.069 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:24.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:24.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:24.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:24.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:24.236 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:24.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:24.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:24.315 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 7 15 21012 20997 187.446 146.188 0.00381221 0.00532804 2024-09-10T18:40:24.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:24.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:24 smithi204 ceph-mon[32357]: pgmap v856: 178 pgs: 178 active+clean; 619 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 62 MiB/s wr, 990 op/s 2024-09-10T18:40:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:24 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:40:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:40:24.368 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:24.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:24.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:24.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:24 smithi028 ceph-mon[39220]: pgmap v856: 178 pgs: 178 active+clean; 619 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 62 MiB/s wr, 990 op/s 2024-09-10T18:40:24.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:24 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:40:24.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:40:24.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:24 smithi028 ceph-mon[29291]: pgmap v856: 178 pgs: 178 active+clean; 619 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 62 MiB/s wr, 990 op/s 2024-09-10T18:40:24.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:24 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:40:24.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:40:24.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:24.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:24.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:24.609 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:24.643 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:24.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:24.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:24.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:24.783 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:24.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:24.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:24.884 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:24.919 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:24.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:24.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:25.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:25.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:25.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:25.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:25.159 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:25.192 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:25.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:25.294 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:25.315 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 8 16 23194 23178 181.053 136.312 0.00452202 0.00551678 2024-09-10T18:40:25.332 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:25.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:25.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:25.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:40:24 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:40:25.434 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:25.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:25.459 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:25.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:25.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:25.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:25.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:25.700 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:25.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:25.731 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:25.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:25.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:25.870 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:25.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:25.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:25.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:25.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:25.995 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:25.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:26.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:26.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:26.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:26.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:26.234 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:26.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:26.270 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:26.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:26.315 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 9 16 25304 25288 175.588 131.875 0.00411584 0.00568801 2024-09-10T18:40:26.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:26.414 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:26.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:26.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:26.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:26 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:26.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:26 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:40:26.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:26 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:40:26.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:26 smithi028 ceph-mon[39220]: pgmap v857: 178 pgs: 178 active+clean; 1.0 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.5 KiB/s rd, 103 MiB/s wr, 1.64k op/s 2024-09-10T18:40:26.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:26 smithi028 ceph-mon[39220]: osdmap e382: 8 total, 8 up, 7 in 2024-09-10T18:40:26.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:26 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:26.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:26 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:40:26.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:26 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:40:26.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:26 smithi028 ceph-mon[29291]: pgmap v857: 178 pgs: 178 active+clean; 1.0 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.5 KiB/s rd, 103 MiB/s wr, 1.64k op/s 2024-09-10T18:40:26.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:26 smithi028 ceph-mon[29291]: osdmap e382: 8 total, 8 up, 7 in 2024-09-10T18:40:26.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:26 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:26.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:26 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:40:26.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:26 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:40:26.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:26 smithi204 ceph-mon[32357]: pgmap v857: 178 pgs: 178 active+clean; 1.0 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.5 KiB/s rd, 103 MiB/s wr, 1.64k op/s 2024-09-10T18:40:26.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:26 smithi204 ceph-mon[32357]: osdmap e382: 8 total, 8 up, 7 in 2024-09-10T18:40:26.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:26.553 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:26.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:26.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:26.656 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:26.681 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:26.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:26.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:26.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:26.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:26.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:26.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:26.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:26.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:26.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:26.979 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:40:26.979 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:40:26.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.980 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:26105 11.e 11:7de3e6e9:::benchmark_data_smithi028_186962_object26104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-10T18:40:26.980 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:40:26.971590+0000", 2024-09-10T18:40:26.980 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0044275490000000002, 2024-09-10T18:40:26.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0044606630000000001, 2024-09-10T18:40:26.980 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:26.980 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:40:26.980 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:26.980 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:40:26.980 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:40:26.980 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 26105 2024-09-10T18:40:26.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.981 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:26.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:26.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.971590+0000", 2024-09-10T18:40:26.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:26.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:26.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.971590+0000", 2024-09-10T18:40:26.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:26.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:26.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.971587+0000", 2024-09-10T18:40:26.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:40:26.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:26.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.971624+0000", 2024-09-10T18:40:26.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.693e-05 2024-09-10T18:40:26.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:26.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.971642+0000", 2024-09-10T18:40:26.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7246000000000001e-05 2024-09-10T18:40:26.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:26.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.971648+0000", 2024-09-10T18:40:26.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3960000000000003e-06 2024-09-10T18:40:26.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:26.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.971704+0000", 2024-09-10T18:40:26.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6310999999999997e-05 2024-09-10T18:40:26.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:26.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.971825+0000", 2024-09-10T18:40:26.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000120303 2024-09-10T18:40:26.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,7", 2024-09-10T18:40:26.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.971969+0000", 2024-09-10T18:40:26.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014437900000000001 2024-09-10T18:40:26.984 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:26.984 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:26.984 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:26.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.985 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:26108 11.e 11:7085b9cb:::benchmark_data_smithi028_186962_object26107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-10T18:40:26.985 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:40:26.973979+0000", 2024-09-10T18:40:26.985 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.002038691, 2024-09-10T18:40:26.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021607499999999999, 2024-09-10T18:40:26.985 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:26.985 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:40:26.985 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:26.985 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:40:26.985 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:40:26.985 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 26108 2024-09-10T18:40:26.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.985 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:26.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:26.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.973979+0000", 2024-09-10T18:40:26.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:26.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:26.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.973979+0000", 2024-09-10T18:40:26.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:26.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:26.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.973976+0000", 2024-09-10T18:40:26.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:40:26.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:26.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.974022+0000", 2024-09-10T18:40:26.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6031e-05 2024-09-10T18:40:26.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:26.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.974036+0000", 2024-09-10T18:40:26.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4090999999999999e-05 2024-09-10T18:40:26.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:26.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.974043+0000", 2024-09-10T18:40:26.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.545e-06 2024-09-10T18:40:26.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:26.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.974101+0000", 2024-09-10T18:40:26.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8189999999999997e-05 2024-09-10T18:40:26.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:26.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.974248+0000", 2024-09-10T18:40:26.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014697399999999999 2024-09-10T18:40:26.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,7", 2024-09-10T18:40:26.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.974384+0000", 2024-09-10T18:40:26.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013563599999999999 2024-09-10T18:40:26.990 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:26.990 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:26.990 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:26.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.990 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.25849.0:26105 11.e e382/374)", 2024-09-10T18:40:26.990 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:40:26.974152+0000", 2024-09-10T18:40:26.990 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.001865418, 2024-09-10T18:40:26.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020580009999999998, 2024-09-10T18:40:26.990 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:26.991 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "queued for pg", 2024-09-10T18:40:26.991 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:26.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:26.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.974152+0000", 2024-09-10T18:40:26.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:26.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:26.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.974152+0000", 2024-09-10T18:40:26.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:26.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:26.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.974151+0000", 2024-09-10T18:40:26.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:26.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:26.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.974158+0000", 2024-09-10T18:40:26.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9970000000000001e-06 2024-09-10T18:40:26.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:26.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.974170+0000", 2024-09-10T18:40:26.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2798e-05 2024-09-10T18:40:26.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:26.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.974176+0000", 2024-09-10T18:40:26.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0499999999999997e-06 2024-09-10T18:40:26.993 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:26.993 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:26.993 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:26.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.993 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25849.0:26109 11.7 e382/374 11:e16c5abb:::benchmark_data_smithi028_186962_object26108:head v 382'1626, mlcod=382'1624)", 2024-09-10T18:40:26.993 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:40:26.975242+0000", 2024-09-10T18:40:26.994 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.000775346, 2024-09-10T18:40:26.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001017977, 2024-09-10T18:40:26.994 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:26.994 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:40:26.994 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:26.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:26.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975242+0000", 2024-09-10T18:40:26.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:26.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:26.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975242+0000", 2024-09-10T18:40:26.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:26.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:26.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975241+0000", 2024-09-10T18:40:26.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:26.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:26.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975267+0000", 2024-09-10T18:40:26.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6662e-05 2024-09-10T18:40:26.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:26.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975280+0000", 2024-09-10T18:40:26.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2445e-05 2024-09-10T18:40:26.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:26.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975286+0000", 2024-09-10T18:40:26.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9549999999999997e-06 2024-09-10T18:40:26.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:26.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975322+0000", 2024-09-10T18:40:26.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6619000000000003e-05 2024-09-10T18:40:26.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:26.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975340+0000", 2024-09-10T18:40:26.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.747e-05 2024-09-10T18:40:26.997 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:26.997 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:26.998 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:26.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.998 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25849.0:26111 11.7 e382/374)", 2024-09-10T18:40:26.998 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:40:26.975459+0000", 2024-09-10T18:40:26.998 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.00055836899999999997, 2024-09-10T18:40:26.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00085959000000000003, 2024-09-10T18:40:26.998 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:26.998 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "queued for pg", 2024-09-10T18:40:26.998 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:26.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:26.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975459+0000", 2024-09-10T18:40:26.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:26.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:26.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975459+0000", 2024-09-10T18:40:26.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:26.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:26.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975458+0000", 2024-09-10T18:40:26.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:26.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:26.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:26.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:27.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975485+0000", 2024-09-10T18:40:27.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7001000000000001e-05 2024-09-10T18:40:27.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:27.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:27.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:27.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975495+0000", 2024-09-10T18:40:27.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0069e-05 2024-09-10T18:40:27.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:27.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:27.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:27.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975500+0000", 2024-09-10T18:40:27.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5790000000000003e-06 2024-09-10T18:40:27.001 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:27.001 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:27.001 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:27.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:27.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:27.001 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.25849.0:26105 11.e e382/374)", 2024-09-10T18:40:27.001 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:40:26.975958+0000", 2024-09-10T18:40:27.001 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 5.9301e-05, 2024-09-10T18:40:27.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00042433999999999999, 2024-09-10T18:40:27.001 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:27.001 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "queued for pg", 2024-09-10T18:40:27.001 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:27.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:27.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:27.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975958+0000", 2024-09-10T18:40:27.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:27.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:27.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:27.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:27.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975958+0000", 2024-09-10T18:40:27.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:27.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:27.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:27.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:27.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975957+0000", 2024-09-10T18:40:27.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:27.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:27.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:27.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:27.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975966+0000", 2024-09-10T18:40:27.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2089999999999994e-06 2024-09-10T18:40:27.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:27.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:27.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:27.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975975+0000", 2024-09-10T18:40:27.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2399999999999996e-06 2024-09-10T18:40:27.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:27.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:27.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:27.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.975980+0000", 2024-09-10T18:40:27.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.69e-06 2024-09-10T18:40:27.004 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:27.004 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:27.004 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:27.004 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:27.004 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-09-10T18:40:27.004 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 6 2024-09-10T18:40:27.004 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:40:27.049 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:27.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:27.073 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:27.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:27.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:27.215 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:27.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:27.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:27.316 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 10 16 26790 26774 167.317 92.875 0.00187757 0.00596549 2024-09-10T18:40:27.317 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:27.338 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:27.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:27.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:27.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:27.464 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:27.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:27.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:27.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:27.606 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:27.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:27.707 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:27.743 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:27.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:27.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:27.796 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:27.845 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:27.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:27.879 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:27.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:27.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-10T18:40:27.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:28.015 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:28.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:28.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:28.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:28.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:28.154 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:28.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:28.255 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:28.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:28.304 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:28.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:28.315 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 11 16 28544 28528 162.071 109.625 0.00516816 0.00616495 2024-09-10T18:40:28.405 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:40:28.405 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:28.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:28.448 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:28.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:28.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:28 smithi028 ceph-mon[39220]: pgmap v859: 178 pgs: 178 active+clean; 1.4 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.32k op/s 2024-09-10T18:40:28.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:28 smithi028 ceph-mon[29291]: pgmap v859: 178 pgs: 178 active+clean; 1.4 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.32k op/s 2024-09-10T18:40:28.544 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:28 smithi204 ceph-mon[32357]: pgmap v859: 178 pgs: 178 active+clean; 1.4 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.32k op/s 2024-09-10T18:40:28.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:28.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:28.585 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:28.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:28.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:28.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:28.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:28.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:28.864 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:28.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:28.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:28.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:28.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:28.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:29.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:29.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:29.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:29.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:29.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:29.280 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:29.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:29.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:29.316 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 12 16 30705 30689 159.82 135.062 0.0133386 0.00625173 2024-09-10T18:40:29.381 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:29.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:29.420 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:29.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:29.522 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:29.558 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:29.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:29.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:29.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:29.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:29.706 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:29.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:29.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:29.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:29.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:29.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:29.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:29.995 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:29.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:30.097 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:30.132 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:30.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:30.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:30.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:30.281 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:30.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:30.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:30.316 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 13 16 32825 32809 157.716 132.5 0.0115016 0.00633495 2024-09-10T18:40:30.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:30.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:30.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:30.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:30.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:30 smithi028 ceph-mon[29291]: pgmap v860: 178 pgs: 178 active+clean; 1.4 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.32k op/s 2024-09-10T18:40:30.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:30 smithi028 ceph-mon[39220]: pgmap v860: 178 pgs: 178 active+clean; 1.4 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.32k op/s 2024-09-10T18:40:30.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:30.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:30.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:30.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:30.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:30 smithi204 ceph-mon[32357]: pgmap v860: 178 pgs: 178 active+clean; 1.4 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.32k op/s 2024-09-10T18:40:30.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:30.674 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:30.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:30.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:30.776 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:30.806 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:30.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:30.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:30.908 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:30.940 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:30.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:30.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:31.041 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:31.072 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:31.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:31.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:31.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:31.221 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:31.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:31.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:31.262 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:40:31.262 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:40:31.263 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:40:31.263 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:40:31.298 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:40:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:40:31.316 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 14 15 34443 34428 153.678 101.188 0.0113855 0.00650196 2024-09-10T18:40:31.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:31.336 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:40:31.336 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:40:31.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:31.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:31.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:31.375 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:40:31.376 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:40:31.415 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:40:31.415 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:40:31.415 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:40:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:40:31] "GET /metrics HTTP/1.1" 200 224800 "" "Prometheus/2.43.0" 2024-09-10T18:40:31.451 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:40:31.451 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:40:31.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:31.487 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:31.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:31.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:31.515 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:40:31.515 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:40:31.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:31.626 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:31.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:31.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:31.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:31.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:31.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:31.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:31.897 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:31.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:31.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:31.919 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:40:32.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:32.037 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:32.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:32.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:31 smithi028 ceph-mon[39220]: pgmap v861: 178 pgs: 178 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 142 MiB/s wr, 2.28k op/s 2024-09-10T18:40:32.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:31 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/933785307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:40:32.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:31 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/828112593' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:40:32.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:31 smithi028 ceph-mon[39220]: from='client.35882 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:40:32.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:31 smithi028 ceph-mon[29291]: pgmap v861: 178 pgs: 178 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 142 MiB/s wr, 2.28k op/s 2024-09-10T18:40:32.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:31 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/933785307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:40:32.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:31 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/828112593' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:40:32.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:31 smithi028 ceph-mon[29291]: from='client.35882 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:40:32.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:32.165 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:32.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:32.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:32.300 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:32.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:32.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:32.316 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 15 16 35958 35942 149.741 94.625 0.00694784 0.00667252 2024-09-10T18:40:32.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:31 smithi204 ceph-mon[32357]: pgmap v861: 178 pgs: 178 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 142 MiB/s wr, 2.28k op/s 2024-09-10T18:40:32.361 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:31 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/933785307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:40:32.362 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:31 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/828112593' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:40:32.362 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:31 smithi204 ceph-mon[32357]: from='client.35882 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:40:32.362 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:40:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:40:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:40:32.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:32.429 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:32.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:32.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:32.531 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:32.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:32.568 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:32.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:32.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:32.710 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:32.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:32.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:32.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:32.845 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:32.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:32.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:32.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:32.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:32.986 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:32.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:33.087 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:33.121 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:33.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:33.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:33.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:32 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:40:33.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:32 smithi028 ceph-mon[39220]: from='client.35882 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:40:33.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:32 smithi028 ceph-mon[39220]: osdmap e383: 8 total, 8 up, 7 in 2024-09-10T18:40:33.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:32 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:40:33.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:32 smithi028 ceph-mon[29291]: from='client.35882 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:40:33.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:32 smithi028 ceph-mon[29291]: osdmap e383: 8 total, 8 up, 7 in 2024-09-10T18:40:33.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:33.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:33.262 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:33.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:33.316 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 16 16 37887 37871 147.916 120.562 0.00671989 0.00675489 2024-09-10T18:40:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:32 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:40:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:32 smithi204 ceph-mon[32357]: from='client.35882 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:40:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:32 smithi204 ceph-mon[32357]: osdmap e383: 8 total, 8 up, 7 in 2024-09-10T18:40:33.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:33.365 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:40:33.365 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:40:33.366 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-10T18:40:33.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-10T18:40:33.389 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:33.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:33.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:33.491 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:33.522 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:33.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:33.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:33.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:33.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:33.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:33.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:33.818 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:33.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:33.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:33.834 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:33.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:33.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:33.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:33.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:33.996 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:33 smithi204 ceph-mon[32357]: 1.12 scrub starts 2024-09-10T18:40:33.996 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:33 smithi204 ceph-mon[32357]: 1.12 scrub ok 2024-09-10T18:40:33.997 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:33 smithi204 ceph-mon[32357]: 3.d scrub starts 2024-09-10T18:40:33.997 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:33 smithi204 ceph-mon[32357]: 3.d scrub ok 2024-09-10T18:40:33.997 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:33 smithi204 ceph-mon[32357]: pgmap v863: 178 pgs: 178 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 124 MiB/s wr, 1.98k op/s 2024-09-10T18:40:34.062 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:34.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:34.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:34.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:34.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:33 smithi028 ceph-mon[29291]: 1.12 scrub starts 2024-09-10T18:40:34.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:33 smithi028 ceph-mon[29291]: 1.12 scrub ok 2024-09-10T18:40:34.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:33 smithi028 ceph-mon[29291]: 3.d scrub starts 2024-09-10T18:40:34.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:33 smithi028 ceph-mon[29291]: 3.d scrub ok 2024-09-10T18:40:34.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:33 smithi028 ceph-mon[29291]: pgmap v863: 178 pgs: 178 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 124 MiB/s wr, 1.98k op/s 2024-09-10T18:40:34.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:33 smithi028 ceph-mon[39220]: 1.12 scrub starts 2024-09-10T18:40:34.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:33 smithi028 ceph-mon[39220]: 1.12 scrub ok 2024-09-10T18:40:34.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:33 smithi028 ceph-mon[39220]: 3.d scrub starts 2024-09-10T18:40:34.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:33 smithi028 ceph-mon[39220]: 3.d scrub ok 2024-09-10T18:40:34.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:33 smithi028 ceph-mon[39220]: pgmap v863: 178 pgs: 178 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 124 MiB/s wr, 1.98k op/s 2024-09-10T18:40:34.200 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:34.234 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:34.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:34.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:34.316 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 17 16 39691 39675 145.848 112.75 0.0145446 0.00685079 2024-09-10T18:40:34.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:34.373 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:34.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:34.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:34.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:40:34.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:34.534 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:34.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:34.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:34.567 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:40:34.567 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:40:34.635 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:34.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:34.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:34.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:34.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:34.819 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:34.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:34.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:34.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:34.971 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:34.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:34.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:35.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:35.108 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:35.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:35.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:35.210 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:35.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:40:35.245 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:35.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:35.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:35.248 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:34 smithi204 ceph-mon[32357]: 1.8 scrub starts 2024-09-10T18:40:35.248 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:34 smithi204 ceph-mon[32357]: 1.8 scrub ok 2024-09-10T18:40:35.248 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:34 smithi204 ceph-mon[32357]: 6.6 scrub starts 2024-09-10T18:40:35.248 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:34 smithi204 ceph-mon[32357]: 6.6 scrub ok 2024-09-10T18:40:35.248 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:40:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:40:35.250 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:34 smithi028 ceph-mon[39220]: 1.8 scrub starts 2024-09-10T18:40:35.250 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:34 smithi028 ceph-mon[39220]: 1.8 scrub ok 2024-09-10T18:40:35.250 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:34 smithi028 ceph-mon[39220]: 6.6 scrub starts 2024-09-10T18:40:35.250 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:34 smithi028 ceph-mon[39220]: 6.6 scrub ok 2024-09-10T18:40:35.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:34 smithi028 ceph-mon[29291]: 1.8 scrub starts 2024-09-10T18:40:35.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:34 smithi028 ceph-mon[29291]: 1.8 scrub ok 2024-09-10T18:40:35.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:34 smithi028 ceph-mon[29291]: 6.6 scrub starts 2024-09-10T18:40:35.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:34 smithi028 ceph-mon[29291]: 6.6 scrub ok 2024-09-10T18:40:35.286 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-10T18:40:35.316 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 18 16 41670 41654 144.616 123.688 0.0148903 0.00690894 2024-09-10T18:40:35.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:35.402 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:35.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:35.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:35.504 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:35.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:35.545 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:35.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:35.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:35.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:35.685 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:35.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:35.756 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:40:35.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:35.825 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:35.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:35.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:35.968 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:35.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:35.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:36.070 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:36.118 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:36.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:36.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:36.211 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:35 smithi204 ceph-mon[32357]: 3.1b scrub starts 2024-09-10T18:40:36.211 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:35 smithi204 ceph-mon[32357]: 3.1b scrub ok 2024-09-10T18:40:36.211 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:35 smithi204 ceph-mon[32357]: 1.f scrub starts 2024-09-10T18:40:36.211 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:35 smithi204 ceph-mon[32357]: 1.f scrub ok 2024-09-10T18:40:36.212 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:35 smithi204 ceph-mon[32357]: 5.4 scrub starts 2024-09-10T18:40:36.212 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:35 smithi204 ceph-mon[32357]: 5.4 scrub ok 2024-09-10T18:40:36.212 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:35 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:36.212 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:35 smithi204 ceph-mon[32357]: pgmap v864: 178 pgs: 178 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 123 MiB/s wr, 1.96k op/s 2024-09-10T18:40:36.212 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:35 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:40:36.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:36.250 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:36.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:36.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:36.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[39220]: 3.1b scrub starts 2024-09-10T18:40:36.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[39220]: 3.1b scrub ok 2024-09-10T18:40:36.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[39220]: 1.f scrub starts 2024-09-10T18:40:36.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[39220]: 1.f scrub ok 2024-09-10T18:40:36.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[39220]: 5.4 scrub starts 2024-09-10T18:40:36.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[39220]: 5.4 scrub ok 2024-09-10T18:40:36.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:36.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[39220]: pgmap v864: 178 pgs: 178 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 123 MiB/s wr, 1.96k op/s 2024-09-10T18:40:36.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:40:36.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[29291]: 3.1b scrub starts 2024-09-10T18:40:36.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[29291]: 3.1b scrub ok 2024-09-10T18:40:36.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[29291]: 1.f scrub starts 2024-09-10T18:40:36.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[29291]: 1.f scrub ok 2024-09-10T18:40:36.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[29291]: 5.4 scrub starts 2024-09-10T18:40:36.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[29291]: 5.4 scrub ok 2024-09-10T18:40:36.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:36.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[29291]: pgmap v864: 178 pgs: 178 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 123 MiB/s wr, 1.96k op/s 2024-09-10T18:40:36.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:35 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:40:36.316 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 19 16 43341 43325 142.501 104.438 0.0106299 0.00701252 2024-09-10T18:40:36.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:36.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:36.383 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:36.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:36.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:36.534 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:36.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:36.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:36.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:36.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:36.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:36.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:36.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:36.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:36.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:36.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:36.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:36.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:36.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:37.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:36 smithi028 ceph-mon[29291]: 6.1f scrub starts 2024-09-10T18:40:37.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:36 smithi028 ceph-mon[29291]: 6.1f scrub ok 2024-09-10T18:40:37.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:36 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:40:37.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:36 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:40:37.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:36 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:40:37.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:36 smithi028 ceph-mon[39220]: 6.1f scrub starts 2024-09-10T18:40:37.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:36 smithi028 ceph-mon[39220]: 6.1f scrub ok 2024-09-10T18:40:37.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:36 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:40:37.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:36 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:40:37.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:36 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:40:37.060 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:37.081 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:37.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:37.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:37.183 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:37.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:37.218 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:37.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:37.317 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:40:37.316225+0000 min lat: 0.00125893 max lat: 0.265717 avg lat: 0.00712274 2024-09-10T18:40:37.317 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:40:37.317 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 20 16 44917 44901 140.3 98.5 0.00884424 0.00712274 2024-09-10T18:40:37.318 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:36 smithi204 ceph-mon[32357]: 6.1f scrub starts 2024-09-10T18:40:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:36 smithi204 ceph-mon[32357]: 6.1f scrub ok 2024-09-10T18:40:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:36 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:40:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:36 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:40:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:36 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:40:37.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:37.355 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:37.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:37.456 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:37.490 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:37.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:37.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:37.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:37.628 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:37.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:37.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:37.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:37.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:37.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:37.903 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:37.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:37.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:37.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-10T18:40:38.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:38.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:38.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:38.139 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:37 smithi204 ceph-mon[32357]: 5.a scrub starts 2024-09-10T18:40:38.139 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:37 smithi204 ceph-mon[32357]: 5.a scrub ok 2024-09-10T18:40:38.139 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:37 smithi204 ceph-mon[32357]: pgmap v865: 178 pgs: 178 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-09-10T18:40:38.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:38.188 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:40:38.188 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:40:38.200 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:38.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:38.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:38.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:37 smithi028 ceph-mon[29291]: 5.a scrub starts 2024-09-10T18:40:38.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:37 smithi028 ceph-mon[29291]: 5.a scrub ok 2024-09-10T18:40:38.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:37 smithi028 ceph-mon[29291]: pgmap v865: 178 pgs: 178 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-09-10T18:40:38.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:37 smithi028 ceph-mon[39220]: 5.a scrub starts 2024-09-10T18:40:38.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:37 smithi028 ceph-mon[39220]: 5.a scrub ok 2024-09-10T18:40:38.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:37 smithi028 ceph-mon[39220]: pgmap v865: 178 pgs: 178 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-09-10T18:40:38.304 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:38.316 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 21 16 46712 46696 138.961 112.188 0.00691575 0.00719119 2024-09-10T18:40:38.336 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:38.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:38.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:38.355 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:38.438 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:38.470 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:38.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:38.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:38.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:38.609 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:38.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:38.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:38.711 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:38.745 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:38.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:38.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:38.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:38.898 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:38.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:38.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:38.949 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-10T18:40:39.000 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:39.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:39.037 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:39.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:39.082 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[29291]: 3.3 scrub starts 2024-09-10T18:40:39.082 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[29291]: 3.3 scrub ok 2024-09-10T18:40:39.082 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[29291]: 1.18 scrub starts 2024-09-10T18:40:39.082 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[29291]: 1.18 scrub ok 2024-09-10T18:40:39.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[29291]: 5.11 scrub starts 2024-09-10T18:40:39.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[29291]: 5.11 scrub ok 2024-09-10T18:40:39.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[29291]: 1.0 scrub starts 2024-09-10T18:40:39.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[29291]: 1.0 scrub ok 2024-09-10T18:40:39.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[29291]: 5.14 scrub starts 2024-09-10T18:40:39.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[29291]: 5.14 scrub ok 2024-09-10T18:40:39.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:40:39.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[29291]: 1.a scrub starts 2024-09-10T18:40:39.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[29291]: 1.a scrub ok 2024-09-10T18:40:39.083 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[39220]: 3.3 scrub starts 2024-09-10T18:40:39.083 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[39220]: 3.3 scrub ok 2024-09-10T18:40:39.083 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[39220]: 1.18 scrub starts 2024-09-10T18:40:39.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[39220]: 1.18 scrub ok 2024-09-10T18:40:39.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[39220]: 5.11 scrub starts 2024-09-10T18:40:39.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[39220]: 5.11 scrub ok 2024-09-10T18:40:39.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[39220]: 1.0 scrub starts 2024-09-10T18:40:39.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[39220]: 1.0 scrub ok 2024-09-10T18:40:39.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[39220]: 5.14 scrub starts 2024-09-10T18:40:39.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[39220]: 5.14 scrub ok 2024-09-10T18:40:39.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:40:39.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[39220]: 1.a scrub starts 2024-09-10T18:40:39.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:38 smithi028 ceph-mon[39220]: 1.a scrub ok 2024-09-10T18:40:39.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:39.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:39.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:39.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:39.317 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 22 16 48462 48446 137.615 109.375 0.00714244 0.00726079 2024-09-10T18:40:39.329 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:39.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:39.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:39.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:38 smithi204 ceph-mon[32357]: 3.3 scrub starts 2024-09-10T18:40:39.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:38 smithi204 ceph-mon[32357]: 3.3 scrub ok 2024-09-10T18:40:39.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:38 smithi204 ceph-mon[32357]: 1.18 scrub starts 2024-09-10T18:40:39.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:38 smithi204 ceph-mon[32357]: 1.18 scrub ok 2024-09-10T18:40:39.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:38 smithi204 ceph-mon[32357]: 5.11 scrub starts 2024-09-10T18:40:39.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:38 smithi204 ceph-mon[32357]: 5.11 scrub ok 2024-09-10T18:40:39.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:38 smithi204 ceph-mon[32357]: 1.0 scrub starts 2024-09-10T18:40:39.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:38 smithi204 ceph-mon[32357]: 1.0 scrub ok 2024-09-10T18:40:39.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:38 smithi204 ceph-mon[32357]: 5.14 scrub starts 2024-09-10T18:40:39.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:38 smithi204 ceph-mon[32357]: 5.14 scrub ok 2024-09-10T18:40:39.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:38 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:40:39.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:38 smithi204 ceph-mon[32357]: 1.a scrub starts 2024-09-10T18:40:39.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:38 smithi204 ceph-mon[32357]: 1.a scrub ok 2024-09-10T18:40:39.376 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:40:39.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:39.489 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:39.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:39.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:39.591 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:39.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:39.627 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:39.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:39.729 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:39.766 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:39.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:39.867 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:39.907 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:39.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:39.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:40.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:40.053 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:40.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:40.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:40.055 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:40:40.056 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:40:40.056 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:40:40.056 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:40:40.056 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:40:40.056 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:40:40.056 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:40:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:40:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:40:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:40:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:40:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:40:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:40:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:40:40.057 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:40:40.058 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:40:40.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:40.193 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:40.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[39220]: 5.1f scrub starts 2024-09-10T18:40:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[39220]: 5.1f scrub ok 2024-09-10T18:40:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[39220]: 3.12 scrub starts 2024-09-10T18:40:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[39220]: 3.12 scrub ok 2024-09-10T18:40:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[39220]: pgmap v866: 178 pgs: 178 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-09-10T18:40:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[39220]: 6.4 scrub starts 2024-09-10T18:40:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[39220]: 6.4 scrub ok 2024-09-10T18:40:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[39220]: 6.1b scrub starts 2024-09-10T18:40:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[39220]: 6.1b scrub ok 2024-09-10T18:40:40.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1810329792' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:40:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[29291]: 5.1f scrub starts 2024-09-10T18:40:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[29291]: 5.1f scrub ok 2024-09-10T18:40:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[29291]: 3.12 scrub starts 2024-09-10T18:40:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[29291]: 3.12 scrub ok 2024-09-10T18:40:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[29291]: pgmap v866: 178 pgs: 178 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-09-10T18:40:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[29291]: 6.4 scrub starts 2024-09-10T18:40:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[29291]: 6.4 scrub ok 2024-09-10T18:40:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[29291]: 6.1b scrub starts 2024-09-10T18:40:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[29291]: 6.1b scrub ok 2024-09-10T18:40:40.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:39 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1810329792' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:40:40.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:40.317 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 23 16 50078 50062 136.023 101 0.00483553 0.00734671 2024-09-10T18:40:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:40.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:40.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:39 smithi204 ceph-mon[32357]: 5.1f scrub starts 2024-09-10T18:40:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:39 smithi204 ceph-mon[32357]: 5.1f scrub ok 2024-09-10T18:40:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:39 smithi204 ceph-mon[32357]: 3.12 scrub starts 2024-09-10T18:40:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:39 smithi204 ceph-mon[32357]: 3.12 scrub ok 2024-09-10T18:40:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:39 smithi204 ceph-mon[32357]: pgmap v866: 178 pgs: 178 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-09-10T18:40:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:39 smithi204 ceph-mon[32357]: 6.4 scrub starts 2024-09-10T18:40:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:39 smithi204 ceph-mon[32357]: 6.4 scrub ok 2024-09-10T18:40:40.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:39 smithi204 ceph-mon[32357]: 6.1b scrub starts 2024-09-10T18:40:40.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:39 smithi204 ceph-mon[32357]: 6.1b scrub ok 2024-09-10T18:40:40.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:39 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1810329792' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:40:40.434 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:40.472 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:40.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:40.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:40.573 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:40.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:40.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:40.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:40.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:40.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:40.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:40.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:40.894 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:40.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:40.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:40.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:41.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:41.042 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:41.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:41.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:40 smithi204 ceph-mon[32357]: 2.1 scrub starts 2024-09-10T18:40:41.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:40 smithi204 ceph-mon[32357]: 2.1 scrub ok 2024-09-10T18:40:41.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:40 smithi204 ceph-mon[32357]: 6.14 scrub starts 2024-09-10T18:40:41.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:40 smithi204 ceph-mon[32357]: 6.14 scrub ok 2024-09-10T18:40:41.144 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:41.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:41.179 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:41.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:41.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:40 smithi028 ceph-mon[29291]: 2.1 scrub starts 2024-09-10T18:40:41.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:40 smithi028 ceph-mon[29291]: 2.1 scrub ok 2024-09-10T18:40:41.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:40 smithi028 ceph-mon[29291]: 6.14 scrub starts 2024-09-10T18:40:41.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:40 smithi028 ceph-mon[29291]: 6.14 scrub ok 2024-09-10T18:40:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:40 smithi028 ceph-mon[39220]: 2.1 scrub starts 2024-09-10T18:40:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:40 smithi028 ceph-mon[39220]: 2.1 scrub ok 2024-09-10T18:40:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:40 smithi028 ceph-mon[39220]: 6.14 scrub starts 2024-09-10T18:40:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:40 smithi028 ceph-mon[39220]: 6.14 scrub ok 2024-09-10T18:40:41.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:41.317 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 24 16 51726 51710 134.646 103 0.00755305 0.00742167 2024-09-10T18:40:41.321 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:41.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:41.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:41.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:41.437 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:40:41.437 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:40:41.437 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-10T18:40:41.437 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:40:41.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:41.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:41.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:41.547 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:41.581 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:41.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:41.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:41.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:41.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:41.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:41.753 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:40:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:40:41] "GET /metrics HTTP/1.1" 200 224883 "" "Prometheus/2.43.0" 2024-09-10T18:40:41.823 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:41.855 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:41.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:41.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:41.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:41.996 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:41.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:41.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:42.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:42.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-10T18:40:42.121 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:42.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:42.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:42.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:42.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:41 smithi028 ceph-mon[29291]: 5.d scrub starts 2024-09-10T18:40:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:41 smithi028 ceph-mon[29291]: 5.d scrub ok 2024-09-10T18:40:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:41 smithi028 ceph-mon[29291]: pgmap v867: 178 pgs: 178 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 109 MiB/s wr, 1.73k op/s 2024-09-10T18:40:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:41 smithi028 ceph-mon[29291]: 4.11 scrub starts 2024-09-10T18:40:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:41 smithi028 ceph-mon[29291]: 4.11 scrub ok 2024-09-10T18:40:42.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:41 smithi028 ceph-mon[39220]: 5.d scrub starts 2024-09-10T18:40:42.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:41 smithi028 ceph-mon[39220]: 5.d scrub ok 2024-09-10T18:40:42.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:41 smithi028 ceph-mon[39220]: pgmap v867: 178 pgs: 178 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 109 MiB/s wr, 1.73k op/s 2024-09-10T18:40:42.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:41 smithi028 ceph-mon[39220]: 4.11 scrub starts 2024-09-10T18:40:42.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:41 smithi028 ceph-mon[39220]: 4.11 scrub ok 2024-09-10T18:40:42.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:42.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:42.317 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 25 16 53110 53094 132.72 86.5 0.00747371 0.00752927 2024-09-10T18:40:42.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:41 smithi204 ceph-mon[32357]: 5.d scrub starts 2024-09-10T18:40:42.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:41 smithi204 ceph-mon[32357]: 5.d scrub ok 2024-09-10T18:40:42.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:41 smithi204 ceph-mon[32357]: pgmap v867: 178 pgs: 178 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 109 MiB/s wr, 1.73k op/s 2024-09-10T18:40:42.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:41 smithi204 ceph-mon[32357]: 4.11 scrub starts 2024-09-10T18:40:42.356 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:41 smithi204 ceph-mon[32357]: 4.11 scrub ok 2024-09-10T18:40:42.357 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:40:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:40:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:40:42.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:42.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:42.395 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:42.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:42.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:42.508 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:40:42.540 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:42.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:42.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:42.642 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:42.663 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:42.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:42.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:42.765 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:42.798 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:42.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:42.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:42.900 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:42.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:42.932 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:42.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:43.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:43.065 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:43.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:43.167 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:43.188 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:43.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:43.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:43.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:42 smithi028 ceph-mon[29291]: 3.a scrub starts 2024-09-10T18:40:43.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:42 smithi028 ceph-mon[29291]: 3.a scrub ok 2024-09-10T18:40:43.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:42 smithi028 ceph-mon[39220]: 3.a scrub starts 2024-09-10T18:40:43.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:42 smithi028 ceph-mon[39220]: 3.a scrub ok 2024-09-10T18:40:43.290 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:43.317 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 26 15 54460 54445 130.863 84.4375 0.00794524 0.00763634 2024-09-10T18:40:43.324 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:43.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:43.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:42 smithi204 ceph-mon[32357]: 3.a scrub starts 2024-09-10T18:40:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:42 smithi204 ceph-mon[32357]: 3.a scrub ok 2024-09-10T18:40:43.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:43.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:43.452 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:43.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:43.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:43.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:43.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:43.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:43.735 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:43.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:43.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:43.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:43.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:43.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:43.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:43.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:44.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:44.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:43 smithi204 ceph-mon[32357]: 4.1a scrub starts 2024-09-10T18:40:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:43 smithi204 ceph-mon[32357]: 4.1a scrub ok 2024-09-10T18:40:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:43 smithi204 ceph-mon[32357]: 5.8 scrub starts 2024-09-10T18:40:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:43 smithi204 ceph-mon[32357]: 5.8 scrub ok 2024-09-10T18:40:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:43 smithi204 ceph-mon[32357]: 5.1a scrub starts 2024-09-10T18:40:44.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:43 smithi204 ceph-mon[32357]: 5.1a scrub ok 2024-09-10T18:40:44.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:43 smithi204 ceph-mon[32357]: pgmap v868: 178 pgs: 178 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-09-10T18:40:44.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:43 smithi204 ceph-mon[32357]: 1.11 scrub starts 2024-09-10T18:40:44.105 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:43 smithi204 ceph-mon[32357]: 1.11 scrub ok 2024-09-10T18:40:44.111 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:44.155 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:44.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:44.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[29291]: 4.1a scrub starts 2024-09-10T18:40:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[29291]: 4.1a scrub ok 2024-09-10T18:40:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[29291]: 5.8 scrub starts 2024-09-10T18:40:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[29291]: 5.8 scrub ok 2024-09-10T18:40:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[29291]: 5.1a scrub starts 2024-09-10T18:40:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[29291]: 5.1a scrub ok 2024-09-10T18:40:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[29291]: pgmap v868: 178 pgs: 178 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-09-10T18:40:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[29291]: 1.11 scrub starts 2024-09-10T18:40:44.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[29291]: 1.11 scrub ok 2024-09-10T18:40:44.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[39220]: 4.1a scrub starts 2024-09-10T18:40:44.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[39220]: 4.1a scrub ok 2024-09-10T18:40:44.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[39220]: 5.8 scrub starts 2024-09-10T18:40:44.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[39220]: 5.8 scrub ok 2024-09-10T18:40:44.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[39220]: 5.1a scrub starts 2024-09-10T18:40:44.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[39220]: 5.1a scrub ok 2024-09-10T18:40:44.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[39220]: pgmap v868: 178 pgs: 178 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-09-10T18:40:44.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[39220]: 1.11 scrub starts 2024-09-10T18:40:44.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:43 smithi028 ceph-mon[39220]: 1.11 scrub ok 2024-09-10T18:40:44.257 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:44.291 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:44.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:44.317 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 27 16 55932 55916 129.421 91.9375 0.0204495 0.00772089 2024-09-10T18:40:44.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:44.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:44.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:44.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:44.463 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:40:44.463 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:40:44.463 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-10T18:40:44.463 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-10T18:40:44.463 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:40:44.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:44.572 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:44.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:44.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:44.674 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:44.715 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:44.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:44.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:44.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:44.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:44.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:44.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:44.991 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:44.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:44.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:45.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:45.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:45.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:45.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:44 smithi028 ceph-mon[29291]: 6.f scrub starts 2024-09-10T18:40:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:44 smithi028 ceph-mon[29291]: 6.f scrub ok 2024-09-10T18:40:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:44 smithi028 ceph-mon[29291]: 4.c scrub starts 2024-09-10T18:40:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:44 smithi028 ceph-mon[29291]: 4.c scrub ok 2024-09-10T18:40:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:44 smithi028 ceph-mon[29291]: 6.1 scrub starts 2024-09-10T18:40:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:44 smithi028 ceph-mon[29291]: 6.1 scrub ok 2024-09-10T18:40:45.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:44 smithi028 ceph-mon[39220]: 6.f scrub starts 2024-09-10T18:40:45.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:44 smithi028 ceph-mon[39220]: 6.f scrub ok 2024-09-10T18:40:45.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:44 smithi028 ceph-mon[39220]: 4.c scrub starts 2024-09-10T18:40:45.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:44 smithi028 ceph-mon[39220]: 4.c scrub ok 2024-09-10T18:40:45.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:44 smithi028 ceph-mon[39220]: 6.1 scrub starts 2024-09-10T18:40:45.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:44 smithi028 ceph-mon[39220]: 6.1 scrub ok 2024-09-10T18:40:45.267 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:45.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:45.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:45.317 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 28 16 57472 57456 128.236 96.25 0.00675543 0.00779297 2024-09-10T18:40:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:44 smithi204 ceph-mon[32357]: 6.f scrub starts 2024-09-10T18:40:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:44 smithi204 ceph-mon[32357]: 6.f scrub ok 2024-09-10T18:40:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:44 smithi204 ceph-mon[32357]: 4.c scrub starts 2024-09-10T18:40:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:44 smithi204 ceph-mon[32357]: 4.c scrub ok 2024-09-10T18:40:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:44 smithi204 ceph-mon[32357]: 6.1 scrub starts 2024-09-10T18:40:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:44 smithi204 ceph-mon[32357]: 6.1 scrub ok 2024-09-10T18:40:45.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:40:45 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:40:45.369 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:45.404 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:45.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:45.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:45.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:45.540 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:45.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:45.642 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:45.673 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:45.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:45.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:45.775 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:45.806 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:45.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:45.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:45.890 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-10T18:40:45.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:45.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:45.937 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:45.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:46.038 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:46.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:46.078 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:46.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:46.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:46.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:46.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:45 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:45 smithi028 ceph-mon[39220]: pgmap v869: 178 pgs: 178 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-10T18:40:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:45 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:45 smithi028 ceph-mon[29291]: pgmap v869: 178 pgs: 178 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-10T18:40:46.299 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:40:46.317 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 29 16 58799 58783 126.674 82.9375 0.0042968 0.0078765 2024-09-10T18:40:46.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:46.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:46.346 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:46.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:45 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:45 smithi204 ceph-mon[32357]: pgmap v869: 178 pgs: 178 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-10T18:40:46.448 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:46.469 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:46.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:46.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:46.571 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:46.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:46.614 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:46.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:46.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:46.757 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:46.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:46.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:46.859 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:46.894 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:46.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:46.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:46.996 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:47.031 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:47.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:47.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:46 smithi204 ceph-mon[32357]: 6.2 scrub starts 2024-09-10T18:40:47.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:46 smithi204 ceph-mon[32357]: 6.2 scrub ok 2024-09-10T18:40:47.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:46 smithi204 ceph-mon[32357]: 5.10 scrub starts 2024-09-10T18:40:47.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:46 smithi204 ceph-mon[32357]: 5.10 scrub ok 2024-09-10T18:40:47.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:46 smithi204 ceph-mon[32357]: 1.16 scrub starts 2024-09-10T18:40:47.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:46 smithi204 ceph-mon[32357]: 1.16 scrub ok 2024-09-10T18:40:47.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:46 smithi204 ceph-mon[32357]: 5.17 scrub starts 2024-09-10T18:40:47.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:46 smithi204 ceph-mon[32357]: 5.17 scrub ok 2024-09-10T18:40:47.133 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:47.169 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:47.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:47.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:47.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:46 smithi028 ceph-mon[39220]: 6.2 scrub starts 2024-09-10T18:40:47.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:46 smithi028 ceph-mon[39220]: 6.2 scrub ok 2024-09-10T18:40:47.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:46 smithi028 ceph-mon[39220]: 5.10 scrub starts 2024-09-10T18:40:47.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:46 smithi028 ceph-mon[39220]: 5.10 scrub ok 2024-09-10T18:40:47.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:46 smithi028 ceph-mon[39220]: 1.16 scrub starts 2024-09-10T18:40:47.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:46 smithi028 ceph-mon[39220]: 1.16 scrub ok 2024-09-10T18:40:47.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:46 smithi028 ceph-mon[39220]: 5.17 scrub starts 2024-09-10T18:40:47.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:46 smithi028 ceph-mon[39220]: 5.17 scrub ok 2024-09-10T18:40:47.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:46 smithi028 ceph-mon[29291]: 6.2 scrub starts 2024-09-10T18:40:47.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:46 smithi028 ceph-mon[29291]: 6.2 scrub ok 2024-09-10T18:40:47.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:46 smithi028 ceph-mon[29291]: 5.10 scrub starts 2024-09-10T18:40:47.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:46 smithi028 ceph-mon[29291]: 5.10 scrub ok 2024-09-10T18:40:47.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:46 smithi028 ceph-mon[29291]: 1.16 scrub starts 2024-09-10T18:40:47.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:46 smithi028 ceph-mon[29291]: 1.16 scrub ok 2024-09-10T18:40:47.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:46 smithi028 ceph-mon[29291]: 5.17 scrub starts 2024-09-10T18:40:47.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:46 smithi028 ceph-mon[29291]: 5.17 scrub ok 2024-09-10T18:40:47.271 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:47.302 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:47.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:47.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:47.317 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 30 16 60291 60275 125.56 93.25 0.00637787 0.00795913 2024-09-10T18:40:47.357 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:47.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:47.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:47.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:47.547 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:47.591 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:47.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:47.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:47.693 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:47.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:47.731 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:47.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:47.832 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:47.870 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:47.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:47.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:47.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:48.010 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:48.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:48.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:48.112 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:48.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[39220]: 4.a scrub starts 2024-09-10T18:40:48.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[39220]: 4.a scrub ok 2024-09-10T18:40:48.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[39220]: 6.1c scrub starts 2024-09-10T18:40:48.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[39220]: 6.1c scrub ok 2024-09-10T18:40:48.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[39220]: 5.5 scrub starts 2024-09-10T18:40:48.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[39220]: 5.5 scrub ok 2024-09-10T18:40:48.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[39220]: pgmap v870: 178 pgs: 178 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-10T18:40:48.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[39220]: 5.13 scrub starts 2024-09-10T18:40:48.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[39220]: 5.13 scrub ok 2024-09-10T18:40:48.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[29291]: 4.a scrub starts 2024-09-10T18:40:48.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[29291]: 4.a scrub ok 2024-09-10T18:40:48.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[29291]: 6.1c scrub starts 2024-09-10T18:40:48.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[29291]: 6.1c scrub ok 2024-09-10T18:40:48.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[29291]: 5.5 scrub starts 2024-09-10T18:40:48.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[29291]: 5.5 scrub ok 2024-09-10T18:40:48.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[29291]: pgmap v870: 178 pgs: 178 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-10T18:40:48.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[29291]: 5.13 scrub starts 2024-09-10T18:40:48.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:48 smithi028 ceph-mon[29291]: 5.13 scrub ok 2024-09-10T18:40:48.156 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:48.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:48.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:48.279 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:48 smithi204 ceph-mon[32357]: 4.a scrub starts 2024-09-10T18:40:48.280 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:48 smithi204 ceph-mon[32357]: 4.a scrub ok 2024-09-10T18:40:48.280 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:48 smithi204 ceph-mon[32357]: 6.1c scrub starts 2024-09-10T18:40:48.280 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:48 smithi204 ceph-mon[32357]: 6.1c scrub ok 2024-09-10T18:40:48.280 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:48 smithi204 ceph-mon[32357]: 5.5 scrub starts 2024-09-10T18:40:48.280 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:48 smithi204 ceph-mon[32357]: 5.5 scrub ok 2024-09-10T18:40:48.280 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:48 smithi204 ceph-mon[32357]: pgmap v870: 178 pgs: 178 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-10T18:40:48.280 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:48 smithi204 ceph-mon[32357]: 5.13 scrub starts 2024-09-10T18:40:48.280 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:48 smithi204 ceph-mon[32357]: 5.13 scrub ok 2024-09-10T18:40:48.281 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:40:48.281 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-10T18:40:48.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-10T18:40:48.281 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:40:48.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.281 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:186582 9.6 9:602ee978:::benchmark_data_smithi028_135703_object51991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-10T18:40:48.281 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:25.728385+0000", 2024-09-10T18:40:48.281 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 382.54760186499999, 2024-09-10T18:40:48.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.27129171000000002, 2024-09-10T18:40:48.281 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.282 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:48.282 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.282 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:40:48.282 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:40:48.282 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 186582 2024-09-10T18:40:48.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.282 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.728385+0000", 2024-09-10T18:40:48.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.728385+0000", 2024-09-10T18:40:48.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.728382+0000", 2024-09-10T18:40:48.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:40:48.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.728390+0000", 2024-09-10T18:40:48.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9929999999999995e-06 2024-09-10T18:40:48.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.728404+0000", 2024-09-10T18:40:48.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3895e-05 2024-09-10T18:40:48.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.728410+0000", 2024-09-10T18:40:48.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5230000000000003e-06 2024-09-10T18:40:48.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.752791+0000", 2024-09-10T18:40:48.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024381679 2024-09-10T18:40:48.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.752881+0000", 2024-09-10T18:40:48.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9269000000000002e-05 2024-09-10T18:40:48.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:40:48.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.752968+0000", 2024-09-10T18:40:48.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7183999999999997e-05 2024-09-10T18:40:48.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:40:48.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.754297+0000", 2024-09-10T18:40:48.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013290050000000001 2024-09-10T18:40:48.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.857961+0000", 2024-09-10T18:40:48.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.103663924 2024-09-10T18:40:48.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.999587+0000", 2024-09-10T18:40:48.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.14162615000000001 2024-09-10T18:40:48.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:40:48.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.999643+0000", 2024-09-10T18:40:48.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6060999999999997e-05 2024-09-10T18:40:48.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:25.999676+0000", 2024-09-10T18:40:48.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3176999999999998e-05 2024-09-10T18:40:48.288 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.288 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.288 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.288 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.34334.0:2429 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 94344803721280 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-10T18:40:48.288 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.462155+0000", 2024-09-10T18:40:48.288 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 107.813831863, 2024-09-10T18:40:48.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46503165600000002, 2024-09-10T18:40:48.289 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.289 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:40:48.289 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.289 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.34334", 2024-09-10T18:40:48.289 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/601695668", 2024-09-10T18:40:48.289 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2429 2024-09-10T18:40:48.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.289 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462155+0000", 2024-09-10T18:40:48.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462155+0000", 2024-09-10T18:40:48.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462152+0000", 2024-09-10T18:40:48.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:40:48.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462160+0000", 2024-09-10T18:40:48.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2139999999999996e-06 2024-09-10T18:40:48.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462177+0000", 2024-09-10T18:40:48.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6606e-05 2024-09-10T18:40:48.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462184+0000", 2024-09-10T18:40:48.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0609999999999998e-06 2024-09-10T18:40:48.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.486107+0000", 2024-09-10T18:40:48.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023923172999999999 2024-09-10T18:40:48.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.486112+0000", 2024-09-10T18:40:48.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5040000000000002e-06 2024-09-10T18:40:48.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.913266+0000", 2024-09-10T18:40:48.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42715417900000002 2024-09-10T18:40:48.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for missing object", 2024-09-10T18:40:48.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.913531+0000", 2024-09-10T18:40:48.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00026490999999999998 2024-09-10T18:40:48.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.924754+0000", 2024-09-10T18:40:48.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011222744999999999 2024-09-10T18:40:48.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-10T18:40:48.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.924844+0000", 2024-09-10T18:40:48.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9617000000000006e-05 2024-09-10T18:40:48.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.925702+0000", 2024-09-10T18:40:48.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00085778299999999998 2024-09-10T18:40:48.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.925767+0000", 2024-09-10T18:40:48.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5029000000000004e-05 2024-09-10T18:40:48.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.927186+0000", 2024-09-10T18:40:48.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014196530000000001 2024-09-10T18:40:48.295 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.295 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.295 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.295 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224013 10.9 10:990f827e:::benchmark_data_smithi028_161947_object87089:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:40:48.295 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.893001+0000", 2024-09-10T18:40:48.295 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 107.382985605, 2024-09-10T18:40:48.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.010061828, 2024-09-10T18:40:48.295 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.295 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:40:48.295 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.295 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:40:48.296 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:40:48.296 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224013 2024-09-10T18:40:48.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.296 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.893001+0000", 2024-09-10T18:40:48.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.893001+0000", 2024-09-10T18:40:48.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.892999+0000", 2024-09-10T18:40:48.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:40:48.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.893005+0000", 2024-09-10T18:40:48.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8290000000000002e-06 2024-09-10T18:40:48.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.893015+0000", 2024-09-10T18:40:48.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4380000000000001e-06 2024-09-10T18:40:48.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.893022+0000", 2024-09-10T18:40:48.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4630000000000004e-06 2024-09-10T18:40:48.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.898346+0000", 2024-09-10T18:40:48.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0053237240000000002 2024-09-10T18:40:48.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.898354+0000", 2024-09-10T18:40:48.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0090000000000007e-06 2024-09-10T18:40:48.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.902998+0000", 2024-09-10T18:40:48.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0046443110000001 2024-09-10T18:40:48.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.903063+0000", 2024-09-10T18:40:48.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4349000000000003e-05 2024-09-10T18:40:48.300 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.300 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.300 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.300 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224031 10.6 10:62b78eac:::benchmark_data_smithi028_161947_object87107:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:40:48.300 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.894085+0000", 2024-09-10T18:40:48.300 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 107.381901563, 2024-09-10T18:40:48.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.026344849, 2024-09-10T18:40:48.300 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.300 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:48.300 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.300 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:40:48.301 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:40:48.301 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224031 2024-09-10T18:40:48.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.301 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894085+0000", 2024-09-10T18:40:48.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894085+0000", 2024-09-10T18:40:48.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894084+0000", 2024-09-10T18:40:48.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:40:48.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894090+0000", 2024-09-10T18:40:48.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9719999999999997e-06 2024-09-10T18:40:48.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894100+0000", 2024-09-10T18:40:48.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9990000000000003e-06 2024-09-10T18:40:48.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894105+0000", 2024-09-10T18:40:48.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3539999999999999e-06 2024-09-10T18:40:48.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894214+0000", 2024-09-10T18:40:48.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010946399999999999 2024-09-10T18:40:48.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894224+0000", 2024-09-10T18:40:48.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.132e-06 2024-09-10T18:40:48.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.901993+0000", 2024-09-10T18:40:48.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0077688789999999 2024-09-10T18:40:48.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.902080+0000", 2024-09-10T18:40:48.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7083000000000006e-05 2024-09-10T18:40:48.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:40:48.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.902157+0000", 2024-09-10T18:40:48.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7247000000000001e-05 2024-09-10T18:40:48.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.908263+0000", 2024-09-10T18:40:48.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061065370000000004 2024-09-10T18:40:48.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.908293+0000", 2024-09-10T18:40:48.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9495000000000001e-05 2024-09-10T18:40:48.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:40:48.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920339+0000", 2024-09-10T18:40:48.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012046347000000001 2024-09-10T18:40:48.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:40:48.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920384+0000", 2024-09-10T18:40:48.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5250000000000002e-05 2024-09-10T18:40:48.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920430+0000", 2024-09-10T18:40:48.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5241000000000003e-05 2024-09-10T18:40:48.307 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.307 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.307 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.307 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224033 10.6 10:6b102ba2:::benchmark_data_smithi028_161947_object87109:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:40:48.307 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.894115+0000", 2024-09-10T18:40:48.307 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 107.38187185300001, 2024-09-10T18:40:48.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0264514709999999, 2024-09-10T18:40:48.307 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.307 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:48.307 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.307 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:40:48.308 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:40:48.308 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224033 2024-09-10T18:40:48.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.308 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894115+0000", 2024-09-10T18:40:48.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894115+0000", 2024-09-10T18:40:48.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894114+0000", 2024-09-10T18:40:48.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999995 2024-09-10T18:40:48.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894118+0000", 2024-09-10T18:40:48.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6430000000000001e-06 2024-09-10T18:40:48.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894123+0000", 2024-09-10T18:40:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:48.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5439999999999998e-06 2024-09-10T18:40:48.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894127+0000", 2024-09-10T18:40:48.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9160000000000003e-06 2024-09-10T18:40:48.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894241+0000", 2024-09-10T18:40:48.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114263 2024-09-10T18:40:48.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894246+0000", 2024-09-10T18:40:48.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7809999999999996e-06 2024-09-10T18:40:48.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.904932+0000", 2024-09-10T18:40:48.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0106856639999999 2024-09-10T18:40:48.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.905031+0000", 2024-09-10T18:40:48.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9630999999999995e-05 2024-09-10T18:40:48.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:40:48.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.905133+0000", 2024-09-10T18:40:48.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101679 2024-09-10T18:40:48.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.912359+0000", 2024-09-10T18:40:48.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0072258110000000004 2024-09-10T18:40:48.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.917542+0000", 2024-09-10T18:40:48.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0051832010000000001 2024-09-10T18:40:48.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:40:48.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920503+0000", 2024-09-10T18:40:48.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029605260000000002 2024-09-10T18:40:48.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:40:48.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920536+0000", 2024-09-10T18:40:48.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3519999999999998e-05 2024-09-10T18:40:48.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920566+0000", 2024-09-10T18:40:48.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9971000000000002e-05 2024-09-10T18:40:48.315 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.315 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.315 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.315 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224038 10.6 10:66bb442b:::benchmark_data_smithi028_161947_object87114:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:40:48.315 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.894131+0000", 2024-09-10T18:40:48.315 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 107.381855782, 2024-09-10T18:40:48.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.026563632, 2024-09-10T18:40:48.315 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.315 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:48.315 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.315 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:40:48.315 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:40:48.316 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224038 2024-09-10T18:40:48.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.316 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894131+0000", 2024-09-10T18:40:48.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894131+0000", 2024-09-10T18:40:48.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894130+0000", 2024-09-10T18:40:48.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999995 2024-09-10T18:40:48.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894133+0000", 2024-09-10T18:40:48.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9119999999999998e-06 2024-09-10T18:40:48.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894138+0000", 2024-09-10T18:40:48.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6940000000000003e-06 2024-09-10T18:40:48.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894142+0000", 2024-09-10T18:40:48.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7680000000000002e-06 2024-09-10T18:40:48.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894266+0000", 2024-09-10T18:40:48.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012440899999999999 2024-09-10T18:40:48.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894270+0000", 2024-09-10T18:40:48.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5349999999999998e-06 2024-09-10T18:40:48.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.908313+0000", 2024-09-10T18:40:48.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0140428459999999 2024-09-10T18:40:48.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.908456+0000", 2024-09-10T18:40:48.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014293500000000001 2024-09-10T18:40:48.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:40:48.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.908562+0000", 2024-09-10T18:40:48.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105519 2024-09-10T18:40:48.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920333+0000", 2024-09-10T18:40:48.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011770994999999999 2024-09-10T18:40:48.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:40:48.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920571+0000", 2024-09-10T18:40:48.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023851100000000001 2024-09-10T18:40:48.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920617+0000", 2024-09-10T18:40:48.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6020999999999999e-05 2024-09-10T18:40:48.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:40:48.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920660+0000", 2024-09-10T18:40:48.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2774000000000001e-05 2024-09-10T18:40:48.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920694+0000", 2024-09-10T18:40:48.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4220000000000001e-05 2024-09-10T18:40:48.321 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.321 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.321 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.322 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224045 10.9 10:92123ef6:::benchmark_data_smithi028_161947_object87121:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:40:48.322 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.894145+0000", 2024-09-10T18:40:48.322 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 107.38184133599999, 2024-09-10T18:40:48.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0239920119999999, 2024-09-10T18:40:48.322 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.322 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:48.322 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.322 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:40:48.322 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:40:48.322 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224045 2024-09-10T18:40:48.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.323 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894145+0000", 2024-09-10T18:40:48.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894145+0000", 2024-09-10T18:40:48.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894145+0000", 2024-09-10T18:40:48.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999995 2024-09-10T18:40:48.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894147+0000", 2024-09-10T18:40:48.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3479999999999998e-06 2024-09-10T18:40:48.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894152+0000", 2024-09-10T18:40:48.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8439999999999997e-06 2024-09-10T18:40:48.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894156+0000", 2024-09-10T18:40:48.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.172e-06 2024-09-10T18:40:48.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.898390+0000", 2024-09-10T18:40:48.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042343989999999998 2024-09-10T18:40:48.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.898395+0000", 2024-09-10T18:40:48.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7049999999999998e-06 2024-09-10T18:40:48.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.903090+0000", 2024-09-10T18:40:48.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.004694923 2024-09-10T18:40:48.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.903199+0000", 2024-09-10T18:40:48.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010871599999999999 2024-09-10T18:40:48.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,7", 2024-09-10T18:40:48.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.903284+0000", 2024-09-10T18:40:48.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5470000000000007e-05 2024-09-10T18:40:48.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.917991+0000", 2024-09-10T18:40:48.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014706933 2024-09-10T18:40:48.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918020+0000", 2024-09-10T18:40:48.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8734e-05 2024-09-10T18:40:48.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:40:48.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918026+0000", 2024-09-10T18:40:48.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9109999999999998e-06 2024-09-10T18:40:48.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:40:48.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918101+0000", 2024-09-10T18:40:48.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5488999999999995e-05 2024-09-10T18:40:48.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918137+0000", 2024-09-10T18:40:48.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5769000000000003e-05 2024-09-10T18:40:48.328 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.328 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.328 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.329 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224047 10.9 10:9e565572:::benchmark_data_smithi028_161947_object87123:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:40:48.329 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.894289+0000", 2024-09-10T18:40:48.329 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 107.381697066, 2024-09-10T18:40:48.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0239835960000001, 2024-09-10T18:40:48.329 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.329 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:48.329 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.329 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:40:48.329 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:40:48.329 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224047 2024-09-10T18:40:48.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.329 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894289+0000", 2024-09-10T18:40:48.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894289+0000", 2024-09-10T18:40:48.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894289+0000", 2024-09-10T18:40:48.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:40:48.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894293+0000", 2024-09-10T18:40:48.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0319999999999997e-06 2024-09-10T18:40:48.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894299+0000", 2024-09-10T18:40:48.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4409999999999998e-06 2024-09-10T18:40:48.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894304+0000", 2024-09-10T18:40:48.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4769999999999997e-06 2024-09-10T18:40:48.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.898414+0000", 2024-09-10T18:40:48.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041101319999999998 2024-09-10T18:40:48.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.898418+0000", 2024-09-10T18:40:48.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5929999999999999e-06 2024-09-10T18:40:48.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.905573+0000", 2024-09-10T18:40:48.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0071550709999999 2024-09-10T18:40:48.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.905662+0000", 2024-09-10T18:40:48.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8850999999999993e-05 2024-09-10T18:40:48.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,7", 2024-09-10T18:40:48.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.905755+0000", 2024-09-10T18:40:48.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2317000000000004e-05 2024-09-10T18:40:48.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:40:48.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918143+0000", 2024-09-10T18:40:48.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012388892 2024-09-10T18:40:48.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918183+0000", 2024-09-10T18:40:48.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0017e-05 2024-09-10T18:40:48.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918207+0000", 2024-09-10T18:40:48.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3074e-05 2024-09-10T18:40:48.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:40:48.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918244+0000", 2024-09-10T18:40:48.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7657999999999998e-05 2024-09-10T18:40:48.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918273+0000", 2024-09-10T18:40:48.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8765999999999999e-05 2024-09-10T18:40:48.336 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.336 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.336 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.336 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224050 10.6 10:60145db6:::benchmark_data_smithi028_161947_object87126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:40:48.336 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.949075+0000", 2024-09-10T18:40:48.336 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 107.326911232, 2024-09-10T18:40:48.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97173985299999999, 2024-09-10T18:40:48.336 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.336 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:48.337 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.337 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:40:48.337 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:40:48.337 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224050 2024-09-10T18:40:48.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.337 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.949075+0000", 2024-09-10T18:40:48.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.949075+0000", 2024-09-10T18:40:48.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.949061+0000", 2024-09-10T18:40:48.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999862 2024-09-10T18:40:48.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.949081+0000", 2024-09-10T18:40:48.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9667999999999999e-05 2024-09-10T18:40:48.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.949098+0000", 2024-09-10T18:40:48.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6889999999999999e-05 2024-09-10T18:40:48.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.949104+0000", 2024-09-10T18:40:48.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1500000000000004e-06 2024-09-10T18:40:48.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.949143+0000", 2024-09-10T18:40:48.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8571000000000001e-05 2024-09-10T18:40:48.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.949150+0000", 2024-09-10T18:40:48.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0160000000000003e-06 2024-09-10T18:40:48.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.910472+0000", 2024-09-10T18:40:48.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.96132260899999999 2024-09-10T18:40:48.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.910572+0000", 2024-09-10T18:40:48.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9650999999999998e-05 2024-09-10T18:40:48.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:40:48.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.910678+0000", 2024-09-10T18:40:48.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010611 2024-09-10T18:40:48.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:40:48.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920576+0000", 2024-09-10T18:40:48.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098980720000000008 2024-09-10T18:40:48.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920726+0000", 2024-09-10T18:40:48.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000150147 2024-09-10T18:40:48.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920753+0000", 2024-09-10T18:40:48.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7117000000000001e-05 2024-09-10T18:40:48.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:40:48.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920786+0000", 2024-09-10T18:40:48.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2502999999999997e-05 2024-09-10T18:40:48.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920815+0000", 2024-09-10T18:40:48.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9295999999999999e-05 2024-09-10T18:40:48.343 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.343 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.343 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.343 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224058 10.6 10:6ba08e13:::benchmark_data_smithi028_161947_object87134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:40:48.343 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.962810+0000", 2024-09-10T18:40:48.344 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 107.313176367, 2024-09-10T18:40:48.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95812768699999995, 2024-09-10T18:40:48.344 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.344 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:48.344 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.344 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:40:48.344 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:40:48.344 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224058 2024-09-10T18:40:48.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.344 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.962810+0000", 2024-09-10T18:40:48.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.962810+0000", 2024-09-10T18:40:48.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.962807+0000", 2024-09-10T18:40:48.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:40:48.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.962816+0000", 2024-09-10T18:40:48.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2430000000000005e-06 2024-09-10T18:40:48.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.962838+0000", 2024-09-10T18:40:48.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2524e-05 2024-09-10T18:40:48.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.962845+0000", 2024-09-10T18:40:48.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.187e-06 2024-09-10T18:40:48.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.962899+0000", 2024-09-10T18:40:48.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3854999999999999e-05 2024-09-10T18:40:48.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.962913+0000", 2024-09-10T18:40:48.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3668e-05 2024-09-10T18:40:48.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.912423+0000", 2024-09-10T18:40:48.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.94950994899999996 2024-09-10T18:40:48.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.912521+0000", 2024-09-10T18:40:48.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7816e-05 2024-09-10T18:40:48.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:40:48.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.912635+0000", 2024-09-10T18:40:48.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114059 2024-09-10T18:40:48.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:40:48.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920581+0000", 2024-09-10T18:40:48.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0079467129999999993 2024-09-10T18:40:48.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920844+0000", 2024-09-10T18:40:48.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00026251600000000002 2024-09-10T18:40:48.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920870+0000", 2024-09-10T18:40:48.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6101999999999998e-05 2024-09-10T18:40:48.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:40:48.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920908+0000", 2024-09-10T18:40:48.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7902999999999999e-05 2024-09-10T18:40:48.350 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920938+0000", 2024-09-10T18:40:48.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9807000000000001e-05 2024-09-10T18:40:48.350 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.350 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.350 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.350 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.351 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224061 10.9 10:9c470190:::benchmark_data_smithi028_161947_object87137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:40:48.351 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.963690+0000", 2024-09-10T18:40:48.351 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 107.312296274, 2024-09-10T18:40:48.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95472792100000003, 2024-09-10T18:40:48.351 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.351 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:48.351 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.351 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:40:48.351 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:40:48.351 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224061 2024-09-10T18:40:48.351 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.352 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.963690+0000", 2024-09-10T18:40:48.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.963690+0000", 2024-09-10T18:40:48.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.963689+0000", 2024-09-10T18:40:48.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:48.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:48.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:48.354 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 31 16 61766 61750 124.483 92.1875 0.00949988 0.00802793 2024-09-10T18:40:48.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.963695+0000", 2024-09-10T18:40:48.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8100000000000003e-06 2024-09-10T18:40:48.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.963706+0000", 2024-09-10T18:40:48.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0927999999999999e-05 2024-09-10T18:40:48.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.963711+0000", 2024-09-10T18:40:48.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9219999999999996e-06 2024-09-10T18:40:48.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.963757+0000", 2024-09-10T18:40:48.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6007000000000002e-05 2024-09-10T18:40:48.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.963765+0000", 2024-09-10T18:40:48.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3939999999999996e-06 2024-09-10T18:40:48.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.910047+0000", 2024-09-10T18:40:48.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.94628242299999998 2024-09-10T18:40:48.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.910156+0000", 2024-09-10T18:40:48.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010876 2024-09-10T18:40:48.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,7", 2024-09-10T18:40:48.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.910244+0000", 2024-09-10T18:40:48.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7503000000000006e-05 2024-09-10T18:40:48.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:40:48.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918148+0000", 2024-09-10T18:40:48.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0079044839999999998 2024-09-10T18:40:48.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918303+0000", 2024-09-10T18:40:48.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015505599999999999 2024-09-10T18:40:48.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918325+0000", 2024-09-10T18:40:48.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1437000000000002e-05 2024-09-10T18:40:48.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:40:48.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918387+0000", 2024-09-10T18:40:48.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2385999999999995e-05 2024-09-10T18:40:48.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918418+0000", 2024-09-10T18:40:48.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1032999999999999e-05 2024-09-10T18:40:48.359 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.359 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.359 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.360 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224063 10.9 10:985fa581:::benchmark_data_smithi028_161947_object87139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:40:48.360 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.964621+0000", 2024-09-10T18:40:48.360 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 107.31136586, 2024-09-10T18:40:48.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95390776799999999, 2024-09-10T18:40:48.360 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.360 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:48.360 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.360 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:40:48.360 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:40:48.360 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224063 2024-09-10T18:40:48.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.360 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.964621+0000", 2024-09-10T18:40:48.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.964621+0000", 2024-09-10T18:40:48.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.964619+0000", 2024-09-10T18:40:48.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:48.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.362 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.362 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.964628+0000", 2024-09-10T18:40:48.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0920000000000004e-06 2024-09-10T18:40:48.362 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.362 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.362 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.362 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.964640+0000", 2024-09-10T18:40:48.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2111e-05 2024-09-10T18:40:48.363 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.964646+0000", 2024-09-10T18:40:48.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6910000000000002e-06 2024-09-10T18:40:48.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.964684+0000", 2024-09-10T18:40:48.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7898999999999997e-05 2024-09-10T18:40:48.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.964692+0000", 2024-09-10T18:40:48.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6760000000000007e-06 2024-09-10T18:40:48.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.912470+0000", 2024-09-10T18:40:48.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.94777832299999998 2024-09-10T18:40:48.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.912560+0000", 2024-09-10T18:40:48.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0186999999999996e-05 2024-09-10T18:40:48.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,7", 2024-09-10T18:40:48.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.912654+0000", 2024-09-10T18:40:48.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3729999999999999e-05 2024-09-10T18:40:48.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:40:48.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918152+0000", 2024-09-10T18:40:48.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054985470000000003 2024-09-10T18:40:48.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918445+0000", 2024-09-10T18:40:48.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029213299999999998 2024-09-10T18:40:48.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918466+0000", 2024-09-10T18:40:48.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1158e-05 2024-09-10T18:40:48.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:40:48.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918501+0000", 2024-09-10T18:40:48.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4919999999999998e-05 2024-09-10T18:40:48.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918528+0000", 2024-09-10T18:40:48.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7661000000000002e-05 2024-09-10T18:40:48.368 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.368 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.368 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.368 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224069 10.9 10:99c920d9:::benchmark_data_smithi028_161947_object87145:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:40:48.368 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.970604+0000", 2024-09-10T18:40:48.368 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 107.30538217, 2024-09-10T18:40:48.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.954060515, 2024-09-10T18:40:48.368 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.369 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:48.369 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.369 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:40:48.369 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:40:48.369 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224069 2024-09-10T18:40:48.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.369 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.970604+0000", 2024-09-10T18:40:48.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.970604+0000", 2024-09-10T18:40:48.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.970602+0000", 2024-09-10T18:40:48.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:40:48.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.970611+0000", 2024-09-10T18:40:48.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2450000000000004e-06 2024-09-10T18:40:48.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.970629+0000", 2024-09-10T18:40:48.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7819e-05 2024-09-10T18:40:48.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.970636+0000", 2024-09-10T18:40:48.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7329999999999997e-06 2024-09-10T18:40:48.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.970685+0000", 2024-09-10T18:40:48.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9771000000000002e-05 2024-09-10T18:40:48.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.970692+0000", 2024-09-10T18:40:48.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1559999999999998e-06 2024-09-10T18:40:48.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.914825+0000", 2024-09-10T18:40:48.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.94413236199999995 2024-09-10T18:40:48.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.914929+0000", 2024-09-10T18:40:48.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0001041 2024-09-10T18:40:48.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,7", 2024-09-10T18:40:48.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.915027+0000", 2024-09-10T18:40:48.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8393000000000005e-05 2024-09-10T18:40:48.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:40:48.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918157+0000", 2024-09-10T18:40:48.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0031293979999999998 2024-09-10T18:40:48.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.922005+0000", 2024-09-10T18:40:48.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038485170000000001 2024-09-10T18:40:48.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.924547+0000", 2024-09-10T18:40:48.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002541387 2024-09-10T18:40:48.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:40:48.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.924628+0000", 2024-09-10T18:40:48.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1588000000000006e-05 2024-09-10T18:40:48.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.924665+0000", 2024-09-10T18:40:48.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6541999999999997e-05 2024-09-10T18:40:48.375 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.375 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.375 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.376 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224070 10.6 10:6d42de77:::benchmark_data_smithi028_161947_object87146:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:40:48.376 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.971750+0000", 2024-09-10T18:40:48.376 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 107.304236305, 2024-09-10T18:40:48.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.94993114000000001, 2024-09-10T18:40:48.376 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.376 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:48.376 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.376 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:40:48.376 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:40:48.376 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224070 2024-09-10T18:40:48.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.376 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.376 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.971750+0000", 2024-09-10T18:40:48.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.971750+0000", 2024-09-10T18:40:48.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.971749+0000", 2024-09-10T18:40:48.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:48.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.971758+0000", 2024-09-10T18:40:48.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6969999999999999e-06 2024-09-10T18:40:48.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.971768+0000", 2024-09-10T18:40:48.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0030000000000001e-05 2024-09-10T18:40:48.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.971772+0000", 2024-09-10T18:40:48.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3229999999999999e-06 2024-09-10T18:40:48.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.971809+0000", 2024-09-10T18:40:48.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7237999999999998e-05 2024-09-10T18:40:48.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.971815+0000", 2024-09-10T18:40:48.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4299999999999997e-06 2024-09-10T18:40:48.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.914826+0000", 2024-09-10T18:40:48.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.943011759 2024-09-10T18:40:48.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.914923+0000", 2024-09-10T18:40:48.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6811000000000005e-05 2024-09-10T18:40:48.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:40:48.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.915020+0000", 2024-09-10T18:40:48.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6992999999999998e-05 2024-09-10T18:40:48.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:40:48.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920586+0000", 2024-09-10T18:40:48.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0055662569999999998 2024-09-10T18:40:48.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920971+0000", 2024-09-10T18:40:48.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00038463800000000003 2024-09-10T18:40:48.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.921576+0000", 2024-09-10T18:40:48.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00060466700000000003 2024-09-10T18:40:48.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:40:48.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.921645+0000", 2024-09-10T18:40:48.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9391e-05 2024-09-10T18:40:48.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.921681+0000", 2024-09-10T18:40:48.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6183000000000002e-05 2024-09-10T18:40:48.382 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.382 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.382 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.383 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224074 10.6 10:69c7ac47:::benchmark_data_smithi028_161947_object87150:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:40:48.383 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.980073+0000", 2024-09-10T18:40:48.383 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 107.295913645, 2024-09-10T18:40:48.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.944869819, 2024-09-10T18:40:48.383 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.383 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:48.383 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.383 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:40:48.383 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:40:48.383 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224074 2024-09-10T18:40:48.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.383 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.980073+0000", 2024-09-10T18:40:48.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.980073+0000", 2024-09-10T18:40:48.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.980071+0000", 2024-09-10T18:40:48.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:48.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.385 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.385 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.980077+0000", 2024-09-10T18:40:48.385 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8939999999999998e-06 2024-09-10T18:40:48.385 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.385 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.385 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.980087+0000", 2024-09-10T18:40:48.385 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0057e-05 2024-09-10T18:40:48.385 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.385 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.385 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.980091+0000", 2024-09-10T18:40:48.385 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0260000000000001e-06 2024-09-10T18:40:48.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.386 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.980132+0000", 2024-09-10T18:40:48.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1056999999999997e-05 2024-09-10T18:40:48.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.386 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.980140+0000", 2024-09-10T18:40:48.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.328e-06 2024-09-10T18:40:48.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.917565+0000", 2024-09-10T18:40:48.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.937425709 2024-09-10T18:40:48.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.917675+0000", 2024-09-10T18:40:48.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000110024 2024-09-10T18:40:48.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:40:48.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.917937+0000", 2024-09-10T18:40:48.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00026186100000000001 2024-09-10T18:40:48.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.921847+0000", 2024-09-10T18:40:48.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039093970000000002 2024-09-10T18:40:48.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:40:48.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.924796+0000", 2024-09-10T18:40:48.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029495699999999999 2024-09-10T18:40:48.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.924827+0000", 2024-09-10T18:40:48.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1102000000000001e-05 2024-09-10T18:40:48.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:40:48.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.924905+0000", 2024-09-10T18:40:48.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7860000000000003e-05 2024-09-10T18:40:48.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.924943+0000", 2024-09-10T18:40:48.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7391000000000002e-05 2024-09-10T18:40:48.389 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.389 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.390 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.390 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.34334.0:2471 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 94344803721280 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:40:48.390 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.218473+0000", 2024-09-10T18:40:48.390 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 84.057513436999997, 2024-09-10T18:40:48.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.047128775, 2024-09-10T18:40:48.390 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.390 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:40:48.390 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.390 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.34334", 2024-09-10T18:40:48.390 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/601695668", 2024-09-10T18:40:48.390 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2471 2024-09-10T18:40:48.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.391 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.218473+0000", 2024-09-10T18:40:48.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.218473+0000", 2024-09-10T18:40:48.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.218472+0000", 2024-09-10T18:40:48.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:48.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.392 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.392 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.218477+0000", 2024-09-10T18:40:48.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1499999999999998e-06 2024-09-10T18:40:48.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.392 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.392 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.218485+0000", 2024-09-10T18:40:48.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4589999999999998e-06 2024-09-10T18:40:48.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.218491+0000", 2024-09-10T18:40:48.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8239999999999996e-06 2024-09-10T18:40:48.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.218545+0000", 2024-09-10T18:40:48.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4008000000000002e-05 2024-09-10T18:40:48.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-10T18:40:48.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.218550+0000", 2024-09-10T18:40:48.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1340000000000003e-06 2024-09-10T18:40:48.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.233529+0000", 2024-09-10T18:40:48.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014979019999999999 2024-09-10T18:40:48.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.233539+0000", 2024-09-10T18:40:48.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3360000000000001e-06 2024-09-10T18:40:48.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.256425+0000", 2024-09-10T18:40:48.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0228867580000001 2024-09-10T18:40:48.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for degraded object", 2024-09-10T18:40:48.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.256708+0000", 2024-09-10T18:40:48.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00028321200000000003 2024-09-10T18:40:48.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.258856+0000", 2024-09-10T18:40:48.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021476680000000001 2024-09-10T18:40:48.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.258949+0000", 2024-09-10T18:40:48.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3283999999999997e-05 2024-09-10T18:40:48.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.265602+0000", 2024-09-10T18:40:48.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0066523099999999998 2024-09-10T18:40:48.396 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.396 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.396 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.397 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253541 10.6 10:6c835e9f:::benchmark_data_smithi028_161947_object116617:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:40:48.397 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.242223+0000", 2024-09-10T18:40:48.397 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 84.033763768, 2024-09-10T18:40:48.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2265690720000002, 2024-09-10T18:40:48.397 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.397 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:48.397 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.397 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:40:48.397 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:40:48.397 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253541 2024-09-10T18:40:48.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.397 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242223+0000", 2024-09-10T18:40:48.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242223+0000", 2024-09-10T18:40:48.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242220+0000", 2024-09-10T18:40:48.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:40:48.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242228+0000", 2024-09-10T18:40:48.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6229999999999997e-06 2024-09-10T18:40:48.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242243+0000", 2024-09-10T18:40:48.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5296000000000001e-05 2024-09-10T18:40:48.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242250+0000", 2024-09-10T18:40:48.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5169999999999996e-06 2024-09-10T18:40:48.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242299+0000", 2024-09-10T18:40:48.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8884999999999997e-05 2024-09-10T18:40:48.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242307+0000", 2024-09-10T18:40:48.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0560000000000005e-06 2024-09-10T18:40:48.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.227655+0000", 2024-09-10T18:40:48.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98534856000000004 2024-09-10T18:40:48.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.227667+0000", 2024-09-10T18:40:48.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1787000000000001e-05 2024-09-10T18:40:48.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.405394+0000", 2024-09-10T18:40:48.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1777274360000001 2024-09-10T18:40:48.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.405530+0000", 2024-09-10T18:40:48.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013596300000000001 2024-09-10T18:40:48.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1,4", 2024-09-10T18:40:48.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.406018+0000", 2024-09-10T18:40:48.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000487601 2024-09-10T18:40:48.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:40:48.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.423091+0000", 2024-09-10T18:40:48.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017072539000000001 2024-09-10T18:40:48.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.429936+0000", 2024-09-10T18:40:48.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068454869999999999 2024-09-10T18:40:48.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468505+0000", 2024-09-10T18:40:48.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.038568526999999998 2024-09-10T18:40:48.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468621+0000", 2024-09-10T18:40:48.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011692999999999999 2024-09-10T18:40:48.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:40:48.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468674+0000", 2024-09-10T18:40:48.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2874000000000002e-05 2024-09-10T18:40:48.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468792+0000", 2024-09-10T18:40:48.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000117383 2024-09-10T18:40:48.404 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.404 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.404 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.405 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253543 10.6 10:606298d4:::benchmark_data_smithi028_161947_object116619:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:40:48.405 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.243923+0000", 2024-09-10T18:40:48.405 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 84.032063644000004, 2024-09-10T18:40:48.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.225144625, 2024-09-10T18:40:48.405 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.405 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:48.405 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.405 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:40:48.405 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:40:48.405 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253543 2024-09-10T18:40:48.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.405 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.405 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.243923+0000", 2024-09-10T18:40:48.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.243923+0000", 2024-09-10T18:40:48.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.243921+0000", 2024-09-10T18:40:48.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:40:48.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.243928+0000", 2024-09-10T18:40:48.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.922e-06 2024-09-10T18:40:48.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.243942+0000", 2024-09-10T18:40:48.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4494e-05 2024-09-10T18:40:48.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.243948+0000", 2024-09-10T18:40:48.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6509999999999998e-06 2024-09-10T18:40:48.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.243988+0000", 2024-09-10T18:40:48.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.002e-05 2024-09-10T18:40:48.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.243996+0000", 2024-09-10T18:40:48.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5730000000000002e-06 2024-09-10T18:40:48.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.227691+0000", 2024-09-10T18:40:48.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98369558000000001 2024-09-10T18:40:48.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.227698+0000", 2024-09-10T18:40:48.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6640000000000001e-06 2024-09-10T18:40:48.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.413292+0000", 2024-09-10T18:40:48.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1855940709999999 2024-09-10T18:40:48.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.413442+0000", 2024-09-10T18:40:48.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000149734 2024-09-10T18:40:48.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1,4", 2024-09-10T18:40:48.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.413594+0000", 2024-09-10T18:40:48.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015274900000000001 2024-09-10T18:40:48.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:40:48.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.423097+0000", 2024-09-10T18:40:48.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095028400000000002 2024-09-10T18:40:48.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.429968+0000", 2024-09-10T18:40:48.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068706779999999999 2024-09-10T18:40:48.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468564+0000", 2024-09-10T18:40:48.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.038596485999999999 2024-09-10T18:40:48.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468913+0000", 2024-09-10T18:40:48.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000348162 2024-09-10T18:40:48.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:40:48.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468958+0000", 2024-09-10T18:40:48.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5558e-05 2024-09-10T18:40:48.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.469067+0000", 2024-09-10T18:40:48.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000109269 2024-09-10T18:40:48.413 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.413 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.413 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.413 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253556 10.6 10:699fc70f:::benchmark_data_smithi028_161947_object116632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:40:48.413 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.273799+0000", 2024-09-10T18:40:48.413 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 84.002186968999993, 2024-09-10T18:40:48.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.195505587, 2024-09-10T18:40:48.413 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.414 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:40:48.414 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.414 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:40:48.414 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:40:48.414 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253556 2024-09-10T18:40:48.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.414 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.273799+0000", 2024-09-10T18:40:48.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.273799+0000", 2024-09-10T18:40:48.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.273798+0000", 2024-09-10T18:40:48.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:40:48.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.273804+0000", 2024-09-10T18:40:48.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.784e-06 2024-09-10T18:40:48.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.273813+0000", 2024-09-10T18:40:48.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8349999999999993e-06 2024-09-10T18:40:48.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.273818+0000", 2024-09-10T18:40:48.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1399999999999999e-06 2024-09-10T18:40:48.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.273852+0000", 2024-09-10T18:40:48.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3973999999999997e-05 2024-09-10T18:40:48.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.273857+0000", 2024-09-10T18:40:48.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5659999999999997e-06 2024-09-10T18:40:48.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.227726+0000", 2024-09-10T18:40:48.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95386886699999995 2024-09-10T18:40:48.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.227734+0000", 2024-09-10T18:40:48.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2579999999999995e-06 2024-09-10T18:40:48.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.418473+0000", 2024-09-10T18:40:48.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.190738742 2024-09-10T18:40:48.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.418607+0000", 2024-09-10T18:40:48.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013377699999999999 2024-09-10T18:40:48.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1,4", 2024-09-10T18:40:48.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.418762+0000", 2024-09-10T18:40:48.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015565800000000001 2024-09-10T18:40:48.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:40:48.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.423103+0000", 2024-09-10T18:40:48.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0043402429999999997 2024-09-10T18:40:48.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.430008+0000", 2024-09-10T18:40:48.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0069057229999999999 2024-09-10T18:40:48.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468852+0000", 2024-09-10T18:40:48.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.038843436000000002 2024-09-10T18:40:48.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:40:48.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.469138+0000", 2024-09-10T18:40:48.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00028564399999999999 2024-09-10T18:40:48.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:40:48.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.469185+0000", 2024-09-10T18:40:48.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7531999999999998e-05 2024-09-10T18:40:48.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.469305+0000", 2024-09-10T18:40:48.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011996499999999999 2024-09-10T18:40:48.421 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.421 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.421 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.422 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.34334.0:2477 5.15 5:a93a5511:::notify.2:head [watch ping cookie 94344803721280 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:40:48.422 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.976177+0000", 2024-09-10T18:40:48.422 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 83.299809504999999, 2024-09-10T18:40:48.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.28319640800000001, 2024-09-10T18:40:48.422 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:40:48.422 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:40:48.422 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:40:48.422 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.34334", 2024-09-10T18:40:48.422 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/601695668", 2024-09-10T18:40:48.422 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2477 2024-09-10T18:40:48.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.422 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:40:48.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:40:48.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.976177+0000", 2024-09-10T18:40:48.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:40:48.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.976177+0000", 2024-09-10T18:40:48.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:40:48.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:40:48.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.976174+0000", 2024-09-10T18:40:48.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:40:48.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:40:48.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.976184+0000", 2024-09-10T18:40:48.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.645e-06 2024-09-10T18:40:48.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:40:48.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.976200+0000", 2024-09-10T18:40:48.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5800000000000001e-05 2024-09-10T18:40:48.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:40:48.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.976209+0000", 2024-09-10T18:40:48.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0790000000000007e-06 2024-09-10T18:40:48.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.976282+0000", 2024-09-10T18:40:48.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3082000000000003e-05 2024-09-10T18:40:48.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:40:48.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.976297+0000", 2024-09-10T18:40:48.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5048999999999999e-05 2024-09-10T18:40:48.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.256741+0000", 2024-09-10T18:40:48.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.280443893 2024-09-10T18:40:48.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for degraded object", 2024-09-10T18:40:48.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.256791+0000", 2024-09-10T18:40:48.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0386999999999997e-05 2024-09-10T18:40:48.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:40:48.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.259216+0000", 2024-09-10T18:40:48.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024254200000000002 2024-09-10T18:40:48.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:40:48.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.259272+0000", 2024-09-10T18:40:48.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5933000000000001e-05 2024-09-10T18:40:48.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:40:48.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:40:48.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:40:48.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.259373+0000", 2024-09-10T18:40:48.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000100932 2024-09-10T18:40:48.427 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.427 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.427 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.427 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:40:48.428 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:40:48.428 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:40:48.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:48.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:48.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:48.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:48.639 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:48.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:48.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:48.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:48.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:48.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:48.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:48.923 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:48.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:48.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:49.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:49.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:49.066 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:49.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:49.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:49.206 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:49.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:49.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:49 smithi028 ceph-mon[39220]: 4.13 scrub starts 2024-09-10T18:40:49.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:49 smithi028 ceph-mon[39220]: 4.13 scrub ok 2024-09-10T18:40:49.241 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:49 smithi028 ceph-mon[29291]: 4.13 scrub starts 2024-09-10T18:40:49.241 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:49 smithi028 ceph-mon[29291]: 4.13 scrub ok 2024-09-10T18:40:49.269 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:49 smithi204 ceph-mon[32357]: 4.13 scrub starts 2024-09-10T18:40:49.269 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:49 smithi204 ceph-mon[32357]: 4.13 scrub ok 2024-09-10T18:40:49.303 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-10T18:40:49.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:49.321 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 32 16 63259 63243 123.495 93.3125 0.0215269 0.00809208 2024-09-10T18:40:49.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:49.344 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:49.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:49.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:40:49.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:49.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:49.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:49.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:49.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:49.635 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:49.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:49.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:49.688 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:40:49.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:49.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:49.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:49.885 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:49.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:49.917 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:49.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:50.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:50.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:50.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:50.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:50 smithi204 ceph-mon[32357]: pgmap v871: 178 pgs: 178 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-10T18:40:50.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:50 smithi204 ceph-mon[32357]: 1.1d scrub starts 2024-09-10T18:40:50.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:50 smithi204 ceph-mon[32357]: 1.1d scrub ok 2024-09-10T18:40:50.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:50 smithi204 ceph-mon[32357]: 6.0 scrub starts 2024-09-10T18:40:50.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:50 smithi204 ceph-mon[32357]: 6.0 scrub ok 2024-09-10T18:40:50.105 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:50 smithi028 ceph-mon[39220]: pgmap v871: 178 pgs: 178 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-10T18:40:50.105 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:50 smithi028 ceph-mon[39220]: 1.1d scrub starts 2024-09-10T18:40:50.105 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:50 smithi028 ceph-mon[39220]: 1.1d scrub ok 2024-09-10T18:40:50.105 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:50 smithi028 ceph-mon[39220]: 6.0 scrub starts 2024-09-10T18:40:50.105 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:50 smithi028 ceph-mon[39220]: 6.0 scrub ok 2024-09-10T18:40:50.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:50 smithi028 ceph-mon[29291]: pgmap v871: 178 pgs: 178 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-10T18:40:50.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:50 smithi028 ceph-mon[29291]: 1.1d scrub starts 2024-09-10T18:40:50.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:50 smithi028 ceph-mon[29291]: 1.1d scrub ok 2024-09-10T18:40:50.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:50 smithi028 ceph-mon[29291]: 6.0 scrub starts 2024-09-10T18:40:50.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:50 smithi028 ceph-mon[29291]: 6.0 scrub ok 2024-09-10T18:40:50.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:50.225 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:50.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:50.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:50.321 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 33 16 64487 64471 122.079 76.75 0.0118136 0.00818651 2024-09-10T18:40:50.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:50.364 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:50.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:50.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:50.466 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:50.470 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:50.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:50.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:50.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:50.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:50.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:50.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:50.739 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:50.776 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:50.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:50.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:50.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:50.919 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:50.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:50.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:51.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:51.037 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:40:51.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:51 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/150582348' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:40:51.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:51 smithi028 ceph-mon[29291]: from='client.35903 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:40:51.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:51 smithi028 ceph-mon[29291]: 4.0 scrub starts 2024-09-10T18:40:51.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:51 smithi028 ceph-mon[29291]: 4.0 scrub ok 2024-09-10T18:40:51.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:51 smithi028 ceph-mon[29291]: 2.2 scrub starts 2024-09-10T18:40:51.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:51 smithi028 ceph-mon[29291]: 2.2 scrub ok 2024-09-10T18:40:51.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:51 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/150582348' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:40:51.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:51 smithi028 ceph-mon[39220]: from='client.35903 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:40:51.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:51 smithi028 ceph-mon[39220]: 4.0 scrub starts 2024-09-10T18:40:51.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:51 smithi028 ceph-mon[39220]: 4.0 scrub ok 2024-09-10T18:40:51.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:51 smithi028 ceph-mon[39220]: 2.2 scrub starts 2024-09-10T18:40:51.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:51 smithi028 ceph-mon[39220]: 2.2 scrub ok 2024-09-10T18:40:51.062 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:51.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:51.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:51.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:51.201 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:51.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:51.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:51.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:51.322 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 34 16 65938 65922 121.152 90.6875 0.00830874 0.00824895 2024-09-10T18:40:51.341 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:51.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:51.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:51 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/150582348' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:40:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:51 smithi204 ceph-mon[32357]: from='client.35903 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:40:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:51 smithi204 ceph-mon[32357]: 4.0 scrub starts 2024-09-10T18:40:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:51 smithi204 ceph-mon[32357]: 4.0 scrub ok 2024-09-10T18:40:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:51 smithi204 ceph-mon[32357]: 2.2 scrub starts 2024-09-10T18:40:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:51 smithi204 ceph-mon[32357]: 2.2 scrub ok 2024-09-10T18:40:51.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:51.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:51.477 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:51.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:51.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:51.618 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:51.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:51.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:51.720 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:51.753 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:40:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:40:51] "GET /metrics HTTP/1.1" 200 224883 "" "Prometheus/2.43.0" 2024-09-10T18:40:51.757 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:51.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:51.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:51.859 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:51.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:51.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:51.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:51.986 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:52.017 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:52.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:52.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:52.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:52.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:52.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:52.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:40:52.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:40:52.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[39220]: 5.1e scrub starts 2024-09-10T18:40:52.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[39220]: from='client.35903 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:40:52.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[39220]: osdmap e384: 8 total, 8 up, 7 in 2024-09-10T18:40:52.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[39220]: 5.1e scrub ok 2024-09-10T18:40:52.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[39220]: pgmap v873: 178 pgs: 178 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-10T18:40:52.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:40:52.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:40:52.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[39220]: 5.16 deep-scrub starts 2024-09-10T18:40:52.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[39220]: 5.16 deep-scrub ok 2024-09-10T18:40:52.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:40:52.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:40:52.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[29291]: 5.1e scrub starts 2024-09-10T18:40:52.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[29291]: from='client.35903 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:40:52.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[29291]: osdmap e384: 8 total, 8 up, 7 in 2024-09-10T18:40:52.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[29291]: 5.1e scrub ok 2024-09-10T18:40:52.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[29291]: pgmap v873: 178 pgs: 178 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-10T18:40:52.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:40:52.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:40:52.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[29291]: 5.16 deep-scrub starts 2024-09-10T18:40:52.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:52 smithi028 ceph-mon[29291]: 5.16 deep-scrub ok 2024-09-10T18:40:52.258 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:52.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:52.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:52.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:52.322 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 35 16 67328 67312 120.173 86.875 0.0229372 0.00831589 2024-09-10T18:40:52.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:52 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:40:52.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:52 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:40:52.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:52 smithi204 ceph-mon[32357]: 5.1e scrub starts 2024-09-10T18:40:52.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:52 smithi204 ceph-mon[32357]: from='client.35903 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:40:52.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:52 smithi204 ceph-mon[32357]: osdmap e384: 8 total, 8 up, 7 in 2024-09-10T18:40:52.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:52 smithi204 ceph-mon[32357]: 5.1e scrub ok 2024-09-10T18:40:52.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:52 smithi204 ceph-mon[32357]: pgmap v873: 178 pgs: 178 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-10T18:40:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:52 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:40:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:52 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:40:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:52 smithi204 ceph-mon[32357]: 5.16 deep-scrub starts 2024-09-10T18:40:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:52 smithi204 ceph-mon[32357]: 5.16 deep-scrub ok 2024-09-10T18:40:52.355 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:40:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:40:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:40:52.401 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:52.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:52.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:52.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:52.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:52.581 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:52.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:52.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:52.682 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:52.721 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:52.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:52.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:52.824 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:52.862 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:52.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:52.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:52.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:53.003 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:53.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:53.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:53.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:53.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:53.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:53.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:53.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:53 smithi028 ceph-mon[29291]: 6.16 scrub starts 2024-09-10T18:40:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:53 smithi028 ceph-mon[29291]: 6.16 scrub ok 2024-09-10T18:40:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:53 smithi028 ceph-mon[29291]: osdmap e385: 8 total, 8 up, 7 in 2024-09-10T18:40:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:53 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:40:53.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:53 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:40:53.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:53 smithi028 ceph-mon[29291]: 1.17 deep-scrub starts 2024-09-10T18:40:53.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:53 smithi028 ceph-mon[29291]: 1.17 deep-scrub ok 2024-09-10T18:40:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:53 smithi028 ceph-mon[39220]: 6.16 scrub starts 2024-09-10T18:40:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:53 smithi028 ceph-mon[39220]: 6.16 scrub ok 2024-09-10T18:40:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:53 smithi028 ceph-mon[39220]: osdmap e385: 8 total, 8 up, 7 in 2024-09-10T18:40:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:53 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:40:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:53 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:40:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:53 smithi028 ceph-mon[39220]: 1.17 deep-scrub starts 2024-09-10T18:40:53.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:53 smithi028 ceph-mon[39220]: 1.17 deep-scrub ok 2024-09-10T18:40:53.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:53.283 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:53.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:53.323 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 36 16 68758 68742 119.314 89.375 0.00926779 0.00837653 2024-09-10T18:40:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:53 smithi204 ceph-mon[32357]: 6.16 scrub starts 2024-09-10T18:40:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:53 smithi204 ceph-mon[32357]: 6.16 scrub ok 2024-09-10T18:40:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:53 smithi204 ceph-mon[32357]: osdmap e385: 8 total, 8 up, 7 in 2024-09-10T18:40:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:53 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:40:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:53 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:40:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:53 smithi204 ceph-mon[32357]: 1.17 deep-scrub starts 2024-09-10T18:40:53.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:53 smithi204 ceph-mon[32357]: 1.17 deep-scrub ok 2024-09-10T18:40:53.385 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:53.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-10T18:40:53.426 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:53.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:53.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:53.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:53.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:53.676 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:53.711 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:53.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:53.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:53.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:53.866 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:53.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:53.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:53.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:54.009 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:54.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:54.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:54.024 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:40:54.025 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:40:54.057 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:40:54.057 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:40:54.096 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:40:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:40:54.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:54.131 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:40:54.131 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:40:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:54.160 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:54.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:54.162 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:40:54.163 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:40:54.197 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:40:54.197 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:40:54.231 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:40:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:40:54.232 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[39220]: 5.1c scrub starts 2024-09-10T18:40:54.232 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[39220]: 5.1c scrub ok 2024-09-10T18:40:54.232 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[39220]: osdmap e386: 8 total, 8 up, 7 in 2024-09-10T18:40:54.232 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[39220]: pgmap v876: 178 pgs: 178 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-10T18:40:54.232 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[39220]: 5.12 scrub starts 2024-09-10T18:40:54.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[39220]: 5.12 scrub ok 2024-09-10T18:40:54.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:40:54.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:40:54.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1006818415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:40:54.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[29291]: 5.1c scrub starts 2024-09-10T18:40:54.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[29291]: 5.1c scrub ok 2024-09-10T18:40:54.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[29291]: osdmap e386: 8 total, 8 up, 7 in 2024-09-10T18:40:54.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[29291]: pgmap v876: 178 pgs: 178 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-10T18:40:54.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[29291]: 5.12 scrub starts 2024-09-10T18:40:54.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[29291]: 5.12 scrub ok 2024-09-10T18:40:54.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:40:54.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:40:54.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:54 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1006818415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:40:54.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:54.267 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:40:54.267 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:40:54.298 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:54.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:54.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:54.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:40:54.323 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 37 16 70117 70101 118.385 84.9375 0.01279 0.0084422 2024-09-10T18:40:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:54 smithi204 ceph-mon[32357]: 5.1c scrub starts 2024-09-10T18:40:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:54 smithi204 ceph-mon[32357]: 5.1c scrub ok 2024-09-10T18:40:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:54 smithi204 ceph-mon[32357]: osdmap e386: 8 total, 8 up, 7 in 2024-09-10T18:40:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:54 smithi204 ceph-mon[32357]: pgmap v876: 178 pgs: 178 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-10T18:40:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:54 smithi204 ceph-mon[32357]: 5.12 scrub starts 2024-09-10T18:40:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:54 smithi204 ceph-mon[32357]: 5.12 scrub ok 2024-09-10T18:40:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:54 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:40:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:54 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:40:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:54 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1006818415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:40:54.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:54.438 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:54.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:54.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:54.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:54.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:54.583 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:54.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:54.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:54.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:54.719 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:54.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:54.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:54.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:54.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:54.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:54.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:54.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:54.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:55.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:40:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:40:55.099 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:55.126 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:55.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:55.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:55.228 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:55.259 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:55.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:55.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:55.323 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 38 16 71326 71310 117.258 75.5625 0.00296178 0.00851156 2024-09-10T18:40:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:55 smithi204 ceph-mon[32357]: 4.1f scrub starts 2024-09-10T18:40:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:55 smithi204 ceph-mon[32357]: 4.1f scrub ok 2024-09-10T18:40:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:55 smithi204 ceph-mon[32357]: 6.e deep-scrub starts 2024-09-10T18:40:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:55 smithi204 ceph-mon[32357]: 6.e deep-scrub ok 2024-09-10T18:40:55.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:55.396 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:55.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:55.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:55.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:55.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:55 smithi028 ceph-mon[29291]: 4.1f scrub starts 2024-09-10T18:40:55.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:55 smithi028 ceph-mon[29291]: 4.1f scrub ok 2024-09-10T18:40:55.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:55 smithi028 ceph-mon[29291]: 6.e deep-scrub starts 2024-09-10T18:40:55.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:55 smithi028 ceph-mon[29291]: 6.e deep-scrub ok 2024-09-10T18:40:55.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:55 smithi028 ceph-mon[39220]: 4.1f scrub starts 2024-09-10T18:40:55.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:55 smithi028 ceph-mon[39220]: 4.1f scrub ok 2024-09-10T18:40:55.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:55 smithi028 ceph-mon[39220]: 6.e deep-scrub starts 2024-09-10T18:40:55.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:55 smithi028 ceph-mon[39220]: 6.e deep-scrub ok 2024-09-10T18:40:55.538 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:55.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:55.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:55.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:55.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:55.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:55.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:55.764 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:55.801 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:55.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:55.903 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:55.932 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:55.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:55.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:56.034 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:56.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:56.074 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:56.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:56.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:56.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:56.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:56.317 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:56.323 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 39 16 72631 72615 116.343 81.5625 0.00290358 0.00858044 2024-09-10T18:40:56.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:56 smithi204 ceph-mon[32357]: 5.3 scrub starts 2024-09-10T18:40:56.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:56 smithi204 ceph-mon[32357]: 5.3 scrub ok 2024-09-10T18:40:56.332 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:56 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:56.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:56 smithi204 ceph-mon[32357]: 4.9 deep-scrub starts 2024-09-10T18:40:56.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:56 smithi204 ceph-mon[32357]: 4.9 deep-scrub ok 2024-09-10T18:40:56.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:56 smithi204 ceph-mon[32357]: pgmap v877: 178 pgs: 178 active+clean; 4.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-10T18:40:56.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:56 smithi204 ceph-mon[32357]: 4.1d scrub starts 2024-09-10T18:40:56.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:56 smithi204 ceph-mon[32357]: 4.1d scrub ok 2024-09-10T18:40:56.333 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:56 smithi204 ceph-mon[32357]: osdmap e387: 8 total, 8 up, 7 in 2024-09-10T18:40:56.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:56.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:56.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:56.451 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:56.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:56.473 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:56.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:56.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[39220]: 5.3 scrub starts 2024-09-10T18:40:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[39220]: 5.3 scrub ok 2024-09-10T18:40:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[39220]: 4.9 deep-scrub starts 2024-09-10T18:40:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[39220]: 4.9 deep-scrub ok 2024-09-10T18:40:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[39220]: pgmap v877: 178 pgs: 178 active+clean; 4.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-10T18:40:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[39220]: 4.1d scrub starts 2024-09-10T18:40:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[39220]: 4.1d scrub ok 2024-09-10T18:40:56.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[39220]: osdmap e387: 8 total, 8 up, 7 in 2024-09-10T18:40:56.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[29291]: 5.3 scrub starts 2024-09-10T18:40:56.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[29291]: 5.3 scrub ok 2024-09-10T18:40:56.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:40:56.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[29291]: 4.9 deep-scrub starts 2024-09-10T18:40:56.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[29291]: 4.9 deep-scrub ok 2024-09-10T18:40:56.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[29291]: pgmap v877: 178 pgs: 178 active+clean; 4.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-10T18:40:56.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[29291]: 4.1d scrub starts 2024-09-10T18:40:56.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[29291]: 4.1d scrub ok 2024-09-10T18:40:56.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:56 smithi028 ceph-mon[29291]: osdmap e387: 8 total, 8 up, 7 in 2024-09-10T18:40:56.574 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:56.594 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:56.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:56.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:56.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:56.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:56.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:56.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:56.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:56.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:56.855 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:56.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:56.956 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:40:56.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:56.979 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:56.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:40:57.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:57.101 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:57.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:57.203 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:57.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:57.238 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:57.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:57.324 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:40:57.323283+0000 min lat: 0.00125893 max lat: 0.413127 avg lat: 0.00874325 2024-09-10T18:40:57.324 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:40:57.324 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 40 16 73191 73175 114.309 35 0.00942704 0.00874325 2024-09-10T18:40:57.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:57.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:57.381 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:57.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:57.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:57.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:57 smithi204 ceph-mon[32357]: 3.1a scrub starts 2024-09-10T18:40:57.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:57 smithi204 ceph-mon[32357]: 3.1a scrub ok 2024-09-10T18:40:57.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:57 smithi204 ceph-mon[32357]: 3.13 scrub starts 2024-09-10T18:40:57.496 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:57 smithi204 ceph-mon[32357]: 3.13 scrub ok 2024-09-10T18:40:57.497 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:57 smithi204 ceph-mon[32357]: 1.1 scrub starts 2024-09-10T18:40:57.497 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:57 smithi204 ceph-mon[32357]: 1.1 scrub ok 2024-09-10T18:40:57.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:57 smithi028 ceph-mon[29291]: 3.1a scrub starts 2024-09-10T18:40:57.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:57 smithi028 ceph-mon[29291]: 3.1a scrub ok 2024-09-10T18:40:57.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:57 smithi028 ceph-mon[29291]: 3.13 scrub starts 2024-09-10T18:40:57.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:57 smithi028 ceph-mon[29291]: 3.13 scrub ok 2024-09-10T18:40:57.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:57 smithi028 ceph-mon[29291]: 1.1 scrub starts 2024-09-10T18:40:57.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:57 smithi028 ceph-mon[29291]: 1.1 scrub ok 2024-09-10T18:40:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:57 smithi028 ceph-mon[39220]: 3.1a scrub starts 2024-09-10T18:40:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:57 smithi028 ceph-mon[39220]: 3.1a scrub ok 2024-09-10T18:40:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:57 smithi028 ceph-mon[39220]: 3.13 scrub starts 2024-09-10T18:40:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:57 smithi028 ceph-mon[39220]: 3.13 scrub ok 2024-09-10T18:40:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:57 smithi028 ceph-mon[39220]: 1.1 scrub starts 2024-09-10T18:40:57.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:57 smithi028 ceph-mon[39220]: 1.1 scrub ok 2024-09-10T18:40:57.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:57.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:57.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:57.625 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:57.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:57.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:57.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:57.804 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:57.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:57.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:57.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:57.930 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:40:57.930 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:40:57.945 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:57.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:58.047 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:58.086 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:58.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:58.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:58.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:58.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:58.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:58.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:58.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:58 smithi028 ceph-mon[29291]: pgmap v879: 178 pgs: 178 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-10T18:40:58.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:58 smithi028 ceph-mon[29291]: 6.1d scrub starts 2024-09-10T18:40:58.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:58 smithi028 ceph-mon[29291]: 6.1d scrub ok 2024-09-10T18:40:58.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:58 smithi028 ceph-mon[39220]: pgmap v879: 178 pgs: 178 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-10T18:40:58.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:58 smithi028 ceph-mon[39220]: 6.1d scrub starts 2024-09-10T18:40:58.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:58 smithi028 ceph-mon[39220]: 6.1d scrub ok 2024-09-10T18:40:58.324 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 41 16 74702 74686 113.825 94.4375 0.00651488 0.00878117 2024-09-10T18:40:58.335 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:40:58.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:58 smithi204 ceph-mon[32357]: pgmap v879: 178 pgs: 178 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-10T18:40:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:58 smithi204 ceph-mon[32357]: 6.1d scrub starts 2024-09-10T18:40:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:58 smithi204 ceph-mon[32357]: 6.1d scrub ok 2024-09-10T18:40:58.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:58.375 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:58.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:40:58.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:58.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:58.516 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:58.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:58.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:58.655 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:58.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:58.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:58.757 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:58.791 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:58.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:58.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:58.891 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:58.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:40:58.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:58.945 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:58.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:40:59.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:59.087 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:59.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:59.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:59.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:40:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:59.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:59.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:40:59.324 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 42 16 76190 76174 113.329 93 0.00766358 0.0088186 2024-09-10T18:40:59.334 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:59.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:59 smithi204 ceph-mon[32357]: 3.4 scrub starts 2024-09-10T18:40:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:40:59 smithi204 ceph-mon[32357]: 3.4 scrub ok 2024-09-10T18:40:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:59.368 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:59.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:59.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:40:59.485 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:59 smithi028 ceph-mon[39220]: 3.4 scrub starts 2024-09-10T18:40:59.485 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:40:59 smithi028 ceph-mon[39220]: 3.4 scrub ok 2024-09-10T18:40:59.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:59 smithi028 ceph-mon[29291]: 3.4 scrub starts 2024-09-10T18:40:59.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:40:59 smithi028 ceph-mon[29291]: 3.4 scrub ok 2024-09-10T18:40:59.507 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:59.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:59.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:40:59.513 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-10T18:40:59.611 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:40:59.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:59.657 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:40:59.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:40:59.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:40:59.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:40:59.817 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:59.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:59.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:40:59.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:40:59.977 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:40:59.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:40:59.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:40:59.978 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:41:00.079 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:00.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:00.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:00.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:00.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:00.259 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:00.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:00.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:00.324 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 43 16 77476 77460 112.562 80.375 0.0109395 0.00887887 2024-09-10T18:41:00.362 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:00.395 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:00.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:00.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:00.437 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:00 smithi204 ceph-mon[32357]: pgmap v880: 178 pgs: 178 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s rd, 61 MiB/s wr, 969 op/s 2024-09-10T18:41:00.437 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:00 smithi204 ceph-mon[32357]: 5.2 scrub starts 2024-09-10T18:41:00.438 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:00 smithi204 ceph-mon[32357]: 5.2 scrub ok 2024-09-10T18:41:00.438 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:00 smithi204 ceph-mon[32357]: 3.b deep-scrub starts 2024-09-10T18:41:00.438 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:00 smithi204 ceph-mon[32357]: 3.b deep-scrub ok 2024-09-10T18:41:00.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:00.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:00 smithi028 ceph-mon[29291]: pgmap v880: 178 pgs: 178 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s rd, 61 MiB/s wr, 969 op/s 2024-09-10T18:41:00.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:00 smithi028 ceph-mon[29291]: 5.2 scrub starts 2024-09-10T18:41:00.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:00 smithi028 ceph-mon[29291]: 5.2 scrub ok 2024-09-10T18:41:00.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:00 smithi028 ceph-mon[29291]: 3.b deep-scrub starts 2024-09-10T18:41:00.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:00 smithi028 ceph-mon[29291]: 3.b deep-scrub ok 2024-09-10T18:41:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:00 smithi028 ceph-mon[39220]: pgmap v880: 178 pgs: 178 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s rd, 61 MiB/s wr, 969 op/s 2024-09-10T18:41:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:00 smithi028 ceph-mon[39220]: 5.2 scrub starts 2024-09-10T18:41:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:00 smithi028 ceph-mon[39220]: 5.2 scrub ok 2024-09-10T18:41:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:00 smithi028 ceph-mon[39220]: 3.b deep-scrub starts 2024-09-10T18:41:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:00 smithi028 ceph-mon[39220]: 3.b deep-scrub ok 2024-09-10T18:41:00.546 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:00.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:00.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:00.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:00.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:00.683 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:00.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:00.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:00.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:00.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:00.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:00.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:00.977 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:00.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:00.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:01.079 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:01.121 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:01.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:01.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:01.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:01.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:01.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:01.277 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:01 smithi204 ceph-mon[32357]: 1.e scrub starts 2024-09-10T18:41:01.277 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:01 smithi204 ceph-mon[32357]: 1.e scrub ok 2024-09-10T18:41:01.277 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:01 smithi204 ceph-mon[32357]: 3.1d scrub starts 2024-09-10T18:41:01.277 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:01 smithi204 ceph-mon[32357]: 3.1d scrub ok 2024-09-10T18:41:01.277 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:01 smithi204 ceph-mon[32357]: 4.12 scrub starts 2024-09-10T18:41:01.277 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:01 smithi204 ceph-mon[32357]: 4.12 scrub ok 2024-09-10T18:41:01.277 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:01 smithi204 ceph-mon[32357]: 1.1b scrub starts 2024-09-10T18:41:01.277 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:01 smithi204 ceph-mon[32357]: 1.1b scrub ok 2024-09-10T18:41:01.277 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:01 smithi204 ceph-mon[32357]: osdmap e388: 8 total, 8 up, 7 in 2024-09-10T18:41:01.277 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:01 smithi204 ceph-mon[32357]: 4.1c scrub starts 2024-09-10T18:41:01.277 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:01 smithi204 ceph-mon[32357]: 4.1c scrub ok 2024-09-10T18:41:01.325 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 44 16 78895 78879 112.019 88.6875 0.00728038 0.00892212 2024-09-10T18:41:01.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:01.412 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:01.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:01.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:01.503 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:41:01] "GET /metrics HTTP/1.1" 200 224876 "" "Prometheus/2.43.0" 2024-09-10T18:41:01.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[29291]: 1.e scrub starts 2024-09-10T18:41:01.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[29291]: 1.e scrub ok 2024-09-10T18:41:01.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[29291]: 3.1d scrub starts 2024-09-10T18:41:01.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[29291]: 3.1d scrub ok 2024-09-10T18:41:01.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[29291]: 4.12 scrub starts 2024-09-10T18:41:01.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[29291]: 4.12 scrub ok 2024-09-10T18:41:01.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[29291]: 1.1b scrub starts 2024-09-10T18:41:01.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[29291]: 1.1b scrub ok 2024-09-10T18:41:01.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[29291]: osdmap e388: 8 total, 8 up, 7 in 2024-09-10T18:41:01.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[29291]: 4.1c scrub starts 2024-09-10T18:41:01.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[29291]: 4.1c scrub ok 2024-09-10T18:41:01.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[39220]: 1.e scrub starts 2024-09-10T18:41:01.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[39220]: 1.e scrub ok 2024-09-10T18:41:01.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[39220]: 3.1d scrub starts 2024-09-10T18:41:01.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[39220]: 3.1d scrub ok 2024-09-10T18:41:01.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[39220]: 4.12 scrub starts 2024-09-10T18:41:01.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[39220]: 4.12 scrub ok 2024-09-10T18:41:01.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[39220]: 1.1b scrub starts 2024-09-10T18:41:01.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[39220]: 1.1b scrub ok 2024-09-10T18:41:01.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[39220]: osdmap e388: 8 total, 8 up, 7 in 2024-09-10T18:41:01.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[39220]: 4.1c scrub starts 2024-09-10T18:41:01.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:01 smithi028 ceph-mon[39220]: 4.1c scrub ok 2024-09-10T18:41:01.514 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:01.544 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:01.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:01.646 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:01.673 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:01.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:01.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:01.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-10T18:41:01.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:01.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:01.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:01.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:01.964 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:01.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:01.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:02.000 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-10T18:41:02.000 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:41:02.066 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:02.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:02.105 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:02.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:02.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:02.249 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:02.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:02.297 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:02 smithi028 ceph-mon[39220]: 1.1a scrub starts 2024-09-10T18:41:02.297 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:02 smithi028 ceph-mon[39220]: 1.1a scrub ok 2024-09-10T18:41:02.297 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:02 smithi028 ceph-mon[39220]: pgmap v882: 178 pgs: 178 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-10T18:41:02.297 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:02 smithi028 ceph-mon[39220]: 4.7 deep-scrub starts 2024-09-10T18:41:02.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:02 smithi028 ceph-mon[39220]: 4.7 deep-scrub ok 2024-09-10T18:41:02.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:02 smithi028 ceph-mon[29291]: 1.1a scrub starts 2024-09-10T18:41:02.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:02 smithi028 ceph-mon[29291]: 1.1a scrub ok 2024-09-10T18:41:02.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:02 smithi028 ceph-mon[29291]: pgmap v882: 178 pgs: 178 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-10T18:41:02.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:02 smithi028 ceph-mon[29291]: 4.7 deep-scrub starts 2024-09-10T18:41:02.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:02 smithi028 ceph-mon[29291]: 4.7 deep-scrub ok 2024-09-10T18:41:02.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 45 16 80350 80334 111.512 90.9375 0.0164607 0.00896228 2024-09-10T18:41:02.350 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:02.383 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:02.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:02.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:02.387 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:02 smithi204 ceph-mon[32357]: 1.1a scrub starts 2024-09-10T18:41:02.387 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:02 smithi204 ceph-mon[32357]: 1.1a scrub ok 2024-09-10T18:41:02.387 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:02 smithi204 ceph-mon[32357]: pgmap v882: 178 pgs: 178 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-10T18:41:02.387 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:02 smithi204 ceph-mon[32357]: 4.7 deep-scrub starts 2024-09-10T18:41:02.387 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:02 smithi204 ceph-mon[32357]: 4.7 deep-scrub ok 2024-09-10T18:41:02.387 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:41:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:41:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:41:02.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:02.528 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:02.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:02.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:02.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:02.669 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:02.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:02.750 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-10T18:41:02.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:02.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:02.829 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:02.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:02.930 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:02.970 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:02.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:02.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:03.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:03.111 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:03.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:03.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:03.151 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:41:03.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:03.218 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:41:03.260 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:03.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:03.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:03.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 46 16 81766 81750 111.012 88.5 0.00824848 0.00900333 2024-09-10T18:41:03.362 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:03.399 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:03.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:03.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:03.414 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:03 smithi204 ceph-mon[32357]: 5.1 scrub starts 2024-09-10T18:41:03.414 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:03 smithi204 ceph-mon[32357]: 5.1 scrub ok 2024-09-10T18:41:03.414 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:03 smithi204 ceph-mon[32357]: 4.15 scrub starts 2024-09-10T18:41:03.414 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:03 smithi204 ceph-mon[32357]: 4.15 scrub ok 2024-09-10T18:41:03.414 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:03 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4146612465' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:41:03.414 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:03 smithi204 ceph-mon[32357]: from='client.35921 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:41:03.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:03.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:03 smithi028 ceph-mon[29291]: 5.1 scrub starts 2024-09-10T18:41:03.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:03 smithi028 ceph-mon[29291]: 5.1 scrub ok 2024-09-10T18:41:03.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:03 smithi028 ceph-mon[29291]: 4.15 scrub starts 2024-09-10T18:41:03.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:03 smithi028 ceph-mon[29291]: 4.15 scrub ok 2024-09-10T18:41:03.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:03 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4146612465' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:41:03.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:03 smithi028 ceph-mon[29291]: from='client.35921 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:41:03.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:03 smithi028 ceph-mon[39220]: 5.1 scrub starts 2024-09-10T18:41:03.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:03 smithi028 ceph-mon[39220]: 5.1 scrub ok 2024-09-10T18:41:03.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:03 smithi028 ceph-mon[39220]: 4.15 scrub starts 2024-09-10T18:41:03.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:03 smithi028 ceph-mon[39220]: 4.15 scrub ok 2024-09-10T18:41:03.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:03 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4146612465' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:41:03.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:03 smithi028 ceph-mon[39220]: from='client.35921 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:41:03.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:03.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:03.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:03.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:03.680 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:03.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:03.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:03.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:03.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:03.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:03.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:03.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:03.971 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:03.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:04.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:04.116 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:04.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:04.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:04.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:04.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:04.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:04.341 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 47 16 83245 83229 110.617 92.4375 0.00893421 0.00903525 2024-09-10T18:41:04.362 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:04.399 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:04.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:04.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:04.421 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:04 smithi204 ceph-mon[32357]: 4.18 scrub starts 2024-09-10T18:41:04.422 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:04 smithi204 ceph-mon[32357]: 4.18 scrub ok 2024-09-10T18:41:04.422 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:04 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:41:04.422 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:04 smithi204 ceph-mon[32357]: from='client.35921 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:41:04.422 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:04 smithi204 ceph-mon[32357]: osdmap e389: 8 total, 8 up, 7 in 2024-09-10T18:41:04.422 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:04 smithi204 ceph-mon[32357]: pgmap v884: 178 pgs: 178 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 60 MiB/s wr, 952 op/s 2024-09-10T18:41:04.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:04 smithi028 ceph-mon[29291]: 4.18 scrub starts 2024-09-10T18:41:04.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:04 smithi028 ceph-mon[29291]: 4.18 scrub ok 2024-09-10T18:41:04.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:04 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:41:04.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:04 smithi028 ceph-mon[29291]: from='client.35921 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:41:04.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:04 smithi028 ceph-mon[29291]: osdmap e389: 8 total, 8 up, 7 in 2024-09-10T18:41:04.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:04 smithi028 ceph-mon[29291]: pgmap v884: 178 pgs: 178 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 60 MiB/s wr, 952 op/s 2024-09-10T18:41:04.481 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:04 smithi028 ceph-mon[39220]: 4.18 scrub starts 2024-09-10T18:41:04.481 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:04 smithi028 ceph-mon[39220]: 4.18 scrub ok 2024-09-10T18:41:04.481 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:04 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:41:04.481 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:04 smithi028 ceph-mon[39220]: from='client.35921 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:41:04.481 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:04 smithi028 ceph-mon[39220]: osdmap e389: 8 total, 8 up, 7 in 2024-09-10T18:41:04.481 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:04 smithi028 ceph-mon[39220]: pgmap v884: 178 pgs: 178 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 60 MiB/s wr, 952 op/s 2024-09-10T18:41:04.501 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:04.534 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:04.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:04.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:04.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:04.668 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:04.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:04.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:04.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:04.795 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:04.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:04.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:04.926 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:04.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:04.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:05.028 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:05.059 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:05.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:05.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:05.130 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:41:05.130 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:41:05.160 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:05.166 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:41:05.166 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:41:05.192 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:05.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:05.199 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:41:05.200 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:41:05.233 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:41:05.233 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:41:05.268 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:41:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:41:05.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:05.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:05 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2501451888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:41:05.307 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:41:05.308 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:41:05.308 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:05 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2501451888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:41:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:05.341 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 48 16 84363 84347 109.768 69.875 0.0135719 0.00910456 2024-09-10T18:41:05.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:05.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:05.344 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:41:05.344 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:41:05.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:05 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2501451888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:41:05.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:41:05 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:41:05.377 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:41:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:41:05.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:05.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:05.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:05.583 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:05.619 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:05.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:05.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:05.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:05.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:05.859 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:05.891 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:05.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:05.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:05.993 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:06.025 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:06.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:06.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:06.127 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:06.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:06.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:06.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:06.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:06.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:06.291 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:06.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:06.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 49 16 85684 85668 109.212 82.5625 0.0128556 0.00915161 2024-09-10T18:41:06.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:06 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:06 smithi204 ceph-mon[32357]: pgmap v885: 178 pgs: 178 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-09-10T18:41:06.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:06 smithi204 ceph-mon[32357]: osdmap e390: 8 total, 8 up, 7 in 2024-09-10T18:41:06.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:06.415 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:06.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:06.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:06 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:06 smithi028 ceph-mon[39220]: pgmap v885: 178 pgs: 178 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-09-10T18:41:06.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:06 smithi028 ceph-mon[39220]: osdmap e390: 8 total, 8 up, 7 in 2024-09-10T18:41:06.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:06 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:06.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:06 smithi028 ceph-mon[29291]: pgmap v885: 178 pgs: 178 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-09-10T18:41:06.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:06 smithi028 ceph-mon[29291]: osdmap e390: 8 total, 8 up, 7 in 2024-09-10T18:41:06.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:06.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:41:06.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:06.554 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:06.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:06.655 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:06.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:06.688 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:06.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:06.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:06.852 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:06.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:06.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:06.954 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:06.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:06.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:07.047 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:41:07.047 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-10T18:41:07.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-10T18:41:07.047 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:41:07.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.047 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:115963 9.e 9:7cc8a211:::benchmark_data_smithi028_135703_object115962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-10T18:41:07.047 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:33:26.098536+0000", 2024-09-10T18:41:07.047 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 460.944178464, 2024-09-10T18:41:07.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.22854117500000001, 2024-09-10T18:41:07.047 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.048 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:07.048 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:07.048 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:41:07.048 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:41:07.048 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 115963 2024-09-10T18:41:07.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.048 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098536+0000", 2024-09-10T18:41:07.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098536+0000", 2024-09-10T18:41:07.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098533+0000", 2024-09-10T18:41:07.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:41:07.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098572+0000", 2024-09-10T18:41:07.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8804999999999999e-05 2024-09-10T18:41:07.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098585+0000", 2024-09-10T18:41:07.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2830000000000001e-05 2024-09-10T18:41:07.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098591+0000", 2024-09-10T18:41:07.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0469999999999999e-06 2024-09-10T18:41:07.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098688+0000", 2024-09-10T18:41:07.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7551000000000002e-05 2024-09-10T18:41:07.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098826+0000", 2024-09-10T18:41:07.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013742999999999999 2024-09-10T18:41:07.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:41:07.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098987+0000", 2024-09-10T18:41:07.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016180599999999999 2024-09-10T18:41:07.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:07.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105991+0000", 2024-09-10T18:41:07.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0070032649999999998 2024-09-10T18:41:07.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.052 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.052 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.106028+0000", 2024-09-10T18:41:07.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7605000000000003e-05 2024-09-10T18:41:07.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.052 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.052 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.326977+0000", 2024-09-10T18:41:07.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.22094904500000001 2024-09-10T18:41:07.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:07.053 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327031+0000", 2024-09-10T18:41:07.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3706999999999999e-05 2024-09-10T18:41:07.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.053 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327077+0000", 2024-09-10T18:41:07.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5705999999999999e-05 2024-09-10T18:41:07.053 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.053 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.053 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.053 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:115964 9.e 9:7700e295:::benchmark_data_smithi028_135703_object115963:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-10T18:41:07.054 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:33:26.098607+0000", 2024-09-10T18:41:07.054 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 460.94410709599998, 2024-09-10T18:41:07.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.22859473999999999, 2024-09-10T18:41:07.054 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.054 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:07.054 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:07.054 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:41:07.054 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:41:07.054 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 115964 2024-09-10T18:41:07.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.054 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098607+0000", 2024-09-10T18:41:07.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098607+0000", 2024-09-10T18:41:07.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098606+0000", 2024-09-10T18:41:07.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:41:07.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098640+0000", 2024-09-10T18:41:07.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4183000000000001e-05 2024-09-10T18:41:07.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098647+0000", 2024-09-10T18:41:07.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6880000000000002e-06 2024-09-10T18:41:07.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.098651+0000", 2024-09-10T18:41:07.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.98e-06 2024-09-10T18:41:07.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.103373+0000", 2024-09-10T18:41:07.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047225339999999996 2024-09-10T18:41:07.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.103468+0000", 2024-09-10T18:41:07.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4591000000000003e-05 2024-09-10T18:41:07.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:41:07.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.103558+0000", 2024-09-10T18:41:07.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0408000000000006e-05 2024-09-10T18:41:07.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.106663+0000", 2024-09-10T18:41:07.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003104557 2024-09-10T18:41:07.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:07.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.106735+0000", 2024-09-10T18:41:07.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2008999999999994e-05 2024-09-10T18:41:07.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327116+0000", 2024-09-10T18:41:07.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.220380839 2024-09-10T18:41:07.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:07.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327169+0000", 2024-09-10T18:41:07.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3189000000000002e-05 2024-09-10T18:41:07.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327202+0000", 2024-09-10T18:41:07.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2540999999999999e-05 2024-09-10T18:41:07.059 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.059 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.059 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.059 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:115965 9.d 9:be537d29:::benchmark_data_smithi028_135703_object115964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-10T18:41:07.060 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:33:26.099655+0000", 2024-09-10T18:41:07.060 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 460.943059414, 2024-09-10T18:41:07.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.22745093699999999, 2024-09-10T18:41:07.060 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.060 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:07.060 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:07.060 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:41:07.061 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:41:07.061 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 115965 2024-09-10T18:41:07.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.061 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.099655+0000", 2024-09-10T18:41:07.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.099655+0000", 2024-09-10T18:41:07.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.099653+0000", 2024-09-10T18:41:07.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:41:07.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.099686+0000", 2024-09-10T18:41:07.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3408000000000003e-05 2024-09-10T18:41:07.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.099700+0000", 2024-09-10T18:41:07.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4396000000000001e-05 2024-09-10T18:41:07.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.099707+0000", 2024-09-10T18:41:07.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5459999999999997e-06 2024-09-10T18:41:07.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.099853+0000", 2024-09-10T18:41:07.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014599900000000001 2024-09-10T18:41:07.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.099979+0000", 2024-09-10T18:41:07.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012636099999999999 2024-09-10T18:41:07.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,5", 2024-09-10T18:41:07.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.100107+0000", 2024-09-10T18:41:07.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012751700000000001 2024-09-10T18:41:07.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:07.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.102820+0000", 2024-09-10T18:41:07.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027130349999999999 2024-09-10T18:41:07.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.109028+0000", 2024-09-10T18:41:07.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0062082170000000002 2024-09-10T18:41:07.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327021+0000", 2024-09-10T18:41:07.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.217992985 2024-09-10T18:41:07.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:07.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327068+0000", 2024-09-10T18:41:07.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6587e-05 2024-09-10T18:41:07.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327105+0000", 2024-09-10T18:41:07.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7750000000000003e-05 2024-09-10T18:41:07.066 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.066 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.066 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.066 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:115969 9.d 9:ba9f0da3:::benchmark_data_smithi028_135703_object115968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-10T18:41:07.067 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:33:26.105312+0000", 2024-09-10T18:41:07.067 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 460.93740202599997, 2024-09-10T18:41:07.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.22189908699999999, 2024-09-10T18:41:07.067 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.067 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:07.067 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:07.067 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:41:07.067 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:41:07.067 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 115969 2024-09-10T18:41:07.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.067 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105312+0000", 2024-09-10T18:41:07.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105312+0000", 2024-09-10T18:41:07.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105309+0000", 2024-09-10T18:41:07.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:41:07.069 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.069 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.069 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.069 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105367+0000", 2024-09-10T18:41:07.069 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7657e-05 2024-09-10T18:41:07.069 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.069 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.069 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.069 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105404+0000", 2024-09-10T18:41:07.069 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6470999999999997e-05 2024-09-10T18:41:07.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105415+0000", 2024-09-10T18:41:07.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1557e-05 2024-09-10T18:41:07.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105478+0000", 2024-09-10T18:41:07.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3323000000000003e-05 2024-09-10T18:41:07.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105614+0000", 2024-09-10T18:41:07.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000135978 2024-09-10T18:41:07.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,5", 2024-09-10T18:41:07.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105771+0000", 2024-09-10T18:41:07.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015666599999999999 2024-09-10T18:41:07.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:07.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.108661+0000", 2024-09-10T18:41:07.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028897369999999999 2024-09-10T18:41:07.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.109245+0000", 2024-09-10T18:41:07.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00058383000000000003 2024-09-10T18:41:07.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327149+0000", 2024-09-10T18:41:07.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21790462799999999 2024-09-10T18:41:07.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:07.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327180+0000", 2024-09-10T18:41:07.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0735999999999998e-05 2024-09-10T18:41:07.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327211+0000", 2024-09-10T18:41:07.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1038999999999999e-05 2024-09-10T18:41:07.073 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.073 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.073 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.073 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:115971 9.d 9:be0cedf3:::benchmark_data_smithi028_135703_object115970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-10T18:41:07.073 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:33:26.105428+0000", 2024-09-10T18:41:07.073 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 460.93728625400001, 2024-09-10T18:41:07.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.22190970199999999, 2024-09-10T18:41:07.073 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.073 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:07.073 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:07.074 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:41:07.074 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:41:07.074 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 115971 2024-09-10T18:41:07.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.074 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105428+0000", 2024-09-10T18:41:07.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105428+0000", 2024-09-10T18:41:07.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.075 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.075 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.105427+0000", 2024-09-10T18:41:07.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:41:07.075 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.075 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.075 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.120120+0000", 2024-09-10T18:41:07.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014693513 2024-09-10T18:41:07.075 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.075 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.075 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.120136+0000", 2024-09-10T18:41:07.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5260999999999998e-05 2024-09-10T18:41:07.076 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.076 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.076 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.076 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.120142+0000", 2024-09-10T18:41:07.076 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4609999999999996e-06 2024-09-10T18:41:07.076 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.076 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.076 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.076 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.120186+0000", 2024-09-10T18:41:07.076 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3498000000000003e-05 2024-09-10T18:41:07.076 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.076 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.076 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.076 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.120304+0000", 2024-09-10T18:41:07.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000118449 2024-09-10T18:41:07.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,5", 2024-09-10T18:41:07.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.120446+0000", 2024-09-10T18:41:07.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014222200000000001 2024-09-10T18:41:07.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:07.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.123629+0000", 2024-09-10T18:41:07.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0031830719999999999 2024-09-10T18:41:07.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.123985+0000", 2024-09-10T18:41:07.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00035541499999999998 2024-09-10T18:41:07.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327254+0000", 2024-09-10T18:41:07.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.20326886 2024-09-10T18:41:07.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:07.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327300+0000", 2024-09-10T18:41:07.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6532000000000001e-05 2024-09-10T18:41:07.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:33:26.327337+0000", 2024-09-10T18:41:07.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7285e-05 2024-09-10T18:41:07.079 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.079 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.079 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.079 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:191077 9.e 9:74bb5a96:::benchmark_data_smithi028_135703_object56486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-10T18:41:07.079 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:31.102004+0000", 2024-09-10T18:41:07.079 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 395.94071018300002, 2024-09-10T18:41:07.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13146555300000001, 2024-09-10T18:41:07.079 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.080 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:07.080 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:07.080 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:41:07.080 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:41:07.080 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 191077 2024-09-10T18:41:07.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.080 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102004+0000", 2024-09-10T18:41:07.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.081 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102004+0000", 2024-09-10T18:41:07.081 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.081 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.081 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.081 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.081 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102003+0000", 2024-09-10T18:41:07.081 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:41:07.081 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.081 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.081 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.081 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102008+0000", 2024-09-10T18:41:07.081 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7439999999999998e-06 2024-09-10T18:41:07.081 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.081 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.082 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.082 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102014+0000", 2024-09-10T18:41:07.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.878e-06 2024-09-10T18:41:07.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.082 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.082 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102019+0000", 2024-09-10T18:41:07.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3560000000000003e-06 2024-09-10T18:41:07.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.082 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.082 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.103206+0000", 2024-09-10T18:41:07.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001187378 2024-09-10T18:41:07.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.103309+0000", 2024-09-10T18:41:07.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103371 2024-09-10T18:41:07.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:41:07.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.103406+0000", 2024-09-10T18:41:07.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6100000000000005e-05 2024-09-10T18:41:07.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:07.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.107609+0000", 2024-09-10T18:41:07.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042029850000000002 2024-09-10T18:41:07.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.107735+0000", 2024-09-10T18:41:07.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012641700000000001 2024-09-10T18:41:07.084 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.233388+0000", 2024-09-10T18:41:07.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.12565284400000001 2024-09-10T18:41:07.084 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.085 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:07.085 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.233434+0000", 2024-09-10T18:41:07.085 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5844999999999999e-05 2024-09-10T18:41:07.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.085 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.085 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.233469+0000", 2024-09-10T18:41:07.085 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5663000000000001e-05 2024-09-10T18:41:07.085 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.085 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.085 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.085 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:191078 9.e 9:724867a8:::benchmark_data_smithi028_135703_object56487:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-10T18:41:07.086 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:31.102030+0000", 2024-09-10T18:41:07.086 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 395.94068358999999, 2024-09-10T18:41:07.086 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.131586658, 2024-09-10T18:41:07.086 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.086 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:07.086 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:07.086 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:41:07.086 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:41:07.086 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 191078 2024-09-10T18:41:07.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.086 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.087 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.087 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102030+0000", 2024-09-10T18:41:07.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.087 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.087 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.087 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102030+0000", 2024-09-10T18:41:07.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.087 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102030+0000", 2024-09-10T18:41:07.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:41:07.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102033+0000", 2024-09-10T18:41:07.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.41e-06 2024-09-10T18:41:07.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102039+0000", 2024-09-10T18:41:07.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9499999999999998e-06 2024-09-10T18:41:07.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.102043+0000", 2024-09-10T18:41:07.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8890000000000004e-06 2024-09-10T18:41:07.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.105506+0000", 2024-09-10T18:41:07.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034628829999999999 2024-09-10T18:41:07.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.105611+0000", 2024-09-10T18:41:07.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010552099999999999 2024-09-10T18:41:07.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:41:07.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.105719+0000", 2024-09-10T18:41:07.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010804200000000001 2024-09-10T18:41:07.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:07.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.107833+0000", 2024-09-10T18:41:07.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021132019999999998 2024-09-10T18:41:07.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.108602+0000", 2024-09-10T18:41:07.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00076962999999999999 2024-09-10T18:41:07.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.233511+0000", 2024-09-10T18:41:07.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.12490902800000001 2024-09-10T18:41:07.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:07.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.233581+0000", 2024-09-10T18:41:07.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0279999999999998e-05 2024-09-10T18:41:07.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:31.233617+0000", 2024-09-10T18:41:07.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5568999999999998e-05 2024-09-10T18:41:07.092 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.092 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.092 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.092 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.35306.0:193358 9.3 9:c47564e4:::benchmark_data_smithi028_135703_object58767:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-10T18:41:07.092 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:34:33.992785+0000", 2024-09-10T18:41:07.093 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 393.04992937600002, 2024-09-10T18:41:07.093 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13214885500000001, 2024-09-10T18:41:07.093 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.093 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:07.093 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:07.093 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.35306", 2024-09-10T18:41:07.093 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/11727973", 2024-09-10T18:41:07.093 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 193358 2024-09-10T18:41:07.093 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.093 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.093 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.093 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.992785+0000", 2024-09-10T18:41:07.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.992785+0000", 2024-09-10T18:41:07.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.992783+0000", 2024-09-10T18:41:07.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:41:07.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.992790+0000", 2024-09-10T18:41:07.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9979999999999997e-06 2024-09-10T18:41:07.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.992803+0000", 2024-09-10T18:41:07.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3283e-05 2024-09-10T18:41:07.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.992809+0000", 2024-09-10T18:41:07.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1959999999999996e-06 2024-09-10T18:41:07.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.992864+0000", 2024-09-10T18:41:07.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4421e-05 2024-09-10T18:41:07.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.992998+0000", 2024-09-10T18:41:07.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000134449 2024-09-10T18:41:07.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:41:07.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.993121+0000", 2024-09-10T18:41:07.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012321899999999999 2024-09-10T18:41:07.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:07.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.995481+0000", 2024-09-10T18:41:07.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023597340000000001 2024-09-10T18:41:07.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:33.996352+0000", 2024-09-10T18:41:07.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00087137300000000005 2024-09-10T18:41:07.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.124849+0000", 2024-09-10T18:41:07.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.12849706699999999 2024-09-10T18:41:07.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:07.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.124899+0000", 2024-09-10T18:41:07.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9917999999999999e-05 2024-09-10T18:41:07.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:34:34.124933+0000", 2024-09-10T18:41:07.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4146000000000001e-05 2024-09-10T18:41:07.098 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.098 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.098 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.099 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:125391 10.c 10:3a369ca4:::benchmark_data_smithi028_161947_object125390:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-09-10T18:41:07.099 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:37:38.683900+0000", 2024-09-10T18:41:07.099 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 208.35881442199999, 2024-09-10T18:41:07.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13223570300000001, 2024-09-10T18:41:07.099 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.099 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:07.099 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:07.099 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:07.099 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:07.099 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 125391 2024-09-10T18:41:07.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.099 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:38.683900+0000", 2024-09-10T18:41:07.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:38.683900+0000", 2024-09-10T18:41:07.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:38.683898+0000", 2024-09-10T18:41:07.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:41:07.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:38.683928+0000", 2024-09-10T18:41:07.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0168e-05 2024-09-10T18:41:07.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:38.683939+0000", 2024-09-10T18:41:07.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0529e-05 2024-09-10T18:41:07.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:38.683944+0000", 2024-09-10T18:41:07.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6210000000000001e-06 2024-09-10T18:41:07.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:38.686065+0000", 2024-09-10T18:41:07.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002121043 2024-09-10T18:41:07.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:38.686186+0000", 2024-09-10T18:41:07.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012036600000000001 2024-09-10T18:41:07.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:41:07.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:38.686324+0000", 2024-09-10T18:41:07.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000138403 2024-09-10T18:41:07.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:07.103 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:07.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:38.689243+0000", 2024-09-10T18:41:07.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029194170000000001 2024-09-10T18:41:07.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:38.690463+0000", 2024-09-10T18:41:07.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001219377 2024-09-10T18:41:07.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:38.816078+0000", 2024-09-10T18:41:07.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.12561528399999999 2024-09-10T18:41:07.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:07.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:38.816111+0000", 2024-09-10T18:41:07.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2673999999999999e-05 2024-09-10T18:41:07.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:37:38.816135+0000", 2024-09-10T18:41:07.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4420999999999999e-05 2024-09-10T18:41:07.106 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.106 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.106 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.106 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14532.0:1059 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e360)", 2024-09-10T18:41:07.106 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.270981+0000", 2024-09-10T18:41:07.106 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 102.771733184, 2024-09-10T18:41:07.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.975137225, 2024-09-10T18:41:07.106 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.106 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:41:07.107 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:07.107 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14532", 2024-09-10T18:41:07.107 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.204:0/1733125751", 2024-09-10T18:41:07.107 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1059 2024-09-10T18:41:07.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.107 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.107 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.107 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.270981+0000", 2024-09-10T18:41:07.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.107 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.107 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.270981+0000", 2024-09-10T18:41:07.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.270978+0000", 2024-09-10T18:41:07.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:41:07.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.270987+0000", 2024-09-10T18:41:07.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1730000000000001e-06 2024-09-10T18:41:07.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.271000+0000", 2024-09-10T18:41:07.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2809999999999999e-05 2024-09-10T18:41:07.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.271006+0000", 2024-09-10T18:41:07.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7749999999999998e-06 2024-09-10T18:41:07.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.271076+0000", 2024-09-10T18:41:07.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9720000000000003e-05 2024-09-10T18:41:07.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:41:07.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.271083+0000", 2024-09-10T18:41:07.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4900000000000003e-06 2024-09-10T18:41:07.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.245829+0000", 2024-09-10T18:41:07.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97474618899999999 2024-09-10T18:41:07.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.245977+0000", 2024-09-10T18:41:07.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014791699999999999 2024-09-10T18:41:07.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.246118+0000", 2024-09-10T18:41:07.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014067999999999999 2024-09-10T18:41:07.111 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.111 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.111 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.112 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253976 10.c 10:31410340:::benchmark_data_smithi028_161947_object117052:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-10T18:41:07.112 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:26.832997+0000", 2024-09-10T18:41:07.112 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 100.20971714300001, 2024-09-10T18:41:07.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.34804471399999998, 2024-09-10T18:41:07.112 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.112 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:07.112 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:07.112 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:07.112 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:07.112 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253976 2024-09-10T18:41:07.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.112 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.832997+0000", 2024-09-10T18:41:07.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.832997+0000", 2024-09-10T18:41:07.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.832994+0000", 2024-09-10T18:41:07.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:41:07.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.833004+0000", 2024-09-10T18:41:07.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4439999999999997e-06 2024-09-10T18:41:07.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.833024+0000", 2024-09-10T18:41:07.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0647999999999999e-05 2024-09-10T18:41:07.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.833032+0000", 2024-09-10T18:41:07.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8660000000000006e-06 2024-09-10T18:41:07.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.833090+0000", 2024-09-10T18:41:07.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8102e-05 2024-09-10T18:41:07.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.833233+0000", 2024-09-10T18:41:07.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014301900000000001 2024-09-10T18:41:07.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3,4", 2024-09-10T18:41:07.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.833375+0000", 2024-09-10T18:41:07.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014154799999999999 2024-09-10T18:41:07.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.836637+0000", 2024-09-10T18:41:07.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032624249999999998 2024-09-10T18:41:07.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:07.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.838250+0000", 2024-09-10T18:41:07.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016129 2024-09-10T18:41:07.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.838798+0000", 2024-09-10T18:41:07.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000548304 2024-09-10T18:41:07.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:27.180820+0000", 2024-09-10T18:41:07.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.34202175699999998 2024-09-10T18:41:07.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:07.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:27.180936+0000", 2024-09-10T18:41:07.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000115751 2024-09-10T18:41:07.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:27.181042+0000", 2024-09-10T18:41:07.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105591 2024-09-10T18:41:07.118 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.118 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.118 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.118 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:257802 10.c 10:3c3928b0:::benchmark_data_smithi028_161947_object120878:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-10T18:41:07.118 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:30.976532+0000", 2024-09-10T18:41:07.118 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 96.066181831999998, 2024-09-10T18:41:07.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.18809384500000001, 2024-09-10T18:41:07.118 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.118 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:07.119 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:07.119 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:07.119 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:07.119 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 257802 2024-09-10T18:41:07.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.119 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.976532+0000", 2024-09-10T18:41:07.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.976532+0000", 2024-09-10T18:41:07.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.976529+0000", 2024-09-10T18:41:07.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:41:07.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.976539+0000", 2024-09-10T18:41:07.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2979999999999997e-06 2024-09-10T18:41:07.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.976556+0000", 2024-09-10T18:41:07.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7498000000000001e-05 2024-09-10T18:41:07.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.976563+0000", 2024-09-10T18:41:07.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2710000000000003e-06 2024-09-10T18:41:07.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.976614+0000", 2024-09-10T18:41:07.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0992000000000002e-05 2024-09-10T18:41:07.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.976734+0000", 2024-09-10T18:41:07.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119633 2024-09-10T18:41:07.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3,4", 2024-09-10T18:41:07.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.976852+0000", 2024-09-10T18:41:07.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011809199999999999 2024-09-10T18:41:07.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981545+0000", 2024-09-10T18:41:07.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004693076 2024-09-10T18:41:07.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164418+0000", 2024-09-10T18:41:07.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.18287271999999999 2024-09-10T18:41:07.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:07.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164429+0000", 2024-09-10T18:41:07.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0662000000000001e-05 2024-09-10T18:41:07.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164464+0000", 2024-09-10T18:41:07.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5716000000000002e-05 2024-09-10T18:41:07.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:07.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164514+0000", 2024-09-10T18:41:07.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9929000000000003e-05 2024-09-10T18:41:07.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164626+0000", 2024-09-10T18:41:07.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111693 2024-09-10T18:41:07.125 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.125 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.125 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.125 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:257806 10.c 10:3487f7a6:::benchmark_data_smithi028_161947_object120882:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-10T18:41:07.125 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:30.981296+0000", 2024-09-10T18:41:07.125 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 96.061418145999994, 2024-09-10T18:41:07.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.18370708299999999, 2024-09-10T18:41:07.126 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.126 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:07.126 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:07.126 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:07.126 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:07.126 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 257806 2024-09-10T18:41:07.126 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.126 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.126 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.126 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981296+0000", 2024-09-10T18:41:07.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.126 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.127 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981296+0000", 2024-09-10T18:41:07.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.127 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981294+0000", 2024-09-10T18:41:07.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:41:07.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.127 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981302+0000", 2024-09-10T18:41:07.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1999999999999994e-06 2024-09-10T18:41:07.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981314+0000", 2024-09-10T18:41:07.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2414e-05 2024-09-10T18:41:07.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981321+0000", 2024-09-10T18:41:07.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2360000000000001e-06 2024-09-10T18:41:07.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981578+0000", 2024-09-10T18:41:07.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00025759099999999999 2024-09-10T18:41:07.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981695+0000", 2024-09-10T18:41:07.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116445 2024-09-10T18:41:07.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3,4", 2024-09-10T18:41:07.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981812+0000", 2024-09-10T18:41:07.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000117277 2024-09-10T18:41:07.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:07.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164637+0000", 2024-09-10T18:41:07.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.182825077 2024-09-10T18:41:07.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164678+0000", 2024-09-10T18:41:07.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1238000000000001e-05 2024-09-10T18:41:07.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164740+0000", 2024-09-10T18:41:07.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1588999999999996e-05 2024-09-10T18:41:07.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164801+0000", 2024-09-10T18:41:07.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0905e-05 2024-09-10T18:41:07.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:07.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164886+0000", 2024-09-10T18:41:07.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5073000000000004e-05 2024-09-10T18:41:07.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.165003+0000", 2024-09-10T18:41:07.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000117201 2024-09-10T18:41:07.131 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.131 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.132 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.132 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:257808 10.c 10:368d1474:::benchmark_data_smithi028_161947_object120884:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-10T18:41:07.132 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:30.981328+0000", 2024-09-10T18:41:07.132 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 96.061386175999999, 2024-09-10T18:41:07.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.183957332, 2024-09-10T18:41:07.132 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.132 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:07.132 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:07.132 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:07.133 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:07.133 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 257808 2024-09-10T18:41:07.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.133 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981328+0000", 2024-09-10T18:41:07.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981328+0000", 2024-09-10T18:41:07.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981327+0000", 2024-09-10T18:41:07.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:41:07.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981330+0000", 2024-09-10T18:41:07.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4249999999999998e-06 2024-09-10T18:41:07.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981336+0000", 2024-09-10T18:41:07.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.998e-06 2024-09-10T18:41:07.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981341+0000", 2024-09-10T18:41:07.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8019999999999999e-06 2024-09-10T18:41:07.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.984066+0000", 2024-09-10T18:41:07.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002724326 2024-09-10T18:41:07.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.984182+0000", 2024-09-10T18:41:07.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116175 2024-09-10T18:41:07.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3,4", 2024-09-10T18:41:07.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.984308+0000", 2024-09-10T18:41:07.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125962 2024-09-10T18:41:07.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:07.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164643+0000", 2024-09-10T18:41:07.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.18033508700000001 2024-09-10T18:41:07.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164709+0000", 2024-09-10T18:41:07.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6027999999999999e-05 2024-09-10T18:41:07.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164771+0000", 2024-09-10T18:41:07.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2257999999999999e-05 2024-09-10T18:41:07.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.165046+0000", 2024-09-10T18:41:07.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000275009 2024-09-10T18:41:07.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:07.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.165098+0000", 2024-09-10T18:41:07.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1671e-05 2024-09-10T18:41:07.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.165285+0000", 2024-09-10T18:41:07.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018733899999999999 2024-09-10T18:41:07.138 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.138 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.139 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.139 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.15693.0:257802 10.c e363/362 ondisk, result = 0)", 2024-09-10T18:41:07.139 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:30.981897+0000", 2024-09-10T18:41:07.139 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 96.060816983999999, 2024-09-10T18:41:07.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.18253246100000001, 2024-09-10T18:41:07.139 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.139 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:41:07.140 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981897+0000", 2024-09-10T18:41:07.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.140 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981897+0000", 2024-09-10T18:41:07.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.140 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981895+0000", 2024-09-10T18:41:07.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:41:07.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981902+0000", 2024-09-10T18:41:07.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9800000000000001e-06 2024-09-10T18:41:07.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981913+0000", 2024-09-10T18:41:07.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0203e-05 2024-09-10T18:41:07.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.981918+0000", 2024-09-10T18:41:07.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7529999999999998e-06 2024-09-10T18:41:07.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164401+0000", 2024-09-10T18:41:07.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.18248287899999999 2024-09-10T18:41:07.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164410+0000", 2024-09-10T18:41:07.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7709999999999995e-06 2024-09-10T18:41:07.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164429+0000", 2024-09-10T18:41:07.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9454999999999999e-05 2024-09-10T18:41:07.143 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.143 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.143 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.143 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.15693.0:257807 10.1 e363/362 10:841c3075:::benchmark_data_smithi028_161947_object120883:head v 363'15959, mlcod=363'15957)", 2024-09-10T18:41:07.143 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:30.985083+0000", 2024-09-10T18:41:07.143 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 96.057631013999995, 2024-09-10T18:41:07.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.18272511, 2024-09-10T18:41:07.143 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.144 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:07.144 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.144 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.144 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.985083+0000", 2024-09-10T18:41:07.144 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.144 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.144 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.985083+0000", 2024-09-10T18:41:07.144 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.985081+0000", 2024-09-10T18:41:07.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:41:07.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.985089+0000", 2024-09-10T18:41:07.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8250000000000005e-06 2024-09-10T18:41:07.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.146 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.162383+0000", 2024-09-10T18:41:07.146 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.177293805 2024-09-10T18:41:07.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.146 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.146 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.146 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.162394+0000", 2024-09-10T18:41:07.146 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1321000000000001e-05 2024-09-10T18:41:07.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.146 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.146 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.162727+0000", 2024-09-10T18:41:07.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000333154 2024-09-10T18:41:07.147 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.162751+0000", 2024-09-10T18:41:07.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3416000000000001e-05 2024-09-10T18:41:07.147 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:07.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.167660+0000", 2024-09-10T18:41:07.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0049094029999999997 2024-09-10T18:41:07.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.148 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.148 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.167808+0000", 2024-09-10T18:41:07.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014801199999999999 2024-09-10T18:41:07.148 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.148 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.148 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.149 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.15693.0:257815 10.8 e363/362 10:1f5c2493:::benchmark_data_smithi028_161947_object120891:head v 363'16105, mlcod=363'16103)", 2024-09-10T18:41:07.149 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:30.986281+0000", 2024-09-10T18:41:07.149 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 96.056433337000001, 2024-09-10T18:41:07.149 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.18100276700000001, 2024-09-10T18:41:07.149 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.150 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:07.150 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.150 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.986281+0000", 2024-09-10T18:41:07.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.150 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.150 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.986281+0000", 2024-09-10T18:41:07.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.150 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.986279+0000", 2024-09-10T18:41:07.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:41:07.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.986287+0000", 2024-09-10T18:41:07.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3340000000000002e-06 2024-09-10T18:41:07.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.162384+0000", 2024-09-10T18:41:07.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.17609651700000001 2024-09-10T18:41:07.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.152 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.162394+0000", 2024-09-10T18:41:07.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8879999999999992e-06 2024-09-10T18:41:07.152 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.152 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.152 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.162711+0000", 2024-09-10T18:41:07.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00031690599999999999 2024-09-10T18:41:07.152 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.152 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.152 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.162817+0000", 2024-09-10T18:41:07.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106598 2024-09-10T18:41:07.152 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:07.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.167228+0000", 2024-09-10T18:41:07.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0044105719999999998 2024-09-10T18:41:07.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.167283+0000", 2024-09-10T18:41:07.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5568000000000001e-05 2024-09-10T18:41:07.153 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.153 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.153 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.153 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.15693.0:257806 10.c e363/362 ondisk, result = 0)", 2024-09-10T18:41:07.154 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:30.993761+0000", 2024-09-10T18:41:07.154 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 96.048953463000004, 2024-09-10T18:41:07.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.17124614899999999, 2024-09-10T18:41:07.154 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.154 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:41:07.154 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.993761+0000", 2024-09-10T18:41:07.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.993761+0000", 2024-09-10T18:41:07.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.993758+0000", 2024-09-10T18:41:07.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:41:07.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:30.993768+0000", 2024-09-10T18:41:07.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1330000000000005e-06 2024-09-10T18:41:07.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.162679+0000", 2024-09-10T18:41:07.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.168911378 2024-09-10T18:41:07.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.156 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.156 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.162688+0000", 2024-09-10T18:41:07.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1819999999999995e-06 2024-09-10T18:41:07.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.156 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.156 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164790+0000", 2024-09-10T18:41:07.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021022580000000001 2024-09-10T18:41:07.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.156 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.157 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.164796+0000", 2024-09-10T18:41:07.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6509999999999998e-06 2024-09-10T18:41:07.157 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.157 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.157 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:31.165007+0000", 2024-09-10T18:41:07.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000210643 2024-09-10T18:41:07.157 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.157 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.157 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.157 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.157 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:25896 11.f 11:f45f1a47:::benchmark_data_smithi028_186962_object25895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-10T18:41:07.157 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:40:26.652734+0000", 2024-09-10T18:41:07.158 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 40.389980027, 2024-09-10T18:41:07.158 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.23825011700000001, 2024-09-10T18:41:07.158 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.158 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:07.158 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:07.158 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:41:07.158 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:41:07.158 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 25896 2024-09-10T18:41:07.158 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.158 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.652734+0000", 2024-09-10T18:41:07.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.652734+0000", 2024-09-10T18:41:07.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.652731+0000", 2024-09-10T18:41:07.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:41:07.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.160 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.652769+0000", 2024-09-10T18:41:07.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7768999999999997e-05 2024-09-10T18:41:07.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.160 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.652780+0000", 2024-09-10T18:41:07.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1573000000000001e-05 2024-09-10T18:41:07.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.160 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.652785+0000", 2024-09-10T18:41:07.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0010000000000001e-06 2024-09-10T18:41:07.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.652829+0000", 2024-09-10T18:41:07.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3315000000000001e-05 2024-09-10T18:41:07.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.652953+0000", 2024-09-10T18:41:07.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012419699999999999 2024-09-10T18:41:07.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:41:07.162 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.653090+0000", 2024-09-10T18:41:07.162 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013666000000000001 2024-09-10T18:41:07.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.162 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.162 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.654688+0000", 2024-09-10T18:41:07.162 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001598173 2024-09-10T18:41:07.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.162 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:07.162 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.654697+0000", 2024-09-10T18:41:07.162 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7649999999999999e-06 2024-09-10T18:41:07.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.162 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.163 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.890890+0000", 2024-09-10T18:41:07.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.23619354100000001 2024-09-10T18:41:07.163 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.163 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.163 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:07.163 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.890942+0000", 2024-09-10T18:41:07.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1672000000000003e-05 2024-09-10T18:41:07.163 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.163 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.163 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.163 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.890984+0000", 2024-09-10T18:41:07.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2302000000000003e-05 2024-09-10T18:41:07.163 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.164 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.164 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.164 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:25915 11.f 11:f90b5c6a:::benchmark_data_smithi028_186962_object25914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-10T18:41:07.164 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:40:26.672576+0000", 2024-09-10T18:41:07.164 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 40.370137888999999, 2024-09-10T18:41:07.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.221930869, 2024-09-10T18:41:07.164 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:07.164 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:07.164 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:07.164 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:41:07.164 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:41:07.164 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 25915 2024-09-10T18:41:07.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.165 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:07.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.165 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:07.165 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672576+0000", 2024-09-10T18:41:07.165 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.165 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.165 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:07.165 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672576+0000", 2024-09-10T18:41:07.165 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:07.165 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.165 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:07.165 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672573+0000", 2024-09-10T18:41:07.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:41:07.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.166 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.166 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:07.166 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672616+0000", 2024-09-10T18:41:07.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2765999999999997e-05 2024-09-10T18:41:07.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.166 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.166 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:07.166 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672634+0000", 2024-09-10T18:41:07.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7462999999999999e-05 2024-09-10T18:41:07.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:07.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672640+0000", 2024-09-10T18:41:07.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7070000000000003e-06 2024-09-10T18:41:07.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:07.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672687+0000", 2024-09-10T18:41:07.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6304000000000003e-05 2024-09-10T18:41:07.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:07.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672796+0000", 2024-09-10T18:41:07.168 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000109098 2024-09-10T18:41:07.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.168 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:41:07.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672918+0000", 2024-09-10T18:41:07.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000122767 2024-09-10T18:41:07.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:07.169 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.674054+0000", 2024-09-10T18:41:07.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001135636 2024-09-10T18:41:07.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.169 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.674228+0000", 2024-09-10T18:41:07.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017409399999999999 2024-09-10T18:41:07.170 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.170 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:07.170 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.894422+0000", 2024-09-10T18:41:07.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.220193417 2024-09-10T18:41:07.170 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.170 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:07.170 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.894467+0000", 2024-09-10T18:41:07.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5510000000000003e-05 2024-09-10T18:41:07.170 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:07.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:07.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:07.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.894507+0000", 2024-09-10T18:41:07.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9913000000000002e-05 2024-09-10T18:41:07.171 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.171 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.171 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.171 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:07.171 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:07.171 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:41:07.183 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:07.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:07.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:07.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:07.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:07.335 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:07.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:07.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 50 16 86933 86917 108.59 78.0625 0.0108237 0.00920411 2024-09-10T18:41:07.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:07.475 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:07.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:07.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:07.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:07.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:07.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:07.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:07.717 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:07.752 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:07.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:07.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:07.833 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:07.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:07.884 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:07.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:07.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:07.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:08.032 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:08.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:08.034 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T18:41:08.133 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:08.188 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:08.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:08.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:08.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:08.326 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:08.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:08.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:08.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 51 15 88388 88373 108.245 91 0.0103653 0.00923397 2024-09-10T18:41:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:08 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:41:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:08 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:41:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:08 smithi204 ceph-mon[32357]: osdmap e391: 8 total, 8 up, 7 in 2024-09-10T18:41:08.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:08 smithi204 ceph-mon[32357]: pgmap v888: 178 pgs: 178 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-10T18:41:08.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:08.465 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:08.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:08.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:08.468 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:41:08.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:08 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:41:08.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:08 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:41:08.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:08 smithi028 ceph-mon[29291]: osdmap e391: 8 total, 8 up, 7 in 2024-09-10T18:41:08.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:08 smithi028 ceph-mon[29291]: pgmap v888: 178 pgs: 178 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-10T18:41:08.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:08 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:41:08.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:08 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:41:08.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:08 smithi028 ceph-mon[39220]: osdmap e391: 8 total, 8 up, 7 in 2024-09-10T18:41:08.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:08 smithi028 ceph-mon[39220]: pgmap v888: 178 pgs: 178 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-10T18:41:08.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:08.607 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:08.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:08.709 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:08.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:08.748 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:08.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:08.849 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:08.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:08.882 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:08.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:08.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:09.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:09.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:09.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:09.159 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:09.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:09.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:41:09.260 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:09.291 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:09.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:09.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:09.341 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 52 16 89876 89860 107.949 92.9375 0.00575234 0.0092589 2024-09-10T18:41:09.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:09.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:09.440 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:09.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:09.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:09.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:09.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:09.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:09.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:09.719 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:09.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:09.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:09.821 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:09.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:09.866 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:09.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:09.967 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:09.995 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:41:09.995 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:41:09.995 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:09 smithi204 ceph-mon[32357]: pgmap v889: 178 pgs: 178 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 848 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-10T18:41:10.002 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:10.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:10.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:10.104 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:10.108 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:41:10.108 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:41:10.108 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:09 smithi028 ceph-mon[29291]: pgmap v889: 178 pgs: 178 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 848 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-10T18:41:10.108 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:41:10.108 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:41:10.108 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:09 smithi028 ceph-mon[39220]: pgmap v889: 178 pgs: 178 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 848 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-10T18:41:10.147 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:10.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:10.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:10.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:10.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:10.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:10.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:10.341 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 53 16 91305 91289 107.597 89.3125 0.00924119 0.00928897 2024-09-10T18:41:10.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:10.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:10.425 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:10.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:10.482 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:41:10.483 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:41:10.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.483 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:91492 11.3 11:caaf223a:::benchmark_data_smithi028_186962_object91491:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-09-10T18:41:10.483 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:10.473640+0000", 2024-09-10T18:41:10.483 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0062984130000000001, 2024-09-10T18:41:10.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0063293890000000004, 2024-09-10T18:41:10.483 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:10.483 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:41:10.483 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:10.484 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:41:10.484 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:41:10.484 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 91492 2024-09-10T18:41:10.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.484 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:10.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:10.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.473640+0000", 2024-09-10T18:41:10.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:10.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:10.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.473640+0000", 2024-09-10T18:41:10.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:10.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:10.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.473638+0000", 2024-09-10T18:41:10.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:41:10.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:10.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.473716+0000", 2024-09-10T18:41:10.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8102000000000005e-05 2024-09-10T18:41:10.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:10.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.473726+0000", 2024-09-10T18:41:10.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0366999999999999e-05 2024-09-10T18:41:10.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:10.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.473731+0000", 2024-09-10T18:41:10.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7910000000000004e-06 2024-09-10T18:41:10.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:10.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.473778+0000", 2024-09-10T18:41:10.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6736000000000001e-05 2024-09-10T18:41:10.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:10.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.473975+0000", 2024-09-10T18:41:10.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019778299999999999 2024-09-10T18:41:10.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,7", 2024-09-10T18:41:10.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.474163+0000", 2024-09-10T18:41:10.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018763699999999999 2024-09-10T18:41:10.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:10.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.478058+0000", 2024-09-10T18:41:10.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038949509999999998 2024-09-10T18:41:10.491 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:10.491 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:10.491 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:10.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.491 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25849.0:91493 11.9 e391/374 11:965683f3:::benchmark_data_smithi028_186962_object91492:head v 391'5754, mlcod=391'5751)", 2024-09-10T18:41:10.491 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:10.477243+0000", 2024-09-10T18:41:10.491 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0026948010000000001, 2024-09-10T18:41:10.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028069869999999999, 2024-09-10T18:41:10.491 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:10.491 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:41:10.492 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:10.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:10.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.477243+0000", 2024-09-10T18:41:10.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:10.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:10.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.477243+0000", 2024-09-10T18:41:10.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:10.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:10.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.477242+0000", 2024-09-10T18:41:10.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:41:10.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:10.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.477274+0000", 2024-09-10T18:41:10.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1736999999999998e-05 2024-09-10T18:41:10.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:10.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.477288+0000", 2024-09-10T18:41:10.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3760999999999999e-05 2024-09-10T18:41:10.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:10.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.477293+0000", 2024-09-10T18:41:10.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2889999999999997e-06 2024-09-10T18:41:10.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:10.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.477359+0000", 2024-09-10T18:41:10.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6214e-05 2024-09-10T18:41:10.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:10.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.477377+0000", 2024-09-10T18:41:10.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8342000000000002e-05 2024-09-10T18:41:10.494 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:10.494 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:10.494 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:10.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.495 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:91497 11.3 11:c20482bb:::benchmark_data_smithi028_186962_object91496:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-09-10T18:41:10.495 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:10.477790+0000", 2024-09-10T18:41:10.495 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.0021476529999999998, 2024-09-10T18:41:10.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023230870000000002, 2024-09-10T18:41:10.495 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:10.495 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:41:10.495 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:10.495 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:41:10.495 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:41:10.495 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 91497 2024-09-10T18:41:10.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.495 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:10.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:10.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.477790+0000", 2024-09-10T18:41:10.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:10.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:10.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.477790+0000", 2024-09-10T18:41:10.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:10.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:10.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.477788+0000", 2024-09-10T18:41:10.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:41:10.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:10.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.477881+0000", 2024-09-10T18:41:10.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3160999999999998e-05 2024-09-10T18:41:10.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:10.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.477891+0000", 2024-09-10T18:41:10.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7489999999999993e-06 2024-09-10T18:41:10.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:10.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.477896+0000", 2024-09-10T18:41:10.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1830000000000002e-06 2024-09-10T18:41:10.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:10.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.478082+0000", 2024-09-10T18:41:10.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018523699999999999 2024-09-10T18:41:10.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:10.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.478229+0000", 2024-09-10T18:41:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014766399999999999 2024-09-10T18:41:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,7", 2024-09-10T18:41:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.478370+0000", 2024-09-10T18:41:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014088300000000001 2024-09-10T18:41:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25849.0:91495 11.8 e391/374 11:12593dad:::benchmark_data_smithi028_186962_object91494:head v 391'5841, mlcod=391'5839)", 2024-09-10T18:41:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:10.478502+0000", 2024-09-10T18:41:10.499 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.001436494, 2024-09-10T18:41:10.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016763609999999999, 2024-09-10T18:41:10.500 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:10.500 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:41:10.500 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:10.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:10.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.478502+0000", 2024-09-10T18:41:10.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:10.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:10.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.478502+0000", 2024-09-10T18:41:10.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:10.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:10.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.478501+0000", 2024-09-10T18:41:10.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:41:10.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:10.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.478548+0000", 2024-09-10T18:41:10.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7718e-05 2024-09-10T18:41:10.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:10.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.478557+0000", 2024-09-10T18:41:10.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0180000000000007e-06 2024-09-10T18:41:10.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:10.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.478562+0000", 2024-09-10T18:41:10.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7010000000000004e-06 2024-09-10T18:41:10.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:10.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.478606+0000", 2024-09-10T18:41:10.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3838999999999998e-05 2024-09-10T18:41:10.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:10.503 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.478623+0000", 2024-09-10T18:41:10.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7540000000000001e-05 2024-09-10T18:41:10.503 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:10.503 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:10.503 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:10.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.503 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.25849.0:91492 11.3 e391/374)", 2024-09-10T18:41:10.503 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:10.479578+0000", 2024-09-10T18:41:10.503 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.00036023799999999998, 2024-09-10T18:41:10.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00065118999999999995, 2024-09-10T18:41:10.503 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:10.504 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "queued for pg", 2024-09-10T18:41:10.504 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:10.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:10.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.479578+0000", 2024-09-10T18:41:10.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:10.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:10.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.479578+0000", 2024-09-10T18:41:10.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:10.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:10.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.479577+0000", 2024-09-10T18:41:10.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:41:10.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:10.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.479582+0000", 2024-09-10T18:41:10.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9710000000000003e-06 2024-09-10T18:41:10.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:10.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.479591+0000", 2024-09-10T18:41:10.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7340000000000003e-06 2024-09-10T18:41:10.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:10.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:10.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:10.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:10.479597+0000", 2024-09-10T18:41:10.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2920000000000003e-06 2024-09-10T18:41:10.506 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:10.506 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:10.506 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:10.506 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:10.506 INFO:teuthology.orchestra.run.smithi204.stdout: ], 2024-09-10T18:41:10.506 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 5 2024-09-10T18:41:10.506 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:41:10.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:10.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:10.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:10.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-10T18:41:10.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:10.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:10.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:10.794 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:10.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:10.827 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:10.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:10.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:10.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:10 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-10T18:41:10.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:10 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-09-10T18:41:10.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:10 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-10T18:41:10.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:10 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-09-10T18:41:10.981 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:10.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:10.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:11.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:11.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:11.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:11.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:11.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:10 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-10T18:41:11.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:10 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]': finished 2024-09-10T18:41:11.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:10 smithi028 ceph-mon[29291]: osdmap e392: 8 total, 8 up, 7 in 2024-09-10T18:41:11.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:10 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-10T18:41:11.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:10 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-09-10T18:41:11.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:10 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-10T18:41:11.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:10 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-09-10T18:41:11.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:10 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-10T18:41:11.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:10 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]': finished 2024-09-10T18:41:11.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:10 smithi028 ceph-mon[39220]: osdmap e392: 8 total, 8 up, 7 in 2024-09-10T18:41:11.217 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:10 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-10T18:41:11.217 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:10 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-09-10T18:41:11.217 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:10 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-10T18:41:11.217 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:10 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-09-10T18:41:11.218 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:10 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-10T18:41:11.218 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:10 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]': finished 2024-09-10T18:41:11.218 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:10 smithi204 ceph-mon[32357]: osdmap e392: 8 total, 8 up, 7 in 2024-09-10T18:41:11.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:11.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:11.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:11.342 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 54 16 92596 92580 107.099 80.6875 0.00576989 0.00933235 2024-09-10T18:41:11.376 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:11.413 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:11.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:11.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:11.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:11.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:11.575 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:11.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:11.666 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:41:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:41:11] "GET /metrics HTTP/1.1" 200 224935 "" "Prometheus/2.43.0" 2024-09-10T18:41:11.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:11.698 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:11.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:11.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:11.800 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:11.822 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:11.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:11.924 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:41:11.957 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:11.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:11.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:12.059 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:12.087 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:12.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:12.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:12.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:12.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:12.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:11 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:41:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:11 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:41:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:11 smithi028 ceph-mon[39220]: pgmap v891: 178 pgs: 178 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-10T18:41:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:11 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2148020351' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:41:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:11 smithi028 ceph-mon[39220]: from='client.25900 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:41:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:11 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:41:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:11 smithi028 ceph-mon[39220]: from='client.25900 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:41:12.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:11 smithi028 ceph-mon[39220]: osdmap e393: 8 total, 8 up, 7 in 2024-09-10T18:41:12.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:11 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:41:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:11 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:41:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:11 smithi028 ceph-mon[29291]: pgmap v891: 178 pgs: 178 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-10T18:41:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:11 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2148020351' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:41:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:11 smithi028 ceph-mon[29291]: from='client.25900 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:41:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:11 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:41:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:11 smithi028 ceph-mon[29291]: from='client.25900 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:41:12.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:11 smithi028 ceph-mon[29291]: osdmap e393: 8 total, 8 up, 7 in 2024-09-10T18:41:12.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:12.342 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 55 16 93635 93619 106.332 64.9375 0.0116688 0.00940014 2024-09-10T18:41:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:11 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:41:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:11 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:41:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:11 smithi204 ceph-mon[32357]: pgmap v891: 178 pgs: 178 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-10T18:41:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:11 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2148020351' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:41:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:11 smithi204 ceph-mon[32357]: from='client.25900 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:41:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:11 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:41:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:11 smithi204 ceph-mon[32357]: from='client.25900 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:41:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:11 smithi204 ceph-mon[32357]: osdmap e393: 8 total, 8 up, 7 in 2024-09-10T18:41:12.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:41:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:41:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:41:12.363 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:12.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:12.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:12.465 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:12.501 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:12.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:12.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:12.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:12.641 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:12.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:12.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:12.743 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:12.777 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-10T18:41:12.789 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:12.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:12.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:12.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:12.928 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:12.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:12.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:13.030 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:13.081 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:13.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:13.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:13.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:13.194 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:41:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:13.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:13.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:13.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:13.342 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 56 16 94984 94968 105.939 84.3125 0.00876874 0.00943474 2024-09-10T18:41:13.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:13.348 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:13.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:13.449 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:13.487 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:13.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:13.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:13.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:13.627 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:13.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:13.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:13.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:13.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:13.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:13.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:13.908 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:13.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:13.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:14.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:14.049 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:14.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:14.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:14.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:14.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:14.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:14.329 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:14.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:14.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:14.342 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 57 16 96504 96488 105.747 95 0.0128211 0.00945229 2024-09-10T18:41:14.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:14 smithi204 ceph-mon[32357]: pgmap v893: 178 pgs: 178 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-10T18:41:14.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:14.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:14.473 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:14.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:14.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:14 smithi028 ceph-mon[29291]: pgmap v893: 178 pgs: 178 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-10T18:41:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:14 smithi028 ceph-mon[39220]: pgmap v893: 178 pgs: 178 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-10T18:41:14.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:14.612 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:14.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:14.713 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:14.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:14.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:14.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:14.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:14.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:14.891 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:14.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:14.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:15.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:15.028 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:15.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:15.076 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:41:15 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:41:15.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:15.129 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:41:15.129 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:41:15.129 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-10T18:41:15.130 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-10T18:41:15.130 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:41:15.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:15.169 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:15.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:15.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:15.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:15.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:15.342 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 58 16 97891 97875 105.418 86.6875 0.00763517 0.00948159 2024-09-10T18:41:15.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:15.443 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:15.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:15.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:15.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:15.578 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:15.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:15.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:15.680 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:15.711 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:15.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:15.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:15.813 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:15.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-10T18:41:15.849 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:15.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:15.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:15.951 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:15.972 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:15.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:15.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:16.074 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:16.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:16.127 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:16.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:16.209 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:41:16.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:16.265 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:16.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:16.342 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 59 16 99064 99048 104.874 73.3125 0.0121853 0.00953079 2024-09-10T18:41:16.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:16.405 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:16.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:16.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:16.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:16 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:16.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:16 smithi028 ceph-mon[39220]: pgmap v894: 178 pgs: 178 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 13 B/s, 0 objects/s recovering 2024-09-10T18:41:16.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:16 smithi028 ceph-mon[39220]: osdmap e394: 8 total, 8 up, 7 in 2024-09-10T18:41:16.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:16 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:16.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:16 smithi028 ceph-mon[29291]: pgmap v894: 178 pgs: 178 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 13 B/s, 0 objects/s recovering 2024-09-10T18:41:16.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:16 smithi028 ceph-mon[29291]: osdmap e394: 8 total, 8 up, 7 in 2024-09-10T18:41:16.507 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:16.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:16.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:16.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:16 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:16 smithi204 ceph-mon[32357]: pgmap v894: 178 pgs: 178 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 13 B/s, 0 objects/s recovering 2024-09-10T18:41:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:16 smithi204 ceph-mon[32357]: osdmap e394: 8 total, 8 up, 7 in 2024-09-10T18:41:16.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:16.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:16.778 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:16.815 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:16.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:16.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:16.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:16.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:16.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:16.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:17.056 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:17.099 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:17.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:17.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:17.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:17.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:17.228 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:17.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:17.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:17.343 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:41:17.342804+0000 min lat: 0.00125893 max lat: 0.413127 avg lat: 0.00954895 2024-09-10T18:41:17.343 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:41:17.343 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 60 16 100554 100538 104.677 93.125 0.0100136 0.00954895 2024-09-10T18:41:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:17.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:17.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:17.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:17.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:17.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:17.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:17.650 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:17.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:17.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:17.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:17.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:17.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:17.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:17.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:17.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:18.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:18.060 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:18.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:18.162 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:18.194 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:18.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:18.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:18.261 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:18 smithi028 ceph-mon[39220]: pgmap v896: 178 pgs: 178 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 816 B/s rd, 73 MiB/s wr, 1.18k op/s; 70 B/s, 1 objects/s recovering 2024-09-10T18:41:18.261 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:18 smithi028 ceph-mon[29291]: pgmap v896: 178 pgs: 178 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 816 B/s rd, 73 MiB/s wr, 1.18k op/s; 70 B/s, 1 objects/s recovering 2024-09-10T18:41:18.296 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:18.325 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:18.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:18.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:18.343 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 61 16 101897 101881 104.337 83.9375 0.00906198 0.00957989 2024-09-10T18:41:18.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:18.471 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:18.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:18.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:18.538 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:18 smithi204 ceph-mon[32357]: pgmap v896: 178 pgs: 178 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 816 B/s rd, 73 MiB/s wr, 1.18k op/s; 70 B/s, 1 objects/s recovering 2024-09-10T18:41:18.557 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:41:18.557 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-10T18:41:18.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-10T18:41:18.557 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:41:18.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.557 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:214410 10.f 10:f1b0b8f0:::benchmark_data_smithi028_161947_object77486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:41:18.557 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.062548+0000", 2024-09-10T18:41:18.557 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 146.48999635999999, 2024-09-10T18:41:18.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45499772199999999, 2024-09-10T18:41:18.557 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.557 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:18.557 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.558 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:18.558 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:18.558 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 214410 2024-09-10T18:41:18.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.558 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062548+0000", 2024-09-10T18:41:18.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062548+0000", 2024-09-10T18:41:18.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.559 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.559 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062546+0000", 2024-09-10T18:41:18.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:41:18.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.559 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.559 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062554+0000", 2024-09-10T18:41:18.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0849999999999997e-06 2024-09-10T18:41:18.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.560 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062566+0000", 2024-09-10T18:41:18.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2384e-05 2024-09-10T18:41:18.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.560 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062572+0000", 2024-09-10T18:41:18.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2299999999999996e-06 2024-09-10T18:41:18.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.560 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062612+0000", 2024-09-10T18:41:18.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9508999999999997e-05 2024-09-10T18:41:18.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062681+0000", 2024-09-10T18:41:18.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9183000000000005e-05 2024-09-10T18:41:18.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3", 2024-09-10T18:41:18.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062748+0000", 2024-09-10T18:41:18.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7360999999999994e-05 2024-09-10T18:41:18.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:18.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.065004+0000", 2024-09-10T18:41:18.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022560739999999998 2024-09-10T18:41:18.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.070537+0000", 2024-09-10T18:41:18.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0055331089999999996 2024-09-10T18:41:18.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517440+0000", 2024-09-10T18:41:18.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44690291100000001 2024-09-10T18:41:18.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:18.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517498+0000", 2024-09-10T18:41:18.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8114e-05 2024-09-10T18:41:18.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.564 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517546+0000", 2024-09-10T18:41:18.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7413999999999997e-05 2024-09-10T18:41:18.564 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.564 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.564 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.564 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.564 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.564 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:214415 10.a 10:58f59aac:::benchmark_data_smithi028_161947_object77491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:41:18.564 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.068100+0000", 2024-09-10T18:41:18.564 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 146.48444494200001, 2024-09-10T18:41:18.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44505853499999998, 2024-09-10T18:41:18.564 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.564 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:18.565 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.565 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:18.565 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:18.565 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 214415 2024-09-10T18:41:18.566 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.566 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.566 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.566 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068100+0000", 2024-09-10T18:41:18.566 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.566 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.566 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.566 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068100+0000", 2024-09-10T18:41:18.566 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.567 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.567 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.567 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.567 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068097+0000", 2024-09-10T18:41:18.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:41:18.567 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.567 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.567 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.567 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068106+0000", 2024-09-10T18:41:18.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9109999999999999e-06 2024-09-10T18:41:18.567 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.568 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.568 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068121+0000", 2024-09-10T18:41:18.568 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4595e-05 2024-09-10T18:41:18.568 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.568 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.568 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068128+0000", 2024-09-10T18:41:18.568 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1099999999999997e-06 2024-09-10T18:41:18.568 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.568 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.568 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068200+0000", 2024-09-10T18:41:18.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2563000000000003e-05 2024-09-10T18:41:18.569 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.569 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.569 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068320+0000", 2024-09-10T18:41:18.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119314 2024-09-10T18:41:18.569 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.569 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,5", 2024-09-10T18:41:18.569 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068445+0000", 2024-09-10T18:41:18.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125384 2024-09-10T18:41:18.569 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.569 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.570 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.570 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512681+0000", 2024-09-10T18:41:18.570 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.444235614 2024-09-10T18:41:18.570 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.570 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.570 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:18.570 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512695+0000", 2024-09-10T18:41:18.570 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3879999999999999e-05 2024-09-10T18:41:18.570 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.570 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.570 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.570 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512953+0000", 2024-09-10T18:41:18.570 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000258576 2024-09-10T18:41:18.571 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.571 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.571 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.513010+0000", 2024-09-10T18:41:18.571 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7293000000000002e-05 2024-09-10T18:41:18.571 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.571 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:18.571 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.513057+0000", 2024-09-10T18:41:18.571 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6354999999999999e-05 2024-09-10T18:41:18.571 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.571 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.572 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.513158+0000", 2024-09-10T18:41:18.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101337 2024-09-10T18:41:18.572 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.572 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.572 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.572 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.572 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.572 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:214416 10.a 10:5d56f650:::benchmark_data_smithi028_161947_object77492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:41:18.572 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.068145+0000", 2024-09-10T18:41:18.572 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 146.48439969500001, 2024-09-10T18:41:18.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44523108900000002, 2024-09-10T18:41:18.572 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.573 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:18.573 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.573 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:18.574 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:18.574 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 214416 2024-09-10T18:41:18.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.574 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068145+0000", 2024-09-10T18:41:18.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068145+0000", 2024-09-10T18:41:18.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.575 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.575 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068144+0000", 2024-09-10T18:41:18.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:41:18.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.575 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.575 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068148+0000", 2024-09-10T18:41:18.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7189999999999999e-06 2024-09-10T18:41:18.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.575 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.575 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068157+0000", 2024-09-10T18:41:18.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0110000000000006e-06 2024-09-10T18:41:18.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068161+0000", 2024-09-10T18:41:18.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8600000000000003e-06 2024-09-10T18:41:18.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.070947+0000", 2024-09-10T18:41:18.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002786232 2024-09-10T18:41:18.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071069+0000", 2024-09-10T18:41:18.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012178199999999999 2024-09-10T18:41:18.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.577 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,5", 2024-09-10T18:41:18.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071193+0000", 2024-09-10T18:41:18.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000124795 2024-09-10T18:41:18.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.577 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:18.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512863+0000", 2024-09-10T18:41:18.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44166944299999999 2024-09-10T18:41:18.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512919+0000", 2024-09-10T18:41:18.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6190000000000002e-05 2024-09-10T18:41:18.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512981+0000", 2024-09-10T18:41:18.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2149000000000004e-05 2024-09-10T18:41:18.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.513207+0000", 2024-09-10T18:41:18.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022604600000000001 2024-09-10T18:41:18.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:18.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.513279+0000", 2024-09-10T18:41:18.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1328999999999994e-05 2024-09-10T18:41:18.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.513376+0000", 2024-09-10T18:41:18.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7270999999999998e-05 2024-09-10T18:41:18.579 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.579 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.579 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.579 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDPGPushReply(10.a 354/353 [PushReplyOp(10:53ea37f5:::benchmark_data_smithi028_161947_object116442:head)])", 2024-09-10T18:41:18.579 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.071316+0000", 2024-09-10T18:41:18.579 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 146.481228587, 2024-09-10T18:41:18.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44120257899999998, 2024-09-10T18:41:18.580 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.580 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:41:18.580 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071316+0000", 2024-09-10T18:41:18.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071316+0000", 2024-09-10T18:41:18.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071315+0000", 2024-09-10T18:41:18.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:41:18.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071320+0000", 2024-09-10T18:41:18.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6399999999999996e-06 2024-09-10T18:41:18.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071328+0000", 2024-09-10T18:41:18.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2549999999999997e-06 2024-09-10T18:41:18.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.582 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071333+0000", 2024-09-10T18:41:18.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9960000000000001e-06 2024-09-10T18:41:18.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.582 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512288+0000", 2024-09-10T18:41:18.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44095521300000001 2024-09-10T18:41:18.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.583 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512519+0000", 2024-09-10T18:41:18.583 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023027599999999999 2024-09-10T18:41:18.583 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.583 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.583 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.583 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.583 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.583 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:214417 10.f 10:f966d268:::benchmark_data_smithi028_161947_object77493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:41:18.583 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.071690+0000", 2024-09-10T18:41:18.583 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 146.48085460600001, 2024-09-10T18:41:18.583 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44675364699999998, 2024-09-10T18:41:18.583 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.583 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:18.583 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.584 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:18.584 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:18.584 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 214417 2024-09-10T18:41:18.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.584 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071690+0000", 2024-09-10T18:41:18.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071690+0000", 2024-09-10T18:41:18.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071687+0000", 2024-09-10T18:41:18.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:41:18.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071695+0000", 2024-09-10T18:41:18.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9359999999999999e-06 2024-09-10T18:41:18.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071710+0000", 2024-09-10T18:41:18.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5092999999999999e-05 2024-09-10T18:41:18.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071718+0000", 2024-09-10T18:41:18.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5329999999999997e-06 2024-09-10T18:41:18.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071782+0000", 2024-09-10T18:41:18.587 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4134000000000006e-05 2024-09-10T18:41:18.587 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.587 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.587 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.587 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072098+0000", 2024-09-10T18:41:18.587 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000316156 2024-09-10T18:41:18.587 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.587 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.587 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3", 2024-09-10T18:41:18.587 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072203+0000", 2024-09-10T18:41:18.587 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010477200000000001 2024-09-10T18:41:18.587 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.587 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.587 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:18.587 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517614+0000", 2024-09-10T18:41:18.588 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44541141000000001 2024-09-10T18:41:18.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.588 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.588 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.518312+0000", 2024-09-10T18:41:18.588 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00069769600000000004 2024-09-10T18:41:18.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.588 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.588 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.518366+0000", 2024-09-10T18:41:18.588 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3857999999999998e-05 2024-09-10T18:41:18.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.588 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:18.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.518409+0000", 2024-09-10T18:41:18.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3569999999999998e-05 2024-09-10T18:41:18.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.518444+0000", 2024-09-10T18:41:18.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4211000000000002e-05 2024-09-10T18:41:18.589 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.589 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.589 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.590 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:214419 10.f 10:f9ab56c8:::benchmark_data_smithi028_161947_object77495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:41:18.590 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.072174+0000", 2024-09-10T18:41:18.590 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 146.48037102699999, 2024-09-10T18:41:18.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45442210100000002, 2024-09-10T18:41:18.590 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.590 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:18.590 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.590 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:18.590 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:18.590 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 214419 2024-09-10T18:41:18.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.591 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072174+0000", 2024-09-10T18:41:18.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072174+0000", 2024-09-10T18:41:18.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072172+0000", 2024-09-10T18:41:18.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:41:18.592 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.592 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.592 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072178+0000", 2024-09-10T18:41:18.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0000000000000002e-06 2024-09-10T18:41:18.592 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.592 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.592 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072189+0000", 2024-09-10T18:41:18.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1032999999999999e-05 2024-09-10T18:41:18.592 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.592 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072196+0000", 2024-09-10T18:41:18.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4280000000000001e-06 2024-09-10T18:41:18.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.514966+0000", 2024-09-10T18:41:18.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44277013500000001 2024-09-10T18:41:18.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.515075+0000", 2024-09-10T18:41:18.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000109749 2024-09-10T18:41:18.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.594 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3", 2024-09-10T18:41:18.594 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.515187+0000", 2024-09-10T18:41:18.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011199700000000001 2024-09-10T18:41:18.594 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.594 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:18.594 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517688+0000", 2024-09-10T18:41:18.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025006770000000002 2024-09-10T18:41:18.594 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.594 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.594 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.526499+0000", 2024-09-10T18:41:18.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088109829999999997 2024-09-10T18:41:18.595 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.595 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.595 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.526546+0000", 2024-09-10T18:41:18.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6965999999999997e-05 2024-09-10T18:41:18.595 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.595 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:18.595 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.526573+0000", 2024-09-10T18:41:18.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7219000000000001e-05 2024-09-10T18:41:18.595 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.595 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.595 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.526596+0000", 2024-09-10T18:41:18.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2323999999999998e-05 2024-09-10T18:41:18.596 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.596 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.596 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.596 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.15693.0:214418 10.e e354/314 10:7f999048:::benchmark_data_smithi028_161947_object77494:head v 354'13271, mlcod=354'13269)", 2024-09-10T18:41:18.596 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.072212+0000", 2024-09-10T18:41:18.596 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 146.48033223300001, 2024-09-10T18:41:18.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44493968699999997, 2024-09-10T18:41:18.596 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.596 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:18.597 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072212+0000", 2024-09-10T18:41:18.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072212+0000", 2024-09-10T18:41:18.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072211+0000", 2024-09-10T18:41:18.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:41:18.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072218+0000", 2024-09-10T18:41:18.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.691e-06 2024-09-10T18:41:18.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072225+0000", 2024-09-10T18:41:18.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7170000000000002e-06 2024-09-10T18:41:18.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072230+0000", 2024-09-10T18:41:18.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5910000000000003e-06 2024-09-10T18:41:18.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.599 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512193+0000", 2024-09-10T18:41:18.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43996238700000001 2024-09-10T18:41:18.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.599 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512221+0000", 2024-09-10T18:41:18.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8779999999999999e-05 2024-09-10T18:41:18.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:18.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517109+0000", 2024-09-10T18:41:18.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004887209 2024-09-10T18:41:18.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517152+0000", 2024-09-10T18:41:18.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3341999999999999e-05 2024-09-10T18:41:18.600 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.600 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.600 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.600 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:214422 10.f 10:f1090f8c:::benchmark_data_smithi028_161947_object77498:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:41:18.600 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.076038+0000", 2024-09-10T18:41:18.601 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 146.47650639, 2024-09-10T18:41:18.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46066008899999999, 2024-09-10T18:41:18.601 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.601 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:18.601 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.601 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:18.601 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:18.601 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 214422 2024-09-10T18:41:18.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.601 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.076038+0000", 2024-09-10T18:41:18.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.076038+0000", 2024-09-10T18:41:18.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.076036+0000", 2024-09-10T18:41:18.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:41:18.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.076050+0000", 2024-09-10T18:41:18.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3974999999999999e-05 2024-09-10T18:41:18.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512179+0000", 2024-09-10T18:41:18.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43612967400000002 2024-09-10T18:41:18.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512192+0000", 2024-09-10T18:41:18.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.312e-05 2024-09-10T18:41:18.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.518478+0000", 2024-09-10T18:41:18.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0062856149999999996 2024-09-10T18:41:18.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.518584+0000", 2024-09-10T18:41:18.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106502 2024-09-10T18:41:18.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3", 2024-09-10T18:41:18.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.518688+0000", 2024-09-10T18:41:18.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103607 2024-09-10T18:41:18.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:18.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.526442+0000", 2024-09-10T18:41:18.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007754337 2024-09-10T18:41:18.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.533804+0000", 2024-09-10T18:41:18.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0073612139999999996 2024-09-10T18:41:18.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.536645+0000", 2024-09-10T18:41:18.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002841732 2024-09-10T18:41:18.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:18.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.536673+0000", 2024-09-10T18:41:18.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7489999999999999e-05 2024-09-10T18:41:18.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.536698+0000", 2024-09-10T18:41:18.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5386999999999999e-05 2024-09-10T18:41:18.607 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.607 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.607 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.607 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224010 10.7 10:e0250792:::benchmark_data_smithi028_161947_object87086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-09-10T18:41:18.607 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.451472+0000", 2024-09-10T18:41:18.607 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 138.10107245200001, 2024-09-10T18:41:18.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.50653015300000004, 2024-09-10T18:41:18.607 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.607 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:41:18.607 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.607 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:18.607 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:18.607 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224010 2024-09-10T18:41:18.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.608 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451472+0000", 2024-09-10T18:41:18.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451472+0000", 2024-09-10T18:41:18.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451470+0000", 2024-09-10T18:41:18.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:41:18.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451477+0000", 2024-09-10T18:41:18.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1470000000000004e-06 2024-09-10T18:41:18.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451490+0000", 2024-09-10T18:41:18.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3033999999999999e-05 2024-09-10T18:41:18.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451496+0000", 2024-09-10T18:41:18.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5269999999999998e-06 2024-09-10T18:41:18.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451563+0000", 2024-09-10T18:41:18.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7187999999999994e-05 2024-09-10T18:41:18.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451693+0000", 2024-09-10T18:41:18.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000129966 2024-09-10T18:41:18.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3,5", 2024-09-10T18:41:18.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451824+0000", 2024-09-10T18:41:18.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013087099999999999 2024-09-10T18:41:18.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:18.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.454501+0000", 2024-09-10T18:41:18.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0026775169999999999 2024-09-10T18:41:18.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.454537+0000", 2024-09-10T18:41:18.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5447999999999997e-05 2024-09-10T18:41:18.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.612 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.457032+0000", 2024-09-10T18:41:18.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024956470000000001 2024-09-10T18:41:18.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.612 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.612 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.473535+0000", 2024-09-10T18:41:18.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016502783 2024-09-10T18:41:18.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.612 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:41:18.612 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.473546+0000", 2024-09-10T18:41:18.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0431000000000001e-05 2024-09-10T18:41:18.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.613 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.952470+0000", 2024-09-10T18:41:18.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47892405900000001 2024-09-10T18:41:18.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.613 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for missing object", 2024-09-10T18:41:18.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.952584+0000", 2024-09-10T18:41:18.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114357 2024-09-10T18:41:18.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.613 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.957919+0000", 2024-09-10T18:41:18.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005335093 2024-09-10T18:41:18.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.958002+0000", 2024-09-10T18:41:18.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3076000000000002e-05 2024-09-10T18:41:18.614 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.614 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.614 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.614 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224020 10.a 10:5e0c363c:::benchmark_data_smithi028_161947_object87096:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-09-10T18:41:18.614 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.457939+0000", 2024-09-10T18:41:18.614 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 138.09460567400001, 2024-09-10T18:41:18.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.48552529, 2024-09-10T18:41:18.614 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.615 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:41:18.615 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.615 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:18.615 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:18.615 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224020 2024-09-10T18:41:18.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.615 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:18.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.457939+0000", 2024-09-10T18:41:18.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.617 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.457939+0000", 2024-09-10T18:41:18.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.618 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.618 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.618 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.618 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.457938+0000", 2024-09-10T18:41:18.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999995 2024-09-10T18:41:18.618 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.618 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.618 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.618 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.457941+0000", 2024-09-10T18:41:18.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.287e-06 2024-09-10T18:41:18.618 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.618 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.618 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.618 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.457947+0000", 2024-09-10T18:41:18.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7259999999999999e-06 2024-09-10T18:41:18.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.619 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.619 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.457951+0000", 2024-09-10T18:41:18.619 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4079999999999998e-06 2024-09-10T18:41:18.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.619 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.619 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.458090+0000", 2024-09-10T18:41:18.619 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000139827 2024-09-10T18:41:18.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.619 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.619 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.458216+0000", 2024-09-10T18:41:18.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012541900000000001 2024-09-10T18:41:18.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,5", 2024-09-10T18:41:18.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.458342+0000", 2024-09-10T18:41:18.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012641700000000001 2024-09-10T18:41:18.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.461711+0000", 2024-09-10T18:41:18.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033687320000000001 2024-09-10T18:41:18.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:18.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463213+0000", 2024-09-10T18:41:18.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0015021119999999999 2024-09-10T18:41:18.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463243+0000", 2024-09-10T18:41:18.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0298999999999999e-05 2024-09-10T18:41:18.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.472753+0000", 2024-09-10T18:41:18.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095097480000000002 2024-09-10T18:41:18.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:41:18.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.472764+0000", 2024-09-10T18:41:18.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.04e-05 2024-09-10T18:41:18.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.943332+0000", 2024-09-10T18:41:18.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47056816499999998 2024-09-10T18:41:18.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.943464+0000", 2024-09-10T18:41:18.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013241699999999999 2024-09-10T18:41:18.622 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.622 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.623 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.623 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224025 10.7 10:e7883bf7:::benchmark_data_smithi028_161947_object87101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-09-10T18:41:18.623 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.463510+0000", 2024-09-10T18:41:18.623 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 138.08903496100001, 2024-09-10T18:41:18.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.49490601499999998, 2024-09-10T18:41:18.623 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.623 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:41:18.623 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.623 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:18.623 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:18.623 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224025 2024-09-10T18:41:18.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.623 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.624 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.624 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.624 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463510+0000", 2024-09-10T18:41:18.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.624 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.624 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.624 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463510+0000", 2024-09-10T18:41:18.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.624 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.624 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.624 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463508+0000", 2024-09-10T18:41:18.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:41:18.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.625 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.625 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463515+0000", 2024-09-10T18:41:18.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4549999999999998e-06 2024-09-10T18:41:18.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.625 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.625 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463529+0000", 2024-09-10T18:41:18.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4250000000000001e-05 2024-09-10T18:41:18.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.625 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.625 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463536+0000", 2024-09-10T18:41:18.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6139999999999997e-06 2024-09-10T18:41:18.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463758+0000", 2024-09-10T18:41:18.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000221652 2024-09-10T18:41:18.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.464491+0000", 2024-09-10T18:41:18.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00073350399999999995 2024-09-10T18:41:18.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3,5", 2024-09-10T18:41:18.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.466575+0000", 2024-09-10T18:41:18.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020836539999999999 2024-09-10T18:41:18.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.479982+0000", 2024-09-10T18:41:18.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013407189999999999 2024-09-10T18:41:18.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:41:18.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.479993+0000", 2024-09-10T18:41:18.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0618999999999999e-05 2024-09-10T18:41:18.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.952817+0000", 2024-09-10T18:41:18.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47282479700000002 2024-09-10T18:41:18.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for missing object", 2024-09-10T18:41:18.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.952911+0000", 2024-09-10T18:41:18.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3918000000000004e-05 2024-09-10T18:41:18.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.958346+0000", 2024-09-10T18:41:18.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054350170000000003 2024-09-10T18:41:18.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.958416+0000", 2024-09-10T18:41:18.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9338999999999995e-05 2024-09-10T18:41:18.629 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.629 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.629 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.629 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224030 10.7 10:eb8e2371:::benchmark_data_smithi028_161947_object87106:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-10T18:41:18.630 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.489381+0000", 2024-09-10T18:41:18.630 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 138.063163949, 2024-09-10T18:41:18.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46926472200000002, 2024-09-10T18:41:18.630 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.630 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:18.630 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.630 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:18.630 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:18.630 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224030 2024-09-10T18:41:18.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.631 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.631 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.631 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.489381+0000", 2024-09-10T18:41:18.631 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.631 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.631 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.631 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.489381+0000", 2024-09-10T18:41:18.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.489379+0000", 2024-09-10T18:41:18.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:41:18.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.489385+0000", 2024-09-10T18:41:18.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.353e-06 2024-09-10T18:41:18.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.633 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.489396+0000", 2024-09-10T18:41:18.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0624e-05 2024-09-10T18:41:18.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.633 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.489400+0000", 2024-09-10T18:41:18.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5770000000000004e-06 2024-09-10T18:41:18.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.633 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.489428+0000", 2024-09-10T18:41:18.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7268e-05 2024-09-10T18:41:18.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:41:18.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.489439+0000", 2024-09-10T18:41:18.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1362000000000001e-05 2024-09-10T18:41:18.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.952941+0000", 2024-09-10T18:41:18.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.463501991 2024-09-10T18:41:18.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.953048+0000", 2024-09-10T18:41:18.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010751600000000001 2024-09-10T18:41:18.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,5", 2024-09-10T18:41:18.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.953164+0000", 2024-09-10T18:41:18.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011588599999999999 2024-09-10T18:41:18.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:18.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.956405+0000", 2024-09-10T18:41:18.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032407389999999999 2024-09-10T18:41:18.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.636 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.636 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.957722+0000", 2024-09-10T18:41:18.636 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013170549999999999 2024-09-10T18:41:18.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.636 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.636 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.958568+0000", 2024-09-10T18:41:18.636 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00084616900000000002 2024-09-10T18:41:18.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.636 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:18.636 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.958597+0000", 2024-09-10T18:41:18.636 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8214000000000001e-05 2024-09-10T18:41:18.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.637 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.637 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.637 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.958645+0000", 2024-09-10T18:41:18.637 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8797000000000001e-05 2024-09-10T18:41:18.637 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.637 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.637 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.637 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.637 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.637 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.34334.0:2468 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94344803719616 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:41:18.637 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.216048+0000", 2024-09-10T18:41:18.637 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 114.336496908, 2024-09-10T18:41:18.637 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.039665386, 2024-09-10T18:41:18.638 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.638 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:41:18.638 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.638 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.34334", 2024-09-10T18:41:18.638 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/601695668", 2024-09-10T18:41:18.638 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2468 2024-09-10T18:41:18.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.638 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216048+0000", 2024-09-10T18:41:18.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216048+0000", 2024-09-10T18:41:18.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.639 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216046+0000", 2024-09-10T18:41:18.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:41:18.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.639 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216053+0000", 2024-09-10T18:41:18.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1040000000000001e-06 2024-09-10T18:41:18.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.640 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.640 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216065+0000", 2024-09-10T18:41:18.640 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2252999999999999e-05 2024-09-10T18:41:18.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.640 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.640 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216071+0000", 2024-09-10T18:41:18.640 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9100000000000002e-06 2024-09-10T18:41:18.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.640 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.640 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.241959+0000", 2024-09-10T18:41:18.640 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025887876000000001 2024-09-10T18:41:18.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.641 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.641 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:41:18.641 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.241964+0000", 2024-09-10T18:41:18.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4979999999999997e-06 2024-09-10T18:41:18.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.641 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.641 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.641 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.247389+0000", 2024-09-10T18:41:18.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0054252889999999 2024-09-10T18:41:18.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.641 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.641 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for missing object", 2024-09-10T18:41:18.641 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.247556+0000", 2024-09-10T18:41:18.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016618199999999999 2024-09-10T18:41:18.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.251327+0000", 2024-09-10T18:41:18.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0037711110000000002 2024-09-10T18:41:18.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for degraded object", 2024-09-10T18:41:18.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.251367+0000", 2024-09-10T18:41:18.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0722000000000001e-05 2024-09-10T18:41:18.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.253854+0000", 2024-09-10T18:41:18.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024868970000000001 2024-09-10T18:41:18.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.253942+0000", 2024-09-10T18:41:18.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7706000000000004e-05 2024-09-10T18:41:18.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.255713+0000", 2024-09-10T18:41:18.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0017709220000000001 2024-09-10T18:41:18.644 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.644 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.644 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.644 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.34334.0:2469 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94344803751536 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:41:18.644 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.216077+0000", 2024-09-10T18:41:18.644 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 114.336467332, 2024-09-10T18:41:18.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0427746659999999, 2024-09-10T18:41:18.644 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.644 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:41:18.644 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.644 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.34334", 2024-09-10T18:41:18.645 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/601695668", 2024-09-10T18:41:18.645 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2469 2024-09-10T18:41:18.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.645 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.645 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.645 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216077+0000", 2024-09-10T18:41:18.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.645 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.645 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216077+0000", 2024-09-10T18:41:18.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216076+0000", 2024-09-10T18:41:18.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:41:18.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216079+0000", 2024-09-10T18:41:18.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9780000000000001e-06 2024-09-10T18:41:18.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216085+0000", 2024-09-10T18:41:18.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8250000000000001e-06 2024-09-10T18:41:18.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216089+0000", 2024-09-10T18:41:18.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.189e-06 2024-09-10T18:41:18.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216134+0000", 2024-09-10T18:41:18.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4165000000000002e-05 2024-09-10T18:41:18.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-10T18:41:18.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216139+0000", 2024-09-10T18:41:18.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6330000000000001e-06 2024-09-10T18:41:18.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.225412+0000", 2024-09-10T18:41:18.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0092728069999999992 2024-09-10T18:41:18.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:41:18.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.225421+0000", 2024-09-10T18:41:18.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6340000000000007e-06 2024-09-10T18:41:18.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.251461+0000", 2024-09-10T18:41:18.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.026040659 2024-09-10T18:41:18.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for degraded object", 2024-09-10T18:41:18.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.251745+0000", 2024-09-10T18:41:18.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00028324900000000001 2024-09-10T18:41:18.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.255059+0000", 2024-09-10T18:41:18.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033147279999999999 2024-09-10T18:41:18.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.255131+0000", 2024-09-10T18:41:18.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1977999999999994e-05 2024-09-10T18:41:18.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.258852+0000", 2024-09-10T18:41:18.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0037205490000000001 2024-09-10T18:41:18.651 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.651 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.651 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.651 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253533 10.d 10:b7985f20:::benchmark_data_smithi028_161947_object116609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-10T18:41:18.651 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.220281+0000", 2024-09-10T18:41:18.651 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 114.332263176, 2024-09-10T18:41:18.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2166046609999999, 2024-09-10T18:41:18.651 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.651 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:41:18.651 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.651 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:18.652 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:18.652 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253533 2024-09-10T18:41:18.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.652 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220281+0000", 2024-09-10T18:41:18.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220281+0000", 2024-09-10T18:41:18.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220280+0000", 2024-09-10T18:41:18.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:41:18.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220296+0000", 2024-09-10T18:41:18.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5908000000000001e-05 2024-09-10T18:41:18.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220304+0000", 2024-09-10T18:41:18.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4589999999999998e-06 2024-09-10T18:41:18.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220310+0000", 2024-09-10T18:41:18.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.242e-06 2024-09-10T18:41:18.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220515+0000", 2024-09-10T18:41:18.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020526999999999999 2024-09-10T18:41:18.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220612+0000", 2024-09-10T18:41:18.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6762000000000006e-05 2024-09-10T18:41:18.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:41:18.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220715+0000", 2024-09-10T18:41:18.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103421 2024-09-10T18:41:18.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.245159+0000", 2024-09-10T18:41:18.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024444429 2024-09-10T18:41:18.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:41:18.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.245168+0000", 2024-09-10T18:41:18.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3359999999999995e-06 2024-09-10T18:41:18.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.233087+0000", 2024-09-10T18:41:18.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98791947300000005 2024-09-10T18:41:18.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:41:18.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.233100+0000", 2024-09-10T18:41:18.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2761e-05 2024-09-10T18:41:18.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.424706+0000", 2024-09-10T18:41:18.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.191606293 2024-09-10T18:41:18.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for missing object", 2024-09-10T18:41:18.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.424824+0000", 2024-09-10T18:41:18.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011730699999999999 2024-09-10T18:41:18.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.436779+0000", 2024-09-10T18:41:18.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011955782999999999 2024-09-10T18:41:18.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.436886+0000", 2024-09-10T18:41:18.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106579 2024-09-10T18:41:18.658 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.658 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.658 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.658 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253542 10.d 10:bdcb9630:::benchmark_data_smithi028_161947_object116618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:41:18.658 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.242110+0000", 2024-09-10T18:41:18.659 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 114.310434878, 2024-09-10T18:41:18.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2379634849999999, 2024-09-10T18:41:18.659 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.659 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:18.659 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.659 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:18.659 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:18.659 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253542 2024-09-10T18:41:18.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.659 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.659 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242110+0000", 2024-09-10T18:41:18.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.660 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242110+0000", 2024-09-10T18:41:18.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242108+0000", 2024-09-10T18:41:18.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:41:18.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242114+0000", 2024-09-10T18:41:18.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7910000000000001e-06 2024-09-10T18:41:18.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242123+0000", 2024-09-10T18:41:18.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1970000000000002e-06 2024-09-10T18:41:18.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242129+0000", 2024-09-10T18:41:18.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1920000000000002e-06 2024-09-10T18:41:18.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.245493+0000", 2024-09-10T18:41:18.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033640319999999999 2024-09-10T18:41:18.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:41:18.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.245498+0000", 2024-09-10T18:41:18.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6500000000000004e-06 2024-09-10T18:41:18.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.233171+0000", 2024-09-10T18:41:18.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98767313099999998 2024-09-10T18:41:18.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:41:18.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.233182+0000", 2024-09-10T18:41:18.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0536e-05 2024-09-10T18:41:18.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.424847+0000", 2024-09-10T18:41:18.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1916651730000001 2024-09-10T18:41:18.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.424963+0000", 2024-09-10T18:41:18.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011628099999999999 2024-09-10T18:41:18.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3,4", 2024-09-10T18:41:18.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.425449+0000", 2024-09-10T18:41:18.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048583300000000002 2024-09-10T18:41:18.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:18.666 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.432621+0000", 2024-09-10T18:41:18.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0071715980000000004 2024-09-10T18:41:18.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.666 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.434929+0000", 2024-09-10T18:41:18.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002308783 2024-09-10T18:41:18.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.452296+0000", 2024-09-10T18:41:18.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017366686999999999 2024-09-10T18:41:18.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.479897+0000", 2024-09-10T18:41:18.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.027600978000000002 2024-09-10T18:41:18.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:18.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.479955+0000", 2024-09-10T18:41:18.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8133000000000001e-05 2024-09-10T18:41:18.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.480073+0000", 2024-09-10T18:41:18.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011787500000000001 2024-09-10T18:41:18.668 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.668 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.668 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.668 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253555 10.b 10:d9889148:::benchmark_data_smithi028_161947_object116631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:41:18.668 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.269929+0000", 2024-09-10T18:41:18.668 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 114.28261531699999, 2024-09-10T18:41:18.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1002267880000001, 2024-09-10T18:41:18.669 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.669 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:18.669 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.669 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:18.669 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:18.669 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253555 2024-09-10T18:41:18.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.669 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.269929+0000", 2024-09-10T18:41:18.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.269929+0000", 2024-09-10T18:41:18.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.269928+0000", 2024-09-10T18:41:18.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:41:18.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.269934+0000", 2024-09-10T18:41:18.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7499999999999997e-06 2024-09-10T18:41:18.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.269944+0000", 2024-09-10T18:41:18.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.893e-06 2024-09-10T18:41:18.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.269949+0000", 2024-09-10T18:41:18.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4360000000000004e-06 2024-09-10T18:41:18.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.270000+0000", 2024-09-10T18:41:18.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1484999999999999e-05 2024-09-10T18:41:18.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:41:18.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.270008+0000", 2024-09-10T18:41:18.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5440000000000001e-06 2024-09-10T18:41:18.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.241802+0000", 2024-09-10T18:41:18.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97179455699999995 2024-09-10T18:41:18.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:41:18.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.241812+0000", 2024-09-10T18:41:18.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5510000000000005e-06 2024-09-10T18:41:18.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.347312+0000", 2024-09-10T18:41:18.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1055002890000001 2024-09-10T18:41:18.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.347439+0000", 2024-09-10T18:41:18.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000126635 2024-09-10T18:41:18.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1,4", 2024-09-10T18:41:18.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.347786+0000", 2024-09-10T18:41:18.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00034768899999999998 2024-09-10T18:41:18.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.352444+0000", 2024-09-10T18:41:18.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0046571440000000002 2024-09-10T18:41:18.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:18.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.352622+0000", 2024-09-10T18:41:18.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017846999999999999 2024-09-10T18:41:18.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.358043+0000", 2024-09-10T18:41:18.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054205030000000001 2024-09-10T18:41:18.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.369964+0000", 2024-09-10T18:41:18.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011921889999999999 2024-09-10T18:41:18.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:18.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.370051+0000", 2024-09-10T18:41:18.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6448999999999999e-05 2024-09-10T18:41:18.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.370156+0000", 2024-09-10T18:41:18.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105078 2024-09-10T18:41:18.676 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.676 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.676 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.676 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:255657 10.a 10:5bf37dbe:::benchmark_data_smithi028_161947_object118733:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-10T18:41:18.676 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:28.727522+0000", 2024-09-10T18:41:18.677 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 109.82502239999999, 2024-09-10T18:41:18.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47986652600000002, 2024-09-10T18:41:18.677 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.677 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:18.677 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.677 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:18.677 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:18.677 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 255657 2024-09-10T18:41:18.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.677 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.727522+0000", 2024-09-10T18:41:18.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.727522+0000", 2024-09-10T18:41:18.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.727520+0000", 2024-09-10T18:41:18.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:41:18.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.727527+0000", 2024-09-10T18:41:18.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4609999999999996e-06 2024-09-10T18:41:18.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.727538+0000", 2024-09-10T18:41:18.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1007e-05 2024-09-10T18:41:18.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.727544+0000", 2024-09-10T18:41:18.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4529999999999999e-06 2024-09-10T18:41:18.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.902982+0000", 2024-09-10T18:41:18.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.17543789900000001 2024-09-10T18:41:18.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.903101+0000", 2024-09-10T18:41:18.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000118615 2024-09-10T18:41:18.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,5", 2024-09-10T18:41:18.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.903224+0000", 2024-09-10T18:41:18.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012264200000000001 2024-09-10T18:41:18.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:18.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.908451+0000", 2024-09-10T18:41:18.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005227654 2024-09-10T18:41:18.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.914591+0000", 2024-09-10T18:41:18.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061396030000000004 2024-09-10T18:41:18.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.914629+0000", 2024-09-10T18:41:18.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7883000000000003e-05 2024-09-10T18:41:18.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.207043+0000", 2024-09-10T18:41:18.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29241404500000001 2024-09-10T18:41:18.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:18.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.207094+0000", 2024-09-10T18:41:18.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1329000000000003e-05 2024-09-10T18:41:18.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.207389+0000", 2024-09-10T18:41:18.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029458899999999999 2024-09-10T18:41:18.683 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.683 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.683 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.683 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:255671 10.a 10:597be30f:::benchmark_data_smithi028_161947_object118747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-10T18:41:18.683 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:28.738638+0000", 2024-09-10T18:41:18.683 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 109.81390645, 2024-09-10T18:41:18.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46912842399999999, 2024-09-10T18:41:18.683 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.683 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:18.683 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.683 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:18.684 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:18.684 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 255671 2024-09-10T18:41:18.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.684 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.738638+0000", 2024-09-10T18:41:18.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.738638+0000", 2024-09-10T18:41:18.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.738637+0000", 2024-09-10T18:41:18.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:41:18.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.738642+0000", 2024-09-10T18:41:18.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2270000000000001e-06 2024-09-10T18:41:18.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.738655+0000", 2024-09-10T18:41:18.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2257e-05 2024-09-10T18:41:18.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.738660+0000", 2024-09-10T18:41:18.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8610000000000003e-06 2024-09-10T18:41:18.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.905752+0000", 2024-09-10T18:41:18.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16709138400000001 2024-09-10T18:41:18.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.905866+0000", 2024-09-10T18:41:18.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114041 2024-09-10T18:41:18.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,5", 2024-09-10T18:41:18.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.905997+0000", 2024-09-10T18:41:18.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013100699999999999 2024-09-10T18:41:18.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:18.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.908495+0000", 2024-09-10T18:41:18.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024976680000000002 2024-09-10T18:41:18.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.914656+0000", 2024-09-10T18:41:18.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006161522 2024-09-10T18:41:18.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.206858+0000", 2024-09-10T18:41:18.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29220204 2024-09-10T18:41:18.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.207448+0000", 2024-09-10T18:41:18.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00059034500000000004 2024-09-10T18:41:18.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:18.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.207654+0000", 2024-09-10T18:41:18.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020553499999999999 2024-09-10T18:41:18.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.207767+0000", 2024-09-10T18:41:18.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011254100000000001 2024-09-10T18:41:18.692 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.692 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.692 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.692 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:255686 10.a 10:58a4d0e9:::benchmark_data_smithi028_161947_object118762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-10T18:41:18.692 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:28.760292+0000", 2024-09-10T18:41:18.692 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 109.792252859, 2024-09-10T18:41:18.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.447912906, 2024-09-10T18:41:18.692 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:41:18.693 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:41:18.693 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:41:18.693 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:41:18.693 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:41:18.693 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 255686 2024-09-10T18:41:18.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.693 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:41:18.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:41:18.693 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.760292+0000", 2024-09-10T18:41:18.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:41:18.693 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.760292+0000", 2024-09-10T18:41:18.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:41:18.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:41:18.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.760289+0000", 2024-09-10T18:41:18.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:41:18.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:41:18.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.760296+0000", 2024-09-10T18:41:18.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2189999999999998e-06 2024-09-10T18:41:18.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:41:18.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.760308+0000", 2024-09-10T18:41:18.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1430000000000001e-05 2024-09-10T18:41:18.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.695 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:41:18.695 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.760314+0000", 2024-09-10T18:41:18.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4910000000000002e-06 2024-09-10T18:41:18.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.695 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:41:18.695 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.908810+0000", 2024-09-10T18:41:18.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.14849517100000001 2024-09-10T18:41:18.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.695 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:41:18.695 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.908933+0000", 2024-09-10T18:41:18.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012294399999999999 2024-09-10T18:41:18.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,5", 2024-09-10T18:41:18.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.909057+0000", 2024-09-10T18:41:18.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012485099999999999 2024-09-10T18:41:18.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:41:18.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.914599+0000", 2024-09-10T18:41:18.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0055419099999999997 2024-09-10T18:41:18.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.914690+0000", 2024-09-10T18:41:18.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0551999999999996e-05 2024-09-10T18:41:18.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.206928+0000", 2024-09-10T18:41:18.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29223781999999998 2024-09-10T18:41:18.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:41:18.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.207823+0000", 2024-09-10T18:41:18.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00089523900000000004 2024-09-10T18:41:18.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:41:18.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.207893+0000", 2024-09-10T18:41:18.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9969999999999996e-05 2024-09-10T18:41:18.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:41:18.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:41:18.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:41:18.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.208205+0000", 2024-09-10T18:41:18.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000311708 2024-09-10T18:41:18.698 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.698 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.698 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.698 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:41:18.698 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:41:18.698 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:41:18.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:18.700 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:18.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:18.801 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:18.833 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:18.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:18.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:18.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:18.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:19.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:19.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:19.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:19.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:19.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:19.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:19.343 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 62 16 103263 103247 104.031 85.375 0.0194453 0.00960788 2024-09-10T18:41:19.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:19.397 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:19.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:19.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:19.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T18:41:19.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:19.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:19.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:19.639 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:19.672 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:19.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:19.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:19.774 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:19.804 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:19.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:19.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:19.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:19.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:20.043 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:20.077 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:20.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:20.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:20.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:20.221 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:20.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:20.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:20.323 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:20.343 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 63 16 104604 104588 103.71 83.8125 0.0106473 0.00963748 2024-09-10T18:41:20.358 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:20.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:20.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:20.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:20.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:20.585 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:20.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:20 smithi204 ceph-mon[32357]: pgmap v897: 178 pgs: 178 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 921 op/s; 54 B/s, 1 objects/s recovering 2024-09-10T18:41:20.607 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:20.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:20.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:41:20.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:20 smithi028 ceph-mon[39220]: pgmap v897: 178 pgs: 178 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 921 op/s; 54 B/s, 1 objects/s recovering 2024-09-10T18:41:20.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:20 smithi028 ceph-mon[29291]: pgmap v897: 178 pgs: 178 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 921 op/s; 54 B/s, 1 objects/s recovering 2024-09-10T18:41:20.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:20.756 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:20.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:20.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:20.897 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:20.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:20.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:20.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:21.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:21.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:21.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:21.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:21.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:21.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:21.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:21.333 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:21.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:21.343 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 64 16 105590 105574 103.053 61.625 0.00610594 0.00969921 2024-09-10T18:41:21.434 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:21.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:21.465 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:21.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:21.566 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:21.590 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:41:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:41:21] "GET /metrics HTTP/1.1" 200 224935 "" "Prometheus/2.43.0" 2024-09-10T18:41:21.605 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:21.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:21.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:21.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:21.746 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:21.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:21.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:21.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:21.883 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:21.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:21.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:21.985 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:22.014 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:22.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:22.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:22.075 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:41:22.075 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:41:22.075 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:41:22.075 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:41:22.075 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:41:22.076 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:41:22.076 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:41:22.076 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:41:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:41:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:41:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:41:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:41:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:41:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:41:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:41:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:41:22.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:22.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:22.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:22.257 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:22.290 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:22.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:22.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:22.344 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 65 16 107053 107037 102.874 91.4375 0.00505779 0.00971579 2024-09-10T18:41:22.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:22.405 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:22 smithi204 ceph-mon[32357]: pgmap v898: 178 pgs: 178 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 47 B/s, 1 objects/s recovering 2024-09-10T18:41:22.406 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:22 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1193514534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:41:22.406 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:41:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:41:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:41:22.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:22.440 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:22.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:22.512 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:22 smithi028 ceph-mon[39220]: pgmap v898: 178 pgs: 178 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 47 B/s, 1 objects/s recovering 2024-09-10T18:41:22.513 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:22 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1193514534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:41:22.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:22 smithi028 ceph-mon[29291]: pgmap v898: 178 pgs: 178 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 47 B/s, 1 objects/s recovering 2024-09-10T18:41:22.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:22 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1193514534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:41:22.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:22.582 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:22.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:22.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:22.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:22.725 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:22.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:22.827 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:22.864 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:22.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:22.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:22.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:23.003 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:23.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:23.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:23.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:23.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:23.146 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:23.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:23.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:23.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:23.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:23.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:23.344 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 66 16 108511 108495 102.696 91.125 0.00809499 0.0097331 2024-09-10T18:41:23.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:23.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:23.426 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:23.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:23.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:23.563 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:23.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:23.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:23.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:23.702 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:23.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:23.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:23.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:23.844 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:23.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:23.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:23.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:23.985 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:23.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:23.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:24.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:24.124 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:24.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:24.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:24.226 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:24.263 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:24.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:24.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:24.344 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 67 16 109769 109753 102.337 78.625 0.00321781 0.00975942 2024-09-10T18:41:24.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:24.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:24.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:24.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:24.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:24.531 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:24.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:24.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:24.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:24 smithi028 ceph-mon[39220]: pgmap v899: 178 pgs: 178 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s; 43 B/s, 0 objects/s recovering 2024-09-10T18:41:24.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:24 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:41:24.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:24 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:41:24.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:41:24.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:24 smithi028 ceph-mon[29291]: pgmap v899: 178 pgs: 178 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s; 43 B/s, 0 objects/s recovering 2024-09-10T18:41:24.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:24 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:41:24.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:24 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:41:24.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:41:24.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:24 smithi204 ceph-mon[32357]: pgmap v899: 178 pgs: 178 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s; 43 B/s, 0 objects/s recovering 2024-09-10T18:41:24.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:24 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:41:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:24 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:41:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:41:24.633 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:24.666 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:24.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:24.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:24.802 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:24.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:24.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:24.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:24.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:24.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:25.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:25.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:25.081 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:25.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:25.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:25.230 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:25.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:25.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:25.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:25.344 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 68 16 111253 111237 102.195 92.75 0.0152613 0.00978089 2024-09-10T18:41:25.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:25 smithi204 ceph-mon[32357]: osdmap e395: 8 total, 8 up, 7 in 2024-09-10T18:41:25.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:41:25 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:41:25.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:25.375 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:25.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:25.390 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:25.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:25.514 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:25.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:25.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:25.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:25.655 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:25.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:25.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:25.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:25 smithi028 ceph-mon[29291]: osdmap e395: 8 total, 8 up, 7 in 2024-09-10T18:41:25.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:25 smithi028 ceph-mon[39220]: osdmap e395: 8 total, 8 up, 7 in 2024-09-10T18:41:25.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:25.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:25.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:25.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:25.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-10T18:41:25.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:25.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:25.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:26.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:26.077 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:26.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:26.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:26.179 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:26.213 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:26.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:26.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:26.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:26.344 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 69 16 112720 112704 102.043 91.6875 0.0080878 0.00979529 2024-09-10T18:41:26.357 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:26.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:26.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:26.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:26.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:26 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:26.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:26 smithi028 ceph-mon[29291]: pgmap v901: 178 pgs: 178 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 552 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-10T18:41:26.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:26 smithi028 ceph-mon[29291]: osdmap e396: 8 total, 8 up, 7 in 2024-09-10T18:41:26.485 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:26 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:26.485 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:26 smithi028 ceph-mon[39220]: pgmap v901: 178 pgs: 178 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 552 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-10T18:41:26.485 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:26 smithi028 ceph-mon[39220]: osdmap e396: 8 total, 8 up, 7 in 2024-09-10T18:41:26.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:26.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:26.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:26 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:26 smithi204 ceph-mon[32357]: pgmap v901: 178 pgs: 178 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 552 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-10T18:41:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:26 smithi204 ceph-mon[32357]: osdmap e396: 8 total, 8 up, 7 in 2024-09-10T18:41:26.605 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:26.634 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:26.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:26.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:26.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:26.779 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:26.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:26.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:26.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:26.921 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:26.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:26.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:27.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:27.058 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:27.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:27.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:27.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:27.201 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:27.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:27.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:27.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:27.304 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:41:27.344 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 70 16 114169 114153 101.879 90.5625 0.011217 0.00981076 2024-09-10T18:41:27.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:27.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:27.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:27.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:27 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/248878136' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:41:27.360 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:27 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/248878136' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:41:27.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:41:27.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:27.484 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:27.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:27.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:27.585 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:27.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:27 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/248878136' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:41:27.617 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:27.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:27.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:27.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:27.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:27.775 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:27.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:27.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:27.914 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:27.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:27.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:28.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:28.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:28.049 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:28.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:28.150 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:28.171 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:28.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:28.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:28.273 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:28.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:28.309 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:28.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:28.345 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 71 16 115588 115572 101.693 88.6875 0.0122816 0.00982901 2024-09-10T18:41:28.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:28.448 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:28.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:28.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:28.551 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:28.586 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:28.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:28.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:28.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:28 smithi204 ceph-mon[32357]: pgmap v903: 178 pgs: 178 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-10T18:41:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:28 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:41:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:28 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/248878136' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:41:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:28 smithi204 ceph-mon[32357]: osdmap e397: 8 total, 8 up, 7 in 2024-09-10T18:41:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:28 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:41:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:28 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:41:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:28 smithi204 ceph-mon[32357]: 1.4 scrub starts 2024-09-10T18:41:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:28 smithi204 ceph-mon[32357]: 3.1f scrub starts 2024-09-10T18:41:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:28 smithi204 ceph-mon[32357]: 3.1f scrub ok 2024-09-10T18:41:28.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:28.710 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:28.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:28.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:28.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[39220]: pgmap v903: 178 pgs: 178 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-10T18:41:28.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:41:28.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/248878136' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:41:28.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[39220]: osdmap e397: 8 total, 8 up, 7 in 2024-09-10T18:41:28.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:41:28.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:41:28.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[39220]: 1.4 scrub starts 2024-09-10T18:41:28.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[39220]: 3.1f scrub starts 2024-09-10T18:41:28.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[39220]: 3.1f scrub ok 2024-09-10T18:41:28.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[29291]: pgmap v903: 178 pgs: 178 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-10T18:41:28.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:41:28.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/248878136' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:41:28.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[29291]: osdmap e397: 8 total, 8 up, 7 in 2024-09-10T18:41:28.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:41:28.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:41:28.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[29291]: 1.4 scrub starts 2024-09-10T18:41:28.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[29291]: 3.1f scrub starts 2024-09-10T18:41:28.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:28 smithi028 ceph-mon[29291]: 3.1f scrub ok 2024-09-10T18:41:28.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:28.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:28.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:28.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:28.953 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:28.993 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:28.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:28.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:29.095 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:29.128 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:29.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:29.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:29.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:29.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:29.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:29.344 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 72 16 117098 117082 101.591 94.375 0.011661 0.00983938 2024-09-10T18:41:29.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:29.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:29.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:29.514 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:29.547 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:29.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:29.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:29 smithi204 ceph-mon[32357]: 1.4 scrub ok 2024-09-10T18:41:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:29 smithi204 ceph-mon[32357]: 4.d scrub starts 2024-09-10T18:41:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:29 smithi204 ceph-mon[32357]: 1.7 scrub starts 2024-09-10T18:41:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:29 smithi204 ceph-mon[32357]: 1.7 scrub ok 2024-09-10T18:41:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:29 smithi204 ceph-mon[32357]: 4.d scrub ok 2024-09-10T18:41:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:29 smithi204 ceph-mon[32357]: osdmap e398: 8 total, 8 up, 7 in 2024-09-10T18:41:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:29 smithi204 ceph-mon[32357]: 5.7 scrub starts 2024-09-10T18:41:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:29 smithi204 ceph-mon[32357]: 5.7 scrub ok 2024-09-10T18:41:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:29 smithi204 ceph-mon[32357]: 1.14 scrub starts 2024-09-10T18:41:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:29 smithi204 ceph-mon[32357]: 1.14 scrub ok 2024-09-10T18:41:29.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[39220]: 1.4 scrub ok 2024-09-10T18:41:29.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[39220]: 4.d scrub starts 2024-09-10T18:41:29.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[39220]: 1.7 scrub starts 2024-09-10T18:41:29.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[39220]: 1.7 scrub ok 2024-09-10T18:41:29.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[39220]: 4.d scrub ok 2024-09-10T18:41:29.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[39220]: osdmap e398: 8 total, 8 up, 7 in 2024-09-10T18:41:29.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[39220]: 5.7 scrub starts 2024-09-10T18:41:29.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[39220]: 5.7 scrub ok 2024-09-10T18:41:29.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[39220]: 1.14 scrub starts 2024-09-10T18:41:29.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[39220]: 1.14 scrub ok 2024-09-10T18:41:29.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[29291]: 1.4 scrub ok 2024-09-10T18:41:29.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[29291]: 4.d scrub starts 2024-09-10T18:41:29.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[29291]: 1.7 scrub starts 2024-09-10T18:41:29.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[29291]: 1.7 scrub ok 2024-09-10T18:41:29.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[29291]: 4.d scrub ok 2024-09-10T18:41:29.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[29291]: osdmap e398: 8 total, 8 up, 7 in 2024-09-10T18:41:29.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[29291]: 5.7 scrub starts 2024-09-10T18:41:29.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[29291]: 5.7 scrub ok 2024-09-10T18:41:29.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[29291]: 1.14 scrub starts 2024-09-10T18:41:29.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:29 smithi028 ceph-mon[29291]: 1.14 scrub ok 2024-09-10T18:41:29.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:29.662 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:29.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:29.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:29.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:29.815 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:29.846 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:29.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:29.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:29.948 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:29.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:29.995 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:29.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:30.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:30.131 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:30.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:30.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:30.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:30.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:30.280 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:30.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:30.345 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 73 16 118597 118581 101.483 93.6875 0.0161864 0.00984954 2024-09-10T18:41:30.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:30.424 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:30.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:30.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:30.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:30 smithi028 ceph-mon[39220]: pgmap v906: 178 pgs: 178 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.0 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T18:41:30.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:30 smithi028 ceph-mon[39220]: 3.15 scrub starts 2024-09-10T18:41:30.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:30 smithi028 ceph-mon[39220]: 3.15 scrub ok 2024-09-10T18:41:30.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:30 smithi028 ceph-mon[39220]: 6.19 scrub starts 2024-09-10T18:41:30.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:30 smithi028 ceph-mon[39220]: 6.19 scrub ok 2024-09-10T18:41:30.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:30 smithi028 ceph-mon[39220]: 5.6 scrub starts 2024-09-10T18:41:30.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:30 smithi028 ceph-mon[39220]: 5.6 scrub ok 2024-09-10T18:41:30.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:30 smithi028 ceph-mon[29291]: pgmap v906: 178 pgs: 178 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.0 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T18:41:30.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:30 smithi028 ceph-mon[29291]: 3.15 scrub starts 2024-09-10T18:41:30.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:30 smithi028 ceph-mon[29291]: 3.15 scrub ok 2024-09-10T18:41:30.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:30 smithi028 ceph-mon[29291]: 6.19 scrub starts 2024-09-10T18:41:30.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:30 smithi028 ceph-mon[29291]: 6.19 scrub ok 2024-09-10T18:41:30.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:30 smithi028 ceph-mon[29291]: 5.6 scrub starts 2024-09-10T18:41:30.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:30 smithi028 ceph-mon[29291]: 5.6 scrub ok 2024-09-10T18:41:30.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:30.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:30.562 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:30.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:30 smithi204 ceph-mon[32357]: pgmap v906: 178 pgs: 178 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.0 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T18:41:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:30 smithi204 ceph-mon[32357]: 3.15 scrub starts 2024-09-10T18:41:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:30 smithi204 ceph-mon[32357]: 3.15 scrub ok 2024-09-10T18:41:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:30 smithi204 ceph-mon[32357]: 6.19 scrub starts 2024-09-10T18:41:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:30 smithi204 ceph-mon[32357]: 6.19 scrub ok 2024-09-10T18:41:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:30 smithi204 ceph-mon[32357]: 5.6 scrub starts 2024-09-10T18:41:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:30 smithi204 ceph-mon[32357]: 5.6 scrub ok 2024-09-10T18:41:30.664 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:30.695 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:30.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:30.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:30.797 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:30.833 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:30.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:30.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:30.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:30.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:30.977 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:30.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:31.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:31.097 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:41:31.098 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:41:31.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:31.120 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:31.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:31.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:31.259 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:31.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:31.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:31.345 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 74 16 120053 120037 101.341 91 0.00524851 0.00986326 2024-09-10T18:41:31.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:31.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:31.397 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:31.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:31.498 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:31.503 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:41:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:41:31] "GET /metrics HTTP/1.1" 200 224955 "" "Prometheus/2.43.0" 2024-09-10T18:41:31.535 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:31.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:31.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:31.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:31.676 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:31.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:31.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:31.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:41:31.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:31.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:31.826 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:31.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:31.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:31.979 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:31.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:31.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:32.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:32.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:32.121 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:32.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:32.222 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:32.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:31 smithi028 ceph-mon[39220]: 5.15 scrub starts 2024-09-10T18:41:32.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:31 smithi028 ceph-mon[39220]: 5.15 scrub ok 2024-09-10T18:41:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:31 smithi028 ceph-mon[39220]: pgmap v907: 178 pgs: 178 active+clean; 7.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.7 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-10T18:41:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:31 smithi028 ceph-mon[39220]: 6.a scrub starts 2024-09-10T18:41:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:31 smithi028 ceph-mon[39220]: 6.a scrub ok 2024-09-10T18:41:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:31 smithi028 ceph-mon[29291]: 5.15 scrub starts 2024-09-10T18:41:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:31 smithi028 ceph-mon[29291]: 5.15 scrub ok 2024-09-10T18:41:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:31 smithi028 ceph-mon[29291]: pgmap v907: 178 pgs: 178 active+clean; 7.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.7 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-10T18:41:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:31 smithi028 ceph-mon[29291]: 6.a scrub starts 2024-09-10T18:41:32.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:31 smithi028 ceph-mon[29291]: 6.a scrub ok 2024-09-10T18:41:32.260 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:32.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:32.345 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 75 15 121536 121521 101.226 92.75 0.00839518 0.0098743 2024-09-10T18:41:32.353 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:41:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:41:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:41:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:31 smithi204 ceph-mon[32357]: 5.15 scrub starts 2024-09-10T18:41:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:31 smithi204 ceph-mon[32357]: 5.15 scrub ok 2024-09-10T18:41:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:31 smithi204 ceph-mon[32357]: pgmap v907: 178 pgs: 178 active+clean; 7.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.7 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-09-10T18:41:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:31 smithi204 ceph-mon[32357]: 6.a scrub starts 2024-09-10T18:41:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:31 smithi204 ceph-mon[32357]: 6.a scrub ok 2024-09-10T18:41:32.362 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:32.399 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:32.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:32.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:32.501 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:32.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:32.534 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:32.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:32.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:32.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-10T18:41:32.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:32.695 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:32.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:32.796 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:32.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:32.833 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:32.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:32.934 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:32.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:32.964 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:32.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:33.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:33.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:33.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:33.211 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:33.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:33.248 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:33.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:33.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:32 smithi028 ceph-mon[29291]: 4.2 scrub starts 2024-09-10T18:41:33.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:32 smithi028 ceph-mon[29291]: 4.2 scrub ok 2024-09-10T18:41:33.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:32 smithi028 ceph-mon[29291]: 6.b scrub starts 2024-09-10T18:41:33.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:32 smithi028 ceph-mon[29291]: 6.b scrub ok 2024-09-10T18:41:33.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:32 smithi028 ceph-mon[29291]: 5.b scrub starts 2024-09-10T18:41:33.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:32 smithi028 ceph-mon[29291]: 5.b scrub ok 2024-09-10T18:41:33.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:32 smithi028 ceph-mon[39220]: 4.2 scrub starts 2024-09-10T18:41:33.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:32 smithi028 ceph-mon[39220]: 4.2 scrub ok 2024-09-10T18:41:33.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:32 smithi028 ceph-mon[39220]: 6.b scrub starts 2024-09-10T18:41:33.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:32 smithi028 ceph-mon[39220]: 6.b scrub ok 2024-09-10T18:41:33.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:32 smithi028 ceph-mon[39220]: 5.b scrub starts 2024-09-10T18:41:33.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:32 smithi028 ceph-mon[39220]: 5.b scrub ok 2024-09-10T18:41:33.345 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 76 16 122992 122976 101.091 90.9375 0.00980007 0.00988759 2024-09-10T18:41:33.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:33.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:32 smithi204 ceph-mon[32357]: 4.2 scrub starts 2024-09-10T18:41:33.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:32 smithi204 ceph-mon[32357]: 4.2 scrub ok 2024-09-10T18:41:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:32 smithi204 ceph-mon[32357]: 6.b scrub starts 2024-09-10T18:41:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:32 smithi204 ceph-mon[32357]: 6.b scrub ok 2024-09-10T18:41:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:32 smithi204 ceph-mon[32357]: 5.b scrub starts 2024-09-10T18:41:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:32 smithi204 ceph-mon[32357]: 5.b scrub ok 2024-09-10T18:41:33.376 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:33.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:33.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:33.479 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:33.510 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:33.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:33.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:33.612 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:33.645 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:33.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:33.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:33.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:33.787 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:33.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:33.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:33.901 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:33.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:33.935 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:33.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:34.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:34.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:34.077 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:34.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:34.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:34.217 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:34.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:34.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:34.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:33 smithi028 ceph-mon[39220]: 4.6 scrub starts 2024-09-10T18:41:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:33 smithi028 ceph-mon[39220]: 4.6 scrub ok 2024-09-10T18:41:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:33 smithi028 ceph-mon[39220]: 11.2 scrub starts 2024-09-10T18:41:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:33 smithi028 ceph-mon[39220]: pgmap v908: 178 pgs: 178 active+clean; 7.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 704 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-10T18:41:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:33 smithi028 ceph-mon[39220]: 1.2 scrub starts 2024-09-10T18:41:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:33 smithi028 ceph-mon[39220]: 1.2 scrub ok 2024-09-10T18:41:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:33 smithi028 ceph-mon[39220]: 5.1b scrub starts 2024-09-10T18:41:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:33 smithi028 ceph-mon[39220]: 5.1b scrub ok 2024-09-10T18:41:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:33 smithi028 ceph-mon[29291]: 4.6 scrub starts 2024-09-10T18:41:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:33 smithi028 ceph-mon[29291]: 4.6 scrub ok 2024-09-10T18:41:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:33 smithi028 ceph-mon[29291]: 11.2 scrub starts 2024-09-10T18:41:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:33 smithi028 ceph-mon[29291]: pgmap v908: 178 pgs: 178 active+clean; 7.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 704 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-10T18:41:34.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:33 smithi028 ceph-mon[29291]: 1.2 scrub starts 2024-09-10T18:41:34.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:33 smithi028 ceph-mon[29291]: 1.2 scrub ok 2024-09-10T18:41:34.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:33 smithi028 ceph-mon[29291]: 5.1b scrub starts 2024-09-10T18:41:34.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:33 smithi028 ceph-mon[29291]: 5.1b scrub ok 2024-09-10T18:41:34.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:34.345 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 77 16 124395 124379 100.917 87.6875 0.014957 0.00990486 2024-09-10T18:41:34.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:33 smithi204 ceph-mon[32357]: 4.6 scrub starts 2024-09-10T18:41:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:33 smithi204 ceph-mon[32357]: 4.6 scrub ok 2024-09-10T18:41:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:33 smithi204 ceph-mon[32357]: 11.2 scrub starts 2024-09-10T18:41:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:33 smithi204 ceph-mon[32357]: pgmap v908: 178 pgs: 178 active+clean; 7.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 704 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-10T18:41:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:33 smithi204 ceph-mon[32357]: 1.2 scrub starts 2024-09-10T18:41:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:33 smithi204 ceph-mon[32357]: 1.2 scrub ok 2024-09-10T18:41:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:33 smithi204 ceph-mon[32357]: 5.1b scrub starts 2024-09-10T18:41:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:33 smithi204 ceph-mon[32357]: 5.1b scrub ok 2024-09-10T18:41:34.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:34.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:34.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:34.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:34.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:34.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:34.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:34.656 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:34.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:34.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:34.757 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:34.791 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:34.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:34.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:34.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:34.929 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:34.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:34.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:35.031 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:35.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:35.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:35.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-10T18:41:35.148 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:35.153 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:35.159 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:35.170 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:35.184 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:35.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:35.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:35.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:34 smithi028 ceph-mon[29291]: 1.13 scrub starts 2024-09-10T18:41:35.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:34 smithi028 ceph-mon[29291]: 1.13 scrub ok 2024-09-10T18:41:35.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:34 smithi028 ceph-mon[29291]: 6.1e scrub starts 2024-09-10T18:41:35.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:34 smithi028 ceph-mon[29291]: 6.1e scrub ok 2024-09-10T18:41:35.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:34 smithi028 ceph-mon[29291]: 6.7 scrub starts 2024-09-10T18:41:35.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:34 smithi028 ceph-mon[29291]: 6.7 scrub ok 2024-09-10T18:41:35.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:34 smithi028 ceph-mon[39220]: 1.13 scrub starts 2024-09-10T18:41:35.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:34 smithi028 ceph-mon[39220]: 1.13 scrub ok 2024-09-10T18:41:35.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:34 smithi028 ceph-mon[39220]: 6.1e scrub starts 2024-09-10T18:41:35.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:34 smithi028 ceph-mon[39220]: 6.1e scrub ok 2024-09-10T18:41:35.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:34 smithi028 ceph-mon[39220]: 6.7 scrub starts 2024-09-10T18:41:35.256 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:34 smithi028 ceph-mon[39220]: 6.7 scrub ok 2024-09-10T18:41:35.286 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:35.313 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:35.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:35.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:35.346 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 78 16 125594 125578 100.584 74.9375 0.0173269 0.00993748 2024-09-10T18:41:35.353 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:41:35 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:41:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:34 smithi204 ceph-mon[32357]: 1.13 scrub starts 2024-09-10T18:41:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:34 smithi204 ceph-mon[32357]: 1.13 scrub ok 2024-09-10T18:41:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:34 smithi204 ceph-mon[32357]: 6.1e scrub starts 2024-09-10T18:41:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:34 smithi204 ceph-mon[32357]: 6.1e scrub ok 2024-09-10T18:41:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:34 smithi204 ceph-mon[32357]: 6.7 scrub starts 2024-09-10T18:41:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:34 smithi204 ceph-mon[32357]: 6.7 scrub ok 2024-09-10T18:41:35.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:35.470 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:35.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:35.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:35.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:35.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:35.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:35.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:35.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:35.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:35.849 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:35.887 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:35.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:35.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:35.989 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:36.019 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:36.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:36.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:36.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:36.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:36.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:36.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:35 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:36.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:35 smithi028 ceph-mon[39220]: pgmap v909: 178 pgs: 178 active+clean; 7.4 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-10T18:41:36.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:35 smithi028 ceph-mon[39220]: 4.5 scrub starts 2024-09-10T18:41:36.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:35 smithi028 ceph-mon[39220]: 4.5 scrub ok 2024-09-10T18:41:36.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:35 smithi028 ceph-mon[39220]: 6.9 scrub starts 2024-09-10T18:41:36.206 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:35 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:36.206 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:35 smithi028 ceph-mon[29291]: pgmap v909: 178 pgs: 178 active+clean; 7.4 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-10T18:41:36.206 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:35 smithi028 ceph-mon[29291]: 4.5 scrub starts 2024-09-10T18:41:36.206 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:35 smithi028 ceph-mon[29291]: 4.5 scrub ok 2024-09-10T18:41:36.206 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:35 smithi028 ceph-mon[29291]: 6.9 scrub starts 2024-09-10T18:41:36.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:36.305 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:36.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:36.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:36.345 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 79 16 126916 126900 100.356 82.625 0.0157705 0.00996018 2024-09-10T18:41:36.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:35 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:36.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:35 smithi204 ceph-mon[32357]: pgmap v909: 178 pgs: 178 active+clean; 7.4 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-10T18:41:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:35 smithi204 ceph-mon[32357]: 4.5 scrub starts 2024-09-10T18:41:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:35 smithi204 ceph-mon[32357]: 4.5 scrub ok 2024-09-10T18:41:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:35 smithi204 ceph-mon[32357]: 6.9 scrub starts 2024-09-10T18:41:36.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:36.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:36.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:36.545 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:36.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:36.581 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:36.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:36.682 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:36.717 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:36.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:36.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:36.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:36.861 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:36.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:36.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:36.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:37.002 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:37.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:37.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:37.104 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:37.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:37.153 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:37.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:37.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:36 smithi028 ceph-mon[39220]: 5.c scrub starts 2024-09-10T18:41:37.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:36 smithi028 ceph-mon[39220]: 5.c scrub ok 2024-09-10T18:41:37.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:36 smithi028 ceph-mon[39220]: 6.9 scrub ok 2024-09-10T18:41:37.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:36 smithi028 ceph-mon[39220]: 3.7 scrub starts 2024-09-10T18:41:37.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:36 smithi028 ceph-mon[39220]: 3.7 scrub ok 2024-09-10T18:41:37.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:36 smithi028 ceph-mon[39220]: 6.12 scrub starts 2024-09-10T18:41:37.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:36 smithi028 ceph-mon[39220]: 6.12 scrub ok 2024-09-10T18:41:37.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:36 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:36 smithi028 ceph-mon[29291]: 5.c scrub starts 2024-09-10T18:41:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:36 smithi028 ceph-mon[29291]: 5.c scrub ok 2024-09-10T18:41:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:36 smithi028 ceph-mon[29291]: 6.9 scrub ok 2024-09-10T18:41:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:36 smithi028 ceph-mon[29291]: 3.7 scrub starts 2024-09-10T18:41:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:36 smithi028 ceph-mon[29291]: 3.7 scrub ok 2024-09-10T18:41:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:36 smithi028 ceph-mon[29291]: 6.12 scrub starts 2024-09-10T18:41:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:36 smithi028 ceph-mon[29291]: 6.12 scrub ok 2024-09-10T18:41:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:36 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:37.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:37.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:37.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:37.346 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-09-10T18:41:37.344881+0000 min lat: 0.00125893 max lat: 0.413127 avg lat: 0.00997784 2024-09-10T18:41:37.346 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T18:41:37.346 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 80 15 128293 128278 100.178 86.125 0.0237533 0.00997784 2024-09-10T18:41:37.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:36 smithi204 ceph-mon[32357]: 5.c scrub starts 2024-09-10T18:41:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:36 smithi204 ceph-mon[32357]: 5.c scrub ok 2024-09-10T18:41:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:36 smithi204 ceph-mon[32357]: 6.9 scrub ok 2024-09-10T18:41:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:36 smithi204 ceph-mon[32357]: 3.7 scrub starts 2024-09-10T18:41:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:36 smithi204 ceph-mon[32357]: 3.7 scrub ok 2024-09-10T18:41:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:36 smithi204 ceph-mon[32357]: 6.12 scrub starts 2024-09-10T18:41:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:36 smithi204 ceph-mon[32357]: 6.12 scrub ok 2024-09-10T18:41:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:36 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:37.400 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:37.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:37.437 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:37.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:37.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:37.572 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:37.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:37.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:37.674 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:37.706 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:37.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:37.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:37.808 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:37.843 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:37.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:37.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:37.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:37.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:37.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:38.097 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:38.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:38.132 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:38.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:38.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:38.279 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:38.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:38.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:38.345 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 81 16 129639 129623 99.9795 84.0625 0.00399683 0.00999744 2024-09-10T18:41:38.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:38 smithi204 ceph-mon[32357]: 11.2 scrub ok 2024-09-10T18:41:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:38 smithi204 ceph-mon[32357]: 6.1a scrub starts 2024-09-10T18:41:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:38 smithi204 ceph-mon[32357]: 6.1a scrub ok 2024-09-10T18:41:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:38 smithi204 ceph-mon[32357]: pgmap v910: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.7 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.0 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T18:41:38.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:38 smithi028 ceph-mon[29291]: 11.2 scrub ok 2024-09-10T18:41:38.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:38 smithi028 ceph-mon[29291]: 6.1a scrub starts 2024-09-10T18:41:38.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:38 smithi028 ceph-mon[29291]: 6.1a scrub ok 2024-09-10T18:41:38.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:38 smithi028 ceph-mon[29291]: pgmap v910: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.7 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.0 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T18:41:38.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:38 smithi028 ceph-mon[39220]: 11.2 scrub ok 2024-09-10T18:41:38.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:38 smithi028 ceph-mon[39220]: 6.1a scrub starts 2024-09-10T18:41:38.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:38 smithi028 ceph-mon[39220]: 6.1a scrub ok 2024-09-10T18:41:38.365 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:38 smithi028 ceph-mon[39220]: pgmap v910: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.7 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.0 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T18:41:38.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:38.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:38.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:38.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:38.523 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:38.563 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:38.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:38.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:38.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:38.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:38.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:38.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:38.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:38.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:38.956 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:38.989 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:38.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:38.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:39.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:39.123 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:41:39.123 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:41:39.123 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[39220]: 3.19 scrub starts 2024-09-10T18:41:39.123 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[39220]: 3.19 scrub ok 2024-09-10T18:41:39.123 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[39220]: osdmap e399: 8 total, 8 up, 7 in 2024-09-10T18:41:39.123 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:39.123 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:39.123 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:41:39.123 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3410131470' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:41:39.124 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[39220]: 5.9 scrub starts 2024-09-10T18:41:39.124 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:41:39.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:41:39.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:41:39.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[29291]: 3.19 scrub starts 2024-09-10T18:41:39.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[29291]: 3.19 scrub ok 2024-09-10T18:41:39.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[29291]: osdmap e399: 8 total, 8 up, 7 in 2024-09-10T18:41:39.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:39.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:39.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:41:39.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3410131470' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:41:39.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[29291]: 5.9 scrub starts 2024-09-10T18:41:39.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:41:39.128 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:39.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:39.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:41:39.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:39.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:39.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:39.345 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 82 15 131115 131100 99.8859 92.3125 0.00631123 0.010007 2024-09-10T18:41:39.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:39 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:41:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:39 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:41:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:39 smithi204 ceph-mon[32357]: 3.19 scrub starts 2024-09-10T18:41:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:39 smithi204 ceph-mon[32357]: 3.19 scrub ok 2024-09-10T18:41:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:39 smithi204 ceph-mon[32357]: osdmap e399: 8 total, 8 up, 7 in 2024-09-10T18:41:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:39 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:41:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:39 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3410131470' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:41:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:39 smithi204 ceph-mon[32357]: 5.9 scrub starts 2024-09-10T18:41:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:41:39.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:39.407 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:39.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:39.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:39.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:39.548 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:39.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:39.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:39.650 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:39.675 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:39.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:39.681 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:39.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:39.782 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:39.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:39.815 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:39.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:39.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:39.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:39.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:40.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[39220]: 5.9 scrub ok 2024-09-10T18:41:40.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[39220]: 3.16 scrub starts 2024-09-10T18:41:40.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[39220]: 3.16 scrub ok 2024-09-10T18:41:40.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[39220]: 1.5 scrub starts 2024-09-10T18:41:40.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[39220]: 1.5 scrub ok 2024-09-10T18:41:40.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:41:40.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:41:40.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3410131470' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:41:40.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[39220]: osdmap e400: 8 total, 8 up, 7 in 2024-09-10T18:41:40.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[39220]: pgmap v913: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.7 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 639 B/s rd, 55 MiB/s wr, 878 op/s 2024-09-10T18:41:40.016 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[39220]: 3.10 scrub starts 2024-09-10T18:41:40.016 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[39220]: 3.10 scrub ok 2024-09-10T18:41:40.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[29291]: 5.9 scrub ok 2024-09-10T18:41:40.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[29291]: 3.16 scrub starts 2024-09-10T18:41:40.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[29291]: 3.16 scrub ok 2024-09-10T18:41:40.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[29291]: 1.5 scrub starts 2024-09-10T18:41:40.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[29291]: 1.5 scrub ok 2024-09-10T18:41:40.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:41:40.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:41:40.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3410131470' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:41:40.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[29291]: osdmap e400: 8 total, 8 up, 7 in 2024-09-10T18:41:40.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[29291]: pgmap v913: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.7 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 639 B/s rd, 55 MiB/s wr, 878 op/s 2024-09-10T18:41:40.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[29291]: 3.10 scrub starts 2024-09-10T18:41:40.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:40 smithi028 ceph-mon[29291]: 3.10 scrub ok 2024-09-10T18:41:40.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:40.094 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:40.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:40.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:40.196 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:40.227 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:40.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:40.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:40.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:40.347 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 83 16 132269 132253 99.5507 72.0625 0.0102149 0.0100407 2024-09-10T18:41:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:40 smithi204 ceph-mon[32357]: 5.9 scrub ok 2024-09-10T18:41:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:40 smithi204 ceph-mon[32357]: 3.16 scrub starts 2024-09-10T18:41:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:40 smithi204 ceph-mon[32357]: 3.16 scrub ok 2024-09-10T18:41:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:40 smithi204 ceph-mon[32357]: 1.5 scrub starts 2024-09-10T18:41:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:40 smithi204 ceph-mon[32357]: 1.5 scrub ok 2024-09-10T18:41:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:40 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:41:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:40 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:41:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:40 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3410131470' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:41:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:40 smithi204 ceph-mon[32357]: osdmap e400: 8 total, 8 up, 7 in 2024-09-10T18:41:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:40 smithi204 ceph-mon[32357]: pgmap v913: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.7 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 639 B/s rd, 55 MiB/s wr, 878 op/s 2024-09-10T18:41:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:40 smithi204 ceph-mon[32357]: 3.10 scrub starts 2024-09-10T18:41:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:40 smithi204 ceph-mon[32357]: 3.10 scrub ok 2024-09-10T18:41:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:40.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:40.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:40.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:40.510 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:40.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:40.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:40.658 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:40.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:40.714 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:41:40.714 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:41:40.748 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:41:40.748 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:41:40.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:40.785 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:41:40.785 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:41:40.798 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:40.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:40.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:40.822 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:41:40.822 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:41:40.867 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:41:40.868 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:41:40.898 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:41:40.898 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:41:40.900 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:40.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:40.941 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:40.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:40.949 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:41:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:41:40.984 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:41:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:41:41.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:41.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:41.075 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:41.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:41.176 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:41.209 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:41.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:41.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:41.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:41 smithi028 ceph-mon[29291]: 11.d scrub starts 2024-09-10T18:41:41.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:41 smithi028 ceph-mon[29291]: 3.f scrub starts 2024-09-10T18:41:41.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:41 smithi028 ceph-mon[29291]: 3.f scrub ok 2024-09-10T18:41:41.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:41 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3296309580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:41:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:41 smithi028 ceph-mon[39220]: 11.d scrub starts 2024-09-10T18:41:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:41 smithi028 ceph-mon[39220]: 3.f scrub starts 2024-09-10T18:41:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:41 smithi028 ceph-mon[39220]: 3.f scrub ok 2024-09-10T18:41:41.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:41 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3296309580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:41:41.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:41.346 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 84 16 133400 133384 99.2071 70.6875 0.0137744 0.0100751 2024-09-10T18:41:41.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:41 smithi204 ceph-mon[32357]: 11.d scrub starts 2024-09-10T18:41:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:41 smithi204 ceph-mon[32357]: 3.f scrub starts 2024-09-10T18:41:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:41 smithi204 ceph-mon[32357]: 3.f scrub ok 2024-09-10T18:41:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:41 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3296309580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:41:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:41.355 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:41.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:41.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:41.492 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:41.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:41.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:41.632 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:41.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:41.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:41.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:41.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:41:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:41:41] "GET /metrics HTTP/1.1" 200 224968 "" "Prometheus/2.43.0" 2024-09-10T18:41:41.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:41.774 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:41.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:41.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:41.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:41.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:42.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:42.059 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:42.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:42.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:42.202 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:42.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:42.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:42.304 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:42.338 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:42.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:42.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:42.345 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 85 16 134569 134553 98.8996 73.0625 0.00625719 0.0101062 2024-09-10T18:41:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:42 smithi204 ceph-mon[32357]: 11.6 deep-scrub starts 2024-09-10T18:41:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:42 smithi204 ceph-mon[32357]: 4.10 scrub starts 2024-09-10T18:41:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:42 smithi204 ceph-mon[32357]: 4.10 scrub ok 2024-09-10T18:41:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:42 smithi204 ceph-mon[32357]: 3.17 scrub starts 2024-09-10T18:41:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:42 smithi204 ceph-mon[32357]: 3.17 scrub ok 2024-09-10T18:41:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:42 smithi204 ceph-mon[32357]: pgmap v914: 178 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 176 active+clean; 8.1 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 33 KiB/s rd, 107 MiB/s wr, 1.76k op/s 2024-09-10T18:41:42.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:41:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:41:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:41:42.441 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:42.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:41:42.469 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:42.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:42.499 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:42 smithi028 ceph-mon[39220]: 11.6 deep-scrub starts 2024-09-10T18:41:42.499 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:42 smithi028 ceph-mon[39220]: 4.10 scrub starts 2024-09-10T18:41:42.499 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:42 smithi028 ceph-mon[39220]: 4.10 scrub ok 2024-09-10T18:41:42.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:42 smithi028 ceph-mon[39220]: 3.17 scrub starts 2024-09-10T18:41:42.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:42 smithi028 ceph-mon[39220]: 3.17 scrub ok 2024-09-10T18:41:42.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:42 smithi028 ceph-mon[39220]: pgmap v914: 178 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 176 active+clean; 8.1 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 33 KiB/s rd, 107 MiB/s wr, 1.76k op/s 2024-09-10T18:41:42.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:42 smithi028 ceph-mon[29291]: 11.6 deep-scrub starts 2024-09-10T18:41:42.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:42 smithi028 ceph-mon[29291]: 4.10 scrub starts 2024-09-10T18:41:42.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:42 smithi028 ceph-mon[29291]: 4.10 scrub ok 2024-09-10T18:41:42.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:42 smithi028 ceph-mon[29291]: 3.17 scrub starts 2024-09-10T18:41:42.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:42 smithi028 ceph-mon[29291]: 3.17 scrub ok 2024-09-10T18:41:42.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:42 smithi028 ceph-mon[29291]: pgmap v914: 178 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 176 active+clean; 8.1 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 33 KiB/s rd, 107 MiB/s wr, 1.76k op/s 2024-09-10T18:41:42.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:42.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:42.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:42.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:42.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:42.748 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:42.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:42.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:42.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:42.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:43.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:43.035 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:43.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:43.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:43.102 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:43 smithi028 ceph-mon[29291]: 1.c scrub starts 2024-09-10T18:41:43.102 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:43 smithi028 ceph-mon[29291]: 1.c scrub ok 2024-09-10T18:41:43.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:43 smithi028 ceph-mon[39220]: 1.c scrub starts 2024-09-10T18:41:43.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:43 smithi028 ceph-mon[39220]: 1.c scrub ok 2024-09-10T18:41:43.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:43.194 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:43.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:43.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:43.296 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:43.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:43.333 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:43.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:43.346 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 86 16 135786 135770 98.634 76.0625 0.00842245 0.010134 2024-09-10T18:41:43.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:43 smithi204 ceph-mon[32357]: 1.c scrub starts 2024-09-10T18:41:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:43 smithi204 ceph-mon[32357]: 1.c scrub ok 2024-09-10T18:41:43.434 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:43.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:43.467 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:43.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:43.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:43.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:43.611 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:43.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:43.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:43.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:43.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:43.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:43.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:43.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:43.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:44.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:44.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:44.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:44.162 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:44.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:44.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:44.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:44 smithi028 ceph-mon[29291]: 11.d scrub ok 2024-09-10T18:41:44.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:44 smithi028 ceph-mon[29291]: 4.1 scrub starts 2024-09-10T18:41:44.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:44 smithi028 ceph-mon[29291]: 4.1 scrub ok 2024-09-10T18:41:44.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:44 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:41:44.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:44 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:41:44.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:44 smithi028 ceph-mon[29291]: pgmap v915: 178 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 176 active+clean; 8.1 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 33 KiB/s rd, 89 MiB/s wr, 1.48k op/s 2024-09-10T18:41:44.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:44 smithi028 ceph-mon[39220]: 11.d scrub ok 2024-09-10T18:41:44.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:44 smithi028 ceph-mon[39220]: 4.1 scrub starts 2024-09-10T18:41:44.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:44 smithi028 ceph-mon[39220]: 4.1 scrub ok 2024-09-10T18:41:44.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:44 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:41:44.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:44 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:41:44.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:44 smithi028 ceph-mon[39220]: pgmap v915: 178 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 176 active+clean; 8.1 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 33 KiB/s rd, 89 MiB/s wr, 1.48k op/s 2024-09-10T18:41:44.264 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:44.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:44.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:44.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:44.346 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 87 16 137020 137004 98.3868 77.125 0.0055727 0.0101593 2024-09-10T18:41:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:44 smithi204 ceph-mon[32357]: 11.d scrub ok 2024-09-10T18:41:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:44 smithi204 ceph-mon[32357]: 4.1 scrub starts 2024-09-10T18:41:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:44 smithi204 ceph-mon[32357]: 4.1 scrub ok 2024-09-10T18:41:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:44 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:41:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:44 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:41:44.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:44 smithi204 ceph-mon[32357]: pgmap v915: 178 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 176 active+clean; 8.1 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 33 KiB/s rd, 89 MiB/s wr, 1.48k op/s 2024-09-10T18:41:44.397 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:44.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:44.431 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:44.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:44.532 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:44.563 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:44.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:44.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:44.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:44.720 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:41:44.720 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-10T18:41:44.720 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-10T18:41:44.720 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-10T18:41:44.720 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:41:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:44.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:44.828 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:44.856 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:44.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:44.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:44.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:44.989 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:44.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:44.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:45.091 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:45.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:45.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:45.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:45.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:41:45.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:45.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:45.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:45.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:45.346 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 88 16 138205 138189 98.1103 74.0625 0.0056161 0.0101859 2024-09-10T18:41:45.353 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:41:45 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:41:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:45 smithi204 ceph-mon[32357]: 1.9 scrub starts 2024-09-10T18:41:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:45 smithi204 ceph-mon[32357]: 1.9 scrub ok 2024-09-10T18:41:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:45 smithi204 ceph-mon[32357]: 4.16 scrub starts 2024-09-10T18:41:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:45 smithi204 ceph-mon[32357]: 4.16 scrub ok 2024-09-10T18:41:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:45 smithi204 ceph-mon[32357]: 5.18 scrub starts 2024-09-10T18:41:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:45 smithi204 ceph-mon[32357]: 5.18 scrub ok 2024-09-10T18:41:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:45 smithi204 ceph-mon[32357]: osdmap e401: 8 total, 8 up, 7 in 2024-09-10T18:41:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:45 smithi204 ceph-mon[32357]: 3.6 scrub starts 2024-09-10T18:41:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:45 smithi204 ceph-mon[32357]: 3.6 scrub ok 2024-09-10T18:41:45.370 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:45.403 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:45.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:45.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:45.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[29291]: 1.9 scrub starts 2024-09-10T18:41:45.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[29291]: 1.9 scrub ok 2024-09-10T18:41:45.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[29291]: 4.16 scrub starts 2024-09-10T18:41:45.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[29291]: 4.16 scrub ok 2024-09-10T18:41:45.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[29291]: 5.18 scrub starts 2024-09-10T18:41:45.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[29291]: 5.18 scrub ok 2024-09-10T18:41:45.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[29291]: osdmap e401: 8 total, 8 up, 7 in 2024-09-10T18:41:45.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[29291]: 3.6 scrub starts 2024-09-10T18:41:45.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[29291]: 3.6 scrub ok 2024-09-10T18:41:45.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[39220]: 1.9 scrub starts 2024-09-10T18:41:45.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[39220]: 1.9 scrub ok 2024-09-10T18:41:45.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[39220]: 4.16 scrub starts 2024-09-10T18:41:45.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[39220]: 4.16 scrub ok 2024-09-10T18:41:45.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[39220]: 5.18 scrub starts 2024-09-10T18:41:45.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[39220]: 5.18 scrub ok 2024-09-10T18:41:45.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[39220]: osdmap e401: 8 total, 8 up, 7 in 2024-09-10T18:41:45.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[39220]: 3.6 scrub starts 2024-09-10T18:41:45.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:45 smithi028 ceph-mon[39220]: 3.6 scrub ok 2024-09-10T18:41:45.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:45.537 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:45.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:45.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:45.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:45.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:45.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:45.778 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:45.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:45.813 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:45.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:45.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:45.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:45.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:45.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:46.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:46.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:46.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:46.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:46.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:46 smithi028 ceph-mon[29291]: 6.14 deep-scrub starts 2024-09-10T18:41:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:46 smithi028 ceph-mon[29291]: 6.14 deep-scrub ok 2024-09-10T18:41:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:46 smithi028 ceph-mon[29291]: 3.9 scrub starts 2024-09-10T18:41:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:46 smithi028 ceph-mon[29291]: 3.9 scrub ok 2024-09-10T18:41:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:46 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:46 smithi028 ceph-mon[29291]: pgmap v917: 178 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 176 active+clean; 8.2 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 51 KiB/s rd, 75 MiB/s wr, 1.28k op/s 2024-09-10T18:41:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:46 smithi028 ceph-mon[29291]: 4.f scrub starts 2024-09-10T18:41:46.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:46 smithi028 ceph-mon[29291]: 4.f scrub ok 2024-09-10T18:41:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:46 smithi028 ceph-mon[39220]: 6.14 deep-scrub starts 2024-09-10T18:41:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:46 smithi028 ceph-mon[39220]: 6.14 deep-scrub ok 2024-09-10T18:41:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:46 smithi028 ceph-mon[39220]: 3.9 scrub starts 2024-09-10T18:41:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:46 smithi028 ceph-mon[39220]: 3.9 scrub ok 2024-09-10T18:41:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:46 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:46 smithi028 ceph-mon[39220]: pgmap v917: 178 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 176 active+clean; 8.2 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 51 KiB/s rd, 75 MiB/s wr, 1.28k op/s 2024-09-10T18:41:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:46 smithi028 ceph-mon[39220]: 4.f scrub starts 2024-09-10T18:41:46.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:46 smithi028 ceph-mon[39220]: 4.f scrub ok 2024-09-10T18:41:46.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:46.256 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:46.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:46.346 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 89 16 139441 139425 97.8759 77.25 0.111666 0.0102109 2024-09-10T18:41:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:46 smithi204 ceph-mon[32357]: 6.14 deep-scrub starts 2024-09-10T18:41:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:46 smithi204 ceph-mon[32357]: 6.14 deep-scrub ok 2024-09-10T18:41:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:46 smithi204 ceph-mon[32357]: 3.9 scrub starts 2024-09-10T18:41:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:46 smithi204 ceph-mon[32357]: 3.9 scrub ok 2024-09-10T18:41:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:46 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:46 smithi204 ceph-mon[32357]: pgmap v917: 178 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 176 active+clean; 8.2 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 51 KiB/s rd, 75 MiB/s wr, 1.28k op/s 2024-09-10T18:41:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:46 smithi204 ceph-mon[32357]: 4.f scrub starts 2024-09-10T18:41:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:46 smithi204 ceph-mon[32357]: 4.f scrub ok 2024-09-10T18:41:46.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:46.401 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:46.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:46.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:46.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:46.533 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:46.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:46.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:46.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-10T18:41:46.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:46.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:46.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:46.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:46.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:46.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:46.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:46.980 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:46.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:46.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:47.082 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:47.116 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:47.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:47.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:47.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:47 smithi028 ceph-mon[29291]: 1.d scrub starts 2024-09-10T18:41:47.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:47 smithi028 ceph-mon[29291]: 1.d scrub ok 2024-09-10T18:41:47.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:47 smithi028 ceph-mon[29291]: 4.e scrub starts 2024-09-10T18:41:47.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:47 smithi028 ceph-mon[29291]: 4.e scrub ok 2024-09-10T18:41:47.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:47 smithi028 ceph-mon[39220]: 1.d scrub starts 2024-09-10T18:41:47.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:47 smithi028 ceph-mon[39220]: 1.d scrub ok 2024-09-10T18:41:47.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:47 smithi028 ceph-mon[39220]: 4.e scrub starts 2024-09-10T18:41:47.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:47 smithi028 ceph-mon[39220]: 4.e scrub ok 2024-09-10T18:41:47.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:47.257 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:47.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:47.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:47.346 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total time run: 90.0235 2024-09-10T18:41:47.346 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total writes made: 140604 2024-09-10T18:41:47.347 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Write size: 65536 2024-09-10T18:41:47.347 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object size: 65536 2024-09-10T18:41:47.347 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Bandwidth (MB/sec): 97.6161 2024-09-10T18:41:47.347 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Bandwidth: 31.3047 2024-09-10T18:41:47.347 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max bandwidth (MB/sec): 215.812 2024-09-10T18:41:47.347 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min bandwidth (MB/sec): 35 2024-09-10T18:41:47.347 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average IOPS: 1561 2024-09-10T18:41:47.347 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev IOPS: 500.875 2024-09-10T18:41:47.347 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max IOPS: 3453 2024-09-10T18:41:47.347 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min IOPS: 560 2024-09-10T18:41:47.347 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average Latency(s): 0.0102388 2024-09-10T18:41:47.347 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Latency(s): 0.0107226 2024-09-10T18:41:47.347 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max latency(s): 0.413127 2024-09-10T18:41:47.348 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min latency(s): 0.00125893 2024-09-10T18:41:47.349 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Cleaning up (deleting benchmark objects) 2024-09-10T18:41:47.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:47 smithi204 ceph-mon[32357]: 1.d scrub starts 2024-09-10T18:41:47.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:47 smithi204 ceph-mon[32357]: 1.d scrub ok 2024-09-10T18:41:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:47 smithi204 ceph-mon[32357]: 4.e scrub starts 2024-09-10T18:41:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:47 smithi204 ceph-mon[32357]: 4.e scrub ok 2024-09-10T18:41:47.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:47.398 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:47.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:47.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:47.501 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:47.534 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:47.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:47.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:47.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:47.666 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:47.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:47.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:47.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:47.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:47.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:47.909 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:47.940 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:47.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:47.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:48.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:48.069 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:48.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:48.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:48.171 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:48.206 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:48.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:48 smithi028 ceph-mon[39220]: pgmap v918: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.4 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 55 KiB/s rd, 98 MiB/s wr, 1.66k op/s 2024-09-10T18:41:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:48 smithi028 ceph-mon[39220]: 1.3 scrub starts 2024-09-10T18:41:48.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:48 smithi028 ceph-mon[39220]: 1.3 scrub ok 2024-09-10T18:41:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:48 smithi028 ceph-mon[29291]: pgmap v918: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.4 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 55 KiB/s rd, 98 MiB/s wr, 1.66k op/s 2024-09-10T18:41:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:48 smithi028 ceph-mon[29291]: 1.3 scrub starts 2024-09-10T18:41:48.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:48 smithi028 ceph-mon[29291]: 1.3 scrub ok 2024-09-10T18:41:48.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:48.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:48.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:48.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:48 smithi204 ceph-mon[32357]: pgmap v918: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.4 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 55 KiB/s rd, 98 MiB/s wr, 1.66k op/s 2024-09-10T18:41:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:48 smithi204 ceph-mon[32357]: 1.3 scrub starts 2024-09-10T18:41:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:48 smithi204 ceph-mon[32357]: 1.3 scrub ok 2024-09-10T18:41:48.444 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:48.474 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:48.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:48.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:48.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:48.614 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:48.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:48.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:48.716 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:48.754 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:48.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:48.856 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:48.884 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:48.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:48.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:48.986 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:49.014 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:49.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:49.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:49.116 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:49.152 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:49.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:49.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:49.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:49.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:49.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:49.303 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:49 smithi028 ceph-mon[39220]: 3.11 scrub starts 2024-09-10T18:41:49.303 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:49 smithi028 ceph-mon[39220]: 3.11 scrub ok 2024-09-10T18:41:49.303 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:49 smithi028 ceph-mon[39220]: 4.3 scrub starts 2024-09-10T18:41:49.303 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:49 smithi028 ceph-mon[39220]: 4.3 scrub ok 2024-09-10T18:41:49.303 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:49 smithi028 ceph-mon[39220]: 5.19 scrub starts 2024-09-10T18:41:49.303 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:49 smithi028 ceph-mon[39220]: 5.19 scrub ok 2024-09-10T18:41:49.303 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:49 smithi028 ceph-mon[29291]: 3.11 scrub starts 2024-09-10T18:41:49.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:49 smithi028 ceph-mon[29291]: 3.11 scrub ok 2024-09-10T18:41:49.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:49 smithi028 ceph-mon[29291]: 4.3 scrub starts 2024-09-10T18:41:49.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:49 smithi028 ceph-mon[29291]: 4.3 scrub ok 2024-09-10T18:41:49.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:49 smithi028 ceph-mon[29291]: 5.19 scrub starts 2024-09-10T18:41:49.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:49 smithi028 ceph-mon[29291]: 5.19 scrub ok 2024-09-10T18:41:49.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:49 smithi204 ceph-mon[32357]: 3.11 scrub starts 2024-09-10T18:41:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:49 smithi204 ceph-mon[32357]: 3.11 scrub ok 2024-09-10T18:41:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:49 smithi204 ceph-mon[32357]: 4.3 scrub starts 2024-09-10T18:41:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:49 smithi204 ceph-mon[32357]: 4.3 scrub ok 2024-09-10T18:41:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:49 smithi204 ceph-mon[32357]: 5.19 scrub starts 2024-09-10T18:41:49.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:49 smithi204 ceph-mon[32357]: 5.19 scrub ok 2024-09-10T18:41:49.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:49.444 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:49.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:49.546 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:49.576 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:49.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:49.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:49.678 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:49.710 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:49.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:49.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:49.840 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:49.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:49.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:49.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:49.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:50.000 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:50.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:41:50.102 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:50.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:50.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:50.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:50.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[39220]: 3.e scrub starts 2024-09-10T18:41:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[39220]: 3.e scrub ok 2024-09-10T18:41:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[39220]: 4.17 scrub starts 2024-09-10T18:41:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[39220]: 4.17 scrub ok 2024-09-10T18:41:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[39220]: 3.1 scrub starts 2024-09-10T18:41:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[39220]: 3.1 scrub ok 2024-09-10T18:41:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[39220]: pgmap v919: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.4 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 44 KiB/s rd, 79 MiB/s wr, 1.33k op/s 2024-09-10T18:41:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[39220]: 11.3 scrub starts 2024-09-10T18:41:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[39220]: 4.8 scrub starts 2024-09-10T18:41:50.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[39220]: 4.8 scrub ok 2024-09-10T18:41:50.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/785707323' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:41:50.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[39220]: from='client.25924 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:41:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[29291]: 3.e scrub starts 2024-09-10T18:41:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[29291]: 3.e scrub ok 2024-09-10T18:41:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[29291]: 4.17 scrub starts 2024-09-10T18:41:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[29291]: 4.17 scrub ok 2024-09-10T18:41:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[29291]: 3.1 scrub starts 2024-09-10T18:41:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[29291]: 3.1 scrub ok 2024-09-10T18:41:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[29291]: pgmap v919: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.4 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 44 KiB/s rd, 79 MiB/s wr, 1.33k op/s 2024-09-10T18:41:50.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[29291]: 11.3 scrub starts 2024-09-10T18:41:50.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[29291]: 4.8 scrub starts 2024-09-10T18:41:50.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[29291]: 4.8 scrub ok 2024-09-10T18:41:50.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/785707323' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:41:50.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:50 smithi028 ceph-mon[29291]: from='client.25924 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:41:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:50.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:50 smithi204 ceph-mon[32357]: 3.e scrub starts 2024-09-10T18:41:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:50 smithi204 ceph-mon[32357]: 3.e scrub ok 2024-09-10T18:41:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:50 smithi204 ceph-mon[32357]: 4.17 scrub starts 2024-09-10T18:41:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:50 smithi204 ceph-mon[32357]: 4.17 scrub ok 2024-09-10T18:41:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:50 smithi204 ceph-mon[32357]: 3.1 scrub starts 2024-09-10T18:41:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:50 smithi204 ceph-mon[32357]: 3.1 scrub ok 2024-09-10T18:41:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:50 smithi204 ceph-mon[32357]: pgmap v919: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.4 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 44 KiB/s rd, 79 MiB/s wr, 1.33k op/s 2024-09-10T18:41:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:50 smithi204 ceph-mon[32357]: 11.3 scrub starts 2024-09-10T18:41:50.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:50 smithi204 ceph-mon[32357]: 4.8 scrub starts 2024-09-10T18:41:50.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:50 smithi204 ceph-mon[32357]: 4.8 scrub ok 2024-09-10T18:41:50.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:50 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/785707323' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:41:50.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:50 smithi204 ceph-mon[32357]: from='client.25924 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:41:50.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:50.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:50.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:50.512 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:50.532 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:50.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:50.634 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:50.670 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:50.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:50.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:50.772 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:50.804 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:50.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:50.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:50.906 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:50.930 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:41:50.930 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:41:50.931 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_4 2024-09-10T18:41:50.931 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_4 by 10 min 8 2024-09-10T18:41:50.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:41:50.937 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:50.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:50.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:51.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:51.065 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:51.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:51.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:51.167 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:51.195 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:51.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:51.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:51 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:41:51.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:51 smithi028 ceph-mon[29291]: 4.1b scrub starts 2024-09-10T18:41:51.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:51 smithi028 ceph-mon[29291]: from='client.25924 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:41:51.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:51 smithi028 ceph-mon[29291]: osdmap e402: 8 total, 8 up, 7 in 2024-09-10T18:41:51.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:51 smithi028 ceph-mon[29291]: 4.1b scrub ok 2024-09-10T18:41:51.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:51 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:41:51.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:51 smithi028 ceph-mon[39220]: 4.1b scrub starts 2024-09-10T18:41:51.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:51 smithi028 ceph-mon[39220]: from='client.25924 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:41:51.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:51 smithi028 ceph-mon[39220]: osdmap e402: 8 total, 8 up, 7 in 2024-09-10T18:41:51.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:51 smithi028 ceph-mon[39220]: 4.1b scrub ok 2024-09-10T18:41:51.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:51.327 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:51.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:51.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:51.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:51 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:41:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:51 smithi204 ceph-mon[32357]: 4.1b scrub starts 2024-09-10T18:41:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:51 smithi204 ceph-mon[32357]: from='client.25924 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:41:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:51 smithi204 ceph-mon[32357]: osdmap e402: 8 total, 8 up, 7 in 2024-09-10T18:41:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:51 smithi204 ceph-mon[32357]: 4.1b scrub ok 2024-09-10T18:41:51.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:51.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:51.469 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:51.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:51.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:51.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:51.596 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:51.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:51.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:51.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:51.731 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:51.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:51.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:41:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:41:51] "GET /metrics HTTP/1.1" 200 224968 "" "Prometheus/2.43.0" 2024-09-10T18:41:51.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:51.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:51.856 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:51.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:51.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:51.990 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:51.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:52.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:52.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:52.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:52.228 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:52.258 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:52.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:52.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:52.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:52 smithi204 ceph-mon[32357]: pgmap v921: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.4 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 23 KiB/s rd, 65 MiB/s wr, 1.55k op/s 2024-09-10T18:41:52.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:41:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:41:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:41:52.360 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:52.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:52.382 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:52.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:52.483 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:52.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:52 smithi028 ceph-mon[29291]: pgmap v921: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.4 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 23 KiB/s rd, 65 MiB/s wr, 1.55k op/s 2024-09-10T18:41:52.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:52 smithi028 ceph-mon[39220]: pgmap v921: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.4 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 23 KiB/s rd, 65 MiB/s wr, 1.55k op/s 2024-09-10T18:41:52.517 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:52.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:52.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:52.619 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:52.650 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:52.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:52.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:52.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:52.786 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:52.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:52.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:52.888 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:52.920 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:52.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:53.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:53.065 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:53.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:53.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:53.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:53.209 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:53.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:53.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:53.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:53.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:53.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:53.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:53.489 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:53.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:53.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:53.591 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:53.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:53.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:53.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:53.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:53.753 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:53.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:53.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:53.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:53.878 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:53.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:53.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:53.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:54.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:54.016 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:54.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:54.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:54.142 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:54.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:54.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:54.243 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:54.274 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:54.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:54.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:54.398 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:54.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:54.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:54.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:54.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:54.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:54 smithi204 ceph-mon[32357]: pgmap v922: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.4 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 20 KiB/s rd, 57 MiB/s wr, 1.35k op/s 2024-09-10T18:41:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:54 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:41:54.640 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:54.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:54.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:54.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:54 smithi028 ceph-mon[39220]: pgmap v922: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.4 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 20 KiB/s rd, 57 MiB/s wr, 1.35k op/s 2024-09-10T18:41:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:54 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:41:54.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:54 smithi028 ceph-mon[29291]: pgmap v922: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.4 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 20 KiB/s rd, 57 MiB/s wr, 1.35k op/s 2024-09-10T18:41:54.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:54 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:41:54.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:54.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:54.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:54.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:54.948 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:54.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:55.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:55.086 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:55.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:55.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:55.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:55.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:55.228 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:55.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:55.329 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:55.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:55 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:41:55.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:55 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:41:55.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:41:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:41:55.361 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:55.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:55.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:55.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:55.491 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:55.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:55.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:55 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:41:55.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:55 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:41:55.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:55 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:41:55.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:55 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:41:55.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:55.620 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:55.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:55.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:55.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:55.760 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:55.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:55.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:55.897 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:55.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:55.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:55.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:56.033 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:56.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:56.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:56.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:56.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:56.174 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:56.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:56.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:56.306 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:56.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:56.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:56.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:56.430 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:56.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:56.532 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:56.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:56.567 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:56.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:56 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:56 smithi204 ceph-mon[32357]: pgmap v923: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.2 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 8.0 KiB/s rd, 40 MiB/s wr, 1.22k op/s 2024-09-10T18:41:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:56 smithi204 ceph-mon[32357]: osdmap e403: 8 total, 8 up, 7 in 2024-09-10T18:41:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:56 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:41:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:56 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:41:56.668 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:56.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:56.695 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:56.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:56.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:56 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:56.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:56 smithi028 ceph-mon[29291]: pgmap v923: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.2 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 8.0 KiB/s rd, 40 MiB/s wr, 1.22k op/s 2024-09-10T18:41:56.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:56 smithi028 ceph-mon[29291]: osdmap e403: 8 total, 8 up, 7 in 2024-09-10T18:41:56.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:56 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:41:56.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:56 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:41:56.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:56 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:41:56.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:56 smithi028 ceph-mon[39220]: pgmap v923: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.2 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 8.0 KiB/s rd, 40 MiB/s wr, 1.22k op/s 2024-09-10T18:41:56.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:56 smithi028 ceph-mon[39220]: osdmap e403: 8 total, 8 up, 7 in 2024-09-10T18:41:56.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:56 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:41:56.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:56 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:41:56.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:56.820 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:56.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:56.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:56.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:56.944 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:56.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:56.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:57.046 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:57.075 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:57.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:57.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:57.177 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:57.205 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:57.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:57.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:57.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:57.343 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:57.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:57.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:57.444 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:57.478 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:57.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:57.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:57.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:57.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:57 smithi204 ceph-mon[32357]: osdmap e404: 8 total, 8 up, 7 in 2024-09-10T18:41:57.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:57.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:57.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:57.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:57.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:57 smithi028 ceph-mon[39220]: osdmap e404: 8 total, 8 up, 7 in 2024-09-10T18:41:57.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:57 smithi028 ceph-mon[29291]: osdmap e404: 8 total, 8 up, 7 in 2024-09-10T18:41:57.756 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:57.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:57.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:57.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:57.893 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:57.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:57.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:58.015 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:58.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:58.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:58.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:58.147 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:58.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:58.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:58.249 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:58.288 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:58.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:58.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:58.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:58.425 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:58.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:58.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:58.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:41:58.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:58.557 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:58.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:41:58.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:41:58 smithi204 ceph-mon[32357]: pgmap v926: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.0 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 1.5 KiB/s rd, 15 MiB/s wr, 1.48k op/s 2024-09-10T18:41:58.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:41:58.692 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:58.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:58.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:41:58.718 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:41:58 smithi028 ceph-mon[39220]: pgmap v926: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.0 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 1.5 KiB/s rd, 15 MiB/s wr, 1.48k op/s 2024-09-10T18:41:58.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:41:58 smithi028 ceph-mon[29291]: pgmap v926: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.0 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 1.5 KiB/s rd, 15 MiB/s wr, 1.48k op/s 2024-09-10T18:41:58.757 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:58.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:58.849 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:58.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:58.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:58.989 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:58.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:58.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:59.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:41:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:59.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:41:59.234 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:41:59.265 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:59.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:59.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:41:59.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:41:59.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:59.452 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:59.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:41:59.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:41:59.580 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:41:59.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:59.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:41:59.682 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:41:59.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:59.723 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:59.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:41:59.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:41:59.867 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:41:59.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:41:59.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:41:59.970 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:00.004 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:00.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:00.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:00.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:00.135 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:00.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:00.236 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:00.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:00.266 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:00.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:00.270 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:42:00.271 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:42:00.309 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:42:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:42:00.341 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:42:00.341 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:42:00.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:00.371 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:42:00.371 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:42:00.401 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:42:00.401 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:42:00.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:00.402 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:00.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:00.430 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:42:00.430 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:42:00.464 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:42:00.465 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:42:00.495 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:42:00.496 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:42:00.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:00 smithi028 ceph-mon[39220]: pgmap v927: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.0 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 742 op/s 2024-09-10T18:42:00.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:00 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4050118389' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:42:00.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:00 smithi028 ceph-mon[29291]: pgmap v927: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.0 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 742 op/s 2024-09-10T18:42:00.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:00 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4050118389' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:42:00.504 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:00.531 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:00.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:00.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:00.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:00 smithi204 ceph-mon[32357]: pgmap v927: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 8.0 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 742 op/s 2024-09-10T18:42:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:00 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4050118389' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:42:00.633 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:00.668 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:00.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:00.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:00.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:00.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:00.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:00.908 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:00.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:00.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:00.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:01.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:01.080 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:01.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:01.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:01.182 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:01.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:01.216 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:01.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:01.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:01.354 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:01.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:01.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:01.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:42:01.395 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:42:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:42:01] "GET /metrics HTTP/1.1" 200 224982 "" "Prometheus/2.43.0" 2024-09-10T18:42:01.456 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:01.489 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:01.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:01.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:01.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:01.631 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:01.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:01.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:01.733 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:01.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:01.767 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:01.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:01.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:01.908 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:01.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:01.976 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:02.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:02.040 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:02.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:02.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:02.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:02.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:02.214 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:02.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:02.245 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:01 smithi028 ceph-mon[29291]: pgmap v928: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 7.6 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-10T18:42:02.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:01 smithi028 ceph-mon[39220]: pgmap v928: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 7.6 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-10T18:42:02.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:02.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:01 smithi204 ceph-mon[32357]: pgmap v928: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 7.6 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-10T18:42:02.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:42:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:42:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:42:02.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:02.358 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:02.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:02.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:02.496 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:02.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:02.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:02.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:02.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:02.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:02.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:02.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:02.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:02.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:02.918 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:02.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:02.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:03.020 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:03.056 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:03.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:03.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:03.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:03.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:03.181 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:03.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:03.282 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:03.315 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:03.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:03.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:03.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:03.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:03.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:03.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:03.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:03.581 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:03.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:03.683 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:03.708 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:03.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:03.733 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:03.756 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:42:03.756 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:42:03.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:03.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:03.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:03.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:03.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:03.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:04.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:42:04.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:04.137 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:04.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:04.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:04.239 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:04.272 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:04.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:04.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:04.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:04.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:04.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:04.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:04.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:04 smithi204 ceph-mon[32357]: pgmap v929: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 7.6 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-10T18:42:04.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:04 smithi028 ceph-mon[29291]: pgmap v929: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 7.6 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-10T18:42:04.653 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:04 smithi028 ceph-mon[39220]: pgmap v929: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 7.6 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-10T18:42:04.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:04.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:04.715 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:04.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:04.816 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:04.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:04.844 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:04.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:04.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:04.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:04.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:05.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:05.125 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:05.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:05.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:05.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:05.264 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:05.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:05.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:05.353 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:42:05 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:42:05.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:05.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:05.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:05.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:05.502 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:05.540 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:05.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:05.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:05.642 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:05.680 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:05.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:05.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:05.782 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:05.819 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:05.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:05.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:05.920 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:05.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T18:42:05.952 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:05.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:05.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:06.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:06.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:06.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:06.200 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:06.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:06.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:06.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:06.334 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:06.367 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:06.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:06.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:06.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:06.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:06.506 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:06.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:06 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:06 smithi204 ceph-mon[32357]: pgmap v930: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 7.5 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 515 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-10T18:42:06.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:06.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:06 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:06.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:06 smithi028 ceph-mon[39220]: pgmap v930: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 7.5 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 515 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-10T18:42:06.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:06 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:06.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:06 smithi028 ceph-mon[29291]: pgmap v930: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 7.5 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 515 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-10T18:42:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:06.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:06.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:06.755 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:06.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:06.794 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:06.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:06.895 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:06.927 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:06.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:06.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:07.029 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:07.056 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:07.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:07.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:07.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:07.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:07.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:07.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:07.318 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:07.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:07.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:07.419 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:07.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:07.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:07.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:07.551 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:07.576 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:07.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:07.647 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:42:07.651 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:42:07.677 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:07.706 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:07.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:07.807 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:07.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:07.838 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:07.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:07.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:07.978 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:07.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:07.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:08.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:08.109 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:08.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:08.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:08.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:08.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:08.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:08.350 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:42:08.378 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:08.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:08.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:08.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:08.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:08.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:08 smithi204 ceph-mon[32357]: pgmap v931: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 7.2 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 937 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T18:42:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:08 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/689822261' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:42:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:08 smithi204 ceph-mon[32357]: from='client.35981 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:42:08.621 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:08.648 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:08.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:08.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:08.751 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:08.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:08 smithi028 ceph-mon[39220]: pgmap v931: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 7.2 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 937 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T18:42:08.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:08 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/689822261' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:42:08.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:08 smithi028 ceph-mon[39220]: from='client.35981 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:42:08.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:08 smithi028 ceph-mon[29291]: pgmap v931: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 7.2 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 937 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T18:42:08.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:08 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/689822261' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:42:08.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:08 smithi028 ceph-mon[29291]: from='client.35981 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:42:08.776 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:08.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:08.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:08.781 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:08.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:08.909 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:08.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:08.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:09.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:09.040 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:09.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:09.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:09.178 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:09.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:09.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:09.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:09.325 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:09.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:09.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:09.465 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:09.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:09.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:09.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:09.601 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:09.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:09.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:09.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:09 smithi204 ceph-mon[32357]: from='client.25930 v1:172.21.15.28:0/4059418355' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:42:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:09 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:42:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:09 smithi204 ceph-mon[32357]: from='client.35981 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:42:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:09 smithi204 ceph-mon[32357]: osdmap e405: 8 total, 8 up, 7 in 2024-09-10T18:42:09.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:42:09.703 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:09.738 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:09.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:09.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:09.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:09 smithi028 ceph-mon[39220]: from='client.25930 v1:172.21.15.28:0/4059418355' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:42:09.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:09 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:42:09.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:09 smithi028 ceph-mon[39220]: from='client.35981 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:42:09.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:09 smithi028 ceph-mon[39220]: osdmap e405: 8 total, 8 up, 7 in 2024-09-10T18:42:09.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:42:09.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:09 smithi028 ceph-mon[29291]: from='client.25930 v1:172.21.15.28:0/4059418355' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:42:09.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:09 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:42:09.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:09 smithi028 ceph-mon[29291]: from='client.35981 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:42:09.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:09 smithi028 ceph-mon[29291]: osdmap e405: 8 total, 8 up, 7 in 2024-09-10T18:42:09.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:42:09.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:09.879 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:09.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:09.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:09.981 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:10.020 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:10.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:10.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:10.121 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:10.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:10.152 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:10.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:10.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:10.293 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:10.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:10.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:10.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:10.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:10.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:10.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:10.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:10.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:10 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:42:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:10 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:42:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:10 smithi204 ceph-mon[32357]: pgmap v933: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 7.2 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-10T18:42:10.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:10 smithi204 ceph-mon[32357]: osdmap e406: 8 total, 8 up, 7 in 2024-09-10T18:42:10.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:10.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:10.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:10.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:10 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:42:10.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:10 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:42:10.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:10 smithi028 ceph-mon[39220]: pgmap v933: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 7.2 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-10T18:42:10.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:10 smithi028 ceph-mon[39220]: osdmap e406: 8 total, 8 up, 7 in 2024-09-10T18:42:10.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:10 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:42:10.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:10 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:42:10.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:10 smithi028 ceph-mon[29291]: pgmap v933: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 7.2 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-10T18:42:10.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:10 smithi028 ceph-mon[29291]: osdmap e406: 8 total, 8 up, 7 in 2024-09-10T18:42:10.823 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:10.853 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:10.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:10.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:10.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:10.992 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:10.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:10.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:11.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:11.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:11.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:11.234 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:11.268 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:11.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:11.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:11.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:11.406 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:11.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:11.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:11.508 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:11.540 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:11.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:11.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:11.584 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:42:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:42:11] "GET /metrics HTTP/1.1" 200 224957 "" "Prometheus/2.43.0" 2024-09-10T18:42:11.642 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:11.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:11.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:11.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:11.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:11.817 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:11.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:11.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:11.922 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:11.960 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:11.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:11.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:12.062 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:12.093 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:12.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:12.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:12.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:12.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:12.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:12.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:11 smithi028 ceph-mon[39220]: pgmap v935: 178 pgs: 178 active+clean; 6.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-10T18:42:12.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:11 smithi028 ceph-mon[29291]: pgmap v935: 178 pgs: 178 active+clean; 6.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-10T18:42:12.333 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:12.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:42:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:42:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:42:12.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:11 smithi204 ceph-mon[32357]: pgmap v935: 178 pgs: 178 active+clean; 6.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-10T18:42:12.366 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:12.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:12.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":931,"stamp":"2024-09-10T18:42:07.200939+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7756716276,"num_objects":118546,"num_object_clones":0,"num_object_copies":355638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118546,"num_whiteouts":0,"num_read":2796,"num_read_kb":2548,"num_write":163862,"num_write_kb":8998700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":334384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67515,"ondisk_log_size":67515,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":29047164,"kb_used_data":22821380,"kb_used_omap":17,"kb_used_meta":6225646,"kb_avail":627168900,"statfs":{"total":671965249536,"available":642220953600,"internally_reserved":0,"allocated":23369093120,"data_stored":23366413400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18329,"internal_metadata":6375061607},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-824836096,"num_objects":-12586,"num_object_clones":0,"num_object_copies":-37758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12586,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12586,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.919547"},"pg_stats":[{"pgid":"1.1d","version":"76'81","reported_seq":158,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279689+0000","last_change":"2024-09-10T18:40:49.241624+0000","last_active":"2024-09-10T18:41:55.279689+0000","last_peered":"2024-09-10T18:41:55.279689+0000","last_clean":"2024-09-10T18:41:55.279689+0000","last_became_active":"2024-09-10T18:34:34.865894+0000","last_became_peered":"2024-09-10T18:34:34.865894+0000","last_unstale":"2024-09-10T18:41:55.279689+0000","last_undegraded":"2024-09-10T18:41:55.279689+0000","last_fullsized":"2024-09-10T18:41:55.279689+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:49.241512+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:40:49.241512+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:16.274244+0000","scrub_duration":0.0070814190000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":462,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279479+0000","last_change":"2024-09-10T18:41:27.836345+0000","last_active":"2024-09-10T18:41:55.279479+0000","last_peered":"2024-09-10T18:41:55.279479+0000","last_clean":"2024-09-10T18:41:55.279479+0000","last_became_active":"2024-09-10T18:39:25.247259+0000","last_became_peered":"2024-09-10T18:39:25.247259+0000","last_unstale":"2024-09-10T18:41:55.279479+0000","last_undegraded":"2024-09-10T18:41:55.279479+0000","last_fullsized":"2024-09-10T18:41:55.279479+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:27.836236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:41:27.836236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:31.756986+0000","scrub_duration":0.0039483269999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"400'25","reported_seq":485,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.272049+0000","last_change":"2024-09-10T18:41:03.025318+0000","last_active":"2024-09-10T18:41:55.272049+0000","last_peered":"2024-09-10T18:41:55.272049+0000","last_clean":"2024-09-10T18:41:55.272049+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:41:55.272049+0000","last_undegraded":"2024-09-10T18:41:55.272049+0000","last_fullsized":"2024-09-10T18:41:55.272049+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'19","last_scrub_stamp":"2024-09-10T18:41:03.025231+0000","last_deep_scrub":"342'19","last_deep_scrub_stamp":"2024-09-10T18:37:49.354781+0000","last_clean_scrub_stamp":"2024-09-10T18:41:03.025231+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:13.569929+0000","scrub_duration":0.005898276,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":469,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.278323+0000","last_change":"2024-09-10T18:41:48.436189+0000","last_active":"2024-09-10T18:41:55.278323+0000","last_peered":"2024-09-10T18:41:55.278323+0000","last_clean":"2024-09-10T18:41:55.278323+0000","last_became_active":"2024-09-10T18:39:00.910095+0000","last_became_peered":"2024-09-10T18:39:00.910095+0000","last_unstale":"2024-09-10T18:41:55.278323+0000","last_undegraded":"2024-09-10T18:41:55.278323+0000","last_fullsized":"2024-09-10T18:41:55.278323+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:48.436106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:41:48.436106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:49.205218+0000","scrub_duration":0.0049447019999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":459,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.272956+0000","last_change":"2024-09-10T18:41:37.007816+0000","last_active":"2024-09-10T18:41:55.272956+0000","last_peered":"2024-09-10T18:41:55.272956+0000","last_clean":"2024-09-10T18:41:55.272956+0000","last_became_active":"2024-09-10T18:39:00.889898+0000","last_became_peered":"2024-09-10T18:39:00.889898+0000","last_unstale":"2024-09-10T18:41:55.272956+0000","last_undegraded":"2024-09-10T18:41:55.272956+0000","last_fullsized":"2024-09-10T18:41:55.272956+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:37.007736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:41:37.007736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:46.764418+0000","scrub_duration":0.0045586869999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'81","reported_seq":146,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283477+0000","last_change":"2024-09-10T18:39:35.192518+0000","last_active":"2024-09-10T18:41:55.283477+0000","last_peered":"2024-09-10T18:41:55.283477+0000","last_clean":"2024-09-10T18:41:55.283477+0000","last_became_active":"2024-09-10T18:34:34.867388+0000","last_became_peered":"2024-09-10T18:34:34.867388+0000","last_unstale":"2024-09-10T18:41:55.283477+0000","last_undegraded":"2024-09-10T18:41:55.283477+0000","last_fullsized":"2024-09-10T18:41:55.283477+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:39:35.192426+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:39:35.192426+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013381291999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":436,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282126+0000","last_change":"2024-09-10T18:40:39.463295+0000","last_active":"2024-09-10T18:41:55.282126+0000","last_peered":"2024-09-10T18:41:55.282126+0000","last_clean":"2024-09-10T18:41:55.282126+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:41:55.282126+0000","last_undegraded":"2024-09-10T18:41:55.282126+0000","last_fullsized":"2024-09-10T18:41:55.282126+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:39.463211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:39.463211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:58.563386+0000","scrub_duration":0.0069721330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":442,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283851+0000","last_change":"2024-09-10T18:39:54.892126+0000","last_active":"2024-09-10T18:41:55.283851+0000","last_peered":"2024-09-10T18:41:55.283851+0000","last_clean":"2024-09-10T18:41:55.283851+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:41:55.283851+0000","last_undegraded":"2024-09-10T18:41:55.283851+0000","last_fullsized":"2024-09-10T18:41:55.283851+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:54.892069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:39:54.892069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001308317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"400'46","reported_seq":520,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.285512+0000","last_change":"2024-09-10T18:39:33.886353+0000","last_active":"2024-09-10T18:41:55.285512+0000","last_peered":"2024-09-10T18:41:55.285512+0000","last_clean":"2024-09-10T18:41:55.285512+0000","last_became_active":"2024-09-10T18:39:25.237997+0000","last_became_peered":"2024-09-10T18:39:25.237997+0000","last_unstale":"2024-09-10T18:41:55.285512+0000","last_undegraded":"2024-09-10T18:41:55.285512+0000","last_fullsized":"2024-09-10T18:41:55.285512+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'34","last_scrub_stamp":"2024-09-10T18:39:33.886231+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:39:33.886231+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027348149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":448,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.271971+0000","last_change":"2024-09-10T18:41:43.981927+0000","last_active":"2024-09-10T18:41:55.271971+0000","last_peered":"2024-09-10T18:41:55.271971+0000","last_clean":"2024-09-10T18:41:55.271971+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:41:55.271971+0000","last_undegraded":"2024-09-10T18:41:55.271971+0000","last_fullsized":"2024-09-10T18:41:55.271971+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:43.981864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:41:43.981864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:58.783321+0000","scrub_duration":0.010372925,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":45,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283045+0000","last_change":"2024-09-10T18:41:00.039020+0000","last_active":"2024-09-10T18:41:55.283045+0000","last_peered":"2024-09-10T18:41:55.283045+0000","last_clean":"2024-09-10T18:41:55.283045+0000","last_became_active":"2024-09-10T18:40:10.986591+0000","last_became_peered":"2024-09-10T18:40:10.986591+0000","last_unstale":"2024-09-10T18:41:55.283045+0000","last_undegraded":"2024-09-10T18:41:55.283045+0000","last_fullsized":"2024-09-10T18:41:55.283045+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:00.038936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_clean_scrub_stamp":"2024-09-10T18:41:00.038936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:10.496537+0000","scrub_duration":0.013791546999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"400'21","reported_seq":494,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.273260+0000","last_change":"2024-09-10T18:40:42.801655+0000","last_active":"2024-09-10T18:41:55.273260+0000","last_peered":"2024-09-10T18:41:55.273260+0000","last_clean":"2024-09-10T18:41:55.273260+0000","last_became_active":"2024-09-10T18:39:25.237946+0000","last_became_peered":"2024-09-10T18:39:25.237946+0000","last_unstale":"2024-09-10T18:41:55.273260+0000","last_undegraded":"2024-09-10T18:41:55.273260+0000","last_fullsized":"2024-09-10T18:41:55.273260+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:40:42.801569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:40:42.801569+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:10.303332+0000","scrub_duration":0.0054863969999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":76,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.277406+0000","last_change":"2024-09-10T18:41:33.904603+0000","last_active":"2024-09-10T18:41:55.277406+0000","last_peered":"2024-09-10T18:41:55.277406+0000","last_clean":"2024-09-10T18:41:55.277406+0000","last_became_active":"2024-09-10T18:39:25.239738+0000","last_became_peered":"2024-09-10T18:39:25.239738+0000","last_unstale":"2024-09-10T18:41:55.277406+0000","last_undegraded":"2024-09-10T18:41:55.277406+0000","last_fullsized":"2024-09-10T18:41:55.277406+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:33.904519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_clean_scrub_stamp":"2024-09-10T18:41:33.904519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:49.028595+0000","scrub_duration":0.0028331189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":65,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.278790+0000","last_change":"2024-09-10T18:39:38.022074+0000","last_active":"2024-09-10T18:41:55.278790+0000","last_peered":"2024-09-10T18:41:55.278790+0000","last_clean":"2024-09-10T18:41:55.278790+0000","last_became_active":"2024-09-10T18:39:25.255362+0000","last_became_peered":"2024-09-10T18:39:25.255362+0000","last_unstale":"2024-09-10T18:41:55.278790+0000","last_undegraded":"2024-09-10T18:41:55.278790+0000","last_fullsized":"2024-09-10T18:41:55.278790+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:38.021856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:39:38.021856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083569290000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"76'81","reported_seq":212,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283625+0000","last_change":"2024-09-10T18:39:35.857769+0000","last_active":"2024-09-10T18:41:55.283625+0000","last_peered":"2024-09-10T18:41:55.283625+0000","last_clean":"2024-09-10T18:41:55.283625+0000","last_became_active":"2024-09-10T18:34:34.863102+0000","last_became_peered":"2024-09-10T18:34:34.863102+0000","last_unstale":"2024-09-10T18:41:55.283625+0000","last_undegraded":"2024-09-10T18:41:55.283625+0000","last_fullsized":"2024-09-10T18:41:55.283625+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:39:35.857664+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:39:35.857664+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0078980279999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":66,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.277985+0000","last_change":"2024-09-10T18:39:49.974639+0000","last_active":"2024-09-10T18:41:55.277985+0000","last_peered":"2024-09-10T18:41:55.277985+0000","last_clean":"2024-09-10T18:41:55.277985+0000","last_became_active":"2024-09-10T18:39:00.915294+0000","last_became_peered":"2024-09-10T18:39:00.915294+0000","last_unstale":"2024-09-10T18:41:55.277985+0000","last_undegraded":"2024-09-10T18:41:55.277985+0000","last_fullsized":"2024-09-10T18:41:55.277985+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:49.974584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:39:49.974584+0000","last_clean_scrub_stamp":"2024-09-10T18:39:49.974584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001326858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"400'11","reported_seq":490,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.272224+0000","last_change":"2024-09-10T18:41:50.059924+0000","last_active":"2024-09-10T18:41:55.272224+0000","last_peered":"2024-09-10T18:41:55.272224+0000","last_clean":"2024-09-10T18:41:55.272224+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:41:55.272224+0000","last_undegraded":"2024-09-10T18:41:55.272224+0000","last_fullsized":"2024-09-10T18:41:55.272224+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'11","last_scrub_stamp":"2024-09-10T18:41:50.059835+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:41:50.059835+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:50.886546+0000","scrub_duration":0.0053330019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":438,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282413+0000","last_change":"2024-09-10T18:40:43.068881+0000","last_active":"2024-09-10T18:41:55.282413+0000","last_peered":"2024-09-10T18:41:55.282413+0000","last_clean":"2024-09-10T18:41:55.282413+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:41:55.282413+0000","last_undegraded":"2024-09-10T18:41:55.282413+0000","last_fullsized":"2024-09-10T18:41:55.282413+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:43.068784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:40:43.068784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037849960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":66,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.278465+0000","last_change":"2024-09-10T18:41:29.917083+0000","last_active":"2024-09-10T18:41:55.278465+0000","last_peered":"2024-09-10T18:41:55.278465+0000","last_clean":"2024-09-10T18:41:55.278465+0000","last_became_active":"2024-09-10T18:39:00.917007+0000","last_became_peered":"2024-09-10T18:39:00.917007+0000","last_unstale":"2024-09-10T18:41:55.278465+0000","last_undegraded":"2024-09-10T18:41:55.278465+0000","last_fullsized":"2024-09-10T18:41:55.278465+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:29.917010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:41:29.917010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:43.729223+0000","scrub_duration":0.0061701890000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'81","reported_seq":305,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.286279+0000","last_change":"2024-09-10T18:39:37.865324+0000","last_active":"2024-09-10T18:41:55.286279+0000","last_peered":"2024-09-10T18:41:55.286279+0000","last_clean":"2024-09-10T18:41:55.286279+0000","last_became_active":"2024-09-10T18:39:25.241941+0000","last_became_peered":"2024-09-10T18:39:25.241941+0000","last_unstale":"2024-09-10T18:41:55.286279+0000","last_undegraded":"2024-09-10T18:41:55.286279+0000","last_fullsized":"2024-09-10T18:41:55.286279+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:39:37.865240+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:39:37.865240+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038263540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":466,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.274493+0000","last_change":"2024-09-10T18:41:34.061934+0000","last_active":"2024-09-10T18:41:55.274493+0000","last_peered":"2024-09-10T18:41:55.274493+0000","last_clean":"2024-09-10T18:41:55.274493+0000","last_became_active":"2024-09-10T18:39:00.895860+0000","last_became_peered":"2024-09-10T18:39:00.895860+0000","last_unstale":"2024-09-10T18:41:55.274493+0000","last_undegraded":"2024-09-10T18:41:55.274493+0000","last_fullsized":"2024-09-10T18:41:55.274493+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:34.061855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:41:34.061855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:43.836690+0000","scrub_duration":0.019856072999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":65,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.272445+0000","last_change":"2024-09-10T18:40:34.769449+0000","last_active":"2024-09-10T18:41:55.272445+0000","last_peered":"2024-09-10T18:41:55.272445+0000","last_clean":"2024-09-10T18:41:55.272445+0000","last_became_active":"2024-09-10T18:39:25.231931+0000","last_became_peered":"2024-09-10T18:39:25.231931+0000","last_unstale":"2024-09-10T18:41:55.272445+0000","last_undegraded":"2024-09-10T18:41:55.272445+0000","last_fullsized":"2024-09-10T18:41:55.272445+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:34.769288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:40:34.769288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:58.863229+0000","scrub_duration":0.0053457419999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"400'39","reported_seq":540,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282707+0000","last_change":"2024-09-10T18:41:00.944804+0000","last_active":"2024-09-10T18:41:55.282707+0000","last_peered":"2024-09-10T18:41:55.282707+0000","last_clean":"2024-09-10T18:41:55.282707+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:41:55.282707+0000","last_undegraded":"2024-09-10T18:41:55.282707+0000","last_fullsized":"2024-09-10T18:41:55.282707+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-09-10T18:41:00.944707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:00.944707+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:20.826468+0000","scrub_duration":0.0070121890000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":459,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279169+0000","last_change":"2024-09-10T18:39:53.330296+0000","last_active":"2024-09-10T18:41:55.279169+0000","last_peered":"2024-09-10T18:41:55.279169+0000","last_clean":"2024-09-10T18:41:55.279169+0000","last_became_active":"2024-09-10T18:39:25.246541+0000","last_became_peered":"2024-09-10T18:39:25.246541+0000","last_unstale":"2024-09-10T18:41:55.279169+0000","last_undegraded":"2024-09-10T18:41:55.279169+0000","last_fullsized":"2024-09-10T18:41:55.279169+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:53.330231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:39:53.330231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020390809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'81","reported_seq":70,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.276891+0000","last_change":"2024-09-10T18:40:37.796593+0000","last_active":"2024-09-10T18:41:55.276891+0000","last_peered":"2024-09-10T18:41:55.276891+0000","last_clean":"2024-09-10T18:41:55.276891+0000","last_became_active":"2024-09-10T18:39:00.907303+0000","last_became_peered":"2024-09-10T18:39:00.907303+0000","last_unstale":"2024-09-10T18:41:55.276891+0000","last_undegraded":"2024-09-10T18:41:55.276891+0000","last_fullsized":"2024-09-10T18:41:55.276891+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:37.796463+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:40:37.796463+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048731449999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":456,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283007+0000","last_change":"2024-09-10T18:40:36.531564+0000","last_active":"2024-09-10T18:41:55.283007+0000","last_peered":"2024-09-10T18:41:55.283007+0000","last_clean":"2024-09-10T18:41:55.283007+0000","last_became_active":"2024-09-10T18:39:00.904769+0000","last_became_peered":"2024-09-10T18:39:00.904769+0000","last_unstale":"2024-09-10T18:41:55.283007+0000","last_undegraded":"2024-09-10T18:41:55.283007+0000","last_fullsized":"2024-09-10T18:41:55.283007+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:36.531474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:36.531474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:57.004056+0000","scrub_duration":0.0050165130000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":442,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.272750+0000","last_change":"2024-09-10T18:40:56.067271+0000","last_active":"2024-09-10T18:41:55.272750+0000","last_peered":"2024-09-10T18:41:55.272750+0000","last_clean":"2024-09-10T18:41:55.272750+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:41:55.272750+0000","last_undegraded":"2024-09-10T18:41:55.272750+0000","last_fullsized":"2024-09-10T18:41:55.272750+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:56.067195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:40:56.067195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:10.446556+0000","scrub_duration":0.0022978320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"400'29","reported_seq":501,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282646+0000","last_change":"2024-09-10T18:40:55.739187+0000","last_active":"2024-09-10T18:41:55.282646+0000","last_peered":"2024-09-10T18:41:55.282646+0000","last_clean":"2024-09-10T18:41:55.282646+0000","last_became_active":"2024-09-10T18:39:11.896502+0000","last_became_peered":"2024-09-10T18:39:11.896502+0000","last_unstale":"2024-09-10T18:41:55.282646+0000","last_undegraded":"2024-09-10T18:41:55.282646+0000","last_fullsized":"2024-09-10T18:41:55.282646+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'23","last_scrub_stamp":"2024-09-10T18:40:55.739083+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:40:55.739083+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:02.459032+0000","scrub_duration":0.0062258410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":444,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.273311+0000","last_change":"2024-09-10T18:40:52.999752+0000","last_active":"2024-09-10T18:41:55.273311+0000","last_peered":"2024-09-10T18:41:55.273311+0000","last_clean":"2024-09-10T18:41:55.273311+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:41:55.273311+0000","last_undegraded":"2024-09-10T18:41:55.273311+0000","last_fullsized":"2024-09-10T18:41:55.273311+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:52.999671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:08.266279+0000","last_clean_scrub_stamp":"2024-09-10T18:40:52.999671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:12.325852+0000","scrub_duration":0.0077083890000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'81","reported_seq":76,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.277344+0000","last_change":"2024-09-10T18:41:00.661030+0000","last_active":"2024-09-10T18:41:55.277344+0000","last_peered":"2024-09-10T18:41:55.277344+0000","last_clean":"2024-09-10T18:41:55.277344+0000","last_became_active":"2024-09-10T18:39:25.240472+0000","last_became_peered":"2024-09-10T18:39:25.240472+0000","last_unstale":"2024-09-10T18:41:55.277344+0000","last_undegraded":"2024-09-10T18:41:55.277344+0000","last_fullsized":"2024-09-10T18:41:55.277344+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:00.660856+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:00.660856+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:13.393804+0000","scrub_duration":0.010899265999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":470,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283276+0000","last_change":"2024-09-10T18:40:47.040541+0000","last_active":"2024-09-10T18:41:55.283276+0000","last_peered":"2024-09-10T18:41:55.283276+0000","last_clean":"2024-09-10T18:41:55.283276+0000","last_became_active":"2024-09-10T18:39:00.891321+0000","last_became_peered":"2024-09-10T18:39:00.891321+0000","last_unstale":"2024-09-10T18:41:55.283276+0000","last_undegraded":"2024-09-10T18:41:55.283276+0000","last_fullsized":"2024-09-10T18:41:55.283276+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:40:47.040456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:47.040456+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:11.288739+0000","scrub_duration":0.0042633510000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":463,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.278453+0000","last_change":"2024-09-10T18:41:37.332757+0000","last_active":"2024-09-10T18:41:55.278453+0000","last_peered":"2024-09-10T18:41:55.278453+0000","last_clean":"2024-09-10T18:41:55.278453+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:41:55.278453+0000","last_undegraded":"2024-09-10T18:41:55.278453+0000","last_fullsized":"2024-09-10T18:41:55.278453+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:37.332672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:41:37.332672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:53.883766+0000","scrub_duration":0.0099109179999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"400'35","reported_seq":353,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283331+0000","last_change":"2024-09-10T18:39:39.111380+0000","last_active":"2024-09-10T18:41:55.283331+0000","last_peered":"2024-09-10T18:41:55.283331+0000","last_clean":"2024-09-10T18:41:55.283331+0000","last_became_active":"2024-09-10T18:39:25.244435+0000","last_became_peered":"2024-09-10T18:39:25.244435+0000","last_unstale":"2024-09-10T18:41:55.283331+0000","last_undegraded":"2024-09-10T18:41:55.283331+0000","last_fullsized":"2024-09-10T18:41:55.283331+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:39:39.111295+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:39:39.111295+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003091229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"362'13","reported_seq":101,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:00.005512+0000","last_change":"2024-09-10T18:40:38.873976+0000","last_active":"2024-09-10T18:42:00.005512+0000","last_peered":"2024-09-10T18:42:00.005512+0000","last_clean":"2024-09-10T18:42:00.005512+0000","last_became_active":"2024-09-10T18:39:25.246551+0000","last_became_peered":"2024-09-10T18:39:25.246551+0000","last_unstale":"2024-09-10T18:42:00.005512+0000","last_undegraded":"2024-09-10T18:42:00.005512+0000","last_fullsized":"2024-09-10T18:42:00.005512+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'13","last_scrub_stamp":"2024-09-10T18:40:38.873884+0000","last_deep_scrub":"217'11","last_deep_scrub_stamp":"2024-09-10T18:30:03.161985+0000","last_clean_scrub_stamp":"2024-09-10T18:40:38.873884+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001632529,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"404'10087","reported_seq":10125,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:05.710413+0000","last_change":"2024-09-10T18:40:05.803585+0000","last_active":"2024-09-10T18:42:05.710413+0000","last_peered":"2024-09-10T18:42:05.710413+0000","last_clean":"2024-09-10T18:42:05.710413+0000","last_became_active":"2024-09-10T18:40:05.803308+0000","last_became_peered":"2024-09-10T18:40:05.803308+0000","last_unstale":"2024-09-10T18:42:05.710413+0000","last_undegraded":"2024-09-10T18:42:05.710413+0000","last_fullsized":"2024-09-10T18:42:05.710413+0000","mapping_epoch":374,"log_start":"394'6400","ondisk_log_start":"394'6400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":0,"log_size":3687,"log_dups_size":0,"ondisk_log_size":3687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475463680,"num_objects":7255,"num_object_clones":0,"num_object_copies":21765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10087,"num_write_kb":554944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":66,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279475+0000","last_change":"2024-09-10T18:40:38.813037+0000","last_active":"2024-09-10T18:41:55.279475+0000","last_peered":"2024-09-10T18:41:55.279475+0000","last_clean":"2024-09-10T18:41:55.279475+0000","last_became_active":"2024-09-10T18:39:00.909261+0000","last_became_peered":"2024-09-10T18:39:00.909261+0000","last_unstale":"2024-09-10T18:41:55.279475+0000","last_undegraded":"2024-09-10T18:41:55.279475+0000","last_fullsized":"2024-09-10T18:41:55.279475+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:38.812860+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:40:38.812860+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:58.388854+0000","scrub_duration":0.0050157300000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"400'46","reported_seq":532,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279560+0000","last_change":"2024-09-10T18:41:45.468203+0000","last_active":"2024-09-10T18:41:55.279560+0000","last_peered":"2024-09-10T18:41:55.279560+0000","last_clean":"2024-09-10T18:41:55.279560+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:41:55.279560+0000","last_undegraded":"2024-09-10T18:41:55.279560+0000","last_fullsized":"2024-09-10T18:41:55.279560+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'46","last_scrub_stamp":"2024-09-10T18:41:45.468076+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:41:45.468076+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:56.701934+0000","scrub_duration":0.0046622160000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":435,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283230+0000","last_change":"2024-09-10T18:39:49.902444+0000","last_active":"2024-09-10T18:41:55.283230+0000","last_peered":"2024-09-10T18:41:55.283230+0000","last_clean":"2024-09-10T18:41:55.283230+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:41:55.283230+0000","last_undegraded":"2024-09-10T18:41:55.283230+0000","last_fullsized":"2024-09-10T18:41:55.283230+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:49.902390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:39:49.902390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001638205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"362'11","reported_seq":733,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:00.016088+0000","last_change":"2024-09-10T18:39:50.576923+0000","last_active":"2024-09-10T18:42:00.016088+0000","last_peered":"2024-09-10T18:42:00.016088+0000","last_clean":"2024-09-10T18:42:00.016088+0000","last_became_active":"2024-09-10T18:39:25.239165+0000","last_became_peered":"2024-09-10T18:39:25.239165+0000","last_unstale":"2024-09-10T18:42:00.016088+0000","last_undegraded":"2024-09-10T18:42:00.016088+0000","last_fullsized":"2024-09-10T18:42:00.016088+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'11","last_scrub_stamp":"2024-09-10T18:39:50.576841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:39:50.576841+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017290719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":76,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.278208+0000","last_change":"2024-09-10T18:39:52.004349+0000","last_active":"2024-09-10T18:41:55.278208+0000","last_peered":"2024-09-10T18:41:55.278208+0000","last_clean":"2024-09-10T18:41:55.278208+0000","last_became_active":"2024-09-10T18:39:25.256173+0000","last_became_peered":"2024-09-10T18:39:25.256173+0000","last_unstale":"2024-09-10T18:41:55.278208+0000","last_undegraded":"2024-09-10T18:41:55.278208+0000","last_fullsized":"2024-09-10T18:41:55.278208+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:52.004281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:39:52.004281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001700701,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"404'10083","reported_seq":10121,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:05.174466+0000","last_change":"2024-09-10T18:40:05.807185+0000","last_active":"2024-09-10T18:42:05.174466+0000","last_peered":"2024-09-10T18:42:05.174466+0000","last_clean":"2024-09-10T18:42:05.174466+0000","last_became_active":"2024-09-10T18:40:05.806936+0000","last_became_peered":"2024-09-10T18:40:05.806936+0000","last_unstale":"2024-09-10T18:42:05.174466+0000","last_undegraded":"2024-09-10T18:42:05.174466+0000","last_fullsized":"2024-09-10T18:42:05.174466+0000","mapping_epoch":374,"log_start":"394'6400","ondisk_log_start":"394'6400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":0,"log_size":3683,"log_dups_size":0,"ondisk_log_size":3683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482017280,"num_objects":7355,"num_object_clones":0,"num_object_copies":22065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10083,"num_write_kb":558016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":378,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.273570+0000","last_change":"2024-09-10T18:41:39.072056+0000","last_active":"2024-09-10T18:41:55.273570+0000","last_peered":"2024-09-10T18:41:55.273570+0000","last_clean":"2024-09-10T18:41:55.273570+0000","last_became_active":"2024-09-10T18:34:32.815839+0000","last_became_peered":"2024-09-10T18:34:32.815839+0000","last_unstale":"2024-09-10T18:41:55.273570+0000","last_undegraded":"2024-09-10T18:41:55.273570+0000","last_fullsized":"2024-09-10T18:41:55.273570+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:39.071980+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:39.071980+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:43.710634+0000","scrub_duration":0.0086581420000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"400'42","reported_seq":529,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282641+0000","last_change":"2024-09-10T18:40:50.663742+0000","last_active":"2024-09-10T18:41:55.282641+0000","last_peered":"2024-09-10T18:41:55.282641+0000","last_clean":"2024-09-10T18:41:55.282641+0000","last_became_active":"2024-09-10T18:39:25.249915+0000","last_became_peered":"2024-09-10T18:39:25.249915+0000","last_unstale":"2024-09-10T18:41:55.282641+0000","last_undegraded":"2024-09-10T18:41:55.282641+0000","last_fullsized":"2024-09-10T18:41:55.282641+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-09-10T18:40:50.663648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:40:50.663648+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012419997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":445,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283865+0000","last_change":"2024-09-10T18:41:35.927468+0000","last_active":"2024-09-10T18:41:55.283865+0000","last_peered":"2024-09-10T18:41:55.283865+0000","last_clean":"2024-09-10T18:41:55.283865+0000","last_became_active":"2024-09-10T18:39:25.236559+0000","last_became_peered":"2024-09-10T18:39:25.236559+0000","last_unstale":"2024-09-10T18:41:55.283865+0000","last_undegraded":"2024-09-10T18:41:55.283865+0000","last_fullsized":"2024-09-10T18:41:55.283865+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:35.927388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:15.765392+0000","last_clean_scrub_stamp":"2024-09-10T18:41:35.927388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:03.513063+0000","scrub_duration":0.004755708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":438,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.272237+0000","last_change":"2024-09-10T18:41:02.057423+0000","last_active":"2024-09-10T18:41:55.272237+0000","last_peered":"2024-09-10T18:41:55.272237+0000","last_clean":"2024-09-10T18:41:55.272237+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:41:55.272237+0000","last_undegraded":"2024-09-10T18:41:55.272237+0000","last_fullsized":"2024-09-10T18:41:55.272237+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:02.057368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:41:02.057368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:25.207206+0000","scrub_duration":0.0069129869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":426,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.273195+0000","last_change":"2024-09-10T18:40:45.874251+0000","last_active":"2024-09-10T18:41:55.273195+0000","last_peered":"2024-09-10T18:41:55.273195+0000","last_clean":"2024-09-10T18:41:55.273195+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:41:55.273195+0000","last_undegraded":"2024-09-10T18:41:55.273195+0000","last_fullsized":"2024-09-10T18:41:55.273195+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:45.874178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:45.874178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:03.408315+0000","scrub_duration":0.010950827999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"404'10276","reported_seq":10312,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:05.036393+0000","last_change":"2024-09-10T18:40:05.803001+0000","last_active":"2024-09-10T18:42:05.036393+0000","last_peered":"2024-09-10T18:42:05.036393+0000","last_clean":"2024-09-10T18:42:05.036393+0000","last_became_active":"2024-09-10T18:40:05.802616+0000","last_became_peered":"2024-09-10T18:40:05.802616+0000","last_unstale":"2024-09-10T18:42:05.036393+0000","last_undegraded":"2024-09-10T18:42:05.036393+0000","last_fullsized":"2024-09-10T18:42:05.036393+0000","mapping_epoch":374,"log_start":"394'6500","ondisk_log_start":"394'6500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":0,"log_size":3776,"log_dups_size":0,"ondisk_log_size":3776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498204672,"num_objects":7602,"num_object_clones":0,"num_object_copies":22806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10276,"num_write_kb":572096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":370,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.281820+0000","last_change":"2024-09-10T18:41:27.793561+0000","last_active":"2024-09-10T18:41:55.281820+0000","last_peered":"2024-09-10T18:41:55.281820+0000","last_clean":"2024-09-10T18:41:55.281820+0000","last_became_active":"2024-09-10T18:39:00.894864+0000","last_became_peered":"2024-09-10T18:39:00.894864+0000","last_unstale":"2024-09-10T18:41:55.281820+0000","last_undegraded":"2024-09-10T18:41:55.281820+0000","last_fullsized":"2024-09-10T18:41:55.281820+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:27.793483+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:27.793483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:44.627147+0000","scrub_duration":0.0048667299999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"400'42","reported_seq":562,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.272497+0000","last_change":"2024-09-10T18:41:43.020411+0000","last_active":"2024-09-10T18:41:55.272497+0000","last_peered":"2024-09-10T18:41:55.272497+0000","last_clean":"2024-09-10T18:41:55.272497+0000","last_became_active":"2024-09-10T18:39:00.903341+0000","last_became_peered":"2024-09-10T18:39:00.903341+0000","last_unstale":"2024-09-10T18:41:55.272497+0000","last_undegraded":"2024-09-10T18:41:55.272497+0000","last_fullsized":"2024-09-10T18:41:55.272497+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'42","last_scrub_stamp":"2024-09-10T18:41:43.020357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:43.020357+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:11.106916+0000","scrub_duration":0.0024664719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":65,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283126+0000","last_change":"2024-09-10T18:41:44.493766+0000","last_active":"2024-09-10T18:41:55.283126+0000","last_peered":"2024-09-10T18:41:55.283126+0000","last_clean":"2024-09-10T18:41:55.283126+0000","last_became_active":"2024-09-10T18:39:25.238382+0000","last_became_peered":"2024-09-10T18:39:25.238382+0000","last_unstale":"2024-09-10T18:41:55.283126+0000","last_undegraded":"2024-09-10T18:41:55.283126+0000","last_fullsized":"2024-09-10T18:41:55.283126+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:44.493678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:41:44.493678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:03.841563+0000","scrub_duration":0.075205465999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":676,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:41:56.292097+0000","last_change":"2024-09-10T18:38:54.835487+0000","last_active":"2024-09-10T18:41:56.292097+0000","last_peered":"2024-09-10T18:41:56.292097+0000","last_clean":"2024-09-10T18:41:56.292097+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:41:56.292097+0000","last_undegraded":"2024-09-10T18:41:56.292097+0000","last_fullsized":"2024-09-10T18:41:56.292097+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:38:54.835389+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:38:54.835389+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013144446000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":436,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283717+0000","last_change":"2024-09-10T18:39:36.159006+0000","last_active":"2024-09-10T18:41:55.283717+0000","last_peered":"2024-09-10T18:41:55.283717+0000","last_clean":"2024-09-10T18:41:55.283717+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:41:55.283717+0000","last_undegraded":"2024-09-10T18:41:55.283717+0000","last_fullsized":"2024-09-10T18:41:55.283717+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:36.158926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:39:36.158926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082744229999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"404'10188","reported_seq":10568,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:05.787273+0000","last_change":"2024-09-10T18:41:42.164909+0000","last_active":"2024-09-10T18:42:05.787273+0000","last_peered":"2024-09-10T18:42:05.787273+0000","last_clean":"2024-09-10T18:42:05.787273+0000","last_became_active":"2024-09-10T18:40:05.804737+0000","last_became_peered":"2024-09-10T18:40:05.804737+0000","last_unstale":"2024-09-10T18:42:05.787273+0000","last_undegraded":"2024-09-10T18:42:05.787273+0000","last_fullsized":"2024-09-10T18:42:05.787273+0000","mapping_epoch":374,"log_start":"391'5637","ondisk_log_start":"391'5637","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'8392","last_scrub_stamp":"2024-09-10T18:41:42.164302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:41:42.164302+0000","objects_scrubbed":8291,"log_size":4551,"log_dups_size":0,"ondisk_log_size":4551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:50.810581+0000","scrub_duration":3.0633734330000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487391256,"num_objects":7438,"num_object_clones":0,"num_object_copies":22314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7438,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10188,"num_write_kb":563969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":449,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283532+0000","last_change":"2024-09-10T18:41:28.176389+0000","last_active":"2024-09-10T18:41:55.283532+0000","last_peered":"2024-09-10T18:41:55.283532+0000","last_clean":"2024-09-10T18:41:55.283532+0000","last_became_active":"2024-09-10T18:34:32.818323+0000","last_became_peered":"2024-09-10T18:34:32.818323+0000","last_unstale":"2024-09-10T18:41:55.283532+0000","last_undegraded":"2024-09-10T18:41:55.283532+0000","last_fullsized":"2024-09-10T18:41:55.283532+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:28.176075+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:28.176075+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:31.023041+0000","scrub_duration":0.024806948999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"400'35","reported_seq":555,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279791+0000","last_change":"2024-09-10T18:41:31.283660+0000","last_active":"2024-09-10T18:41:55.279791+0000","last_peered":"2024-09-10T18:41:55.279791+0000","last_clean":"2024-09-10T18:41:55.279791+0000","last_became_active":"2024-09-10T18:39:00.897648+0000","last_became_peered":"2024-09-10T18:39:00.897648+0000","last_unstale":"2024-09-10T18:41:55.279791+0000","last_undegraded":"2024-09-10T18:41:55.279791+0000","last_fullsized":"2024-09-10T18:41:55.279791+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:41:31.283526+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:41:31.283526+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:45.620858+0000","scrub_duration":0.0028902120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":66,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.278273+0000","last_change":"2024-09-10T18:39:50.954314+0000","last_active":"2024-09-10T18:41:55.278273+0000","last_peered":"2024-09-10T18:41:55.278273+0000","last_clean":"2024-09-10T18:41:55.278273+0000","last_became_active":"2024-09-10T18:39:00.909066+0000","last_became_peered":"2024-09-10T18:39:00.909066+0000","last_unstale":"2024-09-10T18:41:55.278273+0000","last_undegraded":"2024-09-10T18:41:55.278273+0000","last_fullsized":"2024-09-10T18:41:55.278273+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:50.954250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:39:50.954250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001791281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"362'14","reported_seq":521,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:00.005594+0000","last_change":"2024-09-10T18:40:54.997260+0000","last_active":"2024-09-10T18:42:00.005594+0000","last_peered":"2024-09-10T18:42:00.005594+0000","last_clean":"2024-09-10T18:42:00.005594+0000","last_became_active":"2024-09-10T18:39:25.251198+0000","last_became_peered":"2024-09-10T18:39:25.251198+0000","last_unstale":"2024-09-10T18:42:00.005594+0000","last_undegraded":"2024-09-10T18:42:00.005594+0000","last_fullsized":"2024-09-10T18:42:00.005594+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'14","last_scrub_stamp":"2024-09-10T18:40:54.997176+0000","last_deep_scrub":"263'11","last_deep_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_clean_scrub_stamp":"2024-09-10T18:40:54.997176+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:10.610213+0000","scrub_duration":0.0018394329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":65,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282319+0000","last_change":"2024-09-10T18:40:49.659503+0000","last_active":"2024-09-10T18:41:55.282319+0000","last_peered":"2024-09-10T18:41:55.282319+0000","last_clean":"2024-09-10T18:41:55.282319+0000","last_became_active":"2024-09-10T18:39:25.235363+0000","last_became_peered":"2024-09-10T18:39:25.235363+0000","last_unstale":"2024-09-10T18:41:55.282319+0000","last_undegraded":"2024-09-10T18:41:55.282319+0000","last_fullsized":"2024-09-10T18:41:55.282319+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:49.659414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:49.659414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002174053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"404'10023","reported_seq":10061,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:05.711101+0000","last_change":"2024-09-10T18:40:05.807212+0000","last_active":"2024-09-10T18:42:05.711101+0000","last_peered":"2024-09-10T18:42:05.711101+0000","last_clean":"2024-09-10T18:42:05.711101+0000","last_became_active":"2024-09-10T18:40:05.806863+0000","last_became_peered":"2024-09-10T18:40:05.806863+0000","last_unstale":"2024-09-10T18:42:05.711101+0000","last_undegraded":"2024-09-10T18:42:05.711101+0000","last_fullsized":"2024-09-10T18:42:05.711101+0000","mapping_epoch":374,"log_start":"394'6400","ondisk_log_start":"394'6400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":0,"log_size":3623,"log_dups_size":0,"ondisk_log_size":3623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475332608,"num_objects":7253,"num_object_clones":0,"num_object_copies":21759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10023,"num_write_kb":552832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":449,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283071+0000","last_change":"2024-09-10T18:39:53.620517+0000","last_active":"2024-09-10T18:41:55.283071+0000","last_peered":"2024-09-10T18:41:55.283071+0000","last_clean":"2024-09-10T18:41:55.283071+0000","last_became_active":"2024-09-10T18:39:25.249543+0000","last_became_peered":"2024-09-10T18:39:25.249543+0000","last_unstale":"2024-09-10T18:41:55.283071+0000","last_undegraded":"2024-09-10T18:41:55.283071+0000","last_fullsized":"2024-09-10T18:41:55.283071+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:39:53.620447+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:39:53.620447+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00162853,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"400'39","reported_seq":572,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.277224+0000","last_change":"2024-09-10T18:41:47.946062+0000","last_active":"2024-09-10T18:41:55.277224+0000","last_peered":"2024-09-10T18:41:55.277224+0000","last_clean":"2024-09-10T18:41:55.277224+0000","last_became_active":"2024-09-10T18:39:25.256454+0000","last_became_peered":"2024-09-10T18:39:25.256454+0000","last_unstale":"2024-09-10T18:41:55.277224+0000","last_undegraded":"2024-09-10T18:41:55.277224+0000","last_fullsized":"2024-09-10T18:41:55.277224+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'39","last_scrub_stamp":"2024-09-10T18:41:47.945963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:47.945963+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:08.637996+0000","scrub_duration":0.002952414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":463,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279779+0000","last_change":"2024-09-10T18:40:58.166571+0000","last_active":"2024-09-10T18:41:55.279779+0000","last_peered":"2024-09-10T18:41:55.279779+0000","last_clean":"2024-09-10T18:41:55.279779+0000","last_became_active":"2024-09-10T18:39:00.897360+0000","last_became_peered":"2024-09-10T18:39:00.897360+0000","last_unstale":"2024-09-10T18:41:55.279779+0000","last_undegraded":"2024-09-10T18:41:55.279779+0000","last_fullsized":"2024-09-10T18:41:55.279779+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:58.166192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:57.001003+0000","last_clean_scrub_stamp":"2024-09-10T18:40:58.166192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:14.961683+0000","scrub_duration":0.0021571089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":449,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282206+0000","last_change":"2024-09-10T18:40:59.194377+0000","last_active":"2024-09-10T18:41:55.282206+0000","last_peered":"2024-09-10T18:41:55.282206+0000","last_clean":"2024-09-10T18:41:55.282206+0000","last_became_active":"2024-09-10T18:39:25.243348+0000","last_became_peered":"2024-09-10T18:39:25.243348+0000","last_unstale":"2024-09-10T18:41:55.282206+0000","last_undegraded":"2024-09-10T18:41:55.282206+0000","last_fullsized":"2024-09-10T18:41:55.282206+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:59.194295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:40:59.194295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:18.797039+0000","scrub_duration":0.0051170010000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":444,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.278184+0000","last_change":"2024-09-10T18:40:44.261360+0000","last_active":"2024-09-10T18:41:55.278184+0000","last_peered":"2024-09-10T18:41:55.278184+0000","last_clean":"2024-09-10T18:41:55.278184+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:41:55.278184+0000","last_undegraded":"2024-09-10T18:41:55.278184+0000","last_fullsized":"2024-09-10T18:41:55.278184+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:44.261278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:40:44.261278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067495589999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"404'10220","reported_seq":10258,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:04.874408+0000","last_change":"2024-09-10T18:40:05.803024+0000","last_active":"2024-09-10T18:42:04.874408+0000","last_peered":"2024-09-10T18:42:04.874408+0000","last_clean":"2024-09-10T18:42:04.874408+0000","last_became_active":"2024-09-10T18:40:05.802723+0000","last_became_peered":"2024-09-10T18:40:05.802723+0000","last_unstale":"2024-09-10T18:42:04.874408+0000","last_undegraded":"2024-09-10T18:42:04.874408+0000","last_fullsized":"2024-09-10T18:42:04.874408+0000","mapping_epoch":374,"log_start":"394'6500","ondisk_log_start":"394'6500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":0,"log_size":3720,"log_dups_size":0,"ondisk_log_size":3720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491520000,"num_objects":7500,"num_object_clones":0,"num_object_copies":22500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10220,"num_write_kb":567040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":76,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.278542+0000","last_change":"2024-09-10T18:40:56.726808+0000","last_active":"2024-09-10T18:41:55.278542+0000","last_peered":"2024-09-10T18:41:55.278542+0000","last_clean":"2024-09-10T18:41:55.278542+0000","last_became_active":"2024-09-10T18:39:25.246989+0000","last_became_peered":"2024-09-10T18:39:25.246989+0000","last_unstale":"2024-09-10T18:41:55.278542+0000","last_undegraded":"2024-09-10T18:41:55.278542+0000","last_fullsized":"2024-09-10T18:41:55.278542+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:56.726734+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:40:56.726734+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:06.319972+0000","scrub_duration":0.0019705249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"400'54","reported_seq":227,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:00.006327+0000","last_change":"2024-09-10T18:39:50.224997+0000","last_active":"2024-09-10T18:42:00.006327+0000","last_peered":"2024-09-10T18:42:00.006327+0000","last_clean":"2024-09-10T18:42:00.006327+0000","last_became_active":"2024-09-10T18:36:12.922665+0000","last_became_peered":"2024-09-10T18:36:12.922665+0000","last_unstale":"2024-09-10T18:42:00.006327+0000","last_undegraded":"2024-09-10T18:42:00.006327+0000","last_fullsized":"2024-09-10T18:42:00.006327+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'42","last_scrub_stamp":"2024-09-10T18:39:50.224877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:39:50.224877+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002927163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":449,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.272606+0000","last_change":"2024-09-10T18:40:37.768851+0000","last_active":"2024-09-10T18:41:55.272606+0000","last_peered":"2024-09-10T18:41:55.272606+0000","last_clean":"2024-09-10T18:41:55.272606+0000","last_became_active":"2024-09-10T18:39:25.242740+0000","last_became_peered":"2024-09-10T18:39:25.242740+0000","last_unstale":"2024-09-10T18:41:55.272606+0000","last_undegraded":"2024-09-10T18:41:55.272606+0000","last_fullsized":"2024-09-10T18:41:55.272606+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:37.768718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.274219+0000","last_clean_scrub_stamp":"2024-09-10T18:40:37.768718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086019080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":70,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279548+0000","last_change":"2024-09-10T18:40:50.753840+0000","last_active":"2024-09-10T18:41:55.279548+0000","last_peered":"2024-09-10T18:41:55.279548+0000","last_clean":"2024-09-10T18:41:55.279548+0000","last_became_active":"2024-09-10T18:39:00.909579+0000","last_became_peered":"2024-09-10T18:39:00.909579+0000","last_unstale":"2024-09-10T18:41:55.279548+0000","last_undegraded":"2024-09-10T18:41:55.279548+0000","last_fullsized":"2024-09-10T18:41:55.279548+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:40:50.753767+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:40:50.753767+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:19.746906+0000","scrub_duration":0.0096294199999999996,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":65,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279036+0000","last_change":"2024-09-10T18:40:47.146767+0000","last_active":"2024-09-10T18:41:55.279036+0000","last_peered":"2024-09-10T18:41:55.279036+0000","last_clean":"2024-09-10T18:41:55.279036+0000","last_became_active":"2024-09-10T18:39:25.255658+0000","last_became_peered":"2024-09-10T18:39:25.255658+0000","last_unstale":"2024-09-10T18:41:55.279036+0000","last_undegraded":"2024-09-10T18:41:55.279036+0000","last_fullsized":"2024-09-10T18:41:55.279036+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:47.146673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:40:47.146673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003044597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":422,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283481+0000","last_change":"2024-09-10T18:40:34.573578+0000","last_active":"2024-09-10T18:41:55.283481+0000","last_peered":"2024-09-10T18:41:55.283481+0000","last_clean":"2024-09-10T18:41:55.283481+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:41:55.283481+0000","last_undegraded":"2024-09-10T18:41:55.283481+0000","last_fullsized":"2024-09-10T18:41:55.283481+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:34.573495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:34.573495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023351689999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"404'10223","reported_seq":10261,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:04.870419+0000","last_change":"2024-09-10T18:40:05.803578+0000","last_active":"2024-09-10T18:42:04.870419+0000","last_peered":"2024-09-10T18:42:04.870419+0000","last_clean":"2024-09-10T18:42:04.870419+0000","last_became_active":"2024-09-10T18:40:05.803269+0000","last_became_peered":"2024-09-10T18:40:05.803269+0000","last_unstale":"2024-09-10T18:42:04.870419+0000","last_undegraded":"2024-09-10T18:42:04.870419+0000","last_fullsized":"2024-09-10T18:42:04.870419+0000","mapping_epoch":374,"log_start":"394'6500","ondisk_log_start":"394'6500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":0,"log_size":3723,"log_dups_size":0,"ondisk_log_size":3723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489619456,"num_objects":7471,"num_object_clones":0,"num_object_copies":22413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10223,"num_write_kb":566208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":503,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.281798+0000","last_change":"2024-09-10T18:41:33.333442+0000","last_active":"2024-09-10T18:41:55.281798+0000","last_peered":"2024-09-10T18:41:55.281798+0000","last_clean":"2024-09-10T18:41:55.281798+0000","last_became_active":"2024-09-10T18:39:25.240048+0000","last_became_peered":"2024-09-10T18:39:25.240048+0000","last_unstale":"2024-09-10T18:41:55.281798+0000","last_undegraded":"2024-09-10T18:41:55.281798+0000","last_fullsized":"2024-09-10T18:41:55.281798+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:33.333247+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:33.333247+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:47.198960+0000","scrub_duration":0.0057228740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"400'26","reported_seq":39,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.278787+0000","last_change":"2024-09-10T18:41:11.982436+0000","last_active":"2024-09-10T18:41:55.278787+0000","last_peered":"2024-09-10T18:41:55.278787+0000","last_clean":"2024-09-10T18:41:55.278787+0000","last_became_active":"2024-09-10T18:41:11.933724+0000","last_became_peered":"2024-09-10T18:41:11.933724+0000","last_unstale":"2024-09-10T18:41:55.278787+0000","last_undegraded":"2024-09-10T18:41:55.278787+0000","last_fullsized":"2024-09-10T18:41:55.278787+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'22","last_scrub_stamp":"2024-09-10T18:41:01.248929+0000","last_deep_scrub":"388'22","last_deep_scrub_stamp":"2024-09-10T18:41:01.248929+0000","last_clean_scrub_stamp":"2024-09-10T18:41:01.248929+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:05.247203+0000","scrub_duration":0.014772172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":460,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.278033+0000","last_change":"2024-09-10T18:39:58.333861+0000","last_active":"2024-09-10T18:41:55.278033+0000","last_peered":"2024-09-10T18:41:55.278033+0000","last_clean":"2024-09-10T18:41:55.278033+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:41:55.278033+0000","last_undegraded":"2024-09-10T18:41:55.278033+0000","last_fullsized":"2024-09-10T18:41:55.278033+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:58.333803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:39:58.333803+0000","last_clean_scrub_stamp":"2024-09-10T18:39:58.333803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012716559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":494,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279006+0000","last_change":"2024-09-10T18:40:39.001349+0000","last_active":"2024-09-10T18:41:55.279006+0000","last_peered":"2024-09-10T18:41:55.279006+0000","last_clean":"2024-09-10T18:41:55.279006+0000","last_became_active":"2024-09-10T18:39:25.249566+0000","last_became_peered":"2024-09-10T18:39:25.249566+0000","last_unstale":"2024-09-10T18:41:55.279006+0000","last_undegraded":"2024-09-10T18:41:55.279006+0000","last_fullsized":"2024-09-10T18:41:55.279006+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:40:39.001140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:40:39.001140+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094934630000000006,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":487,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279464+0000","last_change":"2024-09-10T18:41:29.980154+0000","last_active":"2024-09-10T18:41:55.279464+0000","last_peered":"2024-09-10T18:41:55.279464+0000","last_clean":"2024-09-10T18:41:55.279464+0000","last_became_active":"2024-09-10T18:39:00.910574+0000","last_became_peered":"2024-09-10T18:39:00.910574+0000","last_unstale":"2024-09-10T18:41:55.279464+0000","last_undegraded":"2024-09-10T18:41:55.279464+0000","last_fullsized":"2024-09-10T18:41:55.279464+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:29.980083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:41:29.980083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:33.276045+0000","scrub_duration":0.010417921,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":436,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283390+0000","last_change":"2024-09-10T18:39:46.257970+0000","last_active":"2024-09-10T18:41:55.283390+0000","last_peered":"2024-09-10T18:41:55.283390+0000","last_clean":"2024-09-10T18:41:55.283390+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:41:55.283390+0000","last_undegraded":"2024-09-10T18:41:55.283390+0000","last_fullsized":"2024-09-10T18:41:55.283390+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:46.257902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:39:46.257902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015239800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"404'10142","reported_seq":10180,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:04.886596+0000","last_change":"2024-09-10T18:40:05.805158+0000","last_active":"2024-09-10T18:42:04.886596+0000","last_peered":"2024-09-10T18:42:04.886596+0000","last_clean":"2024-09-10T18:42:04.886596+0000","last_became_active":"2024-09-10T18:40:05.804888+0000","last_became_peered":"2024-09-10T18:40:05.804888+0000","last_unstale":"2024-09-10T18:42:04.886596+0000","last_undegraded":"2024-09-10T18:42:04.886596+0000","last_fullsized":"2024-09-10T18:42:04.886596+0000","mapping_epoch":374,"log_start":"394'6400","ondisk_log_start":"394'6400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":0,"log_size":3742,"log_dups_size":0,"ondisk_log_size":3742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488112128,"num_objects":7448,"num_object_clones":0,"num_object_copies":22344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10142,"num_write_kb":562880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":373,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279590+0000","last_change":"2024-09-10T18:41:47.469267+0000","last_active":"2024-09-10T18:41:55.279590+0000","last_peered":"2024-09-10T18:41:55.279590+0000","last_clean":"2024-09-10T18:41:55.279590+0000","last_became_active":"2024-09-10T18:34:00.165384+0000","last_became_peered":"2024-09-10T18:34:00.165384+0000","last_unstale":"2024-09-10T18:41:55.279590+0000","last_undegraded":"2024-09-10T18:41:55.279590+0000","last_fullsized":"2024-09-10T18:41:55.279590+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:47.468155+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:47.468155+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:10.750378+0000","scrub_duration":0.0041388789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"400'36","reported_seq":359,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282560+0000","last_change":"2024-09-10T18:41:32.325494+0000","last_active":"2024-09-10T18:41:55.282560+0000","last_peered":"2024-09-10T18:41:55.282560+0000","last_clean":"2024-09-10T18:41:55.282560+0000","last_became_active":"2024-09-10T18:39:25.263465+0000","last_became_peered":"2024-09-10T18:39:25.263465+0000","last_unstale":"2024-09-10T18:41:55.282560+0000","last_undegraded":"2024-09-10T18:41:55.282560+0000","last_fullsized":"2024-09-10T18:41:55.282560+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-09-10T18:41:32.325283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:32.325283+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:01.049093+0000","scrub_duration":0.0058952759999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":75,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.272611+0000","last_change":"2024-09-10T18:41:49.077680+0000","last_active":"2024-09-10T18:41:55.272611+0000","last_peered":"2024-09-10T18:41:55.272611+0000","last_clean":"2024-09-10T18:41:55.272611+0000","last_became_active":"2024-09-10T18:39:25.239068+0000","last_became_peered":"2024-09-10T18:39:25.239068+0000","last_unstale":"2024-09-10T18:41:55.272611+0000","last_undegraded":"2024-09-10T18:41:55.272611+0000","last_fullsized":"2024-09-10T18:41:55.272611+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:49.077609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:41:49.077609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:53.005098+0000","scrub_duration":0.007378697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":1688,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:03.573046+0000","last_change":"2024-09-10T18:39:57.134983+0000","last_active":"2024-09-10T18:42:03.573046+0000","last_peered":"2024-09-10T18:42:03.573046+0000","last_clean":"2024-09-10T18:42:03.573046+0000","last_became_active":"2024-09-10T18:39:25.245563+0000","last_became_peered":"2024-09-10T18:39:25.245563+0000","last_unstale":"2024-09-10T18:42:03.573046+0000","last_undegraded":"2024-09-10T18:42:03.573046+0000","last_fullsized":"2024-09-10T18:42:03.573046+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:39:57.134926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:39:57.134926+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021565859999999998,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1224,"num_read_kb":1219,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":76,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279544+0000","last_change":"2024-09-10T18:41:28.868786+0000","last_active":"2024-09-10T18:41:55.279544+0000","last_peered":"2024-09-10T18:41:55.279544+0000","last_clean":"2024-09-10T18:41:55.279544+0000","last_became_active":"2024-09-10T18:39:25.255881+0000","last_became_peered":"2024-09-10T18:39:25.255881+0000","last_unstale":"2024-09-10T18:41:55.279544+0000","last_undegraded":"2024-09-10T18:41:55.279544+0000","last_fullsized":"2024-09-10T18:41:55.279544+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:28.868225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.170272+0000","last_clean_scrub_stamp":"2024-09-10T18:41:28.868225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:52.177697+0000","scrub_duration":0.0038280509999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":457,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.278622+0000","last_change":"2024-09-10T18:40:39.235217+0000","last_active":"2024-09-10T18:41:55.278622+0000","last_peered":"2024-09-10T18:41:55.278622+0000","last_clean":"2024-09-10T18:41:55.278622+0000","last_became_active":"2024-09-10T18:39:00.897525+0000","last_became_peered":"2024-09-10T18:39:00.897525+0000","last_unstale":"2024-09-10T18:41:55.278622+0000","last_undegraded":"2024-09-10T18:41:55.278622+0000","last_fullsized":"2024-09-10T18:41:55.278622+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:39.235160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:39.235160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:05.453034+0000","scrub_duration":0.0025668739999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"404'10067","reported_seq":10103,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:04.876846+0000","last_change":"2024-09-10T18:40:05.804498+0000","last_active":"2024-09-10T18:42:04.876846+0000","last_peered":"2024-09-10T18:42:04.876846+0000","last_clean":"2024-09-10T18:42:04.876846+0000","last_became_active":"2024-09-10T18:40:05.804097+0000","last_became_peered":"2024-09-10T18:40:05.804097+0000","last_unstale":"2024-09-10T18:42:04.876846+0000","last_undegraded":"2024-09-10T18:42:04.876846+0000","last_fullsized":"2024-09-10T18:42:04.876846+0000","mapping_epoch":374,"log_start":"394'6400","ondisk_log_start":"394'6400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":0,"log_size":3667,"log_dups_size":0,"ondisk_log_size":3667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486998016,"num_objects":7431,"num_object_clones":0,"num_object_copies":22293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10067,"num_write_kb":559936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"400'38","reported_seq":464,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283971+0000","last_change":"2024-09-10T18:41:35.197885+0000","last_active":"2024-09-10T18:41:55.283971+0000","last_peered":"2024-09-10T18:41:55.283971+0000","last_clean":"2024-09-10T18:41:55.283971+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:41:55.283971+0000","last_undegraded":"2024-09-10T18:41:55.283971+0000","last_fullsized":"2024-09-10T18:41:55.283971+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'26","last_scrub_stamp":"2024-09-10T18:41:35.197788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:35.197788+0000","objects_scrubbed":7,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:38.138932+0000","scrub_duration":0.0029282409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":462,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283009+0000","last_change":"2024-09-10T18:39:48.899653+0000","last_active":"2024-09-10T18:41:55.283009+0000","last_peered":"2024-09-10T18:41:55.283009+0000","last_clean":"2024-09-10T18:41:55.283009+0000","last_became_active":"2024-09-10T18:39:00.894778+0000","last_became_peered":"2024-09-10T18:39:00.894778+0000","last_unstale":"2024-09-10T18:41:55.283009+0000","last_undegraded":"2024-09-10T18:41:55.283009+0000","last_fullsized":"2024-09-10T18:41:55.283009+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:48.899587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:39:48.899587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001397414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":512,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282908+0000","last_change":"2024-09-10T18:40:38.039120+0000","last_active":"2024-09-10T18:41:55.282908+0000","last_peered":"2024-09-10T18:41:55.282908+0000","last_clean":"2024-09-10T18:41:55.282908+0000","last_became_active":"2024-09-10T18:39:25.251895+0000","last_became_peered":"2024-09-10T18:39:25.251895+0000","last_unstale":"2024-09-10T18:41:55.282908+0000","last_undegraded":"2024-09-10T18:41:55.282908+0000","last_fullsized":"2024-09-10T18:41:55.282908+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:38.038969+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:40:38.038969+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052669589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":461,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283398+0000","last_change":"2024-09-10T18:40:34.971516+0000","last_active":"2024-09-10T18:41:55.283398+0000","last_peered":"2024-09-10T18:41:55.283398+0000","last_clean":"2024-09-10T18:41:55.283398+0000","last_became_active":"2024-09-10T18:39:00.890829+0000","last_became_peered":"2024-09-10T18:39:00.890829+0000","last_unstale":"2024-09-10T18:41:55.283398+0000","last_undegraded":"2024-09-10T18:41:55.283398+0000","last_fullsized":"2024-09-10T18:41:55.283398+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:34.971366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:40:34.971366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040825239999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":66,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279412+0000","last_change":"2024-09-10T18:41:34.883649+0000","last_active":"2024-09-10T18:41:55.279412+0000","last_peered":"2024-09-10T18:41:55.279412+0000","last_clean":"2024-09-10T18:41:55.279412+0000","last_became_active":"2024-09-10T18:39:00.900703+0000","last_became_peered":"2024-09-10T18:39:00.900703+0000","last_unstale":"2024-09-10T18:41:55.279412+0000","last_undegraded":"2024-09-10T18:41:55.279412+0000","last_fullsized":"2024-09-10T18:41:55.279412+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:34.883463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:41:34.883463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:58.255048+0000","scrub_duration":0.0041636060000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"404'10454","reported_seq":10492,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:05.611631+0000","last_change":"2024-09-10T18:40:05.807291+0000","last_active":"2024-09-10T18:42:05.611631+0000","last_peered":"2024-09-10T18:42:05.611631+0000","last_clean":"2024-09-10T18:42:05.611631+0000","last_became_active":"2024-09-10T18:40:05.807021+0000","last_became_peered":"2024-09-10T18:40:05.807021+0000","last_unstale":"2024-09-10T18:42:05.611631+0000","last_undegraded":"2024-09-10T18:42:05.611631+0000","last_fullsized":"2024-09-10T18:42:05.611631+0000","mapping_epoch":374,"log_start":"393'6000","ondisk_log_start":"393'6000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":0,"log_size":4454,"log_dups_size":0,"ondisk_log_size":4454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489291776,"num_objects":7466,"num_object_clones":0,"num_object_copies":22398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10454,"num_write_kb":573440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"76'81","reported_seq":409,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279754+0000","last_change":"2024-09-10T18:39:54.377371+0000","last_active":"2024-09-10T18:41:55.279754+0000","last_peered":"2024-09-10T18:41:55.279754+0000","last_clean":"2024-09-10T18:41:55.279754+0000","last_became_active":"2024-09-10T18:39:25.263369+0000","last_became_peered":"2024-09-10T18:39:25.263369+0000","last_unstale":"2024-09-10T18:41:55.279754+0000","last_undegraded":"2024-09-10T18:41:55.279754+0000","last_fullsized":"2024-09-10T18:41:55.279754+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:39:54.377316+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:39:54.377316+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027816300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"400'32","reported_seq":498,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.274625+0000","last_change":"2024-09-10T18:41:46.113019+0000","last_active":"2024-09-10T18:41:55.274625+0000","last_peered":"2024-09-10T18:41:55.274625+0000","last_clean":"2024-09-10T18:41:55.274625+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:41:55.274625+0000","last_undegraded":"2024-09-10T18:41:55.274625+0000","last_fullsized":"2024-09-10T18:41:55.274625+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'32","last_scrub_stamp":"2024-09-10T18:41:46.112954+0000","last_deep_scrub":"283'24","last_deep_scrub_stamp":"2024-09-10T18:34:19.404713+0000","last_clean_scrub_stamp":"2024-09-10T18:41:46.112954+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:49.602516+0000","scrub_duration":0.097283912,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":445,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.272673+0000","last_change":"2024-09-10T18:41:44.986845+0000","last_active":"2024-09-10T18:41:55.272673+0000","last_peered":"2024-09-10T18:41:55.272673+0000","last_clean":"2024-09-10T18:41:55.272673+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:41:55.272673+0000","last_undegraded":"2024-09-10T18:41:55.272673+0000","last_fullsized":"2024-09-10T18:41:55.272673+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:44.986765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:41:44.986765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:09.455315+0000","scrub_duration":0.0079498090000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":66,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279248+0000","last_change":"2024-09-10T18:39:54.975996+0000","last_active":"2024-09-10T18:41:55.279248+0000","last_peered":"2024-09-10T18:41:55.279248+0000","last_clean":"2024-09-10T18:41:55.279248+0000","last_became_active":"2024-09-10T18:39:00.907231+0000","last_became_peered":"2024-09-10T18:39:00.907231+0000","last_unstale":"2024-09-10T18:41:55.279248+0000","last_undegraded":"2024-09-10T18:41:55.279248+0000","last_fullsized":"2024-09-10T18:41:55.279248+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:54.975936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:39:54.975936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013369269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":456,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.281921+0000","last_change":"2024-09-10T18:39:41.925032+0000","last_active":"2024-09-10T18:41:55.281921+0000","last_peered":"2024-09-10T18:41:55.281921+0000","last_clean":"2024-09-10T18:41:55.281921+0000","last_became_active":"2024-09-10T18:39:00.894963+0000","last_became_peered":"2024-09-10T18:39:00.894963+0000","last_unstale":"2024-09-10T18:41:55.281921+0000","last_undegraded":"2024-09-10T18:41:55.281921+0000","last_fullsized":"2024-09-10T18:41:55.281921+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:41.924977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:39:41.924977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011688601999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"404'10051","reported_seq":10408,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:05.788375+0000","last_change":"2024-09-10T18:41:36.523759+0000","last_active":"2024-09-10T18:42:05.788375+0000","last_peered":"2024-09-10T18:42:05.788375+0000","last_clean":"2024-09-10T18:42:05.788375+0000","last_became_active":"2024-09-10T18:40:05.803613+0000","last_became_peered":"2024-09-10T18:40:05.803613+0000","last_unstale":"2024-09-10T18:42:05.788375+0000","last_undegraded":"2024-09-10T18:42:05.788375+0000","last_fullsized":"2024-09-10T18:42:05.788375+0000","mapping_epoch":374,"log_start":"391'5504","ondisk_log_start":"391'5504","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'7829","last_scrub_stamp":"2024-09-10T18:41:36.523427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:41:36.523427+0000","objects_scrubbed":7693,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:39.729471+0000","scrub_duration":3.5398984680000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481099776,"num_objects":7341,"num_object_clones":0,"num_object_copies":22023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10051,"num_write_kb":556544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":66,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.277179+0000","last_change":"2024-09-10T18:40:33.759031+0000","last_active":"2024-09-10T18:41:55.277179+0000","last_peered":"2024-09-10T18:41:55.277179+0000","last_clean":"2024-09-10T18:41:55.277179+0000","last_became_active":"2024-09-10T18:39:00.909151+0000","last_became_peered":"2024-09-10T18:39:00.909151+0000","last_unstale":"2024-09-10T18:41:55.277179+0000","last_undegraded":"2024-09-10T18:41:55.277179+0000","last_fullsized":"2024-09-10T18:41:55.277179+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:33.758936+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:40:33.758936+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053087439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"400'53","reported_seq":574,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.273360+0000","last_change":"2024-09-10T18:41:28.183160+0000","last_active":"2024-09-10T18:41:55.273360+0000","last_peered":"2024-09-10T18:41:55.273360+0000","last_clean":"2024-09-10T18:41:55.273360+0000","last_became_active":"2024-09-10T18:38:43.333186+0000","last_became_peered":"2024-09-10T18:38:43.333186+0000","last_unstale":"2024-09-10T18:41:55.273360+0000","last_undegraded":"2024-09-10T18:41:55.273360+0000","last_fullsized":"2024-09-10T18:41:55.273360+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'39","last_scrub_stamp":"2024-09-10T18:41:28.183073+0000","last_deep_scrub":"342'39","last_deep_scrub_stamp":"2024-09-10T18:37:53.386563+0000","last_clean_scrub_stamp":"2024-09-10T18:41:28.183073+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:50.451880+0000","scrub_duration":0.067628692000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":417,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282522+0000","last_change":"2024-09-10T18:40:41.910549+0000","last_active":"2024-09-10T18:41:55.282522+0000","last_peered":"2024-09-10T18:41:55.282522+0000","last_clean":"2024-09-10T18:41:55.282522+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:41:55.282522+0000","last_undegraded":"2024-09-10T18:41:55.282522+0000","last_fullsized":"2024-09-10T18:41:55.282522+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:40:41.910472+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:40:41.910472+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:10.851386+0000","scrub_duration":0.00339047,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":459,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.278694+0000","last_change":"2024-09-10T18:41:35.323346+0000","last_active":"2024-09-10T18:41:55.278694+0000","last_peered":"2024-09-10T18:41:55.278694+0000","last_clean":"2024-09-10T18:41:55.278694+0000","last_became_active":"2024-09-10T18:39:25.249337+0000","last_became_peered":"2024-09-10T18:39:25.249337+0000","last_unstale":"2024-09-10T18:41:55.278694+0000","last_undegraded":"2024-09-10T18:41:55.278694+0000","last_fullsized":"2024-09-10T18:41:55.278694+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:35.323254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:41:35.323254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:59.713569+0000","scrub_duration":0.001933265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":475,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279025+0000","last_change":"2024-09-10T18:40:43.933545+0000","last_active":"2024-09-10T18:41:55.279025+0000","last_peered":"2024-09-10T18:41:55.279025+0000","last_clean":"2024-09-10T18:41:55.279025+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:41:55.279025+0000","last_undegraded":"2024-09-10T18:41:55.279025+0000","last_fullsized":"2024-09-10T18:41:55.279025+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:43.933300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:43.933300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:13.663259+0000","scrub_duration":0.0042666539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"404'10204","reported_seq":10332,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:05.612482+0000","last_change":"2024-09-10T18:41:50.077771+0000","last_active":"2024-09-10T18:42:05.612482+0000","last_peered":"2024-09-10T18:42:05.612482+0000","last_clean":"2024-09-10T18:42:05.612482+0000","last_became_active":"2024-09-10T18:40:05.801520+0000","last_became_peered":"2024-09-10T18:40:05.801520+0000","last_unstale":"2024-09-10T18:42:05.612482+0000","last_undegraded":"2024-09-10T18:42:05.612482+0000","last_fullsized":"2024-09-10T18:42:05.612482+0000","mapping_epoch":374,"log_start":"392'5700","ondisk_log_start":"392'5700","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":2099,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481034240,"num_objects":7340,"num_object_clones":0,"num_object_copies":22020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10204,"num_write_kb":561408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":500,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282048+0000","last_change":"2024-09-10T18:41:43.367065+0000","last_active":"2024-09-10T18:41:55.282048+0000","last_peered":"2024-09-10T18:41:55.282048+0000","last_clean":"2024-09-10T18:41:55.282048+0000","last_became_active":"2024-09-10T18:34:32.827174+0000","last_became_peered":"2024-09-10T18:34:32.827174+0000","last_unstale":"2024-09-10T18:41:55.282048+0000","last_undegraded":"2024-09-10T18:41:55.282048+0000","last_fullsized":"2024-09-10T18:41:55.282048+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:43.366981+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:43.366981+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:56.226795+0000","scrub_duration":0.0018640970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"400'35","reported_seq":215,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283081+0000","last_change":"2024-09-10T18:40:44.058922+0000","last_active":"2024-09-10T18:41:55.283081+0000","last_peered":"2024-09-10T18:41:55.283081+0000","last_clean":"2024-09-10T18:41:55.283081+0000","last_became_active":"2024-09-10T18:32:51.118251+0000","last_became_peered":"2024-09-10T18:32:51.118251+0000","last_unstale":"2024-09-10T18:41:55.283081+0000","last_undegraded":"2024-09-10T18:41:55.283081+0000","last_fullsized":"2024-09-10T18:41:55.283081+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:40:44.058799+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:40:44.058799+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:58.321284+0000","scrub_duration":0.016784752999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":445,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282607+0000","last_change":"2024-09-10T18:40:59.757340+0000","last_active":"2024-09-10T18:41:55.282607+0000","last_peered":"2024-09-10T18:41:55.282607+0000","last_clean":"2024-09-10T18:41:55.282607+0000","last_became_active":"2024-09-10T18:39:25.251419+0000","last_became_peered":"2024-09-10T18:39:25.251419+0000","last_unstale":"2024-09-10T18:41:55.282607+0000","last_undegraded":"2024-09-10T18:41:55.282607+0000","last_fullsized":"2024-09-10T18:41:55.282607+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:59.757255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:59.757255+0000","last_clean_scrub_stamp":"2024-09-10T18:40:59.757255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:20.577534+0000","scrub_duration":0.0056131599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"357'11","reported_seq":762,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:00.006736+0000","last_change":"2024-09-10T18:40:40.952168+0000","last_active":"2024-09-10T18:42:00.006736+0000","last_peered":"2024-09-10T18:42:00.006736+0000","last_clean":"2024-09-10T18:42:00.006736+0000","last_became_active":"2024-09-10T18:39:00.909747+0000","last_became_peered":"2024-09-10T18:39:00.909747+0000","last_unstale":"2024-09-10T18:42:00.006736+0000","last_undegraded":"2024-09-10T18:42:00.006736+0000","last_fullsized":"2024-09-10T18:42:00.006736+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'11","last_scrub_stamp":"2024-09-10T18:40:40.952077+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:40:40.952077+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00940007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":429,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.274443+0000","last_change":"2024-09-10T18:40:53.988053+0000","last_active":"2024-09-10T18:41:55.274443+0000","last_peered":"2024-09-10T18:41:55.274443+0000","last_clean":"2024-09-10T18:41:55.274443+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:41:55.274443+0000","last_undegraded":"2024-09-10T18:41:55.274443+0000","last_fullsized":"2024-09-10T18:41:55.274443+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:53.987811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:53.987811+0000","last_clean_scrub_stamp":"2024-09-10T18:40:53.987811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:19.937073+0000","scrub_duration":0.0078878390000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"404'10188","reported_seq":10226,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:05.174767+0000","last_change":"2024-09-10T18:40:05.806172+0000","last_active":"2024-09-10T18:42:05.174767+0000","last_peered":"2024-09-10T18:42:05.174767+0000","last_clean":"2024-09-10T18:42:05.174767+0000","last_became_active":"2024-09-10T18:40:05.805621+0000","last_became_peered":"2024-09-10T18:40:05.805621+0000","last_unstale":"2024-09-10T18:42:05.174767+0000","last_undegraded":"2024-09-10T18:42:05.174767+0000","last_fullsized":"2024-09-10T18:42:05.174767+0000","mapping_epoch":374,"log_start":"394'6500","ondisk_log_start":"394'6500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":0,"log_size":3688,"log_dups_size":0,"ondisk_log_size":3688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484311040,"num_objects":7390,"num_object_clones":0,"num_object_copies":22170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10188,"num_write_kb":562496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"76'81","reported_seq":172,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.274406+0000","last_change":"2024-09-10T18:41:00.014263+0000","last_active":"2024-09-10T18:41:55.274406+0000","last_peered":"2024-09-10T18:41:55.274406+0000","last_clean":"2024-09-10T18:41:55.274406+0000","last_became_active":"2024-09-10T18:34:32.818251+0000","last_became_peered":"2024-09-10T18:34:32.818251+0000","last_unstale":"2024-09-10T18:41:55.274406+0000","last_undegraded":"2024-09-10T18:41:55.274406+0000","last_fullsized":"2024-09-10T18:41:55.274406+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:00.014178+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:00.014178+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:18.015966+0000","scrub_duration":0.024934580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"400'25","reported_seq":413,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279673+0000","last_change":"2024-09-10T18:39:40.061732+0000","last_active":"2024-09-10T18:41:55.279673+0000","last_peered":"2024-09-10T18:41:55.279673+0000","last_clean":"2024-09-10T18:41:55.279673+0000","last_became_active":"2024-09-10T18:39:25.240389+0000","last_became_peered":"2024-09-10T18:39:25.240389+0000","last_unstale":"2024-09-10T18:41:55.279673+0000","last_undegraded":"2024-09-10T18:41:55.279673+0000","last_fullsized":"2024-09-10T18:41:55.279673+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"302'19","last_scrub_stamp":"2024-09-10T18:39:40.061643+0000","last_deep_scrub":"302'19","last_deep_scrub_stamp":"2024-09-10T18:35:18.836131+0000","last_clean_scrub_stamp":"2024-09-10T18:39:40.061643+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059734920000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"265'2","reported_seq":70,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.277841+0000","last_change":"2024-09-10T18:39:53.939060+0000","last_active":"2024-09-10T18:41:55.277841+0000","last_peered":"2024-09-10T18:41:55.277841+0000","last_clean":"2024-09-10T18:41:55.277841+0000","last_became_active":"2024-09-10T18:39:00.904967+0000","last_became_peered":"2024-09-10T18:39:00.904967+0000","last_unstale":"2024-09-10T18:41:55.277841+0000","last_undegraded":"2024-09-10T18:41:55.277841+0000","last_fullsized":"2024-09-10T18:41:55.277841+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'2","last_scrub_stamp":"2024-09-10T18:39:53.938998+0000","last_deep_scrub":"265'2","last_deep_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_clean_scrub_stamp":"2024-09-10T18:39:53.938998+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001572473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":463,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279470+0000","last_change":"2024-09-10T18:40:36.011493+0000","last_active":"2024-09-10T18:41:55.279470+0000","last_peered":"2024-09-10T18:41:55.279470+0000","last_clean":"2024-09-10T18:41:55.279470+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:41:55.279470+0000","last_undegraded":"2024-09-10T18:41:55.279470+0000","last_fullsized":"2024-09-10T18:41:55.279470+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:36.011401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:40:36.011401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018830089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":65,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.281860+0000","last_change":"2024-09-10T18:41:35.361220+0000","last_active":"2024-09-10T18:41:55.281860+0000","last_peered":"2024-09-10T18:41:55.281860+0000","last_clean":"2024-09-10T18:41:55.281860+0000","last_became_active":"2024-09-10T18:39:25.237150+0000","last_became_peered":"2024-09-10T18:39:25.237150+0000","last_unstale":"2024-09-10T18:41:55.281860+0000","last_undegraded":"2024-09-10T18:41:55.281860+0000","last_fullsized":"2024-09-10T18:41:55.281860+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:35.361130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:41:35.361130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:38.395797+0000","scrub_duration":0.023350010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"404'10268","reported_seq":10306,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:05.175123+0000","last_change":"2024-09-10T18:40:05.806410+0000","last_active":"2024-09-10T18:42:05.175123+0000","last_peered":"2024-09-10T18:42:05.175123+0000","last_clean":"2024-09-10T18:42:05.175123+0000","last_became_active":"2024-09-10T18:40:05.806063+0000","last_became_peered":"2024-09-10T18:40:05.806063+0000","last_unstale":"2024-09-10T18:42:05.175123+0000","last_undegraded":"2024-09-10T18:42:05.175123+0000","last_fullsized":"2024-09-10T18:42:05.175123+0000","mapping_epoch":374,"log_start":"394'6600","ondisk_log_start":"394'6600","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":0,"log_size":3668,"log_dups_size":0,"ondisk_log_size":3668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476708864,"num_objects":7274,"num_object_clones":0,"num_object_copies":21822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10268,"num_write_kb":561344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"76'81","reported_seq":448,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283623+0000","last_change":"2024-09-10T18:40:34.817368+0000","last_active":"2024-09-10T18:41:55.283623+0000","last_peered":"2024-09-10T18:41:55.283623+0000","last_clean":"2024-09-10T18:41:55.283623+0000","last_became_active":"2024-09-10T18:39:00.921868+0000","last_became_peered":"2024-09-10T18:39:00.921868+0000","last_unstale":"2024-09-10T18:41:55.283623+0000","last_undegraded":"2024-09-10T18:41:55.283623+0000","last_fullsized":"2024-09-10T18:41:55.283623+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:34.817276+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:34:15.938079+0000","last_clean_scrub_stamp":"2024-09-10T18:40:34.817276+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027958480000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"400'60","reported_seq":558,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282342+0000","last_change":"2024-09-10T18:40:46.939867+0000","last_active":"2024-09-10T18:41:55.282342+0000","last_peered":"2024-09-10T18:41:55.282342+0000","last_clean":"2024-09-10T18:41:55.282342+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:41:55.282342+0000","last_undegraded":"2024-09-10T18:41:55.282342+0000","last_fullsized":"2024-09-10T18:41:55.282342+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'44","last_scrub_stamp":"2024-09-10T18:40:46.939778+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:40:46.939778+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:07.082574+0000","scrub_duration":0.0070895619999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":472,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283396+0000","last_change":"2024-09-10T18:40:32.930891+0000","last_active":"2024-09-10T18:41:55.283396+0000","last_peered":"2024-09-10T18:41:55.283396+0000","last_clean":"2024-09-10T18:41:55.283396+0000","last_became_active":"2024-09-10T18:39:00.895121+0000","last_became_peered":"2024-09-10T18:39:00.895121+0000","last_unstale":"2024-09-10T18:41:55.283396+0000","last_undegraded":"2024-09-10T18:41:55.283396+0000","last_fullsized":"2024-09-10T18:41:55.283396+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:32.930805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:40:32.930805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017381429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":507,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279223+0000","last_change":"2024-09-10T18:41:32.022773+0000","last_active":"2024-09-10T18:41:55.279223+0000","last_peered":"2024-09-10T18:41:55.279223+0000","last_clean":"2024-09-10T18:41:55.279223+0000","last_became_active":"2024-09-10T18:39:25.240291+0000","last_became_peered":"2024-09-10T18:39:25.240291+0000","last_unstale":"2024-09-10T18:41:55.279223+0000","last_undegraded":"2024-09-10T18:41:55.279223+0000","last_fullsized":"2024-09-10T18:41:55.279223+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:32.022685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:34:21.447291+0000","last_clean_scrub_stamp":"2024-09-10T18:41:32.022685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:44.038691+0000","scrub_duration":0.0069142700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":439,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282819+0000","last_change":"2024-09-10T18:39:47.292863+0000","last_active":"2024-09-10T18:41:55.282819+0000","last_peered":"2024-09-10T18:41:55.282819+0000","last_clean":"2024-09-10T18:41:55.282819+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:41:55.282819+0000","last_undegraded":"2024-09-10T18:41:55.282819+0000","last_fullsized":"2024-09-10T18:41:55.282819+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:47.292801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:04.058646+0000","last_clean_scrub_stamp":"2024-09-10T18:39:47.292801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001761461,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"404'14331","reported_seq":14591,"reported_epoch":404,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-10T18:42:05.791287+0000","last_change":"2024-09-10T18:41:40.109559+0000","last_active":"2024-09-10T18:42:05.791287+0000","last_peered":"2024-09-10T18:42:05.791287+0000","last_clean":"2024-09-10T18:42:05.791287+0000","last_became_active":"2024-09-10T18:40:05.804573+0000","last_became_peered":"2024-09-10T18:40:05.804573+0000","last_unstale":"2024-09-10T18:42:05.791287+0000","last_undegraded":"2024-09-10T18:42:05.791287+0000","last_fullsized":"2024-09-10T18:42:05.791287+0000","mapping_epoch":374,"log_start":"401'9764","ondisk_log_start":"401'9764","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":5446,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 25s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478281728,"num_objects":7298,"num_object_clones":0,"num_object_copies":21894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10142,"num_write_kb":558080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"76'81","reported_seq":390,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283686+0000","last_change":"2024-09-10T18:41:41.914514+0000","last_active":"2024-09-10T18:41:55.283686+0000","last_peered":"2024-09-10T18:41:55.283686+0000","last_clean":"2024-09-10T18:41:55.283686+0000","last_became_active":"2024-09-10T18:39:25.242996+0000","last_became_peered":"2024-09-10T18:39:25.242996+0000","last_unstale":"2024-09-10T18:41:55.283686+0000","last_undegraded":"2024-09-10T18:41:55.283686+0000","last_fullsized":"2024-09-10T18:41:55.283686+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:41.914420+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:41.914420+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:43.841318+0000","scrub_duration":0.0079109539999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"400'29","reported_seq":489,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.272371+0000","last_change":"2024-09-10T18:40:55.034168+0000","last_active":"2024-09-10T18:41:55.272371+0000","last_peered":"2024-09-10T18:41:55.272371+0000","last_clean":"2024-09-10T18:41:55.272371+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:41:55.272371+0000","last_undegraded":"2024-09-10T18:41:55.272371+0000","last_fullsized":"2024-09-10T18:41:55.272371+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'23","last_scrub_stamp":"2024-09-10T18:40:55.034050+0000","last_deep_scrub":"301'23","last_deep_scrub_stamp":"2024-09-10T18:40:55.034050+0000","last_clean_scrub_stamp":"2024-09-10T18:40:55.034050+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:19.662189+0000","scrub_duration":0.0069781490000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":452,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283463+0000","last_change":"2024-09-10T18:41:48.482843+0000","last_active":"2024-09-10T18:41:55.283463+0000","last_peered":"2024-09-10T18:41:55.283463+0000","last_clean":"2024-09-10T18:41:55.283463+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:41:55.283463+0000","last_undegraded":"2024-09-10T18:41:55.283463+0000","last_fullsized":"2024-09-10T18:41:55.283463+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:48.482677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:41:48.482677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:00.981134+0000","scrub_duration":0.052469623,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":487,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.277678+0000","last_change":"2024-09-10T18:40:42.972118+0000","last_active":"2024-09-10T18:41:55.277678+0000","last_peered":"2024-09-10T18:41:55.277678+0000","last_clean":"2024-09-10T18:41:55.277678+0000","last_became_active":"2024-09-10T18:39:25.264561+0000","last_became_peered":"2024-09-10T18:39:25.264561+0000","last_unstale":"2024-09-10T18:41:55.277678+0000","last_undegraded":"2024-09-10T18:41:55.277678+0000","last_fullsized":"2024-09-10T18:41:55.277678+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:42.972018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:40:42.972018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:12.204431+0000","scrub_duration":0.0039715810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":429,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282283+0000","last_change":"2024-09-10T18:41:31.850183+0000","last_active":"2024-09-10T18:41:55.282283+0000","last_peered":"2024-09-10T18:41:55.282283+0000","last_clean":"2024-09-10T18:41:55.282283+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:41:55.282283+0000","last_undegraded":"2024-09-10T18:41:55.282283+0000","last_fullsized":"2024-09-10T18:41:55.282283+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:31.850086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:38:50.727231+0000","last_clean_scrub_stamp":"2024-09-10T18:41:31.850086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:46.735139+0000","scrub_duration":0.0088275699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"404'10244","reported_seq":10281,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:05.225286+0000","last_change":"2024-09-10T18:40:05.802932+0000","last_active":"2024-09-10T18:42:05.225286+0000","last_peered":"2024-09-10T18:42:05.225286+0000","last_clean":"2024-09-10T18:42:05.225286+0000","last_became_active":"2024-09-10T18:40:05.802669+0000","last_became_peered":"2024-09-10T18:40:05.802669+0000","last_unstale":"2024-09-10T18:42:05.225286+0000","last_undegraded":"2024-09-10T18:42:05.225286+0000","last_fullsized":"2024-09-10T18:42:05.225286+0000","mapping_epoch":374,"log_start":"393'6000","ondisk_log_start":"393'6000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":0,"log_size":4244,"log_dups_size":0,"ondisk_log_size":4244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490733568,"num_objects":7488,"num_object_clones":0,"num_object_copies":22464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10244,"num_write_kb":567424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"76'81","reported_seq":172,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282634+0000","last_change":"2024-09-10T18:41:45.417035+0000","last_active":"2024-09-10T18:41:55.282634+0000","last_peered":"2024-09-10T18:41:55.282634+0000","last_clean":"2024-09-10T18:41:55.282634+0000","last_became_active":"2024-09-10T18:34:32.822196+0000","last_became_peered":"2024-09-10T18:34:32.822196+0000","last_unstale":"2024-09-10T18:41:55.282634+0000","last_undegraded":"2024-09-10T18:41:55.282634+0000","last_fullsized":"2024-09-10T18:41:55.282634+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:45.416956+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:45.416956+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:02.570516+0000","scrub_duration":0.0027361949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"400'39","reported_seq":110,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.276967+0000","last_change":"2024-09-10T18:41:49.939121+0000","last_active":"2024-09-10T18:41:55.276967+0000","last_peered":"2024-09-10T18:41:55.276967+0000","last_clean":"2024-09-10T18:41:55.276967+0000","last_became_active":"2024-09-10T18:39:00.902783+0000","last_became_peered":"2024-09-10T18:39:00.902783+0000","last_unstale":"2024-09-10T18:41:55.276967+0000","last_undegraded":"2024-09-10T18:41:55.276967+0000","last_fullsized":"2024-09-10T18:41:55.276967+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'39","last_scrub_stamp":"2024-09-10T18:41:49.939038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:49.939038+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:15.146611+0000","scrub_duration":0.0027044299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":468,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283930+0000","last_change":"2024-09-10T18:41:39.392189+0000","last_active":"2024-09-10T18:41:55.283930+0000","last_peered":"2024-09-10T18:41:55.283930+0000","last_clean":"2024-09-10T18:41:55.283930+0000","last_became_active":"2024-09-10T18:39:25.238037+0000","last_became_peered":"2024-09-10T18:39:25.238037+0000","last_unstale":"2024-09-10T18:41:55.283930+0000","last_undegraded":"2024-09-10T18:41:55.283930+0000","last_fullsized":"2024-09-10T18:41:55.283930+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:41:39.391952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:41:39.391952+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:55.077116+0000","scrub_duration":0.0076173680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":703,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:00.005911+0000","last_change":"2024-09-10T18:41:38.346953+0000","last_active":"2024-09-10T18:42:00.005911+0000","last_peered":"2024-09-10T18:42:00.005911+0000","last_clean":"2024-09-10T18:42:00.005911+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:42:00.005911+0000","last_undegraded":"2024-09-10T18:42:00.005911+0000","last_fullsized":"2024-09-10T18:42:00.005911+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:41:38.346859+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:41:38.346859+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:03.983030+0000","scrub_duration":0.012291041000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":76,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279279+0000","last_change":"2024-09-10T18:41:31.870285+0000","last_active":"2024-09-10T18:41:55.279279+0000","last_peered":"2024-09-10T18:41:55.279279+0000","last_clean":"2024-09-10T18:41:55.279279+0000","last_became_active":"2024-09-10T18:39:25.242197+0000","last_became_peered":"2024-09-10T18:39:25.242197+0000","last_unstale":"2024-09-10T18:41:55.279279+0000","last_undegraded":"2024-09-10T18:41:55.279279+0000","last_fullsized":"2024-09-10T18:41:55.279279+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:31.870211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:41:31.870211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:37.642170+0000","scrub_duration":0.0089146250000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'81","reported_seq":66,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279402+0000","last_change":"2024-09-10T18:40:32.742375+0000","last_active":"2024-09-10T18:41:55.279402+0000","last_peered":"2024-09-10T18:41:55.279402+0000","last_clean":"2024-09-10T18:41:55.279402+0000","last_became_active":"2024-09-10T18:39:00.910728+0000","last_became_peered":"2024-09-10T18:39:00.910728+0000","last_unstale":"2024-09-10T18:41:55.279402+0000","last_undegraded":"2024-09-10T18:41:55.279402+0000","last_fullsized":"2024-09-10T18:41:55.279402+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:32.742274+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:40:32.742274+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058849089999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":438,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.281933+0000","last_change":"2024-09-10T18:41:39.299055+0000","last_active":"2024-09-10T18:41:55.281933+0000","last_peered":"2024-09-10T18:41:55.281933+0000","last_clean":"2024-09-10T18:41:55.281933+0000","last_became_active":"2024-09-10T18:39:25.243214+0000","last_became_peered":"2024-09-10T18:39:25.243214+0000","last_unstale":"2024-09-10T18:41:55.281933+0000","last_undegraded":"2024-09-10T18:41:55.281933+0000","last_fullsized":"2024-09-10T18:41:55.281933+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:39.298969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:41:39.298969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:57.190842+0000","scrub_duration":0.0042839920000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"400'18","reported_seq":460,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283283+0000","last_change":"2024-09-10T18:41:48.983998+0000","last_active":"2024-09-10T18:41:55.283283+0000","last_peered":"2024-09-10T18:41:55.283283+0000","last_clean":"2024-09-10T18:41:55.283283+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:41:55.283283+0000","last_undegraded":"2024-09-10T18:41:55.283283+0000","last_fullsized":"2024-09-10T18:41:55.283283+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'18","last_scrub_stamp":"2024-09-10T18:41:48.983891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:48.983891+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:11.108479+0000","scrub_duration":0.0035174529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":66,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.278283+0000","last_change":"2024-09-10T18:40:51.751346+0000","last_active":"2024-09-10T18:41:55.278283+0000","last_peered":"2024-09-10T18:41:55.278283+0000","last_clean":"2024-09-10T18:41:55.278283+0000","last_became_active":"2024-09-10T18:39:00.916952+0000","last_became_peered":"2024-09-10T18:39:00.916952+0000","last_unstale":"2024-09-10T18:41:55.278283+0000","last_undegraded":"2024-09-10T18:41:55.278283+0000","last_fullsized":"2024-09-10T18:41:55.278283+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:51.751267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:51.751267+0000","last_clean_scrub_stamp":"2024-09-10T18:40:51.751267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:14.849728+0000","scrub_duration":0.0026356539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":433,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282717+0000","last_change":"2024-09-10T18:39:53.274245+0000","last_active":"2024-09-10T18:41:55.282717+0000","last_peered":"2024-09-10T18:41:55.282717+0000","last_clean":"2024-09-10T18:41:55.282717+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:41:55.282717+0000","last_undegraded":"2024-09-10T18:41:55.282717+0000","last_fullsized":"2024-09-10T18:41:55.282717+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:53.274176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:39:53.274176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016102950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'81","reported_seq":379,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279280+0000","last_change":"2024-09-10T18:41:33.337668+0000","last_active":"2024-09-10T18:41:55.279280+0000","last_peered":"2024-09-10T18:41:55.279280+0000","last_clean":"2024-09-10T18:41:55.279280+0000","last_became_active":"2024-09-10T18:34:02.815880+0000","last_became_peered":"2024-09-10T18:34:02.815880+0000","last_unstale":"2024-09-10T18:41:55.279280+0000","last_undegraded":"2024-09-10T18:41:55.279280+0000","last_fullsized":"2024-09-10T18:41:55.279280+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:33.337569+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:33.337569+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:39.439729+0000","scrub_duration":0.0052344720000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"400'25","reported_seq":342,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283714+0000","last_change":"2024-09-10T18:41:43.958089+0000","last_active":"2024-09-10T18:41:55.283714+0000","last_peered":"2024-09-10T18:41:55.283714+0000","last_clean":"2024-09-10T18:41:55.283714+0000","last_became_active":"2024-09-10T18:39:25.236628+0000","last_became_peered":"2024-09-10T18:39:25.236628+0000","last_unstale":"2024-09-10T18:41:55.283714+0000","last_undegraded":"2024-09-10T18:41:55.283714+0000","last_fullsized":"2024-09-10T18:41:55.283714+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'25","last_scrub_stamp":"2024-09-10T18:41:43.957988+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-09-10T18:35:19.802016+0000","last_clean_scrub_stamp":"2024-09-10T18:41:43.957988+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:05.515733+0000","scrub_duration":0.021283768000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":449,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282202+0000","last_change":"2024-09-10T18:41:47.405474+0000","last_active":"2024-09-10T18:41:55.282202+0000","last_peered":"2024-09-10T18:41:55.282202+0000","last_clean":"2024-09-10T18:41:55.282202+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:41:55.282202+0000","last_undegraded":"2024-09-10T18:41:55.282202+0000","last_fullsized":"2024-09-10T18:41:55.282202+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:47.405374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:41:47.405374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:48.024383+0000","scrub_duration":0.0015929049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":431,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283083+0000","last_change":"2024-09-10T18:40:46.624305+0000","last_active":"2024-09-10T18:41:55.283083+0000","last_peered":"2024-09-10T18:41:55.283083+0000","last_clean":"2024-09-10T18:41:55.283083+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:41:55.283083+0000","last_undegraded":"2024-09-10T18:41:55.283083+0000","last_fullsized":"2024-09-10T18:41:55.283083+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:46.624225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:40:46.624225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:58.272601+0000","scrub_duration":0.0068887569999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":475,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282391+0000","last_change":"2024-09-10T18:41:44.176876+0000","last_active":"2024-09-10T18:41:55.282391+0000","last_peered":"2024-09-10T18:41:55.282391+0000","last_clean":"2024-09-10T18:41:55.282391+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:41:55.282391+0000","last_undegraded":"2024-09-10T18:41:55.282391+0000","last_fullsized":"2024-09-10T18:41:55.282391+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:44.176788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:41:44.176788+0000","last_clean_scrub_stamp":"2024-09-10T18:41:44.176788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:00.588555+0000","scrub_duration":0.0078308370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'81","reported_seq":66,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.277037+0000","last_change":"2024-09-10T18:39:55.999435+0000","last_active":"2024-09-10T18:41:55.277037+0000","last_peered":"2024-09-10T18:41:55.277037+0000","last_clean":"2024-09-10T18:41:55.277037+0000","last_became_active":"2024-09-10T18:39:00.902585+0000","last_became_peered":"2024-09-10T18:39:00.902585+0000","last_unstale":"2024-09-10T18:41:55.277037+0000","last_undegraded":"2024-09-10T18:41:55.277037+0000","last_fullsized":"2024-09-10T18:41:55.277037+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:39:55.999377+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:39:55.999377+0000","last_clean_scrub_stamp":"2024-09-10T18:39:55.999377+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001274175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"400'25","reported_seq":85,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.278346+0000","last_change":"2024-09-10T18:41:02.704925+0000","last_active":"2024-09-10T18:41:55.278346+0000","last_peered":"2024-09-10T18:41:55.278346+0000","last_clean":"2024-09-10T18:41:55.278346+0000","last_became_active":"2024-09-10T18:39:00.899968+0000","last_became_peered":"2024-09-10T18:39:00.899968+0000","last_unstale":"2024-09-10T18:41:55.278346+0000","last_undegraded":"2024-09-10T18:41:55.278346+0000","last_fullsized":"2024-09-10T18:41:55.278346+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:41:02.704656+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:41:02.704656+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:32.633326+0000","scrub_duration":0.0081500759999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282802+0000","last_change":"2024-09-10T18:40:39.071149+0000","last_active":"2024-09-10T18:41:55.282802+0000","last_peered":"2024-09-10T18:41:55.282802+0000","last_clean":"2024-09-10T18:41:55.282802+0000","last_became_active":"2024-09-10T18:39:25.245366+0000","last_became_peered":"2024-09-10T18:39:25.245366+0000","last_unstale":"2024-09-10T18:41:55.282802+0000","last_undegraded":"2024-09-10T18:41:55.282802+0000","last_fullsized":"2024-09-10T18:41:55.282802+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:39.071067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_clean_scrub_stamp":"2024-09-10T18:40:39.071067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010317092,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"342'9","reported_seq":686,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:41:56.291468+0000","last_change":"2024-09-10T18:40:38.445567+0000","last_active":"2024-09-10T18:41:56.291468+0000","last_peered":"2024-09-10T18:41:56.291468+0000","last_clean":"2024-09-10T18:41:56.291468+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:41:56.291468+0000","last_undegraded":"2024-09-10T18:41:56.291468+0000","last_fullsized":"2024-09-10T18:41:56.291468+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'9","last_scrub_stamp":"2024-09-10T18:40:38.445473+0000","last_deep_scrub":"342'9","last_deep_scrub_stamp":"2024-09-10T18:37:56.641708+0000","last_clean_scrub_stamp":"2024-09-10T18:40:38.445473+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047172459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":67,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283417+0000","last_change":"2024-09-10T18:39:34.901254+0000","last_active":"2024-09-10T18:41:55.283417+0000","last_peered":"2024-09-10T18:41:55.283417+0000","last_clean":"2024-09-10T18:41:55.283417+0000","last_became_active":"2024-09-10T18:39:00.903167+0000","last_became_peered":"2024-09-10T18:39:00.903167+0000","last_unstale":"2024-09-10T18:41:55.283417+0000","last_undegraded":"2024-09-10T18:41:55.283417+0000","last_fullsized":"2024-09-10T18:41:55.283417+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:34.901182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:39:34.901182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00237777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"76'81","reported_seq":252,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.278675+0000","last_change":"2024-09-10T18:40:43.823600+0000","last_active":"2024-09-10T18:41:55.278675+0000","last_peered":"2024-09-10T18:41:55.278675+0000","last_clean":"2024-09-10T18:41:55.278675+0000","last_became_active":"2024-09-10T18:39:25.249841+0000","last_became_peered":"2024-09-10T18:39:25.249841+0000","last_unstale":"2024-09-10T18:41:55.278675+0000","last_undegraded":"2024-09-10T18:41:55.278675+0000","last_fullsized":"2024-09-10T18:41:55.278675+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:43.823507+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:36:08.710566+0000","last_clean_scrub_stamp":"2024-09-10T18:40:43.823507+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:08.140612+0000","scrub_duration":0.0042527390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"400'30","reported_seq":478,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283140+0000","last_change":"2024-09-10T18:39:39.855932+0000","last_active":"2024-09-10T18:41:55.283140+0000","last_peered":"2024-09-10T18:41:55.283140+0000","last_clean":"2024-09-10T18:41:55.283140+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:41:55.283140+0000","last_undegraded":"2024-09-10T18:41:55.283140+0000","last_fullsized":"2024-09-10T18:41:55.283140+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-10T18:39:39.855841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:39:39.855841+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008828766,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":442,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.284013+0000","last_change":"2024-09-10T18:40:56.107500+0000","last_active":"2024-09-10T18:41:55.284013+0000","last_peered":"2024-09-10T18:41:55.284013+0000","last_clean":"2024-09-10T18:41:55.284013+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:41:55.284013+0000","last_undegraded":"2024-09-10T18:41:55.284013+0000","last_fullsized":"2024-09-10T18:41:55.284013+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:56.107424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:17.826448+0000","last_clean_scrub_stamp":"2024-09-10T18:40:56.107424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:41:58.521555+0000","scrub_duration":0.01055597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"362'11","reported_seq":123,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-10T18:42:00.005519+0000","last_change":"2024-09-10T18:41:30.919843+0000","last_active":"2024-09-10T18:42:00.005519+0000","last_peered":"2024-09-10T18:42:00.005519+0000","last_clean":"2024-09-10T18:42:00.005519+0000","last_became_active":"2024-09-10T18:39:25.256047+0000","last_became_peered":"2024-09-10T18:39:25.256047+0000","last_unstale":"2024-09-10T18:42:00.005519+0000","last_undegraded":"2024-09-10T18:42:00.005519+0000","last_fullsized":"2024-09-10T18:42:00.005519+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'11","last_scrub_stamp":"2024-09-10T18:41:30.919717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:41:30.919717+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:34.339962+0000","scrub_duration":0.0096531840000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":65,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282819+0000","last_change":"2024-09-10T18:40:52.022624+0000","last_active":"2024-09-10T18:41:55.282819+0000","last_peered":"2024-09-10T18:41:55.282819+0000","last_clean":"2024-09-10T18:41:55.282819+0000","last_became_active":"2024-09-10T18:39:25.247171+0000","last_became_peered":"2024-09-10T18:39:25.247171+0000","last_unstale":"2024-09-10T18:41:55.282819+0000","last_undegraded":"2024-09-10T18:41:55.282819+0000","last_fullsized":"2024-09-10T18:41:55.282819+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:52.022343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:52.022343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:13.573283+0000","scrub_duration":0.0053789739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.16","version":"76'81","reported_seq":164,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282644+0000","last_change":"2024-09-10T18:40:46.174454+0000","last_active":"2024-09-10T18:41:55.282644+0000","last_peered":"2024-09-10T18:41:55.282644+0000","last_clean":"2024-09-10T18:41:55.282644+0000","last_became_active":"2024-09-10T18:39:25.253130+0000","last_became_peered":"2024-09-10T18:39:25.253130+0000","last_unstale":"2024-09-10T18:41:55.282644+0000","last_undegraded":"2024-09-10T18:41:55.282644+0000","last_fullsized":"2024-09-10T18:41:55.282644+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:46.174367+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:40:46.174367+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083268330000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"400'27","reported_seq":512,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.272298+0000","last_change":"2024-09-10T18:41:11.943981+0000","last_active":"2024-09-10T18:41:55.272298+0000","last_peered":"2024-09-10T18:41:55.272298+0000","last_clean":"2024-09-10T18:41:55.272298+0000","last_became_active":"2024-09-10T18:41:11.933013+0000","last_became_peered":"2024-09-10T18:41:11.933013+0000","last_unstale":"2024-09-10T18:41:55.272298+0000","last_undegraded":"2024-09-10T18:41:55.272298+0000","last_fullsized":"2024-09-10T18:41:55.272298+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:40:47.920564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:40:47.920564+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:02.836809+0000","scrub_duration":0.019580443999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":442,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.273569+0000","last_change":"2024-09-10T18:39:47.552533+0000","last_active":"2024-09-10T18:41:55.273569+0000","last_peered":"2024-09-10T18:41:55.273569+0000","last_clean":"2024-09-10T18:41:55.273569+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:41:55.273569+0000","last_undegraded":"2024-09-10T18:41:55.273569+0000","last_fullsized":"2024-09-10T18:41:55.273569+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:47.552466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:39:47.552466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001793968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":472,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279634+0000","last_change":"2024-09-10T18:40:53.230753+0000","last_active":"2024-09-10T18:41:55.279634+0000","last_peered":"2024-09-10T18:41:55.279634+0000","last_clean":"2024-09-10T18:41:55.279634+0000","last_became_active":"2024-09-10T18:39:00.897018+0000","last_became_peered":"2024-09-10T18:39:00.897018+0000","last_unstale":"2024-09-10T18:41:55.279634+0000","last_undegraded":"2024-09-10T18:41:55.279634+0000","last_fullsized":"2024-09-10T18:41:55.279634+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:53.230680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_clean_scrub_stamp":"2024-09-10T18:40:53.230680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:02.167295+0000","scrub_duration":0.003669224,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":449,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.286012+0000","last_change":"2024-09-10T18:39:53.911283+0000","last_active":"2024-09-10T18:41:55.286012+0000","last_peered":"2024-09-10T18:41:55.286012+0000","last_clean":"2024-09-10T18:41:55.286012+0000","last_became_active":"2024-09-10T18:39:25.237500+0000","last_became_peered":"2024-09-10T18:39:25.237500+0000","last_unstale":"2024-09-10T18:41:55.286012+0000","last_undegraded":"2024-09-10T18:41:55.286012+0000","last_fullsized":"2024-09-10T18:41:55.286012+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:53.911229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:39:53.911229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001589023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'81","reported_seq":76,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279182+0000","last_change":"2024-09-10T18:40:52.818032+0000","last_active":"2024-09-10T18:41:55.279182+0000","last_peered":"2024-09-10T18:41:55.279182+0000","last_clean":"2024-09-10T18:41:55.279182+0000","last_became_active":"2024-09-10T18:39:25.244612+0000","last_became_peered":"2024-09-10T18:39:25.244612+0000","last_unstale":"2024-09-10T18:41:55.279182+0000","last_undegraded":"2024-09-10T18:41:55.279182+0000","last_fullsized":"2024-09-10T18:41:55.279182+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:52.817945+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:40:52.817945+0000","last_clean_scrub_stamp":"2024-09-10T18:40:52.817945+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:20.054531+0000","scrub_duration":0.061016616000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"400'21","reported_seq":492,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279954+0000","last_change":"2024-09-10T18:41:00.246540+0000","last_active":"2024-09-10T18:41:55.279954+0000","last_peered":"2024-09-10T18:41:55.279954+0000","last_clean":"2024-09-10T18:41:55.279954+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:41:55.279954+0000","last_undegraded":"2024-09-10T18:41:55.279954+0000","last_fullsized":"2024-09-10T18:41:55.279954+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:41:00.246464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:00.246464+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:28.765236+0000","scrub_duration":0.012761635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":445,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283178+0000","last_change":"2024-09-10T18:41:29.335366+0000","last_active":"2024-09-10T18:41:55.283178+0000","last_peered":"2024-09-10T18:41:55.283178+0000","last_clean":"2024-09-10T18:41:55.283178+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:41:55.283178+0000","last_undegraded":"2024-09-10T18:41:55.283178+0000","last_fullsized":"2024-09-10T18:41:55.283178+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:29.335275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:41:29.335275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:44.668182+0000","scrub_duration":0.0032473530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":434,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283916+0000","last_change":"2024-09-10T18:40:47.615302+0000","last_active":"2024-09-10T18:41:55.283916+0000","last_peered":"2024-09-10T18:41:55.283916+0000","last_clean":"2024-09-10T18:41:55.283916+0000","last_became_active":"2024-09-10T18:39:25.251469+0000","last_became_peered":"2024-09-10T18:39:25.251469+0000","last_unstale":"2024-09-10T18:41:55.283916+0000","last_undegraded":"2024-09-10T18:41:55.283916+0000","last_fullsized":"2024-09-10T18:41:55.283916+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:47.615205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:40:47.615205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:09.339759+0000","scrub_duration":0.0030335510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":456,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.277107+0000","last_change":"2024-09-10T18:39:46.020028+0000","last_active":"2024-09-10T18:41:55.277107+0000","last_peered":"2024-09-10T18:41:55.277107+0000","last_clean":"2024-09-10T18:41:55.277107+0000","last_became_active":"2024-09-10T18:39:25.264554+0000","last_became_peered":"2024-09-10T18:39:25.264554+0000","last_unstale":"2024-09-10T18:41:55.277107+0000","last_undegraded":"2024-09-10T18:41:55.277107+0000","last_fullsized":"2024-09-10T18:41:55.277107+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:46.019956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:39:46.019956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018874110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"76'81","reported_seq":181,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.280261+0000","last_change":"2024-09-10T18:41:28.989528+0000","last_active":"2024-09-10T18:41:55.280261+0000","last_peered":"2024-09-10T18:41:55.280261+0000","last_clean":"2024-09-10T18:41:55.280261+0000","last_became_active":"2024-09-10T18:34:02.802580+0000","last_became_peered":"2024-09-10T18:34:02.802580+0000","last_unstale":"2024-09-10T18:41:55.280261+0000","last_undegraded":"2024-09-10T18:41:55.280261+0000","last_fullsized":"2024-09-10T18:41:55.280261+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:28.989434+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:28.989434+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:51.842021+0000","scrub_duration":0.011606899,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"400'27","reported_seq":483,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.285815+0000","last_change":"2024-09-10T18:40:41.493272+0000","last_active":"2024-09-10T18:41:55.285815+0000","last_peered":"2024-09-10T18:41:55.285815+0000","last_clean":"2024-09-10T18:41:55.285815+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:41:55.285815+0000","last_undegraded":"2024-09-10T18:41:55.285815+0000","last_fullsized":"2024-09-10T18:41:55.285815+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:40:41.493074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:40:41.493074+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049833050000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":76,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.278217+0000","last_change":"2024-09-10T18:41:38.954345+0000","last_active":"2024-09-10T18:41:55.278217+0000","last_peered":"2024-09-10T18:41:55.278217+0000","last_clean":"2024-09-10T18:41:55.278217+0000","last_became_active":"2024-09-10T18:39:00.916983+0000","last_became_peered":"2024-09-10T18:39:00.916983+0000","last_unstale":"2024-09-10T18:41:55.278217+0000","last_undegraded":"2024-09-10T18:41:55.278217+0000","last_fullsized":"2024-09-10T18:41:55.278217+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:38.954273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:41:38.954273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:57.986773+0000","scrub_duration":0.0019398480000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":438,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.281031+0000","last_change":"2024-09-10T18:40:46.080092+0000","last_active":"2024-09-10T18:41:55.281031+0000","last_peered":"2024-09-10T18:41:55.281031+0000","last_clean":"2024-09-10T18:41:55.281031+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:41:55.281031+0000","last_undegraded":"2024-09-10T18:41:55.281031+0000","last_fullsized":"2024-09-10T18:41:55.281031+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:46.080018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:40:46.080018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:11.605402+0000","scrub_duration":0.014966508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":455,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282527+0000","last_change":"2024-09-10T18:39:38.888765+0000","last_active":"2024-09-10T18:41:55.282527+0000","last_peered":"2024-09-10T18:41:55.282527+0000","last_clean":"2024-09-10T18:41:55.282527+0000","last_became_active":"2024-09-10T18:39:25.242900+0000","last_became_peered":"2024-09-10T18:39:25.242900+0000","last_unstale":"2024-09-10T18:41:55.282527+0000","last_undegraded":"2024-09-10T18:41:55.282527+0000","last_fullsized":"2024-09-10T18:41:55.282527+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:38.888625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:39:38.888625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020721170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'81","reported_seq":163,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282547+0000","last_change":"2024-09-10T18:39:00.890324+0000","last_active":"2024-09-10T18:41:55.282547+0000","last_peered":"2024-09-10T18:41:55.282547+0000","last_clean":"2024-09-10T18:41:55.282547+0000","last_became_active":"2024-09-10T18:39:00.889998+0000","last_became_peered":"2024-09-10T18:39:00.889998+0000","last_unstale":"2024-09-10T18:41:55.282547+0000","last_undegraded":"2024-09-10T18:41:55.282547+0000","last_fullsized":"2024-09-10T18:41:55.282547+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:38:52.175598+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:38:52.175598+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00138405,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"400'12","reported_seq":458,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282548+0000","last_change":"2024-09-10T18:41:40.949560+0000","last_active":"2024-09-10T18:41:55.282548+0000","last_peered":"2024-09-10T18:41:55.282548+0000","last_clean":"2024-09-10T18:41:55.282548+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:41:55.282548+0000","last_undegraded":"2024-09-10T18:41:55.282548+0000","last_fullsized":"2024-09-10T18:41:55.282548+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'12","last_scrub_stamp":"2024-09-10T18:41:40.949115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:40.949115+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:05.051357+0000","scrub_duration":0.022213717000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":468,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.279101+0000","last_change":"2024-09-10T18:41:40.970661+0000","last_active":"2024-09-10T18:41:55.279101+0000","last_peered":"2024-09-10T18:41:55.279101+0000","last_clean":"2024-09-10T18:41:55.279101+0000","last_became_active":"2024-09-10T18:39:25.242295+0000","last_became_peered":"2024-09-10T18:39:25.242295+0000","last_unstale":"2024-09-10T18:41:55.279101+0000","last_undegraded":"2024-09-10T18:41:55.279101+0000","last_fullsized":"2024-09-10T18:41:55.279101+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:40.970571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:41:40.970571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:46.951491+0000","scrub_duration":0.0072197229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":402,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.284010+0000","last_change":"2024-09-10T18:40:37.829615+0000","last_active":"2024-09-10T18:41:55.284010+0000","last_peered":"2024-09-10T18:41:55.284010+0000","last_clean":"2024-09-10T18:41:55.284010+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:41:55.284010+0000","last_undegraded":"2024-09-10T18:41:55.284010+0000","last_fullsized":"2024-09-10T18:41:55.284010+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:37.829541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:40:37.829541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068983380000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":436,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282327+0000","last_change":"2024-09-10T18:41:36.315811+0000","last_active":"2024-09-10T18:41:55.282327+0000","last_peered":"2024-09-10T18:41:55.282327+0000","last_clean":"2024-09-10T18:41:55.282327+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:41:55.282327+0000","last_undegraded":"2024-09-10T18:41:55.282327+0000","last_fullsized":"2024-09-10T18:41:55.282327+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:36.315732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:41:36.315732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:05.356585+0000","scrub_duration":0.015270002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'81","reported_seq":153,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.273177+0000","last_change":"2024-09-10T18:41:01.033724+0000","last_active":"2024-09-10T18:41:55.273177+0000","last_peered":"2024-09-10T18:41:55.273177+0000","last_clean":"2024-09-10T18:41:55.273177+0000","last_became_active":"2024-09-10T18:34:34.867380+0000","last_became_peered":"2024-09-10T18:34:34.867380+0000","last_unstale":"2024-09-10T18:41:55.273177+0000","last_undegraded":"2024-09-10T18:41:55.273177+0000","last_fullsized":"2024-09-10T18:41:55.273177+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:01.033650+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:01.033650+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:02.408279+0000","scrub_duration":0.0033135310000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":467,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.278893+0000","last_change":"2024-09-10T18:40:57.216412+0000","last_active":"2024-09-10T18:41:55.278893+0000","last_peered":"2024-09-10T18:41:55.278893+0000","last_clean":"2024-09-10T18:41:55.278893+0000","last_became_active":"2024-09-10T18:39:00.909906+0000","last_became_peered":"2024-09-10T18:39:00.909906+0000","last_unstale":"2024-09-10T18:41:55.278893+0000","last_undegraded":"2024-09-10T18:41:55.278893+0000","last_fullsized":"2024-09-10T18:41:55.278893+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:57.216321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_clean_scrub_stamp":"2024-09-10T18:40:57.216321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:12.944857+0000","scrub_duration":0.0069155350000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":435,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283092+0000","last_change":"2024-09-10T18:39:43.927945+0000","last_active":"2024-09-10T18:41:55.283092+0000","last_peered":"2024-09-10T18:41:55.283092+0000","last_clean":"2024-09-10T18:41:55.283092+0000","last_became_active":"2024-09-10T18:39:25.250206+0000","last_became_peered":"2024-09-10T18:39:25.250206+0000","last_unstale":"2024-09-10T18:41:55.283092+0000","last_undegraded":"2024-09-10T18:41:55.283092+0000","last_fullsized":"2024-09-10T18:41:55.283092+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:43.927882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:25:18.030878+0000","last_clean_scrub_stamp":"2024-09-10T18:39:43.927882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017896539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"400'32","reported_seq":534,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.283711+0000","last_change":"2024-09-10T18:40:53.949653+0000","last_active":"2024-09-10T18:41:55.283711+0000","last_peered":"2024-09-10T18:41:55.283711+0000","last_clean":"2024-09-10T18:41:55.283711+0000","last_became_active":"2024-09-10T18:39:00.921476+0000","last_became_peered":"2024-09-10T18:39:00.921476+0000","last_unstale":"2024-09-10T18:41:55.283711+0000","last_undegraded":"2024-09-10T18:41:55.283711+0000","last_fullsized":"2024-09-10T18:41:55.283711+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'24","last_scrub_stamp":"2024-09-10T18:40:53.949577+0000","last_deep_scrub":"369'24","last_deep_scrub_stamp":"2024-09-10T18:39:50.690827+0000","last_clean_scrub_stamp":"2024-09-10T18:40:53.949577+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:07.094803+0000","scrub_duration":0.003590818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":65,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-10T18:41:55.282957+0000","last_change":"2024-09-10T18:40:51.034156+0000","last_active":"2024-09-10T18:41:55.282957+0000","last_peered":"2024-09-10T18:41:55.282957+0000","last_clean":"2024-09-10T18:41:55.282957+0000","last_became_active":"2024-09-10T18:39:25.243964+0000","last_became_peered":"2024-09-10T18:39:25.243964+0000","last_unstale":"2024-09-10T18:41:55.282957+0000","last_undegraded":"2024-09-10T18:41:55.282957+0000","last_fullsized":"2024-09-10T18:41:55.282957+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:51.034067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:40:51.034067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095335249999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":7756120088,"num_objects":118350,"num_object_clones":0,"num_object_copies":355050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118350,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162860,"num_write_kb":8998657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23251988480,"data_stored":23251976264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63844,"ondisk_log_size":63844,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1548,"num_read_kb":1305,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":4566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466944,"data_stored":12426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":1054,"ondisk_log_size":1054,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1224,"num_read_kb":1219,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2511,"ondisk_log_size":2511,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234234,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4225348,"kb_used_data":3343228,"kb_used_omap":1,"kb_used_meta":882110,"kb_avail":89519804,"statfs":{"total":95995035648,"available":91668279296,"internally_reserved":0,"allocated":3423465472,"data_stored":3423078227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":903280984},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":1.379,"15min":1.7090000000000001},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.21099999999999999},"max":{"1min":2.2250000000000001,"5min":29.443999999999999,"15min":75.506},"last":0.40400000000000003},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.53,"15min":1.655},"min":{"1min":0.26500000000000001,"5min":0.25700000000000001,"15min":0.223},"max":{"1min":12.571999999999999,"5min":29.530999999999999,"15min":75.710999999999999},"last":0.28699999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:42:02 2024","interfaces":[{"interface":"back","average":{"1min":1.8160000000000001,"5min":2.1070000000000002,"15min":1.839},"min":{"1min":0.23400000000000001,"5min":0.22800000000000001,"15min":0.214},"max":{"1min":12.461,"5min":98.305999999999997,"15min":98.305999999999997},"last":0.375},{"interface":"front","average":{"1min":1.262,"5min":2.1589999999999998,"15min":1.772},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":9.4969999999999999,"5min":98.381,"15min":98.381},"last":0.307}]},{"osd":2,"last update":"Tue Sep 10 18:42:02 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.994,"15min":1.8280000000000001},"min":{"1min":0.27300000000000002,"5min":0.27100000000000002,"15min":0.26400000000000001},"max":{"1min":5.3840000000000003,"5min":44.728000000000002,"15min":136.648},"last":1.6779999999999999},{"interface":"front","average":{"1min":0.79900000000000004,"5min":1.9330000000000001,"15min":1.8700000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.24399999999999999},"max":{"1min":2.4670000000000001,"5min":44.664999999999999,"15min":136.506},"last":0.44800000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:42:02 2024","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":1.135,"15min":1.2809999999999999},"min":{"1min":0.34999999999999998,"5min":0.30199999999999999,"15min":0.27200000000000002},"max":{"1min":8.7400000000000002,"5min":8.7400000000000002,"15min":38.225000000000001},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.1040000000000001,"15min":1.274},"min":{"1min":0.33600000000000002,"5min":0.29899999999999999,"15min":0.29199999999999998},"max":{"1min":12.516,"5min":12.516,"15min":38.104999999999997},"last":0.42699999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.5009999999999999,"15min":0.90300000000000002},"min":{"1min":0.23899999999999999,"5min":0.22700000000000001,"15min":0.224},"max":{"1min":4.6079999999999997,"5min":23.346,"15min":23.346},"last":0.248},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.3999999999999999,"15min":0.91700000000000004},"min":{"1min":0.30099999999999999,"5min":0.26200000000000001,"15min":0.25},"max":{"1min":5.4219999999999997,"5min":23.323,"15min":23.323},"last":1.6100000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":1.111,"15min":0.97599999999999998},"min":{"1min":0.27900000000000003,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":2.431,"5min":16.756,"15min":16.756},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.873,"5min":1.349,"15min":0.98799999999999999},"min":{"1min":0.35699999999999998,"5min":0.224,"15min":0.224},"max":{"1min":4.5449999999999999,"5min":20.495999999999999,"15min":20.495999999999999},"last":0.32700000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:41:52 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.319,"15min":0.97299999999999998},"min":{"1min":0.32100000000000001,"5min":0.317,"15min":0.27000000000000002},"max":{"1min":4.6689999999999996,"5min":25.902999999999999,"15min":25.902999999999999},"last":1.371},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.3939999999999999,"15min":0.93999999999999995},"min":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.30199999999999999},"max":{"1min":4.944,"5min":24.074000000000002,"15min":24.074000000000002},"last":0.34599999999999997}]}]},{"osd":6,"up_from":144,"seq":618475290813,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4668904,"kb_used_data":3783904,"kb_used_omap":7,"kb_used_meta":884984,"kb_avail":89076248,"statfs":{"total":95995035648,"available":91214077952,"internally_reserved":0,"allocated":3874717696,"data_stored":3874358445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":906223703},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":1.0369999999999999,"15min":1.2549999999999999},"min":{"1min":0.25600000000000001,"5min":0.23300000000000001,"15min":0.192},"max":{"1min":3.6800000000000002,"5min":5.5469999999999997,"15min":30.629999999999999},"last":0.313},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.026,"15min":1.1499999999999999},"min":{"1min":0.27500000000000002,"5min":0.246,"15min":0.217},"max":{"1min":3.9609999999999999,"5min":4.9489999999999998,"15min":30.597000000000001},"last":0.32700000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:42:00 2024","interfaces":[{"interface":"back","average":{"1min":2.774,"5min":1.4059999999999999,"15min":1.1970000000000001},"min":{"1min":0.312,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":23.782,"5min":23.782,"15min":30.984000000000002},"last":2.3410000000000002},{"interface":"front","average":{"1min":2.335,"5min":1.355,"15min":1.262},"min":{"1min":0.32900000000000001,"5min":0.253,"15min":0.24199999999999999},"max":{"1min":23.687999999999999,"5min":23.687999999999999,"15min":30.571000000000002},"last":1.51}]},{"osd":2,"last update":"Tue Sep 10 18:42:00 2024","interfaces":[{"interface":"back","average":{"1min":2.3450000000000002,"5min":1.4490000000000001,"15min":1.3160000000000001},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.25600000000000001},"max":{"1min":23.763999999999999,"5min":23.763999999999999,"15min":39.329999999999998},"last":0.40100000000000002},{"interface":"front","average":{"1min":2.9929999999999999,"5min":1.7210000000000001,"15min":1.3080000000000001},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.23400000000000001},"max":{"1min":23.856000000000002,"5min":23.856000000000002,"15min":39.061999999999998},"last":1.3360000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:42:00 2024","interfaces":[{"interface":"back","average":{"1min":2.3319999999999999,"5min":4.8920000000000003,"15min":2.6619999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.27800000000000002},"max":{"1min":29.670999999999999,"5min":162.61600000000001,"15min":162.61600000000001},"last":1.9299999999999999},{"interface":"front","average":{"1min":2.2970000000000002,"5min":4.8220000000000001,"15min":2.6629999999999998},"min":{"1min":0.39900000000000002,"5min":0.29599999999999999,"15min":0.27100000000000002},"max":{"1min":32.018000000000001,"5min":163.39599999999999,"15min":163.39599999999999},"last":0.29599999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.82999999999999996,"15min":0.69299999999999995},"min":{"1min":0.222,"5min":0.189,"15min":0.189},"max":{"1min":3.9279999999999999,"5min":4.9050000000000002,"15min":5.4950000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.22800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":3.0169999999999999,"5min":11.531000000000001,"15min":11.531000000000001},"last":0.33700000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.79600000000000004,"15min":0.70999999999999996},"min":{"1min":0.26100000000000001,"5min":0.254,"15min":0.24399999999999999},"max":{"1min":2.359,"5min":4.4139999999999997,"15min":4.6619999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.85899999999999999,"15min":0.75600000000000001},"min":{"1min":0.28000000000000003,"5min":0.248,"15min":0.20899999999999999},"max":{"1min":2.0499999999999998,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":0.75}]},{"osd":7,"last update":"Tue Sep 10 18:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":1.0129999999999999,"15min":0.78700000000000003},"min":{"1min":0.34699999999999998,"5min":0.26700000000000002,"15min":0.23799999999999999},"max":{"1min":3.1749999999999998,"5min":11.449,"15min":11.449},"last":1.3069999999999999},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.82099999999999995,"15min":0.73799999999999999},"min":{"1min":0.36199999999999999,"5min":0.32000000000000001,"15min":0.27100000000000002},"max":{"1min":3.754,"5min":5.0149999999999997,"15min":5.5309999999999997},"last":0.46999999999999997}]}]},{"osd":5,"up_from":39,"seq":167503724805,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2792700,"kb_used_data":1901556,"kb_used_omap":1,"kb_used_meta":891134,"kb_avail":90952452,"statfs":{"total":95995035648,"available":93135310848,"internally_reserved":0,"allocated":1947193344,"data_stored":1946814549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912521560},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:41:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":1.0569999999999999,"15min":1.476},"min":{"1min":0.245,"5min":0.224,"15min":0.224},"max":{"1min":1.8100000000000001,"5min":7.9020000000000001,"15min":52.033999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.99299999999999999,"15min":1.407},"min":{"1min":0.38,"5min":0.24399999999999999,"15min":0.22800000000000001},"max":{"1min":4.851,"5min":4.851,"15min":52.167999999999999},"last":0.27900000000000003}]},{"osd":1,"last update":"Tue Sep 10 18:41:16 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.92400000000000004,"15min":1.599},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":2.9670000000000001,"5min":8.7490000000000006,"15min":111.773},"last":1.181},{"interface":"front","average":{"1min":1.2370000000000001,"5min":0.92300000000000004,"15min":1.5820000000000001},"min":{"1min":0.376,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.819,"5min":4.819,"15min":111.73999999999999},"last":1.1919999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:41:16 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.246,"15min":1.3939999999999999},"min":{"1min":0.315,"5min":0.26900000000000002,"15min":0.24299999999999999},"max":{"1min":2.677,"5min":9.8759999999999994,"15min":54.530000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.99299999999999999,"15min":1.28},"min":{"1min":0.27800000000000002,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":2.524,"5min":3.9780000000000002,"15min":54.514000000000003},"last":0.52000000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:41:16 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.298,"15min":1.966},"min":{"1min":0.34000000000000002,"5min":0.29999999999999999,"15min":0.27100000000000002},"max":{"1min":3.6339999999999999,"5min":18.911999999999999,"15min":106.205},"last":0.41599999999999998},{"interface":"front","average":{"1min":1.097,"5min":1.1739999999999999,"15min":1.946},"min":{"1min":0.28999999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.3809999999999998,"5min":18.949999999999999,"15min":105.878},"last":1.004}]},{"osd":4,"last update":"Tue Sep 10 18:41:16 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.86899999999999999,"15min":0.71499999999999997},"min":{"1min":0.35299999999999998,"5min":0.23799999999999999,"15min":0.20799999999999999},"max":{"1min":4.7750000000000004,"5min":4.7750000000000004,"15min":5.8040000000000003},"last":0.753},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.97299999999999998,"15min":0.78800000000000003},"min":{"1min":0.28499999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":3.226,"5min":8.7270000000000003,"15min":8.7270000000000003},"last":0.375}]},{"osd":6,"last update":"Tue Sep 10 18:41:49 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":0.92000000000000004,"15min":0.76500000000000001},"min":{"1min":0.317,"5min":0.29999999999999999,"15min":0.27500000000000002},"max":{"1min":2.5659999999999998,"5min":7.9969999999999999,"15min":7.9969999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.2909999999999999,"15min":0.91200000000000003},"min":{"1min":0.32800000000000001,"5min":0.30299999999999999,"15min":0.27000000000000002},"max":{"1min":8.7400000000000002,"5min":8.7400000000000002,"15min":8.7400000000000002},"last":0.61899999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:41:49 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.0620000000000001,"15min":0.95999999999999996},"min":{"1min":0.35899999999999999,"5min":0.32200000000000001,"15min":0.25900000000000001},"max":{"1min":6.867,"5min":8.3870000000000005,"15min":28.934000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.0289999999999999,"15min":0.86199999999999999},"min":{"1min":0.36099999999999999,"5min":0.314,"15min":0.25900000000000001},"max":{"1min":5.593,"5min":5.593,"15min":5.8259999999999996},"last":0.49299999999999999}]}]},{"osd":4,"up_from":33,"seq":141733921033,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3758264,"kb_used_data":2867248,"kb_used_omap":1,"kb_used_meta":891006,"kb_avail":89986888,"statfs":{"total":95995035648,"available":92146573312,"internally_reserved":0,"allocated":2936061952,"data_stored":2935685956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912390488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:41:52 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.5920000000000001,"15min":1.2609999999999999},"min":{"1min":0.247,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.8080000000000001,"5min":21.468,"15min":21.468},"last":1.8220000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":1.3240000000000001,"15min":1.0800000000000001},"min":{"1min":0.28399999999999997,"5min":0.21299999999999999,"15min":0.189},"max":{"1min":3.0409999999999999,"5min":24.277999999999999,"15min":24.277999999999999},"last":0.38}]},{"osd":1,"last update":"Tue Sep 10 18:41:52 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.3779999999999999,"15min":1.2589999999999999},"min":{"1min":0.246,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":8.2539999999999996,"5min":12.696,"15min":50.045000000000002},"last":2.0390000000000001},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.2629999999999999,"15min":1.296},"min":{"1min":0.29399999999999998,"5min":0.28000000000000003,"15min":0.246},"max":{"1min":4.0380000000000003,"5min":13.632,"15min":50.283000000000001},"last":2.1389999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:41:52 2024","interfaces":[{"interface":"back","average":{"1min":12.504,"5min":4.4640000000000004,"15min":2.0470000000000002},"min":{"1min":0.32800000000000001,"5min":0.251,"15min":0.249},"max":{"1min":169.417,"5min":169.417,"15min":169.417},"last":2.048},{"interface":"front","average":{"1min":12.183999999999999,"5min":4.4980000000000002,"15min":2.0609999999999999},"min":{"1min":0.251,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":169.40100000000001,"5min":169.40100000000001,"15min":169.40100000000001},"last":0.59499999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:41:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.0469999999999999,"15min":1.3500000000000001},"min":{"1min":0.315,"5min":0.26200000000000001,"15min":0.249},"max":{"1min":5.2830000000000004,"5min":7.7699999999999996,"15min":49.990000000000002},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.83799999999999997,"5min":1.508,"15min":1.5329999999999999},"min":{"1min":0.33700000000000002,"5min":0.28999999999999998,"15min":0.28000000000000003},"max":{"1min":2.0299999999999998,"5min":10.119999999999999,"15min":50.764000000000003},"last":1.861}]},{"osd":5,"last update":"Tue Sep 10 18:41:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.83499999999999996,"15min":0.80000000000000004},"min":{"1min":0.30599999999999999,"5min":0.25,"15min":0.20499999999999999},"max":{"1min":2.9420000000000002,"5min":3.492,"15min":22.861000000000001},"last":2.1099999999999999},{"interface":"front","average":{"1min":0.79600000000000004,"5min":1.034,"15min":0.88500000000000001},"min":{"1min":0.32100000000000001,"5min":0.252,"15min":0.22},"max":{"1min":2.7050000000000001,"5min":8.2539999999999996,"15min":19.611999999999998},"last":0.34799999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:41:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.6399999999999999,"15min":0.98199999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.26400000000000001},"max":{"1min":4.4900000000000002,"5min":51.722000000000001,"15min":51.722000000000001},"last":1.998},{"interface":"front","average":{"1min":1.069,"5min":1.5049999999999999,"15min":0.94999999999999996},"min":{"1min":0.36199999999999999,"5min":0.31,"15min":0.30599999999999999},"max":{"1min":2.6960000000000002,"5min":51.862000000000002,"15min":51.862000000000002},"last":2.327}]},{"osd":7,"last update":"Tue Sep 10 18:41:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.022,"15min":0.78900000000000003},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.28799999999999998},"max":{"1min":2.766,"5min":7.6260000000000003,"15min":7.6260000000000003},"last":2.2759999999999998},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0600000000000001,"15min":0.81799999999999995},"min":{"1min":0.375,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.323,"5min":7.8250000000000002,"15min":7.8250000000000002},"last":0.61499999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117263,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4269192,"kb_used_data":3373440,"kb_used_omap":1,"kb_used_meta":895742,"kb_avail":89475960,"statfs":{"total":95995035648,"available":91623383040,"internally_reserved":0,"allocated":3454402560,"data_stored":3454011499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917240152},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:41:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.361,"15min":1.3180000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.18099999999999999},"max":{"1min":3.7149999999999999,"5min":29.981999999999999,"15min":29.981999999999999},"last":0.497},{"interface":"front","average":{"1min":0.83699999999999997,"5min":1.556,"15min":1.486},"min":{"1min":0.218,"5min":0.193,"15min":0.193},"max":{"1min":3.073,"5min":28.754000000000001,"15min":28.754000000000001},"last":0.23899999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:41:26 2024","interfaces":[{"interface":"back","average":{"1min":1.615,"5min":1.3899999999999999,"15min":1.589},"min":{"1min":0.436,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":4.46,"5min":28.539000000000001,"15min":35.406999999999996},"last":0.20499999999999999},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.274,"15min":1.4870000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.9630000000000001,"5min":29.032,"15min":36.902000000000001},"last":0.58999999999999997}]},{"osd":2,"last update":"Tue Sep 10 18:41:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.9059999999999999,"15min":1.4039999999999999},"min":{"1min":0.22800000000000001,"5min":0.191,"15min":0.191},"max":{"1min":2.698,"5min":79.975999999999999,"15min":79.975999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.22,"5min":2.3460000000000001,"15min":1.7589999999999999},"min":{"1min":0.29099999999999998,"5min":0.22,"15min":0.217},"max":{"1min":4.5899999999999999,"5min":78.278000000000006,"15min":78.278000000000006},"last":0.317}]},{"osd":4,"last update":"Tue Sep 10 18:41:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.1879999999999999,"15min":1.1899999999999999},"min":{"1min":0.372,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.726,"5min":6.1699999999999999,"15min":6.5250000000000004},"last":0.36799999999999999},{"interface":"front","average":{"1min":1.036,"5min":1.1619999999999999,"15min":1.1140000000000001},"min":{"1min":0.28399999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.2559999999999998,"5min":4.8499999999999996,"15min":6.1929999999999996},"last":1.476}]},{"osd":5,"last update":"Tue Sep 10 18:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.0309999999999999,"15min":1.2869999999999999},"min":{"1min":0.32700000000000001,"5min":0.32300000000000001,"15min":0.314},"max":{"1min":3.6019999999999999,"5min":6.2370000000000001,"15min":7.3339999999999996},"last":0.34599999999999997},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0409999999999999,"15min":1.23},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":3.1349999999999998,"5min":4.8970000000000002,"15min":6.2249999999999996},"last":0.52300000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:41:56 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.23,"15min":1.3640000000000001},"min":{"1min":0.40999999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.2850000000000001,"5min":6.8129999999999997,"15min":31.815999999999999},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.589,"5min":1.2310000000000001,"15min":1.4219999999999999},"min":{"1min":0.40699999999999997,"5min":0.33200000000000002,"15min":0.315},"max":{"1min":8.7569999999999997,"5min":8.7569999999999997,"15min":31.742000000000001},"last":1.538}]},{"osd":7,"last update":"Tue Sep 10 18:41:56 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.1140000000000001,"15min":1.25},"min":{"1min":0.36799999999999999,"5min":0.308,"15min":0.308},"max":{"1min":2.802,"5min":6.2119999999999997,"15min":6.5730000000000004},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.071,"5min":1.1080000000000001,"15min":1.2390000000000001},"min":{"1min":0.34100000000000003,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.4900000000000002,"5min":4.9370000000000003,"15min":12.612},"last":0.55300000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411603,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4188672,"kb_used_data":3296504,"kb_used_omap":1,"kb_used_meta":892094,"kb_avail":89556480,"statfs":{"total":95995035648,"available":91705835520,"internally_reserved":0,"allocated":3375620096,"data_stored":3375236474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":913504600},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:42:04 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.157,"15min":1.4099999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.17499999999999999},"max":{"1min":3.5169999999999999,"5min":3.8610000000000002,"15min":43.890000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.5149999999999999,"15min":1.7250000000000001},"min":{"1min":0.223,"5min":0.19700000000000001,"15min":0.19400000000000001},"max":{"1min":2.657,"5min":10.307,"15min":44.331000000000003},"last":0.65200000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:42:04 2024","interfaces":[{"interface":"back","average":{"1min":2.0979999999999999,"5min":1.651,"15min":1.5780000000000001},"min":{"1min":0.27700000000000002,"5min":0.222,"15min":0.193},"max":{"1min":11.154,"5min":11.154,"15min":44.575000000000003},"last":1.117},{"interface":"front","average":{"1min":2.1549999999999998,"5min":1.2769999999999999,"15min":1.3400000000000001},"min":{"1min":0.46300000000000002,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":10.637,"5min":10.637,"15min":44.368000000000002},"last":1.484}]},{"osd":3,"last update":"Tue Sep 10 18:41:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.7210000000000001,"15min":1.8120000000000001},"min":{"1min":0.34799999999999998,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.7210000000000001,"5min":45.725000000000001,"15min":61.331000000000003},"last":1.528},{"interface":"front","average":{"1min":1.7310000000000001,"5min":2.0609999999999999,"15min":1.9590000000000001},"min":{"1min":0.33100000000000002,"5min":0.22600000000000001,"15min":0.214},"max":{"1min":6.2329999999999997,"5min":45.677,"15min":61.250999999999998},"last":1.014}]},{"osd":4,"last update":"Tue Sep 10 18:41:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.653,"15min":1.2470000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.24199999999999999},"max":{"1min":5.3129999999999997,"5min":9.8810000000000002,"15min":12.555999999999999},"last":1.095},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.512,"15min":1.135},"min":{"1min":0.35099999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.5550000000000002,"5min":19.696000000000002,"15min":19.696000000000002},"last":1.46}]},{"osd":5,"last update":"Tue Sep 10 18:41:12 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":1.538,"15min":1.2350000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":2.2669999999999999,"5min":25.177,"15min":25.177},"last":1.4330000000000001},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.427,"15min":1.274},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.317},"max":{"1min":3.375,"5min":24.986999999999998,"15min":24.986999999999998},"last":1.5069999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:41:58 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":2.2890000000000001,"15min":1.484},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":5.2770000000000001,"5min":53.359999999999999,"15min":53.359999999999999},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.1379999999999999,"5min":2.0630000000000002,"15min":1.3740000000000001},"min":{"1min":0.42599999999999999,"5min":0.32900000000000001,"15min":0.30099999999999999},"max":{"1min":3.6070000000000002,"5min":53.438000000000002,"15min":53.438000000000002},"last":1.3999999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:42:04 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.371,"15min":1.242},"min":{"1min":0.36699999999999999,"5min":0.36499999999999999,"15min":0.34200000000000003},"max":{"1min":3.2599999999999998,"5min":6.1980000000000004,"15min":10.68},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.7,"15min":1.3759999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.311},"max":{"1min":5.2320000000000002,"5min":9.8499999999999996,"15min":12.596},"last":1.0629999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575128,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5144084,"kb_used_data":4255500,"kb_used_omap":1,"kb_used_meta":888574,"kb_avail":88601068,"statfs":{"total":95995035648,"available":90727493632,"internally_reserved":0,"allocated":4357632000,"data_stored":4357228250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":909900120},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":1.2390000000000001,"15min":1.482},"min":{"1min":0.184,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":1.0329999999999999,"5min":22.042999999999999,"15min":42.420000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":1.173,"15min":1.502},"min":{"1min":0.24299999999999999,"5min":0.19800000000000001,"15min":0.187},"max":{"1min":1.1160000000000001,"5min":5.5140000000000002,"15min":42.548000000000002},"last":0.42299999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:42:04 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3080000000000001,"15min":1.2350000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.19500000000000001},"max":{"1min":11.079000000000001,"5min":11.079000000000001,"15min":11.079000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":1.546,"5min":1.47,"15min":1.3100000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.20999999999999999},"max":{"1min":6.274,"5min":9.0749999999999993,"15min":12.222},"last":3.3340000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:41:34 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.1160000000000001,"15min":1.4319999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":4.6230000000000002,"5min":4.8449999999999998,"15min":43.076000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.1830000000000001,"15min":1.4770000000000001},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.20499999999999999},"max":{"1min":3.117,"5min":5.5739999999999998,"15min":41.93},"last":0.80900000000000005}]},{"osd":4,"last update":"Tue Sep 10 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":2.6539999999999999,"15min":1.6950000000000001},"min":{"1min":0.434,"5min":0.33800000000000002,"15min":0.312},"max":{"1min":2.536,"5min":127.86199999999999,"15min":127.86199999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.4750000000000001,"5min":2.714,"15min":1.8049999999999999},"min":{"1min":0.34899999999999998,"5min":0.33900000000000002,"15min":0.30199999999999999},"max":{"1min":5.5709999999999997,"5min":127.84399999999999,"15min":127.84399999999999},"last":3.4009999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":2.6379999999999999,"15min":1.865},"min":{"1min":0.33900000000000002,"5min":0.29799999999999999,"15min":0.27900000000000003},"max":{"1min":2.23,"5min":118.68000000000001,"15min":118.68000000000001},"last":3.3740000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":2.609,"15min":1.841},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.6840000000000002,"5min":118.36,"15min":118.36},"last":0.60099999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:42:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":6.9699999999999998,"15min":3.3300000000000001},"min":{"1min":0.41699999999999998,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":4.8730000000000002,"5min":389.52499999999998,"15min":389.52499999999998},"last":0.625},{"interface":"front","average":{"1min":0.96999999999999997,"5min":7.4349999999999996,"15min":3.5790000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.181,"5min":389.69499999999999,"15min":389.69499999999999},"last":0.76100000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:42:04 2024","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.3819999999999999,"15min":1.2989999999999999},"min":{"1min":0.44400000000000001,"5min":0.36399999999999999,"15min":0.32700000000000001},"max":{"1min":5.0119999999999996,"5min":16.244,"15min":16.244},"last":5.0119999999999996},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.331,"15min":1.3320000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":5.4669999999999996,"5min":16.550000000000001,"15min":16.550000000000001},"last":0.78900000000000003}]}]},{"osd":0,"up_from":0,"seq":34359738653,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4342743040,"data_stored":4342743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3369734144,"data_stored":3369730072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3415277568,"data_stored":3415277568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2932281344,"data_stored":2932277272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1939406848,"data_stored":1939406848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3866497024,"data_stored":3866492952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3386048512,"data_stored":3386048512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:42:12.425 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-10T18:42:12.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:12.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:12.505 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:12.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:12.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:12.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:12.648 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:12.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:12.749 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:12.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:12.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:12.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:12.927 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:12.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:12.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:13.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:13.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:42:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:13.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:13.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:13.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:13.233 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:13.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:13.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:13.335 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:13.372 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:13.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:13.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:13.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:13.510 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:13.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:13.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:13.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:13.642 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:13.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:13.679 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:13.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:13.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:13.822 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:13.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:13.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:13.953 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:13.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:13.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:14.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:14.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:14.082 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:14.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:14.183 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:14.218 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:14.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:14.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:14.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:14.363 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:14.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:14.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:14.465 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:14 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:42:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:14 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:42:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:14 smithi028 ceph-mon[39220]: pgmap v936: 178 pgs: 178 active+clean; 6.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-10T18:42:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:14 smithi028 ceph-mon[39220]: osdmap e407: 8 total, 8 up, 7 in 2024-09-10T18:42:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:14 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:42:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:14 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:42:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:14 smithi028 ceph-mon[29291]: pgmap v936: 178 pgs: 178 active+clean; 6.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-10T18:42:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:14 smithi028 ceph-mon[29291]: osdmap e407: 8 total, 8 up, 7 in 2024-09-10T18:42:14.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:14.506 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:14.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:14 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:42:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:14 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:42:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:14 smithi204 ceph-mon[32357]: pgmap v936: 178 pgs: 178 active+clean; 6.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-10T18:42:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:14 smithi204 ceph-mon[32357]: osdmap e407: 8 total, 8 up, 7 in 2024-09-10T18:42:14.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:14.642 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:14.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:14.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:14.704 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:14.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:14.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:14.791 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:14.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:14.892 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:14.926 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:14.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:14.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:15.028 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:15.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:15.065 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:15.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:15.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:15.203 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:15.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:15.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:15.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:15.344 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:15.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:15.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:15.353 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:42:15 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:42:15.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:15.482 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:15.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:15.584 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:15.617 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:15.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:15.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:15.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:15.753 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:15.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:15.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:15.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:15.893 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:15.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:15.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:15.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:16.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:16.031 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:16.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:16.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:16.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:16.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:16.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:16.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:16.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:16.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:16.458 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:16.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:16.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:16.479 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:42:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:42:16.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:16 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:16.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:16 smithi028 ceph-mon[39220]: pgmap v938: 178 pgs: 178 active+clean; 6.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 738 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:42:16.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:16 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:16.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:16 smithi028 ceph-mon[29291]: pgmap v938: 178 pgs: 178 active+clean; 6.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 738 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:42:16.511 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:42:16.512 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:42:16.544 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:42:16.544 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:42:16.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:16.580 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:42:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:42:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:16.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:16.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:16.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:16 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:16.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:16 smithi204 ceph-mon[32357]: pgmap v938: 178 pgs: 178 active+clean; 6.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 738 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:42:16.610 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:42:16.610 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:42:16.639 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:42:16.639 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:42:16.672 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:42:16.672 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:42:16.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:16.706 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:42:16.706 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:42:16.741 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:16.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:16.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:16.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:16.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:16.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:16.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:16.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:17.014 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:17.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:17.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:17.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:17.153 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:17.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:17.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:17.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:17.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:17.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:17.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:17.426 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:42:17.427 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:42:17.427 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-10T18:42:17.427 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-10T18:42:17.427 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-10T18:42:17.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-10T18:42:17.432 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:17.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:17.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:17.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:17 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4005720007' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:42:17.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:17 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4005720007' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:42:17.534 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:17.569 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:17.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:17.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:17.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:17 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4005720007' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:42:17.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:17.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:17.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:17.814 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:17.846 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:17.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:17.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:17.948 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:17.978 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:17.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:17.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:18.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:18.113 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:18.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:18.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:18.252 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:18.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:18.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:18.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:18.392 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:18.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:18.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:18.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:18.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:18 smithi028 ceph-mon[29291]: pgmap v939: 178 pgs: 178 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s 2024-09-10T18:42:18.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:18 smithi028 ceph-mon[39220]: pgmap v939: 178 pgs: 178 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s 2024-09-10T18:42:18.530 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:18.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:18.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:18.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:18 smithi204 ceph-mon[32357]: pgmap v939: 178 pgs: 178 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s 2024-09-10T18:42:18.632 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:18.666 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:18.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:18.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:18.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:18.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:18.808 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:18.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:18.909 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:18.946 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:18.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:18.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:19.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:19.073 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:19.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:19.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:19.175 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:19.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:19.210 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:19.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:19.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:19.352 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:19.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:19.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:19.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:19.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:19.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:19.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:19.631 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:19.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:19.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:19.733 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:19.764 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:19.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:19.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:19.809 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:42:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:42:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.25849.0:181273 11.b e407/374 11:d47ccb59:::benchmark_data_smithi028_186962_object40666:head v 407'11447, mlcod=407'11445)", 2024-09-10T18:42:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:42:17.926484+0000", 2024-09-10T18:42:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0053030179999999996, 2024-09-10T18:42:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0053302820000000004, 2024-09-10T18:42:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:42:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:42:19.809 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:42:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:42:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926484+0000", 2024-09-10T18:42:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:42:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926484+0000", 2024-09-10T18:42:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:42:19.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926483+0000", 2024-09-10T18:42:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:42:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:42:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926492+0000", 2024-09-10T18:42:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3200000000000006e-06 2024-09-10T18:42:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:42:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926505+0000", 2024-09-10T18:42:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2262999999999999e-05 2024-09-10T18:42:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:42:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926511+0000", 2024-09-10T18:42:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1099999999999999e-06 2024-09-10T18:42:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:42:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.927122+0000", 2024-09-10T18:42:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00061103399999999999 2024-09-10T18:42:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:42:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.927138+0000", 2024-09-10T18:42:19.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6504e-05 2024-09-10T18:42:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:42:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.25849.0:181277 11.b e407/374 11:d12ede9d:::benchmark_data_smithi028_186962_object40670:head v 407'11448, mlcod=407'11445)", 2024-09-10T18:42:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:42:17.926545+0000", 2024-09-10T18:42:19.813 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0052427350000000001, 2024-09-10T18:42:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0053406590000000002, 2024-09-10T18:42:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:42:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:42:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:42:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:42:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926545+0000", 2024-09-10T18:42:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:42:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926545+0000", 2024-09-10T18:42:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:42:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926544+0000", 2024-09-10T18:42:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:42:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:42:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926552+0000", 2024-09-10T18:42:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2450000000000001e-06 2024-09-10T18:42:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:42:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926556+0000", 2024-09-10T18:42:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7439999999999998e-06 2024-09-10T18:42:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:42:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926560+0000", 2024-09-10T18:42:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.287e-06 2024-09-10T18:42:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:42:19.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.929103+0000", 2024-09-10T18:42:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025436069999999998 2024-09-10T18:42:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:42:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.929119+0000", 2024-09-10T18:42:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5475e-05 2024-09-10T18:42:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:42:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.25849.0:181276 11.6 e407/374 11:692ebed0:::benchmark_data_smithi028_186962_object40669:head v 407'15907, mlcod=407'15905)", 2024-09-10T18:42:19.817 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:42:17.926574+0000", 2024-09-10T18:42:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0052130470000000002, 2024-09-10T18:42:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0053729420000000003, 2024-09-10T18:42:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:42:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:42:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:42:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:42:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926574+0000", 2024-09-10T18:42:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:42:19.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926574+0000", 2024-09-10T18:42:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:42:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926574+0000", 2024-09-10T18:42:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-10T18:42:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:42:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926578+0000", 2024-09-10T18:42:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5449999999999997e-06 2024-09-10T18:42:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:42:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926584+0000", 2024-09-10T18:42:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.733e-06 2024-09-10T18:42:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:42:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.926588+0000", 2024-09-10T18:42:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.484e-06 2024-09-10T18:42:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:42:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.928385+0000", 2024-09-10T18:42:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001797772 2024-09-10T18:42:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:42:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.928406+0000", 2024-09-10T18:42:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0041e-05 2024-09-10T18:42:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:42:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.25849.0:181280 11.b e407/374 11:dceb0170:::benchmark_data_smithi028_186962_object40673:head v 407'11449, mlcod=407'11445)", 2024-09-10T18:42:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:42:17.931243+0000", 2024-09-10T18:42:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.00054403199999999996, 2024-09-10T18:42:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00076274199999999996, 2024-09-10T18:42:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:42:19.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:42:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:42:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:42:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931243+0000", 2024-09-10T18:42:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:42:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931243+0000", 2024-09-10T18:42:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:42:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931242+0000", 2024-09-10T18:42:19.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:42:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:42:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931250+0000", 2024-09-10T18:42:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5879999999999998e-06 2024-09-10T18:42:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:42:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931265+0000", 2024-09-10T18:42:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4064e-05 2024-09-10T18:42:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:42:19.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931271+0000", 2024-09-10T18:42:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3910000000000003e-06 2024-09-10T18:42:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:42:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931616+0000", 2024-09-10T18:42:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00034505000000000001 2024-09-10T18:42:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:42:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931639+0000", 2024-09-10T18:42:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3122e-05 2024-09-10T18:42:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:42:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25849.0:181284 11.c 11:3fadae51:::benchmark_data_smithi028_186962_object40677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-09-10T18:42:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:42:17.931805+0000", 2024-09-10T18:42:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4294967295.9999819, 2024-09-10T18:42:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026066300000000001, 2024-09-10T18:42:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:42:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:42:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:42:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25849", 2024-09-10T18:42:19.825 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:42:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181284 2024-09-10T18:42:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:42:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:42:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931805+0000", 2024-09-10T18:42:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:42:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931805+0000", 2024-09-10T18:42:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:42:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931803+0000", 2024-09-10T18:42:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:42:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:42:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931811+0000", 2024-09-10T18:42:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4980000000000006e-06 2024-09-10T18:42:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:42:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931830+0000", 2024-09-10T18:42:19.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8482999999999999e-05 2024-09-10T18:42:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:42:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931837+0000", 2024-09-10T18:42:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7240000000000003e-06 2024-09-10T18:42:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:42:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931943+0000", 2024-09-10T18:42:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000105827 2024-09-10T18:42:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:42:19.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.932065+0000", 2024-09-10T18:42:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122729 2024-09-10T18:42:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:42:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25849.0:181288 11.0 11:060a66df:::benchmark_data_smithi028_186962_object40681:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-09-10T18:42:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:42:17.931890+0000", 2024-09-10T18:42:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4294967295.9998975, 2024-09-10T18:42:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024360399999999999, 2024-09-10T18:42:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:42:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-10T18:42:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:42:19.829 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25849", 2024-09-10T18:42:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:42:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 181288 2024-09-10T18:42:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:42:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:42:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931890+0000", 2024-09-10T18:42:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:42:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931890+0000", 2024-09-10T18:42:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:19.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:42:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931889+0000", 2024-09-10T18:42:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:42:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:42:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931894+0000", 2024-09-10T18:42:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1309999999999997e-06 2024-09-10T18:42:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:42:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931902+0000", 2024-09-10T18:42:19.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2709999999999992e-06 2024-09-10T18:42:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:42:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.931907+0000", 2024-09-10T18:42:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8590000000000004e-06 2024-09-10T18:42:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:42:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.932005+0000", 2024-09-10T18:42:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8389999999999998e-05 2024-09-10T18:42:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:19.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:42:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:17.932125+0000", 2024-09-10T18:42:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000120044 2024-09-10T18:42:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:42:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:42:19.833 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 6 2024-09-10T18:42:19.833 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:42:19.866 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:19.904 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:19.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:19.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:20.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:20.054 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:20.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:20.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:20.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:20.196 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:20.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:20.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:20.298 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:20.331 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:20.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:20.433 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:20.469 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:20.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:20.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:20.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:20 smithi028 ceph-mon[29291]: pgmap v940: 178 pgs: 178 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T18:42:20.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:20 smithi028 ceph-mon[39220]: pgmap v940: 178 pgs: 178 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T18:42:20.571 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:20.601 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:20.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:20.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:20 smithi204 ceph-mon[32357]: pgmap v940: 178 pgs: 178 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T18:42:20.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-10T18:42:20.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:20.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:20.749 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:20.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:20.850 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:20.880 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:20.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:20.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:20.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:21.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:21.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:21.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:21.147 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:21.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:21.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:21.249 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:21.284 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:21.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:21.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:21.386 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:21.419 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:21.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:21.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:21.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:21.555 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:21.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:21.598 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:42:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:42:21] "GET /metrics HTTP/1.1" 200 224957 "" "Prometheus/2.43.0" 2024-09-10T18:42:21.657 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:21.690 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:21.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:21.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:21.792 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:21.829 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:21.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:21.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:21.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:21.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:21.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:22.086 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:22.123 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:22.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:22.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:22.168 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:21 smithi028 ceph-mon[39220]: pgmap v941: 178 pgs: 178 active+clean; 5.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.54k op/s 2024-09-10T18:42:22.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:21 smithi028 ceph-mon[29291]: pgmap v941: 178 pgs: 178 active+clean; 5.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.54k op/s 2024-09-10T18:42:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:22.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:22.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:22.253 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:22.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:21 smithi204 ceph-mon[32357]: pgmap v941: 178 pgs: 178 active+clean; 5.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.54k op/s 2024-09-10T18:42:22.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:42:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:42:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:42:22.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:22.394 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:22.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:22.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:22.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:22.541 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:22.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:22.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:22.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:22.675 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:22.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:22.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:22.777 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:22.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:22.810 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:22.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:22.911 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:22.945 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:22.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:22.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:42:23.047 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:23.080 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:23.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:23.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:23.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:23.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:23.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:22 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/53902881' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:42:23.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:22 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/53902881' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:42:23.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:23.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:22 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/53902881' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:42:23.363 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:23.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:23.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:23.465 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:23.501 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:23.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:23.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:23.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:23.642 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:23.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:23.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:23.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:42:23.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:23.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:23.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:23.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:23.934 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:23.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:24.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:24.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:24.075 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:24.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:24.176 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:24.212 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:24.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:24.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:42:24.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/53902881' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:42:24.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[39220]: osdmap e408: 8 total, 8 up, 7 in 2024-09-10T18:42:24.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[39220]: 5.e scrub starts 2024-09-10T18:42:24.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[39220]: 5.e scrub ok 2024-09-10T18:42:24.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[39220]: 1.15 scrub starts 2024-09-10T18:42:24.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[39220]: 1.15 scrub ok 2024-09-10T18:42:24.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[39220]: pgmap v943: 178 pgs: 178 active+clean; 5.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-10T18:42:24.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:42:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:42:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/53902881' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:42:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[29291]: osdmap e408: 8 total, 8 up, 7 in 2024-09-10T18:42:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[29291]: 5.e scrub starts 2024-09-10T18:42:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[29291]: 5.e scrub ok 2024-09-10T18:42:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[29291]: 1.15 scrub starts 2024-09-10T18:42:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[29291]: 1.15 scrub ok 2024-09-10T18:42:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[29291]: pgmap v943: 178 pgs: 178 active+clean; 5.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-10T18:42:24.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:23 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:42:24.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:23 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:42:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:23 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/53902881' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:42:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:23 smithi204 ceph-mon[32357]: osdmap e408: 8 total, 8 up, 7 in 2024-09-10T18:42:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:23 smithi204 ceph-mon[32357]: 5.e scrub starts 2024-09-10T18:42:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:23 smithi204 ceph-mon[32357]: 5.e scrub ok 2024-09-10T18:42:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:23 smithi204 ceph-mon[32357]: 1.15 scrub starts 2024-09-10T18:42:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:23 smithi204 ceph-mon[32357]: 1.15 scrub ok 2024-09-10T18:42:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:23 smithi204 ceph-mon[32357]: pgmap v943: 178 pgs: 178 active+clean; 5.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-09-10T18:42:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:23 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:42:24.356 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:24.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:24.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:24.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:24.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:24.496 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:24.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:24.597 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:24.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:24.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:24.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:24.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:24.775 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:24.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:24.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:24.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:24.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:24.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:24.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:25.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:25.055 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:25.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:25.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:25.157 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:25.191 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:25.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:25.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:25.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[39220]: 4.b scrub starts 2024-09-10T18:42:25.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[39220]: 4.b scrub ok 2024-09-10T18:42:25.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[39220]: 11.6 scrub starts 2024-09-10T18:42:25.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[39220]: 6.3 scrub starts 2024-09-10T18:42:25.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[39220]: 5.0 scrub starts 2024-09-10T18:42:25.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[39220]: 5.0 scrub ok 2024-09-10T18:42:25.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[39220]: 3.5 scrub starts 2024-09-10T18:42:25.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[39220]: 6.3 scrub ok 2024-09-10T18:42:25.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[39220]: 3.5 scrub ok 2024-09-10T18:42:25.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[29291]: 4.b scrub starts 2024-09-10T18:42:25.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[29291]: 4.b scrub ok 2024-09-10T18:42:25.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[29291]: 11.6 scrub starts 2024-09-10T18:42:25.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[29291]: 6.3 scrub starts 2024-09-10T18:42:25.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[29291]: 5.0 scrub starts 2024-09-10T18:42:25.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[29291]: 5.0 scrub ok 2024-09-10T18:42:25.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[29291]: 3.5 scrub starts 2024-09-10T18:42:25.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[29291]: 6.3 scrub ok 2024-09-10T18:42:25.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:24 smithi028 ceph-mon[29291]: 3.5 scrub ok 2024-09-10T18:42:25.294 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:25.327 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:25.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:25.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:25.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:42:25 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:42:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:24 smithi204 ceph-mon[32357]: 4.b scrub starts 2024-09-10T18:42:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:24 smithi204 ceph-mon[32357]: 4.b scrub ok 2024-09-10T18:42:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:24 smithi204 ceph-mon[32357]: 11.6 scrub starts 2024-09-10T18:42:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:24 smithi204 ceph-mon[32357]: 6.3 scrub starts 2024-09-10T18:42:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:24 smithi204 ceph-mon[32357]: 5.0 scrub starts 2024-09-10T18:42:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:24 smithi204 ceph-mon[32357]: 5.0 scrub ok 2024-09-10T18:42:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:24 smithi204 ceph-mon[32357]: 3.5 scrub starts 2024-09-10T18:42:25.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:24 smithi204 ceph-mon[32357]: 6.3 scrub ok 2024-09-10T18:42:25.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:24 smithi204 ceph-mon[32357]: 3.5 scrub ok 2024-09-10T18:42:25.429 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:25.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:25.467 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:25.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:25.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:25.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:25.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:25.709 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:25.740 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:25.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:25.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:25.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:25.893 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:25.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:25.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:25.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:26.035 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:26.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:26.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:26.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:25 smithi028 ceph-mon[29291]: 11.3 scrub starts 2024-09-10T18:42:26.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:25 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:26.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:25 smithi028 ceph-mon[29291]: 3.14 scrub starts 2024-09-10T18:42:26.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:25 smithi028 ceph-mon[29291]: 3.14 scrub ok 2024-09-10T18:42:26.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:25 smithi028 ceph-mon[29291]: pgmap v944: 178 pgs: 178 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.54k op/s 2024-09-10T18:42:26.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:25 smithi028 ceph-mon[39220]: 11.3 scrub starts 2024-09-10T18:42:26.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:25 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:26.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:25 smithi028 ceph-mon[39220]: 3.14 scrub starts 2024-09-10T18:42:26.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:25 smithi028 ceph-mon[39220]: 3.14 scrub ok 2024-09-10T18:42:26.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:25 smithi028 ceph-mon[39220]: pgmap v944: 178 pgs: 178 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.54k op/s 2024-09-10T18:42:26.136 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:26.171 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:26.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:26.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:26.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:26.310 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:26.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:25 smithi204 ceph-mon[32357]: 11.3 scrub starts 2024-09-10T18:42:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:25 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:25 smithi204 ceph-mon[32357]: 3.14 scrub starts 2024-09-10T18:42:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:25 smithi204 ceph-mon[32357]: 3.14 scrub ok 2024-09-10T18:42:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:25 smithi204 ceph-mon[32357]: pgmap v944: 178 pgs: 178 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.54k op/s 2024-09-10T18:42:26.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:26.449 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:26.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:26.551 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:26.589 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:26.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:26.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:26.691 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:26.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:26.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:26.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:26.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:26.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:26.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:26.955 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:42:26.955 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "" 2024-09-10T18:42:26.955 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:42:26.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:27.023 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:27.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:27.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:27.125 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:27.166 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:27.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:27.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:26 smithi028 ceph-mon[39220]: 1.6 scrub starts 2024-09-10T18:42:27.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:26 smithi028 ceph-mon[39220]: 1.6 scrub ok 2024-09-10T18:42:27.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:26 smithi028 ceph-mon[39220]: 11.6 scrub ok 2024-09-10T18:42:27.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:26 smithi028 ceph-mon[39220]: 1.1e scrub starts 2024-09-10T18:42:27.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:26 smithi028 ceph-mon[39220]: 1.1e scrub ok 2024-09-10T18:42:27.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:26 smithi028 ceph-mon[39220]: 11.e scrub starts 2024-09-10T18:42:27.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:26 smithi028 ceph-mon[39220]: 11.7 scrub starts 2024-09-10T18:42:27.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:26 smithi028 ceph-mon[29291]: 1.6 scrub starts 2024-09-10T18:42:27.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:26 smithi028 ceph-mon[29291]: 1.6 scrub ok 2024-09-10T18:42:27.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:26 smithi028 ceph-mon[29291]: 11.6 scrub ok 2024-09-10T18:42:27.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:26 smithi028 ceph-mon[29291]: 1.1e scrub starts 2024-09-10T18:42:27.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:26 smithi028 ceph-mon[29291]: 1.1e scrub ok 2024-09-10T18:42:27.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:26 smithi028 ceph-mon[29291]: 11.e scrub starts 2024-09-10T18:42:27.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:26 smithi028 ceph-mon[29291]: 11.7 scrub starts 2024-09-10T18:42:27.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:27.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:27.305 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:27.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:26 smithi204 ceph-mon[32357]: 1.6 scrub starts 2024-09-10T18:42:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:26 smithi204 ceph-mon[32357]: 1.6 scrub ok 2024-09-10T18:42:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:26 smithi204 ceph-mon[32357]: 11.6 scrub ok 2024-09-10T18:42:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:26 smithi204 ceph-mon[32357]: 1.1e scrub starts 2024-09-10T18:42:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:26 smithi204 ceph-mon[32357]: 1.1e scrub ok 2024-09-10T18:42:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:26 smithi204 ceph-mon[32357]: 11.e scrub starts 2024-09-10T18:42:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:26 smithi204 ceph-mon[32357]: 11.7 scrub starts 2024-09-10T18:42:27.407 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:27.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:27.450 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:27.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:27.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:42:27.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:27.573 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:27.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:27.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:27.675 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:27.715 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:27.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:27.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:27.817 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:27.850 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:27.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:27.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:27.952 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:27.987 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:27.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:27.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:28.089 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:28.135 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:28.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:28.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:28.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:28.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:27 smithi028 ceph-mon[39220]: 11.3 scrub ok 2024-09-10T18:42:28.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:27 smithi028 ceph-mon[39220]: 4.14 scrub starts 2024-09-10T18:42:28.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:27 smithi028 ceph-mon[39220]: 4.14 scrub ok 2024-09-10T18:42:28.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:27 smithi028 ceph-mon[39220]: 3.3 scrub starts 2024-09-10T18:42:28.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:27 smithi028 ceph-mon[39220]: pgmap v945: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.52k op/s 2024-09-10T18:42:28.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:27 smithi028 ceph-mon[39220]: 3.3 scrub ok 2024-09-10T18:42:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:27 smithi028 ceph-mon[29291]: 11.3 scrub ok 2024-09-10T18:42:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:27 smithi028 ceph-mon[29291]: 4.14 scrub starts 2024-09-10T18:42:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:27 smithi028 ceph-mon[29291]: 4.14 scrub ok 2024-09-10T18:42:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:27 smithi028 ceph-mon[29291]: 3.3 scrub starts 2024-09-10T18:42:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:27 smithi028 ceph-mon[29291]: pgmap v945: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.52k op/s 2024-09-10T18:42:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:27 smithi028 ceph-mon[29291]: 3.3 scrub ok 2024-09-10T18:42:28.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:28.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:28.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:27 smithi204 ceph-mon[32357]: 11.3 scrub ok 2024-09-10T18:42:28.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:27 smithi204 ceph-mon[32357]: 4.14 scrub starts 2024-09-10T18:42:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:27 smithi204 ceph-mon[32357]: 4.14 scrub ok 2024-09-10T18:42:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:27 smithi204 ceph-mon[32357]: 3.3 scrub starts 2024-09-10T18:42:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:27 smithi204 ceph-mon[32357]: pgmap v945: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.52k op/s 2024-09-10T18:42:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:27 smithi204 ceph-mon[32357]: 3.3 scrub ok 2024-09-10T18:42:28.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:28.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:28.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:28.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:28.523 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:28.558 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:28.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:28.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:28.660 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:28.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:28.695 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:28.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:28.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:28.836 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:28.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:28.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:28.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:28.981 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:28.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:28.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:29.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:29.113 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:29.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:29.129 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:42:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-10T18:42:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-10T18:42:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-10T18:42:29.129 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:42:29.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:28 smithi028 ceph-mon[39220]: 11.1 scrub starts 2024-09-10T18:42:29.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:28 smithi028 ceph-mon[39220]: 6.8 scrub starts 2024-09-10T18:42:29.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:28 smithi028 ceph-mon[39220]: 6.8 scrub ok 2024-09-10T18:42:29.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:28 smithi028 ceph-mon[39220]: 11.7 scrub ok 2024-09-10T18:42:29.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:28 smithi028 ceph-mon[39220]: 6.d scrub starts 2024-09-10T18:42:29.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:28 smithi028 ceph-mon[39220]: 6.d scrub ok 2024-09-10T18:42:29.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:28 smithi028 ceph-mon[39220]: 11.e scrub ok 2024-09-10T18:42:29.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:28 smithi028 ceph-mon[29291]: 11.1 scrub starts 2024-09-10T18:42:29.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:28 smithi028 ceph-mon[29291]: 6.8 scrub starts 2024-09-10T18:42:29.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:28 smithi028 ceph-mon[29291]: 6.8 scrub ok 2024-09-10T18:42:29.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:28 smithi028 ceph-mon[29291]: 11.7 scrub ok 2024-09-10T18:42:29.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:28 smithi028 ceph-mon[29291]: 6.d scrub starts 2024-09-10T18:42:29.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:28 smithi028 ceph-mon[29291]: 6.d scrub ok 2024-09-10T18:42:29.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:28 smithi028 ceph-mon[29291]: 11.e scrub ok 2024-09-10T18:42:29.214 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:29.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:29.248 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:29.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:29.349 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:29.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:28 smithi204 ceph-mon[32357]: 11.1 scrub starts 2024-09-10T18:42:29.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:28 smithi204 ceph-mon[32357]: 6.8 scrub starts 2024-09-10T18:42:29.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:28 smithi204 ceph-mon[32357]: 6.8 scrub ok 2024-09-10T18:42:29.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:28 smithi204 ceph-mon[32357]: 11.7 scrub ok 2024-09-10T18:42:29.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:28 smithi204 ceph-mon[32357]: 6.d scrub starts 2024-09-10T18:42:29.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:28 smithi204 ceph-mon[32357]: 6.d scrub ok 2024-09-10T18:42:29.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:28 smithi204 ceph-mon[32357]: 11.e scrub ok 2024-09-10T18:42:29.381 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:29.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:29.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:29.522 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:29.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:29.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:29.624 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:29.658 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:29.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:29.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:29.759 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:29.787 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:29.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:29.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:29.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:29.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:30.029 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:30.063 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:30.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:30.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:30.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:30.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:29 smithi028 ceph-mon[29291]: 5.a scrub starts 2024-09-10T18:42:30.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:29 smithi028 ceph-mon[29291]: 5.a scrub ok 2024-09-10T18:42:30.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:29 smithi028 ceph-mon[29291]: pgmap v946: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.52k op/s 2024-09-10T18:42:30.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:29 smithi028 ceph-mon[29291]: 3.1b scrub starts 2024-09-10T18:42:30.177 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:29 smithi028 ceph-mon[29291]: 3.1b scrub ok 2024-09-10T18:42:30.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:30 smithi028 ceph-mon[39220]: 5.a scrub starts 2024-09-10T18:42:30.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:30 smithi028 ceph-mon[39220]: 5.a scrub ok 2024-09-10T18:42:30.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:30 smithi028 ceph-mon[39220]: pgmap v946: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.52k op/s 2024-09-10T18:42:30.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:30 smithi028 ceph-mon[39220]: 3.1b scrub starts 2024-09-10T18:42:30.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:30 smithi028 ceph-mon[39220]: 3.1b scrub ok 2024-09-10T18:42:30.208 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:30.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:30.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:30.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:30.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:29 smithi204 ceph-mon[32357]: 5.a scrub starts 2024-09-10T18:42:30.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:29 smithi204 ceph-mon[32357]: 5.a scrub ok 2024-09-10T18:42:30.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:29 smithi204 ceph-mon[32357]: pgmap v946: 178 pgs: 2 active+clean+scrubbing, 176 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.52k op/s 2024-09-10T18:42:30.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:29 smithi204 ceph-mon[32357]: 3.1b scrub starts 2024-09-10T18:42:30.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:29 smithi204 ceph-mon[32357]: 3.1b scrub ok 2024-09-10T18:42:30.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:30.360 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:30.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:30.461 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:30.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:30.493 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:30.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:30.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:30.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:30.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:30.733 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:30.764 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:30.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:30.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:30.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-10T18:42:30.851 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:30.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:30.925 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:30.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:31.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:30 smithi028 ceph-mon[29291]: 11.1 scrub ok 2024-09-10T18:42:31.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:30 smithi028 ceph-mon[29291]: 4.19 scrub starts 2024-09-10T18:42:31.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:31 smithi028 ceph-mon[29291]: 4.19 scrub ok 2024-09-10T18:42:31.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:31 smithi028 ceph-mon[29291]: 3.1c scrub starts 2024-09-10T18:42:31.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:31 smithi028 ceph-mon[29291]: 3.1c scrub ok 2024-09-10T18:42:31.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:31 smithi028 ceph-mon[39220]: 11.1 scrub ok 2024-09-10T18:42:31.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:31 smithi028 ceph-mon[39220]: 4.19 scrub starts 2024-09-10T18:42:31.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:31 smithi028 ceph-mon[39220]: 4.19 scrub ok 2024-09-10T18:42:31.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:31 smithi028 ceph-mon[39220]: 3.1c scrub starts 2024-09-10T18:42:31.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:31 smithi028 ceph-mon[39220]: 3.1c scrub ok 2024-09-10T18:42:31.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:31.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:31.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:31.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:31.167 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:31.196 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:31.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:31.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:31.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:31.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:31.340 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:31.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:31.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:31 smithi204 ceph-mon[32357]: 11.1 scrub ok 2024-09-10T18:42:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:31 smithi204 ceph-mon[32357]: 4.19 scrub starts 2024-09-10T18:42:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:31 smithi204 ceph-mon[32357]: 4.19 scrub ok 2024-09-10T18:42:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:31 smithi204 ceph-mon[32357]: 3.1c scrub starts 2024-09-10T18:42:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:31 smithi204 ceph-mon[32357]: 3.1c scrub ok 2024-09-10T18:42:31.421 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:42:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:42:31] "GET /metrics HTTP/1.1" 200 224959 "" "Prometheus/2.43.0" 2024-09-10T18:42:31.441 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:31.474 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:31.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:31.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:31.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:31.626 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:31.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:31.727 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:31.767 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:31.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:31.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:31.868 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:31.904 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:31.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:31.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:32.006 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:32.040 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:32.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:32.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:32.075 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:42:32.075 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:42:32.075 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:42:32.075 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:42:32.076 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:42:32.076 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:42:32.076 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:42:32.076 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:42:32.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:42:32.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:42:32.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:42:32.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:42:32.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:42:32.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:42:32.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:42:32.077 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:42:32.145 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:32.178 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:32.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:32.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:32.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:32.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:32.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:32.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:42:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:42:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:42:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:32 smithi204 ceph-mon[32357]: 5.d scrub starts 2024-09-10T18:42:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:32 smithi204 ceph-mon[32357]: 5.d scrub ok 2024-09-10T18:42:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:32 smithi204 ceph-mon[32357]: pgmap v947: 178 pgs: 178 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-10T18:42:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:32 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2191981041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:42:32.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:32.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:32 smithi028 ceph-mon[39220]: 5.d scrub starts 2024-09-10T18:42:32.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:32 smithi028 ceph-mon[39220]: 5.d scrub ok 2024-09-10T18:42:32.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:32 smithi028 ceph-mon[39220]: pgmap v947: 178 pgs: 178 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-10T18:42:32.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:32 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2191981041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:42:32.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:32 smithi028 ceph-mon[29291]: 5.d scrub starts 2024-09-10T18:42:32.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:32 smithi028 ceph-mon[29291]: 5.d scrub ok 2024-09-10T18:42:32.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:32 smithi028 ceph-mon[29291]: pgmap v947: 178 pgs: 178 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-10T18:42:32.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:32 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2191981041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:42:32.460 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:32.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:32.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:32.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:32.595 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:32.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:32.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:32.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:32.734 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:32.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:32.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:32.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:32.871 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:32.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:32.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:32.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:33.011 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:33.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:33.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:33.113 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:33.148 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:33.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:33.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:33.250 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:33.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:33 smithi028 ceph-mon[39220]: 1.f scrub starts 2024-09-10T18:42:33.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:33 smithi028 ceph-mon[39220]: 1.f scrub ok 2024-09-10T18:42:33.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:33 smithi028 ceph-mon[39220]: 1.1a scrub starts 2024-09-10T18:42:33.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:33 smithi028 ceph-mon[39220]: 1.1a scrub ok 2024-09-10T18:42:33.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:33 smithi028 ceph-mon[29291]: 1.f scrub starts 2024-09-10T18:42:33.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:33 smithi028 ceph-mon[29291]: 1.f scrub ok 2024-09-10T18:42:33.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:33 smithi028 ceph-mon[29291]: 1.1a scrub starts 2024-09-10T18:42:33.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:33 smithi028 ceph-mon[29291]: 1.1a scrub ok 2024-09-10T18:42:33.282 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:33.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:33.329 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:42:33.329 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:42:33.329 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-10T18:42:33.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:42:33.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:33 smithi204 ceph-mon[32357]: 1.f scrub starts 2024-09-10T18:42:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:33 smithi204 ceph-mon[32357]: 1.f scrub ok 2024-09-10T18:42:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:33 smithi204 ceph-mon[32357]: 1.1a scrub starts 2024-09-10T18:42:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:33 smithi204 ceph-mon[32357]: 1.1a scrub ok 2024-09-10T18:42:33.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:33.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:33.432 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:33.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:33.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:33.591 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:33.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:33.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:33.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:33.729 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:33.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:33.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:33.831 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:33.869 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:33.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:33.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:33.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:34.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:34.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:34.111 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:34.147 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:34.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:34.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:34 smithi028 ceph-mon[29291]: 2.1 scrub starts 2024-09-10T18:42:34.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:34 smithi028 ceph-mon[29291]: 2.1 scrub ok 2024-09-10T18:42:34.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:34 smithi028 ceph-mon[29291]: 1.1c scrub starts 2024-09-10T18:42:34.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:34 smithi028 ceph-mon[29291]: 1.1c scrub ok 2024-09-10T18:42:34.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:34 smithi028 ceph-mon[29291]: pgmap v948: 178 pgs: 178 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 998 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T18:42:34.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:34 smithi028 ceph-mon[29291]: 6.10 scrub starts 2024-09-10T18:42:34.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:34 smithi028 ceph-mon[29291]: 6.10 scrub ok 2024-09-10T18:42:34.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:34 smithi028 ceph-mon[39220]: 2.1 scrub starts 2024-09-10T18:42:34.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:34 smithi028 ceph-mon[39220]: 2.1 scrub ok 2024-09-10T18:42:34.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:34 smithi028 ceph-mon[39220]: 1.1c scrub starts 2024-09-10T18:42:34.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:34 smithi028 ceph-mon[39220]: 1.1c scrub ok 2024-09-10T18:42:34.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:34 smithi028 ceph-mon[39220]: pgmap v948: 178 pgs: 178 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 998 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T18:42:34.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:34 smithi028 ceph-mon[39220]: 6.10 scrub starts 2024-09-10T18:42:34.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:34 smithi028 ceph-mon[39220]: 6.10 scrub ok 2024-09-10T18:42:34.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:34.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:34.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:34 smithi204 ceph-mon[32357]: 2.1 scrub starts 2024-09-10T18:42:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:34 smithi204 ceph-mon[32357]: 2.1 scrub ok 2024-09-10T18:42:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:34 smithi204 ceph-mon[32357]: 1.1c scrub starts 2024-09-10T18:42:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:34 smithi204 ceph-mon[32357]: 1.1c scrub ok 2024-09-10T18:42:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:34 smithi204 ceph-mon[32357]: pgmap v948: 178 pgs: 178 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 998 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T18:42:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:34 smithi204 ceph-mon[32357]: 6.10 scrub starts 2024-09-10T18:42:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:34 smithi204 ceph-mon[32357]: 6.10 scrub ok 2024-09-10T18:42:34.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:34.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:34.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:34.572 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:34.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:34.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:34.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:34.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:34.716 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:34.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:34.817 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:34.847 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:34.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:34.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:34.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:42:34.949 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:34.981 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:34.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:34.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:35.047 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:42:35.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:35.116 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:35.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:35.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:35.218 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:35.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:35.249 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:35.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:35.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:35 smithi028 ceph-mon[29291]: 3.c scrub starts 2024-09-10T18:42:35.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:35 smithi028 ceph-mon[29291]: 3.c scrub ok 2024-09-10T18:42:35.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:35 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2144212196' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:42:35.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:35 smithi028 ceph-mon[39220]: 3.c scrub starts 2024-09-10T18:42:35.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:35 smithi028 ceph-mon[39220]: 3.c scrub ok 2024-09-10T18:42:35.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:35 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2144212196' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:42:35.350 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:35 smithi204 ceph-mon[32357]: 3.c scrub starts 2024-09-10T18:42:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:35 smithi204 ceph-mon[32357]: 3.c scrub ok 2024-09-10T18:42:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:35 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2144212196' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:42:35.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:42:35 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:42:35.382 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:35.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:35.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:35.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:35.524 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:35.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:35.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:35.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:35.663 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:35.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:35.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:35.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:35.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:35.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:35.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:35.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:35.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:36.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:36.064 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:36.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:36.167 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:36.194 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:36.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:36.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:36.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:36.332 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:36.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:36.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:36 smithi204 ceph-mon[32357]: 6.18 scrub starts 2024-09-10T18:42:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:36 smithi204 ceph-mon[32357]: 6.18 scrub ok 2024-09-10T18:42:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:36 smithi204 ceph-mon[32357]: 5.11 scrub starts 2024-09-10T18:42:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:36 smithi204 ceph-mon[32357]: 5.11 scrub ok 2024-09-10T18:42:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:36 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:42:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:36 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:42:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:36 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2144212196' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:42:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:36 smithi204 ceph-mon[32357]: osdmap e409: 8 total, 8 up, 7 in 2024-09-10T18:42:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:36 smithi204 ceph-mon[32357]: 4.1e scrub starts 2024-09-10T18:42:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:36 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:36 smithi204 ceph-mon[32357]: 4.1e scrub ok 2024-09-10T18:42:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:36 smithi204 ceph-mon[32357]: pgmap v950: 178 pgs: 178 active+clean; 4.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-10T18:42:36.434 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:36.472 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:36.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:36.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[29291]: 6.18 scrub starts 2024-09-10T18:42:36.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[29291]: 6.18 scrub ok 2024-09-10T18:42:36.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[29291]: 5.11 scrub starts 2024-09-10T18:42:36.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[29291]: 5.11 scrub ok 2024-09-10T18:42:36.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:42:36.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:42:36.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2144212196' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:42:36.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[29291]: osdmap e409: 8 total, 8 up, 7 in 2024-09-10T18:42:36.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[29291]: 4.1e scrub starts 2024-09-10T18:42:36.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:36.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[29291]: 4.1e scrub ok 2024-09-10T18:42:36.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[29291]: pgmap v950: 178 pgs: 178 active+clean; 4.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-10T18:42:36.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[39220]: 6.18 scrub starts 2024-09-10T18:42:36.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[39220]: 6.18 scrub ok 2024-09-10T18:42:36.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[39220]: 5.11 scrub starts 2024-09-10T18:42:36.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[39220]: 5.11 scrub ok 2024-09-10T18:42:36.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:42:36.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:42:36.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2144212196' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:42:36.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[39220]: osdmap e409: 8 total, 8 up, 7 in 2024-09-10T18:42:36.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[39220]: 4.1e scrub starts 2024-09-10T18:42:36.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:36.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[39220]: 4.1e scrub ok 2024-09-10T18:42:36.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:36 smithi028 ceph-mon[39220]: pgmap v950: 178 pgs: 178 active+clean; 4.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-10T18:42:36.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:36.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:36.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:36.710 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:36.748 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:36.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:36.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:36.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:36.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:36.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:36.994 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:37.030 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:37.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:37.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:37.132 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:37.160 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:37.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:37.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:37.248 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:42:37.248 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:42:37.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:37.264 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:37 smithi028 ceph-mon[39220]: 5.1d scrub starts 2024-09-10T18:42:37.264 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:37 smithi028 ceph-mon[39220]: 5.1d scrub ok 2024-09-10T18:42:37.264 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:37 smithi028 ceph-mon[39220]: 5.1f scrub starts 2024-09-10T18:42:37.264 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:37 smithi028 ceph-mon[39220]: 5.1f scrub ok 2024-09-10T18:42:37.264 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:37 smithi028 ceph-mon[39220]: 6.5 scrub starts 2024-09-10T18:42:37.264 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:37 smithi028 ceph-mon[39220]: 6.5 scrub ok 2024-09-10T18:42:37.264 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:37 smithi028 ceph-mon[39220]: 4.13 scrub starts 2024-09-10T18:42:37.264 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:37 smithi028 ceph-mon[39220]: 4.13 scrub ok 2024-09-10T18:42:37.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:37 smithi028 ceph-mon[29291]: 5.1d scrub starts 2024-09-10T18:42:37.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:37 smithi028 ceph-mon[29291]: 5.1d scrub ok 2024-09-10T18:42:37.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:37 smithi028 ceph-mon[29291]: 5.1f scrub starts 2024-09-10T18:42:37.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:37 smithi028 ceph-mon[29291]: 5.1f scrub ok 2024-09-10T18:42:37.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:37 smithi028 ceph-mon[29291]: 6.5 scrub starts 2024-09-10T18:42:37.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:37 smithi028 ceph-mon[29291]: 6.5 scrub ok 2024-09-10T18:42:37.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:37 smithi028 ceph-mon[29291]: 4.13 scrub starts 2024-09-10T18:42:37.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:37 smithi028 ceph-mon[29291]: 4.13 scrub ok 2024-09-10T18:42:37.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:37.294 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:37.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:37 smithi204 ceph-mon[32357]: 5.1d scrub starts 2024-09-10T18:42:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:37 smithi204 ceph-mon[32357]: 5.1d scrub ok 2024-09-10T18:42:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:37 smithi204 ceph-mon[32357]: 5.1f scrub starts 2024-09-10T18:42:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:37 smithi204 ceph-mon[32357]: 5.1f scrub ok 2024-09-10T18:42:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:37 smithi204 ceph-mon[32357]: 6.5 scrub starts 2024-09-10T18:42:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:37 smithi204 ceph-mon[32357]: 6.5 scrub ok 2024-09-10T18:42:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:37 smithi204 ceph-mon[32357]: 4.13 scrub starts 2024-09-10T18:42:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:37 smithi204 ceph-mon[32357]: 4.13 scrub ok 2024-09-10T18:42:37.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:37.435 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:37.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:37.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:37.570 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:37.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:37.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:37.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:37.712 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:37.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:37.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:37.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:37.867 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:37.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:37.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:37.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:38.008 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:38.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:38.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:38.052 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:38.064 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:38.110 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:38.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:38.145 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:38.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:38.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:38.291 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:38.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:38.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:38.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:38 smithi204 ceph-mon[32357]: 11.4 scrub starts 2024-09-10T18:42:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:38 smithi204 ceph-mon[32357]: pgmap v951: 178 pgs: 178 active+clean; 4.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.50k op/s 2024-09-10T18:42:38.394 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:38.426 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:38.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:38.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:38.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:38 smithi028 ceph-mon[39220]: 11.4 scrub starts 2024-09-10T18:42:38.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:38 smithi028 ceph-mon[39220]: pgmap v951: 178 pgs: 178 active+clean; 4.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.50k op/s 2024-09-10T18:42:38.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:38 smithi028 ceph-mon[29291]: 11.4 scrub starts 2024-09-10T18:42:38.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:38 smithi028 ceph-mon[29291]: pgmap v951: 178 pgs: 178 active+clean; 4.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.50k op/s 2024-09-10T18:42:38.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:38.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-10T18:42:38.555 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:38.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:38.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:38.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:38.721 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:38.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:38.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:38.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:38.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:38.919 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:38.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:39.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:39.046 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:39.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:39.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:39.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:39 smithi028 ceph-mon[29291]: 4.a scrub starts 2024-09-10T18:42:39.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:39 smithi028 ceph-mon[29291]: 4.a scrub ok 2024-09-10T18:42:39.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:42:39.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:39 smithi028 ceph-mon[29291]: 3.8 scrub starts 2024-09-10T18:42:39.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:39 smithi028 ceph-mon[29291]: 3.8 scrub ok 2024-09-10T18:42:39.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:39 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:42:39.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:42:39.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:39 smithi028 ceph-mon[39220]: 4.a scrub starts 2024-09-10T18:42:39.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:39 smithi028 ceph-mon[39220]: 4.a scrub ok 2024-09-10T18:42:39.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:42:39.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:39 smithi028 ceph-mon[39220]: 3.8 scrub starts 2024-09-10T18:42:39.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:39 smithi028 ceph-mon[39220]: 3.8 scrub ok 2024-09-10T18:42:39.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:39 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:42:39.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:42:39.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:39.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:39.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:39.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:39.339 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:39.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:39.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:39 smithi204 ceph-mon[32357]: 4.a scrub starts 2024-09-10T18:42:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:39 smithi204 ceph-mon[32357]: 4.a scrub ok 2024-09-10T18:42:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:42:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:39 smithi204 ceph-mon[32357]: 3.8 scrub starts 2024-09-10T18:42:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:39 smithi204 ceph-mon[32357]: 3.8 scrub ok 2024-09-10T18:42:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:39 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:42:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:42:39.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:39.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:39.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:39.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:39.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:39.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:39.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:39.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:39.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:39.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:39.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:39.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:39.996 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:40.051 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:40.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:40.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:40.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:42:40.122 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[29291]: 4.7 scrub starts 2024-09-10T18:42:40.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[29291]: 11.4 scrub ok 2024-09-10T18:42:40.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[29291]: 4.7 scrub ok 2024-09-10T18:42:40.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[29291]: 4.1f scrub starts 2024-09-10T18:42:40.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[29291]: 4.1f scrub ok 2024-09-10T18:42:40.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:40.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:42:40.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:42:40.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[29291]: pgmap v952: 178 pgs: 178 active+clean; 4.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.50k op/s 2024-09-10T18:42:40.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[29291]: 1.10 scrub starts 2024-09-10T18:42:40.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[29291]: 1.10 scrub ok 2024-09-10T18:42:40.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[39220]: 4.7 scrub starts 2024-09-10T18:42:40.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[39220]: 11.4 scrub ok 2024-09-10T18:42:40.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[39220]: 4.7 scrub ok 2024-09-10T18:42:40.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[39220]: 4.1f scrub starts 2024-09-10T18:42:40.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[39220]: 4.1f scrub ok 2024-09-10T18:42:40.126 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:40.126 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:42:40.126 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:42:40.126 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[39220]: pgmap v952: 178 pgs: 178 active+clean; 4.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.50k op/s 2024-09-10T18:42:40.126 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[39220]: 1.10 scrub starts 2024-09-10T18:42:40.126 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:40 smithi028 ceph-mon[39220]: 1.10 scrub ok 2024-09-10T18:42:40.154 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:40.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:40.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:40.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:40.289 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:40.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:40.321 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:40.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:40 smithi204 ceph-mon[32357]: 4.7 scrub starts 2024-09-10T18:42:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:40 smithi204 ceph-mon[32357]: 11.4 scrub ok 2024-09-10T18:42:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:40 smithi204 ceph-mon[32357]: 4.7 scrub ok 2024-09-10T18:42:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:40 smithi204 ceph-mon[32357]: 4.1f scrub starts 2024-09-10T18:42:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:40 smithi204 ceph-mon[32357]: 4.1f scrub ok 2024-09-10T18:42:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:40 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:40 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:42:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:40 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:42:40.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:40 smithi204 ceph-mon[32357]: pgmap v952: 178 pgs: 178 active+clean; 4.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.50k op/s 2024-09-10T18:42:40.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:40 smithi204 ceph-mon[32357]: 1.10 scrub starts 2024-09-10T18:42:40.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:40 smithi204 ceph-mon[32357]: 1.10 scrub ok 2024-09-10T18:42:40.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:40.462 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:40.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:40.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:40.564 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:40.600 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:40.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:40.702 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:40.735 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:40.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:40.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:40.837 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:40.870 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:40.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:40.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:40.972 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:41.008 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:41.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:41.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:41.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:41.146 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:41.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:41.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:41.248 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:41.277 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:41.280 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:41.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:41.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:41 smithi204 ceph-mon[32357]: 11.f scrub starts 2024-09-10T18:42:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:41 smithi204 ceph-mon[32357]: 1.19 scrub starts 2024-09-10T18:42:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:41 smithi204 ceph-mon[32357]: 1.19 scrub ok 2024-09-10T18:42:41.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:41.386 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:42:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:42:41] "GET /metrics HTTP/1.1" 200 224977 "" "Prometheus/2.43.0" 2024-09-10T18:42:41.387 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:41 smithi028 ceph-mon[39220]: 11.f scrub starts 2024-09-10T18:42:41.387 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:41 smithi028 ceph-mon[39220]: 1.19 scrub starts 2024-09-10T18:42:41.387 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:41 smithi028 ceph-mon[39220]: 1.19 scrub ok 2024-09-10T18:42:41.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:41 smithi028 ceph-mon[29291]: 11.f scrub starts 2024-09-10T18:42:41.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:41 smithi028 ceph-mon[29291]: 1.19 scrub starts 2024-09-10T18:42:41.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:41 smithi028 ceph-mon[29291]: 1.19 scrub ok 2024-09-10T18:42:41.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:41.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:41.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:41.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:41.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:41.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:41.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:41.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:41.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:41.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:41.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:41.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:41.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:41.939 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:41.973 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:41.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:42.074 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:42.108 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:42.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:42.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:42.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:42.246 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:42.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:42.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:42.348 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:42.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:42.388 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:42.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:42.489 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[39220]: 1.b scrub starts 2024-09-10T18:42:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[39220]: 1.b scrub ok 2024-09-10T18:42:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[39220]: 6.15 scrub starts 2024-09-10T18:42:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[39220]: 6.15 scrub ok 2024-09-10T18:42:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[39220]: pgmap v953: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.48k op/s 2024-09-10T18:42:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[39220]: 6.2 deep-scrub starts 2024-09-10T18:42:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[39220]: 6.2 deep-scrub ok 2024-09-10T18:42:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[39220]: 5.3 scrub starts 2024-09-10T18:42:42.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[39220]: 5.3 scrub ok 2024-09-10T18:42:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[29291]: 1.b scrub starts 2024-09-10T18:42:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[29291]: 1.b scrub ok 2024-09-10T18:42:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[29291]: 6.15 scrub starts 2024-09-10T18:42:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[29291]: 6.15 scrub ok 2024-09-10T18:42:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[29291]: pgmap v953: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.48k op/s 2024-09-10T18:42:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[29291]: 6.2 deep-scrub starts 2024-09-10T18:42:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[29291]: 6.2 deep-scrub ok 2024-09-10T18:42:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[29291]: 5.3 scrub starts 2024-09-10T18:42:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:42 smithi028 ceph-mon[29291]: 5.3 scrub ok 2024-09-10T18:42:42.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:42.525 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:42.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:42.603 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:42:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:42:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:42:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:42 smithi204 ceph-mon[32357]: 1.b scrub starts 2024-09-10T18:42:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:42 smithi204 ceph-mon[32357]: 1.b scrub ok 2024-09-10T18:42:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:42 smithi204 ceph-mon[32357]: 6.15 scrub starts 2024-09-10T18:42:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:42 smithi204 ceph-mon[32357]: 6.15 scrub ok 2024-09-10T18:42:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:42 smithi204 ceph-mon[32357]: pgmap v953: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.48k op/s 2024-09-10T18:42:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:42 smithi204 ceph-mon[32357]: 6.2 deep-scrub starts 2024-09-10T18:42:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:42 smithi204 ceph-mon[32357]: 6.2 deep-scrub ok 2024-09-10T18:42:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:42 smithi204 ceph-mon[32357]: 5.3 scrub starts 2024-09-10T18:42:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:42 smithi204 ceph-mon[32357]: 5.3 scrub ok 2024-09-10T18:42:42.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:42.668 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:42.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:42.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:42.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:42.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:42.802 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:42.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:42.903 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:42.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:42.936 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:42.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:43.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:43.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:43.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:43.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:43.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:43.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:43.199 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:43.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:43.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:43.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:43.337 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:43.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:43.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:43.461 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:43.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:43.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:43.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:43 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:42:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:43 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:42:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:43 smithi028 ceph-mon[39220]: 11.f scrub ok 2024-09-10T18:42:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:43 smithi028 ceph-mon[39220]: osdmap e410: 8 total, 8 up, 7 in 2024-09-10T18:42:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:43 smithi028 ceph-mon[39220]: 3.a scrub starts 2024-09-10T18:42:43.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:43 smithi028 ceph-mon[39220]: 3.a scrub ok 2024-09-10T18:42:43.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:43 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:42:43.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:43 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:42:43.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:43 smithi028 ceph-mon[29291]: 11.f scrub ok 2024-09-10T18:42:43.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:43 smithi028 ceph-mon[29291]: osdmap e410: 8 total, 8 up, 7 in 2024-09-10T18:42:43.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:43 smithi028 ceph-mon[29291]: 3.a scrub starts 2024-09-10T18:42:43.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:43 smithi028 ceph-mon[29291]: 3.a scrub ok 2024-09-10T18:42:43.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:43.595 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:43.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:43.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:43.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:43 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:42:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:43 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:42:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:43 smithi204 ceph-mon[32357]: 11.f scrub ok 2024-09-10T18:42:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:43 smithi204 ceph-mon[32357]: osdmap e410: 8 total, 8 up, 7 in 2024-09-10T18:42:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:43 smithi204 ceph-mon[32357]: 3.a scrub starts 2024-09-10T18:42:43.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:43 smithi204 ceph-mon[32357]: 3.a scrub ok 2024-09-10T18:42:43.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:43.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:43.719 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:43.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:43.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:43.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:43.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:43.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:43.999 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:44.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:44.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:44.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:44.123 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:44.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:44.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:44.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:44.264 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:44.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:44.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:44.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:44.405 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:44.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:44.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:44.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[39220]: 5.8 deep-scrub starts 2024-09-10T18:42:44.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[39220]: 5.8 deep-scrub ok 2024-09-10T18:42:44.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[39220]: 2.0 scrub starts 2024-09-10T18:42:44.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[39220]: 2.0 scrub ok 2024-09-10T18:42:44.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[39220]: 4.4 deep-scrub starts 2024-09-10T18:42:44.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[39220]: 4.4 deep-scrub ok 2024-09-10T18:42:44.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[39220]: pgmap v955: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-10T18:42:44.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[39220]: 5.2 deep-scrub starts 2024-09-10T18:42:44.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[39220]: 5.2 deep-scrub ok 2024-09-10T18:42:44.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[29291]: 5.8 deep-scrub starts 2024-09-10T18:42:44.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[29291]: 5.8 deep-scrub ok 2024-09-10T18:42:44.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[29291]: 2.0 scrub starts 2024-09-10T18:42:44.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[29291]: 2.0 scrub ok 2024-09-10T18:42:44.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[29291]: 4.4 deep-scrub starts 2024-09-10T18:42:44.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[29291]: 4.4 deep-scrub ok 2024-09-10T18:42:44.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[29291]: pgmap v955: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-10T18:42:44.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[29291]: 5.2 deep-scrub starts 2024-09-10T18:42:44.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:44 smithi028 ceph-mon[29291]: 5.2 deep-scrub ok 2024-09-10T18:42:44.507 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:44.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:44.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:44.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:44 smithi204 ceph-mon[32357]: 5.8 deep-scrub starts 2024-09-10T18:42:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:44 smithi204 ceph-mon[32357]: 5.8 deep-scrub ok 2024-09-10T18:42:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:44 smithi204 ceph-mon[32357]: 2.0 scrub starts 2024-09-10T18:42:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:44 smithi204 ceph-mon[32357]: 2.0 scrub ok 2024-09-10T18:42:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:44 smithi204 ceph-mon[32357]: 4.4 deep-scrub starts 2024-09-10T18:42:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:44 smithi204 ceph-mon[32357]: 4.4 deep-scrub ok 2024-09-10T18:42:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:44 smithi204 ceph-mon[32357]: pgmap v955: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-10T18:42:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:44 smithi204 ceph-mon[32357]: 5.2 deep-scrub starts 2024-09-10T18:42:44.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:44 smithi204 ceph-mon[32357]: 5.2 deep-scrub ok 2024-09-10T18:42:44.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:44.679 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:44.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:44.781 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:44.814 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:44.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:44.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:44.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:44.958 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:44.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:44.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:45.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:45.096 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:45.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:45.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:45.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:45.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:45.334 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:45.368 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:45.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:45.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:45.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:45 smithi028 ceph-mon[29291]: 3.0 deep-scrub starts 2024-09-10T18:42:45.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:45 smithi028 ceph-mon[29291]: 3.0 deep-scrub ok 2024-09-10T18:42:45.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:45 smithi028 ceph-mon[29291]: 4.1a scrub starts 2024-09-10T18:42:45.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:45 smithi028 ceph-mon[29291]: 4.1a scrub ok 2024-09-10T18:42:45.401 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:45 smithi028 ceph-mon[39220]: 3.0 deep-scrub starts 2024-09-10T18:42:45.401 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:45 smithi028 ceph-mon[39220]: 3.0 deep-scrub ok 2024-09-10T18:42:45.401 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:45 smithi028 ceph-mon[39220]: 4.1a scrub starts 2024-09-10T18:42:45.401 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:45 smithi028 ceph-mon[39220]: 4.1a scrub ok 2024-09-10T18:42:45.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:45.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:45.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:45.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:45 smithi204 ceph-mon[32357]: 3.0 deep-scrub starts 2024-09-10T18:42:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:45 smithi204 ceph-mon[32357]: 3.0 deep-scrub ok 2024-09-10T18:42:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:45 smithi204 ceph-mon[32357]: 4.1a scrub starts 2024-09-10T18:42:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:45 smithi204 ceph-mon[32357]: 4.1a scrub ok 2024-09-10T18:42:45.605 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:42:45 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:42:45.628 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:45.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:45.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:45.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:45.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:45.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:45.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:45.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:45.894 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:45.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:45.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:45.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:46.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:46.031 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:46.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:46.132 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:46.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:46.169 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:46.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:46.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:46.306 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:46.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:46.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:46.408 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:46.440 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:46.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:46 smithi028 ceph-mon[29291]: 11.5 deep-scrub starts 2024-09-10T18:42:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:46 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:46 smithi028 ceph-mon[29291]: 3.1a scrub starts 2024-09-10T18:42:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:46 smithi028 ceph-mon[29291]: 3.1a scrub ok 2024-09-10T18:42:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:46 smithi028 ceph-mon[29291]: pgmap v956: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 3.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T18:42:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:46 smithi028 ceph-mon[29291]: 5.f scrub starts 2024-09-10T18:42:46.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:46 smithi028 ceph-mon[29291]: 5.f scrub ok 2024-09-10T18:42:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:46 smithi028 ceph-mon[39220]: 11.5 deep-scrub starts 2024-09-10T18:42:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:46 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:46 smithi028 ceph-mon[39220]: 3.1a scrub starts 2024-09-10T18:42:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:46 smithi028 ceph-mon[39220]: 3.1a scrub ok 2024-09-10T18:42:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:46 smithi028 ceph-mon[39220]: pgmap v956: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 3.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T18:42:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:46 smithi028 ceph-mon[39220]: 5.f scrub starts 2024-09-10T18:42:46.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:46 smithi028 ceph-mon[39220]: 5.f scrub ok 2024-09-10T18:42:46.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:46.577 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:46.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:46.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:46.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:46 smithi204 ceph-mon[32357]: 11.5 deep-scrub starts 2024-09-10T18:42:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:46 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:46 smithi204 ceph-mon[32357]: 3.1a scrub starts 2024-09-10T18:42:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:46 smithi204 ceph-mon[32357]: 3.1a scrub ok 2024-09-10T18:42:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:46 smithi204 ceph-mon[32357]: pgmap v956: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 3.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T18:42:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:46 smithi204 ceph-mon[32357]: 5.f scrub starts 2024-09-10T18:42:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:46 smithi204 ceph-mon[32357]: 5.f scrub ok 2024-09-10T18:42:46.679 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:46.715 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:46.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:46.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:46.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:46.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:46.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:46.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:46.990 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:46.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:46.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:47.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:47.134 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:47.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:47.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:47.235 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:47.274 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:47.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:47.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:47.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:47.407 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:47.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:47.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:47.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:47 smithi028 ceph-mon[29291]: 6.f scrub starts 2024-09-10T18:42:47.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:47 smithi028 ceph-mon[29291]: 6.f scrub ok 2024-09-10T18:42:47.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:47 smithi028 ceph-mon[29291]: 5.1c scrub starts 2024-09-10T18:42:47.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:47 smithi028 ceph-mon[29291]: 5.1c scrub ok 2024-09-10T18:42:47.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:47 smithi028 ceph-mon[29291]: 3.18 deep-scrub starts 2024-09-10T18:42:47.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:47 smithi028 ceph-mon[29291]: 3.18 deep-scrub ok 2024-09-10T18:42:47.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:47 smithi028 ceph-mon[39220]: 6.f scrub starts 2024-09-10T18:42:47.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:47 smithi028 ceph-mon[39220]: 6.f scrub ok 2024-09-10T18:42:47.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:47 smithi028 ceph-mon[39220]: 5.1c scrub starts 2024-09-10T18:42:47.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:47 smithi028 ceph-mon[39220]: 5.1c scrub ok 2024-09-10T18:42:47.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:47 smithi028 ceph-mon[39220]: 3.18 deep-scrub starts 2024-09-10T18:42:47.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:47 smithi028 ceph-mon[39220]: 3.18 deep-scrub ok 2024-09-10T18:42:47.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:47.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:47.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:47.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:47 smithi204 ceph-mon[32357]: 6.f scrub starts 2024-09-10T18:42:47.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:47 smithi204 ceph-mon[32357]: 6.f scrub ok 2024-09-10T18:42:47.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:47 smithi204 ceph-mon[32357]: 5.1c scrub starts 2024-09-10T18:42:47.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:47 smithi204 ceph-mon[32357]: 5.1c scrub ok 2024-09-10T18:42:47.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:47 smithi204 ceph-mon[32357]: 3.18 deep-scrub starts 2024-09-10T18:42:47.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:47 smithi204 ceph-mon[32357]: 3.18 deep-scrub ok 2024-09-10T18:42:47.632 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:47.679 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:47.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:47.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:47.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:47.804 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:47.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:47.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:47.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:47.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:47.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:48.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:48.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:48.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:48.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:48.202 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:48.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:48.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:48.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:48.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:48.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:48.429 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:48.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:48.471 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:48.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:48.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:48 smithi028 ceph-mon[39220]: 1.18 scrub starts 2024-09-10T18:42:48.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:48 smithi028 ceph-mon[39220]: pgmap v957: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s 2024-09-10T18:42:48.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:48 smithi028 ceph-mon[39220]: 1.18 scrub ok 2024-09-10T18:42:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:48 smithi028 ceph-mon[29291]: 1.18 scrub starts 2024-09-10T18:42:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:48 smithi028 ceph-mon[29291]: pgmap v957: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s 2024-09-10T18:42:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:48 smithi028 ceph-mon[29291]: 1.18 scrub ok 2024-09-10T18:42:48.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:48.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:48 smithi204 ceph-mon[32357]: 1.18 scrub starts 2024-09-10T18:42:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:48 smithi204 ceph-mon[32357]: pgmap v957: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s 2024-09-10T18:42:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:48 smithi204 ceph-mon[32357]: 1.18 scrub ok 2024-09-10T18:42:48.605 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:48.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:48.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:48.741 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:48.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:48.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:48.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:48.875 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:48.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:48.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:48.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:49.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:49.009 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:49.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:49.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:49.134 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:49.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:49.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:49.236 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:49.272 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:49.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:49.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:49.374 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:49.413 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:49.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:49.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:49.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:49 smithi028 ceph-mon[39220]: 4.1c scrub starts 2024-09-10T18:42:49.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:49 smithi028 ceph-mon[39220]: 4.1c scrub ok 2024-09-10T18:42:49.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:49 smithi028 ceph-mon[39220]: 1.12 scrub starts 2024-09-10T18:42:49.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:49 smithi028 ceph-mon[39220]: 1.12 scrub ok 2024-09-10T18:42:49.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:49 smithi028 ceph-mon[39220]: 6.13 deep-scrub starts 2024-09-10T18:42:49.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:49 smithi028 ceph-mon[39220]: 6.13 deep-scrub ok 2024-09-10T18:42:49.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:49 smithi028 ceph-mon[39220]: 1.7 scrub starts 2024-09-10T18:42:49.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:49 smithi028 ceph-mon[39220]: 1.7 scrub ok 2024-09-10T18:42:49.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:49 smithi028 ceph-mon[29291]: 4.1c scrub starts 2024-09-10T18:42:49.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:49 smithi028 ceph-mon[29291]: 4.1c scrub ok 2024-09-10T18:42:49.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:49 smithi028 ceph-mon[29291]: 1.12 scrub starts 2024-09-10T18:42:49.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:49 smithi028 ceph-mon[29291]: 1.12 scrub ok 2024-09-10T18:42:49.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:49 smithi028 ceph-mon[29291]: 6.13 deep-scrub starts 2024-09-10T18:42:49.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:49 smithi028 ceph-mon[29291]: 6.13 deep-scrub ok 2024-09-10T18:42:49.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:49 smithi028 ceph-mon[29291]: 1.7 scrub starts 2024-09-10T18:42:49.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:49 smithi028 ceph-mon[29291]: 1.7 scrub ok 2024-09-10T18:42:49.515 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:49.545 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:49.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:49.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:49 smithi204 ceph-mon[32357]: 4.1c scrub starts 2024-09-10T18:42:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:49 smithi204 ceph-mon[32357]: 4.1c scrub ok 2024-09-10T18:42:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:49 smithi204 ceph-mon[32357]: 1.12 scrub starts 2024-09-10T18:42:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:49 smithi204 ceph-mon[32357]: 1.12 scrub ok 2024-09-10T18:42:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:49 smithi204 ceph-mon[32357]: 6.13 deep-scrub starts 2024-09-10T18:42:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:49 smithi204 ceph-mon[32357]: 6.13 deep-scrub ok 2024-09-10T18:42:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:49 smithi204 ceph-mon[32357]: 1.7 scrub starts 2024-09-10T18:42:49.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:49 smithi204 ceph-mon[32357]: 1.7 scrub ok 2024-09-10T18:42:49.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:49.686 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:49.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:49.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:49.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:49.829 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:49.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:49.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:49.931 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:49.967 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:49.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:49.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:50.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:50.107 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:50.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:50.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:50.209 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:50.239 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:50.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:50.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:50.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:50 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:42:50.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:50 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:42:50.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:50 smithi028 ceph-mon[29291]: pgmap v958: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s 2024-09-10T18:42:50.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:50 smithi028 ceph-mon[29291]: 11.c scrub starts 2024-09-10T18:42:50.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:50 smithi028 ceph-mon[29291]: 1.8 scrub starts 2024-09-10T18:42:50.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:50 smithi028 ceph-mon[29291]: 1.8 scrub ok 2024-09-10T18:42:50.287 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:50 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:42:50.288 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:50 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:42:50.288 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:50 smithi028 ceph-mon[39220]: pgmap v958: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s 2024-09-10T18:42:50.288 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:50 smithi028 ceph-mon[39220]: 11.c scrub starts 2024-09-10T18:42:50.288 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:50 smithi028 ceph-mon[39220]: 1.8 scrub starts 2024-09-10T18:42:50.288 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:50 smithi028 ceph-mon[39220]: 1.8 scrub ok 2024-09-10T18:42:50.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:50.383 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:50.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:50.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:50.485 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:50.524 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:50.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:50.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:50 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:42:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:50 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:42:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:50 smithi204 ceph-mon[32357]: pgmap v958: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s 2024-09-10T18:42:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:50 smithi204 ceph-mon[32357]: 11.c scrub starts 2024-09-10T18:42:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:50 smithi204 ceph-mon[32357]: 1.8 scrub starts 2024-09-10T18:42:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:50 smithi204 ceph-mon[32357]: 1.8 scrub ok 2024-09-10T18:42:50.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:50.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:50.767 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:50.805 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:50.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:50.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:50.907 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:50.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:50.939 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:50.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:51.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:51.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:51.075 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:51.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:51.176 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:51.190 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:42:51.191 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:42:51.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:51.203 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:51.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:51.234 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:42:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:42:51.267 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:42:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:42:51.296 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:42:51.297 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:42:51.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:51.331 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:42:51.331 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:42:51.346 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:51.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:51.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:51.366 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:42:51.366 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:42:51.398 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:42:51.399 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:42:51.416 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:42:51] "GET /metrics HTTP/1.1" 200 224977 "" "Prometheus/2.43.0" 2024-09-10T18:42:51.416 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:42:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:42:51.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[39220]: 3.d scrub starts 2024-09-10T18:42:51.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[39220]: 3.d scrub ok 2024-09-10T18:42:51.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[39220]: osdmap e411: 8 total, 8 up, 7 in 2024-09-10T18:42:51.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[39220]: 11.0 scrub starts 2024-09-10T18:42:51.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[39220]: 1.a scrub starts 2024-09-10T18:42:51.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:42:51.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:42:51.418 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[39220]: 1.a scrub ok 2024-09-10T18:42:51.418 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[39220]: 11.c scrub ok 2024-09-10T18:42:51.418 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[39220]: 6.17 scrub starts 2024-09-10T18:42:51.418 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[39220]: 6.17 scrub ok 2024-09-10T18:42:51.418 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[39220]: 4.5 scrub starts 2024-09-10T18:42:51.418 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[39220]: 4.5 scrub ok 2024-09-10T18:42:51.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[29291]: 3.d scrub starts 2024-09-10T18:42:51.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[29291]: 3.d scrub ok 2024-09-10T18:42:51.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[29291]: osdmap e411: 8 total, 8 up, 7 in 2024-09-10T18:42:51.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[29291]: 11.0 scrub starts 2024-09-10T18:42:51.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[29291]: 1.a scrub starts 2024-09-10T18:42:51.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:42:51.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:42:51.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[29291]: 1.a scrub ok 2024-09-10T18:42:51.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[29291]: 11.c scrub ok 2024-09-10T18:42:51.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[29291]: 6.17 scrub starts 2024-09-10T18:42:51.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[29291]: 6.17 scrub ok 2024-09-10T18:42:51.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[29291]: 4.5 scrub starts 2024-09-10T18:42:51.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:51 smithi028 ceph-mon[29291]: 4.5 scrub ok 2024-09-10T18:42:51.448 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:51.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:51.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:51.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:51.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:51 smithi204 ceph-mon[32357]: 3.d scrub starts 2024-09-10T18:42:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:51 smithi204 ceph-mon[32357]: 3.d scrub ok 2024-09-10T18:42:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:51 smithi204 ceph-mon[32357]: osdmap e411: 8 total, 8 up, 7 in 2024-09-10T18:42:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:51 smithi204 ceph-mon[32357]: 11.0 scrub starts 2024-09-10T18:42:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:51 smithi204 ceph-mon[32357]: 1.a scrub starts 2024-09-10T18:42:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:51 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:42:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:51 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:42:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:51 smithi204 ceph-mon[32357]: 1.a scrub ok 2024-09-10T18:42:51.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:51 smithi204 ceph-mon[32357]: 11.c scrub ok 2024-09-10T18:42:51.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:51 smithi204 ceph-mon[32357]: 6.17 scrub starts 2024-09-10T18:42:51.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:51 smithi204 ceph-mon[32357]: 6.17 scrub ok 2024-09-10T18:42:51.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:51 smithi204 ceph-mon[32357]: 4.5 scrub starts 2024-09-10T18:42:51.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:51 smithi204 ceph-mon[32357]: 4.5 scrub ok 2024-09-10T18:42:51.610 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:51.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:51.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:51.623 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:42:51.628 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:42:51.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:51.752 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:51.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:51.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:51.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:51.895 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:51.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:51.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:51.997 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:52.032 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:52.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:52.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:52.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:52.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:52.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:52.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:52.299 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:52.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:52.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:52.401 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:52.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[29291]: osdmap e412: 8 total, 8 up, 7 in 2024-09-10T18:42:52.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1646413783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:42:52.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[29291]: pgmap v961: 178 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 176 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-09-10T18:42:52.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[29291]: 6.1 scrub starts 2024-09-10T18:42:52.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[29291]: 6.1 scrub ok 2024-09-10T18:42:52.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:42:52.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:42:52.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[29291]: 11.0 scrub ok 2024-09-10T18:42:52.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[29291]: 6.c scrub starts 2024-09-10T18:42:52.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[29291]: 6.c scrub ok 2024-09-10T18:42:52.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[39220]: osdmap e412: 8 total, 8 up, 7 in 2024-09-10T18:42:52.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1646413783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:42:52.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[39220]: pgmap v961: 178 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 176 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-09-10T18:42:52.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[39220]: 6.1 scrub starts 2024-09-10T18:42:52.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[39220]: 6.1 scrub ok 2024-09-10T18:42:52.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:42:52.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:42:52.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[39220]: 11.0 scrub ok 2024-09-10T18:42:52.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[39220]: 6.c scrub starts 2024-09-10T18:42:52.426 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:52 smithi028 ceph-mon[39220]: 6.c scrub ok 2024-09-10T18:42:52.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:52.444 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:52.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:52 smithi204 ceph-mon[32357]: osdmap e412: 8 total, 8 up, 7 in 2024-09-10T18:42:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:52 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1646413783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:42:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:52 smithi204 ceph-mon[32357]: pgmap v961: 178 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 176 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-09-10T18:42:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:52 smithi204 ceph-mon[32357]: 6.1 scrub starts 2024-09-10T18:42:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:52 smithi204 ceph-mon[32357]: 6.1 scrub ok 2024-09-10T18:42:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:52 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:42:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:52 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:42:52.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:52 smithi204 ceph-mon[32357]: 11.0 scrub ok 2024-09-10T18:42:52.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:52 smithi204 ceph-mon[32357]: 6.c scrub starts 2024-09-10T18:42:52.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:52 smithi204 ceph-mon[32357]: 6.c scrub ok 2024-09-10T18:42:52.506 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:42:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:42:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:42:52.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:52.576 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:52.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:52.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:52.678 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:52.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:52.715 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:52.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:52.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:52.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:52.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:52.942 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:52.974 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:52.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:52.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:53.011 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:42:53.012 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-10T18:42:53.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.012 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25849.0:230563 11.2 11:41d6fd55:::benchmark_data_smithi028_186962_object89956:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-10T18:42:53.012 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:42:53.004943+0000", 2024-09-10T18:42:53.012 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0035466479999999999, 2024-09-10T18:42:53.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003573854, 2024-09-10T18:42:53.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:42:53.012 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:42:53.012 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:42:53.012 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25849", 2024-09-10T18:42:53.012 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:42:53.013 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 230563 2024-09-10T18:42:53.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.013 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:42:53.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:42:53.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.004943+0000", 2024-09-10T18:42:53.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:53.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:42:53.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.004943+0000", 2024-09-10T18:42:53.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:53.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:42:53.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.004941+0000", 2024-09-10T18:42:53.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-10T18:42:53.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:42:53.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.004949+0000", 2024-09-10T18:42:53.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2800000000000003e-06 2024-09-10T18:42:53.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:42:53.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.004966+0000", 2024-09-10T18:42:53.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7130999999999999e-05 2024-09-10T18:42:53.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:42:53.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.004973+0000", 2024-09-10T18:42:53.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0439999999999998e-06 2024-09-10T18:42:53.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:42:53.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.008058+0000", 2024-09-10T18:42:53.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030852639999999999 2024-09-10T18:42:53.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:42:53.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.008393+0000", 2024-09-10T18:42:53.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033446700000000002 2024-09-10T18:42:53.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:42:53.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.008519+0000", 2024-09-10T18:42:53.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000126511 2024-09-10T18:42:53.017 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:53.017 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:42:53.017 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:53.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.017 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25849.0:230564 11.6 11:6445e30a:::benchmark_data_smithi028_186962_object89957:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-10T18:42:53.017 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:42:53.004979+0000", 2024-09-10T18:42:53.017 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0035107960000000001, 2024-09-10T18:42:53.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003620889, 2024-09-10T18:42:53.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:42:53.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:42:53.018 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:42:53.018 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25849", 2024-09-10T18:42:53.018 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:42:53.018 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 230564 2024-09-10T18:42:53.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.018 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:42:53.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:42:53.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.004979+0000", 2024-09-10T18:42:53.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:53.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:42:53.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.004979+0000", 2024-09-10T18:42:53.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:53.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:42:53.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.004978+0000", 2024-09-10T18:42:53.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-10T18:42:53.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:42:53.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.004982+0000", 2024-09-10T18:42:53.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1030000000000004e-06 2024-09-10T18:42:53.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:42:53.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.004988+0000", 2024-09-10T18:42:53.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9109999999999998e-06 2024-09-10T18:42:53.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:42:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.004992+0000", 2024-09-10T18:42:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2509999999999996e-06 2024-09-10T18:42:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:42:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.005125+0000", 2024-09-10T18:42:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013305599999999999 2024-09-10T18:42:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:42:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.005251+0000", 2024-09-10T18:42:53.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012542799999999999 2024-09-10T18:42:53.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:42:53.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.005366+0000", 2024-09-10T18:42:53.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000115596 2024-09-10T18:42:53.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:42:53.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.008262+0000", 2024-09-10T18:42:53.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028957409999999999 2024-09-10T18:42:53.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:42:53.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.008365+0000", 2024-09-10T18:42:53.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010309799999999999 2024-09-10T18:42:53.023 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:53.023 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:42:53.023 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:53.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.023 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.25849.0:230556 11.2 e413/374)", 2024-09-10T18:42:53.023 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:42:53.007125+0000", 2024-09-10T18:42:53.023 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0013649440000000001, 2024-09-10T18:42:53.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015561640000000001, 2024-09-10T18:42:53.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:42:53.024 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-09-10T18:42:53.024 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:42:53.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:42:53.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.007125+0000", 2024-09-10T18:42:53.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:53.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:42:53.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.007125+0000", 2024-09-10T18:42:53.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:53.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:42:53.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.007123+0000", 2024-09-10T18:42:53.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-10T18:42:53.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:42:53.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.007130+0000", 2024-09-10T18:42:53.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4219999999999997e-06 2024-09-10T18:42:53.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:42:53.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.007143+0000", 2024-09-10T18:42:53.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3149000000000001e-05 2024-09-10T18:42:53.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:42:53.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.007149+0000", 2024-09-10T18:42:53.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0290000000000002e-06 2024-09-10T18:42:53.026 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:53.026 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:42:53.027 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:53.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.027 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25849.0:230548 11.2 11:4f8aaf33:::benchmark_data_smithi028_186962_object89941:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-10T18:42:53.027 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:42:52.995552+0000", 2024-09-10T18:42:53.027 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.012937402000000001, 2024-09-10T18:42:53.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013185313000000001, 2024-09-10T18:42:53.027 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:42:53.027 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:42:53.027 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:42:53.027 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25849", 2024-09-10T18:42:53.028 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:42:53.028 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 230548 2024-09-10T18:42:53.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.028 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:42:53.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:42:53.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:52.995552+0000", 2024-09-10T18:42:53.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:53.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:42:53.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:52.995552+0000", 2024-09-10T18:42:53.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:53.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:42:53.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:52.995550+0000", 2024-09-10T18:42:53.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:42:53.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:42:53.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:52.995557+0000", 2024-09-10T18:42:53.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4880000000000004e-06 2024-09-10T18:42:53.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:42:53.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:52.995567+0000", 2024-09-10T18:42:53.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7920000000000004e-06 2024-09-10T18:42:53.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:42:53.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:52.995572+0000", 2024-09-10T18:42:53.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5509999999999999e-06 2024-09-10T18:42:53.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:42:53.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:52.997590+0000", 2024-09-10T18:42:53.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020172839999999998 2024-09-10T18:42:53.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:42:53.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:52.997695+0000", 2024-09-10T18:42:53.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010576 2024-09-10T18:42:53.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:42:53.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:52.997803+0000", 2024-09-10T18:42:53.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108157 2024-09-10T18:42:53.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-10T18:42:53.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.001165+0000", 2024-09-10T18:42:53.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033618229999999999 2024-09-10T18:42:53.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:42:53.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.002079+0000", 2024-09-10T18:42:53.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00091339600000000002 2024-09-10T18:42:53.033 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:53.033 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:42:53.033 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:53.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.034 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25849.0:230556 11.2 11:42b0441c:::benchmark_data_smithi028_186962_object89949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-10T18:42:53.034 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-10T18:42:53.002972+0000", 2024-09-10T18:42:53.034 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0055180849999999998, 2024-09-10T18:42:53.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0058536969999999997, 2024-09-10T18:42:53.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-10T18:42:53.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-09-10T18:42:53.034 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-10T18:42:53.034 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25849", 2024-09-10T18:42:53.034 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:42:53.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 230556 2024-09-10T18:42:53.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.035 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-10T18:42:53.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-10T18:42:53.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.002972+0000", 2024-09-10T18:42:53.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:53.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-10T18:42:53.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.002972+0000", 2024-09-10T18:42:53.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-10T18:42:53.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-10T18:42:53.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.002969+0000", 2024-09-10T18:42:53.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-10T18:42:53.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-10T18:42:53.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.002977+0000", 2024-09-10T18:42:53.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3159999999999999e-06 2024-09-10T18:42:53.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-10T18:42:53.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.002993+0000", 2024-09-10T18:42:53.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5750999999999999e-05 2024-09-10T18:42:53.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-10T18:42:53.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.002999+0000", 2024-09-10T18:42:53.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7839999999999998e-06 2024-09-10T18:42:53.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-10T18:42:53.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.003040+0000", 2024-09-10T18:42:53.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0323e-05 2024-09-10T18:42:53.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-10T18:42:53.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.003336+0000", 2024-09-10T18:42:53.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029613600000000001 2024-09-10T18:42:53.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:42:53.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:42:53.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:42:53.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-10T18:42:53.003721+0000", 2024-09-10T18:42:53.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00038530900000000002 2024-09-10T18:42:53.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:53.039 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:42:53.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:53.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:42:53.039 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:42:53.039 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 5 2024-09-10T18:42:53.039 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:42:53.075 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:53.107 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:53.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:53.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:53.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:53.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:53.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:53.344 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:53.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:53.378 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:53.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:53.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:53 smithi028 ceph-mon[39220]: from='client.36011 v1:172.21.15.28:0/516479012' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:42:53.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:53 smithi028 ceph-mon[39220]: 4.18 scrub starts 2024-09-10T18:42:53.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:53 smithi028 ceph-mon[39220]: 4.18 scrub ok 2024-09-10T18:42:53.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:53 smithi028 ceph-mon[39220]: osdmap e413: 8 total, 8 up, 7 in 2024-09-10T18:42:53.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:53 smithi028 ceph-mon[29291]: from='client.36011 v1:172.21.15.28:0/516479012' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:42:53.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:53 smithi028 ceph-mon[29291]: 4.18 scrub starts 2024-09-10T18:42:53.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:53 smithi028 ceph-mon[29291]: 4.18 scrub ok 2024-09-10T18:42:53.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:53 smithi028 ceph-mon[29291]: osdmap e413: 8 total, 8 up, 7 in 2024-09-10T18:42:53.471 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":961,"stamp":"2024-09-10T18:42:51.219357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3527481588,"num_objects":54013,"num_object_clones":0,"num_object_copies":162039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54013,"num_whiteouts":0,"num_read":2841,"num_read_kb":2593,"num_write":228395,"num_write_kb":8998700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":334384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67545,"ondisk_log_size":67545,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16732436,"kb_used_data":10463460,"kb_used_omap":17,"kb_used_meta":6268526,"kb_avail":639483628,"statfs":{"total":671965249536,"available":654831235072,"internally_reserved":0,"allocated":10714583040,"data_stored":10711771592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18329,"internal_metadata":6418970727},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-844038144,"num_objects":-12879,"num_object_clones":0,"num_object_copies":-38637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12879,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12879,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002980"},"pg_stats":[{"pgid":"1.1d","version":"76'81","reported_seq":165,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.127512+0000","last_change":"2024-09-10T18:40:49.241624+0000","last_active":"2024-09-10T18:42:42.127512+0000","last_peered":"2024-09-10T18:42:42.127512+0000","last_clean":"2024-09-10T18:42:42.127512+0000","last_became_active":"2024-09-10T18:34:34.865894+0000","last_became_peered":"2024-09-10T18:34:34.865894+0000","last_unstale":"2024-09-10T18:42:42.127512+0000","last_undegraded":"2024-09-10T18:42:42.127512+0000","last_fullsized":"2024-09-10T18:42:42.127512+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:49.241512+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:40:49.241512+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070814190000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":469,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.122885+0000","last_change":"2024-09-10T18:41:27.836345+0000","last_active":"2024-09-10T18:42:42.122885+0000","last_peered":"2024-09-10T18:42:42.122885+0000","last_clean":"2024-09-10T18:42:42.122885+0000","last_became_active":"2024-09-10T18:39:25.247259+0000","last_became_peered":"2024-09-10T18:39:25.247259+0000","last_unstale":"2024-09-10T18:42:42.122885+0000","last_undegraded":"2024-09-10T18:42:42.122885+0000","last_fullsized":"2024-09-10T18:42:42.122885+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:27.836236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:41:27.836236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039483269999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"400'25","reported_seq":491,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.057480+0000","last_change":"2024-09-10T18:41:03.025318+0000","last_active":"2024-09-10T18:42:35.057480+0000","last_peered":"2024-09-10T18:42:35.057480+0000","last_clean":"2024-09-10T18:42:35.057480+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:42:35.057480+0000","last_undegraded":"2024-09-10T18:42:35.057480+0000","last_fullsized":"2024-09-10T18:42:35.057480+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'19","last_scrub_stamp":"2024-09-10T18:41:03.025231+0000","last_deep_scrub":"342'19","last_deep_scrub_stamp":"2024-09-10T18:37:49.354781+0000","last_clean_scrub_stamp":"2024-09-10T18:41:03.025231+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005898276,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":476,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.133151+0000","last_change":"2024-09-10T18:41:48.436189+0000","last_active":"2024-09-10T18:42:42.133151+0000","last_peered":"2024-09-10T18:42:42.133151+0000","last_clean":"2024-09-10T18:42:42.133151+0000","last_became_active":"2024-09-10T18:39:00.910095+0000","last_became_peered":"2024-09-10T18:39:00.910095+0000","last_unstale":"2024-09-10T18:42:42.133151+0000","last_undegraded":"2024-09-10T18:42:42.133151+0000","last_fullsized":"2024-09-10T18:42:42.133151+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:48.436106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:41:48.436106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049447019999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":465,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.057750+0000","last_change":"2024-09-10T18:41:37.007816+0000","last_active":"2024-09-10T18:42:35.057750+0000","last_peered":"2024-09-10T18:42:35.057750+0000","last_clean":"2024-09-10T18:42:35.057750+0000","last_became_active":"2024-09-10T18:39:00.889898+0000","last_became_peered":"2024-09-10T18:39:00.889898+0000","last_unstale":"2024-09-10T18:42:35.057750+0000","last_undegraded":"2024-09-10T18:42:35.057750+0000","last_fullsized":"2024-09-10T18:42:35.057750+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:37.007736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:41:37.007736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:46.764418+0000","scrub_duration":0.0045586869999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'81","reported_seq":163,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129676+0000","last_change":"2024-09-10T18:42:33.135226+0000","last_active":"2024-09-10T18:42:42.129676+0000","last_peered":"2024-09-10T18:42:42.129676+0000","last_clean":"2024-09-10T18:42:42.129676+0000","last_became_active":"2024-09-10T18:34:34.867388+0000","last_became_peered":"2024-09-10T18:34:34.867388+0000","last_unstale":"2024-09-10T18:42:42.129676+0000","last_undegraded":"2024-09-10T18:42:42.129676+0000","last_fullsized":"2024-09-10T18:42:42.129676+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:33.134846+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:33.134846+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:41.382520+0000","scrub_duration":0.0088763399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":442,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.059090+0000","last_change":"2024-09-10T18:40:39.463295+0000","last_active":"2024-09-10T18:42:35.059090+0000","last_peered":"2024-09-10T18:42:35.059090+0000","last_clean":"2024-09-10T18:42:35.059090+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:42:35.059090+0000","last_undegraded":"2024-09-10T18:42:35.059090+0000","last_fullsized":"2024-09-10T18:42:35.059090+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:39.463211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:39.463211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069721330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":448,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.066633+0000","last_change":"2024-09-10T18:39:54.892126+0000","last_active":"2024-09-10T18:42:35.066633+0000","last_peered":"2024-09-10T18:42:35.066633+0000","last_clean":"2024-09-10T18:42:35.066633+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:42:35.066633+0000","last_undegraded":"2024-09-10T18:42:35.066633+0000","last_fullsized":"2024-09-10T18:42:35.066633+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:54.892069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:39:54.892069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001308317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"400'46","reported_seq":536,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.066808+0000","last_change":"2024-09-10T18:42:30.196325+0000","last_active":"2024-09-10T18:42:35.066808+0000","last_peered":"2024-09-10T18:42:35.066808+0000","last_clean":"2024-09-10T18:42:35.066808+0000","last_became_active":"2024-09-10T18:39:25.237997+0000","last_became_peered":"2024-09-10T18:39:25.237997+0000","last_unstale":"2024-09-10T18:42:35.066808+0000","last_undegraded":"2024-09-10T18:42:35.066808+0000","last_fullsized":"2024-09-10T18:42:35.066808+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'46","last_scrub_stamp":"2024-09-10T18:42:30.196228+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:42:30.196228+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:43.745670+0000","scrub_duration":0.0074592269999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":454,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.057422+0000","last_change":"2024-09-10T18:41:43.981927+0000","last_active":"2024-09-10T18:42:35.057422+0000","last_peered":"2024-09-10T18:42:35.057422+0000","last_clean":"2024-09-10T18:42:35.057422+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:42:35.057422+0000","last_undegraded":"2024-09-10T18:42:35.057422+0000","last_fullsized":"2024-09-10T18:42:35.057422+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:43.981864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:41:43.981864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:58.783321+0000","scrub_duration":0.010372925,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":52,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129831+0000","last_change":"2024-09-10T18:41:00.039020+0000","last_active":"2024-09-10T18:42:42.129831+0000","last_peered":"2024-09-10T18:42:42.129831+0000","last_clean":"2024-09-10T18:42:42.129831+0000","last_became_active":"2024-09-10T18:40:10.986591+0000","last_became_peered":"2024-09-10T18:40:10.986591+0000","last_unstale":"2024-09-10T18:42:42.129831+0000","last_undegraded":"2024-09-10T18:42:42.129831+0000","last_fullsized":"2024-09-10T18:42:42.129831+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:00.038936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_clean_scrub_stamp":"2024-09-10T18:41:00.038936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013791546999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"400'21","reported_seq":510,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:44.184843+0000","last_change":"2024-09-10T18:42:44.184802+0000","last_active":"2024-09-10T18:42:44.184843+0000","last_peered":"2024-09-10T18:42:44.184843+0000","last_clean":"2024-09-10T18:42:44.184843+0000","last_became_active":"2024-09-10T18:39:25.237946+0000","last_became_peered":"2024-09-10T18:39:25.237946+0000","last_unstale":"2024-09-10T18:42:44.184843+0000","last_undegraded":"2024-09-10T18:42:44.184843+0000","last_fullsized":"2024-09-10T18:42:44.184843+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'21","last_scrub_stamp":"2024-09-10T18:42:44.184715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:42:44.184715+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:54.200219+0000","scrub_duration":0.0061211579999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":83,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.118459+0000","last_change":"2024-09-10T18:41:33.904603+0000","last_active":"2024-09-10T18:42:42.118459+0000","last_peered":"2024-09-10T18:42:42.118459+0000","last_clean":"2024-09-10T18:42:42.118459+0000","last_became_active":"2024-09-10T18:39:25.239738+0000","last_became_peered":"2024-09-10T18:39:25.239738+0000","last_unstale":"2024-09-10T18:42:42.118459+0000","last_undegraded":"2024-09-10T18:42:42.118459+0000","last_fullsized":"2024-09-10T18:42:42.118459+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:33.904519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_clean_scrub_stamp":"2024-09-10T18:41:33.904519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028331189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":82,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.131501+0000","last_change":"2024-09-10T18:42:34.174839+0000","last_active":"2024-09-10T18:42:42.131501+0000","last_peered":"2024-09-10T18:42:42.131501+0000","last_clean":"2024-09-10T18:42:42.131501+0000","last_became_active":"2024-09-10T18:39:25.255362+0000","last_became_peered":"2024-09-10T18:39:25.255362+0000","last_unstale":"2024-09-10T18:42:42.131501+0000","last_undegraded":"2024-09-10T18:42:42.131501+0000","last_fullsized":"2024-09-10T18:42:42.131501+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:34.174638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:42:34.174638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:43.645111+0000","scrub_duration":0.005797562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"76'81","reported_seq":228,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.066469+0000","last_change":"2024-09-10T18:42:26.156095+0000","last_active":"2024-09-10T18:42:35.066469+0000","last_peered":"2024-09-10T18:42:35.066469+0000","last_clean":"2024-09-10T18:42:35.066469+0000","last_became_active":"2024-09-10T18:34:34.863102+0000","last_became_peered":"2024-09-10T18:34:34.863102+0000","last_unstale":"2024-09-10T18:42:35.066469+0000","last_undegraded":"2024-09-10T18:42:35.066469+0000","last_fullsized":"2024-09-10T18:42:35.066469+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:26.156014+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:26.156014+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:32.682111+0000","scrub_duration":0.0066787549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":83,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.119101+0000","last_change":"2024-09-10T18:42:30.280858+0000","last_active":"2024-09-10T18:42:42.119101+0000","last_peered":"2024-09-10T18:42:42.119101+0000","last_clean":"2024-09-10T18:42:42.119101+0000","last_became_active":"2024-09-10T18:39:00.915294+0000","last_became_peered":"2024-09-10T18:39:00.915294+0000","last_unstale":"2024-09-10T18:42:42.119101+0000","last_undegraded":"2024-09-10T18:42:42.119101+0000","last_fullsized":"2024-09-10T18:42:42.119101+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:30.280771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:39:49.974584+0000","last_clean_scrub_stamp":"2024-09-10T18:42:30.280771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:31.140049+0000","scrub_duration":0.0020095740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"400'11","reported_seq":496,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.057881+0000","last_change":"2024-09-10T18:41:50.059924+0000","last_active":"2024-09-10T18:42:35.057881+0000","last_peered":"2024-09-10T18:42:35.057881+0000","last_clean":"2024-09-10T18:42:35.057881+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:42:35.057881+0000","last_undegraded":"2024-09-10T18:42:35.057881+0000","last_fullsized":"2024-09-10T18:42:35.057881+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'11","last_scrub_stamp":"2024-09-10T18:41:50.059835+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:41:50.059835+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:50.886546+0000","scrub_duration":0.0053330019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":445,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.128861+0000","last_change":"2024-09-10T18:40:43.068881+0000","last_active":"2024-09-10T18:42:42.128861+0000","last_peered":"2024-09-10T18:42:42.128861+0000","last_clean":"2024-09-10T18:42:42.128861+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:42:42.128861+0000","last_undegraded":"2024-09-10T18:42:42.128861+0000","last_fullsized":"2024-09-10T18:42:42.128861+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:43.068784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:40:43.068784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037849960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":73,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.117833+0000","last_change":"2024-09-10T18:41:29.917083+0000","last_active":"2024-09-10T18:42:42.117833+0000","last_peered":"2024-09-10T18:42:42.117833+0000","last_clean":"2024-09-10T18:42:42.117833+0000","last_became_active":"2024-09-10T18:39:00.917007+0000","last_became_peered":"2024-09-10T18:39:00.917007+0000","last_unstale":"2024-09-10T18:42:42.117833+0000","last_undegraded":"2024-09-10T18:42:42.117833+0000","last_fullsized":"2024-09-10T18:42:42.117833+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:29.917010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:41:29.917010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061701890000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'81","reported_seq":321,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:40.392947+0000","last_change":"2024-09-10T18:42:40.392595+0000","last_active":"2024-09-10T18:42:40.392947+0000","last_peered":"2024-09-10T18:42:40.392947+0000","last_clean":"2024-09-10T18:42:40.392947+0000","last_became_active":"2024-09-10T18:39:25.241941+0000","last_became_peered":"2024-09-10T18:39:25.241941+0000","last_unstale":"2024-09-10T18:42:40.392947+0000","last_undegraded":"2024-09-10T18:42:40.392947+0000","last_fullsized":"2024-09-10T18:42:40.392947+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:40.392516+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:40.392516+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:00.717024+0000","scrub_duration":0.012361784000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":472,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058377+0000","last_change":"2024-09-10T18:41:34.061934+0000","last_active":"2024-09-10T18:42:35.058377+0000","last_peered":"2024-09-10T18:42:35.058377+0000","last_clean":"2024-09-10T18:42:35.058377+0000","last_became_active":"2024-09-10T18:39:00.895860+0000","last_became_peered":"2024-09-10T18:39:00.895860+0000","last_unstale":"2024-09-10T18:42:35.058377+0000","last_undegraded":"2024-09-10T18:42:35.058377+0000","last_fullsized":"2024-09-10T18:42:35.058377+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:34.061855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:41:34.061855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:43.836690+0000","scrub_duration":0.019856072999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":81,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.057993+0000","last_change":"2024-09-10T18:42:29.224932+0000","last_active":"2024-09-10T18:42:35.057993+0000","last_peered":"2024-09-10T18:42:35.057993+0000","last_clean":"2024-09-10T18:42:35.057993+0000","last_became_active":"2024-09-10T18:39:25.231931+0000","last_became_peered":"2024-09-10T18:39:25.231931+0000","last_unstale":"2024-09-10T18:42:35.057993+0000","last_undegraded":"2024-09-10T18:42:35.057993+0000","last_fullsized":"2024-09-10T18:42:35.057993+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:29.224782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:42:29.224782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:55.092649+0000","scrub_duration":0.0076768009999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"400'39","reported_seq":557,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:47.551189+0000","last_change":"2024-09-10T18:42:47.551036+0000","last_active":"2024-09-10T18:42:47.551189+0000","last_peered":"2024-09-10T18:42:47.551189+0000","last_clean":"2024-09-10T18:42:47.551189+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:42:47.551189+0000","last_undegraded":"2024-09-10T18:42:47.551189+0000","last_fullsized":"2024-09-10T18:42:47.551189+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'39","last_scrub_stamp":"2024-09-10T18:42:47.550932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:42:47.550932+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:14.451330+0000","scrub_duration":0.0058170050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":476,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.127891+0000","last_change":"2024-09-10T18:42:35.205645+0000","last_active":"2024-09-10T18:42:42.127891+0000","last_peered":"2024-09-10T18:42:42.127891+0000","last_clean":"2024-09-10T18:42:42.127891+0000","last_became_active":"2024-09-10T18:39:25.246541+0000","last_became_peered":"2024-09-10T18:39:25.246541+0000","last_unstale":"2024-09-10T18:42:42.127891+0000","last_undegraded":"2024-09-10T18:42:42.127891+0000","last_fullsized":"2024-09-10T18:42:42.127891+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:35.205557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:42:35.205557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:42.211397+0000","scrub_duration":0.0048486780000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'81","reported_seq":87,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:47.235340+0000","last_change":"2024-09-10T18:42:47.235267+0000","last_active":"2024-09-10T18:42:47.235340+0000","last_peered":"2024-09-10T18:42:47.235340+0000","last_clean":"2024-09-10T18:42:47.235340+0000","last_became_active":"2024-09-10T18:39:00.907303+0000","last_became_peered":"2024-09-10T18:39:00.907303+0000","last_unstale":"2024-09-10T18:42:47.235340+0000","last_undegraded":"2024-09-10T18:42:47.235340+0000","last_fullsized":"2024-09-10T18:42:47.235340+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:47.235176+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:47.235176+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:01.178362+0000","scrub_duration":0.038577114000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":462,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058240+0000","last_change":"2024-09-10T18:40:36.531564+0000","last_active":"2024-09-10T18:42:35.058240+0000","last_peered":"2024-09-10T18:42:35.058240+0000","last_clean":"2024-09-10T18:42:35.058240+0000","last_became_active":"2024-09-10T18:39:00.904769+0000","last_became_peered":"2024-09-10T18:39:00.904769+0000","last_unstale":"2024-09-10T18:42:35.058240+0000","last_undegraded":"2024-09-10T18:42:35.058240+0000","last_fullsized":"2024-09-10T18:42:35.058240+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:36.531474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:36.531474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050165130000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":458,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:45.180720+0000","last_change":"2024-09-10T18:42:45.180685+0000","last_active":"2024-09-10T18:42:45.180720+0000","last_peered":"2024-09-10T18:42:45.180720+0000","last_clean":"2024-09-10T18:42:45.180720+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:42:45.180720+0000","last_undegraded":"2024-09-10T18:42:45.180720+0000","last_fullsized":"2024-09-10T18:42:45.180720+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:45.180608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:42:45.180608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:02.103777+0000","scrub_duration":0.0058505279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"400'29","reported_seq":507,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058296+0000","last_change":"2024-09-10T18:40:55.739187+0000","last_active":"2024-09-10T18:42:35.058296+0000","last_peered":"2024-09-10T18:42:35.058296+0000","last_clean":"2024-09-10T18:42:35.058296+0000","last_became_active":"2024-09-10T18:39:11.896502+0000","last_became_peered":"2024-09-10T18:39:11.896502+0000","last_unstale":"2024-09-10T18:42:35.058296+0000","last_undegraded":"2024-09-10T18:42:35.058296+0000","last_fullsized":"2024-09-10T18:42:35.058296+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'23","last_scrub_stamp":"2024-09-10T18:40:55.739083+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:40:55.739083+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062258410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":460,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:46.205115+0000","last_change":"2024-09-10T18:42:46.205068+0000","last_active":"2024-09-10T18:42:46.205115+0000","last_peered":"2024-09-10T18:42:46.205115+0000","last_clean":"2024-09-10T18:42:46.205115+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:42:46.205115+0000","last_undegraded":"2024-09-10T18:42:46.205115+0000","last_fullsized":"2024-09-10T18:42:46.205115+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:46.204993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:08.266279+0000","last_clean_scrub_stamp":"2024-09-10T18:42:46.204993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:50.151980+0000","scrub_duration":0.0046309469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'81","reported_seq":83,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.118392+0000","last_change":"2024-09-10T18:41:00.661030+0000","last_active":"2024-09-10T18:42:42.118392+0000","last_peered":"2024-09-10T18:42:42.118392+0000","last_clean":"2024-09-10T18:42:42.118392+0000","last_became_active":"2024-09-10T18:39:25.240472+0000","last_became_peered":"2024-09-10T18:39:25.240472+0000","last_unstale":"2024-09-10T18:42:42.118392+0000","last_undegraded":"2024-09-10T18:42:42.118392+0000","last_fullsized":"2024-09-10T18:42:42.118392+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:00.660856+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:00.660856+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010899265999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":477,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.128915+0000","last_change":"2024-09-10T18:40:47.040541+0000","last_active":"2024-09-10T18:42:42.128915+0000","last_peered":"2024-09-10T18:42:42.128915+0000","last_clean":"2024-09-10T18:42:42.128915+0000","last_became_active":"2024-09-10T18:39:00.891321+0000","last_became_peered":"2024-09-10T18:39:00.891321+0000","last_unstale":"2024-09-10T18:42:42.128915+0000","last_undegraded":"2024-09-10T18:42:42.128915+0000","last_fullsized":"2024-09-10T18:42:42.128915+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:40:47.040456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:47.040456+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042633510000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":470,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.133274+0000","last_change":"2024-09-10T18:41:37.332757+0000","last_active":"2024-09-10T18:42:42.133274+0000","last_peered":"2024-09-10T18:42:42.133274+0000","last_clean":"2024-09-10T18:42:42.133274+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:42:42.133274+0000","last_undegraded":"2024-09-10T18:42:42.133274+0000","last_fullsized":"2024-09-10T18:42:42.133274+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:37.332672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:41:37.332672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:53.883766+0000","scrub_duration":0.0099109179999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"400'35","reported_seq":370,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129741+0000","last_change":"2024-09-10T18:42:35.122912+0000","last_active":"2024-09-10T18:42:42.129741+0000","last_peered":"2024-09-10T18:42:42.129741+0000","last_clean":"2024-09-10T18:42:42.129741+0000","last_became_active":"2024-09-10T18:39:25.244435+0000","last_became_peered":"2024-09-10T18:39:25.244435+0000","last_unstale":"2024-09-10T18:42:42.129741+0000","last_undegraded":"2024-09-10T18:42:42.129741+0000","last_fullsized":"2024-09-10T18:42:42.129741+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'35","last_scrub_stamp":"2024-09-10T18:42:35.122808+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:42:35.122808+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:52.875093+0000","scrub_duration":0.0045093800000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"362'13","reported_seq":127,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:50.014322+0000","last_change":"2024-09-10T18:42:35.587536+0000","last_active":"2024-09-10T18:42:50.014322+0000","last_peered":"2024-09-10T18:42:50.014322+0000","last_clean":"2024-09-10T18:42:50.014322+0000","last_became_active":"2024-09-10T18:39:25.246551+0000","last_became_peered":"2024-09-10T18:39:25.246551+0000","last_unstale":"2024-09-10T18:42:50.014322+0000","last_undegraded":"2024-09-10T18:42:50.014322+0000","last_fullsized":"2024-09-10T18:42:50.014322+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'13","last_scrub_stamp":"2024-09-10T18:42:35.587453+0000","last_deep_scrub":"217'11","last_deep_scrub_stamp":"2024-09-10T18:30:03.161985+0000","last_clean_scrub_stamp":"2024-09-10T18:42:35.587453+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:45.538903+0000","scrub_duration":0.001931735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"411'14117","reported_seq":14229,"reported_epoch":411,"state":"active+clean+scrubbing","last_fresh":"2024-09-10T18:42:50.744556+0000","last_change":"2024-09-10T18:42:50.256500+0000","last_active":"2024-09-10T18:42:50.744556+0000","last_peered":"2024-09-10T18:42:50.744556+0000","last_clean":"2024-09-10T18:42:50.744556+0000","last_became_active":"2024-09-10T18:40:05.803308+0000","last_became_peered":"2024-09-10T18:40:05.803308+0000","last_unstale":"2024-09-10T18:42:50.744556+0000","last_undegraded":"2024-09-10T18:42:50.744556+0000","last_fullsized":"2024-09-10T18:42:50.744556+0000","mapping_epoch":374,"log_start":"406'10500","ondisk_log_start":"406'10500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":1574,"log_size":3617,"log_dups_size":0,"ondisk_log_size":3617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211353600,"num_objects":3225,"num_object_clones":0,"num_object_copies":9675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14117,"num_write_kb":554944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":73,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.118451+0000","last_change":"2024-09-10T18:40:38.813037+0000","last_active":"2024-09-10T18:42:42.118451+0000","last_peered":"2024-09-10T18:42:42.118451+0000","last_clean":"2024-09-10T18:42:42.118451+0000","last_became_active":"2024-09-10T18:39:00.909261+0000","last_became_peered":"2024-09-10T18:39:00.909261+0000","last_unstale":"2024-09-10T18:42:42.118451+0000","last_undegraded":"2024-09-10T18:42:42.118451+0000","last_fullsized":"2024-09-10T18:42:42.118451+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:38.812860+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:40:38.812860+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050157300000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"400'46","reported_seq":539,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.134205+0000","last_change":"2024-09-10T18:41:45.468203+0000","last_active":"2024-09-10T18:42:42.134205+0000","last_peered":"2024-09-10T18:42:42.134205+0000","last_clean":"2024-09-10T18:42:42.134205+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:42:42.134205+0000","last_undegraded":"2024-09-10T18:42:42.134205+0000","last_fullsized":"2024-09-10T18:42:42.134205+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'46","last_scrub_stamp":"2024-09-10T18:41:45.468076+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:41:45.468076+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:56.701934+0000","scrub_duration":0.0046622160000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":451,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:38.416122+0000","last_change":"2024-09-10T18:42:38.415812+0000","last_active":"2024-09-10T18:42:38.416122+0000","last_peered":"2024-09-10T18:42:38.416122+0000","last_clean":"2024-09-10T18:42:38.416122+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:42:38.416122+0000","last_undegraded":"2024-09-10T18:42:38.416122+0000","last_fullsized":"2024-09-10T18:42:38.416122+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:38.415725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:42:38.415725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:42.038913+0000","scrub_duration":0.0081623489999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"362'11","reported_seq":758,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:45.014490+0000","last_change":"2024-09-10T18:42:23.100920+0000","last_active":"2024-09-10T18:42:45.014490+0000","last_peered":"2024-09-10T18:42:45.014490+0000","last_clean":"2024-09-10T18:42:45.014490+0000","last_became_active":"2024-09-10T18:39:25.239165+0000","last_became_peered":"2024-09-10T18:39:25.239165+0000","last_unstale":"2024-09-10T18:42:45.014490+0000","last_undegraded":"2024-09-10T18:42:45.014490+0000","last_fullsized":"2024-09-10T18:42:45.014490+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'11","last_scrub_stamp":"2024-09-10T18:42:23.100815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:42:23.100815+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:41.322512+0000","scrub_duration":0.0035714409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":93,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.123599+0000","last_change":"2024-09-10T18:42:28.272835+0000","last_active":"2024-09-10T18:42:42.123599+0000","last_peered":"2024-09-10T18:42:42.123599+0000","last_clean":"2024-09-10T18:42:42.123599+0000","last_became_active":"2024-09-10T18:39:25.256173+0000","last_became_peered":"2024-09-10T18:39:25.256173+0000","last_unstale":"2024-09-10T18:42:42.123599+0000","last_undegraded":"2024-09-10T18:42:42.123599+0000","last_fullsized":"2024-09-10T18:42:42.123599+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:28.272680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:42:28.272680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:28.506806+0000","scrub_duration":0.0083610449999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"411'14135","reported_seq":14355,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-10T18:42:50.205961+0000","last_change":"2024-09-10T18:42:41.666862+0000","last_active":"2024-09-10T18:42:50.205961+0000","last_peered":"2024-09-10T18:42:50.205961+0000","last_clean":"2024-09-10T18:42:50.205961+0000","last_became_active":"2024-09-10T18:40:05.806936+0000","last_became_peered":"2024-09-10T18:40:05.806936+0000","last_unstale":"2024-09-10T18:42:50.205961+0000","last_undegraded":"2024-09-10T18:42:50.205961+0000","last_fullsized":"2024-09-10T18:42:50.205961+0000","mapping_epoch":374,"log_start":"405'10400","ondisk_log_start":"405'10400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'13440","last_scrub_stamp":"2024-09-10T18:42:41.666772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:42:41.666772+0000","objects_scrubbed":4075,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:03.056289+0000","scrub_duration":1.539427291,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216465408,"num_objects":3303,"num_object_clones":0,"num_object_copies":9909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14135,"num_write_kb":558016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":384,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058358+0000","last_change":"2024-09-10T18:41:39.072056+0000","last_active":"2024-09-10T18:42:35.058358+0000","last_peered":"2024-09-10T18:42:35.058358+0000","last_clean":"2024-09-10T18:42:35.058358+0000","last_became_active":"2024-09-10T18:34:32.815839+0000","last_became_peered":"2024-09-10T18:34:32.815839+0000","last_unstale":"2024-09-10T18:42:35.058358+0000","last_undegraded":"2024-09-10T18:42:35.058358+0000","last_fullsized":"2024-09-10T18:42:35.058358+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:39.071980+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:39.071980+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:43.710634+0000","scrub_duration":0.0086581420000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"400'42","reported_seq":535,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.066623+0000","last_change":"2024-09-10T18:40:50.663742+0000","last_active":"2024-09-10T18:42:35.066623+0000","last_peered":"2024-09-10T18:42:35.066623+0000","last_clean":"2024-09-10T18:42:35.066623+0000","last_became_active":"2024-09-10T18:39:25.249915+0000","last_became_peered":"2024-09-10T18:39:25.249915+0000","last_unstale":"2024-09-10T18:42:35.066623+0000","last_undegraded":"2024-09-10T18:42:35.066623+0000","last_fullsized":"2024-09-10T18:42:35.066623+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-09-10T18:40:50.663648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:40:50.663648+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012419997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":451,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058705+0000","last_change":"2024-09-10T18:41:35.927468+0000","last_active":"2024-09-10T18:42:35.058705+0000","last_peered":"2024-09-10T18:42:35.058705+0000","last_clean":"2024-09-10T18:42:35.058705+0000","last_became_active":"2024-09-10T18:39:25.236559+0000","last_became_peered":"2024-09-10T18:39:25.236559+0000","last_unstale":"2024-09-10T18:42:35.058705+0000","last_undegraded":"2024-09-10T18:42:35.058705+0000","last_fullsized":"2024-09-10T18:42:35.058705+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:35.927388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:15.765392+0000","last_clean_scrub_stamp":"2024-09-10T18:41:35.927388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:03.513063+0000","scrub_duration":0.004755708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":444,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.057498+0000","last_change":"2024-09-10T18:41:02.057423+0000","last_active":"2024-09-10T18:42:35.057498+0000","last_peered":"2024-09-10T18:42:35.057498+0000","last_clean":"2024-09-10T18:42:35.057498+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:42:35.057498+0000","last_undegraded":"2024-09-10T18:42:35.057498+0000","last_fullsized":"2024-09-10T18:42:35.057498+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:02.057368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:41:02.057368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069129869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":442,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:41.283116+0000","last_change":"2024-09-10T18:42:41.283056+0000","last_active":"2024-09-10T18:42:41.283116+0000","last_peered":"2024-09-10T18:42:41.283116+0000","last_clean":"2024-09-10T18:42:41.283116+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:42:41.283116+0000","last_undegraded":"2024-09-10T18:42:41.283116+0000","last_fullsized":"2024-09-10T18:42:41.283116+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:41.282974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:41.282974+0000","last_clean_scrub_stamp":"2024-09-10T18:42:41.282974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:49.635072+0000","scrub_duration":0.021644562999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"410'14416","reported_seq":14691,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:50.085233+0000","last_change":"2024-09-10T18:42:28.301120+0000","last_active":"2024-09-10T18:42:50.085233+0000","last_peered":"2024-09-10T18:42:50.085233+0000","last_clean":"2024-09-10T18:42:50.085233+0000","last_became_active":"2024-09-10T18:40:05.802616+0000","last_became_peered":"2024-09-10T18:40:05.802616+0000","last_unstale":"2024-09-10T18:42:50.085233+0000","last_undegraded":"2024-09-10T18:42:50.085233+0000","last_fullsized":"2024-09-10T18:42:50.085233+0000","mapping_epoch":374,"log_start":"406'10700","ondisk_log_start":"406'10700","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12429","last_scrub_stamp":"2024-09-10T18:42:28.301066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:42:28.301066+0000","objects_scrubbed":5545,"log_size":3716,"log_dups_size":0,"ondisk_log_size":3716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:48.707890+0000","scrub_duration":2.1063137100000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226885632,"num_objects":3462,"num_object_clones":0,"num_object_copies":10386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14416,"num_write_kb":572096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":376,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058067+0000","last_change":"2024-09-10T18:41:27.793561+0000","last_active":"2024-09-10T18:42:35.058067+0000","last_peered":"2024-09-10T18:42:35.058067+0000","last_clean":"2024-09-10T18:42:35.058067+0000","last_became_active":"2024-09-10T18:39:00.894864+0000","last_became_peered":"2024-09-10T18:39:00.894864+0000","last_unstale":"2024-09-10T18:42:35.058067+0000","last_undegraded":"2024-09-10T18:42:35.058067+0000","last_fullsized":"2024-09-10T18:42:35.058067+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:27.793483+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:27.793483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:44.627147+0000","scrub_duration":0.0048667299999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"400'42","reported_seq":568,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.057617+0000","last_change":"2024-09-10T18:41:43.020411+0000","last_active":"2024-09-10T18:42:35.057617+0000","last_peered":"2024-09-10T18:42:35.057617+0000","last_clean":"2024-09-10T18:42:35.057617+0000","last_became_active":"2024-09-10T18:39:00.903341+0000","last_became_peered":"2024-09-10T18:39:00.903341+0000","last_unstale":"2024-09-10T18:42:35.057617+0000","last_undegraded":"2024-09-10T18:42:35.057617+0000","last_fullsized":"2024-09-10T18:42:35.057617+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'42","last_scrub_stamp":"2024-09-10T18:41:43.020357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:43.020357+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:11.106916+0000","scrub_duration":0.0024664719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":72,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.132545+0000","last_change":"2024-09-10T18:41:44.493766+0000","last_active":"2024-09-10T18:42:42.132545+0000","last_peered":"2024-09-10T18:42:42.132545+0000","last_clean":"2024-09-10T18:42:42.132545+0000","last_became_active":"2024-09-10T18:39:25.238382+0000","last_became_peered":"2024-09-10T18:39:25.238382+0000","last_unstale":"2024-09-10T18:42:42.132545+0000","last_undegraded":"2024-09-10T18:42:42.132545+0000","last_fullsized":"2024-09-10T18:42:42.132545+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:44.493678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:41:44.493678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:03.841563+0000","scrub_duration":0.075205465999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":701,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129695+0000","last_change":"2024-09-10T18:42:24.132664+0000","last_active":"2024-09-10T18:42:42.129695+0000","last_peered":"2024-09-10T18:42:42.129695+0000","last_clean":"2024-09-10T18:42:42.129695+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:42:42.129695+0000","last_undegraded":"2024-09-10T18:42:42.129695+0000","last_fullsized":"2024-09-10T18:42:42.129695+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:42:24.132610+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:42:24.132610+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:51.496133+0000","scrub_duration":0.001673162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":453,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.130289+0000","last_change":"2024-09-10T18:42:24.166552+0000","last_active":"2024-09-10T18:42:42.130289+0000","last_peered":"2024-09-10T18:42:42.130289+0000","last_clean":"2024-09-10T18:42:42.130289+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:42:42.130289+0000","last_undegraded":"2024-09-10T18:42:42.130289+0000","last_fullsized":"2024-09-10T18:42:42.130289+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:24.166477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:42:24.166477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:50.908290+0000","scrub_duration":0.039088676000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"411'14305","reported_seq":14691,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-10T18:42:50.850208+0000","last_change":"2024-09-10T18:41:42.164909+0000","last_active":"2024-09-10T18:42:50.850208+0000","last_peered":"2024-09-10T18:42:50.850208+0000","last_clean":"2024-09-10T18:42:50.850208+0000","last_became_active":"2024-09-10T18:40:05.804737+0000","last_became_peered":"2024-09-10T18:40:05.804737+0000","last_unstale":"2024-09-10T18:42:50.850208+0000","last_undegraded":"2024-09-10T18:42:50.850208+0000","last_fullsized":"2024-09-10T18:42:50.850208+0000","mapping_epoch":374,"log_start":"404'9737","ondisk_log_start":"404'9737","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'8392","last_scrub_stamp":"2024-09-10T18:41:42.164302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:41:42.164302+0000","objects_scrubbed":8291,"log_size":4568,"log_dups_size":0,"ondisk_log_size":4568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.0633734330000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217579544,"num_objects":3321,"num_object_clones":0,"num_object_copies":9963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3321,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14305,"num_write_kb":563969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":466,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:48.651360+0000","last_change":"2024-09-10T18:42:48.651294+0000","last_active":"2024-09-10T18:42:48.651360+0000","last_peered":"2024-09-10T18:42:48.651360+0000","last_clean":"2024-09-10T18:42:48.651360+0000","last_became_active":"2024-09-10T18:34:32.818323+0000","last_became_peered":"2024-09-10T18:34:32.818323+0000","last_unstale":"2024-09-10T18:42:48.651360+0000","last_undegraded":"2024-09-10T18:42:48.651360+0000","last_fullsized":"2024-09-10T18:42:48.651360+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:48.651210+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:48.651210+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:51.416433+0000","scrub_duration":0.010854358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"400'35","reported_seq":562,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.127548+0000","last_change":"2024-09-10T18:41:31.283660+0000","last_active":"2024-09-10T18:42:42.127548+0000","last_peered":"2024-09-10T18:42:42.127548+0000","last_clean":"2024-09-10T18:42:42.127548+0000","last_became_active":"2024-09-10T18:39:00.897648+0000","last_became_peered":"2024-09-10T18:39:00.897648+0000","last_unstale":"2024-09-10T18:42:42.127548+0000","last_undegraded":"2024-09-10T18:42:42.127548+0000","last_fullsized":"2024-09-10T18:42:42.127548+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:41:31.283526+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:41:31.283526+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028902120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":83,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.123615+0000","last_change":"2024-09-10T18:42:24.171396+0000","last_active":"2024-09-10T18:42:42.123615+0000","last_peered":"2024-09-10T18:42:42.123615+0000","last_clean":"2024-09-10T18:42:42.123615+0000","last_became_active":"2024-09-10T18:39:00.909066+0000","last_became_peered":"2024-09-10T18:39:00.909066+0000","last_unstale":"2024-09-10T18:42:42.123615+0000","last_undegraded":"2024-09-10T18:42:42.123615+0000","last_fullsized":"2024-09-10T18:42:42.123615+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:24.171307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:42:24.171307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:36.510048+0000","scrub_duration":0.0076532170000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"362'14","reported_seq":547,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:50.014625+0000","last_change":"2024-09-10T18:42:41.670663+0000","last_active":"2024-09-10T18:42:50.014625+0000","last_peered":"2024-09-10T18:42:50.014625+0000","last_clean":"2024-09-10T18:42:50.014625+0000","last_became_active":"2024-09-10T18:39:25.251198+0000","last_became_peered":"2024-09-10T18:39:25.251198+0000","last_unstale":"2024-09-10T18:42:50.014625+0000","last_undegraded":"2024-09-10T18:42:50.014625+0000","last_fullsized":"2024-09-10T18:42:50.014625+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'14","last_scrub_stamp":"2024-09-10T18:42:41.670583+0000","last_deep_scrub":"263'11","last_deep_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_clean_scrub_stamp":"2024-09-10T18:42:41.670583+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:56.976531+0000","scrub_duration":0.0030905279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":71,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.059059+0000","last_change":"2024-09-10T18:40:49.659503+0000","last_active":"2024-09-10T18:42:35.059059+0000","last_peered":"2024-09-10T18:42:35.059059+0000","last_clean":"2024-09-10T18:42:35.059059+0000","last_became_active":"2024-09-10T18:39:25.235363+0000","last_became_peered":"2024-09-10T18:39:25.235363+0000","last_unstale":"2024-09-10T18:42:35.059059+0000","last_undegraded":"2024-09-10T18:42:35.059059+0000","last_fullsized":"2024-09-10T18:42:35.059059+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:49.659414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:49.659414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002174053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"411'14024","reported_seq":14212,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-10T18:42:50.743813+0000","last_change":"2024-09-10T18:42:50.314315+0000","last_active":"2024-09-10T18:42:50.743813+0000","last_peered":"2024-09-10T18:42:50.743813+0000","last_clean":"2024-09-10T18:42:50.743813+0000","last_became_active":"2024-09-10T18:40:05.806863+0000","last_became_peered":"2024-09-10T18:40:05.806863+0000","last_unstale":"2024-09-10T18:42:50.743813+0000","last_undegraded":"2024-09-10T18:42:50.743813+0000","last_fullsized":"2024-09-10T18:42:50.743813+0000","mapping_epoch":374,"log_start":"406'10400","ondisk_log_start":"406'10400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"411'13988","last_scrub_stamp":"2024-09-10T18:42:50.314234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:42:50.314234+0000","objects_scrubbed":3333,"log_size":3624,"log_dups_size":0,"ondisk_log_size":3624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:57.334021+0000","scrub_duration":1.075252565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213123072,"num_objects":3252,"num_object_clones":0,"num_object_copies":9756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14024,"num_write_kb":552832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":466,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.130768+0000","last_change":"2024-09-10T18:42:25.542017+0000","last_active":"2024-09-10T18:42:42.130768+0000","last_peered":"2024-09-10T18:42:42.130768+0000","last_clean":"2024-09-10T18:42:42.130768+0000","last_became_active":"2024-09-10T18:39:25.249543+0000","last_became_peered":"2024-09-10T18:39:25.249543+0000","last_unstale":"2024-09-10T18:42:42.130768+0000","last_undegraded":"2024-09-10T18:42:42.130768+0000","last_fullsized":"2024-09-10T18:42:42.130768+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:25.541937+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:25.541937+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:50.751252+0000","scrub_duration":0.0051482589999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"400'39","reported_seq":579,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.118317+0000","last_change":"2024-09-10T18:41:47.946062+0000","last_active":"2024-09-10T18:42:42.118317+0000","last_peered":"2024-09-10T18:42:42.118317+0000","last_clean":"2024-09-10T18:42:42.118317+0000","last_became_active":"2024-09-10T18:39:25.256454+0000","last_became_peered":"2024-09-10T18:39:25.256454+0000","last_unstale":"2024-09-10T18:42:42.118317+0000","last_undegraded":"2024-09-10T18:42:42.118317+0000","last_fullsized":"2024-09-10T18:42:42.118317+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'39","last_scrub_stamp":"2024-09-10T18:41:47.945963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:47.945963+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:08.637996+0000","scrub_duration":0.002952414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":470,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.128014+0000","last_change":"2024-09-10T18:40:58.166571+0000","last_active":"2024-09-10T18:42:42.128014+0000","last_peered":"2024-09-10T18:42:42.128014+0000","last_clean":"2024-09-10T18:42:42.128014+0000","last_became_active":"2024-09-10T18:39:00.897360+0000","last_became_peered":"2024-09-10T18:39:00.897360+0000","last_unstale":"2024-09-10T18:42:42.128014+0000","last_undegraded":"2024-09-10T18:42:42.128014+0000","last_fullsized":"2024-09-10T18:42:42.128014+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:58.166192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:57.001003+0000","last_clean_scrub_stamp":"2024-09-10T18:40:58.166192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021571089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":466,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:43.761828+0000","last_change":"2024-09-10T18:42:43.761768+0000","last_active":"2024-09-10T18:42:43.761828+0000","last_peered":"2024-09-10T18:42:43.761828+0000","last_clean":"2024-09-10T18:42:43.761828+0000","last_became_active":"2024-09-10T18:39:25.243348+0000","last_became_peered":"2024-09-10T18:39:25.243348+0000","last_unstale":"2024-09-10T18:42:43.761828+0000","last_undegraded":"2024-09-10T18:42:43.761828+0000","last_fullsized":"2024-09-10T18:42:43.761828+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:43.761663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:43.761663+0000","last_clean_scrub_stamp":"2024-09-10T18:42:43.761663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:48.359894+0000","scrub_duration":0.070622993999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":451,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.133198+0000","last_change":"2024-09-10T18:40:44.261360+0000","last_active":"2024-09-10T18:42:42.133198+0000","last_peered":"2024-09-10T18:42:42.133198+0000","last_clean":"2024-09-10T18:42:42.133198+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:42:42.133198+0000","last_undegraded":"2024-09-10T18:42:42.133198+0000","last_fullsized":"2024-09-10T18:42:42.133198+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:44.261278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:40:44.261278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067495589999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"410'14278","reported_seq":14322,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:49.914337+0000","last_change":"2024-09-10T18:40:05.803024+0000","last_active":"2024-09-10T18:42:49.914337+0000","last_peered":"2024-09-10T18:42:49.914337+0000","last_clean":"2024-09-10T18:42:49.914337+0000","last_became_active":"2024-09-10T18:40:05.802723+0000","last_became_peered":"2024-09-10T18:40:05.802723+0000","last_unstale":"2024-09-10T18:42:49.914337+0000","last_undegraded":"2024-09-10T18:42:49.914337+0000","last_fullsized":"2024-09-10T18:42:49.914337+0000","mapping_epoch":374,"log_start":"405'10600","ondisk_log_start":"405'10600","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":0,"log_size":3678,"log_dups_size":0,"ondisk_log_size":3678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225574912,"num_objects":3442,"num_object_clones":0,"num_object_copies":10326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14278,"num_write_kb":567040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":83,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.117914+0000","last_change":"2024-09-10T18:40:56.726808+0000","last_active":"2024-09-10T18:42:42.117914+0000","last_peered":"2024-09-10T18:42:42.117914+0000","last_clean":"2024-09-10T18:42:42.117914+0000","last_became_active":"2024-09-10T18:39:25.246989+0000","last_became_peered":"2024-09-10T18:39:25.246989+0000","last_unstale":"2024-09-10T18:42:42.117914+0000","last_undegraded":"2024-09-10T18:42:42.117914+0000","last_fullsized":"2024-09-10T18:42:42.117914+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:56.726734+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:40:56.726734+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019705249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"410'64","reported_seq":263,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:50.015198+0000","last_change":"2024-09-10T18:42:43.154600+0000","last_active":"2024-09-10T18:42:50.015198+0000","last_peered":"2024-09-10T18:42:50.015198+0000","last_clean":"2024-09-10T18:42:50.015198+0000","last_became_active":"2024-09-10T18:36:12.922665+0000","last_became_peered":"2024-09-10T18:36:12.922665+0000","last_unstale":"2024-09-10T18:42:50.015198+0000","last_undegraded":"2024-09-10T18:42:50.015198+0000","last_fullsized":"2024-09-10T18:42:50.015198+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'64","last_scrub_stamp":"2024-09-10T18:42:43.154380+0000","last_deep_scrub":"410'64","last_deep_scrub_stamp":"2024-09-10T18:42:43.154380+0000","last_clean_scrub_stamp":"2024-09-10T18:42:43.154380+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:52.804641+0000","scrub_duration":0.023655347,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":465,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058098+0000","last_change":"2024-09-10T18:42:27.214322+0000","last_active":"2024-09-10T18:42:35.058098+0000","last_peered":"2024-09-10T18:42:35.058098+0000","last_clean":"2024-09-10T18:42:35.058098+0000","last_became_active":"2024-09-10T18:39:25.242740+0000","last_became_peered":"2024-09-10T18:39:25.242740+0000","last_unstale":"2024-09-10T18:42:35.058098+0000","last_undegraded":"2024-09-10T18:42:35.058098+0000","last_fullsized":"2024-09-10T18:42:35.058098+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:27.214218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.274219+0000","last_clean_scrub_stamp":"2024-09-10T18:42:27.214218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:32.371196+0000","scrub_duration":0.015470144999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":77,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.118522+0000","last_change":"2024-09-10T18:40:50.753840+0000","last_active":"2024-09-10T18:42:42.118522+0000","last_peered":"2024-09-10T18:42:42.118522+0000","last_clean":"2024-09-10T18:42:42.118522+0000","last_became_active":"2024-09-10T18:39:00.909579+0000","last_became_peered":"2024-09-10T18:39:00.909579+0000","last_unstale":"2024-09-10T18:42:42.118522+0000","last_undegraded":"2024-09-10T18:42:42.118522+0000","last_fullsized":"2024-09-10T18:42:42.118522+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:40:50.753767+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:40:50.753767+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096294199999999996,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":72,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.127752+0000","last_change":"2024-09-10T18:40:47.146767+0000","last_active":"2024-09-10T18:42:42.127752+0000","last_peered":"2024-09-10T18:42:42.127752+0000","last_clean":"2024-09-10T18:42:42.127752+0000","last_became_active":"2024-09-10T18:39:25.255658+0000","last_became_peered":"2024-09-10T18:39:25.255658+0000","last_unstale":"2024-09-10T18:42:42.127752+0000","last_undegraded":"2024-09-10T18:42:42.127752+0000","last_fullsized":"2024-09-10T18:42:42.127752+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:47.146673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:40:47.146673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003044597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":428,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.059090+0000","last_change":"2024-09-10T18:40:34.573578+0000","last_active":"2024-09-10T18:42:35.059090+0000","last_peered":"2024-09-10T18:42:35.059090+0000","last_clean":"2024-09-10T18:42:35.059090+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:42:35.059090+0000","last_undegraded":"2024-09-10T18:42:35.059090+0000","last_fullsized":"2024-09-10T18:42:35.059090+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:34.573495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:34.573495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023351689999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"410'14381","reported_seq":14425,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:49.890715+0000","last_change":"2024-09-10T18:40:05.803578+0000","last_active":"2024-09-10T18:42:49.890715+0000","last_peered":"2024-09-10T18:42:49.890715+0000","last_clean":"2024-09-10T18:42:49.890715+0000","last_became_active":"2024-09-10T18:40:05.803269+0000","last_became_peered":"2024-09-10T18:40:05.803269+0000","last_unstale":"2024-09-10T18:42:49.890715+0000","last_undegraded":"2024-09-10T18:42:49.890715+0000","last_fullsized":"2024-09-10T18:42:49.890715+0000","mapping_epoch":374,"log_start":"406'10700","ondisk_log_start":"406'10700","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":0,"log_size":3681,"log_dups_size":0,"ondisk_log_size":3681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217120768,"num_objects":3313,"num_object_clones":0,"num_object_copies":9939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14381,"num_write_kb":566208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":510,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.128684+0000","last_change":"2024-09-10T18:41:33.333442+0000","last_active":"2024-09-10T18:42:42.128684+0000","last_peered":"2024-09-10T18:42:42.128684+0000","last_clean":"2024-09-10T18:42:42.128684+0000","last_became_active":"2024-09-10T18:39:25.240048+0000","last_became_peered":"2024-09-10T18:39:25.240048+0000","last_unstale":"2024-09-10T18:42:42.128684+0000","last_undegraded":"2024-09-10T18:42:42.128684+0000","last_fullsized":"2024-09-10T18:42:42.128684+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:33.333247+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:33.333247+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057228740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"400'26","reported_seq":56,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.134522+0000","last_change":"2024-09-10T18:42:38.544026+0000","last_active":"2024-09-10T18:42:42.134522+0000","last_peered":"2024-09-10T18:42:42.134522+0000","last_clean":"2024-09-10T18:42:42.134522+0000","last_became_active":"2024-09-10T18:41:11.933724+0000","last_became_peered":"2024-09-10T18:41:11.933724+0000","last_unstale":"2024-09-10T18:42:42.134522+0000","last_undegraded":"2024-09-10T18:42:42.134522+0000","last_fullsized":"2024-09-10T18:42:42.134522+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'26","last_scrub_stamp":"2024-09-10T18:42:38.543932+0000","last_deep_scrub":"388'22","last_deep_scrub_stamp":"2024-09-10T18:41:01.248929+0000","last_clean_scrub_stamp":"2024-09-10T18:42:38.543932+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:49.910982+0000","scrub_duration":0.025590214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":477,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:44.171590+0000","last_change":"2024-09-10T18:42:44.171535+0000","last_active":"2024-09-10T18:42:44.171590+0000","last_peered":"2024-09-10T18:42:44.171590+0000","last_clean":"2024-09-10T18:42:44.171590+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:42:44.171590+0000","last_undegraded":"2024-09-10T18:42:44.171590+0000","last_fullsized":"2024-09-10T18:42:44.171590+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:44.171445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:44.171445+0000","last_clean_scrub_stamp":"2024-09-10T18:42:44.171445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:47.474299+0000","scrub_duration":0.015055643,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":511,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.127783+0000","last_change":"2024-09-10T18:42:32.485951+0000","last_active":"2024-09-10T18:42:42.127783+0000","last_peered":"2024-09-10T18:42:42.127783+0000","last_clean":"2024-09-10T18:42:42.127783+0000","last_became_active":"2024-09-10T18:39:25.249566+0000","last_became_peered":"2024-09-10T18:39:25.249566+0000","last_unstale":"2024-09-10T18:42:42.127783+0000","last_undegraded":"2024-09-10T18:42:42.127783+0000","last_fullsized":"2024-09-10T18:42:42.127783+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:42:32.485863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:42:32.485863+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:42.346366+0000","scrub_duration":0.020323474000000001,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":494,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.135192+0000","last_change":"2024-09-10T18:41:29.980154+0000","last_active":"2024-09-10T18:42:42.135192+0000","last_peered":"2024-09-10T18:42:42.135192+0000","last_clean":"2024-09-10T18:42:42.135192+0000","last_became_active":"2024-09-10T18:39:00.910574+0000","last_became_peered":"2024-09-10T18:39:00.910574+0000","last_unstale":"2024-09-10T18:42:42.135192+0000","last_undegraded":"2024-09-10T18:42:42.135192+0000","last_fullsized":"2024-09-10T18:42:42.135192+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:29.980083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:41:29.980083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010417921,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":453,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129603+0000","last_change":"2024-09-10T18:42:36.178847+0000","last_active":"2024-09-10T18:42:42.129603+0000","last_peered":"2024-09-10T18:42:42.129603+0000","last_clean":"2024-09-10T18:42:42.129603+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:42:42.129603+0000","last_undegraded":"2024-09-10T18:42:42.129603+0000","last_fullsized":"2024-09-10T18:42:42.129603+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:36.178756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:42:36.178756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:49.301231+0000","scrub_duration":0.012545664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"410'14224","reported_seq":14268,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:49.925705+0000","last_change":"2024-09-10T18:40:05.805158+0000","last_active":"2024-09-10T18:42:49.925705+0000","last_peered":"2024-09-10T18:42:49.925705+0000","last_clean":"2024-09-10T18:42:49.925705+0000","last_became_active":"2024-09-10T18:40:05.804888+0000","last_became_peered":"2024-09-10T18:40:05.804888+0000","last_unstale":"2024-09-10T18:42:49.925705+0000","last_undegraded":"2024-09-10T18:42:49.925705+0000","last_fullsized":"2024-09-10T18:42:49.925705+0000","mapping_epoch":374,"log_start":"405'10500","ondisk_log_start":"405'10500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":0,"log_size":3724,"log_dups_size":0,"ondisk_log_size":3724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220594176,"num_objects":3366,"num_object_clones":0,"num_object_copies":10098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14224,"num_write_kb":562880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":380,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.130958+0000","last_change":"2024-09-10T18:41:47.469267+0000","last_active":"2024-09-10T18:42:42.130958+0000","last_peered":"2024-09-10T18:42:42.130958+0000","last_clean":"2024-09-10T18:42:42.130958+0000","last_became_active":"2024-09-10T18:34:00.165384+0000","last_became_peered":"2024-09-10T18:34:00.165384+0000","last_unstale":"2024-09-10T18:42:42.130958+0000","last_undegraded":"2024-09-10T18:42:42.130958+0000","last_fullsized":"2024-09-10T18:42:42.130958+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:47.468155+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:47.468155+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:10.750378+0000","scrub_duration":0.0041388789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"400'36","reported_seq":366,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.127998+0000","last_change":"2024-09-10T18:41:32.325494+0000","last_active":"2024-09-10T18:42:42.127998+0000","last_peered":"2024-09-10T18:42:42.127998+0000","last_clean":"2024-09-10T18:42:42.127998+0000","last_became_active":"2024-09-10T18:39:25.263465+0000","last_became_peered":"2024-09-10T18:39:25.263465+0000","last_unstale":"2024-09-10T18:42:42.127998+0000","last_undegraded":"2024-09-10T18:42:42.127998+0000","last_fullsized":"2024-09-10T18:42:42.127998+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-09-10T18:41:32.325283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:32.325283+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:01.049093+0000","scrub_duration":0.0058952759999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":81,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.057715+0000","last_change":"2024-09-10T18:41:49.077680+0000","last_active":"2024-09-10T18:42:35.057715+0000","last_peered":"2024-09-10T18:42:35.057715+0000","last_clean":"2024-09-10T18:42:35.057715+0000","last_became_active":"2024-09-10T18:39:25.239068+0000","last_became_peered":"2024-09-10T18:39:25.239068+0000","last_unstale":"2024-09-10T18:42:35.057715+0000","last_undegraded":"2024-09-10T18:42:35.057715+0000","last_fullsized":"2024-09-10T18:42:35.057715+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:49.077609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:41:49.077609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:53.005098+0000","scrub_duration":0.007378697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":1750,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:49.633443+0000","last_change":"2024-09-10T18:42:43.071764+0000","last_active":"2024-09-10T18:42:49.633443+0000","last_peered":"2024-09-10T18:42:49.633443+0000","last_clean":"2024-09-10T18:42:49.633443+0000","last_became_active":"2024-09-10T18:39:25.245563+0000","last_became_peered":"2024-09-10T18:39:25.245563+0000","last_unstale":"2024-09-10T18:42:49.633443+0000","last_undegraded":"2024-09-10T18:42:49.633443+0000","last_fullsized":"2024-09-10T18:42:49.633443+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:42:43.071674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:42:43.071674+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:49.562883+0000","scrub_duration":0.0074519470000000004,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1269,"num_read_kb":1264,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":83,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.122924+0000","last_change":"2024-09-10T18:41:28.868786+0000","last_active":"2024-09-10T18:42:42.122924+0000","last_peered":"2024-09-10T18:42:42.122924+0000","last_clean":"2024-09-10T18:42:42.122924+0000","last_became_active":"2024-09-10T18:39:25.255881+0000","last_became_peered":"2024-09-10T18:39:25.255881+0000","last_unstale":"2024-09-10T18:42:42.122924+0000","last_undegraded":"2024-09-10T18:42:42.122924+0000","last_fullsized":"2024-09-10T18:42:42.122924+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:28.868225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.170272+0000","last_clean_scrub_stamp":"2024-09-10T18:41:28.868225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:52.177697+0000","scrub_duration":0.0038280509999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":464,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.127564+0000","last_change":"2024-09-10T18:40:39.235217+0000","last_active":"2024-09-10T18:42:42.127564+0000","last_peered":"2024-09-10T18:42:42.127564+0000","last_clean":"2024-09-10T18:42:42.127564+0000","last_became_active":"2024-09-10T18:39:00.897525+0000","last_became_peered":"2024-09-10T18:39:00.897525+0000","last_unstale":"2024-09-10T18:42:42.127564+0000","last_undegraded":"2024-09-10T18:42:42.127564+0000","last_fullsized":"2024-09-10T18:42:42.127564+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:39.235160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:39.235160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025668739999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"410'14052","reported_seq":14096,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:49.894762+0000","last_change":"2024-09-10T18:40:05.804498+0000","last_active":"2024-09-10T18:42:49.894762+0000","last_peered":"2024-09-10T18:42:49.894762+0000","last_clean":"2024-09-10T18:42:49.894762+0000","last_became_active":"2024-09-10T18:40:05.804097+0000","last_became_peered":"2024-09-10T18:40:05.804097+0000","last_unstale":"2024-09-10T18:42:49.894762+0000","last_undegraded":"2024-09-10T18:42:49.894762+0000","last_fullsized":"2024-09-10T18:42:49.894762+0000","mapping_epoch":374,"log_start":"404'10300","ondisk_log_start":"404'10300","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":0,"log_size":3752,"log_dups_size":0,"ondisk_log_size":3752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225837056,"num_objects":3446,"num_object_clones":0,"num_object_copies":10338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14052,"num_write_kb":559936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"400'38","reported_seq":471,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.131738+0000","last_change":"2024-09-10T18:41:35.197885+0000","last_active":"2024-09-10T18:42:42.131738+0000","last_peered":"2024-09-10T18:42:42.131738+0000","last_clean":"2024-09-10T18:42:42.131738+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:42:42.131738+0000","last_undegraded":"2024-09-10T18:42:42.131738+0000","last_fullsized":"2024-09-10T18:42:42.131738+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'26","last_scrub_stamp":"2024-09-10T18:41:35.197788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:35.197788+0000","objects_scrubbed":7,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029282409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":468,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.069007+0000","last_change":"2024-09-10T18:39:48.899653+0000","last_active":"2024-09-10T18:42:35.069007+0000","last_peered":"2024-09-10T18:42:35.069007+0000","last_clean":"2024-09-10T18:42:35.069007+0000","last_became_active":"2024-09-10T18:39:00.894778+0000","last_became_peered":"2024-09-10T18:39:00.894778+0000","last_unstale":"2024-09-10T18:42:35.069007+0000","last_undegraded":"2024-09-10T18:42:35.069007+0000","last_fullsized":"2024-09-10T18:42:35.069007+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:48.899587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:39:48.899587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001397414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":519,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.128867+0000","last_change":"2024-09-10T18:40:38.039120+0000","last_active":"2024-09-10T18:42:42.128867+0000","last_peered":"2024-09-10T18:42:42.128867+0000","last_clean":"2024-09-10T18:42:42.128867+0000","last_became_active":"2024-09-10T18:39:25.251895+0000","last_became_peered":"2024-09-10T18:39:25.251895+0000","last_unstale":"2024-09-10T18:42:42.128867+0000","last_undegraded":"2024-09-10T18:42:42.128867+0000","last_fullsized":"2024-09-10T18:42:42.128867+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:38.038969+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:40:38.038969+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052669589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":468,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129659+0000","last_change":"2024-09-10T18:40:34.971516+0000","last_active":"2024-09-10T18:42:42.129659+0000","last_peered":"2024-09-10T18:42:42.129659+0000","last_clean":"2024-09-10T18:42:42.129659+0000","last_became_active":"2024-09-10T18:39:00.890829+0000","last_became_peered":"2024-09-10T18:39:00.890829+0000","last_unstale":"2024-09-10T18:42:42.129659+0000","last_undegraded":"2024-09-10T18:42:42.129659+0000","last_fullsized":"2024-09-10T18:42:42.129659+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:34.971366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:40:34.971366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040825239999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":73,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.122846+0000","last_change":"2024-09-10T18:41:34.883649+0000","last_active":"2024-09-10T18:42:42.122846+0000","last_peered":"2024-09-10T18:42:42.122846+0000","last_clean":"2024-09-10T18:42:42.122846+0000","last_became_active":"2024-09-10T18:39:00.900703+0000","last_became_peered":"2024-09-10T18:39:00.900703+0000","last_unstale":"2024-09-10T18:42:42.122846+0000","last_undegraded":"2024-09-10T18:42:42.122846+0000","last_fullsized":"2024-09-10T18:42:42.122846+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:34.883463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:41:34.883463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:58.255048+0000","scrub_duration":0.0041636060000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"411'14436","reported_seq":14707,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-10T18:42:50.657134+0000","last_change":"2024-09-10T18:42:29.418325+0000","last_active":"2024-09-10T18:42:50.657134+0000","last_peered":"2024-09-10T18:42:50.657134+0000","last_clean":"2024-09-10T18:42:50.657134+0000","last_became_active":"2024-09-10T18:40:05.807021+0000","last_became_peered":"2024-09-10T18:40:05.807021+0000","last_unstale":"2024-09-10T18:42:50.657134+0000","last_undegraded":"2024-09-10T18:42:50.657134+0000","last_fullsized":"2024-09-10T18:42:50.657134+0000","mapping_epoch":374,"log_start":"404'10000","ondisk_log_start":"404'10000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12600","last_scrub_stamp":"2024-09-10T18:42:29.418266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:42:29.418266+0000","objects_scrubbed":5387,"log_size":4436,"log_dups_size":0,"ondisk_log_size":4436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:37.969432+0000","scrub_duration":1.81290566,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228327424,"num_objects":3484,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14436,"num_write_kb":573440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"76'81","reported_seq":426,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.131132+0000","last_change":"2024-09-10T18:42:41.127739+0000","last_active":"2024-09-10T18:42:42.131132+0000","last_peered":"2024-09-10T18:42:42.131132+0000","last_clean":"2024-09-10T18:42:42.131132+0000","last_became_active":"2024-09-10T18:39:25.263369+0000","last_became_peered":"2024-09-10T18:39:25.263369+0000","last_unstale":"2024-09-10T18:42:42.131132+0000","last_undegraded":"2024-09-10T18:42:42.131132+0000","last_fullsized":"2024-09-10T18:42:42.131132+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:41.127360+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:41.127360+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:56.780587+0000","scrub_duration":0.0078382469999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"400'32","reported_seq":504,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058463+0000","last_change":"2024-09-10T18:41:46.113019+0000","last_active":"2024-09-10T18:42:35.058463+0000","last_peered":"2024-09-10T18:42:35.058463+0000","last_clean":"2024-09-10T18:42:35.058463+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:42:35.058463+0000","last_undegraded":"2024-09-10T18:42:35.058463+0000","last_fullsized":"2024-09-10T18:42:35.058463+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'32","last_scrub_stamp":"2024-09-10T18:41:46.112954+0000","last_deep_scrub":"283'24","last_deep_scrub_stamp":"2024-09-10T18:34:19.404713+0000","last_clean_scrub_stamp":"2024-09-10T18:41:46.112954+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:49.602516+0000","scrub_duration":0.097283912,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":451,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.057709+0000","last_change":"2024-09-10T18:41:44.986845+0000","last_active":"2024-09-10T18:42:35.057709+0000","last_peered":"2024-09-10T18:42:35.057709+0000","last_clean":"2024-09-10T18:42:35.057709+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:42:35.057709+0000","last_undegraded":"2024-09-10T18:42:35.057709+0000","last_fullsized":"2024-09-10T18:42:35.057709+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:44.986765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:41:44.986765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:09.455315+0000","scrub_duration":0.0079498090000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":83,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:45.223940+0000","last_change":"2024-09-10T18:42:45.223626+0000","last_active":"2024-09-10T18:42:45.223940+0000","last_peered":"2024-09-10T18:42:45.223940+0000","last_clean":"2024-09-10T18:42:45.223940+0000","last_became_active":"2024-09-10T18:39:00.907231+0000","last_became_peered":"2024-09-10T18:39:00.907231+0000","last_unstale":"2024-09-10T18:42:45.223940+0000","last_undegraded":"2024-09-10T18:42:45.223940+0000","last_fullsized":"2024-09-10T18:42:45.223940+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:45.223551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:42:45.223551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:51.395703+0000","scrub_duration":0.0026937710000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":462,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058155+0000","last_change":"2024-09-10T18:39:41.925032+0000","last_active":"2024-09-10T18:42:35.058155+0000","last_peered":"2024-09-10T18:42:35.058155+0000","last_clean":"2024-09-10T18:42:35.058155+0000","last_became_active":"2024-09-10T18:39:00.894963+0000","last_became_peered":"2024-09-10T18:39:00.894963+0000","last_unstale":"2024-09-10T18:42:35.058155+0000","last_undegraded":"2024-09-10T18:42:35.058155+0000","last_fullsized":"2024-09-10T18:42:35.058155+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:41.924977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:39:41.924977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011688601999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"411'14025","reported_seq":14389,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-10T18:42:50.831567+0000","last_change":"2024-09-10T18:41:36.523759+0000","last_active":"2024-09-10T18:42:50.831567+0000","last_peered":"2024-09-10T18:42:50.831567+0000","last_clean":"2024-09-10T18:42:50.831567+0000","last_became_active":"2024-09-10T18:40:05.803613+0000","last_became_peered":"2024-09-10T18:40:05.803613+0000","last_unstale":"2024-09-10T18:42:50.831567+0000","last_undegraded":"2024-09-10T18:42:50.831567+0000","last_fullsized":"2024-09-10T18:42:50.831567+0000","mapping_epoch":374,"log_start":"404'9404","ondisk_log_start":"404'9404","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'7829","last_scrub_stamp":"2024-09-10T18:41:36.523427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:41:36.523427+0000","objects_scrubbed":7693,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.5398984680000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220659712,"num_objects":3367,"num_object_clones":0,"num_object_copies":10101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14025,"num_write_kb":556544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":83,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:49.285727+0000","last_change":"2024-09-10T18:42:49.285431+0000","last_active":"2024-09-10T18:42:49.285727+0000","last_peered":"2024-09-10T18:42:49.285727+0000","last_clean":"2024-09-10T18:42:49.285727+0000","last_became_active":"2024-09-10T18:39:00.909151+0000","last_became_peered":"2024-09-10T18:39:00.909151+0000","last_unstale":"2024-09-10T18:42:49.285727+0000","last_undegraded":"2024-09-10T18:42:49.285727+0000","last_fullsized":"2024-09-10T18:42:49.285727+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:49.285168+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:49.285168+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:52.101220+0000","scrub_duration":0.0068966280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"400'53","reported_seq":580,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058299+0000","last_change":"2024-09-10T18:41:28.183160+0000","last_active":"2024-09-10T18:42:35.058299+0000","last_peered":"2024-09-10T18:42:35.058299+0000","last_clean":"2024-09-10T18:42:35.058299+0000","last_became_active":"2024-09-10T18:38:43.333186+0000","last_became_peered":"2024-09-10T18:38:43.333186+0000","last_unstale":"2024-09-10T18:42:35.058299+0000","last_undegraded":"2024-09-10T18:42:35.058299+0000","last_fullsized":"2024-09-10T18:42:35.058299+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'39","last_scrub_stamp":"2024-09-10T18:41:28.183073+0000","last_deep_scrub":"342'39","last_deep_scrub_stamp":"2024-09-10T18:37:53.386563+0000","last_clean_scrub_stamp":"2024-09-10T18:41:28.183073+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:50.451880+0000","scrub_duration":0.067628692000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":434,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.710062+0000","last_change":"2024-09-10T18:42:42.709960+0000","last_active":"2024-09-10T18:42:42.710062+0000","last_peered":"2024-09-10T18:42:42.710062+0000","last_clean":"2024-09-10T18:42:42.710062+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:42:42.710062+0000","last_undegraded":"2024-09-10T18:42:42.710062+0000","last_fullsized":"2024-09-10T18:42:42.710062+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:42:42.709888+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:42:42.709888+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:11.381192+0000","scrub_duration":0.011345328999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":466,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.127637+0000","last_change":"2024-09-10T18:41:35.323346+0000","last_active":"2024-09-10T18:42:42.127637+0000","last_peered":"2024-09-10T18:42:42.127637+0000","last_clean":"2024-09-10T18:42:42.127637+0000","last_became_active":"2024-09-10T18:39:25.249337+0000","last_became_peered":"2024-09-10T18:39:25.249337+0000","last_unstale":"2024-09-10T18:42:42.127637+0000","last_undegraded":"2024-09-10T18:42:42.127637+0000","last_fullsized":"2024-09-10T18:42:42.127637+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:35.323254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:41:35.323254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:59.713569+0000","scrub_duration":0.001933265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":492,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:45.518361+0000","last_change":"2024-09-10T18:42:45.518178+0000","last_active":"2024-09-10T18:42:45.518361+0000","last_peered":"2024-09-10T18:42:45.518361+0000","last_clean":"2024-09-10T18:42:45.518361+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:42:45.518361+0000","last_undegraded":"2024-09-10T18:42:45.518361+0000","last_fullsized":"2024-09-10T18:42:45.518361+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:45.517976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:42:45.517976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:00.331373+0000","scrub_duration":0.019939069,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"411'14163","reported_seq":14530,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-10T18:42:50.624568+0000","last_change":"2024-09-10T18:42:26.520158+0000","last_active":"2024-09-10T18:42:50.624568+0000","last_peered":"2024-09-10T18:42:50.624568+0000","last_clean":"2024-09-10T18:42:50.624568+0000","last_became_active":"2024-09-10T18:40:05.801520+0000","last_became_peered":"2024-09-10T18:40:05.801520+0000","last_unstale":"2024-09-10T18:42:50.624568+0000","last_undegraded":"2024-09-10T18:42:50.624568+0000","last_fullsized":"2024-09-10T18:42:50.624568+0000","mapping_epoch":374,"log_start":"404'9700","ondisk_log_start":"404'9700","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12079","last_scrub_stamp":"2024-09-10T18:42:26.520073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:42:26.520073+0000","objects_scrubbed":5557,"log_size":4463,"log_dups_size":0,"ondisk_log_size":4463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:30.013995+0000","scrub_duration":2.035373323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221577216,"num_objects":3381,"num_object_clones":0,"num_object_copies":10143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14163,"num_write_kb":561408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":507,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129035+0000","last_change":"2024-09-10T18:41:43.367065+0000","last_active":"2024-09-10T18:42:42.129035+0000","last_peered":"2024-09-10T18:42:42.129035+0000","last_clean":"2024-09-10T18:42:42.129035+0000","last_became_active":"2024-09-10T18:34:32.827174+0000","last_became_peered":"2024-09-10T18:34:32.827174+0000","last_unstale":"2024-09-10T18:42:42.129035+0000","last_undegraded":"2024-09-10T18:42:42.129035+0000","last_fullsized":"2024-09-10T18:42:42.129035+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:43.366981+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:43.366981+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:56.226795+0000","scrub_duration":0.0018640970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"400'35","reported_seq":222,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.128867+0000","last_change":"2024-09-10T18:40:44.058922+0000","last_active":"2024-09-10T18:42:42.128867+0000","last_peered":"2024-09-10T18:42:42.128867+0000","last_clean":"2024-09-10T18:42:42.128867+0000","last_became_active":"2024-09-10T18:32:51.118251+0000","last_became_peered":"2024-09-10T18:32:51.118251+0000","last_unstale":"2024-09-10T18:42:42.128867+0000","last_undegraded":"2024-09-10T18:42:42.128867+0000","last_fullsized":"2024-09-10T18:42:42.128867+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:40:44.058799+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:40:44.058799+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016784752999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":451,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.066789+0000","last_change":"2024-09-10T18:40:59.757340+0000","last_active":"2024-09-10T18:42:35.066789+0000","last_peered":"2024-09-10T18:42:35.066789+0000","last_clean":"2024-09-10T18:42:35.066789+0000","last_became_active":"2024-09-10T18:39:25.251419+0000","last_became_peered":"2024-09-10T18:39:25.251419+0000","last_unstale":"2024-09-10T18:42:35.066789+0000","last_undegraded":"2024-09-10T18:42:35.066789+0000","last_fullsized":"2024-09-10T18:42:35.066789+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:59.757255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:59.757255+0000","last_clean_scrub_stamp":"2024-09-10T18:40:59.757255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056131599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"357'11","reported_seq":788,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:50.014765+0000","last_change":"2024-09-10T18:42:30.463021+0000","last_active":"2024-09-10T18:42:50.014765+0000","last_peered":"2024-09-10T18:42:50.014765+0000","last_clean":"2024-09-10T18:42:50.014765+0000","last_became_active":"2024-09-10T18:39:00.909747+0000","last_became_peered":"2024-09-10T18:39:00.909747+0000","last_unstale":"2024-09-10T18:42:50.014765+0000","last_undegraded":"2024-09-10T18:42:50.014765+0000","last_fullsized":"2024-09-10T18:42:50.014765+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'11","last_scrub_stamp":"2024-09-10T18:42:30.462929+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:42:30.462929+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:56.745815+0000","scrub_duration":0.0039261499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":435,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058315+0000","last_change":"2024-09-10T18:40:53.988053+0000","last_active":"2024-09-10T18:42:35.058315+0000","last_peered":"2024-09-10T18:42:35.058315+0000","last_clean":"2024-09-10T18:42:35.058315+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:42:35.058315+0000","last_undegraded":"2024-09-10T18:42:35.058315+0000","last_fullsized":"2024-09-10T18:42:35.058315+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:53.987811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:53.987811+0000","last_clean_scrub_stamp":"2024-09-10T18:40:53.987811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0078878390000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"411'14187","reported_seq":14415,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-10T18:42:50.163198+0000","last_change":"2024-09-10T18:42:38.523136+0000","last_active":"2024-09-10T18:42:50.163198+0000","last_peered":"2024-09-10T18:42:50.163198+0000","last_clean":"2024-09-10T18:42:50.163198+0000","last_became_active":"2024-09-10T18:40:05.805621+0000","last_became_peered":"2024-09-10T18:40:05.805621+0000","last_unstale":"2024-09-10T18:42:50.163198+0000","last_undegraded":"2024-09-10T18:42:50.163198+0000","last_fullsized":"2024-09-10T18:42:50.163198+0000","mapping_epoch":374,"log_start":"405'10500","ondisk_log_start":"405'10500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'13300","last_scrub_stamp":"2024-09-10T18:42:38.523050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:42:38.523050+0000","objects_scrubbed":4335,"log_size":3687,"log_dups_size":0,"ondisk_log_size":3687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:47.124296+0000","scrub_duration":1.356315282,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222232576,"num_objects":3391,"num_object_clones":0,"num_object_copies":10173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14187,"num_write_kb":562496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"76'81","reported_seq":178,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058288+0000","last_change":"2024-09-10T18:41:00.014263+0000","last_active":"2024-09-10T18:42:35.058288+0000","last_peered":"2024-09-10T18:42:35.058288+0000","last_clean":"2024-09-10T18:42:35.058288+0000","last_became_active":"2024-09-10T18:34:32.818251+0000","last_became_peered":"2024-09-10T18:34:32.818251+0000","last_unstale":"2024-09-10T18:42:35.058288+0000","last_undegraded":"2024-09-10T18:42:35.058288+0000","last_fullsized":"2024-09-10T18:42:35.058288+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:00.014178+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:00.014178+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.024934580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"400'25","reported_seq":430,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.131048+0000","last_change":"2024-09-10T18:42:23.354906+0000","last_active":"2024-09-10T18:42:42.131048+0000","last_peered":"2024-09-10T18:42:42.131048+0000","last_clean":"2024-09-10T18:42:42.131048+0000","last_became_active":"2024-09-10T18:39:25.240389+0000","last_became_peered":"2024-09-10T18:39:25.240389+0000","last_unstale":"2024-09-10T18:42:42.131048+0000","last_undegraded":"2024-09-10T18:42:42.131048+0000","last_fullsized":"2024-09-10T18:42:42.131048+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'25","last_scrub_stamp":"2024-09-10T18:42:23.354801+0000","last_deep_scrub":"302'19","last_deep_scrub_stamp":"2024-09-10T18:35:18.836131+0000","last_clean_scrub_stamp":"2024-09-10T18:42:23.354801+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:34.078912+0000","scrub_duration":0.0094838619999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"265'2","reported_seq":87,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.118972+0000","last_change":"2024-09-10T18:42:34.273635+0000","last_active":"2024-09-10T18:42:42.118972+0000","last_peered":"2024-09-10T18:42:42.118972+0000","last_clean":"2024-09-10T18:42:42.118972+0000","last_became_active":"2024-09-10T18:39:00.904967+0000","last_became_peered":"2024-09-10T18:39:00.904967+0000","last_unstale":"2024-09-10T18:42:42.118972+0000","last_undegraded":"2024-09-10T18:42:42.118972+0000","last_fullsized":"2024-09-10T18:42:42.118972+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'2","last_scrub_stamp":"2024-09-10T18:42:34.273553+0000","last_deep_scrub":"265'2","last_deep_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_clean_scrub_stamp":"2024-09-10T18:42:34.273553+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:56.756564+0000","scrub_duration":0.0028652539999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":480,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129918+0000","last_change":"2024-09-10T18:42:28.510786+0000","last_active":"2024-09-10T18:42:42.129918+0000","last_peered":"2024-09-10T18:42:42.129918+0000","last_clean":"2024-09-10T18:42:42.129918+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:42:42.129918+0000","last_undegraded":"2024-09-10T18:42:42.129918+0000","last_fullsized":"2024-09-10T18:42:42.129918+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:28.510725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:42:28.510725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:49.810546+0000","scrub_duration":0.0053696639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":72,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.128847+0000","last_change":"2024-09-10T18:41:35.361220+0000","last_active":"2024-09-10T18:42:42.128847+0000","last_peered":"2024-09-10T18:42:42.128847+0000","last_clean":"2024-09-10T18:42:42.128847+0000","last_became_active":"2024-09-10T18:39:25.237150+0000","last_became_peered":"2024-09-10T18:39:25.237150+0000","last_unstale":"2024-09-10T18:42:42.128847+0000","last_undegraded":"2024-09-10T18:42:42.128847+0000","last_fullsized":"2024-09-10T18:42:42.128847+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:35.361130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:41:35.361130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023350010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"411'15536","reported_seq":15639,"reported_epoch":411,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-10T18:42:50.229983+0000","last_change":"2024-09-10T18:42:45.094903+0000","last_active":"2024-09-10T18:42:50.229983+0000","last_peered":"2024-09-10T18:42:50.229983+0000","last_clean":"2024-09-10T18:42:50.229983+0000","last_became_active":"2024-09-10T18:40:05.806063+0000","last_became_peered":"2024-09-10T18:40:05.806063+0000","last_unstale":"2024-09-10T18:42:50.229983+0000","last_undegraded":"2024-09-10T18:42:50.229983+0000","last_fullsized":"2024-09-10T18:42:50.229983+0000","mapping_epoch":374,"log_start":"407'11801","ondisk_log_start":"407'11801","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:40:04.783635+0000","objects_scrubbed":1317,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 5s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217776128,"num_objects":3323,"num_object_clones":0,"num_object_copies":9969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14219,"num_write_kb":561344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"76'81","reported_seq":465,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.131039+0000","last_change":"2024-09-10T18:42:31.570834+0000","last_active":"2024-09-10T18:42:42.131039+0000","last_peered":"2024-09-10T18:42:42.131039+0000","last_clean":"2024-09-10T18:42:42.131039+0000","last_became_active":"2024-09-10T18:39:00.921868+0000","last_became_peered":"2024-09-10T18:39:00.921868+0000","last_unstale":"2024-09-10T18:42:42.131039+0000","last_undegraded":"2024-09-10T18:42:42.131039+0000","last_fullsized":"2024-09-10T18:42:42.131039+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:31.570752+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:34:15.938079+0000","last_clean_scrub_stamp":"2024-09-10T18:42:31.570752+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:37.950651+0000","scrub_duration":0.00177728,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"400'60","reported_seq":575,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.130993+0000","last_change":"2024-09-10T18:42:37.592570+0000","last_active":"2024-09-10T18:42:42.130993+0000","last_peered":"2024-09-10T18:42:42.130993+0000","last_clean":"2024-09-10T18:42:42.130993+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:42:42.130993+0000","last_undegraded":"2024-09-10T18:42:42.130993+0000","last_fullsized":"2024-09-10T18:42:42.130993+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'60","last_scrub_stamp":"2024-09-10T18:42:37.592486+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:42:37.592486+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:40.264071+0000","scrub_duration":0.0082102519999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":489,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:50.123776+0000","last_change":"2024-09-10T18:42:50.123722+0000","last_active":"2024-09-10T18:42:50.123776+0000","last_peered":"2024-09-10T18:42:50.123776+0000","last_clean":"2024-09-10T18:42:50.123776+0000","last_became_active":"2024-09-10T18:39:00.895121+0000","last_became_peered":"2024-09-10T18:39:00.895121+0000","last_unstale":"2024-09-10T18:42:50.123776+0000","last_undegraded":"2024-09-10T18:42:50.123776+0000","last_fullsized":"2024-09-10T18:42:50.123776+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:50.123643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:42:50.123643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:00.402908+0000","scrub_duration":0.0020317809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":514,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.131404+0000","last_change":"2024-09-10T18:41:32.022773+0000","last_active":"2024-09-10T18:42:42.131404+0000","last_peered":"2024-09-10T18:42:42.131404+0000","last_clean":"2024-09-10T18:42:42.131404+0000","last_became_active":"2024-09-10T18:39:25.240291+0000","last_became_peered":"2024-09-10T18:39:25.240291+0000","last_unstale":"2024-09-10T18:42:42.131404+0000","last_undegraded":"2024-09-10T18:42:42.131404+0000","last_fullsized":"2024-09-10T18:42:42.131404+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:32.022685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:34:21.447291+0000","last_clean_scrub_stamp":"2024-09-10T18:41:32.022685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069142700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":456,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.128786+0000","last_change":"2024-09-10T18:42:28.075463+0000","last_active":"2024-09-10T18:42:42.128786+0000","last_peered":"2024-09-10T18:42:42.128786+0000","last_clean":"2024-09-10T18:42:42.128786+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:42:42.128786+0000","last_undegraded":"2024-09-10T18:42:42.128786+0000","last_fullsized":"2024-09-10T18:42:42.128786+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:28.075334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:04.058646+0000","last_clean_scrub_stamp":"2024-09-10T18:42:28.075334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:47.297976+0000","scrub_duration":0.0028979330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"411'18768","reported_seq":19291,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-10T18:42:50.822856+0000","last_change":"2024-09-10T18:42:25.979322+0000","last_active":"2024-09-10T18:42:50.822856+0000","last_peered":"2024-09-10T18:42:50.822856+0000","last_clean":"2024-09-10T18:42:50.822856+0000","last_became_active":"2024-09-10T18:40:05.804573+0000","last_became_peered":"2024-09-10T18:40:05.804573+0000","last_unstale":"2024-09-10T18:42:50.822856+0000","last_undegraded":"2024-09-10T18:42:50.822856+0000","last_fullsized":"2024-09-10T18:42:50.822856+0000","mapping_epoch":374,"log_start":"404'14164","ondisk_log_start":"404'14164","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'16684","last_scrub_stamp":"2024-09-10T18:42:25.979225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:42:25.979225+0000","objects_scrubbed":5513,"log_size":4604,"log_dups_size":0,"ondisk_log_size":4604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:35.800778+0000","scrub_duration":2.4221987660000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218365952,"num_objects":3332,"num_object_clones":0,"num_object_copies":9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14108,"num_write_kb":558080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"76'81","reported_seq":396,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058423+0000","last_change":"2024-09-10T18:41:41.914514+0000","last_active":"2024-09-10T18:42:35.058423+0000","last_peered":"2024-09-10T18:42:35.058423+0000","last_clean":"2024-09-10T18:42:35.058423+0000","last_became_active":"2024-09-10T18:39:25.242996+0000","last_became_peered":"2024-09-10T18:39:25.242996+0000","last_unstale":"2024-09-10T18:42:35.058423+0000","last_undegraded":"2024-09-10T18:42:35.058423+0000","last_fullsized":"2024-09-10T18:42:35.058423+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:41.914420+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:41.914420+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:43.841318+0000","scrub_duration":0.0079109539999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"400'29","reported_seq":495,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.057547+0000","last_change":"2024-09-10T18:40:55.034168+0000","last_active":"2024-09-10T18:42:35.057547+0000","last_peered":"2024-09-10T18:42:35.057547+0000","last_clean":"2024-09-10T18:42:35.057547+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:42:35.057547+0000","last_undegraded":"2024-09-10T18:42:35.057547+0000","last_fullsized":"2024-09-10T18:42:35.057547+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'23","last_scrub_stamp":"2024-09-10T18:40:55.034050+0000","last_deep_scrub":"301'23","last_deep_scrub_stamp":"2024-09-10T18:40:55.034050+0000","last_clean_scrub_stamp":"2024-09-10T18:40:55.034050+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069781490000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":459,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129815+0000","last_change":"2024-09-10T18:41:48.482843+0000","last_active":"2024-09-10T18:42:42.129815+0000","last_peered":"2024-09-10T18:42:42.129815+0000","last_clean":"2024-09-10T18:42:42.129815+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:42:42.129815+0000","last_undegraded":"2024-09-10T18:42:42.129815+0000","last_fullsized":"2024-09-10T18:42:42.129815+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:48.482677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:41:48.482677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:00.981134+0000","scrub_duration":0.052469623,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":504,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.444130+0000","last_change":"2024-09-10T18:42:42.444073+0000","last_active":"2024-09-10T18:42:42.444130+0000","last_peered":"2024-09-10T18:42:42.444130+0000","last_clean":"2024-09-10T18:42:42.444130+0000","last_became_active":"2024-09-10T18:39:25.264561+0000","last_became_peered":"2024-09-10T18:39:25.264561+0000","last_unstale":"2024-09-10T18:42:42.444130+0000","last_undegraded":"2024-09-10T18:42:42.444130+0000","last_fullsized":"2024-09-10T18:42:42.444130+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:42.443984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:42.443984+0000","last_clean_scrub_stamp":"2024-09-10T18:42:42.443984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:08.013826+0000","scrub_duration":0.001657424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":435,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.059192+0000","last_change":"2024-09-10T18:41:31.850183+0000","last_active":"2024-09-10T18:42:35.059192+0000","last_peered":"2024-09-10T18:42:35.059192+0000","last_clean":"2024-09-10T18:42:35.059192+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:42:35.059192+0000","last_undegraded":"2024-09-10T18:42:35.059192+0000","last_fullsized":"2024-09-10T18:42:35.059192+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:31.850086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:38:50.727231+0000","last_clean_scrub_stamp":"2024-09-10T18:41:31.850086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:46.735139+0000","scrub_duration":0.0088275699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"411'14323","reported_seq":14595,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-10T18:42:50.234250+0000","last_change":"2024-09-10T18:42:28.198283+0000","last_active":"2024-09-10T18:42:50.234250+0000","last_peered":"2024-09-10T18:42:50.234250+0000","last_clean":"2024-09-10T18:42:50.234250+0000","last_became_active":"2024-09-10T18:40:05.802669+0000","last_became_peered":"2024-09-10T18:40:05.802669+0000","last_unstale":"2024-09-10T18:42:50.234250+0000","last_undegraded":"2024-09-10T18:42:50.234250+0000","last_fullsized":"2024-09-10T18:42:50.234250+0000","mapping_epoch":374,"log_start":"404'10100","ondisk_log_start":"404'10100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12367","last_scrub_stamp":"2024-09-10T18:42:28.198206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:04.783635+0000","last_clean_scrub_stamp":"2024-09-10T18:42:28.198206+0000","objects_scrubbed":5450,"log_size":4223,"log_dups_size":0,"ondisk_log_size":4223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:44.623278+0000","scrub_duration":1.9787170679999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223412224,"num_objects":3409,"num_object_clones":0,"num_object_copies":10227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14323,"num_write_kb":567424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"76'81","reported_seq":179,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129050+0000","last_change":"2024-09-10T18:41:45.417035+0000","last_active":"2024-09-10T18:42:42.129050+0000","last_peered":"2024-09-10T18:42:42.129050+0000","last_clean":"2024-09-10T18:42:42.129050+0000","last_became_active":"2024-09-10T18:34:32.822196+0000","last_became_peered":"2024-09-10T18:34:32.822196+0000","last_unstale":"2024-09-10T18:42:42.129050+0000","last_undegraded":"2024-09-10T18:42:42.129050+0000","last_fullsized":"2024-09-10T18:42:42.129050+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:45.416956+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:45.416956+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:02.570516+0000","scrub_duration":0.0027361949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"400'39","reported_seq":117,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.119093+0000","last_change":"2024-09-10T18:41:49.939121+0000","last_active":"2024-09-10T18:42:42.119093+0000","last_peered":"2024-09-10T18:42:42.119093+0000","last_clean":"2024-09-10T18:42:42.119093+0000","last_became_active":"2024-09-10T18:39:00.902783+0000","last_became_peered":"2024-09-10T18:39:00.902783+0000","last_unstale":"2024-09-10T18:42:42.119093+0000","last_undegraded":"2024-09-10T18:42:42.119093+0000","last_fullsized":"2024-09-10T18:42:42.119093+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'39","last_scrub_stamp":"2024-09-10T18:41:49.939038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:49.939038+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:15.146611+0000","scrub_duration":0.0027044299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":475,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129677+0000","last_change":"2024-09-10T18:41:39.392189+0000","last_active":"2024-09-10T18:42:42.129677+0000","last_peered":"2024-09-10T18:42:42.129677+0000","last_clean":"2024-09-10T18:42:42.129677+0000","last_became_active":"2024-09-10T18:39:25.238037+0000","last_became_peered":"2024-09-10T18:39:25.238037+0000","last_unstale":"2024-09-10T18:42:42.129677+0000","last_undegraded":"2024-09-10T18:42:42.129677+0000","last_fullsized":"2024-09-10T18:42:42.129677+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:41:39.391952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:41:39.391952+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:55.077116+0000","scrub_duration":0.0076173680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":719,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:50.014415+0000","last_change":"2024-09-10T18:41:38.346953+0000","last_active":"2024-09-10T18:42:50.014415+0000","last_peered":"2024-09-10T18:42:50.014415+0000","last_clean":"2024-09-10T18:42:50.014415+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:42:50.014415+0000","last_undegraded":"2024-09-10T18:42:50.014415+0000","last_fullsized":"2024-09-10T18:42:50.014415+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:41:38.346859+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:41:38.346859+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:03.983030+0000","scrub_duration":0.012291041000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":83,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.118255+0000","last_change":"2024-09-10T18:41:31.870285+0000","last_active":"2024-09-10T18:42:42.118255+0000","last_peered":"2024-09-10T18:42:42.118255+0000","last_clean":"2024-09-10T18:42:42.118255+0000","last_became_active":"2024-09-10T18:39:25.242197+0000","last_became_peered":"2024-09-10T18:39:25.242197+0000","last_unstale":"2024-09-10T18:42:42.118255+0000","last_undegraded":"2024-09-10T18:42:42.118255+0000","last_fullsized":"2024-09-10T18:42:42.118255+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:31.870211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:41:31.870211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089146250000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'81","reported_seq":83,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:48.242787+0000","last_change":"2024-09-10T18:42:48.242728+0000","last_active":"2024-09-10T18:42:48.242787+0000","last_peered":"2024-09-10T18:42:48.242787+0000","last_clean":"2024-09-10T18:42:48.242787+0000","last_became_active":"2024-09-10T18:39:00.910728+0000","last_became_peered":"2024-09-10T18:39:00.910728+0000","last_unstale":"2024-09-10T18:42:48.242787+0000","last_undegraded":"2024-09-10T18:42:48.242787+0000","last_fullsized":"2024-09-10T18:42:48.242787+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:48.242648+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:48.242648+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:10.021858+0000","scrub_duration":0.0048037560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":445,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.131000+0000","last_change":"2024-09-10T18:41:39.299055+0000","last_active":"2024-09-10T18:42:42.131000+0000","last_peered":"2024-09-10T18:42:42.131000+0000","last_clean":"2024-09-10T18:42:42.131000+0000","last_became_active":"2024-09-10T18:39:25.243214+0000","last_became_peered":"2024-09-10T18:39:25.243214+0000","last_unstale":"2024-09-10T18:42:42.131000+0000","last_undegraded":"2024-09-10T18:42:42.131000+0000","last_fullsized":"2024-09-10T18:42:42.131000+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:39.298969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:41:39.298969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:57.190842+0000","scrub_duration":0.0042839920000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"400'18","reported_seq":466,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058507+0000","last_change":"2024-09-10T18:41:48.983998+0000","last_active":"2024-09-10T18:42:35.058507+0000","last_peered":"2024-09-10T18:42:35.058507+0000","last_clean":"2024-09-10T18:42:35.058507+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:42:35.058507+0000","last_undegraded":"2024-09-10T18:42:35.058507+0000","last_fullsized":"2024-09-10T18:42:35.058507+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'18","last_scrub_stamp":"2024-09-10T18:41:48.983891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:48.983891+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:11.108479+0000","scrub_duration":0.0035174529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":73,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.121863+0000","last_change":"2024-09-10T18:40:51.751346+0000","last_active":"2024-09-10T18:42:42.121863+0000","last_peered":"2024-09-10T18:42:42.121863+0000","last_clean":"2024-09-10T18:42:42.121863+0000","last_became_active":"2024-09-10T18:39:00.916952+0000","last_became_peered":"2024-09-10T18:39:00.916952+0000","last_unstale":"2024-09-10T18:42:42.121863+0000","last_undegraded":"2024-09-10T18:42:42.121863+0000","last_fullsized":"2024-09-10T18:42:42.121863+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:51.751267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:51.751267+0000","last_clean_scrub_stamp":"2024-09-10T18:40:51.751267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026356539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":450,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129407+0000","last_change":"2024-09-10T18:42:41.135735+0000","last_active":"2024-09-10T18:42:42.129407+0000","last_peered":"2024-09-10T18:42:42.129407+0000","last_clean":"2024-09-10T18:42:42.129407+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:42:42.129407+0000","last_undegraded":"2024-09-10T18:42:42.129407+0000","last_fullsized":"2024-09-10T18:42:42.129407+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:41.135603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:42:41.135603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:08.696743+0000","scrub_duration":0.002967669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'81","reported_seq":386,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.131161+0000","last_change":"2024-09-10T18:41:33.337668+0000","last_active":"2024-09-10T18:42:42.131161+0000","last_peered":"2024-09-10T18:42:42.131161+0000","last_clean":"2024-09-10T18:42:42.131161+0000","last_became_active":"2024-09-10T18:34:02.815880+0000","last_became_peered":"2024-09-10T18:34:02.815880+0000","last_unstale":"2024-09-10T18:42:42.131161+0000","last_undegraded":"2024-09-10T18:42:42.131161+0000","last_fullsized":"2024-09-10T18:42:42.131161+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:33.337569+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:33.337569+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052344720000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"400'25","reported_seq":348,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.066475+0000","last_change":"2024-09-10T18:41:43.958089+0000","last_active":"2024-09-10T18:42:35.066475+0000","last_peered":"2024-09-10T18:42:35.066475+0000","last_clean":"2024-09-10T18:42:35.066475+0000","last_became_active":"2024-09-10T18:39:25.236628+0000","last_became_peered":"2024-09-10T18:39:25.236628+0000","last_unstale":"2024-09-10T18:42:35.066475+0000","last_undegraded":"2024-09-10T18:42:35.066475+0000","last_fullsized":"2024-09-10T18:42:35.066475+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'25","last_scrub_stamp":"2024-09-10T18:41:43.957988+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-09-10T18:35:19.802016+0000","last_clean_scrub_stamp":"2024-09-10T18:41:43.957988+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:05.515733+0000","scrub_duration":0.021283768000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":456,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129126+0000","last_change":"2024-09-10T18:41:47.405474+0000","last_active":"2024-09-10T18:42:42.129126+0000","last_peered":"2024-09-10T18:42:42.129126+0000","last_clean":"2024-09-10T18:42:42.129126+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:42:42.129126+0000","last_undegraded":"2024-09-10T18:42:42.129126+0000","last_fullsized":"2024-09-10T18:42:42.129126+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:47.405374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:41:47.405374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015929049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":437,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058314+0000","last_change":"2024-09-10T18:40:46.624305+0000","last_active":"2024-09-10T18:42:35.058314+0000","last_peered":"2024-09-10T18:42:35.058314+0000","last_clean":"2024-09-10T18:42:35.058314+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:42:35.058314+0000","last_undegraded":"2024-09-10T18:42:35.058314+0000","last_fullsized":"2024-09-10T18:42:35.058314+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:46.624225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:40:46.624225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068887569999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":482,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.134434+0000","last_change":"2024-09-10T18:41:44.176876+0000","last_active":"2024-09-10T18:42:42.134434+0000","last_peered":"2024-09-10T18:42:42.134434+0000","last_clean":"2024-09-10T18:42:42.134434+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:42:42.134434+0000","last_undegraded":"2024-09-10T18:42:42.134434+0000","last_fullsized":"2024-09-10T18:42:42.134434+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:44.176788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:41:44.176788+0000","last_clean_scrub_stamp":"2024-09-10T18:41:44.176788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:00.588555+0000","scrub_duration":0.0078308370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'81","reported_seq":83,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.119136+0000","last_change":"2024-09-10T18:42:39.258803+0000","last_active":"2024-09-10T18:42:42.119136+0000","last_peered":"2024-09-10T18:42:42.119136+0000","last_clean":"2024-09-10T18:42:42.119136+0000","last_became_active":"2024-09-10T18:39:00.902585+0000","last_became_peered":"2024-09-10T18:39:00.902585+0000","last_unstale":"2024-09-10T18:42:42.119136+0000","last_undegraded":"2024-09-10T18:42:42.119136+0000","last_fullsized":"2024-09-10T18:42:42.119136+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:39.258696+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:39:55.999377+0000","last_clean_scrub_stamp":"2024-09-10T18:42:39.258696+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:54.780607+0000","scrub_duration":0.0044049809999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"400'25","reported_seq":92,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.123768+0000","last_change":"2024-09-10T18:41:02.704925+0000","last_active":"2024-09-10T18:42:42.123768+0000","last_peered":"2024-09-10T18:42:42.123768+0000","last_clean":"2024-09-10T18:42:42.123768+0000","last_became_active":"2024-09-10T18:39:00.899968+0000","last_became_peered":"2024-09-10T18:39:00.899968+0000","last_unstale":"2024-09-10T18:42:42.123768+0000","last_undegraded":"2024-09-10T18:42:42.123768+0000","last_fullsized":"2024-09-10T18:42:42.123768+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:41:02.704656+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:41:02.704656+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081500759999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":72,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129099+0000","last_change":"2024-09-10T18:40:39.071149+0000","last_active":"2024-09-10T18:42:42.129099+0000","last_peered":"2024-09-10T18:42:42.129099+0000","last_clean":"2024-09-10T18:42:42.129099+0000","last_became_active":"2024-09-10T18:39:25.245366+0000","last_became_peered":"2024-09-10T18:39:25.245366+0000","last_unstale":"2024-09-10T18:42:42.129099+0000","last_undegraded":"2024-09-10T18:42:42.129099+0000","last_fullsized":"2024-09-10T18:42:42.129099+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:39.071067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_clean_scrub_stamp":"2024-09-10T18:40:39.071067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010317092,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"342'9","reported_seq":701,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129081+0000","last_change":"2024-09-10T18:40:38.445567+0000","last_active":"2024-09-10T18:42:42.129081+0000","last_peered":"2024-09-10T18:42:42.129081+0000","last_clean":"2024-09-10T18:42:42.129081+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:42:42.129081+0000","last_undegraded":"2024-09-10T18:42:42.129081+0000","last_fullsized":"2024-09-10T18:42:42.129081+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'9","last_scrub_stamp":"2024-09-10T18:40:38.445473+0000","last_deep_scrub":"342'9","last_deep_scrub_stamp":"2024-09-10T18:37:56.641708+0000","last_clean_scrub_stamp":"2024-09-10T18:40:38.445473+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047172459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":73,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058780+0000","last_change":"2024-09-10T18:39:34.901254+0000","last_active":"2024-09-10T18:42:35.058780+0000","last_peered":"2024-09-10T18:42:35.058780+0000","last_clean":"2024-09-10T18:42:35.058780+0000","last_became_active":"2024-09-10T18:39:00.903167+0000","last_became_peered":"2024-09-10T18:39:00.903167+0000","last_unstale":"2024-09-10T18:42:35.058780+0000","last_undegraded":"2024-09-10T18:42:35.058780+0000","last_fullsized":"2024-09-10T18:42:35.058780+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:34.901182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:39:34.901182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00237777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"76'81","reported_seq":259,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.118057+0000","last_change":"2024-09-10T18:40:43.823600+0000","last_active":"2024-09-10T18:42:42.118057+0000","last_peered":"2024-09-10T18:42:42.118057+0000","last_clean":"2024-09-10T18:42:42.118057+0000","last_became_active":"2024-09-10T18:39:25.249841+0000","last_became_peered":"2024-09-10T18:39:25.249841+0000","last_unstale":"2024-09-10T18:42:42.118057+0000","last_undegraded":"2024-09-10T18:42:42.118057+0000","last_fullsized":"2024-09-10T18:42:42.118057+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:43.823507+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:36:08.710566+0000","last_clean_scrub_stamp":"2024-09-10T18:40:43.823507+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042527390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"400'30","reported_seq":494,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058624+0000","last_change":"2024-09-10T18:42:27.160031+0000","last_active":"2024-09-10T18:42:35.058624+0000","last_peered":"2024-09-10T18:42:35.058624+0000","last_clean":"2024-09-10T18:42:35.058624+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:42:35.058624+0000","last_undegraded":"2024-09-10T18:42:35.058624+0000","last_fullsized":"2024-09-10T18:42:35.058624+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'30","last_scrub_stamp":"2024-09-10T18:42:27.159945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:42:27.159945+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:39.165568+0000","scrub_duration":0.0094521229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":449,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.130434+0000","last_change":"2024-09-10T18:40:56.107500+0000","last_active":"2024-09-10T18:42:42.130434+0000","last_peered":"2024-09-10T18:42:42.130434+0000","last_clean":"2024-09-10T18:42:42.130434+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:42:42.130434+0000","last_undegraded":"2024-09-10T18:42:42.130434+0000","last_fullsized":"2024-09-10T18:42:42.130434+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:56.107424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:17.826448+0000","last_clean_scrub_stamp":"2024-09-10T18:40:56.107424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01055597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"362'11","reported_seq":139,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:50.014366+0000","last_change":"2024-09-10T18:41:30.919843+0000","last_active":"2024-09-10T18:42:50.014366+0000","last_peered":"2024-09-10T18:42:50.014366+0000","last_clean":"2024-09-10T18:42:50.014366+0000","last_became_active":"2024-09-10T18:39:25.256047+0000","last_became_peered":"2024-09-10T18:39:25.256047+0000","last_unstale":"2024-09-10T18:42:50.014366+0000","last_undegraded":"2024-09-10T18:42:50.014366+0000","last_fullsized":"2024-09-10T18:42:50.014366+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'11","last_scrub_stamp":"2024-09-10T18:41:30.919717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:41:30.919717+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096531840000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":72,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129227+0000","last_change":"2024-09-10T18:40:52.022624+0000","last_active":"2024-09-10T18:42:42.129227+0000","last_peered":"2024-09-10T18:42:42.129227+0000","last_clean":"2024-09-10T18:42:42.129227+0000","last_became_active":"2024-09-10T18:39:25.247171+0000","last_became_peered":"2024-09-10T18:39:25.247171+0000","last_unstale":"2024-09-10T18:42:42.129227+0000","last_undegraded":"2024-09-10T18:42:42.129227+0000","last_fullsized":"2024-09-10T18:42:42.129227+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:52.022343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:52.022343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053789739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.16","version":"76'81","reported_seq":171,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.128094+0000","last_change":"2024-09-10T18:40:46.174454+0000","last_active":"2024-09-10T18:42:42.128094+0000","last_peered":"2024-09-10T18:42:42.128094+0000","last_clean":"2024-09-10T18:42:42.128094+0000","last_became_active":"2024-09-10T18:39:25.253130+0000","last_became_peered":"2024-09-10T18:39:25.253130+0000","last_unstale":"2024-09-10T18:42:42.128094+0000","last_undegraded":"2024-09-10T18:42:42.128094+0000","last_fullsized":"2024-09-10T18:42:42.128094+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:46.174367+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:40:46.174367+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083268330000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"400'27","reported_seq":528,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:36.289128+0000","last_change":"2024-09-10T18:42:36.288994+0000","last_active":"2024-09-10T18:42:36.289128+0000","last_peered":"2024-09-10T18:42:36.289128+0000","last_clean":"2024-09-10T18:42:36.289128+0000","last_became_active":"2024-09-10T18:41:11.933013+0000","last_became_peered":"2024-09-10T18:41:11.933013+0000","last_unstale":"2024-09-10T18:42:36.289128+0000","last_undegraded":"2024-09-10T18:42:36.289128+0000","last_fullsized":"2024-09-10T18:42:36.289128+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'27","last_scrub_stamp":"2024-09-10T18:42:36.288831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:42:36.288831+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:57.048695+0000","scrub_duration":0.0051082469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":458,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.058041+0000","last_change":"2024-09-10T18:42:25.151426+0000","last_active":"2024-09-10T18:42:35.058041+0000","last_peered":"2024-09-10T18:42:35.058041+0000","last_clean":"2024-09-10T18:42:35.058041+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:42:35.058041+0000","last_undegraded":"2024-09-10T18:42:35.058041+0000","last_fullsized":"2024-09-10T18:42:35.058041+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:25.151319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:42:25.151319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:38.515432+0000","scrub_duration":0.0047341060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":479,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.127417+0000","last_change":"2024-09-10T18:40:53.230753+0000","last_active":"2024-09-10T18:42:42.127417+0000","last_peered":"2024-09-10T18:42:42.127417+0000","last_clean":"2024-09-10T18:42:42.127417+0000","last_became_active":"2024-09-10T18:39:00.897018+0000","last_became_peered":"2024-09-10T18:39:00.897018+0000","last_unstale":"2024-09-10T18:42:42.127417+0000","last_undegraded":"2024-09-10T18:42:42.127417+0000","last_fullsized":"2024-09-10T18:42:42.127417+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:53.230680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_clean_scrub_stamp":"2024-09-10T18:40:53.230680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003669224,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":455,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.067670+0000","last_change":"2024-09-10T18:39:53.911283+0000","last_active":"2024-09-10T18:42:35.067670+0000","last_peered":"2024-09-10T18:42:35.067670+0000","last_clean":"2024-09-10T18:42:35.067670+0000","last_became_active":"2024-09-10T18:39:25.237500+0000","last_became_peered":"2024-09-10T18:39:25.237500+0000","last_unstale":"2024-09-10T18:42:35.067670+0000","last_undegraded":"2024-09-10T18:42:35.067670+0000","last_fullsized":"2024-09-10T18:42:35.067670+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:39:53.911229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:39:53.911229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001589023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'81","reported_seq":83,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.122761+0000","last_change":"2024-09-10T18:40:52.818032+0000","last_active":"2024-09-10T18:42:42.122761+0000","last_peered":"2024-09-10T18:42:42.122761+0000","last_clean":"2024-09-10T18:42:42.122761+0000","last_became_active":"2024-09-10T18:39:25.244612+0000","last_became_peered":"2024-09-10T18:39:25.244612+0000","last_unstale":"2024-09-10T18:42:42.122761+0000","last_undegraded":"2024-09-10T18:42:42.122761+0000","last_fullsized":"2024-09-10T18:42:42.122761+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:40:52.817945+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:40:52.817945+0000","last_clean_scrub_stamp":"2024-09-10T18:40:52.817945+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.061016616000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"400'21","reported_seq":499,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.127633+0000","last_change":"2024-09-10T18:41:00.246540+0000","last_active":"2024-09-10T18:42:42.127633+0000","last_peered":"2024-09-10T18:42:42.127633+0000","last_clean":"2024-09-10T18:42:42.127633+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:42:42.127633+0000","last_undegraded":"2024-09-10T18:42:42.127633+0000","last_fullsized":"2024-09-10T18:42:42.127633+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-09-10T18:41:00.246464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:00.246464+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012761635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":452,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129891+0000","last_change":"2024-09-10T18:41:29.335366+0000","last_active":"2024-09-10T18:42:42.129891+0000","last_peered":"2024-09-10T18:42:42.129891+0000","last_clean":"2024-09-10T18:42:42.129891+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:42:42.129891+0000","last_undegraded":"2024-09-10T18:42:42.129891+0000","last_fullsized":"2024-09-10T18:42:42.129891+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:29.335275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:41:29.335275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032473530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":440,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.066472+0000","last_change":"2024-09-10T18:40:47.615302+0000","last_active":"2024-09-10T18:42:35.066472+0000","last_peered":"2024-09-10T18:42:35.066472+0000","last_clean":"2024-09-10T18:42:35.066472+0000","last_became_active":"2024-09-10T18:39:25.251469+0000","last_became_peered":"2024-09-10T18:39:25.251469+0000","last_unstale":"2024-09-10T18:42:35.066472+0000","last_undegraded":"2024-09-10T18:42:35.066472+0000","last_fullsized":"2024-09-10T18:42:35.066472+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:47.615205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:40:47.615205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030335510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":473,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.119177+0000","last_change":"2024-09-10T18:42:33.235205+0000","last_active":"2024-09-10T18:42:42.119177+0000","last_peered":"2024-09-10T18:42:42.119177+0000","last_clean":"2024-09-10T18:42:42.119177+0000","last_became_active":"2024-09-10T18:39:25.264554+0000","last_became_peered":"2024-09-10T18:39:25.264554+0000","last_unstale":"2024-09-10T18:42:42.119177+0000","last_undegraded":"2024-09-10T18:42:42.119177+0000","last_fullsized":"2024-09-10T18:42:42.119177+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:33.235134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:42:33.235134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:40.846629+0000","scrub_duration":0.002556516,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"76'81","reported_seq":188,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.131285+0000","last_change":"2024-09-10T18:41:28.989528+0000","last_active":"2024-09-10T18:42:42.131285+0000","last_peered":"2024-09-10T18:42:42.131285+0000","last_clean":"2024-09-10T18:42:42.131285+0000","last_became_active":"2024-09-10T18:34:02.802580+0000","last_became_peered":"2024-09-10T18:34:02.802580+0000","last_unstale":"2024-09-10T18:42:42.131285+0000","last_undegraded":"2024-09-10T18:42:42.131285+0000","last_fullsized":"2024-09-10T18:42:42.131285+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:28.989434+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:28.989434+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:51.842021+0000","scrub_duration":0.011606899,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"400'27","reported_seq":489,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.067377+0000","last_change":"2024-09-10T18:40:41.493272+0000","last_active":"2024-09-10T18:42:35.067377+0000","last_peered":"2024-09-10T18:42:35.067377+0000","last_clean":"2024-09-10T18:42:35.067377+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:42:35.067377+0000","last_undegraded":"2024-09-10T18:42:35.067377+0000","last_fullsized":"2024-09-10T18:42:35.067377+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-09-10T18:40:41.493074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:40:41.493074+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049833050000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":83,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.121794+0000","last_change":"2024-09-10T18:41:38.954345+0000","last_active":"2024-09-10T18:42:42.121794+0000","last_peered":"2024-09-10T18:42:42.121794+0000","last_clean":"2024-09-10T18:42:42.121794+0000","last_became_active":"2024-09-10T18:39:00.916983+0000","last_became_peered":"2024-09-10T18:39:00.916983+0000","last_unstale":"2024-09-10T18:42:42.121794+0000","last_undegraded":"2024-09-10T18:42:42.121794+0000","last_fullsized":"2024-09-10T18:42:42.121794+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:38.954273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:41:38.954273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:42:57.986773+0000","scrub_duration":0.0019398480000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":445,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.128428+0000","last_change":"2024-09-10T18:40:46.080092+0000","last_active":"2024-09-10T18:42:42.128428+0000","last_peered":"2024-09-10T18:42:42.128428+0000","last_clean":"2024-09-10T18:42:42.128428+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:42:42.128428+0000","last_undegraded":"2024-09-10T18:42:42.128428+0000","last_fullsized":"2024-09-10T18:42:42.128428+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:46.080018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:40:46.080018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014966508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":471,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:48.554420+0000","last_change":"2024-09-10T18:42:48.554386+0000","last_active":"2024-09-10T18:42:48.554420+0000","last_peered":"2024-09-10T18:42:48.554420+0000","last_clean":"2024-09-10T18:42:48.554420+0000","last_became_active":"2024-09-10T18:39:25.242900+0000","last_became_peered":"2024-09-10T18:39:25.242900+0000","last_unstale":"2024-09-10T18:42:48.554420+0000","last_undegraded":"2024-09-10T18:42:48.554420+0000","last_fullsized":"2024-09-10T18:42:48.554420+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:48.554301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:48.554301+0000","last_clean_scrub_stamp":"2024-09-10T18:42:48.554301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:01.886073+0000","scrub_duration":0.008030354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'81","reported_seq":180,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.128966+0000","last_change":"2024-09-10T18:42:23.159856+0000","last_active":"2024-09-10T18:42:42.128966+0000","last_peered":"2024-09-10T18:42:42.128966+0000","last_clean":"2024-09-10T18:42:42.128966+0000","last_became_active":"2024-09-10T18:39:00.889998+0000","last_became_peered":"2024-09-10T18:39:00.889998+0000","last_unstale":"2024-09-10T18:42:42.128966+0000","last_undegraded":"2024-09-10T18:42:42.128966+0000","last_fullsized":"2024-09-10T18:42:42.128966+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:23.159712+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:23.159712+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:41.248653+0000","scrub_duration":0.001874744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"400'12","reported_seq":464,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.066511+0000","last_change":"2024-09-10T18:41:40.949560+0000","last_active":"2024-09-10T18:42:35.066511+0000","last_peered":"2024-09-10T18:42:35.066511+0000","last_clean":"2024-09-10T18:42:35.066511+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:42:35.066511+0000","last_undegraded":"2024-09-10T18:42:35.066511+0000","last_fullsized":"2024-09-10T18:42:35.066511+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'12","last_scrub_stamp":"2024-09-10T18:41:40.949115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:40.949115+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:05.051357+0000","scrub_duration":0.022213717000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":475,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.122592+0000","last_change":"2024-09-10T18:41:40.970661+0000","last_active":"2024-09-10T18:42:42.122592+0000","last_peered":"2024-09-10T18:42:42.122592+0000","last_clean":"2024-09-10T18:42:42.122592+0000","last_became_active":"2024-09-10T18:39:25.242295+0000","last_became_peered":"2024-09-10T18:39:25.242295+0000","last_unstale":"2024-09-10T18:42:42.122592+0000","last_undegraded":"2024-09-10T18:42:42.122592+0000","last_fullsized":"2024-09-10T18:42:42.122592+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:40.970571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:41:40.970571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0072197229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":419,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.132454+0000","last_change":"2024-09-10T18:42:34.597769+0000","last_active":"2024-09-10T18:42:42.132454+0000","last_peered":"2024-09-10T18:42:42.132454+0000","last_clean":"2024-09-10T18:42:42.132454+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:42:42.132454+0000","last_undegraded":"2024-09-10T18:42:42.132454+0000","last_fullsized":"2024-09-10T18:42:42.132454+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:34.597693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:42:34.597693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:03.774399+0000","scrub_duration":0.0025145829999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":443,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.128961+0000","last_change":"2024-09-10T18:41:36.315811+0000","last_active":"2024-09-10T18:42:42.128961+0000","last_peered":"2024-09-10T18:42:42.128961+0000","last_clean":"2024-09-10T18:42:42.128961+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:42:42.128961+0000","last_undegraded":"2024-09-10T18:42:42.128961+0000","last_fullsized":"2024-09-10T18:42:42.128961+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:36.315732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:41:36.315732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:05.356585+0000","scrub_duration":0.015270002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'81","reported_seq":169,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-09-10T18:42:35.057881+0000","last_change":"2024-09-10T18:42:32.301668+0000","last_active":"2024-09-10T18:42:35.057881+0000","last_peered":"2024-09-10T18:42:35.057881+0000","last_clean":"2024-09-10T18:42:35.057881+0000","last_became_active":"2024-09-10T18:34:34.867380+0000","last_became_peered":"2024-09-10T18:34:34.867380+0000","last_unstale":"2024-09-10T18:42:35.057881+0000","last_undegraded":"2024-09-10T18:42:35.057881+0000","last_fullsized":"2024-09-10T18:42:35.057881+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:32.301577+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:32.301577+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:43.842433+0000","scrub_duration":0.0077452609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":474,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.127602+0000","last_change":"2024-09-10T18:40:57.216412+0000","last_active":"2024-09-10T18:42:42.127602+0000","last_peered":"2024-09-10T18:42:42.127602+0000","last_clean":"2024-09-10T18:42:42.127602+0000","last_became_active":"2024-09-10T18:39:00.909906+0000","last_became_peered":"2024-09-10T18:39:00.909906+0000","last_unstale":"2024-09-10T18:42:42.127602+0000","last_undegraded":"2024-09-10T18:42:42.127602+0000","last_fullsized":"2024-09-10T18:42:42.127602+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:57.216321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_clean_scrub_stamp":"2024-09-10T18:40:57.216321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069155350000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":451,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:46.566065+0000","last_change":"2024-09-10T18:42:46.565973+0000","last_active":"2024-09-10T18:42:46.566065+0000","last_peered":"2024-09-10T18:42:46.566065+0000","last_clean":"2024-09-10T18:42:46.566065+0000","last_became_active":"2024-09-10T18:39:25.250206+0000","last_became_peered":"2024-09-10T18:39:25.250206+0000","last_unstale":"2024-09-10T18:42:46.566065+0000","last_undegraded":"2024-09-10T18:42:46.566065+0000","last_fullsized":"2024-09-10T18:42:46.566065+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:46.565877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:46.565877+0000","last_clean_scrub_stamp":"2024-09-10T18:42:46.565877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:12.551542+0000","scrub_duration":0.0051506470000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"400'32","reported_seq":551,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.131062+0000","last_change":"2024-09-10T18:42:38.627801+0000","last_active":"2024-09-10T18:42:42.131062+0000","last_peered":"2024-09-10T18:42:42.131062+0000","last_clean":"2024-09-10T18:42:42.131062+0000","last_became_active":"2024-09-10T18:39:00.921476+0000","last_became_peered":"2024-09-10T18:39:00.921476+0000","last_unstale":"2024-09-10T18:42:42.131062+0000","last_undegraded":"2024-09-10T18:42:42.131062+0000","last_fullsized":"2024-09-10T18:42:42.131062+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'32","last_scrub_stamp":"2024-09-10T18:42:38.627699+0000","last_deep_scrub":"369'24","last_deep_scrub_stamp":"2024-09-10T18:39:50.690827+0000","last_clean_scrub_stamp":"2024-09-10T18:42:38.627699+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:05.830328+0000","scrub_duration":0.010970025,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":72,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-10T18:42:42.129339+0000","last_change":"2024-09-10T18:40:51.034156+0000","last_active":"2024-09-10T18:42:42.129339+0000","last_peered":"2024-09-10T18:42:42.129339+0000","last_clean":"2024-09-10T18:42:42.129339+0000","last_became_active":"2024-09-10T18:39:25.243964+0000","last_became_peered":"2024-09-10T18:39:25.243964+0000","last_unstale":"2024-09-10T18:42:42.129339+0000","last_undegraded":"2024-09-10T18:42:42.129339+0000","last_fullsized":"2024-09-10T18:42:42.129339+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:51.034067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:40:51.034067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095335249999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3526885400,"num_objects":53817,"num_object_clones":0,"num_object_copies":161451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53817,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":227393,"num_write_kb":8998657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10565332992,"data_stored":10565320776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63864,"ondisk_log_size":63864,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1548,"num_read_kb":1305,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":4566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466944,"data_stored":12426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":1064,"ondisk_log_size":1064,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1269,"num_read_kb":1264,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2511,"ondisk_log_size":2511,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234243,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2407052,"kb_used_data":1512452,"kb_used_omap":1,"kb_used_meta":894590,"kb_avail":91338100,"statfs":{"total":95995035648,"available":93530214400,"internally_reserved":0,"allocated":1548750848,"data_stored":1548332013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916060504},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":1.379,"15min":1.7090000000000001},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.21099999999999999},"max":{"1min":2.2250000000000001,"5min":29.443999999999999,"15min":75.506},"last":1.71},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.53,"15min":1.655},"min":{"1min":0.26500000000000001,"5min":0.25700000000000001,"15min":0.223},"max":{"1min":12.571999999999999,"5min":29.530999999999999,"15min":75.710999999999999},"last":1.3740000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:42:02 2024","interfaces":[{"interface":"back","average":{"1min":1.8160000000000001,"5min":2.1070000000000002,"15min":1.839},"min":{"1min":0.23400000000000001,"5min":0.22800000000000001,"15min":0.214},"max":{"1min":12.461,"5min":98.305999999999997,"15min":98.305999999999997},"last":1.6870000000000001},{"interface":"front","average":{"1min":1.262,"5min":2.1589999999999998,"15min":1.772},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":9.4969999999999999,"5min":98.381,"15min":98.381},"last":1.415}]},{"osd":2,"last update":"Tue Sep 10 18:42:02 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.994,"15min":1.8280000000000001},"min":{"1min":0.27300000000000002,"5min":0.27100000000000002,"15min":0.26400000000000001},"max":{"1min":5.3840000000000003,"5min":44.728000000000002,"15min":136.648},"last":1.984},{"interface":"front","average":{"1min":0.79900000000000004,"5min":1.9330000000000001,"15min":1.8700000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.24399999999999999},"max":{"1min":2.4670000000000001,"5min":44.664999999999999,"15min":136.506},"last":1.7330000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:42:02 2024","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":1.135,"15min":1.2809999999999999},"min":{"1min":0.34999999999999998,"5min":0.30199999999999999,"15min":0.27200000000000002},"max":{"1min":8.7400000000000002,"5min":8.7400000000000002,"15min":38.225000000000001},"last":2.3170000000000002},{"interface":"front","average":{"1min":1.405,"5min":1.1040000000000001,"15min":1.274},"min":{"1min":0.33600000000000002,"5min":0.29899999999999999,"15min":0.29199999999999998},"max":{"1min":12.516,"5min":12.516,"15min":38.104999999999997},"last":2.2639999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.5009999999999999,"15min":0.90300000000000002},"min":{"1min":0.23899999999999999,"5min":0.22700000000000001,"15min":0.224},"max":{"1min":4.6079999999999997,"5min":23.346,"15min":23.346},"last":1.343},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.3999999999999999,"15min":0.91700000000000004},"min":{"1min":0.30099999999999999,"5min":0.26200000000000001,"15min":0.25},"max":{"1min":5.4219999999999997,"5min":23.323,"15min":23.323},"last":1.915}]},{"osd":5,"last update":"Tue Sep 10 18:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":1.111,"15min":0.97599999999999998},"min":{"1min":0.27900000000000003,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":2.431,"5min":16.756,"15min":16.756},"last":1.6279999999999999},{"interface":"front","average":{"1min":0.873,"5min":1.349,"15min":0.98799999999999999},"min":{"1min":0.35699999999999998,"5min":0.224,"15min":0.224},"max":{"1min":4.5449999999999999,"5min":20.495999999999999,"15min":20.495999999999999},"last":1.3169999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:41:52 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.319,"15min":0.97299999999999998},"min":{"1min":0.32100000000000001,"5min":0.317,"15min":0.27000000000000002},"max":{"1min":4.6689999999999996,"5min":25.902999999999999,"15min":25.902999999999999},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.3939999999999999,"15min":0.93999999999999995},"min":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.30199999999999999},"max":{"1min":4.944,"5min":24.074000000000002,"15min":24.074000000000002},"last":1.3959999999999999}]}]},{"osd":6,"up_from":144,"seq":618475290822,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2647984,"kb_used_data":1753384,"kb_used_omap":7,"kb_used_meta":894584,"kb_avail":91097168,"statfs":{"total":95995035648,"available":93283500032,"internally_reserved":0,"allocated":1795465216,"data_stored":1795074375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":916054103},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":1.0369999999999999,"15min":1.2549999999999999},"min":{"1min":0.25600000000000001,"5min":0.23300000000000001,"15min":0.192},"max":{"1min":3.6800000000000002,"5min":5.5469999999999997,"15min":30.629999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.026,"15min":1.1499999999999999},"min":{"1min":0.27500000000000002,"5min":0.246,"15min":0.217},"max":{"1min":3.9609999999999999,"5min":4.9489999999999998,"15min":30.597000000000001},"last":1.3240000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:42:00 2024","interfaces":[{"interface":"back","average":{"1min":2.774,"5min":1.4059999999999999,"15min":1.1970000000000001},"min":{"1min":0.312,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":23.782,"5min":23.782,"15min":30.984000000000002},"last":1.361},{"interface":"front","average":{"1min":2.335,"5min":1.355,"15min":1.262},"min":{"1min":0.32900000000000001,"5min":0.253,"15min":0.24199999999999999},"max":{"1min":23.687999999999999,"5min":23.687999999999999,"15min":30.571000000000002},"last":0.56000000000000005}]},{"osd":2,"last update":"Tue Sep 10 18:42:00 2024","interfaces":[{"interface":"back","average":{"1min":2.3450000000000002,"5min":1.4490000000000001,"15min":1.3160000000000001},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.25600000000000001},"max":{"1min":23.763999999999999,"5min":23.763999999999999,"15min":39.329999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":2.9929999999999999,"5min":1.7210000000000001,"15min":1.3080000000000001},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.23400000000000001},"max":{"1min":23.856000000000002,"5min":23.856000000000002,"15min":39.061999999999998},"last":1.3009999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:42:00 2024","interfaces":[{"interface":"back","average":{"1min":2.3319999999999999,"5min":4.8920000000000003,"15min":2.6619999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.27800000000000002},"max":{"1min":29.670999999999999,"5min":162.61600000000001,"15min":162.61600000000001},"last":1.4530000000000001},{"interface":"front","average":{"1min":2.2970000000000002,"5min":4.8220000000000001,"15min":2.6629999999999998},"min":{"1min":0.39900000000000002,"5min":0.29599999999999999,"15min":0.27100000000000002},"max":{"1min":32.018000000000001,"5min":163.39599999999999,"15min":163.39599999999999},"last":1.4079999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.82999999999999996,"15min":0.69299999999999995},"min":{"1min":0.222,"5min":0.189,"15min":0.189},"max":{"1min":3.9279999999999999,"5min":4.9050000000000002,"15min":5.4950000000000001},"last":1.3859999999999999},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.22800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":3.0169999999999999,"5min":11.531000000000001,"15min":11.531000000000001},"last":0.61799999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.79600000000000004,"15min":0.70999999999999996},"min":{"1min":0.26100000000000001,"5min":0.254,"15min":0.24399999999999999},"max":{"1min":2.359,"5min":4.4139999999999997,"15min":4.6619999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.85899999999999999,"15min":0.75600000000000001},"min":{"1min":0.28000000000000003,"5min":0.248,"15min":0.20899999999999999},"max":{"1min":2.0499999999999998,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":0.48899999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":1.0129999999999999,"15min":0.78700000000000003},"min":{"1min":0.34699999999999998,"5min":0.26700000000000002,"15min":0.23799999999999999},"max":{"1min":3.1749999999999998,"5min":11.449,"15min":11.449},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.82099999999999995,"15min":0.73799999999999999},"min":{"1min":0.36199999999999999,"5min":0.32000000000000001,"15min":0.27100000000000002},"max":{"1min":3.754,"5min":5.0149999999999997,"15min":5.5309999999999997},"last":1.4259999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724814,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1771908,"kb_used_data":880700,"kb_used_omap":1,"kb_used_meta":891134,"kb_avail":91973244,"statfs":{"total":95995035648,"available":94180601856,"internally_reserved":0,"allocated":901836800,"data_stored":901426415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912521560},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.97499999999999998,"15min":1.2929999999999999},"min":{"1min":0.24299999999999999,"5min":0.224,"15min":0.224},"max":{"1min":1.8939999999999999,"5min":7.9020000000000001,"15min":31.172000000000001},"last":4.274},{"interface":"front","average":{"1min":0.755,"5min":0.89400000000000002,"15min":1.2190000000000001},"min":{"1min":0.27900000000000003,"5min":0.25,"15min":0.24099999999999999},"max":{"1min":2.4620000000000002,"5min":4.851,"15min":32.246000000000002},"last":0.623}]},{"osd":1,"last update":"Tue Sep 10 18:42:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7490000000000001,"5min":1.3660000000000001,"15min":1.7150000000000001},"min":{"1min":0.30599999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":22.623999999999999,"5min":22.623999999999999,"15min":111.773},"last":0.52100000000000002},{"interface":"front","average":{"1min":2.3210000000000002,"5min":1.1899999999999999,"15min":1.6679999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":22.635000000000002,"5min":22.635000000000002,"15min":111.73999999999999},"last":0.65300000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:42:19 2024","interfaces":[{"interface":"back","average":{"1min":2.3279999999999998,"5min":1.5229999999999999,"15min":1.454},"min":{"1min":0.247,"5min":0.247,"15min":0.24299999999999999},"max":{"1min":23.396000000000001,"5min":23.396000000000001,"15min":54.530000000000001},"last":4.3040000000000003},{"interface":"front","average":{"1min":2.8660000000000001,"5min":1.401,"15min":1.389},"min":{"1min":0.39100000000000001,"5min":0.27400000000000002,"15min":0.247},"max":{"1min":22.905999999999999,"5min":22.905999999999999,"15min":54.514000000000003},"last":0.46300000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:42:19 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":1.6140000000000001,"15min":2.0369999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.27100000000000002},"max":{"1min":23.007999999999999,"5min":23.007999999999999,"15min":106.205},"last":4.327},{"interface":"front","average":{"1min":2.5619999999999998,"5min":1.536,"15min":2.0339999999999998},"min":{"1min":0.30499999999999999,"5min":0.28999999999999998,"15min":0.28299999999999997},"max":{"1min":22.931000000000001,"5min":22.931000000000001,"15min":105.878},"last":0.437}]},{"osd":4,"last update":"Tue Sep 10 18:42:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":0.89000000000000001,"15min":0.75800000000000001},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.22900000000000001},"max":{"1min":5.226,"5min":5.226,"15min":5.8040000000000003},"last":0.56499999999999995},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.046,"15min":0.84199999999999997},"min":{"1min":0.27900000000000003,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":5.3230000000000004,"5min":8.7270000000000003,"15min":8.7270000000000003},"last":4.2160000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:41:49 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":0.92000000000000004,"15min":0.76500000000000001},"min":{"1min":0.317,"5min":0.29999999999999999,"15min":0.27500000000000002},"max":{"1min":2.5659999999999998,"5min":7.9969999999999999,"15min":7.9969999999999999},"last":4.2510000000000003},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.2909999999999999,"15min":0.91200000000000003},"min":{"1min":0.32800000000000001,"5min":0.30299999999999999,"15min":0.27000000000000002},"max":{"1min":8.7400000000000002,"5min":8.7400000000000002,"15min":8.7400000000000002},"last":2.8919999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:41:49 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.0620000000000001,"15min":0.95999999999999996},"min":{"1min":0.35899999999999999,"5min":0.32200000000000001,"15min":0.25900000000000001},"max":{"1min":6.867,"5min":8.3870000000000005,"15min":28.934000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.0289999999999999,"15min":0.86199999999999999},"min":{"1min":0.36099999999999999,"5min":0.314,"15min":0.25900000000000001},"max":{"1min":5.593,"5min":5.593,"15min":5.8259999999999996},"last":0.59599999999999997}]}]},{"osd":4,"up_from":33,"seq":141733921042,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2237312,"kb_used_data":1333816,"kb_used_omap":1,"kb_used_meta":903486,"kb_avail":91507840,"statfs":{"total":95995035648,"available":93704028160,"internally_reserved":0,"allocated":1365827584,"data_stored":1365419998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":925170008},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:41:52 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.5920000000000001,"15min":1.2609999999999999},"min":{"1min":0.247,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.8080000000000001,"5min":21.468,"15min":21.468},"last":2.028},{"interface":"front","average":{"1min":0.64700000000000002,"5min":1.3240000000000001,"15min":1.0800000000000001},"min":{"1min":0.28399999999999997,"5min":0.21299999999999999,"15min":0.189},"max":{"1min":3.0409999999999999,"5min":24.277999999999999,"15min":24.277999999999999},"last":0.65800000000000003}]},{"osd":1,"last update":"Tue Sep 10 18:41:52 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.3779999999999999,"15min":1.2589999999999999},"min":{"1min":0.246,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":8.2539999999999996,"5min":12.696,"15min":50.045000000000002},"last":1.986},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.2629999999999999,"15min":1.296},"min":{"1min":0.29399999999999998,"5min":0.28000000000000003,"15min":0.246},"max":{"1min":4.0380000000000003,"5min":13.632,"15min":50.283000000000001},"last":1.853}]},{"osd":2,"last update":"Tue Sep 10 18:41:52 2024","interfaces":[{"interface":"back","average":{"1min":12.504,"5min":4.4640000000000004,"15min":2.0470000000000002},"min":{"1min":0.32800000000000001,"5min":0.251,"15min":0.249},"max":{"1min":169.417,"5min":169.417,"15min":169.417},"last":1.885},{"interface":"front","average":{"1min":12.183999999999999,"5min":4.4980000000000002,"15min":2.0609999999999999},"min":{"1min":0.251,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":169.40100000000001,"5min":169.40100000000001,"15min":169.40100000000001},"last":0.70899999999999996}]},{"osd":3,"last update":"Tue Sep 10 18:41:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.0469999999999999,"15min":1.3500000000000001},"min":{"1min":0.315,"5min":0.26200000000000001,"15min":0.249},"max":{"1min":5.2830000000000004,"5min":7.7699999999999996,"15min":49.990000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.83799999999999997,"5min":1.508,"15min":1.5329999999999999},"min":{"1min":0.33700000000000002,"5min":0.28999999999999998,"15min":0.28000000000000003},"max":{"1min":2.0299999999999998,"5min":10.119999999999999,"15min":50.764000000000003},"last":2.093}]},{"osd":5,"last update":"Tue Sep 10 18:42:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":0.90600000000000003,"15min":0.84599999999999997},"min":{"1min":0.30599999999999999,"5min":0.25,"15min":0.223},"max":{"1min":3.6379999999999999,"5min":3.6379999999999999,"15min":22.861000000000001},"last":1.958},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.96899999999999997,"15min":0.90100000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.22},"max":{"1min":2.976,"5min":7.9119999999999999,"15min":19.611999999999998},"last":0.59699999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:41:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.6399999999999999,"15min":0.98199999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.26400000000000001},"max":{"1min":4.4900000000000002,"5min":51.722000000000001,"15min":51.722000000000001},"last":2.0590000000000002},{"interface":"front","average":{"1min":1.069,"5min":1.5049999999999999,"15min":0.94999999999999996},"min":{"1min":0.36199999999999999,"5min":0.31,"15min":0.30599999999999999},"max":{"1min":2.6960000000000002,"5min":51.862000000000002,"15min":51.862000000000002},"last":1.9139999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:41:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.022,"15min":0.78900000000000003},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.28799999999999998},"max":{"1min":2.766,"5min":7.6260000000000003,"15min":7.6260000000000003},"last":1.9350000000000001},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0600000000000001,"15min":0.81799999999999995},"min":{"1min":0.375,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.323,"5min":7.8250000000000002,"15min":7.8250000000000002},"last":1.7230000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117272,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2439120,"kb_used_data":1539208,"kb_used_omap":1,"kb_used_meta":899902,"kb_avail":91306032,"statfs":{"total":95995035648,"available":93497376768,"internally_reserved":0,"allocated":1576148992,"data_stored":1575726341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":921499992},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:42:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.4099999999999999,"15min":1.26},"min":{"1min":0.26600000000000001,"5min":0.19900000000000001,"15min":0.18099999999999999},"max":{"1min":8.8079999999999998,"5min":29.981999999999999,"15min":29.981999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.4670000000000001,"15min":1.343},"min":{"1min":0.23899999999999999,"5min":0.193,"15min":0.193},"max":{"1min":5.0010000000000003,"5min":28.754000000000001,"15min":28.754000000000001},"last":0.23300000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:42:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.5129999999999999,"15min":1.5509999999999999},"min":{"1min":0.20499999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":5.1879999999999997,"5min":28.539000000000001,"15min":35.406999999999996},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.78,"5min":1.458,"15min":1.4770000000000001},"min":{"1min":0.28899999999999998,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":8.8390000000000004,"5min":29.032,"15min":36.902000000000001},"last":0.46100000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:42:26 2024","interfaces":[{"interface":"back","average":{"1min":2.129,"5min":2.1680000000000001,"15min":1.4410000000000001},"min":{"1min":0.371,"5min":0.191,"15min":0.191},"max":{"1min":8.7210000000000001,"5min":79.975999999999999,"15min":79.975999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.9510000000000001,"5min":2.4630000000000001,"15min":1.744},"min":{"1min":0.23400000000000001,"5min":0.22,"15min":0.217},"max":{"1min":11.242000000000001,"5min":78.278000000000006,"15min":78.278000000000006},"last":0.77300000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:41:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.1879999999999999,"15min":1.1899999999999999},"min":{"1min":0.372,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.726,"5min":6.1699999999999999,"15min":6.5250000000000004},"last":1.663},{"interface":"front","average":{"1min":1.036,"5min":1.1619999999999999,"15min":1.1140000000000001},"min":{"1min":0.28399999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.2559999999999998,"5min":4.8499999999999996,"15min":6.1929999999999996},"last":1.696}]},{"osd":5,"last update":"Tue Sep 10 18:42:12 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.0940000000000001,"15min":1.2130000000000001},"min":{"1min":0.33700000000000002,"5min":0.32300000000000001,"15min":0.314},"max":{"1min":2.6960000000000002,"5min":6.2370000000000001,"15min":6.601},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.0940000000000001,"15min":1.1830000000000001},"min":{"1min":0.316,"5min":0.254,"15min":0.254},"max":{"1min":3.3130000000000002,"5min":4.8970000000000002,"15min":6.2249999999999996},"last":0.371}]},{"osd":6,"last update":"Tue Sep 10 18:41:56 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.23,"15min":1.3640000000000001},"min":{"1min":0.40999999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.2850000000000001,"5min":6.8129999999999997,"15min":31.815999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.589,"5min":1.2310000000000001,"15min":1.4219999999999999},"min":{"1min":0.40699999999999997,"5min":0.33200000000000002,"15min":0.315},"max":{"1min":8.7569999999999997,"5min":8.7569999999999997,"15min":31.742000000000001},"last":2.6960000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:41:56 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.1140000000000001,"15min":1.25},"min":{"1min":0.36799999999999999,"5min":0.308,"15min":0.308},"max":{"1min":2.802,"5min":6.2119999999999997,"15min":6.5730000000000004},"last":4.3769999999999998},{"interface":"front","average":{"1min":1.071,"5min":1.1080000000000001,"15min":1.2390000000000001},"min":{"1min":0.34100000000000003,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.4900000000000002,"5min":4.9370000000000003,"15min":12.612},"last":0.40300000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411612,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2403416,"kb_used_data":1507028,"kb_used_omap":1,"kb_used_meta":896254,"kb_avail":91341736,"statfs":{"total":95995035648,"available":93533937664,"internally_reserved":0,"allocated":1543196672,"data_stored":1542826161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917764440},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:42:04 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.157,"15min":1.4099999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.17499999999999999},"max":{"1min":3.5169999999999999,"5min":3.8610000000000002,"15min":43.890000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.5149999999999999,"15min":1.7250000000000001},"min":{"1min":0.223,"5min":0.19700000000000001,"15min":0.19400000000000001},"max":{"1min":2.657,"5min":10.307,"15min":44.331000000000003},"last":0.57399999999999995}]},{"osd":1,"last update":"Tue Sep 10 18:42:04 2024","interfaces":[{"interface":"back","average":{"1min":2.0979999999999999,"5min":1.651,"15min":1.5780000000000001},"min":{"1min":0.27700000000000002,"5min":0.222,"15min":0.193},"max":{"1min":11.154,"5min":11.154,"15min":44.575000000000003},"last":4.6219999999999999},{"interface":"front","average":{"1min":2.1549999999999998,"5min":1.2769999999999999,"15min":1.3400000000000001},"min":{"1min":0.46300000000000002,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":10.637,"5min":10.637,"15min":44.368000000000002},"last":3.4279999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:42:20 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.8280000000000001,"15min":1.7310000000000001},"min":{"1min":0.38800000000000001,"5min":0.23999999999999999,"15min":0.18099999999999999},"max":{"1min":7.9359999999999999,"5min":45.725000000000001,"15min":61.331000000000003},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.47,"5min":2.113,"15min":1.875},"min":{"1min":0.40300000000000002,"5min":0.22600000000000001,"15min":0.214},"max":{"1min":6.734,"5min":45.677,"15min":61.250999999999998},"last":4.3520000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:41:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.653,"15min":1.2470000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.24199999999999999},"max":{"1min":5.3129999999999997,"5min":9.8810000000000002,"15min":12.555999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.512,"15min":1.135},"min":{"1min":0.35099999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.5550000000000002,"5min":19.696000000000002,"15min":19.696000000000002},"last":0.47199999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:42:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.5900000000000001,"15min":1.248},"min":{"1min":0.49299999999999999,"5min":0.249,"15min":0.249},"max":{"1min":3.6400000000000001,"5min":25.177,"15min":25.177},"last":0.499},{"interface":"front","average":{"1min":1.206,"5min":1.4710000000000001,"15min":1.286},"min":{"1min":0.39400000000000002,"5min":0.32000000000000001,"15min":0.317},"max":{"1min":3.665,"5min":24.986999999999998,"15min":24.986999999999998},"last":3.3919999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:41:58 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":2.2890000000000001,"15min":1.484},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":5.2770000000000001,"5min":53.359999999999999,"15min":53.359999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":1.1379999999999999,"5min":2.0630000000000002,"15min":1.3740000000000001},"min":{"1min":0.42599999999999999,"5min":0.32900000000000001,"15min":0.30099999999999999},"max":{"1min":3.6070000000000002,"5min":53.438000000000002,"15min":53.438000000000002},"last":0.86799999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:42:04 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.371,"15min":1.242},"min":{"1min":0.36699999999999999,"5min":0.36499999999999999,"15min":0.34200000000000003},"max":{"1min":3.2599999999999998,"5min":6.1980000000000004,"15min":10.68},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.7,"15min":1.3759999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.311},"max":{"1min":5.2320000000000002,"5min":9.8499999999999996,"15min":12.596},"last":0.746}]}]},{"osd":1,"up_from":13,"seq":55834575137,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2825644,"kb_used_data":1936872,"kb_used_omap":1,"kb_used_meta":888574,"kb_avail":90919508,"statfs":{"total":95995035648,"available":93101576192,"internally_reserved":0,"allocated":1983356928,"data_stored":1982966289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":909900120},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:42:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.0109999999999999,"15min":1.2829999999999999},"min":{"1min":0.185,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":6.4619999999999997,"5min":6.6760000000000002,"15min":28.893999999999998},"last":0.29299999999999998},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.21,"15min":1.3129999999999999},"min":{"1min":0.23300000000000001,"5min":0.19800000000000001,"15min":0.187},"max":{"1min":7.8700000000000001,"5min":7.8700000000000001,"15min":31.074999999999999},"last":0.48499999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:42:04 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3080000000000001,"15min":1.2350000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.19500000000000001},"max":{"1min":11.079000000000001,"5min":11.079000000000001,"15min":11.079000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.546,"5min":1.47,"15min":1.3100000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.20999999999999999},"max":{"1min":6.274,"5min":9.0749999999999993,"15min":12.222},"last":0.42799999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:42:38 2024","interfaces":[{"interface":"back","average":{"1min":1.8169999999999999,"5min":1.1439999999999999,"15min":1.262},"min":{"1min":0.42599999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":14.121},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.637,"5min":1.2889999999999999,"15min":1.2949999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.20499999999999999},"max":{"1min":7.8079999999999998,"5min":7.8079999999999998,"15min":7.8079999999999998},"last":0.61599999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":2.6440000000000001,"15min":1.6639999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.496,"5min":127.86199999999999,"15min":127.86199999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.4199999999999999,"5min":2.766,"15min":1.7529999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.3579999999999997,"5min":127.84399999999999,"15min":127.84399999999999},"last":0.55600000000000005}]},{"osd":5,"last update":"Tue Sep 10 18:42:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":2.7930000000000001,"15min":1.8700000000000001},"min":{"1min":0.372,"5min":0.29799999999999999,"15min":0.27900000000000003},"max":{"1min":5.117,"5min":118.68000000000001,"15min":118.68000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.4299999999999999,"5min":2.7370000000000001,"15min":1.8280000000000001},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":5.0860000000000003,"5min":118.36,"15min":118.36},"last":0.66600000000000004}]},{"osd":6,"last update":"Tue Sep 10 18:42:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":6.9699999999999998,"15min":3.3300000000000001},"min":{"1min":0.41699999999999998,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":4.8730000000000002,"5min":389.52499999999998,"15min":389.52499999999998},"last":0.745},{"interface":"front","average":{"1min":0.96999999999999997,"5min":7.4349999999999996,"15min":3.5790000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.181,"5min":389.69499999999999,"15min":389.69499999999999},"last":0.59599999999999997}]},{"osd":7,"last update":"Tue Sep 10 18:42:04 2024","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.3819999999999999,"15min":1.2989999999999999},"min":{"1min":0.44400000000000001,"5min":0.36399999999999999,"15min":0.32700000000000001},"max":{"1min":5.0119999999999996,"5min":16.244,"15min":16.244},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.331,"15min":1.3320000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":5.4669999999999996,"5min":16.550000000000001,"15min":16.550000000000001},"last":0.68000000000000005}]}]},{"osd":0,"up_from":0,"seq":34359738661,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1960968192,"data_stored":1960968192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1528696832,"data_stored":1528692760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1560870912,"data_stored":1560870912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1335824384,"data_stored":1335820312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":880476160,"data_stored":880476160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1755385856,"data_stored":1755381784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1543110656,"data_stored":1543110656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:42:53.481 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-10T18:42:53.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pg_num 71 2024-09-10T18:42:53.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:53.518 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:53.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:53.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:53 smithi204 ceph-mon[32357]: from='client.36011 v1:172.21.15.28:0/516479012' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:42:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:53 smithi204 ceph-mon[32357]: 4.18 scrub starts 2024-09-10T18:42:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:53 smithi204 ceph-mon[32357]: 4.18 scrub ok 2024-09-10T18:42:53.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:53 smithi204 ceph-mon[32357]: osdmap e413: 8 total, 8 up, 7 in 2024-09-10T18:42:53.619 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:53.661 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:53.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:53.722 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:53.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:53.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:53.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:53.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:53.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:53.941 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:53.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:53.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:54.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:54.078 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:54.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:54.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:54.180 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:54.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:54.221 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:54.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:54.325 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:54.364 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:54.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:54.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:54.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:54 smithi028 ceph-mon[39220]: 5.6 scrub starts 2024-09-10T18:42:54.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:54 smithi028 ceph-mon[39220]: 5.6 scrub ok 2024-09-10T18:42:54.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:54 smithi028 ceph-mon[39220]: 1.e deep-scrub starts 2024-09-10T18:42:54.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:54 smithi028 ceph-mon[39220]: 1.e deep-scrub ok 2024-09-10T18:42:54.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:54 smithi028 ceph-mon[39220]: pgmap v963: 178 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 176 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 986 op/s 2024-09-10T18:42:54.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:54 smithi028 ceph-mon[39220]: 1.1 scrub starts 2024-09-10T18:42:54.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:54 smithi028 ceph-mon[39220]: 1.1 scrub ok 2024-09-10T18:42:54.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:54 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:42:54.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:54 smithi028 ceph-mon[29291]: 5.6 scrub starts 2024-09-10T18:42:54.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:54 smithi028 ceph-mon[29291]: 5.6 scrub ok 2024-09-10T18:42:54.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:54 smithi028 ceph-mon[29291]: 1.e deep-scrub starts 2024-09-10T18:42:54.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:54 smithi028 ceph-mon[29291]: 1.e deep-scrub ok 2024-09-10T18:42:54.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:54 smithi028 ceph-mon[29291]: pgmap v963: 178 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 176 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 986 op/s 2024-09-10T18:42:54.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:54 smithi028 ceph-mon[29291]: 1.1 scrub starts 2024-09-10T18:42:54.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:54 smithi028 ceph-mon[29291]: 1.1 scrub ok 2024-09-10T18:42:54.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:54 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:42:54.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:54.502 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:54.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:54.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:54 smithi204 ceph-mon[32357]: 5.6 scrub starts 2024-09-10T18:42:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:54 smithi204 ceph-mon[32357]: 5.6 scrub ok 2024-09-10T18:42:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:54 smithi204 ceph-mon[32357]: 1.e deep-scrub starts 2024-09-10T18:42:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:54 smithi204 ceph-mon[32357]: 1.e deep-scrub ok 2024-09-10T18:42:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:54 smithi204 ceph-mon[32357]: pgmap v963: 178 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 176 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 986 op/s 2024-09-10T18:42:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:54 smithi204 ceph-mon[32357]: 1.1 scrub starts 2024-09-10T18:42:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:54 smithi204 ceph-mon[32357]: 1.1 scrub ok 2024-09-10T18:42:54.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:54 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:42:54.605 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:54.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:54.645 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:54.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:54.746 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:54.779 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:54.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:54.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:54.881 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:54.924 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:54.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:54.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:55.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:55.076 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:55.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:55.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:55.178 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:55.211 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:55.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:55.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:55.313 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:55.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:55.347 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:55.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:55.448 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:55.482 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:55.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:55.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:55.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:55.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:42:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:42:55.623 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:55.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:55.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:55.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:55.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:55.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:55.871 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:42:55.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:55.911 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:55.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:42:56.013 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:56.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:56.053 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:56.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:56.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-10T18:42:56.126 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:56.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:56.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:56.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:56.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:56.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:42:56.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:56.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:56.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:56 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:56.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:56 smithi028 ceph-mon[29291]: 1.0 scrub starts 2024-09-10T18:42:56.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:56 smithi028 ceph-mon[29291]: 1.0 scrub ok 2024-09-10T18:42:56.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:56 smithi028 ceph-mon[29291]: pgmap v964: 178 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 176 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.32k op/s 2024-09-10T18:42:56.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:56 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:56.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:56 smithi028 ceph-mon[39220]: 1.0 scrub starts 2024-09-10T18:42:56.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:56 smithi028 ceph-mon[39220]: 1.0 scrub ok 2024-09-10T18:42:56.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:56 smithi028 ceph-mon[39220]: pgmap v964: 178 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 176 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.32k op/s 2024-09-10T18:42:56.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:56.490 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:56.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:56.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:56.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:56.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:56 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:42:56.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:56 smithi204 ceph-mon[32357]: 1.0 scrub starts 2024-09-10T18:42:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:56 smithi204 ceph-mon[32357]: 1.0 scrub ok 2024-09-10T18:42:56.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:56 smithi204 ceph-mon[32357]: pgmap v964: 178 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 176 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.32k op/s 2024-09-10T18:42:56.623 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:56.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:56.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:56.725 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:56.762 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:56.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:56.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:56.864 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:42:56.899 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:56.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:56.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:42:57.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:57.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:57.051 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:57.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:57.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:57.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:57.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:57.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:57.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:57.302 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:42:57.346 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:57.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:57.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:57.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:57.478 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:57.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:57.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:57.580 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:57.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:57 smithi204 ceph-mon[32357]: 5.4 scrub starts 2024-09-10T18:42:57.606 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:57 smithi204 ceph-mon[32357]: 5.4 scrub ok 2024-09-10T18:42:57.606 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:57 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3622147090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:42:57.617 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:57.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:57.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:57.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:57.747 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:57.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:57.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:57.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:57 smithi028 ceph-mon[39220]: 5.4 scrub starts 2024-09-10T18:42:57.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:57 smithi028 ceph-mon[39220]: 5.4 scrub ok 2024-09-10T18:42:57.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:57 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3622147090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:42:57.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:57 smithi028 ceph-mon[29291]: 5.4 scrub starts 2024-09-10T18:42:57.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:57 smithi028 ceph-mon[29291]: 5.4 scrub ok 2024-09-10T18:42:57.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:57 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3622147090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:42:57.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:57.878 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:57.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:57.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:57.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:58.015 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:58.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:58.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:58.116 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:58.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:42:58.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:58.159 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:58.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:42:58.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:58.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:58.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:58.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:58.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:58 smithi028 ceph-mon[29291]: 1.11 scrub starts 2024-09-10T18:42:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:58 smithi028 ceph-mon[29291]: 1.11 scrub ok 2024-09-10T18:42:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:58 smithi028 ceph-mon[29291]: pgmap v965: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T18:42:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:58 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:42:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:58 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3622147090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:42:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:42:58 smithi028 ceph-mon[29291]: osdmap e414: 8 total, 8 up, 7 in 2024-09-10T18:42:58.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:58 smithi028 ceph-mon[39220]: 1.11 scrub starts 2024-09-10T18:42:58.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:58 smithi028 ceph-mon[39220]: 1.11 scrub ok 2024-09-10T18:42:58.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:58 smithi028 ceph-mon[39220]: pgmap v965: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T18:42:58.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:58 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:42:58.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:58 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3622147090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:42:58.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:42:58 smithi028 ceph-mon[39220]: osdmap e414: 8 total, 8 up, 7 in 2024-09-10T18:42:58.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:58.565 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:58.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:58.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:58 smithi204 ceph-mon[32357]: 1.11 scrub starts 2024-09-10T18:42:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:58 smithi204 ceph-mon[32357]: 1.11 scrub ok 2024-09-10T18:42:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:58 smithi204 ceph-mon[32357]: pgmap v965: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T18:42:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:58 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:42:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:58 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3622147090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:42:58.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:42:58 smithi204 ceph-mon[32357]: osdmap e414: 8 total, 8 up, 7 in 2024-09-10T18:42:58.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:58.702 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:58.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:58.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:58.804 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:42:58.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:58.837 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:58.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:42:58.938 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:58.974 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:58.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:58.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:59.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:59.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:59.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:59.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:42:59.258 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:59.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:59.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:42:59.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:59.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:59.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:59.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:59.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:59.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:42:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:59.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:42:59.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:59.672 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:42:59.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:42:59.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:42:59.817 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:59.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:42:59.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:42:59.960 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:42:59.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:42:59.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:00.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:00.103 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:00.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:00.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:00.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:00.235 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:00.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:00.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:00.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:00 smithi028 ceph-mon[29291]: 11.5 deep-scrub ok 2024-09-10T18:43:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:00 smithi028 ceph-mon[29291]: pgmap v967: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 800 op/s 2024-09-10T18:43:00.319 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:00 smithi028 ceph-mon[39220]: 11.5 deep-scrub ok 2024-09-10T18:43:00.319 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:00 smithi028 ceph-mon[39220]: pgmap v967: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 800 op/s 2024-09-10T18:43:00.336 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:00.368 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:00.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:00.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:00.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:00.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:00.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:00.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:00 smithi204 ceph-mon[32357]: 11.5 deep-scrub ok 2024-09-10T18:43:00.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:00 smithi204 ceph-mon[32357]: pgmap v967: 178 pgs: 1 active+clean+scrubbing+deep, 177 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 800 op/s 2024-09-10T18:43:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: specified pg_num 71 < pg_num_max 32 2024-09-10T18:43:00.611 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:00.646 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:00.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:00.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:00.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:00.783 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:00.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:00.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:00.885 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:00.985 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:00.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:00.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:01.087 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:01.119 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:01.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:01.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:01.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:01.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:43:01.259 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:01.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:01.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:01.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:01.414 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:01.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:01.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:01.503 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:43:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:43:01] "GET /metrics HTTP/1.1" 200 225007 "" "Prometheus/2.43.0" 2024-09-10T18:43:01.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:01 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3124637372' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "71"}]: dispatch 2024-09-10T18:43:01.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:01 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3124637372' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "71"}]: dispatch 2024-09-10T18:43:01.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:01.564 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:01.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:01.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:01.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:01 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3124637372' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "71"}]: dispatch 2024-09-10T18:43:01.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:01.711 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:01.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:01.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:01.813 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:01.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:01.848 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:01.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:01.949 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:01.980 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:01.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:01.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:02.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:02.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:02.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:02.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:02.265 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:02.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:02.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:02.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:02.401 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:02.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:02.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:02.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:02.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:02.540 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:02.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:02.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:02 smithi204 ceph-mon[32357]: pgmap v968: 178 pgs: 178 active+clean; 2.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-10T18:43:02.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:43:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:43:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:43:02.641 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:02.680 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:02.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:02.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:02.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:02 smithi028 ceph-mon[29291]: pgmap v968: 178 pgs: 178 active+clean; 2.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-10T18:43:02.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:02 smithi028 ceph-mon[39220]: pgmap v968: 178 pgs: 178 active+clean; 2.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-10T18:43:02.781 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:02.821 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:02.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:02.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:02.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:02.956 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:02.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:03.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:03.093 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:03.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:03.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:03.195 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:03.219 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:03.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:03.321 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:03.357 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:03.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:03.457 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:03.492 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:03.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:03.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:03.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:03.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:03.629 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:03.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:03.731 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:03.765 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:03.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:03.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:03.867 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:03.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:03.889 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:03.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:03.990 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:04.025 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:04.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:04.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:04.166 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:04.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:04.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:04.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:04.305 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:04.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:04.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:04.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:04.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:04.446 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:04.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:04.547 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:04.573 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:04.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:04.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:04 smithi204 ceph-mon[32357]: pgmap v969: 178 pgs: 178 active+clean; 2.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T18:43:04.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:04.698 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:04.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:04.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:04.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:04 smithi028 ceph-mon[29291]: pgmap v969: 178 pgs: 178 active+clean; 2.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T18:43:04.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:04 smithi028 ceph-mon[39220]: pgmap v969: 178 pgs: 178 active+clean; 2.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T18:43:04.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:04.831 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:04.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:04.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:04.933 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:04.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:04.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:05.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:05.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:05.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:05.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:05.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:05.240 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:05.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:05.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:05.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:05.376 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:05.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:05.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:05.478 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:05.509 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:05.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:05.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:05.603 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:43:05 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:43:05.611 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:05.644 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:05.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:05.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:05.746 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:05.768 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:05.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:05.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:05.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:05.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:05.906 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:05.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:06.006 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:06.028 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:06.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:06.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:06.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:06.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:06.153 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:06.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:06.254 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:06.284 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:06.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:06.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:06.386 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:06.415 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:06.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:06.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:06.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:06.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:06.555 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:06.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:06.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:06 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:06.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:06 smithi204 ceph-mon[32357]: pgmap v970: 178 pgs: 178 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T18:43:06.655 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:06.688 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:06.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:06.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:06.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:06 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:06.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:06 smithi028 ceph-mon[39220]: pgmap v970: 178 pgs: 178 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T18:43:06.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:06 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:06.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:06 smithi028 ceph-mon[29291]: pgmap v970: 178 pgs: 178 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T18:43:06.790 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:06.826 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:06.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:06.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:06.925 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-10T18:43:06.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:06.971 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:06.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:06.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:07.073 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:07.107 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:07.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:07.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:07.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:07.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:07.251 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:07.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:07.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:07.385 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:07.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:07.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:07.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:07.526 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:07.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:07.621 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:43:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:43:07.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:07.658 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:43:07.659 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:43:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:07.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:07.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:07.689 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:43:07.689 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:43:07.732 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:43:07.732 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:43:07.760 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:43:07.761 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:43:07.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:07.791 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:43:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:43:07.808 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:07.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:07.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:07.821 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:43:07.822 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:43:07.856 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:43:07.856 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:43:07.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:07.957 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:07.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:07.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:08.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:08.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:08.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:08.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:08.200 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:08.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:08.236 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:08.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:08.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:08.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:08.377 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:08.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:08.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:08.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:08 smithi028 ceph-mon[39220]: pgmap v971: 178 pgs: 178 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T18:43:08.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:08 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3887021809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:43:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:08 smithi028 ceph-mon[29291]: pgmap v971: 178 pgs: 178 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T18:43:08.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:08 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3887021809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:43:08.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:08.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:08.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:08 smithi204 ceph-mon[32357]: pgmap v971: 178 pgs: 178 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T18:43:08.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:08 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3887021809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:43:08.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:08.640 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:08.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:08.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:08.742 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:08.777 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:08.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:08.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:08.879 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:08.912 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:08.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:08.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:08.940 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:43:08.941 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:43:09.014 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:09.044 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:09.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:09.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:09.163 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:09.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:09.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:09.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:09.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:09.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:09.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:09.483 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:09.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:09.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:09.585 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:09.604 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:43:09 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a[62900]: logger=cleanup t=2024-09-10T18:43:09.163723333Z level=info msg="Completed cleanup jobs" duration=26.043934ms 2024-09-10T18:43:09.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:09.622 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:09.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:09.724 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:09.756 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:09.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:09.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:09.859 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:09.891 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:09.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:09.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:09.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:10.033 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:10.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:10.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:43:10.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:43:10.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:09 smithi204 ceph-mon[32357]: pgmap v972: 178 pgs: 178 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 857 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-10T18:43:10.134 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:10.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:10.168 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:10.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:10.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:43:10.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:43:10.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:09 smithi028 ceph-mon[29291]: pgmap v972: 178 pgs: 178 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 857 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-10T18:43:10.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:43:10.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:43:10.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:09 smithi028 ceph-mon[39220]: pgmap v972: 178 pgs: 178 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 857 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-10T18:43:10.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:10.296 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:10.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:10.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-10T18:43:10.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:10.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:10.449 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:10.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:10.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:10.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:10.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:10.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:10.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:10.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:10.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:10.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:10.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:10.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:11.023 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:11.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:11.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:11.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:11.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:11.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:11.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:11.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:11.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:11.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:11.604 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:11.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:11.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:11.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:11.753 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:11.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:11.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:11.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:43:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:43:11] "GET /metrics HTTP/1.1" 200 224984 "" "Prometheus/2.43.0" 2024-09-10T18:43:11.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:43:11.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:11.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:11.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:11.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:11.927 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:43:11.927 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:43:11.927 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-10T18:43:11.927 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-10T18:43:11.927 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-10T18:43:11.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-10T18:43:11.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:12.037 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:12.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:12.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:12.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:11 smithi028 ceph-mon[29291]: pgmap v973: 178 pgs: 178 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s 2024-09-10T18:43:12.047 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:11 smithi028 ceph-mon[39220]: pgmap v973: 178 pgs: 178 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s 2024-09-10T18:43:12.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:12.197 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:12.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:12.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:12.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:11 smithi204 ceph-mon[32357]: pgmap v973: 178 pgs: 178 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s 2024-09-10T18:43:12.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:43:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:43:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:43:12.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:12.355 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:12.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:12.456 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:12.493 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:12.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:12.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:12.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:12.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:12.634 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:12.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:12.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:12.763 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:12.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:12.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:12.864 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:12.897 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:12.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:12.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:12.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:13.036 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:13.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:13.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:13.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:13.180 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:13.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:13.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:13.282 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:13.314 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:13.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:13.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:13.416 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:13.450 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:13.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:13.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:13.586 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:13.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:13.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:13.688 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:13.723 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:13.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:13.825 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:13.858 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:13.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:13.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:13.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:14.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:14.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:14.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:14.141 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:14.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:14.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:14.284 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:14.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:14.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:14.306 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-10T18:43:14.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:14.423 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:14.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:14.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:14 smithi028 ceph-mon[29291]: pgmap v974: 178 pgs: 178 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-10T18:43:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:14 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3620174396' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:43:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:14 smithi028 ceph-mon[39220]: pgmap v974: 178 pgs: 178 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-10T18:43:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:14 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3620174396' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:43:14.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:14.563 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:14.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:14.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:14 smithi204 ceph-mon[32357]: pgmap v974: 178 pgs: 178 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-10T18:43:14.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:14 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3620174396' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:43:14.664 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:14.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:14.694 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:14.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:14.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:14.842 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:14.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:14.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:14.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:14.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:14.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:14.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:15.086 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:15.123 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:15.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:15.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:15.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:15.256 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:15.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:15.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:15.358 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:15.393 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:15.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:15.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:15.494 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:15.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:15 smithi028 ceph-mon[39220]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:43:15.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:15 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3620174396' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:43:15.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:15 smithi028 ceph-mon[39220]: osdmap e415: 8 total, 8 up, 7 in 2024-09-10T18:43:15.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:15 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:43:15.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:15 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:43:15.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:15 smithi028 ceph-mon[29291]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:43:15.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:15 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3620174396' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:43:15.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:15 smithi028 ceph-mon[29291]: osdmap e415: 8 total, 8 up, 7 in 2024-09-10T18:43:15.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:15 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:43:15.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:15 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:43:15.527 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:15.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:15.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:15.603 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:43:15 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:43:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:15 smithi204 ceph-mon[32357]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:43:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:15 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3620174396' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:43:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:15 smithi204 ceph-mon[32357]: osdmap e415: 8 total, 8 up, 7 in 2024-09-10T18:43:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:15 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:43:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:15 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:43:15.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:15.666 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:15.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:15.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:15.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:15.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:15.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:15.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:15.959 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:15.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:15.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:16.061 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:16.095 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:16.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:16.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:16.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:16.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:16.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:16.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:16.371 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:16.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:16.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:16.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:16.511 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:16.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:16.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:16.553 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:43:16.553 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:43:16.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:16 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:16.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:16 smithi028 ceph-mon[29291]: pgmap v976: 178 pgs: 178 active+clean; 1.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.46k op/s 2024-09-10T18:43:16.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:16 smithi028 ceph-mon[29291]: osdmap e416: 8 total, 8 up, 7 in 2024-09-10T18:43:16.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:16 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:43:16.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:16 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:43:16.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:16 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:16.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:16 smithi028 ceph-mon[39220]: pgmap v976: 178 pgs: 178 active+clean; 1.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.46k op/s 2024-09-10T18:43:16.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:16 smithi028 ceph-mon[39220]: osdmap e416: 8 total, 8 up, 7 in 2024-09-10T18:43:16.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:16 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:43:16.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:16 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:43:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:16 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:16 smithi204 ceph-mon[32357]: pgmap v976: 178 pgs: 178 active+clean; 1.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.46k op/s 2024-09-10T18:43:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:16 smithi204 ceph-mon[32357]: osdmap e416: 8 total, 8 up, 7 in 2024-09-10T18:43:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:16 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:43:16.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:16 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:43:16.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:16.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:16.650 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:16.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:16.751 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:16.788 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:16.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:16.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:16.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:16.925 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:16.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:16.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:17.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:17.062 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:17.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:17.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:17.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:17.210 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:17.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:17.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:17.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:43:17.305 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:17.311 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:17.339 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:17.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:17.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:17.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:17 smithi028 ceph-mon[29291]: osdmap e417: 8 total, 8 up, 7 in 2024-09-10T18:43:17.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:17.487 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:17.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:17.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:17.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:17.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:17 smithi204 ceph-mon[32357]: osdmap e417: 8 total, 8 up, 7 in 2024-09-10T18:43:17.621 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:17.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:17.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:17.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:17 smithi028 ceph-mon[39220]: osdmap e417: 8 total, 8 up, 7 in 2024-09-10T18:43:17.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:17.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:17.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:17.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:17.902 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:17.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:17.946 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:43:17.947 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:43:17.981 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:43:17.982 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:43:18.004 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:18.013 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:43:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:43:18.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:18.037 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:18.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:18.050 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:43:18.050 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:43:18.083 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:43:18.083 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:43:18.114 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:43:18.114 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:43:18.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:18.146 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:43:18.146 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:43:18.171 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:18.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:18.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:18.179 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:43:18.179 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:43:18.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:18.322 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:18.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:18.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:18.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:18.461 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:18.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:18.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:18.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:18 smithi028 ceph-mon[39220]: pgmap v979: 178 pgs: 178 active+clean; 1.1 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:43:18.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:18 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/140416091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:43:18.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:18 smithi028 ceph-mon[29291]: pgmap v979: 178 pgs: 178 active+clean; 1.1 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:43:18.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:18 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/140416091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:43:18.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:18.598 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:18.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:18.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:18 smithi204 ceph-mon[32357]: pgmap v979: 178 pgs: 178 active+clean; 1.1 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:43:18.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:18 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/140416091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:43:18.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:18.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:18.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:18.877 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:18.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:18.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:18.979 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:19.001 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:19.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:19.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:19.103 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:19.142 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:19.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:19.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:19.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:19.266 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:19.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:19.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:19.368 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:19.401 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:19.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:19.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:19.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:19.544 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:19.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:19.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:19.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:19.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:19.785 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:19.823 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:19.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:19.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:19.924 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:19.947 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:19.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:19.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:20.049 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:20.070 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:20.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:20.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:20.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:20.211 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:20.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:20.313 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:20.345 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:20.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:20.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:20.448 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:20.481 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:20.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:20.583 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:20.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:20 smithi204 ceph-mon[32357]: pgmap v980: 178 pgs: 178 active+clean; 1.1 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:43:20.614 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:20.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:20.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:20.716 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:20.750 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:20.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:20.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:20 smithi028 ceph-mon[39220]: pgmap v980: 178 pgs: 178 active+clean; 1.1 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:43:20.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:20 smithi028 ceph-mon[29291]: pgmap v980: 178 pgs: 178 active+clean; 1.1 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-10T18:43:20.852 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:20.889 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:20.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:20.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:20.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:21.026 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:21.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:21.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:21.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:21.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:21.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:21.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:21.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:21.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:21.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-10T18:43:21.405 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:21.440 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:21.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:21.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:21.542 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:21.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:21.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:21.639 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:43:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:43:21] "GET /metrics HTTP/1.1" 200 224984 "" "Prometheus/2.43.0" 2024-09-10T18:43:21.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:21.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:21.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:21.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:21.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:21.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:21.970 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:22.010 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:22.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:22.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:22.112 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:22.126 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:22.144 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:22.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:22.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:22.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:21 smithi028 ceph-mon[39220]: pgmap v981: 178 pgs: 178 active+clean; 662 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2.21k op/s 2024-09-10T18:43:22.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:21 smithi028 ceph-mon[29291]: pgmap v981: 178 pgs: 178 active+clean; 662 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2.21k op/s 2024-09-10T18:43:22.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:22.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:22.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:21 smithi204 ceph-mon[32357]: pgmap v981: 178 pgs: 178 active+clean; 662 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2.21k op/s 2024-09-10T18:43:22.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:43:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:43:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:43:22.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:22.428 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:22.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:22.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:22.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:22.566 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:22.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:22.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:22.667 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:22.701 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:22.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:22.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:22.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:22.842 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:22.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:22.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:22.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:22.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:43:23.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:23.006 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:23.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:23.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:23.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:23.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:23.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:22 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:43:23.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:22 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:43:23.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:22 smithi028 ceph-mon[29291]: osdmap e418: 8 total, 8 up, 7 in 2024-09-10T18:43:23.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:22 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:43:23.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:22 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:43:23.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:22 smithi028 ceph-mon[39220]: osdmap e418: 8 total, 8 up, 7 in 2024-09-10T18:43:23.260 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:23.286 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:23.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:23.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:22 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:43:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:22 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:43:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:22 smithi204 ceph-mon[32357]: osdmap e418: 8 total, 8 up, 7 in 2024-09-10T18:43:23.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:23.426 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:23.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:23.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:23.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:23.567 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:23.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:23.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:23.669 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:23.694 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:23.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:23.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:23.796 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:23.829 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:23.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:23.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:23.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:23.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:23.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:24.072 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:24.107 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:24.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:24.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:24.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:24.244 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:24.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:24.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:24.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:23 smithi028 ceph-mon[29291]: pgmap v983: 178 pgs: 178 active+clean; 662 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 645 B/s rd, 0 B/s wr, 1.39k op/s 2024-09-10T18:43:24.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:23 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:43:24.252 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:23 smithi028 ceph-mon[39220]: pgmap v983: 178 pgs: 178 active+clean; 662 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 645 B/s rd, 0 B/s wr, 1.39k op/s 2024-09-10T18:43:24.252 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:23 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:43:24.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:24.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:23 smithi204 ceph-mon[32357]: pgmap v983: 178 pgs: 178 active+clean; 662 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 645 B/s rd, 0 B/s wr, 1.39k op/s 2024-09-10T18:43:24.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:23 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:43:24.387 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:24.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:24.489 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:24.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:24.528 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:24.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:24.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:24.696 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:24.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:24.798 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:24.828 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:24.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:24.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:24.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:24.982 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:24.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:24.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:24.991 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:43:24.991 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "" 2024-09-10T18:43:24.991 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:43:25.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:25.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:25.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:25.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:25.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:25.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:25.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:25.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:25.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:25.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:25.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:25.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:25.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:43:25 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:43:25.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:25.686 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:25.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:25.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:25.830 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:25.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:25.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:25.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:25.969 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:25.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:25.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:26.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:26.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:26.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:26.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:26.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:26.243 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:26.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:26.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:26.344 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:26.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:26.374 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:26.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:26.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:26.506 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:26.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:26.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:26 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:26 smithi204 ceph-mon[32357]: pgmap v984: 178 pgs: 178 active+clean; 504 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 574 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:43:26.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:26 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:26.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:26 smithi028 ceph-mon[29291]: pgmap v984: 178 pgs: 178 active+clean; 504 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 574 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:43:26.605 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:26 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:26.605 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:26 smithi028 ceph-mon[39220]: pgmap v984: 178 pgs: 178 active+clean; 504 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 574 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-10T18:43:26.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:26.642 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:26.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:26.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:26.779 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:26.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:26.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:26.881 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:26.916 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:26.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:26.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:27.018 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:27.055 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:27.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:27.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:27.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:27.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:27.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:27.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:27.273 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Removed 140604 objects 2024-09-10T18:43:27.273 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Clean up completed and total clean up time :99.9239 2024-09-10T18:43:27.289 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:27.310 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:27.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:27.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:27.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:27.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:27.434 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:27.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:27.534 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:27.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:27.556 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:27.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:27.657 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:27.678 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:27.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:27.780 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:27.801 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:27.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:27.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:27.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:27.927 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:27.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:27.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:28.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:28.035 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:43:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-10T18:43:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-10T18:43:28.036 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:43:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:28.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:28.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:28.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:28.183 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:28.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:28.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:28.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:43:28.311 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:28.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:28.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:28.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:28.435 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:28.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:28.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:28.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:28 smithi028 ceph-mon[29291]: pgmap v985: 178 pgs: 178 active+clean; 178 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.53k op/s 2024-09-10T18:43:28.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:28 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2426495227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:43:28.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:28 smithi028 ceph-mon[39220]: pgmap v985: 178 pgs: 178 active+clean; 178 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.53k op/s 2024-09-10T18:43:28.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:28 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2426495227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:43:28.537 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:28.563 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:28.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:28.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:28.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:28 smithi204 ceph-mon[32357]: pgmap v985: 178 pgs: 178 active+clean; 178 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.53k op/s 2024-09-10T18:43:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:28 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2426495227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:43:28.665 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:28.690 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:28.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:28.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:28.792 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:28.818 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:28.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:28.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:28.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:28.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:28.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:29.047 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:29.073 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:29.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:29.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:29.175 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:29.200 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:29.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:29.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:29.301 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:29.325 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:29.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:29.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:29.427 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:29.451 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:29.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:29.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:29.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:29.576 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:29.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:29.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:29.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:29 smithi204 ceph-mon[32357]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:43:29.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:29 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2426495227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:43:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:29 smithi204 ceph-mon[32357]: osdmap e419: 8 total, 8 up, 7 in 2024-09-10T18:43:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:29 smithi204 ceph-mon[32357]: 4.9 scrub starts 2024-09-10T18:43:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:29 smithi204 ceph-mon[32357]: 4.9 scrub ok 2024-09-10T18:43:29.678 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:29.702 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:29.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:29.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:29.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:29 smithi028 ceph-mon[29291]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:43:29.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:29 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2426495227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:43:29.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:29 smithi028 ceph-mon[29291]: osdmap e419: 8 total, 8 up, 7 in 2024-09-10T18:43:29.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:29 smithi028 ceph-mon[29291]: 4.9 scrub starts 2024-09-10T18:43:29.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:29 smithi028 ceph-mon[29291]: 4.9 scrub ok 2024-09-10T18:43:29.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:29 smithi028 ceph-mon[39220]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:43:29.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:29 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2426495227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:43:29.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:29 smithi028 ceph-mon[39220]: osdmap e419: 8 total, 8 up, 7 in 2024-09-10T18:43:29.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:29 smithi028 ceph-mon[39220]: 4.9 scrub starts 2024-09-10T18:43:29.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:29 smithi028 ceph-mon[39220]: 4.9 scrub ok 2024-09-10T18:43:29.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:29.832 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:29.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:29.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:29.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:29.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:29.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:29.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:30.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:30.088 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:30.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:30.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:30.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:30.214 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:30.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:30.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:30.316 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:30.342 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:30.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:30.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:30.443 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:30.468 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:30.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:30.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:30 smithi028 ceph-mon[39220]: pgmap v987: 178 pgs: 178 active+clean; 178 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 967 op/s 2024-09-10T18:43:30.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:30 smithi028 ceph-mon[29291]: pgmap v987: 178 pgs: 178 active+clean; 178 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 967 op/s 2024-09-10T18:43:30.570 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:30.594 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:30.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:30.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:30.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:30 smithi204 ceph-mon[32357]: pgmap v987: 178 pgs: 178 active+clean; 178 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 967 op/s 2024-09-10T18:43:30.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:30.722 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:30.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:30.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:30.824 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:30.849 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:30.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:30.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:30.951 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:30.975 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:30.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:30.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:31.024 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:31.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-10T18:43:31.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:31.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:31.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:31.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:31.240 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:31.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:31.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:31.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:31.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:31.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:31.466 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:31.493 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:31.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:31.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:31.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:31.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:31.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:31.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:31.744 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:31.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:31.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:31.753 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:43:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:43:31] "GET /metrics HTTP/1.1" 200 224988 "" "Prometheus/2.43.0" 2024-09-10T18:43:31.778 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:31.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:31.868 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:31.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:31.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:31.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:31.993 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:31.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:31.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:32.095 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:32.119 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:32.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:32.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:32.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:43:32.221 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:32.246 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:32.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:32.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:32.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:31 smithi028 ceph-mon[29291]: 5.1a scrub starts 2024-09-10T18:43:32.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:31 smithi028 ceph-mon[29291]: 5.1a scrub ok 2024-09-10T18:43:32.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:31 smithi028 ceph-mon[29291]: 1.16 scrub starts 2024-09-10T18:43:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:31 smithi028 ceph-mon[29291]: 1.16 scrub ok 2024-09-10T18:43:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:31 smithi028 ceph-mon[29291]: pgmap v988: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-10T18:43:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:31 smithi028 ceph-mon[29291]: 6.e scrub starts 2024-09-10T18:43:32.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:31 smithi028 ceph-mon[29291]: 6.e scrub ok 2024-09-10T18:43:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:31 smithi028 ceph-mon[39220]: 5.1a scrub starts 2024-09-10T18:43:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:31 smithi028 ceph-mon[39220]: 5.1a scrub ok 2024-09-10T18:43:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:31 smithi028 ceph-mon[39220]: 1.16 scrub starts 2024-09-10T18:43:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:31 smithi028 ceph-mon[39220]: 1.16 scrub ok 2024-09-10T18:43:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:31 smithi028 ceph-mon[39220]: pgmap v988: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-10T18:43:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:31 smithi028 ceph-mon[39220]: 6.e scrub starts 2024-09-10T18:43:32.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:31 smithi028 ceph-mon[39220]: 6.e scrub ok 2024-09-10T18:43:32.348 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:32.353 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:43:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:43:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:43:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:31 smithi204 ceph-mon[32357]: 5.1a scrub starts 2024-09-10T18:43:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:31 smithi204 ceph-mon[32357]: 5.1a scrub ok 2024-09-10T18:43:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:31 smithi204 ceph-mon[32357]: 1.16 scrub starts 2024-09-10T18:43:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:31 smithi204 ceph-mon[32357]: 1.16 scrub ok 2024-09-10T18:43:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:31 smithi204 ceph-mon[32357]: pgmap v988: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-10T18:43:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:31 smithi204 ceph-mon[32357]: 6.e scrub starts 2024-09-10T18:43:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:31 smithi204 ceph-mon[32357]: 6.e scrub ok 2024-09-10T18:43:32.373 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:32.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:32.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:32.469 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-10T18:43:32.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-10T18:43:32.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:32.500 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:32.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:32.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:32.602 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:32.627 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:32.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:32.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:32.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:32.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:32.755 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:32.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:32.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:32.881 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:32.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:32.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:33.009 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:33.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:33.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:33.110 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:33.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:33.137 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:33.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:33.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:33.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:32 smithi028 ceph-mon[39220]: 3.12 scrub starts 2024-09-10T18:43:33.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:32 smithi028 ceph-mon[39220]: 3.12 scrub ok 2024-09-10T18:43:33.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:32 smithi028 ceph-mon[29291]: 3.12 scrub starts 2024-09-10T18:43:33.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:32 smithi028 ceph-mon[29291]: 3.12 scrub ok 2024-09-10T18:43:33.260 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:33.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:33.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:33.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:32 smithi204 ceph-mon[32357]: 3.12 scrub starts 2024-09-10T18:43:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:32 smithi204 ceph-mon[32357]: 3.12 scrub ok 2024-09-10T18:43:33.362 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:33.386 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:33.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:33.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:33.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:33.511 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:33.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:33.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:33.613 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:33.638 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:33.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:33.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:33.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:33.765 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:33.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:33.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:33.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:33.891 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:33.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:33.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:33.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:34.015 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:34.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:34.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:34.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:34.143 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:34.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:34.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:34.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:34.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:33 smithi028 ceph-mon[29291]: 11.2 scrub starts 2024-09-10T18:43:34.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:33 smithi028 ceph-mon[29291]: 11.2 scrub ok 2024-09-10T18:43:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:33 smithi028 ceph-mon[29291]: 3.10 scrub starts 2024-09-10T18:43:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:33 smithi028 ceph-mon[29291]: 3.10 scrub ok 2024-09-10T18:43:34.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:33 smithi028 ceph-mon[29291]: pgmap v989: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-10T18:43:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:33 smithi028 ceph-mon[39220]: 11.2 scrub starts 2024-09-10T18:43:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:33 smithi028 ceph-mon[39220]: 11.2 scrub ok 2024-09-10T18:43:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:33 smithi028 ceph-mon[39220]: 3.10 scrub starts 2024-09-10T18:43:34.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:33 smithi028 ceph-mon[39220]: 3.10 scrub ok 2024-09-10T18:43:34.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:33 smithi028 ceph-mon[39220]: pgmap v989: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-10T18:43:34.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:34.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:33 smithi204 ceph-mon[32357]: 11.2 scrub starts 2024-09-10T18:43:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:33 smithi204 ceph-mon[32357]: 11.2 scrub ok 2024-09-10T18:43:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:33 smithi204 ceph-mon[32357]: 3.10 scrub starts 2024-09-10T18:43:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:33 smithi204 ceph-mon[32357]: 3.10 scrub ok 2024-09-10T18:43:34.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:33 smithi204 ceph-mon[32357]: pgmap v989: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-10T18:43:34.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:34.394 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:34.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:34.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:34.422 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:43:34.423 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:43:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:43:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:43:34.434 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:43:34.434 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:43:34.449 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:43:34.449 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:43:34.465 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:43:34.466 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:43:34.481 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:43:34.481 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:43:34.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:34.497 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:43:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:43:34.513 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:43:34.513 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:43:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:34.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:34.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:34.620 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:34.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:34.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:34.744 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:34.770 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:34.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:34.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:34.871 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:34.898 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:34.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:34.909 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:43:34.909 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:43:34.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:35.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:35.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:35.117 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:35.125 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:35.151 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:35.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:35.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:35.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:34 smithi028 ceph-mon[29291]: 1.1b scrub starts 2024-09-10T18:43:35.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:34 smithi028 ceph-mon[29291]: 1.1b scrub ok 2024-09-10T18:43:35.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:34 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3623670819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:43:35.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:34 smithi028 ceph-mon[39220]: 1.1b scrub starts 2024-09-10T18:43:35.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:34 smithi028 ceph-mon[39220]: 1.1b scrub ok 2024-09-10T18:43:35.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:34 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3623670819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:43:35.253 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:35.279 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:35.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:34 smithi204 ceph-mon[32357]: 1.1b scrub starts 2024-09-10T18:43:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:34 smithi204 ceph-mon[32357]: 1.1b scrub ok 2024-09-10T18:43:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:34 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3623670819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:43:35.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:43:35 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:43:35.380 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:35.404 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:35.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:35.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:35.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:35.527 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-10T18:43:35.527 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T18:43:35.528 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-10T18:43:35.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T18:43:35.530 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:35.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:35.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:35.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:35.655 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:35.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:35.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:35.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-10T18:43:35.758 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:35.781 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:35.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:35.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:35.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:35.907 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:35.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:35.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:36.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:36.031 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:36.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:36.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:36.133 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:36.156 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:36.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:36.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:36.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:35 smithi028 ceph-mon[29291]: 5.5 scrub starts 2024-09-10T18:43:36.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:35 smithi028 ceph-mon[29291]: 5.5 scrub ok 2024-09-10T18:43:36.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:35 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:36.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:35 smithi028 ceph-mon[29291]: pgmap v990: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 806 op/s 2024-09-10T18:43:36.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:35 smithi028 ceph-mon[39220]: 5.5 scrub starts 2024-09-10T18:43:36.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:35 smithi028 ceph-mon[39220]: 5.5 scrub ok 2024-09-10T18:43:36.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:35 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:36.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:35 smithi028 ceph-mon[39220]: pgmap v990: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 806 op/s 2024-09-10T18:43:36.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:36.280 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:36.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:36.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:36.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:35 smithi204 ceph-mon[32357]: 5.5 scrub starts 2024-09-10T18:43:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:35 smithi204 ceph-mon[32357]: 5.5 scrub ok 2024-09-10T18:43:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:35 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:35 smithi204 ceph-mon[32357]: pgmap v990: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 806 op/s 2024-09-10T18:43:36.382 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:36.405 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:36.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:36.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:36.507 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:36.529 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:36.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:36.631 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:36.654 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:36.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:36.756 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:36.779 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:36.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:36.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:36.881 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:36.904 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:36.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:36.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:37.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:37.028 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:37.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:37.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:37.130 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:37.152 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:37.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:37.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:37.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:36 smithi028 ceph-mon[39220]: 5.b scrub starts 2024-09-10T18:43:37.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:36 smithi028 ceph-mon[39220]: 5.b scrub ok 2024-09-10T18:43:37.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:36 smithi028 ceph-mon[39220]: 11.3 scrub starts 2024-09-10T18:43:37.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:36 smithi028 ceph-mon[39220]: 11.3 scrub ok 2024-09-10T18:43:37.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:36 smithi028 ceph-mon[39220]: 5.16 scrub starts 2024-09-10T18:43:37.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:36 smithi028 ceph-mon[39220]: 5.16 scrub ok 2024-09-10T18:43:37.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:36 smithi028 ceph-mon[29291]: 5.b scrub starts 2024-09-10T18:43:37.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:36 smithi028 ceph-mon[29291]: 5.b scrub ok 2024-09-10T18:43:37.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:36 smithi028 ceph-mon[29291]: 11.3 scrub starts 2024-09-10T18:43:37.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:36 smithi028 ceph-mon[29291]: 11.3 scrub ok 2024-09-10T18:43:37.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:36 smithi028 ceph-mon[29291]: 5.16 scrub starts 2024-09-10T18:43:37.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:36 smithi028 ceph-mon[29291]: 5.16 scrub ok 2024-09-10T18:43:37.254 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:37.276 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:37.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:37.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:36 smithi204 ceph-mon[32357]: 5.b scrub starts 2024-09-10T18:43:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:36 smithi204 ceph-mon[32357]: 5.b scrub ok 2024-09-10T18:43:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:36 smithi204 ceph-mon[32357]: 11.3 scrub starts 2024-09-10T18:43:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:36 smithi204 ceph-mon[32357]: 11.3 scrub ok 2024-09-10T18:43:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:36 smithi204 ceph-mon[32357]: 5.16 scrub starts 2024-09-10T18:43:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:36 smithi204 ceph-mon[32357]: 5.16 scrub ok 2024-09-10T18:43:37.378 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:37.400 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:37.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:37.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:37.502 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:37.525 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:37.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:37.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:37.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:37.648 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:37.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:37.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:37.753 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:37.774 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:37.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:37.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:37.876 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:37.897 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:37.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:37.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:38.022 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:38.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:38.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:38.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:37 smithi028 ceph-mon[29291]: 5.12 scrub starts 2024-09-10T18:43:38.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:37 smithi028 ceph-mon[29291]: 5.12 scrub ok 2024-09-10T18:43:38.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:37 smithi028 ceph-mon[29291]: pgmap v991: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 284 op/s 2024-09-10T18:43:38.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:37 smithi028 ceph-mon[29291]: 11.d scrub starts 2024-09-10T18:43:38.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:37 smithi028 ceph-mon[29291]: 11.d scrub ok 2024-09-10T18:43:38.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:37 smithi028 ceph-mon[29291]: 11.a scrub starts 2024-09-10T18:43:38.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:37 smithi028 ceph-mon[29291]: 11.a scrub ok 2024-09-10T18:43:38.066 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:37 smithi028 ceph-mon[39220]: 5.12 scrub starts 2024-09-10T18:43:38.066 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:37 smithi028 ceph-mon[39220]: 5.12 scrub ok 2024-09-10T18:43:38.066 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:37 smithi028 ceph-mon[39220]: pgmap v991: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 284 op/s 2024-09-10T18:43:38.066 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:37 smithi028 ceph-mon[39220]: 11.d scrub starts 2024-09-10T18:43:38.066 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:37 smithi028 ceph-mon[39220]: 11.d scrub ok 2024-09-10T18:43:38.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:37 smithi028 ceph-mon[39220]: 11.a scrub starts 2024-09-10T18:43:38.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:37 smithi028 ceph-mon[39220]: 11.a scrub ok 2024-09-10T18:43:38.123 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:38.144 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:38.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:38.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:38.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:38.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:38.273 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:38.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:38.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:37 smithi204 ceph-mon[32357]: 5.12 scrub starts 2024-09-10T18:43:38.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:37 smithi204 ceph-mon[32357]: 5.12 scrub ok 2024-09-10T18:43:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:37 smithi204 ceph-mon[32357]: pgmap v991: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 284 op/s 2024-09-10T18:43:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:37 smithi204 ceph-mon[32357]: 11.d scrub starts 2024-09-10T18:43:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:37 smithi204 ceph-mon[32357]: 11.d scrub ok 2024-09-10T18:43:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:37 smithi204 ceph-mon[32357]: 11.a scrub starts 2024-09-10T18:43:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:37 smithi204 ceph-mon[32357]: 11.a scrub ok 2024-09-10T18:43:38.374 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:38.396 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:38.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:38.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:38.498 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:38.520 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:38.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:38.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:38.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:38.644 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:38.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:38.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:38.768 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:38.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:38.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:38.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:43:38.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:38.893 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:38.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:38.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:43:38.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:39.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:39.018 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:39.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:39.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:39.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:39.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:39.244 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:39.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:38 smithi028 ceph-mon[29291]: 5.1e scrub starts 2024-09-10T18:43:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:38 smithi028 ceph-mon[29291]: 5.1e scrub ok 2024-09-10T18:43:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:38 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3530068802' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:43:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:38 smithi028 ceph-mon[29291]: from='client.25987 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:43:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:38 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:43:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:38 smithi028 ceph-mon[29291]: 2.2 scrub starts 2024-09-10T18:43:39.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:38 smithi028 ceph-mon[29291]: 2.2 scrub ok 2024-09-10T18:43:39.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:38 smithi028 ceph-mon[39220]: 5.1e scrub starts 2024-09-10T18:43:39.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:38 smithi028 ceph-mon[39220]: 5.1e scrub ok 2024-09-10T18:43:39.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:38 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3530068802' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:43:39.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:38 smithi028 ceph-mon[39220]: from='client.25987 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:43:39.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:38 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:43:39.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:38 smithi028 ceph-mon[39220]: 2.2 scrub starts 2024-09-10T18:43:39.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:38 smithi028 ceph-mon[39220]: 2.2 scrub ok 2024-09-10T18:43:39.265 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:39.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:39.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:38 smithi204 ceph-mon[32357]: 5.1e scrub starts 2024-09-10T18:43:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:38 smithi204 ceph-mon[32357]: 5.1e scrub ok 2024-09-10T18:43:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:38 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3530068802' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:43:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:38 smithi204 ceph-mon[32357]: from='client.25987 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:43:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:38 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:43:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:38 smithi204 ceph-mon[32357]: 2.2 scrub starts 2024-09-10T18:43:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:38 smithi204 ceph-mon[32357]: 2.2 scrub ok 2024-09-10T18:43:39.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:39.388 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:39.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:39.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:39.490 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:39.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:39.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:39.614 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:39.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:39.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:39.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:39.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:39.761 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:39.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:39.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:39.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:39.886 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:39.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:39.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:39.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:40.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:40.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:40.112 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:40.134 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:40.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:40.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:43:40.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:43:40.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[39220]: from='client.25987 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:43:40.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[39220]: osdmap e420: 8 total, 8 up, 7 in 2024-09-10T18:43:40.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:43:40.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[39220]: pgmap v993: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 284 op/s 2024-09-10T18:43:40.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[39220]: 5.1 scrub starts 2024-09-10T18:43:40.213 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[39220]: 5.1 scrub ok 2024-09-10T18:43:40.213 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:43:40.213 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:43:40.213 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:43:40.213 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[39220]: 11.b scrub starts 2024-09-10T18:43:40.213 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[39220]: 11.b scrub ok 2024-09-10T18:43:40.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:43:40.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:43:40.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[29291]: from='client.25987 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:43:40.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[29291]: osdmap e420: 8 total, 8 up, 7 in 2024-09-10T18:43:40.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:43:40.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[29291]: pgmap v993: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 284 op/s 2024-09-10T18:43:40.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[29291]: 5.1 scrub starts 2024-09-10T18:43:40.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[29291]: 5.1 scrub ok 2024-09-10T18:43:40.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:43:40.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:43:40.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:43:40.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[29291]: 11.b scrub starts 2024-09-10T18:43:40.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:39 smithi028 ceph-mon[29291]: 11.b scrub ok 2024-09-10T18:43:40.235 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:40.257 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:40.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:39 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:43:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:39 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:43:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:39 smithi204 ceph-mon[32357]: from='client.25987 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:43:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:39 smithi204 ceph-mon[32357]: osdmap e420: 8 total, 8 up, 7 in 2024-09-10T18:43:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:43:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:39 smithi204 ceph-mon[32357]: pgmap v993: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 284 op/s 2024-09-10T18:43:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:39 smithi204 ceph-mon[32357]: 5.1 scrub starts 2024-09-10T18:43:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:39 smithi204 ceph-mon[32357]: 5.1 scrub ok 2024-09-10T18:43:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:43:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:43:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:39 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:43:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:39 smithi204 ceph-mon[32357]: 11.b scrub starts 2024-09-10T18:43:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:39 smithi204 ceph-mon[32357]: 11.b scrub ok 2024-09-10T18:43:40.358 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:40.379 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:40.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:40.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:40.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:40.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:40.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:40.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:40.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:40.628 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:40.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:40.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:40.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:40.752 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:40.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:40.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:40.853 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:40.875 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:40.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:40.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:40.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:40.998 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:40.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:40.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:41.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:41.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:41.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:41.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:41.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:41.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:40 smithi028 ceph-mon[39220]: 1.f scrub starts 2024-09-10T18:43:41.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:40 smithi028 ceph-mon[39220]: 1.f scrub ok 2024-09-10T18:43:41.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:40 smithi028 ceph-mon[39220]: 4.c scrub starts 2024-09-10T18:43:41.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:40 smithi028 ceph-mon[39220]: 4.c scrub ok 2024-09-10T18:43:41.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:40 smithi028 ceph-mon[39220]: 3.2 scrub starts 2024-09-10T18:43:41.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:40 smithi028 ceph-mon[39220]: 3.2 scrub ok 2024-09-10T18:43:41.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:40 smithi028 ceph-mon[29291]: 1.f scrub starts 2024-09-10T18:43:41.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:40 smithi028 ceph-mon[29291]: 1.f scrub ok 2024-09-10T18:43:41.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:40 smithi028 ceph-mon[29291]: 4.c scrub starts 2024-09-10T18:43:41.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:40 smithi028 ceph-mon[29291]: 4.c scrub ok 2024-09-10T18:43:41.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:40 smithi028 ceph-mon[29291]: 3.2 scrub starts 2024-09-10T18:43:41.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:40 smithi028 ceph-mon[29291]: 3.2 scrub ok 2024-09-10T18:43:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:41.248 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:41.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:41.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:43:41.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:40 smithi204 ceph-mon[32357]: 1.f scrub starts 2024-09-10T18:43:41.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:40 smithi204 ceph-mon[32357]: 1.f scrub ok 2024-09-10T18:43:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:40 smithi204 ceph-mon[32357]: 4.c scrub starts 2024-09-10T18:43:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:40 smithi204 ceph-mon[32357]: 4.c scrub ok 2024-09-10T18:43:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:40 smithi204 ceph-mon[32357]: 3.2 scrub starts 2024-09-10T18:43:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:40 smithi204 ceph-mon[32357]: 3.2 scrub ok 2024-09-10T18:43:41.372 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:41.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:41.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:43:41.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:41.498 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:41.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:41.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:41.503 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:43:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:43:41] "GET /metrics HTTP/1.1" 200 224837 "" "Prometheus/2.43.0" 2024-09-10T18:43:41.600 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:41.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:41.623 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:41.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:41.725 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:43:41.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:41.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:41.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:43:41.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:41.869 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:41.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:41.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:41.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:43:41.995 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:41.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:41.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:43:42.002 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_4' removed 2024-09-10T18:43:42.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:42.121 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:42.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:42.222 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:43:42.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:43:42.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:41 smithi028 ceph-mon[39220]: 6.4 scrub starts 2024-09-10T18:43:42.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:41 smithi028 ceph-mon[39220]: 6.4 scrub ok 2024-09-10T18:43:42.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:41 smithi028 ceph-mon[39220]: pgmap v994: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:43:42.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:41 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3243129426' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:43:42.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:41 smithi028 ceph-mon[39220]: from='client.36062 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:43:42.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:41 smithi028 ceph-mon[39220]: 1.17 scrub starts 2024-09-10T18:43:42.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:41 smithi028 ceph-mon[39220]: 1.17 scrub ok 2024-09-10T18:43:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:41 smithi028 ceph-mon[29291]: 6.4 scrub starts 2024-09-10T18:43:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:41 smithi028 ceph-mon[29291]: 6.4 scrub ok 2024-09-10T18:43:42.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:41 smithi028 ceph-mon[29291]: pgmap v994: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:43:42.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:41 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3243129426' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:43:42.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:41 smithi028 ceph-mon[29291]: from='client.36062 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:43:42.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:41 smithi028 ceph-mon[29291]: 1.17 scrub starts 2024-09-10T18:43:42.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:41 smithi028 ceph-mon[29291]: 1.17 scrub ok 2024-09-10T18:43:42.268 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:42.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:42.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:43:42.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:43:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:43:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:43:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:41 smithi204 ceph-mon[32357]: 6.4 scrub starts 2024-09-10T18:43:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:41 smithi204 ceph-mon[32357]: 6.4 scrub ok 2024-09-10T18:43:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:41 smithi204 ceph-mon[32357]: pgmap v994: 178 pgs: 178 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:43:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:41 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3243129426' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:43:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:41 smithi204 ceph-mon[32357]: from='client.36062 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:43:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:41 smithi204 ceph-mon[32357]: 1.17 scrub starts 2024-09-10T18:43:42.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:41 smithi204 ceph-mon[32357]: 1.17 scrub ok 2024-09-10T18:43:42.370 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:43:42.391 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:42.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:42.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:43:42.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:43:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:42.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:42.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:43:42.618 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:43:42.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:42.641 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-10T18:43:42.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:43:42.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:43:42.747 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-10T18:43:42.758 INFO:tasks.thrashosds:joining thrashosds 2024-09-10T18:43:42.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:43:42.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-09-10T18:43:42.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:43:43.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:42 smithi028 ceph-mon[29291]: from='client.36062 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:43:43.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:42 smithi028 ceph-mon[29291]: osdmap e421: 8 total, 8 up, 7 in 2024-09-10T18:43:43.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:42 smithi028 ceph-mon[29291]: 6.1d scrub starts 2024-09-10T18:43:43.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:42 smithi028 ceph-mon[29291]: 6.1d scrub ok 2024-09-10T18:43:43.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:42 smithi028 ceph-mon[29291]: 6.11 scrub starts 2024-09-10T18:43:43.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:42 smithi028 ceph-mon[29291]: 6.11 scrub ok 2024-09-10T18:43:43.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:42 smithi028 ceph-mon[39220]: from='client.36062 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:43:43.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:42 smithi028 ceph-mon[39220]: osdmap e421: 8 total, 8 up, 7 in 2024-09-10T18:43:43.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:42 smithi028 ceph-mon[39220]: 6.1d scrub starts 2024-09-10T18:43:43.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:42 smithi028 ceph-mon[39220]: 6.1d scrub ok 2024-09-10T18:43:43.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:42 smithi028 ceph-mon[39220]: 6.11 scrub starts 2024-09-10T18:43:43.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:42 smithi028 ceph-mon[39220]: 6.11 scrub ok 2024-09-10T18:43:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:42 smithi204 ceph-mon[32357]: from='client.36062 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:43:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:42 smithi204 ceph-mon[32357]: osdmap e421: 8 total, 8 up, 7 in 2024-09-10T18:43:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:42 smithi204 ceph-mon[32357]: 6.1d scrub starts 2024-09-10T18:43:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:42 smithi204 ceph-mon[32357]: 6.1d scrub ok 2024-09-10T18:43:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:42 smithi204 ceph-mon[32357]: 6.11 scrub starts 2024-09-10T18:43:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:42 smithi204 ceph-mon[32357]: 6.11 scrub ok 2024-09-10T18:43:44.191 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-10T18:43:44.191 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:43:44.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[39220]: 4.a scrub starts 2024-09-10T18:43:44.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[39220]: 4.a scrub ok 2024-09-10T18:43:44.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:43:44.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:43:44.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[39220]: pgmap v996: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:43:44.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[39220]: 1.5 scrub starts 2024-09-10T18:43:44.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[39220]: 1.5 scrub ok 2024-09-10T18:43:44.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[39220]: 3.1e scrub starts 2024-09-10T18:43:44.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[39220]: 3.1e scrub ok 2024-09-10T18:43:44.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[29291]: 4.a scrub starts 2024-09-10T18:43:44.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[29291]: 4.a scrub ok 2024-09-10T18:43:44.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:43:44.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:43:44.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[29291]: pgmap v996: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:43:44.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[29291]: 1.5 scrub starts 2024-09-10T18:43:44.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[29291]: 1.5 scrub ok 2024-09-10T18:43:44.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[29291]: 3.1e scrub starts 2024-09-10T18:43:44.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:44 smithi028 ceph-mon[29291]: 3.1e scrub ok 2024-09-10T18:43:44.241 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:43:44.241 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-10T18:43:44.241 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 422, 2024-09-10T18:43:44.241 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "5eababb2-6fa0-11ef-bcea-c7b262605968", 2024-09-10T18:43:44.241 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-10T18:16:20.680373+0000", 2024-09-10T18:43:44.241 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-10T18:43:43.995648+0000", 2024-09-10T18:43:44.242 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-09-10T18:26:26.611338+0000", 2024-09-10T18:43:44.242 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-09-10T18:39:24.201588+0000", 2024-09-10T18:43:44.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T18:43:44.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5799936, 2024-09-10T18:43:44.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-09-10T18:43:44.242 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-09-10T18:43:44.242 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-09-10T18:43:44.242 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-09-10T18:43:44.242 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-09-10T18:43:44.242 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:43:44.242 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 19, 2024-09-10T18:43:44.242 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T18:43:44.242 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T18:43:44.242 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T18:43:44.242 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-09-10T18:43:44.243 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 11, 2024-09-10T18:43:44.243 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-09-10T18:43:44.243 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-09-10T18:43:44.243 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-09-10T18:43:44.243 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "quincy", 2024-09-10T18:43:44.243 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-09-10T18:43:44.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.243 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-09-10T18:43:44.243 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-09-10T18:43:44.243 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:19:24.013536+0000", 2024-09-10T18:43:44.243 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:43:44.243 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:43:44.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:43:44.243 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:43:44.243 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:43:44.244 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:43:44.244 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:43:44.244 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:43:44.244 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:43:44.244 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:43:44.244 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:43:44.244 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:43:44.244 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 31, 2024-09-10T18:43:44.244 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 31, 2024-09-10T18:43:44.244 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 31, 2024-09-10T18:43:44.244 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 31, 2024-09-10T18:43:44.244 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 31, 2024-09-10T18:43:44.244 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:43:44.244 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:43:44.245 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:43:44.245 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:43:44.245 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:43:44.245 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:43:44.245 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:43:44.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.245 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "300", 2024-09-10T18:43:44.245 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:43:44.245 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:43:44.245 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "287", 2024-09-10T18:43:44.245 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:43:44.245 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:43:44.245 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:43:44.245 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:43:44.245 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:43:44.246 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:43:44.246 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:43:44.246 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:43:44.246 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:43:44.246 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:43:44.246 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:43:44.246 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:43:44.246 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:43:44.246 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:43:44.246 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:43:44.246 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:43:44.246 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:43:44.246 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:43:44.246 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:43:44.247 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:43:44.247 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:43:44.247 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:43:44.247 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:43:44.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.247 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:43:44.247 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:43:44.247 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:43:44.247 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:43:44.247 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:43:44.247 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:43:44.247 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:43:44.247 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:43:44.247 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:43:44.248 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:43:44.248 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:43:44.248 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-10T18:43:44.248 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-09-10T18:43:44.248 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-09-10T18:43:44.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.248 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:43:44.248 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-09-10T18:43:44.248 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.248 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-09-10T18:43:44.248 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-09-10T18:43:44.248 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:30.065209+0000", 2024-09-10T18:43:44.248 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-09-10T18:43:44.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T18:43:44.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:43:44.249 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:43:44.249 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:43:44.249 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:43:44.249 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:43:44.249 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:43:44.249 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:43:44.249 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:43:44.249 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:43:44.249 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:43:44.249 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-09-10T18:43:44.249 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-09-10T18:43:44.249 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-09-10T18:43:44.250 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-09-10T18:43:44.250 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-09-10T18:43:44.250 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:43:44.250 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:43:44.250 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:43:44.250 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:43:44.250 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:43:44.250 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:43:44.250 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:43:44.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.250 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "62", 2024-09-10T18:43:44.250 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:43:44.250 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:43:44.250 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:43:44.250 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:43:44.250 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:43:44.251 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 3, 2024-09-10T18:43:44.251 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 62, 2024-09-10T18:43:44.251 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:43:44.251 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:43:44.251 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:43:44.251 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:43:44.251 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:43:44.251 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:43:44.251 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:43:44.251 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:43:44.251 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:43:44.251 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:43:44.251 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:43:44.251 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:43:44.252 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:43:44.252 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:43:44.252 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:43:44.252 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:43:44.252 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:43:44.252 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:43:44.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:43:44.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.252 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:43:44.252 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:43:44.252 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:43:44.252 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:43:44.252 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:43:44.252 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:43:44.252 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:43:44.253 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:43:44.253 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:43:44.253 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:43:44.253 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:43:44.253 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:43:44.253 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:43:44.253 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-09-10T18:43:44.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.253 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-09-10T18:43:44.253 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-09-10T18:43:44.253 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:30.506441+0000", 2024-09-10T18:43:44.253 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:43:44.253 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:43:44.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:43:44.254 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:43:44.254 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:43:44.254 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:43:44.254 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:43:44.254 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:43:44.254 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:43:44.254 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:43:44.254 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:43:44.254 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:43:44.254 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:43:44.254 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:43:44.254 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:43:44.254 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:43:44.254 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:43:44.255 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:43:44.255 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:43:44.255 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:43:44.255 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:43:44.255 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:43:44.255 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:43:44.255 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:43:44.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.255 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "59", 2024-09-10T18:43:44.255 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:43:44.255 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:43:44.255 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:43:44.255 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:43:44.255 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:43:44.255 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:43:44.256 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:43:44.256 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:43:44.256 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:43:44.256 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:43:44.256 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:43:44.256 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:43:44.256 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:43:44.256 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:43:44.256 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:43:44.256 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:43:44.256 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:43:44.256 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:43:44.256 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:43:44.256 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:43:44.256 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:43:44.257 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:43:44.257 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:43:44.257 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:43:44.257 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:43:44.257 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:43:44.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.257 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:43:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:43:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:43:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:43:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:43:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:43:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:43:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:43:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:43:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:43:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:43:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:43:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:43:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:43:44.258 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-09-10T18:43:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-09-10T18:43:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:32.008130+0000", 2024-09-10T18:43:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:43:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:43:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:43:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:43:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:43:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:43:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:43:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:43:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:43:44.259 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:43:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:43:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:43:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:43:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:43:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:43:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:43:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:43:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:43:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:43:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:43:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:43:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:43:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:43:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:43:44.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "61", 2024-09-10T18:43:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:43:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:43:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:43:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:43:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:43:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:43:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:43:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:43:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:43:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:43:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:43:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:43:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:43:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:43:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:43:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:43:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:43:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:43:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:43:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:43:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:43:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:43:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:43:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:43:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:43:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:43:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.262 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:43:44.263 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:43:44.263 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:43:44.263 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:43:44.263 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:43:44.263 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:43:44.263 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:43:44.263 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:43:44.263 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:43:44.263 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:43:44.263 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:43:44.263 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:43:44.263 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:43:44.263 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:43:44.263 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.264 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-09-10T18:43:44.264 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-09-10T18:43:44.264 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:34.060571+0000", 2024-09-10T18:43:44.264 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:43:44.264 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:43:44.264 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:43:44.264 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:43:44.264 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:43:44.264 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:43:44.264 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:43:44.264 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:43:44.264 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:43:44.264 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:43:44.264 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:43:44.264 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:43:44.265 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:43:44.265 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:43:44.265 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:43:44.265 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:43:44.265 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:43:44.265 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:43:44.265 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:43:44.265 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:43:44.265 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:43:44.265 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:43:44.265 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:43:44.265 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:43:44.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.265 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "63", 2024-09-10T18:43:44.265 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:43:44.266 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:43:44.266 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:43:44.266 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:43:44.266 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:43:44.266 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:43:44.266 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:43:44.266 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:43:44.266 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:43:44.266 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:43:44.266 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:43:44.266 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:43:44.266 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:43:44.266 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:43:44.266 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:43:44.266 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:43:44.266 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:43:44.267 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:43:44.267 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:43:44.267 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:43:44.267 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:43:44.267 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:43:44.267 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:43:44.267 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:43:44.267 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:43:44.267 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:43:44.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.267 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:43:44.267 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:43:44.267 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:43:44.267 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:43:44.267 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:43:44.268 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:43:44.268 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:43:44.268 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:43:44.268 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:43:44.268 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:43:44.268 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:43:44.268 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-10T18:43:44.268 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:43:44.268 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:43:44.268 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.268 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-09-10T18:43:44.268 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-09-10T18:43:44.268 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-10T18:21:36.127330+0000", 2024-09-10T18:43:44.269 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-10T18:43:44.269 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-10T18:43:44.269 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-10T18:43:44.269 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-09-10T18:43:44.269 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-09-10T18:43:44.269 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-10T18:43:44.269 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-10T18:43:44.269 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-10T18:43:44.269 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T18:43:44.270 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T18:43:44.270 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-10T18:43:44.270 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-10T18:43:44.270 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-09-10T18:43:44.270 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-09-10T18:43:44.270 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-09-10T18:43:44.270 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-09-10T18:43:44.270 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-09-10T18:43:44.270 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-10T18:43:44.270 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-10T18:43:44.270 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-10T18:43:44.270 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-10T18:43:44.270 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-10T18:43:44.270 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-10T18:43:44.270 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-10T18:43:44.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.271 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "65", 2024-09-10T18:43:44.271 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-10T18:43:44.271 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T18:43:44.271 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T18:43:44.271 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-10T18:43:44.271 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-10T18:43:44.271 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-10T18:43:44.271 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-10T18:43:44.271 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-10T18:43:44.271 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-10T18:43:44.271 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-10T18:43:44.271 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-10T18:43:44.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-10T18:43:44.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-10T18:43:44.272 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-10T18:43:44.272 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-10T18:43:44.272 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-10T18:43:44.272 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-10T18:43:44.272 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-10T18:43:44.272 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T18:43:44.272 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T18:43:44.272 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T18:43:44.272 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-10T18:43:44.272 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-10T18:43:44.272 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-10T18:43:44.272 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-10T18:43:44.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-10T18:43:44.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.273 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-10T18:43:44.273 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-10T18:43:44.273 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-10T18:43:44.273 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-10T18:43:44.273 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-10T18:43:44.273 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T18:43:44.273 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-10T18:43:44.273 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-10T18:43:44.273 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-10T18:43:44.273 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-10T18:43:44.273 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-10T18:43:44.273 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-10T18:43:44.273 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-09-10T18:43:44.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.274 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-10T18:43:44.274 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-09-10T18:43:44.274 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.274 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.274 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:43:44.274 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:43:44.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.274 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-10T18:43:44.274 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "f1c834d9-252a-4077-b25e-169181d70251", 2024-09-10T18:43:44.274 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:43:44.274 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 0, 2024-09-10T18:43:44.274 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-10T18:43:44.274 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:43:44.274 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:43:44.275 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:43:44.275 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-09-10T18:43:44.275 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 356, 2024-09-10T18:43:44.275 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:43:44.275 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:43:44.275 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:43:44.275 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.275 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.275 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-09-10T18:43:44.275 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:43:44.275 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.275 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.276 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:43:44.276 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.276 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-09-10T18:43:44.276 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:43:44.276 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.276 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.276 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:43:44.276 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.277 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-09-10T18:43:44.277 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:43:44.277 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.277 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.277 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:43:44.277 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.277 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.277 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-09-10T18:43:44.277 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3188372351 2024-09-10T18:43:44.277 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.277 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.278 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/3188372351", 2024-09-10T18:43:44.278 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/3188372351", 2024-09-10T18:43:44.278 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/3188372351", 2024-09-10T18:43:44.278 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/3188372351", 2024-09-10T18:43:44.278 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:43:44.278 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:43:44.278 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:43:44.278 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.278 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-10T18:43:44.278 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "81f1855d-b946-4ca3-b246-38977b9e20fd", 2024-09-10T18:43:44.278 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:43:44.278 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:43:44.278 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:43:44.279 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:43:44.279 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:43:44.279 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:43:44.279 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-10T18:43:44.279 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 374, 2024-09-10T18:43:44.279 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:43:44.279 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:43:44.279 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:43:44.279 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.279 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.279 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-09-10T18:43:44.279 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:43:44.279 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.280 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.280 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:43:44.280 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.280 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.280 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-09-10T18:43:44.280 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:43:44.280 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.280 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.280 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:43:44.280 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.280 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.280 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-09-10T18:43:44.281 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:43:44.281 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.281 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.281 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:43:44.281 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.281 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.281 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-09-10T18:43:44.281 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1369912678 2024-09-10T18:43:44.281 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.281 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.281 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1369912678", 2024-09-10T18:43:44.282 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1369912678", 2024-09-10T18:43:44.282 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1369912678", 2024-09-10T18:43:44.282 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1369912678", 2024-09-10T18:43:44.282 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:43:44.282 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:43:44.282 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:43:44.282 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.282 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-10T18:43:44.282 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "2ad55d2a-36fa-4756-bf22-58d4b049486e", 2024-09-10T18:43:44.282 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:43:44.282 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:43:44.283 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:43:44.283 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:43:44.283 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:43:44.283 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:43:44.283 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-09-10T18:43:44.283 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 392, 2024-09-10T18:43:44.283 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:43:44.283 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:43:44.283 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:43:44.283 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.283 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.283 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-09-10T18:43:44.283 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:43:44.283 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.284 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.284 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:43:44.284 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.284 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.284 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-09-10T18:43:44.284 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:43:44.284 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.284 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.284 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:43:44.284 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.285 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-09-10T18:43:44.285 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:43:44.285 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.285 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.285 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:43:44.285 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.285 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-09-10T18:43:44.285 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1243406003 2024-09-10T18:43:44.285 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.286 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.286 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/1243406003", 2024-09-10T18:43:44.286 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/1243406003", 2024-09-10T18:43:44.286 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/1243406003", 2024-09-10T18:43:44.286 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1243406003", 2024-09-10T18:43:44.286 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:43:44.286 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:43:44.286 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:43:44.286 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.286 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-10T18:43:44.286 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "242729a5-a5e8-44d5-97c8-14d14e970bff", 2024-09-10T18:43:44.286 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:43:44.287 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:43:44.287 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:43:44.287 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:43:44.287 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:43:44.287 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:43:44.287 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-09-10T18:43:44.287 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 374, 2024-09-10T18:43:44.287 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:43:44.287 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:43:44.287 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:43:44.287 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.287 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.288 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-09-10T18:43:44.288 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:43:44.288 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.288 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.288 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:43:44.288 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.288 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.288 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-09-10T18:43:44.288 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:43:44.288 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.288 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.289 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:43:44.289 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.289 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.289 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-09-10T18:43:44.289 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:43:44.289 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.289 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.289 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:43:44.289 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.289 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.289 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-09-10T18:43:44.290 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 67703283 2024-09-10T18:43:44.290 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.290 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.290 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/67703283", 2024-09-10T18:43:44.290 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/67703283", 2024-09-10T18:43:44.290 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/67703283", 2024-09-10T18:43:44.290 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/67703283", 2024-09-10T18:43:44.290 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:43:44.290 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:43:44.290 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:43:44.290 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.290 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-10T18:43:44.291 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "6e38b8ef-e6ed-493f-8eca-755354754787", 2024-09-10T18:43:44.291 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:43:44.291 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:43:44.291 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:43:44.291 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:43:44.291 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:43:44.291 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:43:44.291 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 33, 2024-09-10T18:43:44.291 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 392, 2024-09-10T18:43:44.291 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:43:44.291 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:43:44.291 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:43:44.291 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.291 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.292 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.292 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6800", 2024-09-10T18:43:44.292 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:43:44.292 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.292 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.292 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:43:44.292 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.292 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.292 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6801", 2024-09-10T18:43:44.292 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:43:44.293 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.293 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.293 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:43:44.293 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.293 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.293 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6803", 2024-09-10T18:43:44.293 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:43:44.293 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.293 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.293 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:43:44.293 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.294 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.294 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6802", 2024-09-10T18:43:44.294 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1619669076 2024-09-10T18:43:44.294 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.294 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.294 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6800/1619669076", 2024-09-10T18:43:44.294 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6801/1619669076", 2024-09-10T18:43:44.294 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6803/1619669076", 2024-09-10T18:43:44.294 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6802/1619669076", 2024-09-10T18:43:44.294 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:43:44.294 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:43:44.294 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:43:44.294 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.295 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-10T18:43:44.295 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "a92e0441-66c9-498e-a0ac-6465acdbb1f5", 2024-09-10T18:43:44.295 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:43:44.295 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:43:44.295 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:43:44.295 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:43:44.295 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:43:44.295 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:43:44.295 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 39, 2024-09-10T18:43:44.295 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 374, 2024-09-10T18:43:44.295 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:43:44.295 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:43:44.295 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:43:44.295 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.296 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.296 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6804", 2024-09-10T18:43:44.296 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:43:44.296 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.296 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.296 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:43:44.296 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.296 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.296 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6805", 2024-09-10T18:43:44.296 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:43:44.296 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.296 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.297 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:43:44.297 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.297 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6807", 2024-09-10T18:43:44.297 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:43:44.297 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.297 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.297 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:43:44.297 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.298 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6806", 2024-09-10T18:43:44.298 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2717802880 2024-09-10T18:43:44.298 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.298 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.298 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6804/2717802880", 2024-09-10T18:43:44.298 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6805/2717802880", 2024-09-10T18:43:44.298 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6807/2717802880", 2024-09-10T18:43:44.298 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6806/2717802880", 2024-09-10T18:43:44.298 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:43:44.298 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:43:44.298 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:43:44.298 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.299 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-09-10T18:43:44.299 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "12b02875-6653-45ed-8d2b-b4d33bd88525", 2024-09-10T18:43:44.299 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:43:44.299 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:43:44.299 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:43:44.299 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:43:44.299 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 46, 2024-09-10T18:43:44.299 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 94, 2024-09-10T18:43:44.299 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 144, 2024-09-10T18:43:44.299 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 374, 2024-09-10T18:43:44.299 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 95, 2024-09-10T18:43:44.299 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:43:44.299 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:43:44.300 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.300 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.300 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6808", 2024-09-10T18:43:44.300 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:43:44.300 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.300 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.300 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:43:44.300 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.300 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.300 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6809", 2024-09-10T18:43:44.301 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:43:44.301 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.301 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.301 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:43:44.301 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.301 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.301 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6811", 2024-09-10T18:43:44.301 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:43:44.301 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.301 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.301 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:43:44.302 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.302 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.302 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6810", 2024-09-10T18:43:44.302 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3479311452 2024-09-10T18:43:44.302 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.302 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.302 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6808/3479311452", 2024-09-10T18:43:44.302 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6809/3479311452", 2024-09-10T18:43:44.302 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6811/3479311452", 2024-09-10T18:43:44.302 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6810/3479311452", 2024-09-10T18:43:44.302 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:43:44.302 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:43:44.302 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:43:44.302 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.303 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-09-10T18:43:44.303 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "0c4f7aa9-ff95-4192-a146-ca576eee907f", 2024-09-10T18:43:44.303 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-10T18:43:44.303 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-10T18:43:44.303 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-10T18:43:44.303 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-10T18:43:44.303 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-10T18:43:44.303 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-10T18:43:44.303 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 54, 2024-09-10T18:43:44.303 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 378, 2024-09-10T18:43:44.303 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-10T18:43:44.303 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-10T18:43:44.304 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-10T18:43:44.304 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.304 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6812", 2024-09-10T18:43:44.304 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:43:44.304 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.304 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.304 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-10T18:43:44.304 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.304 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6813", 2024-09-10T18:43:44.305 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:43:44.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.305 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.305 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-10T18:43:44.305 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.305 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.305 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6815", 2024-09-10T18:43:44.305 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:43:44.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.305 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.305 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-10T18:43:44.305 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-10T18:43:44.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.306 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-10T18:43:44.306 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.204:6814", 2024-09-10T18:43:44.306 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3511791879 2024-09-10T18:43:44.306 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.306 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.306 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.204:6812/3511791879", 2024-09-10T18:43:44.306 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.204:6813/3511791879", 2024-09-10T18:43:44.306 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.204:6815/3511791879", 2024-09-10T18:43:44.306 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.204:6814/3511791879", 2024-09-10T18:43:44.306 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-10T18:43:44.306 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-10T18:43:44.306 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-10T18:43:44.306 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.307 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.307 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:43:44.307 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-09-10T18:43:44.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.307 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-10T18:43:44.307 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:43:44.307 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:43:44.307 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:43:44.307 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:43:44.307 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 65536, 2024-09-10T18:43:44.307 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:42:49.200099+0000", 2024-09-10T18:43:44.307 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:43:44.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.307 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-10T18:43:44.308 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:43:44.308 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:43:44.308 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:43:44.308 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:43:44.308 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:43:44.308 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:42:51.321482+0000", 2024-09-10T18:43:44.308 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:43:44.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.308 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-10T18:43:44.308 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:43:44.308 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:43:44.308 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:43:44.308 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:43:44.308 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:43:44.309 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:42:41.660799+0000", 2024-09-10T18:43:44.309 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:43:44.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.309 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-10T18:43:44.309 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:43:44.309 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:43:44.309 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:43:44.309 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:43:44.309 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:43:44.309 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:43:43.218019+0000", 2024-09-10T18:43:44.309 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:43:44.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.310 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-10T18:43:44.310 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:43:44.310 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:43:44.310 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:43:44.310 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:43:44.310 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:43:44.310 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:42:50.277523+0000", 2024-09-10T18:43:44.310 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:43:44.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.310 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-10T18:43:44.311 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:43:44.311 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:43:44.311 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:43:44.311 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:43:44.311 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:43:44.311 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:43:15.529542+0000", 2024-09-10T18:43:44.311 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:43:44.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.311 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-09-10T18:43:44.311 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:43:44.311 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:43:44.311 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:43:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:43:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:43:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:43:14.602091+0000", 2024-09-10T18:43:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 94 2024-09-10T18:43:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-09-10T18:43:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-10T18:43:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-10T18:43:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-10T18:43:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138320759226367, 2024-09-10T18:43:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-10T18:43:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-10T18:43:21.558148+0000", 2024-09-10T18:43:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-10T18:43:44.313 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.313 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:43:44.313 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-09-10T18:43:44.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.313 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.c", 2024-09-10T18:43:44.313 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:43:44.313 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-09-10T18:43:44.313 INFO:teuthology.orchestra.run.smithi028.stdout: 4, 2024-09-10T18:43:44.313 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-10T18:43:44.313 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.313 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.1f", 2024-09-10T18:43:44.313 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:43:44.313 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-10T18:43:44.314 INFO:teuthology.orchestra.run.smithi028.stdout: 2, 2024-09-10T18:43:44.314 INFO:teuthology.orchestra.run.smithi028.stdout: 7 2024-09-10T18:43:44.314 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.314 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "6.17", 2024-09-10T18:43:44.314 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-10T18:43:44.314 INFO:teuthology.orchestra.run.smithi028.stdout: 3, 2024-09-10T18:43:44.314 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-09-10T18:43:44.314 INFO:teuthology.orchestra.run.smithi028.stdout: 5 2024-09-10T18:43:44.314 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.314 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.314 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:43:44.314 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-09-10T18:43:44.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.314 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "3.1d", 2024-09-10T18:43:44.315 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:43:44.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.315 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 5, 2024-09-10T18:43:44.315 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 7 2024-09-10T18:43:44.315 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.315 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.315 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.7", 2024-09-10T18:43:44.315 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:43:44.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.315 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:43:44.315 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-09-10T18:43:44.315 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.315 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.316 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.13", 2024-09-10T18:43:44.316 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:43:44.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.316 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:43:44.316 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-09-10T18:43:44.316 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.316 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.316 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1b", 2024-09-10T18:43:44.316 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:43:44.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.316 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:43:44.316 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-09-10T18:43:44.317 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.317 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.317 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1d", 2024-09-10T18:43:44.317 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:43:44.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.317 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:43:44.317 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-09-10T18:43:44.317 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.317 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.317 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1f", 2024-09-10T18:43:44.317 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-10T18:43:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-10T18:43:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-10T18:43:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-09-10T18:43:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-10T18:43:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-10T18:43:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-09-10T18:43:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-09-10T18:43:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-09-10T18:43:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1745574083": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:43:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3866220552": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:43:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3760775432": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:43:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1746797014": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:43:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1029045895": "2024-09-11T18:16:45.197204+0000", 2024-09-10T18:43:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3310949126": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:43:44.319 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2874160765": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:43:44.319 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2620179974": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:43:44.319 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/421194352": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:43:44.319 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/1077377902": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:43:44.319 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3428757300": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:43:44.319 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1759116360": "2024-09-11T18:22:08.678985+0000", 2024-09-10T18:43:44.319 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2564333506": "2024-09-11T18:17:07.521165+0000", 2024-09-10T18:43:44.319 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/717120167": "2024-09-11T18:22:08.678985+0000" 2024-09-10T18:43:44.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.319 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-09-10T18:43:44.319 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-09-10T18:43:44.319 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-09-10T18:43:44.319 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-09-10T18:43:44.319 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-09-10T18:43:44.319 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-09-10T18:43:44.319 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-09-10T18:43:44.320 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-09-10T18:43:44.320 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-10T18:43:44.320 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-09-10T18:43:44.320 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-09-10T18:43:44.320 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-09-10T18:43:44.320 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-09-10T18:43:44.320 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-09-10T18:43:44.320 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-09-10T18:43:44.320 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-09-10T18:43:44.320 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-09-10T18:43:44.320 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-09-10T18:43:44.320 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-09-10T18:43:44.320 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-09-10T18:43:44.320 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-10T18:43:44.321 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-10T18:43:44.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:44 smithi204 ceph-mon[32357]: 4.a scrub starts 2024-09-10T18:43:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:44 smithi204 ceph-mon[32357]: 4.a scrub ok 2024-09-10T18:43:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:44 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:43:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:44 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:43:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:44 smithi204 ceph-mon[32357]: pgmap v996: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:43:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:44 smithi204 ceph-mon[32357]: 1.5 scrub starts 2024-09-10T18:43:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:44 smithi204 ceph-mon[32357]: 1.5 scrub ok 2024-09-10T18:43:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:44 smithi204 ceph-mon[32357]: 3.1e scrub starts 2024-09-10T18:43:44.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:44 smithi204 ceph-mon[32357]: 3.1e scrub ok 2024-09-10T18:43:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:45.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:45 smithi028 ceph-mon[29291]: osdmap e422: 8 total, 8 up, 7 in 2024-09-10T18:43:45.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:45 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1272401643' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:43:45.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:45 smithi028 ceph-mon[39220]: osdmap e422: 8 total, 8 up, 7 in 2024-09-10T18:43:45.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:45 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1272401643' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:43:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:45 smithi204 ceph-mon[32357]: osdmap e422: 8 total, 8 up, 7 in 2024-09-10T18:43:45.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:45 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1272401643' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T18:43:45.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:43:45 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:43:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:46.031 INFO:tasks.thrashosds.thrasher:j is {'epoch': 422, 'fsid': '5eababb2-6fa0-11ef-bcea-c7b262605968', 'created': '2024-09-10T18:16:20.680373+0000', 'modified': '2024-09-10T18:43:43.995648+0000', 'last_up_change': '2024-09-10T18:26:26.611338+0000', 'last_in_change': '2024-09-10T18:39:24.201588+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T18:19:24.013536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '300', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '287', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-10T18:21:30.065209+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 62, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T18:21:30.506441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T18:21:32.008130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T18:21:34.060571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T18:21:36.127330+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}], 'osds': [{'osd': 0, 'uuid': 'f1c834d9-252a-4077-b25e-169181d70251', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 356, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 3188372351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 3188372351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 3188372351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 3188372351}]}, 'public_addr': '172.21.15.28:6801/3188372351', 'cluster_addr': '172.21.15.28:6802/3188372351', 'heartbeat_back_addr': '172.21.15.28:6804/3188372351', 'heartbeat_front_addr': '172.21.15.28:6803/3188372351', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '81f1855d-b946-4ca3-b246-38977b9e20fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1369912678}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1369912678}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1369912678}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1369912678}]}, 'public_addr': '172.21.15.28:6805/1369912678', 'cluster_addr': '172.21.15.28:6806/1369912678', 'heartbeat_back_addr': '172.21.15.28:6808/1369912678', 'heartbeat_front_addr': '172.21.15.28:6807/1369912678', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2ad55d2a-36fa-4756-bf22-58d4b049486e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 392, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 1243406003}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 1243406003}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 1243406003}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1243406003}]}, 'public_addr': '172.21.15.28:6809/1243406003', 'cluster_addr': '172.21.15.28:6810/1243406003', 'heartbeat_back_addr': '172.21.15.28:6812/1243406003', 'heartbeat_front_addr': '172.21.15.28:6811/1243406003', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '242729a5-a5e8-44d5-97c8-14d14e970bff', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 67703283}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 67703283}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 67703283}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 67703283}]}, 'public_addr': '172.21.15.28:6813/67703283', 'cluster_addr': '172.21.15.28:6814/67703283', 'heartbeat_back_addr': '172.21.15.28:6816/67703283', 'heartbeat_front_addr': '172.21.15.28:6815/67703283', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6e38b8ef-e6ed-493f-8eca-755354754787', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 392, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6800', 'nonce': 1619669076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 1619669076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 1619669076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6802', 'nonce': 1619669076}]}, 'public_addr': '172.21.15.204:6800/1619669076', 'cluster_addr': '172.21.15.204:6801/1619669076', 'heartbeat_back_addr': '172.21.15.204:6803/1619669076', 'heartbeat_front_addr': '172.21.15.204:6802/1619669076', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a92e0441-66c9-498e-a0ac-6465acdbb1f5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6804', 'nonce': 2717802880}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 2717802880}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 2717802880}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6806', 'nonce': 2717802880}]}, 'public_addr': '172.21.15.204:6804/2717802880', 'cluster_addr': '172.21.15.204:6805/2717802880', 'heartbeat_back_addr': '172.21.15.204:6807/2717802880', 'heartbeat_front_addr': '172.21.15.204:6806/2717802880', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '12b02875-6653-45ed-8d2b-b4d33bd88525', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 46, 'last_clean_end': 94, 'up_from': 144, 'up_thru': 374, 'down_at': 95, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6808', 'nonce': 3479311452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 3479311452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 3479311452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6810', 'nonce': 3479311452}]}, 'public_addr': '172.21.15.204:6808/3479311452', 'cluster_addr': '172.21.15.204:6809/3479311452', 'heartbeat_back_addr': '172.21.15.204:6811/3479311452', 'heartbeat_front_addr': '172.21.15.204:6810/3479311452', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0c4f7aa9-ff95-4192-a146-ca576eee907f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 378, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6812', 'nonce': 3511791879}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 3511791879}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 3511791879}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.204:6814', 'nonce': 3511791879}]}, 'public_addr': '172.21.15.204:6812/3511791879', 'cluster_addr': '172.21.15.204:6813/3511791879', 'heartbeat_back_addr': '172.21.15.204:6815/3511791879', 'heartbeat_front_addr': '172.21.15.204:6814/3511791879', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-10T18:42:49.200099+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:42:51.321482+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:42:41.660799+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:43:43.218019+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:42:50.277523+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:43:15.529542+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:43:14.602091+0000', 'dead_epoch': 94}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T18:43:21.558148+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.c', 'osds': [7, 4, 2]}, {'pgid': '5.1f', 'osds': [0, 2, 7]}, {'pgid': '6.17', 'osds': [3, 7, 5]}], 'pg_upmap_items': [{'pgid': '3.1d', 'mappings': [{'from': 5, 'to': 7}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/1745574083': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/3866220552': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:6800/3760775432': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:0/1746797014': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:0/1029045895': '2024-09-11T18:16:45.197204+0000', '172.21.15.28:6800/3310949126': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/2874160765': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/2620179974': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/421194352': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:6800/1077377902': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/3428757300': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/1759116360': '2024-09-11T18:22:08.678985+0000', '172.21.15.28:0/2564333506': '2024-09-11T18:17:07.521165+0000', '172.21.15.28:0/717120167': '2024-09-11T18:22:08.678985+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T18:43:46.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:43:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:46 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:46 smithi204 ceph-mon[32357]: 3.4 scrub starts 2024-09-10T18:43:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:46 smithi204 ceph-mon[32357]: 3.4 scrub ok 2024-09-10T18:43:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:46 smithi204 ceph-mon[32357]: pgmap v998: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 654 B/s rd, 0 op/s 2024-09-10T18:43:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:46 smithi204 ceph-mon[32357]: 1.14 scrub starts 2024-09-10T18:43:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:46 smithi204 ceph-mon[32357]: 1.14 scrub ok 2024-09-10T18:43:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:46 smithi204 ceph-mon[32357]: 6.1e scrub starts 2024-09-10T18:43:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:46 smithi204 ceph-mon[32357]: 6.1e scrub ok 2024-09-10T18:43:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:46 smithi204 ceph-mon[32357]: 5.14 scrub starts 2024-09-10T18:43:46.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:46 smithi204 ceph-mon[32357]: 5.14 scrub ok 2024-09-10T18:43:46.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:46 smithi204 ceph-mon[32357]: 3.1f scrub starts 2024-09-10T18:43:46.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:46 smithi204 ceph-mon[32357]: 3.1f scrub ok 2024-09-10T18:43:46.407 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:46.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[29291]: 3.4 scrub starts 2024-09-10T18:43:46.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[29291]: 3.4 scrub ok 2024-09-10T18:43:46.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[29291]: pgmap v998: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 654 B/s rd, 0 op/s 2024-09-10T18:43:46.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[29291]: 1.14 scrub starts 2024-09-10T18:43:46.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[29291]: 1.14 scrub ok 2024-09-10T18:43:46.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[29291]: 6.1e scrub starts 2024-09-10T18:43:46.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[29291]: 6.1e scrub ok 2024-09-10T18:43:46.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[29291]: 5.14 scrub starts 2024-09-10T18:43:46.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[29291]: 5.14 scrub ok 2024-09-10T18:43:46.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[29291]: 3.1f scrub starts 2024-09-10T18:43:46.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[29291]: 3.1f scrub ok 2024-09-10T18:43:46.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:46.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[39220]: 3.4 scrub starts 2024-09-10T18:43:46.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[39220]: 3.4 scrub ok 2024-09-10T18:43:46.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[39220]: pgmap v998: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 654 B/s rd, 0 op/s 2024-09-10T18:43:46.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[39220]: 1.14 scrub starts 2024-09-10T18:43:46.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[39220]: 1.14 scrub ok 2024-09-10T18:43:46.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[39220]: 6.1e scrub starts 2024-09-10T18:43:46.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[39220]: 6.1e scrub ok 2024-09-10T18:43:46.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[39220]: 5.14 scrub starts 2024-09-10T18:43:46.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[39220]: 5.14 scrub ok 2024-09-10T18:43:46.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[39220]: 3.1f scrub starts 2024-09-10T18:43:46.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:46 smithi028 ceph-mon[39220]: 3.1f scrub ok 2024-09-10T18:43:47.352 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:47 smithi028 ceph-mon[29291]: 3.13 scrub starts 2024-09-10T18:43:47.352 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:47 smithi028 ceph-mon[29291]: 3.13 scrub ok 2024-09-10T18:43:47.352 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:47 smithi028 ceph-mon[29291]: 6.14 deep-scrub starts 2024-09-10T18:43:47.352 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:47 smithi028 ceph-mon[29291]: 6.14 deep-scrub ok 2024-09-10T18:43:47.352 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:47 smithi028 ceph-mon[29291]: 6.6 scrub starts 2024-09-10T18:43:47.352 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:47 smithi028 ceph-mon[29291]: 6.6 scrub ok 2024-09-10T18:43:47.352 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:47 smithi028 ceph-mon[29291]: 4.15 scrub starts 2024-09-10T18:43:47.352 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:47 smithi028 ceph-mon[29291]: 4.15 scrub ok 2024-09-10T18:43:47.352 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:47 smithi028 ceph-mon[39220]: 3.13 scrub starts 2024-09-10T18:43:47.353 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:47 smithi028 ceph-mon[39220]: 3.13 scrub ok 2024-09-10T18:43:47.353 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:47 smithi028 ceph-mon[39220]: 6.14 deep-scrub starts 2024-09-10T18:43:47.353 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:47 smithi028 ceph-mon[39220]: 6.14 deep-scrub ok 2024-09-10T18:43:47.353 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:47 smithi028 ceph-mon[39220]: 6.6 scrub starts 2024-09-10T18:43:47.353 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:47 smithi028 ceph-mon[39220]: 6.6 scrub ok 2024-09-10T18:43:47.353 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:47 smithi028 ceph-mon[39220]: 4.15 scrub starts 2024-09-10T18:43:47.353 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:47 smithi028 ceph-mon[39220]: 4.15 scrub ok 2024-09-10T18:43:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:47 smithi204 ceph-mon[32357]: 3.13 scrub starts 2024-09-10T18:43:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:47 smithi204 ceph-mon[32357]: 3.13 scrub ok 2024-09-10T18:43:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:47 smithi204 ceph-mon[32357]: 6.14 deep-scrub starts 2024-09-10T18:43:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:47 smithi204 ceph-mon[32357]: 6.14 deep-scrub ok 2024-09-10T18:43:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:47 smithi204 ceph-mon[32357]: 6.6 scrub starts 2024-09-10T18:43:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:47 smithi204 ceph-mon[32357]: 6.6 scrub ok 2024-09-10T18:43:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:47 smithi204 ceph-mon[32357]: 4.15 scrub starts 2024-09-10T18:43:47.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:47 smithi204 ceph-mon[32357]: 4.15 scrub ok 2024-09-10T18:43:47.469 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-10T18:43:47.469 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-10T18:43:47.469 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:43:47.469 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:43:47.483 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-10T18:43:47.483 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:43:47.498 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-10T18:43:47.499 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:43:47.515 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-10T18:43:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:43:47.531 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-10T18:43:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:43:47.547 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-09-10T18:43:47.547 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:43:47.563 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-09-10T18:43:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:43:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:48 smithi204 ceph-mon[32357]: 1.1d scrub starts 2024-09-10T18:43:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:48 smithi204 ceph-mon[32357]: 1.1d scrub ok 2024-09-10T18:43:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:48 smithi204 ceph-mon[32357]: 5.1f deep-scrub starts 2024-09-10T18:43:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:48 smithi204 ceph-mon[32357]: 5.1f deep-scrub ok 2024-09-10T18:43:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:48 smithi204 ceph-mon[32357]: 3.1d scrub starts 2024-09-10T18:43:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:48 smithi204 ceph-mon[32357]: 3.1d scrub ok 2024-09-10T18:43:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:48 smithi204 ceph-mon[32357]: 2.1 scrub starts 2024-09-10T18:43:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:48 smithi204 ceph-mon[32357]: 2.1 scrub ok 2024-09-10T18:43:48.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:48 smithi204 ceph-mon[32357]: pgmap v999: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:43:48.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:48 smithi204 ceph-mon[32357]: 6.1a scrub starts 2024-09-10T18:43:48.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:48 smithi204 ceph-mon[32357]: 6.1a scrub ok 2024-09-10T18:43:48.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:48 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4016998064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:43:48.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[29291]: 1.1d scrub starts 2024-09-10T18:43:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[29291]: 1.1d scrub ok 2024-09-10T18:43:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[29291]: 5.1f deep-scrub starts 2024-09-10T18:43:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[29291]: 5.1f deep-scrub ok 2024-09-10T18:43:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[29291]: 3.1d scrub starts 2024-09-10T18:43:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[29291]: 3.1d scrub ok 2024-09-10T18:43:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[29291]: 2.1 scrub starts 2024-09-10T18:43:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[29291]: 2.1 scrub ok 2024-09-10T18:43:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[29291]: pgmap v999: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:43:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[29291]: 6.1a scrub starts 2024-09-10T18:43:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[29291]: 6.1a scrub ok 2024-09-10T18:43:48.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4016998064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:43:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[39220]: 1.1d scrub starts 2024-09-10T18:43:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[39220]: 1.1d scrub ok 2024-09-10T18:43:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[39220]: 5.1f deep-scrub starts 2024-09-10T18:43:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[39220]: 5.1f deep-scrub ok 2024-09-10T18:43:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[39220]: 3.1d scrub starts 2024-09-10T18:43:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[39220]: 3.1d scrub ok 2024-09-10T18:43:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[39220]: 2.1 scrub starts 2024-09-10T18:43:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[39220]: 2.1 scrub ok 2024-09-10T18:43:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[39220]: pgmap v999: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:43:48.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[39220]: 6.1a scrub starts 2024-09-10T18:43:48.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[39220]: 6.1a scrub ok 2024-09-10T18:43:48.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:48 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4016998064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:43:48.806 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-10T18:43:49.067 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:43:49.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:49 smithi204 ceph-mon[32357]: 6.1c scrub starts 2024-09-10T18:43:49.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:49 smithi204 ceph-mon[32357]: 6.1c scrub ok 2024-09-10T18:43:49.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:49 smithi204 ceph-mon[32357]: 4.e scrub starts 2024-09-10T18:43:49.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:49 smithi204 ceph-mon[32357]: 4.e scrub ok 2024-09-10T18:43:49.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:49 smithi204 ceph-mon[32357]: 4.11 scrub starts 2024-09-10T18:43:49.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:49 smithi204 ceph-mon[32357]: 4.11 scrub ok 2024-09-10T18:43:49.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:49 smithi204 ceph-mon[32357]: 5.15 scrub starts 2024-09-10T18:43:49.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:49 smithi204 ceph-mon[32357]: 5.15 scrub ok 2024-09-10T18:43:49.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:49 smithi028 ceph-mon[29291]: 6.1c scrub starts 2024-09-10T18:43:49.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:49 smithi028 ceph-mon[29291]: 6.1c scrub ok 2024-09-10T18:43:49.164 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:49 smithi028 ceph-mon[29291]: 4.e scrub starts 2024-09-10T18:43:49.164 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:49 smithi028 ceph-mon[29291]: 4.e scrub ok 2024-09-10T18:43:49.164 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:49 smithi028 ceph-mon[29291]: 4.11 scrub starts 2024-09-10T18:43:49.164 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:49 smithi028 ceph-mon[29291]: 4.11 scrub ok 2024-09-10T18:43:49.164 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:49 smithi028 ceph-mon[29291]: 5.15 scrub starts 2024-09-10T18:43:49.164 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:49 smithi028 ceph-mon[29291]: 5.15 scrub ok 2024-09-10T18:43:49.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:49 smithi028 ceph-mon[39220]: 6.1c scrub starts 2024-09-10T18:43:49.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:49 smithi028 ceph-mon[39220]: 6.1c scrub ok 2024-09-10T18:43:49.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:49 smithi028 ceph-mon[39220]: 4.e scrub starts 2024-09-10T18:43:49.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:49 smithi028 ceph-mon[39220]: 4.e scrub ok 2024-09-10T18:43:49.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:49 smithi028 ceph-mon[39220]: 4.11 scrub starts 2024-09-10T18:43:49.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:49 smithi028 ceph-mon[39220]: 4.11 scrub ok 2024-09-10T18:43:49.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:49 smithi028 ceph-mon[39220]: 5.15 scrub starts 2024-09-10T18:43:49.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:49 smithi028 ceph-mon[39220]: 5.15 scrub ok 2024-09-10T18:43:49.479 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:50.086 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:50 smithi028 ceph-mon[29291]: 5.10 scrub starts 2024-09-10T18:43:50.086 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:50 smithi028 ceph-mon[29291]: 5.10 scrub ok 2024-09-10T18:43:50.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:50 smithi028 ceph-mon[29291]: pgmap v1000: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T18:43:50.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:50 smithi028 ceph-mon[29291]: 4.d scrub starts 2024-09-10T18:43:50.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:50 smithi028 ceph-mon[29291]: 4.d scrub ok 2024-09-10T18:43:50.087 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:50 smithi028 ceph-mon[39220]: 5.10 scrub starts 2024-09-10T18:43:50.087 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:50 smithi028 ceph-mon[39220]: 5.10 scrub ok 2024-09-10T18:43:50.087 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:50 smithi028 ceph-mon[39220]: pgmap v1000: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T18:43:50.087 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:50 smithi028 ceph-mon[39220]: 4.d scrub starts 2024-09-10T18:43:50.087 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:50 smithi028 ceph-mon[39220]: 4.d scrub ok 2024-09-10T18:43:50.301 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:50 smithi204 ceph-mon[32357]: 5.10 scrub starts 2024-09-10T18:43:50.301 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:50 smithi204 ceph-mon[32357]: 5.10 scrub ok 2024-09-10T18:43:50.301 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:50 smithi204 ceph-mon[32357]: pgmap v1000: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T18:43:50.301 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:50 smithi204 ceph-mon[32357]: 4.d scrub starts 2024-09-10T18:43:50.301 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:50 smithi204 ceph-mon[32357]: 4.d scrub ok 2024-09-10T18:43:50.849 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:43:50.849 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:43:50.849 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-10T18:43:50.849 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:43:51.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:51 smithi028 ceph-mon[39220]: 4.12 scrub starts 2024-09-10T18:43:51.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:51 smithi028 ceph-mon[39220]: 4.12 scrub ok 2024-09-10T18:43:51.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:51 smithi028 ceph-mon[39220]: 6.16 scrub starts 2024-09-10T18:43:51.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:51 smithi028 ceph-mon[39220]: 6.16 scrub ok 2024-09-10T18:43:51.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:51 smithi028 ceph-mon[39220]: 6.a scrub starts 2024-09-10T18:43:51.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:51 smithi028 ceph-mon[39220]: 6.a scrub ok 2024-09-10T18:43:51.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:51 smithi028 ceph-mon[29291]: 4.12 scrub starts 2024-09-10T18:43:51.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:51 smithi028 ceph-mon[29291]: 4.12 scrub ok 2024-09-10T18:43:51.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:51 smithi028 ceph-mon[29291]: 6.16 scrub starts 2024-09-10T18:43:51.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:51 smithi028 ceph-mon[29291]: 6.16 scrub ok 2024-09-10T18:43:51.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:51 smithi028 ceph-mon[29291]: 6.a scrub starts 2024-09-10T18:43:51.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:51 smithi028 ceph-mon[29291]: 6.a scrub ok 2024-09-10T18:43:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:51 smithi204 ceph-mon[32357]: 4.12 scrub starts 2024-09-10T18:43:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:51 smithi204 ceph-mon[32357]: 4.12 scrub ok 2024-09-10T18:43:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:51 smithi204 ceph-mon[32357]: 6.16 scrub starts 2024-09-10T18:43:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:51 smithi204 ceph-mon[32357]: 6.16 scrub ok 2024-09-10T18:43:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:51 smithi204 ceph-mon[32357]: 6.a scrub starts 2024-09-10T18:43:51.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:51 smithi204 ceph-mon[32357]: 6.a scrub ok 2024-09-10T18:43:51.462 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:43:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:43:51] "GET /metrics HTTP/1.1" 200 224837 "" "Prometheus/2.43.0" 2024-09-10T18:43:51.492 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-10T18:43:51.730 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:43:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:43:51.738 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:43:52.026 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-10T18:43:52.120 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:52 smithi204 ceph-mon[32357]: 5.a scrub starts 2024-09-10T18:43:52.120 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:52 smithi204 ceph-mon[32357]: 5.a scrub ok 2024-09-10T18:43:52.121 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:52 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1576644130' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:43:52.121 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:52 smithi204 ceph-mon[32357]: pgmap v1001: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:43:52.121 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:52 smithi204 ceph-mon[32357]: 4.0 scrub starts 2024-09-10T18:43:52.121 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:52 smithi204 ceph-mon[32357]: 4.0 scrub ok 2024-09-10T18:43:52.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:52 smithi028 ceph-mon[29291]: 5.a scrub starts 2024-09-10T18:43:52.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:52 smithi028 ceph-mon[29291]: 5.a scrub ok 2024-09-10T18:43:52.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:52 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1576644130' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:43:52.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:52 smithi028 ceph-mon[29291]: pgmap v1001: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:43:52.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:52 smithi028 ceph-mon[29291]: 4.0 scrub starts 2024-09-10T18:43:52.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:52 smithi028 ceph-mon[29291]: 4.0 scrub ok 2024-09-10T18:43:52.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:52 smithi028 ceph-mon[39220]: 5.a scrub starts 2024-09-10T18:43:52.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:52 smithi028 ceph-mon[39220]: 5.a scrub ok 2024-09-10T18:43:52.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:52 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1576644130' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:43:52.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:52 smithi028 ceph-mon[39220]: pgmap v1001: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:43:52.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:52 smithi028 ceph-mon[39220]: 4.0 scrub starts 2024-09-10T18:43:52.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:52 smithi028 ceph-mon[39220]: 4.0 scrub ok 2024-09-10T18:43:52.526 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":1001,"stamp":"2024-09-10T18:43:51.241801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":2900,"num_read_kb":2652,"num_write":1002,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":334384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3681,"ondisk_log_size":3681,"up":486,"acting":486,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":486,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6287972,"kb_used_data":6892,"kb_used_omap":17,"kb_used_meta":6281006,"kb_avail":649928092,"statfs":{"total":671965249536,"available":665526366208,"internally_reserved":0,"allocated":7057408,"data_stored":4396701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18329,"internal_metadata":6431750247},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.247061"},"pg_stats":[{"pgid":"1.1d","version":"76'81","reported_seq":186,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:46.223735+0000","last_change":"2024-09-10T18:43:46.223709+0000","last_active":"2024-09-10T18:43:46.223735+0000","last_peered":"2024-09-10T18:43:46.223735+0000","last_clean":"2024-09-10T18:43:46.223735+0000","last_became_active":"2024-09-10T18:34:34.865894+0000","last_became_peered":"2024-09-10T18:34:34.865894+0000","last_unstale":"2024-09-10T18:43:46.223735+0000","last_undegraded":"2024-09-10T18:43:46.223735+0000","last_fullsized":"2024-09-10T18:43:46.223735+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:43:46.223650+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:43:46.223650+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:08.879567+0000","scrub_duration":0.0018844000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":490,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:45.902566+0000","last_change":"2024-09-10T18:43:45.902534+0000","last_active":"2024-09-10T18:43:45.902566+0000","last_peered":"2024-09-10T18:43:45.902566+0000","last_clean":"2024-09-10T18:43:45.902566+0000","last_became_active":"2024-09-10T18:39:25.247259+0000","last_became_peered":"2024-09-10T18:39:25.247259+0000","last_unstale":"2024-09-10T18:43:45.902566+0000","last_undegraded":"2024-09-10T18:43:45.902566+0000","last_fullsized":"2024-09-10T18:43:45.902566+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:45.902468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:43:45.902468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:15.079942+0000","scrub_duration":0.001361213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"400'25","reported_seq":513,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.998616+0000","last_change":"2024-09-10T18:42:52.161573+0000","last_active":"2024-09-10T18:43:41.998616+0000","last_peered":"2024-09-10T18:43:41.998616+0000","last_clean":"2024-09-10T18:43:41.998616+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:43:41.998616+0000","last_undegraded":"2024-09-10T18:43:41.998616+0000","last_fullsized":"2024-09-10T18:43:41.998616+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'25","last_scrub_stamp":"2024-09-10T18:42:52.161489+0000","last_deep_scrub":"342'19","last_deep_scrub_stamp":"2024-09-10T18:37:49.354781+0000","last_clean_scrub_stamp":"2024-09-10T18:42:52.161489+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:02.895339+0000","scrub_duration":0.0055438650000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":487,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.191136+0000","last_change":"2024-09-10T18:41:48.436189+0000","last_active":"2024-09-10T18:43:42.191136+0000","last_peered":"2024-09-10T18:43:42.191136+0000","last_clean":"2024-09-10T18:43:42.191136+0000","last_became_active":"2024-09-10T18:39:00.910095+0000","last_became_peered":"2024-09-10T18:39:00.910095+0000","last_unstale":"2024-09-10T18:43:42.191136+0000","last_undegraded":"2024-09-10T18:43:42.191136+0000","last_fullsized":"2024-09-10T18:43:42.191136+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:48.436106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:41:48.436106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049447019999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":487,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:47.249044+0000","last_change":"2024-09-10T18:43:47.249009+0000","last_active":"2024-09-10T18:43:47.249044+0000","last_peered":"2024-09-10T18:43:47.249044+0000","last_clean":"2024-09-10T18:43:47.249044+0000","last_became_active":"2024-09-10T18:39:00.889898+0000","last_became_peered":"2024-09-10T18:39:00.889898+0000","last_unstale":"2024-09-10T18:43:47.249044+0000","last_undegraded":"2024-09-10T18:43:47.249044+0000","last_fullsized":"2024-09-10T18:43:47.249044+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:47.248942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:43:47.248942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:11.047521+0000","scrub_duration":0.001462105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'81","reported_seq":174,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.216123+0000","last_change":"2024-09-10T18:42:33.135226+0000","last_active":"2024-09-10T18:43:42.216123+0000","last_peered":"2024-09-10T18:43:42.216123+0000","last_clean":"2024-09-10T18:43:42.216123+0000","last_became_active":"2024-09-10T18:34:34.867388+0000","last_became_peered":"2024-09-10T18:34:34.867388+0000","last_unstale":"2024-09-10T18:43:42.216123+0000","last_undegraded":"2024-09-10T18:43:42.216123+0000","last_fullsized":"2024-09-10T18:43:42.216123+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:33.134846+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:33.134846+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088763399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":454,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214024+0000","last_change":"2024-09-10T18:40:39.463295+0000","last_active":"2024-09-10T18:43:42.214024+0000","last_peered":"2024-09-10T18:43:42.214024+0000","last_clean":"2024-09-10T18:43:42.214024+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:43:42.214024+0000","last_undegraded":"2024-09-10T18:43:42.214024+0000","last_fullsized":"2024-09-10T18:43:42.214024+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:39.463211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:39.463211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069721330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":470,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:43.730615+0000","last_change":"2024-09-10T18:43:43.730565+0000","last_active":"2024-09-10T18:43:43.730615+0000","last_peered":"2024-09-10T18:43:43.730615+0000","last_clean":"2024-09-10T18:43:43.730615+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:43:43.730615+0000","last_undegraded":"2024-09-10T18:43:43.730615+0000","last_fullsized":"2024-09-10T18:43:43.730615+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:43.730509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:27:14.532784+0000","last_clean_scrub_stamp":"2024-09-10T18:43:43.730509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:56.390554+0000","scrub_duration":0.0017065540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"400'46","reported_seq":548,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.213467+0000","last_change":"2024-09-10T18:42:30.196325+0000","last_active":"2024-09-10T18:43:42.213467+0000","last_peered":"2024-09-10T18:43:42.213467+0000","last_clean":"2024-09-10T18:43:42.213467+0000","last_became_active":"2024-09-10T18:39:25.237997+0000","last_became_peered":"2024-09-10T18:39:25.237997+0000","last_unstale":"2024-09-10T18:43:42.213467+0000","last_undegraded":"2024-09-10T18:43:42.213467+0000","last_fullsized":"2024-09-10T18:43:42.213467+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'46","last_scrub_stamp":"2024-09-10T18:42:30.196228+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:42:30.196228+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074592269999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":466,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.998612+0000","last_change":"2024-09-10T18:41:43.981927+0000","last_active":"2024-09-10T18:43:41.998612+0000","last_peered":"2024-09-10T18:43:41.998612+0000","last_clean":"2024-09-10T18:43:41.998612+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:43:41.998612+0000","last_undegraded":"2024-09-10T18:43:41.998612+0000","last_fullsized":"2024-09-10T18:43:41.998612+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:43.981864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:04.633647+0000","last_clean_scrub_stamp":"2024-09-10T18:41:43.981864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010372925,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":73,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:46.797016+0000","last_change":"2024-09-10T18:43:46.796983+0000","last_active":"2024-09-10T18:43:46.797016+0000","last_peered":"2024-09-10T18:43:46.797016+0000","last_clean":"2024-09-10T18:43:46.797016+0000","last_became_active":"2024-09-10T18:40:10.986591+0000","last_became_peered":"2024-09-10T18:40:10.986591+0000","last_unstale":"2024-09-10T18:43:46.797016+0000","last_undegraded":"2024-09-10T18:43:46.797016+0000","last_fullsized":"2024-09-10T18:43:46.797016+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:46.796922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_clean_scrub_stamp":"2024-09-10T18:43:46.796922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:11.496369+0000","scrub_duration":0.0013795719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"400'21","reported_seq":522,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.998995+0000","last_change":"2024-09-10T18:42:44.184802+0000","last_active":"2024-09-10T18:43:41.998995+0000","last_peered":"2024-09-10T18:43:41.998995+0000","last_clean":"2024-09-10T18:43:41.998995+0000","last_became_active":"2024-09-10T18:39:25.237946+0000","last_became_peered":"2024-09-10T18:39:25.237946+0000","last_unstale":"2024-09-10T18:43:41.998995+0000","last_undegraded":"2024-09-10T18:43:41.998995+0000","last_fullsized":"2024-09-10T18:43:41.998995+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'21","last_scrub_stamp":"2024-09-10T18:42:44.184715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:42:44.184715+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:54.200219+0000","scrub_duration":0.0061211579999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":94,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.193923+0000","last_change":"2024-09-10T18:41:33.904603+0000","last_active":"2024-09-10T18:43:42.193923+0000","last_peered":"2024-09-10T18:43:42.193923+0000","last_clean":"2024-09-10T18:43:42.193923+0000","last_became_active":"2024-09-10T18:39:25.239738+0000","last_became_peered":"2024-09-10T18:39:25.239738+0000","last_unstale":"2024-09-10T18:43:42.193923+0000","last_undegraded":"2024-09-10T18:43:42.193923+0000","last_fullsized":"2024-09-10T18:43:42.193923+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:33.904519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_clean_scrub_stamp":"2024-09-10T18:41:33.904519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028331189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":93,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.191542+0000","last_change":"2024-09-10T18:42:34.174839+0000","last_active":"2024-09-10T18:43:42.191542+0000","last_peered":"2024-09-10T18:43:42.191542+0000","last_clean":"2024-09-10T18:43:42.191542+0000","last_became_active":"2024-09-10T18:39:25.255362+0000","last_became_peered":"2024-09-10T18:39:25.255362+0000","last_unstale":"2024-09-10T18:43:42.191542+0000","last_undegraded":"2024-09-10T18:43:42.191542+0000","last_fullsized":"2024-09-10T18:43:42.191542+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:34.174638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:42:34.174638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005797562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"76'81","reported_seq":240,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214958+0000","last_change":"2024-09-10T18:42:26.156095+0000","last_active":"2024-09-10T18:43:42.214958+0000","last_peered":"2024-09-10T18:43:42.214958+0000","last_clean":"2024-09-10T18:43:42.214958+0000","last_became_active":"2024-09-10T18:34:34.863102+0000","last_became_peered":"2024-09-10T18:34:34.863102+0000","last_unstale":"2024-09-10T18:43:42.214958+0000","last_undegraded":"2024-09-10T18:43:42.214958+0000","last_fullsized":"2024-09-10T18:43:42.214958+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:26.156014+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:26.156014+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066787549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":94,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.194116+0000","last_change":"2024-09-10T18:42:30.280858+0000","last_active":"2024-09-10T18:43:42.194116+0000","last_peered":"2024-09-10T18:43:42.194116+0000","last_clean":"2024-09-10T18:43:42.194116+0000","last_became_active":"2024-09-10T18:39:00.915294+0000","last_became_peered":"2024-09-10T18:39:00.915294+0000","last_unstale":"2024-09-10T18:43:42.194116+0000","last_undegraded":"2024-09-10T18:43:42.194116+0000","last_fullsized":"2024-09-10T18:43:42.194116+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:30.280771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:39:49.974584+0000","last_clean_scrub_stamp":"2024-09-10T18:42:30.280771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020095740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"400'11","reported_seq":508,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.999088+0000","last_change":"2024-09-10T18:41:50.059924+0000","last_active":"2024-09-10T18:43:41.999088+0000","last_peered":"2024-09-10T18:43:41.999088+0000","last_clean":"2024-09-10T18:43:41.999088+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:43:41.999088+0000","last_undegraded":"2024-09-10T18:43:41.999088+0000","last_fullsized":"2024-09-10T18:43:41.999088+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'11","last_scrub_stamp":"2024-09-10T18:41:50.059835+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:41:50.059835+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053330019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":466,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.217816+0000","last_change":"2024-09-10T18:43:31.013191+0000","last_active":"2024-09-10T18:43:42.217816+0000","last_peered":"2024-09-10T18:43:42.217816+0000","last_clean":"2024-09-10T18:43:42.217816+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:43:42.217816+0000","last_undegraded":"2024-09-10T18:43:42.217816+0000","last_fullsized":"2024-09-10T18:43:42.217816+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:31.013126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:43:31.013126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:38.198677+0000","scrub_duration":0.0016739039999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":84,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.193522+0000","last_change":"2024-09-10T18:41:29.917083+0000","last_active":"2024-09-10T18:43:42.193522+0000","last_peered":"2024-09-10T18:43:42.193522+0000","last_clean":"2024-09-10T18:43:42.193522+0000","last_became_active":"2024-09-10T18:39:00.917007+0000","last_became_peered":"2024-09-10T18:39:00.917007+0000","last_unstale":"2024-09-10T18:43:42.193522+0000","last_undegraded":"2024-09-10T18:43:42.193522+0000","last_fullsized":"2024-09-10T18:43:42.193522+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:29.917010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:41:29.917010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061701890000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'81","reported_seq":333,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214716+0000","last_change":"2024-09-10T18:42:40.392595+0000","last_active":"2024-09-10T18:43:42.214716+0000","last_peered":"2024-09-10T18:43:42.214716+0000","last_clean":"2024-09-10T18:43:42.214716+0000","last_became_active":"2024-09-10T18:39:25.241941+0000","last_became_peered":"2024-09-10T18:39:25.241941+0000","last_unstale":"2024-09-10T18:43:42.214716+0000","last_undegraded":"2024-09-10T18:43:42.214716+0000","last_fullsized":"2024-09-10T18:43:42.214716+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:40.392516+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:40.392516+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:00.717024+0000","scrub_duration":0.012361784000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":494,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:45.306658+0000","last_change":"2024-09-10T18:43:45.306621+0000","last_active":"2024-09-10T18:43:45.306658+0000","last_peered":"2024-09-10T18:43:45.306658+0000","last_clean":"2024-09-10T18:43:45.306658+0000","last_became_active":"2024-09-10T18:39:00.895860+0000","last_became_peered":"2024-09-10T18:39:00.895860+0000","last_unstale":"2024-09-10T18:43:45.306658+0000","last_undegraded":"2024-09-10T18:43:45.306658+0000","last_fullsized":"2024-09-10T18:43:45.306658+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:45.306556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:43:45.306556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:51.534754+0000","scrub_duration":0.0036679460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":93,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.999271+0000","last_change":"2024-09-10T18:42:29.224932+0000","last_active":"2024-09-10T18:43:41.999271+0000","last_peered":"2024-09-10T18:43:41.999271+0000","last_clean":"2024-09-10T18:43:41.999271+0000","last_became_active":"2024-09-10T18:39:25.231931+0000","last_became_peered":"2024-09-10T18:39:25.231931+0000","last_unstale":"2024-09-10T18:43:41.999271+0000","last_undegraded":"2024-09-10T18:43:41.999271+0000","last_fullsized":"2024-09-10T18:43:41.999271+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:29.224782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:42:29.224782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:55.092649+0000","scrub_duration":0.0076768009999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"400'39","reported_seq":568,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.997456+0000","last_change":"2024-09-10T18:42:47.551036+0000","last_active":"2024-09-10T18:43:41.997456+0000","last_peered":"2024-09-10T18:43:41.997456+0000","last_clean":"2024-09-10T18:43:41.997456+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:43:41.997456+0000","last_undegraded":"2024-09-10T18:43:41.997456+0000","last_fullsized":"2024-09-10T18:43:41.997456+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'39","last_scrub_stamp":"2024-09-10T18:42:47.550932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:42:47.550932+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:14.451330+0000","scrub_duration":0.0058170050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":487,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.192080+0000","last_change":"2024-09-10T18:42:35.205645+0000","last_active":"2024-09-10T18:43:42.192080+0000","last_peered":"2024-09-10T18:43:42.192080+0000","last_clean":"2024-09-10T18:43:42.192080+0000","last_became_active":"2024-09-10T18:39:25.246541+0000","last_became_peered":"2024-09-10T18:39:25.246541+0000","last_unstale":"2024-09-10T18:43:42.192080+0000","last_undegraded":"2024-09-10T18:43:42.192080+0000","last_fullsized":"2024-09-10T18:43:42.192080+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:35.205557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:42:35.205557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048486780000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'81","reported_seq":98,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.193524+0000","last_change":"2024-09-10T18:42:47.235267+0000","last_active":"2024-09-10T18:43:42.193524+0000","last_peered":"2024-09-10T18:43:42.193524+0000","last_clean":"2024-09-10T18:43:42.193524+0000","last_became_active":"2024-09-10T18:39:00.907303+0000","last_became_peered":"2024-09-10T18:39:00.907303+0000","last_unstale":"2024-09-10T18:43:42.193524+0000","last_undegraded":"2024-09-10T18:43:42.193524+0000","last_fullsized":"2024-09-10T18:43:42.193524+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:47.235176+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:47.235176+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:01.178362+0000","scrub_duration":0.038577114000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":474,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.215315+0000","last_change":"2024-09-10T18:40:36.531564+0000","last_active":"2024-09-10T18:43:42.215315+0000","last_peered":"2024-09-10T18:43:42.215315+0000","last_clean":"2024-09-10T18:43:42.215315+0000","last_became_active":"2024-09-10T18:39:00.904769+0000","last_became_peered":"2024-09-10T18:39:00.904769+0000","last_unstale":"2024-09-10T18:43:42.215315+0000","last_undegraded":"2024-09-10T18:43:42.215315+0000","last_fullsized":"2024-09-10T18:43:42.215315+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:36.531474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:36.531474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050165130000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":470,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.998668+0000","last_change":"2024-09-10T18:42:45.180685+0000","last_active":"2024-09-10T18:43:41.998668+0000","last_peered":"2024-09-10T18:43:41.998668+0000","last_clean":"2024-09-10T18:43:41.998668+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:43:41.998668+0000","last_undegraded":"2024-09-10T18:43:41.998668+0000","last_fullsized":"2024-09-10T18:43:41.998668+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:45.180608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:13.774791+0000","last_clean_scrub_stamp":"2024-09-10T18:42:45.180608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:02.103777+0000","scrub_duration":0.0058505279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"400'29","reported_seq":519,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214459+0000","last_change":"2024-09-10T18:40:55.739187+0000","last_active":"2024-09-10T18:43:42.214459+0000","last_peered":"2024-09-10T18:43:42.214459+0000","last_clean":"2024-09-10T18:43:42.214459+0000","last_became_active":"2024-09-10T18:39:11.896502+0000","last_became_peered":"2024-09-10T18:39:11.896502+0000","last_unstale":"2024-09-10T18:43:42.214459+0000","last_undegraded":"2024-09-10T18:43:42.214459+0000","last_fullsized":"2024-09-10T18:43:42.214459+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'23","last_scrub_stamp":"2024-09-10T18:40:55.739083+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:40:55.739083+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062258410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":472,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.999230+0000","last_change":"2024-09-10T18:42:46.205068+0000","last_active":"2024-09-10T18:43:41.999230+0000","last_peered":"2024-09-10T18:43:41.999230+0000","last_clean":"2024-09-10T18:43:41.999230+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:43:41.999230+0000","last_undegraded":"2024-09-10T18:43:41.999230+0000","last_fullsized":"2024-09-10T18:43:41.999230+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:46.204993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:08.266279+0000","last_clean_scrub_stamp":"2024-09-10T18:42:46.204993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:50.151980+0000","scrub_duration":0.0046309469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'81","reported_seq":104,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.193869+0000","last_change":"2024-09-10T18:43:33.031278+0000","last_active":"2024-09-10T18:43:42.193869+0000","last_peered":"2024-09-10T18:43:42.193869+0000","last_clean":"2024-09-10T18:43:42.193869+0000","last_became_active":"2024-09-10T18:39:25.240472+0000","last_became_peered":"2024-09-10T18:39:25.240472+0000","last_unstale":"2024-09-10T18:43:42.193869+0000","last_undegraded":"2024-09-10T18:43:42.193869+0000","last_fullsized":"2024-09-10T18:43:42.193869+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:43:33.031177+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:43:33.031177+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:48.835724+0000","scrub_duration":0.0015251889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":498,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:47.749991+0000","last_change":"2024-09-10T18:43:47.749957+0000","last_active":"2024-09-10T18:43:47.749991+0000","last_peered":"2024-09-10T18:43:47.749991+0000","last_clean":"2024-09-10T18:43:47.749991+0000","last_became_active":"2024-09-10T18:39:00.891321+0000","last_became_peered":"2024-09-10T18:39:00.891321+0000","last_unstale":"2024-09-10T18:43:47.749991+0000","last_undegraded":"2024-09-10T18:43:47.749991+0000","last_fullsized":"2024-09-10T18:43:47.749991+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:43:47.749889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:43:47.749889+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:57.451937+0000","scrub_duration":0.001622102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":481,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.191283+0000","last_change":"2024-09-10T18:41:37.332757+0000","last_active":"2024-09-10T18:43:42.191283+0000","last_peered":"2024-09-10T18:43:42.191283+0000","last_clean":"2024-09-10T18:43:42.191283+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:43:42.191283+0000","last_undegraded":"2024-09-10T18:43:42.191283+0000","last_fullsized":"2024-09-10T18:43:42.191283+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:37.332672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:41:37.332672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099109179999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"400'35","reported_seq":381,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214223+0000","last_change":"2024-09-10T18:42:35.122912+0000","last_active":"2024-09-10T18:43:42.214223+0000","last_peered":"2024-09-10T18:43:42.214223+0000","last_clean":"2024-09-10T18:43:42.214223+0000","last_became_active":"2024-09-10T18:39:25.244435+0000","last_became_peered":"2024-09-10T18:39:25.244435+0000","last_unstale":"2024-09-10T18:43:42.214223+0000","last_undegraded":"2024-09-10T18:43:42.214223+0000","last_fullsized":"2024-09-10T18:43:42.214223+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'35","last_scrub_stamp":"2024-09-10T18:42:35.122808+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:42:35.122808+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:52.875093+0000","scrub_duration":0.0045093800000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"362'13","reported_seq":160,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:46.724897+0000","last_change":"2024-09-10T18:43:46.724827+0000","last_active":"2024-09-10T18:43:46.724897+0000","last_peered":"2024-09-10T18:43:46.724897+0000","last_clean":"2024-09-10T18:43:46.724897+0000","last_became_active":"2024-09-10T18:39:25.246551+0000","last_became_peered":"2024-09-10T18:39:25.246551+0000","last_unstale":"2024-09-10T18:43:46.724897+0000","last_undegraded":"2024-09-10T18:43:46.724897+0000","last_fullsized":"2024-09-10T18:43:46.724897+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'13","last_scrub_stamp":"2024-09-10T18:43:46.724761+0000","last_deep_scrub":"362'13","last_deep_scrub_stamp":"2024-09-10T18:43:46.724761+0000","last_clean_scrub_stamp":"2024-09-10T18:43:46.724761+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:08.849568+0000","scrub_duration":0.0020838190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":94,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.194138+0000","last_change":"2024-09-10T18:42:50.279085+0000","last_active":"2024-09-10T18:43:42.194138+0000","last_peered":"2024-09-10T18:43:42.194138+0000","last_clean":"2024-09-10T18:43:42.194138+0000","last_became_active":"2024-09-10T18:39:00.909261+0000","last_became_peered":"2024-09-10T18:39:00.909261+0000","last_unstale":"2024-09-10T18:43:42.194138+0000","last_undegraded":"2024-09-10T18:43:42.194138+0000","last_fullsized":"2024-09-10T18:43:42.194138+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:50.279027+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:50.279027+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:53.070607+0000","scrub_duration":0.0039815600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"400'46","reported_seq":550,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.193752+0000","last_change":"2024-09-10T18:41:45.468203+0000","last_active":"2024-09-10T18:43:42.193752+0000","last_peered":"2024-09-10T18:43:42.193752+0000","last_clean":"2024-09-10T18:43:42.193752+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:43:42.193752+0000","last_undegraded":"2024-09-10T18:43:42.193752+0000","last_fullsized":"2024-09-10T18:43:42.193752+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'46","last_scrub_stamp":"2024-09-10T18:41:45.468076+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:41:45.468076+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046622160000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":463,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214452+0000","last_change":"2024-09-10T18:42:38.415812+0000","last_active":"2024-09-10T18:43:42.214452+0000","last_peered":"2024-09-10T18:43:42.214452+0000","last_clean":"2024-09-10T18:43:42.214452+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:43:42.214452+0000","last_undegraded":"2024-09-10T18:43:42.214452+0000","last_fullsized":"2024-09-10T18:43:42.214452+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:38.415725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:42:38.415725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081623489999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"362'11","reported_seq":782,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:45.024798+0000","last_change":"2024-09-10T18:42:23.100920+0000","last_active":"2024-09-10T18:43:45.024798+0000","last_peered":"2024-09-10T18:43:45.024798+0000","last_clean":"2024-09-10T18:43:45.024798+0000","last_became_active":"2024-09-10T18:39:25.239165+0000","last_became_peered":"2024-09-10T18:39:25.239165+0000","last_unstale":"2024-09-10T18:43:45.024798+0000","last_undegraded":"2024-09-10T18:43:45.024798+0000","last_fullsized":"2024-09-10T18:43:45.024798+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'11","last_scrub_stamp":"2024-09-10T18:42:23.100815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:42:23.100815+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035714409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":104,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.194311+0000","last_change":"2024-09-10T18:42:28.272835+0000","last_active":"2024-09-10T18:43:42.194311+0000","last_peered":"2024-09-10T18:43:42.194311+0000","last_clean":"2024-09-10T18:43:42.194311+0000","last_became_active":"2024-09-10T18:39:25.256173+0000","last_became_peered":"2024-09-10T18:39:25.256173+0000","last_unstale":"2024-09-10T18:43:42.194311+0000","last_undegraded":"2024-09-10T18:43:42.194311+0000","last_fullsized":"2024-09-10T18:43:42.194311+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:28.272680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:42:28.272680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083610449999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":406,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:43.309595+0000","last_change":"2024-09-10T18:43:43.309533+0000","last_active":"2024-09-10T18:43:43.309595+0000","last_peered":"2024-09-10T18:43:43.309595+0000","last_clean":"2024-09-10T18:43:43.309595+0000","last_became_active":"2024-09-10T18:34:32.815839+0000","last_became_peered":"2024-09-10T18:34:32.815839+0000","last_unstale":"2024-09-10T18:43:43.309595+0000","last_undegraded":"2024-09-10T18:43:43.309595+0000","last_fullsized":"2024-09-10T18:43:43.309595+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:43:43.309462+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:43:43.309462+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:51.327517+0000","scrub_duration":0.0024218009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"400'42","reported_seq":547,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.213669+0000","last_change":"2024-09-10T18:40:50.663742+0000","last_active":"2024-09-10T18:43:42.213669+0000","last_peered":"2024-09-10T18:43:42.213669+0000","last_clean":"2024-09-10T18:43:42.213669+0000","last_became_active":"2024-09-10T18:39:25.249915+0000","last_became_peered":"2024-09-10T18:39:25.249915+0000","last_unstale":"2024-09-10T18:43:42.213669+0000","last_undegraded":"2024-09-10T18:43:42.213669+0000","last_fullsized":"2024-09-10T18:43:42.213669+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-09-10T18:40:50.663648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:40:50.663648+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012419997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":463,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.215757+0000","last_change":"2024-09-10T18:41:35.927468+0000","last_active":"2024-09-10T18:43:42.215757+0000","last_peered":"2024-09-10T18:43:42.215757+0000","last_clean":"2024-09-10T18:43:42.215757+0000","last_became_active":"2024-09-10T18:39:25.236559+0000","last_became_peered":"2024-09-10T18:39:25.236559+0000","last_unstale":"2024-09-10T18:43:42.215757+0000","last_undegraded":"2024-09-10T18:43:42.215757+0000","last_fullsized":"2024-09-10T18:43:42.215757+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:35.927388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:15.765392+0000","last_clean_scrub_stamp":"2024-09-10T18:41:35.927388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004755708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":466,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.998689+0000","last_change":"2024-09-10T18:43:39.301741+0000","last_active":"2024-09-10T18:43:41.998689+0000","last_peered":"2024-09-10T18:43:41.998689+0000","last_clean":"2024-09-10T18:43:41.998689+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:43:41.998689+0000","last_undegraded":"2024-09-10T18:43:41.998689+0000","last_fullsized":"2024-09-10T18:43:41.998689+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:39.301647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:43:39.301647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:01.701932+0000","scrub_duration":0.0019630849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":454,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.998744+0000","last_change":"2024-09-10T18:42:41.283056+0000","last_active":"2024-09-10T18:43:41.998744+0000","last_peered":"2024-09-10T18:43:41.998744+0000","last_clean":"2024-09-10T18:43:41.998744+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:43:41.998744+0000","last_undegraded":"2024-09-10T18:43:41.998744+0000","last_fullsized":"2024-09-10T18:43:41.998744+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:41.282974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:41.282974+0000","last_clean_scrub_stamp":"2024-09-10T18:42:41.282974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:49.635072+0000","scrub_duration":0.021644562999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":388,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214028+0000","last_change":"2024-09-10T18:41:27.793561+0000","last_active":"2024-09-10T18:43:42.214028+0000","last_peered":"2024-09-10T18:43:42.214028+0000","last_clean":"2024-09-10T18:43:42.214028+0000","last_became_active":"2024-09-10T18:39:00.894864+0000","last_became_peered":"2024-09-10T18:39:00.894864+0000","last_unstale":"2024-09-10T18:43:42.214028+0000","last_undegraded":"2024-09-10T18:43:42.214028+0000","last_fullsized":"2024-09-10T18:43:42.214028+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:27.793483+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:27.793483+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048667299999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"400'42","reported_seq":580,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.998806+0000","last_change":"2024-09-10T18:41:43.020411+0000","last_active":"2024-09-10T18:43:41.998806+0000","last_peered":"2024-09-10T18:43:41.998806+0000","last_clean":"2024-09-10T18:43:41.998806+0000","last_became_active":"2024-09-10T18:39:00.903341+0000","last_became_peered":"2024-09-10T18:39:00.903341+0000","last_unstale":"2024-09-10T18:43:41.998806+0000","last_undegraded":"2024-09-10T18:43:41.998806+0000","last_fullsized":"2024-09-10T18:43:41.998806+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'42","last_scrub_stamp":"2024-09-10T18:41:43.020357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:43.020357+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024664719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":83,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.192813+0000","last_change":"2024-09-10T18:41:44.493766+0000","last_active":"2024-09-10T18:43:42.192813+0000","last_peered":"2024-09-10T18:43:42.192813+0000","last_clean":"2024-09-10T18:43:42.192813+0000","last_became_active":"2024-09-10T18:39:25.238382+0000","last_became_peered":"2024-09-10T18:39:25.238382+0000","last_unstale":"2024-09-10T18:43:42.192813+0000","last_undegraded":"2024-09-10T18:43:42.192813+0000","last_fullsized":"2024-09-10T18:43:42.192813+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:44.493678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:41:44.493678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.075205465999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":726,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:45.684930+0000","last_change":"2024-09-10T18:42:24.132664+0000","last_active":"2024-09-10T18:43:45.684930+0000","last_peered":"2024-09-10T18:43:45.684930+0000","last_clean":"2024-09-10T18:43:45.684930+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:43:45.684930+0000","last_undegraded":"2024-09-10T18:43:45.684930+0000","last_fullsized":"2024-09-10T18:43:45.684930+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:42:24.132610+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:42:24.132610+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:51.496133+0000","scrub_duration":0.001673162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":464,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214751+0000","last_change":"2024-09-10T18:42:24.166552+0000","last_active":"2024-09-10T18:43:42.214751+0000","last_peered":"2024-09-10T18:43:42.214751+0000","last_clean":"2024-09-10T18:43:42.214751+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:43:42.214751+0000","last_undegraded":"2024-09-10T18:43:42.214751+0000","last_fullsized":"2024-09-10T18:43:42.214751+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:24.166477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:42:24.166477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:50.908290+0000","scrub_duration":0.039088676000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":477,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.220560+0000","last_change":"2024-09-10T18:42:48.651294+0000","last_active":"2024-09-10T18:43:42.220560+0000","last_peered":"2024-09-10T18:43:42.220560+0000","last_clean":"2024-09-10T18:43:42.220560+0000","last_became_active":"2024-09-10T18:34:32.818323+0000","last_became_peered":"2024-09-10T18:34:32.818323+0000","last_unstale":"2024-09-10T18:43:42.220560+0000","last_undegraded":"2024-09-10T18:43:42.220560+0000","last_fullsized":"2024-09-10T18:43:42.220560+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:48.651210+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:48.651210+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:51.416433+0000","scrub_duration":0.010854358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"400'35","reported_seq":573,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.191455+0000","last_change":"2024-09-10T18:41:31.283660+0000","last_active":"2024-09-10T18:43:42.191455+0000","last_peered":"2024-09-10T18:43:42.191455+0000","last_clean":"2024-09-10T18:43:42.191455+0000","last_became_active":"2024-09-10T18:39:00.897648+0000","last_became_peered":"2024-09-10T18:39:00.897648+0000","last_unstale":"2024-09-10T18:43:42.191455+0000","last_undegraded":"2024-09-10T18:43:42.191455+0000","last_fullsized":"2024-09-10T18:43:42.191455+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-09-10T18:41:31.283526+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:41:31.283526+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028902120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":94,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.194382+0000","last_change":"2024-09-10T18:42:24.171396+0000","last_active":"2024-09-10T18:43:42.194382+0000","last_peered":"2024-09-10T18:43:42.194382+0000","last_clean":"2024-09-10T18:43:42.194382+0000","last_became_active":"2024-09-10T18:39:00.909066+0000","last_became_peered":"2024-09-10T18:39:00.909066+0000","last_unstale":"2024-09-10T18:43:42.194382+0000","last_undegraded":"2024-09-10T18:43:42.194382+0000","last_fullsized":"2024-09-10T18:43:42.194382+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:24.171307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:42:24.171307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076532170000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"362'14","reported_seq":570,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:45.304888+0000","last_change":"2024-09-10T18:42:41.670663+0000","last_active":"2024-09-10T18:43:45.304888+0000","last_peered":"2024-09-10T18:43:45.304888+0000","last_clean":"2024-09-10T18:43:45.304888+0000","last_became_active":"2024-09-10T18:39:25.251198+0000","last_became_peered":"2024-09-10T18:39:25.251198+0000","last_unstale":"2024-09-10T18:43:45.304888+0000","last_undegraded":"2024-09-10T18:43:45.304888+0000","last_fullsized":"2024-09-10T18:43:45.304888+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'14","last_scrub_stamp":"2024-09-10T18:42:41.670583+0000","last_deep_scrub":"263'11","last_deep_scrub_stamp":"2024-09-10T18:33:01.283648+0000","last_clean_scrub_stamp":"2024-09-10T18:42:41.670583+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:56.976531+0000","scrub_duration":0.0030905279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":83,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.213421+0000","last_change":"2024-09-10T18:40:49.659503+0000","last_active":"2024-09-10T18:43:42.213421+0000","last_peered":"2024-09-10T18:43:42.213421+0000","last_clean":"2024-09-10T18:43:42.213421+0000","last_became_active":"2024-09-10T18:39:25.235363+0000","last_became_peered":"2024-09-10T18:39:25.235363+0000","last_unstale":"2024-09-10T18:43:42.213421+0000","last_undegraded":"2024-09-10T18:43:42.213421+0000","last_fullsized":"2024-09-10T18:43:42.213421+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:49.659414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:40:49.659414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002174053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":477,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.219457+0000","last_change":"2024-09-10T18:42:25.542017+0000","last_active":"2024-09-10T18:43:42.219457+0000","last_peered":"2024-09-10T18:43:42.219457+0000","last_clean":"2024-09-10T18:43:42.219457+0000","last_became_active":"2024-09-10T18:39:25.249543+0000","last_became_peered":"2024-09-10T18:39:25.249543+0000","last_unstale":"2024-09-10T18:43:42.219457+0000","last_undegraded":"2024-09-10T18:43:42.219457+0000","last_fullsized":"2024-09-10T18:43:42.219457+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:25.541937+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:25.541937+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:50.751252+0000","scrub_duration":0.0051482589999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"400'39","reported_seq":590,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.193831+0000","last_change":"2024-09-10T18:41:47.946062+0000","last_active":"2024-09-10T18:43:42.193831+0000","last_peered":"2024-09-10T18:43:42.193831+0000","last_clean":"2024-09-10T18:43:42.193831+0000","last_became_active":"2024-09-10T18:39:25.256454+0000","last_became_peered":"2024-09-10T18:39:25.256454+0000","last_unstale":"2024-09-10T18:43:42.193831+0000","last_undegraded":"2024-09-10T18:43:42.193831+0000","last_fullsized":"2024-09-10T18:43:42.193831+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'39","last_scrub_stamp":"2024-09-10T18:41:47.945963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:47.945963+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002952414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":491,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:45.214463+0000","last_change":"2024-09-10T18:43:45.214424+0000","last_active":"2024-09-10T18:43:45.214463+0000","last_peered":"2024-09-10T18:43:45.214463+0000","last_clean":"2024-09-10T18:43:45.214463+0000","last_became_active":"2024-09-10T18:39:00.897360+0000","last_became_peered":"2024-09-10T18:39:00.897360+0000","last_unstale":"2024-09-10T18:43:45.214463+0000","last_undegraded":"2024-09-10T18:43:45.214463+0000","last_fullsized":"2024-09-10T18:43:45.214463+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:45.214369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:57.001003+0000","last_clean_scrub_stamp":"2024-09-10T18:43:45.214369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:59.198754+0000","scrub_duration":0.0022281430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":477,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.221088+0000","last_change":"2024-09-10T18:42:43.761768+0000","last_active":"2024-09-10T18:43:42.221088+0000","last_peered":"2024-09-10T18:43:42.221088+0000","last_clean":"2024-09-10T18:43:42.221088+0000","last_became_active":"2024-09-10T18:39:25.243348+0000","last_became_peered":"2024-09-10T18:39:25.243348+0000","last_unstale":"2024-09-10T18:43:42.221088+0000","last_undegraded":"2024-09-10T18:43:42.221088+0000","last_fullsized":"2024-09-10T18:43:42.221088+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:43.761663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:43.761663+0000","last_clean_scrub_stamp":"2024-09-10T18:42:43.761663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:48.359894+0000","scrub_duration":0.070622993999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":472,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.191050+0000","last_change":"2024-09-10T18:42:51.229109+0000","last_active":"2024-09-10T18:43:42.191050+0000","last_peered":"2024-09-10T18:43:42.191050+0000","last_clean":"2024-09-10T18:43:42.191050+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:43:42.191050+0000","last_undegraded":"2024-09-10T18:43:42.191050+0000","last_fullsized":"2024-09-10T18:43:42.191050+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:51.228926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:42:51.228926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:20.201013+0000","scrub_duration":0.0077874529999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":104,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.193556+0000","last_change":"2024-09-10T18:42:53.232578+0000","last_active":"2024-09-10T18:43:42.193556+0000","last_peered":"2024-09-10T18:43:42.193556+0000","last_clean":"2024-09-10T18:43:42.193556+0000","last_became_active":"2024-09-10T18:39:25.246989+0000","last_became_peered":"2024-09-10T18:39:25.246989+0000","last_unstale":"2024-09-10T18:43:42.193556+0000","last_undegraded":"2024-09-10T18:43:42.193556+0000","last_fullsized":"2024-09-10T18:43:42.193556+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:53.231004+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:53.231004+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:13.083121+0000","scrub_duration":0.0074062800000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"410'64","reported_seq":286,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:45.213522+0000","last_change":"2024-09-10T18:42:43.154600+0000","last_active":"2024-09-10T18:43:45.213522+0000","last_peered":"2024-09-10T18:43:45.213522+0000","last_clean":"2024-09-10T18:43:45.213522+0000","last_became_active":"2024-09-10T18:36:12.922665+0000","last_became_peered":"2024-09-10T18:36:12.922665+0000","last_unstale":"2024-09-10T18:43:45.213522+0000","last_undegraded":"2024-09-10T18:43:45.213522+0000","last_fullsized":"2024-09-10T18:43:45.213522+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'64","last_scrub_stamp":"2024-09-10T18:42:43.154380+0000","last_deep_scrub":"410'64","last_deep_scrub_stamp":"2024-09-10T18:42:43.154380+0000","last_clean_scrub_stamp":"2024-09-10T18:42:43.154380+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:52.804641+0000","scrub_duration":0.023655347,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":477,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.999888+0000","last_change":"2024-09-10T18:42:27.214322+0000","last_active":"2024-09-10T18:43:41.999888+0000","last_peered":"2024-09-10T18:43:41.999888+0000","last_clean":"2024-09-10T18:43:41.999888+0000","last_became_active":"2024-09-10T18:39:25.242740+0000","last_became_peered":"2024-09-10T18:39:25.242740+0000","last_unstale":"2024-09-10T18:43:41.999888+0000","last_undegraded":"2024-09-10T18:43:41.999888+0000","last_fullsized":"2024-09-10T18:43:41.999888+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:27.214218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.274219+0000","last_clean_scrub_stamp":"2024-09-10T18:42:27.214218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015470144999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":98,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.194219+0000","last_change":"2024-09-10T18:43:38.916884+0000","last_active":"2024-09-10T18:43:42.194219+0000","last_peered":"2024-09-10T18:43:42.194219+0000","last_clean":"2024-09-10T18:43:42.194219+0000","last_became_active":"2024-09-10T18:39:00.909579+0000","last_became_peered":"2024-09-10T18:39:00.909579+0000","last_unstale":"2024-09-10T18:43:42.194219+0000","last_undegraded":"2024-09-10T18:43:42.194219+0000","last_fullsized":"2024-09-10T18:43:42.194219+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:43:38.916818+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:43:38.916818+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:59.275534+0000","scrub_duration":0.001542622,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":93,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.191995+0000","last_change":"2024-09-10T18:43:35.177954+0000","last_active":"2024-09-10T18:43:42.191995+0000","last_peered":"2024-09-10T18:43:42.191995+0000","last_clean":"2024-09-10T18:43:42.191995+0000","last_became_active":"2024-09-10T18:39:25.255658+0000","last_became_peered":"2024-09-10T18:39:25.255658+0000","last_unstale":"2024-09-10T18:43:42.191995+0000","last_undegraded":"2024-09-10T18:43:42.191995+0000","last_fullsized":"2024-09-10T18:43:42.191995+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:35.177898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:43:35.177898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:44.616947+0000","scrub_duration":0.0015975850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":450,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:46.723497+0000","last_change":"2024-09-10T18:43:46.723465+0000","last_active":"2024-09-10T18:43:46.723497+0000","last_peered":"2024-09-10T18:43:46.723497+0000","last_clean":"2024-09-10T18:43:46.723497+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:43:46.723497+0000","last_undegraded":"2024-09-10T18:43:46.723497+0000","last_fullsized":"2024-09-10T18:43:46.723497+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:46.723409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:43:46.723409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:51.420967+0000","scrub_duration":0.0014194520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":521,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.231920+0000","last_change":"2024-09-10T18:41:33.333442+0000","last_active":"2024-09-10T18:43:42.231920+0000","last_peered":"2024-09-10T18:43:42.231920+0000","last_clean":"2024-09-10T18:43:42.231920+0000","last_became_active":"2024-09-10T18:39:25.240048+0000","last_became_peered":"2024-09-10T18:39:25.240048+0000","last_unstale":"2024-09-10T18:43:42.231920+0000","last_undegraded":"2024-09-10T18:43:42.231920+0000","last_fullsized":"2024-09-10T18:43:42.231920+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:33.333247+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:33.333247+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057228740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"400'26","reported_seq":67,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.996619+0000","last_change":"2024-09-10T18:42:38.544026+0000","last_active":"2024-09-10T18:43:41.996619+0000","last_peered":"2024-09-10T18:43:41.996619+0000","last_clean":"2024-09-10T18:43:41.996619+0000","last_became_active":"2024-09-10T18:41:11.933724+0000","last_became_peered":"2024-09-10T18:41:11.933724+0000","last_unstale":"2024-09-10T18:43:41.996619+0000","last_undegraded":"2024-09-10T18:43:41.996619+0000","last_fullsized":"2024-09-10T18:43:41.996619+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'26","last_scrub_stamp":"2024-09-10T18:42:38.543932+0000","last_deep_scrub":"388'22","last_deep_scrub_stamp":"2024-09-10T18:41:01.248929+0000","last_clean_scrub_stamp":"2024-09-10T18:42:38.543932+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:49.910982+0000","scrub_duration":0.025590214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":488,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.190842+0000","last_change":"2024-09-10T18:42:44.171535+0000","last_active":"2024-09-10T18:43:42.190842+0000","last_peered":"2024-09-10T18:43:42.190842+0000","last_clean":"2024-09-10T18:43:42.190842+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:43:42.190842+0000","last_undegraded":"2024-09-10T18:43:42.190842+0000","last_fullsized":"2024-09-10T18:43:42.190842+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:44.171445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:44.171445+0000","last_clean_scrub_stamp":"2024-09-10T18:42:44.171445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:47.474299+0000","scrub_duration":0.015055643,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":532,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:47.210027+0000","last_change":"2024-09-10T18:43:47.209997+0000","last_active":"2024-09-10T18:43:47.210027+0000","last_peered":"2024-09-10T18:43:47.210027+0000","last_clean":"2024-09-10T18:43:47.210027+0000","last_became_active":"2024-09-10T18:39:25.249566+0000","last_became_peered":"2024-09-10T18:39:25.249566+0000","last_unstale":"2024-09-10T18:43:47.210027+0000","last_undegraded":"2024-09-10T18:43:47.210027+0000","last_fullsized":"2024-09-10T18:43:47.210027+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:43:47.209937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:43:47.209937+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:59.440485+0000","scrub_duration":0.001441009,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":515,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.997994+0000","last_change":"2024-09-10T18:42:52.399679+0000","last_active":"2024-09-10T18:43:41.997994+0000","last_peered":"2024-09-10T18:43:41.997994+0000","last_clean":"2024-09-10T18:43:41.997994+0000","last_became_active":"2024-09-10T18:39:00.910574+0000","last_became_peered":"2024-09-10T18:39:00.910574+0000","last_unstale":"2024-09-10T18:43:41.997994+0000","last_undegraded":"2024-09-10T18:43:41.997994+0000","last_fullsized":"2024-09-10T18:43:41.997994+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:52.399594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:42:52.399594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:05.668192+0000","scrub_duration":0.0030284679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":464,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.229379+0000","last_change":"2024-09-10T18:42:36.178847+0000","last_active":"2024-09-10T18:43:42.229379+0000","last_peered":"2024-09-10T18:43:42.229379+0000","last_clean":"2024-09-10T18:43:42.229379+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:43:42.229379+0000","last_undegraded":"2024-09-10T18:43:42.229379+0000","last_fullsized":"2024-09-10T18:43:42.229379+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:36.178756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:42:36.178756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:49.301231+0000","scrub_duration":0.012545664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":391,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.194454+0000","last_change":"2024-09-10T18:41:47.469267+0000","last_active":"2024-09-10T18:43:42.194454+0000","last_peered":"2024-09-10T18:43:42.194454+0000","last_clean":"2024-09-10T18:43:42.194454+0000","last_became_active":"2024-09-10T18:34:00.165384+0000","last_became_peered":"2024-09-10T18:34:00.165384+0000","last_unstale":"2024-09-10T18:43:42.194454+0000","last_undegraded":"2024-09-10T18:43:42.194454+0000","last_fullsized":"2024-09-10T18:43:42.194454+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:47.468155+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:47.468155+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041388789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"400'36","reported_seq":377,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.191793+0000","last_change":"2024-09-10T18:41:32.325494+0000","last_active":"2024-09-10T18:43:42.191793+0000","last_peered":"2024-09-10T18:43:42.191793+0000","last_clean":"2024-09-10T18:43:42.191793+0000","last_became_active":"2024-09-10T18:39:25.263465+0000","last_became_peered":"2024-09-10T18:39:25.263465+0000","last_unstale":"2024-09-10T18:43:42.191793+0000","last_undegraded":"2024-09-10T18:43:42.191793+0000","last_fullsized":"2024-09-10T18:43:42.191793+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-09-10T18:41:32.325283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:32.325283+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058952759999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":93,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.998908+0000","last_change":"2024-09-10T18:41:49.077680+0000","last_active":"2024-09-10T18:43:41.998908+0000","last_peered":"2024-09-10T18:43:41.998908+0000","last_clean":"2024-09-10T18:43:41.998908+0000","last_became_active":"2024-09-10T18:39:25.239068+0000","last_became_peered":"2024-09-10T18:39:25.239068+0000","last_unstale":"2024-09-10T18:43:41.998908+0000","last_undegraded":"2024-09-10T18:43:41.998908+0000","last_fullsized":"2024-09-10T18:43:41.998908+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:49.077609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:41:49.077609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007378697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":1821,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:48.728985+0000","last_change":"2024-09-10T18:42:43.071764+0000","last_active":"2024-09-10T18:43:48.728985+0000","last_peered":"2024-09-10T18:43:48.728985+0000","last_clean":"2024-09-10T18:43:48.728985+0000","last_became_active":"2024-09-10T18:39:25.245563+0000","last_became_peered":"2024-09-10T18:39:25.245563+0000","last_unstale":"2024-09-10T18:43:48.728985+0000","last_undegraded":"2024-09-10T18:43:48.728985+0000","last_fullsized":"2024-09-10T18:43:48.728985+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:42:43.071674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:42:43.071674+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074519470000000004,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1329,"num_read_kb":1324,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":94,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.194931+0000","last_change":"2024-09-10T18:41:28.868786+0000","last_active":"2024-09-10T18:43:42.194931+0000","last_peered":"2024-09-10T18:43:42.194931+0000","last_clean":"2024-09-10T18:43:42.194931+0000","last_became_active":"2024-09-10T18:39:25.255881+0000","last_became_peered":"2024-09-10T18:39:25.255881+0000","last_unstale":"2024-09-10T18:43:42.194931+0000","last_undegraded":"2024-09-10T18:43:42.194931+0000","last_fullsized":"2024-09-10T18:43:42.194931+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:28.868225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.170272+0000","last_clean_scrub_stamp":"2024-09-10T18:41:28.868225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038280509999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":485,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.191355+0000","last_change":"2024-09-10T18:43:41.206735+0000","last_active":"2024-09-10T18:43:42.191355+0000","last_peered":"2024-09-10T18:43:42.191355+0000","last_clean":"2024-09-10T18:43:42.191355+0000","last_became_active":"2024-09-10T18:39:00.897525+0000","last_became_peered":"2024-09-10T18:39:00.897525+0000","last_unstale":"2024-09-10T18:43:42.191355+0000","last_undegraded":"2024-09-10T18:43:42.191355+0000","last_fullsized":"2024-09-10T18:43:42.191355+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:41.206680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:43:41.206680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:57.810552+0000","scrub_duration":0.0014654570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"400'38","reported_seq":492,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.227722+0000","last_change":"2024-09-10T18:42:50.769679+0000","last_active":"2024-09-10T18:43:42.227722+0000","last_peered":"2024-09-10T18:43:42.227722+0000","last_clean":"2024-09-10T18:43:42.227722+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:43:42.227722+0000","last_undegraded":"2024-09-10T18:43:42.227722+0000","last_fullsized":"2024-09-10T18:43:42.227722+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'38","last_scrub_stamp":"2024-09-10T18:42:50.769598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:42:50.769598+0000","objects_scrubbed":7,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:54.640604+0000","scrub_duration":0.063831196000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":490,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214980+0000","last_change":"2024-09-10T18:43:40.704564+0000","last_active":"2024-09-10T18:43:42.214980+0000","last_peered":"2024-09-10T18:43:42.214980+0000","last_clean":"2024-09-10T18:43:42.214980+0000","last_became_active":"2024-09-10T18:39:00.894778+0000","last_became_peered":"2024-09-10T18:39:00.894778+0000","last_unstale":"2024-09-10T18:43:42.214980+0000","last_undegraded":"2024-09-10T18:43:42.214980+0000","last_fullsized":"2024-09-10T18:43:42.214980+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:40.704509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:43:40.704509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:01.658035+0000","scrub_duration":0.001721209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":540,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214428+0000","last_change":"2024-09-10T18:42:55.149653+0000","last_active":"2024-09-10T18:43:42.214428+0000","last_peered":"2024-09-10T18:43:42.214428+0000","last_clean":"2024-09-10T18:43:42.214428+0000","last_became_active":"2024-09-10T18:39:25.251895+0000","last_became_peered":"2024-09-10T18:39:25.251895+0000","last_unstale":"2024-09-10T18:43:42.214428+0000","last_undegraded":"2024-09-10T18:43:42.214428+0000","last_fullsized":"2024-09-10T18:43:42.214428+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:55.149567+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:55.149567+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:02.105687+0000","scrub_duration":0.00408326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":489,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.216062+0000","last_change":"2024-09-10T18:42:56.158057+0000","last_active":"2024-09-10T18:43:42.216062+0000","last_peered":"2024-09-10T18:43:42.216062+0000","last_clean":"2024-09-10T18:43:42.216062+0000","last_became_active":"2024-09-10T18:39:00.890829+0000","last_became_peered":"2024-09-10T18:39:00.890829+0000","last_unstale":"2024-09-10T18:43:42.216062+0000","last_undegraded":"2024-09-10T18:43:42.216062+0000","last_fullsized":"2024-09-10T18:43:42.216062+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:56.157930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:42:56.157930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:05.652530+0000","scrub_duration":0.0021092350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":84,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.194833+0000","last_change":"2024-09-10T18:41:34.883649+0000","last_active":"2024-09-10T18:43:42.194833+0000","last_peered":"2024-09-10T18:43:42.194833+0000","last_clean":"2024-09-10T18:43:42.194833+0000","last_became_active":"2024-09-10T18:39:00.900703+0000","last_became_peered":"2024-09-10T18:39:00.900703+0000","last_unstale":"2024-09-10T18:43:42.194833+0000","last_undegraded":"2024-09-10T18:43:42.194833+0000","last_fullsized":"2024-09-10T18:43:42.194833+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:34.883463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:09.749627+0000","last_clean_scrub_stamp":"2024-09-10T18:41:34.883463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041636060000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"76'81","reported_seq":437,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.194485+0000","last_change":"2024-09-10T18:42:41.127739+0000","last_active":"2024-09-10T18:43:42.194485+0000","last_peered":"2024-09-10T18:43:42.194485+0000","last_clean":"2024-09-10T18:43:42.194485+0000","last_became_active":"2024-09-10T18:39:25.263369+0000","last_became_peered":"2024-09-10T18:39:25.263369+0000","last_unstale":"2024-09-10T18:43:42.194485+0000","last_undegraded":"2024-09-10T18:43:42.194485+0000","last_fullsized":"2024-09-10T18:43:42.194485+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:41.127360+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:41.127360+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:56.780587+0000","scrub_duration":0.0078382469999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"400'32","reported_seq":526,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:48.256786+0000","last_change":"2024-09-10T18:43:48.256748+0000","last_active":"2024-09-10T18:43:48.256786+0000","last_peered":"2024-09-10T18:43:48.256786+0000","last_clean":"2024-09-10T18:43:48.256786+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:43:48.256786+0000","last_undegraded":"2024-09-10T18:43:48.256786+0000","last_fullsized":"2024-09-10T18:43:48.256786+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'32","last_scrub_stamp":"2024-09-10T18:43:48.256679+0000","last_deep_scrub":"283'24","last_deep_scrub_stamp":"2024-09-10T18:34:19.404713+0000","last_clean_scrub_stamp":"2024-09-10T18:43:48.256679+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:50.702455+0000","scrub_duration":0.0022131690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":463,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.998911+0000","last_change":"2024-09-10T18:41:44.986845+0000","last_active":"2024-09-10T18:43:41.998911+0000","last_peered":"2024-09-10T18:43:41.998911+0000","last_clean":"2024-09-10T18:43:41.998911+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:43:41.998911+0000","last_undegraded":"2024-09-10T18:43:41.998911+0000","last_fullsized":"2024-09-10T18:43:41.998911+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:44.986765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:41:44.986765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079498090000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":94,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.194723+0000","last_change":"2024-09-10T18:42:45.223626+0000","last_active":"2024-09-10T18:43:42.194723+0000","last_peered":"2024-09-10T18:43:42.194723+0000","last_clean":"2024-09-10T18:43:42.194723+0000","last_became_active":"2024-09-10T18:39:00.907231+0000","last_became_peered":"2024-09-10T18:39:00.907231+0000","last_unstale":"2024-09-10T18:43:42.194723+0000","last_undegraded":"2024-09-10T18:43:42.194723+0000","last_fullsized":"2024-09-10T18:43:42.194723+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:45.223551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:42:45.223551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:51.395703+0000","scrub_duration":0.0026937710000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":484,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214026+0000","last_change":"2024-09-10T18:42:51.629243+0000","last_active":"2024-09-10T18:43:42.214026+0000","last_peered":"2024-09-10T18:43:42.214026+0000","last_clean":"2024-09-10T18:43:42.214026+0000","last_became_active":"2024-09-10T18:39:00.894963+0000","last_became_peered":"2024-09-10T18:39:00.894963+0000","last_unstale":"2024-09-10T18:43:42.214026+0000","last_undegraded":"2024-09-10T18:43:42.214026+0000","last_fullsized":"2024-09-10T18:43:42.214026+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:51.629160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:42:51.629160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:10.720004+0000","scrub_duration":0.00527298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":94,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.193994+0000","last_change":"2024-09-10T18:42:49.285431+0000","last_active":"2024-09-10T18:43:42.193994+0000","last_peered":"2024-09-10T18:43:42.193994+0000","last_clean":"2024-09-10T18:43:42.193994+0000","last_became_active":"2024-09-10T18:39:00.909151+0000","last_became_peered":"2024-09-10T18:39:00.909151+0000","last_unstale":"2024-09-10T18:43:42.193994+0000","last_undegraded":"2024-09-10T18:43:42.193994+0000","last_fullsized":"2024-09-10T18:43:42.193994+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:49.285168+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:49.285168+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:52.101220+0000","scrub_duration":0.0068966280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"400'53","reported_seq":602,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:49.255517+0000","last_change":"2024-09-10T18:43:49.255479+0000","last_active":"2024-09-10T18:43:49.255517+0000","last_peered":"2024-09-10T18:43:49.255517+0000","last_clean":"2024-09-10T18:43:49.255517+0000","last_became_active":"2024-09-10T18:38:43.333186+0000","last_became_peered":"2024-09-10T18:38:43.333186+0000","last_unstale":"2024-09-10T18:43:49.255517+0000","last_undegraded":"2024-09-10T18:43:49.255517+0000","last_fullsized":"2024-09-10T18:43:49.255517+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'53","last_scrub_stamp":"2024-09-10T18:43:49.255412+0000","last_deep_scrub":"342'39","last_deep_scrub_stamp":"2024-09-10T18:37:53.386563+0000","last_clean_scrub_stamp":"2024-09-10T18:43:49.255412+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:03.977659+0000","scrub_duration":0.0020536460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":445,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.221744+0000","last_change":"2024-09-10T18:42:42.709960+0000","last_active":"2024-09-10T18:43:42.221744+0000","last_peered":"2024-09-10T18:43:42.221744+0000","last_clean":"2024-09-10T18:43:42.221744+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:43:42.221744+0000","last_undegraded":"2024-09-10T18:43:42.221744+0000","last_fullsized":"2024-09-10T18:43:42.221744+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:42:42.709888+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:27:17.554184+0000","last_clean_scrub_stamp":"2024-09-10T18:42:42.709888+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:11.381192+0000","scrub_duration":0.011345328999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":477,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.191421+0000","last_change":"2024-09-10T18:41:35.323346+0000","last_active":"2024-09-10T18:43:42.191421+0000","last_peered":"2024-09-10T18:43:42.191421+0000","last_clean":"2024-09-10T18:43:42.191421+0000","last_became_active":"2024-09-10T18:39:25.249337+0000","last_became_peered":"2024-09-10T18:39:25.249337+0000","last_unstale":"2024-09-10T18:43:42.191421+0000","last_undegraded":"2024-09-10T18:43:42.191421+0000","last_fullsized":"2024-09-10T18:43:42.191421+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:35.323254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:41:35.323254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001933265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":503,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.997444+0000","last_change":"2024-09-10T18:42:45.518178+0000","last_active":"2024-09-10T18:43:41.997444+0000","last_peered":"2024-09-10T18:43:41.997444+0000","last_clean":"2024-09-10T18:43:41.997444+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:43:41.997444+0000","last_undegraded":"2024-09-10T18:43:41.997444+0000","last_fullsized":"2024-09-10T18:43:41.997444+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:45.517976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:42:45.517976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:00.331373+0000","scrub_duration":0.019939069,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":518,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214030+0000","last_change":"2024-09-10T18:41:43.367065+0000","last_active":"2024-09-10T18:43:42.214030+0000","last_peered":"2024-09-10T18:43:42.214030+0000","last_clean":"2024-09-10T18:43:42.214030+0000","last_became_active":"2024-09-10T18:34:32.827174+0000","last_became_peered":"2024-09-10T18:34:32.827174+0000","last_unstale":"2024-09-10T18:43:42.214030+0000","last_undegraded":"2024-09-10T18:43:42.214030+0000","last_fullsized":"2024-09-10T18:43:42.214030+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:43.366981+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:43.366981+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018640970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"400'35","reported_seq":243,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.215783+0000","last_change":"2024-09-10T18:43:39.890749+0000","last_active":"2024-09-10T18:43:42.215783+0000","last_peered":"2024-09-10T18:43:42.215783+0000","last_clean":"2024-09-10T18:43:42.215783+0000","last_became_active":"2024-09-10T18:32:51.118251+0000","last_became_peered":"2024-09-10T18:32:51.118251+0000","last_unstale":"2024-09-10T18:43:42.215783+0000","last_undegraded":"2024-09-10T18:43:42.215783+0000","last_fullsized":"2024-09-10T18:43:42.215783+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'35","last_scrub_stamp":"2024-09-10T18:43:39.890696+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-09-10T18:23:23.566732+0000","last_clean_scrub_stamp":"2024-09-10T18:43:39.890696+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:53.886383+0000","scrub_duration":0.0037577629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":463,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.216359+0000","last_change":"2024-09-10T18:40:59.757340+0000","last_active":"2024-09-10T18:43:42.216359+0000","last_peered":"2024-09-10T18:43:42.216359+0000","last_clean":"2024-09-10T18:43:42.216359+0000","last_became_active":"2024-09-10T18:39:25.251419+0000","last_became_peered":"2024-09-10T18:39:25.251419+0000","last_unstale":"2024-09-10T18:43:42.216359+0000","last_undegraded":"2024-09-10T18:43:42.216359+0000","last_fullsized":"2024-09-10T18:43:42.216359+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:59.757255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:59.757255+0000","last_clean_scrub_stamp":"2024-09-10T18:40:59.757255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056131599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"357'11","reported_seq":811,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:45.024514+0000","last_change":"2024-09-10T18:42:30.463021+0000","last_active":"2024-09-10T18:43:45.024514+0000","last_peered":"2024-09-10T18:43:45.024514+0000","last_clean":"2024-09-10T18:43:45.024514+0000","last_became_active":"2024-09-10T18:39:00.909747+0000","last_became_peered":"2024-09-10T18:39:00.909747+0000","last_unstale":"2024-09-10T18:43:45.024514+0000","last_undegraded":"2024-09-10T18:43:45.024514+0000","last_fullsized":"2024-09-10T18:43:45.024514+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'11","last_scrub_stamp":"2024-09-10T18:42:30.462929+0000","last_deep_scrub":"189'9","last_deep_scrub_stamp":"2024-09-10T18:28:11.017903+0000","last_clean_scrub_stamp":"2024-09-10T18:42:30.462929+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:56.745815+0000","scrub_duration":0.0039261499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":457,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.999811+0000","last_change":"2024-09-10T18:43:31.306531+0000","last_active":"2024-09-10T18:43:41.999811+0000","last_peered":"2024-09-10T18:43:41.999811+0000","last_clean":"2024-09-10T18:43:41.999811+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:43:41.999811+0000","last_undegraded":"2024-09-10T18:43:41.999811+0000","last_fullsized":"2024-09-10T18:43:41.999811+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:31.306467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:53.987811+0000","last_clean_scrub_stamp":"2024-09-10T18:43:31.306467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:48.460616+0000","scrub_duration":0.0015706909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"76'81","reported_seq":200,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.999708+0000","last_change":"2024-09-10T18:42:53.168299+0000","last_active":"2024-09-10T18:43:41.999708+0000","last_peered":"2024-09-10T18:43:41.999708+0000","last_clean":"2024-09-10T18:43:41.999708+0000","last_became_active":"2024-09-10T18:34:32.818251+0000","last_became_peered":"2024-09-10T18:34:32.818251+0000","last_unstale":"2024-09-10T18:43:41.999708+0000","last_undegraded":"2024-09-10T18:43:41.999708+0000","last_fullsized":"2024-09-10T18:43:41.999708+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:53.168123+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:42:53.168123+0000","last_clean_scrub_stamp":"2024-09-10T18:42:53.168123+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:08.938913+0000","scrub_duration":0.0068350160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"400'25","reported_seq":441,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.194357+0000","last_change":"2024-09-10T18:42:23.354906+0000","last_active":"2024-09-10T18:43:42.194357+0000","last_peered":"2024-09-10T18:43:42.194357+0000","last_clean":"2024-09-10T18:43:42.194357+0000","last_became_active":"2024-09-10T18:39:25.240389+0000","last_became_peered":"2024-09-10T18:39:25.240389+0000","last_unstale":"2024-09-10T18:43:42.194357+0000","last_undegraded":"2024-09-10T18:43:42.194357+0000","last_fullsized":"2024-09-10T18:43:42.194357+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'25","last_scrub_stamp":"2024-09-10T18:42:23.354801+0000","last_deep_scrub":"302'19","last_deep_scrub_stamp":"2024-09-10T18:35:18.836131+0000","last_clean_scrub_stamp":"2024-09-10T18:42:23.354801+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094838619999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"265'2","reported_seq":98,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.194072+0000","last_change":"2024-09-10T18:42:34.273635+0000","last_active":"2024-09-10T18:43:42.194072+0000","last_peered":"2024-09-10T18:43:42.194072+0000","last_clean":"2024-09-10T18:43:42.194072+0000","last_became_active":"2024-09-10T18:39:00.904967+0000","last_became_peered":"2024-09-10T18:39:00.904967+0000","last_unstale":"2024-09-10T18:43:42.194072+0000","last_undegraded":"2024-09-10T18:43:42.194072+0000","last_fullsized":"2024-09-10T18:43:42.194072+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'2","last_scrub_stamp":"2024-09-10T18:42:34.273553+0000","last_deep_scrub":"265'2","last_deep_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_clean_scrub_stamp":"2024-09-10T18:42:34.273553+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:56.756564+0000","scrub_duration":0.0028652539999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":491,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.998778+0000","last_change":"2024-09-10T18:42:28.510786+0000","last_active":"2024-09-10T18:43:41.998778+0000","last_peered":"2024-09-10T18:43:41.998778+0000","last_clean":"2024-09-10T18:43:41.998778+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:43:41.998778+0000","last_undegraded":"2024-09-10T18:43:41.998778+0000","last_fullsized":"2024-09-10T18:43:41.998778+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:28.510725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:42:28.510725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:49.810546+0000","scrub_duration":0.0053696639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":83,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.213796+0000","last_change":"2024-09-10T18:41:35.361220+0000","last_active":"2024-09-10T18:43:42.213796+0000","last_peered":"2024-09-10T18:43:42.213796+0000","last_clean":"2024-09-10T18:43:42.213796+0000","last_became_active":"2024-09-10T18:39:25.237150+0000","last_became_peered":"2024-09-10T18:39:25.237150+0000","last_unstale":"2024-09-10T18:43:42.213796+0000","last_undegraded":"2024-09-10T18:43:42.213796+0000","last_fullsized":"2024-09-10T18:43:42.213796+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:35.361130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:41:35.361130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023350010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"76'81","reported_seq":486,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.220626+0000","last_change":"2024-09-10T18:43:39.685178+0000","last_active":"2024-09-10T18:43:42.220626+0000","last_peered":"2024-09-10T18:43:42.220626+0000","last_clean":"2024-09-10T18:43:42.220626+0000","last_became_active":"2024-09-10T18:39:00.921868+0000","last_became_peered":"2024-09-10T18:39:00.921868+0000","last_unstale":"2024-09-10T18:43:42.220626+0000","last_undegraded":"2024-09-10T18:43:42.220626+0000","last_fullsized":"2024-09-10T18:43:42.220626+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:43:39.685121+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:34:15.938079+0000","last_clean_scrub_stamp":"2024-09-10T18:43:39.685121+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:39.706905+0000","scrub_duration":0.001249322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"400'60","reported_seq":596,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.708250+0000","last_change":"2024-09-10T18:43:42.708206+0000","last_active":"2024-09-10T18:43:42.708250+0000","last_peered":"2024-09-10T18:43:42.708250+0000","last_clean":"2024-09-10T18:43:42.708250+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:43:42.708250+0000","last_undegraded":"2024-09-10T18:43:42.708250+0000","last_fullsized":"2024-09-10T18:43:42.708250+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'60","last_scrub_stamp":"2024-09-10T18:43:42.708150+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:43:42.708150+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:04.681120+0000","scrub_duration":0.0025100349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":500,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.232063+0000","last_change":"2024-09-10T18:42:50.123722+0000","last_active":"2024-09-10T18:43:42.232063+0000","last_peered":"2024-09-10T18:43:42.232063+0000","last_clean":"2024-09-10T18:43:42.232063+0000","last_became_active":"2024-09-10T18:39:00.895121+0000","last_became_peered":"2024-09-10T18:39:00.895121+0000","last_unstale":"2024-09-10T18:43:42.232063+0000","last_undegraded":"2024-09-10T18:43:42.232063+0000","last_fullsized":"2024-09-10T18:43:42.232063+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:50.123643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:42:50.123643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:00.402908+0000","scrub_duration":0.0020317809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":535,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.996717+0000","last_change":"2024-09-10T18:43:35.332792+0000","last_active":"2024-09-10T18:43:41.996717+0000","last_peered":"2024-09-10T18:43:41.996717+0000","last_clean":"2024-09-10T18:43:41.996717+0000","last_became_active":"2024-09-10T18:39:25.240291+0000","last_became_peered":"2024-09-10T18:39:25.240291+0000","last_unstale":"2024-09-10T18:43:41.996717+0000","last_undegraded":"2024-09-10T18:43:41.996717+0000","last_fullsized":"2024-09-10T18:43:41.996717+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:35.332738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:34:21.447291+0000","last_clean_scrub_stamp":"2024-09-10T18:43:35.332738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:51.682706+0000","scrub_duration":0.001525294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":467,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214355+0000","last_change":"2024-09-10T18:42:28.075463+0000","last_active":"2024-09-10T18:43:42.214355+0000","last_peered":"2024-09-10T18:43:42.214355+0000","last_clean":"2024-09-10T18:43:42.214355+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:43:42.214355+0000","last_undegraded":"2024-09-10T18:43:42.214355+0000","last_fullsized":"2024-09-10T18:43:42.214355+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:28.075334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:04.058646+0000","last_clean_scrub_stamp":"2024-09-10T18:42:28.075334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:47.297976+0000","scrub_duration":0.0028979330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"76'81","reported_seq":408,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214243+0000","last_change":"2024-09-10T18:41:41.914514+0000","last_active":"2024-09-10T18:43:42.214243+0000","last_peered":"2024-09-10T18:43:42.214243+0000","last_clean":"2024-09-10T18:43:42.214243+0000","last_became_active":"2024-09-10T18:39:25.242996+0000","last_became_peered":"2024-09-10T18:39:25.242996+0000","last_unstale":"2024-09-10T18:43:42.214243+0000","last_undegraded":"2024-09-10T18:43:42.214243+0000","last_fullsized":"2024-09-10T18:43:42.214243+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:41.914420+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:41.914420+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079109539999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"400'29","reported_seq":517,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.998723+0000","last_change":"2024-09-10T18:43:28.352882+0000","last_active":"2024-09-10T18:43:41.998723+0000","last_peered":"2024-09-10T18:43:41.998723+0000","last_clean":"2024-09-10T18:43:41.998723+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:43:41.998723+0000","last_undegraded":"2024-09-10T18:43:41.998723+0000","last_fullsized":"2024-09-10T18:43:41.998723+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'29","last_scrub_stamp":"2024-09-10T18:43:28.352806+0000","last_deep_scrub":"301'23","last_deep_scrub_stamp":"2024-09-10T18:40:55.034050+0000","last_clean_scrub_stamp":"2024-09-10T18:43:28.352806+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:54.227912+0000","scrub_duration":0.0040501570000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":470,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214286+0000","last_change":"2024-09-10T18:41:48.482843+0000","last_active":"2024-09-10T18:43:42.214286+0000","last_peered":"2024-09-10T18:43:42.214286+0000","last_clean":"2024-09-10T18:43:42.214286+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:43:42.214286+0000","last_undegraded":"2024-09-10T18:43:42.214286+0000","last_fullsized":"2024-09-10T18:43:42.214286+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:48.482677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:41:48.482677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.052469623,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":515,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.996369+0000","last_change":"2024-09-10T18:42:42.444073+0000","last_active":"2024-09-10T18:43:41.996369+0000","last_peered":"2024-09-10T18:43:41.996369+0000","last_clean":"2024-09-10T18:43:41.996369+0000","last_became_active":"2024-09-10T18:39:25.264561+0000","last_became_peered":"2024-09-10T18:39:25.264561+0000","last_unstale":"2024-09-10T18:43:41.996369+0000","last_undegraded":"2024-09-10T18:43:41.996369+0000","last_fullsized":"2024-09-10T18:43:41.996369+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:42.443984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:42.443984+0000","last_clean_scrub_stamp":"2024-09-10T18:42:42.443984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:08.013826+0000","scrub_duration":0.001657424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":447,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.216039+0000","last_change":"2024-09-10T18:41:31.850183+0000","last_active":"2024-09-10T18:43:42.216039+0000","last_peered":"2024-09-10T18:43:42.216039+0000","last_clean":"2024-09-10T18:43:42.216039+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:43:42.216039+0000","last_undegraded":"2024-09-10T18:43:42.216039+0000","last_fullsized":"2024-09-10T18:43:42.216039+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:31.850086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:38:50.727231+0000","last_clean_scrub_stamp":"2024-09-10T18:41:31.850086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088275699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"76'81","reported_seq":190,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214427+0000","last_change":"2024-09-10T18:41:45.417035+0000","last_active":"2024-09-10T18:43:42.214427+0000","last_peered":"2024-09-10T18:43:42.214427+0000","last_clean":"2024-09-10T18:43:42.214427+0000","last_became_active":"2024-09-10T18:34:32.822196+0000","last_became_peered":"2024-09-10T18:34:32.822196+0000","last_unstale":"2024-09-10T18:43:42.214427+0000","last_undegraded":"2024-09-10T18:43:42.214427+0000","last_fullsized":"2024-09-10T18:43:42.214427+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:45.416956+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:45.416956+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027361949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"400'39","reported_seq":128,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.193632+0000","last_change":"2024-09-10T18:41:49.939121+0000","last_active":"2024-09-10T18:43:42.193632+0000","last_peered":"2024-09-10T18:43:42.193632+0000","last_clean":"2024-09-10T18:43:42.193632+0000","last_became_active":"2024-09-10T18:39:00.902783+0000","last_became_peered":"2024-09-10T18:39:00.902783+0000","last_unstale":"2024-09-10T18:43:42.193632+0000","last_undegraded":"2024-09-10T18:43:42.193632+0000","last_fullsized":"2024-09-10T18:43:42.193632+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'39","last_scrub_stamp":"2024-09-10T18:41:49.939038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:49.939038+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027044299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":486,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.216940+0000","last_change":"2024-09-10T18:41:39.392189+0000","last_active":"2024-09-10T18:43:42.216940+0000","last_peered":"2024-09-10T18:43:42.216940+0000","last_clean":"2024-09-10T18:43:42.216940+0000","last_became_active":"2024-09-10T18:39:25.238037+0000","last_became_peered":"2024-09-10T18:39:25.238037+0000","last_unstale":"2024-09-10T18:43:42.216940+0000","last_undegraded":"2024-09-10T18:43:42.216940+0000","last_fullsized":"2024-09-10T18:43:42.216940+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:41:39.391952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:41:39.391952+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076173680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":742,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:45.248843+0000","last_change":"2024-09-10T18:41:38.346953+0000","last_active":"2024-09-10T18:43:45.248843+0000","last_peered":"2024-09-10T18:43:45.248843+0000","last_clean":"2024-09-10T18:43:45.248843+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:43:45.248843+0000","last_undegraded":"2024-09-10T18:43:45.248843+0000","last_fullsized":"2024-09-10T18:43:45.248843+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:41:38.346859+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:41:38.346859+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012291041000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":94,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.193741+0000","last_change":"2024-09-10T18:41:31.870285+0000","last_active":"2024-09-10T18:43:42.193741+0000","last_peered":"2024-09-10T18:43:42.193741+0000","last_clean":"2024-09-10T18:43:42.193741+0000","last_became_active":"2024-09-10T18:39:25.242197+0000","last_became_peered":"2024-09-10T18:39:25.242197+0000","last_unstale":"2024-09-10T18:43:42.193741+0000","last_undegraded":"2024-09-10T18:43:42.193741+0000","last_fullsized":"2024-09-10T18:43:42.193741+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:31.870211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:41:31.870211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089146250000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'81","reported_seq":94,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.194209+0000","last_change":"2024-09-10T18:42:48.242728+0000","last_active":"2024-09-10T18:43:42.194209+0000","last_peered":"2024-09-10T18:43:42.194209+0000","last_clean":"2024-09-10T18:43:42.194209+0000","last_became_active":"2024-09-10T18:39:00.910728+0000","last_became_peered":"2024-09-10T18:39:00.910728+0000","last_unstale":"2024-09-10T18:43:42.194209+0000","last_undegraded":"2024-09-10T18:43:42.194209+0000","last_fullsized":"2024-09-10T18:43:42.194209+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:48.242648+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:48.242648+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:10.021858+0000","scrub_duration":0.0048037560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":466,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.213448+0000","last_change":"2024-09-10T18:43:32.791179+0000","last_active":"2024-09-10T18:43:42.213448+0000","last_peered":"2024-09-10T18:43:42.213448+0000","last_clean":"2024-09-10T18:43:42.213448+0000","last_became_active":"2024-09-10T18:39:25.243214+0000","last_became_peered":"2024-09-10T18:39:25.243214+0000","last_unstale":"2024-09-10T18:43:42.213448+0000","last_undegraded":"2024-09-10T18:43:42.213448+0000","last_fullsized":"2024-09-10T18:43:42.213448+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:32.791113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:43:32.791113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:43.534325+0000","scrub_duration":0.0015760069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"400'18","reported_seq":478,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.215563+0000","last_change":"2024-09-10T18:41:48.983998+0000","last_active":"2024-09-10T18:43:42.215563+0000","last_peered":"2024-09-10T18:43:42.215563+0000","last_clean":"2024-09-10T18:43:42.215563+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:43:42.215563+0000","last_undegraded":"2024-09-10T18:43:42.215563+0000","last_fullsized":"2024-09-10T18:43:42.215563+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'18","last_scrub_stamp":"2024-09-10T18:41:48.983891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:48.983891+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035174529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":94,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.194421+0000","last_change":"2024-09-10T18:43:36.962698+0000","last_active":"2024-09-10T18:43:42.194421+0000","last_peered":"2024-09-10T18:43:42.194421+0000","last_clean":"2024-09-10T18:43:42.194421+0000","last_became_active":"2024-09-10T18:39:00.916952+0000","last_became_peered":"2024-09-10T18:39:00.916952+0000","last_unstale":"2024-09-10T18:43:42.194421+0000","last_undegraded":"2024-09-10T18:43:42.194421+0000","last_fullsized":"2024-09-10T18:43:42.194421+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:36.962644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:51.751267+0000","last_clean_scrub_stamp":"2024-09-10T18:43:36.962644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:53.900612+0000","scrub_duration":0.0014840649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":461,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.229307+0000","last_change":"2024-09-10T18:42:41.135735+0000","last_active":"2024-09-10T18:43:42.229307+0000","last_peered":"2024-09-10T18:43:42.229307+0000","last_clean":"2024-09-10T18:43:42.229307+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:43:42.229307+0000","last_undegraded":"2024-09-10T18:43:42.229307+0000","last_fullsized":"2024-09-10T18:43:42.229307+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:41.135603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:42:41.135603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:08.696743+0000","scrub_duration":0.002967669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'81","reported_seq":397,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.193775+0000","last_change":"2024-09-10T18:41:33.337668+0000","last_active":"2024-09-10T18:43:42.193775+0000","last_peered":"2024-09-10T18:43:42.193775+0000","last_clean":"2024-09-10T18:43:42.193775+0000","last_became_active":"2024-09-10T18:34:02.815880+0000","last_became_peered":"2024-09-10T18:34:02.815880+0000","last_unstale":"2024-09-10T18:43:42.193775+0000","last_undegraded":"2024-09-10T18:43:42.193775+0000","last_fullsized":"2024-09-10T18:43:42.193775+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:41:33.337569+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:41:33.337569+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052344720000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"400'25","reported_seq":360,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.215040+0000","last_change":"2024-09-10T18:41:43.958089+0000","last_active":"2024-09-10T18:43:42.215040+0000","last_peered":"2024-09-10T18:43:42.215040+0000","last_clean":"2024-09-10T18:43:42.215040+0000","last_became_active":"2024-09-10T18:39:25.236628+0000","last_became_peered":"2024-09-10T18:39:25.236628+0000","last_unstale":"2024-09-10T18:43:42.215040+0000","last_undegraded":"2024-09-10T18:43:42.215040+0000","last_fullsized":"2024-09-10T18:43:42.215040+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'25","last_scrub_stamp":"2024-09-10T18:41:43.957988+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-09-10T18:35:19.802016+0000","last_clean_scrub_stamp":"2024-09-10T18:41:43.957988+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021283768000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":467,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214365+0000","last_change":"2024-09-10T18:41:47.405474+0000","last_active":"2024-09-10T18:43:42.214365+0000","last_peered":"2024-09-10T18:43:42.214365+0000","last_clean":"2024-09-10T18:43:42.214365+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:43:42.214365+0000","last_undegraded":"2024-09-10T18:43:42.214365+0000","last_fullsized":"2024-09-10T18:43:42.214365+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:47.405374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:41:47.405374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015929049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":449,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.215384+0000","last_change":"2024-09-10T18:40:46.624305+0000","last_active":"2024-09-10T18:43:42.215384+0000","last_peered":"2024-09-10T18:43:42.215384+0000","last_clean":"2024-09-10T18:43:42.215384+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:43:42.215384+0000","last_undegraded":"2024-09-10T18:43:42.215384+0000","last_fullsized":"2024-09-10T18:43:42.215384+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:46.624225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:40:46.624225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068887569999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":503,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:46.250423+0000","last_change":"2024-09-10T18:43:46.250396+0000","last_active":"2024-09-10T18:43:46.250423+0000","last_peered":"2024-09-10T18:43:46.250423+0000","last_clean":"2024-09-10T18:43:46.250423+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:43:46.250423+0000","last_undegraded":"2024-09-10T18:43:46.250423+0000","last_fullsized":"2024-09-10T18:43:46.250423+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:46.250339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:43:46.250339+0000","last_clean_scrub_stamp":"2024-09-10T18:43:46.250339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:04.520351+0000","scrub_duration":0.0014968539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'81","reported_seq":94,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.193724+0000","last_change":"2024-09-10T18:42:39.258803+0000","last_active":"2024-09-10T18:43:42.193724+0000","last_peered":"2024-09-10T18:43:42.193724+0000","last_clean":"2024-09-10T18:43:42.193724+0000","last_became_active":"2024-09-10T18:39:00.902585+0000","last_became_peered":"2024-09-10T18:39:00.902585+0000","last_unstale":"2024-09-10T18:43:42.193724+0000","last_undegraded":"2024-09-10T18:43:42.193724+0000","last_fullsized":"2024-09-10T18:43:42.193724+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:39.258696+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:39:55.999377+0000","last_clean_scrub_stamp":"2024-09-10T18:42:39.258696+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:54.780607+0000","scrub_duration":0.0044049809999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"400'25","reported_seq":113,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:46.864733+0000","last_change":"2024-09-10T18:43:46.864703+0000","last_active":"2024-09-10T18:43:46.864733+0000","last_peered":"2024-09-10T18:43:46.864733+0000","last_clean":"2024-09-10T18:43:46.864733+0000","last_became_active":"2024-09-10T18:39:00.899968+0000","last_became_peered":"2024-09-10T18:39:00.899968+0000","last_unstale":"2024-09-10T18:43:46.864733+0000","last_undegraded":"2024-09-10T18:43:46.864733+0000","last_fullsized":"2024-09-10T18:43:46.864733+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'25","last_scrub_stamp":"2024-09-10T18:43:46.864638+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:43:46.864638+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:01.664633+0000","scrub_duration":0.0017769369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":93,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.217982+0000","last_change":"2024-09-10T18:43:32.061619+0000","last_active":"2024-09-10T18:43:42.217982+0000","last_peered":"2024-09-10T18:43:42.217982+0000","last_clean":"2024-09-10T18:43:42.217982+0000","last_became_active":"2024-09-10T18:39:25.245366+0000","last_became_peered":"2024-09-10T18:39:25.245366+0000","last_unstale":"2024-09-10T18:43:42.217982+0000","last_undegraded":"2024-09-10T18:43:42.217982+0000","last_fullsized":"2024-09-10T18:43:42.217982+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:32.061552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_clean_scrub_stamp":"2024-09-10T18:43:32.061552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:52.702654+0000","scrub_duration":0.001495524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"342'9","reported_seq":736,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:45.722635+0000","last_change":"2024-09-10T18:43:45.722564+0000","last_active":"2024-09-10T18:43:45.722635+0000","last_peered":"2024-09-10T18:43:45.722635+0000","last_clean":"2024-09-10T18:43:45.722635+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:43:45.722635+0000","last_undegraded":"2024-09-10T18:43:45.722635+0000","last_fullsized":"2024-09-10T18:43:45.722635+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'9","last_scrub_stamp":"2024-09-10T18:43:45.722501+0000","last_deep_scrub":"342'9","last_deep_scrub_stamp":"2024-09-10T18:37:56.641708+0000","last_clean_scrub_stamp":"2024-09-10T18:43:45.722501+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:46.159721+0000","scrub_duration":0.0016632299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":95,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.215684+0000","last_change":"2024-09-10T18:42:50.585832+0000","last_active":"2024-09-10T18:43:42.215684+0000","last_peered":"2024-09-10T18:43:42.215684+0000","last_clean":"2024-09-10T18:43:42.215684+0000","last_became_active":"2024-09-10T18:39:00.903167+0000","last_became_peered":"2024-09-10T18:39:00.903167+0000","last_unstale":"2024-09-10T18:43:42.215684+0000","last_undegraded":"2024-09-10T18:43:42.215684+0000","last_fullsized":"2024-09-10T18:43:42.215684+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:50.585744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:42:50.585744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:14.253397+0000","scrub_duration":0.0061391529999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"76'81","reported_seq":280,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.193747+0000","last_change":"2024-09-10T18:42:57.168069+0000","last_active":"2024-09-10T18:43:42.193747+0000","last_peered":"2024-09-10T18:43:42.193747+0000","last_clean":"2024-09-10T18:43:42.193747+0000","last_became_active":"2024-09-10T18:39:25.249841+0000","last_became_peered":"2024-09-10T18:39:25.249841+0000","last_unstale":"2024-09-10T18:43:42.193747+0000","last_undegraded":"2024-09-10T18:43:42.193747+0000","last_fullsized":"2024-09-10T18:43:42.193747+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:57.167991+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:36:08.710566+0000","last_clean_scrub_stamp":"2024-09-10T18:42:57.167991+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:12.764402+0000","scrub_duration":0.0043004580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"400'30","reported_seq":506,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214406+0000","last_change":"2024-09-10T18:42:27.160031+0000","last_active":"2024-09-10T18:43:42.214406+0000","last_peered":"2024-09-10T18:43:42.214406+0000","last_clean":"2024-09-10T18:43:42.214406+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:43:42.214406+0000","last_undegraded":"2024-09-10T18:43:42.214406+0000","last_fullsized":"2024-09-10T18:43:42.214406+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'30","last_scrub_stamp":"2024-09-10T18:42:27.159945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:42:27.159945+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094521229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":470,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:45.842542+0000","last_change":"2024-09-10T18:43:45.842510+0000","last_active":"2024-09-10T18:43:45.842542+0000","last_peered":"2024-09-10T18:43:45.842542+0000","last_clean":"2024-09-10T18:43:45.842542+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:43:45.842542+0000","last_undegraded":"2024-09-10T18:43:45.842542+0000","last_fullsized":"2024-09-10T18:43:45.842542+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:45.842447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:17.826448+0000","last_clean_scrub_stamp":"2024-09-10T18:43:45.842447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:46.225957+0000","scrub_duration":0.0015832750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"362'11","reported_seq":172,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:48.853595+0000","last_change":"2024-09-10T18:43:48.853508+0000","last_active":"2024-09-10T18:43:48.853595+0000","last_peered":"2024-09-10T18:43:48.853595+0000","last_clean":"2024-09-10T18:43:48.853595+0000","last_became_active":"2024-09-10T18:39:25.256047+0000","last_became_peered":"2024-09-10T18:39:25.256047+0000","last_unstale":"2024-09-10T18:43:48.853595+0000","last_undegraded":"2024-09-10T18:43:48.853595+0000","last_fullsized":"2024-09-10T18:43:48.853595+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'11","last_scrub_stamp":"2024-09-10T18:43:48.853369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:43:48.853369+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:52.555525+0000","scrub_duration":0.0023989969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":93,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:49.720299+0000","last_change":"2024-09-10T18:43:49.720270+0000","last_active":"2024-09-10T18:43:49.720299+0000","last_peered":"2024-09-10T18:43:49.720299+0000","last_clean":"2024-09-10T18:43:49.720299+0000","last_became_active":"2024-09-10T18:39:25.247171+0000","last_became_peered":"2024-09-10T18:39:25.247171+0000","last_unstale":"2024-09-10T18:43:49.720299+0000","last_undegraded":"2024-09-10T18:43:49.720299+0000","last_fullsized":"2024-09-10T18:43:49.720299+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:49.720216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:43:49.720216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:53.944377+0000","scrub_duration":0.001252282,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.16","version":"76'81","reported_seq":192,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.191885+0000","last_change":"2024-09-10T18:43:31.212203+0000","last_active":"2024-09-10T18:43:42.191885+0000","last_peered":"2024-09-10T18:43:42.191885+0000","last_clean":"2024-09-10T18:43:42.191885+0000","last_became_active":"2024-09-10T18:39:25.253130+0000","last_became_peered":"2024-09-10T18:39:25.253130+0000","last_unstale":"2024-09-10T18:43:42.191885+0000","last_undegraded":"2024-09-10T18:43:42.191885+0000","last_fullsized":"2024-09-10T18:43:42.191885+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:43:31.212145+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:43:31.212145+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:58.369565+0000","scrub_duration":0.0017569440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"400'27","reported_seq":540,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.999130+0000","last_change":"2024-09-10T18:42:36.288994+0000","last_active":"2024-09-10T18:43:41.999130+0000","last_peered":"2024-09-10T18:43:41.999130+0000","last_clean":"2024-09-10T18:43:41.999130+0000","last_became_active":"2024-09-10T18:41:11.933013+0000","last_became_peered":"2024-09-10T18:41:11.933013+0000","last_unstale":"2024-09-10T18:43:41.999130+0000","last_undegraded":"2024-09-10T18:43:41.999130+0000","last_fullsized":"2024-09-10T18:43:41.999130+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'27","last_scrub_stamp":"2024-09-10T18:42:36.288831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:42:36.288831+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:43:57.048695+0000","scrub_duration":0.0051082469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":470,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.999561+0000","last_change":"2024-09-10T18:42:25.151426+0000","last_active":"2024-09-10T18:43:41.999561+0000","last_peered":"2024-09-10T18:43:41.999561+0000","last_clean":"2024-09-10T18:43:41.999561+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:43:41.999561+0000","last_undegraded":"2024-09-10T18:43:41.999561+0000","last_fullsized":"2024-09-10T18:43:41.999561+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:25.151319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:42:25.151319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047341060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":500,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.191303+0000","last_change":"2024-09-10T18:43:37.205631+0000","last_active":"2024-09-10T18:43:42.191303+0000","last_peered":"2024-09-10T18:43:42.191303+0000","last_clean":"2024-09-10T18:43:42.191303+0000","last_became_active":"2024-09-10T18:39:00.897018+0000","last_became_peered":"2024-09-10T18:39:00.897018+0000","last_unstale":"2024-09-10T18:43:42.191303+0000","last_undegraded":"2024-09-10T18:43:42.191303+0000","last_fullsized":"2024-09-10T18:43:42.191303+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:37.205575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_clean_scrub_stamp":"2024-09-10T18:43:37.205575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:04.721137+0000","scrub_duration":0.0015814519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":477,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.680330+0000","last_change":"2024-09-10T18:43:42.680290+0000","last_active":"2024-09-10T18:43:42.680330+0000","last_peered":"2024-09-10T18:43:42.680330+0000","last_clean":"2024-09-10T18:43:42.680330+0000","last_became_active":"2024-09-10T18:39:25.237500+0000","last_became_peered":"2024-09-10T18:39:25.237500+0000","last_unstale":"2024-09-10T18:43:42.680330+0000","last_undegraded":"2024-09-10T18:43:42.680330+0000","last_fullsized":"2024-09-10T18:43:42.680330+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:42.680232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:43:42.680232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:50.186263+0000","scrub_duration":0.0013361969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'81","reported_seq":104,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.194684+0000","last_change":"2024-09-10T18:43:41.922314+0000","last_active":"2024-09-10T18:43:42.194684+0000","last_peered":"2024-09-10T18:43:42.194684+0000","last_clean":"2024-09-10T18:43:42.194684+0000","last_became_active":"2024-09-10T18:39:25.244612+0000","last_became_peered":"2024-09-10T18:39:25.244612+0000","last_unstale":"2024-09-10T18:43:42.194684+0000","last_undegraded":"2024-09-10T18:43:42.194684+0000","last_fullsized":"2024-09-10T18:43:42.194684+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:43:41.922252+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:40:52.817945+0000","last_clean_scrub_stamp":"2024-09-10T18:43:41.922252+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:06.612587+0000","scrub_duration":0.0012528999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"400'21","reported_seq":520,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:49.265578+0000","last_change":"2024-09-10T18:43:49.265554+0000","last_active":"2024-09-10T18:43:49.265578+0000","last_peered":"2024-09-10T18:43:49.265578+0000","last_clean":"2024-09-10T18:43:49.265578+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:43:49.265578+0000","last_undegraded":"2024-09-10T18:43:49.265578+0000","last_fullsized":"2024-09-10T18:43:49.265578+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'21","last_scrub_stamp":"2024-09-10T18:43:49.265498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:43:49.265498+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:18.263278+0000","scrub_duration":0.001644871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":463,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.229601+0000","last_change":"2024-09-10T18:41:29.335366+0000","last_active":"2024-09-10T18:43:42.229601+0000","last_peered":"2024-09-10T18:43:42.229601+0000","last_clean":"2024-09-10T18:43:42.229601+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:43:42.229601+0000","last_undegraded":"2024-09-10T18:43:42.229601+0000","last_fullsized":"2024-09-10T18:43:42.229601+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:29.335275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:12.858393+0000","last_clean_scrub_stamp":"2024-09-10T18:41:29.335275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032473530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":452,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.216110+0000","last_change":"2024-09-10T18:40:47.615302+0000","last_active":"2024-09-10T18:43:42.216110+0000","last_peered":"2024-09-10T18:43:42.216110+0000","last_clean":"2024-09-10T18:43:42.216110+0000","last_became_active":"2024-09-10T18:39:25.251469+0000","last_became_peered":"2024-09-10T18:39:25.251469+0000","last_unstale":"2024-09-10T18:43:42.216110+0000","last_undegraded":"2024-09-10T18:43:42.216110+0000","last_fullsized":"2024-09-10T18:43:42.216110+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:40:47.615205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:40:47.615205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030335510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":484,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.193825+0000","last_change":"2024-09-10T18:42:33.235205+0000","last_active":"2024-09-10T18:43:42.193825+0000","last_peered":"2024-09-10T18:43:42.193825+0000","last_clean":"2024-09-10T18:43:42.193825+0000","last_became_active":"2024-09-10T18:39:25.264554+0000","last_became_peered":"2024-09-10T18:39:25.264554+0000","last_unstale":"2024-09-10T18:43:42.193825+0000","last_undegraded":"2024-09-10T18:43:42.193825+0000","last_fullsized":"2024-09-10T18:43:42.193825+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:33.235134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:42:33.235134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002556516,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"76'81","reported_seq":209,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:45.250434+0000","last_change":"2024-09-10T18:43:45.250408+0000","last_active":"2024-09-10T18:43:45.250434+0000","last_peered":"2024-09-10T18:43:45.250434+0000","last_clean":"2024-09-10T18:43:45.250434+0000","last_became_active":"2024-09-10T18:34:02.802580+0000","last_became_peered":"2024-09-10T18:34:02.802580+0000","last_unstale":"2024-09-10T18:43:45.250434+0000","last_undegraded":"2024-09-10T18:43:45.250434+0000","last_fullsized":"2024-09-10T18:43:45.250434+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:43:45.250345+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:43:45.250345+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:04.669981+0000","scrub_duration":0.0032821679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"400'27","reported_seq":511,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:48.688706+0000","last_change":"2024-09-10T18:43:48.688671+0000","last_active":"2024-09-10T18:43:48.688706+0000","last_peered":"2024-09-10T18:43:48.688706+0000","last_clean":"2024-09-10T18:43:48.688706+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:43:48.688706+0000","last_undegraded":"2024-09-10T18:43:48.688706+0000","last_fullsized":"2024-09-10T18:43:48.688706+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'27","last_scrub_stamp":"2024-09-10T18:43:48.688610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:43:48.688610+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:10.913753+0000","scrub_duration":0.0021329729999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":94,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.194366+0000","last_change":"2024-09-10T18:41:38.954345+0000","last_active":"2024-09-10T18:43:42.194366+0000","last_peered":"2024-09-10T18:43:42.194366+0000","last_clean":"2024-09-10T18:43:42.194366+0000","last_became_active":"2024-09-10T18:39:00.916983+0000","last_became_peered":"2024-09-10T18:39:00.916983+0000","last_unstale":"2024-09-10T18:43:42.194366+0000","last_undegraded":"2024-09-10T18:43:42.194366+0000","last_fullsized":"2024-09-10T18:43:42.194366+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:38.954273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:41:38.954273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019398480000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":466,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T18:43:48.756802+0000","last_change":"2024-09-10T18:43:48.756764+0000","last_active":"2024-09-10T18:43:48.756802+0000","last_peered":"2024-09-10T18:43:48.756802+0000","last_clean":"2024-09-10T18:43:48.756802+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:43:48.756802+0000","last_undegraded":"2024-09-10T18:43:48.756802+0000","last_fullsized":"2024-09-10T18:43:48.756802+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:48.756697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:43:48.756697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:06.612938+0000","scrub_duration":0.0015511609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":483,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.216291+0000","last_change":"2024-09-10T18:42:48.554386+0000","last_active":"2024-09-10T18:43:42.216291+0000","last_peered":"2024-09-10T18:43:42.216291+0000","last_clean":"2024-09-10T18:43:42.216291+0000","last_became_active":"2024-09-10T18:39:25.242900+0000","last_became_peered":"2024-09-10T18:39:25.242900+0000","last_unstale":"2024-09-10T18:43:42.216291+0000","last_undegraded":"2024-09-10T18:43:42.216291+0000","last_fullsized":"2024-09-10T18:43:42.216291+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:48.554301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:48.554301+0000","last_clean_scrub_stamp":"2024-09-10T18:42:48.554301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:01.886073+0000","scrub_duration":0.008030354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'81","reported_seq":191,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.214344+0000","last_change":"2024-09-10T18:42:23.159856+0000","last_active":"2024-09-10T18:43:42.214344+0000","last_peered":"2024-09-10T18:43:42.214344+0000","last_clean":"2024-09-10T18:43:42.214344+0000","last_became_active":"2024-09-10T18:39:00.889998+0000","last_became_peered":"2024-09-10T18:39:00.889998+0000","last_unstale":"2024-09-10T18:43:42.214344+0000","last_undegraded":"2024-09-10T18:43:42.214344+0000","last_fullsized":"2024-09-10T18:43:42.214344+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:23.159712+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:23.159712+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001874744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"400'12","reported_seq":476,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.213583+0000","last_change":"2024-09-10T18:41:40.949560+0000","last_active":"2024-09-10T18:43:42.213583+0000","last_peered":"2024-09-10T18:43:42.213583+0000","last_clean":"2024-09-10T18:43:42.213583+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:43:42.213583+0000","last_undegraded":"2024-09-10T18:43:42.213583+0000","last_fullsized":"2024-09-10T18:43:42.213583+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'12","last_scrub_stamp":"2024-09-10T18:41:40.949115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:41:40.949115+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022213717000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":486,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.194622+0000","last_change":"2024-09-10T18:41:40.970661+0000","last_active":"2024-09-10T18:43:42.194622+0000","last_peered":"2024-09-10T18:43:42.194622+0000","last_clean":"2024-09-10T18:43:42.194622+0000","last_became_active":"2024-09-10T18:39:25.242295+0000","last_became_peered":"2024-09-10T18:39:25.242295+0000","last_unstale":"2024-09-10T18:43:42.194622+0000","last_undegraded":"2024-09-10T18:43:42.194622+0000","last_fullsized":"2024-09-10T18:43:42.194622+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:40.970571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:41:40.970571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0072197229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":430,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.226597+0000","last_change":"2024-09-10T18:42:34.597769+0000","last_active":"2024-09-10T18:43:42.226597+0000","last_peered":"2024-09-10T18:43:42.226597+0000","last_clean":"2024-09-10T18:43:42.226597+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:43:42.226597+0000","last_undegraded":"2024-09-10T18:43:42.226597+0000","last_fullsized":"2024-09-10T18:43:42.226597+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:34.597693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:42:34.597693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:03.774399+0000","scrub_duration":0.0025145829999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":454,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.217754+0000","last_change":"2024-09-10T18:41:36.315811+0000","last_active":"2024-09-10T18:43:42.217754+0000","last_peered":"2024-09-10T18:43:42.217754+0000","last_clean":"2024-09-10T18:43:42.217754+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:43:42.217754+0000","last_undegraded":"2024-09-10T18:43:42.217754+0000","last_fullsized":"2024-09-10T18:43:42.217754+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:41:36.315732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:41:36.315732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015270002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'81","reported_seq":181,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:41.998949+0000","last_change":"2024-09-10T18:42:32.301668+0000","last_active":"2024-09-10T18:43:41.998949+0000","last_peered":"2024-09-10T18:43:41.998949+0000","last_clean":"2024-09-10T18:43:41.998949+0000","last_became_active":"2024-09-10T18:34:34.867380+0000","last_became_peered":"2024-09-10T18:34:34.867380+0000","last_unstale":"2024-09-10T18:43:41.998949+0000","last_undegraded":"2024-09-10T18:43:41.998949+0000","last_fullsized":"2024-09-10T18:43:41.998949+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:42:32.301577+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:42:32.301577+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077452609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":495,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.191913+0000","last_change":"2024-09-10T18:43:42.191246+0000","last_active":"2024-09-10T18:43:42.191913+0000","last_peered":"2024-09-10T18:43:42.191913+0000","last_clean":"2024-09-10T18:43:42.191913+0000","last_became_active":"2024-09-10T18:39:00.909906+0000","last_became_peered":"2024-09-10T18:39:00.909906+0000","last_unstale":"2024-09-10T18:43:42.191913+0000","last_undegraded":"2024-09-10T18:43:42.191913+0000","last_fullsized":"2024-09-10T18:43:42.191913+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:42.191141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:18.177264+0000","last_clean_scrub_stamp":"2024-09-10T18:43:42.191141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:07.667289+0000","scrub_duration":0.001697453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":463,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.213798+0000","last_change":"2024-09-10T18:42:46.565973+0000","last_active":"2024-09-10T18:43:42.213798+0000","last_peered":"2024-09-10T18:43:42.213798+0000","last_clean":"2024-09-10T18:43:42.213798+0000","last_became_active":"2024-09-10T18:39:25.250206+0000","last_became_peered":"2024-09-10T18:39:25.250206+0000","last_unstale":"2024-09-10T18:43:42.213798+0000","last_undegraded":"2024-09-10T18:43:42.213798+0000","last_fullsized":"2024-09-10T18:43:42.213798+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:42:46.565877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:46.565877+0000","last_clean_scrub_stamp":"2024-09-10T18:42:46.565877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:12.551542+0000","scrub_duration":0.0051506470000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"400'32","reported_seq":562,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.220689+0000","last_change":"2024-09-10T18:42:38.627801+0000","last_active":"2024-09-10T18:43:42.220689+0000","last_peered":"2024-09-10T18:43:42.220689+0000","last_clean":"2024-09-10T18:43:42.220689+0000","last_became_active":"2024-09-10T18:39:00.921476+0000","last_became_peered":"2024-09-10T18:39:00.921476+0000","last_unstale":"2024-09-10T18:43:42.220689+0000","last_undegraded":"2024-09-10T18:43:42.220689+0000","last_fullsized":"2024-09-10T18:43:42.220689+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'32","last_scrub_stamp":"2024-09-10T18:42:38.627699+0000","last_deep_scrub":"369'24","last_deep_scrub_stamp":"2024-09-10T18:39:50.690827+0000","last_clean_scrub_stamp":"2024-09-10T18:42:38.627699+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:44:05.830328+0000","scrub_duration":0.010970025,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":93,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T18:43:42.213925+0000","last_change":"2024-09-10T18:43:37.861381+0000","last_active":"2024-09-10T18:43:42.213925+0000","last_peered":"2024-09-10T18:43:42.213925+0000","last_clean":"2024-09-10T18:43:42.213925+0000","last_became_active":"2024-09-10T18:39:25.243964+0000","last_became_peered":"2024-09-10T18:39:25.243964+0000","last_unstale":"2024-09-10T18:43:42.213925+0000","last_undegraded":"2024-09-10T18:43:42.213925+0000","last_fullsized":"2024-09-10T18:43:42.213925+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:43:37.861329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:43:37.861329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:02.513028+0000","scrub_duration":0.0013473140000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1548,"num_read_kb":1305,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":4566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466944,"data_stored":12426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":1064,"ondisk_log_size":1064,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1329,"num_read_kb":1324,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2511,"ondisk_log_size":2511,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234255,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":891440,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":890430,"kb_avail":92853712,"statfs":{"total":95995035648,"available":95082201088,"internally_reserved":0,"allocated":1024000,"data_stored":637726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":911800664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:43:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":1.2889999999999999,"15min":1.6319999999999999},"min":{"1min":0.254,"5min":0.219,"15min":0.21099999999999999},"max":{"1min":1.71,"5min":29.443999999999999,"15min":75.506},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.4530000000000001,"15min":1.605},"min":{"1min":0.26900000000000002,"5min":0.25700000000000001,"15min":0.223},"max":{"1min":3.48,"5min":29.530999999999999,"15min":75.710999999999999},"last":0.55900000000000005}]},{"osd":1,"last update":"Tue Sep 10 18:43:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":2.173,"15min":1.829},"min":{"1min":0.26500000000000001,"5min":0.22800000000000001,"15min":0.214},"max":{"1min":4.742,"5min":98.305999999999997,"15min":98.305999999999997},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.486,"5min":2.2959999999999998,"15min":1.78},"min":{"1min":0.28199999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.7229999999999999,"5min":98.381,"15min":98.381},"last":0.52600000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:43:02 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.9710000000000001,"15min":1.825},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26400000000000001},"max":{"1min":3.0699999999999998,"5min":44.728000000000002,"15min":136.648},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.956,"15min":1.861},"min":{"1min":0.34499999999999997,"5min":0.27800000000000002,"15min":0.24399999999999999},"max":{"1min":3.6739999999999999,"5min":44.664999999999999,"15min":136.506},"last":0.59899999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:43:02 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.1830000000000001,"15min":1.2969999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":3.3239999999999998,"5min":8.7400000000000002,"15min":38.225000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.1299999999999999,"15min":1.2889999999999999},"min":{"1min":0.375,"5min":0.29899999999999999,"15min":0.29199999999999998},"max":{"1min":4.7990000000000004,"5min":12.516,"15min":38.104999999999997},"last":0.62}]},{"osd":4,"last update":"Tue Sep 10 18:43:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0580000000000001,"15min":0.94799999999999995},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.224},"max":{"1min":3.5609999999999999,"5min":8.4770000000000003,"15min":23.346},"last":0.377},{"interface":"front","average":{"1min":1.101,"5min":0.97499999999999998,"15min":0.96599999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":3.0179999999999998,"5min":5.4219999999999997,"15min":23.323},"last":0.39900000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:43:02 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.92900000000000005,"15min":1.016},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.25800000000000001},"max":{"1min":2.6309999999999998,"5min":7.7119999999999997,"15min":16.756},"last":0.498},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.113,"15min":1.0389999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.224},"max":{"1min":3.5139999999999998,"5min":8.4510000000000005,"15min":20.495999999999999},"last":0.42399999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:42:52 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.016,"15min":1.008},"min":{"1min":0.314,"5min":0.314,"15min":0.28899999999999998},"max":{"1min":3.0939999999999999,"5min":5.976,"15min":25.902999999999999},"last":0.436},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.1679999999999999,"15min":1},"min":{"1min":0.33600000000000002,"5min":0.32400000000000001,"15min":0.30199999999999999},"max":{"1min":6.3029999999999999,"5min":9.9879999999999995,"15min":24.074000000000002},"last":0.45700000000000002}]}]},{"osd":6,"up_from":144,"seq":618475290834,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903952,"kb_used_data":1032,"kb_used_omap":7,"kb_used_meta":902904,"kb_avail":92841200,"statfs":{"total":95995035648,"available":95069388800,"internally_reserved":0,"allocated":1056768,"data_stored":702560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":924573783},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:43:03 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.1140000000000001,"15min":1.208},"min":{"1min":0.248,"5min":0.23300000000000001,"15min":0.192},"max":{"1min":7.0590000000000002,"5min":7.0590000000000002,"15min":30.629999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.032,"15min":1.125},"min":{"1min":0.32700000000000001,"5min":0.246,"15min":0.217},"max":{"1min":2.5169999999999999,"5min":4.9489999999999998,"15min":30.597000000000001},"last":0.53800000000000003}]},{"osd":1,"last update":"Tue Sep 10 18:43:03 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.637,"15min":1.2789999999999999},"min":{"1min":0.314,"5min":0.28899999999999998,"15min":0.23100000000000001},"max":{"1min":11.125999999999999,"5min":23.782,"15min":30.984000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.4339999999999999,"15min":1.2649999999999999},"min":{"1min":0.26800000000000002,"5min":0.253,"15min":0.24199999999999999},"max":{"1min":3.4329999999999998,"5min":23.687999999999999,"15min":30.571000000000002},"last":0.52700000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:43:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.391,"15min":1.306},"min":{"1min":0.28499999999999998,"5min":0.27500000000000002,"15min":0.25600000000000001},"max":{"1min":2.7440000000000002,"5min":23.763999999999999,"15min":39.329999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.6859999999999999,"5min":1.821,"15min":1.369},"min":{"1min":0.44600000000000001,"5min":0.28599999999999998,"15min":0.23400000000000001},"max":{"1min":6.2409999999999997,"5min":23.856000000000002,"15min":39.061999999999998},"last":0.46000000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:43:03 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":4.9249999999999998,"15min":2.597},"min":{"1min":0.307,"5min":0.29499999999999998,"15min":0.27800000000000002},"max":{"1min":3.581,"5min":162.61600000000001,"15min":162.61600000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.7030000000000001,"5min":4.9909999999999997,"15min":2.6320000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.27100000000000002},"max":{"1min":10.942,"5min":163.39599999999999,"15min":163.39599999999999},"last":0.50600000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:43:03 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.82599999999999996,"15min":0.72599999999999998},"min":{"1min":0.251,"5min":0.21099999999999999,"15min":0.189},"max":{"1min":2.3719999999999999,"5min":4.9050000000000002,"15min":5.4950000000000001},"last":0.315},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.89400000000000002,"15min":0.72299999999999998},"min":{"1min":0.33700000000000002,"5min":0.22800000000000001,"15min":0.219},"max":{"1min":2.6890000000000001,"5min":11.531000000000001,"15min":11.531000000000001},"last":0.376}]},{"osd":5,"last update":"Tue Sep 10 18:43:03 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":0.85499999999999998,"15min":0.751},"min":{"1min":0.32700000000000001,"5min":0.254,"15min":0.254},"max":{"1min":2.9209999999999998,"5min":4.4139999999999997,"15min":4.6619999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":1.081,"5min":0.94399999999999995,"15min":0.80100000000000005},"min":{"1min":0.34100000000000003,"5min":0.28000000000000003,"15min":0.20899999999999999},"max":{"1min":3.48,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":0.36099999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:43:03 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.1439999999999999,"15min":0.84799999999999998},"min":{"1min":0.36799999999999999,"5min":0.26700000000000002,"15min":0.23799999999999999},"max":{"1min":4.8019999999999996,"5min":11.449,"15min":11.449},"last":0.40300000000000002},{"interface":"front","average":{"1min":1.204,"5min":0.90600000000000003,"15min":0.78700000000000003},"min":{"1min":0.33100000000000002,"5min":0.32700000000000001,"15min":0.27100000000000002},"max":{"1min":2.8380000000000001,"5min":5.0149999999999997,"15min":5.5309999999999997},"last":0.38900000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724826,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":904616,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":903614,"kb_avail":92840536,"statfs":{"total":95995035648,"available":95068708864,"internally_reserved":0,"allocated":1015808,"data_stored":637984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":925301080},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:43:22 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.89300000000000002,"15min":1.139},"min":{"1min":0.23400000000000001,"5min":0.23200000000000001,"15min":0.224},"max":{"1min":5.7690000000000001,"5min":5.7690000000000001,"15min":26.288},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.83599999999999997,"15min":1.014},"min":{"1min":0.27600000000000002,"5min":0.252,"15min":0.24099999999999999},"max":{"1min":3.4350000000000001,"5min":4.851,"15min":24.178000000000001},"last":0.58999999999999997}]},{"osd":1,"last update":"Tue Sep 10 18:43:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.3859999999999999,"15min":1.7490000000000001},"min":{"1min":0.35299999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.044,"5min":22.623999999999999,"15min":111.773},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.2250000000000001,"15min":1.6899999999999999},"min":{"1min":0.27600000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":2.6200000000000001,"5min":22.635000000000002,"15min":111.73999999999999},"last":0.64400000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:43:22 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.5740000000000001,"15min":1.4059999999999999},"min":{"1min":0.29599999999999999,"5min":0.247,"15min":0.245},"max":{"1min":5.7910000000000004,"5min":23.396000000000001,"15min":54.530000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.385,"15min":1.3260000000000001},"min":{"1min":0.32000000000000001,"5min":0.27400000000000002,"15min":0.247},"max":{"1min":4.5030000000000001,"5min":22.905999999999999,"15min":54.514000000000003},"last":0.72999999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:43:22 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.6319999999999999,"15min":1.8859999999999999},"min":{"1min":0.30099999999999999,"5min":0.29699999999999999,"15min":0.27100000000000002},"max":{"1min":5.8700000000000001,"5min":23.007999999999999,"15min":106.205},"last":0.63200000000000001},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.5680000000000001,"15min":1.9039999999999999},"min":{"1min":0.38,"5min":0.28999999999999998,"15min":0.28299999999999997},"max":{"1min":6.3789999999999996,"5min":22.931000000000001,"15min":105.878},"last":0.65800000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.83699999999999997,"15min":0.77300000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":2.5830000000000002,"5min":5.226,"15min":5.8040000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.038,"15min":0.89400000000000002},"min":{"1min":0.32200000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":5.702,"5min":5.7309999999999999,"15min":8.7270000000000003},"last":0.48699999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:42:50 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":0.99099999999999999,"15min":0.81299999999999994},"min":{"1min":0.31900000000000001,"5min":0.29999999999999999,"15min":0.27500000000000002},"max":{"1min":5.7359999999999998,"5min":7.9969999999999999,"15min":7.9969999999999999},"last":0.443},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.2270000000000001,"15min":0.95499999999999996},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.27000000000000002},"max":{"1min":3.9470000000000001,"5min":8.7400000000000002,"15min":8.7400000000000002},"last":0.57399999999999995}]},{"osd":7,"last update":"Tue Sep 10 18:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":1.054,"15min":0.97899999999999998},"min":{"1min":0.39400000000000002,"5min":0.32200000000000001,"15min":0.25900000000000001},"max":{"1min":3.1680000000000001,"5min":8.3870000000000005,"15min":28.934000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.98899999999999999,"15min":0.88400000000000001},"min":{"1min":0.35299999999999998,"5min":0.314,"15min":0.25900000000000001},"max":{"1min":2.7559999999999998,"5min":5.593,"15min":5.8259999999999996},"last":0.54400000000000004}]}]},{"osd":4,"up_from":33,"seq":141733921054,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900384,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":899326,"kb_avail":92844768,"statfs":{"total":95995035648,"available":95073042432,"internally_reserved":0,"allocated":1073152,"data_stored":702199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920910168},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":1.3320000000000001,"15min":1.2250000000000001},"min":{"1min":0.26100000000000001,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":2.028,"5min":21.468,"15min":21.468},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.1719999999999999,"15min":1.038},"min":{"1min":0.27300000000000002,"5min":0.21299999999999999,"15min":0.189},"max":{"1min":1.742,"5min":24.277999999999999,"15min":24.277999999999999},"last":0.59399999999999997}]},{"osd":1,"last update":"Tue Sep 10 18:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.254,"15min":1.236},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.0390000000000001,"5min":12.696,"15min":50.045000000000002},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.024,"5min":1.2829999999999999,"15min":1.3080000000000001},"min":{"1min":0.28199999999999997,"5min":0.28000000000000003,"15min":0.27100000000000002},"max":{"1min":2.1389999999999998,"5min":13.632,"15min":50.283000000000001},"last":0.52700000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:42:56 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":4.4809999999999999,"15min":2.056},"min":{"1min":0.28100000000000003,"5min":0.251,"15min":0.249},"max":{"1min":2.048,"5min":169.417,"15min":169.417},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.3650000000000002,"15min":2.0550000000000002},"min":{"1min":0.27000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.2610000000000001,"5min":169.40100000000001,"15min":169.40100000000001},"last":0.68000000000000005}]},{"osd":3,"last update":"Tue Sep 10 18:42:56 2024","interfaces":[{"interface":"back","average":{"1min":7.1689999999999996,"5min":2.2829999999999999,"15min":1.7390000000000001},"min":{"1min":0.28499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":116.913,"5min":116.913,"15min":116.913},"last":0.63100000000000001},{"interface":"front","average":{"1min":7.3369999999999997,"5min":2.6859999999999999,"15min":1.952},"min":{"1min":0.35199999999999998,"5min":0.32400000000000001,"15min":0.28000000000000003},"max":{"1min":116.863,"5min":116.863,"15min":116.863},"last":0.54200000000000004}]},{"osd":5,"last update":"Tue Sep 10 18:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.83699999999999997,"15min":0.871},"min":{"1min":0.252,"5min":0.252,"15min":0.223},"max":{"1min":1.974,"5min":3.6379999999999999,"15min":22.861000000000001},"last":0.442},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.77100000000000002,"15min":0.89800000000000002},"min":{"1min":0.309,"5min":0.254,"15min":0.22},"max":{"1min":0.81100000000000005,"5min":4.5819999999999999,"15min":19.611999999999998},"last":0.5}]},{"osd":6,"last update":"Tue Sep 10 18:42:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.173,"15min":1.052},"min":{"1min":0.436,"5min":0.27500000000000002,"15min":0.27400000000000002},"max":{"1min":6.7130000000000001,"5min":6.7130000000000001,"15min":51.722000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.159,"15min":1.0269999999999999},"min":{"1min":0.38,"5min":0.33000000000000002,"15min":0.30599999999999999},"max":{"1min":8.5749999999999993,"5min":8.5749999999999993,"15min":51.862000000000002},"last":0.41099999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:42:56 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.1100000000000001,"15min":0.85499999999999998},"min":{"1min":0.34899999999999998,"5min":0.312,"15min":0.28799999999999998},"max":{"1min":6.4470000000000001,"5min":7.6260000000000003,"15min":7.6260000000000003},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.153,"5min":1.0129999999999999,"15min":0.85499999999999998},"min":{"1min":0.38500000000000001,"5min":0.29299999999999998,"15min":0.28000000000000003},"max":{"1min":6.96,"5min":6.96,"15min":7.8250000000000002},"last":0.56000000000000005}]}]},{"osd":3,"up_from":27,"seq":115964117284,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888500,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":887422,"kb_avail":92856652,"statfs":{"total":95995035648,"available":95085211648,"internally_reserved":0,"allocated":1093632,"data_stored":703542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908720472},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.999,"15min":1.226},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.18099999999999999},"max":{"1min":3.6930000000000001,"5min":8.8079999999999998,"15min":29.981999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.95599999999999996,"15min":1.3380000000000001},"min":{"1min":0.23300000000000001,"5min":0.193,"15min":0.193},"max":{"1min":4.1479999999999997,"5min":5.0010000000000003,"15min":28.754000000000001},"last":0.29899999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:43:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.2869999999999999,"15min":1.5980000000000001},"min":{"1min":0.36699999999999999,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":9.5199999999999996,"5min":9.5199999999999996,"15min":35.406999999999996},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.268,"15min":1.5289999999999999},"min":{"1min":0.41399999999999998,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":8.9499999999999993,"5min":8.9499999999999993,"15min":36.902000000000001},"last":0.35199999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:43:26 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":2.125,"15min":1.423},"min":{"1min":0.23699999999999999,"5min":0.22800000000000001,"15min":0.191},"max":{"1min":3.6699999999999999,"5min":79.975999999999999,"15min":79.975999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.516,"5min":2.3090000000000002,"15min":1.7470000000000001},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":4,"5min":78.278000000000006,"15min":78.278000000000006},"last":0.37}]},{"osd":4,"last update":"Tue Sep 10 18:43:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.1719999999999999,"15min":1.1679999999999999},"min":{"1min":0.315,"5min":0.308,"15min":0.27600000000000002},"max":{"1min":4.8780000000000001,"5min":6.1699999999999999,"15min":6.5250000000000004},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.147,"15min":1.1459999999999999},"min":{"1min":0.376,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.1989999999999998,"5min":5.1989999999999998,"15min":6.1929999999999996},"last":0.46700000000000003}]},{"osd":5,"last update":"Tue Sep 10 18:43:14 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.101,"15min":1.2210000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":4.8479999999999999,"5min":6.2370000000000001,"15min":6.601},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.0649999999999999,"15min":1.238},"min":{"1min":0.371,"5min":0.254,"15min":0.254},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":6.2249999999999996},"last":0.57199999999999995}]},{"osd":6,"last update":"Tue Sep 10 18:43:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.165,"15min":1.3959999999999999},"min":{"1min":0.42499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.093,"5min":6.8129999999999997,"15min":31.815999999999999},"last":0.441},{"interface":"front","average":{"1min":1.256,"5min":1.21,"15min":1.417},"min":{"1min":0.44500000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":2.6960000000000002,"5min":8.7569999999999997,"15min":31.742000000000001},"last":0.51200000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:43:01 2024","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.2310000000000001,"15min":1.2430000000000001},"min":{"1min":0.34200000000000003,"5min":0.308,"15min":0.308},"max":{"1min":4.9000000000000004,"5min":6.2119999999999997,"15min":6.5730000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.355,"5min":1.1040000000000001,"15min":1.2549999999999999},"min":{"1min":0.40300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":5.1180000000000003,"5min":5.1180000000000003,"15min":12.612},"last":0.498}]}]},{"osd":2,"up_from":18,"seq":77309411624,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901416,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":900414,"kb_avail":92843736,"statfs":{"total":95995035648,"available":95071985664,"internally_reserved":0,"allocated":1015808,"data_stored":637229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922024280},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:43:05 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":1.0229999999999999,"15min":1.3120000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.17499999999999999},"max":{"1min":2.6739999999999999,"5min":3.8610000000000002,"15min":43.890000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.746,"5min":1.333,"15min":1.625},"min":{"1min":0.21099999999999999,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":3.1600000000000001,"5min":10.307,"15min":44.331000000000003},"last":0.35199999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:43:05 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.534,"15min":1.5669999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.193},"max":{"1min":4.6219999999999999,"5min":11.154,"15min":44.575000000000003},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.325,"15min":1.369},"min":{"1min":0.36799999999999999,"5min":0.248,"15min":0.19500000000000001},"max":{"1min":3.4279999999999999,"5min":10.637,"15min":44.368000000000002},"last":0.434}]},{"osd":3,"last update":"Tue Sep 10 18:43:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.7729999999999999,"15min":1.7729999999999999},"min":{"1min":0.28599999999999998,"5min":0.23999999999999999,"15min":0.18099999999999999},"max":{"1min":4.7160000000000002,"5min":45.725000000000001,"15min":61.331000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":1.6639999999999999,"5min":2.0649999999999999,"15min":1.9019999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.214},"max":{"1min":4.3520000000000003,"5min":45.677,"15min":61.250999999999998},"last":0.39000000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:43:01 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.425,"15min":1.2350000000000001},"min":{"1min":0.28499999999999998,"5min":0.27400000000000002,"15min":0.24199999999999999},"max":{"1min":2.891,"5min":7.6299999999999999,"15min":12.555999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.357,"15min":1.151},"min":{"1min":0.39600000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":2.7450000000000001,"5min":19.696000000000002,"15min":19.696000000000002},"last":0.47399999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:43:16 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.29,"15min":1.2769999999999999},"min":{"1min":0.499,"5min":0.249,"15min":0.249},"max":{"1min":3.1240000000000001,"5min":19.632999999999999,"15min":25.177},"last":0.57099999999999995},{"interface":"front","average":{"1min":1.351,"5min":1.1950000000000001,"15min":1.329},"min":{"1min":0.37,"5min":0.32000000000000001,"15min":0.317},"max":{"1min":3.3919999999999999,"5min":4.0640000000000001,"15min":24.986999999999998},"last":0.60799999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:43:01 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.6419999999999999,"15min":1.4930000000000001},"min":{"1min":0.33200000000000002,"5min":0.314,"15min":0.314},"max":{"1min":4.0880000000000001,"5min":7.6529999999999996,"15min":53.359999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.171,"5min":1.266,"15min":1.3959999999999999},"min":{"1min":0.41999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.7989999999999999,"5min":5.2599999999999998,"15min":53.438000000000002},"last":0.63600000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:43:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.3480000000000001,"15min":1.222},"min":{"1min":0.38700000000000001,"5min":0.36699999999999999,"15min":0.35399999999999998},"max":{"1min":3.0649999999999999,"5min":6.1980000000000004,"15min":10.68},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.167,"5min":1.5209999999999999,"15min":1.371},"min":{"1min":0.39100000000000001,"5min":0.39000000000000001,"15min":0.376},"max":{"1min":2.9180000000000001,"5min":6.2750000000000004,"15min":12.596},"last":0.58999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575149,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":897664,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":896894,"kb_avail":92847488,"statfs":{"total":95995035648,"available":95075827712,"internally_reserved":0,"allocated":778240,"data_stored":375461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918419800},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.85699999999999998,"15min":1.282},"min":{"1min":0.19500000000000001,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":2.512,"5min":6.6760000000000002,"15min":28.893999999999998},"last":0.219},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.97999999999999998,"15min":1.302},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.187},"max":{"1min":1.9019999999999999,"5min":7.8700000000000001,"15min":31.074999999999999},"last":0.20100000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:43:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7310000000000001,"5min":1.462,"15min":1.2370000000000001},"min":{"1min":0.221,"5min":0.216,"15min":0.19500000000000001},"max":{"1min":7.8380000000000001,"5min":11.079000000000001,"15min":11.079000000000001},"last":0.23400000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.446,"15min":1.2769999999999999},"min":{"1min":0.30499999999999999,"5min":0.216,"15min":0.20999999999999999},"max":{"1min":4.1440000000000001,"5min":9.0749999999999993,"15min":12.222},"last":0.27800000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:43:38 2024","interfaces":[{"interface":"back","average":{"1min":9.3420000000000005,"5min":2.8130000000000002,"15min":1.8320000000000001},"min":{"1min":0.26600000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":148.22900000000001,"5min":148.22900000000001,"15min":148.22900000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":9.0510000000000002,"5min":2.7799999999999998,"15min":1.8320000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.20499999999999999},"max":{"1min":148.65000000000001,"5min":148.65000000000001,"15min":148.65000000000001},"last":0.31900000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":2.4940000000000002,"15min":1.673},"min":{"1min":0.35299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.7050000000000001,"5min":127.86199999999999,"15min":127.86199999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.81399999999999995,"5min":2.6749999999999998,"15min":1.7430000000000001},"min":{"1min":0.40000000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.9490000000000001,"5min":127.84399999999999,"15min":127.84399999999999},"last":0.69999999999999996}]},{"osd":5,"last update":"Tue Sep 10 18:43:13 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":2.7400000000000002,"15min":1.8620000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.27900000000000003},"max":{"1min":4.0700000000000003,"5min":118.68000000000001,"15min":118.68000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.5780000000000001,"5min":2.7410000000000001,"15min":1.8740000000000001},"min":{"1min":0.46300000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.8929999999999998,"5min":118.36,"15min":118.36},"last":0.53800000000000003}]},{"osd":6,"last update":"Tue Sep 10 18:43:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":6.2859999999999996,"15min":3.3580000000000001},"min":{"1min":0.38600000000000001,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":6.8710000000000004,"5min":389.52499999999998,"15min":389.52499999999998},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.546,"5min":6.2709999999999999,"15min":3.5910000000000002},"min":{"1min":0.36499999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":7.7720000000000002,"5min":389.69499999999999,"15min":389.69499999999999},"last":0.58399999999999996}]},{"osd":7,"last update":"Tue Sep 10 18:43:09 2024","interfaces":[{"interface":"back","average":{"1min":1.9590000000000001,"5min":1.476,"15min":1.347},"min":{"1min":0.48399999999999999,"5min":0.36399999999999999,"15min":0.32700000000000001},"max":{"1min":6.8339999999999996,"5min":16.244,"15min":16.244},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.3460000000000001,"15min":1.3320000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":4.6369999999999996,"5min":16.550000000000001,"15min":16.550000000000001},"last":0.56399999999999995}]}]},{"osd":0,"up_from":0,"seq":34359738673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:43:52.534 INFO:tasks.thrashosds.thrasher:Setting 6.17 to [7, 3, 6] 2024-09-10T18:43:52.534 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.17', '7', '3', '6'] 2024-09-10T18:43:52.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pg-upmap 6.17 7 3 6 2024-09-10T18:43:52.599 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:43:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:43:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:43:53.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:53 smithi028 ceph-mon[29291]: from='client.36086 v1:172.21.15.28:0/4031376035' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:43:53.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:53 smithi028 ceph-mon[29291]: 5.2 deep-scrub starts 2024-09-10T18:43:53.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:53 smithi028 ceph-mon[29291]: 5.2 deep-scrub ok 2024-09-10T18:43:53.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:53 smithi028 ceph-mon[29291]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:43:53.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:53 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1576644130' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:43:53.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:53 smithi028 ceph-mon[29291]: osdmap e423: 8 total, 8 up, 7 in 2024-09-10T18:43:53.156 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:53 smithi028 ceph-mon[39220]: from='client.36086 v1:172.21.15.28:0/4031376035' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:43:53.156 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:53 smithi028 ceph-mon[39220]: 5.2 deep-scrub starts 2024-09-10T18:43:53.157 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:53 smithi028 ceph-mon[39220]: 5.2 deep-scrub ok 2024-09-10T18:43:53.157 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:53 smithi028 ceph-mon[39220]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:43:53.157 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:53 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1576644130' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:43:53.157 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:53 smithi028 ceph-mon[39220]: osdmap e423: 8 total, 8 up, 7 in 2024-09-10T18:43:53.226 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:53.280 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:53 smithi204 ceph-mon[32357]: from='client.36086 v1:172.21.15.28:0/4031376035' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:43:53.280 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:53 smithi204 ceph-mon[32357]: 5.2 deep-scrub starts 2024-09-10T18:43:53.280 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:53 smithi204 ceph-mon[32357]: 5.2 deep-scrub ok 2024-09-10T18:43:53.280 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:53 smithi204 ceph-mon[32357]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:43:53.280 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:53 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1576644130' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:43:53.280 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:53 smithi204 ceph-mon[32357]: osdmap e423: 8 total, 8 up, 7 in 2024-09-10T18:43:53.524 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:43:53.525 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:43:53.525 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-10T18:43:53.525 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-10T18:43:53.525 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:43:54.215 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:54 smithi204 ceph-mon[32357]: pgmap v1003: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:43:54.215 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:54 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:43:54.241 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-10T18:43:54.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:54 smithi028 ceph-mon[29291]: pgmap v1003: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:43:54.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:54 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:43:54.288 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:54 smithi028 ceph-mon[39220]: pgmap v1003: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:43:54.288 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:54 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:43:54.492 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:43:55.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:43:55.565 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:43:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:43:55.868 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:56.091 INFO:teuthology.orchestra.run.smithi028.stderr:set 6.17 pg_upmap mapping to [7,3,6] 2024-09-10T18:43:56.115 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:43:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-10T18:43:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-10T18:43:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:43:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:214418 10.e 10:7f999048:::benchmark_data_smithi028_161947_object77494:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:43:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.071610+0000", 2024-09-10T18:43:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 304.04055122800003, 2024-09-10T18:43:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44580072599999998, 2024-09-10T18:43:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:43:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:43:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:43:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:43:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 214418 2024-09-10T18:43:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071610+0000", 2024-09-10T18:43:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071610+0000", 2024-09-10T18:43:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071608+0000", 2024-09-10T18:43:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:43:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071616+0000", 2024-09-10T18:43:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6369999999999999e-06 2024-09-10T18:43:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071629+0000", 2024-09-10T18:43:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3104e-05 2024-09-10T18:43:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071635+0000", 2024-09-10T18:43:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1840000000000004e-06 2024-09-10T18:43:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071687+0000", 2024-09-10T18:43:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1332999999999998e-05 2024-09-10T18:43:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071806+0000", 2024-09-10T18:43:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119496 2024-09-10T18:43:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,6", 2024-09-10T18:43:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071935+0000", 2024-09-10T18:43:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012872200000000001 2024-09-10T18:43:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:43:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.074363+0000", 2024-09-10T18:43:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024283059999999999 2024-09-10T18:43:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.078256+0000", 2024-09-10T18:43:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038927699999999998 2024-09-10T18:43:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517313+0000", 2024-09-10T18:43:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43905738999999999 2024-09-10T18:43:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:43:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517370+0000", 2024-09-10T18:43:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6452000000000001e-05 2024-09-10T18:43:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517411+0000", 2024-09-10T18:43:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1396999999999997e-05 2024-09-10T18:43:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:214433 10.e 10:7bdefcb6:::benchmark_data_smithi028_161947_object77509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:43:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.084131+0000", 2024-09-10T18:43:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 304.02803076499998, 2024-09-10T18:43:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43341650799999998, 2024-09-10T18:43:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:43:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:43:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:43:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:43:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 214433 2024-09-10T18:43:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.084131+0000", 2024-09-10T18:43:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.084131+0000", 2024-09-10T18:43:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.084129+0000", 2024-09-10T18:43:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:43:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.084136+0000", 2024-09-10T18:43:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2899999999999997e-06 2024-09-10T18:43:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.084146+0000", 2024-09-10T18:43:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0216e-05 2024-09-10T18:43:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.084151+0000", 2024-09-10T18:43:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8829999999999997e-06 2024-09-10T18:43:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.084190+0000", 2024-09-10T18:43:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8608e-05 2024-09-10T18:43:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.084279+0000", 2024-09-10T18:43:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9281000000000001e-05 2024-09-10T18:43:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,6", 2024-09-10T18:43:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.084359+0000", 2024-09-10T18:43:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0124999999999996e-05 2024-09-10T18:43:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:43:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.085852+0000", 2024-09-10T18:43:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014931720000000001 2024-09-10T18:43:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.086652+0000", 2024-09-10T18:43:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00079964999999999995 2024-09-10T18:43:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517451+0000", 2024-09-10T18:43:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43079856399999999 2024-09-10T18:43:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:43:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517508+0000", 2024-09-10T18:43:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7785999999999999e-05 2024-09-10T18:43:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517547+0000", 2024-09-10T18:43:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9031e-05 2024-09-10T18:43:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.34334.0:2431 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94344803730768 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-10T18:43:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.462235+0000", 2024-09-10T18:43:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 295.649926134, 2024-09-10T18:43:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45165153400000002, 2024-09-10T18:43:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:43:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:43:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.34334", 2024-09-10T18:43:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/601695668", 2024-09-10T18:43:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2431 2024-09-10T18:43:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462235+0000", 2024-09-10T18:43:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462235+0000", 2024-09-10T18:43:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462234+0000", 2024-09-10T18:43:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:43:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462240+0000", 2024-09-10T18:43:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1940000000000003e-06 2024-09-10T18:43:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462250+0000", 2024-09-10T18:43:56.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0375e-05 2024-09-10T18:43:56.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462256+0000", 2024-09-10T18:43:56.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2820000000000004e-06 2024-09-10T18:43:56.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.484858+0000", 2024-09-10T18:43:56.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.02260204 2024-09-10T18:43:56.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:43:56.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.484867+0000", 2024-09-10T18:43:56.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6010000000000001e-06 2024-09-10T18:43:56.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.895162+0000", 2024-09-10T18:43:56.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.410294571 2024-09-10T18:43:56.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for degraded object", 2024-09-10T18:43:56.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.895932+0000", 2024-09-10T18:43:56.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00077006100000000005 2024-09-10T18:43:56.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.908245+0000", 2024-09-10T18:43:56.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012312599 2024-09-10T18:43:56.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.908353+0000", 2024-09-10T18:43:56.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108498 2024-09-10T18:43:56.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.913887+0000", 2024-09-10T18:43:56.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0055338430000000001 2024-09-10T18:43:56.134 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.134 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.135 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.135 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.34334.0:2470 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94344803730768 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:43:56.135 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.216550+0000", 2024-09-10T18:43:56.135 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 271.89561136200001, 2024-09-10T18:43:56.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0359037019999999, 2024-09-10T18:43:56.135 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.135 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:43:56.135 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:43:56.135 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.34334", 2024-09-10T18:43:56.135 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/601695668", 2024-09-10T18:43:56.135 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2470 2024-09-10T18:43:56.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.136 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216550+0000", 2024-09-10T18:43:56.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216550+0000", 2024-09-10T18:43:56.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216548+0000", 2024-09-10T18:43:56.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:43:56.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216556+0000", 2024-09-10T18:43:56.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5320000000000001e-06 2024-09-10T18:43:56.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216567+0000", 2024-09-10T18:43:56.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1364e-05 2024-09-10T18:43:56.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216573+0000", 2024-09-10T18:43:56.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2310000000000001e-06 2024-09-10T18:43:56.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216619+0000", 2024-09-10T18:43:56.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5479999999999998e-05 2024-09-10T18:43:56.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-10T18:43:56.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216625+0000", 2024-09-10T18:43:56.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0329999999999997e-06 2024-09-10T18:43:56.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.241166+0000", 2024-09-10T18:43:56.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024541344 2024-09-10T18:43:56.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:43:56.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.241237+0000", 2024-09-10T18:43:56.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1252000000000002e-05 2024-09-10T18:43:56.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.239505+0000", 2024-09-10T18:43:56.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.998267248 2024-09-10T18:43:56.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for degraded object", 2024-09-10T18:43:56.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.240339+0000", 2024-09-10T18:43:56.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00083466400000000004 2024-09-10T18:43:56.140 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.249980+0000", 2024-09-10T18:43:56.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0096410620000000006 2024-09-10T18:43:56.140 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.250054+0000", 2024-09-10T18:43:56.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3467999999999995e-05 2024-09-10T18:43:56.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.252454+0000", 2024-09-10T18:43:56.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024001640000000002 2024-09-10T18:43:56.141 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.141 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.141 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.141 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253538 10.1 10:8d536b43:::benchmark_data_smithi028_161947_object116614:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:43:56.141 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.238895+0000", 2024-09-10T18:43:56.141 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 271.87326672299997, 2024-09-10T18:43:56.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2288600299999999, 2024-09-10T18:43:56.142 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.142 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:43:56.142 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:43:56.142 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:43:56.142 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:43:56.142 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253538 2024-09-10T18:43:56.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.142 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.238895+0000", 2024-09-10T18:43:56.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.238895+0000", 2024-09-10T18:43:56.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.238893+0000", 2024-09-10T18:43:56.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:43:56.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.238899+0000", 2024-09-10T18:43:56.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1399999999999997e-06 2024-09-10T18:43:56.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.144 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.144 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.238908+0000", 2024-09-10T18:43:56.144 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8100000000000004e-06 2024-09-10T18:43:56.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.144 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.144 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.238914+0000", 2024-09-10T18:43:56.144 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0299999999999999e-06 2024-09-10T18:43:56.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.144 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.239098+0000", 2024-09-10T18:43:56.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018371699999999999 2024-09-10T18:43:56.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:43:56.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.239109+0000", 2024-09-10T18:43:56.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1073000000000001e-05 2024-09-10T18:43:56.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.221047+0000", 2024-09-10T18:43:56.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98193805899999997 2024-09-10T18:43:56.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.146 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:43:56.146 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.221061+0000", 2024-09-10T18:43:56.146 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4287999999999999e-05 2024-09-10T18:43:56.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.146 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.146 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.146 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.410413+0000", 2024-09-10T18:43:56.146 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.189352075 2024-09-10T18:43:56.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.146 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.146 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.146 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.410505+0000", 2024-09-10T18:43:56.146 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1532000000000004e-05 2024-09-10T18:43:56.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1,7", 2024-09-10T18:43:56.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.410805+0000", 2024-09-10T18:43:56.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000300153 2024-09-10T18:43:56.147 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:43:56.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.424992+0000", 2024-09-10T18:43:56.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014187396 2024-09-10T18:43:56.147 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.425296+0000", 2024-09-10T18:43:56.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000303469 2024-09-10T18:43:56.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.148 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.148 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.464499+0000", 2024-09-10T18:43:56.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.039202846999999999 2024-09-10T18:43:56.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.148 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.148 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.467542+0000", 2024-09-10T18:43:56.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003043736 2024-09-10T18:43:56.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.148 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:43:56.149 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.467630+0000", 2024-09-10T18:43:56.149 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7664999999999995e-05 2024-09-10T18:43:56.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.149 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.149 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.149 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.467755+0000", 2024-09-10T18:43:56.149 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000124738 2024-09-10T18:43:56.149 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.149 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.149 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.149 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.149 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253551 10.1 10:849d8214:::benchmark_data_smithi028_161947_object116627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:43:56.149 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.253322+0000", 2024-09-10T18:43:56.150 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 271.85883997399998, 2024-09-10T18:43:56.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.214642462, 2024-09-10T18:43:56.150 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.150 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:43:56.150 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:43:56.150 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:43:56.150 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:43:56.150 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253551 2024-09-10T18:43:56.150 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.150 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.150 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.253322+0000", 2024-09-10T18:43:56.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.253322+0000", 2024-09-10T18:43:56.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.253320+0000", 2024-09-10T18:43:56.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:43:56.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.253326+0000", 2024-09-10T18:43:56.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3890000000000004e-06 2024-09-10T18:43:56.152 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.152 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.152 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.253334+0000", 2024-09-10T18:43:56.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7709999999999995e-06 2024-09-10T18:43:56.152 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.152 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.152 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.253339+0000", 2024-09-10T18:43:56.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.194e-06 2024-09-10T18:43:56.152 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.152 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.253368+0000", 2024-09-10T18:43:56.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.968e-05 2024-09-10T18:43:56.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:43:56.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.253374+0000", 2024-09-10T18:43:56.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1249999999999998e-06 2024-09-10T18:43:56.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.221089+0000", 2024-09-10T18:43:56.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.96771441899999999 2024-09-10T18:43:56.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:43:56.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.221096+0000", 2024-09-10T18:43:56.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2969999999999998e-06 2024-09-10T18:43:56.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.418112+0000", 2024-09-10T18:43:56.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1970163039999999 2024-09-10T18:43:56.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.418233+0000", 2024-09-10T18:43:56.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000120878 2024-09-10T18:43:56.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1,7", 2024-09-10T18:43:56.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.418368+0000", 2024-09-10T18:43:56.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000134337 2024-09-10T18:43:56.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:43:56.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.424998+0000", 2024-09-10T18:43:56.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006630487 2024-09-10T18:43:56.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.429726+0000", 2024-09-10T18:43:56.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047277359999999997 2024-09-10T18:43:56.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.156 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.156 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.464552+0000", 2024-09-10T18:43:56.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.034826024999999997 2024-09-10T18:43:56.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.156 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.156 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.467795+0000", 2024-09-10T18:43:56.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032427670000000001 2024-09-10T18:43:56.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.157 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:43:56.157 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.467854+0000", 2024-09-10T18:43:56.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9008000000000002e-05 2024-09-10T18:43:56.157 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.157 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.157 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.467964+0000", 2024-09-10T18:43:56.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000110492 2024-09-10T18:43:56.157 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.157 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.157 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.157 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.157 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253557 10.4 10:2d430a53:::benchmark_data_smithi028_161947_object116633:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:43:56.158 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.275676+0000", 2024-09-10T18:43:56.158 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 271.83648595400001, 2024-09-10T18:43:56.158 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2102520019999998, 2024-09-10T18:43:56.158 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.158 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:43:56.158 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:43:56.158 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:43:56.158 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:43:56.158 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253557 2024-09-10T18:43:56.158 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.158 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.158 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.158 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.158 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.275676+0000", 2024-09-10T18:43:56.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.275676+0000", 2024-09-10T18:43:56.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.275674+0000", 2024-09-10T18:43:56.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:43:56.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.275680+0000", 2024-09-10T18:43:56.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7520000000000002e-06 2024-09-10T18:43:56.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.160 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.275690+0000", 2024-09-10T18:43:56.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0193e-05 2024-09-10T18:43:56.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.160 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.275696+0000", 2024-09-10T18:43:56.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.071e-06 2024-09-10T18:43:56.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.275739+0000", 2024-09-10T18:43:56.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3260999999999998e-05 2024-09-10T18:43:56.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:43:56.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.275745+0000", 2024-09-10T18:43:56.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1489999999999999e-06 2024-09-10T18:43:56.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.225070+0000", 2024-09-10T18:43:56.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.94932420399999995 2024-09-10T18:43:56.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.162 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:43:56.162 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.225080+0000", 2024-09-10T18:43:56.162 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0859999999999999e-05 2024-09-10T18:43:56.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.162 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.162 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.423407+0000", 2024-09-10T18:43:56.162 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1983269910000001 2024-09-10T18:43:56.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.162 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.162 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.426495+0000", 2024-09-10T18:43:56.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003087565 2024-09-10T18:43:56.163 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.163 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.163 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1,3", 2024-09-10T18:43:56.163 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.426742+0000", 2024-09-10T18:43:56.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000246829 2024-09-10T18:43:56.163 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.163 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.163 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:43:56.163 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.437266+0000", 2024-09-10T18:43:56.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010523974 2024-09-10T18:43:56.163 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.163 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.465052+0000", 2024-09-10T18:43:56.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.02778657 2024-09-10T18:43:56.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.465093+0000", 2024-09-10T18:43:56.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1043e-05 2024-09-10T18:43:56.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.485759+0000", 2024-09-10T18:43:56.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020665734000000002 2024-09-10T18:43:56.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.165 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:43:56.165 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.485813+0000", 2024-09-10T18:43:56.165 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3619000000000003e-05 2024-09-10T18:43:56.165 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.165 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.165 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.485928+0000", 2024-09-10T18:43:56.165 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114904 2024-09-10T18:43:56.165 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.165 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.165 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.165 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.165 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253558 10.1 10:876bdee7:::benchmark_data_smithi028_161947_object116634:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:43:56.166 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.275705+0000", 2024-09-10T18:43:56.166 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 271.83645625899999, 2024-09-10T18:43:56.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1924562430000001, 2024-09-10T18:43:56.166 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.166 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:43:56.166 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:43:56.166 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:43:56.166 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:43:56.166 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253558 2024-09-10T18:43:56.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.167 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.275705+0000", 2024-09-10T18:43:56.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.275705+0000", 2024-09-10T18:43:56.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.275705+0000", 2024-09-10T18:43:56.168 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999995 2024-09-10T18:43:56.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.168 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.275708+0000", 2024-09-10T18:43:56.168 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7249999999999999e-06 2024-09-10T18:43:56.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.168 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.275720+0000", 2024-09-10T18:43:56.168 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1259e-05 2024-09-10T18:43:56.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.168 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.169 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.275725+0000", 2024-09-10T18:43:56.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.04e-06 2024-09-10T18:43:56.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.169 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.275765+0000", 2024-09-10T18:43:56.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9932999999999999e-05 2024-09-10T18:43:56.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:43:56.169 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.275771+0000", 2024-09-10T18:43:56.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6939999999999998e-06 2024-09-10T18:43:56.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.170 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.170 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.221121+0000", 2024-09-10T18:43:56.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.94534985999999999 2024-09-10T18:43:56.170 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.170 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:43:56.170 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.221132+0000", 2024-09-10T18:43:56.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0376e-05 2024-09-10T18:43:56.170 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.170 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.170 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.421932+0000", 2024-09-10T18:43:56.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2008004329999999 2024-09-10T18:43:56.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.422071+0000", 2024-09-10T18:43:56.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013893 2024-09-10T18:43:56.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1,7", 2024-09-10T18:43:56.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.422210+0000", 2024-09-10T18:43:56.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000138799 2024-09-10T18:43:56.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:43:56.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.425696+0000", 2024-09-10T18:43:56.172 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034863979999999999 2024-09-10T18:43:56.172 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.172 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.172 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.172 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.429773+0000", 2024-09-10T18:43:56.172 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040764900000000003 2024-09-10T18:43:56.172 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.172 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.172 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.172 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.465028+0000", 2024-09-10T18:43:56.172 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.035255005999999998 2024-09-10T18:43:56.172 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.172 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.172 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.173 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468003+0000", 2024-09-10T18:43:56.173 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029758889999999998 2024-09-10T18:43:56.173 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.173 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.173 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:43:56.173 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468054+0000", 2024-09-10T18:43:56.173 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0646000000000003e-05 2024-09-10T18:43:56.173 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.173 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.173 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.173 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468162+0000", 2024-09-10T18:43:56.173 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010744 2024-09-10T18:43:56.173 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.173 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.173 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.174 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.174 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.34334.0:2476 5.e 5:7759931f:::notify.3:head [watch ping cookie 94344803730768 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:43:56.174 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.976253+0000", 2024-09-10T18:43:56.174 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 271.13590836200001, 2024-09-10T18:43:56.174 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.274794708, 2024-09-10T18:43:56.174 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.174 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:43:56.174 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:43:56.174 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.34334", 2024-09-10T18:43:56.174 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/601695668", 2024-09-10T18:43:56.174 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2476 2024-09-10T18:43:56.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.175 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.976253+0000", 2024-09-10T18:43:56.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.175 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.976253+0000", 2024-09-10T18:43:56.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.175 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.976252+0000", 2024-09-10T18:43:56.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:43:56.176 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.176 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.176 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.176 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.976258+0000", 2024-09-10T18:43:56.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9200000000000001e-06 2024-09-10T18:43:56.176 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.176 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.176 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.176 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.976265+0000", 2024-09-10T18:43:56.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9310000000000002e-06 2024-09-10T18:43:56.176 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.176 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.176 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.177 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.976271+0000", 2024-09-10T18:43:56.177 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7689999999999999e-06 2024-09-10T18:43:56.177 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.177 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.177 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.177 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.976330+0000", 2024-09-10T18:43:56.177 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8510000000000001e-05 2024-09-10T18:43:56.177 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.177 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.177 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:43:56.177 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.976341+0000", 2024-09-10T18:43:56.177 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1504e-05 2024-09-10T18:43:56.177 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.177 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.178 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.178 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.240380+0000", 2024-09-10T18:43:56.178 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.26403821300000002 2024-09-10T18:43:56.178 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.178 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.178 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for degraded object", 2024-09-10T18:43:56.178 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.240748+0000", 2024-09-10T18:43:56.178 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00036858399999999998 2024-09-10T18:43:56.178 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.178 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.178 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.178 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.250783+0000", 2024-09-10T18:43:56.178 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010035183 2024-09-10T18:43:56.178 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.179 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.179 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.179 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.250869+0000", 2024-09-10T18:43:56.179 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5609e-05 2024-09-10T18:43:56.179 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.179 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.179 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.179 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.251048+0000", 2024-09-10T18:43:56.179 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017894499999999999 2024-09-10T18:43:56.179 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.179 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.179 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.179 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.179 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253960 10.4 10:2096eec1:::benchmark_data_smithi028_161947_object117036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-10T18:43:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:26.817642+0000", 2024-09-10T18:43:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 269.29451999200001, 2024-09-10T18:43:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.36089289499999999, 2024-09-10T18:43:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:43:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:43:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:43:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:43:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253960 2024-09-10T18:43:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.817642+0000", 2024-09-10T18:43:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.817642+0000", 2024-09-10T18:43:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.817639+0000", 2024-09-10T18:43:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:43:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.182 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.182 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.817647+0000", 2024-09-10T18:43:56.182 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0380000000000002e-06 2024-09-10T18:43:56.182 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.182 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.182 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.182 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.817657+0000", 2024-09-10T18:43:56.182 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0939e-05 2024-09-10T18:43:56.182 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.182 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.182 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.182 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.817664+0000", 2024-09-10T18:43:56.182 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.37e-06 2024-09-10T18:43:56.182 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.182 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.817829+0000", 2024-09-10T18:43:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016520900000000001 2024-09-10T18:43:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.818016+0000", 2024-09-10T18:43:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018723599999999999 2024-09-10T18:43:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1,3", 2024-09-10T18:43:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.818153+0000", 2024-09-10T18:43:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013688799999999999 2024-09-10T18:43:56.183 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.826238+0000", 2024-09-10T18:43:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0080844179999999995 2024-09-10T18:43:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.826270+0000", 2024-09-10T18:43:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2864000000000002e-05 2024-09-10T18:43:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:43:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.830780+0000", 2024-09-10T18:43:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0045099679999999996 2024-09-10T18:43:56.184 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:27.178339+0000", 2024-09-10T18:43:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.34755814899999998 2024-09-10T18:43:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.185 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:43:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:27.178391+0000", 2024-09-10T18:43:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2000999999999999e-05 2024-09-10T18:43:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:27.178534+0000", 2024-09-10T18:43:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000143943 2024-09-10T18:43:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253971 10.4 10:2bc86aca:::benchmark_data_smithi028_161947_object117047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-10T18:43:56.186 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:26.825485+0000", 2024-09-10T18:43:56.187 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 269.28667679400002, 2024-09-10T18:43:56.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.35721280999999999, 2024-09-10T18:43:56.187 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.187 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:43:56.187 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:43:56.187 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:43:56.187 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:43:56.187 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253971 2024-09-10T18:43:56.187 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.187 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.187 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.187 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.187 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.825485+0000", 2024-09-10T18:43:56.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.187 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.188 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.188 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.188 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.825485+0000", 2024-09-10T18:43:56.188 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.188 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.188 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.188 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.188 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.825482+0000", 2024-09-10T18:43:56.188 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:43:56.188 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.188 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.188 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.188 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.825490+0000", 2024-09-10T18:43:56.188 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9799999999999998e-06 2024-09-10T18:43:56.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.825502+0000", 2024-09-10T18:43:56.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1236000000000001e-05 2024-09-10T18:43:56.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.825508+0000", 2024-09-10T18:43:56.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9989999999999997e-06 2024-09-10T18:43:56.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.828266+0000", 2024-09-10T18:43:56.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027587359999999999 2024-09-10T18:43:56.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.828483+0000", 2024-09-10T18:43:56.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021660399999999999 2024-09-10T18:43:56.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1,3", 2024-09-10T18:43:56.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.828619+0000", 2024-09-10T18:43:56.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013557200000000001 2024-09-10T18:43:56.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.830844+0000", 2024-09-10T18:43:56.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022257900000000001 2024-09-10T18:43:56.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:43:56.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.830851+0000", 2024-09-10T18:43:56.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8050000000000001e-06 2024-09-10T18:43:56.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.830894+0000", 2024-09-10T18:43:56.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2929000000000002e-05 2024-09-10T18:43:56.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.192 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.192 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.192 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:27.178570+0000", 2024-09-10T18:43:56.192 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.34767601300000001 2024-09-10T18:43:56.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.192 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.192 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:43:56.192 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:27.182585+0000", 2024-09-10T18:43:56.192 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040150230000000004 2024-09-10T18:43:56.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.192 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.192 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.192 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:27.182698+0000", 2024-09-10T18:43:56.192 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112438 2024-09-10T18:43:56.193 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.193 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.193 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.193 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.193 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.193 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253974 10.4 10:25b7ea01:::benchmark_data_smithi028_161947_object117050:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-10T18:43:56.193 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:26.828777+0000", 2024-09-10T18:43:56.193 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 269.28338501899998, 2024-09-10T18:43:56.193 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.35413370199999999, 2024-09-10T18:43:56.193 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.193 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:43:56.193 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:43:56.193 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:43:56.193 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:43:56.194 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253974 2024-09-10T18:43:56.194 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.194 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.194 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.194 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.194 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.828777+0000", 2024-09-10T18:43:56.194 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.194 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.194 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.194 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.194 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.828777+0000", 2024-09-10T18:43:56.194 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.194 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.194 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.195 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.195 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.828775+0000", 2024-09-10T18:43:56.195 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:43:56.195 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.195 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.195 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.195 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.828781+0000", 2024-09-10T18:43:56.195 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8080000000000001e-06 2024-09-10T18:43:56.195 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.195 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.195 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.195 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.828789+0000", 2024-09-10T18:43:56.195 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9279999999999993e-06 2024-09-10T18:43:56.195 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.196 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.196 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.196 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.828793+0000", 2024-09-10T18:43:56.196 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.968e-06 2024-09-10T18:43:56.196 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.196 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.196 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.196 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.831017+0000", 2024-09-10T18:43:56.196 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002224097 2024-09-10T18:43:56.196 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.196 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.196 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.196 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.831196+0000", 2024-09-10T18:43:56.196 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017928 2024-09-10T18:43:56.196 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.197 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.197 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1,3", 2024-09-10T18:43:56.197 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.831298+0000", 2024-09-10T18:43:56.197 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102067 2024-09-10T18:43:56.197 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.197 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.197 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.197 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.834391+0000", 2024-09-10T18:43:56.197 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030925929999999998 2024-09-10T18:43:56.197 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.197 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.197 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.834421+0000", 2024-09-10T18:43:56.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0431000000000001e-05 2024-09-10T18:43:56.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.198 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:43:56.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.838588+0000", 2024-09-10T18:43:56.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004166739 2024-09-10T18:43:56.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.198 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:27.182736+0000", 2024-09-10T18:43:56.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.34414810299999998 2024-09-10T18:43:56.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:43:56.199 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:27.182804+0000", 2024-09-10T18:43:56.199 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8310000000000002e-05 2024-09-10T18:43:56.199 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.199 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.199 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:27.182910+0000", 2024-09-10T18:43:56.199 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105871 2024-09-10T18:43:56.199 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.199 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.199 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:56 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:43:56.200 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:56 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:43:56.200 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:56 smithi204 ceph-mon[32357]: osdmap e424: 8 total, 8 up, 7 in 2024-09-10T18:43:56.200 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:56 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:56.200 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:56 smithi204 ceph-mon[32357]: pgmap v1005: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:43:56.200 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:56 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3227176759' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [7, 3, 6]}]: dispatch 2024-09-10T18:43:56.200 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:56 smithi204 ceph-mon[32357]: from='client.36092 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [7, 3, 6]}]: dispatch 2024-09-10T18:43:56.200 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.200 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.201 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.201 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25849.0:92933 11.6 e392/374 11:6364bffd:::benchmark_data_smithi028_186962_object92932:head v 392'5763, mlcod=392'5761)", 2024-09-10T18:43:56.201 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:11.597833+0000", 2024-09-10T18:43:56.201 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 164.51432845900001, 2024-09-10T18:43:56.201 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29788762899999999, 2024-09-10T18:43:56.201 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.201 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:43:56.201 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.201 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.201 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.201 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.597833+0000", 2024-09-10T18:43:56.201 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.201 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.201 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.202 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.202 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.597833+0000", 2024-09-10T18:43:56.202 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.202 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.202 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.202 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.202 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.597832+0000", 2024-09-10T18:43:56.202 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:43:56.202 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.202 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.202 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.202 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.597862+0000", 2024-09-10T18:43:56.202 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9859999999999999e-05 2024-09-10T18:43:56.202 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.202 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.203 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.203 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.597870+0000", 2024-09-10T18:43:56.203 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3629999999999996e-06 2024-09-10T18:43:56.203 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.203 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.203 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.203 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.597883+0000", 2024-09-10T18:43:56.203 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3609e-05 2024-09-10T18:43:56.203 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.203 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.203 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.203 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.602221+0000", 2024-09-10T18:43:56.203 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0043378260000000004 2024-09-10T18:43:56.203 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.204 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.204 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.204 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.602238+0000", 2024-09-10T18:43:56.204 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7402999999999999e-05 2024-09-10T18:43:56.204 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.204 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.204 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:43:56.204 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.895688+0000", 2024-09-10T18:43:56.204 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29344992199999997 2024-09-10T18:43:56.204 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.204 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.204 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.204 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.895721+0000", 2024-09-10T18:43:56.204 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2471000000000001e-05 2024-09-10T18:43:56.204 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.205 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.205 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.205 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.205 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.205 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25849.0:92942 11.6 e392/374 11:6c8cd167:::benchmark_data_smithi028_186962_object92941:head v 392'5764, mlcod=392'5761)", 2024-09-10T18:43:56.205 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:11.602911+0000", 2024-09-10T18:43:56.205 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 164.50925030600001, 2024-09-10T18:43:56.205 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29283145399999999, 2024-09-10T18:43:56.205 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.205 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:43:56.205 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.205 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.205 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.206 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.602911+0000", 2024-09-10T18:43:56.206 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.206 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.206 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.206 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.206 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.602911+0000", 2024-09-10T18:43:56.206 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.206 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.206 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.206 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.206 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.602910+0000", 2024-09-10T18:43:56.206 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:43:56.206 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.206 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.207 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.207 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.603009+0000", 2024-09-10T18:43:56.207 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8696000000000005e-05 2024-09-10T18:43:56.207 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.207 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.207 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.207 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.603017+0000", 2024-09-10T18:43:56.207 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5539999999999998e-06 2024-09-10T18:43:56.207 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.207 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.207 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.207 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.603022+0000", 2024-09-10T18:43:56.208 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9450000000000001e-06 2024-09-10T18:43:56.208 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.208 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.208 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.208 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.606269+0000", 2024-09-10T18:43:56.208 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003246451 2024-09-10T18:43:56.208 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.208 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.208 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.208 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.606284+0000", 2024-09-10T18:43:56.208 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5855e-05 2024-09-10T18:43:56.208 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.208 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.208 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:43:56.209 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.895727+0000", 2024-09-10T18:43:56.209 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.28944249599999999 2024-09-10T18:43:56.209 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.209 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.209 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.209 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.895743+0000", 2024-09-10T18:43:56.209 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5832999999999999e-05 2024-09-10T18:43:56.209 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.209 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.209 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.209 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.209 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.209 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25849.0:92947 11.c e392/374 11:3d5bac5c:::benchmark_data_smithi028_186962_object92946:head v 392'5730, mlcod=392'5728)", 2024-09-10T18:43:56.210 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:11.607432+0000", 2024-09-10T18:43:56.210 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 164.50472964900001, 2024-09-10T18:43:56.210 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.288642962, 2024-09-10T18:43:56.210 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.210 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:43:56.210 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.210 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.210 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.210 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607432+0000", 2024-09-10T18:43:56.210 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.210 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.210 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.210 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.211 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607432+0000", 2024-09-10T18:43:56.211 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.211 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.211 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.211 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.211 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607431+0000", 2024-09-10T18:43:56.211 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:43:56.211 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.211 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.211 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.211 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607480+0000", 2024-09-10T18:43:56.211 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.88e-05 2024-09-10T18:43:56.211 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.211 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.212 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.212 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607488+0000", 2024-09-10T18:43:56.212 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7330000000000003e-06 2024-09-10T18:43:56.212 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.212 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.212 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.212 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607493+0000", 2024-09-10T18:43:56.212 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0950000000000003e-06 2024-09-10T18:43:56.212 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.212 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.212 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.212 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607536+0000", 2024-09-10T18:43:56.212 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3226000000000003e-05 2024-09-10T18:43:56.213 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.213 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.213 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.213 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607554+0000", 2024-09-10T18:43:56.213 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7869000000000001e-05 2024-09-10T18:43:56.213 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.213 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.213 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:43:56.213 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.896004+0000", 2024-09-10T18:43:56.213 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.288450757 2024-09-10T18:43:56.213 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.213 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.213 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.213 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.896075+0000", 2024-09-10T18:43:56.213 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0451999999999996e-05 2024-09-10T18:43:56.214 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.214 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.214 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.214 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.214 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.214 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25849.0:92937 11.7 e392/374 11:e32322ce:::benchmark_data_smithi028_186962_object92936:head v 392'5840, mlcod=392'5837)", 2024-09-10T18:43:56.214 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:11.607501+0000", 2024-09-10T18:43:56.214 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 164.50466095600001, 2024-09-10T18:43:56.214 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.285304999, 2024-09-10T18:43:56.214 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.214 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:43:56.214 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.214 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.214 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.215 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607501+0000", 2024-09-10T18:43:56.215 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.215 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.215 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.215 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.215 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607501+0000", 2024-09-10T18:43:56.215 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.215 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.215 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.215 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.215 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607499+0000", 2024-09-10T18:43:56.215 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:43:56.215 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.215 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.216 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.216 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607549+0000", 2024-09-10T18:43:56.216 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0472000000000001e-05 2024-09-10T18:43:56.216 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.216 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.216 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.216 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607558+0000", 2024-09-10T18:43:56.216 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7220000000000005e-06 2024-09-10T18:43:56.216 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.216 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.216 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.216 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607563+0000", 2024-09-10T18:43:56.216 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2089999999999996e-06 2024-09-10T18:43:56.217 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.217 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.217 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.217 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607908+0000", 2024-09-10T18:43:56.217 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00034510500000000001 2024-09-10T18:43:56.217 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.217 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.217 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.217 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607929+0000", 2024-09-10T18:43:56.217 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0115999999999998e-05 2024-09-10T18:43:56.217 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.217 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.217 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:43:56.218 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.892774+0000", 2024-09-10T18:43:56.218 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.284845719 2024-09-10T18:43:56.218 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.218 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.218 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.218 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.892806+0000", 2024-09-10T18:43:56.218 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.133e-05 2024-09-10T18:43:56.218 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.218 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.218 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.218 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.218 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.218 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25849.0:92946 11.d e392/374 11:b49b9694:::benchmark_data_smithi028_186962_object92945:head v 392'5860, mlcod=392'5856)", 2024-09-10T18:43:56.219 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:11.607572+0000", 2024-09-10T18:43:56.219 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 164.50458993800001, 2024-09-10T18:43:56.219 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.289436162, 2024-09-10T18:43:56.219 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.219 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:43:56.219 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.219 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.219 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.219 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607572+0000", 2024-09-10T18:43:56.219 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.219 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.219 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.219 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.220 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607572+0000", 2024-09-10T18:43:56.220 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.220 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.220 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.220 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.220 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607571+0000", 2024-09-10T18:43:56.220 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:43:56.220 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.220 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.220 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.220 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607645+0000", 2024-09-10T18:43:56.220 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4266999999999999e-05 2024-09-10T18:43:56.221 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.221 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.221 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.221 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607652+0000", 2024-09-10T18:43:56.221 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2219999999999996e-06 2024-09-10T18:43:56.221 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.221 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.221 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.221 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607656+0000", 2024-09-10T18:43:56.221 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1949999999999996e-06 2024-09-10T18:43:56.221 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.221 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.221 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.221 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607702+0000", 2024-09-10T18:43:56.222 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.583e-05 2024-09-10T18:43:56.222 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.222 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.222 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.222 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.608316+0000", 2024-09-10T18:43:56.222 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00061384199999999997 2024-09-10T18:43:56.222 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.222 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.222 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:43:56.222 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.896971+0000", 2024-09-10T18:43:56.222 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.28865446099999997 2024-09-10T18:43:56.222 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.222 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.223 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.223 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.897008+0000", 2024-09-10T18:43:56.223 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7292000000000001e-05 2024-09-10T18:43:56.223 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.223 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.223 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.223 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.223 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.223 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25849.0:92941 11.7 e392/374 11:e141045f:::benchmark_data_smithi028_186962_object92940:head v 392'5841, mlcod=392'5838)", 2024-09-10T18:43:56.223 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:11.610525+0000", 2024-09-10T18:43:56.223 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 164.50163696600001, 2024-09-10T18:43:56.223 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.28928263399999998, 2024-09-10T18:43:56.223 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.224 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:43:56.224 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.224 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.224 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.224 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.610525+0000", 2024-09-10T18:43:56.224 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.224 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.224 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.224 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.224 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.610525+0000", 2024-09-10T18:43:56.224 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.224 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.224 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.225 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.225 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.610523+0000", 2024-09-10T18:43:56.225 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:43:56.225 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.225 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.225 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.225 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.610549+0000", 2024-09-10T18:43:56.225 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5437000000000001e-05 2024-09-10T18:43:56.225 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.225 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.225 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.225 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.892772+0000", 2024-09-10T18:43:56.225 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.28222306600000002 2024-09-10T18:43:56.225 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.226 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.226 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.226 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.892781+0000", 2024-09-10T18:43:56.226 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.302e-06 2024-09-10T18:43:56.226 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.226 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.226 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.226 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.892826+0000", 2024-09-10T18:43:56.226 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4922999999999997e-05 2024-09-10T18:43:56.226 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.226 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.226 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.226 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.892843+0000", 2024-09-10T18:43:56.226 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6714999999999998e-05 2024-09-10T18:43:56.227 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.227 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.227 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:43:56.227 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.899766+0000", 2024-09-10T18:43:56.227 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0069236109999999997 2024-09-10T18:43:56.227 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.227 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.227 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.227 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.899807+0000", 2024-09-10T18:43:56.227 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1127000000000001e-05 2024-09-10T18:43:56.227 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.227 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.227 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.227 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.227 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.227 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25849.0:92960 11.6 e392/374 11:6f05a9c1:::benchmark_data_smithi028_186962_object92959:head v 392'5765, mlcod=392'5761)", 2024-09-10T18:43:56.228 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:11.615965+0000", 2024-09-10T18:43:56.228 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 164.49619633699999, 2024-09-10T18:43:56.228 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.28014224500000001, 2024-09-10T18:43:56.228 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.228 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:43:56.228 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.228 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.228 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.228 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.615965+0000", 2024-09-10T18:43:56.228 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.228 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.228 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.228 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.228 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.615965+0000", 2024-09-10T18:43:56.228 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.228 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.229 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.229 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.229 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.615963+0000", 2024-09-10T18:43:56.229 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:43:56.229 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.229 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.229 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.229 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.616041+0000", 2024-09-10T18:43:56.229 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7090999999999998e-05 2024-09-10T18:43:56.229 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.229 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.229 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.229 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.892236+0000", 2024-09-10T18:43:56.230 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.27619505100000002 2024-09-10T18:43:56.230 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.230 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.230 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.230 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.892248+0000", 2024-09-10T18:43:56.230 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1987000000000001e-05 2024-09-10T18:43:56.230 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.230 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.230 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.230 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.892595+0000", 2024-09-10T18:43:56.230 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00034779 2024-09-10T18:43:56.230 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.230 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.230 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.230 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.892619+0000", 2024-09-10T18:43:56.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4153000000000001e-05 2024-09-10T18:43:56.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.231 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.231 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:43:56.231 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.896082+0000", 2024-09-10T18:43:56.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034628060000000001 2024-09-10T18:43:56.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.231 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.231 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.231 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.896107+0000", 2024-09-10T18:43:56.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5202e-05 2024-09-10T18:43:56.231 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.231 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.231 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.232 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.232 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.232 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:92981 11.e 11:755c6f22:::benchmark_data_smithi028_186962_object92980:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-09-10T18:43:56.232 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:11.647049+0000", 2024-09-10T18:43:56.232 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 164.465112172, 2024-09-10T18:43:56.232 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.25514259900000003, 2024-09-10T18:43:56.232 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:43:56.232 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:43:56.232 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:43:56.232 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:43:56.232 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:43:56.232 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 92981 2024-09-10T18:43:56.232 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.232 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:43:56.233 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.233 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:43:56.233 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.647049+0000", 2024-09-10T18:43:56.233 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.233 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.233 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.233 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:43:56.233 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.647049+0000", 2024-09-10T18:43:56.233 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:43:56.233 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.233 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.233 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:43:56.233 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.647046+0000", 2024-09-10T18:43:56.233 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:43:56.233 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:43:56.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.647091+0000", 2024-09-10T18:43:56.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4084e-05 2024-09-10T18:43:56.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:43:56.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.892294+0000", 2024-09-10T18:43:56.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.24520307299999999 2024-09-10T18:43:56.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:43:56.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.892303+0000", 2024-09-10T18:43:56.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6879999999999994e-06 2024-09-10T18:43:56.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:43:56.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.895090+0000", 2024-09-10T18:43:56.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027868290000000002 2024-09-10T18:43:56.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:43:56.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.895211+0000", 2024-09-10T18:43:56.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000120625 2024-09-10T18:43:56.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,7", 2024-09-10T18:43:56.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.895336+0000", 2024-09-10T18:43:56.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000124919 2024-09-10T18:43:56.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.236 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.236 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.898571+0000", 2024-09-10T18:43:56.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003235829 2024-09-10T18:43:56.236 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.236 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:43:56.236 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.899396+0000", 2024-09-10T18:43:56.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00082430999999999997 2024-09-10T18:43:56.236 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.236 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:43:56.236 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.902076+0000", 2024-09-10T18:43:56.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0026805819999999999 2024-09-10T18:43:56.237 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.237 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.237 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:43:56.237 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.902155+0000", 2024-09-10T18:43:56.237 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8973000000000004e-05 2024-09-10T18:43:56.237 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:43:56.237 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:43:56.237 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:43:56.237 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.902192+0000", 2024-09-10T18:43:56.237 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6671000000000002e-05 2024-09-10T18:43:56.237 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.237 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.237 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.237 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:43:56.237 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:43:56.238 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:43:56.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:56 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:43:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:56 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:43:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:56 smithi028 ceph-mon[29291]: osdmap e424: 8 total, 8 up, 7 in 2024-09-10T18:43:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:56 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:56 smithi028 ceph-mon[29291]: pgmap v1005: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:43:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:56 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3227176759' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [7, 3, 6]}]: dispatch 2024-09-10T18:43:56.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:56 smithi028 ceph-mon[29291]: from='client.36092 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [7, 3, 6]}]: dispatch 2024-09-10T18:43:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:56 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:43:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:56 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:43:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:56 smithi028 ceph-mon[39220]: osdmap e424: 8 total, 8 up, 7 in 2024-09-10T18:43:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:56 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:43:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:56 smithi028 ceph-mon[39220]: pgmap v1005: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:43:56.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:56 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3227176759' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [7, 3, 6]}]: dispatch 2024-09-10T18:43:56.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:56 smithi028 ceph-mon[39220]: from='client.36092 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [7, 3, 6]}]: dispatch 2024-09-10T18:43:56.933 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-10T18:43:57.185 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:43:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:57 smithi204 ceph-mon[32357]: from='client.36092 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [7, 3, 6]}]': finished 2024-09-10T18:43:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:57 smithi204 ceph-mon[32357]: osdmap e425: 8 total, 8 up, 7 in 2024-09-10T18:43:57.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:57 smithi028 ceph-mon[39220]: from='client.36092 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [7, 3, 6]}]': finished 2024-09-10T18:43:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:57 smithi028 ceph-mon[39220]: osdmap e425: 8 total, 8 up, 7 in 2024-09-10T18:43:57.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:57 smithi028 ceph-mon[29291]: from='client.36092 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.17", "id": [7, 3, 6]}]': finished 2024-09-10T18:43:57.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:57 smithi028 ceph-mon[29291]: osdmap e425: 8 total, 8 up, 7 in 2024-09-10T18:43:58.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:58 smithi028 ceph-mon[29291]: osdmap e426: 8 total, 8 up, 7 in 2024-09-10T18:43:58.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:58 smithi028 ceph-mon[29291]: pgmap v1008: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 980 B/s rd, 0 op/s 2024-09-10T18:43:58.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:58 smithi028 ceph-mon[39220]: osdmap e426: 8 total, 8 up, 7 in 2024-09-10T18:43:58.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:58 smithi028 ceph-mon[39220]: pgmap v1008: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 980 B/s rd, 0 op/s 2024-09-10T18:43:58.298 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:58 smithi204 ceph-mon[32357]: osdmap e426: 8 total, 8 up, 7 in 2024-09-10T18:43:58.298 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:58 smithi204 ceph-mon[32357]: pgmap v1008: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 980 B/s rd, 0 op/s 2024-09-10T18:43:58.803 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:43:58.804 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:43:58.804 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-10T18:43:58.804 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:43:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-10T18:43:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:43:59 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2975585665' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:43:59.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-10T18:43:59.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:43:59 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2975585665' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:43:59.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:43:59 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2975585665' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:43:59.647 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:43:59.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:44:00.143 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:00 smithi204 ceph-mon[32357]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-10T18:44:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:00 smithi204 ceph-mon[32357]: Cluster is now healthy 2024-09-10T18:44:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:00 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2975585665' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:44:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:00 smithi204 ceph-mon[32357]: osdmap e427: 8 total, 8 up, 7 in 2024-09-10T18:44:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:00 smithi204 ceph-mon[32357]: pgmap v1010: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:44:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:00 smithi204 ceph-mon[32357]: 6.0 scrub starts 2024-09-10T18:44:00.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:00 smithi204 ceph-mon[32357]: 6.0 scrub ok 2024-09-10T18:44:00.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:00 smithi028 ceph-mon[29291]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-10T18:44:00.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:00 smithi028 ceph-mon[29291]: Cluster is now healthy 2024-09-10T18:44:00.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:00 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2975585665' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:44:00.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:00 smithi028 ceph-mon[29291]: osdmap e427: 8 total, 8 up, 7 in 2024-09-10T18:44:00.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:00 smithi028 ceph-mon[29291]: pgmap v1010: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:44:00.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:00 smithi028 ceph-mon[29291]: 6.0 scrub starts 2024-09-10T18:44:00.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:00 smithi028 ceph-mon[29291]: 6.0 scrub ok 2024-09-10T18:44:00.422 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:00 smithi028 ceph-mon[39220]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-10T18:44:00.422 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:00 smithi028 ceph-mon[39220]: Cluster is now healthy 2024-09-10T18:44:00.422 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:00 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2975585665' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:44:00.422 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:00 smithi028 ceph-mon[39220]: osdmap e427: 8 total, 8 up, 7 in 2024-09-10T18:44:00.422 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:00 smithi028 ceph-mon[39220]: pgmap v1010: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:44:00.422 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:00 smithi028 ceph-mon[39220]: 6.0 scrub starts 2024-09-10T18:44:00.422 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:00 smithi028 ceph-mon[39220]: 6.0 scrub ok 2024-09-10T18:44:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:01 smithi204 ceph-mon[32357]: 4.7 scrub starts 2024-09-10T18:44:01.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:01 smithi204 ceph-mon[32357]: 4.7 scrub ok 2024-09-10T18:44:01.461 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:44:01.461 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:44:01.461 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-10T18:44:01.461 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-10T18:44:01.461 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:44:01.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:01 smithi028 ceph-mon[29291]: 4.7 scrub starts 2024-09-10T18:44:01.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:01 smithi028 ceph-mon[29291]: 4.7 scrub ok 2024-09-10T18:44:01.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:01 smithi028 ceph-mon[39220]: 4.7 scrub starts 2024-09-10T18:44:01.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:01 smithi028 ceph-mon[39220]: 4.7 scrub ok 2024-09-10T18:44:01.489 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:44:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:44:01] "GET /metrics HTTP/1.1" 200 222505 "" "Prometheus/2.43.0" 2024-09-10T18:44:02.173 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-10T18:44:02.176 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-10T18:44:02.176 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-09-10T18:44:02.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd in 0 2024-09-10T18:44:02.418 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:44:02.431 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:02.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:02 smithi028 ceph-mon[29291]: pgmap v1011: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T18:44:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:02 smithi028 ceph-mon[39220]: pgmap v1011: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T18:44:02.538 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:02 smithi204 ceph-mon[32357]: pgmap v1011: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T18:44:02.539 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:44:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:44:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:44:03.124 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-10T18:44:03.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[29291]: 6.9 scrub starts 2024-09-10T18:44:03.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[29291]: 6.9 scrub ok 2024-09-10T18:44:03.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[29291]: 1.13 scrub starts 2024-09-10T18:44:03.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[29291]: 1.13 scrub ok 2024-09-10T18:44:03.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2540625040' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:44:03.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[29291]: 4.1b scrub starts 2024-09-10T18:44:03.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[29291]: 4.1b scrub ok 2024-09-10T18:44:03.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[29291]: 1.6 scrub starts 2024-09-10T18:44:03.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[29291]: 1.6 scrub ok 2024-09-10T18:44:03.137 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[39220]: 6.9 scrub starts 2024-09-10T18:44:03.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[39220]: 6.9 scrub ok 2024-09-10T18:44:03.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[39220]: 1.13 scrub starts 2024-09-10T18:44:03.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[39220]: 1.13 scrub ok 2024-09-10T18:44:03.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2540625040' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:44:03.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[39220]: 4.1b scrub starts 2024-09-10T18:44:03.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[39220]: 4.1b scrub ok 2024-09-10T18:44:03.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[39220]: 1.6 scrub starts 2024-09-10T18:44:03.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:03 smithi028 ceph-mon[39220]: 1.6 scrub ok 2024-09-10T18:44:03.322 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:03 smithi204 ceph-mon[32357]: 6.9 scrub starts 2024-09-10T18:44:03.323 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:03 smithi204 ceph-mon[32357]: 6.9 scrub ok 2024-09-10T18:44:03.323 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:03 smithi204 ceph-mon[32357]: 1.13 scrub starts 2024-09-10T18:44:03.323 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:03 smithi204 ceph-mon[32357]: 1.13 scrub ok 2024-09-10T18:44:03.323 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:03 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2540625040' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:44:03.323 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:03 smithi204 ceph-mon[32357]: 4.1b scrub starts 2024-09-10T18:44:03.323 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:03 smithi204 ceph-mon[32357]: 4.1b scrub ok 2024-09-10T18:44:03.323 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:03 smithi204 ceph-mon[32357]: 1.6 scrub starts 2024-09-10T18:44:03.323 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:03 smithi204 ceph-mon[32357]: 1.6 scrub ok 2024-09-10T18:44:04.182 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:04 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2540625040' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:44:04.183 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:04 smithi204 ceph-mon[32357]: osdmap e428: 8 total, 8 up, 7 in 2024-09-10T18:44:04.183 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:04 smithi204 ceph-mon[32357]: pgmap v1013: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 831 B/s rd, 0 op/s 2024-09-10T18:44:04.183 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:04 smithi204 ceph-mon[32357]: 6.1f scrub starts 2024-09-10T18:44:04.183 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:04 smithi204 ceph-mon[32357]: 6.1f scrub ok 2024-09-10T18:44:04.183 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:04 smithi204 ceph-mon[32357]: 6.19 scrub starts 2024-09-10T18:44:04.183 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:04 smithi204 ceph-mon[32357]: 6.19 scrub ok 2024-09-10T18:44:04.184 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:44:04.185 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-10T18:44:04.185 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-10T18:44:04.185 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:44:04.185 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.185 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.34334.0:2429 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 94344803721280 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-10T18:44:04.185 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.462155+0000", 2024-09-10T18:44:04.185 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 303.718072431, 2024-09-10T18:44:04.185 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46503165600000002, 2024-09-10T18:44:04.185 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.185 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:44:04.185 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.185 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.34334", 2024-09-10T18:44:04.185 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/601695668", 2024-09-10T18:44:04.185 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2429 2024-09-10T18:44:04.186 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.186 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.186 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462155+0000", 2024-09-10T18:44:04.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.186 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.186 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462155+0000", 2024-09-10T18:44:04.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.186 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.187 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.187 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462152+0000", 2024-09-10T18:44:04.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:44:04.187 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.187 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.187 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.187 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462160+0000", 2024-09-10T18:44:04.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2139999999999996e-06 2024-09-10T18:44:04.187 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.187 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.187 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.187 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462177+0000", 2024-09-10T18:44:04.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6606e-05 2024-09-10T18:44:04.187 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.188 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.188 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.188 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.462184+0000", 2024-09-10T18:44:04.188 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0609999999999998e-06 2024-09-10T18:44:04.188 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.188 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.188 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.188 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.486107+0000", 2024-09-10T18:44:04.188 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023923172999999999 2024-09-10T18:44:04.188 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.486112+0000", 2024-09-10T18:44:04.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5040000000000002e-06 2024-09-10T18:44:04.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.913266+0000", 2024-09-10T18:44:04.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42715417900000002 2024-09-10T18:44:04.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for missing object", 2024-09-10T18:44:04.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.913531+0000", 2024-09-10T18:44:04.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00026490999999999998 2024-09-10T18:44:04.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.924754+0000", 2024-09-10T18:44:04.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011222744999999999 2024-09-10T18:44:04.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-10T18:44:04.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.924844+0000", 2024-09-10T18:44:04.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9617000000000006e-05 2024-09-10T18:44:04.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.925702+0000", 2024-09-10T18:44:04.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00085778299999999998 2024-09-10T18:44:04.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.925767+0000", 2024-09-10T18:44:04.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5029000000000004e-05 2024-09-10T18:44:04.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.927186+0000", 2024-09-10T18:44:04.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014196530000000001 2024-09-10T18:44:04.191 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.191 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.192 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.192 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.192 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224013 10.9 10:990f827e:::benchmark_data_smithi028_161947_object87089:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:44:04.192 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.893001+0000", 2024-09-10T18:44:04.192 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 303.28722617300002, 2024-09-10T18:44:04.192 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.010061828, 2024-09-10T18:44:04.192 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.192 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:44:04.192 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.192 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:04.192 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:04.192 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224013 2024-09-10T18:44:04.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.192 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.193 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.193 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.193 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.893001+0000", 2024-09-10T18:44:04.193 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.193 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.193 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.193 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.193 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.893001+0000", 2024-09-10T18:44:04.193 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.193 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.193 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.193 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.193 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.892999+0000", 2024-09-10T18:44:04.193 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:44:04.194 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.194 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.194 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.194 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.893005+0000", 2024-09-10T18:44:04.194 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8290000000000002e-06 2024-09-10T18:44:04.194 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.194 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.194 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.194 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.893015+0000", 2024-09-10T18:44:04.194 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4380000000000001e-06 2024-09-10T18:44:04.194 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.194 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.194 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.194 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.893022+0000", 2024-09-10T18:44:04.195 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4630000000000004e-06 2024-09-10T18:44:04.195 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.195 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.195 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.195 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.898346+0000", 2024-09-10T18:44:04.195 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0053237240000000002 2024-09-10T18:44:04.195 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.195 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.195 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.195 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.898354+0000", 2024-09-10T18:44:04.195 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0090000000000007e-06 2024-09-10T18:44:04.195 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.195 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.195 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.196 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.902998+0000", 2024-09-10T18:44:04.196 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0046443110000001 2024-09-10T18:44:04.196 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.196 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.196 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.196 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.903063+0000", 2024-09-10T18:44:04.196 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4349000000000003e-05 2024-09-10T18:44:04.196 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.196 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.196 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.197 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.197 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.197 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224031 10.6 10:62b78eac:::benchmark_data_smithi028_161947_object87107:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:44:04.197 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.894085+0000", 2024-09-10T18:44:04.197 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 303.28614213100002, 2024-09-10T18:44:04.197 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.026344849, 2024-09-10T18:44:04.197 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.197 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:04.197 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.197 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:04.197 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:04.197 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224031 2024-09-10T18:44:04.197 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.197 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.198 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894085+0000", 2024-09-10T18:44:04.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.198 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894085+0000", 2024-09-10T18:44:04.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.198 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894084+0000", 2024-09-10T18:44:04.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:44:04.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.199 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.199 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894090+0000", 2024-09-10T18:44:04.199 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9719999999999997e-06 2024-09-10T18:44:04.199 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.199 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.199 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894100+0000", 2024-09-10T18:44:04.199 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9990000000000003e-06 2024-09-10T18:44:04.199 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.199 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.199 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894105+0000", 2024-09-10T18:44:04.200 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3539999999999999e-06 2024-09-10T18:44:04.200 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.200 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.200 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.200 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894214+0000", 2024-09-10T18:44:04.200 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010946399999999999 2024-09-10T18:44:04.200 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.200 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.200 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.200 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894224+0000", 2024-09-10T18:44:04.200 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.132e-06 2024-09-10T18:44:04.200 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.200 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.200 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.201 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.901993+0000", 2024-09-10T18:44:04.201 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0077688789999999 2024-09-10T18:44:04.201 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.201 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.201 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.201 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.902080+0000", 2024-09-10T18:44:04.201 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7083000000000006e-05 2024-09-10T18:44:04.201 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.201 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.201 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:44:04.201 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.902157+0000", 2024-09-10T18:44:04.201 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7247000000000001e-05 2024-09-10T18:44:04.201 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.201 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.202 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.202 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.908263+0000", 2024-09-10T18:44:04.202 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061065370000000004 2024-09-10T18:44:04.202 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.202 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.202 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.202 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.908293+0000", 2024-09-10T18:44:04.202 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9495000000000001e-05 2024-09-10T18:44:04.202 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.202 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.202 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:04.202 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920339+0000", 2024-09-10T18:44:04.202 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012046347000000001 2024-09-10T18:44:04.202 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.203 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.203 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:04.203 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920384+0000", 2024-09-10T18:44:04.203 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5250000000000002e-05 2024-09-10T18:44:04.203 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.203 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.203 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.203 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920430+0000", 2024-09-10T18:44:04.203 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5241000000000003e-05 2024-09-10T18:44:04.203 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.203 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.203 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.203 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.203 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.204 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224033 10.6 10:6b102ba2:::benchmark_data_smithi028_161947_object87109:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:44:04.204 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.894115+0000", 2024-09-10T18:44:04.204 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 303.28611242099998, 2024-09-10T18:44:04.204 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0264514709999999, 2024-09-10T18:44:04.204 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.204 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:04.204 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.204 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:04.204 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:04.204 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224033 2024-09-10T18:44:04.204 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.204 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.204 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.204 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.205 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894115+0000", 2024-09-10T18:44:04.205 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.205 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.205 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.205 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.205 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894115+0000", 2024-09-10T18:44:04.205 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.205 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.205 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.205 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.205 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894114+0000", 2024-09-10T18:44:04.205 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999995 2024-09-10T18:44:04.205 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.205 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.206 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.206 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894118+0000", 2024-09-10T18:44:04.206 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6430000000000001e-06 2024-09-10T18:44:04.206 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.206 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.206 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.206 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894123+0000", 2024-09-10T18:44:04.206 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5439999999999998e-06 2024-09-10T18:44:04.206 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.206 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.206 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.206 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894127+0000", 2024-09-10T18:44:04.206 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9160000000000003e-06 2024-09-10T18:44:04.207 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.207 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.207 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.207 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894241+0000", 2024-09-10T18:44:04.207 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114263 2024-09-10T18:44:04.207 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.207 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.207 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.207 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894246+0000", 2024-09-10T18:44:04.207 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7809999999999996e-06 2024-09-10T18:44:04.207 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.207 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.207 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.207 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.904932+0000", 2024-09-10T18:44:04.207 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0106856639999999 2024-09-10T18:44:04.208 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.208 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.208 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.208 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.905031+0000", 2024-09-10T18:44:04.208 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9630999999999995e-05 2024-09-10T18:44:04.208 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.208 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.208 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:44:04.208 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.905133+0000", 2024-09-10T18:44:04.208 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101679 2024-09-10T18:44:04.208 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.208 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.208 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.209 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.912359+0000", 2024-09-10T18:44:04.209 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0072258110000000004 2024-09-10T18:44:04.209 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.209 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.209 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.209 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.917542+0000", 2024-09-10T18:44:04.209 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0051832010000000001 2024-09-10T18:44:04.209 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.209 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.209 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:04.209 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920503+0000", 2024-09-10T18:44:04.209 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029605260000000002 2024-09-10T18:44:04.209 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.209 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.209 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:04.210 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920536+0000", 2024-09-10T18:44:04.210 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3519999999999998e-05 2024-09-10T18:44:04.210 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.210 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.210 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.210 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920566+0000", 2024-09-10T18:44:04.210 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9971000000000002e-05 2024-09-10T18:44:04.210 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.210 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.210 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.210 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.210 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.210 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224038 10.6 10:66bb442b:::benchmark_data_smithi028_161947_object87114:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:44:04.210 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.894131+0000", 2024-09-10T18:44:04.210 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 303.28609634999998, 2024-09-10T18:44:04.211 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.026563632, 2024-09-10T18:44:04.211 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.211 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:04.211 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.211 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:04.211 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:04.211 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224038 2024-09-10T18:44:04.211 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.211 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.211 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.211 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.211 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894131+0000", 2024-09-10T18:44:04.211 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.211 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.212 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.212 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.212 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894131+0000", 2024-09-10T18:44:04.212 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.212 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.212 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.212 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.212 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894130+0000", 2024-09-10T18:44:04.212 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999995 2024-09-10T18:44:04.212 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.212 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.212 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.212 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894133+0000", 2024-09-10T18:44:04.212 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9119999999999998e-06 2024-09-10T18:44:04.213 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.213 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.213 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.213 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894138+0000", 2024-09-10T18:44:04.213 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6940000000000003e-06 2024-09-10T18:44:04.213 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.213 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.213 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.213 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894142+0000", 2024-09-10T18:44:04.213 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7680000000000002e-06 2024-09-10T18:44:04.213 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.213 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.213 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.214 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894266+0000", 2024-09-10T18:44:04.214 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012440899999999999 2024-09-10T18:44:04.214 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.214 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.214 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.214 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894270+0000", 2024-09-10T18:44:04.214 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5349999999999998e-06 2024-09-10T18:44:04.214 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.214 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.214 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.214 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.908313+0000", 2024-09-10T18:44:04.214 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0140428459999999 2024-09-10T18:44:04.214 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.214 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.215 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.215 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.908456+0000", 2024-09-10T18:44:04.215 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014293500000000001 2024-09-10T18:44:04.215 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.215 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.215 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:44:04.215 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.908562+0000", 2024-09-10T18:44:04.215 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105519 2024-09-10T18:44:04.215 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.215 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.215 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.215 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920333+0000", 2024-09-10T18:44:04.215 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011770994999999999 2024-09-10T18:44:04.216 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.216 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.216 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:04.216 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920571+0000", 2024-09-10T18:44:04.216 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023851100000000001 2024-09-10T18:44:04.216 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.216 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.216 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.216 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920617+0000", 2024-09-10T18:44:04.216 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6020999999999999e-05 2024-09-10T18:44:04.216 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.216 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.216 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:04.217 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920660+0000", 2024-09-10T18:44:04.217 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2774000000000001e-05 2024-09-10T18:44:04.217 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.217 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.217 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.217 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920694+0000", 2024-09-10T18:44:04.217 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4220000000000001e-05 2024-09-10T18:44:04.217 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.217 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.217 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.217 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.217 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.217 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224045 10.9 10:92123ef6:::benchmark_data_smithi028_161947_object87121:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:44:04.217 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.894145+0000", 2024-09-10T18:44:04.218 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 303.28608190400001, 2024-09-10T18:44:04.218 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0239920119999999, 2024-09-10T18:44:04.218 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.218 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:04.218 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.218 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:04.218 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:04.218 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224045 2024-09-10T18:44:04.218 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.218 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.218 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.218 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.218 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894145+0000", 2024-09-10T18:44:04.218 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.218 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.219 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.219 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.219 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894145+0000", 2024-09-10T18:44:04.219 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.219 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.219 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.219 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.219 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894145+0000", 2024-09-10T18:44:04.219 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999995 2024-09-10T18:44:04.219 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.219 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.219 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.219 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894147+0000", 2024-09-10T18:44:04.219 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3479999999999998e-06 2024-09-10T18:44:04.220 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.220 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.220 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.220 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894152+0000", 2024-09-10T18:44:04.220 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8439999999999997e-06 2024-09-10T18:44:04.220 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.220 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.220 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.220 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894156+0000", 2024-09-10T18:44:04.220 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.172e-06 2024-09-10T18:44:04.220 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.220 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.220 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.220 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.898390+0000", 2024-09-10T18:44:04.221 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042343989999999998 2024-09-10T18:44:04.221 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.221 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.221 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.221 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.898395+0000", 2024-09-10T18:44:04.221 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7049999999999998e-06 2024-09-10T18:44:04.221 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.221 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.221 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.221 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.903090+0000", 2024-09-10T18:44:04.221 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.004694923 2024-09-10T18:44:04.221 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.221 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.222 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.222 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.903199+0000", 2024-09-10T18:44:04.222 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010871599999999999 2024-09-10T18:44:04.222 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.222 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.222 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,7", 2024-09-10T18:44:04.222 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.903284+0000", 2024-09-10T18:44:04.222 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5470000000000007e-05 2024-09-10T18:44:04.222 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.222 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.222 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.222 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.917991+0000", 2024-09-10T18:44:04.222 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014706933 2024-09-10T18:44:04.222 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.223 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.223 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.223 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918020+0000", 2024-09-10T18:44:04.223 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8734e-05 2024-09-10T18:44:04.223 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.223 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.223 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:04.223 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918026+0000", 2024-09-10T18:44:04.223 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9109999999999998e-06 2024-09-10T18:44:04.223 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.223 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.223 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:04.223 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918101+0000", 2024-09-10T18:44:04.223 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5488999999999995e-05 2024-09-10T18:44:04.224 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.224 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.224 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.224 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918137+0000", 2024-09-10T18:44:04.224 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5769000000000003e-05 2024-09-10T18:44:04.224 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.224 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.224 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.224 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.224 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.224 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224047 10.9 10:9e565572:::benchmark_data_smithi028_161947_object87123:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:44:04.224 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.894289+0000", 2024-09-10T18:44:04.224 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 303.28593763399999, 2024-09-10T18:44:04.225 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0239835960000001, 2024-09-10T18:44:04.225 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.225 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:04.225 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.225 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:04.225 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:04.225 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224047 2024-09-10T18:44:04.225 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.225 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.225 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.225 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.225 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894289+0000", 2024-09-10T18:44:04.225 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.226 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.226 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.226 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.226 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894289+0000", 2024-09-10T18:44:04.226 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.227 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.227 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.228 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.228 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894289+0000", 2024-09-10T18:44:04.228 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:44:04.228 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.228 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.228 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.228 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894293+0000", 2024-09-10T18:44:04.228 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0319999999999997e-06 2024-09-10T18:44:04.228 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.228 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.228 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.228 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894299+0000", 2024-09-10T18:44:04.228 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4409999999999998e-06 2024-09-10T18:44:04.229 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.229 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.229 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.229 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.894304+0000", 2024-09-10T18:44:04.229 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4769999999999997e-06 2024-09-10T18:44:04.229 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.229 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.229 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.229 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.898414+0000", 2024-09-10T18:44:04.229 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041101319999999998 2024-09-10T18:44:04.229 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.229 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.229 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.230 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.898418+0000", 2024-09-10T18:44:04.230 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5929999999999999e-06 2024-09-10T18:44:04.230 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.230 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.230 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.230 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.905573+0000", 2024-09-10T18:44:04.230 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0071550709999999 2024-09-10T18:44:04.230 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.230 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.230 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.230 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.905662+0000", 2024-09-10T18:44:04.230 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8850999999999993e-05 2024-09-10T18:44:04.230 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.230 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.231 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,7", 2024-09-10T18:44:04.231 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.905755+0000", 2024-09-10T18:44:04.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2317000000000004e-05 2024-09-10T18:44:04.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.231 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.231 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:04.231 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918143+0000", 2024-09-10T18:44:04.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012388892 2024-09-10T18:44:04.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.231 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.231 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.231 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918183+0000", 2024-09-10T18:44:04.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0017e-05 2024-09-10T18:44:04.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.231 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.232 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.232 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918207+0000", 2024-09-10T18:44:04.232 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3074e-05 2024-09-10T18:44:04.232 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.232 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.232 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:04.232 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918244+0000", 2024-09-10T18:44:04.232 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7657999999999998e-05 2024-09-10T18:44:04.232 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.232 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.232 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.232 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918273+0000", 2024-09-10T18:44:04.232 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8765999999999999e-05 2024-09-10T18:44:04.233 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.233 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.233 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.233 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.233 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.233 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224050 10.6 10:60145db6:::benchmark_data_smithi028_161947_object87126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:44:04.233 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.949075+0000", 2024-09-10T18:44:04.233 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 303.23115180000002, 2024-09-10T18:44:04.233 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97173985299999999, 2024-09-10T18:44:04.233 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.233 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:04.233 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.233 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:04.233 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:04.234 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224050 2024-09-10T18:44:04.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.234 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.949075+0000", 2024-09-10T18:44:04.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.949075+0000", 2024-09-10T18:44:04.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.949061+0000", 2024-09-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999862 2024-09-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.949081+0000", 2024-09-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9667999999999999e-05 2024-09-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.949098+0000", 2024-09-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6889999999999999e-05 2024-09-10T18:44:04.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.949104+0000", 2024-09-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1500000000000004e-06 2024-09-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.949143+0000", 2024-09-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8571000000000001e-05 2024-09-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.236 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.949150+0000", 2024-09-10T18:44:04.237 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0160000000000003e-06 2024-09-10T18:44:04.237 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.237 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.237 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.237 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.910472+0000", 2024-09-10T18:44:04.237 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.96132260899999999 2024-09-10T18:44:04.237 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.237 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.237 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.237 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.910572+0000", 2024-09-10T18:44:04.237 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9650999999999998e-05 2024-09-10T18:44:04.237 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.237 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.238 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:44:04.238 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.910678+0000", 2024-09-10T18:44:04.238 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010611 2024-09-10T18:44:04.238 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.238 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.238 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:04.238 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920576+0000", 2024-09-10T18:44:04.238 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098980720000000008 2024-09-10T18:44:04.238 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.238 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.238 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.238 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920726+0000", 2024-09-10T18:44:04.238 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000150147 2024-09-10T18:44:04.238 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.239 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920753+0000", 2024-09-10T18:44:04.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7117000000000001e-05 2024-09-10T18:44:04.239 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:04.239 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920786+0000", 2024-09-10T18:44:04.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2502999999999997e-05 2024-09-10T18:44:04.239 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.239 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920815+0000", 2024-09-10T18:44:04.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9295999999999999e-05 2024-09-10T18:44:04.239 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.240 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.240 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.240 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.240 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.240 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224058 10.6 10:6ba08e13:::benchmark_data_smithi028_161947_object87134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:44:04.240 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.962810+0000", 2024-09-10T18:44:04.240 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 303.21741693500002, 2024-09-10T18:44:04.240 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95812768699999995, 2024-09-10T18:44:04.240 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.240 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:04.240 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.240 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:04.240 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:04.241 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224058 2024-09-10T18:44:04.241 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.241 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.241 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.241 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.241 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.962810+0000", 2024-09-10T18:44:04.242 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.242 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.242 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.242 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.242 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.962810+0000", 2024-09-10T18:44:04.242 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.242 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.242 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.242 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.243 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.962807+0000", 2024-09-10T18:44:04.243 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:44:04.243 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.243 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.243 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.243 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.962816+0000", 2024-09-10T18:44:04.243 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2430000000000005e-06 2024-09-10T18:44:04.243 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.243 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.244 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.244 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.962838+0000", 2024-09-10T18:44:04.244 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2524e-05 2024-09-10T18:44:04.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.244 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.244 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.244 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.962845+0000", 2024-09-10T18:44:04.244 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.187e-06 2024-09-10T18:44:04.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.244 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.962899+0000", 2024-09-10T18:44:04.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3854999999999999e-05 2024-09-10T18:44:04.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.962913+0000", 2024-09-10T18:44:04.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3668e-05 2024-09-10T18:44:04.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.912423+0000", 2024-09-10T18:44:04.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.94950994899999996 2024-09-10T18:44:04.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.246 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.246 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.246 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.912521+0000", 2024-09-10T18:44:04.246 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7816e-05 2024-09-10T18:44:04.246 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.246 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.246 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:44:04.246 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.912635+0000", 2024-09-10T18:44:04.246 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114059 2024-09-10T18:44:04.246 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.246 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.246 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:04.246 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920581+0000", 2024-09-10T18:44:04.246 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0079467129999999993 2024-09-10T18:44:04.247 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.247 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.247 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.247 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920844+0000", 2024-09-10T18:44:04.247 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00026251600000000002 2024-09-10T18:44:04.247 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.247 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.247 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.247 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920870+0000", 2024-09-10T18:44:04.247 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6101999999999998e-05 2024-09-10T18:44:04.247 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.247 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.247 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:04.247 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920908+0000", 2024-09-10T18:44:04.248 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7902999999999999e-05 2024-09-10T18:44:04.248 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.248 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.248 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.248 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920938+0000", 2024-09-10T18:44:04.248 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9807000000000001e-05 2024-09-10T18:44:04.248 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.248 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.248 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.248 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.248 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.248 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224061 10.9 10:9c470190:::benchmark_data_smithi028_161947_object87137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:44:04.248 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.963690+0000", 2024-09-10T18:44:04.248 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 303.21653684199998, 2024-09-10T18:44:04.249 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95472792100000003, 2024-09-10T18:44:04.249 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.249 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:04.249 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.249 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:04.249 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:04.249 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224061 2024-09-10T18:44:04.249 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.249 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.249 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.249 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.249 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.963690+0000", 2024-09-10T18:44:04.249 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.249 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.250 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.250 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.963690+0000", 2024-09-10T18:44:04.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.250 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.250 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.963689+0000", 2024-09-10T18:44:04.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:44:04.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.250 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.250 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.963695+0000", 2024-09-10T18:44:04.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8100000000000003e-06 2024-09-10T18:44:04.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.251 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.251 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.251 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.963706+0000", 2024-09-10T18:44:04.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0927999999999999e-05 2024-09-10T18:44:04.251 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.251 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.251 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.251 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.963711+0000", 2024-09-10T18:44:04.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9219999999999996e-06 2024-09-10T18:44:04.251 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.251 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.251 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.251 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.963757+0000", 2024-09-10T18:44:04.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6007000000000002e-05 2024-09-10T18:44:04.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.252 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.963765+0000", 2024-09-10T18:44:04.252 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3939999999999996e-06 2024-09-10T18:44:04.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.252 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.910047+0000", 2024-09-10T18:44:04.252 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.94628242299999998 2024-09-10T18:44:04.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.910156+0000", 2024-09-10T18:44:04.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010876 2024-09-10T18:44:04.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.253 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,7", 2024-09-10T18:44:04.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.910244+0000", 2024-09-10T18:44:04.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7503000000000006e-05 2024-09-10T18:44:04.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.253 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:04.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918148+0000", 2024-09-10T18:44:04.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0079044839999999998 2024-09-10T18:44:04.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.254 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.254 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918303+0000", 2024-09-10T18:44:04.254 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015505599999999999 2024-09-10T18:44:04.254 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.254 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.254 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.254 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918325+0000", 2024-09-10T18:44:04.254 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1437000000000002e-05 2024-09-10T18:44:04.254 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.254 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.254 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:04.254 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918387+0000", 2024-09-10T18:44:04.255 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2385999999999995e-05 2024-09-10T18:44:04.255 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.255 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.255 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.255 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918418+0000", 2024-09-10T18:44:04.255 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1032999999999999e-05 2024-09-10T18:44:04.256 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.256 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.256 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.256 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.256 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224063 10.9 10:985fa581:::benchmark_data_smithi028_161947_object87139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:44:04.256 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.964621+0000", 2024-09-10T18:44:04.256 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 303.215606428, 2024-09-10T18:44:04.256 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95390776799999999, 2024-09-10T18:44:04.256 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.256 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:04.256 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.256 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:04.256 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:04.257 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224063 2024-09-10T18:44:04.257 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.257 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.257 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.257 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.257 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.964621+0000", 2024-09-10T18:44:04.257 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.257 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.257 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.257 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.257 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.964621+0000", 2024-09-10T18:44:04.258 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.258 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.258 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.258 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.258 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.964619+0000", 2024-09-10T18:44:04.258 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:44:04.258 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.258 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.258 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.258 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.964628+0000", 2024-09-10T18:44:04.258 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0920000000000004e-06 2024-09-10T18:44:04.258 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.258 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.258 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.259 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.964640+0000", 2024-09-10T18:44:04.259 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2111e-05 2024-09-10T18:44:04.259 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.259 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.259 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.259 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.964646+0000", 2024-09-10T18:44:04.259 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6910000000000002e-06 2024-09-10T18:44:04.259 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.260 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.260 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.260 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.964684+0000", 2024-09-10T18:44:04.260 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7898999999999997e-05 2024-09-10T18:44:04.260 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.260 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.260 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.260 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.964692+0000", 2024-09-10T18:44:04.260 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6760000000000007e-06 2024-09-10T18:44:04.260 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.260 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.260 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.912470+0000", 2024-09-10T18:44:04.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.94777832299999998 2024-09-10T18:44:04.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.912560+0000", 2024-09-10T18:44:04.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0186999999999996e-05 2024-09-10T18:44:04.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,7", 2024-09-10T18:44:04.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.912654+0000", 2024-09-10T18:44:04.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3729999999999999e-05 2024-09-10T18:44:04.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:04.262 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918152+0000", 2024-09-10T18:44:04.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054985470000000003 2024-09-10T18:44:04.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.262 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918445+0000", 2024-09-10T18:44:04.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029213299999999998 2024-09-10T18:44:04.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.263 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918466+0000", 2024-09-10T18:44:04.263 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1158e-05 2024-09-10T18:44:04.263 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.263 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.263 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:04.263 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918501+0000", 2024-09-10T18:44:04.263 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4919999999999998e-05 2024-09-10T18:44:04.263 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.263 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.263 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.263 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918528+0000", 2024-09-10T18:44:04.263 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7661000000000002e-05 2024-09-10T18:44:04.263 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.264 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.264 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.264 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224069 10.9 10:99c920d9:::benchmark_data_smithi028_161947_object87145:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:44:04.264 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.970604+0000", 2024-09-10T18:44:04.264 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 303.20962273800001, 2024-09-10T18:44:04.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.954060515, 2024-09-10T18:44:04.264 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.264 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:04.264 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.264 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:04.264 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:04.264 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224069 2024-09-10T18:44:04.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.265 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.265 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.970604+0000", 2024-09-10T18:44:04.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.265 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.970604+0000", 2024-09-10T18:44:04.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.266 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.266 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.266 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.970602+0000", 2024-09-10T18:44:04.266 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:44:04.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.266 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.266 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.266 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.970611+0000", 2024-09-10T18:44:04.266 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2450000000000004e-06 2024-09-10T18:44:04.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.266 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.266 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.266 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.970629+0000", 2024-09-10T18:44:04.266 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7819e-05 2024-09-10T18:44:04.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.267 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.267 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.970636+0000", 2024-09-10T18:44:04.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7329999999999997e-06 2024-09-10T18:44:04.267 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.267 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.267 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.970685+0000", 2024-09-10T18:44:04.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9771000000000002e-05 2024-09-10T18:44:04.267 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.267 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.267 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.970692+0000", 2024-09-10T18:44:04.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1559999999999998e-06 2024-09-10T18:44:04.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.914825+0000", 2024-09-10T18:44:04.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.94413236199999995 2024-09-10T18:44:04.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.914929+0000", 2024-09-10T18:44:04.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0001041 2024-09-10T18:44:04.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,7", 2024-09-10T18:44:04.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.915027+0000", 2024-09-10T18:44:04.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8393000000000005e-05 2024-09-10T18:44:04.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:04.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.918157+0000", 2024-09-10T18:44:04.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0031293979999999998 2024-09-10T18:44:04.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.922005+0000", 2024-09-10T18:44:04.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038485170000000001 2024-09-10T18:44:04.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.270 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.270 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.924547+0000", 2024-09-10T18:44:04.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002541387 2024-09-10T18:44:04.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.270 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:04.270 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.924628+0000", 2024-09-10T18:44:04.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1588000000000006e-05 2024-09-10T18:44:04.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.271 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.924665+0000", 2024-09-10T18:44:04.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6541999999999997e-05 2024-09-10T18:44:04.271 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.271 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.271 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.271 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.271 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224070 10.6 10:6d42de77:::benchmark_data_smithi028_161947_object87146:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:44:04.271 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.971750+0000", 2024-09-10T18:44:04.271 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 303.208476873, 2024-09-10T18:44:04.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.94993114000000001, 2024-09-10T18:44:04.271 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.271 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:04.272 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.272 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:04.272 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:04.272 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224070 2024-09-10T18:44:04.272 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.272 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.272 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.272 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.272 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.971750+0000", 2024-09-10T18:44:04.272 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.971750+0000", 2024-09-10T18:44:04.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.971749+0000", 2024-09-10T18:44:04.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:44:04.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.971758+0000", 2024-09-10T18:44:04.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6969999999999999e-06 2024-09-10T18:44:04.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.274 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.971768+0000", 2024-09-10T18:44:04.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0030000000000001e-05 2024-09-10T18:44:04.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.274 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.971772+0000", 2024-09-10T18:44:04.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3229999999999999e-06 2024-09-10T18:44:04.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.971809+0000", 2024-09-10T18:44:04.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7237999999999998e-05 2024-09-10T18:44:04.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.971815+0000", 2024-09-10T18:44:04.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4299999999999997e-06 2024-09-10T18:44:04.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.914826+0000", 2024-09-10T18:44:04.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.943011759 2024-09-10T18:44:04.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.914923+0000", 2024-09-10T18:44:04.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6811000000000005e-05 2024-09-10T18:44:04.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:44:04.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.915020+0000", 2024-09-10T18:44:04.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6992999999999998e-05 2024-09-10T18:44:04.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:04.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920586+0000", 2024-09-10T18:44:04.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0055662569999999998 2024-09-10T18:44:04.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.920971+0000", 2024-09-10T18:44:04.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00038463800000000003 2024-09-10T18:44:04.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.921576+0000", 2024-09-10T18:44:04.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00060466700000000003 2024-09-10T18:44:04.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.278 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:04.278 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.921645+0000", 2024-09-10T18:44:04.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9391e-05 2024-09-10T18:44:04.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.278 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.278 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.921681+0000", 2024-09-10T18:44:04.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6183000000000002e-05 2024-09-10T18:44:04.278 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.278 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.278 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.279 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224074 10.6 10:69c7ac47:::benchmark_data_smithi028_161947_object87150:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-10T18:44:04.279 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.980073+0000", 2024-09-10T18:44:04.279 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 303.20015421300002, 2024-09-10T18:44:04.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.944869819, 2024-09-10T18:44:04.279 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.279 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:04.279 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.279 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:04.279 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:04.279 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224074 2024-09-10T18:44:04.279 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.279 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.980073+0000", 2024-09-10T18:44:04.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.980073+0000", 2024-09-10T18:44:04.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.980071+0000", 2024-09-10T18:44:04.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:44:04.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.980077+0000", 2024-09-10T18:44:04.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8939999999999998e-06 2024-09-10T18:44:04.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.980087+0000", 2024-09-10T18:44:04.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0057e-05 2024-09-10T18:44:04.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.980091+0000", 2024-09-10T18:44:04.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0260000000000001e-06 2024-09-10T18:44:04.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.980132+0000", 2024-09-10T18:44:04.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1056999999999997e-05 2024-09-10T18:44:04.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.980140+0000", 2024-09-10T18:44:04.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.328e-06 2024-09-10T18:44:04.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.917565+0000", 2024-09-10T18:44:04.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.937425709 2024-09-10T18:44:04.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.917675+0000", 2024-09-10T18:44:04.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000110024 2024-09-10T18:44:04.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:44:04.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.917937+0000", 2024-09-10T18:44:04.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00026186100000000001 2024-09-10T18:44:04.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.921847+0000", 2024-09-10T18:44:04.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039093970000000002 2024-09-10T18:44:04.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:04.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.924796+0000", 2024-09-10T18:44:04.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029495699999999999 2024-09-10T18:44:04.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.924827+0000", 2024-09-10T18:44:04.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1102000000000001e-05 2024-09-10T18:44:04.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:04.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.924905+0000", 2024-09-10T18:44:04.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7860000000000003e-05 2024-09-10T18:44:04.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:01.924943+0000", 2024-09-10T18:44:04.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7391000000000002e-05 2024-09-10T18:44:04.286 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.286 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.286 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.286 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.34334.0:2471 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 94344803721280 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:44:04.286 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.218473+0000", 2024-09-10T18:44:04.286 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 279.96175400499999, 2024-09-10T18:44:04.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.047128775, 2024-09-10T18:44:04.286 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.286 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:44:04.286 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.287 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.34334", 2024-09-10T18:44:04.287 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/601695668", 2024-09-10T18:44:04.287 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2471 2024-09-10T18:44:04.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.287 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.218473+0000", 2024-09-10T18:44:04.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.218473+0000", 2024-09-10T18:44:04.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.218472+0000", 2024-09-10T18:44:04.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:44:04.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.218477+0000", 2024-09-10T18:44:04.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1499999999999998e-06 2024-09-10T18:44:04.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.218485+0000", 2024-09-10T18:44:04.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4589999999999998e-06 2024-09-10T18:44:04.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.218491+0000", 2024-09-10T18:44:04.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8239999999999996e-06 2024-09-10T18:44:04.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.218545+0000", 2024-09-10T18:44:04.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4008000000000002e-05 2024-09-10T18:44:04.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-10T18:44:04.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.218550+0000", 2024-09-10T18:44:04.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1340000000000003e-06 2024-09-10T18:44:04.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.233529+0000", 2024-09-10T18:44:04.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014979019999999999 2024-09-10T18:44:04.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.233539+0000", 2024-09-10T18:44:04.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3360000000000001e-06 2024-09-10T18:44:04.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.256425+0000", 2024-09-10T18:44:04.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0228867580000001 2024-09-10T18:44:04.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for degraded object", 2024-09-10T18:44:04.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.256708+0000", 2024-09-10T18:44:04.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00028321200000000003 2024-09-10T18:44:04.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.258856+0000", 2024-09-10T18:44:04.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021476680000000001 2024-09-10T18:44:04.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.258949+0000", 2024-09-10T18:44:04.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3283999999999997e-05 2024-09-10T18:44:04.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.265602+0000", 2024-09-10T18:44:04.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0066523099999999998 2024-09-10T18:44:04.295 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.295 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.295 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.295 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253541 10.6 10:6c835e9f:::benchmark_data_smithi028_161947_object116617:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:44:04.295 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.242223+0000", 2024-09-10T18:44:04.295 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 279.93800433600001, 2024-09-10T18:44:04.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2265690720000002, 2024-09-10T18:44:04.295 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.295 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:04.295 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.296 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:04.296 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:04.296 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253541 2024-09-10T18:44:04.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.296 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242223+0000", 2024-09-10T18:44:04.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242223+0000", 2024-09-10T18:44:04.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242220+0000", 2024-09-10T18:44:04.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:44:04.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242228+0000", 2024-09-10T18:44:04.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6229999999999997e-06 2024-09-10T18:44:04.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242243+0000", 2024-09-10T18:44:04.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5296000000000001e-05 2024-09-10T18:44:04.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242250+0000", 2024-09-10T18:44:04.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5169999999999996e-06 2024-09-10T18:44:04.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242299+0000", 2024-09-10T18:44:04.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8884999999999997e-05 2024-09-10T18:44:04.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242307+0000", 2024-09-10T18:44:04.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0560000000000005e-06 2024-09-10T18:44:04.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.227655+0000", 2024-09-10T18:44:04.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98534856000000004 2024-09-10T18:44:04.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.227667+0000", 2024-09-10T18:44:04.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1787000000000001e-05 2024-09-10T18:44:04.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.405394+0000", 2024-09-10T18:44:04.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1777274360000001 2024-09-10T18:44:04.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.405530+0000", 2024-09-10T18:44:04.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013596300000000001 2024-09-10T18:44:04.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1,4", 2024-09-10T18:44:04.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.406018+0000", 2024-09-10T18:44:04.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000487601 2024-09-10T18:44:04.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:04.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.423091+0000", 2024-09-10T18:44:04.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017072539000000001 2024-09-10T18:44:04.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.429936+0000", 2024-09-10T18:44:04.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068454869999999999 2024-09-10T18:44:04.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468505+0000", 2024-09-10T18:44:04.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.038568526999999998 2024-09-10T18:44:04.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468621+0000", 2024-09-10T18:44:04.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011692999999999999 2024-09-10T18:44:04.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:04.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468674+0000", 2024-09-10T18:44:04.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2874000000000002e-05 2024-09-10T18:44:04.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468792+0000", 2024-09-10T18:44:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000117383 2024-09-10T18:44:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253543 10.6 10:606298d4:::benchmark_data_smithi028_161947_object116619:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:44:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.243923+0000", 2024-09-10T18:44:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 279.93630421199998, 2024-09-10T18:44:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.225144625, 2024-09-10T18:44:04.304 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253543 2024-09-10T18:44:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.243923+0000", 2024-09-10T18:44:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.243923+0000", 2024-09-10T18:44:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.243921+0000", 2024-09-10T18:44:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:44:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.243928+0000", 2024-09-10T18:44:04.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.922e-06 2024-09-10T18:44:04.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.243942+0000", 2024-09-10T18:44:04.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4494e-05 2024-09-10T18:44:04.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.243948+0000", 2024-09-10T18:44:04.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6509999999999998e-06 2024-09-10T18:44:04.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.243988+0000", 2024-09-10T18:44:04.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.002e-05 2024-09-10T18:44:04.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.243996+0000", 2024-09-10T18:44:04.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5730000000000002e-06 2024-09-10T18:44:04.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.227691+0000", 2024-09-10T18:44:04.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98369558000000001 2024-09-10T18:44:04.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.227698+0000", 2024-09-10T18:44:04.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6640000000000001e-06 2024-09-10T18:44:04.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.413292+0000", 2024-09-10T18:44:04.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1855940709999999 2024-09-10T18:44:04.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.413442+0000", 2024-09-10T18:44:04.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000149734 2024-09-10T18:44:04.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1,4", 2024-09-10T18:44:04.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.413594+0000", 2024-09-10T18:44:04.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015274900000000001 2024-09-10T18:44:04.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:04.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.423097+0000", 2024-09-10T18:44:04.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095028400000000002 2024-09-10T18:44:04.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.429968+0000", 2024-09-10T18:44:04.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068706779999999999 2024-09-10T18:44:04.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468564+0000", 2024-09-10T18:44:04.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.038596485999999999 2024-09-10T18:44:04.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468913+0000", 2024-09-10T18:44:04.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000348162 2024-09-10T18:44:04.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:04.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468958+0000", 2024-09-10T18:44:04.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5558e-05 2024-09-10T18:44:04.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.469067+0000", 2024-09-10T18:44:04.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000109269 2024-09-10T18:44:04.312 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.312 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.312 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.313 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253556 10.6 10:699fc70f:::benchmark_data_smithi028_161947_object116632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:44:04.313 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.273799+0000", 2024-09-10T18:44:04.313 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 279.90642753700001, 2024-09-10T18:44:04.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.195505587, 2024-09-10T18:44:04.313 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.313 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:04.313 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.313 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:04.314 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:04.314 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253556 2024-09-10T18:44:04.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.314 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.273799+0000", 2024-09-10T18:44:04.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.273799+0000", 2024-09-10T18:44:04.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.273798+0000", 2024-09-10T18:44:04.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:44:04.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.273804+0000", 2024-09-10T18:44:04.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.784e-06 2024-09-10T18:44:04.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.273813+0000", 2024-09-10T18:44:04.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8349999999999993e-06 2024-09-10T18:44:04.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.273818+0000", 2024-09-10T18:44:04.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1399999999999999e-06 2024-09-10T18:44:04.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.273852+0000", 2024-09-10T18:44:04.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3973999999999997e-05 2024-09-10T18:44:04.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.273857+0000", 2024-09-10T18:44:04.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5659999999999997e-06 2024-09-10T18:44:04.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.227726+0000", 2024-09-10T18:44:04.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.95386886699999995 2024-09-10T18:44:04.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:04.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.227734+0000", 2024-09-10T18:44:04.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2579999999999995e-06 2024-09-10T18:44:04.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.418473+0000", 2024-09-10T18:44:04.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.190738742 2024-09-10T18:44:04.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.418607+0000", 2024-09-10T18:44:04.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013377699999999999 2024-09-10T18:44:04.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1,4", 2024-09-10T18:44:04.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.418762+0000", 2024-09-10T18:44:04.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015565800000000001 2024-09-10T18:44:04.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:04.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.423103+0000", 2024-09-10T18:44:04.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0043402429999999997 2024-09-10T18:44:04.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.430008+0000", 2024-09-10T18:44:04.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0069057229999999999 2024-09-10T18:44:04.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.468852+0000", 2024-09-10T18:44:04.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.038843436000000002 2024-09-10T18:44:04.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.469138+0000", 2024-09-10T18:44:04.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00028564399999999999 2024-09-10T18:44:04.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:04.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.469185+0000", 2024-09-10T18:44:04.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7531999999999998e-05 2024-09-10T18:44:04.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.469305+0000", 2024-09-10T18:44:04.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011996499999999999 2024-09-10T18:44:04.322 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.322 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.322 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.322 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:92941 11.7 11:e141045f:::benchmark_data_smithi028_186962_object92940:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-09-10T18:44:04.322 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:11.602505+0000", 2024-09-10T18:44:04.322 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 172.57772177300001, 2024-09-10T18:44:04.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.297579649, 2024-09-10T18:44:04.323 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.323 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:04.323 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.323 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:04.323 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:04.323 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 92941 2024-09-10T18:44:04.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.323 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.602505+0000", 2024-09-10T18:44:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.602505+0000", 2024-09-10T18:44:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.602503+0000", 2024-09-10T18:44:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:44:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.602536+0000", 2024-09-10T18:44:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.239e-05 2024-09-10T18:44:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.602549+0000", 2024-09-10T18:44:04.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2856e-05 2024-09-10T18:44:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.602555+0000", 2024-09-10T18:44:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0440000000000001e-06 2024-09-10T18:44:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.610031+0000", 2024-09-10T18:44:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0074761840000000003 2024-09-10T18:44:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.610146+0000", 2024-09-10T18:44:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114777 2024-09-10T18:44:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:44:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.610280+0000", 2024-09-10T18:44:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000134115 2024-09-10T18:44:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.615651+0000", 2024-09-10T18:44:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0053712300000000003 2024-09-10T18:44:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.615663+0000", 2024-09-10T18:44:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2081e-05 2024-09-10T18:44:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.899993+0000", 2024-09-10T18:44:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.28432955599999998 2024-09-10T18:44:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.900043+0000", 2024-09-10T18:44:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0478e-05 2024-09-10T18:44:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.900085+0000", 2024-09-10T18:44:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1693000000000002e-05 2024-09-10T18:44:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:92950 11.7 11:eafcd1f0:::benchmark_data_smithi028_186962_object92949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-09-10T18:44:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:11.607706+0000", 2024-09-10T18:44:04.329 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 172.572521365, 2024-09-10T18:44:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29540262499999997, 2024-09-10T18:44:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 92950 2024-09-10T18:44:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607706+0000", 2024-09-10T18:44:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607706+0000", 2024-09-10T18:44:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607703+0000", 2024-09-10T18:44:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:44:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607819+0000", 2024-09-10T18:44:04.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0001155 2024-09-10T18:44:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607835+0000", 2024-09-10T18:44:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6634999999999999e-05 2024-09-10T18:44:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.607842+0000", 2024-09-10T18:44:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2439999999999998e-06 2024-09-10T18:44:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.612949+0000", 2024-09-10T18:44:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0051076369999999999 2024-09-10T18:44:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.613068+0000", 2024-09-10T18:44:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000118671 2024-09-10T18:44:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:44:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.613210+0000", 2024-09-10T18:44:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000142344 2024-09-10T18:44:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.620943+0000", 2024-09-10T18:44:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0077323380000000001 2024-09-10T18:44:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.620949+0000", 2024-09-10T18:44:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7440000000000001e-06 2024-09-10T18:44:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.902989+0000", 2024-09-10T18:44:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.28203980400000001 2024-09-10T18:44:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.903071+0000", 2024-09-10T18:44:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1502000000000007e-05 2024-09-10T18:44:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:11.903108+0000", 2024-09-10T18:44:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7598999999999997e-05 2024-09-10T18:44:04.335 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.336 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.336 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.336 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:04.336 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:04.336 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:44:04.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:04 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2540625040' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:44:04.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:04 smithi028 ceph-mon[39220]: osdmap e428: 8 total, 8 up, 7 in 2024-09-10T18:44:04.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:04 smithi028 ceph-mon[39220]: pgmap v1013: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 831 B/s rd, 0 op/s 2024-09-10T18:44:04.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:04 smithi028 ceph-mon[39220]: 6.1f scrub starts 2024-09-10T18:44:04.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:04 smithi028 ceph-mon[39220]: 6.1f scrub ok 2024-09-10T18:44:04.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:04 smithi028 ceph-mon[39220]: 6.19 scrub starts 2024-09-10T18:44:04.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:04 smithi028 ceph-mon[39220]: 6.19 scrub ok 2024-09-10T18:44:04.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:04 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2540625040' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:44:04.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:04 smithi028 ceph-mon[29291]: osdmap e428: 8 total, 8 up, 7 in 2024-09-10T18:44:04.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:04 smithi028 ceph-mon[29291]: pgmap v1013: 162 pgs: 162 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 831 B/s rd, 0 op/s 2024-09-10T18:44:04.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:04 smithi028 ceph-mon[29291]: 6.1f scrub starts 2024-09-10T18:44:04.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:04 smithi028 ceph-mon[29291]: 6.1f scrub ok 2024-09-10T18:44:04.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:04 smithi028 ceph-mon[29291]: 6.19 scrub starts 2024-09-10T18:44:04.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:04 smithi028 ceph-mon[29291]: 6.19 scrub ok 2024-09-10T18:44:05.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-10T18:44:05.130 INFO:teuthology.orchestra.run.smithi028.stderr:marked in osd.0. 2024-09-10T18:44:05.208 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:05 smithi204 ceph-mon[32357]: 4.2 scrub starts 2024-09-10T18:44:05.209 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:05 smithi204 ceph-mon[32357]: 4.2 scrub ok 2024-09-10T18:44:05.209 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:05 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/144828731' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-09-10T18:44:05.291 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:44:05.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:05 smithi028 ceph-mon[39220]: 4.2 scrub starts 2024-09-10T18:44:05.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:05 smithi028 ceph-mon[39220]: 4.2 scrub ok 2024-09-10T18:44:05.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:05 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/144828731' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-09-10T18:44:05.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:05 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a[29287]: 2024-09-10T18:44:05.073+0000 7fed21057640 -1 mon.a@0(leader).osd e428 definitely_dead 0 2024-09-10T18:44:05.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:05 smithi028 ceph-mon[29291]: 4.2 scrub starts 2024-09-10T18:44:05.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:05 smithi028 ceph-mon[29291]: 4.2 scrub ok 2024-09-10T18:44:05.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:05 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/144828731' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-09-10T18:44:05.531 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:44:05 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:44:05.842 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-09-10T18:44:05.842 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-10T18:44:05.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pgp_num 71 2024-09-10T18:44:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:06.482 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:06 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/144828731' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-10T18:44:06.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:06 smithi204 ceph-mon[32357]: osdmap e429: 8 total, 8 up, 8 in 2024-09-10T18:44:06.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:06 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:06.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:06 smithi204 ceph-mon[32357]: pgmap v1015: 162 pgs: 10 peering, 152 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 834 B/s rd, 0 op/s 2024-09-10T18:44:06.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:06 smithi204 ceph-mon[32357]: 5.18 deep-scrub starts 2024-09-10T18:44:06.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:06 smithi204 ceph-mon[32357]: 5.18 deep-scrub ok 2024-09-10T18:44:06.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:06 smithi204 ceph-mon[32357]: 5.17 scrub starts 2024-09-10T18:44:06.483 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:06 smithi204 ceph-mon[32357]: 5.17 scrub ok 2024-09-10T18:44:06.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:06 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/144828731' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-10T18:44:06.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:06 smithi028 ceph-mon[29291]: osdmap e429: 8 total, 8 up, 8 in 2024-09-10T18:44:06.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:06 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:06.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:06 smithi028 ceph-mon[29291]: pgmap v1015: 162 pgs: 10 peering, 152 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 834 B/s rd, 0 op/s 2024-09-10T18:44:06.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:06 smithi028 ceph-mon[29291]: 5.18 deep-scrub starts 2024-09-10T18:44:06.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:06 smithi028 ceph-mon[29291]: 5.18 deep-scrub ok 2024-09-10T18:44:06.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:06 smithi028 ceph-mon[29291]: 5.17 scrub starts 2024-09-10T18:44:06.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:06 smithi028 ceph-mon[29291]: 5.17 scrub ok 2024-09-10T18:44:06.484 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:06 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/144828731' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-10T18:44:06.484 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:06 smithi028 ceph-mon[39220]: osdmap e429: 8 total, 8 up, 8 in 2024-09-10T18:44:06.485 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:06 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:06.485 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:06 smithi028 ceph-mon[39220]: pgmap v1015: 162 pgs: 10 peering, 152 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 834 B/s rd, 0 op/s 2024-09-10T18:44:06.485 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:06 smithi028 ceph-mon[39220]: 5.18 deep-scrub starts 2024-09-10T18:44:06.485 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:06 smithi028 ceph-mon[39220]: 5.18 deep-scrub ok 2024-09-10T18:44:06.485 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:06 smithi028 ceph-mon[39220]: 5.17 scrub starts 2024-09-10T18:44:06.485 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:06 smithi028 ceph-mon[39220]: 5.17 scrub ok 2024-09-10T18:44:07.004 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:44:07.004 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:44:07.004 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-10T18:44:07.004 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:44:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[39220]: osdmap e430: 8 total, 8 up, 8 in 2024-09-10T18:44:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[39220]: 5.d deep-scrub starts 2024-09-10T18:44:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[39220]: 5.d deep-scrub ok 2024-09-10T18:44:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[39220]: 3.9 scrub starts 2024-09-10T18:44:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[39220]: 3.9 scrub ok 2024-09-10T18:44:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[39220]: 6.1b scrub starts 2024-09-10T18:44:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[39220]: 6.1b scrub ok 2024-09-10T18:44:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[39220]: 1.7 scrub starts 2024-09-10T18:44:07.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[39220]: 1.7 scrub ok 2024-09-10T18:44:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[29291]: osdmap e430: 8 total, 8 up, 8 in 2024-09-10T18:44:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[29291]: 5.d deep-scrub starts 2024-09-10T18:44:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[29291]: 5.d deep-scrub ok 2024-09-10T18:44:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[29291]: 3.9 scrub starts 2024-09-10T18:44:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[29291]: 3.9 scrub ok 2024-09-10T18:44:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[29291]: 6.1b scrub starts 2024-09-10T18:44:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[29291]: 6.1b scrub ok 2024-09-10T18:44:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[29291]: 1.7 scrub starts 2024-09-10T18:44:07.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:07 smithi028 ceph-mon[29291]: 1.7 scrub ok 2024-09-10T18:44:07.599 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:07 smithi204 ceph-mon[32357]: osdmap e430: 8 total, 8 up, 8 in 2024-09-10T18:44:07.599 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:07 smithi204 ceph-mon[32357]: 5.d deep-scrub starts 2024-09-10T18:44:07.599 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:07 smithi204 ceph-mon[32357]: 5.d deep-scrub ok 2024-09-10T18:44:07.599 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:07 smithi204 ceph-mon[32357]: 3.9 scrub starts 2024-09-10T18:44:07.599 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:07 smithi204 ceph-mon[32357]: 3.9 scrub ok 2024-09-10T18:44:07.599 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:07 smithi204 ceph-mon[32357]: 6.1b scrub starts 2024-09-10T18:44:07.599 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:07 smithi204 ceph-mon[32357]: 6.1b scrub ok 2024-09-10T18:44:07.599 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:07 smithi204 ceph-mon[32357]: 1.7 scrub starts 2024-09-10T18:44:07.599 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:07 smithi204 ceph-mon[32357]: 1.7 scrub ok 2024-09-10T18:44:07.624 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-10T18:44:07.874 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:44:08.061 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: specified pgp_num 71 > pg_num 31 2024-09-10T18:44:08.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:08 smithi204 ceph-mon[32357]: 3.15 scrub starts 2024-09-10T18:44:08.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:08 smithi204 ceph-mon[32357]: 3.15 scrub ok 2024-09-10T18:44:08.335 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:08 smithi204 ceph-mon[32357]: 6.f scrub starts 2024-09-10T18:44:08.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:08 smithi204 ceph-mon[32357]: 6.f scrub ok 2024-09-10T18:44:08.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:08 smithi204 ceph-mon[32357]: pgmap v1017: 162 pgs: 33 peering, 129 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 91 B/s, 4 objects/s recovering 2024-09-10T18:44:08.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:08 smithi204 ceph-mon[32357]: 4.1 scrub starts 2024-09-10T18:44:08.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:08 smithi204 ceph-mon[32357]: 4.1 scrub ok 2024-09-10T18:44:08.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:08 smithi204 ceph-mon[32357]: 5.7 scrub starts 2024-09-10T18:44:08.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:08 smithi204 ceph-mon[32357]: 5.7 scrub ok 2024-09-10T18:44:08.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:08 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3255813178' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"}]: dispatch 2024-09-10T18:44:08.336 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:08 smithi204 ceph-mon[32357]: from='client.36116 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"}]: dispatch 2024-09-10T18:44:08.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[29291]: 3.15 scrub starts 2024-09-10T18:44:08.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[29291]: 3.15 scrub ok 2024-09-10T18:44:08.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[29291]: 6.f scrub starts 2024-09-10T18:44:08.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[29291]: 6.f scrub ok 2024-09-10T18:44:08.346 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[29291]: pgmap v1017: 162 pgs: 33 peering, 129 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 91 B/s, 4 objects/s recovering 2024-09-10T18:44:08.346 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[29291]: 4.1 scrub starts 2024-09-10T18:44:08.346 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[29291]: 4.1 scrub ok 2024-09-10T18:44:08.346 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[29291]: 5.7 scrub starts 2024-09-10T18:44:08.346 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[29291]: 5.7 scrub ok 2024-09-10T18:44:08.346 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3255813178' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"}]: dispatch 2024-09-10T18:44:08.346 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[29291]: from='client.36116 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"}]: dispatch 2024-09-10T18:44:08.346 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[39220]: 3.15 scrub starts 2024-09-10T18:44:08.346 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[39220]: 3.15 scrub ok 2024-09-10T18:44:08.346 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[39220]: 6.f scrub starts 2024-09-10T18:44:08.346 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[39220]: 6.f scrub ok 2024-09-10T18:44:08.346 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[39220]: pgmap v1017: 162 pgs: 33 peering, 129 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 91 B/s, 4 objects/s recovering 2024-09-10T18:44:08.347 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[39220]: 4.1 scrub starts 2024-09-10T18:44:08.347 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[39220]: 4.1 scrub ok 2024-09-10T18:44:08.347 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[39220]: 5.7 scrub starts 2024-09-10T18:44:08.347 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[39220]: 5.7 scrub ok 2024-09-10T18:44:08.347 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3255813178' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"}]: dispatch 2024-09-10T18:44:08.347 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:08 smithi028 ceph-mon[39220]: from='client.36116 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"}]: dispatch 2024-09-10T18:44:08.880 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-10T18:44:08.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-10T18:44:09.144 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:09.290 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:09 smithi204 ceph-mon[32357]: 3.11 scrub starts 2024-09-10T18:44:09.290 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:09 smithi204 ceph-mon[32357]: 3.11 scrub ok 2024-09-10T18:44:09.290 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:09 smithi204 ceph-mon[32357]: 4.5 scrub starts 2024-09-10T18:44:09.290 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:09 smithi204 ceph-mon[32357]: 4.5 scrub ok 2024-09-10T18:44:09.290 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:09 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:44:09.290 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:44:09.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:09 smithi028 ceph-mon[29291]: 3.11 scrub starts 2024-09-10T18:44:09.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:09 smithi028 ceph-mon[29291]: 3.11 scrub ok 2024-09-10T18:44:09.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:09 smithi028 ceph-mon[29291]: 4.5 scrub starts 2024-09-10T18:44:09.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:09 smithi028 ceph-mon[29291]: 4.5 scrub ok 2024-09-10T18:44:09.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:09 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:44:09.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:44:09.476 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:09 smithi028 ceph-mon[39220]: 3.11 scrub starts 2024-09-10T18:44:09.476 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:09 smithi028 ceph-mon[39220]: 3.11 scrub ok 2024-09-10T18:44:09.476 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:09 smithi028 ceph-mon[39220]: 4.5 scrub starts 2024-09-10T18:44:09.476 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:09 smithi028 ceph-mon[39220]: 4.5 scrub ok 2024-09-10T18:44:09.476 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:09 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:44:09.476 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:44:09.520 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:44:09.520 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:44:09.520 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-10T18:44:09.521 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-10T18:44:09.521 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:44:10.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-10T18:44:10.403 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:44:10.414 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:10 smithi028 ceph-mon[29291]: 1.2 scrub starts 2024-09-10T18:44:10.414 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:10 smithi028 ceph-mon[29291]: 1.2 scrub ok 2024-09-10T18:44:10.414 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:10 smithi028 ceph-mon[29291]: pgmap v1018: 162 pgs: 33 peering, 129 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 89 B/s, 4 objects/s recovering 2024-09-10T18:44:10.414 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:10 smithi028 ceph-mon[29291]: 3.14 scrub starts 2024-09-10T18:44:10.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:10 smithi028 ceph-mon[29291]: 3.14 scrub ok 2024-09-10T18:44:10.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:10 smithi028 ceph-mon[29291]: 4.6 deep-scrub starts 2024-09-10T18:44:10.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:10 smithi028 ceph-mon[29291]: 4.6 deep-scrub ok 2024-09-10T18:44:10.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:10 smithi028 ceph-mon[39220]: 1.2 scrub starts 2024-09-10T18:44:10.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:10 smithi028 ceph-mon[39220]: 1.2 scrub ok 2024-09-10T18:44:10.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:10 smithi028 ceph-mon[39220]: pgmap v1018: 162 pgs: 33 peering, 129 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 89 B/s, 4 objects/s recovering 2024-09-10T18:44:10.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:10 smithi028 ceph-mon[39220]: 3.14 scrub starts 2024-09-10T18:44:10.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:10 smithi028 ceph-mon[39220]: 3.14 scrub ok 2024-09-10T18:44:10.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:10 smithi028 ceph-mon[39220]: 4.6 deep-scrub starts 2024-09-10T18:44:10.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:10 smithi028 ceph-mon[39220]: 4.6 deep-scrub ok 2024-09-10T18:44:10.523 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:10 smithi204 ceph-mon[32357]: 1.2 scrub starts 2024-09-10T18:44:10.523 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:10 smithi204 ceph-mon[32357]: 1.2 scrub ok 2024-09-10T18:44:10.523 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:10 smithi204 ceph-mon[32357]: pgmap v1018: 162 pgs: 33 peering, 129 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 89 B/s, 4 objects/s recovering 2024-09-10T18:44:10.524 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:10 smithi204 ceph-mon[32357]: 3.14 scrub starts 2024-09-10T18:44:10.524 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:10 smithi204 ceph-mon[32357]: 3.14 scrub ok 2024-09-10T18:44:10.524 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:10 smithi204 ceph-mon[32357]: 4.6 deep-scrub starts 2024-09-10T18:44:10.524 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:10 smithi204 ceph-mon[32357]: 4.6 deep-scrub ok 2024-09-10T18:44:10.959 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a: {} 2024-09-10T18:44:10.960 INFO:teuthology.orchestra.run.smithi028.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T18:44:10.972 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b: {} 2024-09-10T18:44:10.972 INFO:teuthology.orchestra.run.smithi028.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T18:44:10.981 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c: {} 2024-09-10T18:44:10.981 INFO:teuthology.orchestra.run.smithi028.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T18:44:11.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:11 smithi028 ceph-mon[29291]: 5.19 scrub starts 2024-09-10T18:44:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:11 smithi028 ceph-mon[29291]: 5.19 scrub ok 2024-09-10T18:44:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:11 smithi028 ceph-mon[29291]: 5.13 scrub starts 2024-09-10T18:44:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:11 smithi028 ceph-mon[29291]: 5.13 scrub ok 2024-09-10T18:44:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:11 smithi028 ceph-mon[29291]: 5.11 scrub starts 2024-09-10T18:44:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:11 smithi028 ceph-mon[29291]: 5.11 scrub ok 2024-09-10T18:44:11.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:11 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1537112024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:44:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:11 smithi028 ceph-mon[39220]: 5.19 scrub starts 2024-09-10T18:44:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:11 smithi028 ceph-mon[39220]: 5.19 scrub ok 2024-09-10T18:44:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:11 smithi028 ceph-mon[39220]: 5.13 scrub starts 2024-09-10T18:44:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:11 smithi028 ceph-mon[39220]: 5.13 scrub ok 2024-09-10T18:44:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:11 smithi028 ceph-mon[39220]: 5.11 scrub starts 2024-09-10T18:44:11.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:11 smithi028 ceph-mon[39220]: 5.11 scrub ok 2024-09-10T18:44:11.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:11 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1537112024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:44:11.531 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:11 smithi204 ceph-mon[32357]: 5.19 scrub starts 2024-09-10T18:44:11.531 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:11 smithi204 ceph-mon[32357]: 5.19 scrub ok 2024-09-10T18:44:11.531 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:11 smithi204 ceph-mon[32357]: 5.13 scrub starts 2024-09-10T18:44:11.532 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:11 smithi204 ceph-mon[32357]: 5.13 scrub ok 2024-09-10T18:44:11.532 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:11 smithi204 ceph-mon[32357]: 5.11 scrub starts 2024-09-10T18:44:11.532 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:11 smithi204 ceph-mon[32357]: 5.11 scrub ok 2024-09-10T18:44:11.532 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:11 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1537112024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:44:11.584 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:44:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:44:11] "GET /metrics HTTP/1.1" 200 222488 "" "Prometheus/2.43.0" 2024-09-10T18:44:11.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-10T18:44:11.980 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:12.055 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:44:12.055 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-10T18:44:12.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-10T18:44:12.056 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:44:12.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.056 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14532.0:1059 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e360)", 2024-09-10T18:44:12.056 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.270981+0000", 2024-09-10T18:44:12.056 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 287.78149853500003, 2024-09-10T18:44:12.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.975137225, 2024-09-10T18:44:12.056 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.056 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:44:12.056 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.056 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14532", 2024-09-10T18:44:12.056 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.204:0/1733125751", 2024-09-10T18:44:12.056 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1059 2024-09-10T18:44:12.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.057 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.270981+0000", 2024-09-10T18:44:12.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.270981+0000", 2024-09-10T18:44:12.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.270978+0000", 2024-09-10T18:44:12.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:44:12.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.270987+0000", 2024-09-10T18:44:12.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1730000000000001e-06 2024-09-10T18:44:12.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.271000+0000", 2024-09-10T18:44:12.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2809999999999999e-05 2024-09-10T18:44:12.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.271006+0000", 2024-09-10T18:44:12.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7749999999999998e-06 2024-09-10T18:44:12.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.271076+0000", 2024-09-10T18:44:12.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9720000000000003e-05 2024-09-10T18:44:12.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:12.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.271083+0000", 2024-09-10T18:44:12.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4900000000000003e-06 2024-09-10T18:44:12.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.245829+0000", 2024-09-10T18:44:12.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97474618899999999 2024-09-10T18:44:12.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.245977+0000", 2024-09-10T18:44:12.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014791699999999999 2024-09-10T18:44:12.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.246118+0000", 2024-09-10T18:44:12.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014067999999999999 2024-09-10T18:44:12.061 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.061 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.061 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.061 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253976 10.c 10:31410340:::benchmark_data_smithi028_161947_object117052:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-10T18:44:12.061 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:26.832997+0000", 2024-09-10T18:44:12.061 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 285.21948249399998, 2024-09-10T18:44:12.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.34804471399999998, 2024-09-10T18:44:12.061 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.061 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.061 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.062 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:12.062 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:12.062 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253976 2024-09-10T18:44:12.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.062 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.832997+0000", 2024-09-10T18:44:12.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.832997+0000", 2024-09-10T18:44:12.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.832994+0000", 2024-09-10T18:44:12.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:44:12.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.833004+0000", 2024-09-10T18:44:12.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4439999999999997e-06 2024-09-10T18:44:12.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.833024+0000", 2024-09-10T18:44:12.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0647999999999999e-05 2024-09-10T18:44:12.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.833032+0000", 2024-09-10T18:44:12.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8660000000000006e-06 2024-09-10T18:44:12.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.833090+0000", 2024-09-10T18:44:12.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8102e-05 2024-09-10T18:44:12.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.833233+0000", 2024-09-10T18:44:12.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014301900000000001 2024-09-10T18:44:12.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3,4", 2024-09-10T18:44:12.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.833375+0000", 2024-09-10T18:44:12.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014154799999999999 2024-09-10T18:44:12.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.836637+0000", 2024-09-10T18:44:12.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032624249999999998 2024-09-10T18:44:12.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:12.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.838250+0000", 2024-09-10T18:44:12.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016129 2024-09-10T18:44:12.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.838798+0000", 2024-09-10T18:44:12.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000548304 2024-09-10T18:44:12.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:27.180820+0000", 2024-09-10T18:44:12.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.34202175699999998 2024-09-10T18:44:12.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:27.180936+0000", 2024-09-10T18:44:12.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000115751 2024-09-10T18:44:12.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:27.181042+0000", 2024-09-10T18:44:12.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105591 2024-09-10T18:44:12.068 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.068 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.068 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.068 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:25896 11.f 11:f45f1a47:::benchmark_data_smithi028_186962_object25895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-10T18:44:12.068 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:40:26.652734+0000", 2024-09-10T18:44:12.068 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.39974537800001, 2024-09-10T18:44:12.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.23825011700000001, 2024-09-10T18:44:12.069 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.069 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.069 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.069 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:12.069 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:12.069 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 25896 2024-09-10T18:44:12.069 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.069 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.069 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.069 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.069 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.652734+0000", 2024-09-10T18:44:12.069 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.069 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.652734+0000", 2024-09-10T18:44:12.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.652731+0000", 2024-09-10T18:44:12.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:44:12.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.652769+0000", 2024-09-10T18:44:12.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7768999999999997e-05 2024-09-10T18:44:12.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.652780+0000", 2024-09-10T18:44:12.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1573000000000001e-05 2024-09-10T18:44:12.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.652785+0000", 2024-09-10T18:44:12.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0010000000000001e-06 2024-09-10T18:44:12.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.652829+0000", 2024-09-10T18:44:12.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3315000000000001e-05 2024-09-10T18:44:12.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.652953+0000", 2024-09-10T18:44:12.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012419699999999999 2024-09-10T18:44:12.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:44:12.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.653090+0000", 2024-09-10T18:44:12.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013666000000000001 2024-09-10T18:44:12.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.073 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.654688+0000", 2024-09-10T18:44:12.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001598173 2024-09-10T18:44:12.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.073 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:12.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.654697+0000", 2024-09-10T18:44:12.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7649999999999999e-06 2024-09-10T18:44:12.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.073 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.890890+0000", 2024-09-10T18:44:12.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.23619354100000001 2024-09-10T18:44:12.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.890942+0000", 2024-09-10T18:44:12.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1672000000000003e-05 2024-09-10T18:44:12.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.890984+0000", 2024-09-10T18:44:12.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2302000000000003e-05 2024-09-10T18:44:12.075 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.075 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.075 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.075 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.075 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:25915 11.f 11:f90b5c6a:::benchmark_data_smithi028_186962_object25914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-10T18:44:12.075 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:40:26.672576+0000", 2024-09-10T18:44:12.075 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.37990324, 2024-09-10T18:44:12.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.221930869, 2024-09-10T18:44:12.075 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.075 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.075 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.075 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:12.076 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:12.076 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 25915 2024-09-10T18:44:12.076 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.076 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.076 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.076 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.076 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672576+0000", 2024-09-10T18:44:12.076 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.076 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.076 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.076 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.076 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672576+0000", 2024-09-10T18:44:12.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672573+0000", 2024-09-10T18:44:12.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:44:12.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672616+0000", 2024-09-10T18:44:12.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2765999999999997e-05 2024-09-10T18:44:12.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672634+0000", 2024-09-10T18:44:12.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7462999999999999e-05 2024-09-10T18:44:12.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672640+0000", 2024-09-10T18:44:12.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7070000000000003e-06 2024-09-10T18:44:12.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672687+0000", 2024-09-10T18:44:12.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6304000000000003e-05 2024-09-10T18:44:12.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672796+0000", 2024-09-10T18:44:12.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000109098 2024-09-10T18:44:12.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:44:12.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.672918+0000", 2024-09-10T18:44:12.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000122767 2024-09-10T18:44:12.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:12.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.674054+0000", 2024-09-10T18:44:12.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001135636 2024-09-10T18:44:12.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.674228+0000", 2024-09-10T18:44:12.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017409399999999999 2024-09-10T18:44:12.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.894422+0000", 2024-09-10T18:44:12.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.220193417 2024-09-10T18:44:12.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.894467+0000", 2024-09-10T18:44:12.081 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5510000000000003e-05 2024-09-10T18:44:12.081 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.081 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.081 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.081 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:40:26.894507+0000", 2024-09-10T18:44:12.081 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9913000000000002e-05 2024-09-10T18:44:12.081 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.085 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.085 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.086 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:104619 11.4 11:201ce33a:::benchmark_data_smithi028_186962_object104618:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-10T18:44:12.086 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:20.353467+0000", 2024-09-10T18:44:12.086 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 171.699012307, 2024-09-10T18:44:12.086 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.30411263500000002, 2024-09-10T18:44:12.086 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.086 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.086 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.086 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:12.086 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:12.086 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 104619 2024-09-10T18:44:12.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.086 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.086 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.086 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.087 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.353467+0000", 2024-09-10T18:44:12.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.087 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.087 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.087 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.353467+0000", 2024-09-10T18:44:12.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.087 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.087 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.087 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.353464+0000", 2024-09-10T18:44:12.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:44:12.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.353522+0000", 2024-09-10T18:44:12.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7807999999999999e-05 2024-09-10T18:44:12.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.353541+0000", 2024-09-10T18:44:12.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8980000000000001e-05 2024-09-10T18:44:12.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.353547+0000", 2024-09-10T18:44:12.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6540000000000004e-06 2024-09-10T18:44:12.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.353677+0000", 2024-09-10T18:44:12.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013072499999999999 2024-09-10T18:44:12.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.353868+0000", 2024-09-10T18:44:12.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019080799999999999 2024-09-10T18:44:12.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,6", 2024-09-10T18:44:12.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.354074+0000", 2024-09-10T18:44:12.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000205498 2024-09-10T18:44:12.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:12.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.360663+0000", 2024-09-10T18:44:12.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0065889479999999999 2024-09-10T18:44:12.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.361760+0000", 2024-09-10T18:44:12.093 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001097134 2024-09-10T18:44:12.093 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.093 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.093 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.093 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.657299+0000", 2024-09-10T18:44:12.093 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29553879500000002 2024-09-10T18:44:12.093 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.093 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.093 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.093 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.657470+0000", 2024-09-10T18:44:12.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017129800000000001 2024-09-10T18:44:12.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.657580+0000", 2024-09-10T18:44:12.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000109681 2024-09-10T18:44:12.094 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.094 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.094 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.094 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:104622 11.f 11:ff0b3a82:::benchmark_data_smithi028_186962_object104621:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-10T18:44:12.094 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:20.353568+0000", 2024-09-10T18:44:12.094 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 171.698911737, 2024-09-10T18:44:12.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.308653018, 2024-09-10T18:44:12.095 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.095 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.095 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.095 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:12.095 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:12.095 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 104622 2024-09-10T18:44:12.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.095 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.353568+0000", 2024-09-10T18:44:12.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.353568+0000", 2024-09-10T18:44:12.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.353567+0000", 2024-09-10T18:44:12.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999995 2024-09-10T18:44:12.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.353598+0000", 2024-09-10T18:44:12.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1207000000000001e-05 2024-09-10T18:44:12.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.353606+0000", 2024-09-10T18:44:12.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7589999999999997e-06 2024-09-10T18:44:12.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.353611+0000", 2024-09-10T18:44:12.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8019999999999999e-06 2024-09-10T18:44:12.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.353666+0000", 2024-09-10T18:44:12.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4858999999999998e-05 2024-09-10T18:44:12.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.353804+0000", 2024-09-10T18:44:12.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000138756 2024-09-10T18:44:12.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:44:12.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.353946+0000", 2024-09-10T18:44:12.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014122599999999999 2024-09-10T18:44:12.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.356677+0000", 2024-09-10T18:44:12.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027310699999999999 2024-09-10T18:44:12.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:12.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.357434+0000", 2024-09-10T18:44:12.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00075741899999999995 2024-09-10T18:44:12.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.662122+0000", 2024-09-10T18:44:12.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.30468771300000003 2024-09-10T18:44:12.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.662185+0000", 2024-09-10T18:44:12.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2800999999999997e-05 2024-09-10T18:44:12.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.662221+0000", 2024-09-10T18:44:12.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6086e-05 2024-09-10T18:44:12.100 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.100 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.101 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.101 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:104627 11.4 11:24ef7918:::benchmark_data_smithi028_186962_object104626:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-10T18:44:12.101 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:20.357323+0000", 2024-09-10T18:44:12.101 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 171.69515639900001, 2024-09-10T18:44:12.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.30076523300000002, 2024-09-10T18:44:12.101 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.101 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.101 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.101 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:12.101 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:12.101 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 104627 2024-09-10T18:44:12.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.102 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.357323+0000", 2024-09-10T18:44:12.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.357323+0000", 2024-09-10T18:44:12.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.357321+0000", 2024-09-10T18:44:12.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:44:12.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.357365+0000", 2024-09-10T18:44:12.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3962999999999999e-05 2024-09-10T18:44:12.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.357378+0000", 2024-09-10T18:44:12.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2982999999999999e-05 2024-09-10T18:44:12.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.357383+0000", 2024-09-10T18:44:12.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3179999999999997e-06 2024-09-10T18:44:12.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.357689+0000", 2024-09-10T18:44:12.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00030609499999999999 2024-09-10T18:44:12.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.357812+0000", 2024-09-10T18:44:12.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000123196 2024-09-10T18:44:12.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,6", 2024-09-10T18:44:12.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.357972+0000", 2024-09-10T18:44:12.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015965 2024-09-10T18:44:12.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:12.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.360670+0000", 2024-09-10T18:44:12.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0026982880000000001 2024-09-10T18:44:12.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.362117+0000", 2024-09-10T18:44:12.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001446611 2024-09-10T18:44:12.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.657617+0000", 2024-09-10T18:44:12.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.295500131 2024-09-10T18:44:12.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.658054+0000", 2024-09-10T18:44:12.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00043693399999999999 2024-09-10T18:44:12.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.107 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.107 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.658088+0000", 2024-09-10T18:44:12.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4424000000000001e-05 2024-09-10T18:44:12.107 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.107 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.107 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.107 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:104631 11.f 11:fb0f82ff:::benchmark_data_smithi028_186962_object104630:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-10T18:44:12.107 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:20.359500+0000", 2024-09-10T18:44:12.107 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 171.69297919900001, 2024-09-10T18:44:12.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.30286099100000002, 2024-09-10T18:44:12.108 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.108 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.108 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.108 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:12.108 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:12.108 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 104631 2024-09-10T18:44:12.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.108 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.359500+0000", 2024-09-10T18:44:12.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.359500+0000", 2024-09-10T18:44:12.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.359499+0000", 2024-09-10T18:44:12.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:44:12.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.359583+0000", 2024-09-10T18:44:12.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4233999999999993e-05 2024-09-10T18:44:12.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.359592+0000", 2024-09-10T18:44:12.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8449999999999992e-06 2024-09-10T18:44:12.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.359597+0000", 2024-09-10T18:44:12.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1630000000000004e-06 2024-09-10T18:44:12.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.359637+0000", 2024-09-10T18:44:12.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0312000000000003e-05 2024-09-10T18:44:12.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.359752+0000", 2024-09-10T18:44:12.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114478 2024-09-10T18:44:12.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:44:12.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.359912+0000", 2024-09-10T18:44:12.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015964000000000001 2024-09-10T18:44:12.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.362175+0000", 2024-09-10T18:44:12.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022636069999999999 2024-09-10T18:44:12.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:12.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.362206+0000", 2024-09-10T18:44:12.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1322999999999998e-05 2024-09-10T18:44:12.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.662278+0000", 2024-09-10T18:44:12.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.30007110300000001 2024-09-10T18:44:12.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.662324+0000", 2024-09-10T18:44:12.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6214000000000002e-05 2024-09-10T18:44:12.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.662361+0000", 2024-09-10T18:44:12.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7308999999999998e-05 2024-09-10T18:44:12.114 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.114 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.114 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.114 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:104635 11.f 11:f4f72d84:::benchmark_data_smithi028_186962_object104634:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-10T18:44:12.114 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:20.364332+0000", 2024-09-10T18:44:12.114 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 171.688147423, 2024-09-10T18:44:12.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29814456299999997, 2024-09-10T18:44:12.114 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.114 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.114 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.115 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:12.115 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:12.115 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 104635 2024-09-10T18:44:12.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.115 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.364332+0000", 2024-09-10T18:44:12.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.364332+0000", 2024-09-10T18:44:12.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.364323+0000", 2024-09-10T18:44:12.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999914 2024-09-10T18:44:12.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.364371+0000", 2024-09-10T18:44:12.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7917999999999998e-05 2024-09-10T18:44:12.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.364385+0000", 2024-09-10T18:44:12.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3682e-05 2024-09-10T18:44:12.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.364391+0000", 2024-09-10T18:44:12.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.981e-06 2024-09-10T18:44:12.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.364431+0000", 2024-09-10T18:44:12.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0181e-05 2024-09-10T18:44:12.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.364514+0000", 2024-09-10T18:44:12.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2378000000000003e-05 2024-09-10T18:44:12.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:44:12.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.364607+0000", 2024-09-10T18:44:12.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3336000000000002e-05 2024-09-10T18:44:12.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:12.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.367304+0000", 2024-09-10T18:44:12.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0026974970000000001 2024-09-10T18:44:12.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.367331+0000", 2024-09-10T18:44:12.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6709000000000001e-05 2024-09-10T18:44:12.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.662400+0000", 2024-09-10T18:44:12.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29506853199999999 2024-09-10T18:44:12.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.662444+0000", 2024-09-10T18:44:12.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4286999999999998e-05 2024-09-10T18:44:12.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.662476+0000", 2024-09-10T18:44:12.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2521000000000003e-05 2024-09-10T18:44:12.120 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.120 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.120 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.120 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:104637 11.f 11:fdb4d8cb:::benchmark_data_smithi028_186962_object104636:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-10T18:44:12.120 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:41:20.365665+0000", 2024-09-10T18:44:12.120 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 171.68681401500001, 2024-09-10T18:44:12.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.30174230200000002, 2024-09-10T18:44:12.120 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.121 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.121 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.121 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:12.121 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:12.121 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 104637 2024-09-10T18:44:12.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.121 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.365665+0000", 2024-09-10T18:44:12.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.365665+0000", 2024-09-10T18:44:12.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.365664+0000", 2024-09-10T18:44:12.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:44:12.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.365729+0000", 2024-09-10T18:44:12.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5548000000000003e-05 2024-09-10T18:44:12.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.365740+0000", 2024-09-10T18:44:12.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0501e-05 2024-09-10T18:44:12.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.365746+0000", 2024-09-10T18:44:12.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0530000000000003e-06 2024-09-10T18:44:12.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.366186+0000", 2024-09-10T18:44:12.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00044049999999999997 2024-09-10T18:44:12.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.366263+0000", 2024-09-10T18:44:12.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6681000000000007e-05 2024-09-10T18:44:12.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:44:12.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.366366+0000", 2024-09-10T18:44:12.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103069 2024-09-10T18:44:12.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:12.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.367464+0000", 2024-09-10T18:44:12.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010977199999999999 2024-09-10T18:44:12.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.367926+0000", 2024-09-10T18:44:12.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00046212100000000002 2024-09-10T18:44:12.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.667350+0000", 2024-09-10T18:44:12.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29942453200000002 2024-09-10T18:44:12.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.126 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.126 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.667383+0000", 2024-09-10T18:44:12.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2101999999999999e-05 2024-09-10T18:44:12.126 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.126 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.126 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:41:20.667408+0000", 2024-09-10T18:44:12.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5007e-05 2024-09-10T18:44:12.126 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.126 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.126 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.126 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.126 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:247377 11.5 11:a1cc6d35:::benchmark_data_smithi028_186962_object106770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-10T18:44:12.127 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:43:04.535893+0000", 2024-09-10T18:44:12.127 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 67.516586356999994, 2024-09-10T18:44:12.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.25236328299999999, 2024-09-10T18:44:12.127 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.127 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.127 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.127 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:12.127 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:12.127 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 247377 2024-09-10T18:44:12.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.127 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.535893+0000", 2024-09-10T18:44:12.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.535893+0000", 2024-09-10T18:44:12.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.535891+0000", 2024-09-10T18:44:12.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:44:12.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.535900+0000", 2024-09-10T18:44:12.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1899999999999995e-06 2024-09-10T18:44:12.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.535913+0000", 2024-09-10T18:44:12.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3604000000000001e-05 2024-09-10T18:44:12.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.535919+0000", 2024-09-10T18:44:12.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4260000000000002e-06 2024-09-10T18:44:12.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.535963+0000", 2024-09-10T18:44:12.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3909000000000003e-05 2024-09-10T18:44:12.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.536061+0000", 2024-09-10T18:44:12.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8913999999999995e-05 2024-09-10T18:44:12.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:44:12.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.536161+0000", 2024-09-10T18:44:12.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9253000000000007e-05 2024-09-10T18:44:12.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.538508+0000", 2024-09-10T18:44:12.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002347029 2024-09-10T18:44:12.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:12.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.538560+0000", 2024-09-10T18:44:12.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2057e-05 2024-09-10T18:44:12.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.788167+0000", 2024-09-10T18:44:12.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.24960754900000001 2024-09-10T18:44:12.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.788216+0000", 2024-09-10T18:44:12.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8241000000000001e-05 2024-09-10T18:44:12.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.788256+0000", 2024-09-10T18:44:12.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0618999999999999e-05 2024-09-10T18:44:12.133 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.133 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.133 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.133 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:247413 11.5 11:ac0625a9:::benchmark_data_smithi028_186962_object106806:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-10T18:44:12.133 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:43:04.562979+0000", 2024-09-10T18:44:12.134 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 67.489499968000004, 2024-09-10T18:44:12.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.22541271900000001, 2024-09-10T18:44:12.134 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.134 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.134 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.134 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:12.134 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:12.134 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 247413 2024-09-10T18:44:12.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.134 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.562979+0000", 2024-09-10T18:44:12.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.562979+0000", 2024-09-10T18:44:12.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.562977+0000", 2024-09-10T18:44:12.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:44:12.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.562985+0000", 2024-09-10T18:44:12.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0830000000000004e-06 2024-09-10T18:44:12.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.562999+0000", 2024-09-10T18:44:12.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3849000000000001e-05 2024-09-10T18:44:12.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.563006+0000", 2024-09-10T18:44:12.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6950000000000003e-06 2024-09-10T18:44:12.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.563060+0000", 2024-09-10T18:44:12.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4373000000000003e-05 2024-09-10T18:44:12.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.563163+0000", 2024-09-10T18:44:12.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010323 2024-09-10T18:44:12.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:44:12.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.563285+0000", 2024-09-10T18:44:12.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012197900000000001 2024-09-10T18:44:12.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:12.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.565605+0000", 2024-09-10T18:44:12.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023199499999999999 2024-09-10T18:44:12.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.565803+0000", 2024-09-10T18:44:12.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019825800000000001 2024-09-10T18:44:12.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.788307+0000", 2024-09-10T18:44:12.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.22250403599999999 2024-09-10T18:44:12.140 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.788350+0000", 2024-09-10T18:44:12.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2268999999999998e-05 2024-09-10T18:44:12.140 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:04.788392+0000", 2024-09-10T18:44:12.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2434000000000001e-05 2024-09-10T18:44:12.141 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.141 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.141 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.141 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:249208 11.f 11:fcb87ca8:::benchmark_data_smithi028_186962_object108601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-10T18:44:12.141 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:43:05.986219+0000", 2024-09-10T18:44:12.141 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 66.066260561000007, 2024-09-10T18:44:12.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.222469629, 2024-09-10T18:44:12.141 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.141 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.141 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.142 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:12.142 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:12.142 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 249208 2024-09-10T18:44:12.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.142 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.986219+0000", 2024-09-10T18:44:12.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.986219+0000", 2024-09-10T18:44:12.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.986217+0000", 2024-09-10T18:44:12.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:44:12.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.986225+0000", 2024-09-10T18:44:12.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5329999999999994e-06 2024-09-10T18:44:12.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.144 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.144 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.986235+0000", 2024-09-10T18:44:12.144 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7329999999999998e-06 2024-09-10T18:44:12.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.144 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.144 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.986240+0000", 2024-09-10T18:44:12.144 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9810000000000003e-06 2024-09-10T18:44:12.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.986313+0000", 2024-09-10T18:44:12.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2818000000000007e-05 2024-09-10T18:44:12.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.986435+0000", 2024-09-10T18:44:12.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012157100000000001 2024-09-10T18:44:12.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:44:12.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.986550+0000", 2024-09-10T18:44:12.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000115587 2024-09-10T18:44:12.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.146 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.146 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:12.146 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.989647+0000", 2024-09-10T18:44:12.146 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030967450000000001 2024-09-10T18:44:12.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.146 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.146 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.146 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.989678+0000", 2024-09-10T18:44:12.146 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1438e-05 2024-09-10T18:44:12.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:06.208607+0000", 2024-09-10T18:44:12.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21892842700000001 2024-09-10T18:44:12.147 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:06.208651+0000", 2024-09-10T18:44:12.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4663000000000003e-05 2024-09-10T18:44:12.147 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:06.208688+0000", 2024-09-10T18:44:12.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6946000000000002e-05 2024-09-10T18:44:12.148 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.148 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.148 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.148 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:249216 11.f 11:f5c07c30:::benchmark_data_smithi028_186962_object108609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-10T18:44:12.148 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:43:05.990897+0000", 2024-09-10T18:44:12.148 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 66.061582278000003, 2024-09-10T18:44:12.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21793504999999999, 2024-09-10T18:44:12.148 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.148 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.148 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.148 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:12.149 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:12.149 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 249216 2024-09-10T18:44:12.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.149 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.149 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.149 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.149 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.990897+0000", 2024-09-10T18:44:12.149 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.149 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.150 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.990897+0000", 2024-09-10T18:44:12.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.150 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.150 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.990895+0000", 2024-09-10T18:44:12.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:44:12.150 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.990901+0000", 2024-09-10T18:44:12.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1639999999999998e-06 2024-09-10T18:44:12.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.990914+0000", 2024-09-10T18:44:12.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2204999999999999e-05 2024-09-10T18:44:12.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.990920+0000", 2024-09-10T18:44:12.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.93e-06 2024-09-10T18:44:12.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.152 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.152 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.990959+0000", 2024-09-10T18:44:12.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9487000000000003e-05 2024-09-10T18:44:12.152 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.152 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.152 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.991065+0000", 2024-09-10T18:44:12.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106196 2024-09-10T18:44:12.152 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.991175+0000", 2024-09-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000110303 2024-09-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.993068+0000", 2024-09-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001892276 2024-09-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.993178+0000", 2024-09-10T18:44:12.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011021 2024-09-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:06.208727+0000", 2024-09-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21554891600000001 2024-09-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:06.208793+0000", 2024-09-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6352000000000004e-05 2024-09-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:06.208832+0000", 2024-09-10T18:44:12.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8850000000000002e-05 2024-09-10T18:44:12.155 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.155 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.155 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.155 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:249218 11.f 11:fef69695:::benchmark_data_smithi028_186962_object108611:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-10T18:44:12.155 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:43:05.995037+0000", 2024-09-10T18:44:12.156 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 66.057441913999995, 2024-09-10T18:44:12.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21390288599999999, 2024-09-10T18:44:12.156 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.156 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.156 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.156 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:12.156 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:12.156 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 249218 2024-09-10T18:44:12.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.156 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.157 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.157 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.995037+0000", 2024-09-10T18:44:12.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.157 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.157 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.157 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.995037+0000", 2024-09-10T18:44:12.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.157 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.157 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.157 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.995036+0000", 2024-09-10T18:44:12.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.995041+0000", 2024-09-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5029999999999999e-06 2024-09-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.995050+0000", 2024-09-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0860000000000008e-06 2024-09-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.158 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.995054+0000", 2024-09-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.155e-06 2024-09-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.995081+0000", 2024-09-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7104e-05 2024-09-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.995156+0000", 2024-09-10T18:44:12.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4589e-05 2024-09-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.995223+0000", 2024-09-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6780999999999996e-05 2024-09-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.996555+0000", 2024-09-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013327530000000001 2024-09-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:05.996670+0000", 2024-09-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114919 2024-09-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:06.208865+0000", 2024-09-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.212195041 2024-09-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:06.208901+0000", 2024-09-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5938e-05 2024-09-10T18:44:12.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.162 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.162 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:06.208940+0000", 2024-09-10T18:44:12.162 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8965999999999999e-05 2024-09-10T18:44:12.162 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.162 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.162 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.163 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:281195 11.4 11:2a5a87cb:::benchmark_data_smithi028_186962_object140588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-09-10T18:44:12.163 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:43:27.214557+0000", 2024-09-10T18:44:12.163 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 44.837922188, 2024-09-10T18:44:12.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.058172283999999998, 2024-09-10T18:44:12.163 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.163 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.163 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.163 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:12.163 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:12.163 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 281195 2024-09-10T18:44:12.163 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.163 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.214557+0000", 2024-09-10T18:44:12.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.214557+0000", 2024-09-10T18:44:12.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.214555+0000", 2024-09-10T18:44:12.165 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:44:12.165 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.165 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.165 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.214561+0000", 2024-09-10T18:44:12.165 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5119999999999999e-06 2024-09-10T18:44:12.165 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.165 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.165 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.214568+0000", 2024-09-10T18:44:12.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4170000000000003e-06 2024-09-10T18:44:12.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.166 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.166 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.166 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.214573+0000", 2024-09-10T18:44:12.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2459999999999997e-06 2024-09-10T18:44:12.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.166 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.166 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.166 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.214623+0000", 2024-09-10T18:44:12.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0016999999999999e-05 2024-09-10T18:44:12.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.166 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.214723+0000", 2024-09-10T18:44:12.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010005699999999999 2024-09-10T18:44:12.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,6", 2024-09-10T18:44:12.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.214828+0000", 2024-09-10T18:44:12.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105116 2024-09-10T18:44:12.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:12.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.217770+0000", 2024-09-10T18:44:12.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029421759999999999 2024-09-10T18:44:12.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.218587+0000", 2024-09-10T18:44:12.168 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00081653900000000002 2024-09-10T18:44:12.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.168 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.272649+0000", 2024-09-10T18:44:12.168 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.054062800000000001 2024-09-10T18:44:12.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.168 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.272692+0000", 2024-09-10T18:44:12.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2778999999999998e-05 2024-09-10T18:44:12.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.169 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.272729+0000", 2024-09-10T18:44:12.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7270000000000001e-05 2024-09-10T18:44:12.169 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.169 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.169 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.169 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:281201 11.4 11:22f93d83:::benchmark_data_smithi028_186962_object140594:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-09-10T18:44:12.169 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:43:27.219899+0000", 2024-09-10T18:44:12.169 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 44.832580024999999, 2024-09-10T18:44:12.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.053023555999999999, 2024-09-10T18:44:12.170 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.170 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.170 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.170 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:12.170 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:12.170 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 281201 2024-09-10T18:44:12.170 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.170 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.219899+0000", 2024-09-10T18:44:12.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.219899+0000", 2024-09-10T18:44:12.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.219897+0000", 2024-09-10T18:44:12.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:44:12.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.172 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.172 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.172 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.219905+0000", 2024-09-10T18:44:12.172 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2980000000000003e-06 2024-09-10T18:44:12.172 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.172 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.172 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.172 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.219916+0000", 2024-09-10T18:44:12.172 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1124999999999999e-05 2024-09-10T18:44:12.172 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.172 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.172 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.172 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.219922+0000", 2024-09-10T18:44:12.173 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.19e-06 2024-09-10T18:44:12.173 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.173 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.173 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.173 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.219965+0000", 2024-09-10T18:44:12.173 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2574999999999998e-05 2024-09-10T18:44:12.173 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.173 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.173 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.173 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.220080+0000", 2024-09-10T18:44:12.173 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000115079 2024-09-10T18:44:12.173 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.173 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.173 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,6", 2024-09-10T18:44:12.174 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.220159+0000", 2024-09-10T18:44:12.174 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9110000000000007e-05 2024-09-10T18:44:12.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.174 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.174 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:12.174 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.222416+0000", 2024-09-10T18:44:12.174 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022567640000000001 2024-09-10T18:44:12.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.174 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.174 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.174 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.272780+0000", 2024-09-10T18:44:12.174 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.050364540999999999 2024-09-10T18:44:12.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.174 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.272864+0000", 2024-09-10T18:44:12.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4251000000000004e-05 2024-09-10T18:44:12.175 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.272893+0000", 2024-09-10T18:44:12.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9028000000000001e-05 2024-09-10T18:44:12.175 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.176 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.272923+0000", 2024-09-10T18:44:12.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9516999999999999e-05 2024-09-10T18:44:12.176 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.176 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.176 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.176 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.176 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.176 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.25849.0:281205 11.f 11:ffc87ee8:::benchmark_data_smithi028_186962_object140598:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-09-10T18:44:12.176 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:43:27.220166+0000", 2024-09-10T18:44:12.176 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 44.832313130000003, 2024-09-10T18:44:12.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.052638580999999997, 2024-09-10T18:44:12.176 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.176 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.177 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.177 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.25849", 2024-09-10T18:44:12.177 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/3196120666", 2024-09-10T18:44:12.177 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 281205 2024-09-10T18:44:12.177 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.177 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.177 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.177 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.177 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.220166+0000", 2024-09-10T18:44:12.177 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.177 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.177 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.177 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.178 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.220166+0000", 2024-09-10T18:44:12.178 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.178 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.178 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.178 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.178 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.220165+0000", 2024-09-10T18:44:12.178 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:44:12.178 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.178 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.178 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.178 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.220171+0000", 2024-09-10T18:44:12.178 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8499999999999999e-06 2024-09-10T18:44:12.178 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.178 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.179 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.179 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.220179+0000", 2024-09-10T18:44:12.179 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0909999999999993e-06 2024-09-10T18:44:12.179 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.179 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.179 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.179 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.220184+0000", 2024-09-10T18:44:12.179 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9699999999999998e-06 2024-09-10T18:44:12.179 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.179 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.179 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.179 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.220244+0000", 2024-09-10T18:44:12.179 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0081999999999998e-05 2024-09-10T18:44:12.179 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.180 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.180 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.180 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.220341+0000", 2024-09-10T18:44:12.180 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7553000000000006e-05 2024-09-10T18:44:12.180 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.180 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.181 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1,2", 2024-09-10T18:44:12.181 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.220434+0000", 2024-09-10T18:44:12.181 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2813999999999996e-05 2024-09-10T18:44:12.181 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.181 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.181 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:12.181 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.222663+0000", 2024-09-10T18:44:12.181 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022289839999999998 2024-09-10T18:44:12.181 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.181 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.181 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.182 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.272242+0000", 2024-09-10T18:44:12.182 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.049578764999999997 2024-09-10T18:44:12.182 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.182 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.182 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:12.182 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.272742+0000", 2024-09-10T18:44:12.182 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00050043400000000001 2024-09-10T18:44:12.182 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.182 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.182 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.182 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.272772+0000", 2024-09-10T18:44:12.182 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9788999999999999e-05 2024-09-10T18:44:12.182 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.183 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.183 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.183 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.272805+0000", 2024-09-10T18:44:12.183 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.269e-05 2024-09-10T18:44:12.183 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.183 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.183 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.183 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.183 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.183 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.25849.0:281204 11.e e418/374 11:7cbea456:::benchmark_data_smithi028_186962_object140597:head v 418'17878, mlcod=418'17875)", 2024-09-10T18:44:12.183 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:43:27.221862+0000", 2024-09-10T18:44:12.183 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 44.830617578999998, 2024-09-10T18:44:12.183 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.050278697999999997, 2024-09-10T18:44:12.184 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.184 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:12.184 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.184 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.184 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.184 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.221862+0000", 2024-09-10T18:44:12.184 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.184 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.184 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.184 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.184 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.221862+0000", 2024-09-10T18:44:12.184 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.185 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.185 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.185 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.185 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.221860+0000", 2024-09-10T18:44:12.185 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:44:12.185 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.185 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.185 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.185 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.221868+0000", 2024-09-10T18:44:12.185 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2439999999999993e-06 2024-09-10T18:44:12.185 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.186 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.221887+0000", 2024-09-10T18:44:12.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8484000000000001e-05 2024-09-10T18:44:12.186 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.186 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.221892+0000", 2024-09-10T18:44:12.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3519999999999997e-06 2024-09-10T18:44:12.186 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.186 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.221949+0000", 2024-09-10T18:44:12.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7024999999999997e-05 2024-09-10T18:44:12.187 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.187 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.187 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.187 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.221966+0000", 2024-09-10T18:44:12.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6827999999999999e-05 2024-09-10T18:44:12.187 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.187 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.187 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:12.187 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.224278+0000", 2024-09-10T18:44:12.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002312316 2024-09-10T18:44:12.187 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.187 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.187 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.187 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:43:27.272140+0000", 2024-09-10T18:44:12.188 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.047862167999999997 2024-09-10T18:44:12.188 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.188 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.188 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.188 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.188 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.188 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.14532.0:1339 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e428)", 2024-09-10T18:44:12.188 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:44:05.753135+0000", 2024-09-10T18:44:12.188 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 6.2993446999999998, 2024-09-10T18:44:12.188 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.41270339700000003, 2024-09-10T18:44:12.189 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:12.189 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:44:12.189 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:12.189 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.14532", 2024-09-10T18:44:12.189 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.204:0/1733125751", 2024-09-10T18:44:12.189 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1339 2024-09-10T18:44:12.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.189 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:12.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:12.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:44:05.753135+0000", 2024-09-10T18:44:12.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:12.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:44:05.753135+0000", 2024-09-10T18:44:12.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:12.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:12.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:44:05.753132+0000", 2024-09-10T18:44:12.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:44:12.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:12.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:44:05.753141+0000", 2024-09-10T18:44:12.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4740000000000005e-06 2024-09-10T18:44:12.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:12.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:44:05.753154+0000", 2024-09-10T18:44:12.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2559e-05 2024-09-10T18:44:12.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:12.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:44:05.753159+0000", 2024-09-10T18:44:12.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6659999999999996e-06 2024-09-10T18:44:12.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.192 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.192 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.192 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:44:05.753224+0000", 2024-09-10T18:44:12.192 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4808999999999996e-05 2024-09-10T18:44:12.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.192 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.192 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:12.192 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:44:05.753233+0000", 2024-09-10T18:44:12.192 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3700000000000001e-06 2024-09-10T18:44:12.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.192 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.193 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:12.193 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:44:06.165584+0000", 2024-09-10T18:44:12.193 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.41235044100000001 2024-09-10T18:44:12.193 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.193 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.193 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:12.193 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:44:06.165719+0000", 2024-09-10T18:44:12.193 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000135637 2024-09-10T18:44:12.193 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:12.193 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:12.193 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:12.193 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:44:06.165838+0000", 2024-09-10T18:44:12.193 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000118588 2024-09-10T18:44:12.194 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.194 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.194 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.194 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:12.194 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:12.194 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:44:12.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:44:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:44:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:44:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:12 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:44:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:12 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:44:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:12 smithi204 ceph-mon[32357]: osdmap e431: 8 total, 8 up, 8 in 2024-09-10T18:44:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:12 smithi204 ceph-mon[32357]: pgmap v1020: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 836 B/s rd, 0 op/s; 53 KiB/s, 9 objects/s recovering 2024-09-10T18:44:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:12 smithi204 ceph-mon[32357]: 3.17 scrub starts 2024-09-10T18:44:12.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:12 smithi204 ceph-mon[32357]: 3.17 scrub ok 2024-09-10T18:44:12.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:12 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:44:12.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:12 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:44:12.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:12 smithi028 ceph-mon[39220]: osdmap e431: 8 total, 8 up, 8 in 2024-09-10T18:44:12.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:12 smithi028 ceph-mon[39220]: pgmap v1020: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 836 B/s rd, 0 op/s; 53 KiB/s, 9 objects/s recovering 2024-09-10T18:44:12.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:12 smithi028 ceph-mon[39220]: 3.17 scrub starts 2024-09-10T18:44:12.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:12 smithi028 ceph-mon[39220]: 3.17 scrub ok 2024-09-10T18:44:12.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:12 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:44:12.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:12 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:44:12.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:12 smithi028 ceph-mon[29291]: osdmap e431: 8 total, 8 up, 8 in 2024-09-10T18:44:12.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:12 smithi028 ceph-mon[29291]: pgmap v1020: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 836 B/s rd, 0 op/s; 53 KiB/s, 9 objects/s recovering 2024-09-10T18:44:12.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:12 smithi028 ceph-mon[29291]: 3.17 scrub starts 2024-09-10T18:44:12.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:12 smithi028 ceph-mon[29291]: 3.17 scrub ok 2024-09-10T18:44:13.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T18:44:13.306 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:44:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:13 smithi204 ceph-mon[32357]: 1.1a scrub starts 2024-09-10T18:44:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:13 smithi204 ceph-mon[32357]: 1.1a scrub ok 2024-09-10T18:44:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:13 smithi204 ceph-mon[32357]: 3.19 scrub starts 2024-09-10T18:44:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:13 smithi204 ceph-mon[32357]: 3.19 scrub ok 2024-09-10T18:44:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:13 smithi204 ceph-mon[32357]: 5.8 scrub starts 2024-09-10T18:44:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:13 smithi204 ceph-mon[32357]: 5.8 scrub ok 2024-09-10T18:44:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:13 smithi204 ceph-mon[32357]: 4.3 scrub starts 2024-09-10T18:44:13.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:13 smithi204 ceph-mon[32357]: 4.3 scrub ok 2024-09-10T18:44:13.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:13 smithi028 ceph-mon[39220]: 1.1a scrub starts 2024-09-10T18:44:13.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:13 smithi028 ceph-mon[39220]: 1.1a scrub ok 2024-09-10T18:44:13.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:13 smithi028 ceph-mon[39220]: 3.19 scrub starts 2024-09-10T18:44:13.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:13 smithi028 ceph-mon[39220]: 3.19 scrub ok 2024-09-10T18:44:13.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:13 smithi028 ceph-mon[39220]: 5.8 scrub starts 2024-09-10T18:44:13.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:13 smithi028 ceph-mon[39220]: 5.8 scrub ok 2024-09-10T18:44:13.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:13 smithi028 ceph-mon[39220]: 4.3 scrub starts 2024-09-10T18:44:13.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:13 smithi028 ceph-mon[39220]: 4.3 scrub ok 2024-09-10T18:44:13.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:13 smithi028 ceph-mon[29291]: 1.1a scrub starts 2024-09-10T18:44:13.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:13 smithi028 ceph-mon[29291]: 1.1a scrub ok 2024-09-10T18:44:13.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:13 smithi028 ceph-mon[29291]: 3.19 scrub starts 2024-09-10T18:44:13.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:13 smithi028 ceph-mon[29291]: 3.19 scrub ok 2024-09-10T18:44:13.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:13 smithi028 ceph-mon[29291]: 5.8 scrub starts 2024-09-10T18:44:13.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:13 smithi028 ceph-mon[29291]: 5.8 scrub ok 2024-09-10T18:44:13.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:13 smithi028 ceph-mon[29291]: 4.3 scrub starts 2024-09-10T18:44:13.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:13 smithi028 ceph-mon[29291]: 4.3 scrub ok 2024-09-10T18:44:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.0 to 1 (10000) 2024-09-10T18:44:14.442 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:14 smithi204 ceph-mon[32357]: pgmap v1021: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 40 KiB/s, 7 objects/s recovering 2024-09-10T18:44:14.442 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:14 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2000013110' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:44:14.442 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:14 smithi204 ceph-mon[32357]: from='client.36134 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:44:14.442 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:14 smithi204 ceph-mon[32357]: 3.16 scrub starts 2024-09-10T18:44:14.443 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:14 smithi204 ceph-mon[32357]: 3.16 scrub ok 2024-09-10T18:44:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:14 smithi028 ceph-mon[39220]: pgmap v1021: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 40 KiB/s, 7 objects/s recovering 2024-09-10T18:44:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:14 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2000013110' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:44:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:14 smithi028 ceph-mon[39220]: from='client.36134 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:44:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:14 smithi028 ceph-mon[39220]: 3.16 scrub starts 2024-09-10T18:44:14.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:14 smithi028 ceph-mon[39220]: 3.16 scrub ok 2024-09-10T18:44:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:14 smithi028 ceph-mon[29291]: pgmap v1021: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 40 KiB/s, 7 objects/s recovering 2024-09-10T18:44:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:14 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2000013110' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:44:14.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:14 smithi028 ceph-mon[29291]: from='client.36134 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:44:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:14 smithi028 ceph-mon[29291]: 3.16 scrub starts 2024-09-10T18:44:14.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:14 smithi028 ceph-mon[29291]: 3.16 scrub ok 2024-09-10T18:44:14.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-10T18:44:15.097 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:44:15.097 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:44:15.098 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-10T18:44:15.098 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:44:15.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:15 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:44:15.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:15 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:44:15.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:15 smithi204 ceph-mon[32357]: from='client.36134 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T18:44:15.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:15 smithi204 ceph-mon[32357]: osdmap e432: 8 total, 8 up, 8 in 2024-09-10T18:44:15.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:15 smithi204 ceph-mon[32357]: 3.6 scrub starts 2024-09-10T18:44:15.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:15 smithi204 ceph-mon[32357]: 3.6 scrub ok 2024-09-10T18:44:15.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:15 smithi204 ceph-mon[32357]: 4.1d scrub starts 2024-09-10T18:44:15.219 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:15 smithi204 ceph-mon[32357]: 4.1d scrub ok 2024-09-10T18:44:15.220 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:15 smithi204 ceph-mon[32357]: 6.7 deep-scrub starts 2024-09-10T18:44:15.220 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:15 smithi204 ceph-mon[32357]: 6.7 deep-scrub ok 2024-09-10T18:44:15.234 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:15.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:44:15.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:44:15.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[39220]: from='client.36134 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T18:44:15.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[39220]: osdmap e432: 8 total, 8 up, 8 in 2024-09-10T18:44:15.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[39220]: 3.6 scrub starts 2024-09-10T18:44:15.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[39220]: 3.6 scrub ok 2024-09-10T18:44:15.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[39220]: 4.1d scrub starts 2024-09-10T18:44:15.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[39220]: 4.1d scrub ok 2024-09-10T18:44:15.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[39220]: 6.7 deep-scrub starts 2024-09-10T18:44:15.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[39220]: 6.7 deep-scrub ok 2024-09-10T18:44:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:44:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:44:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[29291]: from='client.36134 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T18:44:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[29291]: osdmap e432: 8 total, 8 up, 8 in 2024-09-10T18:44:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[29291]: 3.6 scrub starts 2024-09-10T18:44:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[29291]: 3.6 scrub ok 2024-09-10T18:44:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[29291]: 4.1d scrub starts 2024-09-10T18:44:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[29291]: 4.1d scrub ok 2024-09-10T18:44:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[29291]: 6.7 deep-scrub starts 2024-09-10T18:44:15.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:15 smithi028 ceph-mon[29291]: 6.7 deep-scrub ok 2024-09-10T18:44:15.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:44:15 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:44:15.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-10T18:44:15.946 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:44:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:16 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:16 smithi204 ceph-mon[32357]: pgmap v1023: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 40 KiB/s, 3 objects/s recovering 2024-09-10T18:44:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:16 smithi204 ceph-mon[32357]: 5.1b scrub starts 2024-09-10T18:44:16.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:16 smithi204 ceph-mon[32357]: 5.1b scrub ok 2024-09-10T18:44:16.488 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:16 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:16.488 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:16 smithi028 ceph-mon[39220]: pgmap v1023: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 40 KiB/s, 3 objects/s recovering 2024-09-10T18:44:16.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:16 smithi028 ceph-mon[39220]: 5.1b scrub starts 2024-09-10T18:44:16.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:16 smithi028 ceph-mon[39220]: 5.1b scrub ok 2024-09-10T18:44:16.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:16 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:16.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:16 smithi028 ceph-mon[29291]: pgmap v1023: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 40 KiB/s, 3 objects/s recovering 2024-09-10T18:44:16.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:16 smithi028 ceph-mon[29291]: 5.1b scrub starts 2024-09-10T18:44:16.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:16 smithi028 ceph-mon[29291]: 5.1b scrub ok 2024-09-10T18:44:17.178 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-10T18:44:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:17 smithi204 ceph-mon[32357]: 3.b scrub starts 2024-09-10T18:44:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:17 smithi204 ceph-mon[32357]: 3.b scrub ok 2024-09-10T18:44:17.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:17 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/473115548' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:44:17.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:17 smithi028 ceph-mon[39220]: 3.b scrub starts 2024-09-10T18:44:17.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:17 smithi028 ceph-mon[39220]: 3.b scrub ok 2024-09-10T18:44:17.480 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:17 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/473115548' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:44:17.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:17 smithi028 ceph-mon[29291]: 3.b scrub starts 2024-09-10T18:44:17.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:17 smithi028 ceph-mon[29291]: 3.b scrub ok 2024-09-10T18:44:17.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:17 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/473115548' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:44:17.585 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:44:17.585 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-10T18:44:17.585 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-10T18:44:17.586 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-10T18:44:17.586 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:44:17.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-10T18:44:18.162 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:18.316 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:18 smithi204 ceph-mon[32357]: 5.e scrub starts 2024-09-10T18:44:18.316 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:18 smithi204 ceph-mon[32357]: 5.e scrub ok 2024-09-10T18:44:18.316 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:18 smithi204 ceph-mon[32357]: 1.9 deep-scrub starts 2024-09-10T18:44:18.317 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:18 smithi204 ceph-mon[32357]: 1.9 deep-scrub ok 2024-09-10T18:44:18.317 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:18 smithi204 ceph-mon[32357]: 5.c scrub starts 2024-09-10T18:44:18.317 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:18 smithi204 ceph-mon[32357]: 5.c scrub ok 2024-09-10T18:44:18.317 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:18 smithi204 ceph-mon[32357]: 5.6 scrub starts 2024-09-10T18:44:18.317 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:18 smithi204 ceph-mon[32357]: 5.6 scrub ok 2024-09-10T18:44:18.317 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:18 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/473115548' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-10T18:44:18.317 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:18 smithi204 ceph-mon[32357]: osdmap e433: 8 total, 8 up, 8 in 2024-09-10T18:44:18.317 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:18 smithi204 ceph-mon[32357]: pgmap v1025: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 840 B/s rd, 0 op/s 2024-09-10T18:44:18.317 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:18 smithi204 ceph-mon[32357]: 4.16 scrub starts 2024-09-10T18:44:18.317 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:18 smithi204 ceph-mon[32357]: 4.16 scrub ok 2024-09-10T18:44:18.317 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:18 smithi204 ceph-mon[32357]: 4.1f scrub starts 2024-09-10T18:44:18.317 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:18 smithi204 ceph-mon[32357]: 4.1f scrub ok 2024-09-10T18:44:18.341 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-10T18:44:18.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[29291]: 5.e scrub starts 2024-09-10T18:44:18.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[29291]: 5.e scrub ok 2024-09-10T18:44:18.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[29291]: 1.9 deep-scrub starts 2024-09-10T18:44:18.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[29291]: 1.9 deep-scrub ok 2024-09-10T18:44:18.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[29291]: 5.c scrub starts 2024-09-10T18:44:18.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[29291]: 5.c scrub ok 2024-09-10T18:44:18.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[29291]: 5.6 scrub starts 2024-09-10T18:44:18.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[29291]: 5.6 scrub ok 2024-09-10T18:44:18.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/473115548' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-10T18:44:18.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[29291]: osdmap e433: 8 total, 8 up, 8 in 2024-09-10T18:44:18.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[29291]: pgmap v1025: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 840 B/s rd, 0 op/s 2024-09-10T18:44:18.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[29291]: 4.16 scrub starts 2024-09-10T18:44:18.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[29291]: 4.16 scrub ok 2024-09-10T18:44:18.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[29291]: 4.1f scrub starts 2024-09-10T18:44:18.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[29291]: 4.1f scrub ok 2024-09-10T18:44:18.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[39220]: 5.e scrub starts 2024-09-10T18:44:18.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[39220]: 5.e scrub ok 2024-09-10T18:44:18.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[39220]: 1.9 deep-scrub starts 2024-09-10T18:44:18.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[39220]: 1.9 deep-scrub ok 2024-09-10T18:44:18.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[39220]: 5.c scrub starts 2024-09-10T18:44:18.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[39220]: 5.c scrub ok 2024-09-10T18:44:18.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[39220]: 5.6 scrub starts 2024-09-10T18:44:18.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[39220]: 5.6 scrub ok 2024-09-10T18:44:18.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/473115548' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-10T18:44:18.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[39220]: osdmap e433: 8 total, 8 up, 8 in 2024-09-10T18:44:18.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[39220]: pgmap v1025: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 840 B/s rd, 0 op/s 2024-09-10T18:44:18.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[39220]: 4.16 scrub starts 2024-09-10T18:44:18.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[39220]: 4.16 scrub ok 2024-09-10T18:44:18.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[39220]: 4.1f scrub starts 2024-09-10T18:44:18.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:18 smithi028 ceph-mon[39220]: 4.1f scrub ok 2024-09-10T18:44:18.590 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:44:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:19 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:44:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:19 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:44:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:19 smithi204 ceph-mon[32357]: 4.f scrub starts 2024-09-10T18:44:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:19 smithi204 ceph-mon[32357]: 4.f scrub ok 2024-09-10T18:44:19.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:19 smithi204 ceph-mon[32357]: osdmap e434: 8 total, 8 up, 8 in 2024-09-10T18:44:19.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:19 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:44:19.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:19 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:44:19.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:19 smithi028 ceph-mon[39220]: 4.f scrub starts 2024-09-10T18:44:19.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:19 smithi028 ceph-mon[39220]: 4.f scrub ok 2024-09-10T18:44:19.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:19 smithi028 ceph-mon[39220]: osdmap e434: 8 total, 8 up, 8 in 2024-09-10T18:44:19.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:19 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:44:19.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:19 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:44:19.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:19 smithi028 ceph-mon[29291]: 4.f scrub starts 2024-09-10T18:44:19.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:19 smithi028 ceph-mon[29291]: 4.f scrub ok 2024-09-10T18:44:19.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:19 smithi028 ceph-mon[29291]: osdmap e434: 8 total, 8 up, 8 in 2024-09-10T18:44:20.192 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.1 to 1 (10000) 2024-09-10T18:44:20.288 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:20 smithi204 ceph-mon[32357]: 6.2 scrub starts 2024-09-10T18:44:20.288 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:20 smithi204 ceph-mon[32357]: 6.2 scrub ok 2024-09-10T18:44:20.289 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:20 smithi204 ceph-mon[32357]: 3.e scrub starts 2024-09-10T18:44:20.289 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:20 smithi204 ceph-mon[32357]: 3.e scrub ok 2024-09-10T18:44:20.289 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:20 smithi204 ceph-mon[32357]: 4.1c scrub starts 2024-09-10T18:44:20.289 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:20 smithi204 ceph-mon[32357]: 4.1c scrub ok 2024-09-10T18:44:20.289 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:20 smithi204 ceph-mon[32357]: pgmap v1027: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:20.289 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:20 smithi204 ceph-mon[32357]: 4.8 deep-scrub starts 2024-09-10T18:44:20.289 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:20 smithi204 ceph-mon[32357]: 4.8 deep-scrub ok 2024-09-10T18:44:20.289 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:20 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1799898921' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:44:20.289 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:20 smithi204 ceph-mon[32357]: from='client.36146 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:44:20.294 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-10T18:44:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-10T18:44:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-10T18:44:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-10T18:44:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:214410 10.f 10:f1b0b8f0:::benchmark_data_smithi028_161947_object77486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:44:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.062548+0000", 2024-09-10T18:44:20.294 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 328.22324917899999, 2024-09-10T18:44:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45499772199999999, 2024-09-10T18:44:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 214410 2024-09-10T18:44:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062548+0000", 2024-09-10T18:44:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062548+0000", 2024-09-10T18:44:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062546+0000", 2024-09-10T18:44:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:44:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062554+0000", 2024-09-10T18:44:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0849999999999997e-06 2024-09-10T18:44:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062566+0000", 2024-09-10T18:44:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2384e-05 2024-09-10T18:44:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062572+0000", 2024-09-10T18:44:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2299999999999996e-06 2024-09-10T18:44:20.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062612+0000", 2024-09-10T18:44:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9508999999999997e-05 2024-09-10T18:44:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062681+0000", 2024-09-10T18:44:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9183000000000005e-05 2024-09-10T18:44:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3", 2024-09-10T18:44:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.062748+0000", 2024-09-10T18:44:20.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7360999999999994e-05 2024-09-10T18:44:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.065004+0000", 2024-09-10T18:44:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022560739999999998 2024-09-10T18:44:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.070537+0000", 2024-09-10T18:44:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0055331089999999996 2024-09-10T18:44:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517440+0000", 2024-09-10T18:44:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44690291100000001 2024-09-10T18:44:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517498+0000", 2024-09-10T18:44:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8114e-05 2024-09-10T18:44:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517546+0000", 2024-09-10T18:44:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7413999999999997e-05 2024-09-10T18:44:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.300 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:214415 10.a 10:58f59aac:::benchmark_data_smithi028_161947_object77491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:44:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.068100+0000", 2024-09-10T18:44:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 328.21769776100001, 2024-09-10T18:44:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44505853499999998, 2024-09-10T18:44:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 214415 2024-09-10T18:44:20.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068100+0000", 2024-09-10T18:44:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068100+0000", 2024-09-10T18:44:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068097+0000", 2024-09-10T18:44:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:44:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068106+0000", 2024-09-10T18:44:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9109999999999999e-06 2024-09-10T18:44:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068121+0000", 2024-09-10T18:44:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4595e-05 2024-09-10T18:44:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068128+0000", 2024-09-10T18:44:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1099999999999997e-06 2024-09-10T18:44:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068200+0000", 2024-09-10T18:44:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2563000000000003e-05 2024-09-10T18:44:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068320+0000", 2024-09-10T18:44:20.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119314 2024-09-10T18:44:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,5", 2024-09-10T18:44:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068445+0000", 2024-09-10T18:44:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125384 2024-09-10T18:44:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512681+0000", 2024-09-10T18:44:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.444235614 2024-09-10T18:44:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:20.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512695+0000", 2024-09-10T18:44:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3879999999999999e-05 2024-09-10T18:44:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512953+0000", 2024-09-10T18:44:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000258576 2024-09-10T18:44:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.513010+0000", 2024-09-10T18:44:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7293000000000002e-05 2024-09-10T18:44:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.513057+0000", 2024-09-10T18:44:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6354999999999999e-05 2024-09-10T18:44:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.513158+0000", 2024-09-10T18:44:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101337 2024-09-10T18:44:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.307 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:214416 10.a 10:5d56f650:::benchmark_data_smithi028_161947_object77492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:44:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.068145+0000", 2024-09-10T18:44:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 328.21765251400001, 2024-09-10T18:44:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44523108900000002, 2024-09-10T18:44:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 214416 2024-09-10T18:44:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068145+0000", 2024-09-10T18:44:20.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068145+0000", 2024-09-10T18:44:20.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068144+0000", 2024-09-10T18:44:20.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:44:20.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068148+0000", 2024-09-10T18:44:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7189999999999999e-06 2024-09-10T18:44:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068157+0000", 2024-09-10T18:44:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0110000000000006e-06 2024-09-10T18:44:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.068161+0000", 2024-09-10T18:44:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8600000000000003e-06 2024-09-10T18:44:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.070947+0000", 2024-09-10T18:44:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002786232 2024-09-10T18:44:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071069+0000", 2024-09-10T18:44:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012178199999999999 2024-09-10T18:44:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,5", 2024-09-10T18:44:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071193+0000", 2024-09-10T18:44:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000124795 2024-09-10T18:44:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512863+0000", 2024-09-10T18:44:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44166944299999999 2024-09-10T18:44:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512919+0000", 2024-09-10T18:44:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6190000000000002e-05 2024-09-10T18:44:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512981+0000", 2024-09-10T18:44:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2149000000000004e-05 2024-09-10T18:44:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.513207+0000", 2024-09-10T18:44:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022604600000000001 2024-09-10T18:44:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.513279+0000", 2024-09-10T18:44:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1328999999999994e-05 2024-09-10T18:44:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.513376+0000", 2024-09-10T18:44:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7270999999999998e-05 2024-09-10T18:44:20.313 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDPGPushReply(10.a 354/353 [PushReplyOp(10:53ea37f5:::benchmark_data_smithi028_161947_object116442:head)])", 2024-09-10T18:44:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.071316+0000", 2024-09-10T18:44:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 328.214481406, 2024-09-10T18:44:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44120257899999998, 2024-09-10T18:44:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:44:20.314 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071316+0000", 2024-09-10T18:44:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071316+0000", 2024-09-10T18:44:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071315+0000", 2024-09-10T18:44:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:44:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071320+0000", 2024-09-10T18:44:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6399999999999996e-06 2024-09-10T18:44:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071328+0000", 2024-09-10T18:44:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2549999999999997e-06 2024-09-10T18:44:20.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071333+0000", 2024-09-10T18:44:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9960000000000001e-06 2024-09-10T18:44:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512288+0000", 2024-09-10T18:44:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44095521300000001 2024-09-10T18:44:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512519+0000", 2024-09-10T18:44:20.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023027599999999999 2024-09-10T18:44:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:214417 10.f 10:f966d268:::benchmark_data_smithi028_161947_object77493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:44:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.071690+0000", 2024-09-10T18:44:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 328.21410742500001, 2024-09-10T18:44:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44675364699999998, 2024-09-10T18:44:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 214417 2024-09-10T18:44:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071690+0000", 2024-09-10T18:44:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071690+0000", 2024-09-10T18:44:20.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071687+0000", 2024-09-10T18:44:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999971 2024-09-10T18:44:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071695+0000", 2024-09-10T18:44:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9359999999999999e-06 2024-09-10T18:44:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071710+0000", 2024-09-10T18:44:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5092999999999999e-05 2024-09-10T18:44:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071718+0000", 2024-09-10T18:44:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5329999999999997e-06 2024-09-10T18:44:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.071782+0000", 2024-09-10T18:44:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4134000000000006e-05 2024-09-10T18:44:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072098+0000", 2024-09-10T18:44:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000316156 2024-09-10T18:44:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3", 2024-09-10T18:44:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072203+0000", 2024-09-10T18:44:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010477200000000001 2024-09-10T18:44:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517614+0000", 2024-09-10T18:44:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44541141000000001 2024-09-10T18:44:20.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.518312+0000", 2024-09-10T18:44:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00069769600000000004 2024-09-10T18:44:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.518366+0000", 2024-09-10T18:44:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3857999999999998e-05 2024-09-10T18:44:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.518409+0000", 2024-09-10T18:44:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3569999999999998e-05 2024-09-10T18:44:20.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.518444+0000", 2024-09-10T18:44:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4211000000000002e-05 2024-09-10T18:44:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:214419 10.f 10:f9ab56c8:::benchmark_data_smithi028_161947_object77495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:44:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.072174+0000", 2024-09-10T18:44:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 328.21362384600002, 2024-09-10T18:44:20.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45442210100000002, 2024-09-10T18:44:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 214419 2024-09-10T18:44:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072174+0000", 2024-09-10T18:44:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072174+0000", 2024-09-10T18:44:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072172+0000", 2024-09-10T18:44:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:44:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072178+0000", 2024-09-10T18:44:20.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0000000000000002e-06 2024-09-10T18:44:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072189+0000", 2024-09-10T18:44:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1032999999999999e-05 2024-09-10T18:44:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072196+0000", 2024-09-10T18:44:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4280000000000001e-06 2024-09-10T18:44:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.514966+0000", 2024-09-10T18:44:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44277013500000001 2024-09-10T18:44:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.515075+0000", 2024-09-10T18:44:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000109749 2024-09-10T18:44:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3", 2024-09-10T18:44:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.515187+0000", 2024-09-10T18:44:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011199700000000001 2024-09-10T18:44:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517688+0000", 2024-09-10T18:44:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025006770000000002 2024-09-10T18:44:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.526499+0000", 2024-09-10T18:44:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088109829999999997 2024-09-10T18:44:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.526546+0000", 2024-09-10T18:44:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6965999999999997e-05 2024-09-10T18:44:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.526573+0000", 2024-09-10T18:44:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7219000000000001e-05 2024-09-10T18:44:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.526596+0000", 2024-09-10T18:44:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2323999999999998e-05 2024-09-10T18:44:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.330 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.15693.0:214418 10.e e354/314 10:7f999048:::benchmark_data_smithi028_161947_object77494:head v 354'13271, mlcod=354'13269)", 2024-09-10T18:44:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.072212+0000", 2024-09-10T18:44:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 328.21358505199998, 2024-09-10T18:44:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44493968699999997, 2024-09-10T18:44:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072212+0000", 2024-09-10T18:44:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072212+0000", 2024-09-10T18:44:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072211+0000", 2024-09-10T18:44:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:44:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072218+0000", 2024-09-10T18:44:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.691e-06 2024-09-10T18:44:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072225+0000", 2024-09-10T18:44:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7170000000000002e-06 2024-09-10T18:44:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.072230+0000", 2024-09-10T18:44:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5910000000000003e-06 2024-09-10T18:44:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512193+0000", 2024-09-10T18:44:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43996238700000001 2024-09-10T18:44:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512221+0000", 2024-09-10T18:44:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8779999999999999e-05 2024-09-10T18:44:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517109+0000", 2024-09-10T18:44:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004887209 2024-09-10T18:44:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.517152+0000", 2024-09-10T18:44:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3341999999999999e-05 2024-09-10T18:44:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:214422 10.f 10:f1090f8c:::benchmark_data_smithi028_161947_object77498:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-10T18:44:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:38:52.076038+0000", 2024-09-10T18:44:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 328.20975920900003, 2024-09-10T18:44:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46066008899999999, 2024-09-10T18:44:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.335 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 214422 2024-09-10T18:44:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.076038+0000", 2024-09-10T18:44:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.076038+0000", 2024-09-10T18:44:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.076036+0000", 2024-09-10T18:44:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:44:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.076050+0000", 2024-09-10T18:44:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3974999999999999e-05 2024-09-10T18:44:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512179+0000", 2024-09-10T18:44:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43612967400000002 2024-09-10T18:44:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.512192+0000", 2024-09-10T18:44:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.312e-05 2024-09-10T18:44:20.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.518478+0000", 2024-09-10T18:44:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0062856149999999996 2024-09-10T18:44:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.518584+0000", 2024-09-10T18:44:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106502 2024-09-10T18:44:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3", 2024-09-10T18:44:20.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.518688+0000", 2024-09-10T18:44:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103607 2024-09-10T18:44:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.526442+0000", 2024-09-10T18:44:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007754337 2024-09-10T18:44:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.533804+0000", 2024-09-10T18:44:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0073612139999999996 2024-09-10T18:44:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.536645+0000", 2024-09-10T18:44:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002841732 2024-09-10T18:44:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.536673+0000", 2024-09-10T18:44:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7489999999999999e-05 2024-09-10T18:44:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:38:52.536698+0000", 2024-09-10T18:44:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5386999999999999e-05 2024-09-10T18:44:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224010 10.7 10:e0250792:::benchmark_data_smithi028_161947_object87086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-09-10T18:44:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.451472+0000", 2024-09-10T18:44:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 319.83432527100001, 2024-09-10T18:44:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.50653015300000004, 2024-09-10T18:44:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.343 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:44:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224010 2024-09-10T18:44:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451472+0000", 2024-09-10T18:44:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451472+0000", 2024-09-10T18:44:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451470+0000", 2024-09-10T18:44:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:44:20.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451477+0000", 2024-09-10T18:44:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1470000000000004e-06 2024-09-10T18:44:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451490+0000", 2024-09-10T18:44:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3033999999999999e-05 2024-09-10T18:44:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451496+0000", 2024-09-10T18:44:20.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5269999999999998e-06 2024-09-10T18:44:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451563+0000", 2024-09-10T18:44:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7187999999999994e-05 2024-09-10T18:44:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451693+0000", 2024-09-10T18:44:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000129966 2024-09-10T18:44:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3,5", 2024-09-10T18:44:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.451824+0000", 2024-09-10T18:44:20.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013087099999999999 2024-09-10T18:44:20.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:20.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.454501+0000", 2024-09-10T18:44:20.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0026775169999999999 2024-09-10T18:44:20.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.454537+0000", 2024-09-10T18:44:20.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5447999999999997e-05 2024-09-10T18:44:20.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.457032+0000", 2024-09-10T18:44:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024956470000000001 2024-09-10T18:44:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.473535+0000", 2024-09-10T18:44:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016502783 2024-09-10T18:44:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.473546+0000", 2024-09-10T18:44:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0431000000000001e-05 2024-09-10T18:44:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.952470+0000", 2024-09-10T18:44:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47892405900000001 2024-09-10T18:44:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for missing object", 2024-09-10T18:44:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.952584+0000", 2024-09-10T18:44:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114357 2024-09-10T18:44:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.957919+0000", 2024-09-10T18:44:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005335093 2024-09-10T18:44:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.958002+0000", 2024-09-10T18:44:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3076000000000002e-05 2024-09-10T18:44:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224020 10.a 10:5e0c363c:::benchmark_data_smithi028_161947_object87096:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-09-10T18:44:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.457939+0000", 2024-09-10T18:44:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 319.82785849300001, 2024-09-10T18:44:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.48552529, 2024-09-10T18:44:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:44:20.351 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224020 2024-09-10T18:44:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.457939+0000", 2024-09-10T18:44:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.457939+0000", 2024-09-10T18:44:20.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.457938+0000", 2024-09-10T18:44:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999995 2024-09-10T18:44:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.457941+0000", 2024-09-10T18:44:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.287e-06 2024-09-10T18:44:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.457947+0000", 2024-09-10T18:44:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7259999999999999e-06 2024-09-10T18:44:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.457951+0000", 2024-09-10T18:44:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4079999999999998e-06 2024-09-10T18:44:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.458090+0000", 2024-09-10T18:44:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000139827 2024-09-10T18:44:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.458216+0000", 2024-09-10T18:44:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012541900000000001 2024-09-10T18:44:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,5", 2024-09-10T18:44:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.458342+0000", 2024-09-10T18:44:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012641700000000001 2024-09-10T18:44:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.461711+0000", 2024-09-10T18:44:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033687320000000001 2024-09-10T18:44:20.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463213+0000", 2024-09-10T18:44:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0015021119999999999 2024-09-10T18:44:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463243+0000", 2024-09-10T18:44:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0298999999999999e-05 2024-09-10T18:44:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.472753+0000", 2024-09-10T18:44:20.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095097480000000002 2024-09-10T18:44:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.472764+0000", 2024-09-10T18:44:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.04e-05 2024-09-10T18:44:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.943332+0000", 2024-09-10T18:44:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47056816499999998 2024-09-10T18:44:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.943464+0000", 2024-09-10T18:44:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013241699999999999 2024-09-10T18:44:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224025 10.7 10:e7883bf7:::benchmark_data_smithi028_161947_object87101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-09-10T18:44:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.463510+0000", 2024-09-10T18:44:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 319.82228778000001, 2024-09-10T18:44:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.49490601499999998, 2024-09-10T18:44:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:44:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.358 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224025 2024-09-10T18:44:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463510+0000", 2024-09-10T18:44:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463510+0000", 2024-09-10T18:44:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463508+0000", 2024-09-10T18:44:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:44:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463515+0000", 2024-09-10T18:44:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4549999999999998e-06 2024-09-10T18:44:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463529+0000", 2024-09-10T18:44:20.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4250000000000001e-05 2024-09-10T18:44:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463536+0000", 2024-09-10T18:44:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6139999999999997e-06 2024-09-10T18:44:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.463758+0000", 2024-09-10T18:44:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000221652 2024-09-10T18:44:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.464491+0000", 2024-09-10T18:44:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00073350399999999995 2024-09-10T18:44:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 2,3,5", 2024-09-10T18:44:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.466575+0000", 2024-09-10T18:44:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020836539999999999 2024-09-10T18:44:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.479982+0000", 2024-09-10T18:44:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013407189999999999 2024-09-10T18:44:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.362 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.479993+0000", 2024-09-10T18:44:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0618999999999999e-05 2024-09-10T18:44:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.952817+0000", 2024-09-10T18:44:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47282479700000002 2024-09-10T18:44:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for missing object", 2024-09-10T18:44:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.952911+0000", 2024-09-10T18:44:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3918000000000004e-05 2024-09-10T18:44:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.958346+0000", 2024-09-10T18:44:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054350170000000003 2024-09-10T18:44:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.958416+0000", 2024-09-10T18:44:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9338999999999995e-05 2024-09-10T18:44:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:224030 10.7 10:eb8e2371:::benchmark_data_smithi028_161947_object87106:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-10T18:44:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:00.489381+0000", 2024-09-10T18:44:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 319.79641676799997, 2024-09-10T18:44:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46926472200000002, 2024-09-10T18:44:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 224030 2024-09-10T18:44:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.489381+0000", 2024-09-10T18:44:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.489381+0000", 2024-09-10T18:44:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.489379+0000", 2024-09-10T18:44:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:44:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.489385+0000", 2024-09-10T18:44:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.353e-06 2024-09-10T18:44:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.489396+0000", 2024-09-10T18:44:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0624e-05 2024-09-10T18:44:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.489400+0000", 2024-09-10T18:44:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5770000000000004e-06 2024-09-10T18:44:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.489428+0000", 2024-09-10T18:44:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7268e-05 2024-09-10T18:44:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.489439+0000", 2024-09-10T18:44:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1362000000000001e-05 2024-09-10T18:44:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.952941+0000", 2024-09-10T18:44:20.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.463501991 2024-09-10T18:44:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.953048+0000", 2024-09-10T18:44:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010751600000000001 2024-09-10T18:44:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3,5", 2024-09-10T18:44:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.953164+0000", 2024-09-10T18:44:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011588599999999999 2024-09-10T18:44:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:20.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.956405+0000", 2024-09-10T18:44:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032407389999999999 2024-09-10T18:44:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.957722+0000", 2024-09-10T18:44:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013170549999999999 2024-09-10T18:44:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.958568+0000", 2024-09-10T18:44:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00084616900000000002 2024-09-10T18:44:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.958597+0000", 2024-09-10T18:44:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8214000000000001e-05 2024-09-10T18:44:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:00.958645+0000", 2024-09-10T18:44:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8797000000000001e-05 2024-09-10T18:44:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.371 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.34334.0:2468 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94344803719616 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:44:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.216048+0000", 2024-09-10T18:44:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 296.06974972699999, 2024-09-10T18:44:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.039665386, 2024-09-10T18:44:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:44:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.34334", 2024-09-10T18:44:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/601695668", 2024-09-10T18:44:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2468 2024-09-10T18:44:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216048+0000", 2024-09-10T18:44:20.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216048+0000", 2024-09-10T18:44:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216046+0000", 2024-09-10T18:44:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999981 2024-09-10T18:44:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216053+0000", 2024-09-10T18:44:20.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1040000000000001e-06 2024-09-10T18:44:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216065+0000", 2024-09-10T18:44:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2252999999999999e-05 2024-09-10T18:44:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216071+0000", 2024-09-10T18:44:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9100000000000002e-06 2024-09-10T18:44:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.241959+0000", 2024-09-10T18:44:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025887876000000001 2024-09-10T18:44:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.241964+0000", 2024-09-10T18:44:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4979999999999997e-06 2024-09-10T18:44:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.247389+0000", 2024-09-10T18:44:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0054252889999999 2024-09-10T18:44:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for missing object", 2024-09-10T18:44:20.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.247556+0000", 2024-09-10T18:44:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016618199999999999 2024-09-10T18:44:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.251327+0000", 2024-09-10T18:44:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0037711110000000002 2024-09-10T18:44:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for degraded object", 2024-09-10T18:44:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.251367+0000", 2024-09-10T18:44:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0722000000000001e-05 2024-09-10T18:44:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.253854+0000", 2024-09-10T18:44:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024868970000000001 2024-09-10T18:44:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.253942+0000", 2024-09-10T18:44:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7706000000000004e-05 2024-09-10T18:44:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.255713+0000", 2024-09-10T18:44:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0017709220000000001 2024-09-10T18:44:20.377 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.34334.0:2469 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94344803751536 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:44:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.216077+0000", 2024-09-10T18:44:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 296.06972015100001, 2024-09-10T18:44:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0427746659999999, 2024-09-10T18:44:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-10T18:44:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.34334", 2024-09-10T18:44:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/601695668", 2024-09-10T18:44:20.378 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2469 2024-09-10T18:44:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216077+0000", 2024-09-10T18:44:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216077+0000", 2024-09-10T18:44:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216076+0000", 2024-09-10T18:44:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:44:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216079+0000", 2024-09-10T18:44:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9780000000000001e-06 2024-09-10T18:44:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216085+0000", 2024-09-10T18:44:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8250000000000001e-06 2024-09-10T18:44:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216089+0000", 2024-09-10T18:44:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.189e-06 2024-09-10T18:44:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216134+0000", 2024-09-10T18:44:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4165000000000002e-05 2024-09-10T18:44:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-10T18:44:20.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.216139+0000", 2024-09-10T18:44:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6330000000000001e-06 2024-09-10T18:44:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.225412+0000", 2024-09-10T18:44:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0092728069999999992 2024-09-10T18:44:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.225421+0000", 2024-09-10T18:44:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6340000000000007e-06 2024-09-10T18:44:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.251461+0000", 2024-09-10T18:44:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.026040659 2024-09-10T18:44:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for degraded object", 2024-09-10T18:44:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.251745+0000", 2024-09-10T18:44:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00028324900000000001 2024-09-10T18:44:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.255059+0000", 2024-09-10T18:44:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033147279999999999 2024-09-10T18:44:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.255131+0000", 2024-09-10T18:44:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1977999999999994e-05 2024-09-10T18:44:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.258852+0000", 2024-09-10T18:44:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0037205490000000001 2024-09-10T18:44:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253533 10.d 10:b7985f20:::benchmark_data_smithi028_161947_object116609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-10T18:44:20.384 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.220281+0000", 2024-09-10T18:44:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 296.065515995, 2024-09-10T18:44:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2166046609999999, 2024-09-10T18:44:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-10T18:44:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253533 2024-09-10T18:44:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.385 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220281+0000", 2024-09-10T18:44:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220281+0000", 2024-09-10T18:44:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220280+0000", 2024-09-10T18:44:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:44:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220296+0000", 2024-09-10T18:44:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5908000000000001e-05 2024-09-10T18:44:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220304+0000", 2024-09-10T18:44:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4589999999999998e-06 2024-09-10T18:44:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220310+0000", 2024-09-10T18:44:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.242e-06 2024-09-10T18:44:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220515+0000", 2024-09-10T18:44:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020526999999999999 2024-09-10T18:44:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220612+0000", 2024-09-10T18:44:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6762000000000006e-05 2024-09-10T18:44:20.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-10T18:44:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.220715+0000", 2024-09-10T18:44:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103421 2024-09-10T18:44:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.245159+0000", 2024-09-10T18:44:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024444429 2024-09-10T18:44:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.245168+0000", 2024-09-10T18:44:20.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3359999999999995e-06 2024-09-10T18:44:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.233087+0000", 2024-09-10T18:44:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98791947300000005 2024-09-10T18:44:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.233100+0000", 2024-09-10T18:44:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2761e-05 2024-09-10T18:44:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.390 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.424706+0000", 2024-09-10T18:44:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.191606293 2024-09-10T18:44:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for missing object", 2024-09-10T18:44:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.424824+0000", 2024-09-10T18:44:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011730699999999999 2024-09-10T18:44:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.436779+0000", 2024-09-10T18:44:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011955782999999999 2024-09-10T18:44:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.436886+0000", 2024-09-10T18:44:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106579 2024-09-10T18:44:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253542 10.d 10:bdcb9630:::benchmark_data_smithi028_161947_object116618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:44:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.242110+0000", 2024-09-10T18:44:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 296.043687697, 2024-09-10T18:44:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2379634849999999, 2024-09-10T18:44:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.392 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253542 2024-09-10T18:44:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242110+0000", 2024-09-10T18:44:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242110+0000", 2024-09-10T18:44:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242108+0000", 2024-09-10T18:44:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:44:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242114+0000", 2024-09-10T18:44:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7910000000000001e-06 2024-09-10T18:44:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242123+0000", 2024-09-10T18:44:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1970000000000002e-06 2024-09-10T18:44:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.242129+0000", 2024-09-10T18:44:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1920000000000002e-06 2024-09-10T18:44:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.245493+0000", 2024-09-10T18:44:20.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033640319999999999 2024-09-10T18:44:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.245498+0000", 2024-09-10T18:44:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6500000000000004e-06 2024-09-10T18:44:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.233171+0000", 2024-09-10T18:44:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98767313099999998 2024-09-10T18:44:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.233182+0000", 2024-09-10T18:44:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0536e-05 2024-09-10T18:44:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.424847+0000", 2024-09-10T18:44:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1916651730000001 2024-09-10T18:44:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.424963+0000", 2024-09-10T18:44:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011628099999999999 2024-09-10T18:44:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,3,4", 2024-09-10T18:44:20.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.425449+0000", 2024-09-10T18:44:20.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048583300000000002 2024-09-10T18:44:20.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:20.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.432621+0000", 2024-09-10T18:44:20.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0071715980000000004 2024-09-10T18:44:20.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.434929+0000", 2024-09-10T18:44:20.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002308783 2024-09-10T18:44:20.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.452296+0000", 2024-09-10T18:44:20.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017366686999999999 2024-09-10T18:44:20.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.479897+0000", 2024-09-10T18:44:20.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.027600978000000002 2024-09-10T18:44:20.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:20.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.479955+0000", 2024-09-10T18:44:20.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8133000000000001e-05 2024-09-10T18:44:20.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.480073+0000", 2024-09-10T18:44:20.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011787500000000001 2024-09-10T18:44:20.400 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.400 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.400 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.400 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:253555 10.b 10:d9889148:::benchmark_data_smithi028_161947_object116631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-10T18:44:20.400 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:24.269929+0000", 2024-09-10T18:44:20.400 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 296.01586813599999, 2024-09-10T18:44:20.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1002267880000001, 2024-09-10T18:44:20.400 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.401 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:20.401 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.401 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:20.401 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:20.401 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 253555 2024-09-10T18:44:20.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.401 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.269929+0000", 2024-09-10T18:44:20.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.269929+0000", 2024-09-10T18:44:20.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.269928+0000", 2024-09-10T18:44:20.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:44:20.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.269934+0000", 2024-09-10T18:44:20.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7499999999999997e-06 2024-09-10T18:44:20.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.269944+0000", 2024-09-10T18:44:20.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.893e-06 2024-09-10T18:44:20.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.269949+0000", 2024-09-10T18:44:20.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4360000000000004e-06 2024-09-10T18:44:20.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.270000+0000", 2024-09-10T18:44:20.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1484999999999999e-05 2024-09-10T18:44:20.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:20.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:24.270008+0000", 2024-09-10T18:44:20.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5440000000000001e-06 2024-09-10T18:44:20.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.241802+0000", 2024-09-10T18:44:20.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97179455699999995 2024-09-10T18:44:20.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-10T18:44:20.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:25.241812+0000", 2024-09-10T18:44:20.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5510000000000005e-06 2024-09-10T18:44:20.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.405 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.347312+0000", 2024-09-10T18:44:20.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1055002890000001 2024-09-10T18:44:20.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.405 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.347439+0000", 2024-09-10T18:44:20.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000126635 2024-09-10T18:44:20.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,1,4", 2024-09-10T18:44:20.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.347786+0000", 2024-09-10T18:44:20.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00034768899999999998 2024-09-10T18:44:20.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.352444+0000", 2024-09-10T18:44:20.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0046571440000000002 2024-09-10T18:44:20.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:20.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.352622+0000", 2024-09-10T18:44:20.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017846999999999999 2024-09-10T18:44:20.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.358043+0000", 2024-09-10T18:44:20.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054205030000000001 2024-09-10T18:44:20.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.369964+0000", 2024-09-10T18:44:20.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011921889999999999 2024-09-10T18:44:20.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:20.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.370051+0000", 2024-09-10T18:44:20.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6448999999999999e-05 2024-09-10T18:44:20.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:26.370156+0000", 2024-09-10T18:44:20.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105078 2024-09-10T18:44:20.408 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.408 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.408 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.408 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:255657 10.a 10:5bf37dbe:::benchmark_data_smithi028_161947_object118733:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-10T18:44:20.408 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:28.727522+0000", 2024-09-10T18:44:20.408 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 291.558275219, 2024-09-10T18:44:20.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47986652600000002, 2024-09-10T18:44:20.409 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.409 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:20.409 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.409 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:20.409 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:20.409 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 255657 2024-09-10T18:44:20.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.409 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.727522+0000", 2024-09-10T18:44:20.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.727522+0000", 2024-09-10T18:44:20.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.727520+0000", 2024-09-10T18:44:20.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999986 2024-09-10T18:44:20.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.727527+0000", 2024-09-10T18:44:20.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4609999999999996e-06 2024-09-10T18:44:20.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.727538+0000", 2024-09-10T18:44:20.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1007e-05 2024-09-10T18:44:20.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.727544+0000", 2024-09-10T18:44:20.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4529999999999999e-06 2024-09-10T18:44:20.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.902982+0000", 2024-09-10T18:44:20.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.17543789900000001 2024-09-10T18:44:20.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.903101+0000", 2024-09-10T18:44:20.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000118615 2024-09-10T18:44:20.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,5", 2024-09-10T18:44:20.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.903224+0000", 2024-09-10T18:44:20.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012264200000000001 2024-09-10T18:44:20.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:20.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.908451+0000", 2024-09-10T18:44:20.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005227654 2024-09-10T18:44:20.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.914591+0000", 2024-09-10T18:44:20.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061396030000000004 2024-09-10T18:44:20.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.914629+0000", 2024-09-10T18:44:20.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7883000000000003e-05 2024-09-10T18:44:20.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.207043+0000", 2024-09-10T18:44:20.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29241404500000001 2024-09-10T18:44:20.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:20.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.207094+0000", 2024-09-10T18:44:20.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1329000000000003e-05 2024-09-10T18:44:20.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.207389+0000", 2024-09-10T18:44:20.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029458899999999999 2024-09-10T18:44:20.415 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.415 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.415 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.415 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:255671 10.a 10:597be30f:::benchmark_data_smithi028_161947_object118747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-10T18:44:20.415 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:28.738638+0000", 2024-09-10T18:44:20.415 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 291.54715926900002, 2024-09-10T18:44:20.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46912842399999999, 2024-09-10T18:44:20.415 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.415 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:20.416 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.416 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:20.416 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:20.416 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 255671 2024-09-10T18:44:20.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.416 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.738638+0000", 2024-09-10T18:44:20.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.738638+0000", 2024-09-10T18:44:20.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.738637+0000", 2024-09-10T18:44:20.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.999999 2024-09-10T18:44:20.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.738642+0000", 2024-09-10T18:44:20.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2270000000000001e-06 2024-09-10T18:44:20.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.738655+0000", 2024-09-10T18:44:20.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2257e-05 2024-09-10T18:44:20.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.738660+0000", 2024-09-10T18:44:20.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8610000000000003e-06 2024-09-10T18:44:20.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.905752+0000", 2024-09-10T18:44:20.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16709138400000001 2024-09-10T18:44:20.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.905866+0000", 2024-09-10T18:44:20.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114041 2024-09-10T18:44:20.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,5", 2024-09-10T18:44:20.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.905997+0000", 2024-09-10T18:44:20.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013100699999999999 2024-09-10T18:44:20.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:20.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.908495+0000", 2024-09-10T18:44:20.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024976680000000002 2024-09-10T18:44:20.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.914656+0000", 2024-09-10T18:44:20.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006161522 2024-09-10T18:44:20.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.206858+0000", 2024-09-10T18:44:20.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29220204 2024-09-10T18:44:20.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.207448+0000", 2024-09-10T18:44:20.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00059034500000000004 2024-09-10T18:44:20.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:20.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.207654+0000", 2024-09-10T18:44:20.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020553499999999999 2024-09-10T18:44:20.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.207767+0000", 2024-09-10T18:44:20.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011254100000000001 2024-09-10T18:44:20.421 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.421 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.422 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.422 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.15693.0:255686 10.a 10:58a4d0e9:::benchmark_data_smithi028_161947_object118762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-10T18:44:20.422 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-10T18:39:28.760292+0000", 2024-09-10T18:44:20.422 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 291.525505678, 2024-09-10T18:44:20.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.447912906, 2024-09-10T18:44:20.422 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-10T18:44:20.422 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:44:20.422 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-10T18:44:20.422 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.15693", 2024-09-10T18:44:20.422 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "v1:172.21.15.28:0/1889247092", 2024-09-10T18:44:20.422 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 255686 2024-09-10T18:44:20.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.423 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-10T18:44:20.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-10T18:44:20.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.760292+0000", 2024-09-10T18:44:20.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-10T18:44:20.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.760292+0000", 2024-09-10T18:44:20.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-10T18:44:20.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-10T18:44:20.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.760289+0000", 2024-09-10T18:44:20.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4294967295.9999976 2024-09-10T18:44:20.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-10T18:44:20.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.760296+0000", 2024-09-10T18:44:20.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2189999999999998e-06 2024-09-10T18:44:20.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-10T18:44:20.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.760308+0000", 2024-09-10T18:44:20.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1430000000000001e-05 2024-09-10T18:44:20.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-10T18:44:20.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.760314+0000", 2024-09-10T18:44:20.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4910000000000002e-06 2024-09-10T18:44:20.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-10T18:44:20.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.908810+0000", 2024-09-10T18:44:20.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.14849517100000001 2024-09-10T18:44:20.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-10T18:44:20.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.908933+0000", 2024-09-10T18:44:20.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012294399999999999 2024-09-10T18:44:20.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,2,5", 2024-09-10T18:44:20.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.909057+0000", 2024-09-10T18:44:20.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012485099999999999 2024-09-10T18:44:20.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-10T18:44:20.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.914599+0000", 2024-09-10T18:44:20.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0055419099999999997 2024-09-10T18:44:20.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:28.914690+0000", 2024-09-10T18:44:20.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0551999999999996e-05 2024-09-10T18:44:20.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.206928+0000", 2024-09-10T18:44:20.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29223781999999998 2024-09-10T18:44:20.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:44:20.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.207823+0000", 2024-09-10T18:44:20.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00089523900000000004 2024-09-10T18:44:20.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-10T18:44:20.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.207893+0000", 2024-09-10T18:44:20.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9969999999999996e-05 2024-09-10T18:44:20.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-10T18:44:20.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-10T18:44:20.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-10T18:44:20.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-10T18:39:29.208205+0000", 2024-09-10T18:44:20.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000311708 2024-09-10T18:44:20.428 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.428 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.428 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.428 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-10T18:44:20.428 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-10T18:44:20.428 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-10T18:44:20.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[29291]: 6.2 scrub starts 2024-09-10T18:44:20.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[29291]: 6.2 scrub ok 2024-09-10T18:44:20.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[29291]: 3.e scrub starts 2024-09-10T18:44:20.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[29291]: 3.e scrub ok 2024-09-10T18:44:20.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[29291]: 4.1c scrub starts 2024-09-10T18:44:20.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[29291]: 4.1c scrub ok 2024-09-10T18:44:20.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[29291]: pgmap v1027: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:20.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[29291]: 4.8 deep-scrub starts 2024-09-10T18:44:20.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[29291]: 4.8 deep-scrub ok 2024-09-10T18:44:20.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1799898921' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:44:20.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[29291]: from='client.36146 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:44:20.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[39220]: 6.2 scrub starts 2024-09-10T18:44:20.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[39220]: 6.2 scrub ok 2024-09-10T18:44:20.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[39220]: 3.e scrub starts 2024-09-10T18:44:20.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[39220]: 3.e scrub ok 2024-09-10T18:44:20.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[39220]: 4.1c scrub starts 2024-09-10T18:44:20.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[39220]: 4.1c scrub ok 2024-09-10T18:44:20.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[39220]: pgmap v1027: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:20.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[39220]: 4.8 deep-scrub starts 2024-09-10T18:44:20.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[39220]: 4.8 deep-scrub ok 2024-09-10T18:44:20.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1799898921' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:44:20.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:20 smithi028 ceph-mon[39220]: from='client.36146 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:44:20.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-10T18:44:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:21.503 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:44:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:44:21] "GET /metrics HTTP/1.1" 200 222488 "" "Prometheus/2.43.0" 2024-09-10T18:44:21.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:21 smithi028 ceph-mon[29291]: from='client.36146 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T18:44:21.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:21 smithi028 ceph-mon[29291]: osdmap e435: 8 total, 8 up, 8 in 2024-09-10T18:44:21.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:21 smithi028 ceph-mon[29291]: 3.1 scrub starts 2024-09-10T18:44:21.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:21 smithi028 ceph-mon[29291]: 3.1 scrub ok 2024-09-10T18:44:21.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:21 smithi028 ceph-mon[29291]: 3.a scrub starts 2024-09-10T18:44:21.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:21 smithi028 ceph-mon[29291]: 3.a scrub ok 2024-09-10T18:44:21.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:21 smithi028 ceph-mon[39220]: from='client.36146 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T18:44:21.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:21 smithi028 ceph-mon[39220]: osdmap e435: 8 total, 8 up, 8 in 2024-09-10T18:44:21.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:21 smithi028 ceph-mon[39220]: 3.1 scrub starts 2024-09-10T18:44:21.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:21 smithi028 ceph-mon[39220]: 3.1 scrub ok 2024-09-10T18:44:21.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:21 smithi028 ceph-mon[39220]: 3.a scrub starts 2024-09-10T18:44:21.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:21 smithi028 ceph-mon[39220]: 3.a scrub ok 2024-09-10T18:44:21.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:21 smithi204 ceph-mon[32357]: from='client.36146 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T18:44:21.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:21 smithi204 ceph-mon[32357]: osdmap e435: 8 total, 8 up, 8 in 2024-09-10T18:44:21.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:21 smithi204 ceph-mon[32357]: 3.1 scrub starts 2024-09-10T18:44:21.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:21 smithi204 ceph-mon[32357]: 3.1 scrub ok 2024-09-10T18:44:21.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:21 smithi204 ceph-mon[32357]: 3.a scrub starts 2024-09-10T18:44:21.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:21 smithi204 ceph-mon[32357]: 3.a scrub ok 2024-09-10T18:44:22.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[29291]: 3.f scrub starts 2024-09-10T18:44:22.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[29291]: 3.f scrub ok 2024-09-10T18:44:22.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[29291]: pgmap v1029: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T18:44:22.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[29291]: 4.1e scrub starts 2024-09-10T18:44:22.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[29291]: 4.1e scrub ok 2024-09-10T18:44:22.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[29291]: 1.4 scrub starts 2024-09-10T18:44:22.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[29291]: 1.4 scrub ok 2024-09-10T18:44:22.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[29291]: 3.1c scrub starts 2024-09-10T18:44:22.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[29291]: 3.1c scrub ok 2024-09-10T18:44:22.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[39220]: 3.f scrub starts 2024-09-10T18:44:22.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[39220]: 3.f scrub ok 2024-09-10T18:44:22.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[39220]: pgmap v1029: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T18:44:22.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[39220]: 4.1e scrub starts 2024-09-10T18:44:22.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[39220]: 4.1e scrub ok 2024-09-10T18:44:22.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[39220]: 1.4 scrub starts 2024-09-10T18:44:22.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[39220]: 1.4 scrub ok 2024-09-10T18:44:22.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[39220]: 3.1c scrub starts 2024-09-10T18:44:22.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:22 smithi028 ceph-mon[39220]: 3.1c scrub ok 2024-09-10T18:44:22.505 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:44:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:44:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:44:22.505 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:22 smithi204 ceph-mon[32357]: 3.f scrub starts 2024-09-10T18:44:22.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:22 smithi204 ceph-mon[32357]: 3.f scrub ok 2024-09-10T18:44:22.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:22 smithi204 ceph-mon[32357]: pgmap v1029: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T18:44:22.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:22 smithi204 ceph-mon[32357]: 4.1e scrub starts 2024-09-10T18:44:22.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:22 smithi204 ceph-mon[32357]: 4.1e scrub ok 2024-09-10T18:44:22.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:22 smithi204 ceph-mon[32357]: 1.4 scrub starts 2024-09-10T18:44:22.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:22 smithi204 ceph-mon[32357]: 1.4 scrub ok 2024-09-10T18:44:22.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:22 smithi204 ceph-mon[32357]: 3.1c scrub starts 2024-09-10T18:44:22.506 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:22 smithi204 ceph-mon[32357]: 3.1c scrub ok 2024-09-10T18:44:23.205 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-10T18:44:23.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:23 smithi028 ceph-mon[29291]: 5.1c scrub starts 2024-09-10T18:44:23.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:23 smithi028 ceph-mon[29291]: 5.1c scrub ok 2024-09-10T18:44:23.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:23 smithi028 ceph-mon[29291]: 3.5 scrub starts 2024-09-10T18:44:23.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:23 smithi028 ceph-mon[29291]: 3.5 scrub ok 2024-09-10T18:44:23.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:23 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2112702506' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:44:23.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:23 smithi028 ceph-mon[29291]: from='client.36152 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:44:23.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:23 smithi028 ceph-mon[39220]: 5.1c scrub starts 2024-09-10T18:44:23.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:23 smithi028 ceph-mon[39220]: 5.1c scrub ok 2024-09-10T18:44:23.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:23 smithi028 ceph-mon[39220]: 3.5 scrub starts 2024-09-10T18:44:23.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:23 smithi028 ceph-mon[39220]: 3.5 scrub ok 2024-09-10T18:44:23.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:23 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2112702506' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:44:23.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:23 smithi028 ceph-mon[39220]: from='client.36152 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:44:23.516 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:23 smithi204 ceph-mon[32357]: 5.1c scrub starts 2024-09-10T18:44:23.516 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:23 smithi204 ceph-mon[32357]: 5.1c scrub ok 2024-09-10T18:44:23.516 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:23 smithi204 ceph-mon[32357]: 3.5 scrub starts 2024-09-10T18:44:23.516 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:23 smithi204 ceph-mon[32357]: 3.5 scrub ok 2024-09-10T18:44:23.516 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:23 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2112702506' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:44:23.516 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:23 smithi204 ceph-mon[32357]: from='client.36152 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:44:23.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-10T18:44:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:24.468 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:24 smithi028 ceph-mon[39220]: 1.3 deep-scrub starts 2024-09-10T18:44:24.468 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:24 smithi028 ceph-mon[39220]: 1.3 deep-scrub ok 2024-09-10T18:44:24.468 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:24 smithi028 ceph-mon[39220]: from='client.36152 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-10T18:44:24.468 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:24 smithi028 ceph-mon[39220]: osdmap e436: 8 total, 8 up, 8 in 2024-09-10T18:44:24.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:24 smithi028 ceph-mon[39220]: pgmap v1031: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:24.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:44:24.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:24 smithi028 ceph-mon[39220]: 6.d scrub starts 2024-09-10T18:44:24.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:24 smithi028 ceph-mon[39220]: 6.d scrub ok 2024-09-10T18:44:24.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:24 smithi028 ceph-mon[29291]: 1.3 deep-scrub starts 2024-09-10T18:44:24.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:24 smithi028 ceph-mon[29291]: 1.3 deep-scrub ok 2024-09-10T18:44:24.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:24 smithi028 ceph-mon[29291]: from='client.36152 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-10T18:44:24.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:24 smithi028 ceph-mon[29291]: osdmap e436: 8 total, 8 up, 8 in 2024-09-10T18:44:24.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:24 smithi028 ceph-mon[29291]: pgmap v1031: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:24.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:44:24.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:24 smithi028 ceph-mon[29291]: 6.d scrub starts 2024-09-10T18:44:24.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:24 smithi028 ceph-mon[29291]: 6.d scrub ok 2024-09-10T18:44:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:24 smithi204 ceph-mon[32357]: 1.3 deep-scrub starts 2024-09-10T18:44:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:24 smithi204 ceph-mon[32357]: 1.3 deep-scrub ok 2024-09-10T18:44:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:24 smithi204 ceph-mon[32357]: from='client.36152 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-10T18:44:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:24 smithi204 ceph-mon[32357]: osdmap e436: 8 total, 8 up, 8 in 2024-09-10T18:44:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:24 smithi204 ceph-mon[32357]: pgmap v1031: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:24.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:44:24.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:24 smithi204 ceph-mon[32357]: 6.d scrub starts 2024-09-10T18:44:24.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:24 smithi204 ceph-mon[32357]: 6.d scrub ok 2024-09-10T18:44:25.586 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:25 smithi028 ceph-mon[29291]: 1.b scrub starts 2024-09-10T18:44:25.586 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:25 smithi028 ceph-mon[29291]: 1.b scrub ok 2024-09-10T18:44:25.586 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:25 smithi028 ceph-mon[39220]: 1.b scrub starts 2024-09-10T18:44:25.586 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:25 smithi028 ceph-mon[39220]: 1.b scrub ok 2024-09-10T18:44:25.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:25 smithi204 ceph-mon[32357]: 1.b scrub starts 2024-09-10T18:44:25.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:25 smithi204 ceph-mon[32357]: 1.b scrub ok 2024-09-10T18:44:25.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:44:25 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:44:26.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:26 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:26 smithi204 ceph-mon[32357]: pgmap v1032: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 724 B/s rd, 0 op/s 2024-09-10T18:44:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:26 smithi204 ceph-mon[32357]: 6.b scrub starts 2024-09-10T18:44:26.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:26 smithi204 ceph-mon[32357]: 6.b scrub ok 2024-09-10T18:44:26.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:26 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:26.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:26 smithi028 ceph-mon[29291]: pgmap v1032: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 724 B/s rd, 0 op/s 2024-09-10T18:44:26.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:26 smithi028 ceph-mon[29291]: 6.b scrub starts 2024-09-10T18:44:26.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:26 smithi028 ceph-mon[29291]: 6.b scrub ok 2024-09-10T18:44:26.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:26 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:26.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:26 smithi028 ceph-mon[39220]: pgmap v1032: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 724 B/s rd, 0 op/s 2024-09-10T18:44:26.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:26 smithi028 ceph-mon[39220]: 6.b scrub starts 2024-09-10T18:44:26.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:26 smithi028 ceph-mon[39220]: 6.b scrub ok 2024-09-10T18:44:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.2 to 1 (10000) 2024-09-10T18:44:27.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:27 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3786955451' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:44:27.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:27 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3786955451' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:44:27.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:27 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3786955451' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:44:28.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-10T18:44:28.367 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:28.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:28 smithi028 ceph-mon[29291]: 1.d scrub starts 2024-09-10T18:44:28.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:28 smithi028 ceph-mon[29291]: 1.d scrub ok 2024-09-10T18:44:28.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:28 smithi028 ceph-mon[29291]: pgmap v1033: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:44:28.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:28 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3786955451' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T18:44:28.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:28 smithi028 ceph-mon[29291]: osdmap e437: 8 total, 8 up, 8 in 2024-09-10T18:44:28.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:28 smithi028 ceph-mon[29291]: 4.b scrub starts 2024-09-10T18:44:28.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:28 smithi028 ceph-mon[29291]: 4.b scrub ok 2024-09-10T18:44:28.490 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:28 smithi028 ceph-mon[39220]: 1.d scrub starts 2024-09-10T18:44:28.490 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:28 smithi028 ceph-mon[39220]: 1.d scrub ok 2024-09-10T18:44:28.490 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:28 smithi028 ceph-mon[39220]: pgmap v1033: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:44:28.490 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:28 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3786955451' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T18:44:28.490 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:28 smithi028 ceph-mon[39220]: osdmap e437: 8 total, 8 up, 8 in 2024-09-10T18:44:28.490 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:28 smithi028 ceph-mon[39220]: 4.b scrub starts 2024-09-10T18:44:28.490 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:28 smithi028 ceph-mon[39220]: 4.b scrub ok 2024-09-10T18:44:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:28 smithi204 ceph-mon[32357]: 1.d scrub starts 2024-09-10T18:44:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:28 smithi204 ceph-mon[32357]: 1.d scrub ok 2024-09-10T18:44:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:28 smithi204 ceph-mon[32357]: pgmap v1033: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:44:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:28 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3786955451' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T18:44:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:28 smithi204 ceph-mon[32357]: osdmap e437: 8 total, 8 up, 8 in 2024-09-10T18:44:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:28 smithi204 ceph-mon[32357]: 4.b scrub starts 2024-09-10T18:44:28.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:28 smithi204 ceph-mon[32357]: 4.b scrub ok 2024-09-10T18:44:29.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:29 smithi028 ceph-mon[29291]: 3.3 scrub starts 2024-09-10T18:44:29.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:29 smithi028 ceph-mon[29291]: 3.3 scrub ok 2024-09-10T18:44:29.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:29 smithi028 ceph-mon[29291]: 5.9 scrub starts 2024-09-10T18:44:29.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:29 smithi028 ceph-mon[29291]: 5.9 scrub ok 2024-09-10T18:44:29.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:29 smithi028 ceph-mon[29291]: 3.0 scrub starts 2024-09-10T18:44:29.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:29 smithi028 ceph-mon[29291]: 3.0 scrub ok 2024-09-10T18:44:29.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:29 smithi028 ceph-mon[39220]: 3.3 scrub starts 2024-09-10T18:44:29.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:29 smithi028 ceph-mon[39220]: 3.3 scrub ok 2024-09-10T18:44:29.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:29 smithi028 ceph-mon[39220]: 5.9 scrub starts 2024-09-10T18:44:29.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:29 smithi028 ceph-mon[39220]: 5.9 scrub ok 2024-09-10T18:44:29.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:29 smithi028 ceph-mon[39220]: 3.0 scrub starts 2024-09-10T18:44:29.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:29 smithi028 ceph-mon[39220]: 3.0 scrub ok 2024-09-10T18:44:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:29 smithi204 ceph-mon[32357]: 3.3 scrub starts 2024-09-10T18:44:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:29 smithi204 ceph-mon[32357]: 3.3 scrub ok 2024-09-10T18:44:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:29 smithi204 ceph-mon[32357]: 5.9 scrub starts 2024-09-10T18:44:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:29 smithi204 ceph-mon[32357]: 5.9 scrub ok 2024-09-10T18:44:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:29 smithi204 ceph-mon[32357]: 3.0 scrub starts 2024-09-10T18:44:29.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:29 smithi204 ceph-mon[32357]: 3.0 scrub ok 2024-09-10T18:44:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-10T18:44:30.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:30 smithi028 ceph-mon[29291]: pgmap v1035: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:44:30.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:30 smithi028 ceph-mon[29291]: 3.7 scrub starts 2024-09-10T18:44:30.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:30 smithi028 ceph-mon[29291]: 3.7 scrub ok 2024-09-10T18:44:30.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:30 smithi028 ceph-mon[29291]: 4.4 scrub starts 2024-09-10T18:44:30.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:30 smithi028 ceph-mon[29291]: 4.4 scrub ok 2024-09-10T18:44:30.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:30 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4060681802' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:44:30.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:30 smithi028 ceph-mon[39220]: pgmap v1035: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:44:30.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:30 smithi028 ceph-mon[39220]: 3.7 scrub starts 2024-09-10T18:44:30.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:30 smithi028 ceph-mon[39220]: 3.7 scrub ok 2024-09-10T18:44:30.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:30 smithi028 ceph-mon[39220]: 4.4 scrub starts 2024-09-10T18:44:30.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:30 smithi028 ceph-mon[39220]: 4.4 scrub ok 2024-09-10T18:44:30.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:30 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4060681802' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:44:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:30 smithi204 ceph-mon[32357]: pgmap v1035: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:44:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:30 smithi204 ceph-mon[32357]: 3.7 scrub starts 2024-09-10T18:44:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:30 smithi204 ceph-mon[32357]: 3.7 scrub ok 2024-09-10T18:44:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:30 smithi204 ceph-mon[32357]: 4.4 scrub starts 2024-09-10T18:44:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:30 smithi204 ceph-mon[32357]: 4.4 scrub ok 2024-09-10T18:44:30.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:30 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4060681802' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:44:31.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-10T18:44:31.373 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:31.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:31 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4060681802' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-10T18:44:31.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:31 smithi028 ceph-mon[29291]: osdmap e438: 8 total, 8 up, 8 in 2024-09-10T18:44:31.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:31 smithi028 ceph-mon[29291]: 5.f scrub starts 2024-09-10T18:44:31.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:31 smithi028 ceph-mon[29291]: 5.f scrub ok 2024-09-10T18:44:31.496 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:44:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:44:31] "GET /metrics HTTP/1.1" 200 222525 "" "Prometheus/2.43.0" 2024-09-10T18:44:31.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:31 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4060681802' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-10T18:44:31.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:31 smithi028 ceph-mon[39220]: osdmap e438: 8 total, 8 up, 8 in 2024-09-10T18:44:31.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:31 smithi028 ceph-mon[39220]: 5.f scrub starts 2024-09-10T18:44:31.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:31 smithi028 ceph-mon[39220]: 5.f scrub ok 2024-09-10T18:44:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:31 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4060681802' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-10T18:44:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:31 smithi204 ceph-mon[32357]: osdmap e438: 8 total, 8 up, 8 in 2024-09-10T18:44:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:31 smithi204 ceph-mon[32357]: 5.f scrub starts 2024-09-10T18:44:31.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:31 smithi204 ceph-mon[32357]: 5.f scrub ok 2024-09-10T18:44:32.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:32 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:44:32.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:32 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:44:32.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:32 smithi028 ceph-mon[29291]: 6.12 scrub starts 2024-09-10T18:44:32.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:32 smithi028 ceph-mon[29291]: 6.12 scrub ok 2024-09-10T18:44:32.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:32 smithi028 ceph-mon[29291]: pgmap v1037: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:44:32.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:32 smithi028 ceph-mon[29291]: osdmap e439: 8 total, 8 up, 8 in 2024-09-10T18:44:32.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:32 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:44:32.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:32 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:44:32.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:32 smithi028 ceph-mon[39220]: 6.12 scrub starts 2024-09-10T18:44:32.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:32 smithi028 ceph-mon[39220]: 6.12 scrub ok 2024-09-10T18:44:32.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:32 smithi028 ceph-mon[39220]: pgmap v1037: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:44:32.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:32 smithi028 ceph-mon[39220]: osdmap e439: 8 total, 8 up, 8 in 2024-09-10T18:44:32.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:44:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:44:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:44:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:32 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:44:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:32 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:44:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:32 smithi204 ceph-mon[32357]: 6.12 scrub starts 2024-09-10T18:44:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:32 smithi204 ceph-mon[32357]: 6.12 scrub ok 2024-09-10T18:44:32.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:32 smithi204 ceph-mon[32357]: pgmap v1037: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:44:32.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:32 smithi204 ceph-mon[32357]: osdmap e439: 8 total, 8 up, 8 in 2024-09-10T18:44:33.334 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.3 to 1 (10000) 2024-09-10T18:44:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:33 smithi204 ceph-mon[32357]: 4.13 scrub starts 2024-09-10T18:44:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:33 smithi204 ceph-mon[32357]: 4.13 scrub ok 2024-09-10T18:44:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:33 smithi204 ceph-mon[32357]: 6.10 deep-scrub starts 2024-09-10T18:44:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:33 smithi204 ceph-mon[32357]: 6.10 deep-scrub ok 2024-09-10T18:44:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:33 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3812014229' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:44:33.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:33 smithi204 ceph-mon[32357]: from='client.36170 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:44:33.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:33 smithi028 ceph-mon[29291]: 4.13 scrub starts 2024-09-10T18:44:33.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:33 smithi028 ceph-mon[29291]: 4.13 scrub ok 2024-09-10T18:44:33.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:33 smithi028 ceph-mon[29291]: 6.10 deep-scrub starts 2024-09-10T18:44:33.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:33 smithi028 ceph-mon[29291]: 6.10 deep-scrub ok 2024-09-10T18:44:33.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:33 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3812014229' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:44:33.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:33 smithi028 ceph-mon[29291]: from='client.36170 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:44:33.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:33 smithi028 ceph-mon[39220]: 4.13 scrub starts 2024-09-10T18:44:33.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:33 smithi028 ceph-mon[39220]: 4.13 scrub ok 2024-09-10T18:44:33.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:33 smithi028 ceph-mon[39220]: 6.10 deep-scrub starts 2024-09-10T18:44:33.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:33 smithi028 ceph-mon[39220]: 6.10 deep-scrub ok 2024-09-10T18:44:33.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:33 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3812014229' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:44:33.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:33 smithi028 ceph-mon[39220]: from='client.36170 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:44:34.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-10T18:44:34.386 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:34.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:34 smithi028 ceph-mon[29291]: pgmap v1039: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:34.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:34 smithi028 ceph-mon[29291]: from='client.36170 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T18:44:34.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:34 smithi028 ceph-mon[29291]: osdmap e440: 8 total, 8 up, 8 in 2024-09-10T18:44:34.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:34 smithi028 ceph-mon[29291]: 1.8 scrub starts 2024-09-10T18:44:34.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:34 smithi028 ceph-mon[29291]: 1.8 scrub ok 2024-09-10T18:44:34.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:34 smithi028 ceph-mon[39220]: pgmap v1039: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:34.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:34 smithi028 ceph-mon[39220]: from='client.36170 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T18:44:34.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:34 smithi028 ceph-mon[39220]: osdmap e440: 8 total, 8 up, 8 in 2024-09-10T18:44:34.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:34 smithi028 ceph-mon[39220]: 1.8 scrub starts 2024-09-10T18:44:34.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:34 smithi028 ceph-mon[39220]: 1.8 scrub ok 2024-09-10T18:44:34.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:34 smithi204 ceph-mon[32357]: pgmap v1039: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:34.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:34 smithi204 ceph-mon[32357]: from='client.36170 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T18:44:34.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:34 smithi204 ceph-mon[32357]: osdmap e440: 8 total, 8 up, 8 in 2024-09-10T18:44:34.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:34 smithi204 ceph-mon[32357]: 1.8 scrub starts 2024-09-10T18:44:34.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:34 smithi204 ceph-mon[32357]: 1.8 scrub ok 2024-09-10T18:44:35.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:44:35 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:44:35.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:35 smithi204 ceph-mon[32357]: 3.1a deep-scrub starts 2024-09-10T18:44:35.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:35 smithi204 ceph-mon[32357]: 3.1a deep-scrub ok 2024-09-10T18:44:35.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:35 smithi204 ceph-mon[32357]: 3.1b deep-scrub starts 2024-09-10T18:44:35.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:35 smithi204 ceph-mon[32357]: 3.1b deep-scrub ok 2024-09-10T18:44:35.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:35 smithi204 ceph-mon[32357]: 5.1d deep-scrub starts 2024-09-10T18:44:35.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:35 smithi204 ceph-mon[32357]: 5.1d deep-scrub ok 2024-09-10T18:44:35.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:35 smithi028 ceph-mon[39220]: 3.1a deep-scrub starts 2024-09-10T18:44:35.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:35 smithi028 ceph-mon[39220]: 3.1a deep-scrub ok 2024-09-10T18:44:35.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:35 smithi028 ceph-mon[39220]: 3.1b deep-scrub starts 2024-09-10T18:44:35.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:35 smithi028 ceph-mon[39220]: 3.1b deep-scrub ok 2024-09-10T18:44:35.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:35 smithi028 ceph-mon[39220]: 5.1d deep-scrub starts 2024-09-10T18:44:35.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:35 smithi028 ceph-mon[39220]: 5.1d deep-scrub ok 2024-09-10T18:44:35.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:35 smithi028 ceph-mon[29291]: 3.1a deep-scrub starts 2024-09-10T18:44:35.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:35 smithi028 ceph-mon[29291]: 3.1a deep-scrub ok 2024-09-10T18:44:35.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:35 smithi028 ceph-mon[29291]: 3.1b deep-scrub starts 2024-09-10T18:44:35.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:35 smithi028 ceph-mon[29291]: 3.1b deep-scrub ok 2024-09-10T18:44:35.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:35 smithi028 ceph-mon[29291]: 5.1d deep-scrub starts 2024-09-10T18:44:35.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:35 smithi028 ceph-mon[29291]: 5.1d deep-scrub ok 2024-09-10T18:44:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:36 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:36 smithi204 ceph-mon[32357]: pgmap v1041: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:36 smithi204 ceph-mon[32357]: 6.1 scrub starts 2024-09-10T18:44:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:36 smithi204 ceph-mon[32357]: 6.1 scrub ok 2024-09-10T18:44:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:36 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/250309831' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:44:36.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:36 smithi204 ceph-mon[32357]: from='client.36176 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:44:36.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:36 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:36.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:36 smithi028 ceph-mon[29291]: pgmap v1041: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:36.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:36 smithi028 ceph-mon[29291]: 6.1 scrub starts 2024-09-10T18:44:36.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:36 smithi028 ceph-mon[29291]: 6.1 scrub ok 2024-09-10T18:44:36.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:36 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/250309831' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:44:36.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:36 smithi028 ceph-mon[29291]: from='client.36176 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:44:36.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:36 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:36.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:36 smithi028 ceph-mon[39220]: pgmap v1041: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:36.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:36 smithi028 ceph-mon[39220]: 6.1 scrub starts 2024-09-10T18:44:36.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:36 smithi028 ceph-mon[39220]: 6.1 scrub ok 2024-09-10T18:44:36.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:36 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/250309831' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:44:36.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:36 smithi028 ceph-mon[39220]: from='client.36176 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:44:37.013 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-10T18:44:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:37 smithi204 ceph-mon[32357]: 4.18 scrub starts 2024-09-10T18:44:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:37 smithi204 ceph-mon[32357]: 4.18 scrub ok 2024-09-10T18:44:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:37 smithi204 ceph-mon[32357]: 4.10 scrub starts 2024-09-10T18:44:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:37 smithi204 ceph-mon[32357]: 4.10 scrub ok 2024-09-10T18:44:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:37 smithi204 ceph-mon[32357]: from='client.36176 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-10T18:44:37.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:37 smithi204 ceph-mon[32357]: osdmap e441: 8 total, 8 up, 8 in 2024-09-10T18:44:37.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:37 smithi028 ceph-mon[39220]: 4.18 scrub starts 2024-09-10T18:44:37.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:37 smithi028 ceph-mon[39220]: 4.18 scrub ok 2024-09-10T18:44:37.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:37 smithi028 ceph-mon[39220]: 4.10 scrub starts 2024-09-10T18:44:37.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:37 smithi028 ceph-mon[39220]: 4.10 scrub ok 2024-09-10T18:44:37.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:37 smithi028 ceph-mon[39220]: from='client.36176 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-10T18:44:37.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:37 smithi028 ceph-mon[39220]: osdmap e441: 8 total, 8 up, 8 in 2024-09-10T18:44:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:37 smithi028 ceph-mon[29291]: 4.18 scrub starts 2024-09-10T18:44:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:37 smithi028 ceph-mon[29291]: 4.18 scrub ok 2024-09-10T18:44:37.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:37 smithi028 ceph-mon[29291]: 4.10 scrub starts 2024-09-10T18:44:37.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:37 smithi028 ceph-mon[29291]: 4.10 scrub ok 2024-09-10T18:44:37.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:37 smithi028 ceph-mon[29291]: from='client.36176 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-10T18:44:37.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:37 smithi028 ceph-mon[29291]: osdmap e441: 8 total, 8 up, 8 in 2024-09-10T18:44:37.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-10T18:44:38.079 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:38.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:38 smithi028 ceph-mon[29291]: 4.1a scrub starts 2024-09-10T18:44:38.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:38 smithi028 ceph-mon[29291]: 4.1a scrub ok 2024-09-10T18:44:38.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:38 smithi028 ceph-mon[29291]: 1.a scrub starts 2024-09-10T18:44:38.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:38 smithi028 ceph-mon[29291]: 1.a scrub ok 2024-09-10T18:44:38.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:38 smithi028 ceph-mon[29291]: pgmap v1043: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:38.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:38 smithi028 ceph-mon[39220]: 4.1a scrub starts 2024-09-10T18:44:38.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:38 smithi028 ceph-mon[39220]: 4.1a scrub ok 2024-09-10T18:44:38.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:38 smithi028 ceph-mon[39220]: 1.a scrub starts 2024-09-10T18:44:38.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:38 smithi028 ceph-mon[39220]: 1.a scrub ok 2024-09-10T18:44:38.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:38 smithi028 ceph-mon[39220]: pgmap v1043: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:38 smithi204 ceph-mon[32357]: 4.1a scrub starts 2024-09-10T18:44:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:38 smithi204 ceph-mon[32357]: 4.1a scrub ok 2024-09-10T18:44:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:38 smithi204 ceph-mon[32357]: 1.a scrub starts 2024-09-10T18:44:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:38 smithi204 ceph-mon[32357]: 1.a scrub ok 2024-09-10T18:44:38.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:38 smithi204 ceph-mon[32357]: pgmap v1043: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:39 smithi204 ceph-mon[32357]: 1.15 deep-scrub starts 2024-09-10T18:44:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:39 smithi204 ceph-mon[32357]: 1.15 deep-scrub ok 2024-09-10T18:44:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:39 smithi204 ceph-mon[32357]: 1.c scrub starts 2024-09-10T18:44:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:39 smithi204 ceph-mon[32357]: 1.c scrub ok 2024-09-10T18:44:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:44:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:39 smithi204 ceph-mon[32357]: 1.16 scrub starts 2024-09-10T18:44:39.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:39 smithi204 ceph-mon[32357]: 1.16 scrub ok 2024-09-10T18:44:39.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:39 smithi028 ceph-mon[29291]: 1.15 deep-scrub starts 2024-09-10T18:44:39.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:39 smithi028 ceph-mon[29291]: 1.15 deep-scrub ok 2024-09-10T18:44:39.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:39 smithi028 ceph-mon[29291]: 1.c scrub starts 2024-09-10T18:44:39.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:39 smithi028 ceph-mon[29291]: 1.c scrub ok 2024-09-10T18:44:39.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:44:39.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:39 smithi028 ceph-mon[29291]: 1.16 scrub starts 2024-09-10T18:44:39.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:39 smithi028 ceph-mon[29291]: 1.16 scrub ok 2024-09-10T18:44:39.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:39 smithi028 ceph-mon[39220]: 1.15 deep-scrub starts 2024-09-10T18:44:39.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:39 smithi028 ceph-mon[39220]: 1.15 deep-scrub ok 2024-09-10T18:44:39.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:39 smithi028 ceph-mon[39220]: 1.c scrub starts 2024-09-10T18:44:39.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:39 smithi028 ceph-mon[39220]: 1.c scrub ok 2024-09-10T18:44:39.663 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:44:39.663 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:39 smithi028 ceph-mon[39220]: 1.16 scrub starts 2024-09-10T18:44:39.663 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:39 smithi028 ceph-mon[39220]: 1.16 scrub ok 2024-09-10T18:44:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.4 to 1 (10000) 2024-09-10T18:44:40.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:40 smithi028 ceph-mon[29291]: pgmap v1044: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2024-09-10T18:44:40.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:40 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:44:40.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:40 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1105235259' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:44:40.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:40 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:40.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:40 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:44:40.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:40 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:44:40.650 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:40 smithi028 ceph-mon[39220]: pgmap v1044: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2024-09-10T18:44:40.650 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:40 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:44:40.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:40 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1105235259' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:44:40.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:40 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:40.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:40 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:44:40.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:40 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:44:40.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:40 smithi204 ceph-mon[32357]: pgmap v1044: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2024-09-10T18:44:40.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:40 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:44:40.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:40 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1105235259' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:44:40.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:40 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:40.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:40 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:44:40.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:40 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:44:41.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-10T18:44:41.357 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:44:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:44:41] "GET /metrics HTTP/1.1" 200 222505 "" "Prometheus/2.43.0" 2024-09-10T18:44:41.388 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:41.628 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:41 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1105235259' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T18:44:41.628 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:41 smithi028 ceph-mon[39220]: osdmap e442: 8 total, 8 up, 8 in 2024-09-10T18:44:41.628 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:41 smithi028 ceph-mon[39220]: 5.3 deep-scrub starts 2024-09-10T18:44:41.628 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:41 smithi028 ceph-mon[39220]: 5.3 deep-scrub ok 2024-09-10T18:44:41.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:41 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1105235259' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T18:44:41.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:41 smithi028 ceph-mon[29291]: osdmap e442: 8 total, 8 up, 8 in 2024-09-10T18:44:41.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:41 smithi028 ceph-mon[29291]: 5.3 deep-scrub starts 2024-09-10T18:44:41.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:41 smithi028 ceph-mon[29291]: 5.3 deep-scrub ok 2024-09-10T18:44:41.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:41 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1105235259' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T18:44:41.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:41 smithi204 ceph-mon[32357]: osdmap e442: 8 total, 8 up, 8 in 2024-09-10T18:44:41.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:41 smithi204 ceph-mon[32357]: 5.3 deep-scrub starts 2024-09-10T18:44:41.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:41 smithi204 ceph-mon[32357]: 5.3 deep-scrub ok 2024-09-10T18:44:42.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[29291]: 1.1c deep-scrub starts 2024-09-10T18:44:42.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[29291]: 1.1c deep-scrub ok 2024-09-10T18:44:42.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[29291]: 1.10 scrub starts 2024-09-10T18:44:42.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[29291]: 1.10 scrub ok 2024-09-10T18:44:42.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[29291]: pgmap v1046: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:44:42.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[29291]: 1.f deep-scrub starts 2024-09-10T18:44:42.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[29291]: 1.f deep-scrub ok 2024-09-10T18:44:42.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:44:42.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:44:42.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[39220]: 1.1c deep-scrub starts 2024-09-10T18:44:42.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[39220]: 1.1c deep-scrub ok 2024-09-10T18:44:42.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[39220]: 1.10 scrub starts 2024-09-10T18:44:42.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[39220]: 1.10 scrub ok 2024-09-10T18:44:42.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[39220]: pgmap v1046: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:44:42.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[39220]: 1.f deep-scrub starts 2024-09-10T18:44:42.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[39220]: 1.f deep-scrub ok 2024-09-10T18:44:42.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:44:42.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:42 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:44:42.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:44:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:44:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:44:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:42 smithi204 ceph-mon[32357]: 1.1c deep-scrub starts 2024-09-10T18:44:42.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:42 smithi204 ceph-mon[32357]: 1.1c deep-scrub ok 2024-09-10T18:44:42.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:42 smithi204 ceph-mon[32357]: 1.10 scrub starts 2024-09-10T18:44:42.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:42 smithi204 ceph-mon[32357]: 1.10 scrub ok 2024-09-10T18:44:42.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:42 smithi204 ceph-mon[32357]: pgmap v1046: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:44:42.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:42 smithi204 ceph-mon[32357]: 1.f deep-scrub starts 2024-09-10T18:44:42.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:42 smithi204 ceph-mon[32357]: 1.f deep-scrub ok 2024-09-10T18:44:42.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:42 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:44:42.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:42 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:44:43.371 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-10T18:44:43.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[29291]: 3.c scrub starts 2024-09-10T18:44:43.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[29291]: 3.c scrub ok 2024-09-10T18:44:43.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[29291]: osdmap e443: 8 total, 8 up, 8 in 2024-09-10T18:44:43.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:44:43.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:44:43.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[29291]: 1.11 scrub starts 2024-09-10T18:44:43.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[29291]: 1.11 scrub ok 2024-09-10T18:44:43.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2677144527' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:44:43.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[29291]: from='client.36188 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:44:43.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[39220]: 3.c scrub starts 2024-09-10T18:44:43.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[39220]: 3.c scrub ok 2024-09-10T18:44:43.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[39220]: osdmap e443: 8 total, 8 up, 8 in 2024-09-10T18:44:43.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:44:43.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:44:43.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[39220]: 1.11 scrub starts 2024-09-10T18:44:43.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[39220]: 1.11 scrub ok 2024-09-10T18:44:43.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2677144527' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:44:43.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:43 smithi028 ceph-mon[39220]: from='client.36188 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:44:43.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:43 smithi204 ceph-mon[32357]: 3.c scrub starts 2024-09-10T18:44:43.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:43 smithi204 ceph-mon[32357]: 3.c scrub ok 2024-09-10T18:44:43.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:43 smithi204 ceph-mon[32357]: osdmap e443: 8 total, 8 up, 8 in 2024-09-10T18:44:43.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:43 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:44:43.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:43 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:44:43.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:43 smithi204 ceph-mon[32357]: 1.11 scrub starts 2024-09-10T18:44:43.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:43 smithi204 ceph-mon[32357]: 1.11 scrub ok 2024-09-10T18:44:43.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:43 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2677144527' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:44:43.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:43 smithi204 ceph-mon[32357]: from='client.36188 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:44:44.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-10T18:44:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:44.631 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:44 smithi028 ceph-mon[39220]: pgmap v1048: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 819 B/s rd, 0 op/s 2024-09-10T18:44:44.631 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:44 smithi028 ceph-mon[39220]: from='client.36188 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-10T18:44:44.631 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:44 smithi028 ceph-mon[39220]: osdmap e444: 8 total, 8 up, 8 in 2024-09-10T18:44:44.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:44 smithi028 ceph-mon[29291]: pgmap v1048: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 819 B/s rd, 0 op/s 2024-09-10T18:44:44.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:44 smithi028 ceph-mon[29291]: from='client.36188 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-10T18:44:44.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:44 smithi028 ceph-mon[29291]: osdmap e444: 8 total, 8 up, 8 in 2024-09-10T18:44:44.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:44 smithi204 ceph-mon[32357]: pgmap v1048: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 819 B/s rd, 0 op/s 2024-09-10T18:44:44.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:44 smithi204 ceph-mon[32357]: from='client.36188 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-10T18:44:44.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:44 smithi204 ceph-mon[32357]: osdmap e444: 8 total, 8 up, 8 in 2024-09-10T18:44:45.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:45 smithi028 ceph-mon[29291]: 6.8 scrub starts 2024-09-10T18:44:45.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:45 smithi028 ceph-mon[29291]: 6.8 scrub ok 2024-09-10T18:44:45.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:45 smithi028 ceph-mon[39220]: 6.8 scrub starts 2024-09-10T18:44:45.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:45 smithi028 ceph-mon[39220]: 6.8 scrub ok 2024-09-10T18:44:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:45 smithi204 ceph-mon[32357]: 6.8 scrub starts 2024-09-10T18:44:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:45 smithi204 ceph-mon[32357]: 6.8 scrub ok 2024-09-10T18:44:45.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:44:45 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:44:46.379 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.5 to 1 (10000) 2024-09-10T18:44:46.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:46 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:46.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:46 smithi028 ceph-mon[29291]: pgmap v1050: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:46.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:46 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1959042724' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:44:46.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:46 smithi028 ceph-mon[29291]: from='client.36194 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:44:46.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:46 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:46.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:46 smithi028 ceph-mon[39220]: pgmap v1050: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:46.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:46 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1959042724' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:44:46.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:46 smithi028 ceph-mon[39220]: from='client.36194 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:44:46.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:46 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:46.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:46 smithi204 ceph-mon[32357]: pgmap v1050: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:46.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:46 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1959042724' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:44:46.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:46 smithi204 ceph-mon[32357]: from='client.36194 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:44:47.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-10T18:44:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:47.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[39220]: 1.e scrub starts 2024-09-10T18:44:47.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[39220]: 1.e scrub ok 2024-09-10T18:44:47.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[39220]: 6.5 scrub starts 2024-09-10T18:44:47.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[39220]: 6.5 scrub ok 2024-09-10T18:44:47.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[39220]: from='client.36194 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T18:44:47.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[39220]: osdmap e445: 8 total, 8 up, 8 in 2024-09-10T18:44:47.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[39220]: 6.18 scrub starts 2024-09-10T18:44:47.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[39220]: 6.18 scrub ok 2024-09-10T18:44:47.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[39220]: 4.17 scrub starts 2024-09-10T18:44:47.673 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[39220]: 4.17 scrub ok 2024-09-10T18:44:47.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[29291]: 1.e scrub starts 2024-09-10T18:44:47.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[29291]: 1.e scrub ok 2024-09-10T18:44:47.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[29291]: 6.5 scrub starts 2024-09-10T18:44:47.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[29291]: 6.5 scrub ok 2024-09-10T18:44:47.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[29291]: from='client.36194 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T18:44:47.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[29291]: osdmap e445: 8 total, 8 up, 8 in 2024-09-10T18:44:47.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[29291]: 6.18 scrub starts 2024-09-10T18:44:47.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[29291]: 6.18 scrub ok 2024-09-10T18:44:47.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[29291]: 4.17 scrub starts 2024-09-10T18:44:47.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:47 smithi028 ceph-mon[29291]: 4.17 scrub ok 2024-09-10T18:44:47.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:47 smithi204 ceph-mon[32357]: 1.e scrub starts 2024-09-10T18:44:47.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:47 smithi204 ceph-mon[32357]: 1.e scrub ok 2024-09-10T18:44:47.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:47 smithi204 ceph-mon[32357]: 6.5 scrub starts 2024-09-10T18:44:47.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:47 smithi204 ceph-mon[32357]: 6.5 scrub ok 2024-09-10T18:44:47.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:47 smithi204 ceph-mon[32357]: from='client.36194 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T18:44:47.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:47 smithi204 ceph-mon[32357]: osdmap e445: 8 total, 8 up, 8 in 2024-09-10T18:44:47.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:47 smithi204 ceph-mon[32357]: 6.18 scrub starts 2024-09-10T18:44:47.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:47 smithi204 ceph-mon[32357]: 6.18 scrub ok 2024-09-10T18:44:47.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:47 smithi204 ceph-mon[32357]: 4.17 scrub starts 2024-09-10T18:44:47.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:47 smithi204 ceph-mon[32357]: 4.17 scrub ok 2024-09-10T18:44:48.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:48 smithi028 ceph-mon[29291]: 1.18 scrub starts 2024-09-10T18:44:48.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:48 smithi028 ceph-mon[29291]: 1.18 scrub ok 2024-09-10T18:44:48.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:48 smithi028 ceph-mon[29291]: pgmap v1052: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:48.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:48 smithi028 ceph-mon[29291]: 5.5 scrub starts 2024-09-10T18:44:48.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:48 smithi028 ceph-mon[29291]: 5.5 scrub ok 2024-09-10T18:44:48.652 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:48 smithi028 ceph-mon[39220]: 1.18 scrub starts 2024-09-10T18:44:48.652 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:48 smithi028 ceph-mon[39220]: 1.18 scrub ok 2024-09-10T18:44:48.652 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:48 smithi028 ceph-mon[39220]: pgmap v1052: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:48.652 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:48 smithi028 ceph-mon[39220]: 5.5 scrub starts 2024-09-10T18:44:48.652 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:48 smithi028 ceph-mon[39220]: 5.5 scrub ok 2024-09-10T18:44:48.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:48 smithi204 ceph-mon[32357]: 1.18 scrub starts 2024-09-10T18:44:48.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:48 smithi204 ceph-mon[32357]: 1.18 scrub ok 2024-09-10T18:44:48.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:48 smithi204 ceph-mon[32357]: pgmap v1052: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:44:48.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:48 smithi204 ceph-mon[32357]: 5.5 scrub starts 2024-09-10T18:44:48.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:48 smithi204 ceph-mon[32357]: 5.5 scrub ok 2024-09-10T18:44:49.391 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-10T18:44:49.743 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:49 smithi028 ceph-mon[29291]: 1.12 scrub starts 2024-09-10T18:44:49.744 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:49 smithi028 ceph-mon[29291]: 1.12 scrub ok 2024-09-10T18:44:49.744 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:49 smithi028 ceph-mon[29291]: 5.1e scrub starts 2024-09-10T18:44:49.744 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:49 smithi028 ceph-mon[29291]: 5.1e scrub ok 2024-09-10T18:44:49.744 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:49 smithi028 ceph-mon[29291]: 1.1e deep-scrub starts 2024-09-10T18:44:49.744 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:49 smithi028 ceph-mon[29291]: 1.1e deep-scrub ok 2024-09-10T18:44:49.744 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:49 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2681889481' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:44:49.744 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:49 smithi028 ceph-mon[39220]: 1.12 scrub starts 2024-09-10T18:44:49.744 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:49 smithi028 ceph-mon[39220]: 1.12 scrub ok 2024-09-10T18:44:49.744 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:49 smithi028 ceph-mon[39220]: 5.1e scrub starts 2024-09-10T18:44:49.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:49 smithi028 ceph-mon[39220]: 5.1e scrub ok 2024-09-10T18:44:49.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:49 smithi028 ceph-mon[39220]: 1.1e deep-scrub starts 2024-09-10T18:44:49.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:49 smithi028 ceph-mon[39220]: 1.1e deep-scrub ok 2024-09-10T18:44:49.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:49 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2681889481' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:44:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:49 smithi204 ceph-mon[32357]: 1.12 scrub starts 2024-09-10T18:44:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:49 smithi204 ceph-mon[32357]: 1.12 scrub ok 2024-09-10T18:44:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:49 smithi204 ceph-mon[32357]: 5.1e scrub starts 2024-09-10T18:44:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:49 smithi204 ceph-mon[32357]: 5.1e scrub ok 2024-09-10T18:44:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:49 smithi204 ceph-mon[32357]: 1.1e deep-scrub starts 2024-09-10T18:44:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:49 smithi204 ceph-mon[32357]: 1.1e deep-scrub ok 2024-09-10T18:44:49.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:49 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2681889481' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:44:50.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-10T18:44:50.321 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:50.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:50 smithi028 ceph-mon[39220]: pgmap v1053: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-09-10T18:44:50.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:50 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2681889481' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-10T18:44:50.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:50 smithi028 ceph-mon[39220]: osdmap e446: 8 total, 8 up, 8 in 2024-09-10T18:44:50.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:50 smithi028 ceph-mon[29291]: pgmap v1053: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-09-10T18:44:50.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:50 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2681889481' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-10T18:44:50.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:50 smithi028 ceph-mon[29291]: osdmap e446: 8 total, 8 up, 8 in 2024-09-10T18:44:50.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:50 smithi204 ceph-mon[32357]: pgmap v1053: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-09-10T18:44:50.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:50 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2681889481' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-10T18:44:50.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:50 smithi204 ceph-mon[32357]: osdmap e446: 8 total, 8 up, 8 in 2024-09-10T18:44:51.586 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:44:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:44:51] "GET /metrics HTTP/1.1" 200 222505 "" "Prometheus/2.43.0" 2024-09-10T18:44:51.586 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:51 smithi028 ceph-mon[29291]: 6.e scrub starts 2024-09-10T18:44:51.587 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:51 smithi028 ceph-mon[29291]: 6.e scrub ok 2024-09-10T18:44:51.587 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:51 smithi028 ceph-mon[29291]: 6.3 scrub starts 2024-09-10T18:44:51.587 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:51 smithi028 ceph-mon[29291]: 6.3 scrub ok 2024-09-10T18:44:51.587 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:51 smithi028 ceph-mon[29291]: 5.1f deep-scrub starts 2024-09-10T18:44:51.587 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:51 smithi028 ceph-mon[29291]: 5.1f deep-scrub ok 2024-09-10T18:44:51.587 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:51 smithi028 ceph-mon[39220]: 6.e scrub starts 2024-09-10T18:44:51.587 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:51 smithi028 ceph-mon[39220]: 6.e scrub ok 2024-09-10T18:44:51.587 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:51 smithi028 ceph-mon[39220]: 6.3 scrub starts 2024-09-10T18:44:51.587 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:51 smithi028 ceph-mon[39220]: 6.3 scrub ok 2024-09-10T18:44:51.587 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:51 smithi028 ceph-mon[39220]: 5.1f deep-scrub starts 2024-09-10T18:44:51.587 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:51 smithi028 ceph-mon[39220]: 5.1f deep-scrub ok 2024-09-10T18:44:51.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:51 smithi204 ceph-mon[32357]: 6.e scrub starts 2024-09-10T18:44:51.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:51 smithi204 ceph-mon[32357]: 6.e scrub ok 2024-09-10T18:44:51.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:51 smithi204 ceph-mon[32357]: 6.3 scrub starts 2024-09-10T18:44:51.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:51 smithi204 ceph-mon[32357]: 6.3 scrub ok 2024-09-10T18:44:51.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:51 smithi204 ceph-mon[32357]: 5.1f deep-scrub starts 2024-09-10T18:44:51.855 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:51 smithi204 ceph-mon[32357]: 5.1f deep-scrub ok 2024-09-10T18:44:52.399 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:44:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:44:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:44:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:52 smithi028 ceph-mon[29291]: 3.d scrub starts 2024-09-10T18:44:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:52 smithi028 ceph-mon[29291]: 3.d scrub ok 2024-09-10T18:44:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:52 smithi028 ceph-mon[29291]: pgmap v1055: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:44:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:52 smithi028 ceph-mon[29291]: 4.19 scrub starts 2024-09-10T18:44:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:52 smithi028 ceph-mon[29291]: 4.19 scrub ok 2024-09-10T18:44:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:52 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/1323049392' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:44:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:52 smithi028 ceph-mon[29291]: from='client.36206 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:44:52.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:52 smithi028 ceph-mon[39220]: 3.d scrub starts 2024-09-10T18:44:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:52 smithi028 ceph-mon[39220]: 3.d scrub ok 2024-09-10T18:44:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:52 smithi028 ceph-mon[39220]: pgmap v1055: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:44:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:52 smithi028 ceph-mon[39220]: 4.19 scrub starts 2024-09-10T18:44:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:52 smithi028 ceph-mon[39220]: 4.19 scrub ok 2024-09-10T18:44:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:52 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/1323049392' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:44:52.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:52 smithi028 ceph-mon[39220]: from='client.36206 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:44:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:52 smithi204 ceph-mon[32357]: 3.d scrub starts 2024-09-10T18:44:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:52 smithi204 ceph-mon[32357]: 3.d scrub ok 2024-09-10T18:44:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:52 smithi204 ceph-mon[32357]: pgmap v1055: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:44:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:52 smithi204 ceph-mon[32357]: 4.19 scrub starts 2024-09-10T18:44:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:52 smithi204 ceph-mon[32357]: 4.19 scrub ok 2024-09-10T18:44:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:52 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/1323049392' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:44:52.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:52 smithi204 ceph-mon[32357]: from='client.36206 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:44:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.7 to 1 (10000) 2024-09-10T18:44:53.662 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:53 smithi204 ceph-mon[32357]: 2.0 scrub starts 2024-09-10T18:44:53.662 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:53 smithi204 ceph-mon[32357]: 2.0 scrub ok 2024-09-10T18:44:53.663 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:53 smithi204 ceph-mon[32357]: 3.12 scrub starts 2024-09-10T18:44:53.663 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:53 smithi204 ceph-mon[32357]: 3.12 scrub ok 2024-09-10T18:44:53.663 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:53 smithi204 ceph-mon[32357]: 1.1 scrub starts 2024-09-10T18:44:53.663 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:53 smithi204 ceph-mon[32357]: 1.1 scrub ok 2024-09-10T18:44:53.663 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:53 smithi204 ceph-mon[32357]: 5.b scrub starts 2024-09-10T18:44:53.663 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:53 smithi204 ceph-mon[32357]: 5.b scrub ok 2024-09-10T18:44:53.663 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:53 smithi204 ceph-mon[32357]: from='client.36206 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T18:44:53.663 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:53 smithi204 ceph-mon[32357]: osdmap e447: 8 total, 8 up, 8 in 2024-09-10T18:44:53.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[39220]: 2.0 scrub starts 2024-09-10T18:44:53.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[39220]: 2.0 scrub ok 2024-09-10T18:44:53.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[39220]: 3.12 scrub starts 2024-09-10T18:44:53.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[39220]: 3.12 scrub ok 2024-09-10T18:44:53.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[39220]: 1.1 scrub starts 2024-09-10T18:44:53.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[39220]: 1.1 scrub ok 2024-09-10T18:44:53.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[39220]: 5.b scrub starts 2024-09-10T18:44:53.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[39220]: 5.b scrub ok 2024-09-10T18:44:53.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[39220]: from='client.36206 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T18:44:53.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[39220]: osdmap e447: 8 total, 8 up, 8 in 2024-09-10T18:44:53.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[29291]: 2.0 scrub starts 2024-09-10T18:44:53.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[29291]: 2.0 scrub ok 2024-09-10T18:44:53.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[29291]: 3.12 scrub starts 2024-09-10T18:44:53.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[29291]: 3.12 scrub ok 2024-09-10T18:44:53.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[29291]: 1.1 scrub starts 2024-09-10T18:44:53.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[29291]: 1.1 scrub ok 2024-09-10T18:44:53.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[29291]: 5.b scrub starts 2024-09-10T18:44:53.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[29291]: 5.b scrub ok 2024-09-10T18:44:53.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[29291]: from='client.36206 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T18:44:53.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:53 smithi028 ceph-mon[29291]: osdmap e447: 8 total, 8 up, 8 in 2024-09-10T18:44:53.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-10T18:44:53.990 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:54 smithi028 ceph-mon[39220]: 4.e scrub starts 2024-09-10T18:44:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:54 smithi028 ceph-mon[39220]: 4.e scrub ok 2024-09-10T18:44:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:54 smithi028 ceph-mon[39220]: pgmap v1057: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-09-10T18:44:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:54 smithi028 ceph-mon[39220]: 6.16 scrub starts 2024-09-10T18:44:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:54 smithi028 ceph-mon[39220]: 6.16 scrub ok 2024-09-10T18:44:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:54 smithi028 ceph-mon[39220]: 4.14 scrub starts 2024-09-10T18:44:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:54 smithi028 ceph-mon[39220]: 4.14 scrub ok 2024-09-10T18:44:54.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:54 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:44:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:54 smithi028 ceph-mon[29291]: 4.e scrub starts 2024-09-10T18:44:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:54 smithi028 ceph-mon[29291]: 4.e scrub ok 2024-09-10T18:44:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:54 smithi028 ceph-mon[29291]: pgmap v1057: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-09-10T18:44:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:54 smithi028 ceph-mon[29291]: 6.16 scrub starts 2024-09-10T18:44:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:54 smithi028 ceph-mon[29291]: 6.16 scrub ok 2024-09-10T18:44:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:54 smithi028 ceph-mon[29291]: 4.14 scrub starts 2024-09-10T18:44:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:54 smithi028 ceph-mon[29291]: 4.14 scrub ok 2024-09-10T18:44:54.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:54 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:44:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:54 smithi204 ceph-mon[32357]: 4.e scrub starts 2024-09-10T18:44:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:54 smithi204 ceph-mon[32357]: 4.e scrub ok 2024-09-10T18:44:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:54 smithi204 ceph-mon[32357]: pgmap v1057: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-09-10T18:44:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:54 smithi204 ceph-mon[32357]: 6.16 scrub starts 2024-09-10T18:44:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:54 smithi204 ceph-mon[32357]: 6.16 scrub ok 2024-09-10T18:44:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:54 smithi204 ceph-mon[32357]: 4.14 scrub starts 2024-09-10T18:44:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:54 smithi204 ceph-mon[32357]: 4.14 scrub ok 2024-09-10T18:44:54.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:54 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:44:55.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:55 smithi028 ceph-mon[29291]: 1.5 scrub starts 2024-09-10T18:44:55.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:55 smithi028 ceph-mon[29291]: 1.5 scrub ok 2024-09-10T18:44:55.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:55 smithi028 ceph-mon[39220]: 1.5 scrub starts 2024-09-10T18:44:55.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:55 smithi028 ceph-mon[39220]: 1.5 scrub ok 2024-09-10T18:44:55.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:55 smithi204 ceph-mon[32357]: 1.5 scrub starts 2024-09-10T18:44:55.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:55 smithi204 ceph-mon[32357]: 1.5 scrub ok 2024-09-10T18:44:55.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:44:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:44:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-10T18:44:56.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:56 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:56.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:56 smithi028 ceph-mon[39220]: pgmap v1058: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:44:56.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:56 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/134543979' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:44:56.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:56 smithi028 ceph-mon[39220]: from='client.36212 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:44:56.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:56 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:56.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:56 smithi028 ceph-mon[29291]: pgmap v1058: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:44:56.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:56 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/134543979' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:44:56.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:56 smithi028 ceph-mon[29291]: from='client.36212 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:44:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:56 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:44:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:56 smithi204 ceph-mon[32357]: pgmap v1058: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:44:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:56 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/134543979' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:44:56.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:56 smithi204 ceph-mon[32357]: from='client.36212 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:44:57.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-10T18:44:57.404 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:57.673 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:57 smithi028 ceph-mon[39220]: from='client.36212 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-10T18:44:57.673 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:57 smithi028 ceph-mon[39220]: osdmap e448: 8 total, 8 up, 8 in 2024-09-10T18:44:57.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:57 smithi028 ceph-mon[29291]: from='client.36212 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-10T18:44:57.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:57 smithi028 ceph-mon[29291]: osdmap e448: 8 total, 8 up, 8 in 2024-09-10T18:44:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:57 smithi204 ceph-mon[32357]: from='client.36212 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-10T18:44:57.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:57 smithi204 ceph-mon[32357]: osdmap e448: 8 total, 8 up, 8 in 2024-09-10T18:44:58.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:58 smithi028 ceph-mon[29291]: 6.1e scrub starts 2024-09-10T18:44:58.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:58 smithi028 ceph-mon[29291]: 6.1e scrub ok 2024-09-10T18:44:58.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:58 smithi028 ceph-mon[29291]: pgmap v1060: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:44:58.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:58 smithi028 ceph-mon[29291]: 6.17 scrub starts 2024-09-10T18:44:58.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:58 smithi028 ceph-mon[29291]: 6.17 scrub ok 2024-09-10T18:44:58.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:58 smithi028 ceph-mon[39220]: 6.1e scrub starts 2024-09-10T18:44:58.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:58 smithi028 ceph-mon[39220]: 6.1e scrub ok 2024-09-10T18:44:58.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:58 smithi028 ceph-mon[39220]: pgmap v1060: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:44:58.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:58 smithi028 ceph-mon[39220]: 6.17 scrub starts 2024-09-10T18:44:58.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:58 smithi028 ceph-mon[39220]: 6.17 scrub ok 2024-09-10T18:44:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:58 smithi204 ceph-mon[32357]: 6.1e scrub starts 2024-09-10T18:44:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:58 smithi204 ceph-mon[32357]: 6.1e scrub ok 2024-09-10T18:44:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:58 smithi204 ceph-mon[32357]: pgmap v1060: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:44:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:58 smithi204 ceph-mon[32357]: 6.17 scrub starts 2024-09-10T18:44:58.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:58 smithi204 ceph-mon[32357]: 6.17 scrub ok 2024-09-10T18:44:59.424 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.6 to 1 (10000) 2024-09-10T18:44:59.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:59 smithi028 ceph-mon[39220]: 1.1b scrub starts 2024-09-10T18:44:59.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:59 smithi028 ceph-mon[39220]: 1.1b scrub ok 2024-09-10T18:44:59.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:59 smithi028 ceph-mon[39220]: 6.4 scrub starts 2024-09-10T18:44:59.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:59 smithi028 ceph-mon[39220]: 6.4 scrub ok 2024-09-10T18:44:59.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:59 smithi028 ceph-mon[39220]: 5.4 scrub starts 2024-09-10T18:44:59.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:59 smithi028 ceph-mon[39220]: 5.4 scrub ok 2024-09-10T18:44:59.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:44:59 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/280742316' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:44:59.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:59 smithi028 ceph-mon[29291]: 1.1b scrub starts 2024-09-10T18:44:59.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:59 smithi028 ceph-mon[29291]: 1.1b scrub ok 2024-09-10T18:44:59.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:59 smithi028 ceph-mon[29291]: 6.4 scrub starts 2024-09-10T18:44:59.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:59 smithi028 ceph-mon[29291]: 6.4 scrub ok 2024-09-10T18:44:59.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:59 smithi028 ceph-mon[29291]: 5.4 scrub starts 2024-09-10T18:44:59.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:59 smithi028 ceph-mon[29291]: 5.4 scrub ok 2024-09-10T18:44:59.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:44:59 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/280742316' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:44:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:59 smithi204 ceph-mon[32357]: 1.1b scrub starts 2024-09-10T18:44:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:59 smithi204 ceph-mon[32357]: 1.1b scrub ok 2024-09-10T18:44:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:59 smithi204 ceph-mon[32357]: 6.4 scrub starts 2024-09-10T18:44:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:59 smithi204 ceph-mon[32357]: 6.4 scrub ok 2024-09-10T18:44:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:59 smithi204 ceph-mon[32357]: 5.4 scrub starts 2024-09-10T18:44:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:59 smithi204 ceph-mon[32357]: 5.4 scrub ok 2024-09-10T18:44:59.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:44:59 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/280742316' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:45:00.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-10T18:45:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:00 smithi028 ceph-mon[39220]: 1.17 scrub starts 2024-09-10T18:45:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:00 smithi028 ceph-mon[39220]: 1.17 scrub ok 2024-09-10T18:45:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:00 smithi028 ceph-mon[39220]: pgmap v1061: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:45:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:00 smithi028 ceph-mon[39220]: 3.4 deep-scrub starts 2024-09-10T18:45:00.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:00 smithi028 ceph-mon[39220]: 3.4 deep-scrub ok 2024-09-10T18:45:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:00 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/280742316' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T18:45:00.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:00 smithi028 ceph-mon[39220]: osdmap e449: 8 total, 8 up, 8 in 2024-09-10T18:45:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:00 smithi028 ceph-mon[29291]: 1.17 scrub starts 2024-09-10T18:45:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:00 smithi028 ceph-mon[29291]: 1.17 scrub ok 2024-09-10T18:45:00.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:00 smithi028 ceph-mon[29291]: pgmap v1061: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:45:00.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:00 smithi028 ceph-mon[29291]: 3.4 deep-scrub starts 2024-09-10T18:45:00.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:00 smithi028 ceph-mon[29291]: 3.4 deep-scrub ok 2024-09-10T18:45:00.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:00 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/280742316' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T18:45:00.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:00 smithi028 ceph-mon[29291]: osdmap e449: 8 total, 8 up, 8 in 2024-09-10T18:45:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:00 smithi204 ceph-mon[32357]: 1.17 scrub starts 2024-09-10T18:45:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:00 smithi204 ceph-mon[32357]: 1.17 scrub ok 2024-09-10T18:45:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:00 smithi204 ceph-mon[32357]: pgmap v1061: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:45:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:00 smithi204 ceph-mon[32357]: 3.4 deep-scrub starts 2024-09-10T18:45:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:00 smithi204 ceph-mon[32357]: 3.4 deep-scrub ok 2024-09-10T18:45:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:00 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/280742316' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T18:45:00.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:00 smithi204 ceph-mon[32357]: osdmap e449: 8 total, 8 up, 8 in 2024-09-10T18:45:01.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:01 smithi028 ceph-mon[29291]: 4.9 scrub starts 2024-09-10T18:45:01.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:01 smithi028 ceph-mon[29291]: 4.9 scrub ok 2024-09-10T18:45:01.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:01 smithi028 ceph-mon[29291]: 5.16 scrub starts 2024-09-10T18:45:01.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:01 smithi028 ceph-mon[29291]: 5.16 scrub ok 2024-09-10T18:45:01.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:45:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:45:01] "GET /metrics HTTP/1.1" 200 222492 "" "Prometheus/2.43.0" 2024-09-10T18:45:01.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:01 smithi028 ceph-mon[39220]: 4.9 scrub starts 2024-09-10T18:45:01.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:01 smithi028 ceph-mon[39220]: 4.9 scrub ok 2024-09-10T18:45:01.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:01 smithi028 ceph-mon[39220]: 5.16 scrub starts 2024-09-10T18:45:01.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:01 smithi028 ceph-mon[39220]: 5.16 scrub ok 2024-09-10T18:45:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:01 smithi204 ceph-mon[32357]: 4.9 scrub starts 2024-09-10T18:45:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:01 smithi204 ceph-mon[32357]: 4.9 scrub ok 2024-09-10T18:45:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:01 smithi204 ceph-mon[32357]: 5.16 scrub starts 2024-09-10T18:45:01.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:01 smithi204 ceph-mon[32357]: 5.16 scrub ok 2024-09-10T18:45:02.429 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:45:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:45:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:45:02.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:02 smithi028 ceph-mon[29291]: 5.15 scrub starts 2024-09-10T18:45:02.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:02 smithi028 ceph-mon[29291]: 5.15 scrub ok 2024-09-10T18:45:02.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:02 smithi028 ceph-mon[29291]: pgmap v1063: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:45:02.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:02 smithi028 ceph-mon[39220]: 5.15 scrub starts 2024-09-10T18:45:02.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:02 smithi028 ceph-mon[39220]: 5.15 scrub ok 2024-09-10T18:45:02.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:02 smithi028 ceph-mon[39220]: pgmap v1063: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:45:02.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:02 smithi204 ceph-mon[32357]: 5.15 scrub starts 2024-09-10T18:45:02.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:02 smithi204 ceph-mon[32357]: 5.15 scrub ok 2024-09-10T18:45:02.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:02 smithi204 ceph-mon[32357]: pgmap v1063: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:45:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-10T18:45:03.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:03 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/729590821' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:45:03.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:03 smithi028 ceph-mon[39220]: from='client.36224 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:45:03.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:03 smithi028 ceph-mon[39220]: 1.0 deep-scrub starts 2024-09-10T18:45:03.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:03 smithi028 ceph-mon[39220]: 1.0 deep-scrub ok 2024-09-10T18:45:03.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:03 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/729590821' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:45:03.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:03 smithi028 ceph-mon[29291]: from='client.36224 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:45:03.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:03 smithi028 ceph-mon[29291]: 1.0 deep-scrub starts 2024-09-10T18:45:03.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:03 smithi028 ceph-mon[29291]: 1.0 deep-scrub ok 2024-09-10T18:45:03.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:03 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/729590821' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:45:03.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:03 smithi204 ceph-mon[32357]: from='client.36224 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:45:03.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:03 smithi204 ceph-mon[32357]: 1.0 deep-scrub starts 2024-09-10T18:45:03.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:03 smithi204 ceph-mon[32357]: 1.0 deep-scrub ok 2024-09-10T18:45:04.236 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-10T18:45:04.237 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-10T18:45:04.237 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-10T18:45:04.237 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-10T18:45:04.237 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-10T18:45:04.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T18:45:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[39220]: pgmap v1064: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:45:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[39220]: from='client.36224 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-10T18:45:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[39220]: osdmap e450: 8 total, 8 up, 8 in 2024-09-10T18:45:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:45:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:45:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[39220]: 3.8 scrub starts 2024-09-10T18:45:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[39220]: 3.8 scrub ok 2024-09-10T18:45:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[39220]: 3.10 scrub starts 2024-09-10T18:45:04.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[39220]: 3.10 scrub ok 2024-09-10T18:45:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[29291]: pgmap v1064: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:45:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[29291]: from='client.36224 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-10T18:45:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[29291]: osdmap e450: 8 total, 8 up, 8 in 2024-09-10T18:45:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:45:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:45:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[29291]: 3.8 scrub starts 2024-09-10T18:45:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[29291]: 3.8 scrub ok 2024-09-10T18:45:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[29291]: 3.10 scrub starts 2024-09-10T18:45:04.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:04 smithi028 ceph-mon[29291]: 3.10 scrub ok 2024-09-10T18:45:04.505 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:04.708 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:04 smithi204 ceph-mon[32357]: pgmap v1064: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:45:04.709 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:04 smithi204 ceph-mon[32357]: from='client.36224 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-10T18:45:04.709 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:04 smithi204 ceph-mon[32357]: osdmap e450: 8 total, 8 up, 8 in 2024-09-10T18:45:04.709 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:04 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:45:04.709 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:04 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:45:04.709 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:04 smithi204 ceph-mon[32357]: 3.8 scrub starts 2024-09-10T18:45:04.709 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:04 smithi204 ceph-mon[32357]: 3.8 scrub ok 2024-09-10T18:45:04.709 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:04 smithi204 ceph-mon[32357]: 3.10 scrub starts 2024-09-10T18:45:04.709 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:04 smithi204 ceph-mon[32357]: 3.10 scrub ok 2024-09-10T18:45:05.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:45:05 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:45:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:05 smithi204 ceph-mon[32357]: 5.1 scrub starts 2024-09-10T18:45:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:05 smithi204 ceph-mon[32357]: 5.1 scrub ok 2024-09-10T18:45:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:05 smithi204 ceph-mon[32357]: 2.2 scrub starts 2024-09-10T18:45:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:05 smithi204 ceph-mon[32357]: 2.2 scrub ok 2024-09-10T18:45:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:05 smithi204 ceph-mon[32357]: osdmap e451: 8 total, 8 up, 8 in 2024-09-10T18:45:05.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:05 smithi204 ceph-mon[32357]: 6.13 scrub starts 2024-09-10T18:45:05.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:05 smithi204 ceph-mon[32357]: 6.13 scrub ok 2024-09-10T18:45:05.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:05 smithi204 ceph-mon[32357]: 6.14 scrub starts 2024-09-10T18:45:05.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:05 smithi204 ceph-mon[32357]: 6.14 scrub ok 2024-09-10T18:45:05.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[39220]: 5.1 scrub starts 2024-09-10T18:45:05.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[39220]: 5.1 scrub ok 2024-09-10T18:45:05.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[39220]: 2.2 scrub starts 2024-09-10T18:45:05.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[39220]: 2.2 scrub ok 2024-09-10T18:45:05.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[39220]: osdmap e451: 8 total, 8 up, 8 in 2024-09-10T18:45:05.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[39220]: 6.13 scrub starts 2024-09-10T18:45:05.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[39220]: 6.13 scrub ok 2024-09-10T18:45:05.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[39220]: 6.14 scrub starts 2024-09-10T18:45:05.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[39220]: 6.14 scrub ok 2024-09-10T18:45:05.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[29291]: 5.1 scrub starts 2024-09-10T18:45:05.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[29291]: 5.1 scrub ok 2024-09-10T18:45:05.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[29291]: 2.2 scrub starts 2024-09-10T18:45:05.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[29291]: 2.2 scrub ok 2024-09-10T18:45:05.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[29291]: osdmap e451: 8 total, 8 up, 8 in 2024-09-10T18:45:05.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[29291]: 6.13 scrub starts 2024-09-10T18:45:05.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[29291]: 6.13 scrub ok 2024-09-10T18:45:05.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[29291]: 6.14 scrub starts 2024-09-10T18:45:05.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:05 smithi028 ceph-mon[29291]: 6.14 scrub ok 2024-09-10T18:45:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:45:06.387 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":452,"fsid":"5eababb2-6fa0-11ef-bcea-c7b262605968","created":"2024-09-10T18:16:20.680373+0000","modified":"2024-09-10T18:45:05.438931+0000","last_up_change":"2024-09-10T18:26:26.611338+0000","last_in_change":"2024-09-10T18:44:05.125356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T18:19:24.013536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"300","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"287","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-10T18:21:30.065209+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":62,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-10T18:21:30.506441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-10T18:21:32.008130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-10T18:21:34.060571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-10T18:21:36.127330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"f1c834d9-252a-4077-b25e-169181d70251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":429,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6801","nonce":3188372351}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6802","nonce":3188372351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6804","nonce":3188372351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6803","nonce":3188372351}]},"public_addr":"172.21.15.28:6801/3188372351","cluster_addr":"172.21.15.28:6802/3188372351","heartbeat_back_addr":"172.21.15.28:6804/3188372351","heartbeat_front_addr":"172.21.15.28:6803/3188372351","state":["exists","up"]},{"osd":1,"uuid":"81f1855d-b946-4ca3-b246-38977b9e20fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":429,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6805","nonce":1369912678}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6806","nonce":1369912678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6808","nonce":1369912678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6807","nonce":1369912678}]},"public_addr":"172.21.15.28:6805/1369912678","cluster_addr":"172.21.15.28:6806/1369912678","heartbeat_back_addr":"172.21.15.28:6808/1369912678","heartbeat_front_addr":"172.21.15.28:6807/1369912678","state":["exists","up"]},{"osd":2,"uuid":"2ad55d2a-36fa-4756-bf22-58d4b049486e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":429,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6809","nonce":1243406003}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6810","nonce":1243406003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6812","nonce":1243406003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6811","nonce":1243406003}]},"public_addr":"172.21.15.28:6809/1243406003","cluster_addr":"172.21.15.28:6810/1243406003","heartbeat_back_addr":"172.21.15.28:6812/1243406003","heartbeat_front_addr":"172.21.15.28:6811/1243406003","state":["exists","up"]},{"osd":3,"uuid":"242729a5-a5e8-44d5-97c8-14d14e970bff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":429,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6813","nonce":67703283}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6814","nonce":67703283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6816","nonce":67703283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6815","nonce":67703283}]},"public_addr":"172.21.15.28:6813/67703283","cluster_addr":"172.21.15.28:6814/67703283","heartbeat_back_addr":"172.21.15.28:6816/67703283","heartbeat_front_addr":"172.21.15.28:6815/67703283","state":["exists","up"]},{"osd":4,"uuid":"6e38b8ef-e6ed-493f-8eca-755354754787","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":429,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6800","nonce":1619669076}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6801","nonce":1619669076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6803","nonce":1619669076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6802","nonce":1619669076}]},"public_addr":"172.21.15.204:6800/1619669076","cluster_addr":"172.21.15.204:6801/1619669076","heartbeat_back_addr":"172.21.15.204:6803/1619669076","heartbeat_front_addr":"172.21.15.204:6802/1619669076","state":["exists","up"]},{"osd":5,"uuid":"a92e0441-66c9-498e-a0ac-6465acdbb1f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":429,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6804","nonce":2717802880}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6805","nonce":2717802880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6807","nonce":2717802880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6806","nonce":2717802880}]},"public_addr":"172.21.15.204:6804/2717802880","cluster_addr":"172.21.15.204:6805/2717802880","heartbeat_back_addr":"172.21.15.204:6807/2717802880","heartbeat_front_addr":"172.21.15.204:6806/2717802880","state":["exists","up"]},{"osd":6,"uuid":"12b02875-6653-45ed-8d2b-b4d33bd88525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":46,"last_clean_end":94,"up_from":144,"up_thru":429,"down_at":95,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6808","nonce":3479311452}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6809","nonce":3479311452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6811","nonce":3479311452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6810","nonce":3479311452}]},"public_addr":"172.21.15.204:6808/3479311452","cluster_addr":"172.21.15.204:6809/3479311452","heartbeat_back_addr":"172.21.15.204:6811/3479311452","heartbeat_front_addr":"172.21.15.204:6810/3479311452","state":["exists","up"]},{"osd":7,"uuid":"0c4f7aa9-ff95-4192-a146-ca576eee907f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":429,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6812","nonce":3511791879}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6813","nonce":3511791879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6815","nonce":3511791879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6814","nonce":3511791879}]},"public_addr":"172.21.15.204:6812/3511791879","cluster_addr":"172.21.15.204:6813/3511791879","heartbeat_back_addr":"172.21.15.204:6815/3511791879","heartbeat_front_addr":"172.21.15.204:6814/3511791879","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:44:14.020799+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:44:17.383265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:44:10.944152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:45:03.458375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:45:05.147201+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:44:42.396034+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:44:41.708466+0000","dead_epoch":94},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:44:30.704660+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"4.c","osds":[7,4,2]},{"pgid":"5.1f","osds":[0,2,7]},{"pgid":"6.17","osds":[7,3,6]}],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":7}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1745574083":"2024-09-11T18:22:08.678985+0000","172.21.15.28:0/3866220552":"2024-09-11T18:16:45.197204+0000","172.21.15.28:6800/3760775432":"2024-09-11T18:16:45.197204+0000","172.21.15.28:0/1746797014":"2024-09-11T18:16:45.197204+0000","172.21.15.28:0/1029045895":"2024-09-11T18:16:45.197204+0000","172.21.15.28:6800/3310949126":"2024-09-11T18:17:07.521165+0000","172.21.15.28:0/2874160765":"2024-09-11T18:17:07.521165+0000","172.21.15.28:0/2620179974":"2024-09-11T18:22:08.678985+0000","172.21.15.28:0/421194352":"2024-09-11T18:17:07.521165+0000","172.21.15.28:6800/1077377902":"2024-09-11T18:22:08.678985+0000","172.21.15.28:0/3428757300":"2024-09-11T18:22:08.678985+0000","172.21.15.28:0/1759116360":"2024-09-11T18:22:08.678985+0000","172.21.15.28:0/2564333506":"2024-09-11T18:17:07.521165+0000","172.21.15.28:0/717120167":"2024-09-11T18:22:08.678985+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T18:45:06.745 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[29291]: 4.d scrub starts 2024-09-10T18:45:06.745 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[29291]: 4.d scrub ok 2024-09-10T18:45:06.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[29291]: 4.15 scrub starts 2024-09-10T18:45:06.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[29291]: 4.15 scrub ok 2024-09-10T18:45:06.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:45:06.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:45:06.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:06.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[29291]: pgmap v1067: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:45:06.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[29291]: osdmap e452: 8 total, 8 up, 8 in 2024-09-10T18:45:06.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3342318811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:45:06.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[39220]: 4.d scrub starts 2024-09-10T18:45:06.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[39220]: 4.d scrub ok 2024-09-10T18:45:06.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[39220]: 4.15 scrub starts 2024-09-10T18:45:06.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[39220]: 4.15 scrub ok 2024-09-10T18:45:06.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:45:06.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:45:06.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:06.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[39220]: pgmap v1067: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:45:06.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[39220]: osdmap e452: 8 total, 8 up, 8 in 2024-09-10T18:45:06.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:06 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3342318811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:45:06.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:06 smithi204 ceph-mon[32357]: 4.d scrub starts 2024-09-10T18:45:06.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:06 smithi204 ceph-mon[32357]: 4.d scrub ok 2024-09-10T18:45:06.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:06 smithi204 ceph-mon[32357]: 4.15 scrub starts 2024-09-10T18:45:06.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:06 smithi204 ceph-mon[32357]: 4.15 scrub ok 2024-09-10T18:45:06.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:06 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:45:06.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:06 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:45:06.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:06 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:06.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:06 smithi204 ceph-mon[32357]: pgmap v1067: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:45:06.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:06 smithi204 ceph-mon[32357]: osdmap e452: 8 total, 8 up, 8 in 2024-09-10T18:45:06.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:06 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3342318811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:45:07.208 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-10T18:45:07.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T18:45:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:08.736 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:08 smithi028 ceph-mon[39220]: pgmap v1069: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:45:08.736 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:08 smithi028 ceph-mon[39220]: 5.12 scrub starts 2024-09-10T18:45:08.736 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:08 smithi028 ceph-mon[39220]: 5.12 scrub ok 2024-09-10T18:45:08.736 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:08 smithi028 ceph-mon[29291]: pgmap v1069: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:45:08.736 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:08 smithi028 ceph-mon[29291]: 5.12 scrub starts 2024-09-10T18:45:08.736 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:08 smithi028 ceph-mon[29291]: 5.12 scrub ok 2024-09-10T18:45:08.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:08 smithi204 ceph-mon[32357]: pgmap v1069: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:45:08.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:08 smithi204 ceph-mon[32357]: 5.12 scrub starts 2024-09-10T18:45:08.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:08 smithi204 ceph-mon[32357]: 5.12 scrub ok 2024-09-10T18:45:09.461 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:45:09.462 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":452,"fsid":"5eababb2-6fa0-11ef-bcea-c7b262605968","created":"2024-09-10T18:16:20.680373+0000","modified":"2024-09-10T18:45:05.438931+0000","last_up_change":"2024-09-10T18:26:26.611338+0000","last_in_change":"2024-09-10T18:44:05.125356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T18:19:24.013536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"300","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"287","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-10T18:21:30.065209+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":62,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-10T18:21:30.506441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-10T18:21:32.008130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-10T18:21:34.060571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-10T18:21:36.127330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"f1c834d9-252a-4077-b25e-169181d70251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":429,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6801","nonce":3188372351}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6802","nonce":3188372351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6804","nonce":3188372351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6803","nonce":3188372351}]},"public_addr":"172.21.15.28:6801/3188372351","cluster_addr":"172.21.15.28:6802/3188372351","heartbeat_back_addr":"172.21.15.28:6804/3188372351","heartbeat_front_addr":"172.21.15.28:6803/3188372351","state":["exists","up"]},{"osd":1,"uuid":"81f1855d-b946-4ca3-b246-38977b9e20fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":429,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6805","nonce":1369912678}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6806","nonce":1369912678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6808","nonce":1369912678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6807","nonce":1369912678}]},"public_addr":"172.21.15.28:6805/1369912678","cluster_addr":"172.21.15.28:6806/1369912678","heartbeat_back_addr":"172.21.15.28:6808/1369912678","heartbeat_front_addr":"172.21.15.28:6807/1369912678","state":["exists","up"]},{"osd":2,"uuid":"2ad55d2a-36fa-4756-bf22-58d4b049486e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":429,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6809","nonce":1243406003}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6810","nonce":1243406003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6812","nonce":1243406003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6811","nonce":1243406003}]},"public_addr":"172.21.15.28:6809/1243406003","cluster_addr":"172.21.15.28:6810/1243406003","heartbeat_back_addr":"172.21.15.28:6812/1243406003","heartbeat_front_addr":"172.21.15.28:6811/1243406003","state":["exists","up"]},{"osd":3,"uuid":"242729a5-a5e8-44d5-97c8-14d14e970bff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":429,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6813","nonce":67703283}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6814","nonce":67703283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6816","nonce":67703283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6815","nonce":67703283}]},"public_addr":"172.21.15.28:6813/67703283","cluster_addr":"172.21.15.28:6814/67703283","heartbeat_back_addr":"172.21.15.28:6816/67703283","heartbeat_front_addr":"172.21.15.28:6815/67703283","state":["exists","up"]},{"osd":4,"uuid":"6e38b8ef-e6ed-493f-8eca-755354754787","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":429,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6800","nonce":1619669076}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6801","nonce":1619669076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6803","nonce":1619669076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6802","nonce":1619669076}]},"public_addr":"172.21.15.204:6800/1619669076","cluster_addr":"172.21.15.204:6801/1619669076","heartbeat_back_addr":"172.21.15.204:6803/1619669076","heartbeat_front_addr":"172.21.15.204:6802/1619669076","state":["exists","up"]},{"osd":5,"uuid":"a92e0441-66c9-498e-a0ac-6465acdbb1f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":429,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6804","nonce":2717802880}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6805","nonce":2717802880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6807","nonce":2717802880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6806","nonce":2717802880}]},"public_addr":"172.21.15.204:6804/2717802880","cluster_addr":"172.21.15.204:6805/2717802880","heartbeat_back_addr":"172.21.15.204:6807/2717802880","heartbeat_front_addr":"172.21.15.204:6806/2717802880","state":["exists","up"]},{"osd":6,"uuid":"12b02875-6653-45ed-8d2b-b4d33bd88525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":46,"last_clean_end":94,"up_from":144,"up_thru":429,"down_at":95,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6808","nonce":3479311452}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6809","nonce":3479311452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6811","nonce":3479311452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6810","nonce":3479311452}]},"public_addr":"172.21.15.204:6808/3479311452","cluster_addr":"172.21.15.204:6809/3479311452","heartbeat_back_addr":"172.21.15.204:6811/3479311452","heartbeat_front_addr":"172.21.15.204:6810/3479311452","state":["exists","up"]},{"osd":7,"uuid":"0c4f7aa9-ff95-4192-a146-ca576eee907f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":429,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6812","nonce":3511791879}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6813","nonce":3511791879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6815","nonce":3511791879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.204:6814","nonce":3511791879}]},"public_addr":"172.21.15.204:6812/3511791879","cluster_addr":"172.21.15.204:6813/3511791879","heartbeat_back_addr":"172.21.15.204:6815/3511791879","heartbeat_front_addr":"172.21.15.204:6814/3511791879","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:44:14.020799+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:44:17.383265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:44:10.944152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:45:03.458375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:45:05.147201+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:44:42.396034+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:44:41.708466+0000","dead_epoch":94},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:44:30.704660+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"4.c","osds":[7,4,2]},{"pgid":"5.1f","osds":[0,2,7]},{"pgid":"6.17","osds":[7,3,6]}],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":7}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1745574083":"2024-09-11T18:22:08.678985+0000","172.21.15.28:0/3866220552":"2024-09-11T18:16:45.197204+0000","172.21.15.28:6800/3760775432":"2024-09-11T18:16:45.197204+0000","172.21.15.28:0/1746797014":"2024-09-11T18:16:45.197204+0000","172.21.15.28:0/1029045895":"2024-09-11T18:16:45.197204+0000","172.21.15.28:6800/3310949126":"2024-09-11T18:17:07.521165+0000","172.21.15.28:0/2874160765":"2024-09-11T18:17:07.521165+0000","172.21.15.28:0/2620179974":"2024-09-11T18:22:08.678985+0000","172.21.15.28:0/421194352":"2024-09-11T18:17:07.521165+0000","172.21.15.28:6800/1077377902":"2024-09-11T18:22:08.678985+0000","172.21.15.28:0/3428757300":"2024-09-11T18:22:08.678985+0000","172.21.15.28:0/1759116360":"2024-09-11T18:22:08.678985+0000","172.21.15.28:0/2564333506":"2024-09-11T18:17:07.521165+0000","172.21.15.28:0/717120167":"2024-09-11T18:22:08.678985+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T18:45:09.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:09 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:45:09.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:09 smithi028 ceph-mon[29291]: 1.14 scrub starts 2024-09-10T18:45:09.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:09 smithi028 ceph-mon[29291]: 1.14 scrub ok 2024-09-10T18:45:09.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:09 smithi028 ceph-mon[29291]: 4.a scrub starts 2024-09-10T18:45:09.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:09 smithi028 ceph-mon[29291]: 4.a scrub ok 2024-09-10T18:45:09.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:09 smithi028 ceph-mon[29291]: 6.15 scrub starts 2024-09-10T18:45:09.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:09 smithi028 ceph-mon[29291]: 6.15 scrub ok 2024-09-10T18:45:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:09 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:45:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:09 smithi028 ceph-mon[39220]: 1.14 scrub starts 2024-09-10T18:45:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:09 smithi028 ceph-mon[39220]: 1.14 scrub ok 2024-09-10T18:45:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:09 smithi028 ceph-mon[39220]: 4.a scrub starts 2024-09-10T18:45:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:09 smithi028 ceph-mon[39220]: 4.a scrub ok 2024-09-10T18:45:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:09 smithi028 ceph-mon[39220]: 6.15 scrub starts 2024-09-10T18:45:09.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:09 smithi028 ceph-mon[39220]: 6.15 scrub ok 2024-09-10T18:45:09.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:09 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:45:09.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:09 smithi204 ceph-mon[32357]: 1.14 scrub starts 2024-09-10T18:45:09.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:09 smithi204 ceph-mon[32357]: 1.14 scrub ok 2024-09-10T18:45:09.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:09 smithi204 ceph-mon[32357]: 4.a scrub starts 2024-09-10T18:45:09.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:09 smithi204 ceph-mon[32357]: 4.a scrub ok 2024-09-10T18:45:09.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:09 smithi204 ceph-mon[32357]: 6.15 scrub starts 2024-09-10T18:45:09.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:09 smithi204 ceph-mon[32357]: 6.15 scrub ok 2024-09-10T18:45:10.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-10T18:45:10.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-10T18:45:10.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-10T18:45:10.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-10T18:45:10.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-10T18:45:10.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-10T18:45:10.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-10T18:45:10.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-10T18:45:10.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:10 smithi028 ceph-mon[39220]: 6.1d deep-scrub starts 2024-09-10T18:45:10.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:10 smithi028 ceph-mon[39220]: 6.1d deep-scrub ok 2024-09-10T18:45:10.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:10 smithi028 ceph-mon[39220]: pgmap v1070: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:45:10.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:10 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3609586864' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:45:10.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:10 smithi028 ceph-mon[29291]: 6.1d deep-scrub starts 2024-09-10T18:45:10.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:10 smithi028 ceph-mon[29291]: 6.1d deep-scrub ok 2024-09-10T18:45:10.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:10 smithi028 ceph-mon[29291]: pgmap v1070: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:45:10.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:10 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3609586864' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:45:10.788 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:10.848 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:10.853 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:10 smithi204 ceph-mon[32357]: 6.1d deep-scrub starts 2024-09-10T18:45:10.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:10 smithi204 ceph-mon[32357]: 6.1d deep-scrub ok 2024-09-10T18:45:10.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:10 smithi204 ceph-mon[32357]: pgmap v1070: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:45:10.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:10 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3609586864' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:45:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:11.061 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:11 smithi028 ceph-mon[29291]: 3.1f scrub starts 2024-09-10T18:45:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:11 smithi028 ceph-mon[29291]: 3.1f scrub ok 2024-09-10T18:45:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:11 smithi028 ceph-mon[29291]: 5.2 scrub starts 2024-09-10T18:45:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:11 smithi028 ceph-mon[29291]: 5.2 scrub ok 2024-09-10T18:45:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:11 smithi028 ceph-mon[29291]: 5.1a scrub starts 2024-09-10T18:45:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:11 smithi028 ceph-mon[29291]: 5.1a scrub ok 2024-09-10T18:45:11.505 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:45:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:45:11] "GET /metrics HTTP/1.1" 200 222519 "" "Prometheus/2.43.0" 2024-09-10T18:45:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:11 smithi028 ceph-mon[39220]: 3.1f scrub starts 2024-09-10T18:45:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:11 smithi028 ceph-mon[39220]: 3.1f scrub ok 2024-09-10T18:45:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:11 smithi028 ceph-mon[39220]: 5.2 scrub starts 2024-09-10T18:45:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:11 smithi028 ceph-mon[39220]: 5.2 scrub ok 2024-09-10T18:45:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:11 smithi028 ceph-mon[39220]: 5.1a scrub starts 2024-09-10T18:45:11.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:11 smithi028 ceph-mon[39220]: 5.1a scrub ok 2024-09-10T18:45:11.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:11 smithi204 ceph-mon[32357]: 3.1f scrub starts 2024-09-10T18:45:11.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:11 smithi204 ceph-mon[32357]: 3.1f scrub ok 2024-09-10T18:45:11.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:11 smithi204 ceph-mon[32357]: 5.2 scrub starts 2024-09-10T18:45:11.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:11 smithi204 ceph-mon[32357]: 5.2 scrub ok 2024-09-10T18:45:11.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:11 smithi204 ceph-mon[32357]: 5.1a scrub starts 2024-09-10T18:45:11.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:11 smithi204 ceph-mon[32357]: 5.1a scrub ok 2024-09-10T18:45:12.459 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:45:12 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:45:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:45:12.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:12 smithi028 ceph-mon[29291]: pgmap v1071: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:45:12.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:12 smithi028 ceph-mon[29291]: 5.0 scrub starts 2024-09-10T18:45:12.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:12 smithi028 ceph-mon[29291]: 5.0 scrub ok 2024-09-10T18:45:12.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:12 smithi028 ceph-mon[29291]: 2.1 scrub starts 2024-09-10T18:45:12.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:12 smithi028 ceph-mon[29291]: 2.1 scrub ok 2024-09-10T18:45:12.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:12 smithi028 ceph-mon[39220]: pgmap v1071: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:45:12.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:12 smithi028 ceph-mon[39220]: 5.0 scrub starts 2024-09-10T18:45:12.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:12 smithi028 ceph-mon[39220]: 5.0 scrub ok 2024-09-10T18:45:12.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:12 smithi028 ceph-mon[39220]: 2.1 scrub starts 2024-09-10T18:45:12.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:12 smithi028 ceph-mon[39220]: 2.1 scrub ok 2024-09-10T18:45:12.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:12 smithi204 ceph-mon[32357]: pgmap v1071: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:45:12.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:12 smithi204 ceph-mon[32357]: 5.0 scrub starts 2024-09-10T18:45:12.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:12 smithi204 ceph-mon[32357]: 5.0 scrub ok 2024-09-10T18:45:12.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:12 smithi204 ceph-mon[32357]: 2.1 scrub starts 2024-09-10T18:45:12.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:12 smithi204 ceph-mon[32357]: 2.1 scrub ok 2024-09-10T18:45:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:13 smithi028 ceph-mon[39220]: 1.1d deep-scrub starts 2024-09-10T18:45:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:13 smithi028 ceph-mon[39220]: 1.1d deep-scrub ok 2024-09-10T18:45:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:13 smithi028 ceph-mon[39220]: 1.6 scrub starts 2024-09-10T18:45:13.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:13 smithi028 ceph-mon[39220]: 1.6 scrub ok 2024-09-10T18:45:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:13 smithi028 ceph-mon[29291]: 1.1d deep-scrub starts 2024-09-10T18:45:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:13 smithi028 ceph-mon[29291]: 1.1d deep-scrub ok 2024-09-10T18:45:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:13 smithi028 ceph-mon[29291]: 1.6 scrub starts 2024-09-10T18:45:13.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:13 smithi028 ceph-mon[29291]: 1.6 scrub ok 2024-09-10T18:45:13.546 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:13.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:13 smithi204 ceph-mon[32357]: 1.1d deep-scrub starts 2024-09-10T18:45:13.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:13 smithi204 ceph-mon[32357]: 1.1d deep-scrub ok 2024-09-10T18:45:13.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:13 smithi204 ceph-mon[32357]: 1.6 scrub starts 2024-09-10T18:45:13.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:13 smithi204 ceph-mon[32357]: 1.6 scrub ok 2024-09-10T18:45:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:14.055 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:14.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:14 smithi028 ceph-mon[29291]: pgmap v1072: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:45:14.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:14 smithi028 ceph-mon[29291]: 3.13 scrub starts 2024-09-10T18:45:14.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:14 smithi028 ceph-mon[29291]: 3.13 scrub ok 2024-09-10T18:45:14.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:14 smithi028 ceph-mon[29291]: 3.18 scrub starts 2024-09-10T18:45:14.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:14 smithi028 ceph-mon[29291]: 3.18 scrub ok 2024-09-10T18:45:14.613 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:14 smithi028 ceph-mon[39220]: pgmap v1072: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:45:14.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:14 smithi028 ceph-mon[39220]: 3.13 scrub starts 2024-09-10T18:45:14.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:14 smithi028 ceph-mon[39220]: 3.13 scrub ok 2024-09-10T18:45:14.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:14 smithi028 ceph-mon[39220]: 3.18 scrub starts 2024-09-10T18:45:14.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:14 smithi028 ceph-mon[39220]: 3.18 scrub ok 2024-09-10T18:45:14.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:14 smithi204 ceph-mon[32357]: pgmap v1072: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:45:14.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:14 smithi204 ceph-mon[32357]: 3.13 scrub starts 2024-09-10T18:45:14.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:14 smithi204 ceph-mon[32357]: 3.13 scrub ok 2024-09-10T18:45:14.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:14 smithi204 ceph-mon[32357]: 3.18 scrub starts 2024-09-10T18:45:14.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:14 smithi204 ceph-mon[32357]: 3.18 scrub ok 2024-09-10T18:45:15.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:45:15 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:45:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:15 smithi204 ceph-mon[32357]: 6.9 deep-scrub starts 2024-09-10T18:45:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:15 smithi204 ceph-mon[32357]: 6.9 deep-scrub ok 2024-09-10T18:45:15.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:15 smithi204 ceph-mon[32357]: 4.7 scrub starts 2024-09-10T18:45:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:15 smithi204 ceph-mon[32357]: 4.7 scrub ok 2024-09-10T18:45:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:15 smithi204 ceph-mon[32357]: 4.c deep-scrub starts 2024-09-10T18:45:15.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:15 smithi204 ceph-mon[32357]: 4.c deep-scrub ok 2024-09-10T18:45:15.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:15 smithi028 ceph-mon[39220]: 6.9 deep-scrub starts 2024-09-10T18:45:15.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:15 smithi028 ceph-mon[39220]: 6.9 deep-scrub ok 2024-09-10T18:45:15.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:15 smithi028 ceph-mon[39220]: 4.7 scrub starts 2024-09-10T18:45:15.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:15 smithi028 ceph-mon[39220]: 4.7 scrub ok 2024-09-10T18:45:15.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:15 smithi028 ceph-mon[39220]: 4.c deep-scrub starts 2024-09-10T18:45:15.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:15 smithi028 ceph-mon[39220]: 4.c deep-scrub ok 2024-09-10T18:45:15.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:15 smithi028 ceph-mon[29291]: 6.9 deep-scrub starts 2024-09-10T18:45:15.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:15 smithi028 ceph-mon[29291]: 6.9 deep-scrub ok 2024-09-10T18:45:15.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:15 smithi028 ceph-mon[29291]: 4.7 scrub starts 2024-09-10T18:45:15.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:15 smithi028 ceph-mon[29291]: 4.7 scrub ok 2024-09-10T18:45:15.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:15 smithi028 ceph-mon[29291]: 4.c deep-scrub starts 2024-09-10T18:45:15.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:15 smithi028 ceph-mon[29291]: 4.c deep-scrub ok 2024-09-10T18:45:16.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:16 smithi028 ceph-mon[29291]: 6.a scrub starts 2024-09-10T18:45:16.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:16 smithi028 ceph-mon[29291]: 6.a scrub ok 2024-09-10T18:45:16.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:16 smithi028 ceph-mon[29291]: pgmap v1073: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:45:16.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:16 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:16.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:16 smithi028 ceph-mon[39220]: 6.a scrub starts 2024-09-10T18:45:16.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:16 smithi028 ceph-mon[39220]: 6.a scrub ok 2024-09-10T18:45:16.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:16 smithi028 ceph-mon[39220]: pgmap v1073: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:45:16.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:16 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:16.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:16 smithi204 ceph-mon[32357]: 6.a scrub starts 2024-09-10T18:45:16.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:16 smithi204 ceph-mon[32357]: 6.a scrub ok 2024-09-10T18:45:16.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:16 smithi204 ceph-mon[32357]: pgmap v1073: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:45:16.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:16 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:17.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:17 smithi028 ceph-mon[29291]: 4.3 scrub starts 2024-09-10T18:45:17.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:17 smithi028 ceph-mon[29291]: 4.3 scrub ok 2024-09-10T18:45:17.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:17 smithi028 ceph-mon[29291]: 5.a scrub starts 2024-09-10T18:45:17.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:17 smithi028 ceph-mon[29291]: 5.a scrub ok 2024-09-10T18:45:17.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:17 smithi028 ceph-mon[39220]: 4.3 scrub starts 2024-09-10T18:45:17.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:17 smithi028 ceph-mon[39220]: 4.3 scrub ok 2024-09-10T18:45:17.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:17 smithi028 ceph-mon[39220]: 5.a scrub starts 2024-09-10T18:45:17.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:17 smithi028 ceph-mon[39220]: 5.a scrub ok 2024-09-10T18:45:17.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:17 smithi204 ceph-mon[32357]: 4.3 scrub starts 2024-09-10T18:45:17.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:17 smithi204 ceph-mon[32357]: 4.3 scrub ok 2024-09-10T18:45:17.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:17 smithi204 ceph-mon[32357]: 5.a scrub starts 2024-09-10T18:45:17.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:17 smithi204 ceph-mon[32357]: 5.a scrub ok 2024-09-10T18:45:18.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:18 smithi028 ceph-mon[29291]: 6.7 scrub starts 2024-09-10T18:45:18.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:18 smithi028 ceph-mon[29291]: 6.7 scrub ok 2024-09-10T18:45:18.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:18 smithi028 ceph-mon[29291]: pgmap v1074: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2024-09-10T18:45:18.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:18 smithi028 ceph-mon[39220]: 6.7 scrub starts 2024-09-10T18:45:18.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:18 smithi028 ceph-mon[39220]: 6.7 scrub ok 2024-09-10T18:45:18.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:18 smithi028 ceph-mon[39220]: pgmap v1074: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2024-09-10T18:45:18.818 INFO:teuthology.orchestra.run.smithi028.stdout:141733921072 2024-09-10T18:45:18.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-10T18:45:18.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:18 smithi204 ceph-mon[32357]: 6.7 scrub starts 2024-09-10T18:45:18.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:18 smithi204 ceph-mon[32357]: 6.7 scrub ok 2024-09-10T18:45:18.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:18 smithi204 ceph-mon[32357]: pgmap v1074: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2024-09-10T18:45:19.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:19 smithi028 ceph-mon[29291]: 6.f scrub starts 2024-09-10T18:45:19.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:19 smithi028 ceph-mon[29291]: 6.f scrub ok 2024-09-10T18:45:19.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:19 smithi028 ceph-mon[29291]: pgmap v1075: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:45:19.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:19 smithi028 ceph-mon[39220]: 6.f scrub starts 2024-09-10T18:45:19.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:19 smithi028 ceph-mon[39220]: 6.f scrub ok 2024-09-10T18:45:19.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:19 smithi028 ceph-mon[39220]: pgmap v1075: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:45:19.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:19 smithi204 ceph-mon[32357]: 6.f scrub starts 2024-09-10T18:45:19.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:19 smithi204 ceph-mon[32357]: 6.f scrub ok 2024-09-10T18:45:19.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:19 smithi204 ceph-mon[32357]: pgmap v1075: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:45:20.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:20 smithi028 ceph-mon[39220]: 1.19 scrub starts 2024-09-10T18:45:20.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:20 smithi028 ceph-mon[39220]: 1.19 scrub ok 2024-09-10T18:45:20.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:20 smithi028 ceph-mon[39220]: 4.12 scrub starts 2024-09-10T18:45:20.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:20 smithi028 ceph-mon[39220]: 4.12 scrub ok 2024-09-10T18:45:20.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:20 smithi028 ceph-mon[29291]: 1.19 scrub starts 2024-09-10T18:45:20.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:20 smithi028 ceph-mon[29291]: 1.19 scrub ok 2024-09-10T18:45:20.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:20 smithi028 ceph-mon[29291]: 4.12 scrub starts 2024-09-10T18:45:20.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:20 smithi028 ceph-mon[29291]: 4.12 scrub ok 2024-09-10T18:45:20.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:20 smithi204 ceph-mon[32357]: 1.19 scrub starts 2024-09-10T18:45:20.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:20 smithi204 ceph-mon[32357]: 1.19 scrub ok 2024-09-10T18:45:20.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:20 smithi204 ceph-mon[32357]: 4.12 scrub starts 2024-09-10T18:45:20.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:20 smithi204 ceph-mon[32357]: 4.12 scrub ok 2024-09-10T18:45:21.461 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:45:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:45:21] "GET /metrics HTTP/1.1" 200 222519 "" "Prometheus/2.43.0" 2024-09-10T18:45:21.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:21 smithi028 ceph-mon[29291]: 6.1a scrub starts 2024-09-10T18:45:21.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:21 smithi028 ceph-mon[29291]: 6.1a scrub ok 2024-09-10T18:45:21.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:21 smithi028 ceph-mon[29291]: pgmap v1076: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:45:21.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:21 smithi028 ceph-mon[29291]: 6.1c scrub starts 2024-09-10T18:45:21.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:21 smithi028 ceph-mon[29291]: 6.1c scrub ok 2024-09-10T18:45:21.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:21 smithi028 ceph-mon[39220]: 6.1a scrub starts 2024-09-10T18:45:21.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:21 smithi028 ceph-mon[39220]: 6.1a scrub ok 2024-09-10T18:45:21.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:21 smithi028 ceph-mon[39220]: pgmap v1076: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:45:21.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:21 smithi028 ceph-mon[39220]: 6.1c scrub starts 2024-09-10T18:45:21.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:21 smithi028 ceph-mon[39220]: 6.1c scrub ok 2024-09-10T18:45:21.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:21 smithi204 ceph-mon[32357]: 6.1a scrub starts 2024-09-10T18:45:21.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:21 smithi204 ceph-mon[32357]: 6.1a scrub ok 2024-09-10T18:45:21.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:21 smithi204 ceph-mon[32357]: pgmap v1076: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:45:21.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:21 smithi204 ceph-mon[32357]: 6.1c scrub starts 2024-09-10T18:45:21.854 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:21 smithi204 ceph-mon[32357]: 6.1c scrub ok 2024-09-10T18:45:22.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:45:22 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:45:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:45:23.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:23 smithi028 ceph-mon[29291]: 6.c scrub starts 2024-09-10T18:45:23.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:23 smithi028 ceph-mon[29291]: 6.c scrub ok 2024-09-10T18:45:23.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:23 smithi028 ceph-mon[39220]: 6.c scrub starts 2024-09-10T18:45:23.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:23 smithi028 ceph-mon[39220]: 6.c scrub ok 2024-09-10T18:45:23.191 INFO:teuthology.orchestra.run.smithi028.stdout:115964117303 2024-09-10T18:45:23.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-10T18:45:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:23 smithi204 ceph-mon[32357]: 6.c scrub starts 2024-09-10T18:45:23.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:23 smithi204 ceph-mon[32357]: 6.c scrub ok 2024-09-10T18:45:24.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:24 smithi028 ceph-mon[29291]: 4.1b scrub starts 2024-09-10T18:45:24.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:24 smithi028 ceph-mon[29291]: 4.1b scrub ok 2024-09-10T18:45:24.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:24 smithi028 ceph-mon[29291]: pgmap v1077: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-10T18:45:24.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:24 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:45:24.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:24 smithi028 ceph-mon[39220]: 4.1b scrub starts 2024-09-10T18:45:24.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:24 smithi028 ceph-mon[39220]: 4.1b scrub ok 2024-09-10T18:45:24.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:24 smithi028 ceph-mon[39220]: pgmap v1077: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-10T18:45:24.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:24 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:45:24.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:24 smithi204 ceph-mon[32357]: 4.1b scrub starts 2024-09-10T18:45:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:24 smithi204 ceph-mon[32357]: 4.1b scrub ok 2024-09-10T18:45:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:24 smithi204 ceph-mon[32357]: pgmap v1077: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-10T18:45:24.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:24 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:45:24.630 INFO:teuthology.orchestra.run.smithi028.stdout:34359738693 2024-09-10T18:45:24.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-10T18:45:25.200 INFO:teuthology.orchestra.run.smithi028.stdout:231928234274 2024-09-10T18:45:25.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-10T18:45:25.353 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:45:25 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:45:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:25 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:45:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:25 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:45:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:25 smithi204 ceph-mon[32357]: osdmap e453: 8 total, 8 up, 8 in 2024-09-10T18:45:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:25 smithi204 ceph-mon[32357]: 5.10 scrub starts 2024-09-10T18:45:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:25 smithi204 ceph-mon[32357]: 5.10 scrub ok 2024-09-10T18:45:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:25 smithi204 ceph-mon[32357]: 5.14 scrub starts 2024-09-10T18:45:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:25 smithi204 ceph-mon[32357]: 5.14 scrub ok 2024-09-10T18:45:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:25 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:45:25.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:25 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:45:25.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:45:25.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:45:25.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[29291]: osdmap e453: 8 total, 8 up, 8 in 2024-09-10T18:45:25.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[29291]: 5.10 scrub starts 2024-09-10T18:45:25.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[29291]: 5.10 scrub ok 2024-09-10T18:45:25.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[29291]: 5.14 scrub starts 2024-09-10T18:45:25.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[29291]: 5.14 scrub ok 2024-09-10T18:45:25.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:45:25.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:45:25.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:45:25.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:45:25.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[39220]: osdmap e453: 8 total, 8 up, 8 in 2024-09-10T18:45:25.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[39220]: 5.10 scrub starts 2024-09-10T18:45:25.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[39220]: 5.10 scrub ok 2024-09-10T18:45:25.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[39220]: 5.14 scrub starts 2024-09-10T18:45:25.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[39220]: 5.14 scrub ok 2024-09-10T18:45:25.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:45:25.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:25 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:45:26.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:26 smithi028 ceph-mon[39220]: 4.1f scrub starts 2024-09-10T18:45:26.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:26 smithi028 ceph-mon[39220]: 4.1f scrub ok 2024-09-10T18:45:26.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:26 smithi028 ceph-mon[39220]: osdmap e454: 8 total, 8 up, 8 in 2024-09-10T18:45:26.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:26 smithi028 ceph-mon[39220]: 1.13 deep-scrub starts 2024-09-10T18:45:26.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:26 smithi028 ceph-mon[39220]: 1.13 deep-scrub ok 2024-09-10T18:45:26.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:26 smithi028 ceph-mon[39220]: pgmap v1080: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:45:26.210 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:26 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:26.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:26 smithi028 ceph-mon[29291]: 4.1f scrub starts 2024-09-10T18:45:26.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:26 smithi028 ceph-mon[29291]: 4.1f scrub ok 2024-09-10T18:45:26.211 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:26 smithi028 ceph-mon[29291]: osdmap e454: 8 total, 8 up, 8 in 2024-09-10T18:45:26.211 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:26 smithi028 ceph-mon[29291]: 1.13 deep-scrub starts 2024-09-10T18:45:26.211 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:26 smithi028 ceph-mon[29291]: 1.13 deep-scrub ok 2024-09-10T18:45:26.211 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:26 smithi028 ceph-mon[29291]: pgmap v1080: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:45:26.211 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:26 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:26 smithi204 ceph-mon[32357]: 4.1f scrub starts 2024-09-10T18:45:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:26 smithi204 ceph-mon[32357]: 4.1f scrub ok 2024-09-10T18:45:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:26 smithi204 ceph-mon[32357]: osdmap e454: 8 total, 8 up, 8 in 2024-09-10T18:45:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:26 smithi204 ceph-mon[32357]: 1.13 deep-scrub starts 2024-09-10T18:45:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:26 smithi204 ceph-mon[32357]: 1.13 deep-scrub ok 2024-09-10T18:45:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:26 smithi204 ceph-mon[32357]: pgmap v1080: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:45:26.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:26 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:27.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[29291]: 3.15 deep-scrub starts 2024-09-10T18:45:27.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[29291]: 3.15 deep-scrub ok 2024-09-10T18:45:27.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[29291]: 5.18 scrub starts 2024-09-10T18:45:27.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[29291]: 5.18 scrub ok 2024-09-10T18:45:27.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[29291]: 5.8 scrub starts 2024-09-10T18:45:27.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[29291]: 5.8 scrub ok 2024-09-10T18:45:27.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[29291]: 5.19 scrub starts 2024-09-10T18:45:27.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[29291]: 5.19 scrub ok 2024-09-10T18:45:27.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[29291]: 6.0 scrub starts 2024-09-10T18:45:27.248 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[29291]: 6.0 scrub ok 2024-09-10T18:45:27.248 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[39220]: 3.15 deep-scrub starts 2024-09-10T18:45:27.248 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[39220]: 3.15 deep-scrub ok 2024-09-10T18:45:27.248 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[39220]: 5.18 scrub starts 2024-09-10T18:45:27.248 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[39220]: 5.18 scrub ok 2024-09-10T18:45:27.248 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[39220]: 5.8 scrub starts 2024-09-10T18:45:27.248 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[39220]: 5.8 scrub ok 2024-09-10T18:45:27.248 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[39220]: 5.19 scrub starts 2024-09-10T18:45:27.248 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[39220]: 5.19 scrub ok 2024-09-10T18:45:27.248 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[39220]: 6.0 scrub starts 2024-09-10T18:45:27.248 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:27 smithi028 ceph-mon[39220]: 6.0 scrub ok 2024-09-10T18:45:27.271 INFO:teuthology.orchestra.run.smithi028.stdout:618475290853 2024-09-10T18:45:27.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-10T18:45:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:27 smithi204 ceph-mon[32357]: 3.15 deep-scrub starts 2024-09-10T18:45:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:27 smithi204 ceph-mon[32357]: 3.15 deep-scrub ok 2024-09-10T18:45:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:27 smithi204 ceph-mon[32357]: 5.18 scrub starts 2024-09-10T18:45:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:27 smithi204 ceph-mon[32357]: 5.18 scrub ok 2024-09-10T18:45:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:27 smithi204 ceph-mon[32357]: 5.8 scrub starts 2024-09-10T18:45:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:27 smithi204 ceph-mon[32357]: 5.8 scrub ok 2024-09-10T18:45:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:27 smithi204 ceph-mon[32357]: 5.19 scrub starts 2024-09-10T18:45:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:27 smithi204 ceph-mon[32357]: 5.19 scrub ok 2024-09-10T18:45:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:27 smithi204 ceph-mon[32357]: 6.0 scrub starts 2024-09-10T18:45:27.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:27 smithi204 ceph-mon[32357]: 6.0 scrub ok 2024-09-10T18:45:27.692 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:27.725 INFO:teuthology.orchestra.run.smithi028.stdout:167503724844 2024-09-10T18:45:27.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-10T18:45:28.260 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:28 smithi028 ceph-mon[29291]: 3.9 scrub starts 2024-09-10T18:45:28.261 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:28 smithi028 ceph-mon[29291]: 3.9 scrub ok 2024-09-10T18:45:28.261 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:28 smithi028 ceph-mon[29291]: pgmap v1081: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:45:28.261 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:28 smithi028 ceph-mon[39220]: 3.9 scrub starts 2024-09-10T18:45:28.261 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:28 smithi028 ceph-mon[39220]: 3.9 scrub ok 2024-09-10T18:45:28.261 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:28 smithi028 ceph-mon[39220]: pgmap v1081: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:45:28.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:28 smithi204 ceph-mon[32357]: 3.9 scrub starts 2024-09-10T18:45:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:28 smithi204 ceph-mon[32357]: 3.9 scrub ok 2024-09-10T18:45:28.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:28 smithi204 ceph-mon[32357]: pgmap v1081: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:45:29.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:29 smithi028 ceph-mon[29291]: 6.19 scrub starts 2024-09-10T18:45:29.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:29 smithi028 ceph-mon[29291]: 6.19 scrub ok 2024-09-10T18:45:29.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:29 smithi028 ceph-mon[29291]: 6.6 scrub starts 2024-09-10T18:45:29.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:29 smithi028 ceph-mon[29291]: 6.6 scrub ok 2024-09-10T18:45:29.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:29 smithi028 ceph-mon[39220]: 6.19 scrub starts 2024-09-10T18:45:29.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:29 smithi028 ceph-mon[39220]: 6.19 scrub ok 2024-09-10T18:45:29.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:29 smithi028 ceph-mon[39220]: 6.6 scrub starts 2024-09-10T18:45:29.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:29 smithi028 ceph-mon[39220]: 6.6 scrub ok 2024-09-10T18:45:29.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:29 smithi204 ceph-mon[32357]: 6.19 scrub starts 2024-09-10T18:45:29.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:29 smithi204 ceph-mon[32357]: 6.19 scrub ok 2024-09-10T18:45:29.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:29 smithi204 ceph-mon[32357]: 6.6 scrub starts 2024-09-10T18:45:29.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:29 smithi204 ceph-mon[32357]: 6.6 scrub ok 2024-09-10T18:45:30.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:30 smithi028 ceph-mon[39220]: 4.1 deep-scrub starts 2024-09-10T18:45:30.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:30 smithi028 ceph-mon[39220]: 4.1 deep-scrub ok 2024-09-10T18:45:30.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:30 smithi028 ceph-mon[39220]: 5.1b scrub starts 2024-09-10T18:45:30.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:30 smithi028 ceph-mon[39220]: 5.1b scrub ok 2024-09-10T18:45:30.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:30 smithi028 ceph-mon[39220]: pgmap v1082: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:45:30.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:30 smithi028 ceph-mon[29291]: 4.1 deep-scrub starts 2024-09-10T18:45:30.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:30 smithi028 ceph-mon[29291]: 4.1 deep-scrub ok 2024-09-10T18:45:30.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:30 smithi028 ceph-mon[29291]: 5.1b scrub starts 2024-09-10T18:45:30.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:30 smithi028 ceph-mon[29291]: 5.1b scrub ok 2024-09-10T18:45:30.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:30 smithi028 ceph-mon[29291]: pgmap v1082: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:45:30.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:30 smithi204 ceph-mon[32357]: 4.1 deep-scrub starts 2024-09-10T18:45:30.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:30 smithi204 ceph-mon[32357]: 4.1 deep-scrub ok 2024-09-10T18:45:30.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:30 smithi204 ceph-mon[32357]: 5.1b scrub starts 2024-09-10T18:45:30.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:30 smithi204 ceph-mon[32357]: 5.1b scrub ok 2024-09-10T18:45:30.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:30 smithi204 ceph-mon[32357]: pgmap v1082: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:45:30.882 INFO:teuthology.orchestra.run.smithi028.stdout:77309411644 2024-09-10T18:45:30.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-10T18:45:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:31.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:31 smithi028 ceph-mon[39220]: 5.6 scrub starts 2024-09-10T18:45:31.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:31 smithi028 ceph-mon[39220]: 5.6 scrub ok 2024-09-10T18:45:31.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:31 smithi028 ceph-mon[39220]: 3.a deep-scrub starts 2024-09-10T18:45:31.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:31 smithi028 ceph-mon[39220]: 3.a deep-scrub ok 2024-09-10T18:45:31.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:31 smithi028 ceph-mon[39220]: 4.8 scrub starts 2024-09-10T18:45:31.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:31 smithi028 ceph-mon[39220]: 4.8 scrub ok 2024-09-10T18:45:31.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:31 smithi028 ceph-mon[39220]: 3.6 scrub starts 2024-09-10T18:45:31.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:31 smithi028 ceph-mon[39220]: 3.6 scrub ok 2024-09-10T18:45:31.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:31 smithi028 ceph-mon[29291]: 5.6 scrub starts 2024-09-10T18:45:31.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:31 smithi028 ceph-mon[29291]: 5.6 scrub ok 2024-09-10T18:45:31.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:31 smithi028 ceph-mon[29291]: 3.a deep-scrub starts 2024-09-10T18:45:31.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:31 smithi028 ceph-mon[29291]: 3.a deep-scrub ok 2024-09-10T18:45:31.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:31 smithi028 ceph-mon[29291]: 4.8 scrub starts 2024-09-10T18:45:31.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:31 smithi028 ceph-mon[29291]: 4.8 scrub ok 2024-09-10T18:45:31.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:31 smithi028 ceph-mon[29291]: 3.6 scrub starts 2024-09-10T18:45:31.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:31 smithi028 ceph-mon[29291]: 3.6 scrub ok 2024-09-10T18:45:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:31 smithi204 ceph-mon[32357]: 5.6 scrub starts 2024-09-10T18:45:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:31 smithi204 ceph-mon[32357]: 5.6 scrub ok 2024-09-10T18:45:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:31 smithi204 ceph-mon[32357]: 3.a deep-scrub starts 2024-09-10T18:45:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:31 smithi204 ceph-mon[32357]: 3.a deep-scrub ok 2024-09-10T18:45:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:31 smithi204 ceph-mon[32357]: 4.8 scrub starts 2024-09-10T18:45:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:31 smithi204 ceph-mon[32357]: 4.8 scrub ok 2024-09-10T18:45:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:31 smithi204 ceph-mon[32357]: 3.6 scrub starts 2024-09-10T18:45:31.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:31 smithi204 ceph-mon[32357]: 3.6 scrub ok 2024-09-10T18:45:31.422 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:45:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:45:31] "GET /metrics HTTP/1.1" 200 222494 "" "Prometheus/2.43.0" 2024-09-10T18:45:32.005 INFO:teuthology.orchestra.run.smithi028.stdout:141733921074 2024-09-10T18:45:32.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:32 smithi028 ceph-mon[29291]: pgmap v1083: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:45:32.346 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:32 smithi028 ceph-mon[29291]: 4.6 scrub starts 2024-09-10T18:45:32.346 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:32 smithi028 ceph-mon[29291]: 4.6 scrub ok 2024-09-10T18:45:32.346 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:32 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/260018298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T18:45:32.346 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:32 smithi028 ceph-mon[39220]: pgmap v1083: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:45:32.346 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:32 smithi028 ceph-mon[39220]: 4.6 scrub starts 2024-09-10T18:45:32.346 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:32 smithi028 ceph-mon[39220]: 4.6 scrub ok 2024-09-10T18:45:32.346 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:32 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/260018298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T18:45:32.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:45:32 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:45:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:45:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:32 smithi204 ceph-mon[32357]: pgmap v1083: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:45:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:32 smithi204 ceph-mon[32357]: 4.6 scrub starts 2024-09-10T18:45:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:32 smithi204 ceph-mon[32357]: 4.6 scrub ok 2024-09-10T18:45:32.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:32 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/260018298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T18:45:33.052 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733921072 got 141733921074 for osd.4 2024-09-10T18:45:33.052 DEBUG:teuthology.parallel:result is None 2024-09-10T18:45:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:33 smithi204 ceph-mon[32357]: 3.1c deep-scrub starts 2024-09-10T18:45:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:33 smithi204 ceph-mon[32357]: 3.1c deep-scrub ok 2024-09-10T18:45:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:33 smithi204 ceph-mon[32357]: 5.c scrub starts 2024-09-10T18:45:33.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:33 smithi204 ceph-mon[32357]: 5.c scrub ok 2024-09-10T18:45:33.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:33 smithi028 ceph-mon[29291]: 3.1c deep-scrub starts 2024-09-10T18:45:33.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:33 smithi028 ceph-mon[29291]: 3.1c deep-scrub ok 2024-09-10T18:45:33.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:33 smithi028 ceph-mon[29291]: 5.c scrub starts 2024-09-10T18:45:33.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:33 smithi028 ceph-mon[29291]: 5.c scrub ok 2024-09-10T18:45:33.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:33 smithi028 ceph-mon[39220]: 3.1c deep-scrub starts 2024-09-10T18:45:33.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:33 smithi028 ceph-mon[39220]: 3.1c deep-scrub ok 2024-09-10T18:45:33.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:33 smithi028 ceph-mon[39220]: 5.c scrub starts 2024-09-10T18:45:33.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:33 smithi028 ceph-mon[39220]: 5.c scrub ok 2024-09-10T18:45:33.818 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:33.849 INFO:teuthology.orchestra.run.smithi028.stdout:55834575169 2024-09-10T18:45:33.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-10T18:45:34.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:34 smithi204 ceph-mon[32357]: 5.d scrub starts 2024-09-10T18:45:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:34 smithi204 ceph-mon[32357]: 5.d scrub ok 2024-09-10T18:45:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:34 smithi204 ceph-mon[32357]: pgmap v1084: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 997 B/s rd, 0 op/s 2024-09-10T18:45:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:34 smithi204 ceph-mon[32357]: 3.1e deep-scrub starts 2024-09-10T18:45:34.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:34 smithi204 ceph-mon[32357]: 3.1e deep-scrub ok 2024-09-10T18:45:34.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:34 smithi028 ceph-mon[29291]: 5.d scrub starts 2024-09-10T18:45:34.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:34 smithi028 ceph-mon[29291]: 5.d scrub ok 2024-09-10T18:45:34.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:34 smithi028 ceph-mon[29291]: pgmap v1084: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 997 B/s rd, 0 op/s 2024-09-10T18:45:34.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:34 smithi028 ceph-mon[29291]: 3.1e deep-scrub starts 2024-09-10T18:45:34.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:34 smithi028 ceph-mon[29291]: 3.1e deep-scrub ok 2024-09-10T18:45:34.363 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:34 smithi028 ceph-mon[39220]: 5.d scrub starts 2024-09-10T18:45:34.363 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:34 smithi028 ceph-mon[39220]: 5.d scrub ok 2024-09-10T18:45:34.363 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:34 smithi028 ceph-mon[39220]: pgmap v1084: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 997 B/s rd, 0 op/s 2024-09-10T18:45:34.363 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:34 smithi028 ceph-mon[39220]: 3.1e deep-scrub starts 2024-09-10T18:45:34.363 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:34 smithi028 ceph-mon[39220]: 3.1e deep-scrub ok 2024-09-10T18:45:35.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:45:35 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:45:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:35 smithi204 ceph-mon[32357]: 1.2 scrub starts 2024-09-10T18:45:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:35 smithi204 ceph-mon[32357]: 1.2 scrub ok 2024-09-10T18:45:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:35 smithi204 ceph-mon[32357]: 4.1c deep-scrub starts 2024-09-10T18:45:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:35 smithi204 ceph-mon[32357]: 4.1c deep-scrub ok 2024-09-10T18:45:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:35 smithi204 ceph-mon[32357]: 4.5 scrub starts 2024-09-10T18:45:35.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:35 smithi204 ceph-mon[32357]: 4.5 scrub ok 2024-09-10T18:45:35.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:35 smithi028 ceph-mon[29291]: 1.2 scrub starts 2024-09-10T18:45:35.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:35 smithi028 ceph-mon[29291]: 1.2 scrub ok 2024-09-10T18:45:35.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:35 smithi028 ceph-mon[29291]: 4.1c deep-scrub starts 2024-09-10T18:45:35.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:35 smithi028 ceph-mon[29291]: 4.1c deep-scrub ok 2024-09-10T18:45:35.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:35 smithi028 ceph-mon[29291]: 4.5 scrub starts 2024-09-10T18:45:35.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:35 smithi028 ceph-mon[29291]: 4.5 scrub ok 2024-09-10T18:45:35.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:35 smithi028 ceph-mon[39220]: 1.2 scrub starts 2024-09-10T18:45:35.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:35 smithi028 ceph-mon[39220]: 1.2 scrub ok 2024-09-10T18:45:35.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:35 smithi028 ceph-mon[39220]: 4.1c deep-scrub starts 2024-09-10T18:45:35.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:35 smithi028 ceph-mon[39220]: 4.1c deep-scrub ok 2024-09-10T18:45:35.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:35 smithi028 ceph-mon[39220]: 4.5 scrub starts 2024-09-10T18:45:35.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:35 smithi028 ceph-mon[39220]: 4.5 scrub ok 2024-09-10T18:45:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:36.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[29291]: 1.7 scrub starts 2024-09-10T18:45:36.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[29291]: 1.7 scrub ok 2024-09-10T18:45:36.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[29291]: 4.1e scrub starts 2024-09-10T18:45:36.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[29291]: 4.1e scrub ok 2024-09-10T18:45:36.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[29291]: pgmap v1085: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-09-10T18:45:36.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:36.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[29291]: 3.2 scrub starts 2024-09-10T18:45:36.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[29291]: 3.2 scrub ok 2024-09-10T18:45:36.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[29291]: 3.1d scrub starts 2024-09-10T18:45:36.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[29291]: 3.1d scrub ok 2024-09-10T18:45:36.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[39220]: 1.7 scrub starts 2024-09-10T18:45:36.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[39220]: 1.7 scrub ok 2024-09-10T18:45:36.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[39220]: 4.1e scrub starts 2024-09-10T18:45:36.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[39220]: 4.1e scrub ok 2024-09-10T18:45:36.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[39220]: pgmap v1085: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-09-10T18:45:36.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:36.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[39220]: 3.2 scrub starts 2024-09-10T18:45:36.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[39220]: 3.2 scrub ok 2024-09-10T18:45:36.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[39220]: 3.1d scrub starts 2024-09-10T18:45:36.332 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:36 smithi028 ceph-mon[39220]: 3.1d scrub ok 2024-09-10T18:45:36.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:36 smithi204 ceph-mon[32357]: 1.7 scrub starts 2024-09-10T18:45:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:36 smithi204 ceph-mon[32357]: 1.7 scrub ok 2024-09-10T18:45:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:36 smithi204 ceph-mon[32357]: 4.1e scrub starts 2024-09-10T18:45:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:36 smithi204 ceph-mon[32357]: 4.1e scrub ok 2024-09-10T18:45:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:36 smithi204 ceph-mon[32357]: pgmap v1085: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-09-10T18:45:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:36 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:36 smithi204 ceph-mon[32357]: 3.2 scrub starts 2024-09-10T18:45:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:36 smithi204 ceph-mon[32357]: 3.2 scrub ok 2024-09-10T18:45:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:36 smithi204 ceph-mon[32357]: 3.1d scrub starts 2024-09-10T18:45:36.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:36 smithi204 ceph-mon[32357]: 3.1d scrub ok 2024-09-10T18:45:36.392 INFO:teuthology.orchestra.run.smithi028.stdout:231928234276 2024-09-10T18:45:37.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:45:37.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:45:37.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[39220]: osdmap e455: 8 total, 8 up, 8 in 2024-09-10T18:45:37.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[39220]: 3.1 scrub starts 2024-09-10T18:45:37.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[39220]: 3.1 scrub ok 2024-09-10T18:45:37.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[39220]: 4.0 scrub starts 2024-09-10T18:45:37.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[39220]: 4.0 scrub ok 2024-09-10T18:45:37.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[39220]: 3.11 scrub starts 2024-09-10T18:45:37.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[39220]: 3.11 scrub ok 2024-09-10T18:45:37.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/4183975885' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T18:45:37.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:45:37.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:45:37.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[29291]: osdmap e455: 8 total, 8 up, 8 in 2024-09-10T18:45:37.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[29291]: 3.1 scrub starts 2024-09-10T18:45:37.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[29291]: 3.1 scrub ok 2024-09-10T18:45:37.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[29291]: 4.0 scrub starts 2024-09-10T18:45:37.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[29291]: 4.0 scrub ok 2024-09-10T18:45:37.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[29291]: 3.11 scrub starts 2024-09-10T18:45:37.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[29291]: 3.11 scrub ok 2024-09-10T18:45:37.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:37 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/4183975885' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T18:45:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:37 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:45:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:37 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:45:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:37 smithi204 ceph-mon[32357]: osdmap e455: 8 total, 8 up, 8 in 2024-09-10T18:45:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:37 smithi204 ceph-mon[32357]: 3.1 scrub starts 2024-09-10T18:45:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:37 smithi204 ceph-mon[32357]: 3.1 scrub ok 2024-09-10T18:45:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:37 smithi204 ceph-mon[32357]: 4.0 scrub starts 2024-09-10T18:45:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:37 smithi204 ceph-mon[32357]: 4.0 scrub ok 2024-09-10T18:45:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:37 smithi204 ceph-mon[32357]: 3.11 scrub starts 2024-09-10T18:45:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:37 smithi204 ceph-mon[32357]: 3.11 scrub ok 2024-09-10T18:45:37.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:37 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/4183975885' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T18:45:38.134 INFO:teuthology.orchestra.run.smithi028.stdout:34359738696 2024-09-10T18:45:38.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:38 smithi028 ceph-mon[29291]: pgmap v1087: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:45:38.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:38 smithi028 ceph-mon[29291]: 3.e scrub starts 2024-09-10T18:45:38.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:38 smithi028 ceph-mon[29291]: 3.e scrub ok 2024-09-10T18:45:38.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:38 smithi028 ceph-mon[39220]: pgmap v1087: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:45:38.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:38 smithi028 ceph-mon[39220]: 3.e scrub starts 2024-09-10T18:45:38.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:38 smithi028 ceph-mon[39220]: 3.e scrub ok 2024-09-10T18:45:38.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:38 smithi204 ceph-mon[32357]: pgmap v1087: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:45:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:38 smithi204 ceph-mon[32357]: 3.e scrub starts 2024-09-10T18:45:38.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:38 smithi204 ceph-mon[32357]: 3.e scrub ok 2024-09-10T18:45:38.930 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738693 got 34359738696 for osd.0 2024-09-10T18:45:38.930 DEBUG:teuthology.parallel:result is None 2024-09-10T18:45:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:39 smithi204 ceph-mon[32357]: 5.e scrub starts 2024-09-10T18:45:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:39 smithi204 ceph-mon[32357]: 5.e scrub ok 2024-09-10T18:45:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:39 smithi204 ceph-mon[32357]: 5.7 scrub starts 2024-09-10T18:45:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:39 smithi204 ceph-mon[32357]: 5.7 scrub ok 2024-09-10T18:45:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:39 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3869364391' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T18:45:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:39 smithi204 ceph-mon[32357]: 3.17 scrub starts 2024-09-10T18:45:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:39 smithi204 ceph-mon[32357]: 3.17 scrub ok 2024-09-10T18:45:39.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:39 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:45:39.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:39 smithi028 ceph-mon[29291]: 5.e scrub starts 2024-09-10T18:45:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:39 smithi028 ceph-mon[29291]: 5.e scrub ok 2024-09-10T18:45:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:39 smithi028 ceph-mon[29291]: 5.7 scrub starts 2024-09-10T18:45:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:39 smithi028 ceph-mon[29291]: 5.7 scrub ok 2024-09-10T18:45:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:39 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3869364391' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T18:45:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:39 smithi028 ceph-mon[29291]: 3.17 scrub starts 2024-09-10T18:45:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:39 smithi028 ceph-mon[29291]: 3.17 scrub ok 2024-09-10T18:45:39.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:39 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:45:39.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:39 smithi028 ceph-mon[39220]: 5.e scrub starts 2024-09-10T18:45:39.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:39 smithi028 ceph-mon[39220]: 5.e scrub ok 2024-09-10T18:45:39.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:39 smithi028 ceph-mon[39220]: 5.7 scrub starts 2024-09-10T18:45:39.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:39 smithi028 ceph-mon[39220]: 5.7 scrub ok 2024-09-10T18:45:39.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:39 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3869364391' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T18:45:39.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:39 smithi028 ceph-mon[39220]: 3.17 scrub starts 2024-09-10T18:45:39.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:39 smithi028 ceph-mon[39220]: 3.17 scrub ok 2024-09-10T18:45:39.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:39 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:45:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:115964117306 2024-09-10T18:45:40.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:40 smithi028 ceph-mon[39220]: 3.5 scrub starts 2024-09-10T18:45:40.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:40 smithi028 ceph-mon[39220]: 3.5 scrub ok 2024-09-10T18:45:40.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:40 smithi028 ceph-mon[39220]: 3.19 scrub starts 2024-09-10T18:45:40.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:40 smithi028 ceph-mon[39220]: 3.19 scrub ok 2024-09-10T18:45:40.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:40 smithi028 ceph-mon[39220]: pgmap v1088: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:45:40.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:40 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3719024590' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T18:45:40.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:40 smithi028 ceph-mon[29291]: 3.5 scrub starts 2024-09-10T18:45:40.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:40 smithi028 ceph-mon[29291]: 3.5 scrub ok 2024-09-10T18:45:40.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:40 smithi028 ceph-mon[29291]: 3.19 scrub starts 2024-09-10T18:45:40.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:40 smithi028 ceph-mon[29291]: 3.19 scrub ok 2024-09-10T18:45:40.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:40 smithi028 ceph-mon[29291]: pgmap v1088: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:45:40.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:40 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3719024590' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T18:45:40.228 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234274 got 231928234276 for osd.7 2024-09-10T18:45:40.228 DEBUG:teuthology.parallel:result is None 2024-09-10T18:45:40.353 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:40 smithi204 ceph-mon[32357]: 3.5 scrub starts 2024-09-10T18:45:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:40 smithi204 ceph-mon[32357]: 3.5 scrub ok 2024-09-10T18:45:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:40 smithi204 ceph-mon[32357]: 3.19 scrub starts 2024-09-10T18:45:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:40 smithi204 ceph-mon[32357]: 3.19 scrub ok 2024-09-10T18:45:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:40 smithi204 ceph-mon[32357]: pgmap v1088: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:45:40.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:40 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3719024590' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T18:45:40.434 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:40.889 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117303 got 115964117306 for osd.3 2024-09-10T18:45:40.889 DEBUG:teuthology.parallel:result is None 2024-09-10T18:45:41.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:41 smithi028 ceph-mon[29291]: 5.11 scrub starts 2024-09-10T18:45:41.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:41 smithi028 ceph-mon[29291]: 5.11 scrub ok 2024-09-10T18:45:41.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:41 smithi028 ceph-mon[29291]: 3.16 scrub starts 2024-09-10T18:45:41.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:41 smithi028 ceph-mon[29291]: 3.16 scrub ok 2024-09-10T18:45:41.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:41 smithi028 ceph-mon[29291]: 3.0 scrub starts 2024-09-10T18:45:41.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:41 smithi028 ceph-mon[29291]: 3.0 scrub ok 2024-09-10T18:45:41.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:41 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:45:41.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:41 smithi028 ceph-mon[39220]: 5.11 scrub starts 2024-09-10T18:45:41.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:41 smithi028 ceph-mon[39220]: 5.11 scrub ok 2024-09-10T18:45:41.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:41 smithi028 ceph-mon[39220]: 3.16 scrub starts 2024-09-10T18:45:41.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:41 smithi028 ceph-mon[39220]: 3.16 scrub ok 2024-09-10T18:45:41.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:41 smithi028 ceph-mon[39220]: 3.0 scrub starts 2024-09-10T18:45:41.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:41 smithi028 ceph-mon[39220]: 3.0 scrub ok 2024-09-10T18:45:41.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:41 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:45:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:41 smithi204 ceph-mon[32357]: 5.11 scrub starts 2024-09-10T18:45:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:41 smithi204 ceph-mon[32357]: 5.11 scrub ok 2024-09-10T18:45:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:41 smithi204 ceph-mon[32357]: 3.16 scrub starts 2024-09-10T18:45:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:41 smithi204 ceph-mon[32357]: 3.16 scrub ok 2024-09-10T18:45:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:41 smithi204 ceph-mon[32357]: 3.0 scrub starts 2024-09-10T18:45:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:41 smithi204 ceph-mon[32357]: 3.0 scrub ok 2024-09-10T18:45:41.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:41 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:45:41.503 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:45:41 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:45:41] "GET /metrics HTTP/1.1" 200 222488 "" "Prometheus/2.43.0" 2024-09-10T18:45:42.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:42 smithi028 ceph-mon[39220]: pgmap v1089: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-10T18:45:42.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:42 smithi028 ceph-mon[39220]: 1.d scrub starts 2024-09-10T18:45:42.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:42 smithi028 ceph-mon[39220]: 1.d scrub ok 2024-09-10T18:45:42.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:42 smithi028 ceph-mon[29291]: pgmap v1089: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-10T18:45:42.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:42 smithi028 ceph-mon[29291]: 1.d scrub starts 2024-09-10T18:45:42.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:42 smithi028 ceph-mon[29291]: 1.d scrub ok 2024-09-10T18:45:42.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:42 smithi204 ceph-mon[32357]: pgmap v1089: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-10T18:45:42.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:42 smithi204 ceph-mon[32357]: 1.d scrub starts 2024-09-10T18:45:42.586 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:42 smithi204 ceph-mon[32357]: 1.d scrub ok 2024-09-10T18:45:42.586 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:45:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:45:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:45:43.288 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:43 smithi028 ceph-mon[39220]: 6.12 scrub starts 2024-09-10T18:45:43.288 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:43 smithi028 ceph-mon[39220]: 6.12 scrub ok 2024-09-10T18:45:43.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:43 smithi028 ceph-mon[29291]: 6.12 scrub starts 2024-09-10T18:45:43.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:43 smithi028 ceph-mon[29291]: 6.12 scrub ok 2024-09-10T18:45:43.319 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:43 smithi204 ceph-mon[32357]: 6.12 scrub starts 2024-09-10T18:45:43.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:43 smithi204 ceph-mon[32357]: 6.12 scrub ok 2024-09-10T18:45:43.473 INFO:teuthology.orchestra.run.smithi028.stdout:618475290857 2024-09-10T18:45:44.205 INFO:tasks.cephadm.ceph_manager.ceph:need seq 618475290853 got 618475290857 for osd.6 2024-09-10T18:45:44.205 DEBUG:teuthology.parallel:result is None 2024-09-10T18:45:44.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:44 smithi028 ceph-mon[39220]: pgmap v1090: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-10T18:45:44.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:44 smithi028 ceph-mon[39220]: 4.16 deep-scrub starts 2024-09-10T18:45:44.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:44 smithi028 ceph-mon[39220]: 4.16 deep-scrub ok 2024-09-10T18:45:44.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:44 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/3306073074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T18:45:44.407 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:44 smithi028 ceph-mon[29291]: pgmap v1090: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-10T18:45:44.407 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:44 smithi028 ceph-mon[29291]: 4.16 deep-scrub starts 2024-09-10T18:45:44.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:44 smithi028 ceph-mon[29291]: 4.16 deep-scrub ok 2024-09-10T18:45:44.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:44 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/3306073074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T18:45:44.419 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:44 smithi204 ceph-mon[32357]: pgmap v1090: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-10T18:45:44.419 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:44 smithi204 ceph-mon[32357]: 4.16 deep-scrub starts 2024-09-10T18:45:44.419 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:44 smithi204 ceph-mon[32357]: 4.16 deep-scrub ok 2024-09-10T18:45:44.419 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:44 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/3306073074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T18:45:44.794 INFO:teuthology.orchestra.run.smithi028.stdout:167503724849 2024-09-10T18:45:45.145 INFO:teuthology.orchestra.run.smithi028.stdout:77309411647 2024-09-10T18:45:45.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[29291]: 1.1a deep-scrub starts 2024-09-10T18:45:45.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[29291]: 1.1a deep-scrub ok 2024-09-10T18:45:45.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[29291]: 4.2 scrub starts 2024-09-10T18:45:45.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[29291]: 4.2 scrub ok 2024-09-10T18:45:45.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[29291]: 4.11 scrub starts 2024-09-10T18:45:45.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[29291]: 4.11 scrub ok 2024-09-10T18:45:45.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:45:45.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:45:45.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/411477742' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T18:45:45.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[39220]: 1.1a deep-scrub starts 2024-09-10T18:45:45.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[39220]: 1.1a deep-scrub ok 2024-09-10T18:45:45.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[39220]: 4.2 scrub starts 2024-09-10T18:45:45.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[39220]: 4.2 scrub ok 2024-09-10T18:45:45.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[39220]: 4.11 scrub starts 2024-09-10T18:45:45.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[39220]: 4.11 scrub ok 2024-09-10T18:45:45.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:45:45.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:45:45.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:45 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/411477742' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T18:45:45.603 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:45:45 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:45:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:45 smithi204 ceph-mon[32357]: 1.1a deep-scrub starts 2024-09-10T18:45:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:45 smithi204 ceph-mon[32357]: 1.1a deep-scrub ok 2024-09-10T18:45:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:45 smithi204 ceph-mon[32357]: 4.2 scrub starts 2024-09-10T18:45:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:45 smithi204 ceph-mon[32357]: 4.2 scrub ok 2024-09-10T18:45:45.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:45 smithi204 ceph-mon[32357]: 4.11 scrub starts 2024-09-10T18:45:45.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:45 smithi204 ceph-mon[32357]: 4.11 scrub ok 2024-09-10T18:45:45.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:45 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:45:45.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:45 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:45:45.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:45 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/411477742' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T18:45:45.631 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724844 got 167503724849 for osd.5 2024-09-10T18:45:45.631 DEBUG:teuthology.parallel:result is None 2024-09-10T18:45:46.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[39220]: 3.14 scrub starts 2024-09-10T18:45:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[39220]: 3.14 scrub ok 2024-09-10T18:45:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[39220]: 4.f scrub starts 2024-09-10T18:45:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[39220]: 4.f scrub ok 2024-09-10T18:45:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/296936069' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T18:45:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[39220]: pgmap v1091: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-10T18:45:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[39220]: 6.11 scrub starts 2024-09-10T18:45:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[39220]: 6.11 scrub ok 2024-09-10T18:45:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[39220]: 6.10 scrub starts 2024-09-10T18:45:46.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[39220]: 6.10 scrub ok 2024-09-10T18:45:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[29291]: 3.14 scrub starts 2024-09-10T18:45:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[29291]: 3.14 scrub ok 2024-09-10T18:45:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[29291]: 4.f scrub starts 2024-09-10T18:45:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[29291]: 4.f scrub ok 2024-09-10T18:45:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/296936069' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T18:45:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[29291]: pgmap v1091: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-10T18:45:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[29291]: 6.11 scrub starts 2024-09-10T18:45:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[29291]: 6.11 scrub ok 2024-09-10T18:45:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[29291]: 6.10 scrub starts 2024-09-10T18:45:46.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:46 smithi028 ceph-mon[29291]: 6.10 scrub ok 2024-09-10T18:45:46.286 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411644 got 77309411647 for osd.2 2024-09-10T18:45:46.286 DEBUG:teuthology.parallel:result is None 2024-09-10T18:45:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:46 smithi204 ceph-mon[32357]: 3.14 scrub starts 2024-09-10T18:45:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:46 smithi204 ceph-mon[32357]: 3.14 scrub ok 2024-09-10T18:45:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:46 smithi204 ceph-mon[32357]: 4.f scrub starts 2024-09-10T18:45:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:46 smithi204 ceph-mon[32357]: 4.f scrub ok 2024-09-10T18:45:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:46 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/296936069' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T18:45:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:46 smithi204 ceph-mon[32357]: pgmap v1091: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-10T18:45:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:46 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:46 smithi204 ceph-mon[32357]: 6.11 scrub starts 2024-09-10T18:45:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:46 smithi204 ceph-mon[32357]: 6.11 scrub ok 2024-09-10T18:45:46.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:46 smithi204 ceph-mon[32357]: 6.10 scrub starts 2024-09-10T18:45:46.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:46 smithi204 ceph-mon[32357]: 6.10 scrub ok 2024-09-10T18:45:47.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:47 smithi028 ceph-mon[29291]: 5.1c deep-scrub starts 2024-09-10T18:45:47.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:47 smithi028 ceph-mon[29291]: 5.1c deep-scrub ok 2024-09-10T18:45:47.374 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:47 smithi028 ceph-mon[39220]: 5.1c deep-scrub starts 2024-09-10T18:45:47.374 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:47 smithi028 ceph-mon[39220]: 5.1c deep-scrub ok 2024-09-10T18:45:47.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:47 smithi204 ceph-mon[32357]: 5.1c deep-scrub starts 2024-09-10T18:45:47.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:47 smithi204 ceph-mon[32357]: 5.1c deep-scrub ok 2024-09-10T18:45:48.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:48 smithi028 ceph-mon[29291]: 5.1d scrub starts 2024-09-10T18:45:48.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:48 smithi028 ceph-mon[29291]: 5.1d scrub ok 2024-09-10T18:45:48.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:48 smithi028 ceph-mon[29291]: pgmap v1092: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 823 B/s rd, 0 op/s 2024-09-10T18:45:48.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:48 smithi028 ceph-mon[39220]: 5.1d scrub starts 2024-09-10T18:45:48.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:48 smithi028 ceph-mon[39220]: 5.1d scrub ok 2024-09-10T18:45:48.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:48 smithi028 ceph-mon[39220]: pgmap v1092: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 823 B/s rd, 0 op/s 2024-09-10T18:45:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:48 smithi204 ceph-mon[32357]: 5.1d scrub starts 2024-09-10T18:45:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:48 smithi204 ceph-mon[32357]: 5.1d scrub ok 2024-09-10T18:45:48.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:48 smithi204 ceph-mon[32357]: pgmap v1092: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 823 B/s rd, 0 op/s 2024-09-10T18:45:48.753 INFO:teuthology.orchestra.run.smithi028.stdout:55834575173 2024-09-10T18:45:49.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:49 smithi028 ceph-mon[29291]: 4.1a scrub starts 2024-09-10T18:45:49.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:49 smithi028 ceph-mon[29291]: 4.1a scrub ok 2024-09-10T18:45:49.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:49 smithi028 ceph-mon[29291]: 6.d scrub starts 2024-09-10T18:45:49.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:49 smithi028 ceph-mon[29291]: 6.d scrub ok 2024-09-10T18:45:49.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:49 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/393397292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T18:45:49.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:49 smithi028 ceph-mon[39220]: 4.1a scrub starts 2024-09-10T18:45:49.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:49 smithi028 ceph-mon[39220]: 4.1a scrub ok 2024-09-10T18:45:49.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:49 smithi028 ceph-mon[39220]: 6.d scrub starts 2024-09-10T18:45:49.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:49 smithi028 ceph-mon[39220]: 6.d scrub ok 2024-09-10T18:45:49.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:49 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/393397292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T18:45:49.525 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575169 got 55834575173 for osd.1 2024-09-10T18:45:49.525 DEBUG:teuthology.parallel:result is None 2024-09-10T18:45:49.526 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-10T18:45:49.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:45:49.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:49 smithi204 ceph-mon[32357]: 4.1a scrub starts 2024-09-10T18:45:49.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:49 smithi204 ceph-mon[32357]: 4.1a scrub ok 2024-09-10T18:45:49.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:49 smithi204 ceph-mon[32357]: 6.d scrub starts 2024-09-10T18:45:49.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:49 smithi204 ceph-mon[32357]: 6.d scrub ok 2024-09-10T18:45:49.605 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:49 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/393397292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T18:45:50.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:50 smithi028 ceph-mon[29291]: pgmap v1093: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:45:50.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:50 smithi028 ceph-mon[29291]: 1.9 scrub starts 2024-09-10T18:45:50.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:50 smithi028 ceph-mon[29291]: 1.9 scrub ok 2024-09-10T18:45:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:50 smithi028 ceph-mon[39220]: pgmap v1093: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:45:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:50 smithi028 ceph-mon[39220]: 1.9 scrub starts 2024-09-10T18:45:50.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:50 smithi028 ceph-mon[39220]: 1.9 scrub ok 2024-09-10T18:45:50.603 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:50 smithi204 ceph-mon[32357]: pgmap v1093: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:45:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:50 smithi204 ceph-mon[32357]: 1.9 scrub starts 2024-09-10T18:45:50.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:50 smithi204 ceph-mon[32357]: 1.9 scrub ok 2024-09-10T18:45:50.798 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:51.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:45:51 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:45:51] "GET /metrics HTTP/1.1" 200 222488 "" "Prometheus/2.43.0" 2024-09-10T18:45:52.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:51 smithi028 ceph-mon[29291]: 3.c scrub starts 2024-09-10T18:45:52.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:51 smithi028 ceph-mon[29291]: 3.c scrub ok 2024-09-10T18:45:52.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:51 smithi028 ceph-mon[29291]: 6.1 deep-scrub starts 2024-09-10T18:45:52.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:51 smithi028 ceph-mon[29291]: 6.1 deep-scrub ok 2024-09-10T18:45:52.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:51 smithi028 ceph-mon[29291]: 3.f deep-scrub starts 2024-09-10T18:45:52.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:51 smithi028 ceph-mon[29291]: 3.f deep-scrub ok 2024-09-10T18:45:52.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:51 smithi028 ceph-mon[29291]: pgmap v1094: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:45:52.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:51 smithi028 ceph-mon[39220]: 3.c scrub starts 2024-09-10T18:45:52.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:51 smithi028 ceph-mon[39220]: 3.c scrub ok 2024-09-10T18:45:52.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:51 smithi028 ceph-mon[39220]: 6.1 deep-scrub starts 2024-09-10T18:45:52.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:51 smithi028 ceph-mon[39220]: 6.1 deep-scrub ok 2024-09-10T18:45:52.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:51 smithi028 ceph-mon[39220]: 3.f deep-scrub starts 2024-09-10T18:45:52.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:51 smithi028 ceph-mon[39220]: 3.f deep-scrub ok 2024-09-10T18:45:52.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:51 smithi028 ceph-mon[39220]: pgmap v1094: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:45:52.354 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:45:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:45:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:45:52.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:51 smithi204 ceph-mon[32357]: 3.c scrub starts 2024-09-10T18:45:52.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:51 smithi204 ceph-mon[32357]: 3.c scrub ok 2024-09-10T18:45:52.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:51 smithi204 ceph-mon[32357]: 6.1 deep-scrub starts 2024-09-10T18:45:52.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:51 smithi204 ceph-mon[32357]: 6.1 deep-scrub ok 2024-09-10T18:45:52.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:51 smithi204 ceph-mon[32357]: 3.f deep-scrub starts 2024-09-10T18:45:52.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:51 smithi204 ceph-mon[32357]: 3.f deep-scrub ok 2024-09-10T18:45:52.355 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:51 smithi204 ceph-mon[32357]: pgmap v1094: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:45:53.113 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:45:53.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[39220]: 1.a scrub starts 2024-09-10T18:45:53.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[39220]: 1.a scrub ok 2024-09-10T18:45:53.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[39220]: 1.1c scrub starts 2024-09-10T18:45:53.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[39220]: 1.1c scrub ok 2024-09-10T18:45:53.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[39220]: 6.1b scrub starts 2024-09-10T18:45:53.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[39220]: 6.1b scrub ok 2024-09-10T18:45:53.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:45:53.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:45:53.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:45:53.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:45:53.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:45:53.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[29291]: 1.a scrub starts 2024-09-10T18:45:53.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[29291]: 1.a scrub ok 2024-09-10T18:45:53.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[29291]: 1.1c scrub starts 2024-09-10T18:45:53.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[29291]: 1.1c scrub ok 2024-09-10T18:45:53.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[29291]: 6.1b scrub starts 2024-09-10T18:45:53.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[29291]: 6.1b scrub ok 2024-09-10T18:45:53.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:45:53.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:45:53.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:45:53.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:45:53.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:52 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:45:53.122 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:45:53.149 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:52 smithi204 ceph-mon[32357]: 1.a scrub starts 2024-09-10T18:45:53.149 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:52 smithi204 ceph-mon[32357]: 1.a scrub ok 2024-09-10T18:45:53.149 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:52 smithi204 ceph-mon[32357]: 1.1c scrub starts 2024-09-10T18:45:53.149 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:52 smithi204 ceph-mon[32357]: 1.1c scrub ok 2024-09-10T18:45:53.149 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:52 smithi204 ceph-mon[32357]: 6.1b scrub starts 2024-09-10T18:45:53.149 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:52 smithi204 ceph-mon[32357]: 6.1b scrub ok 2024-09-10T18:45:53.149 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:52 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:45:53.149 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:52 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:45:53.150 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:52 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:45:53.150 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:52 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:45:53.150 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:52 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:45:53.863 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":1094,"stamp":"2024-09-10T18:45:51.273154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":3014,"num_read_kb":2766,"num_write":1002,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":663865,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3712,"ondisk_log_size":3712,"up":486,"acting":486,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":486,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7176356,"kb_used_data":7652,"kb_used_omap":19,"kb_used_meta":7168620,"kb_avail":742784860,"statfs":{"total":767960285184,"available":760611696640,"internally_reserved":0,"allocated":7835648,"data_stored":4757188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20033,"internal_metadata":7340667327},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003188"},"pg_stats":[{"pgid":"1.1d","version":"76'81","reported_seq":229,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.108195+0000","last_change":"2024-09-10T18:45:12.273543+0000","last_active":"2024-09-10T18:45:25.108195+0000","last_peered":"2024-09-10T18:45:25.108195+0000","last_clean":"2024-09-10T18:45:25.108195+0000","last_became_active":"2024-09-10T18:34:34.865894+0000","last_became_peered":"2024-09-10T18:34:34.865894+0000","last_unstale":"2024-09-10T18:45:25.108195+0000","last_undegraded":"2024-09-10T18:45:25.108195+0000","last_fullsized":"2024-09-10T18:45:25.108195+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:12.273477+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:45:12.273477+0000","last_clean_scrub_stamp":"2024-09-10T18:45:12.273477+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:22.469989+0000","scrub_duration":0.001515665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.658317+0000","last_change":"2024-09-10T18:45:10.445143+0000","last_active":"2024-09-10T18:45:25.658317+0000","last_peered":"2024-09-10T18:45:25.658317+0000","last_clean":"2024-09-10T18:45:25.658317+0000","last_became_active":"2024-09-10T18:44:06.170062+0000","last_became_peered":"2024-09-10T18:44:06.170062+0000","last_unstale":"2024-09-10T18:45:25.658317+0000","last_undegraded":"2024-09-10T18:45:25.658317+0000","last_fullsized":"2024-09-10T18:45:25.658317+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:10.445040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:45:10.445040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:14.416296+0000","scrub_duration":0.0033800380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"400'25","reported_seq":555,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.036738+0000","last_change":"2024-09-10T18:44:35.666624+0000","last_active":"2024-09-10T18:45:25.036738+0000","last_peered":"2024-09-10T18:45:25.036738+0000","last_clean":"2024-09-10T18:45:25.036738+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:45:25.036738+0000","last_undegraded":"2024-09-10T18:45:25.036738+0000","last_fullsized":"2024-09-10T18:45:25.036738+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'25","last_scrub_stamp":"2024-09-10T18:44:35.666557+0000","last_deep_scrub":"342'19","last_deep_scrub_stamp":"2024-09-10T18:37:49.354781+0000","last_clean_scrub_stamp":"2024-09-10T18:44:35.666557+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:45.011647+0000","scrub_duration":0.00194746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":540,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:26.120412+0000","last_change":"2024-09-10T18:45:26.120376+0000","last_active":"2024-09-10T18:45:26.120412+0000","last_peered":"2024-09-10T18:45:26.120412+0000","last_clean":"2024-09-10T18:45:26.120412+0000","last_became_active":"2024-09-10T18:39:00.910095+0000","last_became_peered":"2024-09-10T18:39:00.910095+0000","last_unstale":"2024-09-10T18:45:26.120412+0000","last_undegraded":"2024-09-10T18:45:26.120412+0000","last_fullsized":"2024-09-10T18:45:26.120412+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:26.120321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:45:26.120321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:45.597849+0000","scrub_duration":0.0019125699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":529,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.036431+0000","last_change":"2024-09-10T18:45:19.719242+0000","last_active":"2024-09-10T18:45:25.036431+0000","last_peered":"2024-09-10T18:45:25.036431+0000","last_clean":"2024-09-10T18:45:25.036431+0000","last_became_active":"2024-09-10T18:39:00.889898+0000","last_became_peered":"2024-09-10T18:39:00.889898+0000","last_unstale":"2024-09-10T18:45:25.036431+0000","last_undegraded":"2024-09-10T18:45:25.036431+0000","last_fullsized":"2024-09-10T18:45:25.036431+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:19.719163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:45:19.719163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:41.285618+0000","scrub_duration":0.0015216979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'81","reported_seq":217,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405440+0000","last_change":"2024-09-10T18:44:41.052661+0000","last_active":"2024-09-10T18:45:25.405440+0000","last_peered":"2024-09-10T18:45:25.405440+0000","last_clean":"2024-09-10T18:45:25.405440+0000","last_became_active":"2024-09-10T18:34:34.867388+0000","last_became_peered":"2024-09-10T18:34:34.867388+0000","last_unstale":"2024-09-10T18:45:25.405440+0000","last_undegraded":"2024-09-10T18:45:25.405440+0000","last_fullsized":"2024-09-10T18:45:25.405440+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:41.052593+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:44:41.052593+0000","last_clean_scrub_stamp":"2024-09-10T18:44:41.052593+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:41.420911+0000","scrub_duration":0.0013664650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":496,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.035490+0000","last_change":"2024-09-10T18:44:06.629971+0000","last_active":"2024-09-10T18:45:25.035490+0000","last_peered":"2024-09-10T18:45:25.035490+0000","last_clean":"2024-09-10T18:45:25.035490+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:45:25.035490+0000","last_undegraded":"2024-09-10T18:45:25.035490+0000","last_fullsized":"2024-09-10T18:45:25.035490+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:06.629919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:44:06.629919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021700869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":512,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:33.394610+0000","last_change":"2024-09-10T18:45:33.394562+0000","last_active":"2024-09-10T18:45:33.394610+0000","last_peered":"2024-09-10T18:45:33.394610+0000","last_clean":"2024-09-10T18:45:33.394610+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:45:33.394610+0000","last_undegraded":"2024-09-10T18:45:33.394610+0000","last_fullsized":"2024-09-10T18:45:33.394610+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:33.394489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:45:33.394489+0000","last_clean_scrub_stamp":"2024-09-10T18:45:33.394489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:55.815361+0000","scrub_duration":0.0021405780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"400'46","reported_seq":616,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.035352+0000","last_change":"2024-09-10T18:44:51.373960+0000","last_active":"2024-09-10T18:45:25.035352+0000","last_peered":"2024-09-10T18:45:25.035352+0000","last_clean":"2024-09-10T18:45:25.035352+0000","last_became_active":"2024-09-10T18:44:06.166307+0000","last_became_peered":"2024-09-10T18:44:06.166307+0000","last_unstale":"2024-09-10T18:45:25.035352+0000","last_undegraded":"2024-09-10T18:45:25.035352+0000","last_fullsized":"2024-09-10T18:45:25.035352+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'46","last_scrub_stamp":"2024-09-10T18:44:51.373894+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:44:51.373894+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:08.264078+0000","scrub_duration":0.0020169509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":518,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.540949+0000","last_change":"2024-09-10T18:45:25.540900+0000","last_active":"2024-09-10T18:45:25.540949+0000","last_peered":"2024-09-10T18:45:25.540949+0000","last_clean":"2024-09-10T18:45:25.540949+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:45:25.540949+0000","last_undegraded":"2024-09-10T18:45:25.540949+0000","last_fullsized":"2024-09-10T18:45:25.540949+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:25.540825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:44:05.367986+0000","last_clean_scrub_stamp":"2024-09-10T18:45:25.540825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:40.345042+0000","scrub_duration":0.001622484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":116,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:35.403360+0000","last_change":"2024-09-10T18:45:35.403291+0000","last_active":"2024-09-10T18:45:35.403360+0000","last_peered":"2024-09-10T18:45:35.403360+0000","last_clean":"2024-09-10T18:45:35.403360+0000","last_became_active":"2024-09-10T18:40:10.986591+0000","last_became_peered":"2024-09-10T18:40:10.986591+0000","last_unstale":"2024-09-10T18:45:35.403360+0000","last_undegraded":"2024-09-10T18:45:35.403360+0000","last_fullsized":"2024-09-10T18:45:35.403360+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:35.403223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_clean_scrub_stamp":"2024-09-10T18:45:35.403223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:48.759914+0000","scrub_duration":0.001477007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"400'21","reported_seq":594,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:47.743449+0000","last_change":"2024-09-10T18:45:47.743397+0000","last_active":"2024-09-10T18:45:47.743449+0000","last_peered":"2024-09-10T18:45:47.743449+0000","last_clean":"2024-09-10T18:45:47.743449+0000","last_became_active":"2024-09-10T18:44:06.172812+0000","last_became_peered":"2024-09-10T18:44:06.172812+0000","last_unstale":"2024-09-10T18:45:47.743449+0000","last_undegraded":"2024-09-10T18:45:47.743449+0000","last_fullsized":"2024-09-10T18:45:47.743449+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'21","last_scrub_stamp":"2024-09-10T18:45:47.743315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:45:47.743315+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:50.375271+0000","scrub_duration":0.001988802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":157,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:29.129220+0000","last_change":"2024-09-10T18:45:29.129164+0000","last_active":"2024-09-10T18:45:29.129220+0000","last_peered":"2024-09-10T18:45:29.129220+0000","last_clean":"2024-09-10T18:45:29.129220+0000","last_became_active":"2024-09-10T18:44:06.170431+0000","last_became_peered":"2024-09-10T18:44:06.170431+0000","last_unstale":"2024-09-10T18:45:29.129220+0000","last_undegraded":"2024-09-10T18:45:29.129220+0000","last_fullsized":"2024-09-10T18:45:29.129220+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:29.129096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_clean_scrub_stamp":"2024-09-10T18:45:29.129096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:44.573196+0000","scrub_duration":0.001575012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.657895+0000","last_change":"2024-09-10T18:44:46.446472+0000","last_active":"2024-09-10T18:45:25.657895+0000","last_peered":"2024-09-10T18:45:25.657895+0000","last_clean":"2024-09-10T18:45:25.657895+0000","last_became_active":"2024-09-10T18:44:06.167736+0000","last_became_peered":"2024-09-10T18:44:06.167736+0000","last_unstale":"2024-09-10T18:45:25.657895+0000","last_undegraded":"2024-09-10T18:45:25.657895+0000","last_fullsized":"2024-09-10T18:45:25.657895+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:46.446420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:44:46.446420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:04.856827+0000","scrub_duration":0.0021950250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'81","reported_seq":282,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.036255+0000","last_change":"2024-09-10T18:44:48.453566+0000","last_active":"2024-09-10T18:45:25.036255+0000","last_peered":"2024-09-10T18:45:25.036255+0000","last_clean":"2024-09-10T18:45:25.036255+0000","last_became_active":"2024-09-10T18:34:34.863102+0000","last_became_peered":"2024-09-10T18:34:34.863102+0000","last_unstale":"2024-09-10T18:45:25.036255+0000","last_undegraded":"2024-09-10T18:45:25.036255+0000","last_fullsized":"2024-09-10T18:45:25.036255+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:48.453498+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:44:48.453498+0000","last_clean_scrub_stamp":"2024-09-10T18:44:48.453498+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:07.561005+0000","scrub_duration":0.002023249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":147,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:32.128950+0000","last_change":"2024-09-10T18:45:32.128903+0000","last_active":"2024-09-10T18:45:32.128950+0000","last_peered":"2024-09-10T18:45:32.128950+0000","last_clean":"2024-09-10T18:45:32.128950+0000","last_became_active":"2024-09-10T18:39:00.915294+0000","last_became_peered":"2024-09-10T18:39:00.915294+0000","last_unstale":"2024-09-10T18:45:32.128950+0000","last_undegraded":"2024-09-10T18:45:32.128950+0000","last_fullsized":"2024-09-10T18:45:32.128950+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:32.128822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:45:32.128822+0000","last_clean_scrub_stamp":"2024-09-10T18:45:32.128822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:54.224216+0000","scrub_duration":0.0016067970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"400'11","reported_seq":560,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.037423+0000","last_change":"2024-09-10T18:45:22.596902+0000","last_active":"2024-09-10T18:45:25.037423+0000","last_peered":"2024-09-10T18:45:25.037423+0000","last_clean":"2024-09-10T18:45:25.037423+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:45:25.037423+0000","last_undegraded":"2024-09-10T18:45:25.037423+0000","last_fullsized":"2024-09-10T18:45:25.037423+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'11","last_scrub_stamp":"2024-09-10T18:45:22.596826+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:45:22.596826+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:25.786042+0000","scrub_duration":0.002692945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":509,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.404789+0000","last_change":"2024-09-10T18:45:11.329820+0000","last_active":"2024-09-10T18:45:25.404789+0000","last_peered":"2024-09-10T18:45:25.404789+0000","last_clean":"2024-09-10T18:45:25.404789+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:45:25.404789+0000","last_undegraded":"2024-09-10T18:45:25.404789+0000","last_fullsized":"2024-09-10T18:45:25.404789+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:11.329716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:45:11.329716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:25.688528+0000","scrub_duration":0.00160012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:28.134865+0000","last_change":"2024-09-10T18:45:28.134804+0000","last_active":"2024-09-10T18:45:28.134865+0000","last_peered":"2024-09-10T18:45:28.134865+0000","last_clean":"2024-09-10T18:45:28.134865+0000","last_became_active":"2024-09-10T18:39:00.917007+0000","last_became_peered":"2024-09-10T18:39:00.917007+0000","last_unstale":"2024-09-10T18:45:28.134865+0000","last_undegraded":"2024-09-10T18:45:28.134865+0000","last_fullsized":"2024-09-10T18:45:28.134865+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:28.134742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:45:28.134742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:29.684542+0000","scrub_duration":0.0017479450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'81","reported_seq":385,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.035795+0000","last_change":"2024-09-10T18:45:19.531979+0000","last_active":"2024-09-10T18:45:25.035795+0000","last_peered":"2024-09-10T18:45:25.035795+0000","last_clean":"2024-09-10T18:45:25.035795+0000","last_became_active":"2024-09-10T18:44:06.166987+0000","last_became_peered":"2024-09-10T18:44:06.166987+0000","last_unstale":"2024-09-10T18:45:25.035795+0000","last_undegraded":"2024-09-10T18:45:25.035795+0000","last_fullsized":"2024-09-10T18:45:25.035795+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:19.531921+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:45:19.531921+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:23.320065+0000","scrub_duration":0.0017219480000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":536,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.036408+0000","last_change":"2024-09-10T18:44:56.718221+0000","last_active":"2024-09-10T18:45:25.036408+0000","last_peered":"2024-09-10T18:45:25.036408+0000","last_clean":"2024-09-10T18:45:25.036408+0000","last_became_active":"2024-09-10T18:39:00.895860+0000","last_became_peered":"2024-09-10T18:39:00.895860+0000","last_unstale":"2024-09-10T18:45:25.036408+0000","last_undegraded":"2024-09-10T18:45:25.036408+0000","last_fullsized":"2024-09-10T18:45:25.036408+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:56.718155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:44:56.718155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:10.050934+0000","scrub_duration":0.004132035,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.658263+0000","last_change":"2024-09-10T18:44:34.380158+0000","last_active":"2024-09-10T18:45:25.658263+0000","last_peered":"2024-09-10T18:45:25.658263+0000","last_clean":"2024-09-10T18:45:25.658263+0000","last_became_active":"2024-09-10T18:44:06.163606+0000","last_became_peered":"2024-09-10T18:44:06.163606+0000","last_unstale":"2024-09-10T18:45:25.658263+0000","last_undegraded":"2024-09-10T18:45:25.658263+0000","last_fullsized":"2024-09-10T18:45:25.658263+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:34.380067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:44:34.380067+0000","last_clean_scrub_stamp":"2024-09-10T18:44:34.380067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:47.855233+0000","scrub_duration":0.0019231789999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"454'46","reported_seq":628,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:33.697862+0000","last_change":"2024-09-10T18:45:33.697748+0000","last_active":"2024-09-10T18:45:33.697862+0000","last_peered":"2024-09-10T18:45:33.697862+0000","last_clean":"2024-09-10T18:45:33.697862+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:45:33.697862+0000","last_undegraded":"2024-09-10T18:45:33.697862+0000","last_fullsized":"2024-09-10T18:45:33.697862+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'46","last_scrub_stamp":"2024-09-10T18:45:33.697655+0000","last_deep_scrub":"454'46","last_deep_scrub_stamp":"2024-09-10T18:45:33.697655+0000","last_clean_scrub_stamp":"2024-09-10T18:45:33.697655+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:58.862583+0000","scrub_duration":0.0063720590000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":550,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:47.066380+0000","last_change":"2024-09-10T18:45:47.066352+0000","last_active":"2024-09-10T18:45:47.066380+0000","last_peered":"2024-09-10T18:45:47.066380+0000","last_clean":"2024-09-10T18:45:47.066380+0000","last_became_active":"2024-09-10T18:44:06.162348+0000","last_became_peered":"2024-09-10T18:44:06.162348+0000","last_unstale":"2024-09-10T18:45:47.066380+0000","last_undegraded":"2024-09-10T18:45:47.066380+0000","last_fullsized":"2024-09-10T18:45:47.066380+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:47.066292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:44:35.066309+0000","last_clean_scrub_stamp":"2024-09-10T18:45:47.066292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:59.223230+0000","scrub_duration":0.0016254850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'81","reported_seq":141,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.177847+0000","last_change":"2024-09-10T18:44:47.099266+0000","last_active":"2024-09-10T18:45:25.177847+0000","last_peered":"2024-09-10T18:45:25.177847+0000","last_clean":"2024-09-10T18:45:25.177847+0000","last_became_active":"2024-09-10T18:39:00.907303+0000","last_became_peered":"2024-09-10T18:39:00.907303+0000","last_unstale":"2024-09-10T18:45:25.177847+0000","last_undegraded":"2024-09-10T18:45:25.177847+0000","last_fullsized":"2024-09-10T18:45:25.177847+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:47.099153+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:47.099153+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:58.313101+0000","scrub_duration":0.0024115510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":516,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.036532+0000","last_change":"2024-09-10T18:44:03.627161+0000","last_active":"2024-09-10T18:45:25.036532+0000","last_peered":"2024-09-10T18:45:25.036532+0000","last_clean":"2024-09-10T18:45:25.036532+0000","last_became_active":"2024-09-10T18:39:00.904769+0000","last_became_peered":"2024-09-10T18:39:00.904769+0000","last_unstale":"2024-09-10T18:45:25.036532+0000","last_undegraded":"2024-09-10T18:45:25.036532+0000","last_fullsized":"2024-09-10T18:45:25.036532+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:03.627094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:44:03.627094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001532537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":512,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.036807+0000","last_change":"2024-09-10T18:44:33.681320+0000","last_active":"2024-09-10T18:45:25.036807+0000","last_peered":"2024-09-10T18:45:25.036807+0000","last_clean":"2024-09-10T18:45:25.036807+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:45:25.036807+0000","last_undegraded":"2024-09-10T18:45:25.036807+0000","last_fullsized":"2024-09-10T18:45:25.036807+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:33.681184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:44:33.681184+0000","last_clean_scrub_stamp":"2024-09-10T18:44:33.681184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:45.487221+0000","scrub_duration":0.0027611559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"400'29","reported_seq":561,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.036004+0000","last_change":"2024-09-10T18:44:14.615569+0000","last_active":"2024-09-10T18:45:25.036004+0000","last_peered":"2024-09-10T18:45:25.036004+0000","last_clean":"2024-09-10T18:45:25.036004+0000","last_became_active":"2024-09-10T18:39:11.896502+0000","last_became_peered":"2024-09-10T18:39:11.896502+0000","last_unstale":"2024-09-10T18:45:25.036004+0000","last_undegraded":"2024-09-10T18:45:25.036004+0000","last_fullsized":"2024-09-10T18:45:25.036004+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'29","last_scrub_stamp":"2024-09-10T18:44:14.615508+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:44:14.615508+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:41.084923+0000","scrub_duration":0.0028561379999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":524,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.768307+0000","last_change":"2024-09-10T18:45:45.768259+0000","last_active":"2024-09-10T18:45:45.768307+0000","last_peered":"2024-09-10T18:45:45.768307+0000","last_clean":"2024-09-10T18:45:45.768307+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:45:45.768307+0000","last_undegraded":"2024-09-10T18:45:45.768307+0000","last_fullsized":"2024-09-10T18:45:45.768307+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:45.768182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:45:45.768182+0000","last_clean_scrub_stamp":"2024-09-10T18:45:45.768182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:03.418329+0000","scrub_duration":0.0014202220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'81","reported_seq":157,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178340+0000","last_change":"2024-09-10T18:44:58.038568+0000","last_active":"2024-09-10T18:45:25.178340+0000","last_peered":"2024-09-10T18:45:25.178340+0000","last_clean":"2024-09-10T18:45:25.178340+0000","last_became_active":"2024-09-10T18:44:06.171035+0000","last_became_peered":"2024-09-10T18:44:06.171035+0000","last_unstale":"2024-09-10T18:45:25.178340+0000","last_undegraded":"2024-09-10T18:45:25.178340+0000","last_fullsized":"2024-09-10T18:45:25.178340+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:58.038484+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:58.038484+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:07.496990+0000","scrub_duration":0.0017201479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":541,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405285+0000","last_change":"2024-09-10T18:45:21.342131+0000","last_active":"2024-09-10T18:45:25.405285+0000","last_peered":"2024-09-10T18:45:25.405285+0000","last_clean":"2024-09-10T18:45:25.405285+0000","last_became_active":"2024-09-10T18:39:00.891321+0000","last_became_peered":"2024-09-10T18:39:00.891321+0000","last_unstale":"2024-09-10T18:45:25.405285+0000","last_undegraded":"2024-09-10T18:45:25.405285+0000","last_fullsized":"2024-09-10T18:45:25.405285+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:45:21.342062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:45:21.342062+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:32.535184+0000","scrub_duration":0.0017807109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":534,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:39.100615+0000","last_change":"2024-09-10T18:45:39.100584+0000","last_active":"2024-09-10T18:45:39.100615+0000","last_peered":"2024-09-10T18:45:39.100615+0000","last_clean":"2024-09-10T18:45:39.100615+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:45:39.100615+0000","last_undegraded":"2024-09-10T18:45:39.100615+0000","last_fullsized":"2024-09-10T18:45:39.100615+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:39.100524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:45:39.100524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:41.815546+0000","scrub_duration":0.001315004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"400'35","reported_seq":67,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:35.267010+0000","last_change":"2024-09-10T18:45:35.266959+0000","last_active":"2024-09-10T18:45:35.267010+0000","last_peered":"2024-09-10T18:45:35.267010+0000","last_clean":"2024-09-10T18:45:35.267010+0000","last_became_active":"2024-09-10T18:44:06.163102+0000","last_became_peered":"2024-09-10T18:44:06.163102+0000","last_unstale":"2024-09-10T18:45:35.267010+0000","last_undegraded":"2024-09-10T18:45:35.267010+0000","last_fullsized":"2024-09-10T18:45:35.267010+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'35","last_scrub_stamp":"2024-09-10T18:45:35.266893+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:45:35.266893+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:03.821912+0000","scrub_duration":0.0025491289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"430'14","reported_seq":70,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:40.043343+0000","last_change":"2024-09-10T18:44:50.370594+0000","last_active":"2024-09-10T18:45:40.043343+0000","last_peered":"2024-09-10T18:45:40.043343+0000","last_clean":"2024-09-10T18:45:40.043343+0000","last_became_active":"2024-09-10T18:44:06.169046+0000","last_became_peered":"2024-09-10T18:44:06.169046+0000","last_unstale":"2024-09-10T18:45:40.043343+0000","last_undegraded":"2024-09-10T18:45:40.043343+0000","last_fullsized":"2024-09-10T18:45:40.043343+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'14","last_scrub_stamp":"2024-09-10T18:44:50.370539+0000","last_deep_scrub":"430'14","last_deep_scrub_stamp":"2024-09-10T18:44:50.370539+0000","last_clean_scrub_stamp":"2024-09-10T18:44:50.370539+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:09.273392+0000","scrub_duration":0.0015470880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178370+0000","last_change":"2024-09-10T18:44:37.049017+0000","last_active":"2024-09-10T18:45:25.178370+0000","last_peered":"2024-09-10T18:45:25.178370+0000","last_clean":"2024-09-10T18:45:25.178370+0000","last_became_active":"2024-09-10T18:39:00.909261+0000","last_became_peered":"2024-09-10T18:39:00.909261+0000","last_unstale":"2024-09-10T18:45:25.178370+0000","last_undegraded":"2024-09-10T18:45:25.178370+0000","last_fullsized":"2024-09-10T18:45:25.178370+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:37.048955+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:37.048955+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:48.141729+0000","scrub_duration":0.0031844540000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"400'46","reported_seq":603,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.078293+0000","last_change":"2024-09-10T18:45:45.078248+0000","last_active":"2024-09-10T18:45:45.078293+0000","last_peered":"2024-09-10T18:45:45.078293+0000","last_clean":"2024-09-10T18:45:45.078293+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:45:45.078293+0000","last_undegraded":"2024-09-10T18:45:45.078293+0000","last_fullsized":"2024-09-10T18:45:45.078293+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'46","last_scrub_stamp":"2024-09-10T18:45:45.078152+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:45:45.078152+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:12.093029+0000","scrub_duration":0.0021668640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":505,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.035524+0000","last_change":"2024-09-10T18:45:03.468180+0000","last_active":"2024-09-10T18:45:25.035524+0000","last_peered":"2024-09-10T18:45:25.035524+0000","last_clean":"2024-09-10T18:45:25.035524+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:45:25.035524+0000","last_undegraded":"2024-09-10T18:45:25.035524+0000","last_fullsized":"2024-09-10T18:45:25.035524+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:03.468079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:45:03.468079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:23.740107+0000","scrub_duration":0.002418201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"430'12","reported_seq":872,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.044362+0000","last_change":"2024-09-10T18:45:37.589204+0000","last_active":"2024-09-10T18:45:45.044362+0000","last_peered":"2024-09-10T18:45:45.044362+0000","last_clean":"2024-09-10T18:45:45.044362+0000","last_became_active":"2024-09-10T18:44:06.174179+0000","last_became_peered":"2024-09-10T18:44:06.174179+0000","last_unstale":"2024-09-10T18:45:45.044362+0000","last_undegraded":"2024-09-10T18:45:45.044362+0000","last_fullsized":"2024-09-10T18:45:45.044362+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'12","last_scrub_stamp":"2024-09-10T18:45:37.589138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:45:37.589138+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:00.198236+0000","scrub_duration":0.0041030880000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":167,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:48.048480+0000","last_change":"2024-09-10T18:45:48.048409+0000","last_active":"2024-09-10T18:45:48.048480+0000","last_peered":"2024-09-10T18:45:48.048480+0000","last_clean":"2024-09-10T18:45:48.048480+0000","last_became_active":"2024-09-10T18:44:06.170517+0000","last_became_peered":"2024-09-10T18:44:06.170517+0000","last_unstale":"2024-09-10T18:45:48.048480+0000","last_undegraded":"2024-09-10T18:45:48.048480+0000","last_fullsized":"2024-09-10T18:45:48.048480+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:48.048342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:45:48.048342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:54.246653+0000","scrub_duration":0.0016251130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":448,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.038363+0000","last_change":"2024-09-10T18:44:53.672481+0000","last_active":"2024-09-10T18:45:25.038363+0000","last_peered":"2024-09-10T18:45:25.038363+0000","last_clean":"2024-09-10T18:45:25.038363+0000","last_became_active":"2024-09-10T18:34:32.815839+0000","last_became_peered":"2024-09-10T18:34:32.815839+0000","last_unstale":"2024-09-10T18:45:25.038363+0000","last_undegraded":"2024-09-10T18:45:25.038363+0000","last_fullsized":"2024-09-10T18:45:25.038363+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:53.672428+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:53.672428+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:00.070252+0000","scrub_duration":0.0033041030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"400'42","reported_seq":627,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.351924+0000","last_change":"2024-09-10T18:45:36.351883+0000","last_active":"2024-09-10T18:45:36.351924+0000","last_peered":"2024-09-10T18:45:36.351924+0000","last_clean":"2024-09-10T18:45:36.351924+0000","last_became_active":"2024-09-10T18:44:06.163204+0000","last_became_peered":"2024-09-10T18:44:06.163204+0000","last_unstale":"2024-09-10T18:45:36.351924+0000","last_undegraded":"2024-09-10T18:45:36.351924+0000","last_fullsized":"2024-09-10T18:45:36.351924+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'42","last_scrub_stamp":"2024-09-10T18:45:36.351816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:45:36.351816+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:53.580868+0000","scrub_duration":0.0020778580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":515,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.036724+0000","last_change":"2024-09-10T18:44:29.407186+0000","last_active":"2024-09-10T18:45:25.036724+0000","last_peered":"2024-09-10T18:45:25.036724+0000","last_clean":"2024-09-10T18:45:25.036724+0000","last_became_active":"2024-09-10T18:44:06.161964+0000","last_became_peered":"2024-09-10T18:44:06.161964+0000","last_unstale":"2024-09-10T18:45:25.036724+0000","last_undegraded":"2024-09-10T18:45:25.036724+0000","last_fullsized":"2024-09-10T18:45:25.036724+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:29.407120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:15.765392+0000","last_clean_scrub_stamp":"2024-09-10T18:44:29.407120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:57.792649+0000","scrub_duration":0.0015259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":508,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.037073+0000","last_change":"2024-09-10T18:45:03.758819+0000","last_active":"2024-09-10T18:45:25.037073+0000","last_peered":"2024-09-10T18:45:25.037073+0000","last_clean":"2024-09-10T18:45:25.037073+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:45:25.037073+0000","last_undegraded":"2024-09-10T18:45:25.037073+0000","last_fullsized":"2024-09-10T18:45:25.037073+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:03.758749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:45:03.758749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:07.832659+0000","scrub_duration":0.0017596980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":496,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.036958+0000","last_change":"2024-09-10T18:44:18.548351+0000","last_active":"2024-09-10T18:45:25.036958+0000","last_peered":"2024-09-10T18:45:25.036958+0000","last_clean":"2024-09-10T18:45:25.036958+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:45:25.036958+0000","last_undegraded":"2024-09-10T18:45:25.036958+0000","last_fullsized":"2024-09-10T18:45:25.036958+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:18.548258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:41.282974+0000","last_clean_scrub_stamp":"2024-09-10T18:44:18.548258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:47.727148+0000","scrub_duration":0.0017483010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":430,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.035936+0000","last_change":"2024-09-10T18:44:21.578381+0000","last_active":"2024-09-10T18:45:25.035936+0000","last_peered":"2024-09-10T18:45:25.035936+0000","last_clean":"2024-09-10T18:45:25.035936+0000","last_became_active":"2024-09-10T18:39:00.894864+0000","last_became_peered":"2024-09-10T18:39:00.894864+0000","last_unstale":"2024-09-10T18:45:25.035936+0000","last_undegraded":"2024-09-10T18:45:25.035936+0000","last_fullsized":"2024-09-10T18:45:25.035936+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:21.578319+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:21.578319+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:42.924555+0000","scrub_duration":0.0014340749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"454'49","reported_seq":639,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:28.565366+0000","last_change":"2024-09-10T18:45:28.565307+0000","last_active":"2024-09-10T18:45:28.565366+0000","last_peered":"2024-09-10T18:45:28.565366+0000","last_clean":"2024-09-10T18:45:28.565366+0000","last_became_active":"2024-09-10T18:39:00.903341+0000","last_became_peered":"2024-09-10T18:39:00.903341+0000","last_unstale":"2024-09-10T18:45:28.565366+0000","last_undegraded":"2024-09-10T18:45:28.565366+0000","last_fullsized":"2024-09-10T18:45:28.565366+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'49","last_scrub_stamp":"2024-09-10T18:45:28.565240+0000","last_deep_scrub":"454'49","last_deep_scrub_stamp":"2024-09-10T18:45:28.565240+0000","last_clean_scrub_stamp":"2024-09-10T18:45:28.565240+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:44.001456+0000","scrub_duration":0.0054877279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":55,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:30.336251+0000","last_change":"2024-09-10T18:45:30.336217+0000","last_active":"2024-09-10T18:45:30.336251+0000","last_peered":"2024-09-10T18:45:30.336251+0000","last_clean":"2024-09-10T18:45:30.336251+0000","last_became_active":"2024-09-10T18:44:06.173448+0000","last_became_peered":"2024-09-10T18:44:06.173448+0000","last_unstale":"2024-09-10T18:45:30.336251+0000","last_undegraded":"2024-09-10T18:45:30.336251+0000","last_fullsized":"2024-09-10T18:45:30.336251+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:30.336164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:45:30.336164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:00.126568+0000","scrub_duration":0.0017263249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":792,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.044544+0000","last_change":"2024-09-10T18:45:11.461223+0000","last_active":"2024-09-10T18:45:45.044544+0000","last_peered":"2024-09-10T18:45:45.044544+0000","last_clean":"2024-09-10T18:45:45.044544+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:45:45.044544+0000","last_undegraded":"2024-09-10T18:45:45.044544+0000","last_fullsized":"2024-09-10T18:45:45.044544+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:45:11.461149+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:45:11.461149+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:34.810193+0000","scrub_duration":0.001663917,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":507,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.404823+0000","last_change":"2024-09-10T18:44:50.042378+0000","last_active":"2024-09-10T18:45:25.404823+0000","last_peered":"2024-09-10T18:45:25.404823+0000","last_clean":"2024-09-10T18:45:25.404823+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:45:25.404823+0000","last_undegraded":"2024-09-10T18:45:25.404823+0000","last_fullsized":"2024-09-10T18:45:25.404823+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:50.042313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:44:50.042313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:03.300708+0000","scrub_duration":0.0013999069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":530,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:34.796505+0000","last_change":"2024-09-10T18:45:34.796453+0000","last_active":"2024-09-10T18:45:34.796505+0000","last_peered":"2024-09-10T18:45:34.796505+0000","last_clean":"2024-09-10T18:45:34.796505+0000","last_became_active":"2024-09-10T18:34:32.818323+0000","last_became_peered":"2024-09-10T18:34:32.818323+0000","last_unstale":"2024-09-10T18:45:34.796505+0000","last_undegraded":"2024-09-10T18:45:34.796505+0000","last_fullsized":"2024-09-10T18:45:34.796505+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:34.796382+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:45:34.796382+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:03.611573+0000","scrub_duration":0.0018127449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"400'35","reported_seq":626,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:44.088828+0000","last_change":"2024-09-10T18:45:44.088792+0000","last_active":"2024-09-10T18:45:44.088828+0000","last_peered":"2024-09-10T18:45:44.088828+0000","last_clean":"2024-09-10T18:45:44.088828+0000","last_became_active":"2024-09-10T18:39:00.897648+0000","last_became_peered":"2024-09-10T18:39:00.897648+0000","last_unstale":"2024-09-10T18:45:44.088828+0000","last_undegraded":"2024-09-10T18:45:44.088828+0000","last_fullsized":"2024-09-10T18:45:44.088828+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'35","last_scrub_stamp":"2024-09-10T18:45:44.088721+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:45:44.088721+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:59.547739+0000","scrub_duration":0.0023048360000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":147,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:39.042019+0000","last_change":"2024-09-10T18:45:39.041977+0000","last_active":"2024-09-10T18:45:39.042019+0000","last_peered":"2024-09-10T18:45:39.042019+0000","last_clean":"2024-09-10T18:45:39.042019+0000","last_became_active":"2024-09-10T18:39:00.909066+0000","last_became_peered":"2024-09-10T18:39:00.909066+0000","last_unstale":"2024-09-10T18:45:39.042019+0000","last_undegraded":"2024-09-10T18:45:39.042019+0000","last_fullsized":"2024-09-10T18:45:39.042019+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:39.041862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:45:39.041862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:44.128944+0000","scrub_duration":0.0021130200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"430'15","reported_seq":69,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:40.043435+0000","last_change":"2024-09-10T18:44:40.433166+0000","last_active":"2024-09-10T18:45:40.043435+0000","last_peered":"2024-09-10T18:45:40.043435+0000","last_clean":"2024-09-10T18:45:40.043435+0000","last_became_active":"2024-09-10T18:44:06.170798+0000","last_became_peered":"2024-09-10T18:44:06.170798+0000","last_unstale":"2024-09-10T18:45:40.043435+0000","last_undegraded":"2024-09-10T18:45:40.043435+0000","last_fullsized":"2024-09-10T18:45:40.043435+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'15","last_scrub_stamp":"2024-09-10T18:44:40.433110+0000","last_deep_scrub":"430'15","last_deep_scrub_stamp":"2024-09-10T18:44:40.433110+0000","last_clean_scrub_stamp":"2024-09-10T18:44:40.433110+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:01.497967+0000","scrub_duration":0.0031203229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:26.303262+0000","last_change":"2024-09-10T18:45:26.303226+0000","last_active":"2024-09-10T18:45:26.303262+0000","last_peered":"2024-09-10T18:45:26.303262+0000","last_clean":"2024-09-10T18:45:26.303262+0000","last_became_active":"2024-09-10T18:44:06.168574+0000","last_became_peered":"2024-09-10T18:44:06.168574+0000","last_unstale":"2024-09-10T18:45:26.303262+0000","last_undegraded":"2024-09-10T18:45:26.303262+0000","last_fullsized":"2024-09-10T18:45:26.303262+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:26.303170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:45:26.303170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:44.559014+0000","scrub_duration":0.0011554740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":540,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.382294+0000","last_change":"2024-09-10T18:45:12.938053+0000","last_active":"2024-09-10T18:45:25.382294+0000","last_peered":"2024-09-10T18:45:25.382294+0000","last_clean":"2024-09-10T18:45:25.382294+0000","last_became_active":"2024-09-10T18:44:06.171130+0000","last_became_peered":"2024-09-10T18:44:06.171130+0000","last_unstale":"2024-09-10T18:45:25.382294+0000","last_undegraded":"2024-09-10T18:45:25.382294+0000","last_fullsized":"2024-09-10T18:45:25.382294+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:12.937962+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:45:12.937962+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:34.068642+0000","scrub_duration":0.001852795,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"400'39","reported_seq":73,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.658271+0000","last_change":"2024-09-10T18:45:16.353481+0000","last_active":"2024-09-10T18:45:25.658271+0000","last_peered":"2024-09-10T18:45:25.658271+0000","last_clean":"2024-09-10T18:45:25.658271+0000","last_became_active":"2024-09-10T18:44:06.160218+0000","last_became_peered":"2024-09-10T18:44:06.160218+0000","last_unstale":"2024-09-10T18:45:25.658271+0000","last_undegraded":"2024-09-10T18:45:25.658271+0000","last_fullsized":"2024-09-10T18:45:25.658271+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'39","last_scrub_stamp":"2024-09-10T18:45:16.353408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:45:16.353408+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:41.419564+0000","scrub_duration":0.0024552179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":534,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.108169+0000","last_change":"2024-09-10T18:44:59.294759+0000","last_active":"2024-09-10T18:45:25.108169+0000","last_peered":"2024-09-10T18:45:25.108169+0000","last_clean":"2024-09-10T18:45:25.108169+0000","last_became_active":"2024-09-10T18:39:00.897360+0000","last_became_peered":"2024-09-10T18:39:00.897360+0000","last_unstale":"2024-09-10T18:45:25.108169+0000","last_undegraded":"2024-09-10T18:45:25.108169+0000","last_fullsized":"2024-09-10T18:45:25.108169+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:59.294705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:44:59.294705+0000","last_clean_scrub_stamp":"2024-09-10T18:44:59.294705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:07.677285+0000","scrub_duration":0.0015780189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":540,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.381919+0000","last_change":"2024-09-10T18:45:10.886010+0000","last_active":"2024-09-10T18:45:25.381919+0000","last_peered":"2024-09-10T18:45:25.381919+0000","last_clean":"2024-09-10T18:45:25.381919+0000","last_became_active":"2024-09-10T18:44:06.170664+0000","last_became_peered":"2024-09-10T18:44:06.170664+0000","last_unstale":"2024-09-10T18:45:25.381919+0000","last_undegraded":"2024-09-10T18:45:25.381919+0000","last_fullsized":"2024-09-10T18:45:25.381919+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:10.885938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:43:51.751769+0000","last_clean_scrub_stamp":"2024-09-10T18:45:10.885938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:18.607347+0000","scrub_duration":0.001637416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":515,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.107418+0000","last_change":"2024-09-10T18:44:36.057332+0000","last_active":"2024-09-10T18:45:25.107418+0000","last_peered":"2024-09-10T18:45:25.107418+0000","last_clean":"2024-09-10T18:45:25.107418+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:45:25.107418+0000","last_undegraded":"2024-09-10T18:45:25.107418+0000","last_fullsized":"2024-09-10T18:45:25.107418+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:36.057276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:44:36.057276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:40.780596+0000","scrub_duration":0.0018953100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":157,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178052+0000","last_change":"2024-09-10T18:44:52.958954+0000","last_active":"2024-09-10T18:45:25.178052+0000","last_peered":"2024-09-10T18:45:25.178052+0000","last_clean":"2024-09-10T18:45:25.178052+0000","last_became_active":"2024-09-10T18:44:06.171188+0000","last_became_peered":"2024-09-10T18:44:06.171188+0000","last_unstale":"2024-09-10T18:45:25.178052+0000","last_undegraded":"2024-09-10T18:45:25.178052+0000","last_fullsized":"2024-09-10T18:45:25.178052+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:52.958889+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:52.958889+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:55.212589+0000","scrub_duration":0.0015928489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"410'64","reported_seq":353,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.044299+0000","last_change":"2024-09-10T18:44:30.068625+0000","last_active":"2024-09-10T18:45:45.044299+0000","last_peered":"2024-09-10T18:45:45.044299+0000","last_clean":"2024-09-10T18:45:45.044299+0000","last_became_active":"2024-09-10T18:36:12.922665+0000","last_became_peered":"2024-09-10T18:36:12.922665+0000","last_unstale":"2024-09-10T18:45:45.044299+0000","last_undegraded":"2024-09-10T18:45:45.044299+0000","last_fullsized":"2024-09-10T18:45:45.044299+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'64","last_scrub_stamp":"2024-09-10T18:44:30.068561+0000","last_deep_scrub":"410'64","last_deep_scrub_stamp":"2024-09-10T18:42:43.154380+0000","last_clean_scrub_stamp":"2024-09-10T18:44:30.068561+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:52.195667+0000","scrub_duration":0.0021106079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":529,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.037677+0000","last_change":"2024-09-10T18:44:27.556717+0000","last_active":"2024-09-10T18:45:25.037677+0000","last_peered":"2024-09-10T18:45:25.037677+0000","last_clean":"2024-09-10T18:45:25.037677+0000","last_became_active":"2024-09-10T18:44:06.172679+0000","last_became_peered":"2024-09-10T18:44:06.172679+0000","last_unstale":"2024-09-10T18:45:25.037677+0000","last_undegraded":"2024-09-10T18:45:25.037677+0000","last_fullsized":"2024-09-10T18:45:25.037677+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:27.556656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.274219+0000","last_clean_scrub_stamp":"2024-09-10T18:44:27.556656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:50.380293+0000","scrub_duration":0.0032193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":141,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178457+0000","last_change":"2024-09-10T18:45:04.135777+0000","last_active":"2024-09-10T18:45:25.178457+0000","last_peered":"2024-09-10T18:45:25.178457+0000","last_clean":"2024-09-10T18:45:25.178457+0000","last_became_active":"2024-09-10T18:39:00.909579+0000","last_became_peered":"2024-09-10T18:39:00.909579+0000","last_unstale":"2024-09-10T18:45:25.178457+0000","last_undegraded":"2024-09-10T18:45:25.178457+0000","last_fullsized":"2024-09-10T18:45:25.178457+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:45:04.135669+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:45:04.135669+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:04.780106+0000","scrub_duration":0.0030437379999999998,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.658201+0000","last_change":"2024-09-10T18:44:47.459708+0000","last_active":"2024-09-10T18:45:25.658201+0000","last_peered":"2024-09-10T18:45:25.658201+0000","last_clean":"2024-09-10T18:45:25.658201+0000","last_became_active":"2024-09-10T18:44:06.167169+0000","last_became_peered":"2024-09-10T18:44:06.167169+0000","last_unstale":"2024-09-10T18:45:25.658201+0000","last_undegraded":"2024-09-10T18:45:25.658201+0000","last_fullsized":"2024-09-10T18:45:25.658201+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:47.459654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:44:47.459654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:53.413488+0000","scrub_duration":0.001576222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":492,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:28.447189+0000","last_change":"2024-09-10T18:45:28.447152+0000","last_active":"2024-09-10T18:45:28.447189+0000","last_peered":"2024-09-10T18:45:28.447189+0000","last_clean":"2024-09-10T18:45:28.447189+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:45:28.447189+0000","last_undegraded":"2024-09-10T18:45:28.447189+0000","last_fullsized":"2024-09-10T18:45:28.447189+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:28.447098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:45:28.447098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:57.380889+0000","scrub_duration":0.001909354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":584,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:33.416285+0000","last_change":"2024-09-10T18:45:33.416237+0000","last_active":"2024-09-10T18:45:33.416285+0000","last_peered":"2024-09-10T18:45:33.416285+0000","last_clean":"2024-09-10T18:45:33.416285+0000","last_became_active":"2024-09-10T18:44:06.160358+0000","last_became_peered":"2024-09-10T18:44:06.160358+0000","last_unstale":"2024-09-10T18:45:33.416285+0000","last_undegraded":"2024-09-10T18:45:33.416285+0000","last_fullsized":"2024-09-10T18:45:33.416285+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:33.416167+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:45:33.416167+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:50.618981+0000","scrub_duration":0.0016950649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"400'26","reported_seq":142,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.276354+0000","last_change":"2024-09-10T18:45:14.771571+0000","last_active":"2024-09-10T18:45:25.276354+0000","last_peered":"2024-09-10T18:45:25.276354+0000","last_clean":"2024-09-10T18:45:25.276354+0000","last_became_active":"2024-09-10T18:44:06.173529+0000","last_became_peered":"2024-09-10T18:44:06.173529+0000","last_unstale":"2024-09-10T18:45:25.276354+0000","last_undegraded":"2024-09-10T18:45:25.276354+0000","last_fullsized":"2024-09-10T18:45:25.276354+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'26","last_scrub_stamp":"2024-09-10T18:45:14.771498+0000","last_deep_scrub":"388'22","last_deep_scrub_stamp":"2024-09-10T18:41:01.248929+0000","last_clean_scrub_stamp":"2024-09-10T18:45:14.771498+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:27.715714+0000","scrub_duration":0.0021619310000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":541,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:40.118123+0000","last_change":"2024-09-10T18:45:40.118074+0000","last_active":"2024-09-10T18:45:40.118123+0000","last_peered":"2024-09-10T18:45:40.118123+0000","last_clean":"2024-09-10T18:45:40.118123+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:45:40.118123+0000","last_undegraded":"2024-09-10T18:45:40.118123+0000","last_fullsized":"2024-09-10T18:45:40.118123+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:40.118015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:44.171445+0000","last_clean_scrub_stamp":"2024-09-10T18:45:40.118015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:56.250485+0000","scrub_duration":0.0016665600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":589,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.275889+0000","last_change":"2024-09-10T18:45:11.772433+0000","last_active":"2024-09-10T18:45:25.275889+0000","last_peered":"2024-09-10T18:45:25.275889+0000","last_clean":"2024-09-10T18:45:25.275889+0000","last_became_active":"2024-09-10T18:44:06.170923+0000","last_became_peered":"2024-09-10T18:44:06.170923+0000","last_unstale":"2024-09-10T18:45:25.275889+0000","last_undegraded":"2024-09-10T18:45:25.275889+0000","last_fullsized":"2024-09-10T18:45:25.275889+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:45:11.772332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:45:11.772332+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:22.236606+0000","scrub_duration":0.001552512,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":568,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:29.760795+0000","last_change":"2024-09-10T18:45:29.760751+0000","last_active":"2024-09-10T18:45:29.760795+0000","last_peered":"2024-09-10T18:45:29.760795+0000","last_clean":"2024-09-10T18:45:29.760795+0000","last_became_active":"2024-09-10T18:39:00.910574+0000","last_became_peered":"2024-09-10T18:39:00.910574+0000","last_unstale":"2024-09-10T18:45:29.760795+0000","last_undegraded":"2024-09-10T18:45:29.760795+0000","last_fullsized":"2024-09-10T18:45:29.760795+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:29.760692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:45:29.760692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:58.558452+0000","scrub_duration":0.0016954629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":507,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405511+0000","last_change":"2024-09-10T18:44:46.062649+0000","last_active":"2024-09-10T18:45:25.405511+0000","last_peered":"2024-09-10T18:45:25.405511+0000","last_clean":"2024-09-10T18:45:25.405511+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:45:25.405511+0000","last_undegraded":"2024-09-10T18:45:25.405511+0000","last_fullsized":"2024-09-10T18:45:25.405511+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:46.062578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:44:46.062578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:11.953294+0000","scrub_duration":0.0016260160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":434,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.107881+0000","last_change":"2024-09-10T18:44:23.030550+0000","last_active":"2024-09-10T18:45:25.107881+0000","last_peered":"2024-09-10T18:45:25.107881+0000","last_clean":"2024-09-10T18:45:25.107881+0000","last_became_active":"2024-09-10T18:34:00.165384+0000","last_became_peered":"2024-09-10T18:34:00.165384+0000","last_unstale":"2024-09-10T18:45:25.107881+0000","last_undegraded":"2024-09-10T18:45:25.107881+0000","last_fullsized":"2024-09-10T18:45:25.107881+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:23.030493+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:44:23.030493+0000","last_clean_scrub_stamp":"2024-09-10T18:44:23.030493+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:47.160426+0000","scrub_duration":0.0018700380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"430'42","reported_seq":75,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:31.301959+0000","last_change":"2024-09-10T18:45:31.301887+0000","last_active":"2024-09-10T18:45:31.301959+0000","last_peered":"2024-09-10T18:45:31.301959+0000","last_clean":"2024-09-10T18:45:31.301959+0000","last_became_active":"2024-09-10T18:44:06.176362+0000","last_became_peered":"2024-09-10T18:44:06.176362+0000","last_unstale":"2024-09-10T18:45:31.301959+0000","last_undegraded":"2024-09-10T18:45:31.301959+0000","last_fullsized":"2024-09-10T18:45:31.301959+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'42","last_scrub_stamp":"2024-09-10T18:45:31.301813+0000","last_deep_scrub":"430'42","last_deep_scrub_stamp":"2024-09-10T18:44:09.491797+0000","last_clean_scrub_stamp":"2024-09-10T18:45:31.301813+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:42.237027+0000","scrub_duration":0.0017048230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":55,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.302296+0000","last_change":"2024-09-10T18:45:36.302252+0000","last_active":"2024-09-10T18:45:36.302296+0000","last_peered":"2024-09-10T18:45:36.302296+0000","last_clean":"2024-09-10T18:45:36.302296+0000","last_became_active":"2024-09-10T18:44:06.164270+0000","last_became_peered":"2024-09-10T18:44:06.164270+0000","last_unstale":"2024-09-10T18:45:36.302296+0000","last_undegraded":"2024-09-10T18:45:36.302296+0000","last_fullsized":"2024-09-10T18:45:36.302296+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:36.302197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:45:36.302197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:03.497915+0000","scrub_duration":0.001270715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":1997,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:48.317325+0000","last_change":"2024-09-10T18:44:52.094924+0000","last_active":"2024-09-10T18:45:48.317325+0000","last_peered":"2024-09-10T18:45:48.317325+0000","last_clean":"2024-09-10T18:45:48.317325+0000","last_became_active":"2024-09-10T18:44:06.165291+0000","last_became_peered":"2024-09-10T18:44:06.165291+0000","last_unstale":"2024-09-10T18:45:48.317325+0000","last_undegraded":"2024-09-10T18:45:48.317325+0000","last_fullsized":"2024-09-10T18:45:48.317325+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:44:52.094828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:44:52.094828+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:14.362433+0000","scrub_duration":0.002081846,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1446,"num_read_kb":1441,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":157,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:38.074401+0000","last_change":"2024-09-10T18:45:38.074366+0000","last_active":"2024-09-10T18:45:38.074401+0000","last_peered":"2024-09-10T18:45:38.074401+0000","last_clean":"2024-09-10T18:45:38.074401+0000","last_became_active":"2024-09-10T18:44:06.172014+0000","last_became_peered":"2024-09-10T18:44:06.172014+0000","last_unstale":"2024-09-10T18:45:38.074401+0000","last_undegraded":"2024-09-10T18:45:38.074401+0000","last_fullsized":"2024-09-10T18:45:38.074401+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:38.074301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.170272+0000","last_clean_scrub_stamp":"2024-09-10T18:45:38.074301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:48.727311+0000","scrub_duration":0.001632566,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":528,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.107697+0000","last_change":"2024-09-10T18:44:58.257403+0000","last_active":"2024-09-10T18:45:25.107697+0000","last_peered":"2024-09-10T18:45:25.107697+0000","last_clean":"2024-09-10T18:45:25.107697+0000","last_became_active":"2024-09-10T18:39:00.897525+0000","last_became_peered":"2024-09-10T18:39:00.897525+0000","last_unstale":"2024-09-10T18:45:25.107697+0000","last_undegraded":"2024-09-10T18:45:25.107697+0000","last_fullsized":"2024-09-10T18:45:25.107697+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:58.257331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:44:58.257331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:25.650629+0000","scrub_duration":0.0015563700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"400'38","reported_seq":545,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:33.834208+0000","last_change":"2024-09-10T18:45:33.834140+0000","last_active":"2024-09-10T18:45:33.834208+0000","last_peered":"2024-09-10T18:45:33.834208+0000","last_clean":"2024-09-10T18:45:33.834208+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:45:33.834208+0000","last_undegraded":"2024-09-10T18:45:33.834208+0000","last_fullsized":"2024-09-10T18:45:33.834208+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'38","last_scrub_stamp":"2024-09-10T18:45:33.834062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:45:33.834062+0000","objects_scrubbed":7,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:43.745833+0000","scrub_duration":0.0022866380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":532,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:35.332074+0000","last_change":"2024-09-10T18:45:35.331998+0000","last_active":"2024-09-10T18:45:35.332074+0000","last_peered":"2024-09-10T18:45:35.332074+0000","last_clean":"2024-09-10T18:45:35.332074+0000","last_became_active":"2024-09-10T18:39:00.894778+0000","last_became_peered":"2024-09-10T18:39:00.894778+0000","last_unstale":"2024-09-10T18:45:35.332074+0000","last_undegraded":"2024-09-10T18:45:35.332074+0000","last_fullsized":"2024-09-10T18:45:35.332074+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:35.331930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:45:35.331930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:42.960860+0000","scrub_duration":0.001699964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":593,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405285+0000","last_change":"2024-09-10T18:45:03.266337+0000","last_active":"2024-09-10T18:45:25.405285+0000","last_peered":"2024-09-10T18:45:25.405285+0000","last_clean":"2024-09-10T18:45:25.405285+0000","last_became_active":"2024-09-10T18:44:06.165287+0000","last_became_peered":"2024-09-10T18:44:06.165287+0000","last_unstale":"2024-09-10T18:45:25.405285+0000","last_undegraded":"2024-09-10T18:45:25.405285+0000","last_fullsized":"2024-09-10T18:45:25.405285+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:03.266267+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:45:03.266267+0000","last_clean_scrub_stamp":"2024-09-10T18:45:03.266267+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:20.499388+0000","scrub_duration":0.0019289680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":532,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405433+0000","last_change":"2024-09-10T18:44:59.162745+0000","last_active":"2024-09-10T18:45:25.405433+0000","last_peered":"2024-09-10T18:45:25.405433+0000","last_clean":"2024-09-10T18:45:25.405433+0000","last_became_active":"2024-09-10T18:39:00.890829+0000","last_became_peered":"2024-09-10T18:39:00.890829+0000","last_unstale":"2024-09-10T18:45:25.405433+0000","last_undegraded":"2024-09-10T18:45:25.405433+0000","last_fullsized":"2024-09-10T18:45:25.405433+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:59.162685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:44:59.162685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:10.659296+0000","scrub_duration":0.001394831,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.177795+0000","last_change":"2024-09-10T18:45:17.143890+0000","last_active":"2024-09-10T18:45:25.177795+0000","last_peered":"2024-09-10T18:45:25.177795+0000","last_clean":"2024-09-10T18:45:25.177795+0000","last_became_active":"2024-09-10T18:39:00.900703+0000","last_became_peered":"2024-09-10T18:39:00.900703+0000","last_unstale":"2024-09-10T18:45:25.177795+0000","last_undegraded":"2024-09-10T18:45:25.177795+0000","last_fullsized":"2024-09-10T18:45:25.177795+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:17.143815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:44:14.867416+0000","last_clean_scrub_stamp":"2024-09-10T18:45:17.143815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:45.666741+0000","scrub_duration":0.0013807190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"76'81","reported_seq":490,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.108063+0000","last_change":"2024-09-10T18:44:25.005414+0000","last_active":"2024-09-10T18:45:25.108063+0000","last_peered":"2024-09-10T18:45:25.108063+0000","last_clean":"2024-09-10T18:45:25.108063+0000","last_became_active":"2024-09-10T18:44:06.163834+0000","last_became_peered":"2024-09-10T18:44:06.163834+0000","last_unstale":"2024-09-10T18:45:25.108063+0000","last_undegraded":"2024-09-10T18:45:25.108063+0000","last_fullsized":"2024-09-10T18:45:25.108063+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:25.005338+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:25.005338+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:52.017381+0000","scrub_duration":0.001623403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"400'32","reported_seq":568,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.038090+0000","last_change":"2024-09-10T18:44:52.696594+0000","last_active":"2024-09-10T18:45:25.038090+0000","last_peered":"2024-09-10T18:45:25.038090+0000","last_clean":"2024-09-10T18:45:25.038090+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:45:25.038090+0000","last_undegraded":"2024-09-10T18:45:25.038090+0000","last_fullsized":"2024-09-10T18:45:25.038090+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'32","last_scrub_stamp":"2024-09-10T18:44:52.696522+0000","last_deep_scrub":"283'24","last_deep_scrub_stamp":"2024-09-10T18:34:19.404713+0000","last_clean_scrub_stamp":"2024-09-10T18:44:52.696522+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:00.970889+0000","scrub_duration":0.0022833799999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":515,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:26.555864+0000","last_change":"2024-09-10T18:45:26.555814+0000","last_active":"2024-09-10T18:45:26.555864+0000","last_peered":"2024-09-10T18:45:26.555864+0000","last_clean":"2024-09-10T18:45:26.555864+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:45:26.555864+0000","last_undegraded":"2024-09-10T18:45:26.555864+0000","last_fullsized":"2024-09-10T18:45:26.555864+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:26.555747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:45:26.555747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:35.118776+0000","scrub_duration":0.0016912190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.177800+0000","last_change":"2024-09-10T18:44:30.955440+0000","last_active":"2024-09-10T18:45:25.177800+0000","last_peered":"2024-09-10T18:45:25.177800+0000","last_clean":"2024-09-10T18:45:25.177800+0000","last_became_active":"2024-09-10T18:39:00.907231+0000","last_became_peered":"2024-09-10T18:39:00.907231+0000","last_unstale":"2024-09-10T18:45:25.177800+0000","last_undegraded":"2024-09-10T18:45:25.177800+0000","last_fullsized":"2024-09-10T18:45:25.177800+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:30.955372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:44:30.955372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:52.363842+0000","scrub_duration":0.003736241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":526,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.036047+0000","last_change":"2024-09-10T18:45:22.450293+0000","last_active":"2024-09-10T18:45:25.036047+0000","last_peered":"2024-09-10T18:45:25.036047+0000","last_clean":"2024-09-10T18:45:25.036047+0000","last_became_active":"2024-09-10T18:39:00.894963+0000","last_became_peered":"2024-09-10T18:39:00.894963+0000","last_unstale":"2024-09-10T18:45:25.036047+0000","last_undegraded":"2024-09-10T18:45:25.036047+0000","last_fullsized":"2024-09-10T18:45:25.036047+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:22.450241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:45:22.450241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:49.769151+0000","scrub_duration":0.0015896980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178120+0000","last_change":"2024-09-10T18:44:33.961034+0000","last_active":"2024-09-10T18:45:25.178120+0000","last_peered":"2024-09-10T18:45:25.178120+0000","last_clean":"2024-09-10T18:45:25.178120+0000","last_became_active":"2024-09-10T18:39:00.909151+0000","last_became_peered":"2024-09-10T18:39:00.909151+0000","last_unstale":"2024-09-10T18:45:25.178120+0000","last_undegraded":"2024-09-10T18:45:25.178120+0000","last_fullsized":"2024-09-10T18:45:25.178120+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:33.960921+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:33.960921+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:03.520006+0000","scrub_duration":0.0025747090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"400'53","reported_seq":644,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.038175+0000","last_change":"2024-09-10T18:45:04.735964+0000","last_active":"2024-09-10T18:45:25.038175+0000","last_peered":"2024-09-10T18:45:25.038175+0000","last_clean":"2024-09-10T18:45:25.038175+0000","last_became_active":"2024-09-10T18:38:43.333186+0000","last_became_peered":"2024-09-10T18:38:43.333186+0000","last_unstale":"2024-09-10T18:45:25.038175+0000","last_undegraded":"2024-09-10T18:45:25.038175+0000","last_fullsized":"2024-09-10T18:45:25.038175+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'53","last_scrub_stamp":"2024-09-10T18:45:04.735870+0000","last_deep_scrub":"342'39","last_deep_scrub_stamp":"2024-09-10T18:37:53.386563+0000","last_clean_scrub_stamp":"2024-09-10T18:45:04.735870+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:14.811693+0000","scrub_duration":0.0029873159999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":498,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:29.833256+0000","last_change":"2024-09-10T18:45:29.833199+0000","last_active":"2024-09-10T18:45:29.833256+0000","last_peered":"2024-09-10T18:45:29.833256+0000","last_clean":"2024-09-10T18:45:29.833256+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:45:29.833256+0000","last_undegraded":"2024-09-10T18:45:29.833256+0000","last_fullsized":"2024-09-10T18:45:29.833256+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:45:29.833136+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:45:29.833136+0000","last_clean_scrub_stamp":"2024-09-10T18:45:29.833136+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:40.817774+0000","scrub_duration":0.0021491370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":540,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:32.136004+0000","last_change":"2024-09-10T18:45:32.135970+0000","last_active":"2024-09-10T18:45:32.136004+0000","last_peered":"2024-09-10T18:45:32.136004+0000","last_clean":"2024-09-10T18:45:32.136004+0000","last_became_active":"2024-09-10T18:44:06.162385+0000","last_became_peered":"2024-09-10T18:44:06.162385+0000","last_unstale":"2024-09-10T18:45:32.136004+0000","last_undegraded":"2024-09-10T18:45:32.136004+0000","last_fullsized":"2024-09-10T18:45:32.136004+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:32.135918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:45:32.135918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:55.179790+0000","scrub_duration":0.0010348410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":556,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.276436+0000","last_change":"2024-09-10T18:45:18.788891+0000","last_active":"2024-09-10T18:45:25.276436+0000","last_peered":"2024-09-10T18:45:25.276436+0000","last_clean":"2024-09-10T18:45:25.276436+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:45:25.276436+0000","last_undegraded":"2024-09-10T18:45:25.276436+0000","last_fullsized":"2024-09-10T18:45:25.276436+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:18.788806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:45:18.788806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:43.440299+0000","scrub_duration":0.0019326199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":571,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:49.352499+0000","last_change":"2024-09-10T18:45:49.352441+0000","last_active":"2024-09-10T18:45:49.352499+0000","last_peered":"2024-09-10T18:45:49.352499+0000","last_clean":"2024-09-10T18:45:49.352499+0000","last_became_active":"2024-09-10T18:34:32.827174+0000","last_became_peered":"2024-09-10T18:34:32.827174+0000","last_unstale":"2024-09-10T18:45:49.352499+0000","last_undegraded":"2024-09-10T18:45:49.352499+0000","last_fullsized":"2024-09-10T18:45:49.352499+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:49.352364+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:44:16.864682+0000","last_clean_scrub_stamp":"2024-09-10T18:45:49.352364+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:05.821643+0000","scrub_duration":0.001932034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"400'35","reported_seq":286,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405153+0000","last_change":"2024-09-10T18:45:15.324234+0000","last_active":"2024-09-10T18:45:25.405153+0000","last_peered":"2024-09-10T18:45:25.405153+0000","last_clean":"2024-09-10T18:45:25.405153+0000","last_became_active":"2024-09-10T18:32:51.118251+0000","last_became_peered":"2024-09-10T18:32:51.118251+0000","last_unstale":"2024-09-10T18:45:25.405153+0000","last_undegraded":"2024-09-10T18:45:25.405153+0000","last_fullsized":"2024-09-10T18:45:25.405153+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'35","last_scrub_stamp":"2024-09-10T18:45:15.324166+0000","last_deep_scrub":"400'35","last_deep_scrub_stamp":"2024-09-10T18:45:15.324166+0000","last_clean_scrub_stamp":"2024-09-10T18:45:15.324166+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:21.787018+0000","scrub_duration":0.0020122489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":515,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.035906+0000","last_change":"2024-09-10T18:44:16.635241+0000","last_active":"2024-09-10T18:45:25.035906+0000","last_peered":"2024-09-10T18:45:25.035906+0000","last_clean":"2024-09-10T18:45:25.035906+0000","last_became_active":"2024-09-10T18:44:06.161655+0000","last_became_peered":"2024-09-10T18:44:06.161655+0000","last_unstale":"2024-09-10T18:45:25.035906+0000","last_undegraded":"2024-09-10T18:45:25.035906+0000","last_fullsized":"2024-09-10T18:45:25.035906+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:16.635176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:59.757255+0000","last_clean_scrub_stamp":"2024-09-10T18:44:16.635176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:45.437654+0000","scrub_duration":0.001634649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"357'11","reported_seq":888,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.044457+0000","last_change":"2024-09-10T18:45:32.726221+0000","last_active":"2024-09-10T18:45:45.044457+0000","last_peered":"2024-09-10T18:45:45.044457+0000","last_clean":"2024-09-10T18:45:45.044457+0000","last_became_active":"2024-09-10T18:39:00.909747+0000","last_became_peered":"2024-09-10T18:39:00.909747+0000","last_unstale":"2024-09-10T18:45:45.044457+0000","last_undegraded":"2024-09-10T18:45:45.044457+0000","last_fullsized":"2024-09-10T18:45:45.044457+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'11","last_scrub_stamp":"2024-09-10T18:45:32.726145+0000","last_deep_scrub":"357'11","last_deep_scrub_stamp":"2024-09-10T18:44:06.249311+0000","last_clean_scrub_stamp":"2024-09-10T18:45:32.726145+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:49.065008+0000","scrub_duration":0.0020634149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":499,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.037919+0000","last_change":"2024-09-10T18:44:49.693689+0000","last_active":"2024-09-10T18:45:25.037919+0000","last_peered":"2024-09-10T18:45:25.037919+0000","last_clean":"2024-09-10T18:45:25.037919+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:45:25.037919+0000","last_undegraded":"2024-09-10T18:45:25.037919+0000","last_fullsized":"2024-09-10T18:45:25.037919+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:49.693531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:53.987811+0000","last_clean_scrub_stamp":"2024-09-10T18:44:49.693531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:13.648425+0000","scrub_duration":0.0021128950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"76'81","reported_seq":242,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.037891+0000","last_change":"2024-09-10T18:44:45.718577+0000","last_active":"2024-09-10T18:45:25.037891+0000","last_peered":"2024-09-10T18:45:25.037891+0000","last_clean":"2024-09-10T18:45:25.037891+0000","last_became_active":"2024-09-10T18:34:32.818251+0000","last_became_peered":"2024-09-10T18:34:32.818251+0000","last_unstale":"2024-09-10T18:45:25.037891+0000","last_undegraded":"2024-09-10T18:45:25.037891+0000","last_fullsized":"2024-09-10T18:45:25.037891+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:45.718509+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:42:53.168123+0000","last_clean_scrub_stamp":"2024-09-10T18:44:45.718509+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:54.289759+0000","scrub_duration":0.0018204969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"400'25","reported_seq":55,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.658163+0000","last_change":"2024-09-10T18:44:27.399562+0000","last_active":"2024-09-10T18:45:25.658163+0000","last_peered":"2024-09-10T18:45:25.658163+0000","last_clean":"2024-09-10T18:45:25.658163+0000","last_became_active":"2024-09-10T18:44:06.169580+0000","last_became_peered":"2024-09-10T18:44:06.169580+0000","last_unstale":"2024-09-10T18:45:25.658163+0000","last_undegraded":"2024-09-10T18:45:25.658163+0000","last_fullsized":"2024-09-10T18:45:25.658163+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'25","last_scrub_stamp":"2024-09-10T18:44:27.399496+0000","last_deep_scrub":"302'19","last_deep_scrub_stamp":"2024-09-10T18:35:18.836131+0000","last_clean_scrub_stamp":"2024-09-10T18:44:27.399496+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:47.178093+0000","scrub_duration":0.002811334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"265'2","reported_seq":151,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:50.111077+0000","last_change":"2024-09-10T18:45:50.111017+0000","last_active":"2024-09-10T18:45:50.111077+0000","last_peered":"2024-09-10T18:45:50.111077+0000","last_clean":"2024-09-10T18:45:50.111077+0000","last_became_active":"2024-09-10T18:39:00.904967+0000","last_became_peered":"2024-09-10T18:39:00.904967+0000","last_unstale":"2024-09-10T18:45:50.111077+0000","last_undegraded":"2024-09-10T18:45:50.111077+0000","last_fullsized":"2024-09-10T18:45:50.111077+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'2","last_scrub_stamp":"2024-09-10T18:45:50.110941+0000","last_deep_scrub":"265'2","last_deep_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_clean_scrub_stamp":"2024-09-10T18:45:50.110941+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:58.081284+0000","scrub_duration":0.0018958600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":544,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.275923+0000","last_change":"2024-09-10T18:45:16.751583+0000","last_active":"2024-09-10T18:45:25.275923+0000","last_peered":"2024-09-10T18:45:25.275923+0000","last_clean":"2024-09-10T18:45:25.275923+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:45:25.275923+0000","last_undegraded":"2024-09-10T18:45:25.275923+0000","last_fullsized":"2024-09-10T18:45:25.275923+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:16.751528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:45:16.751528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:44.514774+0000","scrub_duration":0.0017661840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.658050+0000","last_change":"2024-09-10T18:45:14.424451+0000","last_active":"2024-09-10T18:45:25.658050+0000","last_peered":"2024-09-10T18:45:25.658050+0000","last_clean":"2024-09-10T18:45:25.658050+0000","last_became_active":"2024-09-10T18:44:06.168584+0000","last_became_peered":"2024-09-10T18:44:06.168584+0000","last_unstale":"2024-09-10T18:45:25.658050+0000","last_undegraded":"2024-09-10T18:45:25.658050+0000","last_fullsized":"2024-09-10T18:45:25.658050+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:14.424387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:45:14.424387+0000","last_clean_scrub_stamp":"2024-09-10T18:45:14.424387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:28.659520+0000","scrub_duration":0.0016728649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'81","reported_seq":529,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.381460+0000","last_change":"2024-09-10T18:44:41.625296+0000","last_active":"2024-09-10T18:45:25.381460+0000","last_peered":"2024-09-10T18:45:25.381460+0000","last_clean":"2024-09-10T18:45:25.381460+0000","last_became_active":"2024-09-10T18:39:00.921868+0000","last_became_peered":"2024-09-10T18:39:00.921868+0000","last_unstale":"2024-09-10T18:45:25.381460+0000","last_undegraded":"2024-09-10T18:45:25.381460+0000","last_fullsized":"2024-09-10T18:45:25.381460+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:41.625226+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:44:41.625226+0000","last_clean_scrub_stamp":"2024-09-10T18:44:41.625226+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:50.236128+0000","scrub_duration":0.0016452070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"400'60","reported_seq":639,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.381987+0000","last_change":"2024-09-10T18:45:08.864502+0000","last_active":"2024-09-10T18:45:25.381987+0000","last_peered":"2024-09-10T18:45:25.381987+0000","last_clean":"2024-09-10T18:45:25.381987+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:45:25.381987+0000","last_undegraded":"2024-09-10T18:45:25.381987+0000","last_fullsized":"2024-09-10T18:45:25.381987+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'60","last_scrub_stamp":"2024-09-10T18:45:08.864434+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:45:08.864434+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:28.004968+0000","scrub_duration":0.002365561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":543,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405820+0000","last_change":"2024-09-10T18:44:51.049789+0000","last_active":"2024-09-10T18:45:25.405820+0000","last_peered":"2024-09-10T18:45:25.405820+0000","last_clean":"2024-09-10T18:45:25.405820+0000","last_became_active":"2024-09-10T18:39:00.895121+0000","last_became_peered":"2024-09-10T18:39:00.895121+0000","last_unstale":"2024-09-10T18:45:25.405820+0000","last_undegraded":"2024-09-10T18:45:25.405820+0000","last_fullsized":"2024-09-10T18:45:25.405820+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:51.049732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:44:51.049732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:54.189937+0000","scrub_duration":0.001321523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":588,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.276063+0000","last_change":"2024-09-10T18:44:52.960451+0000","last_active":"2024-09-10T18:45:25.276063+0000","last_peered":"2024-09-10T18:45:25.276063+0000","last_clean":"2024-09-10T18:45:25.276063+0000","last_became_active":"2024-09-10T18:44:06.171739+0000","last_became_peered":"2024-09-10T18:44:06.171739+0000","last_unstale":"2024-09-10T18:45:25.276063+0000","last_undegraded":"2024-09-10T18:45:25.276063+0000","last_fullsized":"2024-09-10T18:45:25.276063+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:52.960390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:34:21.447291+0000","last_clean_scrub_stamp":"2024-09-10T18:44:52.960390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:05.338263+0000","scrub_duration":0.001197523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":510,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405276+0000","last_change":"2024-09-10T18:44:44.066629+0000","last_active":"2024-09-10T18:45:25.405276+0000","last_peered":"2024-09-10T18:45:25.405276+0000","last_clean":"2024-09-10T18:45:25.405276+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:45:25.405276+0000","last_undegraded":"2024-09-10T18:45:25.405276+0000","last_fullsized":"2024-09-10T18:45:25.405276+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:44.066562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:04.058646+0000","last_clean_scrub_stamp":"2024-09-10T18:44:44.066562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:02.662427+0000","scrub_duration":0.0029830899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"76'81","reported_seq":464,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.036240+0000","last_change":"2024-09-10T18:44:38.497287+0000","last_active":"2024-09-10T18:45:25.036240+0000","last_peered":"2024-09-10T18:45:25.036240+0000","last_clean":"2024-09-10T18:45:25.036240+0000","last_became_active":"2024-09-10T18:44:06.161273+0000","last_became_peered":"2024-09-10T18:44:06.161273+0000","last_unstale":"2024-09-10T18:45:25.036240+0000","last_undegraded":"2024-09-10T18:45:25.036240+0000","last_fullsized":"2024-09-10T18:45:25.036240+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:38.497228+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:38.497228+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:07.171855+0000","scrub_duration":0.001927752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"400'29","reported_seq":559,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.037121+0000","last_change":"2024-09-10T18:44:59.729207+0000","last_active":"2024-09-10T18:45:25.037121+0000","last_peered":"2024-09-10T18:45:25.037121+0000","last_clean":"2024-09-10T18:45:25.037121+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:45:25.037121+0000","last_undegraded":"2024-09-10T18:45:25.037121+0000","last_fullsized":"2024-09-10T18:45:25.037121+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'29","last_scrub_stamp":"2024-09-10T18:44:59.729140+0000","last_deep_scrub":"301'23","last_deep_scrub_stamp":"2024-09-10T18:40:55.034050+0000","last_clean_scrub_stamp":"2024-09-10T18:44:59.729140+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:27.147300+0000","scrub_duration":0.0038454589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":523,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:37.340815+0000","last_change":"2024-09-10T18:45:37.340781+0000","last_active":"2024-09-10T18:45:37.340815+0000","last_peered":"2024-09-10T18:45:37.340815+0000","last_clean":"2024-09-10T18:45:37.340815+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:45:37.340815+0000","last_undegraded":"2024-09-10T18:45:37.340815+0000","last_fullsized":"2024-09-10T18:45:37.340815+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:37.340714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:45:37.340714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:41.344523+0000","scrub_duration":0.001535542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":578,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.761278+0000","last_change":"2024-09-10T18:45:25.761240+0000","last_active":"2024-09-10T18:45:25.761278+0000","last_peered":"2024-09-10T18:45:25.761278+0000","last_clean":"2024-09-10T18:45:25.761278+0000","last_became_active":"2024-09-10T18:44:06.171685+0000","last_became_peered":"2024-09-10T18:44:06.171685+0000","last_unstale":"2024-09-10T18:45:25.761278+0000","last_undegraded":"2024-09-10T18:45:25.761278+0000","last_fullsized":"2024-09-10T18:45:25.761278+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:25.761185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:42.443984+0000","last_clean_scrub_stamp":"2024-09-10T18:45:25.761185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:36.399679+0000","scrub_duration":0.001397183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":489,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.035545+0000","last_change":"2024-09-10T18:44:25.486602+0000","last_active":"2024-09-10T18:45:25.035545+0000","last_peered":"2024-09-10T18:45:25.035545+0000","last_clean":"2024-09-10T18:45:25.035545+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:45:25.035545+0000","last_undegraded":"2024-09-10T18:45:25.035545+0000","last_fullsized":"2024-09-10T18:45:25.035545+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:25.486535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:38:50.727231+0000","last_clean_scrub_stamp":"2024-09-10T18:44:25.486535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:48.376106+0000","scrub_duration":0.0018253130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"76'81","reported_seq":243,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:41.369034+0000","last_change":"2024-09-10T18:45:41.369001+0000","last_active":"2024-09-10T18:45:41.369034+0000","last_peered":"2024-09-10T18:45:41.369034+0000","last_clean":"2024-09-10T18:45:41.369034+0000","last_became_active":"2024-09-10T18:34:32.822196+0000","last_became_peered":"2024-09-10T18:34:32.822196+0000","last_unstale":"2024-09-10T18:45:41.369034+0000","last_undegraded":"2024-09-10T18:45:41.369034+0000","last_fullsized":"2024-09-10T18:45:41.369034+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:41.368947+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:45:41.368947+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:03.686724+0000","scrub_duration":0.001398479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"434'46","reported_seq":188,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:30.091086+0000","last_change":"2024-09-10T18:45:30.091033+0000","last_active":"2024-09-10T18:45:30.091086+0000","last_peered":"2024-09-10T18:45:30.091086+0000","last_clean":"2024-09-10T18:45:30.091086+0000","last_became_active":"2024-09-10T18:39:00.902783+0000","last_became_peered":"2024-09-10T18:39:00.902783+0000","last_unstale":"2024-09-10T18:45:30.091086+0000","last_undegraded":"2024-09-10T18:45:30.091086+0000","last_fullsized":"2024-09-10T18:45:30.091086+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'46","last_scrub_stamp":"2024-09-10T18:45:30.090965+0000","last_deep_scrub":"434'46","last_deep_scrub_stamp":"2024-09-10T18:44:19.904898+0000","last_clean_scrub_stamp":"2024-09-10T18:45:30.090965+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:49.288365+0000","scrub_duration":0.0021495899999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":545,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405939+0000","last_change":"2024-09-10T18:44:20.936428+0000","last_active":"2024-09-10T18:45:25.405939+0000","last_peered":"2024-09-10T18:45:25.405939+0000","last_clean":"2024-09-10T18:45:25.405939+0000","last_became_active":"2024-09-10T18:44:06.161378+0000","last_became_peered":"2024-09-10T18:44:06.161378+0000","last_unstale":"2024-09-10T18:45:25.405939+0000","last_undegraded":"2024-09-10T18:45:25.405939+0000","last_fullsized":"2024-09-10T18:45:25.405939+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:44:20.936373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:44:20.936373+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:40.266629+0000","scrub_duration":0.001433365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":809,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.044369+0000","last_change":"2024-09-10T18:44:28.022333+0000","last_active":"2024-09-10T18:45:45.044369+0000","last_peered":"2024-09-10T18:45:45.044369+0000","last_clean":"2024-09-10T18:45:45.044369+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:45:45.044369+0000","last_undegraded":"2024-09-10T18:45:45.044369+0000","last_fullsized":"2024-09-10T18:45:45.044369+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:44:28.022263+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:44:28.022263+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016602500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":157,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178233+0000","last_change":"2024-09-10T18:45:15.139557+0000","last_active":"2024-09-10T18:45:25.178233+0000","last_peered":"2024-09-10T18:45:25.178233+0000","last_clean":"2024-09-10T18:45:25.178233+0000","last_became_active":"2024-09-10T18:44:06.172086+0000","last_became_peered":"2024-09-10T18:44:06.172086+0000","last_unstale":"2024-09-10T18:45:25.178233+0000","last_undegraded":"2024-09-10T18:45:25.178233+0000","last_fullsized":"2024-09-10T18:45:25.178233+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:15.139489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:45:15.139489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:22.413742+0000","scrub_duration":0.001470343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'81","reported_seq":137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178406+0000","last_change":"2024-09-10T18:44:48.054541+0000","last_active":"2024-09-10T18:45:25.178406+0000","last_peered":"2024-09-10T18:45:25.178406+0000","last_clean":"2024-09-10T18:45:25.178406+0000","last_became_active":"2024-09-10T18:39:00.910728+0000","last_became_peered":"2024-09-10T18:39:00.910728+0000","last_unstale":"2024-09-10T18:45:25.178406+0000","last_undegraded":"2024-09-10T18:45:25.178406+0000","last_fullsized":"2024-09-10T18:45:25.178406+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:48.054466+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:48.054466+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:00.133566+0000","scrub_duration":0.0015792390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":519,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.381734+0000","last_change":"2024-09-10T18:45:03.801355+0000","last_active":"2024-09-10T18:45:25.381734+0000","last_peered":"2024-09-10T18:45:25.381734+0000","last_clean":"2024-09-10T18:45:25.381734+0000","last_became_active":"2024-09-10T18:44:06.170932+0000","last_became_peered":"2024-09-10T18:44:06.170932+0000","last_unstale":"2024-09-10T18:45:25.381734+0000","last_undegraded":"2024-09-10T18:45:25.381734+0000","last_fullsized":"2024-09-10T18:45:25.381734+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:03.801290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:45:03.801290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:27.883971+0000","scrub_duration":0.001687937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"400'18","reported_seq":520,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.036651+0000","last_change":"2024-09-10T18:44:46.467864+0000","last_active":"2024-09-10T18:45:25.036651+0000","last_peered":"2024-09-10T18:45:25.036651+0000","last_clean":"2024-09-10T18:45:25.036651+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:45:25.036651+0000","last_undegraded":"2024-09-10T18:45:25.036651+0000","last_fullsized":"2024-09-10T18:45:25.036651+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'18","last_scrub_stamp":"2024-09-10T18:44:46.467804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:44:46.467804+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:11.798982+0000","scrub_duration":0.002535575,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178857+0000","last_change":"2024-09-10T18:45:00.067239+0000","last_active":"2024-09-10T18:45:25.178857+0000","last_peered":"2024-09-10T18:45:25.178857+0000","last_clean":"2024-09-10T18:45:25.178857+0000","last_became_active":"2024-09-10T18:39:00.916952+0000","last_became_peered":"2024-09-10T18:39:00.916952+0000","last_unstale":"2024-09-10T18:45:25.178857+0000","last_undegraded":"2024-09-10T18:45:25.178857+0000","last_fullsized":"2024-09-10T18:45:25.178857+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:00.067157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:51.751267+0000","last_clean_scrub_stamp":"2024-09-10T18:45:00.067157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:02.011190+0000","scrub_duration":0.0016814779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":504,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405437+0000","last_change":"2024-09-10T18:45:09.307456+0000","last_active":"2024-09-10T18:45:25.405437+0000","last_peered":"2024-09-10T18:45:25.405437+0000","last_clean":"2024-09-10T18:45:25.405437+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:45:25.405437+0000","last_undegraded":"2024-09-10T18:45:25.405437+0000","last_fullsized":"2024-09-10T18:45:25.405437+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:09.307383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:45:09.307383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:29.778842+0000","scrub_duration":0.0014754760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'81","reported_seq":450,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.108628+0000","last_change":"2024-09-10T18:45:25.108578+0000","last_active":"2024-09-10T18:45:25.108628+0000","last_peered":"2024-09-10T18:45:25.108628+0000","last_clean":"2024-09-10T18:45:25.108628+0000","last_became_active":"2024-09-10T18:34:02.815880+0000","last_became_peered":"2024-09-10T18:34:02.815880+0000","last_unstale":"2024-09-10T18:45:25.108628+0000","last_undegraded":"2024-09-10T18:45:25.108628+0000","last_fullsized":"2024-09-10T18:45:25.108628+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:25.108523+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:45:25.108523+0000","last_clean_scrub_stamp":"2024-09-10T18:45:25.108523+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:45.294250+0000","scrub_duration":0.0022515410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"400'25","reported_seq":65,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:43.386224+0000","last_change":"2024-09-10T18:45:43.386199+0000","last_active":"2024-09-10T18:45:43.386224+0000","last_peered":"2024-09-10T18:45:43.386224+0000","last_clean":"2024-09-10T18:45:43.386224+0000","last_became_active":"2024-09-10T18:44:06.163688+0000","last_became_peered":"2024-09-10T18:44:06.163688+0000","last_unstale":"2024-09-10T18:45:43.386224+0000","last_undegraded":"2024-09-10T18:45:43.386224+0000","last_fullsized":"2024-09-10T18:45:43.386224+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'25","last_scrub_stamp":"2024-09-10T18:45:43.386142+0000","last_deep_scrub":"400'25","last_deep_scrub_stamp":"2024-09-10T18:45:43.386142+0000","last_clean_scrub_stamp":"2024-09-10T18:45:43.386142+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:00.538488+0000","scrub_duration":0.0017439249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":520,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.365843+0000","last_change":"2024-09-10T18:45:36.365755+0000","last_active":"2024-09-10T18:45:36.365843+0000","last_peered":"2024-09-10T18:45:36.365843+0000","last_clean":"2024-09-10T18:45:36.365843+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:45:36.365843+0000","last_undegraded":"2024-09-10T18:45:36.365843+0000","last_fullsized":"2024-09-10T18:45:36.365843+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:36.365691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:45:36.365691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:43.155806+0000","scrub_duration":0.001656331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":491,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.036573+0000","last_change":"2024-09-10T18:44:05.597703+0000","last_active":"2024-09-10T18:45:25.036573+0000","last_peered":"2024-09-10T18:45:25.036573+0000","last_clean":"2024-09-10T18:45:25.036573+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:45:25.036573+0000","last_undegraded":"2024-09-10T18:45:25.036573+0000","last_fullsized":"2024-09-10T18:45:25.036573+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:05.597644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:44:05.597644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017041560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":546,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.276086+0000","last_change":"2024-09-10T18:45:04.887089+0000","last_active":"2024-09-10T18:45:25.276086+0000","last_peered":"2024-09-10T18:45:25.276086+0000","last_clean":"2024-09-10T18:45:25.276086+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:45:25.276086+0000","last_undegraded":"2024-09-10T18:45:25.276086+0000","last_fullsized":"2024-09-10T18:45:25.276086+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:04.887033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:43:46.250339+0000","last_clean_scrub_stamp":"2024-09-10T18:45:04.887033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:23.332796+0000","scrub_duration":0.0029547129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'81","reported_seq":137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178009+0000","last_change":"2024-09-10T18:44:41.105004+0000","last_active":"2024-09-10T18:45:25.178009+0000","last_peered":"2024-09-10T18:45:25.178009+0000","last_clean":"2024-09-10T18:45:25.178009+0000","last_became_active":"2024-09-10T18:39:00.902585+0000","last_became_peered":"2024-09-10T18:39:00.902585+0000","last_unstale":"2024-09-10T18:45:25.178009+0000","last_undegraded":"2024-09-10T18:45:25.178009+0000","last_fullsized":"2024-09-10T18:45:25.178009+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:41.104948+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:39:55.999377+0000","last_clean_scrub_stamp":"2024-09-10T18:44:41.104948+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:52.886626+0000","scrub_duration":0.00129419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"400'25","reported_seq":156,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178735+0000","last_change":"2024-09-10T18:45:05.099475+0000","last_active":"2024-09-10T18:45:25.178735+0000","last_peered":"2024-09-10T18:45:25.178735+0000","last_clean":"2024-09-10T18:45:25.178735+0000","last_became_active":"2024-09-10T18:39:00.899968+0000","last_became_peered":"2024-09-10T18:39:00.899968+0000","last_unstale":"2024-09-10T18:45:25.178735+0000","last_undegraded":"2024-09-10T18:45:25.178735+0000","last_fullsized":"2024-09-10T18:45:25.178735+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'25","last_scrub_stamp":"2024-09-10T18:45:05.099413+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:45:05.099413+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:07.431929+0000","scrub_duration":0.0020260790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.657776+0000","last_change":"2024-09-10T18:44:52.329429+0000","last_active":"2024-09-10T18:45:25.657776+0000","last_peered":"2024-09-10T18:45:25.657776+0000","last_clean":"2024-09-10T18:45:25.657776+0000","last_became_active":"2024-09-10T18:44:06.159130+0000","last_became_peered":"2024-09-10T18:44:06.159130+0000","last_unstale":"2024-09-10T18:45:25.657776+0000","last_undegraded":"2024-09-10T18:45:25.657776+0000","last_fullsized":"2024-09-10T18:45:25.657776+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:52.329371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_clean_scrub_stamp":"2024-09-10T18:44:52.329371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:20.234171+0000","scrub_duration":0.001513637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"342'9","reported_seq":802,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.044552+0000","last_change":"2024-09-10T18:45:24.494003+0000","last_active":"2024-09-10T18:45:45.044552+0000","last_peered":"2024-09-10T18:45:45.044552+0000","last_clean":"2024-09-10T18:45:45.044552+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:45:45.044552+0000","last_undegraded":"2024-09-10T18:45:45.044552+0000","last_fullsized":"2024-09-10T18:45:45.044552+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'9","last_scrub_stamp":"2024-09-10T18:45:24.493865+0000","last_deep_scrub":"342'9","last_deep_scrub_stamp":"2024-09-10T18:37:56.641708+0000","last_clean_scrub_stamp":"2024-09-10T18:45:24.493865+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:28.176007+0000","scrub_duration":0.0022316329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":53,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405832+0000","last_change":"2024-09-10T18:44:58.132531+0000","last_active":"2024-09-10T18:45:25.405832+0000","last_peered":"2024-09-10T18:45:25.405832+0000","last_clean":"2024-09-10T18:45:25.405832+0000","last_became_active":"2024-09-10T18:43:57.678270+0000","last_became_peered":"2024-09-10T18:43:57.678270+0000","last_unstale":"2024-09-10T18:45:25.405832+0000","last_undegraded":"2024-09-10T18:45:25.405832+0000","last_fullsized":"2024-09-10T18:45:25.405832+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:58.132467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:44:58.132467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:26.266039+0000","scrub_duration":0.0017044510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"76'81","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.657803+0000","last_change":"2024-09-10T18:44:42.473565+0000","last_active":"2024-09-10T18:45:25.657803+0000","last_peered":"2024-09-10T18:45:25.657803+0000","last_clean":"2024-09-10T18:45:25.657803+0000","last_became_active":"2024-09-10T18:44:06.160532+0000","last_became_peered":"2024-09-10T18:44:06.160532+0000","last_unstale":"2024-09-10T18:45:25.657803+0000","last_undegraded":"2024-09-10T18:45:25.657803+0000","last_fullsized":"2024-09-10T18:45:25.657803+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:42.473478+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:36:08.710566+0000","last_clean_scrub_stamp":"2024-09-10T18:44:42.473478+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:56.284220+0000","scrub_duration":0.0026086019999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"400'30","reported_seq":548,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.036337+0000","last_change":"2024-09-10T18:44:53.381662+0000","last_active":"2024-09-10T18:45:25.036337+0000","last_peered":"2024-09-10T18:45:25.036337+0000","last_clean":"2024-09-10T18:45:25.036337+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:45:25.036337+0000","last_undegraded":"2024-09-10T18:45:25.036337+0000","last_fullsized":"2024-09-10T18:45:25.036337+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'30","last_scrub_stamp":"2024-09-10T18:44:53.381596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:44:53.381596+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:21.940755+0000","scrub_duration":0.0040286710000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":513,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405121+0000","last_change":"2024-09-10T18:45:13.414109+0000","last_active":"2024-09-10T18:45:25.405121+0000","last_peered":"2024-09-10T18:45:25.405121+0000","last_clean":"2024-09-10T18:45:25.405121+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:45:25.405121+0000","last_undegraded":"2024-09-10T18:45:25.405121+0000","last_fullsized":"2024-09-10T18:45:25.405121+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:13.414043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:17.826448+0000","last_clean_scrub_stamp":"2024-09-10T18:45:13.414043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:18.715631+0000","scrub_duration":0.0017876610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"430'12","reported_seq":253,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.044362+0000","last_change":"2024-09-10T18:45:01.049536+0000","last_active":"2024-09-10T18:45:45.044362+0000","last_peered":"2024-09-10T18:45:45.044362+0000","last_clean":"2024-09-10T18:45:45.044362+0000","last_became_active":"2024-09-10T18:44:06.170582+0000","last_became_peered":"2024-09-10T18:44:06.170582+0000","last_unstale":"2024-09-10T18:45:45.044362+0000","last_undegraded":"2024-09-10T18:45:45.044362+0000","last_fullsized":"2024-09-10T18:45:45.044362+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'12","last_scrub_stamp":"2024-09-10T18:45:01.049470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:45:01.049470+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:14.721784+0000","scrub_duration":0.002164067,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.658375+0000","last_change":"2024-09-10T18:44:53.287994+0000","last_active":"2024-09-10T18:45:25.658375+0000","last_peered":"2024-09-10T18:45:25.658375+0000","last_clean":"2024-09-10T18:45:25.658375+0000","last_became_active":"2024-09-10T18:44:06.160884+0000","last_became_peered":"2024-09-10T18:44:06.160884+0000","last_unstale":"2024-09-10T18:45:25.658375+0000","last_undegraded":"2024-09-10T18:45:25.658375+0000","last_fullsized":"2024-09-10T18:45:25.658375+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:53.287932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:44:53.287932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:59.964334+0000","scrub_duration":0.0037275199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'81","reported_seq":245,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.108149+0000","last_change":"2024-09-10T18:44:39.065232+0000","last_active":"2024-09-10T18:45:25.108149+0000","last_peered":"2024-09-10T18:45:25.108149+0000","last_clean":"2024-09-10T18:45:25.108149+0000","last_became_active":"2024-09-10T18:44:06.171808+0000","last_became_peered":"2024-09-10T18:44:06.171808+0000","last_unstale":"2024-09-10T18:45:25.108149+0000","last_undegraded":"2024-09-10T18:45:25.108149+0000","last_fullsized":"2024-09-10T18:45:25.108149+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:39.065169+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:39.065169+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:07.369806+0000","scrub_duration":0.001359952,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"400'27","reported_seq":582,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.037318+0000","last_change":"2024-09-10T18:44:31.637993+0000","last_active":"2024-09-10T18:45:25.037318+0000","last_peered":"2024-09-10T18:45:25.037318+0000","last_clean":"2024-09-10T18:45:25.037318+0000","last_became_active":"2024-09-10T18:41:11.933013+0000","last_became_peered":"2024-09-10T18:41:11.933013+0000","last_unstale":"2024-09-10T18:45:25.037318+0000","last_undegraded":"2024-09-10T18:45:25.037318+0000","last_fullsized":"2024-09-10T18:45:25.037318+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'27","last_scrub_stamp":"2024-09-10T18:44:31.637922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:44:31.637922+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:46.999860+0000","scrub_duration":0.0038029119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":522,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:44.721292+0000","last_change":"2024-09-10T18:45:44.721249+0000","last_active":"2024-09-10T18:45:44.721292+0000","last_peered":"2024-09-10T18:45:44.721292+0000","last_clean":"2024-09-10T18:45:44.721292+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:45:44.721292+0000","last_undegraded":"2024-09-10T18:45:44.721292+0000","last_fullsized":"2024-09-10T18:45:44.721292+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:44.721190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:45:44.721190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:13.673819+0000","scrub_duration":0.001445244,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":543,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.107887+0000","last_change":"2024-09-10T18:45:07.310043+0000","last_active":"2024-09-10T18:45:25.107887+0000","last_peered":"2024-09-10T18:45:25.107887+0000","last_clean":"2024-09-10T18:45:25.107887+0000","last_became_active":"2024-09-10T18:39:00.897018+0000","last_became_peered":"2024-09-10T18:39:00.897018+0000","last_unstale":"2024-09-10T18:45:25.107887+0000","last_undegraded":"2024-09-10T18:45:25.107887+0000","last_fullsized":"2024-09-10T18:45:25.107887+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:07.309981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_clean_scrub_stamp":"2024-09-10T18:45:07.309981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:16.957321+0000","scrub_duration":0.0016989010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":529,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.332929+0000","last_change":"2024-09-10T18:45:45.332886+0000","last_active":"2024-09-10T18:45:45.332929+0000","last_peered":"2024-09-10T18:45:45.332929+0000","last_clean":"2024-09-10T18:45:45.332929+0000","last_became_active":"2024-09-10T18:44:06.162639+0000","last_became_peered":"2024-09-10T18:44:06.162639+0000","last_unstale":"2024-09-10T18:45:45.332929+0000","last_undegraded":"2024-09-10T18:45:45.332929+0000","last_fullsized":"2024-09-10T18:45:45.332929+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:45.332820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:45:45.332820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:57.513815+0000","scrub_duration":0.0012939239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'81","reported_seq":157,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178788+0000","last_change":"2024-09-10T18:44:59.072853+0000","last_active":"2024-09-10T18:45:25.178788+0000","last_peered":"2024-09-10T18:45:25.178788+0000","last_clean":"2024-09-10T18:45:25.178788+0000","last_became_active":"2024-09-10T18:44:06.173792+0000","last_became_peered":"2024-09-10T18:44:06.173792+0000","last_unstale":"2024-09-10T18:45:25.178788+0000","last_undegraded":"2024-09-10T18:45:25.178788+0000","last_fullsized":"2024-09-10T18:45:25.178788+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:59.072782+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:40:52.817945+0000","last_clean_scrub_stamp":"2024-09-10T18:44:59.072782+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:04.923069+0000","scrub_duration":0.001624995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"400'21","reported_seq":563,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.108089+0000","last_change":"2024-09-10T18:45:20.200436+0000","last_active":"2024-09-10T18:45:25.108089+0000","last_peered":"2024-09-10T18:45:25.108089+0000","last_clean":"2024-09-10T18:45:25.108089+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:45:25.108089+0000","last_undegraded":"2024-09-10T18:45:25.108089+0000","last_fullsized":"2024-09-10T18:45:25.108089+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'21","last_scrub_stamp":"2024-09-10T18:45:20.200370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:45:20.200370+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:20.664309+0000","scrub_duration":0.002329121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":516,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.406122+0000","last_change":"2024-09-10T18:45:25.406050+0000","last_active":"2024-09-10T18:45:25.406122+0000","last_peered":"2024-09-10T18:45:25.406122+0000","last_clean":"2024-09-10T18:45:25.406122+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:45:25.406122+0000","last_undegraded":"2024-09-10T18:45:25.406122+0000","last_fullsized":"2024-09-10T18:45:25.406122+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:25.405978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:45:25.405978+0000","last_clean_scrub_stamp":"2024-09-10T18:45:25.405978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:37.463319+0000","scrub_duration":0.002680263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":504,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.035617+0000","last_change":"2024-09-10T18:44:10.599072+0000","last_active":"2024-09-10T18:45:25.035617+0000","last_peered":"2024-09-10T18:45:25.035617+0000","last_clean":"2024-09-10T18:45:25.035617+0000","last_became_active":"2024-09-10T18:44:06.160812+0000","last_became_peered":"2024-09-10T18:44:06.160812+0000","last_unstale":"2024-09-10T18:45:25.035617+0000","last_undegraded":"2024-09-10T18:45:25.035617+0000","last_fullsized":"2024-09-10T18:45:25.035617+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:10.599007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:44:10.599007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001387208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":55,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.474986+0000","last_change":"2024-09-10T18:45:45.474936+0000","last_active":"2024-09-10T18:45:45.474986+0000","last_peered":"2024-09-10T18:45:45.474986+0000","last_clean":"2024-09-10T18:45:45.474986+0000","last_became_active":"2024-09-10T18:44:06.165448+0000","last_became_peered":"2024-09-10T18:44:06.165448+0000","last_unstale":"2024-09-10T18:45:45.474986+0000","last_undegraded":"2024-09-10T18:45:45.474986+0000","last_fullsized":"2024-09-10T18:45:45.474986+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:45.474880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:44:32.354637+0000","last_clean_scrub_stamp":"2024-09-10T18:45:45.474880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:46.680776+0000","scrub_duration":0.0013512000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'81","reported_seq":252,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.276037+0000","last_change":"2024-09-10T18:45:08.825395+0000","last_active":"2024-09-10T18:45:25.276037+0000","last_peered":"2024-09-10T18:45:25.276037+0000","last_clean":"2024-09-10T18:45:25.276037+0000","last_became_active":"2024-09-10T18:34:02.802580+0000","last_became_peered":"2024-09-10T18:34:02.802580+0000","last_unstale":"2024-09-10T18:45:25.276037+0000","last_undegraded":"2024-09-10T18:45:25.276037+0000","last_fullsized":"2024-09-10T18:45:25.276037+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:08.825329+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:45:08.825329+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:26.403111+0000","scrub_duration":0.001539328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"400'27","reported_seq":553,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:44.335805+0000","last_change":"2024-09-10T18:45:44.335762+0000","last_active":"2024-09-10T18:45:44.335805+0000","last_peered":"2024-09-10T18:45:44.335805+0000","last_clean":"2024-09-10T18:45:44.335805+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:45:44.335805+0000","last_undegraded":"2024-09-10T18:45:44.335805+0000","last_fullsized":"2024-09-10T18:45:44.335805+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'27","last_scrub_stamp":"2024-09-10T18:45:44.335686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:45:44.335686+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:53.349846+0000","scrub_duration":0.002076423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":147,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:40.008833+0000","last_change":"2024-09-10T18:45:40.008801+0000","last_active":"2024-09-10T18:45:40.008833+0000","last_peered":"2024-09-10T18:45:40.008833+0000","last_clean":"2024-09-10T18:45:40.008833+0000","last_became_active":"2024-09-10T18:39:00.916983+0000","last_became_peered":"2024-09-10T18:39:00.916983+0000","last_unstale":"2024-09-10T18:45:40.008833+0000","last_undegraded":"2024-09-10T18:45:40.008833+0000","last_fullsized":"2024-09-10T18:45:40.008833+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:40.008738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:45:40.008738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:59.962523+0000","scrub_duration":0.0015682090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":509,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.404466+0000","last_change":"2024-09-10T18:45:24.381281+0000","last_active":"2024-09-10T18:45:25.404466+0000","last_peered":"2024-09-10T18:45:25.404466+0000","last_clean":"2024-09-10T18:45:25.404466+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:45:25.404466+0000","last_undegraded":"2024-09-10T18:45:25.404466+0000","last_fullsized":"2024-09-10T18:45:25.404466+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:24.381213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:45:24.381213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:42.220148+0000","scrub_duration":0.0016045880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":535,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.035814+0000","last_change":"2024-09-10T18:45:04.469996+0000","last_active":"2024-09-10T18:45:25.035814+0000","last_peered":"2024-09-10T18:45:25.035814+0000","last_clean":"2024-09-10T18:45:25.035814+0000","last_became_active":"2024-09-10T18:44:06.161239+0000","last_became_peered":"2024-09-10T18:44:06.161239+0000","last_unstale":"2024-09-10T18:45:25.035814+0000","last_undegraded":"2024-09-10T18:45:25.035814+0000","last_fullsized":"2024-09-10T18:45:25.035814+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:04.469907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:48.554301+0000","last_clean_scrub_stamp":"2024-09-10T18:45:04.469907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:08.225190+0000","scrub_duration":0.0021947099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'81","reported_seq":234,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405339+0000","last_change":"2024-09-10T18:44:37.962063+0000","last_active":"2024-09-10T18:45:25.405339+0000","last_peered":"2024-09-10T18:45:25.405339+0000","last_clean":"2024-09-10T18:45:25.405339+0000","last_became_active":"2024-09-10T18:39:00.889998+0000","last_became_peered":"2024-09-10T18:39:00.889998+0000","last_unstale":"2024-09-10T18:45:25.405339+0000","last_undegraded":"2024-09-10T18:45:25.405339+0000","last_fullsized":"2024-09-10T18:45:25.405339+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:37.961994+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:44:37.961994+0000","last_clean_scrub_stamp":"2024-09-10T18:44:37.961994+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:42.727197+0000","scrub_duration":0.0015827720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"400'12","reported_seq":518,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.035296+0000","last_change":"2024-09-10T18:44:36.504023+0000","last_active":"2024-09-10T18:45:25.035296+0000","last_peered":"2024-09-10T18:45:25.035296+0000","last_clean":"2024-09-10T18:45:25.035296+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:45:25.035296+0000","last_undegraded":"2024-09-10T18:45:25.035296+0000","last_fullsized":"2024-09-10T18:45:25.035296+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'12","last_scrub_stamp":"2024-09-10T18:44:36.503958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:44:36.503958+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:54.761400+0000","scrub_duration":0.0019983959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":55,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:38.332944+0000","last_change":"2024-09-10T18:45:38.332918+0000","last_active":"2024-09-10T18:45:38.332944+0000","last_peered":"2024-09-10T18:45:38.332944+0000","last_clean":"2024-09-10T18:45:38.332944+0000","last_became_active":"2024-09-10T18:44:06.161983+0000","last_became_peered":"2024-09-10T18:44:06.161983+0000","last_unstale":"2024-09-10T18:45:38.332944+0000","last_undegraded":"2024-09-10T18:45:38.332944+0000","last_fullsized":"2024-09-10T18:45:38.332944+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:38.332851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:45:38.332851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:06.659426+0000","scrub_duration":0.0015862020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":483,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:39.790686+0000","last_change":"2024-09-10T18:45:39.790652+0000","last_active":"2024-09-10T18:45:39.790686+0000","last_peered":"2024-09-10T18:45:39.790686+0000","last_clean":"2024-09-10T18:45:39.790686+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:45:39.790686+0000","last_undegraded":"2024-09-10T18:45:39.790686+0000","last_fullsized":"2024-09-10T18:45:39.790686+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:39.790582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:45:39.790582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:54.285917+0000","scrub_duration":0.0015055019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":507,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:42.374048+0000","last_change":"2024-09-10T18:45:42.374018+0000","last_active":"2024-09-10T18:45:42.374048+0000","last_peered":"2024-09-10T18:45:42.374048+0000","last_clean":"2024-09-10T18:45:42.374048+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:45:42.374048+0000","last_undegraded":"2024-09-10T18:45:42.374048+0000","last_fullsized":"2024-09-10T18:45:42.374048+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:42.373965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:45:42.373965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:44.649548+0000","scrub_duration":0.001595735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'81","reported_seq":233,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:43.717565+0000","last_change":"2024-09-10T18:45:43.717524+0000","last_active":"2024-09-10T18:45:43.717565+0000","last_peered":"2024-09-10T18:45:43.717565+0000","last_clean":"2024-09-10T18:45:43.717565+0000","last_became_active":"2024-09-10T18:34:34.867380+0000","last_became_peered":"2024-09-10T18:34:34.867380+0000","last_unstale":"2024-09-10T18:45:43.717565+0000","last_undegraded":"2024-09-10T18:45:43.717565+0000","last_fullsized":"2024-09-10T18:45:43.717565+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:43.717432+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:45:43.717432+0000","last_clean_scrub_stamp":"2024-09-10T18:45:43.717432+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:51.521155+0000","scrub_duration":0.001472758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":538,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.107207+0000","last_change":"2024-09-10T18:45:09.237487+0000","last_active":"2024-09-10T18:45:25.107207+0000","last_peered":"2024-09-10T18:45:25.107207+0000","last_clean":"2024-09-10T18:45:25.107207+0000","last_became_active":"2024-09-10T18:39:00.909906+0000","last_became_peered":"2024-09-10T18:39:00.909906+0000","last_unstale":"2024-09-10T18:45:25.107207+0000","last_undegraded":"2024-09-10T18:45:25.107207+0000","last_fullsized":"2024-09-10T18:45:25.107207+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:09.237426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:45:09.237426+0000","last_clean_scrub_stamp":"2024-09-10T18:45:09.237426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:11.341214+0000","scrub_duration":0.0016153669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":515,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.035451+0000","last_change":"2024-09-10T18:45:13.493539+0000","last_active":"2024-09-10T18:45:25.035451+0000","last_peered":"2024-09-10T18:45:25.035451+0000","last_clean":"2024-09-10T18:45:25.035451+0000","last_became_active":"2024-09-10T18:44:06.161345+0000","last_became_peered":"2024-09-10T18:44:06.161345+0000","last_unstale":"2024-09-10T18:45:25.035451+0000","last_undegraded":"2024-09-10T18:45:25.035451+0000","last_fullsized":"2024-09-10T18:45:25.035451+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:13.493469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:46.565877+0000","last_clean_scrub_stamp":"2024-09-10T18:45:13.493469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:26.435784+0000","scrub_duration":0.0013431490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"400'32","reported_seq":615,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.382437+0000","last_change":"2024-09-10T18:45:24.857886+0000","last_active":"2024-09-10T18:45:25.382437+0000","last_peered":"2024-09-10T18:45:25.382437+0000","last_clean":"2024-09-10T18:45:25.382437+0000","last_became_active":"2024-09-10T18:39:00.921476+0000","last_became_peered":"2024-09-10T18:39:00.921476+0000","last_unstale":"2024-09-10T18:45:25.382437+0000","last_undegraded":"2024-09-10T18:45:25.382437+0000","last_fullsized":"2024-09-10T18:45:25.382437+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'32","last_scrub_stamp":"2024-09-10T18:45:24.857809+0000","last_deep_scrub":"369'24","last_deep_scrub_stamp":"2024-09-10T18:39:50.690827+0000","last_clean_scrub_stamp":"2024-09-10T18:45:24.857809+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:25.278534+0000","scrub_duration":0.0029675330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.658213+0000","last_change":"2024-09-10T18:44:48.447412+0000","last_active":"2024-09-10T18:45:25.658213+0000","last_peered":"2024-09-10T18:45:25.658213+0000","last_clean":"2024-09-10T18:45:25.658213+0000","last_became_active":"2024-09-10T18:44:06.169267+0000","last_became_peered":"2024-09-10T18:44:06.169267+0000","last_unstale":"2024-09-10T18:45:25.658213+0000","last_undegraded":"2024-09-10T18:45:25.658213+0000","last_fullsized":"2024-09-10T18:45:25.658213+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:48.447347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:44:48.447347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:16.752300+0000","scrub_duration":0.0019764130000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1545,"num_read_kb":1302,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":5886,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":1091,"ondisk_log_size":1091,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1058,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1446,"num_read_kb":1441,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":797,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2511,"ondisk_log_size":2511,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234280,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":891440,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":890430,"kb_avail":92853712,"statfs":{"total":95995035648,"available":95082201088,"internally_reserved":0,"allocated":1024000,"data_stored":635883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":911800664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:45:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.63,"15min":1.4970000000000001},"min":{"1min":0.311,"5min":0.254,"15min":0.21099999999999999},"max":{"1min":0.70699999999999996,"5min":3.8519999999999999,"15min":75.506},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.92700000000000005,"15min":1.506},"min":{"1min":0.34699999999999998,"5min":0.26500000000000001,"15min":0.223},"max":{"1min":0.72499999999999998,"5min":12.571999999999999,"15min":75.710999999999999},"last":0.495}]},{"osd":1,"last update":"Tue Sep 10 18:45:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":1.056,"15min":1.5189999999999999},"min":{"1min":0.436,"5min":0.23400000000000001,"15min":0.214},"max":{"1min":0.78900000000000003,"5min":12.461,"15min":98.305999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.137,"15min":1.5129999999999999},"min":{"1min":0.39000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.77300000000000002,"5min":9.4969999999999999,"15min":98.381},"last":0.55500000000000005}]},{"osd":2,"last update":"Tue Sep 10 18:45:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.94399999999999995,"15min":1.792},"min":{"1min":0.39900000000000002,"5min":0.26900000000000002,"15min":0.26800000000000002},"max":{"1min":0.75700000000000001,"5min":5.3840000000000003,"15min":136.648},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.85699999999999998,"15min":1.7889999999999999},"min":{"1min":0.38300000000000001,"5min":0.27800000000000002,"15min":0.24399999999999999},"max":{"1min":0.72799999999999998,"5min":5.7859999999999996,"15min":136.506},"last":0.625}]},{"osd":3,"last update":"Tue Sep 10 18:45:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":1.0720000000000001,"15min":1.278},"min":{"1min":0.316,"5min":0.251,"15min":0.251},"max":{"1min":0.69199999999999995,"5min":8.7400000000000002,"15min":38.225000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":1.036,"15min":1.2310000000000001},"min":{"1min":0.50700000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.77700000000000002,"5min":12.516,"15min":38.104999999999997},"last":0.57299999999999995}]},{"osd":4,"last update":"Tue Sep 10 18:45:06 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.85099999999999998,"15min":0.92800000000000005},"min":{"1min":0.28399999999999997,"5min":0.23899999999999999,"15min":0.224},"max":{"1min":0.66100000000000003,"5min":8.4770000000000003,"15min":23.346},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.80600000000000005,"15min":0.94399999999999995},"min":{"1min":0.31900000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.80400000000000005,"5min":5.4219999999999997,"15min":23.323},"last":0.58799999999999997}]},{"osd":5,"last update":"Tue Sep 10 18:45:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.746,"15min":0.93500000000000005},"min":{"1min":0.33500000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.71099999999999997,"5min":4.0179999999999998,"15min":16.756},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.89100000000000001,"15min":1.008},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.224},"max":{"1min":0.72399999999999998,"5min":8.4510000000000005,"15min":20.495999999999999},"last":0.64200000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.89000000000000001,"15min":0.98899999999999999},"min":{"1min":0.38400000000000001,"5min":0.314,"15min":0.28899999999999998},"max":{"1min":0.72199999999999998,"5min":4.6689999999999996,"15min":25.902999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":1.0600000000000001,"15min":1.004},"min":{"1min":0.38700000000000001,"5min":0.33600000000000002,"15min":0.30199999999999999},"max":{"1min":0.73099999999999998,"5min":9.9879999999999995,"15min":24.074000000000002},"last":0.61399999999999999}]}]},{"osd":6,"up_from":144,"seq":618475290859,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903964,"kb_used_data":1044,"kb_used_omap":7,"kb_used_meta":902904,"kb_avail":92841188,"statfs":{"total":95995035648,"available":95069376512,"internally_reserved":0,"allocated":1069056,"data_stored":701047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":924573783},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.94799999999999995,"15min":1.071},"min":{"1min":0.308,"5min":0.23300000000000001,"15min":0.192},"max":{"1min":0.85199999999999998,"5min":14.228999999999999,"15min":14.228999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.83599999999999997,"15min":1.0129999999999999},"min":{"1min":0.26200000000000001,"5min":0.246,"15min":0.217},"max":{"1min":0.81599999999999995,"5min":4.4960000000000004,"15min":18.093},"last":0.437}]},{"osd":1,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":1.5129999999999999,"15min":1.147},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76200000000000001,"5min":23.782,"15min":23.782},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.56100000000000005,"5min":1.1519999999999999,"15min":1.163},"min":{"1min":0.35899999999999999,"5min":0.26800000000000002,"15min":0.246},"max":{"1min":0.86599999999999999,"5min":23.687999999999999,"15min":23.687999999999999},"last":0.39700000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":1.1220000000000001,"15min":1.276},"min":{"1min":0.39100000000000001,"5min":0.27500000000000002,"15min":0.25600000000000001},"max":{"1min":0.79800000000000004,"5min":23.763999999999999,"15min":39.329999999999998},"last":0.505},{"interface":"front","average":{"1min":0.55500000000000005,"5min":1.5820000000000001,"15min":1.3140000000000001},"min":{"1min":0.36899999999999999,"5min":0.314,"15min":0.26200000000000001},"max":{"1min":0.77600000000000002,"5min":23.856000000000002,"15min":39.061999999999998},"last":0.47299999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":1.236,"15min":2.3570000000000002},"min":{"1min":0.34300000000000003,"5min":0.29499999999999998,"15min":0.28199999999999997},"max":{"1min":0.88100000000000001,"5min":29.670999999999999,"15min":162.61600000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":1.4239999999999999,"15min":2.4380000000000002},"min":{"1min":0.35199999999999998,"5min":0.29599999999999999,"15min":0.27100000000000002},"max":{"1min":0.73899999999999999,"5min":32.018000000000001,"15min":163.39599999999999},"last":0.52100000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.189},"max":{"1min":0.72899999999999998,"5min":3.9279999999999999,"15min":4.9050000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.748,"15min":0.72099999999999997},"min":{"1min":0.27600000000000002,"5min":0.22800000000000001,"15min":0.219},"max":{"1min":0.68200000000000005,"5min":3.093,"15min":11.531000000000001},"last":0.34799999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":1.9450000000000001,"15min":1.171},"min":{"1min":0.32100000000000001,"5min":0.26100000000000001,"15min":0.254},"max":{"1min":0.73499999999999999,"5min":129.69900000000001,"15min":129.69900000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":2.0459999999999998,"15min":1.224},"min":{"1min":0.33800000000000002,"5min":0.28000000000000003,"15min":0.20899999999999999},"max":{"1min":0.78500000000000003,"5min":130.04400000000001,"15min":130.04400000000001},"last":0.377}]},{"osd":7,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.0860000000000001,"15min":0.91100000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.23799999999999999},"max":{"1min":0.78200000000000003,"5min":10.103999999999999,"15min":11.449},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.98399999999999999,"15min":0.81899999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.27100000000000002},"max":{"1min":0.76200000000000001,"5min":9.8049999999999997,"15min":9.8049999999999997},"last":0.59699999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724851,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":887984,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":886974,"kb_avail":92857168,"statfs":{"total":95995035648,"available":95085740032,"internally_reserved":0,"allocated":1024000,"data_stored":636361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908261720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.93700000000000006,"15min":1.135},"min":{"1min":0.37,"5min":0.23400000000000001,"15min":0.224},"max":{"1min":0.82799999999999996,"5min":19.123999999999999,"15min":26.288},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.71199999999999997,"15min":0.95399999999999996},"min":{"1min":0.35999999999999999,"5min":0.27600000000000002,"15min":0.24099999999999999},"max":{"1min":0.73199999999999998,"5min":4.851,"15min":24.178000000000001},"last":0.39500000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":1.4139999999999999,"15min":1.7390000000000001},"min":{"1min":0.26700000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.84199999999999997,"5min":22.623999999999999,"15min":111.773},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.143,"15min":1.653},"min":{"1min":0.46000000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.76500000000000001,"5min":22.635000000000002,"15min":111.73999999999999},"last":0.34899999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":1.2210000000000001,"15min":1.306},"min":{"1min":0.38,"5min":0.247,"15min":0.245},"max":{"1min":0.76200000000000001,"5min":23.396000000000001,"15min":54.530000000000001},"last":0.623},{"interface":"front","average":{"1min":0.57499999999999996,"5min":1.2070000000000001,"15min":1.2010000000000001},"min":{"1min":0.311,"5min":0.27800000000000002,"15min":0.247},"max":{"1min":0.81100000000000005,"5min":22.905999999999999,"15min":54.514000000000003},"last":0.52000000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.143,"15min":1.7829999999999999},"min":{"1min":0.45300000000000001,"5min":0.29699999999999999,"15min":0.27100000000000002},"max":{"1min":0.74199999999999999,"5min":23.007999999999999,"15min":106.205},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":1.244,"15min":1.823},"min":{"1min":0.34499999999999997,"5min":0.28999999999999998,"15min":0.28299999999999997},"max":{"1min":0.76500000000000001,"5min":22.931000000000001,"15min":105.878},"last":0.373}]},{"osd":4,"last update":"Tue Sep 10 18:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.70899999999999996,"15min":0.75},"min":{"1min":0.29599999999999999,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":0.625,"5min":5.226,"15min":5.8040000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.88800000000000001,"15min":0.84599999999999997},"min":{"1min":0.32900000000000001,"5min":0.27900000000000003,"15min":0.23999999999999999},"max":{"1min":0.70599999999999996,"5min":5.702,"15min":8.7270000000000003},"last":0.64800000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.873,"15min":0.76700000000000002},"min":{"1min":0.318,"5min":0.317,"15min":0.27500000000000002},"max":{"1min":0.78900000000000003,"5min":5.7359999999999998,"15min":7.9969999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.46999999999999997,"5min":1.0289999999999999,"15min":0.91700000000000004},"min":{"1min":0.32900000000000001,"5min":0.28499999999999998,"15min":0.27000000000000002},"max":{"1min":0.753,"5min":8.7400000000000002,"15min":8.7400000000000002},"last":0.63400000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.94999999999999996,"15min":0.95799999999999996},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.25900000000000001},"max":{"1min":0.77600000000000002,"5min":6.867,"15min":28.934000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.502,"5min":0.88700000000000001,"15min":0.86699999999999999},"min":{"1min":0.38200000000000001,"5min":0.27300000000000002,"15min":0.25900000000000001},"max":{"1min":0.71499999999999997,"5min":5.593,"15min":5.8259999999999996},"last":0.53800000000000003}]}]},{"osd":4,"up_from":33,"seq":141733921079,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900076,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":899326,"kb_avail":92845076,"statfs":{"total":95995035648,"available":95073357824,"internally_reserved":0,"allocated":757760,"data_stored":373006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920910168},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.67000000000000004,"15min":1.083},"min":{"1min":0.30499999999999999,"5min":0.247,"15min":0.20899999999999999},"max":{"1min":0.68999999999999995,"5min":4.7050000000000001,"15min":21.468},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.57199999999999995,"15min":0.94099999999999995},"min":{"1min":0.34899999999999998,"5min":0.27000000000000002,"15min":0.189},"max":{"1min":0.73899999999999999,"5min":3.0409999999999999,"15min":24.277999999999999},"last":0.374}]},{"osd":1,"last update":"Tue Sep 10 18:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.89500000000000002,"15min":1.2130000000000001},"min":{"1min":0.41099999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.66700000000000004,"5min":8.2539999999999996,"15min":50.045000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.82499999999999996,"15min":1.25},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.27100000000000002},"max":{"1min":0.68000000000000005,"5min":4.0380000000000003,"15min":50.283000000000001},"last":0.40000000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":3.0830000000000002,"15min":2.008},"min":{"1min":0.32400000000000001,"5min":0.28100000000000003,"15min":0.249},"max":{"1min":0.71799999999999997,"5min":169.417,"15min":169.417},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.52400000000000002,"5min":2.996,"15min":2.004},"min":{"1min":0.39400000000000002,"5min":0.251,"15min":0.24099999999999999},"max":{"1min":0.78700000000000003,"5min":169.40100000000001,"15min":169.40100000000001},"last":0.501}]},{"osd":3,"last update":"Tue Sep 10 18:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":2.069,"15min":1.5700000000000001},"min":{"1min":0.34499999999999997,"5min":0.27200000000000002,"15min":0.26200000000000001},"max":{"1min":0.70399999999999996,"5min":116.913,"15min":116.913},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":2.1920000000000002,"15min":1.7849999999999999},"min":{"1min":0.41199999999999998,"5min":0.309,"15min":0.28999999999999998},"max":{"1min":0.68899999999999995,"5min":116.863,"15min":116.863},"last":0.59799999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.69099999999999995,"15min":0.83399999999999996},"min":{"1min":0.35599999999999998,"5min":0.252,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":3.6379999999999999,"15min":22.861000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.57499999999999996,"15min":0.85199999999999998},"min":{"1min":0.30299999999999999,"5min":0.254,"15min":0.22},"max":{"1min":0.72099999999999997,"5min":2.976,"15min":19.611999999999998},"last":0.55100000000000005}]},{"osd":6,"last update":"Tue Sep 10 18:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.90100000000000002,"15min":1.0309999999999999},"min":{"1min":0.29799999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.61799999999999999,"5min":6.7130000000000001,"15min":51.722000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.878,"15min":0.99399999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.59999999999999998,"5min":8.5749999999999993,"15min":51.862000000000002},"last":0.62}]},{"osd":7,"last update":"Tue Sep 10 18:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.871,"15min":0.83199999999999996},"min":{"1min":0.35399999999999998,"5min":0.312,"15min":0.28799999999999998},"max":{"1min":0.747,"5min":6.4470000000000001,"15min":7.6260000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.78900000000000003,"15min":0.81599999999999995},"min":{"1min":0.35599999999999998,"5min":0.31,"15min":0.28000000000000003},"max":{"1min":0.67900000000000005,"5min":6.96,"15min":7.8250000000000002},"last":0.60799999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117309,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":892664,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":891582,"kb_avail":92852488,"statfs":{"total":95995035648,"available":95080947712,"internally_reserved":0,"allocated":1097728,"data_stored":701530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912980312},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.81200000000000006,"15min":1.089},"min":{"1min":0.23899999999999999,"5min":0.19900000000000001,"15min":0.18099999999999999},"max":{"1min":0.47899999999999998,"5min":8.8079999999999998,"15min":29.981999999999999},"last":0.37},{"interface":"front","average":{"1min":0.374,"5min":0.68600000000000005,"15min":1.2190000000000001},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.193},"max":{"1min":0.56599999999999995,"5min":5.0010000000000003,"15min":28.754000000000001},"last":0.48299999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":1.0820000000000001,"15min":1.4159999999999999},"min":{"1min":0.253,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":0.505,"5min":9.5199999999999996,"15min":35.406999999999996},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":1.0760000000000001,"15min":1.391},"min":{"1min":0.24199999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.51500000000000001,"5min":8.9499999999999993,"15min":36.902000000000001},"last":0.443}]},{"osd":2,"last update":"Tue Sep 10 18:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":1.0349999999999999,"15min":1.3120000000000001},"min":{"1min":0.247,"5min":0.22800000000000001,"15min":0.191},"max":{"1min":0.58499999999999996,"5min":8.7210000000000001,"15min":79.975999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.373,"5min":1.085,"15min":1.5580000000000001},"min":{"1min":0.28000000000000003,"5min":0.23400000000000001,"15min":0.22},"max":{"1min":0.502,"5min":11.242000000000001,"15min":78.278000000000006},"last":0.498}]},{"osd":4,"last update":"Tue Sep 10 18:45:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":1.046,"15min":1.111},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.27600000000000002},"max":{"1min":0.76600000000000001,"5min":4.8780000000000001,"15min":6.5250000000000004},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.97599999999999998,"15min":1.085},"min":{"1min":0.35599999999999998,"5min":0.28399999999999997,"15min":0.27300000000000002},"max":{"1min":0.82899999999999996,"5min":5.1989999999999998,"15min":6.1929999999999996},"last":0.76200000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:45:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.93500000000000005,"15min":1.0940000000000001},"min":{"1min":0.41799999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.80200000000000005,"5min":4.8479999999999999,"15min":6.601},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.94699999999999995,"15min":1.127},"min":{"1min":0.503,"5min":0.254,"15min":0.254},"max":{"1min":0.78400000000000003,"5min":5.0410000000000004,"15min":6.2249999999999996},"last":0.82099999999999995}]},{"osd":6,"last update":"Tue Sep 10 18:45:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":1.0229999999999999,"15min":1.248},"min":{"1min":0.44400000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.75700000000000001,"5min":6.8129999999999997,"15min":31.815999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":1.091,"15min":1.335},"min":{"1min":0.46000000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.86699999999999999,"5min":8.7569999999999997,"15min":31.742000000000001},"last":0.70799999999999996}]},{"osd":7,"last update":"Tue Sep 10 18:45:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":1.113,"15min":1.1539999999999999},"min":{"1min":0.38300000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.77100000000000002,"5min":4.9000000000000004,"15min":6.5730000000000004},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.628,"5min":1.022,"15min":1.198},"min":{"1min":0.47599999999999998,"5min":0.34100000000000003,"15min":0.31900000000000001},"max":{"1min":0.84599999999999997,"5min":5.1180000000000003,"15min":12.612},"last":0.71899999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411649,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901420,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":900414,"kb_avail":92843732,"statfs":{"total":95995035648,"available":95071981568,"internally_reserved":0,"allocated":1019904,"data_stored":635496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922024280},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.628,"15min":1.1200000000000001},"min":{"1min":0.223,"5min":0.216,"15min":0.17499999999999999},"max":{"1min":0.54400000000000004,"5min":3.5169999999999999,"15min":26.585999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.70499999999999996,"15min":1.3380000000000001},"min":{"1min":0.25600000000000001,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":0.54600000000000004,"5min":5.8879999999999999,"15min":26.48},"last":0.443}]},{"osd":1,"last update":"Tue Sep 10 18:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":1.071,"15min":1.274},"min":{"1min":0.25700000000000001,"5min":0.22500000000000001,"15min":0.19400000000000001},"max":{"1min":0.52600000000000002,"5min":11.154,"15min":15.750999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.375,"5min":1.0409999999999999,"15min":1.1479999999999999},"min":{"1min":0.27200000000000002,"5min":0.248,"15min":0.19500000000000001},"max":{"1min":0.54200000000000004,"5min":10.637,"15min":14.988},"last":0.41699999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:45:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.96899999999999997,"15min":1.5720000000000001},"min":{"1min":0.29899999999999999,"5min":0.28599999999999998,"15min":0.18099999999999999},"max":{"1min":0.57599999999999996,"5min":7.9359999999999999,"15min":61.331000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":1.1399999999999999,"15min":1.7070000000000001},"min":{"1min":0.32000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.79400000000000004,"5min":6.734,"15min":61.250999999999998},"last":0.372}]},{"osd":4,"last update":"Tue Sep 10 18:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.92200000000000004,"15min":1.212},"min":{"1min":0.378,"5min":0.249,"15min":0.249},"max":{"1min":0.82899999999999996,"5min":5.3129999999999997,"15min":12.555999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.90900000000000003,"15min":1.129},"min":{"1min":0.39500000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81799999999999995,"5min":3.6909999999999998,"15min":19.696000000000002},"last":0.58699999999999997}]},{"osd":5,"last update":"Tue Sep 10 18:45:20 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.89500000000000002,"15min":1.2150000000000001},"min":{"1min":0.47499999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.91700000000000004,"5min":3.6400000000000001,"15min":25.177},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.97099999999999997,"15min":1.2729999999999999},"min":{"1min":0.40300000000000002,"5min":0.32000000000000001,"15min":0.317},"max":{"1min":0.86599999999999999,"5min":3.665,"15min":24.986999999999998},"last":0.61599999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.99399999999999999,"15min":1.4570000000000001},"min":{"1min":0.40799999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78500000000000003,"5min":5.2770000000000001,"15min":53.359999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.97999999999999998,"15min":1.351},"min":{"1min":0.441,"5min":0.34799999999999998,"15min":0.32900000000000001},"max":{"1min":0.84699999999999998,"5min":3.6960000000000002,"15min":53.438000000000002},"last":0.54700000000000004}]},{"osd":7,"last update":"Tue Sep 10 18:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.97699999999999998,"15min":1.1459999999999999},"min":{"1min":0.40799999999999997,"5min":0.36699999999999999,"15min":0.35399999999999998},"max":{"1min":0.80000000000000004,"5min":6.1980000000000004,"15min":10.68},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.624,"5min":1.121,"15min":1.3109999999999999},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.95599999999999996,"5min":5.2320000000000002,"15min":12.596},"last":0.69699999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575174,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901808,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":901054,"kb_avail":92843344,"statfs":{"total":95995035648,"available":95071584256,"internally_reserved":0,"allocated":761856,"data_stored":373306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922679640},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:45:49 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.625,"15min":1.093},"min":{"1min":0.25,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":2.2210000000000001,"5min":6.4619999999999997,"15min":28.893999999999998},"last":0.314},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.66200000000000003,"15min":1.1240000000000001},"min":{"1min":0.247,"5min":0.20100000000000001,"15min":0.19800000000000001},"max":{"1min":0.5,"5min":7.8700000000000001,"15min":31.074999999999999},"last":0.40100000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.94499999999999995,"15min":1.1100000000000001},"min":{"1min":0.23200000000000001,"5min":0.216,"15min":0.19500000000000001},"max":{"1min":0.53500000000000003,"5min":11.079000000000001,"15min":11.079000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.93899999999999995,"15min":1.151},"min":{"1min":0.248,"5min":0.21199999999999999,"15min":0.20999999999999999},"max":{"1min":0.496,"5min":6.274,"15min":12.222},"last":0.46000000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":2.593,"15min":1.6299999999999999},"min":{"1min":0.251,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":2.1869999999999998,"5min":148.22900000000001,"15min":148.22900000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.496,"5min":2.5059999999999998,"15min":1.6259999999999999},"min":{"1min":0.27300000000000002,"5min":0.24399999999999999,"15min":0.20499999999999999},"max":{"1min":2.1190000000000002,"5min":148.65000000000001,"15min":148.65000000000001},"last":0.29499999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:45:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.81100000000000005,"15min":1.528},"min":{"1min":0.39900000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.86699999999999999,"5min":3.496,"15min":127.86199999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":1.0569999999999999,"15min":1.649},"min":{"1min":0.39900000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.92100000000000004,"5min":6.7690000000000001,"15min":127.84399999999999},"last":0.72399999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":1.0269999999999999,"15min":1.679},"min":{"1min":0.40600000000000003,"5min":0.32300000000000001,"15min":0.27900000000000003},"max":{"1min":0.88900000000000001,"5min":6.7939999999999996,"15min":118.68000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":1.0589999999999999,"15min":1.7150000000000001},"min":{"1min":0.46899999999999997,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.014,"5min":6.8929999999999998,"15min":118.36},"last":0.76600000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":4.7130000000000001,"15min":3.2919999999999998},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":0.86099999999999999,"5min":389.52499999999998,"15min":389.52499999999998},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":4.4589999999999996,"15min":3.4769999999999999},"min":{"1min":0.378,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93100000000000005,"5min":389.69499999999999,"15min":389.69499999999999},"last":0.69099999999999995}]},{"osd":7,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":1.3560000000000001,"15min":1.268},"min":{"1min":0.40699999999999997,"5min":0.40600000000000003,"15min":0.32700000000000001},"max":{"1min":0.79000000000000004,"5min":16.244,"15min":16.244},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.0700000000000001,"15min":1.222},"min":{"1min":0.45500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.89500000000000002,"5min":16.550000000000001,"15min":16.550000000000001},"last":0.73799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738698,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":897000,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":895934,"kb_avail":92848152,"statfs":{"total":95995035648,"available":95076507648,"internally_reserved":0,"allocated":1081344,"data_stored":700559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917436760},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:45:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.65600000000000003,"15min":1.577},"min":{"1min":0.23699999999999999,"5min":0.20499999999999999,"15min":0.185},"max":{"1min":0.66100000000000003,"5min":5.7610000000000001,"15min":146.44499999999999},"last":0.316},{"interface":"front","average":{"1min":0.372,"5min":0.60199999999999998,"15min":1.6160000000000001},"min":{"1min":0.223,"5min":0.221,"15min":0.20100000000000001},"max":{"1min":0.57899999999999996,"5min":2.8660000000000001,"15min":148.77199999999999},"last":0.218}]},{"osd":2,"last update":"Tue Sep 10 18:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":3.589,"15min":2.613},"min":{"1min":0.29699999999999999,"5min":0.222,"15min":0.19700000000000001},"max":{"1min":0.52400000000000002,"5min":102.571,"15min":146.56999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.37,"5min":3.4550000000000001,"15min":2.3740000000000001},"min":{"1min":0.248,"5min":0.21299999999999999,"15min":0.21099999999999999},"max":{"1min":0.68300000000000005,"5min":101.916,"15min":146.601},"last":0.38300000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:45:38 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.3919999999999999,"15min":1.764},"min":{"1min":0.27300000000000002,"5min":0.26400000000000001,"15min":0.23400000000000001},"max":{"1min":0.68600000000000005,"5min":43.179000000000002,"15min":140.68299999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.41699999999999998,"5min":1.4390000000000001,"15min":1.905},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.80300000000000005,"5min":43.226999999999997,"15min":140.572},"last":0.27200000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.76700000000000002,"15min":1.0840000000000001},"min":{"1min":0.41299999999999998,"5min":0.311,"15min":0.27500000000000002},"max":{"1min":0.77500000000000002,"5min":5.5700000000000003,"15min":8.3699999999999992},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.72599999999999998,"15min":1.0569999999999999},"min":{"1min":0.40300000000000002,"5min":0.33400000000000002,"15min":0.29899999999999999},"max":{"1min":0.75800000000000001,"5min":3.6459999999999999,"15min":8.7710000000000008},"last":0.72699999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.75600000000000001,"15min":1.107},"min":{"1min":0.42099999999999999,"5min":0.32500000000000001,"15min":0.28799999999999998},"max":{"1min":0.97399999999999998,"5min":5.3920000000000003,"15min":8.1259999999999994},"last":0.755},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64500000000000002,"15min":1.1040000000000001},"min":{"1min":0.47899999999999998,"5min":0.33500000000000002,"15min":0.29799999999999999},"max":{"1min":0.93000000000000005,"5min":3.6400000000000001,"15min":20.460000000000001},"last":0.624}]},{"osd":6,"last update":"Tue Sep 10 18:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.83799999999999997,"15min":2.5},"min":{"1min":0.436,"5min":0.379,"15min":0.28899999999999998},"max":{"1min":0.75,"5min":5.5439999999999996,"15min":400.83699999999999},"last":0.63},{"interface":"front","average":{"1min":0.627,"5min":0.73499999999999999,"15min":2.399},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.33300000000000002},"max":{"1min":0.89400000000000002,"5min":2.3700000000000001,"15min":400.77999999999997},"last":0.77800000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.80300000000000005,"15min":1.2010000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.32200000000000001},"max":{"1min":0.86399999999999999,"5min":3.9350000000000001,"15min":10.75},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.751,"15min":1.0640000000000001},"min":{"1min":0.52500000000000002,"5min":0.38,"15min":0.378},"max":{"1min":0.86599999999999999,"5min":2.4350000000000001,"15min":11.218999999999999},"last":0.60299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:45:53.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:45:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:54.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[29291]: 1.f scrub starts 2024-09-10T18:45:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[29291]: 1.f scrub ok 2024-09-10T18:45:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[29291]: 1.3 deep-scrub starts 2024-09-10T18:45:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[29291]: 1.3 deep-scrub ok 2024-09-10T18:45:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[29291]: from='client.16458 v1:172.21.15.28:0/3820855264' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:45:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[29291]: 1.15 scrub starts 2024-09-10T18:45:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[29291]: 1.15 scrub ok 2024-09-10T18:45:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[29291]: pgmap v1095: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:45:54.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:45:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[39220]: 1.f scrub starts 2024-09-10T18:45:54.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[39220]: 1.f scrub ok 2024-09-10T18:45:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[39220]: 1.3 deep-scrub starts 2024-09-10T18:45:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[39220]: 1.3 deep-scrub ok 2024-09-10T18:45:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[39220]: from='client.16458 v1:172.21.15.28:0/3820855264' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:45:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[39220]: 1.15 scrub starts 2024-09-10T18:45:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[39220]: 1.15 scrub ok 2024-09-10T18:45:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[39220]: pgmap v1095: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:45:54.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:54 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:45:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:54 smithi204 ceph-mon[32357]: 1.f scrub starts 2024-09-10T18:45:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:54 smithi204 ceph-mon[32357]: 1.f scrub ok 2024-09-10T18:45:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:54 smithi204 ceph-mon[32357]: 1.3 deep-scrub starts 2024-09-10T18:45:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:54 smithi204 ceph-mon[32357]: 1.3 deep-scrub ok 2024-09-10T18:45:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:54 smithi204 ceph-mon[32357]: from='client.16458 v1:172.21.15.28:0/3820855264' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:45:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:54 smithi204 ceph-mon[32357]: 1.15 scrub starts 2024-09-10T18:45:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:54 smithi204 ceph-mon[32357]: 1.15 scrub ok 2024-09-10T18:45:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:54 smithi204 ceph-mon[32357]: pgmap v1095: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:45:54.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:54 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:45:55.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:55 smithi028 ceph-mon[29291]: 5.f scrub starts 2024-09-10T18:45:55.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:55 smithi028 ceph-mon[29291]: 5.f scrub ok 2024-09-10T18:45:55.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:55 smithi028 ceph-mon[29291]: 4.b scrub starts 2024-09-10T18:45:55.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:55 smithi028 ceph-mon[29291]: 4.b scrub ok 2024-09-10T18:45:55.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:55 smithi028 ceph-mon[39220]: 5.f scrub starts 2024-09-10T18:45:55.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:55 smithi028 ceph-mon[39220]: 5.f scrub ok 2024-09-10T18:45:55.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:55 smithi028 ceph-mon[39220]: 4.b scrub starts 2024-09-10T18:45:55.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:55 smithi028 ceph-mon[39220]: 4.b scrub ok 2024-09-10T18:45:55.304 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:55 smithi204 ceph-mon[32357]: 5.f scrub starts 2024-09-10T18:45:55.304 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:55 smithi204 ceph-mon[32357]: 5.f scrub ok 2024-09-10T18:45:55.304 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:55 smithi204 ceph-mon[32357]: 4.b scrub starts 2024-09-10T18:45:55.304 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:55 smithi204 ceph-mon[32357]: 4.b scrub ok 2024-09-10T18:45:55.603 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:45:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug there is no tcmu-runner data available 2024-09-10T18:45:56.192 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-10T18:45:56.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[29291]: 4.18 scrub starts 2024-09-10T18:45:56.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[29291]: purged_snaps scrub starts 2024-09-10T18:45:56.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[29291]: purged_snaps scrub ok 2024-09-10T18:45:56.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[29291]: 4.18 scrub ok 2024-09-10T18:45:56.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[29291]: osdmap e456: 8 total, 8 up, 8 in 2024-09-10T18:45:56.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[29291]: 1.10 scrub starts 2024-09-10T18:45:56.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[29291]: 1.10 scrub ok 2024-09-10T18:45:56.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[29291]: pgmap v1097: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:45:56.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[29291]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:56.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[39220]: 4.18 scrub starts 2024-09-10T18:45:56.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[39220]: purged_snaps scrub starts 2024-09-10T18:45:56.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[39220]: purged_snaps scrub ok 2024-09-10T18:45:56.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[39220]: 4.18 scrub ok 2024-09-10T18:45:56.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[39220]: osdmap e456: 8 total, 8 up, 8 in 2024-09-10T18:45:56.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[39220]: 1.10 scrub starts 2024-09-10T18:45:56.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[39220]: 1.10 scrub ok 2024-09-10T18:45:56.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[39220]: pgmap v1097: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:45:56.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:56 smithi028 ceph-mon[39220]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-10T18:45:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:56 smithi204 ceph-mon[32357]: 4.18 scrub starts 2024-09-10T18:45:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:56 smithi204 ceph-mon[32357]: purged_snaps scrub starts 2024-09-10T18:45:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:56 smithi204 ceph-mon[32357]: purged_snaps scrub ok 2024-09-10T18:45:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:56 smithi204 ceph-mon[32357]: 4.18 scrub ok 2024-09-10T18:45:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:56 smithi204 ceph-mon[32357]: osdmap e456: 8 total, 8 up, 8 in 2024-09-10T18:45:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:56 smithi204 ceph-mon[32357]: 1.10 scrub starts 2024-09-10T18:45:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:56 smithi204 ceph-mon[32357]: 1.10 scrub ok 2024-09-10T18:45:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:56 smithi204 ceph-mon[32357]: pgmap v1097: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:45:56.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:56 smithi204 ceph-mon[32357]: from='client.24457 v1:172.21.15.204:0/2200010475' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:45:57.147 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":1097,"stamp":"2024-09-10T18:45:55.274543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":3014,"num_read_kb":2766,"num_write":1002,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":663865,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3712,"ondisk_log_size":3712,"up":486,"acting":486,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":486,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7176356,"kb_used_data":7652,"kb_used_omap":19,"kb_used_meta":7168620,"kb_avail":742784860,"statfs":{"total":767960285184,"available":760611696640,"internally_reserved":0,"allocated":7835648,"data_stored":4757188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20033,"internal_metadata":7340667327},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003145"},"pg_stats":[{"pgid":"1.1d","version":"76'81","reported_seq":229,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.108195+0000","last_change":"2024-09-10T18:45:12.273543+0000","last_active":"2024-09-10T18:45:25.108195+0000","last_peered":"2024-09-10T18:45:25.108195+0000","last_clean":"2024-09-10T18:45:25.108195+0000","last_became_active":"2024-09-10T18:34:34.865894+0000","last_became_peered":"2024-09-10T18:34:34.865894+0000","last_unstale":"2024-09-10T18:45:25.108195+0000","last_undegraded":"2024-09-10T18:45:25.108195+0000","last_fullsized":"2024-09-10T18:45:25.108195+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:12.273477+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:45:12.273477+0000","last_clean_scrub_stamp":"2024-09-10T18:45:12.273477+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:22.469989+0000","scrub_duration":0.001515665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.658317+0000","last_change":"2024-09-10T18:45:10.445143+0000","last_active":"2024-09-10T18:45:25.658317+0000","last_peered":"2024-09-10T18:45:25.658317+0000","last_clean":"2024-09-10T18:45:25.658317+0000","last_became_active":"2024-09-10T18:44:06.170062+0000","last_became_peered":"2024-09-10T18:44:06.170062+0000","last_unstale":"2024-09-10T18:45:25.658317+0000","last_undegraded":"2024-09-10T18:45:25.658317+0000","last_fullsized":"2024-09-10T18:45:25.658317+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:10.445040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.642495+0000","last_clean_scrub_stamp":"2024-09-10T18:45:10.445040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:14.416296+0000","scrub_duration":0.0033800380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"400'25","reported_seq":566,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:54.633809+0000","last_change":"2024-09-10T18:45:54.633749+0000","last_active":"2024-09-10T18:45:54.633809+0000","last_peered":"2024-09-10T18:45:54.633809+0000","last_clean":"2024-09-10T18:45:54.633809+0000","last_became_active":"2024-09-10T18:26:27.648457+0000","last_became_peered":"2024-09-10T18:26:27.648457+0000","last_unstale":"2024-09-10T18:45:54.633809+0000","last_undegraded":"2024-09-10T18:45:54.633809+0000","last_fullsized":"2024-09-10T18:45:54.633809+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'25","last_scrub_stamp":"2024-09-10T18:45:54.633691+0000","last_deep_scrub":"342'19","last_deep_scrub_stamp":"2024-09-10T18:37:49.354781+0000","last_clean_scrub_stamp":"2024-09-10T18:45:54.633691+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:58.655714+0000","scrub_duration":0.001916168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":540,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:26.120412+0000","last_change":"2024-09-10T18:45:26.120376+0000","last_active":"2024-09-10T18:45:26.120412+0000","last_peered":"2024-09-10T18:45:26.120412+0000","last_clean":"2024-09-10T18:45:26.120412+0000","last_became_active":"2024-09-10T18:39:00.910095+0000","last_became_peered":"2024-09-10T18:39:00.910095+0000","last_unstale":"2024-09-10T18:45:26.120412+0000","last_undegraded":"2024-09-10T18:45:26.120412+0000","last_fullsized":"2024-09-10T18:45:26.120412+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:26.120321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:45:26.120321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:45.597849+0000","scrub_duration":0.0019125699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":530,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.367948+0000","last_change":"2024-09-10T18:45:19.719242+0000","last_active":"2024-09-10T18:45:36.367948+0000","last_peered":"2024-09-10T18:45:36.367948+0000","last_clean":"2024-09-10T18:45:36.367948+0000","last_became_active":"2024-09-10T18:39:00.889898+0000","last_became_peered":"2024-09-10T18:39:00.889898+0000","last_unstale":"2024-09-10T18:45:36.367948+0000","last_undegraded":"2024-09-10T18:45:36.367948+0000","last_fullsized":"2024-09-10T18:45:36.367948+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:19.719163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:18.749558+0000","last_clean_scrub_stamp":"2024-09-10T18:45:19.719163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:41.285618+0000","scrub_duration":0.0015216979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'81","reported_seq":217,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405440+0000","last_change":"2024-09-10T18:44:41.052661+0000","last_active":"2024-09-10T18:45:25.405440+0000","last_peered":"2024-09-10T18:45:25.405440+0000","last_clean":"2024-09-10T18:45:25.405440+0000","last_became_active":"2024-09-10T18:34:34.867388+0000","last_became_peered":"2024-09-10T18:34:34.867388+0000","last_unstale":"2024-09-10T18:45:25.405440+0000","last_undegraded":"2024-09-10T18:45:25.405440+0000","last_fullsized":"2024-09-10T18:45:25.405440+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:41.052593+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:44:41.052593+0000","last_clean_scrub_stamp":"2024-09-10T18:44:41.052593+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:41.420911+0000","scrub_duration":0.0013664650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":507,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:52.371225+0000","last_change":"2024-09-10T18:45:52.371179+0000","last_active":"2024-09-10T18:45:52.371225+0000","last_peered":"2024-09-10T18:45:52.371225+0000","last_clean":"2024-09-10T18:45:52.371225+0000","last_became_active":"2024-09-10T18:26:27.636811+0000","last_became_peered":"2024-09-10T18:26:27.636811+0000","last_unstale":"2024-09-10T18:45:52.371225+0000","last_undegraded":"2024-09-10T18:45:52.371225+0000","last_fullsized":"2024-09-10T18:45:52.371225+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:52.371115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:45:52.371115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:12.387043+0000","scrub_duration":0.0016417560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":513,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660880+0000","last_change":"2024-09-10T18:45:33.394562+0000","last_active":"2024-09-10T18:45:36.660880+0000","last_peered":"2024-09-10T18:45:36.660880+0000","last_clean":"2024-09-10T18:45:36.660880+0000","last_became_active":"2024-09-10T18:26:27.636855+0000","last_became_peered":"2024-09-10T18:26:27.636855+0000","last_unstale":"2024-09-10T18:45:36.660880+0000","last_undegraded":"2024-09-10T18:45:36.660880+0000","last_fullsized":"2024-09-10T18:45:36.660880+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:33.394489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:45:33.394489+0000","last_clean_scrub_stamp":"2024-09-10T18:45:33.394489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:55.815361+0000","scrub_duration":0.0021405780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"400'46","reported_seq":617,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.659834+0000","last_change":"2024-09-10T18:44:51.373960+0000","last_active":"2024-09-10T18:45:36.659834+0000","last_peered":"2024-09-10T18:45:36.659834+0000","last_clean":"2024-09-10T18:45:36.659834+0000","last_became_active":"2024-09-10T18:44:06.166307+0000","last_became_peered":"2024-09-10T18:44:06.166307+0000","last_unstale":"2024-09-10T18:45:36.659834+0000","last_undegraded":"2024-09-10T18:45:36.659834+0000","last_fullsized":"2024-09-10T18:45:36.659834+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'46","last_scrub_stamp":"2024-09-10T18:44:51.373894+0000","last_deep_scrub":"123'22","last_deep_scrub_stamp":"2024-09-10T18:25:13.118335+0000","last_clean_scrub_stamp":"2024-09-10T18:44:51.373894+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:08.264078+0000","scrub_duration":0.0020169509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":519,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.367406+0000","last_change":"2024-09-10T18:45:25.540900+0000","last_active":"2024-09-10T18:45:36.367406+0000","last_peered":"2024-09-10T18:45:36.367406+0000","last_clean":"2024-09-10T18:45:36.367406+0000","last_became_active":"2024-09-10T18:26:27.648759+0000","last_became_peered":"2024-09-10T18:26:27.648759+0000","last_unstale":"2024-09-10T18:45:36.367406+0000","last_undegraded":"2024-09-10T18:45:36.367406+0000","last_fullsized":"2024-09-10T18:45:36.367406+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:25.540825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:44:05.367986+0000","last_clean_scrub_stamp":"2024-09-10T18:45:25.540825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:40.345042+0000","scrub_duration":0.001622484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":116,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:35.403360+0000","last_change":"2024-09-10T18:45:35.403291+0000","last_active":"2024-09-10T18:45:35.403360+0000","last_peered":"2024-09-10T18:45:35.403360+0000","last_clean":"2024-09-10T18:45:35.403360+0000","last_became_active":"2024-09-10T18:40:10.986591+0000","last_became_peered":"2024-09-10T18:40:10.986591+0000","last_unstale":"2024-09-10T18:45:35.403360+0000","last_undegraded":"2024-09-10T18:45:35.403360+0000","last_fullsized":"2024-09-10T18:45:35.403360+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:35.403223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:33:16.963102+0000","last_clean_scrub_stamp":"2024-09-10T18:45:35.403223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:48.759914+0000","scrub_duration":0.001477007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"400'21","reported_seq":595,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:47.743467+0000","last_change":"2024-09-10T18:45:47.743397+0000","last_active":"2024-09-10T18:45:47.743467+0000","last_peered":"2024-09-10T18:45:47.743467+0000","last_clean":"2024-09-10T18:45:47.743467+0000","last_became_active":"2024-09-10T18:44:06.172812+0000","last_became_peered":"2024-09-10T18:44:06.172812+0000","last_unstale":"2024-09-10T18:45:47.743467+0000","last_undegraded":"2024-09-10T18:45:47.743467+0000","last_fullsized":"2024-09-10T18:45:47.743467+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'21","last_scrub_stamp":"2024-09-10T18:45:47.743315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:45:47.743315+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:50.375271+0000","scrub_duration":0.001988802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":157,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:29.129220+0000","last_change":"2024-09-10T18:45:29.129164+0000","last_active":"2024-09-10T18:45:29.129220+0000","last_peered":"2024-09-10T18:45:29.129220+0000","last_clean":"2024-09-10T18:45:29.129220+0000","last_became_active":"2024-09-10T18:44:06.170431+0000","last_became_peered":"2024-09-10T18:44:06.170431+0000","last_unstale":"2024-09-10T18:45:29.129220+0000","last_undegraded":"2024-09-10T18:45:29.129220+0000","last_fullsized":"2024-09-10T18:45:29.129220+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:29.129096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:58.589631+0000","last_clean_scrub_stamp":"2024-09-10T18:45:29.129096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:44.573196+0000","scrub_duration":0.001575012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.657895+0000","last_change":"2024-09-10T18:44:46.446472+0000","last_active":"2024-09-10T18:45:25.657895+0000","last_peered":"2024-09-10T18:45:25.657895+0000","last_clean":"2024-09-10T18:45:25.657895+0000","last_became_active":"2024-09-10T18:44:06.167736+0000","last_became_peered":"2024-09-10T18:44:06.167736+0000","last_unstale":"2024-09-10T18:45:25.657895+0000","last_undegraded":"2024-09-10T18:45:25.657895+0000","last_fullsized":"2024-09-10T18:45:25.657895+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:46.446420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:44:46.446420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:04.856827+0000","scrub_duration":0.0021950250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'81","reported_seq":283,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660763+0000","last_change":"2024-09-10T18:44:48.453566+0000","last_active":"2024-09-10T18:45:36.660763+0000","last_peered":"2024-09-10T18:45:36.660763+0000","last_clean":"2024-09-10T18:45:36.660763+0000","last_became_active":"2024-09-10T18:34:34.863102+0000","last_became_peered":"2024-09-10T18:34:34.863102+0000","last_unstale":"2024-09-10T18:45:36.660763+0000","last_undegraded":"2024-09-10T18:45:36.660763+0000","last_fullsized":"2024-09-10T18:45:36.660763+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:48.453498+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:44:48.453498+0000","last_clean_scrub_stamp":"2024-09-10T18:44:48.453498+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:07.561005+0000","scrub_duration":0.002023249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":147,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:32.128950+0000","last_change":"2024-09-10T18:45:32.128903+0000","last_active":"2024-09-10T18:45:32.128950+0000","last_peered":"2024-09-10T18:45:32.128950+0000","last_clean":"2024-09-10T18:45:32.128950+0000","last_became_active":"2024-09-10T18:39:00.915294+0000","last_became_peered":"2024-09-10T18:39:00.915294+0000","last_unstale":"2024-09-10T18:45:32.128950+0000","last_undegraded":"2024-09-10T18:45:32.128950+0000","last_fullsized":"2024-09-10T18:45:32.128950+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:32.128822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:45:32.128822+0000","last_clean_scrub_stamp":"2024-09-10T18:45:32.128822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:54.224216+0000","scrub_duration":0.0016067970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"400'11","reported_seq":561,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.367899+0000","last_change":"2024-09-10T18:45:22.596902+0000","last_active":"2024-09-10T18:45:36.367899+0000","last_peered":"2024-09-10T18:45:36.367899+0000","last_clean":"2024-09-10T18:45:36.367899+0000","last_became_active":"2024-09-10T18:30:11.029110+0000","last_became_peered":"2024-09-10T18:30:11.029110+0000","last_unstale":"2024-09-10T18:45:36.367899+0000","last_undegraded":"2024-09-10T18:45:36.367899+0000","last_fullsized":"2024-09-10T18:45:36.367899+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'11","last_scrub_stamp":"2024-09-10T18:45:22.596826+0000","last_deep_scrub":"85'7","last_deep_scrub_stamp":"2024-09-10T18:22:44.365196+0000","last_clean_scrub_stamp":"2024-09-10T18:45:22.596826+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:25.786042+0000","scrub_duration":0.002692945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":509,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.404789+0000","last_change":"2024-09-10T18:45:11.329820+0000","last_active":"2024-09-10T18:45:25.404789+0000","last_peered":"2024-09-10T18:45:25.404789+0000","last_clean":"2024-09-10T18:45:25.404789+0000","last_became_active":"2024-09-10T18:21:35.624078+0000","last_became_peered":"2024-09-10T18:21:35.624078+0000","last_unstale":"2024-09-10T18:45:25.404789+0000","last_undegraded":"2024-09-10T18:45:25.404789+0000","last_fullsized":"2024-09-10T18:45:25.404789+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:11.329716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:45:11.329716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:25.688528+0000","scrub_duration":0.00160012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:28.134865+0000","last_change":"2024-09-10T18:45:28.134804+0000","last_active":"2024-09-10T18:45:28.134865+0000","last_peered":"2024-09-10T18:45:28.134865+0000","last_clean":"2024-09-10T18:45:28.134865+0000","last_became_active":"2024-09-10T18:39:00.917007+0000","last_became_peered":"2024-09-10T18:39:00.917007+0000","last_unstale":"2024-09-10T18:45:28.134865+0000","last_undegraded":"2024-09-10T18:45:28.134865+0000","last_fullsized":"2024-09-10T18:45:28.134865+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:28.134742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:07.786118+0000","last_clean_scrub_stamp":"2024-09-10T18:45:28.134742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:29.684542+0000","scrub_duration":0.0017479450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'81","reported_seq":386,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660494+0000","last_change":"2024-09-10T18:45:19.531979+0000","last_active":"2024-09-10T18:45:36.660494+0000","last_peered":"2024-09-10T18:45:36.660494+0000","last_clean":"2024-09-10T18:45:36.660494+0000","last_became_active":"2024-09-10T18:44:06.166987+0000","last_became_peered":"2024-09-10T18:44:06.166987+0000","last_unstale":"2024-09-10T18:45:36.660494+0000","last_undegraded":"2024-09-10T18:45:36.660494+0000","last_fullsized":"2024-09-10T18:45:36.660494+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:19.531921+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:45:19.531921+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:23.320065+0000","scrub_duration":0.0017219480000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":537,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.368320+0000","last_change":"2024-09-10T18:44:56.718221+0000","last_active":"2024-09-10T18:45:36.368320+0000","last_peered":"2024-09-10T18:45:36.368320+0000","last_clean":"2024-09-10T18:45:36.368320+0000","last_became_active":"2024-09-10T18:39:00.895860+0000","last_became_peered":"2024-09-10T18:39:00.895860+0000","last_unstale":"2024-09-10T18:45:36.368320+0000","last_undegraded":"2024-09-10T18:45:36.368320+0000","last_fullsized":"2024-09-10T18:45:36.368320+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:56.718155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:44:56.718155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:10.050934+0000","scrub_duration":0.004132035,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.658263+0000","last_change":"2024-09-10T18:44:34.380158+0000","last_active":"2024-09-10T18:45:25.658263+0000","last_peered":"2024-09-10T18:45:25.658263+0000","last_clean":"2024-09-10T18:45:25.658263+0000","last_became_active":"2024-09-10T18:44:06.163606+0000","last_became_peered":"2024-09-10T18:44:06.163606+0000","last_unstale":"2024-09-10T18:45:25.658263+0000","last_undegraded":"2024-09-10T18:45:25.658263+0000","last_fullsized":"2024-09-10T18:45:25.658263+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:34.380067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:44:34.380067+0000","last_clean_scrub_stamp":"2024-09-10T18:44:34.380067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:47.855233+0000","scrub_duration":0.0019231789999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"454'46","reported_seq":628,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:33.697862+0000","last_change":"2024-09-10T18:45:33.697748+0000","last_active":"2024-09-10T18:45:33.697862+0000","last_peered":"2024-09-10T18:45:33.697862+0000","last_clean":"2024-09-10T18:45:33.697862+0000","last_became_active":"2024-09-10T18:21:33.617495+0000","last_became_peered":"2024-09-10T18:21:33.617495+0000","last_unstale":"2024-09-10T18:45:33.697862+0000","last_undegraded":"2024-09-10T18:45:33.697862+0000","last_fullsized":"2024-09-10T18:45:33.697862+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'46","last_scrub_stamp":"2024-09-10T18:45:33.697655+0000","last_deep_scrub":"454'46","last_deep_scrub_stamp":"2024-09-10T18:45:33.697655+0000","last_clean_scrub_stamp":"2024-09-10T18:45:33.697655+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:58.862583+0000","scrub_duration":0.0063720590000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":550,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:47.066380+0000","last_change":"2024-09-10T18:45:47.066352+0000","last_active":"2024-09-10T18:45:47.066380+0000","last_peered":"2024-09-10T18:45:47.066380+0000","last_clean":"2024-09-10T18:45:47.066380+0000","last_became_active":"2024-09-10T18:44:06.162348+0000","last_became_peered":"2024-09-10T18:44:06.162348+0000","last_unstale":"2024-09-10T18:45:47.066380+0000","last_undegraded":"2024-09-10T18:45:47.066380+0000","last_fullsized":"2024-09-10T18:45:47.066380+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:47.066292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:44:35.066309+0000","last_clean_scrub_stamp":"2024-09-10T18:45:47.066292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:59.223230+0000","scrub_duration":0.0016254850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'81","reported_seq":141,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.177847+0000","last_change":"2024-09-10T18:44:47.099266+0000","last_active":"2024-09-10T18:45:25.177847+0000","last_peered":"2024-09-10T18:45:25.177847+0000","last_clean":"2024-09-10T18:45:25.177847+0000","last_became_active":"2024-09-10T18:39:00.907303+0000","last_became_peered":"2024-09-10T18:39:00.907303+0000","last_unstale":"2024-09-10T18:45:25.177847+0000","last_undegraded":"2024-09-10T18:45:25.177847+0000","last_fullsized":"2024-09-10T18:45:25.177847+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:47.099153+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:47.099153+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:58.313101+0000","scrub_duration":0.0024115510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":517,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660549+0000","last_change":"2024-09-10T18:44:03.627161+0000","last_active":"2024-09-10T18:45:36.660549+0000","last_peered":"2024-09-10T18:45:36.660549+0000","last_clean":"2024-09-10T18:45:36.660549+0000","last_became_active":"2024-09-10T18:39:00.904769+0000","last_became_peered":"2024-09-10T18:39:00.904769+0000","last_unstale":"2024-09-10T18:45:36.660549+0000","last_undegraded":"2024-09-10T18:45:36.660549+0000","last_fullsized":"2024-09-10T18:45:36.660549+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:03.627094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:44:03.627094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001532537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":513,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.367545+0000","last_change":"2024-09-10T18:44:33.681320+0000","last_active":"2024-09-10T18:45:36.367545+0000","last_peered":"2024-09-10T18:45:36.367545+0000","last_clean":"2024-09-10T18:45:36.367545+0000","last_became_active":"2024-09-10T18:21:31.527999+0000","last_became_peered":"2024-09-10T18:21:31.527999+0000","last_unstale":"2024-09-10T18:45:36.367545+0000","last_undegraded":"2024-09-10T18:45:36.367545+0000","last_fullsized":"2024-09-10T18:45:36.367545+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:33.681184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:44:33.681184+0000","last_clean_scrub_stamp":"2024-09-10T18:44:33.681184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027611559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"400'29","reported_seq":562,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660262+0000","last_change":"2024-09-10T18:44:14.615569+0000","last_active":"2024-09-10T18:45:36.660262+0000","last_peered":"2024-09-10T18:45:36.660262+0000","last_clean":"2024-09-10T18:45:36.660262+0000","last_became_active":"2024-09-10T18:39:11.896502+0000","last_became_peered":"2024-09-10T18:39:11.896502+0000","last_unstale":"2024-09-10T18:45:36.660262+0000","last_undegraded":"2024-09-10T18:45:36.660262+0000","last_fullsized":"2024-09-10T18:45:36.660262+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'29","last_scrub_stamp":"2024-09-10T18:44:14.615508+0000","last_deep_scrub":"88'17","last_deep_scrub_stamp":"2024-09-10T18:23:02.323736+0000","last_clean_scrub_stamp":"2024-09-10T18:44:14.615508+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028561379999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":525,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.768324+0000","last_change":"2024-09-10T18:45:45.768259+0000","last_active":"2024-09-10T18:45:45.768324+0000","last_peered":"2024-09-10T18:45:45.768324+0000","last_clean":"2024-09-10T18:45:45.768324+0000","last_became_active":"2024-09-10T18:21:35.623659+0000","last_became_peered":"2024-09-10T18:21:35.623659+0000","last_unstale":"2024-09-10T18:45:45.768324+0000","last_undegraded":"2024-09-10T18:45:45.768324+0000","last_fullsized":"2024-09-10T18:45:45.768324+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:45.768182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:45:45.768182+0000","last_clean_scrub_stamp":"2024-09-10T18:45:45.768182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:03.418329+0000","scrub_duration":0.0014202220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'81","reported_seq":157,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178340+0000","last_change":"2024-09-10T18:44:58.038568+0000","last_active":"2024-09-10T18:45:25.178340+0000","last_peered":"2024-09-10T18:45:25.178340+0000","last_clean":"2024-09-10T18:45:25.178340+0000","last_became_active":"2024-09-10T18:44:06.171035+0000","last_became_peered":"2024-09-10T18:44:06.171035+0000","last_unstale":"2024-09-10T18:45:25.178340+0000","last_undegraded":"2024-09-10T18:45:25.178340+0000","last_fullsized":"2024-09-10T18:45:25.178340+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:58.038484+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:58.038484+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:07.496990+0000","scrub_duration":0.0017201479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":541,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405285+0000","last_change":"2024-09-10T18:45:21.342131+0000","last_active":"2024-09-10T18:45:25.405285+0000","last_peered":"2024-09-10T18:45:25.405285+0000","last_clean":"2024-09-10T18:45:25.405285+0000","last_became_active":"2024-09-10T18:39:00.891321+0000","last_became_peered":"2024-09-10T18:39:00.891321+0000","last_unstale":"2024-09-10T18:45:25.405285+0000","last_undegraded":"2024-09-10T18:45:25.405285+0000","last_fullsized":"2024-09-10T18:45:25.405285+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-10T18:45:21.342062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:45:21.342062+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:32.535184+0000","scrub_duration":0.0017807109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":534,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:39.100615+0000","last_change":"2024-09-10T18:45:39.100584+0000","last_active":"2024-09-10T18:45:39.100615+0000","last_peered":"2024-09-10T18:45:39.100615+0000","last_clean":"2024-09-10T18:45:39.100615+0000","last_became_active":"2024-09-10T18:21:31.534315+0000","last_became_peered":"2024-09-10T18:21:31.534315+0000","last_unstale":"2024-09-10T18:45:39.100615+0000","last_undegraded":"2024-09-10T18:45:39.100615+0000","last_fullsized":"2024-09-10T18:45:39.100615+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:39.100524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:45:39.100524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:41.815546+0000","scrub_duration":0.001315004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"400'35","reported_seq":67,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:35.267010+0000","last_change":"2024-09-10T18:45:35.266959+0000","last_active":"2024-09-10T18:45:35.267010+0000","last_peered":"2024-09-10T18:45:35.267010+0000","last_clean":"2024-09-10T18:45:35.267010+0000","last_became_active":"2024-09-10T18:44:06.163102+0000","last_became_peered":"2024-09-10T18:44:06.163102+0000","last_unstale":"2024-09-10T18:45:35.267010+0000","last_undegraded":"2024-09-10T18:45:35.267010+0000","last_fullsized":"2024-09-10T18:45:35.267010+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'35","last_scrub_stamp":"2024-09-10T18:45:35.266893+0000","last_deep_scrub":"86'15","last_deep_scrub_stamp":"2024-09-10T18:22:49.298455+0000","last_clean_scrub_stamp":"2024-09-10T18:45:35.266893+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:03.821912+0000","scrub_duration":0.0025491289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"430'14","reported_seq":71,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.044312+0000","last_change":"2024-09-10T18:44:50.370594+0000","last_active":"2024-09-10T18:45:45.044312+0000","last_peered":"2024-09-10T18:45:45.044312+0000","last_clean":"2024-09-10T18:45:45.044312+0000","last_became_active":"2024-09-10T18:44:06.169046+0000","last_became_peered":"2024-09-10T18:44:06.169046+0000","last_unstale":"2024-09-10T18:45:45.044312+0000","last_undegraded":"2024-09-10T18:45:45.044312+0000","last_fullsized":"2024-09-10T18:45:45.044312+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'14","last_scrub_stamp":"2024-09-10T18:44:50.370539+0000","last_deep_scrub":"430'14","last_deep_scrub_stamp":"2024-09-10T18:44:50.370539+0000","last_clean_scrub_stamp":"2024-09-10T18:44:50.370539+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:09.273392+0000","scrub_duration":0.0015470880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"76'81","reported_seq":137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178370+0000","last_change":"2024-09-10T18:44:37.049017+0000","last_active":"2024-09-10T18:45:25.178370+0000","last_peered":"2024-09-10T18:45:25.178370+0000","last_clean":"2024-09-10T18:45:25.178370+0000","last_became_active":"2024-09-10T18:39:00.909261+0000","last_became_peered":"2024-09-10T18:39:00.909261+0000","last_unstale":"2024-09-10T18:45:25.178370+0000","last_undegraded":"2024-09-10T18:45:25.178370+0000","last_fullsized":"2024-09-10T18:45:25.178370+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:37.048955+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:37.048955+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:48.141729+0000","scrub_duration":0.0031844540000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"400'46","reported_seq":603,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.078293+0000","last_change":"2024-09-10T18:45:45.078248+0000","last_active":"2024-09-10T18:45:45.078293+0000","last_peered":"2024-09-10T18:45:45.078293+0000","last_clean":"2024-09-10T18:45:45.078293+0000","last_became_active":"2024-09-10T18:21:33.619013+0000","last_became_peered":"2024-09-10T18:21:33.619013+0000","last_unstale":"2024-09-10T18:45:45.078293+0000","last_undegraded":"2024-09-10T18:45:45.078293+0000","last_fullsized":"2024-09-10T18:45:45.078293+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'46","last_scrub_stamp":"2024-09-10T18:45:45.078152+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-09-10T18:22:40.007685+0000","last_clean_scrub_stamp":"2024-09-10T18:45:45.078152+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:12.093029+0000","scrub_duration":0.0021668640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":506,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660031+0000","last_change":"2024-09-10T18:45:03.468180+0000","last_active":"2024-09-10T18:45:36.660031+0000","last_peered":"2024-09-10T18:45:36.660031+0000","last_clean":"2024-09-10T18:45:36.660031+0000","last_became_active":"2024-09-10T18:21:31.532819+0000","last_became_peered":"2024-09-10T18:21:31.532819+0000","last_unstale":"2024-09-10T18:45:36.660031+0000","last_undegraded":"2024-09-10T18:45:36.660031+0000","last_fullsized":"2024-09-10T18:45:36.660031+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:03.468079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:45:03.468079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:23.740107+0000","scrub_duration":0.002418201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"430'12","reported_seq":874,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-09-10T18:45:55.048657+0000","last_change":"2024-09-10T18:45:37.589204+0000","last_active":"2024-09-10T18:45:55.048657+0000","last_peered":"2024-09-10T18:45:55.048657+0000","last_clean":"2024-09-10T18:45:55.048657+0000","last_became_active":"2024-09-10T18:44:06.174179+0000","last_became_peered":"2024-09-10T18:44:06.174179+0000","last_unstale":"2024-09-10T18:45:55.048657+0000","last_undegraded":"2024-09-10T18:45:55.048657+0000","last_fullsized":"2024-09-10T18:45:55.048657+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'12","last_scrub_stamp":"2024-09-10T18:45:37.589138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:45:37.589138+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:00.198236+0000","scrub_duration":0.0041030880000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":167,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:48.048480+0000","last_change":"2024-09-10T18:45:48.048409+0000","last_active":"2024-09-10T18:45:48.048480+0000","last_peered":"2024-09-10T18:45:48.048480+0000","last_clean":"2024-09-10T18:45:48.048480+0000","last_became_active":"2024-09-10T18:44:06.170517+0000","last_became_peered":"2024-09-10T18:44:06.170517+0000","last_unstale":"2024-09-10T18:45:48.048480+0000","last_undegraded":"2024-09-10T18:45:48.048480+0000","last_fullsized":"2024-09-10T18:45:48.048480+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:48.048342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:21.511676+0000","last_clean_scrub_stamp":"2024-09-10T18:45:48.048342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:54.246653+0000","scrub_duration":0.0016251130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'81","reported_seq":449,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.368737+0000","last_change":"2024-09-10T18:44:53.672481+0000","last_active":"2024-09-10T18:45:36.368737+0000","last_peered":"2024-09-10T18:45:36.368737+0000","last_clean":"2024-09-10T18:45:36.368737+0000","last_became_active":"2024-09-10T18:34:32.815839+0000","last_became_peered":"2024-09-10T18:34:32.815839+0000","last_unstale":"2024-09-10T18:45:36.368737+0000","last_undegraded":"2024-09-10T18:45:36.368737+0000","last_fullsized":"2024-09-10T18:45:36.368737+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:53.672428+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:53.672428+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:00.070252+0000","scrub_duration":0.0033041030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"400'42","reported_seq":628,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.659941+0000","last_change":"2024-09-10T18:45:36.351883+0000","last_active":"2024-09-10T18:45:36.659941+0000","last_peered":"2024-09-10T18:45:36.659941+0000","last_clean":"2024-09-10T18:45:36.659941+0000","last_became_active":"2024-09-10T18:44:06.163204+0000","last_became_peered":"2024-09-10T18:44:06.163204+0000","last_unstale":"2024-09-10T18:45:36.659941+0000","last_undegraded":"2024-09-10T18:45:36.659941+0000","last_fullsized":"2024-09-10T18:45:36.659941+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'42","last_scrub_stamp":"2024-09-10T18:45:36.351816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:45:36.351816+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:53.580868+0000","scrub_duration":0.0020778580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":516,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660926+0000","last_change":"2024-09-10T18:44:29.407186+0000","last_active":"2024-09-10T18:45:36.660926+0000","last_peered":"2024-09-10T18:45:36.660926+0000","last_clean":"2024-09-10T18:45:36.660926+0000","last_became_active":"2024-09-10T18:44:06.161964+0000","last_became_peered":"2024-09-10T18:44:06.161964+0000","last_unstale":"2024-09-10T18:45:36.660926+0000","last_undegraded":"2024-09-10T18:45:36.660926+0000","last_fullsized":"2024-09-10T18:45:36.660926+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:29.407120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:15.765392+0000","last_clean_scrub_stamp":"2024-09-10T18:44:29.407120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:57.792649+0000","scrub_duration":0.0015259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":509,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.367575+0000","last_change":"2024-09-10T18:45:03.758819+0000","last_active":"2024-09-10T18:45:36.367575+0000","last_peered":"2024-09-10T18:45:36.367575+0000","last_clean":"2024-09-10T18:45:36.367575+0000","last_became_active":"2024-09-10T18:21:35.623688+0000","last_became_peered":"2024-09-10T18:21:35.623688+0000","last_unstale":"2024-09-10T18:45:36.367575+0000","last_undegraded":"2024-09-10T18:45:36.367575+0000","last_fullsized":"2024-09-10T18:45:36.367575+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:03.758749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:04.334337+0000","last_clean_scrub_stamp":"2024-09-10T18:45:03.758749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:07.832659+0000","scrub_duration":0.0017596980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":497,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.367783+0000","last_change":"2024-09-10T18:44:18.548351+0000","last_active":"2024-09-10T18:45:36.367783+0000","last_peered":"2024-09-10T18:45:36.367783+0000","last_clean":"2024-09-10T18:45:36.367783+0000","last_became_active":"2024-09-10T18:21:37.642499+0000","last_became_peered":"2024-09-10T18:21:37.642499+0000","last_unstale":"2024-09-10T18:45:36.367783+0000","last_undegraded":"2024-09-10T18:45:36.367783+0000","last_fullsized":"2024-09-10T18:45:36.367783+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:18.548258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:41.282974+0000","last_clean_scrub_stamp":"2024-09-10T18:44:18.548258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017483010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"76'81","reported_seq":431,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660027+0000","last_change":"2024-09-10T18:44:21.578381+0000","last_active":"2024-09-10T18:45:36.660027+0000","last_peered":"2024-09-10T18:45:36.660027+0000","last_clean":"2024-09-10T18:45:36.660027+0000","last_became_active":"2024-09-10T18:39:00.894864+0000","last_became_peered":"2024-09-10T18:39:00.894864+0000","last_unstale":"2024-09-10T18:45:36.660027+0000","last_undegraded":"2024-09-10T18:45:36.660027+0000","last_fullsized":"2024-09-10T18:45:36.660027+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:21.578319+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:21.578319+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014340749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"454'49","reported_seq":640,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.367824+0000","last_change":"2024-09-10T18:45:28.565307+0000","last_active":"2024-09-10T18:45:36.367824+0000","last_peered":"2024-09-10T18:45:36.367824+0000","last_clean":"2024-09-10T18:45:36.367824+0000","last_became_active":"2024-09-10T18:39:00.903341+0000","last_became_peered":"2024-09-10T18:39:00.903341+0000","last_unstale":"2024-09-10T18:45:36.367824+0000","last_undegraded":"2024-09-10T18:45:36.367824+0000","last_fullsized":"2024-09-10T18:45:36.367824+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'49","last_scrub_stamp":"2024-09-10T18:45:28.565240+0000","last_deep_scrub":"454'49","last_deep_scrub_stamp":"2024-09-10T18:45:28.565240+0000","last_clean_scrub_stamp":"2024-09-10T18:45:28.565240+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:44.001456+0000","scrub_duration":0.0054877279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":55,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:30.336251+0000","last_change":"2024-09-10T18:45:30.336217+0000","last_active":"2024-09-10T18:45:30.336251+0000","last_peered":"2024-09-10T18:45:30.336251+0000","last_clean":"2024-09-10T18:45:30.336251+0000","last_became_active":"2024-09-10T18:44:06.173448+0000","last_became_peered":"2024-09-10T18:44:06.173448+0000","last_unstale":"2024-09-10T18:45:30.336251+0000","last_undegraded":"2024-09-10T18:45:30.336251+0000","last_fullsized":"2024-09-10T18:45:30.336251+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:30.336164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:45:30.336164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:00.126568+0000","scrub_duration":0.0017263249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"88'9","reported_seq":794,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-09-10T18:45:55.047561+0000","last_change":"2024-09-10T18:45:11.461223+0000","last_active":"2024-09-10T18:45:55.047561+0000","last_peered":"2024-09-10T18:45:55.047561+0000","last_clean":"2024-09-10T18:45:55.047561+0000","last_became_active":"2024-09-10T18:21:35.622807+0000","last_became_peered":"2024-09-10T18:21:35.622807+0000","last_unstale":"2024-09-10T18:45:55.047561+0000","last_undegraded":"2024-09-10T18:45:55.047561+0000","last_fullsized":"2024-09-10T18:45:55.047561+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-09-10T18:45:11.461149+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-09-10T18:23:07.387603+0000","last_clean_scrub_stamp":"2024-09-10T18:45:11.461149+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:34.810193+0000","scrub_duration":0.001663917,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":507,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.404823+0000","last_change":"2024-09-10T18:44:50.042378+0000","last_active":"2024-09-10T18:45:25.404823+0000","last_peered":"2024-09-10T18:45:25.404823+0000","last_clean":"2024-09-10T18:45:25.404823+0000","last_became_active":"2024-09-10T18:26:27.639239+0000","last_became_peered":"2024-09-10T18:26:27.639239+0000","last_unstale":"2024-09-10T18:45:25.404823+0000","last_undegraded":"2024-09-10T18:45:25.404823+0000","last_fullsized":"2024-09-10T18:45:25.404823+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:50.042313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:06.527847+0000","last_clean_scrub_stamp":"2024-09-10T18:44:50.042313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:03.300708+0000","scrub_duration":0.0013999069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'81","reported_seq":530,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:34.796505+0000","last_change":"2024-09-10T18:45:34.796453+0000","last_active":"2024-09-10T18:45:34.796505+0000","last_peered":"2024-09-10T18:45:34.796505+0000","last_clean":"2024-09-10T18:45:34.796505+0000","last_became_active":"2024-09-10T18:34:32.818323+0000","last_became_peered":"2024-09-10T18:34:32.818323+0000","last_unstale":"2024-09-10T18:45:34.796505+0000","last_undegraded":"2024-09-10T18:45:34.796505+0000","last_fullsized":"2024-09-10T18:45:34.796505+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:34.796382+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:45:34.796382+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:03.611573+0000","scrub_duration":0.0018127449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"400'35","reported_seq":626,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:44.088828+0000","last_change":"2024-09-10T18:45:44.088792+0000","last_active":"2024-09-10T18:45:44.088828+0000","last_peered":"2024-09-10T18:45:44.088828+0000","last_clean":"2024-09-10T18:45:44.088828+0000","last_became_active":"2024-09-10T18:39:00.897648+0000","last_became_peered":"2024-09-10T18:39:00.897648+0000","last_unstale":"2024-09-10T18:45:44.088828+0000","last_undegraded":"2024-09-10T18:45:44.088828+0000","last_fullsized":"2024-09-10T18:45:44.088828+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'35","last_scrub_stamp":"2024-09-10T18:45:44.088721+0000","last_deep_scrub":"123'15","last_deep_scrub_stamp":"2024-09-10T18:31:00.545612+0000","last_clean_scrub_stamp":"2024-09-10T18:45:44.088721+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:59.547739+0000","scrub_duration":0.0023048360000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":147,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:39.042019+0000","last_change":"2024-09-10T18:45:39.041977+0000","last_active":"2024-09-10T18:45:39.042019+0000","last_peered":"2024-09-10T18:45:39.042019+0000","last_clean":"2024-09-10T18:45:39.042019+0000","last_became_active":"2024-09-10T18:39:00.909066+0000","last_became_peered":"2024-09-10T18:39:00.909066+0000","last_unstale":"2024-09-10T18:45:39.042019+0000","last_undegraded":"2024-09-10T18:45:39.042019+0000","last_fullsized":"2024-09-10T18:45:39.042019+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:39.041862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:03.772241+0000","last_clean_scrub_stamp":"2024-09-10T18:45:39.041862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:44.128944+0000","scrub_duration":0.0021130200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"430'15","reported_seq":70,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.044412+0000","last_change":"2024-09-10T18:44:40.433166+0000","last_active":"2024-09-10T18:45:45.044412+0000","last_peered":"2024-09-10T18:45:45.044412+0000","last_clean":"2024-09-10T18:45:45.044412+0000","last_became_active":"2024-09-10T18:44:06.170798+0000","last_became_peered":"2024-09-10T18:44:06.170798+0000","last_unstale":"2024-09-10T18:45:45.044412+0000","last_undegraded":"2024-09-10T18:45:45.044412+0000","last_fullsized":"2024-09-10T18:45:45.044412+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'15","last_scrub_stamp":"2024-09-10T18:44:40.433110+0000","last_deep_scrub":"430'15","last_deep_scrub_stamp":"2024-09-10T18:44:40.433110+0000","last_clean_scrub_stamp":"2024-09-10T18:44:40.433110+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:01.497967+0000","scrub_duration":0.0031203229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:26.303262+0000","last_change":"2024-09-10T18:45:26.303226+0000","last_active":"2024-09-10T18:45:26.303262+0000","last_peered":"2024-09-10T18:45:26.303262+0000","last_clean":"2024-09-10T18:45:26.303262+0000","last_became_active":"2024-09-10T18:44:06.168574+0000","last_became_peered":"2024-09-10T18:44:06.168574+0000","last_unstale":"2024-09-10T18:45:26.303262+0000","last_undegraded":"2024-09-10T18:45:26.303262+0000","last_fullsized":"2024-09-10T18:45:26.303262+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:26.303170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:45:26.303170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:44.559014+0000","scrub_duration":0.0011554740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"76'81","reported_seq":540,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.382294+0000","last_change":"2024-09-10T18:45:12.938053+0000","last_active":"2024-09-10T18:45:25.382294+0000","last_peered":"2024-09-10T18:45:25.382294+0000","last_clean":"2024-09-10T18:45:25.382294+0000","last_became_active":"2024-09-10T18:44:06.171130+0000","last_became_peered":"2024-09-10T18:44:06.171130+0000","last_unstale":"2024-09-10T18:45:25.382294+0000","last_undegraded":"2024-09-10T18:45:25.382294+0000","last_fullsized":"2024-09-10T18:45:25.382294+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:12.937962+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:45:12.937962+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:34.068642+0000","scrub_duration":0.001852795,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"400'39","reported_seq":73,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.658271+0000","last_change":"2024-09-10T18:45:16.353481+0000","last_active":"2024-09-10T18:45:25.658271+0000","last_peered":"2024-09-10T18:45:25.658271+0000","last_clean":"2024-09-10T18:45:25.658271+0000","last_became_active":"2024-09-10T18:44:06.160218+0000","last_became_peered":"2024-09-10T18:44:06.160218+0000","last_unstale":"2024-09-10T18:45:25.658271+0000","last_undegraded":"2024-09-10T18:45:25.658271+0000","last_fullsized":"2024-09-10T18:45:25.658271+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'39","last_scrub_stamp":"2024-09-10T18:45:16.353408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:45:16.353408+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:41.419564+0000","scrub_duration":0.0024552179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":534,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.108169+0000","last_change":"2024-09-10T18:44:59.294759+0000","last_active":"2024-09-10T18:45:25.108169+0000","last_peered":"2024-09-10T18:45:25.108169+0000","last_clean":"2024-09-10T18:45:25.108169+0000","last_became_active":"2024-09-10T18:39:00.897360+0000","last_became_peered":"2024-09-10T18:39:00.897360+0000","last_unstale":"2024-09-10T18:45:25.108169+0000","last_undegraded":"2024-09-10T18:45:25.108169+0000","last_fullsized":"2024-09-10T18:45:25.108169+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:59.294705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:44:59.294705+0000","last_clean_scrub_stamp":"2024-09-10T18:44:59.294705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:07.677285+0000","scrub_duration":0.0015780189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":540,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.381919+0000","last_change":"2024-09-10T18:45:10.886010+0000","last_active":"2024-09-10T18:45:25.381919+0000","last_peered":"2024-09-10T18:45:25.381919+0000","last_clean":"2024-09-10T18:45:25.381919+0000","last_became_active":"2024-09-10T18:44:06.170664+0000","last_became_peered":"2024-09-10T18:44:06.170664+0000","last_unstale":"2024-09-10T18:45:25.381919+0000","last_undegraded":"2024-09-10T18:45:25.381919+0000","last_fullsized":"2024-09-10T18:45:25.381919+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:10.885938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:43:51.751769+0000","last_clean_scrub_stamp":"2024-09-10T18:45:10.885938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:18.607347+0000","scrub_duration":0.001637416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":515,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.107418+0000","last_change":"2024-09-10T18:44:36.057332+0000","last_active":"2024-09-10T18:45:25.107418+0000","last_peered":"2024-09-10T18:45:25.107418+0000","last_clean":"2024-09-10T18:45:25.107418+0000","last_became_active":"2024-09-10T18:26:27.638055+0000","last_became_peered":"2024-09-10T18:26:27.638055+0000","last_unstale":"2024-09-10T18:45:25.107418+0000","last_undegraded":"2024-09-10T18:45:25.107418+0000","last_fullsized":"2024-09-10T18:45:25.107418+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:36.057276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:55.520136+0000","last_clean_scrub_stamp":"2024-09-10T18:44:36.057276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:40.780596+0000","scrub_duration":0.0018953100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'81","reported_seq":157,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178052+0000","last_change":"2024-09-10T18:44:52.958954+0000","last_active":"2024-09-10T18:45:25.178052+0000","last_peered":"2024-09-10T18:45:25.178052+0000","last_clean":"2024-09-10T18:45:25.178052+0000","last_became_active":"2024-09-10T18:44:06.171188+0000","last_became_peered":"2024-09-10T18:44:06.171188+0000","last_unstale":"2024-09-10T18:45:25.178052+0000","last_undegraded":"2024-09-10T18:45:25.178052+0000","last_fullsized":"2024-09-10T18:45:25.178052+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:52.958889+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:52.958889+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:55.212589+0000","scrub_duration":0.0015928489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"410'64","reported_seq":353,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.044299+0000","last_change":"2024-09-10T18:44:30.068625+0000","last_active":"2024-09-10T18:45:45.044299+0000","last_peered":"2024-09-10T18:45:45.044299+0000","last_clean":"2024-09-10T18:45:45.044299+0000","last_became_active":"2024-09-10T18:36:12.922665+0000","last_became_peered":"2024-09-10T18:36:12.922665+0000","last_unstale":"2024-09-10T18:45:45.044299+0000","last_undegraded":"2024-09-10T18:45:45.044299+0000","last_fullsized":"2024-09-10T18:45:45.044299+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'64","last_scrub_stamp":"2024-09-10T18:44:30.068561+0000","last_deep_scrub":"410'64","last_deep_scrub_stamp":"2024-09-10T18:42:43.154380+0000","last_clean_scrub_stamp":"2024-09-10T18:44:30.068561+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:52.195667+0000","scrub_duration":0.0021106079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":530,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.368372+0000","last_change":"2024-09-10T18:44:27.556717+0000","last_active":"2024-09-10T18:45:36.368372+0000","last_peered":"2024-09-10T18:45:36.368372+0000","last_clean":"2024-09-10T18:45:36.368372+0000","last_became_active":"2024-09-10T18:44:06.172679+0000","last_became_peered":"2024-09-10T18:44:06.172679+0000","last_unstale":"2024-09-10T18:45:36.368372+0000","last_undegraded":"2024-09-10T18:45:36.368372+0000","last_fullsized":"2024-09-10T18:45:36.368372+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:27.556656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.274219+0000","last_clean_scrub_stamp":"2024-09-10T18:44:27.556656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032193880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"203'3","reported_seq":141,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178457+0000","last_change":"2024-09-10T18:45:04.135777+0000","last_active":"2024-09-10T18:45:25.178457+0000","last_peered":"2024-09-10T18:45:25.178457+0000","last_clean":"2024-09-10T18:45:25.178457+0000","last_became_active":"2024-09-10T18:39:00.909579+0000","last_became_peered":"2024-09-10T18:39:00.909579+0000","last_unstale":"2024-09-10T18:45:25.178457+0000","last_undegraded":"2024-09-10T18:45:25.178457+0000","last_fullsized":"2024-09-10T18:45:25.178457+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'3","last_scrub_stamp":"2024-09-10T18:45:04.135669+0000","last_deep_scrub":"203'3","last_deep_scrub_stamp":"2024-09-10T18:29:04.922744+0000","last_clean_scrub_stamp":"2024-09-10T18:45:04.135669+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:04.780106+0000","scrub_duration":0.0030437379999999998,"objects_trimmed":0,"snaptrim_duration":0.00091906300000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.658201+0000","last_change":"2024-09-10T18:44:47.459708+0000","last_active":"2024-09-10T18:45:25.658201+0000","last_peered":"2024-09-10T18:45:25.658201+0000","last_clean":"2024-09-10T18:45:25.658201+0000","last_became_active":"2024-09-10T18:44:06.167169+0000","last_became_peered":"2024-09-10T18:44:06.167169+0000","last_unstale":"2024-09-10T18:45:25.658201+0000","last_undegraded":"2024-09-10T18:45:25.658201+0000","last_fullsized":"2024-09-10T18:45:25.658201+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:47.459654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:44:47.459654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:53.413488+0000","scrub_duration":0.001576222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":493,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660712+0000","last_change":"2024-09-10T18:45:28.447152+0000","last_active":"2024-09-10T18:45:36.660712+0000","last_peered":"2024-09-10T18:45:36.660712+0000","last_clean":"2024-09-10T18:45:36.660712+0000","last_became_active":"2024-09-10T18:21:37.641462+0000","last_became_peered":"2024-09-10T18:21:37.641462+0000","last_unstale":"2024-09-10T18:45:36.660712+0000","last_undegraded":"2024-09-10T18:45:36.660712+0000","last_fullsized":"2024-09-10T18:45:36.660712+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:28.447098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:45:28.447098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:57.380889+0000","scrub_duration":0.001909354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'81","reported_seq":584,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:33.416285+0000","last_change":"2024-09-10T18:45:33.416237+0000","last_active":"2024-09-10T18:45:33.416285+0000","last_peered":"2024-09-10T18:45:33.416285+0000","last_clean":"2024-09-10T18:45:33.416285+0000","last_became_active":"2024-09-10T18:44:06.160358+0000","last_became_peered":"2024-09-10T18:44:06.160358+0000","last_unstale":"2024-09-10T18:45:33.416285+0000","last_undegraded":"2024-09-10T18:45:33.416285+0000","last_fullsized":"2024-09-10T18:45:33.416285+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:33.416167+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:45:33.416167+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:50.618981+0000","scrub_duration":0.0016950649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"400'26","reported_seq":142,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.276354+0000","last_change":"2024-09-10T18:45:14.771571+0000","last_active":"2024-09-10T18:45:25.276354+0000","last_peered":"2024-09-10T18:45:25.276354+0000","last_clean":"2024-09-10T18:45:25.276354+0000","last_became_active":"2024-09-10T18:44:06.173529+0000","last_became_peered":"2024-09-10T18:44:06.173529+0000","last_unstale":"2024-09-10T18:45:25.276354+0000","last_undegraded":"2024-09-10T18:45:25.276354+0000","last_fullsized":"2024-09-10T18:45:25.276354+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'26","last_scrub_stamp":"2024-09-10T18:45:14.771498+0000","last_deep_scrub":"388'22","last_deep_scrub_stamp":"2024-09-10T18:41:01.248929+0000","last_clean_scrub_stamp":"2024-09-10T18:45:14.771498+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:27.715714+0000","scrub_duration":0.0021619310000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":541,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:40.118123+0000","last_change":"2024-09-10T18:45:40.118074+0000","last_active":"2024-09-10T18:45:40.118123+0000","last_peered":"2024-09-10T18:45:40.118123+0000","last_clean":"2024-09-10T18:45:40.118123+0000","last_became_active":"2024-09-10T18:26:27.641296+0000","last_became_peered":"2024-09-10T18:26:27.641296+0000","last_unstale":"2024-09-10T18:45:40.118123+0000","last_undegraded":"2024-09-10T18:45:40.118123+0000","last_fullsized":"2024-09-10T18:45:40.118123+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:40.118015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:44.171445+0000","last_clean_scrub_stamp":"2024-09-10T18:45:40.118015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:56.250485+0000","scrub_duration":0.0016665600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"60'1","reported_seq":589,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.275889+0000","last_change":"2024-09-10T18:45:11.772433+0000","last_active":"2024-09-10T18:45:25.275889+0000","last_peered":"2024-09-10T18:45:25.275889+0000","last_clean":"2024-09-10T18:45:25.275889+0000","last_became_active":"2024-09-10T18:44:06.170923+0000","last_became_peered":"2024-09-10T18:44:06.170923+0000","last_unstale":"2024-09-10T18:45:25.275889+0000","last_undegraded":"2024-09-10T18:45:25.275889+0000","last_fullsized":"2024-09-10T18:45:25.275889+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'1","last_scrub_stamp":"2024-09-10T18:45:11.772332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:45:11.772332+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:22.236606+0000","scrub_duration":0.001552512,"objects_trimmed":0,"snaptrim_duration":0.000955812,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":568,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:29.760795+0000","last_change":"2024-09-10T18:45:29.760751+0000","last_active":"2024-09-10T18:45:29.760795+0000","last_peered":"2024-09-10T18:45:29.760795+0000","last_clean":"2024-09-10T18:45:29.760795+0000","last_became_active":"2024-09-10T18:39:00.910574+0000","last_became_peered":"2024-09-10T18:39:00.910574+0000","last_unstale":"2024-09-10T18:45:29.760795+0000","last_undegraded":"2024-09-10T18:45:29.760795+0000","last_fullsized":"2024-09-10T18:45:29.760795+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:29.760692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:53.488434+0000","last_clean_scrub_stamp":"2024-09-10T18:45:29.760692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:58.558452+0000","scrub_duration":0.0016954629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":507,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405511+0000","last_change":"2024-09-10T18:44:46.062649+0000","last_active":"2024-09-10T18:45:25.405511+0000","last_peered":"2024-09-10T18:45:25.405511+0000","last_clean":"2024-09-10T18:45:25.405511+0000","last_became_active":"2024-09-10T18:26:27.638796+0000","last_became_peered":"2024-09-10T18:26:27.638796+0000","last_unstale":"2024-09-10T18:45:25.405511+0000","last_undegraded":"2024-09-10T18:45:25.405511+0000","last_fullsized":"2024-09-10T18:45:25.405511+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:46.062578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:44:46.062578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:11.953294+0000","scrub_duration":0.0016260160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"76'81","reported_seq":434,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.107881+0000","last_change":"2024-09-10T18:44:23.030550+0000","last_active":"2024-09-10T18:45:25.107881+0000","last_peered":"2024-09-10T18:45:25.107881+0000","last_clean":"2024-09-10T18:45:25.107881+0000","last_became_active":"2024-09-10T18:34:00.165384+0000","last_became_peered":"2024-09-10T18:34:00.165384+0000","last_unstale":"2024-09-10T18:45:25.107881+0000","last_undegraded":"2024-09-10T18:45:25.107881+0000","last_fullsized":"2024-09-10T18:45:25.107881+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:23.030493+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:44:23.030493+0000","last_clean_scrub_stamp":"2024-09-10T18:44:23.030493+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:47.160426+0000","scrub_duration":0.0018700380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"430'42","reported_seq":75,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:31.301959+0000","last_change":"2024-09-10T18:45:31.301887+0000","last_active":"2024-09-10T18:45:31.301959+0000","last_peered":"2024-09-10T18:45:31.301959+0000","last_clean":"2024-09-10T18:45:31.301959+0000","last_became_active":"2024-09-10T18:44:06.176362+0000","last_became_peered":"2024-09-10T18:44:06.176362+0000","last_unstale":"2024-09-10T18:45:31.301959+0000","last_undegraded":"2024-09-10T18:45:31.301959+0000","last_fullsized":"2024-09-10T18:45:31.301959+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'42","last_scrub_stamp":"2024-09-10T18:45:31.301813+0000","last_deep_scrub":"430'42","last_deep_scrub_stamp":"2024-09-10T18:44:09.491797+0000","last_clean_scrub_stamp":"2024-09-10T18:45:31.301813+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:42.237027+0000","scrub_duration":0.0017048230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":55,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.302296+0000","last_change":"2024-09-10T18:45:36.302252+0000","last_active":"2024-09-10T18:45:36.302296+0000","last_peered":"2024-09-10T18:45:36.302296+0000","last_clean":"2024-09-10T18:45:36.302296+0000","last_became_active":"2024-09-10T18:44:06.164270+0000","last_became_peered":"2024-09-10T18:44:06.164270+0000","last_unstale":"2024-09-10T18:45:36.302296+0000","last_undegraded":"2024-09-10T18:45:36.302296+0000","last_fullsized":"2024-09-10T18:45:36.302296+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:36.302197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:24:18.269034+0000","last_clean_scrub_stamp":"2024-09-10T18:45:36.302197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:03.497915+0000","scrub_duration":0.001270715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":1997,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:48.317325+0000","last_change":"2024-09-10T18:44:52.094924+0000","last_active":"2024-09-10T18:45:48.317325+0000","last_peered":"2024-09-10T18:45:48.317325+0000","last_clean":"2024-09-10T18:45:48.317325+0000","last_became_active":"2024-09-10T18:44:06.165291+0000","last_became_peered":"2024-09-10T18:44:06.165291+0000","last_unstale":"2024-09-10T18:45:48.317325+0000","last_undegraded":"2024-09-10T18:45:48.317325+0000","last_fullsized":"2024-09-10T18:45:48.317325+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T18:44:52.094828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:44:52.094828+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:14.362433+0000","scrub_duration":0.002081846,"objects_trimmed":0,"snaptrim_duration":0.0027724939999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1446,"num_read_kb":1441,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":157,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:38.074401+0000","last_change":"2024-09-10T18:45:38.074366+0000","last_active":"2024-09-10T18:45:38.074401+0000","last_peered":"2024-09-10T18:45:38.074401+0000","last_clean":"2024-09-10T18:45:38.074401+0000","last_became_active":"2024-09-10T18:44:06.172014+0000","last_became_peered":"2024-09-10T18:44:06.172014+0000","last_unstale":"2024-09-10T18:45:38.074401+0000","last_undegraded":"2024-09-10T18:45:38.074401+0000","last_fullsized":"2024-09-10T18:45:38.074401+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:38.074301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:01.170272+0000","last_clean_scrub_stamp":"2024-09-10T18:45:38.074301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:48.727311+0000","scrub_duration":0.001632566,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":528,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.107697+0000","last_change":"2024-09-10T18:44:58.257403+0000","last_active":"2024-09-10T18:45:25.107697+0000","last_peered":"2024-09-10T18:45:25.107697+0000","last_clean":"2024-09-10T18:45:25.107697+0000","last_became_active":"2024-09-10T18:39:00.897525+0000","last_became_peered":"2024-09-10T18:39:00.897525+0000","last_unstale":"2024-09-10T18:45:25.107697+0000","last_undegraded":"2024-09-10T18:45:25.107697+0000","last_fullsized":"2024-09-10T18:45:25.107697+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:58.257331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:44:58.257331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:25.650629+0000","scrub_duration":0.0015563700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"400'38","reported_seq":545,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:33.834208+0000","last_change":"2024-09-10T18:45:33.834140+0000","last_active":"2024-09-10T18:45:33.834208+0000","last_peered":"2024-09-10T18:45:33.834208+0000","last_clean":"2024-09-10T18:45:33.834208+0000","last_became_active":"2024-09-10T18:26:27.644641+0000","last_became_peered":"2024-09-10T18:26:27.644641+0000","last_unstale":"2024-09-10T18:45:33.834208+0000","last_undegraded":"2024-09-10T18:45:33.834208+0000","last_fullsized":"2024-09-10T18:45:33.834208+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'38","last_scrub_stamp":"2024-09-10T18:45:33.834062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:45:33.834062+0000","objects_scrubbed":7,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:43.745833+0000","scrub_duration":0.0022866380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":533,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660247+0000","last_change":"2024-09-10T18:45:35.331998+0000","last_active":"2024-09-10T18:45:36.660247+0000","last_peered":"2024-09-10T18:45:36.660247+0000","last_clean":"2024-09-10T18:45:36.660247+0000","last_became_active":"2024-09-10T18:39:00.894778+0000","last_became_peered":"2024-09-10T18:39:00.894778+0000","last_unstale":"2024-09-10T18:45:36.660247+0000","last_undegraded":"2024-09-10T18:45:36.660247+0000","last_fullsized":"2024-09-10T18:45:36.660247+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:35.331930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:45:35.331930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:42.960860+0000","scrub_duration":0.001699964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'81","reported_seq":593,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405285+0000","last_change":"2024-09-10T18:45:03.266337+0000","last_active":"2024-09-10T18:45:25.405285+0000","last_peered":"2024-09-10T18:45:25.405285+0000","last_clean":"2024-09-10T18:45:25.405285+0000","last_became_active":"2024-09-10T18:44:06.165287+0000","last_became_peered":"2024-09-10T18:44:06.165287+0000","last_unstale":"2024-09-10T18:45:25.405285+0000","last_undegraded":"2024-09-10T18:45:25.405285+0000","last_fullsized":"2024-09-10T18:45:25.405285+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:03.266267+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:45:03.266267+0000","last_clean_scrub_stamp":"2024-09-10T18:45:03.266267+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:20.499388+0000","scrub_duration":0.0019289680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":532,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405433+0000","last_change":"2024-09-10T18:44:59.162745+0000","last_active":"2024-09-10T18:45:25.405433+0000","last_peered":"2024-09-10T18:45:25.405433+0000","last_clean":"2024-09-10T18:45:25.405433+0000","last_became_active":"2024-09-10T18:39:00.890829+0000","last_became_peered":"2024-09-10T18:39:00.890829+0000","last_unstale":"2024-09-10T18:45:25.405433+0000","last_undegraded":"2024-09-10T18:45:25.405433+0000","last_fullsized":"2024-09-10T18:45:25.405433+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:59.162685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:44:59.162685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:10.659296+0000","scrub_duration":0.001394831,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.177795+0000","last_change":"2024-09-10T18:45:17.143890+0000","last_active":"2024-09-10T18:45:25.177795+0000","last_peered":"2024-09-10T18:45:25.177795+0000","last_clean":"2024-09-10T18:45:25.177795+0000","last_became_active":"2024-09-10T18:39:00.900703+0000","last_became_peered":"2024-09-10T18:39:00.900703+0000","last_unstale":"2024-09-10T18:45:25.177795+0000","last_undegraded":"2024-09-10T18:45:25.177795+0000","last_fullsized":"2024-09-10T18:45:25.177795+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:17.143815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:44:14.867416+0000","last_clean_scrub_stamp":"2024-09-10T18:45:17.143815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:45.666741+0000","scrub_duration":0.0013807190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"76'81","reported_seq":490,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.108063+0000","last_change":"2024-09-10T18:44:25.005414+0000","last_active":"2024-09-10T18:45:25.108063+0000","last_peered":"2024-09-10T18:45:25.108063+0000","last_clean":"2024-09-10T18:45:25.108063+0000","last_became_active":"2024-09-10T18:44:06.163834+0000","last_became_peered":"2024-09-10T18:44:06.163834+0000","last_unstale":"2024-09-10T18:45:25.108063+0000","last_undegraded":"2024-09-10T18:45:25.108063+0000","last_fullsized":"2024-09-10T18:45:25.108063+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:25.005338+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:25.005338+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:52.017381+0000","scrub_duration":0.001623403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"400'32","reported_seq":569,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.368510+0000","last_change":"2024-09-10T18:44:52.696594+0000","last_active":"2024-09-10T18:45:36.368510+0000","last_peered":"2024-09-10T18:45:36.368510+0000","last_clean":"2024-09-10T18:45:36.368510+0000","last_became_active":"2024-09-10T18:26:27.648566+0000","last_became_peered":"2024-09-10T18:26:27.648566+0000","last_unstale":"2024-09-10T18:45:36.368510+0000","last_undegraded":"2024-09-10T18:45:36.368510+0000","last_fullsized":"2024-09-10T18:45:36.368510+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'32","last_scrub_stamp":"2024-09-10T18:44:52.696522+0000","last_deep_scrub":"283'24","last_deep_scrub_stamp":"2024-09-10T18:34:19.404713+0000","last_clean_scrub_stamp":"2024-09-10T18:44:52.696522+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:00.970889+0000","scrub_duration":0.0022833799999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":516,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.368074+0000","last_change":"2024-09-10T18:45:26.555814+0000","last_active":"2024-09-10T18:45:36.368074+0000","last_peered":"2024-09-10T18:45:36.368074+0000","last_clean":"2024-09-10T18:45:36.368074+0000","last_became_active":"2024-09-10T18:21:31.528370+0000","last_became_peered":"2024-09-10T18:21:31.528370+0000","last_unstale":"2024-09-10T18:45:36.368074+0000","last_undegraded":"2024-09-10T18:45:36.368074+0000","last_fullsized":"2024-09-10T18:45:36.368074+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:26.555747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:45:26.555747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:35.118776+0000","scrub_duration":0.0016912190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.177800+0000","last_change":"2024-09-10T18:44:30.955440+0000","last_active":"2024-09-10T18:45:25.177800+0000","last_peered":"2024-09-10T18:45:25.177800+0000","last_clean":"2024-09-10T18:45:25.177800+0000","last_became_active":"2024-09-10T18:39:00.907231+0000","last_became_peered":"2024-09-10T18:39:00.907231+0000","last_unstale":"2024-09-10T18:45:25.177800+0000","last_undegraded":"2024-09-10T18:45:25.177800+0000","last_fullsized":"2024-09-10T18:45:25.177800+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:30.955372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:00.387887+0000","last_clean_scrub_stamp":"2024-09-10T18:44:30.955372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:52.363842+0000","scrub_duration":0.003736241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":527,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660217+0000","last_change":"2024-09-10T18:45:22.450293+0000","last_active":"2024-09-10T18:45:36.660217+0000","last_peered":"2024-09-10T18:45:36.660217+0000","last_clean":"2024-09-10T18:45:36.660217+0000","last_became_active":"2024-09-10T18:39:00.894963+0000","last_became_peered":"2024-09-10T18:39:00.894963+0000","last_unstale":"2024-09-10T18:45:36.660217+0000","last_undegraded":"2024-09-10T18:45:36.660217+0000","last_fullsized":"2024-09-10T18:45:36.660217+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:22.450241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:45:22.450241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:49.769151+0000","scrub_duration":0.0015896980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'81","reported_seq":137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178120+0000","last_change":"2024-09-10T18:44:33.961034+0000","last_active":"2024-09-10T18:45:25.178120+0000","last_peered":"2024-09-10T18:45:25.178120+0000","last_clean":"2024-09-10T18:45:25.178120+0000","last_became_active":"2024-09-10T18:39:00.909151+0000","last_became_peered":"2024-09-10T18:39:00.909151+0000","last_unstale":"2024-09-10T18:45:25.178120+0000","last_undegraded":"2024-09-10T18:45:25.178120+0000","last_fullsized":"2024-09-10T18:45:25.178120+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:33.960921+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:33.960921+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:03.520006+0000","scrub_duration":0.0025747090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"400'53","reported_seq":645,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.368491+0000","last_change":"2024-09-10T18:45:04.735964+0000","last_active":"2024-09-10T18:45:36.368491+0000","last_peered":"2024-09-10T18:45:36.368491+0000","last_clean":"2024-09-10T18:45:36.368491+0000","last_became_active":"2024-09-10T18:38:43.333186+0000","last_became_peered":"2024-09-10T18:38:43.333186+0000","last_unstale":"2024-09-10T18:45:36.368491+0000","last_undegraded":"2024-09-10T18:45:36.368491+0000","last_fullsized":"2024-09-10T18:45:36.368491+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'53","last_scrub_stamp":"2024-09-10T18:45:04.735870+0000","last_deep_scrub":"342'39","last_deep_scrub_stamp":"2024-09-10T18:37:53.386563+0000","last_clean_scrub_stamp":"2024-09-10T18:45:04.735870+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:14.811693+0000","scrub_duration":0.0029873159999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"165'2","reported_seq":498,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:29.833256+0000","last_change":"2024-09-10T18:45:29.833199+0000","last_active":"2024-09-10T18:45:29.833256+0000","last_peered":"2024-09-10T18:45:29.833256+0000","last_clean":"2024-09-10T18:45:29.833256+0000","last_became_active":"2024-09-10T18:26:27.644292+0000","last_became_peered":"2024-09-10T18:26:27.644292+0000","last_unstale":"2024-09-10T18:45:29.833256+0000","last_undegraded":"2024-09-10T18:45:29.833256+0000","last_fullsized":"2024-09-10T18:45:29.833256+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'2","last_scrub_stamp":"2024-09-10T18:45:29.833136+0000","last_deep_scrub":"165'2","last_deep_scrub_stamp":"2024-09-10T18:45:29.833136+0000","last_clean_scrub_stamp":"2024-09-10T18:45:29.833136+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:40.817774+0000","scrub_duration":0.0021491370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":540,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:32.136004+0000","last_change":"2024-09-10T18:45:32.135970+0000","last_active":"2024-09-10T18:45:32.136004+0000","last_peered":"2024-09-10T18:45:32.136004+0000","last_clean":"2024-09-10T18:45:32.136004+0000","last_became_active":"2024-09-10T18:44:06.162385+0000","last_became_peered":"2024-09-10T18:44:06.162385+0000","last_unstale":"2024-09-10T18:45:32.136004+0000","last_undegraded":"2024-09-10T18:45:32.136004+0000","last_fullsized":"2024-09-10T18:45:32.136004+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:32.135918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:45:32.135918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:55.179790+0000","scrub_duration":0.0010348410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":556,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.276436+0000","last_change":"2024-09-10T18:45:18.788891+0000","last_active":"2024-09-10T18:45:25.276436+0000","last_peered":"2024-09-10T18:45:25.276436+0000","last_clean":"2024-09-10T18:45:25.276436+0000","last_became_active":"2024-09-10T18:21:37.642527+0000","last_became_peered":"2024-09-10T18:21:37.642527+0000","last_unstale":"2024-09-10T18:45:25.276436+0000","last_undegraded":"2024-09-10T18:45:25.276436+0000","last_fullsized":"2024-09-10T18:45:25.276436+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:18.788806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:45:18.788806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:43.440299+0000","scrub_duration":0.0019326199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"76'81","reported_seq":571,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:49.352499+0000","last_change":"2024-09-10T18:45:49.352441+0000","last_active":"2024-09-10T18:45:49.352499+0000","last_peered":"2024-09-10T18:45:49.352499+0000","last_clean":"2024-09-10T18:45:49.352499+0000","last_became_active":"2024-09-10T18:34:32.827174+0000","last_became_peered":"2024-09-10T18:34:32.827174+0000","last_unstale":"2024-09-10T18:45:49.352499+0000","last_undegraded":"2024-09-10T18:45:49.352499+0000","last_fullsized":"2024-09-10T18:45:49.352499+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:49.352364+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:44:16.864682+0000","last_clean_scrub_stamp":"2024-09-10T18:45:49.352364+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:05.821643+0000","scrub_duration":0.001932034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"400'35","reported_seq":286,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405153+0000","last_change":"2024-09-10T18:45:15.324234+0000","last_active":"2024-09-10T18:45:25.405153+0000","last_peered":"2024-09-10T18:45:25.405153+0000","last_clean":"2024-09-10T18:45:25.405153+0000","last_became_active":"2024-09-10T18:32:51.118251+0000","last_became_peered":"2024-09-10T18:32:51.118251+0000","last_unstale":"2024-09-10T18:45:25.405153+0000","last_undegraded":"2024-09-10T18:45:25.405153+0000","last_fullsized":"2024-09-10T18:45:25.405153+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'35","last_scrub_stamp":"2024-09-10T18:45:15.324166+0000","last_deep_scrub":"400'35","last_deep_scrub_stamp":"2024-09-10T18:45:15.324166+0000","last_clean_scrub_stamp":"2024-09-10T18:45:15.324166+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:21.787018+0000","scrub_duration":0.0020122489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":516,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660495+0000","last_change":"2024-09-10T18:44:16.635241+0000","last_active":"2024-09-10T18:45:36.660495+0000","last_peered":"2024-09-10T18:45:36.660495+0000","last_clean":"2024-09-10T18:45:36.660495+0000","last_became_active":"2024-09-10T18:44:06.161655+0000","last_became_peered":"2024-09-10T18:44:06.161655+0000","last_unstale":"2024-09-10T18:45:36.660495+0000","last_undegraded":"2024-09-10T18:45:36.660495+0000","last_fullsized":"2024-09-10T18:45:36.660495+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:16.635176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:59.757255+0000","last_clean_scrub_stamp":"2024-09-10T18:44:16.635176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001634649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"357'11","reported_seq":888,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.044457+0000","last_change":"2024-09-10T18:45:32.726221+0000","last_active":"2024-09-10T18:45:45.044457+0000","last_peered":"2024-09-10T18:45:45.044457+0000","last_clean":"2024-09-10T18:45:45.044457+0000","last_became_active":"2024-09-10T18:39:00.909747+0000","last_became_peered":"2024-09-10T18:39:00.909747+0000","last_unstale":"2024-09-10T18:45:45.044457+0000","last_undegraded":"2024-09-10T18:45:45.044457+0000","last_fullsized":"2024-09-10T18:45:45.044457+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'11","last_scrub_stamp":"2024-09-10T18:45:32.726145+0000","last_deep_scrub":"357'11","last_deep_scrub_stamp":"2024-09-10T18:44:06.249311+0000","last_clean_scrub_stamp":"2024-09-10T18:45:32.726145+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:49.065008+0000","scrub_duration":0.0020634149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":500,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.368220+0000","last_change":"2024-09-10T18:44:49.693689+0000","last_active":"2024-09-10T18:45:36.368220+0000","last_peered":"2024-09-10T18:45:36.368220+0000","last_clean":"2024-09-10T18:45:36.368220+0000","last_became_active":"2024-09-10T18:21:37.641977+0000","last_became_peered":"2024-09-10T18:21:37.641977+0000","last_unstale":"2024-09-10T18:45:36.368220+0000","last_undegraded":"2024-09-10T18:45:36.368220+0000","last_fullsized":"2024-09-10T18:45:36.368220+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:49.693531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:53.987811+0000","last_clean_scrub_stamp":"2024-09-10T18:44:49.693531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:13.648425+0000","scrub_duration":0.0021128950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"76'81","reported_seq":243,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.368221+0000","last_change":"2024-09-10T18:44:45.718577+0000","last_active":"2024-09-10T18:45:36.368221+0000","last_peered":"2024-09-10T18:45:36.368221+0000","last_clean":"2024-09-10T18:45:36.368221+0000","last_became_active":"2024-09-10T18:34:32.818251+0000","last_became_peered":"2024-09-10T18:34:32.818251+0000","last_unstale":"2024-09-10T18:45:36.368221+0000","last_undegraded":"2024-09-10T18:45:36.368221+0000","last_fullsized":"2024-09-10T18:45:36.368221+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:45.718509+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:42:53.168123+0000","last_clean_scrub_stamp":"2024-09-10T18:44:45.718509+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018204969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"400'25","reported_seq":55,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.658163+0000","last_change":"2024-09-10T18:44:27.399562+0000","last_active":"2024-09-10T18:45:25.658163+0000","last_peered":"2024-09-10T18:45:25.658163+0000","last_clean":"2024-09-10T18:45:25.658163+0000","last_became_active":"2024-09-10T18:44:06.169580+0000","last_became_peered":"2024-09-10T18:44:06.169580+0000","last_unstale":"2024-09-10T18:45:25.658163+0000","last_undegraded":"2024-09-10T18:45:25.658163+0000","last_fullsized":"2024-09-10T18:45:25.658163+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'25","last_scrub_stamp":"2024-09-10T18:44:27.399496+0000","last_deep_scrub":"302'19","last_deep_scrub_stamp":"2024-09-10T18:35:18.836131+0000","last_clean_scrub_stamp":"2024-09-10T18:44:27.399496+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:47.178093+0000","scrub_duration":0.002811334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"265'2","reported_seq":151,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:50.111077+0000","last_change":"2024-09-10T18:45:50.111017+0000","last_active":"2024-09-10T18:45:50.111077+0000","last_peered":"2024-09-10T18:45:50.111077+0000","last_clean":"2024-09-10T18:45:50.111077+0000","last_became_active":"2024-09-10T18:39:00.904967+0000","last_became_peered":"2024-09-10T18:39:00.904967+0000","last_unstale":"2024-09-10T18:45:50.111077+0000","last_undegraded":"2024-09-10T18:45:50.111077+0000","last_fullsized":"2024-09-10T18:45:50.111077+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'2","last_scrub_stamp":"2024-09-10T18:45:50.110941+0000","last_deep_scrub":"265'2","last_deep_scrub_stamp":"2024-09-10T18:33:11.047305+0000","last_clean_scrub_stamp":"2024-09-10T18:45:50.110941+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:58.081284+0000","scrub_duration":0.0018958600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":544,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.275923+0000","last_change":"2024-09-10T18:45:16.751583+0000","last_active":"2024-09-10T18:45:25.275923+0000","last_peered":"2024-09-10T18:45:25.275923+0000","last_clean":"2024-09-10T18:45:25.275923+0000","last_became_active":"2024-09-10T18:21:35.624663+0000","last_became_peered":"2024-09-10T18:21:35.624663+0000","last_unstale":"2024-09-10T18:45:25.275923+0000","last_undegraded":"2024-09-10T18:45:25.275923+0000","last_fullsized":"2024-09-10T18:45:25.275923+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:16.751528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:45:16.751528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:44.514774+0000","scrub_duration":0.0017661840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.658050+0000","last_change":"2024-09-10T18:45:14.424451+0000","last_active":"2024-09-10T18:45:25.658050+0000","last_peered":"2024-09-10T18:45:25.658050+0000","last_clean":"2024-09-10T18:45:25.658050+0000","last_became_active":"2024-09-10T18:44:06.168584+0000","last_became_peered":"2024-09-10T18:44:06.168584+0000","last_unstale":"2024-09-10T18:45:25.658050+0000","last_undegraded":"2024-09-10T18:45:25.658050+0000","last_fullsized":"2024-09-10T18:45:25.658050+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:14.424387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:45:14.424387+0000","last_clean_scrub_stamp":"2024-09-10T18:45:14.424387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:28.659520+0000","scrub_duration":0.0016728649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'81","reported_seq":529,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.381460+0000","last_change":"2024-09-10T18:44:41.625296+0000","last_active":"2024-09-10T18:45:25.381460+0000","last_peered":"2024-09-10T18:45:25.381460+0000","last_clean":"2024-09-10T18:45:25.381460+0000","last_became_active":"2024-09-10T18:39:00.921868+0000","last_became_peered":"2024-09-10T18:39:00.921868+0000","last_unstale":"2024-09-10T18:45:25.381460+0000","last_undegraded":"2024-09-10T18:45:25.381460+0000","last_fullsized":"2024-09-10T18:45:25.381460+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:41.625226+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:44:41.625226+0000","last_clean_scrub_stamp":"2024-09-10T18:44:41.625226+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:50.236128+0000","scrub_duration":0.0016452070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"400'60","reported_seq":639,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.381987+0000","last_change":"2024-09-10T18:45:08.864502+0000","last_active":"2024-09-10T18:45:25.381987+0000","last_peered":"2024-09-10T18:45:25.381987+0000","last_clean":"2024-09-10T18:45:25.381987+0000","last_became_active":"2024-09-10T18:29:26.995237+0000","last_became_peered":"2024-09-10T18:29:26.995237+0000","last_unstale":"2024-09-10T18:45:25.381987+0000","last_undegraded":"2024-09-10T18:45:25.381987+0000","last_fullsized":"2024-09-10T18:45:25.381987+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'60","last_scrub_stamp":"2024-09-10T18:45:08.864434+0000","last_deep_scrub":"86'28","last_deep_scrub_stamp":"2024-09-10T18:22:54.863323+0000","last_clean_scrub_stamp":"2024-09-10T18:45:08.864434+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:28.004968+0000","scrub_duration":0.002365561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":543,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405820+0000","last_change":"2024-09-10T18:44:51.049789+0000","last_active":"2024-09-10T18:45:25.405820+0000","last_peered":"2024-09-10T18:45:25.405820+0000","last_clean":"2024-09-10T18:45:25.405820+0000","last_became_active":"2024-09-10T18:39:00.895121+0000","last_became_peered":"2024-09-10T18:39:00.895121+0000","last_unstale":"2024-09-10T18:45:25.405820+0000","last_undegraded":"2024-09-10T18:45:25.405820+0000","last_fullsized":"2024-09-10T18:45:25.405820+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:51.049732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:44:51.049732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:54.189937+0000","scrub_duration":0.001321523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":588,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.276063+0000","last_change":"2024-09-10T18:44:52.960451+0000","last_active":"2024-09-10T18:45:25.276063+0000","last_peered":"2024-09-10T18:45:25.276063+0000","last_clean":"2024-09-10T18:45:25.276063+0000","last_became_active":"2024-09-10T18:44:06.171739+0000","last_became_peered":"2024-09-10T18:44:06.171739+0000","last_unstale":"2024-09-10T18:45:25.276063+0000","last_undegraded":"2024-09-10T18:45:25.276063+0000","last_fullsized":"2024-09-10T18:45:25.276063+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:52.960390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:34:21.447291+0000","last_clean_scrub_stamp":"2024-09-10T18:44:52.960390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:05.338263+0000","scrub_duration":0.001197523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":510,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405276+0000","last_change":"2024-09-10T18:44:44.066629+0000","last_active":"2024-09-10T18:45:25.405276+0000","last_peered":"2024-09-10T18:45:25.405276+0000","last_clean":"2024-09-10T18:45:25.405276+0000","last_became_active":"2024-09-10T18:21:37.641999+0000","last_became_peered":"2024-09-10T18:21:37.641999+0000","last_unstale":"2024-09-10T18:45:25.405276+0000","last_undegraded":"2024-09-10T18:45:25.405276+0000","last_fullsized":"2024-09-10T18:45:25.405276+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:44.066562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:37:04.058646+0000","last_clean_scrub_stamp":"2024-09-10T18:44:44.066562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:02.662427+0000","scrub_duration":0.0029830899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"76'81","reported_seq":465,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660553+0000","last_change":"2024-09-10T18:44:38.497287+0000","last_active":"2024-09-10T18:45:36.660553+0000","last_peered":"2024-09-10T18:45:36.660553+0000","last_clean":"2024-09-10T18:45:36.660553+0000","last_became_active":"2024-09-10T18:44:06.161273+0000","last_became_peered":"2024-09-10T18:44:06.161273+0000","last_unstale":"2024-09-10T18:45:36.660553+0000","last_undegraded":"2024-09-10T18:45:36.660553+0000","last_fullsized":"2024-09-10T18:45:36.660553+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:38.497228+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:38.497228+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:07.171855+0000","scrub_duration":0.001927752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"400'29","reported_seq":560,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.367672+0000","last_change":"2024-09-10T18:44:59.729207+0000","last_active":"2024-09-10T18:45:36.367672+0000","last_peered":"2024-09-10T18:45:36.367672+0000","last_clean":"2024-09-10T18:45:36.367672+0000","last_became_active":"2024-09-10T18:21:33.614622+0000","last_became_peered":"2024-09-10T18:21:33.614622+0000","last_unstale":"2024-09-10T18:45:36.367672+0000","last_undegraded":"2024-09-10T18:45:36.367672+0000","last_fullsized":"2024-09-10T18:45:36.367672+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'29","last_scrub_stamp":"2024-09-10T18:44:59.729140+0000","last_deep_scrub":"301'23","last_deep_scrub_stamp":"2024-09-10T18:40:55.034050+0000","last_clean_scrub_stamp":"2024-09-10T18:44:59.729140+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:27.147300+0000","scrub_duration":0.0038454589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":523,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:37.340815+0000","last_change":"2024-09-10T18:45:37.340781+0000","last_active":"2024-09-10T18:45:37.340815+0000","last_peered":"2024-09-10T18:45:37.340815+0000","last_clean":"2024-09-10T18:45:37.340815+0000","last_became_active":"2024-09-10T18:21:31.531707+0000","last_became_peered":"2024-09-10T18:21:31.531707+0000","last_unstale":"2024-09-10T18:45:37.340815+0000","last_undegraded":"2024-09-10T18:45:37.340815+0000","last_fullsized":"2024-09-10T18:45:37.340815+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:37.340714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:12.489588+0000","last_clean_scrub_stamp":"2024-09-10T18:45:37.340714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:41.344523+0000","scrub_duration":0.001535542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":578,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.761278+0000","last_change":"2024-09-10T18:45:25.761240+0000","last_active":"2024-09-10T18:45:25.761278+0000","last_peered":"2024-09-10T18:45:25.761278+0000","last_clean":"2024-09-10T18:45:25.761278+0000","last_became_active":"2024-09-10T18:44:06.171685+0000","last_became_peered":"2024-09-10T18:44:06.171685+0000","last_unstale":"2024-09-10T18:45:25.761278+0000","last_undegraded":"2024-09-10T18:45:25.761278+0000","last_fullsized":"2024-09-10T18:45:25.761278+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:25.761185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:42.443984+0000","last_clean_scrub_stamp":"2024-09-10T18:45:25.761185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:36.399679+0000","scrub_duration":0.001397183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":490,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660099+0000","last_change":"2024-09-10T18:44:25.486602+0000","last_active":"2024-09-10T18:45:36.660099+0000","last_peered":"2024-09-10T18:45:36.660099+0000","last_clean":"2024-09-10T18:45:36.660099+0000","last_became_active":"2024-09-10T18:21:37.644552+0000","last_became_peered":"2024-09-10T18:21:37.644552+0000","last_unstale":"2024-09-10T18:45:36.660099+0000","last_undegraded":"2024-09-10T18:45:36.660099+0000","last_fullsized":"2024-09-10T18:45:36.660099+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:25.486535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:38:50.727231+0000","last_clean_scrub_stamp":"2024-09-10T18:44:25.486535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018253130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"76'81","reported_seq":243,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:41.369034+0000","last_change":"2024-09-10T18:45:41.369001+0000","last_active":"2024-09-10T18:45:41.369034+0000","last_peered":"2024-09-10T18:45:41.369034+0000","last_clean":"2024-09-10T18:45:41.369034+0000","last_became_active":"2024-09-10T18:34:32.822196+0000","last_became_peered":"2024-09-10T18:34:32.822196+0000","last_unstale":"2024-09-10T18:45:41.369034+0000","last_undegraded":"2024-09-10T18:45:41.369034+0000","last_fullsized":"2024-09-10T18:45:41.369034+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:41.368947+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:45:41.368947+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:03.686724+0000","scrub_duration":0.001398479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"434'46","reported_seq":188,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:30.091086+0000","last_change":"2024-09-10T18:45:30.091033+0000","last_active":"2024-09-10T18:45:30.091086+0000","last_peered":"2024-09-10T18:45:30.091086+0000","last_clean":"2024-09-10T18:45:30.091086+0000","last_became_active":"2024-09-10T18:39:00.902783+0000","last_became_peered":"2024-09-10T18:39:00.902783+0000","last_unstale":"2024-09-10T18:45:30.091086+0000","last_undegraded":"2024-09-10T18:45:30.091086+0000","last_fullsized":"2024-09-10T18:45:30.091086+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'46","last_scrub_stamp":"2024-09-10T18:45:30.090965+0000","last_deep_scrub":"434'46","last_deep_scrub_stamp":"2024-09-10T18:44:19.904898+0000","last_clean_scrub_stamp":"2024-09-10T18:45:30.090965+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:49.288365+0000","scrub_duration":0.0021495899999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"58'2","reported_seq":545,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405939+0000","last_change":"2024-09-10T18:44:20.936428+0000","last_active":"2024-09-10T18:45:25.405939+0000","last_peered":"2024-09-10T18:45:25.405939+0000","last_clean":"2024-09-10T18:45:25.405939+0000","last_became_active":"2024-09-10T18:44:06.161378+0000","last_became_peered":"2024-09-10T18:44:06.161378+0000","last_unstale":"2024-09-10T18:45:25.405939+0000","last_undegraded":"2024-09-10T18:45:25.405939+0000","last_fullsized":"2024-09-10T18:45:25.405939+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-09-10T18:44:20.936373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:30.506537+0000","last_clean_scrub_stamp":"2024-09-10T18:44:20.936373+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:40.266629+0000","scrub_duration":0.001433365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"233'11","reported_seq":809,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.044369+0000","last_change":"2024-09-10T18:44:28.022333+0000","last_active":"2024-09-10T18:45:45.044369+0000","last_peered":"2024-09-10T18:45:45.044369+0000","last_clean":"2024-09-10T18:45:45.044369+0000","last_became_active":"2024-09-10T18:26:27.651073+0000","last_became_peered":"2024-09-10T18:26:27.651073+0000","last_unstale":"2024-09-10T18:45:45.044369+0000","last_undegraded":"2024-09-10T18:45:45.044369+0000","last_fullsized":"2024-09-10T18:45:45.044369+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-09-10T18:44:28.022263+0000","last_deep_scrub":"233'11","last_deep_scrub_stamp":"2024-09-10T18:31:03.524066+0000","last_clean_scrub_stamp":"2024-09-10T18:44:28.022263+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016602500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":157,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178233+0000","last_change":"2024-09-10T18:45:15.139557+0000","last_active":"2024-09-10T18:45:25.178233+0000","last_peered":"2024-09-10T18:45:25.178233+0000","last_clean":"2024-09-10T18:45:25.178233+0000","last_became_active":"2024-09-10T18:44:06.172086+0000","last_became_peered":"2024-09-10T18:44:06.172086+0000","last_unstale":"2024-09-10T18:45:25.178233+0000","last_undegraded":"2024-09-10T18:45:25.178233+0000","last_fullsized":"2024-09-10T18:45:25.178233+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:15.139489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:45:15.139489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:22.413742+0000","scrub_duration":0.001470343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'81","reported_seq":137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178406+0000","last_change":"2024-09-10T18:44:48.054541+0000","last_active":"2024-09-10T18:45:25.178406+0000","last_peered":"2024-09-10T18:45:25.178406+0000","last_clean":"2024-09-10T18:45:25.178406+0000","last_became_active":"2024-09-10T18:39:00.910728+0000","last_became_peered":"2024-09-10T18:39:00.910728+0000","last_unstale":"2024-09-10T18:45:25.178406+0000","last_undegraded":"2024-09-10T18:45:25.178406+0000","last_fullsized":"2024-09-10T18:45:25.178406+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:48.054466+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:48.054466+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:00.133566+0000","scrub_duration":0.0015792390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":519,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.381734+0000","last_change":"2024-09-10T18:45:03.801355+0000","last_active":"2024-09-10T18:45:25.381734+0000","last_peered":"2024-09-10T18:45:25.381734+0000","last_clean":"2024-09-10T18:45:25.381734+0000","last_became_active":"2024-09-10T18:44:06.170932+0000","last_became_peered":"2024-09-10T18:44:06.170932+0000","last_unstale":"2024-09-10T18:45:25.381734+0000","last_undegraded":"2024-09-10T18:45:25.381734+0000","last_fullsized":"2024-09-10T18:45:25.381734+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:03.801290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:23:08.899949+0000","last_clean_scrub_stamp":"2024-09-10T18:45:03.801290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:27.883971+0000","scrub_duration":0.001687937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"400'18","reported_seq":521,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660835+0000","last_change":"2024-09-10T18:44:46.467864+0000","last_active":"2024-09-10T18:45:36.660835+0000","last_peered":"2024-09-10T18:45:36.660835+0000","last_clean":"2024-09-10T18:45:36.660835+0000","last_became_active":"2024-09-10T18:21:34.045739+0000","last_became_peered":"2024-09-10T18:21:34.045739+0000","last_unstale":"2024-09-10T18:45:36.660835+0000","last_undegraded":"2024-09-10T18:45:36.660835+0000","last_fullsized":"2024-09-10T18:45:36.660835+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'18","last_scrub_stamp":"2024-09-10T18:44:46.467804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:44:46.467804+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:11.798982+0000","scrub_duration":0.002535575,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178857+0000","last_change":"2024-09-10T18:45:00.067239+0000","last_active":"2024-09-10T18:45:25.178857+0000","last_peered":"2024-09-10T18:45:25.178857+0000","last_clean":"2024-09-10T18:45:25.178857+0000","last_became_active":"2024-09-10T18:39:00.916952+0000","last_became_peered":"2024-09-10T18:39:00.916952+0000","last_unstale":"2024-09-10T18:45:25.178857+0000","last_undegraded":"2024-09-10T18:45:25.178857+0000","last_fullsized":"2024-09-10T18:45:25.178857+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:00.067157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:40:51.751267+0000","last_clean_scrub_stamp":"2024-09-10T18:45:00.067157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:02.011190+0000","scrub_duration":0.0016814779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":504,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405437+0000","last_change":"2024-09-10T18:45:09.307456+0000","last_active":"2024-09-10T18:45:25.405437+0000","last_peered":"2024-09-10T18:45:25.405437+0000","last_clean":"2024-09-10T18:45:25.405437+0000","last_became_active":"2024-09-10T18:26:27.643199+0000","last_became_peered":"2024-09-10T18:26:27.643199+0000","last_unstale":"2024-09-10T18:45:25.405437+0000","last_undegraded":"2024-09-10T18:45:25.405437+0000","last_fullsized":"2024-09-10T18:45:25.405437+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:09.307383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:45:09.307383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:29.778842+0000","scrub_duration":0.0014754760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'81","reported_seq":450,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.108628+0000","last_change":"2024-09-10T18:45:25.108578+0000","last_active":"2024-09-10T18:45:25.108628+0000","last_peered":"2024-09-10T18:45:25.108628+0000","last_clean":"2024-09-10T18:45:25.108628+0000","last_became_active":"2024-09-10T18:34:02.815880+0000","last_became_peered":"2024-09-10T18:34:02.815880+0000","last_unstale":"2024-09-10T18:45:25.108628+0000","last_undegraded":"2024-09-10T18:45:25.108628+0000","last_fullsized":"2024-09-10T18:45:25.108628+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:25.108523+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:45:25.108523+0000","last_clean_scrub_stamp":"2024-09-10T18:45:25.108523+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:45.294250+0000","scrub_duration":0.0022515410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"400'25","reported_seq":65,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:43.386224+0000","last_change":"2024-09-10T18:45:43.386199+0000","last_active":"2024-09-10T18:45:43.386224+0000","last_peered":"2024-09-10T18:45:43.386224+0000","last_clean":"2024-09-10T18:45:43.386224+0000","last_became_active":"2024-09-10T18:44:06.163688+0000","last_became_peered":"2024-09-10T18:44:06.163688+0000","last_unstale":"2024-09-10T18:45:43.386224+0000","last_undegraded":"2024-09-10T18:45:43.386224+0000","last_fullsized":"2024-09-10T18:45:43.386224+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'25","last_scrub_stamp":"2024-09-10T18:45:43.386142+0000","last_deep_scrub":"400'25","last_deep_scrub_stamp":"2024-09-10T18:45:43.386142+0000","last_clean_scrub_stamp":"2024-09-10T18:45:43.386142+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:00.538488+0000","scrub_duration":0.0017439249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":520,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.365843+0000","last_change":"2024-09-10T18:45:36.365755+0000","last_active":"2024-09-10T18:45:36.365843+0000","last_peered":"2024-09-10T18:45:36.365843+0000","last_clean":"2024-09-10T18:45:36.365843+0000","last_became_active":"2024-09-10T18:26:27.650654+0000","last_became_peered":"2024-09-10T18:26:27.650654+0000","last_unstale":"2024-09-10T18:45:36.365843+0000","last_undegraded":"2024-09-10T18:45:36.365843+0000","last_fullsized":"2024-09-10T18:45:36.365843+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:36.365691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:56.100012+0000","last_clean_scrub_stamp":"2024-09-10T18:45:36.365691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:43.155806+0000","scrub_duration":0.001656331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":492,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660749+0000","last_change":"2024-09-10T18:44:05.597703+0000","last_active":"2024-09-10T18:45:36.660749+0000","last_peered":"2024-09-10T18:45:36.660749+0000","last_clean":"2024-09-10T18:45:36.660749+0000","last_became_active":"2024-09-10T18:21:35.623690+0000","last_became_peered":"2024-09-10T18:21:35.623690+0000","last_unstale":"2024-09-10T18:45:36.660749+0000","last_undegraded":"2024-09-10T18:45:36.660749+0000","last_fullsized":"2024-09-10T18:45:36.660749+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:05.597644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:44:05.597644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017041560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":546,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.276086+0000","last_change":"2024-09-10T18:45:04.887089+0000","last_active":"2024-09-10T18:45:25.276086+0000","last_peered":"2024-09-10T18:45:25.276086+0000","last_clean":"2024-09-10T18:45:25.276086+0000","last_became_active":"2024-09-10T18:21:37.637292+0000","last_became_peered":"2024-09-10T18:21:37.637292+0000","last_unstale":"2024-09-10T18:45:25.276086+0000","last_undegraded":"2024-09-10T18:45:25.276086+0000","last_fullsized":"2024-09-10T18:45:25.276086+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:04.887033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:43:46.250339+0000","last_clean_scrub_stamp":"2024-09-10T18:45:04.887033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:23.332796+0000","scrub_duration":0.0029547129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'81","reported_seq":137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178009+0000","last_change":"2024-09-10T18:44:41.105004+0000","last_active":"2024-09-10T18:45:25.178009+0000","last_peered":"2024-09-10T18:45:25.178009+0000","last_clean":"2024-09-10T18:45:25.178009+0000","last_became_active":"2024-09-10T18:39:00.902585+0000","last_became_peered":"2024-09-10T18:39:00.902585+0000","last_unstale":"2024-09-10T18:45:25.178009+0000","last_undegraded":"2024-09-10T18:45:25.178009+0000","last_fullsized":"2024-09-10T18:45:25.178009+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:41.104948+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:39:55.999377+0000","last_clean_scrub_stamp":"2024-09-10T18:44:41.104948+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:52.886626+0000","scrub_duration":0.00129419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"400'25","reported_seq":156,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178735+0000","last_change":"2024-09-10T18:45:05.099475+0000","last_active":"2024-09-10T18:45:25.178735+0000","last_peered":"2024-09-10T18:45:25.178735+0000","last_clean":"2024-09-10T18:45:25.178735+0000","last_became_active":"2024-09-10T18:39:00.899968+0000","last_became_peered":"2024-09-10T18:39:00.899968+0000","last_unstale":"2024-09-10T18:45:25.178735+0000","last_undegraded":"2024-09-10T18:45:25.178735+0000","last_fullsized":"2024-09-10T18:45:25.178735+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'25","last_scrub_stamp":"2024-09-10T18:45:05.099413+0000","last_deep_scrub":"165'13","last_deep_scrub_stamp":"2024-09-10T18:27:17.468619+0000","last_clean_scrub_stamp":"2024-09-10T18:45:05.099413+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:07.431929+0000","scrub_duration":0.0020260790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.657776+0000","last_change":"2024-09-10T18:44:52.329429+0000","last_active":"2024-09-10T18:45:25.657776+0000","last_peered":"2024-09-10T18:45:25.657776+0000","last_clean":"2024-09-10T18:45:25.657776+0000","last_became_active":"2024-09-10T18:44:06.159130+0000","last_became_peered":"2024-09-10T18:44:06.159130+0000","last_unstale":"2024-09-10T18:45:25.657776+0000","last_undegraded":"2024-09-10T18:45:25.657776+0000","last_fullsized":"2024-09-10T18:45:25.657776+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:52.329371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:54.375851+0000","last_clean_scrub_stamp":"2024-09-10T18:44:52.329371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:20.234171+0000","scrub_duration":0.001513637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"342'9","reported_seq":804,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-09-10T18:45:55.048999+0000","last_change":"2024-09-10T18:45:24.494003+0000","last_active":"2024-09-10T18:45:55.048999+0000","last_peered":"2024-09-10T18:45:55.048999+0000","last_clean":"2024-09-10T18:45:55.048999+0000","last_became_active":"2024-09-10T18:21:35.626805+0000","last_became_peered":"2024-09-10T18:21:35.626805+0000","last_unstale":"2024-09-10T18:45:55.048999+0000","last_undegraded":"2024-09-10T18:45:55.048999+0000","last_fullsized":"2024-09-10T18:45:55.048999+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'9","last_scrub_stamp":"2024-09-10T18:45:24.493865+0000","last_deep_scrub":"342'9","last_deep_scrub_stamp":"2024-09-10T18:37:56.641708+0000","last_clean_scrub_stamp":"2024-09-10T18:45:24.493865+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:28.176007+0000","scrub_duration":0.0022316329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":53,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405832+0000","last_change":"2024-09-10T18:44:58.132531+0000","last_active":"2024-09-10T18:45:25.405832+0000","last_peered":"2024-09-10T18:45:25.405832+0000","last_clean":"2024-09-10T18:45:25.405832+0000","last_became_active":"2024-09-10T18:43:57.678270+0000","last_became_peered":"2024-09-10T18:43:57.678270+0000","last_unstale":"2024-09-10T18:45:25.405832+0000","last_undegraded":"2024-09-10T18:45:25.405832+0000","last_fullsized":"2024-09-10T18:45:25.405832+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:58.132467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:44:58.132467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:26.266039+0000","scrub_duration":0.0017044510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"76'81","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.657803+0000","last_change":"2024-09-10T18:44:42.473565+0000","last_active":"2024-09-10T18:45:25.657803+0000","last_peered":"2024-09-10T18:45:25.657803+0000","last_clean":"2024-09-10T18:45:25.657803+0000","last_became_active":"2024-09-10T18:44:06.160532+0000","last_became_peered":"2024-09-10T18:44:06.160532+0000","last_unstale":"2024-09-10T18:45:25.657803+0000","last_undegraded":"2024-09-10T18:45:25.657803+0000","last_fullsized":"2024-09-10T18:45:25.657803+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:42.473478+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:36:08.710566+0000","last_clean_scrub_stamp":"2024-09-10T18:44:42.473478+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:56.284220+0000","scrub_duration":0.0026086019999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"400'30","reported_seq":549,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660641+0000","last_change":"2024-09-10T18:44:53.381662+0000","last_active":"2024-09-10T18:45:36.660641+0000","last_peered":"2024-09-10T18:45:36.660641+0000","last_clean":"2024-09-10T18:45:36.660641+0000","last_became_active":"2024-09-10T18:21:34.046153+0000","last_became_peered":"2024-09-10T18:21:34.046153+0000","last_unstale":"2024-09-10T18:45:36.660641+0000","last_undegraded":"2024-09-10T18:45:36.660641+0000","last_fullsized":"2024-09-10T18:45:36.660641+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'30","last_scrub_stamp":"2024-09-10T18:44:53.381596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:44:53.381596+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:21.940755+0000","scrub_duration":0.0040286710000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":513,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405121+0000","last_change":"2024-09-10T18:45:13.414109+0000","last_active":"2024-09-10T18:45:25.405121+0000","last_peered":"2024-09-10T18:45:25.405121+0000","last_clean":"2024-09-10T18:45:25.405121+0000","last_became_active":"2024-09-10T18:21:31.531052+0000","last_became_peered":"2024-09-10T18:21:31.531052+0000","last_unstale":"2024-09-10T18:45:25.405121+0000","last_undegraded":"2024-09-10T18:45:25.405121+0000","last_fullsized":"2024-09-10T18:45:25.405121+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:13.414043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:35:17.826448+0000","last_clean_scrub_stamp":"2024-09-10T18:45:13.414043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:18.715631+0000","scrub_duration":0.0017876610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"430'12","reported_seq":253,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.044362+0000","last_change":"2024-09-10T18:45:01.049536+0000","last_active":"2024-09-10T18:45:45.044362+0000","last_peered":"2024-09-10T18:45:45.044362+0000","last_clean":"2024-09-10T18:45:45.044362+0000","last_became_active":"2024-09-10T18:44:06.170582+0000","last_became_peered":"2024-09-10T18:44:06.170582+0000","last_unstale":"2024-09-10T18:45:45.044362+0000","last_undegraded":"2024-09-10T18:45:45.044362+0000","last_fullsized":"2024-09-10T18:45:45.044362+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'12","last_scrub_stamp":"2024-09-10T18:45:01.049470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:45:01.049470+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:14.721784+0000","scrub_duration":0.002164067,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.658375+0000","last_change":"2024-09-10T18:44:53.287994+0000","last_active":"2024-09-10T18:45:25.658375+0000","last_peered":"2024-09-10T18:45:25.658375+0000","last_clean":"2024-09-10T18:45:25.658375+0000","last_became_active":"2024-09-10T18:44:06.160884+0000","last_became_peered":"2024-09-10T18:44:06.160884+0000","last_unstale":"2024-09-10T18:45:25.658375+0000","last_undegraded":"2024-09-10T18:45:25.658375+0000","last_fullsized":"2024-09-10T18:45:25.658375+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:53.287932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:44:53.287932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:59.964334+0000","scrub_duration":0.0037275199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'81","reported_seq":245,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.108149+0000","last_change":"2024-09-10T18:44:39.065232+0000","last_active":"2024-09-10T18:45:25.108149+0000","last_peered":"2024-09-10T18:45:25.108149+0000","last_clean":"2024-09-10T18:45:25.108149+0000","last_became_active":"2024-09-10T18:44:06.171808+0000","last_became_peered":"2024-09-10T18:44:06.171808+0000","last_unstale":"2024-09-10T18:45:25.108149+0000","last_undegraded":"2024-09-10T18:45:25.108149+0000","last_fullsized":"2024-09-10T18:45:25.108149+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:39.065169+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:44:39.065169+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:07.369806+0000","scrub_duration":0.001359952,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"400'27","reported_seq":583,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.368021+0000","last_change":"2024-09-10T18:44:31.637993+0000","last_active":"2024-09-10T18:45:36.368021+0000","last_peered":"2024-09-10T18:45:36.368021+0000","last_clean":"2024-09-10T18:45:36.368021+0000","last_became_active":"2024-09-10T18:41:11.933013+0000","last_became_peered":"2024-09-10T18:41:11.933013+0000","last_unstale":"2024-09-10T18:45:36.368021+0000","last_undegraded":"2024-09-10T18:45:36.368021+0000","last_fullsized":"2024-09-10T18:45:36.368021+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'27","last_scrub_stamp":"2024-09-10T18:44:31.637922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:44:31.637922+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038029119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":523,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:44.721310+0000","last_change":"2024-09-10T18:45:44.721249+0000","last_active":"2024-09-10T18:45:44.721310+0000","last_peered":"2024-09-10T18:45:44.721310+0000","last_clean":"2024-09-10T18:45:44.721310+0000","last_became_active":"2024-09-10T18:26:27.651814+0000","last_became_peered":"2024-09-10T18:26:27.651814+0000","last_unstale":"2024-09-10T18:45:44.721310+0000","last_undegraded":"2024-09-10T18:45:44.721310+0000","last_fullsized":"2024-09-10T18:45:44.721310+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:44.721190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:57.246282+0000","last_clean_scrub_stamp":"2024-09-10T18:45:44.721190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:13.673819+0000","scrub_duration":0.001445244,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":543,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.107887+0000","last_change":"2024-09-10T18:45:07.310043+0000","last_active":"2024-09-10T18:45:25.107887+0000","last_peered":"2024-09-10T18:45:25.107887+0000","last_clean":"2024-09-10T18:45:25.107887+0000","last_became_active":"2024-09-10T18:39:00.897018+0000","last_became_peered":"2024-09-10T18:39:00.897018+0000","last_unstale":"2024-09-10T18:45:25.107887+0000","last_undegraded":"2024-09-10T18:45:25.107887+0000","last_fullsized":"2024-09-10T18:45:25.107887+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:07.309981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:31:56.674462+0000","last_clean_scrub_stamp":"2024-09-10T18:45:07.309981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:16.957321+0000","scrub_duration":0.0016989010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":530,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.332948+0000","last_change":"2024-09-10T18:45:45.332886+0000","last_active":"2024-09-10T18:45:45.332948+0000","last_peered":"2024-09-10T18:45:45.332948+0000","last_clean":"2024-09-10T18:45:45.332948+0000","last_became_active":"2024-09-10T18:44:06.162639+0000","last_became_peered":"2024-09-10T18:44:06.162639+0000","last_unstale":"2024-09-10T18:45:45.332948+0000","last_undegraded":"2024-09-10T18:45:45.332948+0000","last_fullsized":"2024-09-10T18:45:45.332948+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:45.332820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:28:57.310730+0000","last_clean_scrub_stamp":"2024-09-10T18:45:45.332820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:57.513815+0000","scrub_duration":0.0012939239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'81","reported_seq":157,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.178788+0000","last_change":"2024-09-10T18:44:59.072853+0000","last_active":"2024-09-10T18:45:25.178788+0000","last_peered":"2024-09-10T18:45:25.178788+0000","last_clean":"2024-09-10T18:45:25.178788+0000","last_became_active":"2024-09-10T18:44:06.173792+0000","last_became_peered":"2024-09-10T18:44:06.173792+0000","last_unstale":"2024-09-10T18:45:25.178788+0000","last_undegraded":"2024-09-10T18:45:25.178788+0000","last_fullsized":"2024-09-10T18:45:25.178788+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:59.072782+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:40:52.817945+0000","last_clean_scrub_stamp":"2024-09-10T18:44:59.072782+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:04.923069+0000","scrub_duration":0.001624995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"400'21","reported_seq":563,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.108089+0000","last_change":"2024-09-10T18:45:20.200436+0000","last_active":"2024-09-10T18:45:25.108089+0000","last_peered":"2024-09-10T18:45:25.108089+0000","last_clean":"2024-09-10T18:45:25.108089+0000","last_became_active":"2024-09-10T18:26:27.637983+0000","last_became_peered":"2024-09-10T18:26:27.637983+0000","last_unstale":"2024-09-10T18:45:25.108089+0000","last_undegraded":"2024-09-10T18:45:25.108089+0000","last_fullsized":"2024-09-10T18:45:25.108089+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'21","last_scrub_stamp":"2024-09-10T18:45:20.200370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:45:20.200370+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:20.664309+0000","scrub_duration":0.002329121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":516,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.406122+0000","last_change":"2024-09-10T18:45:25.406050+0000","last_active":"2024-09-10T18:45:25.406122+0000","last_peered":"2024-09-10T18:45:25.406122+0000","last_clean":"2024-09-10T18:45:25.406122+0000","last_became_active":"2024-09-10T18:21:31.536307+0000","last_became_peered":"2024-09-10T18:21:31.536307+0000","last_unstale":"2024-09-10T18:45:25.406122+0000","last_undegraded":"2024-09-10T18:45:25.406122+0000","last_fullsized":"2024-09-10T18:45:25.406122+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:25.405978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:45:25.405978+0000","last_clean_scrub_stamp":"2024-09-10T18:45:25.405978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:37.463319+0000","scrub_duration":0.002680263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":505,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660406+0000","last_change":"2024-09-10T18:44:10.599072+0000","last_active":"2024-09-10T18:45:36.660406+0000","last_peered":"2024-09-10T18:45:36.660406+0000","last_clean":"2024-09-10T18:45:36.660406+0000","last_became_active":"2024-09-10T18:44:06.160812+0000","last_became_peered":"2024-09-10T18:44:06.160812+0000","last_unstale":"2024-09-10T18:45:36.660406+0000","last_undegraded":"2024-09-10T18:45:36.660406+0000","last_fullsized":"2024-09-10T18:45:36.660406+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:10.599007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:26:10.052296+0000","last_clean_scrub_stamp":"2024-09-10T18:44:10.599007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001387208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":55,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:45.474986+0000","last_change":"2024-09-10T18:45:45.474936+0000","last_active":"2024-09-10T18:45:45.474986+0000","last_peered":"2024-09-10T18:45:45.474986+0000","last_clean":"2024-09-10T18:45:45.474986+0000","last_became_active":"2024-09-10T18:44:06.165448+0000","last_became_peered":"2024-09-10T18:44:06.165448+0000","last_unstale":"2024-09-10T18:45:45.474986+0000","last_undegraded":"2024-09-10T18:45:45.474986+0000","last_fullsized":"2024-09-10T18:45:45.474986+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:45.474880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:44:32.354637+0000","last_clean_scrub_stamp":"2024-09-10T18:45:45.474880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:46.680776+0000","scrub_duration":0.0013512000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'81","reported_seq":252,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.276037+0000","last_change":"2024-09-10T18:45:08.825395+0000","last_active":"2024-09-10T18:45:25.276037+0000","last_peered":"2024-09-10T18:45:25.276037+0000","last_clean":"2024-09-10T18:45:25.276037+0000","last_became_active":"2024-09-10T18:34:02.802580+0000","last_became_peered":"2024-09-10T18:34:02.802580+0000","last_unstale":"2024-09-10T18:45:25.276037+0000","last_undegraded":"2024-09-10T18:45:25.276037+0000","last_fullsized":"2024-09-10T18:45:25.276037+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:08.825329+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:22:17.024553+0000","last_clean_scrub_stamp":"2024-09-10T18:45:08.825329+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:26.403111+0000","scrub_duration":0.001539328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"400'27","reported_seq":554,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:44.335843+0000","last_change":"2024-09-10T18:45:44.335762+0000","last_active":"2024-09-10T18:45:44.335843+0000","last_peered":"2024-09-10T18:45:44.335843+0000","last_clean":"2024-09-10T18:45:44.335843+0000","last_became_active":"2024-09-10T18:26:27.637320+0000","last_became_peered":"2024-09-10T18:26:27.637320+0000","last_unstale":"2024-09-10T18:45:44.335843+0000","last_undegraded":"2024-09-10T18:45:44.335843+0000","last_fullsized":"2024-09-10T18:45:44.335843+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'27","last_scrub_stamp":"2024-09-10T18:45:44.335686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:45:44.335686+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:53.349846+0000","scrub_duration":0.002076423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":147,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:40.008833+0000","last_change":"2024-09-10T18:45:40.008801+0000","last_active":"2024-09-10T18:45:40.008833+0000","last_peered":"2024-09-10T18:45:40.008833+0000","last_clean":"2024-09-10T18:45:40.008833+0000","last_became_active":"2024-09-10T18:39:00.916983+0000","last_became_peered":"2024-09-10T18:39:00.916983+0000","last_unstale":"2024-09-10T18:45:40.008833+0000","last_undegraded":"2024-09-10T18:45:40.008833+0000","last_fullsized":"2024-09-10T18:45:40.008833+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:40.008738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:22:40.685879+0000","last_clean_scrub_stamp":"2024-09-10T18:45:40.008738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:59.962523+0000","scrub_duration":0.0015682090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":509,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.404466+0000","last_change":"2024-09-10T18:45:24.381281+0000","last_active":"2024-09-10T18:45:25.404466+0000","last_peered":"2024-09-10T18:45:25.404466+0000","last_clean":"2024-09-10T18:45:25.404466+0000","last_became_active":"2024-09-10T18:26:27.650828+0000","last_became_peered":"2024-09-10T18:26:27.650828+0000","last_unstale":"2024-09-10T18:45:25.404466+0000","last_undegraded":"2024-09-10T18:45:25.404466+0000","last_fullsized":"2024-09-10T18:45:25.404466+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:24.381213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:45:24.381213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:42.220148+0000","scrub_duration":0.0016045880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":536,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660402+0000","last_change":"2024-09-10T18:45:04.469996+0000","last_active":"2024-09-10T18:45:36.660402+0000","last_peered":"2024-09-10T18:45:36.660402+0000","last_clean":"2024-09-10T18:45:36.660402+0000","last_became_active":"2024-09-10T18:44:06.161239+0000","last_became_peered":"2024-09-10T18:44:06.161239+0000","last_unstale":"2024-09-10T18:45:36.660402+0000","last_undegraded":"2024-09-10T18:45:36.660402+0000","last_fullsized":"2024-09-10T18:45:36.660402+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:04.469907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:48.554301+0000","last_clean_scrub_stamp":"2024-09-10T18:45:04.469907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:08.225190+0000","scrub_duration":0.0021947099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'81","reported_seq":234,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.405339+0000","last_change":"2024-09-10T18:44:37.962063+0000","last_active":"2024-09-10T18:45:25.405339+0000","last_peered":"2024-09-10T18:45:25.405339+0000","last_clean":"2024-09-10T18:45:25.405339+0000","last_became_active":"2024-09-10T18:39:00.889998+0000","last_became_peered":"2024-09-10T18:39:00.889998+0000","last_unstale":"2024-09-10T18:45:25.405339+0000","last_undegraded":"2024-09-10T18:45:25.405339+0000","last_fullsized":"2024-09-10T18:45:25.405339+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:44:37.961994+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:44:37.961994+0000","last_clean_scrub_stamp":"2024-09-10T18:44:37.961994+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:45:42.727197+0000","scrub_duration":0.0015827720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"400'12","reported_seq":519,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.659820+0000","last_change":"2024-09-10T18:44:36.504023+0000","last_active":"2024-09-10T18:45:36.659820+0000","last_peered":"2024-09-10T18:45:36.659820+0000","last_clean":"2024-09-10T18:45:36.659820+0000","last_became_active":"2024-09-10T18:26:27.634110+0000","last_became_peered":"2024-09-10T18:26:27.634110+0000","last_unstale":"2024-09-10T18:45:36.659820+0000","last_undegraded":"2024-09-10T18:45:36.659820+0000","last_fullsized":"2024-09-10T18:45:36.659820+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'12","last_scrub_stamp":"2024-09-10T18:44:36.503958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:32.587099+0000","last_clean_scrub_stamp":"2024-09-10T18:44:36.503958+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019983959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":55,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:38.332944+0000","last_change":"2024-09-10T18:45:38.332918+0000","last_active":"2024-09-10T18:45:38.332944+0000","last_peered":"2024-09-10T18:45:38.332944+0000","last_clean":"2024-09-10T18:45:38.332944+0000","last_became_active":"2024-09-10T18:44:06.161983+0000","last_became_peered":"2024-09-10T18:44:06.161983+0000","last_unstale":"2024-09-10T18:45:38.332944+0000","last_undegraded":"2024-09-10T18:45:38.332944+0000","last_fullsized":"2024-09-10T18:45:38.332944+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:38.332851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:30:05.204467+0000","last_clean_scrub_stamp":"2024-09-10T18:45:38.332851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:47:06.659426+0000","scrub_duration":0.0015862020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":483,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:39.790686+0000","last_change":"2024-09-10T18:45:39.790652+0000","last_active":"2024-09-10T18:45:39.790686+0000","last_peered":"2024-09-10T18:45:39.790686+0000","last_clean":"2024-09-10T18:45:39.790686+0000","last_became_active":"2024-09-10T18:26:27.651642+0000","last_became_peered":"2024-09-10T18:26:27.651642+0000","last_unstale":"2024-09-10T18:45:39.790686+0000","last_undegraded":"2024-09-10T18:45:39.790686+0000","last_fullsized":"2024-09-10T18:45:39.790686+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:39.790582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:34.598351+0000","last_clean_scrub_stamp":"2024-09-10T18:45:39.790582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:54.285917+0000","scrub_duration":0.0015055019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":507,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:42.374048+0000","last_change":"2024-09-10T18:45:42.374018+0000","last_active":"2024-09-10T18:45:42.374048+0000","last_peered":"2024-09-10T18:45:42.374048+0000","last_clean":"2024-09-10T18:45:42.374048+0000","last_became_active":"2024-09-10T18:21:37.642811+0000","last_became_peered":"2024-09-10T18:21:37.642811+0000","last_unstale":"2024-09-10T18:45:42.374048+0000","last_undegraded":"2024-09-10T18:45:42.374048+0000","last_fullsized":"2024-09-10T18:45:42.374048+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:42.373965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:21:36.612407+0000","last_clean_scrub_stamp":"2024-09-10T18:45:42.373965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:44.649548+0000","scrub_duration":0.001595735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'81","reported_seq":234,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:43.717584+0000","last_change":"2024-09-10T18:45:43.717524+0000","last_active":"2024-09-10T18:45:43.717584+0000","last_peered":"2024-09-10T18:45:43.717584+0000","last_clean":"2024-09-10T18:45:43.717584+0000","last_became_active":"2024-09-10T18:34:34.867380+0000","last_became_peered":"2024-09-10T18:34:34.867380+0000","last_unstale":"2024-09-10T18:45:43.717584+0000","last_undegraded":"2024-09-10T18:45:43.717584+0000","last_fullsized":"2024-09-10T18:45:43.717584+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":287,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'81","last_scrub_stamp":"2024-09-10T18:45:43.717432+0000","last_deep_scrub":"76'81","last_deep_scrub_stamp":"2024-09-10T18:45:43.717432+0000","last_clean_scrub_stamp":"2024-09-10T18:45:43.717432+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:51.521155+0000","scrub_duration":0.001472758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":538,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.107207+0000","last_change":"2024-09-10T18:45:09.237487+0000","last_active":"2024-09-10T18:45:25.107207+0000","last_peered":"2024-09-10T18:45:25.107207+0000","last_clean":"2024-09-10T18:45:25.107207+0000","last_became_active":"2024-09-10T18:39:00.909906+0000","last_became_peered":"2024-09-10T18:39:00.909906+0000","last_unstale":"2024-09-10T18:45:25.107207+0000","last_undegraded":"2024-09-10T18:45:25.107207+0000","last_fullsized":"2024-09-10T18:45:25.107207+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:09.237426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:45:09.237426+0000","last_clean_scrub_stamp":"2024-09-10T18:45:09.237426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:11.341214+0000","scrub_duration":0.0016153669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":516,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-10T18:45:36.660026+0000","last_change":"2024-09-10T18:45:13.493539+0000","last_active":"2024-09-10T18:45:36.660026+0000","last_peered":"2024-09-10T18:45:36.660026+0000","last_clean":"2024-09-10T18:45:36.660026+0000","last_became_active":"2024-09-10T18:44:06.161345+0000","last_became_peered":"2024-09-10T18:44:06.161345+0000","last_unstale":"2024-09-10T18:45:36.660026+0000","last_undegraded":"2024-09-10T18:45:36.660026+0000","last_fullsized":"2024-09-10T18:45:36.660026+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:45:13.493469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:42:46.565877+0000","last_clean_scrub_stamp":"2024-09-10T18:45:13.493469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:26.435784+0000","scrub_duration":0.0013431490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"400'32","reported_seq":615,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.382437+0000","last_change":"2024-09-10T18:45:24.857886+0000","last_active":"2024-09-10T18:45:25.382437+0000","last_peered":"2024-09-10T18:45:25.382437+0000","last_clean":"2024-09-10T18:45:25.382437+0000","last_became_active":"2024-09-10T18:39:00.921476+0000","last_became_peered":"2024-09-10T18:39:00.921476+0000","last_unstale":"2024-09-10T18:45:25.382437+0000","last_undegraded":"2024-09-10T18:45:25.382437+0000","last_fullsized":"2024-09-10T18:45:25.382437+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'32","last_scrub_stamp":"2024-09-10T18:45:24.857809+0000","last_deep_scrub":"369'24","last_deep_scrub_stamp":"2024-09-10T18:39:50.690827+0000","last_clean_scrub_stamp":"2024-09-10T18:45:24.857809+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:25.278534+0000","scrub_duration":0.0029675330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":45,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T18:45:25.658213+0000","last_change":"2024-09-10T18:44:48.447412+0000","last_active":"2024-09-10T18:45:25.658213+0000","last_peered":"2024-09-10T18:45:25.658213+0000","last_clean":"2024-09-10T18:45:25.658213+0000","last_became_active":"2024-09-10T18:44:06.169267+0000","last_became_peered":"2024-09-10T18:44:06.169267+0000","last_unstale":"2024-09-10T18:45:25.658213+0000","last_undegraded":"2024-09-10T18:45:25.658213+0000","last_fullsized":"2024-09-10T18:45:25.658213+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:44:48.447347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:29:05.092434+0000","last_clean_scrub_stamp":"2024-09-10T18:44:48.447347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:46:16.752300+0000","scrub_duration":0.0019764130000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1545,"num_read_kb":1302,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":5886,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},"log_size":1091,"ondisk_log_size":1091,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1058,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1446,"num_read_kb":1441,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":797,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2511,"ondisk_log_size":2511,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234280,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":891440,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":890430,"kb_avail":92853712,"statfs":{"total":95995035648,"available":95082201088,"internally_reserved":0,"allocated":1024000,"data_stored":635883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":911800664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:45:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.63,"15min":1.4970000000000001},"min":{"1min":0.311,"5min":0.254,"15min":0.21099999999999999},"max":{"1min":0.70699999999999996,"5min":3.8519999999999999,"15min":75.506},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.92700000000000005,"15min":1.506},"min":{"1min":0.34699999999999998,"5min":0.26500000000000001,"15min":0.223},"max":{"1min":0.72499999999999998,"5min":12.571999999999999,"15min":75.710999999999999},"last":0.495}]},{"osd":1,"last update":"Tue Sep 10 18:45:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":1.056,"15min":1.5189999999999999},"min":{"1min":0.436,"5min":0.23400000000000001,"15min":0.214},"max":{"1min":0.78900000000000003,"5min":12.461,"15min":98.305999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.137,"15min":1.5129999999999999},"min":{"1min":0.39000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.77300000000000002,"5min":9.4969999999999999,"15min":98.381},"last":0.55500000000000005}]},{"osd":2,"last update":"Tue Sep 10 18:45:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.94399999999999995,"15min":1.792},"min":{"1min":0.39900000000000002,"5min":0.26900000000000002,"15min":0.26800000000000002},"max":{"1min":0.75700000000000001,"5min":5.3840000000000003,"15min":136.648},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.85699999999999998,"15min":1.7889999999999999},"min":{"1min":0.38300000000000001,"5min":0.27800000000000002,"15min":0.24399999999999999},"max":{"1min":0.72799999999999998,"5min":5.7859999999999996,"15min":136.506},"last":0.625}]},{"osd":3,"last update":"Tue Sep 10 18:45:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":1.0720000000000001,"15min":1.278},"min":{"1min":0.316,"5min":0.251,"15min":0.251},"max":{"1min":0.69199999999999995,"5min":8.7400000000000002,"15min":38.225000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":1.036,"15min":1.2310000000000001},"min":{"1min":0.50700000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.77700000000000002,"5min":12.516,"15min":38.104999999999997},"last":0.57299999999999995}]},{"osd":4,"last update":"Tue Sep 10 18:45:06 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.85099999999999998,"15min":0.92800000000000005},"min":{"1min":0.28399999999999997,"5min":0.23899999999999999,"15min":0.224},"max":{"1min":0.66100000000000003,"5min":8.4770000000000003,"15min":23.346},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.80600000000000005,"15min":0.94399999999999995},"min":{"1min":0.31900000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.80400000000000005,"5min":5.4219999999999997,"15min":23.323},"last":0.58799999999999997}]},{"osd":5,"last update":"Tue Sep 10 18:45:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.746,"15min":0.93500000000000005},"min":{"1min":0.33500000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.71099999999999997,"5min":4.0179999999999998,"15min":16.756},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.89100000000000001,"15min":1.008},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.224},"max":{"1min":0.72399999999999998,"5min":8.4510000000000005,"15min":20.495999999999999},"last":0.64200000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.89000000000000001,"15min":0.98899999999999999},"min":{"1min":0.38400000000000001,"5min":0.314,"15min":0.28899999999999998},"max":{"1min":0.72199999999999998,"5min":4.6689999999999996,"15min":25.902999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":1.0600000000000001,"15min":1.004},"min":{"1min":0.38700000000000001,"5min":0.33600000000000002,"15min":0.30199999999999999},"max":{"1min":0.73099999999999998,"5min":9.9879999999999995,"15min":24.074000000000002},"last":0.61399999999999999}]}]},{"osd":6,"up_from":144,"seq":618475290859,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903964,"kb_used_data":1044,"kb_used_omap":7,"kb_used_meta":902904,"kb_avail":92841188,"statfs":{"total":95995035648,"available":95069376512,"internally_reserved":0,"allocated":1069056,"data_stored":701047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8105,"internal_metadata":924573783},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.94799999999999995,"15min":1.071},"min":{"1min":0.308,"5min":0.23300000000000001,"15min":0.192},"max":{"1min":0.85199999999999998,"5min":14.228999999999999,"15min":14.228999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.83599999999999997,"15min":1.0129999999999999},"min":{"1min":0.26200000000000001,"5min":0.246,"15min":0.217},"max":{"1min":0.81599999999999995,"5min":4.4960000000000004,"15min":18.093},"last":0.437}]},{"osd":1,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":1.5129999999999999,"15min":1.147},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76200000000000001,"5min":23.782,"15min":23.782},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.56100000000000005,"5min":1.1519999999999999,"15min":1.163},"min":{"1min":0.35899999999999999,"5min":0.26800000000000002,"15min":0.246},"max":{"1min":0.86599999999999999,"5min":23.687999999999999,"15min":23.687999999999999},"last":0.39700000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":1.1220000000000001,"15min":1.276},"min":{"1min":0.39100000000000001,"5min":0.27500000000000002,"15min":0.25600000000000001},"max":{"1min":0.79800000000000004,"5min":23.763999999999999,"15min":39.329999999999998},"last":0.505},{"interface":"front","average":{"1min":0.55500000000000005,"5min":1.5820000000000001,"15min":1.3140000000000001},"min":{"1min":0.36899999999999999,"5min":0.314,"15min":0.26200000000000001},"max":{"1min":0.77600000000000002,"5min":23.856000000000002,"15min":39.061999999999998},"last":0.47299999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":1.236,"15min":2.3570000000000002},"min":{"1min":0.34300000000000003,"5min":0.29499999999999998,"15min":0.28199999999999997},"max":{"1min":0.88100000000000001,"5min":29.670999999999999,"15min":162.61600000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":1.4239999999999999,"15min":2.4380000000000002},"min":{"1min":0.35199999999999998,"5min":0.29599999999999999,"15min":0.27100000000000002},"max":{"1min":0.73899999999999999,"5min":32.018000000000001,"15min":163.39599999999999},"last":0.52100000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.189},"max":{"1min":0.72899999999999998,"5min":3.9279999999999999,"15min":4.9050000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.748,"15min":0.72099999999999997},"min":{"1min":0.27600000000000002,"5min":0.22800000000000001,"15min":0.219},"max":{"1min":0.68200000000000005,"5min":3.093,"15min":11.531000000000001},"last":0.34799999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":1.9450000000000001,"15min":1.171},"min":{"1min":0.32100000000000001,"5min":0.26100000000000001,"15min":0.254},"max":{"1min":0.73499999999999999,"5min":129.69900000000001,"15min":129.69900000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":2.0459999999999998,"15min":1.224},"min":{"1min":0.33800000000000002,"5min":0.28000000000000003,"15min":0.20899999999999999},"max":{"1min":0.78500000000000003,"5min":130.04400000000001,"15min":130.04400000000001},"last":0.377}]},{"osd":7,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":1.0860000000000001,"15min":0.91100000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.23799999999999999},"max":{"1min":0.78200000000000003,"5min":10.103999999999999,"15min":11.449},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.98399999999999999,"15min":0.81899999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.27100000000000002},"max":{"1min":0.76200000000000001,"5min":9.8049999999999997,"15min":9.8049999999999997},"last":0.59699999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724851,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":887984,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":886974,"kb_avail":92857168,"statfs":{"total":95995035648,"available":95085740032,"internally_reserved":0,"allocated":1024000,"data_stored":636361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908261720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.93700000000000006,"15min":1.135},"min":{"1min":0.37,"5min":0.23400000000000001,"15min":0.224},"max":{"1min":0.82799999999999996,"5min":19.123999999999999,"15min":26.288},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.71199999999999997,"15min":0.95399999999999996},"min":{"1min":0.35999999999999999,"5min":0.27600000000000002,"15min":0.24099999999999999},"max":{"1min":0.73199999999999998,"5min":4.851,"15min":24.178000000000001},"last":0.39500000000000002}]},{"osd":1,"last update":"Tue Sep 10 18:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":1.4139999999999999,"15min":1.7390000000000001},"min":{"1min":0.26700000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.84199999999999997,"5min":22.623999999999999,"15min":111.773},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.143,"15min":1.653},"min":{"1min":0.46000000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.76500000000000001,"5min":22.635000000000002,"15min":111.73999999999999},"last":0.34899999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":1.2210000000000001,"15min":1.306},"min":{"1min":0.38,"5min":0.247,"15min":0.245},"max":{"1min":0.76200000000000001,"5min":23.396000000000001,"15min":54.530000000000001},"last":0.623},{"interface":"front","average":{"1min":0.57499999999999996,"5min":1.2070000000000001,"15min":1.2010000000000001},"min":{"1min":0.311,"5min":0.27800000000000002,"15min":0.247},"max":{"1min":0.81100000000000005,"5min":22.905999999999999,"15min":54.514000000000003},"last":0.52000000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.143,"15min":1.7829999999999999},"min":{"1min":0.45300000000000001,"5min":0.29699999999999999,"15min":0.27100000000000002},"max":{"1min":0.74199999999999999,"5min":23.007999999999999,"15min":106.205},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":1.244,"15min":1.823},"min":{"1min":0.34499999999999997,"5min":0.28999999999999998,"15min":0.28299999999999997},"max":{"1min":0.76500000000000001,"5min":22.931000000000001,"15min":105.878},"last":0.373}]},{"osd":4,"last update":"Tue Sep 10 18:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.70899999999999996,"15min":0.75},"min":{"1min":0.29599999999999999,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":0.625,"5min":5.226,"15min":5.8040000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.88800000000000001,"15min":0.84599999999999997},"min":{"1min":0.32900000000000001,"5min":0.27900000000000003,"15min":0.23999999999999999},"max":{"1min":0.70599999999999996,"5min":5.702,"15min":8.7270000000000003},"last":0.64800000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.873,"15min":0.76700000000000002},"min":{"1min":0.318,"5min":0.317,"15min":0.27500000000000002},"max":{"1min":0.78900000000000003,"5min":5.7359999999999998,"15min":7.9969999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.46999999999999997,"5min":1.0289999999999999,"15min":0.91700000000000004},"min":{"1min":0.32900000000000001,"5min":0.28499999999999998,"15min":0.27000000000000002},"max":{"1min":0.753,"5min":8.7400000000000002,"15min":8.7400000000000002},"last":0.63400000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.94999999999999996,"15min":0.95799999999999996},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.25900000000000001},"max":{"1min":0.77600000000000002,"5min":6.867,"15min":28.934000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.502,"5min":0.88700000000000001,"15min":0.86699999999999999},"min":{"1min":0.38200000000000001,"5min":0.27300000000000002,"15min":0.25900000000000001},"max":{"1min":0.71499999999999997,"5min":5.593,"15min":5.8259999999999996},"last":0.53800000000000003}]}]},{"osd":4,"up_from":33,"seq":141733921080,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900076,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":899326,"kb_avail":92845076,"statfs":{"total":95995035648,"available":95073357824,"internally_reserved":0,"allocated":757760,"data_stored":373006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920910168},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.67000000000000004,"15min":1.083},"min":{"1min":0.30499999999999999,"5min":0.247,"15min":0.20899999999999999},"max":{"1min":0.68999999999999995,"5min":4.7050000000000001,"15min":21.468},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.57199999999999995,"15min":0.94099999999999995},"min":{"1min":0.34899999999999998,"5min":0.27000000000000002,"15min":0.189},"max":{"1min":0.73899999999999999,"5min":3.0409999999999999,"15min":24.277999999999999},"last":0.53500000000000003}]},{"osd":1,"last update":"Tue Sep 10 18:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.89500000000000002,"15min":1.2130000000000001},"min":{"1min":0.41099999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.66700000000000004,"5min":8.2539999999999996,"15min":50.045000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.82499999999999996,"15min":1.25},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.27100000000000002},"max":{"1min":0.68000000000000005,"5min":4.0380000000000003,"15min":50.283000000000001},"last":0.51500000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":3.0830000000000002,"15min":2.008},"min":{"1min":0.32400000000000001,"5min":0.28100000000000003,"15min":0.249},"max":{"1min":0.71799999999999997,"5min":169.417,"15min":169.417},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":2.996,"15min":2.004},"min":{"1min":0.39400000000000002,"5min":0.251,"15min":0.24099999999999999},"max":{"1min":0.78700000000000003,"5min":169.40100000000001,"15min":169.40100000000001},"last":0.58399999999999996}]},{"osd":3,"last update":"Tue Sep 10 18:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":2.069,"15min":1.5700000000000001},"min":{"1min":0.34499999999999997,"5min":0.27200000000000002,"15min":0.26200000000000001},"max":{"1min":0.70399999999999996,"5min":116.913,"15min":116.913},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.53200000000000003,"5min":2.1920000000000002,"15min":1.7849999999999999},"min":{"1min":0.41199999999999998,"5min":0.309,"15min":0.28999999999999998},"max":{"1min":0.68899999999999995,"5min":116.863,"15min":116.863},"last":0.55200000000000005}]},{"osd":5,"last update":"Tue Sep 10 18:45:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.69099999999999995,"15min":0.83399999999999996},"min":{"1min":0.35599999999999998,"5min":0.252,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":3.6379999999999999,"15min":22.861000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.57499999999999996,"15min":0.85199999999999998},"min":{"1min":0.30299999999999999,"5min":0.254,"15min":0.22},"max":{"1min":0.72099999999999997,"5min":2.976,"15min":19.611999999999998},"last":0.35199999999999998}]},{"osd":6,"last update":"Tue Sep 10 18:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.90100000000000002,"15min":1.0309999999999999},"min":{"1min":0.29799999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.61799999999999999,"5min":6.7130000000000001,"15min":51.722000000000001},"last":0.38},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.878,"15min":0.99399999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.59999999999999998,"5min":8.5749999999999993,"15min":51.862000000000002},"last":0.42099999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.871,"15min":0.83199999999999996},"min":{"1min":0.35399999999999998,"5min":0.312,"15min":0.28799999999999998},"max":{"1min":0.747,"5min":6.4470000000000001,"15min":7.6260000000000003},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.78900000000000003,"15min":0.81599999999999995},"min":{"1min":0.35599999999999998,"5min":0.31,"15min":0.28000000000000003},"max":{"1min":0.67900000000000005,"5min":6.96,"15min":7.8250000000000002},"last":0.45700000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117310,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":892664,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":891582,"kb_avail":92852488,"statfs":{"total":95995035648,"available":95080947712,"internally_reserved":0,"allocated":1097728,"data_stored":701530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912980312},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.81200000000000006,"15min":1.089},"min":{"1min":0.23899999999999999,"5min":0.19900000000000001,"15min":0.18099999999999999},"max":{"1min":0.47899999999999998,"5min":8.8079999999999998,"15min":29.981999999999999},"last":0.22},{"interface":"front","average":{"1min":0.374,"5min":0.68600000000000005,"15min":1.2190000000000001},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.193},"max":{"1min":0.56599999999999995,"5min":5.0010000000000003,"15min":28.754000000000001},"last":0.35299999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":1.0820000000000001,"15min":1.4159999999999999},"min":{"1min":0.253,"5min":0.20499999999999999,"15min":0.19600000000000001},"max":{"1min":0.505,"5min":9.5199999999999996,"15min":35.406999999999996},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":1.0760000000000001,"15min":1.391},"min":{"1min":0.24199999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.51500000000000001,"5min":8.9499999999999993,"15min":36.902000000000001},"last":0.30599999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":1.0349999999999999,"15min":1.3120000000000001},"min":{"1min":0.247,"5min":0.22800000000000001,"15min":0.191},"max":{"1min":0.58499999999999996,"5min":8.7210000000000001,"15min":79.975999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.373,"5min":1.085,"15min":1.5580000000000001},"min":{"1min":0.28000000000000003,"5min":0.23400000000000001,"15min":0.22},"max":{"1min":0.502,"5min":11.242000000000001,"15min":78.278000000000006},"last":0.28799999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:45:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":1.046,"15min":1.111},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.27600000000000002},"max":{"1min":0.76600000000000001,"5min":4.8780000000000001,"15min":6.5250000000000004},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.97599999999999998,"15min":1.085},"min":{"1min":0.35599999999999998,"5min":0.28399999999999997,"15min":0.27300000000000002},"max":{"1min":0.82899999999999996,"5min":5.1989999999999998,"15min":6.1929999999999996},"last":0.45300000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:45:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.93500000000000005,"15min":1.0940000000000001},"min":{"1min":0.41799999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.80200000000000005,"5min":4.8479999999999999,"15min":6.601},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.94699999999999995,"15min":1.127},"min":{"1min":0.503,"5min":0.254,"15min":0.254},"max":{"1min":0.78400000000000003,"5min":5.0410000000000004,"15min":6.2249999999999996},"last":0.59499999999999997}]},{"osd":6,"last update":"Tue Sep 10 18:45:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":1.0229999999999999,"15min":1.248},"min":{"1min":0.44400000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.75700000000000001,"5min":6.8129999999999997,"15min":31.815999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":1.091,"15min":1.335},"min":{"1min":0.46000000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.86699999999999999,"5min":8.7569999999999997,"15min":31.742000000000001},"last":0.60299999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:45:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":1.113,"15min":1.1539999999999999},"min":{"1min":0.38300000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.77100000000000002,"5min":4.9000000000000004,"15min":6.5730000000000004},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.628,"5min":1.022,"15min":1.198},"min":{"1min":0.47599999999999998,"5min":0.34100000000000003,"15min":0.31900000000000001},"max":{"1min":0.84599999999999997,"5min":5.1180000000000003,"15min":12.612},"last":0.51300000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411649,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901420,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":900414,"kb_avail":92843732,"statfs":{"total":95995035648,"available":95071981568,"internally_reserved":0,"allocated":1019904,"data_stored":635496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922024280},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.628,"15min":1.1200000000000001},"min":{"1min":0.223,"5min":0.216,"15min":0.17499999999999999},"max":{"1min":0.54400000000000004,"5min":3.5169999999999999,"15min":26.585999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.70499999999999996,"15min":1.3380000000000001},"min":{"1min":0.25600000000000001,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":0.54600000000000004,"5min":5.8879999999999999,"15min":26.48},"last":0.443}]},{"osd":1,"last update":"Tue Sep 10 18:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":1.071,"15min":1.274},"min":{"1min":0.25700000000000001,"5min":0.22500000000000001,"15min":0.19400000000000001},"max":{"1min":0.52600000000000002,"5min":11.154,"15min":15.750999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.375,"5min":1.0409999999999999,"15min":1.1479999999999999},"min":{"1min":0.27200000000000002,"5min":0.248,"15min":0.19500000000000001},"max":{"1min":0.54200000000000004,"5min":10.637,"15min":14.988},"last":0.41699999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:45:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.96899999999999997,"15min":1.5720000000000001},"min":{"1min":0.29899999999999999,"5min":0.28599999999999998,"15min":0.18099999999999999},"max":{"1min":0.57599999999999996,"5min":7.9359999999999999,"15min":61.331000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":1.1399999999999999,"15min":1.7070000000000001},"min":{"1min":0.32000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.79400000000000004,"5min":6.734,"15min":61.250999999999998},"last":0.372}]},{"osd":4,"last update":"Tue Sep 10 18:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.92200000000000004,"15min":1.212},"min":{"1min":0.378,"5min":0.249,"15min":0.249},"max":{"1min":0.82899999999999996,"5min":5.3129999999999997,"15min":12.555999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.90900000000000003,"15min":1.129},"min":{"1min":0.39500000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81799999999999995,"5min":3.6909999999999998,"15min":19.696000000000002},"last":0.58699999999999997}]},{"osd":5,"last update":"Tue Sep 10 18:45:20 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.89500000000000002,"15min":1.2150000000000001},"min":{"1min":0.47499999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.91700000000000004,"5min":3.6400000000000001,"15min":25.177},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.97099999999999997,"15min":1.2729999999999999},"min":{"1min":0.40300000000000002,"5min":0.32000000000000001,"15min":0.317},"max":{"1min":0.86599999999999999,"5min":3.665,"15min":24.986999999999998},"last":0.61599999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.99399999999999999,"15min":1.4570000000000001},"min":{"1min":0.40799999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78500000000000003,"5min":5.2770000000000001,"15min":53.359999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.97999999999999998,"15min":1.351},"min":{"1min":0.441,"5min":0.34799999999999998,"15min":0.32900000000000001},"max":{"1min":0.84699999999999998,"5min":3.6960000000000002,"15min":53.438000000000002},"last":0.54700000000000004}]},{"osd":7,"last update":"Tue Sep 10 18:45:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.97699999999999998,"15min":1.1459999999999999},"min":{"1min":0.40799999999999997,"5min":0.36699999999999999,"15min":0.35399999999999998},"max":{"1min":0.80000000000000004,"5min":6.1980000000000004,"15min":10.68},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.624,"5min":1.121,"15min":1.3109999999999999},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.95599999999999996,"5min":5.2320000000000002,"15min":12.596},"last":0.69699999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575174,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901808,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":901054,"kb_avail":92843344,"statfs":{"total":95995035648,"available":95071584256,"internally_reserved":0,"allocated":761856,"data_stored":373306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922679640},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:45:49 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.625,"15min":1.093},"min":{"1min":0.25,"5min":0.184,"15min":0.16600000000000001},"max":{"1min":2.2210000000000001,"5min":6.4619999999999997,"15min":28.893999999999998},"last":0.314},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.66200000000000003,"15min":1.1240000000000001},"min":{"1min":0.247,"5min":0.20100000000000001,"15min":0.19800000000000001},"max":{"1min":0.5,"5min":7.8700000000000001,"15min":31.074999999999999},"last":0.40100000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.94499999999999995,"15min":1.1100000000000001},"min":{"1min":0.23200000000000001,"5min":0.216,"15min":0.19500000000000001},"max":{"1min":0.53500000000000003,"5min":11.079000000000001,"15min":11.079000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.93899999999999995,"15min":1.151},"min":{"1min":0.248,"5min":0.21199999999999999,"15min":0.20999999999999999},"max":{"1min":0.496,"5min":6.274,"15min":12.222},"last":0.46000000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":2.593,"15min":1.6299999999999999},"min":{"1min":0.251,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":2.1869999999999998,"5min":148.22900000000001,"15min":148.22900000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.496,"5min":2.5059999999999998,"15min":1.6259999999999999},"min":{"1min":0.27300000000000002,"5min":0.24399999999999999,"15min":0.20499999999999999},"max":{"1min":2.1190000000000002,"5min":148.65000000000001,"15min":148.65000000000001},"last":0.29499999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:45:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.81100000000000005,"15min":1.528},"min":{"1min":0.39900000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.86699999999999999,"5min":3.496,"15min":127.86199999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":1.0569999999999999,"15min":1.649},"min":{"1min":0.39900000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.92100000000000004,"5min":6.7690000000000001,"15min":127.84399999999999},"last":0.72399999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":1.0269999999999999,"15min":1.679},"min":{"1min":0.40600000000000003,"5min":0.32300000000000001,"15min":0.27900000000000003},"max":{"1min":0.88900000000000001,"5min":6.7939999999999996,"15min":118.68000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":1.0589999999999999,"15min":1.7150000000000001},"min":{"1min":0.46899999999999997,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.014,"5min":6.8929999999999998,"15min":118.36},"last":0.76600000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":4.7130000000000001,"15min":3.2919999999999998},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":0.86099999999999999,"5min":389.52499999999998,"15min":389.52499999999998},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":4.4589999999999996,"15min":3.4769999999999999},"min":{"1min":0.378,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93100000000000005,"5min":389.69499999999999,"15min":389.69499999999999},"last":0.69099999999999995}]},{"osd":7,"last update":"Tue Sep 10 18:45:10 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":1.3560000000000001,"15min":1.268},"min":{"1min":0.40699999999999997,"5min":0.40600000000000003,"15min":0.32700000000000001},"max":{"1min":0.79000000000000004,"5min":16.244,"15min":16.244},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.0700000000000001,"15min":1.222},"min":{"1min":0.45500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.89500000000000002,"5min":16.550000000000001,"15min":16.550000000000001},"last":0.73799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738699,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":897000,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":895934,"kb_avail":92848152,"statfs":{"total":95995035648,"available":95076507648,"internally_reserved":0,"allocated":1081344,"data_stored":700559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917436760},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:45:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.65600000000000003,"15min":1.577},"min":{"1min":0.23699999999999999,"5min":0.20499999999999999,"15min":0.185},"max":{"1min":0.66100000000000003,"5min":5.7610000000000001,"15min":146.44499999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.60199999999999998,"15min":1.6160000000000001},"min":{"1min":0.223,"5min":0.221,"15min":0.20100000000000001},"max":{"1min":0.57899999999999996,"5min":2.8660000000000001,"15min":148.77199999999999},"last":0.33200000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":3.589,"15min":2.613},"min":{"1min":0.29699999999999999,"5min":0.222,"15min":0.19700000000000001},"max":{"1min":0.52400000000000002,"5min":102.571,"15min":146.56999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.37,"5min":3.4550000000000001,"15min":2.3740000000000001},"min":{"1min":0.248,"5min":0.21299999999999999,"15min":0.21099999999999999},"max":{"1min":0.68300000000000005,"5min":101.916,"15min":146.601},"last":0.312}]},{"osd":3,"last update":"Tue Sep 10 18:45:38 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.3919999999999999,"15min":1.764},"min":{"1min":0.27300000000000002,"5min":0.26400000000000001,"15min":0.23400000000000001},"max":{"1min":0.68600000000000005,"5min":43.179000000000002,"15min":140.68299999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":1.4390000000000001,"15min":1.905},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.80300000000000005,"5min":43.226999999999997,"15min":140.572},"last":0.35599999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.76700000000000002,"15min":1.0840000000000001},"min":{"1min":0.41299999999999998,"5min":0.311,"15min":0.27500000000000002},"max":{"1min":0.77500000000000002,"5min":5.5700000000000003,"15min":8.3699999999999992},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.72599999999999998,"15min":1.0569999999999999},"min":{"1min":0.40300000000000002,"5min":0.33400000000000002,"15min":0.29899999999999999},"max":{"1min":0.75800000000000001,"5min":3.6459999999999999,"15min":8.7710000000000008},"last":0.55600000000000005}]},{"osd":5,"last update":"Tue Sep 10 18:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.75600000000000001,"15min":1.107},"min":{"1min":0.42099999999999999,"5min":0.32500000000000001,"15min":0.28799999999999998},"max":{"1min":0.97399999999999998,"5min":5.3920000000000003,"15min":8.1259999999999994},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64500000000000002,"15min":1.1040000000000001},"min":{"1min":0.47899999999999998,"5min":0.33500000000000002,"15min":0.29799999999999999},"max":{"1min":0.93000000000000005,"5min":3.6400000000000001,"15min":20.460000000000001},"last":0.57099999999999995}]},{"osd":6,"last update":"Tue Sep 10 18:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.83799999999999997,"15min":2.5},"min":{"1min":0.436,"5min":0.379,"15min":0.28899999999999998},"max":{"1min":0.75,"5min":5.5439999999999996,"15min":400.83699999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.73499999999999999,"15min":2.399},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.33300000000000002},"max":{"1min":0.89400000000000002,"5min":2.3700000000000001,"15min":400.77999999999997},"last":0.66700000000000004}]},{"osd":7,"last update":"Tue Sep 10 18:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.80300000000000005,"15min":1.2010000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.32200000000000001},"max":{"1min":0.86399999999999999,"5min":3.9350000000000001,"15min":10.75},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.751,"15min":1.0640000000000001},"min":{"1min":0.52500000000000002,"5min":0.38,"15min":0.378},"max":{"1min":0.86599999999999999,"5min":2.4350000000000001,"15min":11.218999999999999},"last":0.627}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3503,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:45:57.156 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-10T18:45:57.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-10T18:45:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:57 smithi204 ceph-mon[32357]: 1.b deep-scrub starts 2024-09-10T18:45:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:57 smithi204 ceph-mon[32357]: 1.b deep-scrub ok 2024-09-10T18:45:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:57 smithi204 ceph-mon[32357]: 1.1 scrub starts 2024-09-10T18:45:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:57 smithi204 ceph-mon[32357]: 1.1 scrub ok 2024-09-10T18:45:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:57 smithi204 ceph-mon[32357]: from='client.16464 v1:172.21.15.28:0/2843785657' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:45:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:57 smithi204 ceph-mon[32357]: 6.1f scrub starts 2024-09-10T18:45:57.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:57 smithi204 ceph-mon[32357]: 6.1f scrub ok 2024-09-10T18:45:57.419 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:45:57.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:57 smithi028 ceph-mon[39220]: 1.b deep-scrub starts 2024-09-10T18:45:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:57 smithi028 ceph-mon[39220]: 1.b deep-scrub ok 2024-09-10T18:45:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:57 smithi028 ceph-mon[39220]: 1.1 scrub starts 2024-09-10T18:45:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:57 smithi028 ceph-mon[39220]: 1.1 scrub ok 2024-09-10T18:45:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:57 smithi028 ceph-mon[39220]: from='client.16464 v1:172.21.15.28:0/2843785657' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:45:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:57 smithi028 ceph-mon[39220]: 6.1f scrub starts 2024-09-10T18:45:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:57 smithi028 ceph-mon[39220]: 6.1f scrub ok 2024-09-10T18:45:57.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:57 smithi028 ceph-mon[29291]: 1.b deep-scrub starts 2024-09-10T18:45:57.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:57 smithi028 ceph-mon[29291]: 1.b deep-scrub ok 2024-09-10T18:45:57.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:57 smithi028 ceph-mon[29291]: 1.1 scrub starts 2024-09-10T18:45:57.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:57 smithi028 ceph-mon[29291]: 1.1 scrub ok 2024-09-10T18:45:57.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:57 smithi028 ceph-mon[29291]: from='client.16464 v1:172.21.15.28:0/2843785657' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:45:57.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:57 smithi028 ceph-mon[29291]: 6.1f scrub starts 2024-09-10T18:45:57.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:57 smithi028 ceph-mon[29291]: 6.1f scrub ok 2024-09-10T18:45:58.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:58 smithi028 ceph-mon[29291]: pgmap v1098: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:45:58.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:58 smithi028 ceph-mon[39220]: pgmap v1098: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:45:58.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:58 smithi204 ceph-mon[32357]: pgmap v1098: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:45:59.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-mon[29291]: 3.1a deep-scrub starts 2024-09-10T18:45:59.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-mon[29291]: 3.1a deep-scrub ok 2024-09-10T18:45:59.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-mon[29291]: 5.9 scrub starts 2024-09-10T18:45:59.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-mon[29291]: 5.9 scrub ok 2024-09-10T18:45:59.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-mon[29291]: 5.13 scrub starts 2024-09-10T18:45:59.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-mon[29291]: 5.13 scrub ok 2024-09-10T18:45:59.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-mon[29291]: 3.1b scrub starts 2024-09-10T18:45:59.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-mon[29291]: 3.1b scrub ok 2024-09-10T18:45:59.254 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-mon[39220]: 3.1a deep-scrub starts 2024-09-10T18:45:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-mon[39220]: 3.1a deep-scrub ok 2024-09-10T18:45:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-mon[39220]: 5.9 scrub starts 2024-09-10T18:45:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-mon[39220]: 5.9 scrub ok 2024-09-10T18:45:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-mon[39220]: 5.13 scrub starts 2024-09-10T18:45:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-mon[39220]: 5.13 scrub ok 2024-09-10T18:45:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-mon[39220]: 3.1b scrub starts 2024-09-10T18:45:59.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-mon[39220]: 3.1b scrub ok 2024-09-10T18:45:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:59 smithi204 ceph-mon[32357]: 3.1a deep-scrub starts 2024-09-10T18:45:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:59 smithi204 ceph-mon[32357]: 3.1a deep-scrub ok 2024-09-10T18:45:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:59 smithi204 ceph-mon[32357]: 5.9 scrub starts 2024-09-10T18:45:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:59 smithi204 ceph-mon[32357]: 5.9 scrub ok 2024-09-10T18:45:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:59 smithi204 ceph-mon[32357]: 5.13 scrub starts 2024-09-10T18:45:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:59 smithi204 ceph-mon[32357]: 5.13 scrub ok 2024-09-10T18:45:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:59 smithi204 ceph-mon[32357]: 3.1b scrub starts 2024-09-10T18:45:59.354 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:45:59 smithi204 ceph-mon[32357]: 3.1b scrub ok 2024-09-10T18:45:59.567 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:45:59] ENGINE Bus STOPPING 2024-09-10T18:45:59.567 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:45:59] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T18:45:59.567 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:45:59] ENGINE Bus STOPPED 2024-09-10T18:45:59.567 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:45:59] ENGINE Bus STARTING 2024-09-10T18:45:59.567 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:45:59] ENGINE Serving on http://:::9283 2024-09-10T18:45:59.567 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:45:59 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:45:59] ENGINE Bus STARTED 2024-09-10T18:46:00.161 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-10T18:46:00.171 INFO:tasks.cephadm:Teardown begin 2024-09-10T18:46:00.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-10T18:46:00.215 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-10T18:46:00.295 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:46:00 smithi204 ceph-mon[32357]: pgmap v1099: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:46:00.296 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:46:00 smithi204 ceph-mon[32357]: from='client.? v1:172.21.15.28:0/2735936433' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T18:46:00.296 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:46:00 smithi204 ceph-mon[32357]: from='client.36350 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T18:46:00.296 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:46:00 smithi204 ceph-mon[32357]: from='client.36350 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-10T18:46:00.296 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:46:00 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:46:00.296 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:46:00 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:00.296 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:46:00 smithi204 ceph-mon[32357]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:46:00.296 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:46:00 smithi204 ceph-mon[32357]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:46:00.297 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-10T18:46:00.297 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-10T18:46:00.315 DEBUG:teuthology.orchestra.run.smithi204:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-10T18:46:00.333 INFO:tasks.cephadm:Stopping all daemons... 2024-09-10T18:46:00.333 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-10T18:46:00.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mon.a 2024-09-10T18:46:00.379 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-mon[39220]: pgmap v1099: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:46:00.379 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-mon[39220]: from='client.? v1:172.21.15.28:0/2735936433' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T18:46:00.379 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-mon[39220]: from='client.36350 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T18:46:00.379 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-mon[39220]: from='client.36350 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-10T18:46:00.379 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:46:00.380 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:00.380 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-mon[39220]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:46:00.380 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-mon[39220]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:46:00.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-mon[29291]: pgmap v1099: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:46:00.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-mon[29291]: from='client.? v1:172.21.15.28:0/2735936433' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T18:46:00.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-mon[29291]: from='client.36350 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T18:46:00.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-mon[29291]: from='client.36350 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-10T18:46:00.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:46:00.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:00.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-mon[29291]: from='mgr.34472 v1:172.21.15.28:0/3270023362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:46:00.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-mon[29291]: from='mgr.34472 ' entity='mgr.y' 2024-09-10T18:46:00.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:46:00 smithi028 systemd[1]: Stopping Ceph mon.a for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:46:00.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a[29287]: 2024-09-10T18:46:00.652+0000 7fed26862640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T18:46:00.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:46:00 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a[29287]: 2024-09-10T18:46:00.652+0000 7fed26862640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-10T18:46:00.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:46:00 smithi028 podman[215390]: 2024-09-10 18:46:00.693653217 +0000 UTC m=+0.138805235 container died c1cb1eb6243125d19433dfd1a26fe957d501c01e7abf3c4f110ef30df115a11d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T18:46:01.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:46:01 smithi028 podman[215390]: 2024-09-10 18:46:01.169137914 +0000 UTC m=+0.614289923 container cleanup c1cb1eb6243125d19433dfd1a26fe957d501c01e7abf3c4f110ef30df115a11d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-10T18:46:01.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:46:01 smithi028 bash[215390]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a 2024-09-10T18:46:01.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 10 18:46:01 smithi028 podman[215402]: 2024-09-10 18:46:01.487688855 +0000 UTC m=+0.790521541 container remove c1cb1eb6243125d19433dfd1a26fe957d501c01e7abf3c4f110ef30df115a11d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-a, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True) 2024-09-10T18:46:01.599 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:46:01 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:46:01] "GET /metrics HTTP/1.1" 200 222480 "" "Prometheus/2.43.0" 2024-09-10T18:46:01.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mon.a.service' 2024-09-10T18:46:01.725 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T18:46:01.726 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-10T18:46:01.726 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-10T18:46:01.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mon.c 2024-09-10T18:46:02.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:46:01 smithi028 systemd[1]: Stopping Ceph mon.c for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:46:02.324 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:46:02 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:46:02] ENGINE Bus STOPPING 2024-09-10T18:46:02.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:46:02 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-c[39216]: 2024-09-10T18:46:02.029+0000 7f2625728640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T18:46:02.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:46:02 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-c[39216]: 2024-09-10T18:46:02.029+0000 7f2625728640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-10T18:46:02.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:46:02 smithi028 podman[215511]: 2024-09-10 18:46:02.061762611 +0000 UTC m=+0.146914209 container died 9f8c661432c8db15ed754ed487235fa154e42bdbece867bc97c207bc9cd9625b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True) 2024-09-10T18:46:02.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:46:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x[34193]: ::ffff:172.21.15.204 - - [10/Sep/2024:18:46:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:46:02.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:46:02 smithi028 podman[215511]: 2024-09-10 18:46:02.549176652 +0000 UTC m=+0.634328247 container cleanup 9f8c661432c8db15ed754ed487235fa154e42bdbece867bc97c207bc9cd9625b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-c, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-10T18:46:02.754 INFO:journalctl@ceph.mon.c.smithi028.stdout:Sep 10 18:46:02 smithi028 bash[215511]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-c 2024-09-10T18:46:02.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:46:02 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:46:02] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T18:46:02.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:46:02 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:46:02] ENGINE Bus STOPPED 2024-09-10T18:46:02.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:46:02 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:46:02] ENGINE Bus STARTING 2024-09-10T18:46:02.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:46:02 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:46:02] ENGINE Serving on http://:::9283 2024-09-10T18:46:02.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:46:02 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: [10/Sep/2024:18:46:02] ENGINE Bus STARTED 2024-09-10T18:46:02.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mon.c.service' 2024-09-10T18:46:02.977 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T18:46:02.977 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-10T18:46:02.977 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-10T18:46:02.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mon.b 2024-09-10T18:46:03.299 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:46:03 smithi204 systemd[1]: Stopping Ceph mon.b for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:46:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:46:03 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-b[32353]: 2024-09-10T18:46:03.296+0000 7f0ac1cf5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T18:46:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:46:03 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-b[32353]: 2024-09-10T18:46:03.296+0000 7f0ac1cf5640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-10T18:46:03.604 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:46:03 smithi204 podman[157515]: 2024-09-10 18:46:03.329589182 +0000 UTC m=+0.142388956 container died edec062d1cefa105b8ec5bf10f41072af5e48fc7c2340ea08909c462279cea02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-10T18:46:04.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:46:03 smithi204 podman[157515]: 2024-09-10 18:46:03.798938592 +0000 UTC m=+0.611738411 container cleanup edec062d1cefa105b8ec5bf10f41072af5e48fc7c2340ea08909c462279cea02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-10T18:46:04.104 INFO:journalctl@ceph.mon.b.smithi204.stdout:Sep 10 18:46:03 smithi204 bash[157515]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mon-b 2024-09-10T18:46:04.244 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mon.b.service' 2024-09-10T18:46:04.284 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T18:46:04.285 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-10T18:46:04.285 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-10T18:46:04.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mgr.y 2024-09-10T18:46:04.571 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:46:04 smithi028 systemd[1]: Stopping Ceph mgr.y for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:46:04.922 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:46:04 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:46:04.568+0000 7f2e23809640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T18:46:04.922 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:46:04 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y[29518]: 2024-09-10T18:46:04.568+0000 7f2e23809640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-10T18:46:04.922 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:46:04 smithi028 podman[215645]: 2024-09-10 18:46:04.600880771 +0000 UTC m=+0.122843610 container died c0b9aeca110e9d6332e41bb2df929e07e373e3c587a9b0307fa532faa160ac80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-10T18:46:05.253 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:46:05 smithi028 podman[215645]: 2024-09-10 18:46:05.137315531 +0000 UTC m=+0.659278370 container cleanup c0b9aeca110e9d6332e41bb2df929e07e373e3c587a9b0307fa532faa160ac80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-10T18:46:05.254 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:46:05 smithi028 bash[215645]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y 2024-09-10T18:46:05.616 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Sep 10 18:46:05 smithi028 podman[215657]: 2024-09-10 18:46:05.363998337 +0000 UTC m=+0.751485618 container remove c0b9aeca110e9d6332e41bb2df929e07e373e3c587a9b0307fa532faa160ac80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-y, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T18:46:05.620 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mgr.y.service' 2024-09-10T18:46:05.655 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T18:46:05.656 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-10T18:46:05.656 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-10T18:46:05.656 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mgr.x 2024-09-10T18:46:05.983 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:46:05 smithi204 systemd[1]: Stopping Ceph mgr.x for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:46:06.273 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:46:05 smithi204 podman[157637]: 2024-09-10 18:46:05.982822287 +0000 UTC m=+0.137499205 container died 0d6adb1156510a78cb8d96dbf8d82449cc570459dd5b977915f9f8206c345904 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-10T18:46:06.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:46:06 smithi204 podman[157637]: 2024-09-10 18:46:06.379061068 +0000 UTC m=+0.533737997 container cleanup 0d6adb1156510a78cb8d96dbf8d82449cc570459dd5b977915f9f8206c345904 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-10T18:46:06.604 INFO:journalctl@ceph.mgr.x.smithi204.stdout:Sep 10 18:46:06 smithi204 bash[157637]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-mgr-x 2024-09-10T18:46:06.760 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@mgr.x.service' 2024-09-10T18:46:06.795 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T18:46:06.796 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-10T18:46:06.796 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-10T18:46:06.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.0 2024-09-10T18:46:07.085 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:46:06 smithi028 systemd[1]: Stopping Ceph osd.0 for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:46:07.503 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:46:07 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-0[43198]: 2024-09-10T18:46:07.083+0000 7fa7bbcee640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T18:46:07.503 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:46:07 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-0[43198]: 2024-09-10T18:46:07.083+0000 7fa7bbcee640 -1 osd.0 456 *** Got signal Terminated *** 2024-09-10T18:46:07.504 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:46:07 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-0[43198]: 2024-09-10T18:46:07.083+0000 7fa7bbcee640 -1 osd.0 456 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T18:46:12.869 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:46:12 smithi028 podman[215763]: 2024-09-10 18:46:12.595731926 +0000 UTC m=+5.606139151 container died 47afe10483eb16aff856e1db4e25c1b4f8476870ca190a9b71d053145500b5d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-10T18:46:13.254 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:46:13 smithi028 podman[215763]: 2024-09-10 18:46:13.091849168 +0000 UTC m=+6.102256392 container cleanup 47afe10483eb16aff856e1db4e25c1b4f8476870ca190a9b71d053145500b5d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True) 2024-09-10T18:46:13.254 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:46:13 smithi028 bash[215763]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-0 2024-09-10T18:46:13.745 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:46:13 smithi028 podman[215780]: 2024-09-10 18:46:13.426706914 +0000 UTC m=+0.822534710 container remove 47afe10483eb16aff856e1db4e25c1b4f8476870ca190a9b71d053145500b5d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-10T18:46:14.003 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:46:13 smithi028 podman[215869]: 2024-09-10 18:46:13.647363782 +0000 UTC m=+0.016228918 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T18:46:14.004 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:46:13 smithi028 podman[215869]: 2024-09-10 18:46:13.769433562 +0000 UTC m=+0.138298752 container create 3675c6c943473f287101a0bdaf5dc93da9cc424a6ccd47d7463bcb1cf3c19f6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-10T18:46:14.278 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:46:14 smithi028 podman[215869]: 2024-09-10 18:46:14.046838378 +0000 UTC m=+0.415703525 container init 3675c6c943473f287101a0bdaf5dc93da9cc424a6ccd47d7463bcb1cf3c19f6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T18:46:14.278 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:46:14 smithi028 podman[215869]: 2024-09-10 18:46:14.051973133 +0000 UTC m=+0.420838269 container start 3675c6c943473f287101a0bdaf5dc93da9cc424a6ccd47d7463bcb1cf3c19f6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-10T18:46:14.278 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:46:14 smithi028 podman[215869]: 2024-09-10 18:46:14.138401061 +0000 UTC m=+0.507266200 container attach 3675c6c943473f287101a0bdaf5dc93da9cc424a6ccd47d7463bcb1cf3c19f6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-10T18:46:14.644 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 10 18:46:14 smithi028 podman[215869]: 2024-09-10 18:46:14.384354103 +0000 UTC m=+0.753219242 container died 3675c6c943473f287101a0bdaf5dc93da9cc424a6ccd47d7463bcb1cf3c19f6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T18:46:15.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.0.service' 2024-09-10T18:46:15.190 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T18:46:15.190 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-10T18:46:15.190 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-10T18:46:15.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.1 2024-09-10T18:46:15.504 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:46:15 smithi028 systemd[1]: Stopping Ceph osd.1 for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:46:16.004 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:46:15 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-1[49149]: 2024-09-10T18:46:15.529+0000 7fcc54b1d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T18:46:16.004 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:46:15 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-1[49149]: 2024-09-10T18:46:15.529+0000 7fcc54b1d640 -1 osd.1 456 *** Got signal Terminated *** 2024-09-10T18:46:16.004 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:46:15 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-1[49149]: 2024-09-10T18:46:15.529+0000 7fcc54b1d640 -1 osd.1 456 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T18:46:21.339 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:46:21 smithi028 podman[215985]: 2024-09-10 18:46:21.053722592 +0000 UTC m=+5.634626595 container died 9b53f26bce44c9e934dd6f3e5f10cf634cdce1cad1d6f60ee2aa058f2e3090b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-10T18:46:21.753 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:46:21 smithi028 podman[215985]: 2024-09-10 18:46:21.495134926 +0000 UTC m=+6.076038919 container cleanup 9b53f26bce44c9e934dd6f3e5f10cf634cdce1cad1d6f60ee2aa058f2e3090b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-1, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-10T18:46:21.754 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:46:21 smithi028 bash[215985]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-1 2024-09-10T18:46:22.125 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:46:21 smithi028 podman[215999]: 2024-09-10 18:46:21.805054878 +0000 UTC m=+0.748780982 container remove 9b53f26bce44c9e934dd6f3e5f10cf634cdce1cad1d6f60ee2aa058f2e3090b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-1, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T18:46:22.426 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:46:22 smithi028 podman[216090]: 2024-09-10 18:46:22.027393631 +0000 UTC m=+0.016207491 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T18:46:22.426 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:46:22 smithi028 podman[216090]: 2024-09-10 18:46:22.130954987 +0000 UTC m=+0.119768837 container create aa61afa26c76e954bc90922b0e7bde1072d20220bd81f6ae8f036f81428a4fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-10T18:46:22.426 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:46:22 smithi028 podman[216090]: 2024-09-10 18:46:22.425125469 +0000 UTC m=+0.413939320 container init aa61afa26c76e954bc90922b0e7bde1072d20220bd81f6ae8f036f81428a4fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-10T18:46:22.754 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:46:22 smithi028 podman[216090]: 2024-09-10 18:46:22.428696719 +0000 UTC m=+0.417510575 container start aa61afa26c76e954bc90922b0e7bde1072d20220bd81f6ae8f036f81428a4fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T18:46:22.754 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:46:22 smithi028 podman[216090]: 2024-09-10 18:46:22.549935281 +0000 UTC m=+0.538749137 container attach aa61afa26c76e954bc90922b0e7bde1072d20220bd81f6ae8f036f81428a4fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-1-deactivate, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-10T18:46:22.754 INFO:journalctl@ceph.osd.1.smithi028.stdout:Sep 10 18:46:22 smithi028 podman[216090]: 2024-09-10 18:46:22.607292935 +0000 UTC m=+0.596106786 container died aa61afa26c76e954bc90922b0e7bde1072d20220bd81f6ae8f036f81428a4fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T18:46:23.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.1.service' 2024-09-10T18:46:23.331 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T18:46:23.331 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-10T18:46:23.331 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-10T18:46:23.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.2 2024-09-10T18:46:23.641 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:46:23 smithi028 systemd[1]: Stopping Ceph osd.2 for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:46:24.003 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:46:23 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-2[54713]: 2024-09-10T18:46:23.639+0000 7fea7e866640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T18:46:24.003 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:46:23 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-2[54713]: 2024-09-10T18:46:23.639+0000 7fea7e866640 -1 osd.2 456 *** Got signal Terminated *** 2024-09-10T18:46:24.003 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:46:23 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-2[54713]: 2024-09-10T18:46:23.639+0000 7fea7e866640 -1 osd.2 456 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T18:46:29.359 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:46:29 smithi028 podman[216195]: 2024-09-10 18:46:29.074467796 +0000 UTC m=+5.545988009 container died 607a55225ab4901916ad6c4a6271c7f684b287d5d0521efcaac3c84fdea90e55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-10T18:46:29.733 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:46:29 smithi028 podman[216195]: 2024-09-10 18:46:29.514498954 +0000 UTC m=+5.986019164 container cleanup 607a55225ab4901916ad6c4a6271c7f684b287d5d0521efcaac3c84fdea90e55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-10T18:46:29.733 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:46:29 smithi028 bash[216195]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-2 2024-09-10T18:46:30.003 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:46:29 smithi028 podman[216210]: 2024-09-10 18:46:29.732578133 +0000 UTC m=+0.649209741 container remove 607a55225ab4901916ad6c4a6271c7f684b287d5d0521efcaac3c84fdea90e55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-10T18:46:30.470 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:46:30 smithi028 podman[216297]: 2024-09-10 18:46:30.084825718 +0000 UTC m=+0.016693029 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T18:46:30.470 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:46:30 smithi028 podman[216297]: 2024-09-10 18:46:30.191628403 +0000 UTC m=+0.123495704 container create 5e6f5766be40135df9b0edd24e983f324480ebacc688172099adc5e18338916b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T18:46:30.470 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:46:30 smithi028 podman[216297]: 2024-09-10 18:46:30.402418821 +0000 UTC m=+0.334286140 container init 5e6f5766be40135df9b0edd24e983f324480ebacc688172099adc5e18338916b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T18:46:30.470 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:46:30 smithi028 podman[216297]: 2024-09-10 18:46:30.405749632 +0000 UTC m=+0.337616932 container start 5e6f5766be40135df9b0edd24e983f324480ebacc688172099adc5e18338916b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, ceph=True) 2024-09-10T18:46:30.470 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:46:30 smithi028 podman[216297]: 2024-09-10 18:46:30.468946553 +0000 UTC m=+0.400813862 container attach 5e6f5766be40135df9b0edd24e983f324480ebacc688172099adc5e18338916b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T18:46:30.754 INFO:journalctl@ceph.osd.2.smithi028.stdout:Sep 10 18:46:30 smithi028 podman[216297]: 2024-09-10 18:46:30.58764354 +0000 UTC m=+0.519510841 container died 5e6f5766be40135df9b0edd24e983f324480ebacc688172099adc5e18338916b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-10T18:46:31.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.2.service' 2024-09-10T18:46:31.263 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T18:46:31.263 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-10T18:46:31.263 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-10T18:46:31.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.3 2024-09-10T18:46:31.503 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 10 18:46:31 smithi028 systemd[1]: Stopping Ceph osd.3 for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:46:32.004 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 10 18:46:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-3[60920]: 2024-09-10T18:46:31.591+0000 7f93b1ba7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T18:46:32.004 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 10 18:46:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-3[60920]: 2024-09-10T18:46:31.591+0000 7f93b1ba7640 -1 osd.3 456 *** Got signal Terminated *** 2024-09-10T18:46:32.004 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 10 18:46:31 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-3[60920]: 2024-09-10T18:46:31.591+0000 7f93b1ba7640 -1 osd.3 456 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T18:46:34.025 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:33 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:33.664+0000 7f9c61244640 -1 osd.4 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:07.033057+0000 front 2024-09-10T18:46:07.032970+0000 (oldest deadline 2024-09-10T18:46:32.932959+0000) 2024-09-10T18:46:34.886 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:34.626+0000 7f9c61244640 -1 osd.4 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:07.033057+0000 front 2024-09-10T18:46:07.032970+0000 (oldest deadline 2024-09-10T18:46:32.932959+0000) 2024-09-10T18:46:35.301 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:34 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:34.885+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:36.103 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:35 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:35.672+0000 7f9c61244640 -1 osd.4 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:07.033057+0000 front 2024-09-10T18:46:07.032970+0000 (oldest deadline 2024-09-10T18:46:32.932959+0000) 2024-09-10T18:46:36.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:35 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:35.899+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:36.902 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:36 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:36.635+0000 7f9c61244640 -1 osd.4 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:07.033057+0000 front 2024-09-10T18:46:07.032970+0000 (oldest deadline 2024-09-10T18:46:32.932959+0000) 2024-09-10T18:46:37.185 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:36 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:36.901+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:37.185 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:37 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:37.148+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:37.376 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 10 18:46:37 smithi028 podman[216400]: 2024-09-10 18:46:37.075186599 +0000 UTC m=+5.582086015 container died 87d523b53f1d9ff1952e9c12489a7d69c801d7f0ad83626f6797efd7fc1452c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-10T18:46:37.591 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:37 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:37.183+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.654743+0000 front 2024-09-10T18:46:11.654904+0000 (oldest deadline 2024-09-10T18:46:36.855477+0000) 2024-09-10T18:46:37.693 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 10 18:46:37 smithi028 podman[216400]: 2024-09-10 18:46:37.532982175 +0000 UTC m=+6.039881633 container cleanup 87d523b53f1d9ff1952e9c12489a7d69c801d7f0ad83626f6797efd7fc1452c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-3, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T18:46:37.694 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 10 18:46:37 smithi028 bash[216400]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-3 2024-09-10T18:46:37.854 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:37 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:37.589+0000 7f9c61244640 -1 osd.4 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:07.033057+0000 front 2024-09-10T18:46:07.032970+0000 (oldest deadline 2024-09-10T18:46:32.932959+0000) 2024-09-10T18:46:38.003 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 10 18:46:37 smithi028 podman[216416]: 2024-09-10 18:46:37.692789785 +0000 UTC m=+0.610186201 container remove 87d523b53f1d9ff1952e9c12489a7d69c801d7f0ad83626f6797efd7fc1452c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-3, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-10T18:46:38.188 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:37 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:37.864+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:38.188 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:38 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:38.110+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:38.413 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 10 18:46:38 smithi028 podman[216504]: 2024-09-10 18:46:38.006154452 +0000 UTC m=+0.016033361 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T18:46:38.413 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 10 18:46:38 smithi028 podman[216504]: 2024-09-10 18:46:38.110188209 +0000 UTC m=+0.120067106 container create 68466fc675f5855be847c584df6759d6ab74a5f479fc518cb101fef83d38b675 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-10T18:46:38.413 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 10 18:46:38 smithi028 podman[216504]: 2024-09-10 18:46:38.329048822 +0000 UTC m=+0.338927716 container init 68466fc675f5855be847c584df6759d6ab74a5f479fc518cb101fef83d38b675 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-10T18:46:38.413 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 10 18:46:38 smithi028 podman[216504]: 2024-09-10 18:46:38.332627491 +0000 UTC m=+0.342506395 container start 68466fc675f5855be847c584df6759d6ab74a5f479fc518cb101fef83d38b675 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-10T18:46:38.413 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 10 18:46:38 smithi028 podman[216504]: 2024-09-10 18:46:38.412126952 +0000 UTC m=+0.422005846 container attach 68466fc675f5855be847c584df6759d6ab74a5f479fc518cb101fef83d38b675 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-3-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-10T18:46:38.542 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:38 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:38.186+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.654743+0000 front 2024-09-10T18:46:11.654904+0000 (oldest deadline 2024-09-10T18:46:36.855477+0000) 2024-09-10T18:46:38.754 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 10 18:46:38 smithi028 podman[216504]: 2024-09-10 18:46:38.516252364 +0000 UTC m=+0.526131278 container died 68466fc675f5855be847c584df6759d6ab74a5f479fc518cb101fef83d38b675 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-10T18:46:38.843 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:38 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:38.540+0000 7f9c61244640 -1 osd.4 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:07.033057+0000 front 2024-09-10T18:46:07.032970+0000 (oldest deadline 2024-09-10T18:46:32.932959+0000) 2024-09-10T18:46:39.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:38 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:38.841+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:39.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.3.service' 2024-09-10T18:46:39.301 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T18:46:39.301 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-10T18:46:39.301 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-10T18:46:39.301 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.4 2024-09-10T18:46:39.495 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:39 smithi204 systemd[1]: Stopping Ceph osd.4 for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:46:39.495 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:39 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:39.168+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.654743+0000 front 2024-09-10T18:46:11.654904+0000 (oldest deadline 2024-09-10T18:46:36.855477+0000) 2024-09-10T18:46:39.495 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:39 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:39.120+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:39.854 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:39 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:39.492+0000 7f9c61244640 -1 osd.4 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:07.033057+0000 front 2024-09-10T18:46:07.032970+0000 (oldest deadline 2024-09-10T18:46:32.932959+0000) 2024-09-10T18:46:39.854 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:39 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:39.598+0000 7f9c67c32640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T18:46:39.854 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:39 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:39.598+0000 7f9c67c32640 -1 osd.4 456 *** Got signal Terminated *** 2024-09-10T18:46:39.854 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:39 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:39.598+0000 7f9c67c32640 -1 osd.4 456 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T18:46:40.172 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:39 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:39.881+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:40.172 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:40 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:40.107+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:40.499 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:40 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:40.170+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.654743+0000 front 2024-09-10T18:46:11.654904+0000 (oldest deadline 2024-09-10T18:46:36.855477+0000) 2024-09-10T18:46:40.853 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:40 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:40.497+0000 7f9c61244640 -1 osd.4 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:07.033057+0000 front 2024-09-10T18:46:07.032970+0000 (oldest deadline 2024-09-10T18:46:32.932959+0000) 2024-09-10T18:46:41.144 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:40 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:40.853+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:41.517 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:41 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:41.142+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.654743+0000 front 2024-09-10T18:46:11.654904+0000 (oldest deadline 2024-09-10T18:46:36.855477+0000) 2024-09-10T18:46:41.517 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:41 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:41.147+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:41.854 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:41 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:41.516+0000 7f9c61244640 -1 osd.4 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:07.033057+0000 front 2024-09-10T18:46:07.032970+0000 (oldest deadline 2024-09-10T18:46:32.932959+0000) 2024-09-10T18:46:42.152 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:41 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:41.875+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:42.502 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:42.180+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.654743+0000 front 2024-09-10T18:46:11.654904+0000 (oldest deadline 2024-09-10T18:46:36.855477+0000) 2024-09-10T18:46:42.502 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:42.180+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.057075+0000 front 2024-09-10T18:46:19.057063+0000 (oldest deadline 2024-09-10T18:46:41.356952+0000) 2024-09-10T18:46:42.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:42.151+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:42.503 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:42.151+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:42.847 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:42.500+0000 7f9c61244640 -1 osd.4 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:07.033057+0000 front 2024-09-10T18:46:07.032970+0000 (oldest deadline 2024-09-10T18:46:32.932959+0000) 2024-09-10T18:46:42.847 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:42.500+0000 7f9c61244640 -1 osd.4 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.033947+0000 front 2024-09-10T18:46:17.034024+0000 (oldest deadline 2024-09-10T18:46:41.733914+0000) 2024-09-10T18:46:43.103 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:42 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:42.845+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:43.509 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:43 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:43.211+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.654743+0000 front 2024-09-10T18:46:11.654904+0000 (oldest deadline 2024-09-10T18:46:36.855477+0000) 2024-09-10T18:46:43.509 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:43 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:43.211+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.057075+0000 front 2024-09-10T18:46:19.057063+0000 (oldest deadline 2024-09-10T18:46:41.356952+0000) 2024-09-10T18:46:43.510 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:43 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:43.172+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:43.510 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:43 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:43.172+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:43.853 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:43 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:43.507+0000 7f9c61244640 -1 osd.4 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:07.033057+0000 front 2024-09-10T18:46:07.032970+0000 (oldest deadline 2024-09-10T18:46:32.932959+0000) 2024-09-10T18:46:43.854 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:43 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:43.507+0000 7f9c61244640 -1 osd.4 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.033947+0000 front 2024-09-10T18:46:17.034024+0000 (oldest deadline 2024-09-10T18:46:41.733914+0000) 2024-09-10T18:46:44.167 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:43 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:43.877+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:44.167 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:43 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:43.877+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.487911+0000 front 2024-09-10T18:46:19.487984+0000 (oldest deadline 2024-09-10T18:46:42.987826+0000) 2024-09-10T18:46:44.540 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:44.165+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.654743+0000 front 2024-09-10T18:46:11.654904+0000 (oldest deadline 2024-09-10T18:46:36.855477+0000) 2024-09-10T18:46:44.540 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:44.165+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.057075+0000 front 2024-09-10T18:46:19.057063+0000 (oldest deadline 2024-09-10T18:46:41.356952+0000) 2024-09-10T18:46:44.540 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:44.178+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:44.540 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:44.178+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:44.854 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:44.538+0000 7f9c61244640 -1 osd.4 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:07.033057+0000 front 2024-09-10T18:46:07.032970+0000 (oldest deadline 2024-09-10T18:46:32.932959+0000) 2024-09-10T18:46:44.854 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4[37119]: 2024-09-10T18:46:44.538+0000 7f9c61244640 -1 osd.4 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.033947+0000 front 2024-09-10T18:46:17.034024+0000 (oldest deadline 2024-09-10T18:46:41.733914+0000) 2024-09-10T18:46:45.161 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:44.896+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:45.161 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:44 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:44.896+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.487911+0000 front 2024-09-10T18:46:19.487984+0000 (oldest deadline 2024-09-10T18:46:42.987826+0000) 2024-09-10T18:46:45.534 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:45 smithi204 podman[157772]: 2024-09-10 18:46:45.197899431 +0000 UTC m=+5.697206591 container died bb44c28ab61a7c230e01a0c1e61733270a966fe03b2141a98c5fa04ec2a1154f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T18:46:45.534 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:45 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:45.203+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:45.535 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:45 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:45.203+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:45.535 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:45 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:45.159+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.654743+0000 front 2024-09-10T18:46:11.654904+0000 (oldest deadline 2024-09-10T18:46:36.855477+0000) 2024-09-10T18:46:45.535 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:45 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:45.159+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.057075+0000 front 2024-09-10T18:46:19.057063+0000 (oldest deadline 2024-09-10T18:46:41.356952+0000) 2024-09-10T18:46:45.789 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:45 smithi204 podman[157772]: 2024-09-10 18:46:45.740578894 +0000 UTC m=+6.239886055 container cleanup bb44c28ab61a7c230e01a0c1e61733270a966fe03b2141a98c5fa04ec2a1154f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-10T18:46:45.789 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:45 smithi204 bash[157772]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4 2024-09-10T18:46:46.104 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:45 smithi204 podman[157787]: 2024-09-10 18:46:45.942316852 +0000 UTC m=+0.740233102 container remove bb44c28ab61a7c230e01a0c1e61733270a966fe03b2141a98c5fa04ec2a1154f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T18:46:46.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:45 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:45.942+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:46.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:45 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:45.942+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.487911+0000 front 2024-09-10T18:46:19.487984+0000 (oldest deadline 2024-09-10T18:46:42.987826+0000) 2024-09-10T18:46:46.500 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:46 smithi204 podman[157876]: 2024-09-10 18:46:46.255788554 +0000 UTC m=+0.016087601 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T18:46:46.500 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:46 smithi204 podman[157876]: 2024-09-10 18:46:46.376984885 +0000 UTC m=+0.137283920 container create d48a4d9ba8721029dd42c4ffe2445683a70ed41a82c5e156db0960ee150fb996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-10T18:46:46.500 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:46 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:46.204+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.654743+0000 front 2024-09-10T18:46:11.654904+0000 (oldest deadline 2024-09-10T18:46:36.855477+0000) 2024-09-10T18:46:46.500 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:46 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:46.204+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.057075+0000 front 2024-09-10T18:46:19.057063+0000 (oldest deadline 2024-09-10T18:46:41.356952+0000) 2024-09-10T18:46:46.500 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:46 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:46.178+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:46.500 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:46 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:46.178+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:46.854 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:46 smithi204 podman[157876]: 2024-09-10 18:46:46.621402607 +0000 UTC m=+0.381701642 container init d48a4d9ba8721029dd42c4ffe2445683a70ed41a82c5e156db0960ee150fb996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4-deactivate, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-10T18:46:46.854 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:46 smithi204 podman[157876]: 2024-09-10 18:46:46.624782608 +0000 UTC m=+0.385081649 container start d48a4d9ba8721029dd42c4ffe2445683a70ed41a82c5e156db0960ee150fb996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-10T18:46:46.854 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:46 smithi204 podman[157876]: 2024-09-10 18:46:46.704795822 +0000 UTC m=+0.465094857 container attach d48a4d9ba8721029dd42c4ffe2445683a70ed41a82c5e156db0960ee150fb996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-10T18:46:47.159 INFO:journalctl@ceph.osd.4.smithi204.stdout:Sep 10 18:46:46 smithi204 podman[157876]: 2024-09-10 18:46:46.854233679 +0000 UTC m=+0.614532719 container died d48a4d9ba8721029dd42c4ffe2445683a70ed41a82c5e156db0960ee150fb996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T18:46:47.160 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:46 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:46.898+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:47.160 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:46 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:46.898+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.487911+0000 front 2024-09-10T18:46:19.487984+0000 (oldest deadline 2024-09-10T18:46:42.987826+0000) 2024-09-10T18:46:47.493 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:47 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:47.158+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.654743+0000 front 2024-09-10T18:46:11.654904+0000 (oldest deadline 2024-09-10T18:46:36.855477+0000) 2024-09-10T18:46:47.493 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:47 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:47.158+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.057075+0000 front 2024-09-10T18:46:19.057063+0000 (oldest deadline 2024-09-10T18:46:41.356952+0000) 2024-09-10T18:46:47.493 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:47 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:47.189+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:47.493 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:47 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:47.189+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:47.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.4.service' 2024-09-10T18:46:47.563 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T18:46:47.564 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-10T18:46:47.564 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-10T18:46:47.564 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.5 2024-09-10T18:46:47.783 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:47 smithi204 systemd[1]: Stopping Ceph osd.5 for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:46:48.103 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:47 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:47.851+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:48.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:47 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:47.851+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.487911+0000 front 2024-09-10T18:46:19.487984+0000 (oldest deadline 2024-09-10T18:46:42.987826+0000) 2024-09-10T18:46:48.104 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:47 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:47.928+0000 7fb201b2e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T18:46:48.104 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:47 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:47.928+0000 7fb201b2e640 -1 osd.5 456 *** Got signal Terminated *** 2024-09-10T18:46:48.104 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:47 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:47.928+0000 7fb201b2e640 -1 osd.5 456 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T18:46:48.604 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:48 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:48.159+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.654743+0000 front 2024-09-10T18:46:11.654904+0000 (oldest deadline 2024-09-10T18:46:36.855477+0000) 2024-09-10T18:46:48.604 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:48 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:48.159+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.057075+0000 front 2024-09-10T18:46:19.057063+0000 (oldest deadline 2024-09-10T18:46:41.356952+0000) 2024-09-10T18:46:48.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:48 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:48.170+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:48.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:48 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:48.170+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:49.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:48 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:48.853+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:49.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:48 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:48.853+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.487911+0000 front 2024-09-10T18:46:19.487984+0000 (oldest deadline 2024-09-10T18:46:42.987826+0000) 2024-09-10T18:46:49.604 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:49 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:49.189+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.654743+0000 front 2024-09-10T18:46:11.654904+0000 (oldest deadline 2024-09-10T18:46:36.855477+0000) 2024-09-10T18:46:49.604 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:49 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:49.189+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.057075+0000 front 2024-09-10T18:46:19.057063+0000 (oldest deadline 2024-09-10T18:46:41.356952+0000) 2024-09-10T18:46:49.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:49 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:49.184+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:49.605 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:49 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:49.184+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:50.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:49 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:49.851+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:50.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:49 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:49.851+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.487911+0000 front 2024-09-10T18:46:19.487984+0000 (oldest deadline 2024-09-10T18:46:42.987826+0000) 2024-09-10T18:46:50.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:49 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:49.851+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:27.589444+0000 front 2024-09-10T18:46:27.589545+0000 (oldest deadline 2024-09-10T18:46:49.289309+0000) 2024-09-10T18:46:50.603 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:50 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:50.155+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.654743+0000 front 2024-09-10T18:46:11.654904+0000 (oldest deadline 2024-09-10T18:46:36.855477+0000) 2024-09-10T18:46:50.604 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:50 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:50.155+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.057075+0000 front 2024-09-10T18:46:19.057063+0000 (oldest deadline 2024-09-10T18:46:41.356952+0000) 2024-09-10T18:46:50.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:50 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:50.191+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:50.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:50 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:50.191+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:50.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:50 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:50.191+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:46:51.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:50 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:50.841+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:51.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:50 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:50.841+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.487911+0000 front 2024-09-10T18:46:19.487984+0000 (oldest deadline 2024-09-10T18:46:42.987826+0000) 2024-09-10T18:46:51.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:50 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:50.841+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:27.589444+0000 front 2024-09-10T18:46:27.589545+0000 (oldest deadline 2024-09-10T18:46:49.289309+0000) 2024-09-10T18:46:51.604 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:51 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:51.165+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.654743+0000 front 2024-09-10T18:46:11.654904+0000 (oldest deadline 2024-09-10T18:46:36.855477+0000) 2024-09-10T18:46:51.604 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:51 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:51.165+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.057075+0000 front 2024-09-10T18:46:19.057063+0000 (oldest deadline 2024-09-10T18:46:41.356952+0000) 2024-09-10T18:46:51.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:51 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:51.161+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:51.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:51 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:51.161+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:51.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:51 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:51.161+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:46:52.134 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:51 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:51.874+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:52.134 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:51 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:51.874+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.487911+0000 front 2024-09-10T18:46:19.487984+0000 (oldest deadline 2024-09-10T18:46:42.987826+0000) 2024-09-10T18:46:52.135 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:51 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:51.874+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:27.589444+0000 front 2024-09-10T18:46:27.589545+0000 (oldest deadline 2024-09-10T18:46:49.289309+0000) 2024-09-10T18:46:52.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:52.188+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:52.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:52.188+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:52.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:52.188+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:46:52.604 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:52.133+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.654743+0000 front 2024-09-10T18:46:11.654904+0000 (oldest deadline 2024-09-10T18:46:36.855477+0000) 2024-09-10T18:46:52.604 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5[42953]: 2024-09-10T18:46:52.133+0000 7fb1fc142640 -1 osd.5 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.057075+0000 front 2024-09-10T18:46:19.057063+0000 (oldest deadline 2024-09-10T18:46:41.356952+0000) 2024-09-10T18:46:53.197 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:52.902+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:53.197 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:52.902+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.487911+0000 front 2024-09-10T18:46:19.487984+0000 (oldest deadline 2024-09-10T18:46:42.987826+0000) 2024-09-10T18:46:53.197 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:52 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:52.902+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:27.589444+0000 front 2024-09-10T18:46:27.589545+0000 (oldest deadline 2024-09-10T18:46:49.289309+0000) 2024-09-10T18:46:53.579 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:53 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:53.195+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:53.579 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:53 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:53.195+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:53.579 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:53 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:53.195+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:46:53.579 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:53 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:53.195+0000 7f4844cef640 -1 osd.7 456 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1475 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e456) 2024-09-10T18:46:53.854 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:53 smithi204 podman[157980]: 2024-09-10 18:46:53.578307294 +0000 UTC m=+5.788930141 container died 0af635970a235881cb4b57aa2e343a97f0ba3ad6b174f8b36a10952e5751a230 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T18:46:54.165 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:53 smithi204 podman[157980]: 2024-09-10 18:46:53.970970526 +0000 UTC m=+6.181593361 container cleanup 0af635970a235881cb4b57aa2e343a97f0ba3ad6b174f8b36a10952e5751a230 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-10T18:46:54.165 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:53 smithi204 bash[157980]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5 2024-09-10T18:46:54.165 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:53 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:53.939+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:54.165 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:53 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:53.939+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.487911+0000 front 2024-09-10T18:46:19.487984+0000 (oldest deadline 2024-09-10T18:46:42.987826+0000) 2024-09-10T18:46:54.165 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:53 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:53.939+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:27.589444+0000 front 2024-09-10T18:46:27.589545+0000 (oldest deadline 2024-09-10T18:46:49.289309+0000) 2024-09-10T18:46:54.579 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:54 smithi204 podman[158009]: 2024-09-10 18:46:54.16405975 +0000 UTC m=+0.580482258 container remove 0af635970a235881cb4b57aa2e343a97f0ba3ad6b174f8b36a10952e5751a230 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-10T18:46:54.579 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:54.235+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:54.579 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:54.235+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:54.579 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:54.235+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:46:54.580 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:54.235+0000 7f4844cef640 -1 osd.7 456 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1475 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e456) 2024-09-10T18:46:54.843 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:54 smithi204 podman[158095]: 2024-09-10 18:46:54.481236163 +0000 UTC m=+0.015990147 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T18:46:54.843 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:54 smithi204 podman[158095]: 2024-09-10 18:46:54.610828434 +0000 UTC m=+0.145582454 container create 6a958fb4fee03cf4124519e039d1eecfee9eb39a3762760d274eff620ddf64d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-10T18:46:55.104 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:54 smithi204 podman[158095]: 2024-09-10 18:46:54.842686431 +0000 UTC m=+0.377440405 container init 6a958fb4fee03cf4124519e039d1eecfee9eb39a3762760d274eff620ddf64d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5-deactivate, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-10T18:46:55.104 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:54 smithi204 podman[158095]: 2024-09-10 18:46:54.846087123 +0000 UTC m=+0.380841097 container start 6a958fb4fee03cf4124519e039d1eecfee9eb39a3762760d274eff620ddf64d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5-deactivate, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-10T18:46:55.104 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:54 smithi204 podman[158095]: 2024-09-10 18:46:54.925864145 +0000 UTC m=+0.460618132 container attach 6a958fb4fee03cf4124519e039d1eecfee9eb39a3762760d274eff620ddf64d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-10T18:46:55.104 INFO:journalctl@ceph.osd.5.smithi204.stdout:Sep 10 18:46:55 smithi204 podman[158119]: 2024-09-10 18:46:55.052492147 +0000 UTC m=+0.016196984 container died 6a958fb4fee03cf4124519e039d1eecfee9eb39a3762760d274eff620ddf64d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T18:46:55.105 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:54.913+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:55.105 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:54.913+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.487911+0000 front 2024-09-10T18:46:19.487984+0000 (oldest deadline 2024-09-10T18:46:42.987826+0000) 2024-09-10T18:46:55.105 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:54 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:54.914+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:27.589444+0000 front 2024-09-10T18:46:27.589545+0000 (oldest deadline 2024-09-10T18:46:49.289309+0000) 2024-09-10T18:46:55.572 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:55.279+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:55.573 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:55.279+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:55.573 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:55.279+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:46:55.573 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:55.279+0000 7f4844cef640 -1 osd.7 456 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1475 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e456) 2024-09-10T18:46:55.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.5.service' 2024-09-10T18:46:55.636 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T18:46:55.637 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-10T18:46:55.637 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-10T18:46:55.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.6 2024-09-10T18:46:55.853 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:55 smithi204 systemd[1]: Stopping Ceph osd.6 for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:46:56.297 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:55.913+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:56.298 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:55.913+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.487911+0000 front 2024-09-10T18:46:19.487984+0000 (oldest deadline 2024-09-10T18:46:42.987826+0000) 2024-09-10T18:46:56.298 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:55.913+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:27.589444+0000 front 2024-09-10T18:46:27.589545+0000 (oldest deadline 2024-09-10T18:46:49.289309+0000) 2024-09-10T18:46:56.298 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:55.965+0000 7fc92582e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T18:46:56.298 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:55.965+0000 7fc92582e640 -1 osd.6 456 *** Got signal Terminated *** 2024-09-10T18:46:56.298 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:55 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:55.965+0000 7fc92582e640 -1 osd.6 456 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T18:46:56.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:56 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:56.296+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:56.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:56 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:56.296+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:56.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:56 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:56.296+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:46:56.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:56 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:56.296+0000 7f4844cef640 -1 osd.7 456 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1475 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e456) 2024-09-10T18:46:57.343 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:56 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:56.961+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:57.343 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:56 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:56.961+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.487911+0000 front 2024-09-10T18:46:19.487984+0000 (oldest deadline 2024-09-10T18:46:42.987826+0000) 2024-09-10T18:46:57.343 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:56 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:56.961+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:27.589444+0000 front 2024-09-10T18:46:27.589545+0000 (oldest deadline 2024-09-10T18:46:49.289309+0000) 2024-09-10T18:46:57.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:57 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:57.341+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:57.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:57 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:57.341+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:57.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:57 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:57.341+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:46:57.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:57 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:57.341+0000 7f4844cef640 -1 osd.7 456 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1475 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e456) 2024-09-10T18:46:58.258 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:57 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:57.932+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:58.259 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:57 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:57.932+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.487911+0000 front 2024-09-10T18:46:19.487984+0000 (oldest deadline 2024-09-10T18:46:42.987826+0000) 2024-09-10T18:46:58.259 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:57 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:57.932+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:27.589444+0000 front 2024-09-10T18:46:27.589545+0000 (oldest deadline 2024-09-10T18:46:49.289309+0000) 2024-09-10T18:46:58.259 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:57 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:57.932+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-09-10T18:46:35.690734+0000 front 2024-09-10T18:46:35.690696+0000 (oldest deadline 2024-09-10T18:46:57.390718+0000) 2024-09-10T18:46:58.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:58 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:58.375+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:58.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:58 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:58.375+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:58.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:58 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:58.375+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:46:58.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:58 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:58.375+0000 7f4844cef640 -1 osd.7 456 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1475 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e456) 2024-09-10T18:46:59.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:58 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:58.919+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:46:59.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:58 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:58.919+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.487911+0000 front 2024-09-10T18:46:19.487984+0000 (oldest deadline 2024-09-10T18:46:42.987826+0000) 2024-09-10T18:46:59.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:58 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:58.919+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:27.589444+0000 front 2024-09-10T18:46:27.589545+0000 (oldest deadline 2024-09-10T18:46:49.289309+0000) 2024-09-10T18:46:59.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:58 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:58.919+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-09-10T18:46:35.690734+0000 front 2024-09-10T18:46:35.690696+0000 (oldest deadline 2024-09-10T18:46:57.390718+0000) 2024-09-10T18:46:59.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:59 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:59.371+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:46:59.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:59 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:59.371+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:46:59.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:59 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:59.371+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:46:59.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:46:59 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:46:59.371+0000 7f4844cef640 -1 osd.7 456 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1475 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e456) 2024-09-10T18:47:00.354 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:59 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:59.954+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:09.586428+0000 front 2024-09-10T18:46:09.586459+0000 (oldest deadline 2024-09-10T18:46:34.286325+0000) 2024-09-10T18:47:00.354 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:59 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:59.954+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:19.487911+0000 front 2024-09-10T18:46:19.487984+0000 (oldest deadline 2024-09-10T18:46:42.987826+0000) 2024-09-10T18:47:00.354 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:59 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:59.954+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:27.589444+0000 front 2024-09-10T18:46:27.589545+0000 (oldest deadline 2024-09-10T18:46:49.289309+0000) 2024-09-10T18:47:00.354 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:46:59 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6[79002]: 2024-09-10T18:46:59.954+0000 7fc923649640 -1 osd.6 456 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-09-10T18:46:35.690734+0000 front 2024-09-10T18:46:35.690696+0000 (oldest deadline 2024-09-10T18:46:57.390718+0000) 2024-09-10T18:47:00.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:00 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:00.410+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:47:00.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:00 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:00.410+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:47:00.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:00 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:00.410+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:47:00.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:00 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:00.410+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-09-10T18:46:35.838973+0000 front 2024-09-10T18:46:35.839024+0000 (oldest deadline 2024-09-10T18:46:59.938916+0000) 2024-09-10T18:47:00.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:00 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:00.410+0000 7f4844cef640 -1 osd.7 456 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1475 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e456) 2024-09-10T18:47:01.676 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:47:01 smithi204 podman[158197]: 2024-09-10 18:47:01.400150149 +0000 UTC m=+5.544693044 container died b69a2462c2d6225f7d03e831b050b6dd43fd5ae6724fbb70a7d6710b12df4053 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T18:47:01.676 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:01 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:01.375+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:47:01.677 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:01 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:01.375+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:47:01.677 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:01 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:01.375+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:47:01.677 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:01 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:01.375+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-09-10T18:46:35.838973+0000 front 2024-09-10T18:46:35.839024+0000 (oldest deadline 2024-09-10T18:46:59.938916+0000) 2024-09-10T18:47:01.677 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:01 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:01.375+0000 7f4844cef640 -1 osd.7 456 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1475 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e456) 2024-09-10T18:47:02.103 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:47:01 smithi204 podman[158197]: 2024-09-10 18:47:01.790885918 +0000 UTC m=+5.935428761 container cleanup b69a2462c2d6225f7d03e831b050b6dd43fd5ae6724fbb70a7d6710b12df4053 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T18:47:02.104 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:47:01 smithi204 bash[158197]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6 2024-09-10T18:47:02.459 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:47:02 smithi204 podman[158213]: 2024-09-10 18:47:02.150737922 +0000 UTC m=+0.733242168 container remove b69a2462c2d6225f7d03e831b050b6dd43fd5ae6724fbb70a7d6710b12df4053 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2) 2024-09-10T18:47:02.459 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:02.365+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:47:02.460 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:02.365+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:47:02.460 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:02.365+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:47:02.460 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:02.365+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-09-10T18:46:35.838973+0000 front 2024-09-10T18:46:35.839024+0000 (oldest deadline 2024-09-10T18:46:59.938916+0000) 2024-09-10T18:47:02.460 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:02 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:02.365+0000 7f4844cef640 -1 osd.7 456 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1475 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e456) 2024-09-10T18:47:02.781 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:47:02 smithi204 podman[158302]: 2024-09-10 18:47:02.361454748 +0000 UTC m=+0.015811394 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T18:47:02.781 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:47:02 smithi204 podman[158302]: 2024-09-10 18:47:02.468786262 +0000 UTC m=+0.123142936 container create ba757ea2425867c9c82c929ded090621ea0b517d0e32409dcaf39151f7e3ab9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-09-10T18:47:02.781 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:47:02 smithi204 podman[158302]: 2024-09-10 18:47:02.704450869 +0000 UTC m=+0.358807505 container init ba757ea2425867c9c82c929ded090621ea0b517d0e32409dcaf39151f7e3ab9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True) 2024-09-10T18:47:02.781 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:47:02 smithi204 podman[158302]: 2024-09-10 18:47:02.707785272 +0000 UTC m=+0.362141905 container start ba757ea2425867c9c82c929ded090621ea0b517d0e32409dcaf39151f7e3ab9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-deactivate, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-10T18:47:02.781 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:47:02 smithi204 podman[158302]: 2024-09-10 18:47:02.779427957 +0000 UTC m=+0.433784590 container attach ba757ea2425867c9c82c929ded090621ea0b517d0e32409dcaf39151f7e3ab9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-10T18:47:03.103 INFO:journalctl@ceph.osd.6.smithi204.stdout:Sep 10 18:47:02 smithi204 podman[158302]: 2024-09-10 18:47:02.878885425 +0000 UTC m=+0.533242068 container died ba757ea2425867c9c82c929ded090621ea0b517d0e32409dcaf39151f7e3ab9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T18:47:03.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:03 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:03.349+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:47:03.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:03 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:03.349+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:47:03.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:03 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:03.349+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:47:03.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:03 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:03.349+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-09-10T18:46:35.838973+0000 front 2024-09-10T18:46:35.839024+0000 (oldest deadline 2024-09-10T18:46:59.938916+0000) 2024-09-10T18:47:03.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:03 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:03.349+0000 7f4844cef640 -1 osd.7 456 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1475 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e456) 2024-09-10T18:47:03.747 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.6.service' 2024-09-10T18:47:03.792 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T18:47:03.793 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-10T18:47:03.793 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-10T18:47:03.793 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.7 2024-09-10T18:47:04.103 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:03 smithi204 systemd[1]: Stopping Ceph osd.7 for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:47:04.360 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:04.133+0000 7f484b6dd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T18:47:04.360 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:04.133+0000 7f484b6dd640 -1 osd.7 456 *** Got signal Terminated *** 2024-09-10T18:47:04.361 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:04.133+0000 7f484b6dd640 -1 osd.7 456 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T18:47:04.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:04.390+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:47:04.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:04.391+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:47:04.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:04.391+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:47:04.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:04.391+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-09-10T18:46:35.838973+0000 front 2024-09-10T18:46:35.839024+0000 (oldest deadline 2024-09-10T18:46:59.938916+0000) 2024-09-10T18:47:04.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:04 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:04.391+0000 7f4844cef640 -1 osd.7 456 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1475 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e456) 2024-09-10T18:47:05.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:05 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:05.364+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:47:05.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:05 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:05.364+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:47:05.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:05 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:05.364+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:47:05.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:05 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:05.364+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-09-10T18:46:35.838973+0000 front 2024-09-10T18:46:35.839024+0000 (oldest deadline 2024-09-10T18:46:59.938916+0000) 2024-09-10T18:47:05.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:05 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:05.364+0000 7f4844cef640 -1 osd.7 456 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1475 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e456) 2024-09-10T18:47:06.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:06 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:06.384+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:47:06.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:06 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:06.384+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:47:06.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:06 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:06.384+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:47:06.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:06 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:06.384+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-09-10T18:46:35.838973+0000 front 2024-09-10T18:46:35.839024+0000 (oldest deadline 2024-09-10T18:46:59.938916+0000) 2024-09-10T18:47:06.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:06 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:06.384+0000 7f4844cef640 -1 osd.7 456 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1475 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e456) 2024-09-10T18:47:07.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:07.344+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:47:07.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:07.344+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:47:07.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:07.344+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:47:07.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:07.344+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-09-10T18:46:35.838973+0000 front 2024-09-10T18:46:35.839024+0000 (oldest deadline 2024-09-10T18:46:59.938916+0000) 2024-09-10T18:47:07.604 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:07 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:07.344+0000 7f4844cef640 -1 osd.7 456 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1475 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e456) 2024-09-10T18:47:08.853 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:08.376+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-09-10T18:46:11.236286+0000 front 2024-09-10T18:46:11.236304+0000 (oldest deadline 2024-09-10T18:46:37.136322+0000) 2024-09-10T18:47:08.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:08.376+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-09-10T18:46:17.136865+0000 front 2024-09-10T18:46:17.136931+0000 (oldest deadline 2024-09-10T18:46:41.836803+0000) 2024-09-10T18:47:08.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:08.376+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-09-10T18:46:23.537630+0000 front 2024-09-10T18:46:23.537603+0000 (oldest deadline 2024-09-10T18:46:49.437534+0000) 2024-09-10T18:47:08.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:08.376+0000 7f4844cef640 -1 osd.7 456 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-09-10T18:46:35.838973+0000 front 2024-09-10T18:46:35.839024+0000 (oldest deadline 2024-09-10T18:46:59.938916+0000) 2024-09-10T18:47:08.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:08 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7[54295]: 2024-09-10T18:47:08.376+0000 7f4844cef640 -1 osd.7 456 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14532.0:1475 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e456) 2024-09-10T18:47:09.958 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:09 smithi204 podman[158405]: 2024-09-10 18:47:09.651780724 +0000 UTC m=+5.629038742 container died bf71d599b4929f0bc4d93d3dd702edf1160cc6ba731c892e5c9150bc5221acf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-10T18:47:10.272 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:10 smithi204 podman[158405]: 2024-09-10 18:47:10.044929678 +0000 UTC m=+6.022187650 container cleanup bf71d599b4929f0bc4d93d3dd702edf1160cc6ba731c892e5c9150bc5221acf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-10T18:47:10.272 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:10 smithi204 bash[158405]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7 2024-09-10T18:47:10.578 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:10 smithi204 podman[158434]: 2024-09-10 18:47:10.270996592 +0000 UTC m=+0.608414972 container remove bf71d599b4929f0bc4d93d3dd702edf1160cc6ba731c892e5c9150bc5221acf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T18:47:10.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:10 smithi204 podman[158522]: 2024-09-10 18:47:10.480655485 +0000 UTC m=+0.015847800 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T18:47:10.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:10 smithi204 podman[158522]: 2024-09-10 18:47:10.580162295 +0000 UTC m=+0.115354604 container create 813ac26d26437aa0e7c22596f68394e4525cbd8f584bc2d9a79ac59e60ead71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T18:47:10.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:10 smithi204 podman[158522]: 2024-09-10 18:47:10.791360335 +0000 UTC m=+0.326552637 container init 813ac26d26437aa0e7c22596f68394e4525cbd8f584bc2d9a79ac59e60ead71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2) 2024-09-10T18:47:10.854 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:10 smithi204 podman[158522]: 2024-09-10 18:47:10.79516906 +0000 UTC m=+0.330361371 container start 813ac26d26437aa0e7c22596f68394e4525cbd8f584bc2d9a79ac59e60ead71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T18:47:11.295 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:10 smithi204 podman[158522]: 2024-09-10 18:47:10.877042124 +0000 UTC m=+0.412234427 container attach 813ac26d26437aa0e7c22596f68394e4525cbd8f584bc2d9a79ac59e60ead71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-10T18:47:11.295 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:10 smithi204 podman[158522]: 2024-09-10 18:47:10.970121719 +0000 UTC m=+0.505314031 container died 813ac26d26437aa0e7c22596f68394e4525cbd8f584bc2d9a79ac59e60ead71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-10T18:47:11.560 INFO:journalctl@ceph.osd.7.smithi204.stdout:Sep 10 18:47:11 smithi204 podman[158545]: 2024-09-10 18:47:11.536381935 +0000 UTC m=+0.560096329 container remove 813ac26d26437aa0e7c22596f68394e4525cbd8f584bc2d9a79ac59e60ead71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-10T18:47:11.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@osd.7.service' 2024-09-10T18:47:11.605 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T18:47:11.606 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-10T18:47:11.606 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-10T18:47:11.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@rgw.foo.a 2024-09-10T18:47:11.908 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Sep 10 18:47:11 smithi028 systemd[1]: Stopping Ceph rgw.foo.a for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:47:12.169 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Sep 10 18:47:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-rgw-foo-a[65718]: 2024-09-10T18:47:11.905+0000 7fd162ffd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T18:47:12.169 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Sep 10 18:47:11 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-rgw-foo-a[65718]: 2024-09-10T18:47:11.905+0000 7fd173cad7c0 -1 shutting down 2024-09-10T18:47:21.911 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:47:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a[78449]: ts=2024-09-10T18:47:21.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-09-10T18:47:21.911 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:47:21 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a[78449]: ts=2024-09-10T18:47:21.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi204.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.204:8443: connect: connection refused" 2024-09-10T18:47:22.188 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Sep 10 18:47:21 smithi028 bash[216607]: time="2024-09-10T18:47:21Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-09-10T18:47:22.188 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Sep 10 18:47:21 smithi028 podman[216607]: 2024-09-10 18:47:21.924552932 +0000 UTC m=+10.111050137 container died cb342bdaacb4a7cda1a920996d21595af0d5ace824fa182730b397aa2c89c5a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-rgw-foo-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T18:47:22.716 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Sep 10 18:47:22 smithi028 podman[216607]: 2024-09-10 18:47:22.438925449 +0000 UTC m=+10.625422651 container cleanup cb342bdaacb4a7cda1a920996d21595af0d5ace824fa182730b397aa2c89c5a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-rgw-foo-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-10T18:47:22.716 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Sep 10 18:47:22 smithi028 bash[216607]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-rgw-foo-a 2024-09-10T18:47:22.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@rgw.foo.a.service' 2024-09-10T18:47:22.922 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T18:47:22.923 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-10T18:47:22.923 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-10T18:47:22.923 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@prometheus.a 2024-09-10T18:47:23.215 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:47:22 smithi204 systemd[1]: Stopping Ceph prometheus.a for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:47:23.490 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:47:23 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:47:23.213Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-10T18:47:23.490 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:47:23 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:47:23.214Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-10T18:47:23.490 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:47:23 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:47:23.214Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-10T18:47:23.490 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:47:23 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:47:23.214Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-10T18:47:23.490 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:47:23 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:47:23.214Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-10T18:47:23.490 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:47:23 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:47:23.214Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-10T18:47:23.490 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:47:23 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:47:23.214Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-10T18:47:23.491 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:47:23 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:47:23.214Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-10T18:47:23.491 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:47:23 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:47:23.223Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-10T18:47:23.491 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:47:23 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:47:23.223Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-10T18:47:23.491 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:47:23 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:47:23.223Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-10T18:47:23.491 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:47:23 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a[68625]: ts=2024-09-10T18:47:23.223Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-10T18:47:23.491 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:47:23 smithi204 podman[158625]: 2024-09-10 18:47:23.234373629 +0000 UTC m=+0.114822064 container died 18fcf77c26a7c6edeb2ec241aeac651efd7c8e965b327fdabe9ca92764b541bd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T18:47:23.853 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:47:23 smithi204 podman[158625]: 2024-09-10 18:47:23.580027516 +0000 UTC m=+0.460475949 container cleanup 18fcf77c26a7c6edeb2ec241aeac651efd7c8e965b327fdabe9ca92764b541bd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T18:47:23.854 INFO:journalctl@ceph.prometheus.a.smithi204.stdout:Sep 10 18:47:23 smithi204 bash[158625]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-prometheus-a 2024-09-10T18:47:24.010 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@prometheus.a.service' 2024-09-10T18:47:24.045 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T18:47:24.045 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-10T18:47:24.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-10T18:47:26.071 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:47:25 smithi028 systemd[1]: Stopping Ceph alertmanager.a for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:47:26.387 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:47:26 smithi028 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a[78449]: ts=2024-09-10T18:47:26.069Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-10T18:47:26.387 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:47:26 smithi028 podman[217059]: 2024-09-10 18:47:26.110357816 +0000 UTC m=+0.143294849 container died dc1a8a4268b4437f96588c0d28faa4fa187b90695b65b5909b59da02a7ac333f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T18:47:26.753 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:47:26 smithi028 podman[217059]: 2024-09-10 18:47:26.577665385 +0000 UTC m=+0.610602418 container cleanup dc1a8a4268b4437f96588c0d28faa4fa187b90695b65b5909b59da02a7ac333f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T18:47:26.753 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:47:26 smithi028 bash[217059]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a 2024-09-10T18:47:27.091 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:47:26 smithi028 podman[217071]: 2024-09-10 18:47:26.836979521 +0000 UTC m=+0.725865363 container remove dc1a8a4268b4437f96588c0d28faa4fa187b90695b65b5909b59da02a7ac333f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T18:47:27.091 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:47:26 smithi028 podman[217071]: 2024-09-10 18:47:26.937862284 +0000 UTC m=+0.826748128 volume remove 8e8b5fa0e1bf9b688f601f74fbca3acb39777d64fe48f98b032eb068148e50a1 2024-09-10T18:47:27.091 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:47:27 smithi028 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-10T18:47:27.091 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:47:27 smithi028 systemd[1]: Stopped Ceph alertmanager.a for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:47:27.091 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Sep 10 18:47:27 smithi028 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@alertmanager.a.service: Consumed 1.957s CPU time. 2024-09-10T18:47:27.504 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:47:27 smithi028 systemd[1]: Stopping Ceph node-exporter.a for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:47:27.791 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:47:27 smithi028 podman[217177]: 2024-09-10 18:47:27.534252973 +0000 UTC m=+0.116690528 container died 5922a8d5ce88c530d86d87987c0ef8d2851e1bdf9a5a9b8914607aead6e26939 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T18:47:28.141 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:47:27 smithi028 podman[217177]: 2024-09-10 18:47:27.873037814 +0000 UTC m=+0.455475368 container cleanup 5922a8d5ce88c530d86d87987c0ef8d2851e1bdf9a5a9b8914607aead6e26939 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T18:47:28.141 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:47:27 smithi028 bash[217177]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a 2024-09-10T18:47:28.464 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:47:28 smithi028 podman[217191]: 2024-09-10 18:47:28.140652197 +0000 UTC m=+0.610134905 container remove 5922a8d5ce88c530d86d87987c0ef8d2851e1bdf9a5a9b8914607aead6e26939 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T18:47:28.465 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:47:28 smithi028 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-10T18:47:28.465 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:47:28 smithi028 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-10T18:47:28.465 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:47:28 smithi028 systemd[1]: Stopped Ceph node-exporter.a for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:47:28.465 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Sep 10 18:47:28 smithi028 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@node-exporter.a.service: Consumed 5.160s CPU time. 2024-09-10T18:47:29.010 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-10T18:47:30.628 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:47:30 smithi204 systemd[1]: Stopping Ceph iscsi.iscsi.a for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:47:31.103 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:47:30 smithi204 ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a[59429]: debug Shutdown received 2024-09-10T18:47:41.015 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:47:40 smithi204 bash[158998]: time="2024-09-10T18:47:40Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-09-10T18:47:41.015 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:47:40 smithi204 podman[158998]: 2024-09-10 18:47:40.649040358 +0000 UTC m=+10.134100566 container died 2293f4ff0e2d1a20132028e77cb28b3764c225e0f0fc8cb1ffdd21597d2fe696 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-10T18:47:41.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:47:41 smithi204 podman[158998]: 2024-09-10 18:47:41.117380407 +0000 UTC m=+10.602440616 container cleanup 2293f4ff0e2d1a20132028e77cb28b3764c225e0f0fc8cb1ffdd21597d2fe696 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T18:47:41.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:47:41 smithi204 bash[158998]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a 2024-09-10T18:47:41.853 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:47:41 smithi204 podman[159011]: 2024-09-10 18:47:41.460756981 +0000 UTC m=+0.807683226 container remove 2293f4ff0e2d1a20132028e77cb28b3764c225e0f0fc8cb1ffdd21597d2fe696 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-10T18:47:41.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:47:41 smithi204 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-10T18:47:41.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:47:41 smithi204 podman[159058]: 2024-09-10 18:47:41.69101073 +0000 UTC m=+0.119225357 container died bc8c373e717fffdd4ba80b54d9b654ccbeae9b0756bf6e05630f5647cf6e2729 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-10T18:47:42.269 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:47:42 smithi204 podman[159058]: 2024-09-10 18:47:42.029951491 +0000 UTC m=+0.458166118 container cleanup bc8c373e717fffdd4ba80b54d9b654ccbeae9b0756bf6e05630f5647cf6e2729 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-10T18:47:42.269 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:47:42 smithi204 bash[159058]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu 2024-09-10T18:47:42.269 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:47:42 smithi204 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-10T18:47:42.270 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:47:42 smithi204 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Unit process 59319 (conmon) remains running after unit stopped. 2024-09-10T18:47:42.270 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:47:42 smithi204 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Unit process 159070 (podman) remains running after unit stopped. 2024-09-10T18:47:42.270 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:47:42 smithi204 systemd[1]: Stopped Ceph iscsi.iscsi.a for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:47:42.270 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:47:42 smithi204 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Consumed 3.102s CPU time. 2024-09-10T18:47:42.551 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:47:42 smithi204 systemd[1]: Stopping Ceph grafana.a for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:47:42.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi204.stdout:Sep 10 18:47:42 smithi204 podman[159070]: 2024-09-10 18:47:42.367319594 +0000 UTC m=+0.680200442 container remove bc8c373e717fffdd4ba80b54d9b654ccbeae9b0756bf6e05630f5647cf6e2729 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-10T18:47:42.853 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:47:42 smithi204 podman[159197]: 2024-09-10 18:47:42.722924089 +0000 UTC m=+0.119752346 container died 8c25b510bf7d685a2332cc1bb0b49437ba4d22610ec9258ea7c080cc6e586e0c (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897, version=8.5, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, name=ubi8, vcs-type=git, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, distribution-scope=public, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-09-10T18:47:43.256 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:47:43 smithi204 podman[159197]: 2024-09-10 18:47:43.045373695 +0000 UTC m=+0.442201945 container cleanup 8c25b510bf7d685a2332cc1bb0b49437ba4d22610ec9258ea7c080cc6e586e0c (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a, release=239.1651231664, version=8.5, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, maintainer=Paul Cuzner , io.buildah.version=1.32.0) 2024-09-10T18:47:43.256 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:47:43 smithi204 bash[159197]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a 2024-09-10T18:47:43.604 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:47:43 smithi204 podman[159210]: 2024-09-10 18:47:43.254963206 +0000 UTC m=+0.532655749 container remove 8c25b510bf7d685a2332cc1bb0b49437ba4d22610ec9258ea7c080cc6e586e0c (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., version=8.5, io.buildah.version=1.32.0, name=ubi8, architecture=x86_64, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public) 2024-09-10T18:47:43.604 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:47:43 smithi204 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-10T18:47:43.604 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:47:43 smithi204 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-10T18:47:43.604 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:47:43 smithi204 systemd[1]: Stopped Ceph grafana.a for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:47:43.604 INFO:journalctl@ceph.grafana.a.smithi204.stdout:Sep 10 18:47:43 smithi204 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@grafana.a.service: Consumed 13.587s CPU time. 2024-09-10T18:47:44.103 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:47:43 smithi204 systemd[1]: Stopping Ceph node-exporter.b for 5eababb2-6fa0-11ef-bcea-c7b262605968... 2024-09-10T18:47:44.103 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:47:43 smithi204 podman[159321]: 2024-09-10 18:47:43.918903095 +0000 UTC m=+0.110819475 container died 4c2732cbae6f7c897b6115f82ff2d8b57c44c9239f3f32c8ec4ad9b511c9721d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T18:47:44.604 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:47:44 smithi204 podman[159321]: 2024-09-10 18:47:44.24976006 +0000 UTC m=+0.441676437 container cleanup 4c2732cbae6f7c897b6115f82ff2d8b57c44c9239f3f32c8ec4ad9b511c9721d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T18:47:44.605 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:47:44 smithi204 bash[159321]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b 2024-09-10T18:47:44.605 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:47:44 smithi204 podman[159333]: 2024-09-10 18:47:44.426132467 +0000 UTC m=+0.510963405 container remove 4c2732cbae6f7c897b6115f82ff2d8b57c44c9239f3f32c8ec4ad9b511c9721d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-5eababb2-6fa0-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T18:47:44.605 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:47:44 smithi204 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-10T18:47:45.009 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:47:44 smithi204 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-10T18:47:45.009 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:47:44 smithi204 systemd[1]: Stopped Ceph node-exporter.b for 5eababb2-6fa0-11ef-bcea-c7b262605968. 2024-09-10T18:47:45.009 INFO:journalctl@ceph.node-exporter.b.smithi204.stdout:Sep 10 18:47:44 smithi204 systemd[1]: ceph-5eababb2-6fa0-11ef-bcea-c7b262605968@node-exporter.b.service: Consumed 4.385s CPU time. 2024-09-10T18:47:45.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-10T18:47:45.386 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-10T18:47:45.414 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-10T18:47:45.416 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900206/remote/smithi028/crash 2024-09-10T18:47:45.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/crash -- . 2024-09-10T18:47:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:tar: /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-10T18:47:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:tar: Error is not recoverable: exiting now 2024-09-10T18:47:45.470 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900206/remote/smithi204/crash 2024-09-10T18:47:45.471 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/crash -- . 2024-09-10T18:47:45.510 INFO:teuthology.orchestra.run.smithi204.stderr:tar: /var/lib/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-10T18:47:45.510 INFO:teuthology.orchestra.run.smithi204.stderr:tar: Error is not recoverable: exiting now 2024-09-10T18:47:45.511 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-10T18:47:45.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-10T18:47:45.541 INFO:teuthology.orchestra.run.smithi028.stderr:grep: /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph.log: No such file or directory 2024-09-10T18:47:45.543 INFO:tasks.cephadm:Compressing logs... 2024-09-10T18:47:45.543 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-10T18:47:45.586 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-10T18:47:45.621 INFO:teuthology.orchestra.run.smithi204.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/tcmu-runner.log: No such file or directory 2024-09-10T18:47:45.622 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-10T18:47:45.622 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/objectstore_tool.2.log 2024-09-10T18:47:45.622 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.7.log 2024-09-10T18:47:45.622 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/tcmu-runner.log: /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/objectstore_tool.2.log: 62.9% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/tcmu-runner.log.gz 2024-09-10T18:47:45.622 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-10T18:47:45.622 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.4.log 2024-09-10T18:47:45.622 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.6.log 2024-09-10T18:47:45.622 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.5.log 2024-09-10T18:47:45.622 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-mgr.x.log 2024-09-10T18:47:45.622 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-mon.b.log 2024-09-10T18:47:45.622 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-10T18:47:45.629 INFO:teuthology.orchestra.run.smithi028.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.0.log 2024-09-10T18:47:45.629 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T18:47:45.630 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.1.log 2024-09-10T18:47:45.630 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-10T18:47:45.631 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.0.log: /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-mon.c.log 2024-09-10T18:47:45.631 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-mon.a.log 2024-09-10T18:47:45.631 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.2.log 2024-09-10T18:47:45.631 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-mgr.y.log 2024-09-10T18:47:45.632 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log 2024-09-10T18:47:45.632 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.3.log 2024-09-10T18:47:45.632 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-10T18:47:45.632 INFO:teuthology.orchestra.run.smithi028.stderr: 58.6% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-09-10T18:47:45.672 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-mgr.x.log: /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 91.0% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-10T18:47:45.681 INFO:teuthology.orchestra.run.smithi204.stderr: 92.1% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-mgr.x.log.gz 2024-09-10T18:47:45.721 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-10T18:47:45.775 INFO:teuthology.orchestra.run.smithi204.stderr: 94.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-10T18:47:45.808 INFO:teuthology.orchestra.run.smithi204.stderr: 95.1% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/objectstore_tool.2.log.gz 2024-09-10T18:47:46.250 INFO:teuthology.orchestra.run.smithi028.stderr: 91.3% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-10T18:47:46.529 INFO:teuthology.orchestra.run.smithi204.stderr: 90.7% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-mon.b.log.gz 2024-09-10T18:47:48.774 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-mgr.y.log.gz 2024-09-10T18:47:54.875 INFO:teuthology.orchestra.run.smithi028.stderr: 90.7% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-mon.c.log.gz 2024-09-10T18:48:01.558 INFO:teuthology.orchestra.run.smithi028.stderr: 90.2% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-mon.a.log.gz 2024-09-10T18:49:11.999 INFO:teuthology.orchestra.run.smithi204.stderr: 93.4% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.6.log.gz 2024-09-10T18:49:31.725 INFO:teuthology.orchestra.run.smithi028.stderr: 93.5% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.3.log.gz 2024-09-10T18:50:19.202 INFO:teuthology.orchestra.run.smithi204.stderr: 93.5% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.5.log.gz 2024-09-10T18:51:39.738 INFO:teuthology.orchestra.run.smithi204.stderr: 93.4% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.4.log.gz 2024-09-10T18:51:51.538 INFO:teuthology.orchestra.run.smithi204.stderr: 93.6% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.7.log.gz 2024-09-10T18:51:51.540 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-10T18:51:51.540 INFO:teuthology.orchestra.run.smithi204.stderr:real 4m5.939s 2024-09-10T18:51:51.540 INFO:teuthology.orchestra.run.smithi204.stderr:user 4m19.378s 2024-09-10T18:51:51.540 INFO:teuthology.orchestra.run.smithi204.stderr:sys 0m11.602s 2024-09-10T18:52:25.709 INFO:teuthology.orchestra.run.smithi028.stderr: 93.6% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.2.log.gz 2024-09-10T18:53:52.020 INFO:teuthology.orchestra.run.smithi028.stderr: 93.4% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.1.log.gz 2024-09-10T18:53:53.428 INFO:teuthology.orchestra.run.smithi028.stderr: 93.4% -- replaced with /var/log/ceph/5eababb2-6fa0-11ef-bcea-c7b262605968/ceph-osd.0.log.gz 2024-09-10T18:53:53.431 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-10T18:53:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:real 6m7.832s 2024-09-10T18:53:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:user 5m31.154s 2024-09-10T18:53:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m13.559s 2024-09-10T18:53:53.432 INFO:tasks.cephadm:Archiving logs... 2024-09-10T18:53:53.432 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900206/remote/smithi028/log 2024-09-10T18:53:53.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-10T18:54:40.355 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900206/remote/smithi204/log 2024-09-10T18:54:40.356 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-10T18:55:19.447 INFO:tasks.cephadm:Removing cluster... 2024-09-10T18:55:19.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 --force 2024-09-10T18:55:20.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5eababb2-6fa0-11ef-bcea-c7b262605968 --force 2024-09-10T18:55:21.166 INFO:tasks.cephadm:Removing cephadm ... 2024-09-10T18:55:21.167 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-10T18:55:21.184 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-10T18:55:21.200 INFO:tasks.cephadm:Teardown complete 2024-09-10T18:55:21.200 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-10T18:55:21.209 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-10T18:55:21.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T18:55:21.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T18:55:21.345 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-10T18:55:21.354 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-10T18:55:21.355 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T18:55:21.357 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T18:55:21.370 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-09-10T18:55:21.372 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-09-10T18:55:21.395 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T18:55:21.395 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-09-10T18:55:21.395 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv01.front.sepia.ceph.com 3 6 377 3 +45us[ +32us] +/- 46ms 2024-09-10T18:55:21.395 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 3 +1096us[+1096us] +/- 75ms 2024-09-10T18:55:21.395 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 3 -1035us[-1035us] +/- 74ms 2024-09-10T18:55:21.395 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:55:21.397 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T18:55:21.397 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-09-10T18:55:21.397 INFO:teuthology.orchestra.run.smithi204.stdout:^* hv01.front.sepia.ceph.com 3 6 377 4 +23us[ +36us] +/- 46ms 2024-09-10T18:55:21.397 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 2 +1071us[+1071us] +/- 75ms 2024-09-10T18:55:21.397 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 -1094us[-1081us] +/- 74ms 2024-09-10T18:55:21.397 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:55:21.398 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-10T18:55:21.407 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-10T18:55:21.407 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-10T18:55:21.415 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-10T18:55:21.524 DEBUG:teuthology.orchestra.run.smithi204:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-10T18:55:21.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T18:55:21.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:21.617 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T18:55:21.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:21.644 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-10T18:55:21.658 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-10T18:55:21.675 INFO:teuthology.task.internal:Duration was 2887.625164 seconds 2024-09-10T18:55:21.675 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-10T18:55:21.691 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-10T18:55:21.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T18:55:21.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T18:55:21.749 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T18:55:21.766 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T18:55:22.061 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-10T18:55:22.062 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-09-10T18:55:22.062 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T18:55:22.083 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-09-10T18:55:22.083 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T18:55:22.118 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-10T18:55:22.119 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T18:55:22.126 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T18:55:22.188 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-10T18:55:22.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T18:55:22.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T18:55:23.802 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-10T18:55:23.817 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-10T18:55:23.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T18:55:23.858 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T18:55:23.979 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-10T18:55:23.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T18:55:23.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T18:55:24.014 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-09-10T18:55:24.022 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-09-10T18:55:24.053 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T18:55:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:24.092 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T18:55:24.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:24.108 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-10T18:55:24.119 INFO:teuthology.task.internal:Transferring archived files... 2024-09-10T18:55:24.120 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900206/remote/smithi028 2024-09-10T18:55:24.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T18:55:24.210 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900206/remote/smithi204 2024-09-10T18:55:24.210 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T18:55:24.250 INFO:teuthology.task.internal:Removing archive directory... 2024-09-10T18:55:24.250 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T18:55:24.252 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T18:55:24.299 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-10T18:55:24.310 INFO:teuthology.task.internal:Not uploading archives. 2024-09-10T18:55:24.310 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-10T18:55:24.320 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-10T18:55:24.320 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T18:55:24.322 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T18:55:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 18:55 /home/ubuntu/cephtest 2024-09-10T18:55:24.355 INFO:teuthology.orchestra.run.smithi204.stdout: 265336 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 18:55 /home/ubuntu/cephtest 2024-09-10T18:55:24.357 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-10T18:55:24.367 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-10T18:55:24.389 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} duration: 2887.6251635551453 flavor: default owner: scheduled_gabrioux@teuthology success: true 2024-09-10T18:55:24.389 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T18:55:24.455 INFO:teuthology.run:pass